The page
I built a pipeline that turns written articles into audio, using an AI text-to-speech service behind AWS Lambda. On a normal day it was quiet. When a lot of articles published at once, audio started arriving late, and some never arrived.
Triage
# throttles and errors on the function, last 3 hours
aws cloudwatch get-metric-statistics --namespace AWS/Lambda \
--metric-name Throttles --dimensions Name=FunctionName,Value=tts-render \
--start-time "$(date -u -d '-3 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 300 --statistics Sum --output table
# timeouts in the logs
aws logs tail /aws/lambda/tts-render --since 3h --filter-pattern "Task timed out"
# is there a backlog, and how old is it?
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
--attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage
# concurrency settings on the function
aws lambda get-function-concurrency --function-name tts-render
Root cause
Lambda scaled out quickly when a burst arrived, and the text-to-speech API behind it had its own rate limit. Too many concurrent calls got rate limited, retried, timed out, and some messages ran out of retries.
The fix
Capped the function's reserved concurrency to what the downstream API could handle, so the queue absorbs bursts instead of the API. Retries got backoff, and failed messages go to a dead-letter queue with an alarm.
aws lambda put-function-concurrency --function-name tts-render \
--reserved-concurrent-executions 20
What I learnt
Serverless scales to whatever comes in. The services behind it usually don't. A queue with a concurrency cap matches the two.
What I do now
- Set Lambda concurrency from the downstream limit, not from the incoming load.
- Alert on queue age, which tells you about user-visible delay.
- Dead-letter queue and an alarm on it for anything that can fail.
stage 2 builds pipelines like this → see the engagements