What do I do with my webhook dead letter queue?
Treat every entry as a bug report with the evidence attached. Group the entries by error, fix the cause of the largest group, retry that group through the fixed code, and delete an entry only after its retry succeeds. Then alert on the age of the oldest entry, not on the count: a dead-letter queue nobody is told about fills up silently, and an event that sits in it for months is data you have stopped looking for.
Turn the queue into a work list
// Turn a dead-letter queue into a work list: group by error, largest first, and flag stale entries.
// entries: [{ id, error, failedAt }]; now and maxAgeMs in milliseconds
export function triageDeadLetters (entries, now, maxAgeMs) {
const groups = new Map()
let oldest = null
for (const e of entries) {
const g = groups.get(e.error) ?? { error: e.error, ids: [], oldest: e.failedAt }
g.ids.push(e.id)
g.oldest = Math.min(g.oldest, e.failedAt)
groups.set(e.error, g)
if (oldest == null || e.failedAt < oldest) oldest = e.failedAt
}
const work = [...groups.values()].sort((a, b) => b.ids.length - a.ids.length)
return { work, alert: oldest != null && now - oldest > maxAgeMs }
}Why not retry everything
An entry is there because retrying did not fix it. Either it failed every retry, or, like a validation error, it went straight to the queue because another attempt would fail the same way. A validation error or a mapping bug fails again, identically, until the cause is fixed; retrying without a fix only moves the timestamp.
Why alert on age
A queue that holds a handful of entries for months looks small on any dashboard that counts. The oldest entry’s age is the number that says someone should have looked already.
In Slashbin
An event that fails Golden Model validation lands in the dead-letter queue with the validation error attached and its original payload kept byte for byte, so you fix the model, republish, and retry the queue. See Retry vs. Replay for the difference. See Golden Model.
Need somewhere for the next project’s webhooks to land? Slashbin’s Free plan takes an account, not a card.
Start Free
(opens in a new tab)Retry vs. Replay
Retry and Replay are different features. Retry re-runs a dead-lettered entry (one that failed during ingest or transform, before delivery) after you fix its cause: in the Console, open Dead Letter Queue and use Retry or Retry All; in the CLI, run slashbin dlq retry <id> or slashbin dlq retry-all. Replay re-drives stored history for a data stream, for example after you change a model. Replay is billed and delivers to every destination of the data stream. To recover dead letters, retry them; use Replay only to re-send stored history.