What do I do with my webhook dead letter queue?
Treat every entry as a bug report with the evidence attached. Group the entries by error, fix the cause of the largest group, replay that group through the fixed code, and delete an entry only after its replay succeeds. Then alert on the age of the oldest entry, not on the count: a dead-letter queue nobody is told about fills up silently, and an event that sits in it for months is data you have stopped looking for.
Turn the queue into a work list
// Turn a dead-letter queue into a work list: group by error, largest first, and flag stale entries.
// entries: [{ id, error, failedAt }]; now and maxAgeMs in milliseconds
export function triageDeadLetters (entries, now, maxAgeMs) {
const groups = new Map()
let oldest = null
for (const e of entries) {
const g = groups.get(e.error) ?? { error: e.error, ids: [], oldest: e.failedAt }
g.ids.push(e.id)
g.oldest = Math.min(g.oldest, e.failedAt)
groups.set(e.error, g)
if (oldest == null || e.failedAt < oldest) oldest = e.failedAt
}
const work = [...groups.values()].sort((a, b) => b.ids.length - a.ids.length)
return { work, alert: oldest != null && now - oldest > maxAgeMs }
}Why not retry everything
An entry is there because retrying did not fix it. Either it failed every retry, or, like a validation error, it went straight to the queue because another attempt would fail the same way. A validation error or a mapping bug fails again, identically, until the cause is fixed; retrying without a fix only moves the timestamp.
Why alert on age
A queue that holds a handful of entries for months looks small on any dashboard that counts. The oldest entry's age is the number that says someone should have looked already.
In Slashbin
An event that fails Golden Model validation lands in the dead-letter queue with the validation error attached and its original payload kept byte for byte, so you fix the model, republish, and retry the queue. See Golden Model.
Need somewhere for the next project's webhooks to land? Slashbin's Free plan takes an account, not a card.
Start Free
(opens in a new tab)