How do I paginate through a webhook backfill?
A webhook only delivers what happens after you subscribe, so history has to come from the vendor's list API. Page through it with the cursor the API returns, skip anything created at or after the moment your webhook subscription started, and upsert every item by its id. The webhooks cover everything from that moment on, and the id makes the overlap harmless.
The loop
// Backfill history from a paginated list API, up to the moment live webhooks took over.
// fetchPage(cursor) -> { items: [{ id, created }], nextCursor } (nextCursor is null on the last page)
// upsert(item) writes keyed on item.id, so an item seen twice leaves one row.
// until: when the webhook subscription started, in the same units as item.created.
export async function backfill ({ fetchPage, upsert, until }) {
let cursor = null
let written = 0
do {
const { items, nextCursor } = await fetchPage(cursor)
for (const item of items) {
if (item.created >= until) continue // live webhooks own this one
await upsert(item)
written++
}
cursor = nextCursor
} while (cursor != null)
return written
}If it stops halfway
Save the cursor after each page and restart from it — How do I resume a webhook sync from a cursor? has that loop.
After the backfill
Backfill once. From then on a missed delivery is a replay problem, not a pagination problem: a gateway that stores each raw webhook lets you replay it through your current code without paging the vendor again.
Need somewhere for the next project's webhooks to land? Slashbin's Free plan takes an account, not a card.
Start Free
(opens in a new tab)