How do I resume a webhook sync from a cursor?

Save the cursor only after every item on a page has been handled, key every write on the item's id so handling it twice is harmless, and on restart fetch from the saved cursor. A crash then repeats at most one page and never skips one. Saving the cursor before the page is handled is the bug: a crash after the save loses the rest of that page for good.

The loop

// Resume a paginated sync from its last saved cursor.
// fetchPage(cursor) -> { items, nextCursor }  (nextCursor is null on the last page)
// handle(item) must be idempotent: key writes on item.id.
// loadCursor() / saveCursor(cursor) persist the checkpoint (a row, a file, a key).
export async function syncFromCursor ({ fetchPage, handle, loadCursor, saveCursor }) {
  let cursor = await loadCursor() // null on the first run
  for (;;) {
    const { items, nextCursor } = await fetchPage(cursor)
    for (const item of items) await handle(item)
    if (nextCursor == null) return cursor
    await saveCursor(nextCursor) // only after the whole page is handled
    cursor = nextCursor
  }
}

Why not save the cursor first

Saving first means a crash between the save and the last item skips those items permanently: the next run starts after them. Saving after means a crash repeats part of one page, and idempotent writes absorb the repeat.

When the sync only exists to catch up on webhooks

If you page through the vendor's API because your endpoint was down and missed webhooks, a gateway that stores each webhook's raw payload removes the job: you replay the stored events through your current code instead of paging the vendor again. History from before your webhook subscription still comes from the API — see How do I paginate through a webhook backfill?

Related

• How do I paginate through a webhook backfill?