Why is my webhook sync not progressing?

A sync that runs but never advances is fetching the same page again and again: either the cursor it saves is not the one the last page returned, or one item on that page fails every run and stops the loop before the save. Check it directly by recording the cursor before and after each run. If items were fetched and the cursor did not move, the sync is stalled, and the page at that cursor holds the item or the field to fix.

A progress check around each run

// Run one pass of a cursor-based sync and fail loudly when it did not advance.
// runPass(cursor) -> { fetched, resumeFrom, done }: resumeFrom is where the next pass starts,
// done is true when the pass reached the end of the data
// loadCursor() / saveCursor(cursor) persist the checkpoint
export async function runPassWithProgressCheck ({ runPass, loadCursor, saveCursor }) {
  const before = await loadCursor()
  const { fetched, resumeFrom, done } = await runPass(before)
  if (fetched > 0 && !done && resumeFrom === before) {
    throw new Error(`sync stalled: ${fetched} item(s) fetched, cursor still ${before}`)
  }
  if (resumeFrom !== before) await saveCursor(resumeFrom)
  return { before, after: resumeFrom, fetched }
}

The three usual causes

1. The wrong cursor is saved. The run saves the cursor it requested with, not the next cursor the response returned, so every run starts where the last one did.
2. One item fails every time. The same item throws on every run, the loop stops before saving, and the next run fetches that page again. The error log repeats one id.
3. A timestamp cursor over identical timestamps. A cursor of updated_at with >= fetches the same page forever once more than a page of rows share one timestamp; with > it skips the rest of them. Use the timestamp and the id together as the cursor.

If the job itself is stuck

A sync that shows as running but does nothing may not be running at all: see How do I resume a webhook sync from a cursor? for a loop that saves its cursor only after a whole page is handled.

When the sync only exists to catch up on webhooks

If you page through the sender's API because your endpoint missed webhooks, and a gateway stored those webhooks when they arrived, you can replay the stored events instead.

Related

• Why is my webhook data missing fields?
• How do I resume a webhook sync from a cursor?