How do I resume a webhook job after a crash?

Give every job a lease instead of a running flag. A worker claims a job by writing its own id and an expiry time, renews the expiry while it works, saves a checkpoint as it goes, and marks the job done at the end. A job left behind by a crashed worker is then a job whose lease has expired, and the next worker claims it and resumes from the saved checkpoint. A plain running flag is what leaves a crashed job stuck: nothing ever sets it back.

Claiming a job

// Claim the next job whose lease is empty or expired; a crashed worker's job frees itself.
// listOpenJobs() -> [{ id, leaseUntil, checkpoint }] for jobs not yet done
// compareAndSet(id, expectedLeaseUntil, patch) -> true only if leaseUntil still equals expectedLeaseUntil
export async function claimJob ({ listOpenJobs, compareAndSet, workerId, now, leaseMs }) {
  for (const job of await listOpenJobs()) {
    if (job.leaseUntil != null && job.leaseUntil > now) continue // a live worker holds it
    const lease = { leaseOwner: workerId, leaseUntil: now + leaseMs }
    if (await compareAndSet(job.id, job.leaseUntil, lease)) return { ...job, ...lease } // resume from job.checkpoint
  }
  return null
}

Why the claim is a compare-and-set

Two workers can read the same expired lease at the same moment. The claim succeeds only for the one whose write still finds the lease it read; the other moves to the next job. In SQL that is an UPDATE … WHERE id = $1 AND lease_until IS NOT DISTINCT FROM $2 that reports one row changed.

Renew well inside the lease

Renew the lease every third of its length, and checkpoint after each unit of work — a page, a batch, an item. A worker that stops renewing has crashed or hung, and either way the job should move.

Why deleting the jobs table fixes it

Deleting the table clears every stuck flag, which is why it works — and it also throws away every checkpoint, so each job starts over. With leases there is nothing to delete. For the cursor inside a job, see How do I resume a webhook sync from a cursor?

Related

• Do I need a queue for webhooks?
• How do I resume a webhook sync from a cursor?