How do I handle vendor API rate limits when backfilling?
Treat a 429 as an instruction, not an error: wait as long as the response's Retry-After header says, or back off exponentially with jitter when it says nothing, and then retry the same request. Send one request at a time per API key unless the sender documents a higher limit, and save your cursor after every page, so being throttled slows the backfill down instead of restarting it.
A request wrapper that waits its turn
// Fetch with rate-limit handling: honour Retry-After on 429, back off with jitter on 429 and 5xx.
// request() -> a fetch Response; sleep(ms) -> Promise
// setTimeout fires at once for delays over 2^31 - 1 ms (about 24.8 days), so long waits are slept in chunks.
const MAX_TIMER_MS = 2 ** 31 - 1
async function sleepFor (ms) {
for (let left = ms; left > 0; left -= MAX_TIMER_MS) await new Promise((r) => setTimeout(r, Math.min(left, MAX_TIMER_MS)))
}
export async function fetchWithRateLimit (request, { sleep = sleepFor, maxAttempts = 6, baseMs = 1000, capMs = 60000, now = Date.now } = {}) {
for (let attempt = 1; ; attempt++) {
const res = await request()
if (res.status !== 429 && res.status < 500) return res
if (attempt >= maxAttempts) throw new Error(`gave up after ${attempt} attempts: HTTP ${res.status}`)
const header = res.headers.get('retry-after')
let waitMs
if (header != null && /^\d+$/.test(header.trim())) waitMs = Number(header) * 1000
else if (header != null && !Number.isNaN(Date.parse(header))) waitMs = Math.max(0, Date.parse(header) - now())
else waitMs = Math.min(capMs, baseMs * 2 ** (attempt - 1)) * (0.5 + Math.random() / 2)
await sleep(waitMs)
}
}Why jitter
Without it, every worker that was throttled at the same moment retries at the same moment and is throttled again. A random share of the wait spreads them out.
Keep the cursor
A throttled request is retried, not the whole backfill. Wrap each page fetch in the function above and save the cursor after each page, as in How do I paginate through a webhook backfill?
After the backfill
Backfill history once. From then on the sender's webhooks carry each change as it happens, and a delivery your endpoint missed is a replay of the stored payload rather than another trip through the rate limit.
Need somewhere for the next project's webhooks to land? Slashbin's Free plan takes an account, not a card.
Start Free
(opens in a new tab)