Skip to Content
Public Beta: Direct engineering support available. Join Discord →

How do I handle vendor API rate limits when backfilling?

Treat a 429 as an instruction, not an error: wait as long as the response’s Retry-After header says, or back off exponentially with jitter when it says nothing, and then retry the same request. Send one request at a time per API key unless the sender documents a higher limit, and save your cursor after every page, so being throttled slows the backfill down instead of restarting it.

A request wrapper that waits its turn

// Fetch with rate-limit handling: honour Retry-After on 429, back off with jitter on 429 and 5xx. // request() -> a fetch Response; sleep(ms) -> Promise // setTimeout fires at once for delays over 2^31 - 1 ms (about 24.8 days), so long waits are slept in chunks. const MAX_TIMER_MS = 2 ** 31 - 1 async function sleepFor (ms) { for (let left = ms; left > 0; left -= MAX_TIMER_MS) await new Promise((r) => setTimeout(r, Math.min(left, MAX_TIMER_MS))) } export async function fetchWithRateLimit (request, { sleep = sleepFor, maxAttempts = 6, baseMs = 1000, capMs = 60000, now = Date.now } = {}) { for (let attempt = 1; ; attempt++) { const res = await request() if (res.status !== 429 && res.status < 500) return res if (attempt >= maxAttempts) throw new Error(`gave up after ${attempt} attempts: HTTP ${res.status}`) const header = res.headers.get('retry-after') let waitMs if (header != null && /^\d+$/.test(header.trim())) waitMs = Number(header) * 1000 else if (header != null && !Number.isNaN(Date.parse(header))) waitMs = Math.max(0, Date.parse(header) - now()) else waitMs = Math.min(capMs, baseMs * 2 ** (attempt - 1)) * (0.5 + Math.random() / 2) await sleep(waitMs) } }

Why jitter

Without it, every worker that was throttled at the same moment retries at the same moment and is throttled again. A random share of the wait spreads them out.

Keep the cursor

A throttled request is retried, not the whole backfill. Wrap each page fetch in the function above and save the cursor after each page, as in How do I paginate through a webhook backfill?

After the backfill

Backfill history once. From then on the sender’s webhooks carry each change as it happens, and a delivery your endpoint missed is a replay of the stored payload rather than another trip through the rate limit.

Need somewhere for the next project’s webhooks to land? Slashbin’s Free plan takes an account, not a card.

Start Free

(opens in a new tab)