Paginate a partner collection

The loop

let cursor = null;
do {
  const url = `.../applications?limit=100${cursor ? `&cursor=${cursor}` : ''}`;
  const page = await get(url);
  process(page.items);
  cursor = page.next_cursor;
} while (cursor);

Treat next_cursor as opaque and stop when it is null — see the pagination reference.

Page at the right size

Each page is one request against your bucket, so page at a reasonable limit (tens to a hundred) rather than one item at a time. Cursor paging is stable under concurrent inserts, so you can iterate a large, live collection without skipping or repeating rows.

Back off if throttled

If a page returns a 429, apply your back-off and resume from the same cursor — the cursor is stable, so you lose no progress. Persist the cursor if the job may be interrupted.