n8n Split in Batches Node: Processing Large Datasets Without Hitting Rate Limits

Most n8n workflows work fine until you give them 10,000 records.

Then you hit API rate limits. Or the HTTP Request node times out. Or you run out of memory. Or a downstream database starts rejecting connections because you opened too many at once.

The Split in Batches node is the solution. Here is how it works and how to use it correctly in production.

What Split in Batches Does

Split in Batches divides an array of items into smaller groups (batches) and processes them sequentially. Instead of sending 5,000 API calls at once, you send 50 at a time, with each batch completing before the next begins.

The node has one key parameter: Batch Size — how many items to process per iteration.

After each batch executes, the workflow loops back to Split in Batches for the next batch. When all batches are processed, the workflow continues to the next node in the main flow.

Basic Setup

The simplest Split in Batches pattern:

1. Input: A node that returns many items (database query, list fetch, etc.)
2. Split in Batches: Set batch size (50-100 is usually safe)
3. Processing nodes: HTTP Request, database write, etc. — these operate on one batch at a time
4. Loop back to Split in Batches via a connection from the last processing node
5. Continue to the next part of your workflow once all batches complete

The loop is explicit: you draw a connection from the last node in your processing chain back to Split in Batches. n8n handles the iteration automatically.

Choosing the Right Batch Size

Batch size depends on two factors: the rate limit of your downstream API and the memory/timeout constraints of your n8n instance.

For external APIs with rate limits:

Most APIs express rate limits as “N requests per second” or “N requests per minute.” Work backwards:

• Salesforce: 100 concurrent API calls → batch size 50 with no sleep
• HubSpot: 100 requests per 10 seconds → batch size 50, add 2-second sleep between batches
• Mailchimp: 10 requests per second → batch size 5-10

Add a Wait node after your HTTP Request to introduce deliberate pauses between batches when needed.

For database operations:

• PostgreSQL: batch size 100-500 is usually safe for inserts
• Single-record upserts via HTTP API: batch size 25-50
• Bulk operations (if the API supports them): batch size 500-1000

For n8n memory:

If you are processing items with large payloads (base64-encoded files, long text), reduce the batch size. A batch size of 10 might be appropriate for 5MB records.

Handling Errors Within a Batch

By default, if one item in a batch fails, the entire batch fails. You have two options:

Option 1: Continue on Fail. Enable “Continue On Fail” on the processing node. Failed items pass through with an error flag instead of stopping the batch. Add a downstream IF node to separate errors from successes.

Option 2: Process one at a time. Set batch size to 1. This is slower but gives you full control — you can handle each error individually before proceeding.

For critical data pipelines, batch size 1 with full error handling is worth the performance cost.

Tracking Progress

Split in Batches provides metadata you can use for logging or conditional logic:

• $node["Split in Batches"].context.currentRunIndex — which batch is currently executing (0-indexed)
• $node["Split in Batches"].context.noItemsLeft — boolean, true when the last batch has been processed

Use these to log progress, send intermediate notifications, or conditionally skip processing based on batch number.

Example — log every 10 batches:

const batchIndex = $node["Split in Batches"].context.currentRunIndex;
if (batchIndex % 10 === 0) {
  // send progress notification
}

Combining with Aggregation

After all batches complete, the next node in the main flow receives only the last batch’s output by default.

If you need to aggregate results from all batches (for example, to write a summary report), use an Aggregate node or a Code node to collect results as the loop runs. One approach: write partial results to a database or Redis key during each batch, then read the full results after the loop completes.

Common Mistakes

Not closing the loop. The most common mistake is forgetting to draw the connection from the last processing node back to Split in Batches. Without it, the workflow processes only the first batch and exits.

Setting batch size too high for the first run. Start conservative (25-50) and increase after verifying that your downstream targets handle it. It is much easier to increase batch size than to recover from a mass rate-limit ban.

Mixing Split in Batches with parallel branches. Split in Batches processes sequentially. If you have a parallel branch inside the loop, n8n may behave unexpectedly. Keep the loop body linear.

Ignoring the last batch. If your total item count is not divisible by your batch size, the last batch will be smaller. Make sure your processing logic handles variable-size batches correctly.

A Practical Example: Syncing 50,000 Contacts

You fetch 50,000 contact records from a database and need to upsert them into HubSpot (rate limit: 100 requests/10 seconds).

1. Database node → returns 50,000 items
2. Split in Batches (batch size: 80)
3. HTTP Request → HubSpot Upsert Contact
4. Wait node (1 second delay)
5. Loop back to Split in Batches
6. After loop: Code node to summarize results

At 80 contacts per batch with a 1-second wait, you process 50,000 contacts in about 10-11 minutes — well within HubSpot’s limits and with zero manual intervention.

Split in Batches turns “this will crash” into “this will just take a few minutes.” That is usually the right trade.