GTM Galaxy

Article

Stopping the 429 Storm: Building a Batched API Dispatcher for High-Volume CRM Ingestion

Claudine Baumbach / / Pacific

Most GTM tech stacks rely on a brittle point-to-point connection: a lead fills out a form, a webhook fires, and an automation platform like n8n immediately pushes that data into Salesforce or HubSpot. This works until it doesn't. When you launch a viral campaign or a product update and 5,000 events hit your endpoint in 60 seconds, your CRM starts throwing 429 Too Many Requests errors.

Standard automation tools respond by retrying the failed calls, which only compounds the congestion. If you are on Salesforce, you might also watch your daily API quota vanish by mid-morning because every single update counts as a separate billable event. This is the 429 storm. The solution isn't adding more retry logic; it’s decoupling ingestion from execution using a stateful transactional outbox.

The Unit Economics of API Calls

Direct 1:1 integrations are expensive and fragile. Sending 200 individual POST requests to create 200 contacts consumes 200 API calls and hits concurrency guardrails.

CRM vendors provide batch endpoints specifically to handle this volume, but they require you to aggregate the data yourself:

  • Salesforce sObject Collections: You can update or create up to 200 records in a single request. Salesforce counts this entire batch as exactly one API call toward your daily limit.
  • HubSpot Batch API: Most CRM object endpoints allow up to 100 records per request. Buffering these prevents the 423 Locked errors that occur when multiple concurrent processes try to sync the same record simultaneously.

The Architecture: A Stateful Outbox

Instead of letting your webhook hit the CRM directly, hit a PostgreSQL table. This table acts as a buffer, absorbing high-frequency spikes and allowing you to dispatch records at a controlled cadence.

A minimal crm_outbox schema looks like this:

CREATE TABLE crm_outbox (
    id SERIAL PRIMARY KEY,
    object_type VARCHAR(50), -- 'lead', 'contact', 'deal'
    payload JSONB,
    status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'failed'
    attempts INTEGER DEFAULT 0,
    error_message TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

When a webhook arrives, your first automation does one thing: write a row to this table and return a 200 OK. This moves the pressure of the traffic spike from the CRM’s API to your own database, which handles high-concurrency writes with ease.

Building the Dispatcher Logic

Next, you need a "Dispatcher" workflow (running on a cron, e.g., every 30 seconds) to pull pending records and format them for the CRM.

To avoid race conditions where two dispatcher instances pick up the same records, use a FOR UPDATE SKIP LOCKED query. This ensures that even if you have multiple instances of n8n running, they won't duplicate efforts:

WITH batch AS (
  SELECT id FROM crm_outbox
  WHERE status = 'pending' 
  OR (status = 'failed' AND attempts < 3)
  LIMIT 100
  FOR UPDATE SKIP LOCKED
)
UPDATE crm_outbox
SET status = 'processing'
FROM batch
WHERE crm_outbox.id = batch.id
RETURNING crm_outbox.id, crm_outbox.payload;

Handling Partial Success and 'allOrNone'

Batching introduces a complication: what happens if 98 records are valid but 2 have bad data?

In Salesforce, set the allOrNone parameter to false. This prevents a single validation error from rolling back the entire batch. Salesforce will return an array of result objects, which you must iterate through to reconcile your outbox.

{
  "allOrNone": false,
  "records": [
    { "attributes": {"type": "Lead"}, "Email": "valid@test.com", "LastName": "Smith" },
    { "attributes": {"type": "Lead"}, "Email": "invalid-email", "LastName": "Jones" }
  ]
}

HubSpot handles this similarly, often returning a 207 Multi-Status. Your dispatcher must inspect the response body. For every successful record, delete the row or mark it synced. For failures, log the error message back to the crm_outbox and increment the attempts counter. This gives you a deterministic audit trail of why specific leads are stuck.

Trade-offs and Maintenance

Batching is not a silver bullet. The primary trade-off is latency. If your dispatcher runs every 60 seconds, a lead might wait a minute before appearing in the CRM. If you have high-priority "speed to lead" workflows—like an immediate Slack alert for a demo request—use a hybrid approach: push high-priority events immediately and route bulk telemetry or enrichment data through the outbox.

There is also a maintenance tax. At high volumes (e.g., 500k events/day), your outbox table will accumulate bloat. You need a simple cleanup job to prune records that are older than 7 days and marked as synced:

DELETE FROM crm_outbox WHERE status = 'synced' AND created_at < NOW() - INTERVAL '7 days';

When to Buy: The Reverse ETL Question

Platforms like Census or Hightouch are essentially managed outboxes. They handle batching, retries, and rate limiting natively. If you have the budget and prefer a managed UI over SQL and n8n nodes, they are the correct choice.

However, Reverse ETL tools often have sync intervals that are slower than a custom dispatcher (e.g., 15-minute minimums on lower tiers) and may charge per-record or per-sync-frequency. Building your own dispatcher with Postgres and n8n gives you sub-minute latency and total control over the logic without the recurring "sync tax." For a GTM engineer, it is a reusable pattern that transforms a fragile integration into a resilient revenue system.

— C.B.