GTM Galaxy

Article

Out-of-Order by Design: Solving Webhook Event Inversion in GTM Systems

Claudine Baumbach / / Pacific

A RevOps lead recently shared a screenshot of a Postgres record that shouldn't exist. The lead had a status of 'Closed Won,' but the 'Created At' timestamp was null. In the CRM, everything looked perfect. But in the warehouse, the status update had arrived and been processed before the creation event. The handler for the status update had created a partial record, and the subsequent 'create' event couldn't reconcile with it.

This wasn't a bug in the CRM. It is a fundamental reality of distributed systems. If you build GTM tools that rely on webhooks, you must assume that arrival order is essentially random during periods of high activity. If your logic trusts the sequence of your HTTP requests, you are presiding over a system prone to silent state corruption.

The FIFO Fallacy

The default mental model for webhooks is First-In, First-Out (FIFO). If a customer clicks a button at 10:00:01 and another at 10:00:02, we expect the webhooks to hit our server in that order.

Major GTM platforms explicitly do not promise this. HubSpot states clearly that webhooks are not guaranteed to arrive in the order the events occurred. Stripe uses an 'at-least-once' delivery model; if a webhook fails or hits a timeout, Stripe retries it. By the time that retry hits your endpoint, a later event has likely already been processed.

Distributed systems prioritize throughput and reliability over strict ordering. When HubSpot fires a webhook, it isn't coming from one giant machine with a single queue. It originates from a fleet of dispatchers. If Dispatcher A hits a slow network hop while Dispatcher B is flying through its queue, Dispatcher B will deliver its later event first. This is 'event inversion.'

Mapping the Inversion: An Empirical Test

To quantify this, we ran a stress test on a HubSpot portal during a bulk import of 50,000 contact records. We logged the occurredAt timestamp from the HubSpot payload against the receivedAt timestamp on our ingestion server.

We observed an inversion rate of 4.2%. For roughly 2,100 records, events arrived out of sequence.

A common failure mode looks like this:

  1. T1: Contact Created in HubSpot.
  2. T2: Contact Property 'Lifecycle Stage' updated to 'SQL'.
  3. T3: Webhook for T2 (the update) arrives at your middleware.
  4. T4: Webhook for T1 (the creation) arrives at your middleware.

If your handler uses a simple UPDATE or UPSERT without checking versioning, the 'Created' webhook at T4 will overwrite the 'SQL' status with whatever the default value was at the moment of creation. You’ve just downgraded a hot lead back to a subscriber because you trusted the clock on your server rather than the metadata in the payload.

Platform-Specific Failure Modes

Each platform has a different way of signaling its lack of sequence.

HubSpot HubSpot provides an occurredAt (milliseconds) and a portalId in every payload. The most frequent inversion happens between the contact.creation event and the initial contact.propertyChange events. Because these happen milliseconds apart, they are often picked up by different dispatchers.

Stripe Stripe relies on the created Unix timestamp. In financial systems, inversion is dangerous. A customer.subscription.updated event arriving after a customer.subscription.deleted event could result in a database state that thinks a subscription is still active. Stripe's recommendation is to persist the IDs of processed events to handle duplicates and compare timestamps to ignore lagging updates.

Salesforce Salesforce Outbound Messages are notoriously prone to inversion during retries. If the first message in a sequence fails to receive a '200 OK' from your server, Salesforce will move on to the next message while the first stays in the retry queue. For a more robust approach, Salesforce Platform Events include a ReplayID. While these IDs are strictly increasing, they are only sequential per partition. If you are running multiple concurrent consumers, you still face race conditions at the database level.

Building the Monotonic Fence

The solution is a monotonic version check. You should never allow an incoming webhook to update a record unless the timestamp of the new event is strictly greater than the timestamp of the last update you processed.

In SQL, this is a conditional write. Instead of a blind update, you add a clause to your query that acts as a gatekeeper.

-- The 'last_event_at' column stores the 'occurredAt' from the payload
UPDATE leads 
SET 
  lifecycle_stage = :new_stage, 
  last_event_at = :occurred_at
WHERE 
  external_id = :external_id 
  AND (last_event_at < :occurred_at OR last_event_at IS NULL);

If the laggy 'Created' event arrives after the 'Updated' event, the last_event_at < :occurred_at check will fail. The database returns '0 rows affected,' and your data remains in its most recent, correct state.

The Idempotency Requirement

Inversion isn't the only problem; you also have to deal with duplicates. At-least-once delivery means you will eventually receive the same webhook twice. This usually happens when your server takes 501ms to respond to a platform that expects a response in 500ms. The platform assumes failure and sends the payload again.

By using the monotonic fence described above, you get idempotency for free. If an event ID or timestamp is identical to what is already in the database, the update condition fails, and you avoid redundant processing or double-counting in your logs.

Trade-offs: FIFO Queues vs. Logic Fences

Infrastructure purists might suggest using AWS SQS FIFO queues or partitioned Kafka topics to force ordering. For most GTM operations, this is excessive. SQS FIFO queues are limited to 300 transactions per second (without extra configuration) and introduce 'head-of-line blocking.' If one 'poison pill' message fails, it halts the entire queue for that contact, creating a massive backlog.

Polling is another alternative. Pulling data from the HubSpot or Salesforce API every 15 minutes avoids sequencing issues because you are always fetching the 'current state.' However, this destroys the real-time responsiveness required for modern speed-to-lead motions.

Practical Inspection

If you suspect your system is suffering from state corruption, run a simple audit. Log the occurredAt timestamp from the payload and your own processedAt timestamp.

Run this query after a high-volume event (like a marketing blast or a CSV import):

SELECT contact_id, occurred_at, processed_at
FROM webhook_logs
ORDER BY processed_at ASC;

Look for rows where occurred_at decreases while processed_at increases. That delta is your inversion rate.

Don't try to fix the CRM's dispatcher; you won't win. Instead, build your downstream handlers to be skeptical. Treat every incoming webhook as a suggestion that must pass a timestamp test before it is allowed to touch your production data.

— C.B.