Your Order Flow Stops at 3am and Nobody Notices Until Morning

Your order sync ran fine all week. Then at 3am on Saturday an access token expired. Your store kept taking wholesale orders. Nothing reached the ERP.

No email went out. The dashboard stayed green, because a job that never starts cannot turn red.

On Monday your warehouse manager asks why the pick list is short. Someone opens the logs. Forty orders are sitting in the store with no match downstream, and the two oldest were promised for Monday delivery.

The short answer: Integrations are built for the day they work, not the night they stop. Add alerts that reach a person with the order reference attached, a dead letter queue somebody reviews, a daily count comparing orders in your store against orders downstream, and a heartbeat for every scheduled job. Monitoring is what turns an integration into an automation.

A silent integration failure is a break in an automated flow that stops work moving without producing any signal a human being actually sees.

Almost every integration is tested on the happy path. An order is created, the API accepts it, the record appears downstream, everyone signs off. That test proves the flow works when both systems are awake, the token is valid and nothing is under load. It proves nothing about the other case, and the other case is the one that costs money.

Stated plainly: an automation without monitoring is a manual process with extra delay. A person still finds the problem. They just find it later, with more orders affected, and after your buyers have noticed.

How does an order flow stop without telling you?

Each of these breaks differently, and each one needs a different thing watching it. The middle column is what the failure looks like from the outside, which is usually all you have.

Failure mode	What it looks like	How you detect it
Access token expired or revoked	Everything stops at once. Calls return 401 or 403	Alert specifically on authentication errors. Put credential expiry dates in a calendar
Webhook delivered once, never retried	One or two orders missing, the rest perfectly fine	Log a receipt per event. Reconcile daily counts
Queue blocked behind one poisoned message	Everything after a certain timestamp stops, nothing before it	Alarm on queue depth and on the age of the oldest message
Rate limiting, HTTP 429	Intermittent gaps, worse on busy days and during bulk edits	Count 429 responses. Alert when retries are exhausted, not on the first one
Schema or required field changed at one end	One order type fails, every other type works	Validate the payload and alert on validation errors by type
Timeout mid-write	A half-written record, or a second copy of the same order	Reconcile field by field. Flag orders with no line items or no totals
Scheduled job did not run	Nothing. No errors, no records, no signal at all	A heartbeat. Alert on the absence of a run, not on a failed one

Notice the pattern. Five of these produce an error somewhere. Two produce nothing, and those two are the expensive ones. Error alerts can only tell you about work that was attempted.

The 429 row is a documented, ordinary condition rather than an emergency. RFC 6585 defines 429 Too Many Requests as the signal that a client has sent too many requests in a given period, and says the response may include a Retry-After header telling you how long to wait. Shopify's API rate limit documentation describes a leaky bucket that refills at a fixed rate and instructs apps to stop calling and back off when they are throttled. Handled properly, 429 costs you nothing. Handled as a hard failure, it drops orders on your busiest days.

The timeout row deserves its own reading, because a write that succeeded but reported failure is how the same wholesale order ends up in your system twice. That mechanism is covered in why the same wholesale order arrives twice.

What do the platforms actually promise when delivery fails?

Two documented examples, both from systems B2B teams genuinely run. These are published behaviors, not estimates.

Shopify webhooks stop trying, then stop existing. Shopify's webhook troubleshooting documentation states that Shopify retries failed webhook calls up to eight times over a four-hour period, that your endpoint has five seconds to respond before a delivery counts as failed, and that after eight failed attempts Shopify stops attempting delivery. It goes further: if failures persist, the subscription is removed, and removed subscriptions receive no deliveries until you create them again.

Read that as an operations fact. A four-hour outage on your receiving endpoint that starts at 3am is enough to lose every order in that window permanently, and enough to leave you with no subscription at all by morning. Nothing on your side errors, because nothing on your side is called. The only controls that catch this are a heartbeat on the receiver and a daily count.

Accounting tokens expire on a clock, not on an event. Xero's OAuth 2.0 FAQ documents a 30-minute access token and refresh tokens that expire after 60 days if unused, after which the user has to reauthorize the app. That is a dated fuse on every connection you have. An integration that goes quiet over a slow trading period can pass 60 days without anyone touching it, and the reconnection then needs a human with the right login.

Both cases share a shape. The vendor documented exactly how it fails. Nobody on the buying side read it during the build, because the build was about making it work.

Third-party logistics providers add their own version of this, covered separately in 3PL integration errors.

What does a sync that stopped on Friday night cost?

Numbers first, then the part you cannot invoice. These are illustrative. Put in your own order volume and hourly rate.

Say your order flow stops at 21:00 on Friday and somebody spots it at 09:00 on Monday.

Line	Illustrative number
Wholesale orders stuck over the weekend	18
Manual re-entry, 8 minutes each	2.4 hours
Re-entry cost at €35 per hour	€84
Orders that missed Monday's 3PL cutoff	18
Orders short-shipped because stock levels never came back to the store	3
Credit notes and reshipments at €40 each	€120
Customer service time chasing it, 3 hours	€105
Visible total	about €309

Illustrative figures. Redo the arithmetic with yours.

The €309 is not the problem. The problem is 18 orders that now ship on Tuesday instead of Monday, to buyers who run replenishment on a schedule and rate you on exactly this. One of them is your largest account.

And the cost scales with detection time, not with order volume. The same failure caught at 07:00 on Saturday costs one phone call and a token refresh.

What does good monitoring look like?

Five things. None of them require a monitoring platform.

1. Every failure lands somewhere a human sees, with the order reference attached. A Slack channel or a shared inbox is fine. The message must carry the order number or purchase order reference, the target system, and the error. An alert that says "sync failed" sends someone into the logs. An alert that says "order WH-4182 rejected by ERP: SKU not found" can be acted on from a phone.

2. A dead letter queue that is reviewed, not just written to. The pattern is standard infrastructure. Amazon's SQS dead-letter queue documentation describes a queue that receives messages which failed processing after a configured number of attempts, set by maxReceiveCount, so they can be isolated and diagnosed rather than lost, and it points to setting a CloudWatch alarm on messages arriving there. That alarm is the part teams skip. Whatever you build on, the rule is the same: failed messages need somewhere to land, and a named person needs a standing 10-minute slot twice a week to open it. A dead letter queue nobody opens is a folder where orders go to be forgotten.

3. A daily reconciliation count. Its own section, below.

4. A heartbeat for anything scheduled. If a job should run every hour, have it report that it ran, then alert when the report does not arrive. This is the only way to detect a job that silently did not start, and it is usually a few lines of code.

5. A named owner and a short runbook. One page. What this integration does, what breaks most often, how to replay a failed message, who to call at the other end, and how to enter an order by hand if the flow is down for a day. Name a person, not a team. Teams do not get out of bed.

Why does daily reconciliation catch what error alerts miss?

Error alerts watch the work that was attempted. Reconciliation watches the work that should have existed.

The control is simple. Once a day, count the orders created in your store yesterday. Count the orders downstream with a matching reference. Compare. If the numbers differ, list the references that are missing.

Two numbers and a list. That catches the webhook that exhausted its eight Shopify retries at 4am, the job that never ran and the queue that quietly stalled, because none of those leave an error on your side to alert on.

Run it in the morning, before your dispatch cutoff, so there is still time to act. Count in both directions if orders can also be created downstream.

If you are scoping a new connection, this belongs in the build, not in a later phase. It is on our B2B e-commerce integration checklist for that reason.

How do you avoid alert fatigue?

The failure mode after no alerts is too many alerts.

This is well documented ground. Google's SRE book chapter on monitoring distributed systems puts it bluntly: when pages occur too frequently, people second-guess, skim or ignore incoming alerts, sometimes ignoring a real page masked by the noise. It also sets the test worth stealing, that every page should be actionable, and that effective alerting systems have good signal and very low noise.

Translate that into three rules for an order flow:

• Alert on the outcome, not the attempt. A single 429 followed by a successful retry is not news. Retries exhausted is news.
• Aggregate the repeats. Fifty orders rejected for the same reason is one alert with a count, not fifty alerts.
• Every alert needs an action. If nobody would do anything differently on receiving it, it is a log line.

Then hold yourself to a number. If your alert channel gets more than a handful of messages on a normal day, the noise is the bug, and fixing it comes before adding anything new.

When is monitoring not worth building?

Say no when:

• The volume is tiny. Five orders a week through a flow where a person already checks every one. They are the monitoring.
• The flow is not time critical. A nightly marketing export that lands a day late costs nothing. Reconcile it weekly and skip the alerting.
• You would build a monitoring system instead of a webhook. The first version is a message into a channel you already use.
• The integration is being replaced this quarter. Instrument the one that survives.
• Nobody will own it. Alerts with no owner are noise with a timestamp.

How do you measure whether it worked?

Write these down now, before you change anything.

• Time to detect. From the moment a flow stops to the moment a person knows. This is the number the whole exercise moves.
• Silent failures found by reconciliation. Breaks that produced no error at all. Above zero means alerting alone was never enough.
• Alerts per normal day, and the share somebody acted on.
• Age of the oldest item in the dead letter queue.
• Orders affected per incident. This falls as time to detect falls.

Check again after a few normal weeks. Time to detect dropping from two days to twenty minutes is the result, even in a month where nothing broke.

Do this in the next hour

1. List every integration that moves orders, stock or invoices. Most teams find more than they expected.
2. Open the vendor documentation for each one and write down what it does when your endpoint fails, and when its credentials expire. Two of those answers will surprise you.
3. Add one alert to the flow with the highest order value, with the order reference in the message.
4. Add the daily reconciliation count for that same flow.
5. Put a name and a phone number next to it.

That is a morning's work, and it covers the failures that cost the most. It is the same Find, Prioritize, Automate methodology applied to reliability instead of speed, and it is how we build software integrations in the first place.

Not sure what your current setup does when it fails? Book a 30-minute call and we will go through one flow with you.

FAQ

What is a silent integration failure?

It is a break in an automated flow that stops work moving without producing a signal anyone sees. Common examples are a scheduled job that never started, a webhook whose retries were exhausted overnight, and a queue stalled behind one bad message. None of these generate an error on your side to alert on, which is why counting records in both systems catches them and error alerting does not.

How do I monitor webhooks for my e-commerce store?

Log a receipt for every event you accept, including the event identifier, so you can tell a repeat from a new one. Alert when processing fails after retries are exhausted, not on the first failure. Then compare daily counts between the sending system and the receiving one, because a delivery that was abandoned after its final retry leaves no trace on your side at all.

How many times does Shopify retry a failed webhook?

Shopify's documentation states that it retries failed webhook calls up to eight times over a four-hour period, and that your endpoint must respond within five seconds or the delivery counts as failed. After eight failed attempts Shopify stops attempting delivery, and if failures persist the subscription is removed and has to be created again. Treat a four-hour outage as permanent order loss.

What is a dead letter queue and do I need one?

A dead letter queue is where messages go after they fail processing repeatedly, so they are isolated for diagnosis instead of lost. You need one as soon as failing a message means losing an order. The important part is the review: alarm on messages arriving there, and book a short recurring slot for a named person to clear it, because a dead letter queue nobody opens is just a place orders go to be forgotten.

How often should I reconcile orders between my store and my ERP?

Daily is right for most B2B teams, run in the morning and before your dispatch cutoff so there is time to act on what it finds. Count orders created yesterday in the store, count the matching records downstream, and list any references that are missing. If your order volume is high or your margin per order is large, run it more often.

How do I stop integration alerts from becoming noise?

Alert on outcomes rather than attempts, so a retry that succeeded stays silent and only exhausted retries reach a person. Aggregate repeats into one message with a count. Then apply the test from Google's SRE practice: every alert should be actionable. If nobody would do anything differently on receiving it, it belongs in a log, not in the channel people are supposed to read.

Sources

• Shopify: troubleshoot webhooks supports the eight retry attempts over four hours, the five-second response deadline, and removal of the subscription when failures persist.
• Shopify: API rate limits supports the leaky bucket model, the 429 throttle response and the instruction to stop calling and back off.
• IETF RFC 6585: Additional HTTP Status Codes supports the definition of 429 Too Many Requests and the Retry-After header.
• Xero: OAuth 2.0 FAQ supports the 30-minute access token and the 60-day expiry of unused refresh tokens requiring reauthorization.
• Amazon SQS: using dead-letter queues supports the dead letter queue pattern, maxReceiveCount, and alarming on messages that arrive there.
• Google SRE Book: Monitoring Distributed Systems supports the alert fatigue argument, the "every page should be actionable" test, and good signal with very low noise.
• Google Cloud Pub/Sub: subscription overview supports at-least-once delivery as the default behaviour of message systems, and the requirement that consumers be idempotent.