What it actually costs when an automation breaks, and nobody notices

The pitch for automation is always about what happens when it works. Nobody sells you the Tuesday in March when a credential expires, the workflow stops, and forty enquiries go nowhere for eleven days before a customer mentions it. That failure is not rare, it is the normal way these things break, and the cost of it is almost entirely determined by decisions you make before launch rather than by what goes wrong.

Silence is the failure mode

When a person stops doing their job, you find out. They are visibly not there, work piles visibly up, someone asks. Automation fails differently: it stops, and everything downstream looks exactly the same as a quiet week.

That is the whole problem in one sentence. A broken workflow and a slow month are indistinguishable from the outside, so the clock on discovery is set by whoever happens to notice, and often that is a customer.

The five ways these things actually break

In rough order of how often we see them. None are exotic and none require anything unusual to have happened.

  • A credential expires. A token, an app password, an OAuth grant that a platform decided to time-box. Nothing was changed by anyone, and the workflow simply stops. This is the most common cause by a wide margin, and it is the most preventable.
  • An upstream system changes shape. A field gets renamed, a form gets an extra question, an API version is retired with notice nobody read. The workflow keeps running and starts writing the wrong thing into the right place, which is worse than stopping.
  • A rate limit or a quota is reached. Usually on the busiest day of the year, because that is what a busy day is. The workflow does not fail loudly, it fails partially, which is the hardest kind to spot.
  • A silent partial failure. Four of five steps complete. The record is created, the notification never sends. Every dashboard says the automation ran, because it did.
  • The person who understood it left. Not a technical failure at all, and the most expensive one on the list. The system keeps working right up until the day it needs a decision, and then nobody can make it.

What it costs, in hours and in risk

We do not put a currency figure on this and we would not trust anyone who did without seeing your numbers first. The two honest units are hours and risk, and they behave very differently.

The same outage, measured two ways
FailureThe hours costThe risk cost
Enquiry capture stopsRebuilding the list by hand from six inboxes, if it can be rebuilt at allEnquiries that were never recorded anywhere and cannot be recovered
Follow-up sequence stopsManually working through everyone who slipped, in a rushCustomers who quietly went elsewhere and will not say why
Records stop syncingReconciling two systems that have drifted, which grows with every dayDecisions made on numbers that were wrong the whole time
A sequence loops or double-sendsMinutes, technicallyComplaints, unsubscribes, and a sending reputation that takes months to repair
Invoices stop going outA day of catch-up paperworkCash timing, and the awkward call about an invoice that is now very late

Why the loud failures are the cheap ones

An automation that crashes noisily at 2am is a good automation. Someone gets told, someone fixes it, and the damage is bounded by the response time.

The expensive failures are the polite ones. A workflow that catches its own errors, logs them somewhere nobody reads, and carries on can run wrong for months. We have found systems quietly discarding a share of their input for most of a year, in businesses that were entirely happy with them, because everything visible looked normal.

This is why "it has error handling" is not reassuring on its own. The question is not whether errors are caught. It is who is told, and how quickly they notice they have been told.

  • Fails loudly and stops: cheapest. Bounded by how fast someone responds.
  • Fails loudly and continues: manageable, if the alert reaches a person rather than a channel.
  • Fails quietly and stops: expensive, discovered by a customer, usually weeks later.
  • Fails quietly and continues with bad data: the worst outcome available, because the cleanup is longer than the outage and you cannot tell which records to trust.

Who gets paged at 2am

This is the question that separates a system somebody owns from a system somebody built. It needs an answer before launch, and the answer has to be a person's name.

For most small businesses the honest answer is: nobody, and that is fine, as long as it is deliberate. Very little in a local business genuinely needs a 2am response. What everything needs is a defined morning owner and an alert that will actually be seen.

  1. Decide what is genuinely urgent. For most businesses, only two things: anything that talks to customers doing so incorrectly, and anything that has stopped capturing enquiries. Everything else can wait until morning without harm.
  2. Send alerts where the person already looks. Not a dedicated inbox created for the purpose, which is a place alerts go to be ignored. Their actual phone, their actual working channel.
  3. Alert on silence, not only on errors. This is the one that gets skipped and it is the one that catches the expensive failures. If the workflow normally runs forty times a day, running zero times is the alert, and no error will ever fire to tell you.
  4. Name one owner, in writing. Not the agency, not "the team". A person whose job description includes reading the exception queue.
  5. Write the runbook while it is boring. Two paragraphs per workflow: what it does, what it touches, how to turn it off safely. Written now it takes twenty minutes. Written during an outage it does not exist.

What ownership has to mean

A large share of the pain here is not technical, it is contractual, and it is decided at the point you agree the work rather than at the point something breaks.

We build systems that clients own, which is easy to say and only means something if it survives the specific test of us disappearing tomorrow.

  • Access, in your own accounts. The workflows, the credentials, the code, the data, all sitting in accounts you control and can revoke us from.
  • Documentation a competent stranger can use. The test is not whether it is documented. It is whether a developer who has never met us could pick it up, which is the situation you are actually planning for.
  • An exit that is not a hostage negotiation. Nothing that only runs on our infrastructure, nothing that stops working if a relationship ends.
  • An honest map of the dependencies. Every third party the system needs, and what happens to you when each one changes. This is the section vendors leave out and it is the one that matters in year two.

How to check what you already have

If you have automations running right now, five questions will tell you where you stand. Most businesses fail three of them, and finding out on a quiet afternoon is considerably better than finding out in March.

  1. How would you know if it stopped? If the honest answer involves noticing something feels quiet, you have no monitoring, whatever the tool's dashboard says.
  2. When did it last actually run? Go and look, right now. This one finds real problems surprisingly often.
  3. Who gets told, and did they see the last one? An alert channel nobody reads is not monitoring, it is the appearance of monitoring, which is worse because it stops you looking.
  4. Could someone else turn it off? If the only person who can safely stop it is unavailable, you have a single point of failure wearing a person's name.
  5. What changed upstream in the last six months? Any tool it touches that has updated, changed a form, or altered a field. That is your list of things to re-test.

The uncomfortable conclusion

Automation moves work; it does not remove it. What used to be forty small manual tasks becomes one system plus a much smaller ongoing responsibility, and that trade is overwhelmingly worth making. But the responsibility is not zero, and pretending otherwise is how businesses end up worse off than before they started.

The businesses that get burned are rarely the ones that built something complicated. They are the ones that built something and then stopped thinking about it, because they were told it would just run.

If you are at the point of choosing what to build, the ninety day roadmap covers the order and the tests for finishing each stage, and build versus buy is worth reading first, because the cheapest system to own is often the one you did not build.

FAQ

How often do these things actually break?
Expect something to need attention a few times a year per system, most of it credential and upstream-change work rather than anything dramatic. That is manageable and predictable. What is not manageable is finding out about it from a customer.
Does a managed service solve this?
It moves who does the work, not whether it is needed, and it is a reasonable choice. What it does not remove is your need to know what the system touches and how to turn it off, because at some point you will want to change vendor or bring it in-house.
We are small. Is monitoring overkill?
The monitoring that matters is not complicated: alert when a workflow that normally runs stops running, and send it somewhere a person already looks. That is an afternoon of work and it catches most of what this article describes.
What should we ask a vendor before signing?
Who gets alerted and how, what happens to the system if the relationship ends, where the credentials live, and what the documentation looks like. Ask to see the documentation from another project. A vendor who cannot show you one has answered the question.
Is a simpler system less likely to break?
Fewer moving parts fail less often, yes. But simple systems break the same way complicated ones do, through expired credentials and upstream changes, and simple systems are more likely to have been left unmonitored precisely because they felt too small to bother with.