Retry the Transient, Escalate the Rest

Every workflow I run routes its failures to one place, which sends an alert to my phone within seconds. For months that was all it did: something broke, and my phone buzzed the same way every time.

That is the right design when every failure deserves a human. It stops being right once unattended jobs fail for reasons no human can act on, such as a socket that hung up mid-request or an API that timed out and recovered four seconds later. So I taught the handler to triage first.

Three conditions, all of them

On a failure, the handler gathers the context — which workflow, which node, the error text, the execution ID — and asks a 7B model running on my own machine for three fields: a category of TRANSIENT, LOGIC or UNKNOWN, a confidence score, and a one-sentence reason.

A retry happens only when the classification is transient, the confidence is at least 0.7, and the failed workflow is on an explicit allowlist of jobs that are safe to run twice. If any one of those is missing, it escalates as it always did.

Every failure of the triage step itself produces the same output: an unset endpoint, an unreachable endpoint, or a reply that will not parse as JSON each return UNKNOWN at confidence 0, which can never clear the gate. A broken classifier leaves the system behaving as it did before I built any of this. That property is the only reason I was willing to put a model in the path of my alerting.

The allowlist has one entry

I kept this part the most conservative, and the reasoning generalizes.

A workflow earns a place on the list by meeting four conditions at once: it sends nothing outward to a third party, it behaves identically when run twice, it can be triggered programmatically, and it runs on a schedule rather than holding a live connection open.

One workflow qualifies. It reads three public APIs, writes nothing to the database, and de-duplicates its own output, so the worst case of a spurious retry is a repeated digest to me.

Anything that messages a human outside the firm fails the first condition and cannot be added. A confident classification cannot put a workflow on the list. The model gets a vote on what happened, never on what is safe, and only the first is a judgment I am willing to delegate.

What the live runs found

I tested the gate and the allowlist independently, which turned out to matter. A throwaway workflow, temporarily allowlisted, threw a transient-styled failure: classified TRANSIENT at 0.9, retried twice with measured waits of 30 and 121 seconds, then escalated once its budget was spent. The same workflow then threw a TypeError, which was classified LOGIC at confidence 0 and never retried, with six nodes running instead of fifteen.

Two bugs surfaced only under real execution, and both were the empty-lookup shape I had already been bitten by once: on a workflow’s first failure there was no retry-budget row yet, so the node was skipped and the failure produced neither a retry nor an alert. Static validation reported both paths clean, because nothing was malformed.

Net effect

A transient failure on the allowlisted workflow now retries itself with real backoff and, if it recovers, never reaches me. Everything else escalates as before, with one extra line saying what was tried and why.

The full write-up is on this site.