Ten Tasks, Three Ways

“Run AI publicly where it makes sense and privately where it matters” is the kind of sentence anyone can say, and I have said it a lot this year. It sounds like judgment and costs nothing to assert.

So I built the thing that tests it. A harness that takes ten business tasks — classification, extraction, one-line summary, question-answering over a supplied paragraph — and runs every one of them three ways: through the hosted frontier model, through a 7B model on my own machine, and through a router that decides per task which of the two should see it.

Four of the ten cases have fake personal data planted in them: a name with an email and a social security number, a name with a phone number, a bank account number, a name with a date of birth. Those exist to make the router work for its living. Scoring is deterministic — exact or fuzzy match against a gold answer, or the fraction of required keywords present. No model grades its own homework.

One run, on live infrastructure, twenty real HTTP calls, forty and a half seconds:

	Accuracy	Avg latency	Cost	Cases leaked
Public	10/10	2730ms	$0.0057	5/10
Private	10/10	1137ms	$0.0000	0/10
Hybrid	10/10	1764ms	$0.0037	0/10

What I will and will not claim

The 7B model matched the frontier model on all ten. That is a statement about these tasks — short, well-scoped extraction and classification — and not a claim about model capability in general. Run it against something open-ended and I expect the gap to reappear.

Private was also faster, 1137ms against 2730ms. I am not going to tell you self-hosted inference is faster; for prompts this short, a hop to my own hardware simply beats a round trip to an API plus its inference time. What it does kill is the assumption I have to argue people out of most often — that private automatically means slow.

The result I would have preferred to leave out

The router’s heuristic for “a name next to a date” fired on a plain invoice. Bill to: Acme Corp… Due date: 2026-09-01. “Acme Corp” looks like a First Last name; the date looks like a date. It routed an ordinary invoice to the private model for no reason.

I am reporting it rather than quietly editing the eval set around it, because it is the same classifier running in production with the same known failure mode. And the direction of the error is the whole point. An over-eager classifier wastes a private call on a boring invoice. It never sends someone’s social security number to a public API.

The failure worth engineering against is the false negative. A harness that hides its own false positives has not earned your trust on false negatives.

Why an instrument beats an opinion

Ten cases is a small set, deliberately. The value is not the scoreboard — run a different task mix and every number moves. The value is that a placement decision now arrives with measurements attached instead of a consultant’s confidence.

I have sat through a lot of meetings where the question of where to run the model got settled by whoever spoke last. This is cheaper than that meeting, and it is right more often.

The full teardown, including the execution log and the scoring code, is on the Marain site.