The Counting Problem

In 2013 the Indian government’s official records put malaria deaths in the country at 440. The government’s own medical certification of cause of death report put them at 5,428. A committee of the Indian Council of Medical Research put them at 40,297. The Lancet put them at 205,000.

I came across those four numbers sitting next to each other while running a stakeholder audit for Malaria No More in India, and the finding that mattered was not a health finding. State funding was allocated against reported incidence. The count was wrong by somewhere between ten and a hundred times, so the funding was wrong, so the delivery was wrong. Nobody in that chain was lying. Each of them was reporting the number their instrument produced.

The sharpest single piece of evidence in the study came from an accredited social health activist in Junagadh Block, Kalahandi, in Odisha. She was paid between twenty-five and seventy-five rupees for malaria work and better rates for maternal reporting, which meant she was financially rewarded for spending her time elsewhere and quietly penalized for finding malaria. A national policy adviser was interviewed for the same study. The village health worker had the finding.

What a group has agreed to count

That audit is the clearest example I have of the only thing I have ever really worked on, which is the gap between what drives an organization and what it reports.

A newsroom that counts page views produces one kind of journalism. A newsroom that counts corrections produces a different one, with the same reporters. A company that counts booked pipeline builds a different sales organization than one that counts the time between a customer’s problem and its resolution. Once the counting is settled, most of the behavior is settled with it, whatever the values poster in reception says.

Every organization I have worked in has a gap between the list of things that actually move it and the list of things it reports every month. Every one of them believes its own gap is smaller than it is.

My CV reads as though I have had four or five careers. Journalism, advertising, a consultancy advising boards, companies of my own, streaming. For a long time I picked whichever one a room seemed likeliest to find interesting and left the rest out, because the alternative was explaining a résumé that looks like four people took turns with it.

They were the same job, and it took me too long to see it. Working out what is worth counting is an empirical problem. Making the count legible to the people who have to act on it is a storytelling problem. Building the instrument that does the counting reliably, without you in the room, is an engineering problem. Drop any one and you get a failure I now recognize on sight: a finding nobody acts on, a story nobody can verify, a machine measuring the wrong thing to four decimal places.

Twenty-eight years of that, in four versions.

In a newsroom, the gap was between what mattered and what was countable. I was a journalist from 2001 to 2007, and the honest position for most of it was that nobody knew what was working. Ratings arrived late and described the wrong thing. Then the internet arrived with numbers attached to everything, and we found that the countable thing and the valuable thing were not merely different but could be opposed. A story that mattered and a story that traveled were two different products, and only one of them showed up in the report.

At Publicis, the gap was the product. Six years, most of them spent working out what a marketing dollar actually bought in a market where the tooling changed faster than the accounting could follow. I grew the group’s digital practice in India from one office to four and from ten people to more than a hundred. What we were really selling, under the campaigns and the media plans, was a defensible answer to a question the client’s finance director kept asking.

At Xynteo, the gap moved up a floor. As Chief Digital Officer I sat with executive teams at some of the largest companies in the world and asked whether their strategies were working. Almost none had built the instrument that would tell them, and not because they were unserious. Building it was expensive, slow, and required a kind of attention that quarterly life does not protect. The strategy got measured by whether it survived contact with the board, which measures something, just not the thing.

Then it became my own problem. I have started four companies and sold two of them. The first, in 1998, ran on a content management system I wrote by hand. The most recent was Laminar, which let media companies launch a full streaming service — backend through monetization and analytics — in eight weeks rather than years, and which was acquired in December 2023. What running a company teaches you about this is that the scarce input is not capital or engineering. It is knowing which of the forty things on the roadmap actually moves the business, when all forty have advocates and three matter.

What changed this year

For twenty-eight years, the reason organizations did not measure the thing that mattered was cost. The instrument meant an engineer, a data pipeline, a quarter of someone’s roadmap, and a political fight about whose number was right. So it did not get built, a proxy stood in for the truth, and everyone agreed not to look too hard at the seam.

That cost has fallen far enough to change the decision. I spent this summer building the kind of instrumentation that used to require a team, and the marginal cost of running it lands at fractions of a cent. A researched dossier on a company, a few cents. Extracting the fields from an invoice, about a penny. A harness that puts ten business tasks through three routes — a hosted frontier model, a small model on hardware I already own, and a router choosing between them per task — scoring each on accuracy, latency, cost, and what left the building: one afternoon of setup, then effectively nothing per run.

When the instrument cost a quarter of an engineering roadmap, “we don’t measure that” was a resource decision. It is now a choice.

What did not change

The obvious conclusion, that we are about to get much better at measuring things, is premature. Cheap instruments do not produce good measurement any more than cheap cameras produced good photographs. They produce more measurement, most of it aimed at the wrong thing.

The hard part was never the building. It is knowing which question is worth an instrument, which proxy is lying to you, and which number an organization will actually act on when it arrives. That is judgment, and judgment has not got cheaper by a cent. It has become more valuable, because there is now nothing standing between a confident wrong idea and a working implementation of it.

I know the price of getting that wrong, because I have paid it. In 2019 my company published seat assessments for Uttar Pradesh, built from models that scored hope, fear, aspiration and resentment. The assessments were badly wrong. The models had been signaling that fear and resentment should be weighted much higher than I had set them, and I overrode them, because I did not believe anger was running that hot. The signal was in the data and I discounted it because it did not match my read of the country.

That is why every system I build now logs a human override with a written reason. The failure that costs you is never a broken instrument, because a broken instrument gets found. It is an instrument that is right in a direction you find implausible, which is precisely the moment it is easiest to talk yourself out of.

So the useful version of the question has changed shape. It used to be can we measure this, which has an engineering answer and no longer costs anything to obtain. It is now should we, and will anyone do anything differently when the number arrives, which never had one.

Twenty-eight years, one question, and for the first time the instrument is not the expensive part. The choosing is, and the choosing is the work I intend to spend the next twenty-eight years on.