Why companies pay millions to parallelise their own overhead

In 2015, three researchers, Frank McSherry, Michael Isard and Derek Murray, published a short paper with a title that should have ended several product roadmaps: “Scalability! But at what COST?” Their test was simple. They took the published benchmarks from the leading distributed graph-processing systems, the kind that spread one computation across hundreds of cores, and ran the same workloads on a single thread of McSherry’s laptop.

The laptop won. Not narrowly, and not on toy problems. One task was finding connected components in Twitter’s follower graph: 1.5 billion follow relationships, with the job of working out which accounts link to which. GraphLab, a leading system of the day, took 242 seconds using 128 cores. The laptop, running a simple technique from the 1970s called union-find, finished in 15. PageRank, the algorithm Google originally used to rank web pages, told the same story. The single thread completed 20 iterations in 300 seconds, or 110 with a better memory layout. GraphX needed 128 cores to take 419.

The paper turned this into a metric: COST, the Configuration that Outperforms a Single Thread. In plain terms, how many cores does a distributed system need before it beats one competent thread on one machine? For several published systems the answer was in the hundreds. For some the authors recorded it as unbounded: no configuration of any size ever caught up.

That paper is a decade old. It has been cited, admired, and almost entirely ignored. Distributed systems are supposed to trade simplicity for performance: you accept complexity and get speed in return. Measured, they often trade away both, and send the bill to the infrastructure budget. One worked example shows the size of the bill.

A $1.4 million system doing a $57,000 job

Consider a mid-market SaaS company with 55 engineers. Its event pipeline handles 2 billion events a day: clicks, telemetry, billing signals. The architecture is the standard one. A Kafka cluster of nine machines queues the events; a Flink deployment of 128 virtual CPUs processes them; a fleet of smaller services consumes the results. (Kafka and Flink are the industry’s default tools for moving and processing streams of events. A virtual CPU is one rented core.) Three and a half platform engineers keep it all alive. The design passed review without argument, because every component is an industry standard.

Now do the division. Two billion events a day is 23,000 events per second. At 500 bytes per event, the daily volume is one terabyte, which works out at 12 megabytes per second. For scale: one modern solid-state drive reads 7,000 megabytes per second, and one core can parse several hundred thousand small JSON records per second. This is a single-machine workload with a hundredfold margin to spare at peak.

What the company pays to treat it as a distributed one:

Compute. 128 virtual CPUs of stream processing at $0.10 per core-hour, running around the clock: $112,000 a year. The Kafka machines, with their storage and replication: roughly $90,000. Then cross-zone transfer, the charge cloud providers levy when data moves between their data centres. An event pipeline that replicates everything moves a great deal of data between data centres: another $60,000.

People. The platform engineers exist because the architecture exists. Queues need rebalancing; checkpoints need tuning; lagging consumers need watching; message formats need governing. At a loaded cost of $250,000 per engineer, 3.5 of them cost $875,000 a year. This is the largest number in the system and the least examined, because it is booked as headcount rather than infrastructure.

Incidents. Systems built from many coordinating parts fail in coordination-specific ways. A queue rebalances itself in the middle of a deploy. A checkpoint corrupts. A system that promises to process each event exactly once will, under failure, process some events twice. One serious incident a quarter, at $60,000 in engineering time and customer clean-up, is a conservative rate: $240,000 a year.

The comparison nobody priced. Two servers at $10,000 each, a primary and a spare, spread over three years, plus a fifth of one engineer to own the ingest process: about $57,000 a year. And the single process is faster, not merely cheaper. The work of packing data up, sending it across the network and unpacking it again, the work that consumed most of the cluster’s cores, is simply gone. This is the detail the scaling chart never shows. The distributed version is not a faster system that costs more; it is usually a slower system that costs more.

A 24x multiple, paid to run a workload the cheaper option clears with room to spare. The company is invented; the ratios are not. They fall straight out of McSherry’s arithmetic. If a system needs 64 cores to match one well-used thread, those cores are not solving the business problem. They are solving the system’s own coordination problem. At $0.10 per core-hour, that is $56,000 a year spent parallelising overhead.

The hardware moved; the reflex did not

The distributed-by-default habit formed between 2004 and 2012, when it was frequently correct. Google published its MapReduce paper in 2004, describing how to split one job across thousands of cheap machines, because no single machine could hold a copy of the web. A commodity server of that era had four cores and 8 gigabytes of memory. The data did not fit on one machine, so the only path was outward, and coordination overhead was the price of admission.

Every number in that calculation has since moved by a factor of a hundred or more. A single server now ships with 192 cores. Cloud providers rent machines with 24 terabytes of memory by the hour. One solid-state drive reads 14 gigabytes per second, and a standard server case holds twenty-four of them. A $5,000 workstation with 32 cores and 256 gigabytes of memory is a more capable computer than the clusters MapReduce was invented for. The Twitter graph that humbled GraphLab fits in the memory of a mid-range laptop.

The data did not grow to match. Jordan Tigani, a founding engineer on Google’s BigQuery, wrote in 2023 that most customers he had observed held less than a terabyte of data in total, and that the median data warehouse was far smaller. The industry built tooling for the workloads of five companies and sold it to five hundred thousand.

The companies that noticed have published the results. Amazon’s own Prime Video team wrote in 2023 that merging its stream-monitoring service from a scatter of small cloud functions into one ordinary program cut infrastructure cost by 90%: an argument against reflexive distribution, published by the world’s largest seller of distribution. Segment, a data company, reversed its microservices migration after the overhead of running more than a hundred separate services consumed the team that built them. Stack Overflow served over a billion page views a month from one program running on nine web servers, processors idling in single digits, while conference stages filled with architectures serving a fraction of the traffic on ten times the hardware.

None of this says scale is fictional. It says the threshold at which distribution pays has moved so far up that most companies will never cross it, and the ones who assume they have crossed it rarely check.

Nobody is paid to measure the comparison

The measurement is a week of work and it almost never happens, because every party in the decision has a reason not to take it.

Frameworks grade their own homework. A distributed system is benchmarked against other distributed systems, or against itself on fewer machines. A chart showing throughput rising from 8 nodes to 64 reads as success. Whether one node, or a plain program with no framework at all, would beat the whole curve is a question the benchmark suite is not built to ask. And the framework itself narrows the field: the 15-second union-find solution was unavailable to the 128-core systems, not because it was obscure but because it did not fit the style of computation the frameworks require. The framework forces a worse algorithm, then takes credit for scaling it.

Careers reward complexity. An engineer who operates a Kafka-and-Flink pipeline across forty services has a CV. An engineer who replaced it with a process reading from a local disk has an anecdote. Promotion committees, conference programmes and job specifications all price distributed-systems experience at a premium, so the labour market pays engineers to manufacture the experience. Nobody has ever been asked in an interview to justify the cluster they did not build.

Vendors sell node-hours. Cloud pricing charges by the machine and by the data moved. A customer who merges 64 cores of coordination overhead onto one machine has, from the vendor’s side, destroyed revenue. The commercial gravity of the cloud, its reference architectures and certification tracks included, pulls toward many-machine designs for reasons that have nothing to do with the customer’s workload.

Premature scaling wears the costume of prudence. “Will it scale?” ends conversations. “What does this cost before we need it to scale?” starts arguments. Building for a hundred times the actual load is treated as diligence; building for the actual load is treated as risk. The arithmetic says the opposite: the difference is seven figures a year, spent on nothing customers can see.

The habit has already reached the newest layer of the stack. AI systems that look up documents before answering route one question through an embedding service, a vector database, a reranker and an orchestration layer: six network hops to produce an answer a simpler setup on one machine returns faster. The components changed. The reflex did not.

Four honest reasons to distribute

The argument is not that clusters are always wrong. It is that distribution is a cost to be justified by measurement, not a default justified by convention. Four justifications survive scrutiny, and each is narrower than commonly assumed.

The data does not fit. Petabyte working sets exist. But “does not fit” means does not fit in 24 terabytes of memory or a shelf of fast drives after compression and a sensible layout; it does not mean “does not fit the default configuration of the tool already chosen”. Most datasets that arrive in planning documents as petabyte problems fit on one drive.

Availability requires replicas. Surviving the failure of a machine takes more than one machine. It does not take forty services. A primary, a spare and a script that switches between them buy most of the availability at a fraction of the coordination. The difference between two machines and a distributed system is the difference between redundancy and choreography.

Latency requires geography. Users in Sydney and Frankfurt cannot both sit 10 milliseconds from Virginia; signals in fibre only travel so fast. Geographic distribution is physics. It justifies deploying the same simple system in three regions, not breaking that system into pieces.

The organisation needs partitioning. Two hundred engineers cannot all work on one deployable program without trampling each other. Systems end up shaped like the teams that build them; the observation is old enough to have a name, Conway’s law. But this is an argument about team boundaries, and it is routinely used at a tenth of the scale where it holds. A 30-engineer company running 40 microservices has not partitioned its organisation. It has fractured it.

Each of these can be tested cheaply, before the architecture is bought. Which points to the remedy.

The fix costs one engineer one week

Before any distributed architecture is approved, someone builds the single-machine baseline: the workload, implemented competently, on one good server. Real data volumes, real access patterns, no framework. The result becomes a number in the design review, and the proposed architecture must beat that number by a margin that justifies its operating cost.

This is not a novel engineering practice. It is an ordinary financial control applied to a category of spending that has escaped it. No CFO approves a $1.4 million annual commitment on the strength of a chart showing that the commitment scales. Yet that chart, throughput rising with node count, is what every architecture review currently shows. The baseline turns a matter of taste into a purchasing decision.

The discipline teaches as it goes, which is its second effect. Teams that build the baseline learn their workload’s true shape: the dataset that fits in memory, the peak that is a tenth of the estimate, the one query that dominates everything. Some will still build the cluster, and will build a better one, because they know what it must beat. Most will find that the comparison nobody ran was the better system all along: faster, cheaper, and comprehensible to the people on call for it.

McSherry and his co-authors ended their paper by suggesting that new systems be measured against the hardware already on an engineer’s desk. A decade on, the hardware is a hundred times better and the question is still missing from most design reviews. Asking it costs nothing. Not asking it costs, at mid-market scale, seven figures a year.

The laptop is still winning. The industry is still not keeping score.