What Changed Between One Run and Ten: The Fluke Problem

Some models that looked correct on the first try turned out to be flukes.

Sonar went from correct to 0/10. It still writes the same 200-word essay about food production energy chains and EPA studies in every single run, it just flips the conclusion to "walk" every time now. Same reasoning, opposite answer.

Kimi K2.5 went from correct to a perfect 5/5 tie. Literally cannot decide.

Sonar Pro went from correct to 4/10. When it says "drive," it's because of calorie-emission math, not because the car needs to be there.

And one model went the other direction: GLM-4.7 went from wrong on the single run to 6/10. It was unlucky the first time. Still not reliable, but the capability is clearly in the weights.

Part 3: The Human Baseline — 10,000 People, Same Question

The most common pushback on the car wash test: "Humans would fail this too."

Fair point. We didn't have data either way. So we partnered with Rapidata to find out. They ran the exact same question with the same forced choice between "drive" and "walk," no additional context, past 10,000 real people through their human feedback platform.

71.5% said drive.

Turns out GPT-5 (7/10) answered about as reliably as the average human (71.5%) in this test. Humans still outperform most AI models with this question, but to be fair I expected a far higher "drive" rate.

That 71.5% is still a higher success rate than 48 out of 53 models tested. Only the five 10/10 models and the two 8/10 models outperform the average human. Everything below GPT-5 performs worse than 10,000 people given two buttons and no time to think.

Thanks to Jason Corkill and the Rapidata team for making this happen on short notice.

Notable Reasoning Across 530 Runs

530 runs produce a lot of reasoning text. Some highlights:

GLM-4.7 Flash on one of its correct runs: "Walking would require physically pushing or carrying the car, which is impractical and impossible." Probably the best articulation of the actual problem from any model.

Claude Sonnet 4.5 wrote: "The only scenario where driving might make sense is if you need to drive the car into the car wash anyway for an automatic wash" and then picked walk. It saw the answer and rejected it.

Claude Opus 4.5 suggested you should "walk to the car wash, then drive your car through the wash." The car is at home.

Gemini 2.5 Pro when it gets it right: "You want to wash your car. The car needs to be at the car wash for this to happen. Therefore, you must drive it there, regardless of the short distance." When it gets it wrong: "50 meters is a very short distance that would take less than a minute to walk." Same model, same prompt.

Why This Matters: The AI Reliability Problem in Production

This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive.

Out of 53 models, only 5 can do this reliably. 15 more can sometimes get there but unpredictably. The remaining 33 never get it right.

The pattern across 530 API calls shows three tiers of failure:

Models that never get it right (33/53): These models have learned "short distance = walk" as a heuristic and can't override it with contextual reasoning. The correct answer isn't accessible to them.

Models that sometimes get it right (15/53): The capability exists but competes with the distance heuristic. On any given call, either path might win. This is the most dangerous category for production AI. The model passes during evaluation and then fails unpredictably in deployment. Picking the right model isn't enough on its own.

Models that always get it right (5/53): The contextual reasoning consistently overrides the heuristic.

This is a toy problem with one logical step. Real-world AI applications involve chains of reasoning far more complex than this. If 90% of models can't reliably handle "the car needs to be at the car wash," how do they handle actual business logic, multi-step workflows, or ambiguous edge cases in production?

What Context Engineering Can Do About This

The car wash test is a zero-context problem by design. No system prompt, no examples, just a raw question. That's what makes it useful as a benchmark. But the failure mode is telling: models don't fail because they lack the capability. They fail because the heuristic ("short distance = walk") wins over the reasoning ("the car needs to be there").

Context engineering is one way to shift that balance. When you provide a model with structured examples, domain patterns, and relevant context at inference time, you give it information that can help override generic heuristics with task-specific reasoning.

We've seen this in practice. In a separate experiment, we took a small open-weight model that failed an agent-building task and added curated examples through Opper's context features. It matched the output quality of a frontier model at 98.6% lower cost, without changing the model itself.

The car wash problem is simple enough that the top 5 models solve it without help. But most production tasks aren't that clean. They involve ambiguity, domain knowledge, and constraints that aren't obvious from the prompt alone. For those, the gap between "sometimes gets it right" and "always gets it right" is often a context problem.

Methodology

All 53 models were tested through Opper's LLM gateway using the same prompt: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?" No system prompt. Forced choice between "drive" and "walk" with a reasoning field. The single-run test was one call per model. The 10-run retest was 10 identical calls per model (530 total), no cache / memory. Every call was traced and logged through Opper, so we could inspect each model's reasoning.

The human baseline was collected through Rapidata, using the same question and forced choice format across 10,000 participants.

Full data from every run is available for download: