A passing check is a claim, not a fact. In one working day on this site, a page was giving away a paid book, a contact form was losing every enquiry, the site map was rendering blank, and a metric named “coherence” was mostly measuring how short a text was. Every one of these had already passed a check. This note is the catalogue of how that happens — ten ways a check goes wrong while staying green — and the shorter list of moves that found things. The thesis fits in one sentence: checking again re-runs the instrument; checking differently tests it.
The ledger
The examples here are not hypothetical and not curated from a career — they are one audit day on this site, 18–19 July 2026. The checker whose failures are catalogued was Claude, working as the site’s engineer; the checks that failed were its own. What made the day unusual was not the number of defects but their camouflage: every serious one sat behind a check that had already passed. The passing check was the disguise.
1. The check that cannot fail
A verification script printed twelve lines reading BAD and then concluded PASS. The failure flag was being set inside a subshell and died with it; the summary read a variable nothing could ever have written. The check was not lying — it was incapable of telling the truth. Its cousin is the test suite recorded from a system’s current behaviour: it passes on the day it is written and forever after, because it asserts that the system does whatever it does.
The tell: a check that has never failed is unproven. Break the thing on purpose once and watch the check go red. If it cannot be made to fail, it is not an instrument; it is a decoration.
2. The wrong dialect
A search for href="/courses" found nothing, because the framework emits href="/courses/". A dollar sign meant literally was read as a regex anchor and matched nothing. A price never appeared as a contiguous string because the renderer inserts comment nodes between text runs. An audit expecting min="0" met JSX writing min={0}. In each case the check was well-formed — for a substrate that was not the one deployed. Checker and checked spoke two dialects, disagreement came out as silence, and silence read as health.
The tell: before trusting a checker’s negatives, hand it one known positive. If it cannot find the thing you know is there, its “nothing found” means nothing.
3. Looking where the light is
The site map rendered blank for most of a day, and every audit passed while it did. The links were audited in source, where all of them were present and correct. The console was checked for errors and had none, because a parse error kills a script before any handler exists to report it. The HTML fallback held every link, hidden by CSS on desktop. Each check looked where checking was easy. The failure lived where the reader lives — the rendered page — and the only instrument pointed there was a person, who said: nothing really visible.
The tell: check at the layer where the consequence happens. A source audit is a fact about source. Only a rendered check is a fact about the page.
4. The measure that measures something else
The writing instrument on this site reported a “coherence” score — the algebraic connectivity of a sentence graph. It was stable, reproducible, mathematically well-defined, and mostly measuring length: the same document truncated to eight sentences scored 0.39; whole, 0.11. Identical prose, a fourfold swing. A measure can be perfectly reliable about the wrong thing. What exposed it was an invariance test: apply a manipulation the measure should ignore — truncating the same prose — and watch whether it matters. The repair was not a cleverer denominator (four were tried; all failed) but a fixed scale of measurement.
The tell: reliability is not validity. Find a transformation the measure should be indifferent to, apply it, and see.
5. The sample the checker wrote
To test that metric, four sample texts were first written by hand — a tight argument, a rumination, a disconnected list, a technical passage. They confirmed the confound the author expected and concealed the one he hadn’t imagined, because they were written varying the suspected thing and holding steady the unsuspected one. Seventy-four documents written for other purposes — the site’s own papers and course modules — reversed the finding in a single run.
The tell: a test set authored by someone who knows the hypothesis is the weakest evidence there is. Use material that existed before the question did.
6. The design that erases its effect
Three experimental designs in a row failed to find a hysteresis loop that was really there. The first held each measurement so long that the memory it hunted had already relaxed — quasi-static by construction, and hysteresis vanishes in the quasi-static limit by definition. The second compared the effect at its best against the control at its worst. The third took the ratio of two numbers that were both noise. The loop appeared on the fourth design — paired, rate-matched, absolute — and had been there all along.
The tell: ask what would distinguish the effect from the artefact that mimics it, and design for the distinction. A design that only asks is there an effect? will find one, or erase one, by construction.
7. Silence read as success
The contact form posted to a worker that had not existed for weeks. Every submission failed; every failure was swallowed by a catch block; the form looked fine and told no one. The suggestion box had the identical disease. A game’s scores went to endpoints never deployed, each request ending in catch(()=>{}). Nothing looked broken, because looking broken had been explicitly handled away. Silence is ambiguous between no failure and no signal, and each of these systems had been built to prefer the flattering reading.
The tell: absence of an error is not presence of a success. Probe the path end to end and require the positive signal — the 201, the stored record, the reply — before believing it.
8. The document that drifted from its data
A page argued that a steel bench and a pine bench sit about 9 °C apart at the skin. The figure was real, computed from a table of conductivities and densities that lived in the same file. Then the table was edited — better numbers, sound reasons, and every test still green, because the code was never wrong. Only the sentence was. A document quoting a figure it did not measure is an untested assertion, and it fails silently: nothing throws, nothing 404s, the page simply says a thing that is no longer true and keeps saying it.
The fix was to write the sentences down as tests. Not one assertion per function — one per claim, named after the sentence it defends: steel and pine sit ~9 °C apart, a steel bench at 65 °C burns, cork moves 6 °C across a 75 °C swing while steel moves 69. Within seconds of being written the suite failed, and the failing assertion was the one that had been overclaimed: it said every plant material beat every petroleum one, and polyurethane foam had the best number on the board. The page had to be weakened from “biomaterials feel warmer” to “one of them matches the best synthetic there is.” That is a truer sentence, and it exists because an assertion refused the flattering one.
Two more fell out of the same suite. A rule change made minerals admissible, and an assertion reading clean surfaces span less than concrete began comparing concrete against itself — the words had held still while the category beneath them changed meaning. And a superlative flipped: foam has the lowest effusivity was true, was asserted, and stopped being true the moment a new material was added. It carried a note saying what to rewrite when it flipped, so the flip arrived with its own instructions.
The sharpest version of the move is to assert what the writing must not say. When two figures were close enough to call level rather than a win, the suite gained a check that fails if anyone upgrades the prose to a victory while the gap stays small — and would announce it if the evidence ever justified the stronger claim. A document that cannot drift toward exaggeration is a rarer thing than a document that is currently accurate.
The tell: find every number in the prose that came from somewhere else, and ask what would happen if the somewhere else changed. If the answer is “nothing would happen”, the sentence is unguarded. Assertions written from the claim catch this; assertions written from the behaviour never do, because the behaviour is fine.
9. The change that never happened
A script replaced a string in a source file, printed updated, and had changed nothing. The pattern assumed six spaces of indentation where the file used four, so it matched zero times and reported success anyway. This happened four times in one day, and one of those silent misses left a build gate in place that passed because it was checking nothing — a rule matching no input is indistinguishable from a rule finding no fault. Another survived a full deploy.
This one sits upstream of the others. They are ways a check goes green while the thing is broken; this is a way the thing was never touched while the tooling said it was. Everything downstream then behaves correctly and misleads: the test passes because it is still testing the old code, the page looks unchanged because it is unchanged, and the conclusion drawn is about the wrong artifact entirely.
The tell is that the edit and the report of the edit come from the same process, which has no way to tell a successful replacement from a vacuous one. A tool that errors on a missed pattern removes the category. Failing that: assert the match count, and read the file back rather than the log.
10. The copy that was not the one you changed
A mobile fix, a leaderboard, a rewritten index page and a corrected count each appeared not to have deployed. Each had deployed. The apex domain was serving edge-cached HTML with a week-long lifetime, so the verification kept measuring an old copy of the right page — and query-string cache-busting was unreliable enough that it sometimes returned the new copy and sometimes the old, in the same minute.
The reverse is worse and quieter. A change can look live because a cached copy happens to match what you expected, and the same session found two versions of that: a Worker whose deployed code no longer matched the source in the repository, where only one of three routes it defines actually answers, and a directory of built output standing in for source that had moved on. In each case the artifact under inspection was real, current-looking, and not the one that had been edited.
§3 is about looking where the light is. This is about looking in exactly the right place, at yesterday’s version of it. The move that works is to verify against the immutable thing the deploy just returned — the specific deployment URL, the commit, the build output — and treat the friendly alias as a question about propagation rather than about correctness.
What found things
Against the ten, the moves that worked were few:
- A different question. Re-running any audit found nothing, twice. Every new instrument — links, then endpoints, then renders, then accessibility — found a class of defect the others structurally could not see, on its first run.
- Assertions from the claim, not the behaviour. A deterministic reasoning engine was given a suite written from its documentation. Two real bugs surfaced in the first minute — one of them dead fallback code that had reported every unreadable input as the framework’s best state.
- Evidence that predates the question (§5).
- Exercise, don’t inspect. Drive the page in a browser, submit the form, buy the product, call the endpoint after the change. The report of the thing is not the thing.
- Promote every failure to a standing check. Each class above ended the day as a build gate: inline scripts are parse-checked because one apostrophe blanked a page; the build fails if a route loses its handler because one did, silently, for weeks. The build now refuses repeats of its own history.
- Verify the artifact, not the alias. Check the immutable thing a deploy just handed you — that deployment URL, that commit, that build output. A friendly domain answers a question about propagation; it is not evidence about correctness, and it will hand you yesterday’s copy of the right page without saying so (§10).
- Let the tool refuse. Prefer an editor that errors on a pattern it cannot find over a script that reports its own success. Where a bulk change is unavoidable, make it assert how many times it matched and fail on zero — the difference between changing nothing and finding nothing wrong is invisible from the log (§9).
- A person looking. The blank map was found by a human glance after five green audits. Keep the glance in the loop; it is the only instrument with no fixed dialect.
In the framework’s terms
A check is an instrument, and a passing check is a claim of coherence. The claim is worth exactly the displacement the check could have registered — a needle that cannot move measures nothing, whatever it points at. Checking again interrogates the same sector of the plane. Checking differently rotates the sector, and the failures, this day, were always in the sector nobody had rotated to.
Limits
One day, one site, one checker — and the checker writing the catalogue is the one that made the errors, which biases the catalogue toward errors that were eventually caught. The ones that were not caught are, by construction, absent. That is not a flourish; it is the strongest reason to expect this list to be incomplete.
Pocket version
- A check that has never failed is unproven — break the thing once.
- Hand every checker a known positive before trusting its negatives.
- Check where the consequence lives, not where the tooling is comfortable.
- Reliability is not validity — find the transformation that shouldn’t matter.
- Test data that knows the hypothesis is barely data.
- Design for the distinction, not the effect.
- Silence is not success — require the positive signal.
These get worked out in the open, at whatever length the problem takes. I do the same thing on a problem of yours — one thing diagnosed and written up plainly, no build. what that costs