So let me get to the nut of this thing before I do my usual meandering. Recently, someone accused me of using AI to write this old post from about a year ago, specifically highlighting the section about Ta-Nehisi Coates. I replied by saying that the post contained no AI writing; nothing I publish has been written or edited by an LLM. My accuser proceeded to say that the AI detection tool Pangram flagged that portion as 100% AI written with high confidence, a result I replicated. (Pangram report.) As the man who wrote that piece, I knew that I wasn’t guilty, but I also knew that wasn’t going to fly with other people. I was sufficiently annoyed that I paid for a Pangram license and found that, when I entered the entire essay, Pangram cleared it as 100% human written with high confidence. (Pangram report.) That is to say, the essay of about 5,000 words was declared 100% human written with high confidence even though it contains the section of about 300 words which was declared 100% AI written with high confidence. Subsequently, I’ve found that I can produce the same sort of inconsistency in the opposite direction - I can break the section that was called 100% AI into pieces that are then called 100% human written. (Pangram report of an example.) So pieces that Pangram call 100% human written are part of a larger piece that Pangram calls 100% AI written which in turn is part of an essay that Pangram calls 100% human written, a Russian doll of contradictory results.

If nothing else, there’s a fundamental inconsistency here: it doesn’t make sense for a system to say that it has high confidence that an essay is 100% human written but also that a piece of that essay is 100% AI-written, again with high confidence, and also to say that the piece itself has subsections that are 100% human written…. These claims are mutually exclusive and must necessarily erode our confidence in the instrument. Experimenting, I’ve also found that I can pretty reliably get Pangram to declare large sections of text to be AI-written or human-written depending on the larger textual context I place those sections in - that is, I can take AI-generated text and embed it in human-generated text and have Pangram declare it 100% human, and I can take human-generated text and embed it in AI-generated text and have Pangram declare it 100% AI. I’ve also found that it’s pretty easy to induce false positives, that is, to write texts myself that Pangram identifies as AI-written. All of this makes me hesitant about the Pangram tool, and in particular the way many people use it, as a one-shot gotcha machine.

In a purely self-interested sense, the most obvious thing for me to do would be to point to this 100% human written outcome for the whole essay, say “See???,” and get indignant about having been accused. But I think there’s a lot to talk about here, and I think that people who write for a living have to have these conversations. These tools are being used right now, and with consequences, and we have to work this stuff out. So here goes.

The percentage meter seems clearly broken. One frustration of discussing Pangram results is that many people seem to think that the percentage that’s expressed is the confidence level - that is, that a 100% AI result is saying “we are 100% sure that this text is AI generated.” But that’s not what’s being measured there. The percentage is supposed to indicate what portion of the text is suspected to be LLM written. The degree of confidence is flagged there underneath “AI Generated,” although annoyingly the flag only appears when confidence is high, with no “Confidence Low” flag that ever appears, in my experience. This is all fine and good - the percentage AI generated and confidence numbers are different things that should be easy to interpret. The first problem is that many, many people are clearly interpreting “100% AI” to mean “100% confidence.” The second problem is that the percentage meter just does not appear to work! Anecdotally, a really suspiciously high portion of all Pangram outputs are 100%, whether 100% AI or 100% human. This is suspicious not only because a lot of LLM writing that gets published is likely hybrid, integrated with a writer’s own words, but also because the basic processes through which detectors like Pangram function seem likely to produce fewer polar outcomes. And a detector that spits out a lot of 100%s seems easier to break.

So let’s break the percentage meter. This is easy to do. Here’s a paragraph I’ve copy and pasted from a piece I wrote in 2017, which I hope you will accept as given was not written with the help of an LLM. I have appended three sentences to the end of the paragraph that were written by ChatGPT, using the original paragraph as a prompt, highlighted in red. The human-written, pre-LLM section is 239 words, while ChatGPT’s portion is 71 words long.

The gambler's fallacy is when you expect a certain periodicity in outcomes when you have no reason to expect it. That is, you look at events that happened in the recent past, and say "that is an unusually high/low number of times for that event to happen, so therefore what will follow is an unusually low/high number of times for it to happen." The classic case is roulette: you're walking along the casino floor, and you see the electronic sign showing that a roulette table has hit black 10 times in a row. You know the odds of this are very small, so you rush over to place a bet on red. But of course that's not justified: the table doesn't "know" it has come up black 10 times in a row. You've still got the same (bad) odds of hitting red, 47.4%. You're still playing with the same house edge. A coin that's just come up heads 50 times in a row has the same odds of being heads again as being tails again. The expectation that non-periodic random events are governed by some sort of god of reciprocal probabilities is the source of tons of bad human reasoning - and journalism is absolutely stuffed with it. You see it any time people point out that a particular event hasn't happened in a long time, so therefore we've got an increased chance of it happening in the future. But unless the process has some actual mechanism of correction or reversion, the mere passage of time changes nothing. Droughts do not make rain “due,” losing streaks do not create future wins, and a long period without a crisis does not by itself make a crisis more likely tomorrow. The past can tell you something about the underlying probability of an event, but it cannot compel randomness to balance its books.

So, ideally, Pangram would flag this as 23% AI generated; that is, indeed, what the exact percentages are by wordcount. Here’s what actually happens (report):

So, yeah, not great. That passage is three-quarters human written, dominantly human written, and yet Pangram says with high confidence that it’s 100% AI. I know there are a lot of people who would endorse a “one drop” rule with LLM assistance, but Pangram itself is saying that its technology operates a certain way, and it doesn’t appear to operate that way; certainly you can break it quite consistently. Pangram’s own examples suggest that it can return mixed results with a specific percentage. Look, they even give you human vs robot %:

And yet in dozens of trials I’ve never gotten a mixed report like this. They’ve all been either 100% human or 100% AI, even though I’ve been Frankensteining human and LLM writing together constantly for this exercise. For whatever reason, Pangram seems to want to find 100% results, even though users are clearly trusting it to detect what portion of a text is LLM generated. This all seems quite bad to me! This technology is being used in a way that has severe professional consequences for some people. It should work the way it’s supposed to work; it should offer a “percentage AI written” that’s actually a reflection of how much of the text was LLM generated, and if it can’t do that, they should drop the percentiles and just give a quantitative, numeral confidence figure.

Here’s where we can do a “you try it at home” exercise, if you have a Pangram account. Take human written text, append LLM-generated text, and see if you get a 100% AI result. With enough naked ChatGPT text you’ll get there. And if you’d like, you can try and go the other direction - if you have a long enough human-written piece, you can stick LLM-generated text somewhere in it and see what you get back. I think you’ll find that it’s easy to pass such text through as 100% human. This additionally means that you can easily get specific passages into texts flagged as 100% human or 100% AI, again fairly consistently. I just don’t think that’s ideal, and it disturbs me how cavalier some people are about it.

I have no idea what you do with conflicting results like these. I suppose many would say that the much longer overall essay has many more datapoints and the whole-essay analysis should thus be considered more authoritative than the 300-word passage analysis. But I can assure you that this won’t stop people who don’t like me from declaring some sort of victory! This is what I worry about with this whole controversy - we’re very likely to end up with everybody confirming their priors. People who don’t like me will say that the short passage analysis proves that that portion was LLM written, while people who do like me (I assure you, a few such people exist) will say that the whole-essay analysis has more data and is obviously the right level of perspective. Nobody will be convinced of anything. The nightmarish possibility is that people will start cutting up essays that pass Pangram’s test into small pieces, looking to find parts that get flagged as AI, so that nobody ever feels confident about anything.

If 50 words is not sufficient to have confidence in a result, you shouldn’t be able to put 50 words in the detector. Several people on Notes said some version of “Well of course nothing is 100%, but the more data that Pangram has, the more accurate it will be, and that 300 word chunk that got flagged just wasn’t long enough.” But the minimum amount of text Pangram will accept is 50 words, so it should work on 50 words. I’m obviously not equipped to discuss the technical specifics here, but since I heard some version of this many times, I want to say clearly: if a particular selection of text provides insufficient data to get accurate results, you shouldn’t be able to use the detector on that selection of text. At the very least, if the confidence in a conclusion is low, that low level of confidence should be prominently displayed in the results.

Even a very small error rate will produce false outcomes regularly given enough repetitions. I’ve written more than 5000 blog posts, newsletter entries, and freelance essays in my career; I can’t be sure because my old fredrikdeboer.com blog was lost years ago, but it may be closer to 6000. But let’s narrow down: since ChatGPT was released to the public in 2022, I’ve published about 1200 posts here, and those are the ones that are relevant. Now, one frustration I have with this discussion is how often people quote error rates that come straight from Pangram, which doesn’t seem particularly rigorous to me. But let’s be generous and use the company’s own numbers. Pangram reports a false positive rate of 0.19%, and claims something closer to one in ten thousand on academic essays. (That is, the kind turned in by college students, presumably Pangram’s most common usage.) At those rates, 1,200 posts of mine yields an expected two or three spurious “AI written” verdicts - small, but not zero. But! Even that considerably understates the exposure to potential error, because “newsletter post” or “essay” or “document” isn’t the only unit these tools actually operate on. Pangram returns sentence-level “highlights” and a separate AI-assist judgment, so every paragraph becomes its own opportunity to be wrong. There’s thirty paragraphs in the post that got flagged. Let’s say my average is half that. At that rate, my post-ChatGPT archive represents something like 18,000 chances to get flagged, and at one in ten thousand you would expect a few passages in there to come back marked as machine-written no matter what I did or didn’t do. This is why a “one drop rule” doesn’t work. An accusation should come from a large-scale investigation of a writer’s corpus. But that’s hard and time intensive and, most importantly, doesn’t fit with the desire to own one’s enemies online, which drives a lot of this stuff.

And, of course, “Freddie deBoer” is not the right unit of analysis, but rather the whole community of people who write. That community produces a lot of text, and so even at 1 in 10,000, there will be an immense number of false positives being generated, if you’re testing everybody. And as I suggested above, some people’s natural writing style is more likely to be flagged than that of others, even with no LLM assistance. Whatever Pangram keys on is going to get keyed on over and over again; there’s inevitably a bias against whoever happens to write near the boundary. My own writing doesn’t appear to have that problem, but other people’s will. Can’t you imagine the outrage if it’s eventually found that, say, Chinese international students are more likely to produce false positives with their writing, thanks to underlying second language consistencies? Those consistencies exist; back when I taught Chinese undergrads at Purdue, they would very often write “now a day” instead of “nowadays,” including when they were writing live and unscripted in the computer lab. It all worries me.

Despite Pangram’s reputation for avoiding false positives, it’s very easy to induce them. A lot of Pangram enthusiasts boast that the system is very unlikely to produce a false positive - that is, it’s unlikely to say that actually human-written writing was written by an LLM. Well, here’s a hundred words that I wrote myself that Pangram flags as 100% AI written with high confidence.

Free school lunch appears compassionate, yet its underlying logic is unmistakably collectivist. Once the state assumes responsibility for feeding children, it subtly redefines nourishment as a public entitlement rather than a private obligation. This normalization expands expectations: first lunch, then housing, healthcare, transportation, and ultimately the centralized distribution of all essential goods. Citizens become conditioned to view government provision as both natural and necessary. Individual responsibility erodes, markets lose legitimacy, and bureaucratic planning gains moral authority. Thus, the cafeteria tray becomes more than a meal; it becomes a symbolic gateway through which communism quietly enters democratic society, one sandwich daily.

It took me maybe fifteen minutes to achieve this outcome. I could certainly do it again. For this exercise I deliberately attempted to avoid the most cliche signs of AI writing - “It’s not X, it’s Y,” the term “load bearing,” the word “delve,” etc. And yet getting this outcome was still quite easy. Again, if you have a Pangram license, I encourage you to give it a try yourself.

… and yes, the ability to come up with “inorganic” false positives does matter. Defenders of Pangram tend to dismiss these artificial exercises - “well, sure, you can make it give you a false positive if you try, but people don’t do that in the real world, so it doesn’t matter.” I find this wrongheaded. If I can reliably produce human text that trips the model, then I’ve learned what the model is actually keying on, and it isn’t human or AI authorship; it’s keying on a set of surface regularities that correlate with human or AI authorship in the Pangram training corpus. In other words, the ability to induce a false positive points to the fact that Pangram results depend on textual artifacts, and crucially the textual artifacts I’m exploiting on purpose are ones that some people use accidentally, by disposition - second-language writers and writers who write in a (shall we say) formulaic register are obvious potential victims here. When people talk about the tolerances of the Pangram system, in regards to deliberate false positives, they tend to say things like “You don’t expect a bridge to be able to hold a weight designed to overload it.” But that’s not equivalent. The right analogy isn’t a stress test that a bridge passes at loads no truck will ever carry; it’s discovering that the bridge fails at a load some trucks routinely carry, simply by virtue of being those trucks, and being told it doesn’t count because you drove the truck onto the bridge with the express purpose of overloading it. If you follow me.

Inevitably, a system like Pangram doesn’t really identify authorship directly, which would be impossible through only textual evidence; instead, such a system is always responding to surface features of prose that a human can intentionally or accidentally reproduce. Pangram and its defenders insist that it avoids Type I error. But the fact that I can induce Type I error, even if artificially, should undermine our confidence in that claim; surely there are some human beings out there who naturally write the way I write when I’m trying to intentionally trigger Pangram. I mean, as ugly as that paragraph is, does it really seem wildly out of the ordinary to you? It doesn’t, to me. I’ve taught writing to hundreds of students. Trust me when I say that some of them right in styles that are… unique. Is it possible for a human to accidentally write that paragraph? Sure. And again, with Pangram increasingly being used as a gotcha in internet warfare, the possibility of a false positive becomes more and more troubling.

The piece in question is an almost uniquely unlikely candidate to have been AI written. I didn’t use any AI assistance on that post and never do, on anything, and I could give you the whole story about how this is the only thing I enjoy doing, how I have dealt with graphomania for my entire adult life, etc etc. But of course this won’t mean much to you because I would have obvious professional and social incentives to lie about using AI if in fact I was guilty of doing so. Still, I think you’ll agree that this would be a weird piece for me to use AI on. For one, it’s about as pure an exercise in stylistics as I can imagine writing; I was inspired by my feeling that while there’s plenty of writing about writers out there, but very little of it talks about what we actually love in the writers we admire. And I was working in the tradition of this piece from 2013, where I attempted a similar task. The thing is, the piece is weird even by my standards; it includes phrases like “a Lisa Frank sticker crying blood” and “Black gay Susan Sontag grandchild” and so on. And the flagged portion on Ta-Nehisi Coates includes the phrase “the universal rabbi of liberal virtue.” If LLMs tend to produce featureless, formulaic prose, then I don’t know how one would come up with this piece. I’ve been amused trying to think of what LLM prompt would result in this piece of writing.

My accuser, for the record, said that he doesn’t think the whole thing was LLM generated, just the passage about Coates. And it’s just like… why? Why would I do that, exactly? The piece was already considered way too long by the standards of many of my readers. I could have deleted the Coates section at my whim if I wanted to. The whole point of this exercise was to look deeply into the work of writers I like; if I didn’t have particular feelings about Coates’s corpus, I just would have left him off. And as that 2013 piece shows, I’ve been thinking about his particular strengths as a writer for a long time. Just doesn’t track.

Another thing I think is relevant is that the piece had just about zero selfish reason to exist; I wrote it for no other reason than to write it. It’s the very definition of a “one for me” piece, a weird, distaff textual experiment that even many of my most devoted readers didn’t particularly like, an expression of an obsessive interest in prose style, conceived at an aperture that very few people care about. And my instinct that this was a purely “for me” post proved correct - I knew it wouldn’t result in many views or subscriptions and as predicted it did not. A year later it’s attracted fewer than 40,000 views; average posts do about double that number. According to Substack’s metrics, thirty-five of those readers chose to subscribe at the free tier after reading it; I’ve published nineteen posts for this newsletter that have attracted more than ten times as many free subscribers as that. Two readers chose to pay for a subscription thanks to this post, if Substack’s metrics are to be believed; the top ten revenue-generating posts in my history all brought in more than a hundred paid subscribers. I also can’t be said to have been motivated to just get something out there for my readers, as I published seventeen pieces that month and four that week. There was no reason for that piece to exist other than that I wanted to write it.

It’s written in an unusual register, but that’s not unusual for me - here and here and here and here are other pieces that are not in my usual voice. So, you know, man… I don’t know. Even if you want to come at me in this way, this particular essay seems like a weird choice on many levels.

Here’s the most important thing, for me: most writers would not be able to respond as I have done here. I mean a few things by that. First, I ponied up for a paid Pangram account to investigate all of this, and many wouldn’t be able to justify the expense. (It’s really deeply unfortunate that people may feel the need to pay Pangram just to clear their names.) More importantly, I’m just not someone who’s particularly vulnerable to these accusations. I have twenty years of experience, and in those twenty years I’ve always had a reputation for being unusually prolific. I’m very confident in my abilities and very precious about my work. I have no publication or boss to worry about. I’m performatively indifferent to the opinions of my professional peers. And I’m also pathologically recalcitrant, antisocial and defiant in a manner that has hurt my life and my career in innumerable ways but which is occasionally useful, as it is here. I’m also very used to people taking swings at me. I didn’t panic about this accusation because I knew I didn’t use an LLM on that piece, but I also never panic when someone comes for me because, well… they’ve been coming for me for a long time.

Most writers don’t have the luxury of any of that. And I really worry for the young, the inexperienced, and those who lack my psychotic disorder-enabled level of self-confidence and my life experience. Some 22 year old could catch an allegation like this and find themselves completely unequipped to respond, and that’s really depressing. And look, I fucking hate AI writing. I hate what it’s doing to the only thing I really care about. I hate writers for using LLMs to do their work for them. I hate Silicon Valley for doing this to my profession. I really, truly, deeply do. This piece is inevitably going to be represented as being soft on AI writing when it’s not. If I can use a deeply ill-advised analogy here, it’s like the unreliability of sexual assault accusations that stem from “recovered” memories, which I’ve been writing about. Several people accused me of indifference to child sexual abuse because I have pointed out that the research record is damning when it comes to the reliability of recovered memories. But that presumes the very idea that I’m disputing - if it’s the case that recovered memories are in fact not real memories at all, as the research record strongly concludes, then criticizing the idea of recovered memories can’t harm the effort against child sexual abuse. Right? Well, it’s the same thing here - pointing out the weaknesses in Pangram is only hurting the cause in fighting undisclosed AI writing if in fact Pangram never gets it wrong. But some degree of error is literally inevitable.

Look, I think Pangram can be a genuinely useful tool as part of a broader effort to figure out what’s being written by machines, which is unfortunately a thing we have to worry about now. And I think the company has made a number of smart decisions - the choice to abandon perplexity-based detection, which punishes anyone whose prose tends be predictable, seems like the right one, from my amateur perspective. They’ve also been unusually public about how their proprietary system is built and evaluated and stress tested. But I’m sorry to say that we are going to be living with this problem for the rest of our lives, and right now we’re trying to fight that problem using tools built in the last eighteen months against models that will be replaced within the eighteen after that. It’s just very early. And the mathematics of the situation guarantee that some quantity of false positives will always exist; you can drive the number down, and Pangram appears to has driven it pretty far down, but you can’t drive it to zero. And a rate low enough to be statistically negligible is not the same as a rate low enough to be morally meaningless, especially when a specific person’s reputation is what’s at stake. I’m not saying not to use Pangram. But you have to use it with the epistemological modesty appropriate to a technology this young, and it’s never going to be the last word on anyone.

Let’s face it: writer culture is, at bottom, a culture of resentment toward other writers. So much of what defines the professional and social cultures of writing lies in the petty grievances and jealousies that writers always harbor towards each other. (Aren’t you the guy who always says professional writers are too clubby, popularity-seeking, and obsessed with being liked? Yes I am! I’m afraid it’s both, at the same time.) Writers fixate on the columns they should have gotten, the woman whose novel sold for more than it should have, the kid on Substack with more subscribers and less talent…. Lord knows there’s a lot of that in me. A machine that promises to reveal which of your rivals are frauds is a tempting weapon to deploy. But we have to be careful and be fair.

I started this piece out of annoyance and a desire to defend myself, but I found myself genuinely deeply interested in the linguistic and statistical and ethical puzzles that’s embedded in all of this. I tried my best to be fair and useful. And now, inevitably, some are going to say that the fact that I put so much effort in this must mean that I’m guilty and trying to save face. Don’t you just hate it here? Isn’t this all such a fucking drag?