Do LLMs “Write?”: The Mechanics of AI Writing vs. Human Writing
A Professional Writer’s Guide to Identifying AI Writing (Pt. 2)
This essay is pt. 2 of a series on how to identify AI-generated writing. For pt. 1, check out The Writer’s Contract.
If you want to understand how LLMs “write,” it helps to start with a basic understanding of how LLMs work more broadly. I know you’re probably here for the listicle-style writing tips, and I swear on my life that a cornucopia of them is headed your way, but stay with me on this one. Understanding next token prediction, the foundational function of all LLMs, is really helpful for wrapping your head around why AI writes the way it does and where its writing capabilities begin and end.
I am not an expert in this subject and would never claim to be. This essay is not an authoritative, expert account of LLM mechanics. It is, however, a thoroughly researched explanation that has helped me make sense of how I engage with LLMs and can help you too.
So without further ado, let’s get into it. What makes the way LLMs write different from the way humans do.
How Humans Write
Speaking generally, when you or I write a sentence, our process quickly and subliminally goes a little like this: we form an idea (sometimes more of just a pre-verbal lump than an actual “idea”) that we want to express to someone else, and with that idea in mind, we sort through the filing cabinet of words we know to find the best combination possible for making that idea legible to someone else. A lot of the time, this process is happening autonomically in a way we don’t really detect. However, if you’ve ever had the experience of writing being “hard,” it’s probably because you’ve encountered the conscious version of this process where you are actively fishing around for the right words or the right sequence of words, and you just can’t quite land one.
In the human brain, that challenge takes on a certain urgency because being understood is consequential to us. For us, picking the right words and putting them in the right order can feel stressful and threatening because our survival-motivated brains register being misunderstood as a social threat that jeopardizes our place in the herd.
Accordingly, you can think of human writing as defined by three particular things: a starting idea, a selection process for finding the words to communicate that idea, and a sense of emotional importance that motivates us to communicate in more precise and effective ways.
Professional writing, the type you consume in magazines, books, and journals, also includes a recursive editing process where writers hack away at their original creations to try to find even better words and arrangements, but for the purpose of this piece, all you need to keep in mind is that idea + emotional stakes + word selection process = writing.
How LLMs Write
What AI does when it “writes” bears little resemblance to this process.
In layperson’s terms, LLMs like ChatGPT and Claude perform one, and arguably only one core function: next token prediction. Essentially, they “write” by taking your prompts, breaking them into tokens, converting those tokens into numbers, throwing those numbers through many layers of learned calculations, and outputting a probability distribution over possible next tokens so that they can pick the best one.
The way it “learns” (gets trained) to do that is by being repeatedly adjusted while it sorts through huge amounts of training text. These adjustments make it so that, when given different token sequences, the model becomes increasingly good at assigning a high probability to the token that actually came next in the training data. With new and improved internal parameters (numerical weights and biases), the model functionally encodes patterns in grammar, word choice, style, facts, and language structure so that when you open up your chat interface, the model behind it can use learned numerical patterns to score possible next tokens, pick the highest probability one, and then do the same thing over and over and over again until you have a response.
TL;DR—no matter how much it looks like reasoning, explaining, thinking, or writing, all LLMs really do is very advanced statistics homework, over and over and over. It’s just recursive pattern completion that involves no human-like relationship to meaning, intent, or truth.
The reason I want you to understand this is that, because all LLMs are doing is prediction, the way they write does not and cannot include our three ingredients for human writing. LLMs don’t have ideas. They don’t have truths. Even when you give them a prompt, the prompt is not an idea or a truth; it’s just a mathematical sequence of tokens that would likely be followed by the sequence of tokens it gives back. At no point in that process is there an idea, a pre-verbal lump, or a premise. There is never an idea, fledgingly and unformed but real.
There is also no word selection process. While next-token prediction seems like the same thing as finding the right word for your sentence, when you fish around in your brain for words, you implicitly have a goal for what that word must do, and you are indexing your entire lexical database to find the one that best achieves the goal. But AI has no goals or conscious lexical inventory. It has what is functionally a very large map of mathematical relationships between words, but it has no understanding of those words’ relationships to the world (because…again…it does not experience the world), so the idea of a “best” or most precise word is beyond its comprehension. It’s not even trying to predict the most probable word statistically. Just one that’s probbaly not wrong.
Perhaps most importantly, there are no emotional stakes. What drives you and me to write, and to write well, is a particular emotional discomfort that is unique to being entities cursed with consciousness. We try to say what we mean exactly as we mean it because we have entire interior universes inside of us that can only be known if we put the right words for describing them in the right order, and that is both daunting and, if you ask me, profound. The mechanics of it all aside, the stakes are what make writing interesting. And for AI… there are none. You can snark at it all you want (and I do), but it’s of no consequence to your LLM if you’re rude because it communicated poorly.
One more technical thing. It is easy to assume (and it is partially true) that if AI is writing based on probabilities, then what it’s producing must be the most common patterns in human writing. Think: standard sentence structures, familiar transitions, genre-specific norms, etc. It is true that models reflect what appears most often, but that is not the only thing determining which tokens models spit out. While they may not be searching for the most accurate way to say something, they are programmed to avoid being obviously wrong. In some sense, AI is producing the “average” of all the human inputs it is trained on, but it is more likely than humans to be generic, balanced, and low-risk in its choices. You’ll understand the significance of this as we get into the series, but just keep it in mind.
There is no amount of prompting that can make AI understand or care about what it’s “writing.” It doesn’t know what it’s doing, has no destination, no determination to express something well, and no understanding of what it’s expressing. LLMs produce text sequences that resemble human writing but are fundamentally not produced by writing. And accordingly, even the best-disguised prose will often still give off this ephemeral uncanny valley feeling like a pencil drawing that almost resembles photography, but just still… isn’t. It won’t reach for the right word because it has no motive to, and it wouldn’t dare say something risky because it can’t.
All the things about human writing — the intimacy, the frustration, the familiarity, the honesty, the embarrassment — AI cannot give them to you, not only because it’s a machine writing but because it’s not writing at all.