I think most people who are interested in how LLMs work get the idea that they are basically token-predicting machines. They take some input, turn it into tokens, 'predict' a next token, stick it on the end of the input, and that becomes a new input. Then it runs the cycle again; repeat to fade…

The bit that I think fewer people understand is the 'sampling' step, where the LLM actually 'chooses' the token. If more people understood it, I think you'd be seeing it everywhere, because its honestly kind of mind-blowing.

The linguistic trap

There's this thing that happens a lot when people talk, write – or maybe even think¹ – about AI or LLMs. As soon as you ascribe 'decisions' to the LLM, you're invoking a subject that doesn't exist. A decision cannot just "happen" – there has to be a thing that's accountable for making the decision. Otherwise, it isn't a "decision" – its just an… outcome – an event, or an incident.

So; inside the chatbot there's an LLM that spits out tokens that get turned into text, and inside the LLM there is a neural network with billions of 'neurons' arranged into layers, doing calculations, sending the results to each other, and out comes a token, and inside the neurons are big matrix calculations running in parallel with some constants pre-calculated by a training process, and somewhere inside all those calculations is… some kind of demon, choosing tokens.

A dice² doesn't 'choose' or 'decide' to land on a one; someone rolls the dice, and that's the outcome. The only decisions were to roll the dice, and to do something with a particular outcome. (Say – rolling a 20 on a 20 sided dice passes a perception check in a game of D&D.³. The decision is using a random number generator to choose an outcome in the game; not the dice making a decision about the game.) A calculator doesn't 'choose' a number to display when its given a calculation. The calculation happens; the result is outputted. A dice was rolled; someone got a pay rise. Neither the calculator or the dice are making 'decisions'. If someone makes a decision to plug the roll of a dice into a company's HR system to choose who gets a pay rise – the dice isn't 'choosing'. The only demon here is the nut who plugged a dice into the HR system.

Because if there is a thing that chooses which token to append to the string of tokens that was fed into it, then that's the same thing that is accountable for the decisions that the LLM makes. It doesn't exist. When you notice someone talking about AI 'choosing' or 'thinking', remember that there is no choice without a chooser, no thought without a thinker. Its just a linguistic trap. I think that naming it as a 'demon' is a useful way to point out – even if only to yourself - that the demon doesn't exist.

Its really hard to avoid the kind of language that 'invokes the demon', because these things are agents. They make choices! They do clever things! I mean, there must be something that wrote all the code in my vibe-coding projects, and I know that it wasn't me! (Some of them use languages I don't even understand!)

The passive voice might be the 'best' way to describe it, but it sounds weird. A decision just 'happened'? Yep – it wasn't 'made', because there is no 'thing' – no demon – that can 'decide'. Its linguistically uncomfortable, but at least its honest. 'Code was generated', and now I've got an app - which is cool – but I'm not the 'builder' who 'built' it. The grammar of the English language wasn't designed to describe 'creations' that don't have a 'creator', or 'designs' with no 'designer'.⁴

But inside an LLM, this sampling step is the 'token choosing' part, which is the part where the real 'magic' of large language models sits. Because that is literally its one job – predicting a token.

I really want to shine the spotlight on it, because I find it fascinating.

The 'perversion' of writing

On Daring Fireball: Anthropic’s ‘Watermark’ Text Adulteration in Claude Is a Perversion of Writing, John Gruber writes;

My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate. That is in fact exactly what they plan to do.

I don't think that's what they plan to do, not because I have some insight into what anyone at Anthropic thinks about anything, but because if this works the way I understand it then… it simply isn't. If something is 'corrupted', then there has to be a 'purity' there in the first place, and that isn't the same place that 'watermarking' is happening.

First off – this part is (I think) spot on;

The exact words we choose when writing matter. I want any LLM I use to choose the very best, most precise words at every single decision point. An obvious constraint that I accept is time and computation. Within the constraint of executing inference quickly, and at a certain cost per token, I want the best words. This constraint matches human writing.

OK - but there's a third constraint quietly⁵ added here; time, computation, and 'matching human writing'. We want the best, most precise words at every single decision point, we want it fast, we want it cheap, and we also want it to match human writing. (If it doesn't, then the whole fingerprinting thing is kind of redundant anyway.)

Gruber continues;

At each decision point for generating the next token, the model makes a choice. With these semantic watermarking techniques, they make different choices for some tokens based on word lists that could be called “green” and “red”. At each decision point, they’re a little more likely to pick a word from the green list than the red list. That doesn’t mean they never choose words from the red list. Just that they’re less likely to than they would if the adulterated marking technique weren’t in place.

It's this point where 'the model makes a choice' point where I think Gruber goes off-track. The model doesn't "choose" – there is no demon inside it making a choice. (Words matter.)

This 'choice' is actually a two-step process, and I'm going to explain why there is no demon in either one of them.

LLM's sampling stage

First, the neural network performs its calculations. For a one billion parameter model, there are a billion calculations going on here. All boiling down to multiplications and additions, all 100% deterministic. The output of all this is going to be a single token, but the final step before that single token is a list of a few hundred thousand numbers; a score for every possible token in the model's 'token vocabulary'.

Technically, these scores are logits – but you can think of them as probability scores.^5b The goal of the model is to design and train it to have the highest scoring token be the "best" prediction. The "very best, most precise words at every single decision point".

Watermarking doesn't even touch this first step. It's purely deterministic; run it a thousand times, and you get exactly the same output from the same model a thousand times. There is no demon here, making (or corrupting) any choices – just numbers being added and multiplied a couple of billion times for every token.⁶

The second step is the actual 'sampling' – the token 'choice'. If there is a demon, then this is where it comes in (spoiler: there isn't a demon here either. I was just trying to make maths sound a little more exciting for a moment.)

If you've heard of 'temperature', you'll already get the idea that simply picking the most probable next token every time… doesn't quite work. It gives you the exact same answer every time, but its also kind of… boring. It feels mechanical, or robotic. "[T]he very best, most precise words at every single decision point" – yes, the best decision for every single token comes out of the super-well-trained model; but this is where we hit against the constraint about matching human writing. There's not even a real need for a fingerprint here – read 200 tokens of zero-temperature LLM output and you can smell the robots. You know that there's no demon there. (Well, you know if you're a writer – if everyone could smell the machines then 'slop' wouldn't be a thing. But these things really do have a stink.)

Turning up the temperature means sometimes picking the second most probable token instead. Instead of picking the one that scores highest all of the time, only pick it some of the time.

How often? Well – remember, we've got a score for every single possible token; the goal was to have the best token be the highest scoring, but we actually got a score for how possible any and everything was. But we know that sometimes, the 'best' isn't actually the one with the highest score. But, if we throw in a bit of randomness at this stage, the whole of the output becomes 'better'.

This is the point where the 'perversion of writing' is already happening; the point where the virtual dice is rolled and entropy gets introduced into the system. This is where a thousand identical prompts now get a thousand different outputs. This is where the demon leaves its fingerprints on the output, making it a better match for 'human' writing.

The argument that fingerprinted outputs are worse ('adulterated', 'corrupted') is saying that by contaminating the randomness by injecting some order and reducing the entropy makes the quality of the output worse.

Gruber's mistake is benchmarking the fingerprinted output against an platonic ideal of zero-temperature, human-like writing. That's a contradiction; you can have zero-temperature, precise token choices at every single decision point, or you can have human-like writing. At least today, you can't have both.

The neural network is the calculator, the temperature-influenced sampling is the roll of the dice. The fingerprinting is just adding some new rules to the dice roll.

Will a thousand identical prompts still get a thousand different responses? Yep. Will those thousand different responses be 'better'? Who knows?

The purity of randomness

Sampling is an event that 'just happens' – there is no 'sampling demon' making a choice. Usually, the token with the highest score comes out. Sometimes, it doesn't. We can't predict it, and its going to be different every single time - its random. The individual token choices don't look 'better' (if they did, it would surely be baked into the determinsitic model) – but the paragraphs do. And, if you kind of squint, now it kind of looks like there's a demon writing them instead of a robot.

Apparently, what billions of dollars of investment and the dedicated efforts of some of the smartest experts in the field have come up with as the best way to do that is… to throw in a sprinkle of randomness. The essential ingredient that generates something like 'human writing' isn't about the architecture of the neural network, or the weights from the training data – it's chaos, extracted from an entropy source. (Maybe thats where the demon is?)

But there is no writing without a writer, and an LLM – or the roll of a dice, or a million monkeys with a million typewriters – will never be a writer. It can produce something that passes as art, but it is not an artist. Of everything in the writer's office, an LLM will always be more closely related to the word processor than the fingers on the keyboard, or the figure in the chair. That little dash of randomness – a sprinkle of chaos into the otherwise deterministic output – just makes it feel a bit more like there's a demon inside the machine, cosplaying as a writer. Its a convincing component of the illusion that we all seem to kind of get carried along with – at least linguistically. That illusion that there's a thing that we're talking to. (I wrote something almost exactly four years ago about the weirdness of how we should treat a thing that doesn't really exist but is sending us replies anyway.)

So; that's what we need to be benchmarking the 'adulterated' fingerprinted output against. Not a kind of platonic ideal of the perfect superintelligence – the digital god whose output would be blasphemy to edit, but the one that's been injected with random noise, that makes every response to the same question a little bit different.

Why I kind of semi-secretly hope Gruber is actually right

If you ask the same chatbot the same question a thousand times, is there a 'best' answer out of a thousand different responses? Probably. Can we figure out which one it is? No – because if we did, we'd put that into the model. But we do know that adding a little bit of chaos seems to get us closer to 'best', most of the time. Does it matter if we impose a little bit of order to the chaos, adding a special secret pattern?

I honestly don't know. But it would be cool as fuck if it did, because it would mean that there is some sort of magical 'purity' inside pure chaos. It's weird enough that injecting chaos is already how we get the "best" output from our LLMs, but if it turns out that interfering with the random number generator and making it slightly less random for the sake of an identifiable watermark also makes the output qualitatively worse… Well, it probably wouldn't be the weirdest thing about AI – but it would have a good chance of making the top five⁷. Maybe we'll see quantifiable benchmarks for how long Claude can keep running a task without losing focus, with and without the entropy demon leaving fingerprints all over its output. Maybe the demon's choice is special, and there's a magical quality that comes from the purity of injecting the most random numbers.

I'm not saying that magic exists… but I'm also not not saying it.

I'm just not betting all-in against there being something magical about pure randomness. I'd love to see evidence for it. But still – I am sceptical that reducing the 'purity' of the probability sampling is going to somehow be a perversion of the output of an LLM.

(Which is still not 'writing'.)

Depends on your view on the Sapir-Whorf hypothesis ↩
Yes, I know that a single dice is called a 'die'. ↩
Nat 20, let's go. ↩
It might be that there's a reason for that; I don't know. Douglas Adams's "Is there an artificial God?" speech goes into this idea much better than I ever could; Man the maker looks at his world and says 'So who made this then?' Who made this? - you can see why it's a treacherous question. ↩
Yes, I said "quietly". I also liberally use em-dashes, and I type every single one of them by hand with ⌥+- – fuck you, I'm not a robot. ↩
Logits get converted to probabilities by the softmax step, which is also where temperature comes in. ↩
*Sigh*. Ok, there's also rectified linear units, which you could argue do something with booleans that isn't quite 'just addition or subtraction'. You can have that argument with your favourite LLM if you like. The point is that if one multiplication or addition operation doesn't have a demon inside it, you don't suddenly manifest a demon because there is a chain of a billion of them. ↩
Its not just the sampling point where adding randomness makes AI work better; the training process of any model has to start out with a random seed – if all the weights are set to zero, all the neurons in a layer are essentially identical. They need some entropy before the training process can actually start working properly, otherwise it just gets stuck. Perhaps there's a door open here for the demon to appear in a sequel… ↩

Some Random Nerd

The linguistic trap

The 'perversion' of writing

LLM's sampling stage

The purity of randomness

Why I kind of semi-secretly hope Gruber is actually right