This is also why LLMs are called auto-regressive. They can keep doing multiple forward passes over their own output until a <stop> token is generated.

This is a simplified overview of where I'm skipping RoPE, the MLP layers in between, token sampling at the end, and much more.

Where @ is the dot-product of two matrices.

In reality, multiple LLM conversations are processed in parallel. So inference is batched—where we process \(B\) chats concurrently. This means our input sequence \( X \in \mathbb{R}_{B \times N \times d}\).

Work the math out on paper to verify it tracks.

In our Python code, just the transpose arguments change:

Only, there's one trouble with our implementation of attention: it reads too much data from memory. Let's look at a single matmul, the \(K = X.W_k\). Companies that serve models will allow you to chat with them for up to 200k or so tokens. For a single K@W_k matmul, it looks like this:

Notice that the output is another \(\mathbb{R}_{B \times N \times d}\) tensor.

As established in the matmul cost section, to compute each \(O^b \in \mathbb{R}_{N \times d}\), we need \(d(N+d)\) memory reads and \(2Nd^2\) compute operations.

For a batch size of \( B \) (number of concurrent conversations), we get:

1. Floating-point operations: \(2BNd^{2}\).
2. Memory accesses: \(Bd(N+d)\).

Assume N to be roughly 200k, and d to be 8192 (most common outside frontier labs). Meaning that to generate one token for a single user, we need 26 trillion floating-point ops and 1.7 billion memory accesses. This is with the tiled matmuls. That's way more compute ops than memory reads. In fact, we're doing four orders of magnitude more compute than memory accesses. The next batch of input will have to wait tens of thousands of cycles for the GPU to finish with the current batch.

On diagramming the above matmuls out on paper, you'll notice a key detail— we're wasting far too many resources to re-compute the matmul products for tokens that were already processed in a previous iteration.

Recall that LLMs are auto-regressive. They:

1. Take a list of tokens \(X\), do a bunch of matmuls.
2. Repeatedly do attention(X, weights) at L (for L layers), and generate a new token \(x\)
3. append \(x\) to \(X\) (the chat thus far).
4. Put the output of 3 back into step 1, until a "STOP" token is generated.

To avoid re-processing the entire chat history again for every new word, inference engines will cache the \(K,V\) pairs for reuse.

Reducing Compute with KV-Cache

The intermediate output on every chat, namely \(K\) and \(V\), is cached at every layer, and stored in a region of VRAM called the KV Cache. Inference engines like vLLM allow programmers to decide what percentage of VRAM should be pre-allocated for this.

For a batch size of \( B \) (number of concurrent conversations), we get:

1. ~26.2 million memory accesses
2. ~52.4 million ops

Meaning that for every memory access made, we need only perform two operations rather than 10 thousand. For the entire batch, we're doing 2*B operations per memory access. This is fantastic! Now, let's pull out the spec-sheet for the fastest GPU available and figure out how many tokens we can generate per second (and for how many users).

How much does a token cost?

Let's take the NVIDIA B200 as our leading example for the remainder of this. From a web search, you'll find that it has the following specs:

1. Memory bandwidth: 8 TB/s (Or \(8*10^{12} \) bytes accessed per second).
2. Compute intensity: 4500 TFLOP/s (Or \(4500 * 10^{12}\) bytes crunched per second).

See that? A Blackwell class GPU can crunch bytes 562 times faster than it can load them. Put differently, to get the most out of such a chip, we should be doing 562 computations for every byte loaded. Any more, and we have memory bandwidth sitting idle (e.g: without a KV-cache). Any less, and we have compute cores sitting idle.

Currently, we're doing 2*B compute-ops per byte read. So, how many users should we serve to fully exhaust a B200's compute and bandwidth budget?

\( 2B = 562 \implies B = 331 \)

With a single NVIDIA B200 GPU, we should be serving 331 users concurrently to get the most out of our investment. Of course, this is a theoretical ceiling. In reality, VRAM is limited. We'll have to squeeze the model weights in there along with the huge KV-cache.

How many users can you serve realistically?

We'll assume a 32B dense model, as they've have gotten quite good for production use and a B200 can comfortably serve them. This could be a Gemma, Qwen, DeepSeek, whatever.

That's more VRAM than our GPU has!

Since both happen in parallel, the compute cores are idle 98% of the time.

Every 24ms, we generate B=6 tokens. For 1s (=1000ms), we generate roughly 250 tokens for 6 users, or about 40 tokens per user per second.

Assuming the LLM output is meant for reading (unlike, say, building SQL queries in the background), 40 tokens per second is beyond most people's reading speed.

Dollar cost per user

This largely depends on whether you own or rent your hardware. At $40,000 per B200, your lifetime cost per user is 40_000/num_users.

In the 100% duty cycle case (worst for cost), that's 6k$ per user. Realistically, serving 300 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill.

If you rent the GPU, the cost is more straightforward. At an hourly rate of $4, your hourly cost per user is 4/num_users. For num_users=300 you get an hourly rate of about $0.013 per user, or $9.36 per month.

As an AI company, you'll have more than one GPU (I pray). For model-sizes that span multiple GPUs, our math is still directionally valid, but the use of napkins is ill-advised.

Backmatter