The Architectures
Over the last few years, research on large language models has exploded, and with it came a wave of new architectures, techniques and design tricks. The variety of models available at the end of 2026 is far greater than what existed only a year ago.
The idea of this study is to understand how these different architectures affect performance, what each of them gains and what each gives up. These are exactly the questions an AI engineer faces when designing a solution around a model. All the models used here were selected to run on a computer with an 8 GB GPU, and together they try to cover the range of architectures found in today’s open-source landscape.
| Model | Parameters (total / active) | Architecture | Weights | Date of Release |
|---|---|---|---|---|
| Llama 3.1 8B | 8.0B / 8.0B | Dense, attention (GQA) | Q4_K_M | Jul/2024 |
| Llama 3.2 3B | 3.2B / 3.2B | Dense, attention (GQA) | Q8_0 | Sep/2024 |
| Qwen2.5 7B | 7.6B / 7.6B | Dense, attention (GQA) | Q4_K_M | Sep/2024 |
| Qwen3-VL 4B | 4.4B / 4.4B | Dense, attention | Q8_0 | Sep/2025 |
| Qwen3.6-35B-A3B | 35B / 3B | MoE, Gated DeltaNet + attention (30 + 10 layers) | UD-Q4_K_XL | Apr/2026 |
| Gemma 3 4B | 4.3B / 4.3B | Dense, local + global attention | Q8_0 | Aug/2025 |
| Gemma 4 E4B | ~8B / 4.5B effective | Dense, attention, per-layer embeddings | UD-Q4_K_XL | Apr/2026 |
| Phi-4-multimodal | 5.6B / 5.6B | Dense, attention | Q4_K_M | Feb/2025 |
| gpt-oss-20b | 21B / 3.6B | MoE, local + global attention | MXFP4 (original) | Aug/2025 |
| Nemotron 3 Nano Omni | 30B / 3B | MoE, Mamba-2 + attention | UD-Q4_K_XL | Apr/2026 |
| LFM3 1.7B | 1.7B / 1.7B | Dense, short convolutions + local and global attention (13 + 5 + 6 layers) | BF16 (original) | Oct/2026 |
| Ternary Bonsai 2 27B | 27B / 27B | Dense, recurrent + attention (48 + 16 layers), ternary weights | PTQ1_0 (~1.6 bits) | Sep/2026 |
The Dense Transformer
Every model in this study descends from the same origin, the Transformer [1]. In its original form it is a stack of identical blocks, each made of an attention layer, where every token gathers information from the tokens before it, and a feed-forward network (FFN), which then transforms each token on its own. The model is dense: every parameter takes part in the computation of every token.
For several years, progress came mostly from making this structure larger. The scaling laws [2], and later the Chinchilla results [3], showed that loss falls predictably as parameters and training data grow, and models moved from billions to hundreds of billions of parameters. The architecture itself barely changed. What changed was the bill, and it arrives in two forms.
The first, the weights, are the learned parameters of the model: the projection matrices of the attention layers and, above all, the matrices of the FFNs, which hold most of the parameters. In a dense model, generating a token means reading every one of these weights once, so both the memory footprint and the cost per token grow with the model’s size. The second, the KV cache memory, is less visible and grows while the model is being used. To avoid recomputing the whole past at every step, attention stores the keys and values of every previous token in this cache, whose size per token is
where the factor 2 accounts for keys and values. The cache grows linearly with the context, and on long prompts it can become larger than the model itself. Most of the architectural ideas of the last three years can be read as answers to one of these two costs, and the most interesting ones attack both.
Attention, Made Cheaper
An attention layer is split into several heads that work in parallel. Each head projects every token into a query, a key and a value: the query of the current token is compared with the keys of all previous tokens, and the resulting scores decide how much of each value is taken. Having several heads lets each one focus on a different kind of relation between tokens. In the original multi-head attention [1], each query head has its own keys and values, so every head adds to the KV cache. Grouped-query attention (GQA) [4] lets several query heads share a single key/value head, with almost no loss in quality. Llama 3.2 3B has 24 query heads but only 8 key/value heads, which makes its cache three times smaller. Even so, with 28 layers and a head dimension of 128, the formula still gives 112 KiB per token, so at 32K tokens the cache reaches about 3.5 GiB, more than the model’s own 8-bit weights.
A second idea restricts how far back a layer can look. In sliding-window attention, a layer only attends to a fixed window of recent tokens, so its cache stops growing once the window is full. Gemma 3 interleaves several of these local layers for each global one, and gpt-oss alternates between the two. Only the global layers carry the full history, and the cost of long contexts falls in proportion. DeepSeek followed a third path with Multi-head Latent Attention, compressing keys and values into a small latent vector before caching them. The three approaches share the same principle: attention remains the mechanism, but the model stores less of it.
Mixture of Experts
Mixture of Experts (MoE) [5] addresses the other cost, the computation per token. Each block’s single FFN is replaced by many smaller FFNs, the experts, and a small router decides which few of them each token passes through. The model can then hold a very large number of parameters while each token only touches a small fraction of them. The naming convention reflects this split: Qwen3.6-35B-A3B has 35B parameters in total, of which about 3B are active for any given token.
The idea dates back to 2017, but it entered open models through Switch Transformer [6], then Mixtral 8x7B [7] at the end of 2023, and DeepSeek-V3 [8], with 671B parameters and 37B active. By 2026 sparsity had become the default for large open-weight releases, and it has reached sizes that run on a desktop. Three models in this study are MoE: gpt-oss-20b, Qwen3.6-35B-A3B and Nemotron 3 Nano Omni, all with between 3B and 3.6B active parameters.
Sparsity decouples two quantities that a dense model keeps tied together. Compute follows the active parameters, so these models need about as much computation per token as a 3B dense model. Memory still follows the total, since every expert has to be stored somewhere. This asymmetry is what makes them practical outside a datacenter: because each token reads only a handful of experts, the experts can live in system RAM while attention and the shared layers stay on the GPU. The constraint then moves away from the GPU’s compute and towards the bandwidth of RAM and of the link between the two.
Beyond Attention
Attention’s cache grows with every token, and no amount of grouping or windowing changes that for the layers that keep the full history. A separate line of research replaces attention, in most layers, with a recurrent state of fixed size. Instead of keeping every past token, the model reads the sequence step by step and updates a matrix that summarizes everything seen so far. Memory stays constant regardless of the context length, and so does the cost of each new token.
State space models took this idea furthest. Mamba [9] made the state update depend on the input itself, a selective state space model, which closed much of the quality gap with attention, and Mamba-2 [10] reformulated it to train efficiently on modern GPUs. In parallel, linear attention evolved into Gated DeltaNet [11]. It keeps a key-value memory matrix and, at each step, corrects what that memory currently returns for a key instead of simply adding to it, while a gate decides how much of the past to forget.
What these layers give up is exact recall. Everything the model knows about the past has to pass through a state of fixed size, so retrieving one specific detail from far back in a long prompt becomes harder. The designs of 2025 and 2026 resolved this by becoming hybrids: mostly recurrent layers, with a few full-attention layers kept for precise retrieval. Qwen3.6-35B-A3B places 10 attention layers among 30 Gated DeltaNet layers, and Nemotron 3 Nano Omni does the same with Mamba-2. These two are also MoE, which makes them the most complete answer to the two costs in this study: sparse FFNs for the compute, and a memory that barely grows with the context, since only their few attention layers build a cache. Ternary Bonsai 2 27B keeps the same layout as Qwen3.6, 16 attention layers among 64, but with a dense FFN instead of experts.
Liquid AI took a different route with LFM3 1.7B, replacing most attention layers with short convolutions [12] instead of a recurrent state. A gated short convolution mixes each token with only the few tokens just before it, 6 in LFM3, so it keeps no summary of the distant past, only those last few tokens. Looking further back is left to attention: of its 24 layers, 13 are convolutions, 5 use sliding-window attention over the last 256 tokens and only 6 are global [13]. The design divides the work. Most of what a language model computes, such as grammar and the next few words, depends on nearby text and can be handled by cheap convolutions, while only a few layers need to see the whole context.
Small Models, Larger Capacity
At the other end of the scale, the question is how much capability fits in a few billion parameters. Part of the answer came from training rather than architecture. Small models are now trained on far more tokens than Chinchilla would recommend, and are often distilled from a larger model of the same family, which is how Llama 3.2 3B, Gemma 3 4B and Qwen3-VL 4B reach their level. LFM3 1.7B, the smallest model in this study, was trained on 22 trillion tokens, about 13,000 for each of its parameters.
Gemma 4 E4B adds an idea inherited from Gemma 3n, the per-layer embeddings. Every model starts by looking up each token in its embedding table, which turns the token into a vector. In Gemma 4 E4B, this table has dimension 2,560. Gemma 4 E4B also gives each of its 42 layers its own embedding table, of dimension 256. When a token enters a layer, the layer looks up the token in its table, converts the resulting embedding to dimension 2,560 with a small matrix, and adds it to the token’s vector. Together, the 42 per-layer tables and the main table hold about 3.5B of the roughly 8B parameters in the file. Since a lookup involves no computation, the tables do not need fast memory and can stay in RAM. For the same reason, Google leaves them out and counts only the other 4.5B, the effective parameters that give E4B its name. Other models count their embedding table as part of their size, so what sets Gemma 4 E4B apart is the 42 extra tables, not the way they work.
Quantization
All the previous ideas change the network. Quantization leaves the network intact and changes how each weight is stored. Since generating a token means reading every active weight from memory, fewer bits per weight reduce the memory footprint and speed up generation at the same time, at the price of some precision.
The techniques matured quickly. Rounding to 8 bits with a scale per small block of weights, the Q8_0 format, turned out to lose almost nothing and became the safe default. Going down to 4 bits required more care, and methods such as GPTQ [14] and AWQ [15] used a small amount of calibration data to decide how to round each weight, or which weights to protect. llama.cpp’s k-quants (Q4_K_M, Q5_K_M) took a more pragmatic path, grouping weights into blocks with their own scales and keeping the most sensitive tensors at higher precision, and Unsloth’s dynamic UD-Q4_K_XL files extend this by choosing the precision layer by layer. Eventually low precision moved into the release itself: gpt-oss was published by OpenAI with its experts already in MXFP4 [16], a 4-bit floating-point format with one shared scale per block of 32 values. Its 4-bit file is not a compressed copy of the model, it is the model. LFM3 1.7B is small enough to skip quantization: its original 16-bit file takes 3.4 GB, and it is the only model in this study that runs unquantized.
Ternary Weights
The natural limit of this trend is a weight with almost no bits at all. BitNet b1.58 [17] showed in 2024 that a model trained from scratch with weights restricted to -1, 0 and +1 can match a 16-bit model of the same size. Three values carry log2(3) ≈ 1.58 bits of information, hence the name, and with such weights matrix multiplications reduce mostly to additions and subtractions.
Ternary Bonsai 2 27B, from PrismML, brings this idea to a 27B dense hybrid derived from Qwen3.8-27B, with three recurrent layers for each attention layer. Its weights occupy 5.95 GB instead of the roughly 54 GB they would take in 16 bits, the footprint of a model almost ten times smaller. PrismML reports that it retains 98.2% of the original model’s benchmark average. Unlike the MoE models, which hold many parameters but read only a few of them for each token, Bonsai reads all 27B parameters for every token, and this stays cheap because each one takes only about 1.58 bits.
How a Model Writes an Answer
A local LLM works with two kinds of memory. The first is the VRAM, the GPU’s own memory. The study here used an RTX 4060, which has 8 GB of VRAM, read by the GPU chip, the processor that does the math, at about 272 GB/s. The second is the RAM, the computer’s system memory, 32 GB in this setup, which the CPU reads several times more slowly. A PCIe link connects the two and moves about 13 GB/s in practice, roughly twenty times less than the VRAM, so any data that crosses it during generation slows the whole process down.
Before any request arrives, the server starts and reserves space in VRAM for the KV cache (in RAM only for layers that llama.cpp moves there). The size of this space is set by the -c option, the context size, measured in tokens. The space is taken at startup whether it is used or not, so a larger -c leaves less VRAM for the weights, while speed depends only on how many tokens are actually stored. When a conversation reaches -c, generation stops, which means that everything in it, the input, the reasoning and the output, must always fit within the -c value.
When a request arrives, the tokenizer first splits the text into tokens, pieces of words that each have an id number. The model then looks up each id in the embedding table and takes the matching row as that token’s vector. Each token needs only one row, so llama.cpp keeps this table in RAM, and for this reason every model in the study has a few hundred megabytes of weights outside the VRAM.
With the prompt turned into vectors, the model reads all of it in a single pass, called prefill, which fills the KV cache. The vectors go through the layers one after another. In each layer, attention lets every token gather information from the tokens before it and stores that token’s keys and values in the cache, so they do not need to be computed again later, and the FFN then transforms each token on its own. In the MoE models the router and its selected experts take the place of the FFN, in the hybrids most layers update a recurrent state instead of writing to the cache, and in LFM3 most layers are short convolutions that keep only the last few tokens. llama.cpp processes the prompt in batches of 512 tokens, and each weight read from VRAM is used for all 512 tokens before the next weight is read. For this reason prefill is limited by how fast the GPU chip computes, not by how fast memory is read.
Once the prompt is in the cache, the model starts writing the answer one token at a time, a phase called decode. Each new token passes through all the layers, and the last layer gives a score to every token in the vocabulary. The model takes the token with the highest score, a choice called greedy decoding and used throughout this study, appends it to the text and passes it through the layers again. Since every step produces a single token, there is no batch to share each weight across. For every token the model reads all active weights and the whole filled cache once and does little math with each value, so decode is limited by memory bandwidth. The time per token is therefore close to the bytes read divided by the bandwidth, and when the cache grows as large as the weights, writing slows to half.
When the model does not fit in VRAM, the excess goes to RAM in one of three ways. In the first, llama.cpp chooses at startup which weights go to RAM, and during decode the CPU computes those parts where they are. The MoE models run this way, with attention and the shared layers in VRAM and the experts in RAM. During prefill, llama.cpp still lets the GPU chip do the math, so it copies those weights across the PCIe link for every batch of 512 tokens. In the second, the GPU driver places whatever llama.cpp requests beyond the VRAM in shared GPU memory, which is system RAM that the GPU borrows. Nothing fails, but each read from it crosses the PCIe link, and the speed drops sharply. Not every driver offers this fallback; where it is missing, the request fails and the model does not start. In the third, the RAM itself is full, and the operating system starts paging, moving memory to the SSD, which is slower still.
From Architecture to Behavior
Each of the architectures in this post carries an expectation about how the model should behave. In the dense attention models, the KV cache should grow exactly as the formula predicts, and generation should slow down as the cache fills. In the hybrids, memory should stay nearly flat as the context grows, and the open question is whether their handful of attention layers is enough to retrieve information from far back in the prompt. LFM3 should behave in a similar way, since only its attention layers build a cache, with the added question of whether convolutions that see just a few tokens back leave enough for retrieval. The MoE models should generate at a pace set by their active parameters rather than their total, while depending far more on memory bandwidth once their experts leave the GPU. Bonsai should behave like a much larger model while occupying the memory of a much smaller one. The next post puts these expectations to the test, running every model with the same code, prompts and measurements, with completely controlled environments.
References
[1] Vaswani et al., Attention Is All You Need, 2017. arXiv:1706.03762
[2] Kaplan et al., Scaling Laws for Neural Language Models, 2020. arXiv:2001.08361
[3] Hoffmann et al., Training Compute-Optimal Large Language Models, 2022. arXiv:2203.15556
[4] Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023. arXiv:2305.13245
[5] Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, 2017. arXiv:1701.06538
[6] Fedus et al., Switch Transformers, 2021. arXiv:2101.03961
[7] Jiang et al., Mixtral of Experts, 2024. arXiv:2401.04088
[8] DeepSeek-AI, DeepSeek-V3 Technical Report, 2024. arXiv:2412.19437
[9] Gu and Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023. arXiv:2312.00752
[10] Dao and Gu, Transformers are SSMs (Mamba-2), 2024. arXiv:2405.21060
[11] Yang et al., Gated Delta Networks: Improving Mamba2 with Delta Rule, 2024. arXiv:2412.06464
[12] Liquid AI, LFM2 Technical Report, 2025. arXiv:2511.23404
[13] Liquid AI, LFM3-1.7B model card, 2026. https://huggingface.co/LiquidAI/LFM3-1.7B
[14] Frantar et al., GPTQ, 2022. arXiv:2210.17323
[15] Lin et al., AWQ, 2023. arXiv:2306.00978
[16] Open Compute Project, Microscaling (MX) Formats Specification, 2023.
[17] Ma et al., The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, 2024. arXiv:2402.17764