潜在格子语言模型
Latent Lattice Language Models

原始链接: https://shivamduggal4.github.io/latent-lattice-language-models/

Latent Lattice Language Models 引入逐层的水平潜在流,这些潜在流在 token 间持久存在,与垂直残差流互补。与主要用于序列记忆的循环状态不同,这些潜在变量直接条件化注意力、KV 缓存写入、MLP 计算和解码。在 20B FineWeb-Edu token 上,一个 406M 参数的 Lattice 模型将验证困惑度从 15.41 降至 14.40,相较参数匹配的 Transformer;配合超连接/输入残差,进一步降至 13.61。收益扩展至 Gated DeltaNet、Mamba-2、Triadic GDN 以及混合循环模型,在所有测试的原生状态尺寸下;部分 Lattice 变体表现优于原生状态大四倍的基线模型。该工作将持久的逐层状态识别为扩展语言模型计算的互补维度。

Hacker News 链接到“Latent Lattice Language Models。”讨论有五条评论:一条认为该想法类似于通过在整个序列上保持 EMA 来扩展线性注意力中的短卷积,并批评缺少 beyond-2k-context 或 needle-in-haystack 评估,预测 EMA 无法保留细粒度检索信息。另一条询问 0.98 困惑度下降有多显著。另一条称赞使用每层状态来扩展计算。另一条提出可能与 Infini-attention 有关。另一条表达对 RNN 再次流行感到兴奋。总体而言,评论者质疑评估深度和解释,同时指出概念联系。
相关文章

原文
Latent Lattice Language Models

Shivam Duggal1, Zongze Wu2, William T. Freeman1, Antonio Torralba1, Richard Zhang2, Eli Shechtman2

1MIT2Adobe

Abstract

Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer–SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream’s attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN & Lattice-Triadic-GDN variants outperforming pure baselines with 4× larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

The Lattice of Latents

Vertical residual streams flow through depth; horizontal latents persist across tokens. Switch between training and inference, and compare models.

Input injectionablation

Paused · click to play

Click the figure to pause. Input injection can be switched on or off at any moment; the animation continues from the same point.

Results

Interactive versions of the main results. Hover over any bar or point for details.

Language modeling after 20B-token pretraining

LLLM (dmodel, dr) has a vertical residual-stream width dmodel and horizontal-latent width dr.

View

Transformer LLLM (single pass) LLLM + Jacobi refiner (J3)

GFLOP/token: floating-point operations per token, excluding attention scores (which add only ≈ 0.1 GFLOP/token) and the embedding lookup.
J3 runs three sweeps; its sequential rollout uses a single pass at inference (click Show full table for results).

Show full table

Scaling pretraining tokens: 10B → 20B → 30B

FineWeb-Edu validation perplexity as training continues. Circle area is proportional to parameter count.

Both LLLMs use only 22 layers vs. 40 for the Transformer 40L, so with the same attention width (1024) they keep a 45% smaller KV cache.

Lattice of Latents vs. prior residual-stream enhancements

400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. The Transformer and its single enhancements appear first, then the Lattice and its single enhancements, then two-enhancement combinations.

Transformer variants Lattice variants (ours)

HC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the single +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual.

Deconstructing the Lattice

A hard regime: the vertical residual stream is squeezed to 256 dimensions. Press play to add the horizontal-update enhancements one at a time and watch them recover most of the gap to a full-width Transformer. 10B-token pretraining.

+ adds a component; ↻ replaces one. Parameter counts are shown under each bar.

Recurrent backbones

Pure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) across native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart. Bars appear from highest to lowest perplexity, each at its state-size position.

Pure backbone + Lattice (ours)

Native-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.

联系我们 contact @ memedata.com