Depth that grows with the sequence.
Each token extends the recurrent path through the full decoder. After \(t\) tokens, that path traverses \(tL_D\) decoder blocks while the per-token block count stays fixed.
原文
Parallel work around a recurrent core.
Batch known-token encoder work and independent decoder updates. Reuse weights and memory, and checkpoint activations while preserving the reference computation.
One transition from sampling to replay.
Rebuild the full history under current parameters, including prompt states and decoder SWA KV. Keep recorded behavior probabilities tied to the actual sampler.
The complete state matters.
Causal encoderKnown-token parallelism · global KV memory
→
Recurrent decoderEncoder cross-attention · local decoder SWA
Carry forward: recurrent output + decoder SWA KV
The previous output enters the next merge. Each SWA layer reads its own recent keys and values.
\[H_t=(s_t,C_t^D),\qquad H_0=(s_\star,\varnothing).\]
\[(s_t,C_t^D)=D_\phi\!\left(\operatorname{Merge}(e_t,s_{t-1});M_{\le t},C_{t-1}^D,t\right).\]
\[p_\Theta(x_{t+1}\mid x_{1:t})=\operatorname{softmax}\!\left(W_o\operatorname{RMSNorm}_o(s_t)\right)_{x_{t+1}}.\]
Here \(M_{\le t}\) is global encoder memory, \(s_t\) is the recurrent output, and \(C_t^D\) contains layerwise decoder KV. A SWA window of \(W\) includes the current token and retains at most \(W-1\) historical entries for the next update.
The concrete configuration uses 48 encoder layers and 48 decoder layers, with compatible attention and FFN weights shared across stages. The temporal path traverses \(48t\) decoder blocks after \(t\) tokens. Each token executes 96 logical blocks; decoder cross-attention means these blocks do not all have equal FLOPs.