Latent Lattice Language Models
Researchers introduce the Lattice of Latents, an architectural enhancement for language models that maintains a set of per-layer horizontal latent streams running recurrently over time alongside the vertical residual stream. Unlike hybrid Transformer-SSM architectures where states serve mainly as internal memory, these persistent companion states directly condition attention, KV-cache writes, MLP computation, and decoding. Experiments on Transformers and state-space models show that the Lattice improves validation perplexity and enables smaller models to outperform baselines with much larger native states. Open article →