Inkling 2 Mini prefill accelerated in SGLang

advprop.ai7 min read

The author explores whether prefill for looped language models like Inkling 2 Mini can be accelerated by computing only a single pass of the loop with projection-only layers for the rest. By implementing this in SGLang and training the projections using distillation, time to first token was significantly reduced. However, untrained models suffered massive benchmark drops, which post-training on the original model's outputs largely mitigated. Open article →