Vaibhaav-Tiwari/deepseek-v3-jax: DeepSeek-V3 in 190 lines of pure JAX

An engineer has implemented the DeepSeek-V3 architecture—including Multi-head Latent Attention, DeepSeekMoE, auxiliary-loss-free balancing, and multi-token prediction—in 190 lines of pure JAX. Pretrained from scratch on public-domain detective fiction using a MacBook Air, the demonstration model achieves a validation loss decrease from 5.02 to 3.51 over 12,000 steps. Open article →