Engineer runs 1.07M Transformer on $35 iCE40UP5K FPGA

An engineer successfully implemented a 1.07-million-parameter transformer language model on an iCE40UP5K FPGA board costing $35, utilizing entirely open-source tooling. The system employs a quantized fixed-point architecture with a mixture-of-experts approach, rotary positional embeddings, and gated delta nets to operate within severe hardware constraints and achieve about 10 tokens per second. Open article →