LLM inference prefill compute-bound, decode memory-bound analysis
The author analyzes the computational dynamics of large language model inference, distinguishing between compute-bound prefill phases and memory-bandwidth-bound decode phases. The analysis demonstrates that because the weight reuse factor during decode is approximately one, performance is constrained by memory bandwidth rather than floating-point operations. Consequently, the memory wall governs both edge AI and datacenter decode economics, highlighting a shared architectural need for memory-centric optimizations. Open article →