Apple MacBook Pro memory bandwidth predicts token speed

The author examines how memory bandwidth acts as the primary bottleneck during the decode phase of large language model inference on a laptop. By measuring time per output token across twelve model configurations on an Apple Silicon MacBook Pro, the author demonstrates that a linear model using model size and an effective bandwidth of 163 GB/s accurately predicts token generation speed. Open article →