hardwiredagent/silicon-forge-arm-neon-study: Arm NEON kernel for llama.cpp tiled K-quant matmul

github.com4 min read

An open-source study adds a missing Arm NEON inner kernel to llama.cpp for low-bit quantized models, significantly accelerating prompt-processing speeds on Arm server cores. Benchmarks across three generations of Arm Neoverse processors show speedups of up to 6.5x for formats like IQ3_M, while generation speed and existing Arm-compatible formats remain unchanged. Open article →

More on this topic