MINE memory-efficient LLM inference integrates into vLLM

Researchers Emil Schätzle and Markus Püschel present MINE, a memory-efficient inference engine that accelerates single-stream Large Language Model inference through a lossless entropy coding approach. By integrating into vLLM as a plugin and utilizing a GPU entropy coding scheme called dtANS-LLM, MINE exploits weight distributions to trade compression ratio against decoding cost. On Llama 3.1 8B Instruct, MINE achieves significant speedups over existing baselines while being available as open source. Open article →