Author runs Qwen and Gemma on GTX 1080
An author details how they successfully repurposed an outdated Nvidia GTX 1080 graphics card to run large language models locally using Proxmox, Debian LXC passthrough, and the llama.cpp engine with a Vulkan backend. By utilizing Mixture-of-Experts models like Qwen and Gemma, the VRAM-constrained, 10-year-old GPU achieved surprisingly high token generation speeds by offloading part of the expert weights to system RAM. Open article →