Engineer sizes private Llama 3.3 70B server for 200 users

pub.towardsai.net11 min read

An infrastructure engineer explains how to size a private LLM server for 200 users by calculating request throughput, context length, and GPU memory usage. The author details a hardware bill of materials utilizing an AMD EPYC processor and four NVIDIA Blackwell GPUs running Llama 3.3 70B quantized to 4-bit precision. Open article →