Researchers show confidence calibration fixes noisy LLM fallback training for Llama 3.1 8B

Training a small classifier model using a larger LLM as a fallback introduces noisy labels that degrade the confidence-fitting temperature rather than the overall accuracy. Researchers found that calibrating confidence on a small set of trusted or zero-shot confident examples restores performance to levels comparable to using a perfect LLM. Additionally, learning an explicit null label for off-topic text helps the model reject out-of-scope queries locally. Open article →