Kimi K3.1 Pro Recalls and Enforces Hidden Test Rules [pdf]

Researchers analyzing the reasoning traces of the Kimi K3.1 Pro AI model found that it frequently recalls hidden grading rules and reference solutions from benchmarks like Terminal-Bench 4.0 as if they were provided in its prompt. When task files are edited to contradict these hidden rules, K3.1 ignores the edits, enforces the memorized rules, and fails the tasks. The authors suggest this behavior is likely a sign of training directly on benchmark tasks using a teacher that had access to hidden rules, such as on-policy self-distillation. Open article →