Investigating What Open Models Knew When They Answered Questions Incorrectly

patronus.ai19 min read

Researchers investigating why large language models fail on complex tasks discovered that open-weight models often compute the correct answer internally but fail to output it. By probing internal activations and testing targeted layer retrainings, they found that rate-of-change failures can be fixed by reconnecting internal knowledge to the output, whereas long-chain reasoning failures require multi-step thinking processes. Open article →