I let LLMs play a 1988 detective gamebook 646 times

An author tested how well various large language models could play a 1988 detective gamebook across 646 games. The author found that no model won on the first attempt, but capable models learned from retries and could eventually solve the game if failures pointed clearly back to their causes. However, when failures stemmed from early sensible decisions whose consequences were hidden until the very end, the models consistently failed to trace the error and often blamed the wrong cause. Open article →