Language Model Harnesses Can Be Compositional Generalizers

www.alphaxiv.org10 min read

Researchers explore how neurosymbolic language model harnesses, such as recursive language models (RLMs), can serve as compositional generalizers to improve data efficiency and handle long-horizon tasks. By externalizing context and programmatically decomposing problems into locally in-distribution sub-tasks, these harnesses enable models to transfer across domains and generalize to tasks significantly longer than their training data. Furthermore, the authors introduce surrogate harness training to train models on fully synthetic environments with fake tool calls, improving downstream performance on complex long-context benchmarks while using fewer tokens. Open article →

More on this topic