Researchers show Bradley-Terry leaderboards manipulable

Researchers found that arena-style language model leaderboards using preference-based evaluations and the Bradley-Terry model are highly vulnerable to manipulation strategies. Unscrupulous providers can boost a model's rank by up to 15 to 20 places using strategic voting, prompting, or nomination, and combine these attacks for an increase of up to 45 places. The authors describe approaches to safeguard against each of these manipulation vectors. Open article →