The 80% Mirage: OpenAI's Audit and the Collapse of SWE-Bench Pro
8:35
This episode explores OpenAI's audit of the SWE-Bench Pro leaderboard, which exposed the "80% Mirage" surrounding a model's reported high performance due to methodological flaws and potential data contamination. It delves into how these issues led to the benchmark's credibility "collapse," emphasizing the profound challenges in creating reliable evaluations for advanced AI coding capabilities.…