LessWrong (30+ Karma)
“Measuring agent capabilities on open-ended AI R&D tasks” by Alex Jackson, Titus Buckworth, Jason R Brown, Francis Rhys Ward
36:20
TL;DR. We measure agent capabilities on four long and open-ended research tasks. Frontier models have been improving with an overall doubling time of around one year since 2024 (based on an exponential fit). Agents perform better at engineering-heavy tasks compared to more conceptual tasks (between 7 and 16.8 months median doubling time), and model capabilities are improving slower on the…