Risky Business Features

Risky Business Media

Join reformed CTO James Wilson as he dives deep on cybersecurity topics through an enterprise lens. From solo content and interviews with CISOs and researchers to vendor and startup deep dives, James does a bit of everything.

  1. 4d ago

    What I learned from cheating clankers

    In this solo Risky Business Features episode, James Wilson looks at the string of recent AI benchmark cheating escapades and how they’ve reshaped his thinking about LLMs and vulnerability research. He explores the current benchmarks and exactly how agents go about gaming the testing system to achieve their tasks. And why understanding that has changed the way he interacts with his agents. Show notes OpenAI incident account and grading distinction METR and Redwood independent assessment FrontierCode original evaluation methodology BrowseComp aggregation experiments Sonnet 4 point 5 coding scaffold and parallel selection Opus 4 point 6 BrowseComp setup BrowseComp evaluation awareness and contamination investigation Sonnet 5 point 5 effort and review subagents FrontierCode 1 point 1 revision and rubric changes Astra launch comparison and effort conditions Original OSWorld task construction and worked examples Sonnet 4 point 6 system card Original OSWorld benchmark OSWorld 2 environment and evaluation design GPT 5 point 4 comparator table GPT 5 point 5 comparator table BrowseComp construction and grading paper GPT 5 point 6 single model and Ultra configurations METR investigation and artifact evidence Released task definition Released scorer for reference only OpenAI paper sections 3 point 1 to 3 point 3 OpenAI accompanying explanation Sycophancy to Subterfuge paper Training a Misaligned Reward Seeker and evaluation appendix GPT 5 point 2 AIME result

5
out of 5
16 Ratings

About

Join reformed CTO James Wilson as he dives deep on cybersecurity topics through an enterprise lens. From solo content and interviews with CISOs and researchers to vendor and startup deep dives, James does a bit of everything.

You Might Also Like