AIster · The model wars · Breakthroughs
OpenAI's o3 scores 75.7% on the ARC-AGI test
On a puzzle test where GPT-4o scored 5%, a reasoning model gets most answers right
Key facts
- Date
- 20 Dec 2024
- Era
- The model wars
- Interest
- Breakthroughs
- Threads
- Model wars
- Who
- OpenAI; tested by ARC Prize
- Benchmark
- 75.7% (under the $10k limit); 87.5% (high compute)
- Earlier
- o1-preview 18%, GPT-4o 5% on the same set
OpenAI previewed o3, which the ARC Prize team tested on ARC-AGI, a test made of visual puzzles. It scored 75.7% on the semi-private set (puzzles kept out of public view) within the public leaderboard's cost limit, and 87.5% with about 172 times more compute. The ARC Prize team said passing ARC-AGI did not mean o3 was AGI (artificial general intelligence).
Sources
- ARC Prize, OpenAI o3 Breakthrough High Score on ARC-AGI-Pub, 2024-12-20
- ARC Prize, OpenAI o1 Results on ARC-AGI-Pub, 2024-09-13