AIster · The model wars · Breakthroughs

OpenAI's o1 reasons before it answers

Spending more computing time on each answer becomes a second way to improve AI

Key facts

Date

12 Sep 2024

Era

The model wars

Interest

Breakthroughs

Threads

Model wars

Who

OpenAI

What

o1-preview and o1-mini (full o1 followed on 5 Dec 2024)

Benchmark

IMO qualifying exam: 83% vs GPT-4o's 13% (OpenAI)

OpenAI released o1-preview and o1-mini on 12 September 2024, the first models in a series built to reason before they answer. Instead of replying straight away, o1 works through a long internal chain of thought and only then gives a response. Before launch the project had been reported under the codename Strawberry.

The numbers OpenAI said that on a qualifying exam for the International Mathematics Olympiad, GPT-4o solved 13% of problems while o1 solved 83%. OpenAI also reported that it reached the 89th percentile in Codeforces programming contests. These were OpenAI's figures for o1. What users got in September was o1-preview, alongside the smaller o1-mini, which was aimed at code generation; the full o1 followed on 5 December 2024.

Slower and dearer Thinking costs time and tokens. In the API, o1-preview cost $15 per million input tokens and $60 per million output tokens, six times the price of GPT-4o. ChatGPT Plus and Team subscribers were at first limited to 30 messages a week with o1-preview and 50 with o1-mini.

Independent tests showed the trade-off. The ARC Prize team ran o1-preview on its ARC-AGI puzzles and found it matched Claude 3.5 Sonnet on the public tasks, at 21.2% against 21%, but took 70 hours on the 400 public tasks, compared with about 30 minutes for GPT-4o and Claude 3.5 Sonnet. On the harder semi-private set it scored 18%.

Why it mattered Before o1, the main lever for better models was training: more data, more parameters, more compute before release. o1 showed a second lever, spending more computation while the model answers, often called test-time compute.

Questions people ask

When was OpenAI o1 released?

OpenAI released o1-preview and o1-mini on 12 September 2024 to ChatGPT Plus and Team users and to developers through the API. The full o1 model followed on 5 December 2024.

How is o1 different from GPT-4o?

o1 reasons step by step before it answers, which makes it slower and more expensive but much stronger on maths and coding. OpenAI reported that o1 solved 83% of problems on a qualifying exam for the International Mathematics Olympiad, against 13% for GPT-4o.

Sources

1. OpenAI, Introducing OpenAI o1, 2024-09-12
2. TechCrunch, OpenAI unveils o1, a model that can fact-check itself, 2024-09-12
3. ARC Prize, OpenAI o1 Results on ARC-AGI-Pub, 2024-09-13