AIster · The model wars · Launches

Anthropic's Claude 3 edges past GPT-4

A year after GPT-4, another lab matched it on tests and in users' blind votes

Key facts

Date

4 Mar 2024

Era

The model wars

Interest

Launches

Threads

Model wars

Who

Anthropic

What

Claude 3 family: Haiku, Sonnet, Opus

Benchmark

MMLU (5-shot): Opus 86.8% vs GPT-4 86.4%, as reported by Anthropic

Arena

First place from 27 Mar 2024 until an updated GPT-4 retook it on 12 Apr

Anthropic released the Claude 3 family on 4 March 2024: three models called Haiku, Sonnet and Opus, in rising order of size and capability. Opus and Sonnet were available that day in claude.ai and through the API; Haiku followed later. The models launched with a 200,000-token context window, and Anthropic said all three could accept inputs of more than a million tokens.

What the numbers said Anthropic's launch post said Opus "outperforms its peers on most of the common evaluation benchmarks for AI systems", naming undergraduate-level knowledge (MMLU), graduate-level reasoning (GPQA) and basic mathematics (GSM8K). In the model card's results table, Opus scored 86.8% on MMLU in the 5-shot setting (shown five worked examples first), against 86.4% for GPT-4. On GPQA Diamond, a set of graduate-level science questions released in November 2023, Opus scored 50.4% (0-shot, chain of thought) against 35.7% for GPT-4.

Those comparisons need care. The Claude 3 figures were Anthropic's own runs, while the GPT-4 figures came from OpenAI's technical report, published a year earlier. On MMLU the margin was 0.4 percentage points.

The leaderboard A different kind of test came from users. On Chatbot Arena, run by the LMSYS group, people put a question to two anonymous models and vote for the better answer. On 27 March 2024 Claude 3 Opus moved into first place, knocking GPT-4 off the top of the leaderboard for the first time. The lead was short: by 12 April an updated GPT-4 had taken first place again.

Why it mattered For about a year after its release, GPT-4 was the model others were measured against. Claude 3 showed that another lab could match it both on the standard tests and in blind votes by users.

Questions people ask

When was Claude 3 released?

Anthropic announced the Claude 3 family (Haiku, Sonnet and Opus) on 4 March 2024. Opus and Sonnet were available that day; Haiku followed later.

Did Claude 3 beat GPT-4?

On Anthropic's reported figures, Claude 3 Opus scored 86.8% on MMLU against GPT-4's 86.4%. On 27 March 2024 it also took first place on the Chatbot Arena leaderboard, which an updated GPT-4 regained on 12 April.

Sources

1. Anthropic, Introducing the next generation of Claude, 2024-03-04
2. Anthropic, The Claude 3 Model Family: Opus, Sonnet, Haiku (model card), 2024-03-04
3. The Decoder, Updated GPT-4 is ahead of Claude 3 Opus in the Chatbot Arena benchmark, 2024-04-12