AIster · Prologue · Launches

OpenAI unveils GPT-3

A 175-billion-parameter model showed scale alone could unlock new abilities

Key facts

Date

28 May 2020

Era

Prologue

Interest

Launches

Threads

Model wars, Labs & people

Paper

“Language Models are Few-Shot Learners”, 31 authors

Size

175 billion parameters

Context window

2,048 tokens

API

Private beta from 11 Jun 2020

The paper

On 28 May 2020, 31 OpenAI authors led by Tom B. Brown posted “Language Models are Few-Shot Learners” to arXiv. It introduced GPT-3, a language model with 175 billion parameters, which the paper said was 10 times larger than any previous non-sparse language model.

The core finding was about scale. Earlier models were usually fine-tuned for each task with extra training. GPT-3 was instead given a task description and a handful of examples in its prompt, and then left to carry on. The paper reported that making the model bigger substantially improved this “few-shot” performance across translation, question answering, arithmetic and other tests. It also noted that human evaluators had difficulty telling some GPT-3 news articles apart from ones written by people.

What it learned from

GPT-3 was trained on a weighted mix of text: Common Crawl web data made up 60 per cent, WebText2 22 per cent, two book collections 8 per cent each and Wikipedia 3 per cent. It could consider 2,048 tokens (small chunks of text, often parts of words) at a time.

The API

On 11 June 2020 OpenAI launched an API giving developers access to models from the GPT-3 family, through a private beta with a waitlist. OpenAI called it a commercial product and said it could be tried on “virtually any English language task”. OpenAI explained its choice not to release the model openly: “it feels inherently safer to release them via an API and broaden access over time, rather than release an open source model where access cannot be adjusted if it turns out to have harmful applications.” Early users included the text adventure game AI Dungeon.

Reaction

In September 2020 The Guardian published an opinion piece that it had assembled and edited from several essays GPT-3 wrote to its prompt. On 22 September 2020 Microsoft secured an exclusive licence to the underlying model, while OpenAI kept the public API running. Views were split: philosopher David Chalmers called it “one of the most interesting and important AI systems ever produced”, while others criticised its accuracy, bias and the environmental cost of training.

Why it matters

GPT-3 was the model that convinced much of the tech industry that large language models were worth building. Its API became the first way many start-ups built products on top of a general-purpose model, and ChatGPT, launched in November 2022, was fine-tuned from a model in the successor GPT-3.5 series.

Questions people ask

When was GPT-3 released?

OpenAI described GPT-3 in a paper posted on 28 May 2020 and opened access to it through a private-beta API on 11 June 2020.

How big is GPT-3?

GPT-3 has 175 billion parameters. OpenAI's paper said this was 10 times more than any previous non-sparse language model.

Sources

1. arXiv, Language Models are Few-Shot Learners, 2020-05-28
2. OpenAI, OpenAI API, 2020-06-11
3. TechCrunch, OpenAI makes an all-purpose API for its text-based AI capabilities, 2020-06-11