AIster · Prologue · Breakthroughs

“Attention Is All You Need” introduces the Transformer

The Google paper behind the T in GPT and nearly every large language model since

Key facts

Date

12 Jun 2017

Era

Prologue

Interest

Breakthroughs

Threads

Model wars, Open vs closed

Authors

Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin

Posted

arXiv, 12 Jun 2017

Benchmark

28.4 BLEU English–German; 41.0 BLEU English–French

Training

3.5 days on eight GPUs (English–French)

The paper

On 12 June 2017, eight authors posted a paper to arXiv titled “Attention Is All You Need”: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. It proposed a new neural network architecture, the Transformer, that relies only on attention mechanisms and drops the recurrence and convolutions used by earlier language systems.

What attention does

Older language models typically read a sentence word by word, carrying information forward as they went. The Transformer instead uses what the paper calls self-attention. In the words of Google's announcement, it “applies a self-attention mechanism which directly models relationships between all words in a sentence, regardless of their respective position.”

Google's example was two sentences in which the word “it” refers to different things: in one to an animal, in the other to a street. When translating into French, where the pronoun's gender depends on the noun it refers to, the model needs to know which is meant. Google showed that the Transformer's attention picked out the right noun in each case.

The results

The paper tested the design on machine translation. It reported 28.4 BLEU on the WMT 2014 English-to-German task and 41.0 BLEU on English-to-French, the latter after training for 3.5 days on eight GPUs. Because attention can look at all words at once, the work can be spread across chips in parallel, which made training faster than with older designs.

What came next

The paper was about translation, but the architecture turned out to be general. On 11 June 2018 OpenAI published “Improving Language Understanding by Generative Pre-Training”, which introduced the first GPT model. GPT stands for generative pre-trained transformer, and GPT models are built on the Transformer architecture.

That line runs through GPT-3 in 2020 and on to ChatGPT at the end of 2022. For a newcomer, this is the single most important technical paper in the story: almost every well-known chatbot today is a descendant of the design it describes.

Questions people ask

What is the Transformer in AI?

The Transformer is a neural network architecture introduced in the 2017 paper “Attention Is All You Need”. It uses self-attention to relate every word in a sentence to every other word at once, without reading them in order.

What does GPT stand for?

GPT stands for generative pre-trained transformer. GPT models are large language models built on the Transformer architecture from the 2017 paper; OpenAI introduced the first one in June 2018.

Sources

1. arXiv, Attention Is All You Need, 2017-06-12
2. Google Research, Transformer: A Novel Neural Network Architecture for Language Understanding, 2017-08-31
3. OpenAI, Improving language understanding with unsupervised learning, 2018-06-11