AIster · Prologue · Breakthroughs

AlexNet wins ImageNet and deep learning takes off

A neural network trained on GPUs won an image contest by a wide margin and redirected AI

Key facts

Date

Oct 2012

Era

Prologue

Interest

Breakthroughs

Threads

Model wars, Compute

Team

SuperVision: Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton (University of Toronto)

Top-5 error

0.15315 (runner-up ISI: 0.26172)

Model size

60 million parameters

Training data

1.3 million images, 1,000 classes

The contest

The ImageNet Large Scale Visual Recognition Challenge asked teams to build software that could label photographs across 1,000 categories. In 2012 the submission deadline was 30 September. Preliminary results went to participants on 8 October, the winners were presented at a workshop at the ECCV conference in Florence on 12 October, and full results were published on 13 October.

The result

The winning team, entered as SuperVision, was Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton from the University of Toronto. On the main classification task their best entry had a top-5 error of 0.15315, meaning the correct label was missing from its five guesses about 15 per cent of the time. The second-placed team, ISI, scored 0.26172. In a contest where progress was usually measured in small steps, the gap was striking.

The network became known as AlexNet.

What was different

The paper describing the system, “ImageNet Classification with Deep Convolutional Neural Networks”, appeared at the NIPS conference (now NeurIPS) later that year. It describes a deep convolutional neural network (a design suited to images) with 60 million parameters, trained on 1.3 million high-resolution images. It had five convolutional layers followed by fully connected layers.

Two practical choices stood out. The team wrote an efficient implementation that ran on GPUs, the graphics chips built for video games, which made training a network this large feasible. They also used a then-new regularisation method to stop the network simply memorising its training data.

None of the core ideas were entirely new. What changed was the combination of a large labelled dataset, cheap parallel hardware and a deeper network than most researchers had thought worth trying.

Why it matters

AlexNet is widely treated as the point where deep learning moved from a minority interest to the main approach in AI. The recipe of more data, more compute and bigger neural networks is the same one behind today's large language models.

The people involved went on to shape the next decade. Sutskever became research director of OpenAI when it launched in 2015.

Questions people ask

What is AlexNet?

AlexNet is the deep convolutional neural network built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton at the University of Toronto. It won the 2012 ImageNet image-recognition challenge by a wide margin and is often cited as the start of the deep learning boom.

Why was AlexNet important?

It showed that a large neural network trained on GPUs with lots of labelled data could beat established methods by a clear margin. Its winning top-5 error of 0.15315 compared with 0.26172 for the next team, and the field moved towards deep learning afterwards.

Sources

1. ImageNet, ImageNet Large Scale Visual Recognition Competition 2012 (ILSVRC2012) results, 2012-10-13
2. ImageNet, Large Scale Visual Recognition Challenge 2012, 2012-10-13
3. NeurIPS Proceedings, ImageNet Classification with Deep Convolutional Neural Networks, 2012-12-01