AIster · The model wars · Launches

OpenAI unveils GPT-4o, one model for text, vision and voice

Voice replies in about a third of a second, and OpenAI's newest model on the free plan

Key facts

Date

13 May 2024

Era

The model wars

Interest

Launches

Threads

Model wars

Who

OpenAI

What

GPT-4o (“o” for omni), trained end to end on text, vision and audio

Speed

Audio replies in as little as 232 ms, 320 ms on average (OpenAI)

Old Voice Mode

2.8 s average with GPT-3.5, 5.4 s with GPT-4

OpenAI announced GPT-4o on 13 May 2024. The "o" stands for omni. Earlier ChatGPT features stitched separate systems together; GPT-4o was trained end to end across text, vision and audio, so the same neural network processes every input and output.

Faster voice replies ChatGPT already had a Voice Mode, but it was a pipeline of three models: one transcribed speech to text, GPT-3.5 or GPT-4 wrote the reply, and a third turned the text back into speech. OpenAI put the average delay at 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. GPT-4o, OpenAI said, could respond to audio in as little as 232 milliseconds, with an average of 320 milliseconds, "similar to human response time in a conversation". In the old pipeline, OpenAI said, the main model could not hear tone, multiple speakers or background noise, and could not laugh, sing or express emotion; GPT-4o took in the audio directly.

Free, with limits OpenAI made GPT-4o available on ChatGPT's free plan, with Plus subscribers getting five times as many messages. The new voice mode shown in the launch demos was not part of that first release; OpenAI said an alpha version would follow in the coming weeks.

The Sky voice One of the voices in the demos, called Sky, drew comparisons with Scarlett Johansson, who voiced an AI assistant in the 2013 film Her. After the launch event Sam Altman, OpenAI's chief executive, posted a single word on X: "her". On 20 May Johansson said Altman had approached her in September 2023 to voice ChatGPT and that she had declined. The same day Altman said: "The voice of Sky is not Scarlett Johansson's, and it was never intended to resemble hers." He added: "Out of respect for Ms. Johansson, we have paused using Sky's voice in our products." OpenAI said the voice belonged to a different professional actress using her own natural speaking voice.

Why it mattered GPT-4o moved OpenAI's newest model from a typed chat window towards spoken conversation with little delay, and put it on the free plan. The Sky episode, a week later, showed how quickly questions about voice and likeness would follow.

Questions people ask

When did GPT-4o come out?

OpenAI announced GPT-4o on 13 May 2024 and made it available in ChatGPT, including on the free plan, with Plus subscribers getting five times as many messages.

Why did OpenAI remove the Sky voice?

On 20 May 2024 Scarlett Johansson said Sky sounded like her and that she had turned down Sam Altman's request to voice ChatGPT. OpenAI said Sky was a different actress's natural voice, but paused it out of respect for Johansson.

Sources

1. OpenAI, Hello GPT-4o, 2024-05-13
2. OpenAI, How the voices for ChatGPT were chosen, 2024-05-19
3. TechCrunch, Scarlett Johansson says that OpenAI approached her to use her voice, 2024-05-20