To call ChatGPT, the free text-generating AI developed by San Francisco-based startup OpenAI, a hit is a huge understatement.
In December, ChatGPT was estimated to have over 100 million monthly active users. It has attracted a lot of media attention and spawned numerous social media memes. It has been used to write hundreds of ebooks on Amazon’s Kindle store. And it is credited with having co-authored at least one scientific article.
But because OpenAI is a company — albeit one with a maximum profit — it had to monetize ChatGPT somehow or investors would get nervous. It took a step in that direction with the launch of a premium service, ChatGPT Plus, in February. And it’s taken a bigger step today by introducing an API that allows any company to build ChatGPT technology into their apps, websites, products, and services.
An API was always the plan. So says Greg Brockman, the president and chairman of OpenAI (and also one of the co-founders). He spoke to me via video call yesterday afternoon prior to the launch of the ChatGPT API.
“It takes us a little bit of time to get these APIs to a certain quality level,” said Brockman. “I think it’s like, just being able to meet demand and scale.”
Brockman says the ChatGPT API is powered by the same AI model behind OpenAI’s wildly popular ChatGPT, called “gpt-3.5-turbo.” GPT-3.5 is the most powerful text-generating model OpenAI offers through its API suite today; the nickname “turbo” refers to an optimized, more responsive version of GPT-3.5 that OpenAI is quietly testing for ChatGPT.
Priced at $0.002 per 1,000 tokens, or about 750 words, Brockman claims the API can power a range of experiences, including “non-chat” applications. Snap, Quizlet, Instacart, and Shopify are among the early adopters.
The initial motivation behind developing gpt-3.5-turbo might have been to reduce ChatGPT’s gigantic computing costs. Sam Altman, CEO of OpenAI, once called ChatGPT’s cost “eye-popping,” estimating it at a few cents per chat in computing costs. (With over a million users, that’s likely to add up quickly.)
But Brockman says gpt-3.5 turbo has been improved in other ways.
“When you build an AI-powered tutor, you never want the tutor to just give the student an answer. You want it to always explain and help them learn – that’s an example of the kind of system you should be able to build [with the API]’ said Brockman. “We think this is going to be something that will just make the API much more usable and accessible.”
The ChatGPT API supports My AI, Snap’s recently announced chatbot for Snapchat+ subscribers, and Quizlet’s new Q-Chat virtual tutor feature. Shopify used the ChatGPT API to build a personalized assistant for shopping recommendations, while Instacart used it to create Ask Instacart, an upcoming toll that allows Instacart customers to ask questions about food and get “shoppable” answers based on product data from the company’s retail partners.
“Grocery shopping can take a lot of mental strain, involving many factors, including budget, health and diet, personal taste, seasonality, culinary skills, prep time, and recipe inspiration,” Instacart lead architect JJ Zhuang told me via email. “What if AI could take over that mental burden, and we could help the family leaders who are usually responsible for grocery shopping, meal planning, and putting food on the table — and make grocery shopping really fun? Instacart’s AI system, when integrated with OpenAI’s ChatGPT, allows us to do just that, and we’re excited to start experimenting with what’s possible in the Instacart app.”
Image Credits: Instacart
Those who have been closely following the ChatGPT saga might be wondering if it’s ripe for release – and rightly so.
In the beginning, users could push ChatGPT into answering questions in racist and sexist ways, a reflection of the biased data that ChatGPT was initially trained on. (ChatGPT’s training data spans a wide variety of Internet content, namely ebooks, Reddit posts, and Wikipedia articles.) ChatGPT also fabricates facts without revealing that it does, a phenomenon in AI known as hallucination.
ChatGPT – and similar systems – are also susceptible to prompt-based attacks, or malicious hostile prompts that lead them to perform tasks that were not part of their original objectives. Entire communities on Reddit have formed around finding ways to “jailbreak” ChatGPT and bypass all the protections OpenAI has put in place. In one of the less offensive examples, an employee of startup Scale AI was able to get ChatGPT to reveal information about its internal technical operation.
Brands would undoubtedly not want to get in the crosshairs. Brockman is confident they won’t be. Why? One reason, he says, is the continued improvements to the backend – in some cases at the expense of Kenyan contract workers. But Brockman highlighted a new (and decidedly less controversial) approach that OpenAI calls Chat Markup Language, or ChatML. ChatML feeds text to the ChatGPT API as a string of messages along with metadata. That’s in contrast to the standard ChatGPT, which uses plain text represented as a series of tokens. (For example, the word “fantastic” would be split into the tokens “fan”, “bag”, and “tic”.)
For example, given the question “What are some interesting party ideas for my 30th birthday?” a developer can choose to add an additional prompt to that prompt, such as “You are a fun conversational chatbot designed to help users with the questions they ask. You must answer truthfully and in a fun way!” or “You are a bot” before being processed by the ChatGPT API. These instructions, according to Brockman, help to better tailor and filter the responses from the ChatGPT model.
“We are moving to a higher level API. If you have a more structured way of representing input to the system, saying ‘this is from the developer’ or ‘this is from the user’… I would expect that as a developer you can actually be more robust [using ChatML] against these kinds of fast attacks,” said Brockman.
Another change that will (hopefully) prevent unintended ChatGPT behavior is more frequent model updates. With the release of gpt-3.5-turbo, developers will be automatically upgraded to OpenAI’s latest stable model by default, says Brockman, starting with gpt-3.5-turbo-0301 (released today). However, developers have the option to stick with an older model if they choose, which may negate the benefit somewhat.
Whether they choose to update to the latest model or not, Brockman notes that some customers – mainly large enterprises with correspondingly large budgets – will gain greater control over system performance through the introduction of dedicated capacity plans. First described in documentation leaked earlier this month, OpenAI’s special capacity plans, launched today, let customers pay for a computing infrastructure allocation to run an OpenAI model, e.g. gpt-3.5-turbo. (It’s Azure on the backend, by the way.)
In addition to “full control” over instance load – normally calls to the OpenAI API occur on shared compute resources – dedicated capacity allows customers to enable features such as longer context limits. Context limits refer to the text that the model considers before generating additional text; longer context limits allow the model to essentially “remember” more text. While higher context limits may not solve all bias and toxicity issues, they may lead to less hallucination in models such as gpt-3.5-turbo.
Brockman says special-capacity customers can expect gpt-3.5 turbo models with a context window of up to 16k, meaning they can use four times the tokens of the standard ChatGPT model. That could allow someone to paste in pages and pages of tax codes and get reasonable answers from the model, for example – a feat not possible today.
Brockman hinted at a general release in the future, but not anytime soon.
“The context windows are starting to creep in, and part of the reason we’re just dedicated capacity customers right now is because there are a lot of performance trade-offs on our end,” Brockman said. “Maybe eventually we can offer an on-demand version of the same thing.”
Given OpenAI’s mounting pressure to turn a profit following a multi-billion dollar investment from Microsoft, that wouldn’t be very surprising.