The new text generator GPT-3 (Generative Pre-trained Transformer) can write texts in the style of Shakespeare, create program codes, and translate foreign languages and legal paragraphs. Is this method the answer to all our questions or does it end in tragedy, as it does not find a suitable method among the 175 billion machine learning parameters? The Australian philosopher, David Chalmers, says “it is one of the most interesting and important AI methods ever developed”.
The basic technique of statistical language modelling is used to automatically gain knowledge concerning linguistic structures, such as word order probabilities, syntactic relationships, and dialogue progression. However, while older models can only rely on very few words to predict the next word, GPT-3 involves several paragraphs: Up to 2,048 so-called “tokens” (which can be words or parts of a code) are taken into account by the AI method in its prediction. The developers used more than 570 gigabytes of text for their training, which corresponds to about one trillion words from the Internet, including forum entries, blog posts and editorials. That’s a lot, but notquite enough for a novel. Even if filters have been announced, the problem of racist or sexist submissions, known as “biases,” remain.
The autoregressive language model uses deep learning to create “human” texts. It is the third-generation speech prediction model of OpenAI, an artificial intelligence research lab. OpenAI is backed by financial investors such as the tech billionaires Elon Musk (Tesla), Peter Thiel (Co-Founder of Paypal) and Reid Hoffman (Co-Founder of LinkedIn). In addition, Microsoft provided one billion dollars last year. Is it clever marketing when the techies, otherwise brimming with hubris, warned of the potential dangers of GPT-3 when they presented the original paper on May 28, 2020, and called for further research in order to minimize the risk?
Past approaches, such as Google’s text generator, BERT, rely on so-called fine-tuning. Here the AI method, following the basic training, must be adapted to the respective requirements, for example of a translation, with task-specific data sets. With GPT-3 this complex process is often unnecessary. As the training data of GPT-3 was very comprehensive, no further training is required for certain language tasks.
OpenAI now offers an API for accessing GPT-3 for commercial use. This is a universal interface that allows users to try out virtually any task in English. Specific AI know-how and cost-intensive IT infrastructure are no longer required. The beta version is currently available for selected users. Since then, videos, screenshots and reports have been causing a hype on social networks.
“Artificial intelligences should not be rashly given human qualities. But AI methods will change many fields. There is no doubt about that.” – Bernd Zipper
In a few clear sentences the method sums up what is behind any complex, legal paragraphs. Such examples suggest that the model could deduce or even understand the meaning of language. These in turn would be indications of an intelligence that is attributed to humans. But how close does GPT-3 really come to this ideal? Is language AI as versatile as the human brain, only much faster? “The capabilities of GPT-3 are very impressive, but not surprising given the size of the model,” said Sina Zarrieß, Professor of Machine Learning and Language Technology at the University of Jena. But there are also limitations. “Anyone who takes a closer look at the texts, which appear quite authentic, will notice that there is often no real sense behind the surface,” says Zarrieß. Again, and again, the model finds absurd and wrong solutions for tasks that would normally not cause any problems for people. GPT-3 also has difficulty with absurd questions such as: “Will an ice cube melt if I put it in the refrigerator? The same applies to questions of understanding the content, for example, whether one statement necessarily leads to another.
Statistical models are based on pure probability theory. The method therefore has no way to understand or interpret what it has learned. So, if GPT-3 suddenly seems to master programming languages, this is simply because it has seen a code and stored its pattern. Although it does have a certain charm if you think about the creation of a website and how the command “create a button in the color of the fire department” creates a red button.
However, it also became clear that research on AI-supported text generators still has a long way to go. The answer to the question about the meaning of life is still probably 42, at least if GPT-3 has stored “The Hitchhiker’s Guide to the Galaxy” by the British author Douglas Adams.
In the second part, which is still to come, I will discuss the application possibilities for our industry.