GPT-3- Part 1: To be, or not to be, that is the question

0

The new text generator GPT-3 (Generative Pre-trained Transformer) can write texts in the style of Shakespeare, create program codes, and translate foreign languages and legal paragraphs. Is this method the answer to all our questions or does it end in tragedy, as it does not find a suitable method among the 175 billion machine learning parameters? The Australian philosopher, David Chalmers, says “it is one of the most interesting and important AI methods ever developed”.

The basic technique of statistical language modelling is used to automatically gain knowledge concerning linguistic structures, such as word order probabilities, syntactic relationships, and dialogue progression. However, while older models can only rely on very few words to predict the next word, GPT-3 involves several paragraphs: Up to 2,048 so-called “tokens” (which can be words or parts of a code) are taken into account by the AI method in its prediction. The developers used more than 570 gigabytes of text for their training, which corresponds to about one trillion words from the Internet, including forum entries, blog posts and editorials. That’s a lot, but notquite enough for a novel. Even if filters have been announced, the problem of racist or sexist submissions, known as “biases,” remain.

The autoregressive language model uses deep learning to create “human” texts. It is the third-generation speech prediction model of OpenAI, an artificial intelligence research lab. OpenAI is backed by financial investors such as the tech billionaires Elon Musk (Tesla), Peter Thiel (Co-Founder of Paypal) and Reid Hoffman (Co-Founder of LinkedIn). In addition, Microsoft provided one billion dollars last year. Is it clever marketing when the techies, otherwise brimming with hubris, warned of the potential dangers of GPT-3 when they presented the original paper on May 28, 2020, and called for further research in order to minimize the risk?

Past approaches, such as Google’s text generator, BERT, rely on so-called fine-tuning. Here the AI method, following the basic training, must be adapted to the respective requirements, for example of a translation, with task-specific data sets. With GPT-3 this complex process is often unnecessary. As the training data of GPT-3 was very comprehensive, no further training is required for certain language tasks.

OpenAI now offers an API for accessing GPT-3 for commercial use. This is a universal interface that allows users to try out virtually any task in English. Specific AI know-how and cost-intensive IT infrastructure are no longer required. The beta version is currently available for selected users. Since then, videos, screenshots and reports have been causing a hype on social networks.

Artificial intelligences should not be rashly given human qualities. But AI methods will change many fields. There is no doubt about that. – Bernd Zipper

In a few clear sentences the method sums up what is behind any complex, legal paragraphs. Such examples suggest that the model could deduce or even understand the meaning of language. These in turn would be indications of an intelligence that is attributed to humans. But how close does GPT-3 really come to this ideal? Is language AI as versatile as the human brain, only much faster? “The capabilities of GPT-3 are very impressive, but not surprising given the size of the model,” said Sina Zarrieß, Professor of Machine Learning and Language Technology at the University of Jena. But there are also limitations. “Anyone who takes a closer look at the texts, which appear quite authentic, will notice that there is often no real sense behind the surface,” says  Zarrieß. Again, and again, the model finds absurd and wrong solutions for tasks that would normally not cause any problems for people. GPT-3 also has difficulty with absurd questions such as: “Will an ice cube melt if I put it in the refrigerator? The same applies to questions of understanding the content, for example, whether one statement necessarily leads to another.

Statistical models are based on pure probability theory. The method therefore has no way to understand or interpret what it has learned. So, if GPT-3 suddenly seems to master programming languages, this is simply because it has seen a code and stored its pattern. Although it does have a certain charm if you think about the creation of a website and how the command “create a button in the color of the fire department” creates a red button.

Bernd Zipper My Take: GPT-3 has fascinating capabilities, there is no question about that. OpenAI started as a non-profit artificial intelligence research lab in San Francisco. According to OpenAI, the goal is still to ensure that artificial intelligence benefits everyone. But the development of commercial products is one way to ensure that sufficient funding is available. I am sure that this will give the cause even more drive. Especially since the current version is still in beta status and will be continuously improved.

However, it also became clear that research on AI-supported text generators still has a long way to go. The answer to the question about the meaning of life is still probably 42, at least if GPT-3 has stored “The Hitchhiker’s Guide to the Galaxy” by the British author Douglas Adams.

In the second part, which is still to come, I will discuss the application possibilities for our industry.

Summary
GPT-3- Part 1: To be, or not to be, that is the question
Article Name
GPT-3- Part 1: To be, or not to be, that is the question
Description
The new text generator GPT-3 (Generative Pre-trained Transformer) can write texts in the style of Shakespeare, create program codes, and translate foreign languages and legal paragraphs. Is this method the answer to all questions or does it end up as a tragedy because it does not find a suitable method among the 175 billion machine learning parameters?
Author
Publisher Name
Beyond-Print.net

Founder and CEO of zipcon consulting GmbH, one of the leading consulting companies for the print and media industry in Central Europe. The technology and strategy consultant and his team actively support practical implementation in a wide variety of customer projects. His work involves developing visions, concepts and strategies for the players active in the print production process across a wide range of industries. His areas of expertise include online print, mass customization, strategy and technological assessment for print, and the development of new strategies in the print and media environment. Bernd Zipper is the creator and chairman of Initiative Online Print e.V. and, in addition to his consulting activities, is an author, lecturer and sought-after speaker, orator and moderator. His visionary lectures are regarded worldwide as trend-setting management recommendations for the print and media industry. (Profiles also in Xing, LinkedIn).

Leave A Comment

Data protection
, Owner: (Registered business address: Germany), processes personal data only to the extent strictly necessary for the operation of this website. All details in the privacy policy.
Data protection
, Owner: (Registered business address: Germany), processes personal data only to the extent strictly necessary for the operation of this website. All details in the privacy policy.