What is GPT-3?

The following text, an excerpt from an interview given by Ashwin Rao, an AI expert & Lecturer at Stanford. University, explains GPT-3 in a simple way:

First some background on OpenAI, the creator of GPT-3. OpenAI is a non-profit AI research company based in San Francisco and their mission is to develop Artificial General Intelligence that benefits all of humanity. GPT is the abbreviation for Generative Pre-Training and GPT-3 is the latest version they recently released. But the fact that GPT-3 figures out the correct code from a simple comment string is indeed very impressive. This has a lot of implications for fairly mundane coding activities we often do, and this will help developers focus on higher-value tasks and more creative work.

GPT-3 used a CommonCrawl dataset with about a trillion words and augmented it with some more web links scraping, books datasets, and English-language Wikipedia.  The big technical innovation in GPT-3 is an approach called meta-learning. Let me explain meta-learning by contrasting it with the current approach for language and vision models. The current approach is to pre-train deep neural networks with a large and diverse data set, and then developers fine-tune that model with their own domain-specific data. GPT-3’s meta-learning takes a different approach where during the pre-training, the model learns the context of the different tasks in the pre-training dataset. During inference time, we are given an English description of the task and a very small set of demonstrations of the task. During inference, GPT-3 identifies and understands the task based on the meta-learning it had done during pre-training, and efficiently adapts to the task given at inference-time. This is called in-context learning and this approach doesn’t require any fine-tuning of the model from the developer. The developer simply has to provide a English description of the task and a small set of task demonstrations.

This is a promising step in the direction of how humans learn – in human learning, we are simultaneously learning a lot of different tasks and concepts in our lives, and we always understand the nuances distinguishing the various tasks and concepts. So when we are presented with an activity, we figure out the nature of that activity, and we quickly specialize ourselves to that activity.

GPT-3 is impressive. It has been made accessible to the world through APIs and currently GPT-3 is busy following commands like obituary writing, making a button with Watermelon colors etc. While GPT-3 can read this text and write a Sushi recipe book for my daughter in a few seconds, what it cannot do is write about the connection between itself and the Beirut Blast.

Let me tell you what is common between GPT-3 and the Beirut blast. Both demonstrate lack of thinking skills.

Wired wrote : GPT-3 often spews contradictions or nonsense, because its statistical word-stringing is not guided by any intent or a coherent understanding of reality. “It doesn’t have any internal model of the world, or any world, and so it can’t do reasoning that would require such a model,” says Melanie Mitchell, a professor at the Santa Fe Institute and author of Artificial Intelligence: A Guide for Thinking Humans. In her experiments, GPT-3 struggles with questions that involve reasoning by analogy.

GPT-3 lacks reasoning and concept blending. Concept blending is a way of thinking that we humans engage in where one concept is applied to another one in order to arrive at a new concept. For example, rebounding skills in basketball and your business is a form of concept blending. Read the article here. Imagine concept blending as mental models. Imagine it as Jugaad in a crude way.

And like GPT-3, the Beirut blast lacked critical thinking and reasoning. The main cause for the blast was a large amount of stored ammonium nitrate kept without the required attention. This is a thinking error that the world could have avoided. Like Wired wrote, GPT-3 spews nonsense, similarly the Beirut blast showed us how humans spew nonsense.

During my childhood days, sometimes we were left with a tad moist Diwali firecrackers. We would keep these in the sun to make it crisp enough to be fired up again. Sometimes the exposure to the sun would be too long & harsh and the firecrackers would burst by itself. We had learnt the value of the right amount of exposure to heat. The Beirut blast brought back those memories and awareness to the fact that some officials didn’t bother to analyse risks from the large, stored amount of ammonium nitrate.  

Lives have been lost. Hospitals, homes, offices have been damaged because it looks like some people didn’t think critically. It makes us think about the lessons that we can take from this event. Safety norms, processes, risk analysis, technological alerts for disasters etc. One lesson I see is that critical thinking and decision making is not meant only for the senior management, it is also for the floor manager & ground level workers. It is a skill that is a necessity for human beings.

Everytime I begin to write my column, I scan my weekly reading notes and events for the idea and context setting. Like last week’s article on Nudge theory, that was widely read, was triggered by Abhijit’s Banerjee’s research paper on Covid19 messages. My this week’s note has three points:  introduction to GPT-3, Beirut blast impact and a comment from a SHIFT course user. He said, “the course is extremely good & has very rich content, lots of matters and examples but what is the immediate gain?”

As a closing note, critical thinking is the blending of concepts through consciously training our minds. Thinking doesn’t happen by sitting in a corner. Thinking is developed through habits and ideas. While I am yet to act on the feedback for SHIFT, the release of GPT-3 and the Beirut blast leaves us with the thought that although technological advancement will improve our living but our lives are still dependent on the way we think or the way we lack thinking.  

Hello GPT-3, can you please write a note on the importance of critical thinking lessons for humans?

PS: The Beirut blast has left several people suffering. You can help by donating through this link : https://www.savethechildren.org.uk/how-you-can-help/emergencies/beirut-explosion