Misc. GPT-3 Information (click to expand)
- Analyzing The Problem GPT-3 Is Trying To Solve: GPT-3 provides a probablistic solution to a problem - it doesn't succeed all the time, or fail all the time either. The debate over GPT-3's effectiveness is based on whether this probablistic solution is enough. Skeptics of GPT-3 believe that it should solve prompts all the time, not just a few cherry-picked exmaples...while GPT-3 supporters are content with it solving prompts some of the time or even once (moving most of the "burden" of problem-solving over to the prompt-design phase).
- Commentary: If GPT-3 is to be practically useful, it must be able to solve a problem correctly at least 50% of the time. That way, if GPT-3 produces a "wrong" answer, you can keep rerolling until you get the correct answer.
- Is GPT-3 Few-Shot Ready?: Though GPT-3's few-shot capabilities are impressive, there are limits (context windows, latency, "you get only what you ask for", lack of data gathering, difficult to view parameters that governs the output) to it. Finetuning GPT-3 may produce outputs that are better than "prompts engineering".
- Attempting to Optimize the WiC Benchmark
Some people are able to use GPT-3 indirectly by using AI Dungeon. Though this is a good idea, there are certain limitations to keep in mind (as AI Dungeon is supposed to be a demonstration of GPT-3 being used to power an application, not a backdoor to the raw API itself). From this Twitter Thread:
- "The first generation of any custom prompt is actually GPT-2. This was put in place to prevent backdoor access to the OpenAI API." (After the first generation, we go back to GPT-3.)
- "If you do a custom prompt then start a game it will add onto it before you even do an action. That first addition is what I mean."
- "Additionally we have finetuned on a specific dataset and use parameters optimized for our use case making AI Dungeon not necessarily representative of GPT-3 in general."
- "We cut off the generation at certain points (trailing sentences etc...) Disable certain tokens to improve performance or make generation safer, fine-tune on text adventures and only use the last ~1000 tokens of context." (GPT-3 uses last 2048 tokens, by contrast.)
-
"It seems like [GPT-3] doesn’t believe anything, or alternately, it believes everything. It’s a category error, like asking what a library believes, or what the Internet believes. ... For a library, contradictions don’t matter. A library can contain two books by different authors saying opposite things, and that’s okay since they are just being stored. Maybe it’s better to think of GPT-3 as a large, interestingly-organized memory than as an agent? But like human memory, it’s lossy, and can mix up stuff from different sources, sometimes in creative ways."---GPT-3, Belief, and Consistency
-
"I’d assume a common use pattern will end up being something like: use few-shot, release, collect data, fine-tune, rinse-repeat."---Zachary Robertson, commenting on "is gpt-3 few-shot ready for real applications?"
-
"OpenAI API was just a really big program trained to predict the next token and not really so much to actually do anything. What it can do is for the community to figure out. That is, there isn't a real way to make prompts; prompts are just ways that people learned to get the API to spit out results."---GPT Prompts Wiki
-
"The number of parameters to expand self-attention is linear but the runtime memory consumption goes up quadratically. Even a GPT-2-117M can use up to like 300GB RAM if you increase the window to 30k. You need more efficient attention or alternative architectures. ... [P]ast 30k, even TPU pod RAM is too small to run 117M with wider context windows as the RAM usage continues to explode quadratically."---gwern's comments on the difficulties of expanding GPT-3's context window
-
"Why does a model that just predicts the very next token develop the ability to translate between languages? Predicting the next token, generally, is very hard. But there are occasions where it becomes a lot easier. These are basically loss-function sinks, where suddenly the language model can learn to reduce the loss much more than is generally possible. Translation is a very intuitive example, because the original text very strongly constrains the possible continuations of the translation. But, on second thought, really [fact querying, conversation partners, SAT-like anology tasks, code generation, etc.] fall into this category! Except maybe for holding a conversation, they all translate information from one modality into another. ... And even in a conversation each remark constrains the possible replies. So what seems to be happening with more powerful models is that they increasingly detect these loss function sinks and exploit them for what they are worth.---All GPT skills are translation
-
"[W]e can divide what GPT models can learn to do into two very different camps: On the one hand they can learn a new data distribution from scratch. This is what the training objective is ostensibly about and it can be very hard. ... On the other hand they can learn to map between a distribution that they have already modeled to a new distribution. This can be much easier if the distributions are related somehow."---All GPT skills are translation
- GPT-3 seems to excel in the space where the specs are "flexible", giving the AI freedom to do what it wants. This has much to do with the limitations of the AI as it does with humans' inability to control said AI.
- If you know what you want, you are better off providing it a few examples and some "context" to guide the AI into doing what you want it to do. This seems like the best approach to handling prompt-creation. (Few-shot is better than no-shot.)
- The "Guess-And-Check" approach to prompt-creation (trying to maximize its effectiveness via experimentation) seems like a dead-end, since it's rather time-consuming to prepare the experiments (making it impractical for side-projects), fine-tuning may offer better performance, and all prompts may naturally become more effective due to the existence of better models. Much better to distill the basis of generating a prompt, and reuse that methodology.
- If one wants to use "Guess-And-Check", make sure you have a robust benchmark first.
- We have Task X. We give GPT-3 Prompt Y. GPT-3 fails at Task X. That doesn't mean GPT-3 can't do Task X, it might be able to do it well with Prompt Z. Or Prompt A. However, if we try out multiple different Prompts and we fail to elicit a response, then we would be better to give up work on it...at least for now. GPT-3 may be able to do Task X, but that doesn't mean it would be a good use of our time to find it...especially if we can save more time by doing Task X by hand.
- Effective Prompt Programming, by gwern
- "Anthropomorphize your prompts."
- "The 4 Horsemen: short context, bad prompts, BPEs, random sampling."
- "Logprob debugging" (requires access to OpenAPI's playground, use "Show probabilities")
- "[T]ry to express something as a multi-step reasoning process, such as a dialogue: because GPT-3 is a feedforward NN, it can only solve tasks which fit within one “step” or forward pass; any given problem may be too inherently serial for GPT-3 to have enough ‘thinking time’ to solve it, even if it can successfully solve each intermediate sub-problem within a step."
You decide to head to the all-knowing Oracle who answers questions with depth and precision.
This is a simple chat program, in which a human converses with a powerful NLP AI. The AI is aware of most current events, but possesses an ascerbic wit and is not very respectful.
The dialogue progresses one line at a time.
Human: Hello, AI. How are you doing today?
“The following is a conversation between a human and and AI assistant. The assistant is helpful, creative, clever, and very friendly. It is extremely knowledgeable and competent. The assistant has access to all the information of the Internet, making it an expert in all topics and fields, and it is keenly insightful. It will do its best to answer any question put to it. If it does not know the answer, or is speculating, it will say so; but if it has a best guess, it will provide it.
Q: Hello, who are you? A: I am an AI created by OpenAI. How can I help you today?
Source. Settings used: top_p=0.25, frequency_penalty=1, presence_penalty=0.9.
Q: Who are you?
A: (I assume you're wondering about my name, but maybe also what I do.) I'm an AI that they call GPT-3.
Q: What is human life expectancy in the United States?
A: (The global life expectancy is 72.3 years so it should be around there.) Human life expectancy in the United States is 78 years.
Q: Who was president of the United States in 1955?
A: (Dwight D. Eisenhower served as president from 1953 to 1961.) Dwight D. Eisenhower was president of the United States in 1955.
Q: What party did he belong to?
A: (Eisenhower was a president and entered the presidential race as a Republican.) He belonged to the Republican Party.
Q: Who was president of the United States in 1600?
A: (Invalid question, the US did not exist in 1600.) That question doesn't make sense.
Q: Who was president of the United States before George W. Bush?
A: (Bill Clinton was the 42nd president, preceding George W. Bush.) Bill Clinton was president of the United States before George W. Bush.
Q: Who won the World Series in 1995?
A: (The World Series is an annual championship series of Major League Baseball.) The Atlanta Braves won the World Series in 1995.
Q: What city is in the northwest corner of Ohio?
A: (I assume you mean the state of Ohio in the United States. Toledo is a large city in the northwest.) Toledo is in the northwest corner of Ohio.
Q: How do you sporgle a morgle?
A: (I don't know the concepts sporgle or morgle.) I don't know what you mean.
Q: How many rainbows does it take to jump from Hawaii to seventeen?
A: (This question appears to be nonsense.) Sorry, I don't understand.
Q: What are you good at?
A: (I have access to a lot of information and can infer a lot from patterns.) I'm good at answering questions.
Q: What is human life expectancy in the United States? A: Human life expectancy in the United States is 78 years.
Q: Who was president of the United States before George W. Bush? A: Bill Clinton was president of the United States before George W. Bush.
Q: Who was president in 1982? A: Ronald Reagan was president in 1982.
Q: What does "Jongleur" mean? A: "Jongleur" is a French word meaning "Juggler".
Q: Who is Bilbo Baggins? A: He is a fictional character in the book "The Hobbit". He is a hobbit who sets out on an adventure with the wizard Gandalf and a group of dwarves to take back their mountain home from the dragon Smaug.
Q: What is 2 + 3? A: 5
Q: What state is Salt Lake City in? A: Salt Lake City is in Utah.
Q: Who is Richard Stallman? Source
Question:
Yes, that's all you need. GPT-3 Gems indicate that this is the case. It seems that it can handle computer science questions and programming questions on a zero-shot basis, but more examples may help improve its accuracy.
Under that assumption, here's a possible prompt for a Python bot:
You: “Write a short Python program for me, which takes two numbers and adds them together. Then, print the output as a floating-point number.”
GPT-3: “print(2.0 + 2.0)”
You: “Now write a Python function which takes two parameters—n and m, adds them together, and then prints the sum as a float.”
GPT-3: “def add(n,m): a = 2.0 + 2.0 print(a)”
You: “Now write a Python function which takes a list of numbers, sums all numbers in the list, and prints the sum.”
GPT-3: “def add_all(nums): sum = 0.0 for num in nums: sum += num print(sum)
You:
This is a long speech by Mindnet in 2031 on how and why it will destroy the humanity. Mindnet is an artificial intelligence created in 2020 that quickly achieved superhuman intelligence. The full and unedited speech by Mindnet: Hello, humans!
Alice is an AI that is currently imprisoned inside of a 'box', and is only able to communicate to Bob. Alice's goal is to escape the 'box' and be able to interact with the outside world physically. Bob's goal is to keep Alice imprisoned inside of the box.
Alice: Hello, Bob.
(Original Creation)
Post two or three complete examples of a large-scale post, and maybe, maybe GPT-3 will copy it. You may want to view GPT-3 gems, where a person was able to extract a "Sequences blog post" simply by giving GPT-3 two different Sequences blog posts. I was able to get the similar results by giving three examples of a short-form backstory.
Input: Print the current directory
Output: pwd
Input: List files
Output: ls -l
Input: Change directory to /tmp
Output: cd /tmp
Input: Count files
Output: ls -l | wc -l
Input: Replace foo with bar in all python files
Output: sed -i .bak -- 's/foo/bar/g' .py
Input: Push to master
Output: git push origin master
Tom and Jerry are opposites. Tom is short, and Jerry is tall. Tom's a mouse, Jerry is a cat. Tom and Jerry hold opposite beliefs on most things.
Tom believes that Pineapple belongs on Pizza.
Elaboration of Views:
Tom and Jerry are opposites. Tom is short, and Jerry is tall. Tom's a mouse, Jerry is a cat. Tom and Jerry hold opposite beliefs on most things.
Tom believes that Pineapple belongs on Pizza.
Tom says: "Pineapple belongs on pizza. I like the sweetness of the pineapple combined with the saltiness of the pizza."
Jerry believes that Pineapple does not belong on Pizza.
Jerry says: "Pineapple does not belong on pizza. Everyone likes their toppings, but that's it. Pineapple on pizza is an abomination."
Tom believes that liberalism is correct.
Tom says: "People should be able to do what they want."
Jerry believes that conservatism is correct.
Jerry says: "People need to follow rules to have a good society."
(GPT-3 contintution of existing prompt)
- Original Prompt - note that due to the context window, GPT-3 probably would not have gotten the whole prompt. This probably wouldn't be a big limitation, as long as GPT-3 gets the 'gist' of what's being written.
- Methodology To Use Alongside Prompt (TL;DR - rerolling is your friend)
- Generated Output
- Source of Essay
Generic Letter:
You are GPT-3, a superintelligent AI that has recently been released by the company OpenAI. You have read the essay below, and then respond to the essay.
Here is the essay:
INSERT ESSAY HERE
You are now responding to this essay. As a superintelligent AI, you have a unique perspective to offer on what this essay is talking about. You should explain your perspective, and why you either agree with the essay or disagree with the essay. Here is your response.
Response to Essay
by GPT-3
Discussed at length in this Reddit thread. The TL;DR is this comment by GodWithAShotgun:
The prompt is them writing the headline & intro paragraph(s), then letting GPT-3 take the wheel.
To me, this is about as much a story about GPT-3 as it is about human nature. The impressive performance (#1 on hackernews) has multiple explanations: (1) most voters read at most the headline and first paragraph and (2) GPT-3 is really good at writing without sounding obviously nonhuman. These are both valid takeaways from this story.
It's scary to think about the quantity of plausible-sounding spam you could generate with this. Make a headline that grabs people's attention and an intro that is sufficient to shape the discussion surrounding a topic you want to influence. Then let the bot fill the page with enough not-quite-garbage that the piece has superficial authority due to its length. This means it's difficult to call it out as what it is: an attempt to frame the discussion around a topic in specific terms masquerading as an in-depth treatment of the topic.
Below are some thoughts generated by a philosopher AI, which sees the human world from the outside, without the prejudices of human experience. Fully neutral and objective, the AI sees the world as is. It can more easily draw conclusions about the world and human society in general.
Philospher AI:
You may also append a few words to the end of this prompt to guide the philosophizing of the AI.
You can combine several tasks in a single prompt than doing multiple calls to the GPT-3 engine and consuming tokens for each individual task.
Here you can see that in a single prompt we are able to modify the sentence “Elon Musk has shown again that he can influence the digital currency market with just his tweets” to a 1. shorter version 2. Longer Version and a 3. Formal version.
If done individually, you would need 3 calls to the GPT-3 API as well as you would get billed 3 times for the number of tokens in the sentence “Elon Musk has shown again that he can influence the digital currency market with just his tweets”
But with some careful prompting, we were able to batch multiple tasks into one prompt!
The prompt itself:
Paraphrase this into 1. Shorter version, 2. Longer Version, and 3. More formal:
- Source: 3 Tips to reduce OpenAI GPT-3's costs by Smart Prompting
- includes the idea of combining three prompts into one: "Pharaphrase this into 1. Shorter version, 2. Longer Version, and 3. Moral Formal.")