Interesting how using the word "yo" seems to be important to this trick, since it's not a common English word for this style of writing. This seems to be a way of exposing a deeper-level signal to the surface as a probability on a single word, bypassing the English interface.
Once you're doing tricks like that, it's getting a bit unclear whether it's really a natural-language interface anymore. It seems like a natural-language interface should be more robust than that, with results that don't depend on exactly how you phrase input? (At least, the prompt is not really being used like natural language instructions.)
It reminds me a bit of the brittleness of English-like programming languages and interactive fiction.
I briefly tested it with 'That doesn't make sense.' which worked, but switched back to yo because I'm looking at the probability of the first word and 'That' is too common to assume that the rest of the completion is 'doesn't make sense.'
I'd try with a few different responses according to the type of nonsense question: "That doesn't make any sense", "How could I know that?"/"No one knows that", "That's a matter of opinion", etc.
> Once you're doing tricks like that, it's getting a bit unclear whether it's really a natural-language interface anymore.
It's not really a natural language interface. It tries to predict the word that comes next in a stream of text.
It knows how to answer questions because it has examples of journalist interviews, etc, in its training data corpus. It also happens to know lots of facts because they occur in the training data.
Adding a "prelude" before a question you want answered to provide context is a completely reasonable way to tell it what kind of task you want it to perform. If you wanted to make a "natural language" interface with GPT-3 for a given domain, you could choose a prompt string to put before every question to get the type of answer you would like.
Alternatively one could fine-tune GPT3 for the exact kind of task one would like to perform. But the surprising thing about GPT-3 is that this is often not necessary: simply adding some clever context is enough.
Once you're doing tricks like that, it's getting a bit unclear whether it's really a natural-language interface anymore. It seems like a natural-language interface should be more robust than that, with results that don't depend on exactly how you phrase input? (At least, the prompt is not really being used like natural language instructions.)
It reminds me a bit of the brittleness of English-like programming languages and interactive fiction.