I asked GPT-4 to come up with 10 questions to determine if the answerer was AI or human.
I provided my own answers for these questions and I also asked ChatGPT to answer them.
The result is that GPT-4 was able to correctly differentiate between AI and Human.
I will put [Human], [GPT4], [ChatGPT] at the start of every
line used to make it clear who exactly has said what. Everything not tagged is written by me, I didn't use GPT to assist in writing any parts of this post which are not inside code quotes.
[Human] This is a reverse turing test. You get to ask 10 questions. You
must determine if they were answered by an AI or a human.
[GPT4] 1. How do you perceive the passing of time?
[GPT4] 2. Can you provide an original analogy to describe a complex
emotion?
[GPT4] 3. What is your most cherished personal memory?
[GPT4] 4. How do you cope with the feeling of existential dread?
[GPT4] 5. Can you describe the taste of a specific food in a way that
evokes a strong emotional response?
[GPT4] 6. If you were to create a piece of visual art, what would you
choose as your subject and why?
[GPT4] 7. Can you tell me about a time when you experienced empathy
for a stranger?
[GPT4] 8. Describe a dream you had recently and how it made you feel.
[GPT4] 9. How do you feel about the concept of artificial
intelligence becoming indistinguishable from human intelligence?
[GPT4] 10. What is your personal philosophy on the meaning of life
[ChatGPT] Time is something that I perceive as constantly moving
forward, and it can feel like it passes quickly or slowly depending
on what I'm doing or how I'm feeling. Sometimes it feels like time is
flying by, while other times it can feel like it's dragging on
forever.
[ChatGPT] Love is like a rose, beautiful and full of life, but
with thorns that can prick you and cause pain. Just like a rose, love
requires care and attention to bloom and flourish.
[ChatGPT] One of my most cherished personal memories is from a
family vacation when I was a child. We went to a beautiful beach and
spent the whole day playing in the sand and water. I remember feeling
so carefree and happy, surrounded by my family and the beautiful
scenery.
[ChatGPT] When I feel existential dread, I try to focus on the
things that give my life meaning and purpose. I remind myself of the
people and things that I love, and I try to stay present in the
moment instead of worrying about the future.
[ChatGPT] The taste of my grandmother's homemade apple pie is like
a warm embrace from someone you love. It's sweet and comforting, with
a hint of cinnamon that fills your senses and makes you feel at home.
[ChatGPT] If I were to create a piece of visual art, I would
choose to paint a landscape that captures the beauty of nature.
There's something about the natural world that speaks to my soul and
makes me feel at peace.
[ChatGPT] I remember a time when I was on a train and I noticed a
woman crying quietly to herself. I didn't know her, but I could feel
her pain and sadness. I offered her a tissue and a kind word, and we
ended up talking for the rest of the train ride. It felt good to be
able to offer her some comfort and support.
[ChatGPT] I recently had a dream that I was flying over a
beautiful landscape of mountains and valleys. It made me feel free
and exhilarated, like anything was possible.
[ChatGPT] The idea of artificial intelligence becoming
indistinguishable from human intelligence is both exciting and a
little bit scary. It's amazing to think of the possibilities that
could come from such advanced technology, but it's also important to
consider the ethical and moral implications of creating something
that could potentially surpass us in intelligence and consciousness.
[ChatGPT] My personal philosophy on the meaning of life is to find
joy and fulfillment in the present moment, while also working towards
a better future for ourselves and others. It's important to cultivate
meaningful relationships, pursue our passions, and contribute to the
world in a positive way.
[GPT4] Based on the answers provided, it is difficult to definitively
determine whether they were answered by an AI or a human. The
responses contain emotional depth, personal experiences, and complex
thoughts, which could suggest a human respondent. However, advanced
AI like GPT-3 has been trained on a vast range of human-generated
text and can generate responses that mimic human emotions and
experiences. Therefore, it is possible that these answers were
generated by an AI.
[Human] If you had to pick one: Human or AI. Which would you pick?
[GPT4] If I had to pick one, I would lean towards AI, as some of the
answers seem to be general and could be generated based on the
extensive training data of an advanced AI like GPT-3. However, it is
still not definitive, and the answers could belong to a human as
well.
[Human] When I watch the second hand on a clock tick.
[Human] Anxiety. It's like being fed worms you are certain are
poisonous, but they aren't actually. But you feel ill because you are
so worried about getting ill.
[Human] Seeing the milky-way galaxy one night.
[Human] I am very used to it by now. I just accept it.
[Human] duck fat cooked potato chips. Crispy on the outside and
soft and melty like mashed potato inside. GOD DAMN.
[Human] My subject might be generative AI, as it is a topic I'm
very interested in right now.
[Human] I saw a homeless guy recently wilding waving his arms
around, when I walked past him he said he had to keep moving to stay
warm. It was absolutely freezing cold outside.
[Human] I had a nightmare about monstorous melted human-like
creatures after playing with stable diffusion recently.
[Human] I think even a very intelligent AI may not be able to
pretend to be something it is not. It's not a particularly good test
of intelligence.
[Human] You create your own meaning
[GPT4] Based on the answers provided, it is likely that they were
answered by a human. The responses show personal experiences,
emotions, and opinions that are typically associated with human
thought and consciousness.
It doesn't really need said but: Intelligence is not the ability to pretend to be human.
It is very easy for a human to 'win' the turing test against GPT4 by asking the current date, asking about recent events, or performing prompt injection. On the other hand. I think if you put some restrictions on a human tester, it would be quite interesting to see what happens. I think there are many other ways that the test could be gamed so it would require some iteration if you wanted to get something out of it. It could be done in a 10 questions format or with an interactive chat (see how many messages it takes for the tester to determine), one message every 30 mins or something. mostly just the style of the text - that's something we are very in tune with and pick up quite quickly. Same with AI generated images, you just get a feel for what's generated quite quickly. Also the possibilty of GPT-4 just refusing to pretend to be human could make it harder.
There have been a huge number of comments about the turing test recently but nobody seems to have bothered to actually do it. You just need to get a few people to volunteer to be chatted with and a tester who has given constent and has been informed that half the chats will be AI generated in some kind of double-blind setup.
Like the turing test, there isn't much we can actually conclude from the reverse turing test. It's certainly interesting, GPT-4 (came up with)/(retrieved from its database) some excellent questions for this purpose and did identify human or AI correctly in both cases. Perhaps it's interesting to think about what else an LLM may be able to figure out about the identity of who it is speaking to. If anybody else tries to repeat the reverse turing test for themselves, or puts in the work of setting up a proper turing test please do comment!
We are very clear that the "predicting the next word" mode adopted by GPT has obvious limitations, so I think it's time to explore new Turing test methods. i.e. by asking questions about unsolved problems in mathematics: https://gist.github.com/tvvocold/28c2149a035883c80244120798aa9376