OpenAI's GPT-4 consistently "beats The Turing Test" in the news. But we are very clear that the "predicting the next word" mode adopted by GPT has obvious limitations, the strengths and weaknesses of GPT-4 are qualitatively the same as before. The problem of hallucinations is not solved; reliability is not solved; planning on complex tasks is (as the authors themselves acknowledge) not solved. I think it's time to explore new Turing test methods.
One of the better ways I can think of to determine if AI have human thinking skills is by asking questions about unsolved problems in mathematics. By raising the question that there is no answer yet, in other words, by putting AI and humans in the same situation with complex & unsolved problems, to detect whether it can have original thinking and complex reasoning capabilities.
i.e.
- GPT-4's answer:
- Bard's answer
As the result showed GPT4 has read discussion over such well known problems. So this output could be synthesized from memory rather than from intelligence.
If you try to teach it something you just made up, you can have a better gauge over its intelligence. In the linked example, I accessed that its intelligence is roughly equivalent to a human between 9 and 15 years old. Not smart enough to come up with original insights into unsolved math problems yet, which is actually out of reach for the vast majority of humans.
https://twitter.com/kailuowang/status/1639703840814448641?t=vgBC_J8xEuvOUcLaAqGPrw&s=09