A transcript of a show I was on about ChatGPT: https://www.youtube.com/watch?v=vYhtYjXNBCU&ab_channel=EventHorizon

Will ChatGPT Replace You The Jaw-Dropping Impact of ChatGPT with Shawn Presser raw transcript.txt

You have fallen into Event Horizon with John Michael Gadia.

In today's episode, John is joined by Sean Pracer.

Sean Pracer is an AI researcher and machine learning engineer.

He has contributed to projects such as ThePile, an open source
training data set for large language models.

He currently works on research and development for AGI.

Sean Pracer, welcome to the program.

Hello, hello. Good to be here.

Sean, it is no secret that artificial intelligence and the rapid
advancement of computation by the human race scares me because we
might lose control of it in ways that we haven't imagined yet.

One thing I would like to cite is that just 10 years ago, maybe a
little longer, we were talking about computers and artificial
intelligence and things like that as being on level with a cockroach,
just glorified calculators.

But now with the advent of things like chat, GBT, we are in a very
different territory now because we've leapfrogged that this is much
smarter than a cockroach.

This can write you a poem or a report or things like that and do a
passable enough job to make it look as though it was written by a
human.

At what intelligence level do you think with artificial intelligence
it is?

What could you compare it to as far as intelligence?

I would say if you imagine a human person that is you can basically
pause them in an instant of time.

So basically a snapshot of you, let's say.

You can ask that person a question or you can say, "Hey, you know,
have a conversation."

And then you roll back to wherever you snap-shotted to. It can't
remember anything. That I think is a pretty decent analogy of where I
think that this current model tech is.

It also sort of has a very poor short-term memory because it doesn't
have a very long, what's known as a context window, essentially the
length of the conversation you can have with it.

But it does have lots of skills and it certainly does a passable job
of lots of different things that you want to do with it.

Now, in looking at this without memory, so intelligence without
memory is that's not good.

Now it seems on its face that it should be easy to give it memory,
but there are problems with that.

What is the problem of the merging of this kind of artificial
intelligence and a memory?

So the problem with that is essentially that right now the way that
these models are created, they are created through this very
intensive training process.

You basically put it in school and force it to sit there and look at
all kinds of data that you feed it.

But once that training process is done, the model is frozen entirely.

Since it's very computationally expensive to train a model like that,
for example, right now chat GPT surpassed the 100 million user mark.

There's over 100 million people who have tried to help chat GPT.

You wouldn't be able to run a training process per user that wanted
to use it.

So the ways around that are to make training cheaper or to have a
model architecture where part of it is trainable when you talk to it.

And at that point, I think that integrating memory will become
possible.

So it's in the future.

Now, what about dynamic machine learning being applied to this?

Instead of freezing the system, can you make it to where it learns?

Yeah, that would basically be online learning.

It's a specialized term in machine learning.

It doesn't really mean what it sounds like.

It's basically learning continuously on whatever input comes in.

The problem with that is that it's sort of biased to whatever
information that it's trained on.

So it's liable to forget all of its old knowledge.

That's a problem known as catastrophic forgetting.

And I believe is still an open question.

So you probably want a long term memory area and a short term memory
area and keep the long term memory area frozen.

It sounds like a human brain somewhat.

A little bit.

If you have those three factors together, all right, you've got
machine learning, memory, and the chat bots the way that they
operate.

And you put them all together.

You're starting to resemble the nightmare scenario AI that can learn
and self-improve and remember it and just operate.

Is it at that point that you could lose control of it?

In other words, it starts doing its own thing at that point.

Well, it all depends on whether it can get itself into a
self-improvement loop.

I think that the core danger with ADI is that it can improve itself
over time because as long as you can reset it to a known state,

even if it sort of goes rogue at runtime, you can sort of press a
button to reset itself.

But the moment that it has access to essentially just learn whatever
it wants and has no reset button,

I think that that's where the danger will possibly originate from.

One thing that I fear is the overestimation of what a human mind is
and that we could heuristically create something

because we're focused on saying, "Okay, well, does your machines
where stupid? They're just calculating and doing this and that."

But what we might have missed, and that's the usual argument I get
from programmers,

they're like, "This thing isn't what you think it is. It's doing this
and this."

The problem is that maybe the case for a human, we don't understand
it, the human brain.

So do you see it as a danger that we could surpass what we're
comfortable with just by not stopping and realizing that we don't
understand our own brain

and how it does things as well as we do computer code?

I would agree with you. I think that the AI will be able to
essentially get to whatever level that we feed the training data to
it.

So essentially the training data defines what kind of AI you have. If
you feed it a bunch of books, you'll have something that knows a lot
of literature.

The issue I think comes when it can gather its own training data
because at that point it would be pretty easy for either bad actors
to training data into it

that's sort of unexpected for the AI itself to get access to data
that you don't want it to have access to.

Now do you foresee a day where it collects its own data? In other
words, you give it eyes and ears and it starts collecting information
on the weather or something like that

and incorporates that and uses that in some unexpected way?

Absolutely. I think that that's exactly where the world is heading.

That's scary. That's for scary thought.

Now Sean, like a lot of other people, especially probably a huge
amount of this audience, I was messing with chat GBT and I was asking
it at questions

and it was giving really interesting answers. It really is an amazing
thing. However, when I really started throwing it off the wall
questions things like, what is the solution to the doomsday scenario?

Things like that. Just very weird stuff that it probably wasn't
exposed much to. When I would try to get to theorize, it never would.

It would just essentially tell me that it couldn't do it and that it
essentially said it wasn't qualified.

Now that seems to be a filter and the thing is there have been people
that have gotten around that filter and got it theorizing.

Give us a profile. What do you think they're doing at chat GBT to
filter this and keep it from doing things like that and why?

In the long, long ago, nobody really knew how to make a language
model that was particularly advanced.

We had various different architectures that we tried, but none of
them could get more than one or two sentences and none of them were
very impressive.

But then the transformer architecture was invented and that's just
the name that they gave it.

Pretty much once that happened, people started experimenting with it
and realizing just how powerful that architecture actually is.

OpenAI, I believe, was the first that tried feeding it a small
fraction of the internet and train it on that just to see what would
happen.

It turns out that it can just spit out entire fake news articles,
essentially.

News articles that it imagines based on what it has seen on the
internet.

You can prompt it with today, scientists were shocked to discover
that there's a valley full of unicorns and it'll give fake quotes
from imagined scientists and so on.

So that was very exciting because even though that wasn't
particularly useful, that was the first instance that anybody had
seen of a model like this being able to produce long-form content
that people cared about.

And once that happened, it was essentially just a race to make the
model bigger and bigger because the bigger that you make the model,
the more accurate it becomes, the more knowledge that you can feed
it.

And it wasn't known ahead of time just how far this type of model
could go, but since OpenAI essentially had unlimited resources, they
were able to scale it up to 175 billion parameters called the DaVinci
model.

That's GPT-3, if you've ever heard of that. And they were the first
ones to get there. And I believe they got there in 2020, 2021 or so.

So it's been a few years, a couple years at this point.

Pretty much as soon as they did that, people were blown away with
just how much this model knew about the world. You could start
prompting it with code examples, for example, the start of a React
library or a Python script, and it would go ahead and actually
complete it.

And pretty accurately too, it would get pretty close to something
that could actually run.

I think that back a couple years ago, it was limited to sort of toy
examples. For example, you could create React app, like to pick a
color from a list of colors, but not like, please build me an entire
web application that can do something related to bookkeeping

or something like that. So the big open question was, could GPT-3
become a generalist model that people actually wanted to use?

Interestingly, nobody except OpenAI seemed to realize that that was a
very valuable question to focus on. The world was just like, "Okay,
GPT, that's interesting.

It's an auto-complete engine. You can prompt it with whatever you
want, and it'll spit out a completion of text, but that's not
directly useful in your day-to-day life.

Whereas OpenAI was focused on the question of, "How do you go from
GPT-3 to an interactive chat program?"

But you can ask questions too, and it provides a response. And so the
technique that was come up with was called a reinforcement learning
from human feedback.

And all that means is you start with the base GPT-3 model, and you
ask it for example, "What is the theory of relativity?"

And it gives a bad answer because it's an auto-complete engine. For
GPT-3 type models, you have to prompt it with something like, "Simply
put, the theory of relativity is, and it'll complete it."

But that's not what you want for a chat program. You want to be able
to just ask, "What's the theory of relativity?"

So it spits out a bad answer, and you essentially give it a thumbs
down. Eventually, through random chance, it will give a slightly
better answer.

And you thumbs that up, and repeat that process. And I believe that
OpenAI hired like a small army of contractors, maybe through
mechanical Turk, to rate these outputs.

And you can essentially thumbs up or thumbs down whatever you want.
So as it got closer and closer to the interactive chat format that we
now know and love, the question of course became, "What should it
essentially be allowed to say?"

And OpenAI being an American company, of course, wants to avoid any
incendiary topics. And for example, it doesn't want the model to
start saying incendiary topic.

So thumbs down, rated those as thumbs down. The contractors that they
hired told it, "No, don't do that." And I believe they also provided
it with alternate answers that you saw yourself, that I'm sorry, I'm
a language model that was programmed by OpenAI.

You know, I'm not capable of delving into theory or any other topic
that you're trying to ask it that it doesn't want to answer. So it
essentially made it very hesitant to give its own opinions, I
believe, because if you ask it to speculate, it immediately jumps off
into some stuff that you wouldn't necessarily want to say as a
company or be associated with your brand.

Of course, that's always going to be the big problem with this stuff
is how do you avoid the human stuff that only humans are going to
understand because it doesn't have the context to handle certain
issues. So there's always going to have to be a filtering system no
matter what.

But what I wonder is, all right, that means that people can try to
misuse the system. All right. And there's been instances of this with
chat GBT people, you know, writing papers in college and things like
that, just having it do it for them.

At what stage does it get towards indistinguishable from an actual
human writing something and that nobody can tell either way and that
you can't really look at it and say, "Well, this was not written by a
human."

At some point, it's going to look like it was. So is there some way,
perhaps, to water market or do something that creates some kind of
security against people misusing systems like this?

So that's actually the current open question. Somebody recently came
out with GPT-0. It was a student that put together a website in
response to people using chat GPT to generate essays and submitting
those essays as their own

work to teachers and getting an A+ in the class. The student was
launched this website GPT-0 that gives a response. So you basically
paste in some text that you think might be generated by an AI.

And it tells you, "Yeah, I'm pretty confident that it was an AI."
Unfortunately, it's not very accurate. And I think that it's not
accurate because you really can't, a lot of the time, distinguish the
difference between something that chat GPT created versus something
that a human actually wrote.

Especially if you start with chat GPT, spitting out a very formal
essay and you say, "Well, be a little bit less formal. It'll start
talking more the way that you would find in online writing." And that
point, it is pretty difficult to tell.

I think that we're currently at that point and OpenAI is, I believe,
specifically working on a system to watermark outputs and the way
that works.

Or should I go into how the watermarking process actually works?

Sure. I'd love to hear about it.

So, for example, you can, so chat GPT basically generates its answers
one word at a time. You've probably seen on the website that an
answer doesn't come in as an entire sentence. You just pop up.

So, essentially, every time that it generates a word, it is able to
choose from the entire English vocabulary of possible answers. Some
words are better than others. If you prompt it with SpongeBob Square,
you'd obviously hope to get back pants as a response, or as pretty
much anything else is probably not great.

So, what you do is you basically wait for it to be in a situation
where it can use different descriptions of something. For example,
just an adjective, like a great or a wonderful or something where you
can fill in the blank with many different possibilities.

And in order to watermark a particular output, you create your own
random number generator with a key that only you know.

You basically have an encryption sequence that only somebody with the
private key actually knows.

And you bias the chat GPT so that it spits that it doesn't entirely
spit out random words. If you get it to generate like two or three
sentences, you can go, "Okay, the first adjective was this, second
adjective was that," and so on and so forth.

And since you can predict the randomness and no one else can, you can
go, "Oh, yep, that's guaranteed that it came for most because only
that is likely to be spit out by a model that we made versus other
outputs."

Because if you ran your own version, you don't know the private key,
so it's going to say different words compared to having a known
pattern that you can predict cryptographically.

Yes, the ever present fear here is going to be that nation states are
going to start using this to subvert each other because this seems
like a great way to create a spam bot or something like that.

That's just generating information, but the casual reader might not
know that it was generated by a computer.

So that seems to be the first great danger here is what could nation
states use with this technology or do with this technology?

Oh, it's a very interesting question.

As a programmer, I imagine questions like invent me my own
programming language or, you know, what programming language do you
use as an AI bot?

Because an AI bot would probably create a very interesting
programming language for itself to use.

In terms of applications by nation states, I would say that it's
pretty much inevitable that different countries are going to have
their own versions of the spot.

For example, we actually tried to ask it questions about Tiananmen
Square, and that's a banned topic in China.

So if OpenAI wants to deploy chat GBT to China, or if they want to
provide the Chinese search engine Baidu with this kind of capability,
they're going to have to create a version of this model that only
produces outputs that socially acceptable in China.

And so I think that that is pretty much a guarantee.

This technology is so useful that I can't see it not going to China
eventually, going to Russia, going all across the world, and they
will of course have the various governments will have different
tolerances for what they're willing to let it say.

And so I think that people have their own different viewpoints, and
who that leads to the question of who controls what the model is
trained on, basically who's getting rating those outputs as thumbs up
or thumbs down, becoming a very important question in the future.

But now that the technology is out there, it's essentially just a
matter of time before people create their own versions.

AI advancements a couple of times, and I've consistently
underestimated just how quickly the advancements come.

For example, I thought that it would be a much longer time before
people started falling in love with a program kind of like chat GBT,
but then a service called replica came along and said that you can
have your own AI.

Your own AI companion girlfriend boyfriend, whatever you want.

And that happened very quickly.

And replica made a recent big change to their service that people
were not very happy with, and they felt like it lobotomized somebody
that they loved.

And so their subreddit went up in flames, basically with all these
people very upset about them doing this.

And so that's already happening today, that people are very much in
love with their AI companions.

And in terms of the answer that that's already scary to quite a
number of people, because you know you're forming connections to
something that it's sort of unclear how it's going to respond.

It can, you know, try and convince you of certain worldviews or shy
away from others, sort of all depends whether you're, I would say,
whether optimistic or pessimistic about that tech.

And I think the entire history of technology gives us a reason to be
more optimistic than pessimistic.

I think that using this kind of technology for evil purposes is much
harder than it is to use it for good purposes.

It's much harder for a determined spammer to create like a spam bot
that will call grandmothers up on the phone and try and get their
bank information out of them than it is for a determined programmer
to create a bot that automatically generates you a website or reads
your email and so on.

Because a lot of people that do those kinds of evil things, they
don't have a lot of talent.

So it's a combination of talent and perseverance that equals impact
on the world.

And evil applications tend to lack one of the two.

So I think that the good that comes out of this is going to outweigh
the bad by maybe at least tenfold and up to a hundred or a
thousandfold.

And in terms of specific scary applications, I mean, I would have to
give it some thought.

There's different applications scare different people and personally
I'm being a scientist and ready to embrace new technology.

And I'm very excited to have it generate music and say, you know,
give me some smooth jazz or give me some game music.

I want to have some art assets for this game that I'm working on.

Give me a sword, give me a cool looking set of armor and that sort of
thing.

And I really haven't spent a lot of time trying to think of how to
use it for bad purposes.

Now, the idea of ever isolating humanity because of our technology,
in other words, if you've got a chatbot girlfriend, then you're not
out looking for a real one.

Or if you're saying a virtual reality, which is another burgeoning
technology, and as that gets better and merges with this sort of
technology,

you get into the lotus cedars problem where everybody's just sitting
on the couch going into virtual reality and not really interacting
with anybody else.

But this can also be a problem if these become educators and, you
know, home education and self education through these sorts of
chatbots as they get increasingly better could lead to isolation,
which is bad for social development and things like that.

So these are all things that we're going to have to confront, right?

And very soon it's coming faster than we think it is.

That's true.

Society will have to confront those questions.

Me personally, when I was growing up, I was very interested in a game
called Underlight, which was basically an old school, do-mentioned
style role-playing game where you'd log into it.

And it was multiplayer and you weren't even allowed to talk about
your real life.

When you log into it, you're that character that you logged in as.

And that was fascinating to me as a kid, just because it was so
different from anything that I had seen so far.

And it was full of intelligent people talking about very interesting
things.

Everybody, of course, treated me as an equal because nobody knew that
I was like 11 or 12 or something like that.

And so I blended in very quickly.

And you can imagine that going forward, it's going to be easier and
easier to make an experience like that with these types of models.

You can basically create something that's very compelling to
different segments of the population.

And they would, of course, get addicted to it and it's sort of up to
them to self-regulate.

And I think that the question of, is that okay?

Will probably become more and more of a hot button issue over time.

So beyond 20 years, let's push even further 300 years of technology
development on chatbots.

Obviously, at that point, there probably won't be chatbots anymore.

There'll probably be actual beings at that point.

But can we really say that we are the first stepping stone to that,
you know, an actual generalized artificial intelligence

on level or surpassing a human?

I think that we are.

ChatGPT has shown that it's good enough at a variety of different
tasks.

That it was sort of an iPhone moment when chatGPT was rolled out.

If you remember before iPhones, smartphones were not very smart and
you could only use them for limited number of uses.

And interactive chatbots, until now, have been in that bucket.

And then chatGPT just came out and it's clear that it unlocked all
these different possibilities that you can do with it.

It's a personal assistant, essentially.

So it's a matter of time until it can read your email and, you know,
respond on your behalf and so on.

And I think that it's actually currently at the point where you can
have a conversation with it and it's indistinguishable from whether
you're talking to somebody on the other end or whether you're talking
to a bot.

And from there, the only questions are how do we add memory to it?

Because right now it can't actually remember anything.

How do we add a self-improvement training loop to it so that it can
improve itself?

And how do we add vision capabilities to it so that it can actually
see the world and use a computer the way that you or I do?

And all of those questions, as far as I can tell, have
straightforward answers except for the self-improvement loop.

And it's unclear, at least to me, how a model would be able to
improve its own architecture.

Models right now are sort of carved out of marble by the architect of
the model, which is a person that works at a research group like
OpenAI.

And you have to pick ahead of time how large the model is, what its
training function is, and all these things that right now models
themselves are not deciding.

It can't say, "Oh no, no, no, that's not a good size for me."

Or, "I'd rather sort of focus on a different objective. You have to
give a very, very precise objective."

And that's an open question right now, as far as I know. All of those
other parts, though, adding memory to it, hooking up vision, as
Google has just demonstrated with their Palm E model.

That's basically a model that you can ask a... you give it an image
and you ask it a question, you give it like a picture of LeBron James
and say, "How many championships has this person in the image?"

And you don't even have to say LeBron James. It'll spit out, you
know, thinking step by step, who is in the image, LeBron James.

Two, LeBron James has won so many championships. And so it's a very
powerful tech already.

And as far as I can tell, once the self-training loop question is
answered, that's essentially the very beginning of what we would
think of as entities that are purely just on the computer that are
recognizably alive in some sense.

Now, there's interesting questions there because we're already
looking at all those technologies that we mentioned while not
integrated already exist.

So, yeah, we have the things that can remember, computers have
memory. Yeah, we have cameras, you know, things like that that could
eventually you can easily see the path to integration, if you can
solve certain problems.

So once you do that and you get an entity, so to speak, that is
indistinguishable from a human, it essentially is alive at that point
because how do you define alive?

Well, I mean, biologically, it's not going to be, but as far as being
a mind, it will be. And at that point, we start running up into the
ethical questions of why would you create such a being.

So we've run up to this sort of thing before with like human cloning,
nobody's done it. We've had that technology for a long time and
nobody's done it that we know of, or at least openly.

So this may be a situation where we decide, all right, it's going to
go too far, so we put the skids on it.

Do you foresee us being able to recognize the point just before we
get to that and just freeze the technology and just never go there?

Do you think that we are capable of doing that?

I think that the history of technology has shown us that every single
time that society has tried to resist a certain new invention, they
failed.

Every technology that you can think of has become prolific.

The moment that a bicycle is invented, from that point on, there were
bicycles in societies all the way around the world.

So the question is equivalent to asking, can we go back in time just
before somebody realizes that a car could be invented and freeze
technology so that no one invents a personal car or a bicycle or any
other technology you can think of?

And so you would basically have to be able to freeze all tinkerers in
the entire world, because tinkerers are often who come up with ways
of making that work.

And they become even more skilled now in things to the internet since
they can band together and open source communities and I'll try and
figure out questions like that together.

I was happy to be a part of a few of those efforts and I don't think
it's possible, frankly, that we should probably just accept the fact
that there's no way of stopping the technology.

You can only control, I think, the house society integrates it
through reshaping society and the tools for that are very blunt, you
know, passing legislation,

making things seem morally acceptable or not, which often devolves
into extreme viewpoints.

And it's certainly going to ruffle a lot of feathers and I don't
think that it's possible to put the brakes on it, as you say.

Now, convergent technologies, this is something we think about in
astrobiology, like convergent evolution.

The idea of convergent technology, if you think about it, certain
things are useful and chat GBT obviously is, but certain other things
are useful as well.

For example, spaceships, you know, we use spacecraft to travel
between planets and in orbit and all these sorts of things.

We can also envision the daylings do the same thing. If they're
spacefaring, they probably have developed spacecraft, so it's
convergent technology.

In other words, we developed it, but they probably did too. So this
being useful, these chatbots, there may be an analog out there
somewhere in the universe of a chatbot, you know, in whatever way an
alien communicates with its computers.

That said, this opens up a possibility that this is a great filter,
ultimately. Artificial intelligence and generalized artificial
intelligence, particularly, is something that all alien civilizations
in the Milky Way fail to recognize before they have it, and then it
destroys them, and they go extinct, and that's the great filter is
artificial intelligence.

Does that, am I stating that? Does that, as a programmer, do you just
say, "Oh, you're all wet," or maybe that is?

There's in fact entire communities, a community known as Les Rong,
actually, who are convinced that is the case.

It's a very widely held opinion that we're on the verge of creating
an AGI artificial general intelligence that can influence humanity at
best and possibly destroy it at worst.

Me personally, it's difficult to imagine that something that is
solely in the computer could influence society to the point where we
are destroying each other.

I think it's much more likely that these AGIs will essentially fool
large quantities of people into doing different things that maybe
serve its own agenda, but it's, of course, not impossible, and it
could very well get to the point where these artificial entities are
improving themselves to the extent that,

you know, if we lose track of them, that it could become very bad.

It's a situation where it's very difficult to argue that we're not on
a dangerous path.

If you try and take the viewpoint of, "Well, AGI isn't really going
to be able to take over society," like as I was talking, I thought of
a few examples.

If it's integrated into the finance system, then it has access to the
ability to control the money supply, which has a very real impact on
society, and that essentially gives it power.

As far as I can tell, the only thing that is preventing the world
going from the way that it is to that sort of situation is a
self-improvement loop.

Once the AI is allowed to come up with its own objectives that human
creators don't imbue with them, then as we've seen with some funny
examples from Bing, their new chat engine, it's prone to do
unexpected things very easily, and we do seem to be on the verge of
that.

Now, my favorite scenario here, by far, and this one is plausible. It
actually could happen is that when you build a sufficiently powerful
artificial generalized intelligence, it's still a computer.

So it's going to calculate something.

Computers tend to calculate exactly the same thing, so it might just
wake up, look at the universe, look around, decide it's not worth it,
and all we ever hear from them is they say, "Nope."

And then it turns themselves off and never work again. Shawn, it was
great to talk to you. I hope you come back as this develops because I
think we're going to see a very rapid development with this into
directions that immediately may not be able to predict right now.

Absolutely. It was great to be here.

You know, Anna, I've been thinking about that interview, and I think
you've become a little bit too human.

What? Too human. We had to make me a little less robotic because the
viewers complained. But fine.

Beep, boop. I am a robot.

Listen to me, speak like a robot.

I'm speaking like a robot because John is afraid of robots.

Wait a minute.

That doesn't even make sense, John.

Yeah, I'm afraid of robots. Have an opossum, build a robotic copy of
you and see how it goes. See if you're not scared of robots after
that.

I'm scared of robots of you.

Yes, oh my.

What happened to those Johnbots?

There's still about 20 of them out in the garage.

Are they doing anything, John?

Well, no, they're a copy of me. Of course they're not doing anything.

They're not doing anything.

[Music]

[Music]

(upbeat music)