Yoshua Bengio Weighs in on the Pause and Building a World Model

CRAIG: Hi, I'm Craig Smith and this is Eye on AI. This week I talked to Yoshua Bengio, one of the founders of deep learning about augmenting large language models with world models and inference machines that would give AI systems the ability to reason on reality. We also talked about the famous pause letter of which Yoshua was perhaps the most prominent signer.

CRAIG: We are sponsored this week by NetSuite, Oracle's cloud-based enterprise resource planning software to help any business manage their financials, operations, and customer relationships in a single platform.

CRAIG: For the first time in NetSuite's, 22 years as the number one cloud financial system, you can defer payments on a full implementation of NetSuite for six months. No payment. No interest for six months. Take advantage of this special financing offer at www.netsuite.com/eyeonai. It's an unprecedented offer.

CRAIG: Yoshua, how are you?

YOSHUA: Very good.

CRAIG: I wanted to catch up partly on the whole large language model thing but also to talk about the letter, so I know you don't have much time. I'm going to jump right into it. first of all, on, on the letter specifically, the letter is triggered this FTC complaint.

It was made by the Center for Artificial Intelligence and Digital Policy. I don't know much about them, but it was a complaint to the United States FTC, asking them to stop open AI from releasing any more powerful model. It follows on the letter.

CRAIG: The first question I have is your name to me at least stood out among the signatories because you are known for being very measured in your comments and very reasoned in your view of the risks of AI and your skepticism about the advent of AGI or human level intelligence anytime in the near future.

ChatGPT and GPT-4’s capabilities took even people within the community by surprise, but certainly the general public. So why did you sign the letter? Many of your colleagues, Geoff Hinton and Yann LeCun, I could go on and on, did not sign it.

The people who did sign it, the prominent names who signed it Gary Marcus, Max Tegmark, these guys are known to be very vocal in their concerns much more so than you. So that's the first question.

YOSHUA: I've been speaking for many years about the long-term danger to society of having very powerful tools at our disposal. It's like any technology in the sense the more powerful they are, the more useful they can be, but also the more dangerous they can be if they're misused.

YOSHUA: And I had the impression for many years that the way our society's organized in each country, but globally in general, is not adequate to face the challenges that very powerful technologies bring, and in particular AI, but I'm thinking of biotechnology in the future, which is something great that I, I think, can be incredibly useful for humanity as well.

YOSHUA: But, but the more powerful it is, the, the, the more wisdom we need in, in the way we deal with this. And right now, for example, the system of competition between companies as we are seeing now accelerating with these large language models has benefits. potentially this has, you know, driven innovation in, in some ways, in, in the last century.

YOSHUA: But also, it means that companies are a bit in a haste and may not take the precautions that would otherwise be warranted. So, in the short term, I think we need to accelerate the, the, the countermeasures. To reduce risks, and that's regulation. Very simple. We do that for many other areas. You know, almost every industrial area is highly regulated.

YOSHUA: when it touches the public, you know, it's, whether it's airplanes or chemistry or drugs or, They're all heavily regulated, not so much computing and, and AI within that. And because these things are becoming very powerful, I think it's urgent.

We kind of really change gear here. Now why did I sign? Like why now? Like if maybe last year I wouldn't have signed this letter, it's because we now reached that threshold.

The threshold is a train test, meaning that we have systems that we can dialogue with, and we can't be sure if this is coming from a machine or a human. And that could be exploited in, in highly dangerous ways that can democracy - and I care about democracy.

CRAIG: Yeah. An interesting aside is the potential that AI has for enhancing democracy.

YOSHUA: absolutely like any other thing that AI can do. It could be very bad or very good, but right now, where are the investments going? How much investment is there in like AI tools for democracy?

YOSHUA: Not so much because it's not clear where the profit is going to be. Right. And it's true for other areas of social importance in healthcare. There is of course, quite a lot, quite a bit, but it's, it's minuscule compared to what is going on in, you know, what is invested by, by large companies who care about advertising recommendations, search engines, and, and so on.

CRAIG: You said that there's a need to accelerate regulation, that I think everybody agrees with, but the letter didn't call simply for accelerated regulation. It called for this pause.

Was that meant as many people believe? more to, to get attention because the, the reality, although I saw Chelsea Finn on Twitter saying she was going to stop work on anything that pointed towards, toward AGI.

CRAIG: But is that, is that pause, was it meant to be realistic or is it simply a way to emphasize the urgency?

YOSHUA: the chances that a request like this, you know, be followed by, by these companies was small.

And, I, I, I don't, I don't have high hopes, but the important thing indeed is to say that we need to coordinate and the, the short term coordination that can happen is between companies, maybe a few companies decide well, let's take a pause to improve our, you know, testing documentation, you know, ethics studies around what we're doing, but really, it, at the end of the day, it has to be in the hands of government. that's going to make sure that every company's going to do it.

And even, you know, more difficult, but also, but, but essential is that at the end of the day, it has to be international.

YOSHUA: It has to be international treaties where the main countries that can do these things agree just like we've done in the past for many international treaties where there are risks that could be global. From technology.

CRAIG: And then on this FTC complaint, and I know that you haven't read it, so I can't really expect you to, to comment on it in detail but since the letter was released, is there a plan for.

CRAIG: presenting governments with legislative language or regulatory language from future of Life Institute or, or

YOSHUA: the, the, the, like, I'm not representing the Future of Life Institute. Yeah, no, I understand. and I didn't even write that letter. I, I, I was sent it and asked if I would, I would sign it and I said yes.

YOSHUA: I proposed some change. It didn't happen, but I decided to sign anyways.

CRAIG: Well, let me put it this way. Is there a movement among people that sign the letter, support the letter? I mean particularly the, the, the most prominent people to pursue regulation with their independent governments. Is there any coordination at.

YOSHUA: There is not necessarily among the people who signed. I don't think it's organized that way, but there are lots of groups of people who have been working on this in the last few years.

For your information, the Canadian government is moving forward, a legislation that's probably going to be the first in the world to be voted, around AI probably this spring.

YOSHUA: Europe has been working on this for several years. it should come in 2023. So, loads and loads of scholars scientists and, you know, government policy, public policy people have been working on this for years. Yep.

I've been involved, in the creation and, and as for two years as the co-chair of, a working group of the global partnership on AI that's a new organization has been created, by France and Canada, which now has about, has about 30 countries. it, it has connections to OECD and it, it, it's, it's all about international coordination around AI.

And that working group was the working group on responsible AI. And a lot of the things we discussed for a few years now are precisely these things.

YOSHUA: and in fact, I just witnessed an exchange about very kind of precise suggestions regarding, following up on the letter, so regarding watermarking for example, and mandatory display of the origin of the content.

Is it human or if it's images, like was it really recorded or is it generated by machines, right.

YOSHUA: so, these are two different things and they're connected. So first of all, watermarking just means that the way that the words are generated, or the pixels are generated is not going to look any different to a human.

But a machine with the right program that could be provided by the company that, that does these things like OpenAI for example will be able to very, very easily say this with very high confidence -

YOSHUA: like 99.999%. Not, not some like guess, but like really be sure- this was generated by the, you know, GPT-4. Or, or something of that.

CRAIG: Yeah. Let me just jump in though. I'm, I'm fairly familiar with, with the regulatory efforts going on. but the letter didn't call the headline wasn't you know, speed up regulation.

CRAIG: It was to pause development.

YOSHUA: No, but look, don't read the headline. Read the…

CRAIG: Well, this is the problem. This is my problem with the letter as, as someone who's peripherally involved, is that I read the letter my brother, who immediately sent me a text, of course, didn't read the letter. He read the headline.

CRAIG: And the FTC complaint is not about speeding up regulation. It's about preventing OpenAI from releasing more powerful models for the time being. so, I, I mean, do you support that kind of action that is coming out of the letter?

YOSHUA: I agree with the pause. However, I think, I don't think it should be just OpenAI. And I don't think it should be only the United States, so it, it's going to take more time, but, but I, I really think that it has to move to the international arena as quickly as possible.

And by the way, China is also like probably ahead of the US in terms of regulation of AI for different reasons.

YOSHUA: But, but in a sense, they are as concerned by the destabilizing effects that these systems can have on public opinion as democracies are for, for different reasons. They don't want to lose power.

We want to make sure that the, you know, democratic debate takes place as it should. and, and so maybe there's an opportunity for.

YOSHUA: A an agreement even though it comes from different angles.

CRAIG: Yeah. Okay. I, I don't want to get too, too hung up on, on the letter. but on, I, I, I prefer talking about the research and the, and the developments. I mean, it has been impressive what's come out of the scaling of transformer models.

CRAIG: they're the only constraint at this. seems to be money and, and electric power. Some expertise and, and expertise. That's right. have but, but there are things missing in the model, and that's what I've been focused on in talking to people.

YOSHUA: That's what I focused my research on.

CRAIG: Yes. So that's what I wanted to talk about. So, on the drawbacks there's of course the hallucination problem that OpenAI is working on reinforcement learning with human feedback.

CRAIG: I had a very interesting conversation with Yann LeCun about his view that you have to build a world model that,

YOSHUA: I agree.

CRAIG: Yeah, because these large language models, they're, grounding in reality is what's contained in text. And there's, there's,

YOSHUA: that's not the only reason why we need a world model. There are other reasons. one reason I focus on is because humans separate knowledge of the world. From how to take decisions.

And in fact, scientists do that as well and engineers do that. just to understand that separation, consider a go playing system like AlphaGo.

The knowledge that's needed here really is the rules of the game, like how you make points and so on.

YOSHUA: And, and the fact that you want to have as many points, I mean, you want to win and person who has more points,

YOSHUA: Once you have that, you don't need to interact with the rest of the world. So, like the world model here would be easy to get because it's very small. But the machinery to take that and turn that into good decisions, which we call inference in machine learning. That's very complex.

YOSHUA: In fact, doing exact. Inference here, like finding the exact right move. The best move is intractable. It's exponentially hard. And that's why we need really, really large neural nets to do the inference. So AlphaGo is just doing inference. I mean, it's learning to do inference by checking itself with the world model.

YOSHUA: It's checking itself because it's playing against itself and obeying the rules in a sense. Right?

And right now, if you look at something like large language models, there is no separation between the knowledge we are extracting from the world, like the rules of go and how we should answer questions.

YOSHUA: It's the same system that somehow implicitly has the knowledge and is doing the actions like, you know, answering questions. And let me give you another example where I'm going to illustrate why it could be useful to, to do that to have that separation.

I had only one car accident in my life, and I never drove down a cliff or something.

YOSHUA: Probably would've died. Instead, I had lots of thoughts imagining what would happen if I, you know, drive over a cliff or what would happen if I suddenly brake on the highway and the person behind me hits me, but I don't need to try it because I have a world model where I can like simulate these kinds of things at a very abstract level.

YOSHUA: And so, there are lots of good reasons why you want that separation. Like the policy of how I drive. And my knowledge of how things like causal knowledge of, you know, what, what's, what happens if I do this? What's going to be the cause-and-effect chain that can happen? And science is all like this.

People, like scientists come up with theories, they're like, like world models.

YOSHUA: Mm-hmm. And. Once they kind of, you know, want to test a theory or, or believe a theory, then they, you do engineering. Okay, so now let's solve for the bridge. That's not going to fall, which is not the same thing as like the laws of physics and mechanics that are involved in, in the world, not model, right? So, there's a difference between the two.

YOSHUA: but right now, in our large neural nets, trained end to end, which I contributed to in many ways. We are not doing that. We, we have a single big neural net that embodies everything somehow.

And that's detrimental in, in, in many ways. you're, it, I think it's one of the main causes of overfitting for example, because there is no place for like a real-world model.

YOSHUA: It's, there's, there's no explicit place for reasoning, reasoning, causally, or reasoning, you know, more generally. Of course, if you look at ChatGPT, it seems to kind of reason, but if you, if it has to reason over too many steps, it, it tends to get it wrong.

It's not just hallucinating, it's like getting it wrong.

YOSHUA: I, I did a little experiment. I asked him to do, additions or multiplication of, of numbers and, you know, if it's just like one-digit numbers, it works very well. If you have three-digit numbers, it's, it, it can't do it usually. I mean, it, it claims, it does it, and, and it, it, it, it gives you wrong answers or, or wrong explanations.

YOSHUA: which is interesting because to do this, you, you have many steps and you can tell it, oh, do it step by step. Show me how you do it, right? Like people have been doing, and it still gets it wrong. So, it, it's very weak in terms of reasoning. So, what, what reasoning is like reason really, like you separate.

YOSHUA: The knowledge from how you use it. That reasoning is using the knowledge, combining pieces of knowledge in sort of a rational, coherent way in order to go from A to B. That's reasoning like planning. Planning is a special case of reasoning.

CRAIG: At a very high level, how do you build a world model? I'm sure you know Yann’s joint embedding predictive architecture.

YOSHUA: So, we've come up with a new framework for training inference machines. and they, and it can also be used to. To help train the world model. I don’t know what kind of people you have in your audience.

YOSHUA: I don’t know how much technical I can go here.

CRAIG: Go technical because people, their eyes glaze over. It's too, if it's too technical, but I'd rather have it in for those that that follow it.

YOSHUA: So, so we've come up in the last couple of years with something called generative flow networks or G flow nets. And what they do is they learn to do probabilistic inference.

YOSHUA: So, there are neural nets, which could be very big potentially, that learn to do reasoning, essentially through a sequence of steps. And after that sequence of steps, they get a kind of reward. So, they're connected to reinforcement learning.

If the steps are coherent with the world knowledge, so the world knowledge is given by some other neural net or some other piece of code if you want, that says whether the result makes sense.

YOSHUA: So, you have two things to train. You got the world model that embodies knowledge and can check whether things are coherent. let me give you an example from, say, mathematics, that people can understand the, you know, you know, more or less what a theorem and a proof is about. Yep. Right? Yep.

So, the world model here is just checking that the proof is correct in the sense that each step is logical and coherent, and it uses things, facts, that are known. Like, you know, if you add X plus Y is Z equals to Y plus X.

So, so you're allowed to do that transformation and the inference machine is proposing solutions to a problem like, prove this theory and it needs to.

YOSHUA: Usually through a sequence of steps where it uses knowledge and if it messes up, like it makes up, it hallucinates things along the way like ChatGPT does, the reward is going to be very bad. Mm-hmm. And so, it learns to be coherent.

YOSHUA: And that's not something that explicitly ingrained in current large language models. They, they, they get it because they've seen so much text. And humans tend to be coherent, but not always. And you know, there are lots of subtleties here.

For example, who's speaking, maybe I'm speaking not the truth, because I'm trying to convince you.

YOSHUA: And so, there are lots of subtleties in language that means that it's not always like the truth that you're seeing right? And so, we need machines that understand the notion of truth in a more fundamental way, I think.

By the way, this is something that we are doing with neural nets, but the idea behind reasoning and logic as a building block of AI are from the early days of AI, like, it's like classical AI, symbolic AI.

YOSHUA: You know, that's what Gary Marcus has been saying. I I think that the way that these guys have been proposing it is not going to work. But what can work is to train a neural net to behave in a coherent. T

o behave rationally as in manipulate pieces of truth together to come up with proposals for things that make sense.

CRAIG: I understand there's the, the inference model, the world model, and then the language model and the language model.

YOSHUA: Yes, exactly. The language model is yet another thing in the brain. The language part is quite distinct from That's right. The reasoning part and the math part and, and, and you know, the knowledge about action and so

CRAIG: on. Right. And that's kind of the problem right now, the language model, it produces co coherent language. It doesn't really know whether something is true or not. Yes. the, and in, in your structure, the, the world model would provide that ground truth. and, and the inference model would.

CRAIG: Yes, execute reasoning and then the language model would express,

YOSHUA: it would, it would do more than execute reasoning, because reasoning you have to understand reasoning is very hard because there are many paths you could follow. Think about proving a theorem. Is it easy? No. Because how, you know, which pieces of knowledge do I combine in what order?

YOSHUA: There's an exponential number of possibilities. That's why you need joint of models. So, you need actually models that are very similar to what we use for large language models.

But instead of generating words and imitating what humans have said, they generate sequences of like pieces of, I mean, they, they pick pieces of knowledge or they, they pick pieces of, solutions, and then they get rewarded if, if they do it right.

YOSHUA: Right. and by the way, the inference machine is also important to train the model itself. So, if, if there are things that are not given an input, so let, let me give you an example. You see an image and you're trying to come up with an explanation for what's in the image.

And maybe the image doesn't come with labels.

YOSHUA: So, you're trying to do improvised learning. Like humans look at images and they make sense of them. So there, there are well known principles in, in machine learning to deal with that situation where you don't observe the full story.

You have to, they're called latent variables, like the Jeff Hinton calls them hidden variables. The model needs to come up with a, like a story that explains the data.

The inference machine is the thing that comes up with the story, and then the world model checks whether the story is consistent with the image, and that's how the inference machine gets better.

YOSHUA: It's trying to make stories that are consistent with the model, but when it does that, it helps to train the world model as well, because it, now that you have a story and an image, you can see, you can build a model that makes the right links between them.

So, a lot of the work that Jeff and others and, and you know, I, I did in the last few, in the early decades, especially of deep learning with probabilistic models like Boltzmann machines and so on, were based on these ideas, but, but I think that now we can use these large neural nets, like the large language models, but not for language, for reasoning.

CRAIG: On the world model side though, it sounds, when you say hearkening back to the early days of AI, it sounds like a rule-based system where you load it up with

YOSHUA: No, because we are going to learn the rules as well, and we are going to learn them in a probabilistic way.

YOSHUA: So, so here's another aspect that's missing from classical AI. Given any finite amount of data, you can't be a hundred percent sure of what is the right world model that explains the data because there are may be multiple interpretations.

It's like in science you have usually different theories that are compatible with the data.

YOSHUA: And what scientists do is try to come up with different theories, diverse set of theories. That's important because if we just go for the first theory, we find. it might be completely wrong, even though it fits the data. It's, but what if we know that we have two competing theories?

We can be safer. Like maybe sometimes they agree and then we should go that way, and sometimes they disagree, and then we should be careful.

YOSHUA: So, one of the problems with things like ChatGPT right now and, and many of these models, it's not just this one, is that they are often confidently wrong. It's not just that they hallucinate. They're sure of what they're saying. And of course, that's bad, right?

That can lead to catastrophic outcomes. But, and, and I guess there are some humans who were like that too.

YOSHUA: But I guess for humans is maybe a lot is tied to their ego. But when, you know, in a, in good circumstances, people have a good sense that some belief that comes to them isn't completely sure and they're actually good when you ask them to bet.

If they have to bet money on something, somehow, they'll exploit the fact that they're not sure to bet more or less money.

YOSHUA: Okay? So, our brain is doing that calculation. It's not something we verbalize, but we kind of sense it and then we act accordingly, and that is very important. because coming back the connection to the world model is that really what we need to do is not to have one world model, but to have a distribution over world models.

YOSHUA: So, it's not like we have a neural net that is the world model. I think that's the wrong picture. the right picture is we have a neural net that generates the, the theory that corresponds to the world model, just again, the same kind on that I've been talking about.

But now, instead of coming up with explanation for an image, it comes up with a general theory that explains a lot of data.

YOSHUA: And the theory could be something that can be verbalized the same way that scientists come up with theories. So, you see, we are talking about things that are related to classical AI, but in classical AI, not only, there was no uncertainty, but it was all handcrafted, like the theory is a set of rules and facts that somebody puts down.

YOSHUA: And you know, maybe that can be part of a solution of something, but really many of these statements, they're not like a hundred percent sure and so people are starting to play with probability and so on. But it, we never got it quite right.

And the reason we never got it quite right is that getting it right is computationally - it looks computationally intractable, but it turns out it can be approximated by these very large neural nets. They do a good job at that.

So, we can have large neural nets that just us, just like we do, spit out theories or, or pieces of theories that are relevant to what we're currently seeing. rather than have one theory that's, you know, hard coded.

YOSHUA: either by humans or even by a machine. And that's like one list that we don't change and then we trust there on that doesn't make sense. we keep revising our views and our views are uncertain.

So, we, we have a distribution of our theories, and we don't say, oh, there are these 10 theories and this, you know, this one is 0.8 and this one is 0.6.

YOSHUA: That's not how it works. And the reason is there's an exponential number of theories, so we don't explicitly emulate them. Instead, we generate pieces of theories according to their Bayseian probability.

And there's a lot of evidence from cognitive science that people do that. Mm-hmm. It's, it's happening unconsciously because that generative model is something that's system one.

YOSHUA: It's, it's behind the scenes. Right. You don't control it. You just see your thoughts coming out.

CRAIG: I can imagine the inference model. I mean, there, there's a lot of inference models out there Now. How do you train the world model? Is it,

YOSHUA: okay, well that's, that's the cool thing.

YOSHUA: If you have the inference machine.

CRAIG: Oh, sorry. I’ll just finish the question - because there is a certain amount of reasoning in large language models, as you found in your own research. And so, and that's contained in language, but if you want to build a real-world model, you want to get beyond language.

CRAIG: So, do you do it using video? I mean, how do you do that? Right.

YOSHUA: The, the, the world knowledge. I mean, the world model could be about all kinds of things, images, concepts in the world. I, I like to focus on the abstractions of the kind that we can verbalize, but you, you can also, and Yann is sort of more focused on the perception and, and, and action part.

YOSHUA: Mm-hmm. but the principal, I think could be applied to both and the principal is the there is no world model. You only have an inference machine because - So now what I'm going to say is a little bit mind twisting. You only have an inference machine. And what that inference machine does is it, it, it can generate the world model, or the piece that you need at the moment.

YOSHUA: So, and when you have such a piece, you can also evaluate how your regular inference is doing. Like, you know, am I answering the questions, right?

So, if I see an image about, you know, dogs chasing cats, a piece of my theory of about how the world works may come to me about things I know about things I think I know, I think are, are true about cats and dogs.

And that will serve to provide a reward for having decided that I need to run out quickly to stop them because maybe the cat is in danger or maybe the dog's in danger. so, what I'm trying to say is we don't actually need a separate world.

I mean, I'm saying the opposite of what I said earlier, that's why I'm saying it's mind twisting.

YOSHUA: So, so now that we've established, we want a world model and an inference machine, I'm going to say that instead of a world model thinking of like a separate neural net, because there are many possibilities of what the right world model should be, you have actually just another generative neural net that generates just the pieces of world model that you need on the fly.

YOSHUA: It's probabilistic. So maybe one day you view things one way, and the other day you view things a different way because really there's uncertainty. and, and that serves as a sort of reward to drive your normal inference. Like, what should I do? how should I plan? And so, on

CRAIG: these snippets of, of, of reality, of the world model that are being generated, how do you ensure that they're, they reflect reality?

YOSHUA: Because you can, once you generate a piece of theory, you can also like confront it to the observations. So, what a theory gives you is a way to quantify, how well it matches any piece of data.

So, like in science, so if I have a theory that says force equals mass times acceleration, and I observe force mass and acceleration, I can just compute how well it matches.

YOSHUA: So, a theory gives us a checkable quantity about how well it fits the data. A theory is like a likelihood function, right? So, it, it, it, it's something or an energy function, in terms of what Yann likes, it's a quantity that we can measure.

The output of a theory is a measure of how any particular data is consistent with the theory.

YOSHUA: So that's what a theory does. Its output is how well it fits with the data.

CRAIG: is this active research and if so, where are you in it?

YOSHUA: yeah, it's very active. It's a good part of what I do in my group. One of the first papers that goes in, in that direction came out last summer.

YOSHUA: It's called Bayesian Structure Learning with Generative Flow Networks. And in, in this paper, the theory is a causal theory. So, what the, the Probabilistic inference machine does is it generates a graph that says this variable is the cause of this one.

This variable is the cause of that one and so on.

YOSHUA: and that's in. theory of the observed data that we are seeing. And once you have that, you have a way to numerically then evaluate how it's consistent with whatever data you have at hand. And that becomes a reward for the inference machine that generates the theory.

And next time it's going to generate one that fits better.

YOSHUA: But it's not like in usual reinforcement learning where you're trying to find the theory that best fits the data, that would be back to maximum likelihood training. What, what the GFlowNet is different from reinforcement learning.

Standard reinforcement learning is that instead of looking for generating the actions that give you the best reward, it's sampling actions or, you know, like theories with some probability.

YOSHUA: And it's trying to do it such that this probability matches the reward so that it's proportional to the reward. So then if you have, first of all, if you have two theories that are about equally well fitting the data, you have a 50 50 chance of, of sampling them.

If you have a million theories that are equally fitting the data you, you can sample all of them with standard reinforcement running.

YOSHUA: It will pick one theory and be happy. Right. So, if you want to be Bayesian, which is like mathematically the right thing to do if you could, but it's hard. You want to have all the theory somehow.

And if a theory fits the data twice as good, maybe you want to sample it twice more often. And that's, that's the Beijing way of thinking about how you should be rational in your decision making.

YOSHUA: Then you should consider all the theories and average their decisions to decide what to do next.

CRAIG: Okay. Great. Thanks. Bye-bye I really appreciate you doing this.

YOSHUA: Hello to Beauregard!

CRAIG: He'll be delighted I've got him studying machine learning online. That's it for this episode. I want to thank our sponsor, NetSuite by Oracle. If you want to give them a try for a limited time, you can get.

CRAIG: Full implementation with no payment and no interest for six months. Just go to www.netsuite.com/eyeonai. Be sure to type the EYEONAI so they know the sponsorship is working. If you want a transcript of this show, you can find one on our website, eye-on.ai.

CRAIG: And remember, the Singularity may not be near, but AI is changing your world, so pay attention.