my fondness memory oh it's usually when I discover something that I think nobody has seen before but that is that happens very rarely because most of the things you think of somebody else has done before this episode is sponsored by numerai are you a data scientist looking to make a real world impact with your skills do you love competing against the best Minds in the world well introducing numeri the Revolutionary Cutting Edge AI driven hedge fund that's changing the game for good Numerica mines a competitive data science tournament with powerful clean stock market data enabling you to predict the market like never before sign up now become part of the elite Community taking the stock market by storm and I'll see you on the leaderboard wonderful so um today is a momentous occasion what an episode of mlst we're going to have we're joined not by a Godfather of AI but the father of AI you again schmidtuber the researcher responsible for leading the research groups which invented much of the technology which has powered the Deep learning Revolution it's long been a dream to get you on the podcast you again um it feels like the day has finally arrived so welcome to mlst thank you Tim for these very kind words and it's very generous introduction so on on that let's let's discuss the credit assignment problem in machine learning now you've dedicated a significant amount of time uh researching and Publishing the actual history of the field and there's a significant Divergence between the public narrative and what actually happened and amazingly no one has pointed out any factual inaccuracies in your accounts but the incorrect perceptions still persevere now I particularly enjoyed reading your history of the breakthroughs in machine learning going back to ancient times and of course even remarking on the very first computer scientist leibenitz and for example you pointed out the history of who invented backprop and and the CNN and you explained it there wasn't really a neural network winter at all in the 1970s so could you just sketch out a little bit of that history so that's a challenge um actually um computer science history and Computing history started maybe 2000 years ago when um when here on of Alexandria built the first program controlled machine that was 2000 years ago and the first century basically and um and he basically built an automaton that was programmed through a cable which was wrapped around a rotating cylinder which had certain knobs and then um there was a weight which pulled it down and the whole apparatus was able to direct the movements of little robots of little puppets in an automatic theater that as far as I know was the first program controlled machine in the history of mankind even before that there were other machines the ancient Greeks had even earlier the antiquitera mechanism which was kind of a clerk and astronomical clerk but then more recently um we have seen many additional advances and you mentioned leibniz of course who is of special interest to our field because he not only is called the first computer scientist because he had the first machine with a memory and that was in the 1600s 80s I think he not only had the the first machine that could do all the basic arithmetic operations which are additional multiplication division and um subtraction then he not only had his first ideas for a universal Problem Solver that would solve all kinds of questions even philosophical questions just through computation and he Not only was the forest who had this algebra of thought which is deductively equivalent to the to the much later Boolean algebra so in many ways he was a Pioneer but especially in our field and deep learning he contributed something essential which is really Central for this field which is the chain rule I think 1676 that's when he published that and that's what is now being trained to uh what is now being used to train very deep artificial neural networks and also shallow neural networks and recurrent neural networks and everything that we are using in modern AI is really um in many ways depending on that early work but then of course there was so much additional work um the first neural networks as we know them they came out um they came up about around 1800s that's when Gauss and legendre had the linear neural networks the linear perceptrons in the sense that they were linear without being um without having any non-differential aspect to it so these first neural networks back then were called a method of least squares and the the Training Method was regression and the error function was exactly the same that we used today and it was basically just a network with a set of inputs and a set of outputs and a linear mapping from the inputs to the outputs and you could learn to adjust the weights of these connections so that was the first linear neural network and and many additional later developments led to what we have today you you had this you had this beautiful uh statement you said that machine learning is the science of credit assignment and we should apply that same that same science to the field itself and I guess uh what I'm really curious about is first if you could educate our listeners just a bit on what credit assignment is in the context of say machine learning and why you think it's important that that should apply to the field in general you know why should we care about credit assignment why should we study the history of of um you know the developments in the field why is it important I am interested in credit assignment not only in machine learning but also in the history of machine learning because machine learning itself is the science of credit assignment what does that mean suppose you have a complicated machine which is um influencing the world in a way that leads to the solution of a problem and maybe the machine solves the problem but then the big question is which are the components of these many components were responsible some of them were active a long time ago and others later and early actions set the stage or later actions now if you want to improve the performance of the machine you should figure out how um the the components contribute to the overall success and this is um what creditor Simon is about and in machine learning in general we have a system consisting of a lot um of many um machine learning engineers and mathematicians and um and Hardware Builders and all kinds of people and there you also would like to figure out which parts of the system are responsible for later successes yeah and it's it's a brilliant point and and I completely agree with you by the way and I think the way I think about it is you've got this giant architecture of humanity and in it are these certain nodes it may be an individual maybe a research group and if they come up with things that are very helpful right you want to try and direct more attention more resources you know at that that nodule at that node right because it's it's likely to come up with additional you know very very important things and if we don't get that right we're just not optimizing the the algorithm of science as a whole that's right yes machine learning and Science in general is um based on this principle of credit assignment where credit usually it doesn't come in form of money sometimes also in form of money but in a form of reputation and then the whole system is set up such that you create an incentive for people who have worked on improving some method to um to credit those who maybe came up with the original method and to just have these chain uh these chains of credit assignment that make clear who did what when because the whole system is based on this incentive and yes those who are then credited with certain valuable contributions they also can get reasonable jobs and and the within the economy and so on but that's more like the secondary consequence of the basic principle and that's why our PhD advisors teach their PhD students to be meticulous when it comes to credit assignment to past work so one last question if I may I've really enjoyed studying the history of of advancement because I found that when I go back and read original Source materials you know let's say Einstein's first paper on diffusion okay or anything like that you know because they're breaking new ground they're kind of considering like a wider array of possibilities and then over time you know the field becomes more and more focused kind of on a narrower Avenue of that and you can go back and look at the original work and actually gain a lot of inspiration for alternative approaches or alternative con you know considerations so in a sense it's it's kind of in the in the sense of forgetting is as important as learning you know sometimes we need to go back to go down a different branch of the tree if you will and expand the the breadth of the search a little bit I'm curious if you've noticed that same phenomenon yes science in general is about failure and um 99 of all scientific activity is about creating failures but then you learn from these um failures and you do backtracking and you will go back to a previous decision point where you maybe um made the wrong decision and pursued a wrong Avenue but now you have a branching point and you pursue an alternative and um and in a field that is rapidly moving forward you don't go back very far usually you just go back to a recent paper which came out five months ago and maybe you have a little Improvement there and then maybe there's yet another little Improvement there and some parts of our field are at the moment a little bit like that where PhD students are moving in who just look at the most recent papers and then find a way of improving it a little bit and you know two percent better results on this particular Benchmark and then the same guys are also reviewing at Major conferences papers by similar students and so on and so then sometimes what happens is that um no very deep backtracking is happening just because of the actors aren't really aware of the entire search tree that has already been explored in the past on the other hand science has this way of healing itself and um since you can gain reputation by identifying maybe more relevant points branching points uh new um you have this incentive within the whole system to um to improve things as much as you can sometimes by going back much further yeah so um there's been a lot of discussion in the discourse around this concept of AI existential risk and you again you've published quite a few pieces about this recently um prominently in in the guardian and in Forbes actually and one of the things I wanted to focus on is this concept of recursive self-improvement because that seems to be one of the the plausible explanations that these folks give and of course when it comes to recursive self-improvement you are an expert in this field I mean go go Dell machines come to mind immediately so I want to um kind of explore asymptotes and limitations this whole idea of because the self-improvement is very sexy isn't it in fact it is the the one idea that motivated um me to to do all of this and so my first paper ever 1987 that was my diploma thesis and it was about this because the self-improvement thing so it was about a machine that learns something in a domain but not only that it also learns on top of that to um learn a Better Learning algorithm based on experience and the lower level domains and then also recursively learns to um improve the way it improves the way it learns and then also recursively learns to improve the way it improves the way it improves the way it lands and yeah I call that meta learning and back then I had this hierarchy with in principle infinite itself Improvement or in the in the recursive way uh otherwise I was limited by the limited time that you run the system like that and uh and then um of course the motivation behind that is that um you don't want to have an artificial system that is stuck uh always with the same old human designed learning algorithm no you want something that improves that learning algorithm um without any limitations except for the limitations of physics and computability and so much of what I have been doing since then is really about that self-improvement and different settings where you have on the one hand reinforcement learning systems that learn in an environment to better interact and better create ways of learning from these interactions to learn faster and too long to improve the way of learning faster and so on and then also a gradient base systems artificial neural networks that land through gradient descent which is a pre-wired a human designed learning algorithm to come up with a better learning algorithm that works better in a given set of environments than the original human designed one and um yeah that started around 1992 neural networks had learned to run their own learning algorithms on the recurrent Network themselves so you have a network which has standard connections and input units and output units but then you have these special output units which are used to address connections within the system within this Recon Network and they can read and write them and suddenly because it's a recurrent Network and therefore it is a general purpose computer suddenly you can run arbitrary algorithms on this recurrent Network including arbitrary learning algorithms that translate incoming signals not only the input signals but also the evaluation signals like reinforcement signals or error signals into weight changes fast weight changes where the weight changes are not dictated any longer so this gradient descent method but no now the network itself is learning to do that but the initial weight Matrix is still learn through gradient descent which is propagating through all these self-referential Dynamics in a way that improves as a learning algorithm running on the network itself that was 1992 and back then compute was really really slow was a million times more expensive than today and you couldn't do much with it but now in recent works all of that is working out really nicely and has become popular and we have just if you look at the past few years a whole series of papers just on that so that's that's the fast weight programming that you're referring to uh just yes um yeah yeah so it's fast weight programmers that where you have a part of the network that learns to quickly reprogram another part of the network or the original version of that was actually two networks so one is a slow Network and then there's another one a fast Network and this loan Network learns to um generate weight changes for the second Network and the program of the second Network are its weights so the weight Matrix of the second Network that is the program of the second Network and the first one what does it do it generates outputs it allowance to generate outputs that um cause weight changes in the second Network and these rate changes are being applied to patterns to input patterns to queries for example and then okay the first Network essentially learns to program the second Network and and essentially the first network has a linear algorithm for the second Network and the first system of that kind 1991 that was really um based on on keys and values so the first Network learns to program the second Network by giving it keys and values and it says now take SEC network take this key and this value and Associate both of them through an outer product which just means that those connections those units are strongly active they get connected through stronger connections and um the mathematical way of um describing that is the outer product between key and value so that's how the first Network would program the second Network and the important thing was that the first Network had to invent good keys and good values depending on the context um of the input stream coming in so it used the context to generate what is today called an attention mapping which is then being applied to queries and this is this was a first step before right before the most General Next Step which is then really about learning a learning algorithm running on the network itself for the weights of the network itself could I press you a tiny bit on this concept of um meta learning and convergence and asymptotes now one of the reasons um I think why the x-risk people believe that it will just go on forever is they believe in this idea of a pure intelligence one that doesn't have physical limitations in in the real world and I'm quite amenable to this ecological idea of intelligence that it does you know the world is a computer basically as well as the actual brain that we're building um so surely it must hit some kind of asymptote do you have any intuition on what those limitations would be so you are talking about the ongoing acceleration of um computers computing power and and limitations thereof is that what you have in mind here well that that's one part of it so even if you just scale Transformers I think there would be some kind of asymptote but we're talking here about meta learning learning to learn how to learn and recursive self-improvement and it's similar to this idea of reflection self-reflection and language models it actually improves the performance with successive steps of reflection and then it levels off it it reaches an asymptote I just believe that there are asymptotes everywhere and that's the reason why I don't think recursive self-improvement will go on forever but I just wondered if you had any intuitions on what those impressions are you are totally right there are certain uh algorithms that we have discovered and in past decades which are already optimal in a way um such that you cannot really improve them any further and no self-improvement and no fancy machine will ever be able to further improve them there are certainly certain sorting algorithms that under given limitations are optimal and you can further improve them well that's that's one of the limits then of course there are the fundamental limitations of what's computable first identified by a good girl in 1931 he just showed that there are certain things that no computational process can ever achieve no computational theorem proven can prove or disprove certain theorems in a language and a computer in a in a symbolic language that is powerful enough to encode certain simple principles simple simple principles of arithmetics and stuff like that and so what he showed was that the fundamental limitations to RF computation and therefore there are there are fundamental limitations to any AI based on computation I'm glad you brought that topic up because it's one of our one of our favorite things to discuss which is do you think the human mind ultimately reduces to just an effective computation and so subject to those those same limits or do you think there's any you know uh known or unknown physics that that give us some out in which uh the brain can do a computation that amounts to hyper computation yeah since we have no evidence that the brain can compute something that is not computable in the traditional sense and Google sends and tourings and churches sends and everybody who has worked on this field um since we have no evidence we shouldn't assume that's the case as soon as someone shows that people can compute certain things or approve certain theorems that machines cannot prove given the same um initial conditions we should looked more closely but you know there are many things that might be possible in fairy tales and we are not really exploring them because the probability of you know coming up with interesting results is so low fair enough so I interrupted there so you mentioned so far two asymptotes one being of the mathematical kind where there's just mathematical proofs that certain things are optimal the other one being the limits of computation itself what other asymptotes do you see applying to or putting bounds on recursive self-improvement oh the most obvious thing is probably light speed and the the um the limits locality physical computation yes and we know those now for several decades we have happily enjoyed uh the fact that every five years computers getting 10 times cheaper and this process started long before um Moore's law was um defined in the 60s I believe because even in 1941 already when the when twoza built the first program controlled computer in this law apparently was active so back then he could could compute maybe one instruction per second and since then you know every 10 years a factor of 100 every 30 years a fact of a million more or less until today and and there's no no reason to believe it won't hold for a couple of additional decades because the the physical limits are much further out the physical limits that we know are the premium Unlimited discovered I think in 1983 and they basically say that one kilogram of matter cannot compute more than 10 to the 51 instructions per second so that's a lot of compute but it's limited um to to give you an um idea of how much compute that is I also have a kilogram of computer in here and and probably it cannot compute 10 to the 20 instructions per second otherwise my head would explode because of the heat problem but maybe it can compute something that is not so far from 10 to the 20 instructions maybe 10 to the 17 something like that although most of my neurons are not active as we speak um because again otherwise my head would just evaporate now so if you have an upper limit of 10 to the 20 instructions per brain then the upper limit of all of humankind would be 10 billion times that that individual limit and that will be 10 to the 30 instructions per second and you see um it's still far away from the 10 to the 51 instructions per second that in principle one kilogram of matter could compute and now we have we have more than 10 to the 30 kilograms of matter in the solar system and there's some so if if um if the current Trend continuous at some point much of that is going to be used for computation but then it will have to slow down even if the exponential acceleration will still be with us for a couple of decades because at some point it is going to be a polynomial law because uh due to the limits of light speed at some point it will be harder and harder to acquire additional Mass once you have reached the limits of physical computation per kilogram the only way to expand is to go outwards on you know find additional stars and additional matter further away from the solar system and there um you will get um a polynomial um acceleration or a polynomial growth at best so it will be much worse and the current exponential growth that we are still enjoying sure but I would say you know the existential threat um you know that that's more than sufficient to supply an existential threat and let me just let me just put this a little bit differently which is um and I agree with you on this which is you're quoted as saying that traditional humans won't play a significant role in spreading intelligence Across the Universe and and I think you're right I think we kind of share a vision of something like the Von Neumann probes that go out into space and you know form this you know star-spanning civilization of machines and artificial intelligence that have transcended you know biological limitations so I guess my question to you is once that space fairing star spanning you know civilization exists if it becomes misaligned with us and decides that we're in the way right isn't that an existential threat I mean might they just you know repurpose the Earth regardless of whether we're here or not for for their own aims yeah I'm often getting these questions and um and there is no proof that uh we will be safe forever or something like that uh on the other hand it's also very clear as far as I can judge that all of this cannot be stopped and it can be channeled in a very natural and um I think good way in a way that is good for humankind now um first of all at the moment we have a tremendous bias towards good AI meaning AI that is good for humans why because there's this intense commercial pressure to create stuff that humans want to buy and they like to buy only stuff they think is good for them which means that all the companies that are in that are trying to devise AI products they are maximally incentivized to generate AI products that are good for those guys who are buying them or at least um where the customers think it's good for them so um it's a circle 95 so maybe five percent of all AI research is really about AI weapons and one has to be worried about that when all this has to be worried about weapons research but there's a tremendous bias towards good AI so that is one of the reasons why you can be a little bit optimistic for the future I'm always trying to point out the two types of AIS there are those who are um just tools of users human human users and the others that invent their own goals and they pursue their own goals and both of them we have had for a long time now for the AI tools it's kind of clear there's a human and a human wants to achieve something and so it uses he uses or she uses that tool to achieve certain ends and and most of those are of the type let's improve Healthcare and let's facilitate translation from one person to another one in another nation and just make life easier and make human lives longer and healthier okay so that that's the AI tools but then there are the other um uh AIS which also have existed in my lab for at least 13 two years which invent their own goals and um they are a little bit like little scientists um where you have um an incentive to explore the environment through actions through experiments self-invented experiments that tell you more about how the world works such that you can become a better and better and more and more General Problem Solver in that world and so these um AIS they have for a long time created their own goals and now of course the interesting question is um these more interesting AIS what are they going to do once they are once they have been scaled up and can compete or maybe outperform humans and everything they want to achieve on the one hand the AI tools and there the greatest warriors what are the other humans going to do to me with their AI tools so in the extreme case you have people who are using AI weapons against you and maybe your neighbor is um has bought a little drone for uh 300 dollars and it has face recognition and it has a little gripper and it flies across the Hedge and puts some poison into your coffee or something like that so then the the problem is not the the AI which is trying to enslave humans or something silly like that no it's your neighbor or the other human and generally speaking you have to be much more afraid of um other humans then you have to be of AIS even those who Define or set themselves their own goals because um you must mostly worry about those with whom you Shack owns so if you share goals then suddenly there is a potential of conflict because maybe there's only one schnitzel over there and two persons want to eat the Schnitzel and suddenly they have a reason to fight against each other generally speaking if you share goals then you can do two things you can either collaborate or compete an extreme form of collaboration would be to um maybe marry another person and set up a family and master life together and an extreme form of competition would be War and and those who share goals they have many more incentives to interact than those who don't share goals and so uh humans are mostly interested in other humans because they share similar goals and because they give them reason to collaborate or to compete more CEOs at certain companies are interested in other CEOs of completing companies and five-year-old girls are mostly interested in other five-year-old girls and the super smart AIS of the future who set themselves their own goals they will be mostly interested in other super smart AIS of the future who set themselves their own goals there is not so much competition uh and there are not so many sharp goals between biological beings such as humans and a new type of life as you mentioned can expand into uh the universe and can multiply in a way that is completely um infeasible for biological beings so there's a certain um long-term protection at least so lack of interest on the other side okay brilliant there's a few things I wanted to touch on there we will get on to what it means for goals to emerge from systems later and you started off by saying that um humans will buy products that make them feel good and Facebook is quite an interesting example to play with actually because Facebook is a little bit like an AI system which is a collective intelligence and humans use Facebook but they have some idea that it might cause them harm and the thing with population ethics is we we know that our moral reasoning kind of decays over space and even more so over time and part of the reason why time is so difficult is because it's predictive we don't actually know what's going to happen in the future so our kind of reasoning about establishing what the value of something is is very very faulty and I think that's one of the reasons why these people would say that we don't really know what's good for us I do completely agree with you though that the problem I think is humans rather than ai's on their own yes these are good points feel free to uh offer some thoughts yes I mean it would that's a a whole separate discussion isn't it uh when you discuss the limitations of what's predictable and um and how people often fail to see what's good for them well I think maybe so you've already you've already said that there's no proof that we'll be safe forever right like I mean there could there could come an existential risk you know from AI so I think our question to you is do you have sympathy for the folks who say we need to be putting more resources into researching alignment like we need to develop the tools in order to allow it to be easier for people to construct AI that is aligned for the goals and to make sure that you know that it doesn't that it doesn't have uh unintended unintended consequences like in other words there may not be a proof that we can go forever and be safe for AI but we at least want to develop the basic mechanics that we need to safely develop and deploy AI don't we yes and I sympathize with those who are devoting their lives to alignment issues and trying to build AIS aligned with humans I've used them as part of the evolution of all kinds of other ideas that come up as not only nations compete with other nations but companies compete with other companies and shareholders of different companies compete with shareholders of different companies and so on and um and so there is such a huge set of different human goals which are not aligned with each other that makes me doubt that you will come up with a general system that all humans can accept simply because if you put 10 humans in a room and ask them what is good they they will give you 10 different opinions however I sympathize with some with this goal and um and and it's good that people are worried and they spend resources on solving some of these issues in the long run however I think there is no way of stopping all kinds of AIS from having all kinds of goals that have very little to do with humans the the universe itself is built in a certain way that apparently derives it from very simple initial conditions to more and more complexity and now we have reached a certain stage after 13.8 billion years of evolution and it seems clear that this cannot be the end of it because the universe is still young it's going to be much older than it is now now there is this drive built and drive um RC Cosmos to become more complex and it seems clear that Civilization a civilization like ours is um is a stepping stone on towards something that is more complex and um could I touch on a couple of things here the bootloader example is kind of where I want to go with this so um a lot of the ideas of of this movement can be traced back to um Derek parfitt who is a philosopher he was a moral realist so he thought there was such a thing as a moral fact and I'm a bit of a relativist myself and actually if you trace this tree of complexity and how humans evolve over time we might just be a stepping stone to a kind of Rich diverse transhumanist future where we become the thing over time that we're so scared of and I think the lens that we're using here about what's right and what's wrong is kind of like I was saying before it's a snapshot of humanity now and we kind of think of it as just this monolithic single thing so does it really work when you project out to how we're going to evolve in the future well first of all humankind is not a monolithic thing so many Aussies arguments go like we should not do that because of that we should not do that because of that but there is no us there's no we there are only um almost 10 billion different people and they all have different ideas about what's good for them and so um for thousands of years we had these evolutions of ideas and of devices and philosophies competing partially competing and partially compatible with each other which in the end led to the current values that some people agree with and other people over there they agree with different values nevertheless there are certain values that have become more popular than others more successful more evolutionary with more success during the evolution of ideas and so um given this entire context of evolution of Concepts and um accepted ideas of what should be done or what is worth being supported and what's not worth being supported all of this has changed a lot if we look back 200 years the average people in the west had different ideas of what's good then today and and this evolution of ideas is not going to stop any time soon just a final question on this um there is a very real existential risk right now of nuclear Armageddon a real risk right now and if I were a rational person I would be devoting all of my effort into that and other risks Associated so do you think it's a little bit weird that so much focuses on this AIX risk to me it's indeed we had now there are all these letters coming out warning us the dangers of AI and I think some of the guys who are writing these letters they are just seeking attention because they know that um AI dystopia are attracting more attention than documentaries about the benefits of AI and Healthcare and stuff like that but generally speaking I am much more worried about nuclear bombs than about AI weapons [Music] a nuclear bomb a big one can wipe out 10 million people a big city within a few milliseconds without a face recognition just like that without any AI and so in that sense it's much more harmful than the comparatively harmless AI weapons and that we have today and that we can currently conceive of so yes I'm much more worried about a 60 year old technology that can wipe out civilization within two hours without any AI well I guess um since we're we're not really going to worry about AI for the moment we can uh we can turn our attention back to discussing with you uh how we develop AI so um you know I'm really curious with with just the really the the vast you know breadth and depth of your of your knowledge over the the history of AI and the state of the art I'm curious you know which current approaches you're you're most excited about and or what's on the horizon um that you know for any of our listeners out there thinking about um going into AI research machine learning research you know what may be Alternatives that aren't getting enough attention should they should they look into studying and and perhaps choosing to research at the moment the Limelight is on um language models large language models which pass the tooling tests and do all kinds of things that seemed inconceivable just a couple of years ago at least to some of those who are now surprised but of course that is just a tiny part of what's going to be important to develop true um Ai and AGI artificial general intelligence on the other hand the roots of what we need to develop through AI also come from the previous Millennium they are not new and of course what you need is an environment to interact with and you need an agent that can manipulate the environment and you need a way of learning to improve the rewards that you get from this environment as you are interacting it with it within a single lifetime so one of the important aspects of reinforcement learning what we are not talking about is that you have only one single life you don't have repeatable episodes like in most of traditional enforcement language no you have only one single life and in the beginning you know nothing and then after 30 of your life is over you know something about life and all you know is the data that you collected during these first 30 um percent of your life and now there is an infinite almost infinite possibility set of possibilities uh Futures and from this little well short experience you have to generalize somehow and try to select action sequences that lead to the most promising Futures that you can shape yourself through your actions now to achieve all of that you need to build a model of the world a predictive model of the world which means that you have to be able to learn over time and to predict the consequences of your actions such that you can use this model of the world that you are acquiring there to plan to plan ahead and you want to do that in a way that isn't it's a naive way which we had in 1990 which is millisecond by millisecond planning where you say okay now I'm moving from A to B and the way to do it is first move that little pinky muscle a little bit and move it a little bit more and move it a little bit more and then get up and so no you want to do that in a high level way in a in a hierarchical way in a way that allows you to to focus on the important apps are concepts for example as you are trying to go from from your home to Bay Beijing you decompose this whole future into a couple of sub calls you say a first important step is to go to the cap station and get a taxi to the airport and then in the airport you will find your your plane and then for nine hours nothing is going to happen and you exit in Beijing and have to find another cab and so on so you you don't do millisecond by millisecond detailed planning no you have a high level planning to just reduce the computational effort and focus on the essentials of what you want to do so that is something that most current systems don't do but for a long time we have had systems like that and they are getting more sophisticated over time important you have a predictive model of the world that is not just focusing on the pixels and predicting the how does the video change as I'm moving my hand back and forth the video that I get through my cameras my eyes and so on and no higher level Concepts that that reflect islands of predictability many things are not predictable but certain abstract representations of these things are predictable and so how can you discover these higher level Concepts that you need to efficiently think about your own future options and select those that are most promising in the single life yeah yeah yeah this is really interesting so we've been speaking with Carl friston for example and he talks about this collective intelligence where you have this multi-agent cybernetic framework which is causally closed and one of the things we're talking about here really is not the model itself people talk about chat GPT and it's just a model and people have configured it in Arrangements that have varying degrees of autonomy and in the future we will develop these Collective intelligences and they're not just predicting the actions and behaviors of other agents but even the world that we're in is a computer to some extent so when you imbue agents with this kind of creativity and autonomy that's the thing that I don't think people really understand what might emerge from that it's related to this discussion about what kind of goals might emerge from that do you have any intuition on on what that would look like yeah let me give you just the simplest example um that we had in 1990 as well 32 years ago of a system that sets itself it's ongoings and it consists of two artificial neural networks and I know that contrasting is very interested in that and only recently uh for the first time in my life I um I was on a paper uh where he was co-author just a year ago and uh and so back then it was really about a reinforcement learning agent and it interacts with the world and it generates actions that change the world and then there is another Network which just is trying to predict the consequences of the actions in the environment so the reactions of the environment to these actions and so that becomes a world model and then the what kind of goal was there which was different from traditional goals well in the beginning this model of the world this prediction machine which is a model of the world a wild model um knows nothing so it has high error as it is trying to predict the next thing as it is trying to predict the reactants are seen environment to the actions of the agent so um as this second network is trying to reduce its prediction error through gradient descent through back propagation essentially the other one is trying to generate actions outputs that maximize the same error so basically the goal the cells invent a goal if you will of the first network is to generate an action with whose consequences cannot yet be predicted by the other network by the model of the world so the first network is generating outputs that surprise the second Network so certainly you have an incentive where the first network is trying to invent actions experiments that fool or that surprise the second Network and that was called artificial in curiosity uh so now suddenly you have a little agent which a little bit like a baby doesn't learn by imitating the parents no it learns by inventing its own little supports and it's trying to surprise itself and have fun by playing with the toys and and um observing new unpredictable things which however become predictable over time and therefore become boring and then it has another incentive to invent the additional experiments such that it still can surprise um its model of the world which in turn is improving and so on as a team does that does that also have the effect of of making the network which is trying to predict does it have the effect of making it more robust and more generalizable like almost a form of you know regularization kind of built in in this this pairing yeah you can build into that Network all kinds of regularizers and orthogonal concept which is also very important um so that was just the first version that was really in 1990 and then we have had a we had a long string of papers just on improvements of this original concept of artificial curiosity so this old system is basically what you um what you now know as Gans generative advisorial networks because the first network is generating a probability distribution over outputs and the second network is then um predicting the consequences of these outputs and the environment and if you if the output is an image then the consequence can be either this image as of a certain type yes or not no and then that's all that's the prediction machine the world model predicts in that simple case and you minimize the first Network minimizes the same error function that the second one maximizes so then you have basically again but then um you don't have what you just mentioned yet the regularizer as a scientist what you really want to learn is a model of the world that extracts the regularities and the environment that um that um that finds predictable things which are regular in the sense that there's a short explanation thereof for example if you have falling objects in a video then they all fall in the same way they accelerate in the same way which means it's predictable what these objects do if you see two of the frames you can predict the third frame pretty well and the law behind that is very simple this means that you can greatly compress the video that is coming in because you can instead of storing all the pixels you can compute many of these pixels by just looking at two successive frames and predicting the third frame or maybe three successive frames and predicting the fourth frame something like that and you only have to encode the deviations from the prediction so everything else you don't have to store separately which means you once you understand gravity you can greatly greatly compress the video so that's what you really want to do and so the more advanced version of artificial curiosity is about that where you have a motivation to find a description of the data which is coming in of the video of the falling apples for example that um is simpler than the one that you had before so before you had a simple explanation of the data you needed so many bits so many bits to um to swipe the data and afterwards only so many and the difference between before and after that is the reward that you get so that's the true we want that controller the first neural network should get in response to the improvements of the second Network which are now measured in terms of compression progress so first I needed so many resources to impose the data but then I discovered this regularity of gravity and I can greatly compress all kinds of videos that that are reflecting the concept of gravity and certainly I have a huge insight into the nature of the world and that is my true Joy scientific as a as a scientist my true joy as a scientist that I want to encode in a little number which is given as a reward to the guy who is inventing the experiments that lead to the data to the data with the falling apples for example right well and of course this is this has been a challenge in machine learning you know since the beginning which is okay as we add more and more parameters how do we prevent it from learning spurious information with those parameters and instead have it focus on parsimonious explanations on regular explanations on things that in this Universe are more likely to generalize you know to unseen examples and so I think my question to you is does this setup that you describe is it a form of that and or what is the state of the art you know these days for helping to push or nudge neural networks towards learning parsimonious models for the world rather than highly detailed spurious susceptible to you know high frequency uh anomalies and adversarial examples and all this sort of thing yes what is the current state of the art in regularizing descriptors of the data such as neural networks such that you get simple explanations of the data such that you get short programs that compute the data in other words such that the description of the data is a short program that computes a much larger raw data and um and how close can we get to the limits which are given through this concept concept of algorithmic information or call mcgarve complexity tomograph complexity of any data is the length of the shortest program on some general computer that computes it since in our field the general computers are returned neural networks we want to find a simple Recon Network that computes all this data and given one computation of the data we want to find an even simpler one so we want to ask this idea of compression progress and here I have to say although we have lots of regularizers invented through out the past few decades there's nothing that is really convincing I think one of the really important missing things is um to to make that work in a way that is truly convincing that is as convincing as chat GPT is today in the much more limited domain of of generating text from previously observed texts and stuff a very old idea of um I think the 1980s was to have weight decay in a neural network which basically is the idea that all the weights should um have um have an incentive to become close to zero such that you can prune them and so people built in a regularizer that just punished weights for being large or being very negative but that didn't work really well and something better was flat minimum search that was 1998 and first Arthur my brilliant student back then roughly the same time when the lstm paper came out and uh and there the idea is if you have um if you plot so weights of a neural network on on the XX on the x-axis and you plot the error on the y-axis and um given the the weights you have high or low error and then there is for example a sharp error function which has a sharp minimum which um which goes like that can you see my finger so here here is the x-axis here's the y-axis here's the arrow and the error for a certain weight is really really low but then for a different weight in the environment in the vicinity it's um hi again which would be very different from a flat minimum which would be like this and so um here's the error and it's going down and for many many ways it is low the error and then it goes up again so if you're a sharp very sharp well versus a very broad well yes a sharp well versus a broad well now if you are in a shop well you have to specify the weights with a lot with with high precision so you have to spend many bits of information on encoding the the weights of this network as opposed to a large um to a flat minimum where it doesn't matter if you you know perturb the weights um because the error remains uh low in this flat minimum so what you really want to find is is a network that has low complexity in the sense that you can describe the good Network source with low error with very few bits of information and suddenly if you maximize or if you minimize that flap minimum second order error function then suddenly you have a preference for Networks that um that for example do this you you have a hidden unit and the outgoing weights they have certain values but if you give a very negative way to the hidden unit then it doesn't matter what all these outgoing weights do and flat minimum minimum search likes to uh find weight matrices like that where one singing weight can eliminate many others which you certainly don't need any longer such as the description complexity of the whole thing is much lower than in the beginning when you when you just had a random initialization of all these ways so that is much more General than weight Decay because weight Decay doesn't like these strong ways it wants to remove them but sometimes it's really good to have a very negative weight coming to a hidden unit which is Switched Off through that weight such that all the outgoing connections are meaningless but it's not um what you what you it's very nice it's a very nice principle but it's not as general as finding the shortest program on a universal computer that computes the weight Matrix that is solving your problem how do you think we're going to get to that point how do you think what approaches are going to lead us to finding things that approach comograph complexity yeah and I think that part has again a lot to do with meta learning and as a system is able to run its own learning algorithm on the network itself it can suddenly speak about the um algorithms in form of weight matrices and it can discuss Concepts such as the complexity of a weight Matrix and then you can conduct a search in this space of networks that generates weight matrices and then you suddenly on in the game so you are playing the right game and then it's more a question of how to um yeah choose an initial learning algorithm such as grain descent to come up with something that computes The Simple Solutions which you really want to see in the end sorry very recent papers on that on on aspects of that um came out uh just um a while ago with my students Vincent Hermann and Luis quieres and um and uh Francesco facho and my poster Kazuki area and Robert Choi does also um and then the idea is really to have one network that computes an experiment and the experiment itself is the weight Matrix of a Recon Network so there is a generator of an experiment which can be anything that describes a computational interaction with an environment so a program so that experiment is then executed in the real world there's a prediction machine that predicts the outcome of the experiment before the algorithm is executed and so then there is just a yes or no question either the following outcome will occur or not either it will occur or not but now the entire setup is such that you don't have predictions all the time about every single Pixel no you just have something which is very abstract and which is just about whether a certain unit of the Recon network is going to be on or off at the end of the experiment and this internal on and off unit can represent any computational question any questions that you can ask at all and now the the task of the experiment generator which is another Network which generates a recurrent Network weight Matrix which represents the experiment the task of this experiment generator is to again come up with something that surprises the um the prediction machine which looks at the experiment and says yeah it's going to work or not and uh and suddenly you are again in this old game except that now you have this world of abstractions where the abstractions can be anything that is computable interesting really cool pretty cool could we spend the last 10 minutes or so just talking about some of the the current AI landscape so in particular the capabilities of gpt4 um the moat building thing and and the the power that companies like uh Google and open AI have and um also the potential for open source so maybe we'll just start with the you know the very current capabilities of of gpt4 are you impressed with it what do you think I'm impressed in the sense that um I like the outcomes that you get there and um it wasn't obvious a couple of years ago that it would become so good on the other hand of course it's not yet this full AGI thing and it is not really close to um to justifying uh those fears that some researchers sometimes now um document and um in letters and public letters and so on so to me it's a little bit um like a Visa view because for for many decades I have had discussions like that and people said you are crazy when I said that within my lifetime I want to build something that is smarter than myself and now suddenly in recent years um some of the guys who said it's never going to happen suddenly they just look at chat gbt and they think oh now we are really close to AGI and whatever uh so um I I don't share these um [Music] extreme um I'm less impressed than some of those guys let me say that right the open source movement that you mentioned you you want to ask a specific question about that right well yeah that was that famous Google memo that got leaked and when the waits for llama from Facebook went out within about two or three weeks um it was a valing pretty similar to chat GPT you know with this um Laura fine tuning and the open source Community has just exploded you know you can now run it on your laptop and there is some question whether there is a significant gap between the capability you know is is it just a pilot trick is it really as good potentially or could it be as good as some of the next best models from open AI but I guess the question is do you think that we need open AI to to have the best models no of course not um no I'm very convinced I'll see open source movement and have supported that some people say the open source movement is maybe six or eight months behind the large companies that are now um coming out with these models and um I think the the best way of making sure that there won't be dominance through some large company is to support the open source movement because how can a large company compete against all these brilliant PhD students around the world who are so motivated to you know within a few days create something that is a little bit better than what the last guy has um put out there on GitHub and whatever so I'm I'm very convinced that this open source movement is going to make sure that there won't be a huge mode for a long time I'm reading between the lines here but I would guess you would be opposed to legislation like the EU is considering where you know the very tight restrictions on generative models you know onerous onerous kind of uh approval processes and things like that because that's going to have this chilling effect on on open source Innovation and and the little guys wouldn't it yes I have signed letters um which which support the open source movement and whenever I get a chance to um maybe influence some EU politicians then I'm trying to contribute to making sure that they don't don't shoot themselves in the foot by um by by killing killing Innovation through the open source movement so you certainly want to avoid that uh lots of different open source movements around the world so if one big entity fails to support open source or even makes it harder for open source there will still be lots of other entities which won't follow and so no matter what's going to happen on the political level I think open source is not going away I guess just in closing you've been in this game for decades now and what is I know it's a bit of a strange question to ask but what's your fondest memory in your career my fondest memory oh it's usually when I discover something that I think nobody has seen before but that is uh that happens very rarely because most of the things you think of somebody else has done before but yeah yeah um what usually happens is um you and and this has happened many times not many times but quite a few times in my career since the 80s as a scientist who publishes stuff yeah some of you think oh that is the solution to all these problems and now I really figured out a way of building this Universal system which learns how to improve itself and learns the way to improve the way it improves itself and so on and now we are done and now all is that's necessary is to scale it up and it's going to solve everything and then you think a little bit long about it and maybe you have a couple of Publications but then it turns out something is missing you know something important is missing and yeah and actually it's not that great and actually you have to think her to add something important to it which then for a brief moment looks like see greatest thing since sliced bread and and then you get excited again but then suddenly you realize oh it's still not finished something important is missing and so it goes back and forth like that I think that's the life of a scientist the greatest Choice um those moments where you have an Insight where suddenly Things fall into place such that along the lines of what we discussed before the description length of some solution to a problem suddenly it shrinks because two puzzle pieces they suddenly match and uh and become one or become one in the sense that they fit each other such as suddenly you have the shared line between the two puzzle pieces one is negative and the other one is positive and certainly the whole thing is um much more compressible than uh there's some of the things separately so um these these things that's what's driving um scientists like myself I guess wonderful um Professor you again Schmidt it's been an absolute honor thank you so much for coming on the show today thank you it was such a pleasure talking to you