Transcript for:
AI and the Intelligence Explosion

there would then unquestionably be an intelligence explosion and the intelligence of man would be left far behind thus the first Ultra intelligent machine is the last invention that man need ever make that is the premise of the movie The Matrix very scary stuff Leopold Ashen brener is a former open aai employee who got fired for apparently leaking information but what he says is that he was just warning the board about their sever lack of Safety and Security protocols within the company and then he put together a 165 page Manifesto talking about what he believes is the future of not only hitting AGI by 2027 but then achieving super intelligence by the end of this decade and he says very few people are actually aware of what is coming and I read all 165 pages and I'm going to break the whole thing down for you all right so here it is Leopold Ashen burner situational awareness the decade ahead I read through everything and I highlighted the most important bits so let's get into it first and foremost dedicated to Ilia Suk who I believe Ilia and Leopold were the ones who really kicked off this going to the board and telling them about all of their security concerns so he dedicated this entire thing to Ilia and he also gave a thank you to Yan Ley and Ilia again and if you remember Yan is somebody who just left open AI with Ilia just about a couple weeks ago over the same security concerns so obviously I'm not going to read this whole thing but especially in the introduction there are a lot of really interesting bits so I'm going to try to cover it all over the past year the Talk of the Town the town being San Francisco has shifted from1 billion compute clusters to1 billion do compute clusters to trillion dollar clusters every six months a zero is added to the boardroom plans and that is something that we're seeing the explosion of Nvidia stock the acquisition the massive acquisition of gpus by Leading tech companies is unlike anything we've ever seen in the history of technology and one thing that I haven't thought a lot about is the power needed to power all of this compute and it seems like behind the scenes there is a fierce scramble to secure every Power contract still available for the rest of the decades and we're going to touch a lot more on the power consumption how we're going to power all of these massive server Farms American big business is gearing up to pour trillions of dollars into a long unseen mobilization of American industrial might by the end of the decade American electricity production will have grown tens of percent that is crazy to think about all of these compute clusters being built out need electricity they feed on electricity that is how they run and we literally do not have enough electricity in the United States currently to power everything that we're going to need to power to to reach AGI and super intelligence the AGI race has begun now for his predictions by 2025 2026 these machines will outpaced college graduates we're basically almost there by the end of the decade they will be smarter than you or I we will have super intelligence in the true sense of the word and to be clear he defines super intelligence as what is beyond AGI AGI is one thing super intelligence is another and I'm going to show you what that's all about soon it's kind of scary to think about along the way National Security Forces not seen in half a century will be Unleashed and before long quote unquote the project will be on and I will get to what the project is and what it entails later on if we're lucky we'll be in an allout race with the CCP that is the government of China if we're unlucky and allout War and to be clear he is extremely concerned with the CCP and them as an adversary to the US within the context of AGI and super intelligence in fact if we take a look at this Business Insider article about him getting fired he says I ruffled some feathers Ashen Brer said he wrote a memo after a major security incident he didn't specify what it was he shared it with a couple open aai board members and then he wrote that the company security was egregiously insufficient in protecting against theft of key algorithmic secrets from foreign actors human resources at open aai then gave him a warning about the memo telling him it was racist and unconstructive to worry about the Chinese communist his party now who knows if this is actually true this is the business insiders report coming from Leopold himself I don't know but either way to think that worrying about the CCP hacking us private entities is racist I mean this happens and has happened many many times we've lost a lot of intellectual property due to exactly that now back to his paper he says that everyone is talking about AI but few have the faintest glimmer of what is about to hit them so he is very much in the camp that we're going to have intelligence explosion he believes AGI is coming very soon and shortly behind that super intelligence that we might not be able to control then he talks about the naysayers Nvidia analysts still think 2024 might be close to the peak mainstream pundits are stuck on the willful blindness of it's just predicting the next word and that is in reference to how Transformers work currently large language models do just predict the next word but if you scale that up enough that may become super intelligence at most they entertain another internet scale technological change before long the world will wake up and he says there are only a few hundred people most of them in San Francisco and the AI Labs that have situational awareness and that again is the title of this paper situational awareness meaning they actually understand what is coming all right so in chapter 1 from GPT 4 to AGI counting the O and the O are orders of magnitude those are 10 exchanges in either computes or intelligence whatever we're measuring that is an order of magnitude change AGI by 2027 is strikingly possible that is right around the corner gpt2 to gbt 4 took us from preschooler to Smart high schooler abilities in four years tracing trend lines in compute so 0.5 orders of magnitude per year algorithmic efficiencies another 0.5 orders of magnitude per year and un hobbling gains from chapot to agent we should expect another preschooler to high schooler size qualitative Jump by 2027 and I will explain what all of this means through this paper he starts by saying GPT fors capabilities came as a shock to many and that is true I don't think anybody was really understanding how powerful these systems were I think a lot of people were shocked by GPT 3.5 not a lot of people understood what is possible with large language models but GPT 4 was merely the continuation of a decade of Breakneck progress in deep learning a decade earlier so just 10 years earlier models could barely identify simple images of cats and dogs four years earlier gpt2 could barely string together semi- plausible sentences now we are rapidly saturating all the benchmarks we can come up with and yet this dramatic progress has merely been the result of consistent Trends in scaling up deep learning now that is what a lot of people say if we just continue to scale up large language models we can reach AGI and Beyond Yan laon and the team at meta is of a different thought they do do not believe that the underlying architecture behind Transformers behind large language models is enough to scale up and reach AGI and Beyond and what happens when we hit AGI that's when we will have the explosion of artificial intelligence but why how does that actually happen because of AGI and again AGI itself isn't the explosion in intelligence that just means that artificial intelligence is good enough to do things better than humans on average so let's see I make the following claim it is strikingly plausible that by 2027 models will be able to do the work of an AI researcher engineer that doesn't require believing in scii it just requires believing in straight lines on a graph here it is this is the effective compute on the Y AIS normalized to GPT 4 and then this is the timeline on the xaxis as we see here this is where we've been all of a sudden we have crossed over this GPT 4 line and we will continue all the way up to here we're automated AI researcher why is is that the mark though why is that the Benchmark that we have to measure against well if AI is as smart or smarter than an AI researcher and engineer then all of a sudden we can deploy hundreds of them thousands of them millions of them all working 24 hours a day in parallel doing research and compounding the effects of that research and that development of AI so it is this sudden explosion in our ability to find new research Dev develop that research and then deploy that research here is something really interesting that I found publicly things have been quiet for a year since the gbt 4 release as the next generation of models has been in the oven leading some to Proclaim stagnation and that deep learning is hitting a wall and Gary Marcus is one of those people who thinks that large language models have hit a wall but by counting the O's orders of magnitude we get a Peak at what we should actually expect all right so then he continues to talk about the leap from GPT 2 to gbt 4 and it was not a onetime gain we are racing through o extremely rapidly and the numbers indicate we should expect another 100,000x effective compute scale up resulting in another gpt2 to GPT 4 size qualitative jump over 4 years four years that's crazy and that doesn't just mean a better chatbot gains should take us from chatbots to agents of course you know I'm very bullish on Agents from a tool to something that looks more like a drop in remote worker replacement another jump like that very well could take us to AGI to models as smart as phds or experts that can work beside us as co-workers perhaps most importantly if these AI systems could automate AI research itself that would set in motion intense feedback loops that I've been describing so far that I hadn't thought enough about the fact that the second we create something that can iterate on itself and actually do research SE Arch on making itself better that feedback loop is going to be insane and that's kind of the premise of the movie The Matrix even now barely anyone is pricing all this in so let's take a second and let's talk about the progress to date so we have gpt2 in 2019 can you believe that by the way 2019 just 5 years ago preschooler wow it can string together a few plausible sentences gpt2 could barely count to five without getting tripped up so here's the example Le and this is from the gpt2 paper you can check it out yourself but it's just an example of gpt2 having reading comprehension then gpt3 in 2020 Elementary schooler wow with just some few shot examples that can do some simple useful tasks it was also commercially useful in a few narrow ways for example gpt3 could generate simple copy for SEO and marketing and here it's showing what was very impressive at the time so a few examples including a little coding example with CSS it looks like then GPT 4 in 2023 a smart high schooler wow it can write some pretty sophisticated code and iteratively debug it can write intelligently and sophisticatedly about complicated topics it can reason through difficult high school competition math it's beating the vast majority of high schoolers on whatever test we can give it and then again some examples and these are crazy complex examples that in my mind are Beyond high schoolers but GPT 4 as it says here is still somewhat uneven for some tasks it's much better than than smart high schoolers while there are other tasks it can't yet do one perfect example of that is give me 10 sentences that end in the word Apple in fact probably an 8-year-old can do that but because of the way that Transformers work it really struggles with questions like that basically anything with planning it really struggles with so here we go here's the timeline preschooler Elementary schooler smart high schooler and then what is next now let's take a look at this graph here this is kind of in insane so all the way back to 1998 we can see how long this progress took and these are just different benchmarks that they measured artificial intelligence against so you can see how long it took to get better but then all of a sudden right around 2016 we started to have this explosion in capabilities and all of these trend lines are going directly vertical their Improvement is substantial and here is an example from GPT 3.5 which was 2022 to GPT 4 which is 2023 here are their performance in a bunch of different exams from the bar exam elsat sat gr Etc look at GPT 3.5 10th percentile 40th percentile 87th perc all but now look at gp4 I mean the GRE verbal 99th per the bar exam 90th per the LSAT 88% sat 97% so really outperforming humans on most of these tests on average and then he goes on to talk about the Skeptics over and over again year after year Skeptics have claimed deep learning won't be able to do X and have been quickly proven wrong with the example here's Yan Leon again Yan laon is the head of meta AI predicting in 2022 that even GPT 5000 won't be able to reason about physical interactions with the real world GPT 4 obviously does it with ease a year later now I don't necessarily believe that it does not do it with ease and Yan laon gave some really good examples of gb4 not being able to be a world model in itself and I made an entire video about it where he had a discussion with Lex Friedman check out that video and Yan Leon believes that we're going to need some additional technology in addition to Transformers to actually reach a true World model he's been inventing something called I believe V jeppa and so you can also check that out here Gary Marcus walls predicted after gbt2 being solved by gpt3 and the walls he predicted after gpt3 being solved by gbt 4 and so on now the hardest unsolved benchmarks are tests like GP QA a set of PhD level biology chemistry and physics questions many of the questions read like gibberish to me and even phds in other scientific Fields spending 30 minutes plus with Google barely score above random chance Cloud 3 Opus currently gets 60% right near instantly kind of nuts here is another example where just scaling up compute had incredible results so this is the three examples of different computes based on Sora and Sora is the textto video generation model by open AI here is the base compute and you can see one screenshot from this video you can barely tell what's going on then with 4X compute you know it's a dog there's a person in the background then 32 times compute a highly detailed video very precise I don't see any mistakes or hallucinations in this video so really showing how just strictly scaling up compute can lead to incredible results with each o of effective compute models predictably reliably get better so he talks about the progress of gpts and AI in general and breaks them down into three categories so let's go over these because I think these are really interesting number one is compute we're building more gpus we're building better gpus and there's just going to be more compute out in the world and if you throw compute at these models they get better two algorithmic efficiencies which I think kind of fly under the radar of people understanding how a simple algorithmic efficiency can result in huge gains and then finally un hobbling gains so with simple algorithmic improvements like reinforcement learning from Human feedback Chain of Thought tools and Scaffolding we can unlock significant latent capabilities un hobbling gains another word for that is Agents so he's talking about a lot of things that we do with agents today as the UN hobbling gains so Chain of Thought or having multiple agents working together being able to use tools being able to use scaffolding so yeah un hobbling is a big way to unlock gains now let's go into depth about compute when Moore's law was in its Heyday it was comparatively glacial perhaps 1 to 1.5 o per decade we are seeing much more rapid scale UPS in compute close to 5x the speed of Moore's Law instead because of Mammoth investment now whether that investment is going to pay off is yet to be seen but we've seen Nvidia stock price Skyrocket that is because every major tech company is buying up gpus as quickly as they can and they are putting in place the infrastructure for AI down the road now here's the thing we've spent so much money on gpus as a country and in the world as a whole yet the revenues from artificial intelligence and that investment only is in the order of tens of billions of dollars and most of which is captured by open AI currently next I want to touch on this the SF Rumor Mill San Francisco is AB breasted with dramatic Tales of huge GPU orders in additional two o of compute a cluster in the tens of billions seems very likely to happen by the end of 2027 even a cluster closer to plus three o of compute a hundred billion dollar seems plausible and is rumored to be in the works at Microsoft open AI now we've already heard the rumors that Sam Altman is Raising 7 trillion for compute so the next order of magnitude is certainly there now let's talk about algorithmic efficiencies algorithmic progress is probably a similarly important driver of progress here is an example here's the relative inference cost of a 50% performance on the math benchmark so in July 2022 we had the relative cost I believe this is in tokens minurva 540b all the way up there GPT 4 much lower and then Gemini 1.5 flash with fuse shot very inexpensive now so the cost of running inference on these models is dropping dramatically inference efficiency improved by nearly three orders of magnitude 1,000 times in less than 2 years and he says that there are two kinds of algorithmic progress those that simply result in better base models and that straightforwardly act as compute efficiencies or compute multipliers for example a better algorithm might allow us to achieve the same performance but with 10x less training compute and both of those things are equally important here he goes on to describe the price drop of gp4 so since the GPT 4 release a year ago open AI prices for GPT 4 level models have fallen another 6 or 4X for input and output with the release of GPT 40 Gemini 1.5 flash recently released offers between GPT 3.75 level and GPT 4 level performance while costing 85 and 57x less input to output than the original GPT 4 then he goes on to talk about mixture of experts which we've talked about at length on this channel and how you can have these massive models but then the inference only uses a fraction of the active model and so that is another huge inference in algorithmic gain so put together public information suggests that the gpt2 to GPT 4 jump included one to two orders of magnitude algorithmic efficiency gains and we can see that here while compute efficiencies will become harder to find as we pick the low hanging fruit AI lab investments in money and talent to find new algorithmic improvements are growing rapidly so even though we're kind of taking all that lwh hanging fruit in algorithmic efficiencies and it will become harder and harder to find those gains our investment to find those gains are outstripping the rate at which we're losing the ability to find gains if that makes sense now let's talk about another problem that a lot of people are talking about the data wall which basically means we're running out of data and if you watch this Channel at all you already know where we're going to go with this synthetic data but let's talk about it there is a potentially important source of variance for all of this we're running out of internet data that could mean that very soon a naive approach to pre-training larger language models on more scrape data could start hitting serious bottlenecks basically all of the most modern Frontier models have used all of the internet's data there is no more data and as I've said a lot in the past looking at private or proprietary data sets like X like Reddit Google Apple these data sets are going to increase in value very very quickly and that could be a differentiation between different AI organizations so here it says llama 3 was trained on 15 trillion tokens common crawl and and lists a few other things then he goes on to say you can go somewhat further by repeating the data but academic work on this suggests that repetition only gets you so far repeating the data meaning look at the data over and over again but there are other things to think about when we're thinking about this limit on data this isn't to be understated we've been writing the scaling curves writing the wave of the language modeling pre-training Paradigm and without something new here this Paradigm will at least naively run out despite the massive Investments we Plateau all of the AI labs are rumored to be making massive research bets on new algorithmic improvements or approaches to get around this researchers are purportedly trying many strategies from synthetic data to self-play and RL approaches now synthetic data we've talked about that just means using one model to create a bunch of data for another model and I'm not so sure that that is a long-term solution but what could be a long-term solution and something that Sam Alman has talked about is is actually being able to do a lot more with your existing data he also talks about self-play here and if you remember alphao which is Google's AI that learn to play go better than any human it basically just started with learning the rules of the game and then created its own synthetic data and in this example it works well because it just played games over and over with itself and every time it won it would use that to train the model every time it lost it would use that as a negative signal and it played thousands and millions millions of games until it got to the point where it was better than humans and so that could be another technique that we use but one thing that Sam Alman said and one thing that also Leopold is saying is it should be possible to find ways to train models with much better sample efficiency so taking that existing set of data and getting more out of it and he gives a really good analogy for how that would work consider how you or I would learn from a really dense math textbook so what modern llms do during training is essentially very very quickly skim the textbook the words just flying by not spending much brain power on it rather when you or I read that math textbook we read a couple Pages slowly then have an internal monologue about the material in our head and talk about it with a few Study Buddies read another page or two then try some practice problems fail try them again in a different way get some feedback on those problems try again until we get the problem right and so on and then eventually the material just clicks that is not how Transformers work we just give it ACH of data it looks at it once and then it tries to guess the next token and it just doesn't really think about the information at all so imagine we could apply this new learning technique to large language models and they would actually think and go over the same data over and over again they would be able to get so much more out of the existing data sets and there is also a lot of garbage data on the internet so just being able to curate and having higher quality data sets and getting more out of those data sets could unlock huge performance gains so here he talks about that current Frontier models like llama 3 are trained on the internet and the internet is mostly crap like e-commerce or SEO or whatever many llms spend the vast majority of their training compute on this crap rather than on the really high quality data imagine if you could spend GPT 4 level compute on entirely extremely high quality data it could be much much more capable model and then here he explains how that could actually be a differentiator for different AI labs this also means that we should expect more variance between the different labs in the coming years compared to today which is a great thing basically all of these models are becoming commoditized and that is something that I deeply believed but maybe after reading this I don't believe it as much as I did up until recently the state-of-the-art techniques were published so everyone was basically doing the same thing on the same data set and then he goes on to say if everybody is figuring out new techniques for leveraging the same amount of data Whoever has the best techniques the best algorithms is going to win and that's a differentiator now open source will have a much harder time competing because if you're having these huge algorithmic gains internally he's saying that these AI Labs probably won't want to share that whether that's true or not I'm not so sure because a lot of these researchers like their work being out there so again let's have a thought experiment imagine if when asked to solve a hard math problem you had to instantly answer with the very first thing that came to mind that's basically how llm solve math most of us problem solve through step by step and we able to give a little bit of that step by byep strategy to large language models with Chain of Thought reasoning and even with that it's still not great here he talks about the different potential algorithmic gains we've already talked about rhf we've talked about Chain of Thought scaffolding which is Chain of Thought Plus+ rather than just asking the model to solve a problem have one model have a plan of attack another propose a bunch of possible solutions he's basically describing agents then we have tools again agents use tools so that's external math libraries that's searching the internet Etc and then context length improvements will also help us improve the models themselves and then posttraining improvements so he talks about how gp4 and its current state is a lot different from the original GPT 4 that came out another way he describes large language models as being hobbled is they do not have long-term memory they can't use a computer they still mostly don't think before they speak they can only engage in short back and forth dialogues rather than going away for a day or a week thinking about the problem and they are mostly not personalized to you for your application and here let's wrap it up by 2027 rather than a chatbot you're going to have something that looks more like an agent more like a coworker now to get from this chatbot to an agent cooworker he talks about a few things one is solving the onboarding problem every time that you release a model or you start to use a new model it's essentially just a raw model that knows nothing about you so it's kind of like hiring somebody new every single time now if we had an automated way to give models all of the information about your company about yourself to help them quote unquote onboard board that would be a huge time saer and I think what he's describing is basically just rag cuz he says being able to have very long context to onboard models to basically feed in all of this additional information all this context is good but I think rag is also a great solution to that then he says the test time compute overhang and basically what that means is when we give a model a prompt it comes back to us immediately if we allowed them and this takes a technological unlock if we allowed them to take our prompt and go back and think about it actually spend time on it days or weeks maybe a timeline we Define they hypothetically would come back with something that is much more effective and higher quality now I've talked about this iterative Loop that models can take and this is something that I wrote so let me try to explain something that I've attempted in the past I've tried to take a model give it a prompt let's say write 10 sentences that end in the word Apple so it does that and let's say it fails one of them I take the response and I simply repackage it in another prompt give it back and say are you sure make sure all of them end in the word apple and if not please iterate and give me a better answer and every single time you do this without some kind of external feedback mechanism there's no way for the model to understand I made a mistake every single time I've done this it says yeah that's right it looks right to me now there's ways around this with coding for example you can iterate on coding really easily because the model will provide code you'll run the code and then you'll provide back either a success or any errors that you see and that's why large language models are able to iterate on code so well but on other things that don't have this external validation it's nearly impossible for a large language model's answer to improve just by asking it again or to check its work they could also have different large language models checking each other's work and maybe one is better than another but even then some external feedback is necessary then he also describes large language models being able to use a computer chat GPT right now is basically like a human that sits in an isolated box that you can text we will we will simply enable models to use a computer like a human would so why not why can't it search the internet why can't it write code and execute code and again that's what agents are that means joining your Zoom calls researching things online messaging and emailing people reading shared docs using your apps and Dev tooling and so on something I really believe in a model in itself by itself independently is never going to be as powerful as if it had access to tools but now let's move past all of this and talk about what actually happens when models can do AI research themselves once models can automate AI research itself that's enough enough to kick off intense feedback loops and we could very quickly make further progress the automated AI Engineers themselves solving all the remaining bottleneck to fully automate everything in particular millions of automated researchers could very plausibly compress impress a decade of further algorithmic progress into a year or less AGI will merely be a small taste of the super intelligence soon to follow and more on that in the next chapter and we're going to touch a lot on that all right chapter two from AGI to Super intelligence the intelligence explosion AI progress won't stop at human level hundreds of millions of agis could automate AI research compressing a decade of algorithmic progress into one year we would rapidly go from Human level to vastly superhuman AI systems the power and the Peril of super intelligence would be dramatic all right then this is a quote from i j good 1965 where he's basically describing the Matrix let an ultra intelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever since the design of machines is one of those intellectual activities an ultra intelligent machine could design even better machines there would then unquestionably be an intelligent explosion and the intelligence of man would be left far behind thus the first Ultra intelligent machine is the last invention that man need ever make that is the premise of the movie The Matrix very scary stuff I'm going just going to read that last sentence again thus the first Ultra intelligent machine is the last invention that man need ever make okay so next the bomb and the super so I'm not going to get too deep into what he's talking about here but he basically comp compares the nuclear bomb to what we're experiencing now with artificial intelligence and specifically he says that going from nothing to having that first nuclear bomb was just a baby step but going from the bomb to the hydrogen the super bombs was just as important if not more important than actually discovering that first bomb and that's because the increase in energy the increase in Devastation between the bomb and the hydrogen bomb was so dramatic at compared to having nothing to just the bomb so here he talks about that discovery of the first atomic bomb but then just 7 years later teller's hydrogen bomb multiplied yields a thousandfold once again a single bomb with more explosive power than all of the bombs dropped in the entirety of World War II combined and that is what he's saying is the difference between getting the AGI and then Super intelligence once we get to AGI we won't just have one AGI I'll walk through the numbers later but given infrance GPU fleets by then will likely be able to run many millions of them perhaps 100 million human equivalents and soon after at 10x plus human speed so he's saying all of these millions or hundreds of millions of AI researchers will be improving itself self-improvement and that is the intelligence explosion so here is a graph a potential scenario for the intelligence explosion we have a pretty linear Improvement right here and then all of a sudden we have AGI discovered or AI researchers discovered and then Super intelligence or the automated AI research just explodes now here's something that I've touched on in the past if we have ai systems that are just vastly smarter than humans at a certain point we're not even going to be able to understand what they are how they work we already don't understand how large language models work and the inside of them it's basically a black box but imagine they get so sophisticated that we can't even begin to understand how they're working why they make certain decisions that is basically just handing over complete trust to these AI systems here's an example I like to give let's think about coding the way that coding looks today it looks like natural language and the reason it looks like natural language is because humans are really bad at coding so it has to look like natural language to help us but in the future if large language models are writing all of our code it doesn't have to look like natural language it can look like anything it could be ones and zeros only it could be symbols it could be something we can't even imagine so that is what he is describing now applying super intelligence to R&D and other fields explosive progress would broaden from just ml research soon they'd solve robotics make dramatic leaps across other fields of Science and Technology within years and an industrial explosion would follow super intelligence would likely provide a decisive military advantage and unfold Untold powers of Destruction we will be faced with one of the most intense and volatile moments of human history here he goes on to describe that AI research can be done by AI itself and once that happens that is the kickoff to that explosion of intelligence so let's even dive into that a little deeper in addition to their raw quantitative Advantage automated AI researchers will have other enormous advantages over human researchers they'll be able to read every single ml paper if it written and have been able to deeply think about every single previous experiment ever run at the lab and in parallel learn from each other they'll be easily able to write millions of lines of complex code keep the entire codebase in context and spend human decades checking and rechecking every line of code for bugs and optimization you won't have to individually train up each automated AI researcher instead you can just teach and onboard one of them and then make replicas and then vast numbers of automated AI researchers will be able to share context enabling much more efficient collaboration and coordination compared to human researchers very scary stuff here he talks about possible bottlenecks to this envisioned future I'm just going to touch on it I'm not going to dive into it too much limited compute obviously if we're not able to build out our Computing infrastructure fast enough for whatever reason energ or silicon we're not going to be able to achieve this then he talks about the long tail so basically after we get all the lwh hanging fruit all of that additional research and Discovery costs a lot more to discover meaning it takes a lot more compute to get there to get additional gains and then there may be inherent limits to algorithmic progress okay now let's move on to what super intelligence actually means and why it's so powerful why it's going to change the world according to Leo uphold whether or not you agree with the strongest form of these arguments whether we can get a less than one year of intelligence explosion or it takes a few years it is clear we must confront the possibility of super intelligence the AI systems will likely have by the end of this decade will be unimaginably powerful so here is human progress here's time and what does it feel like to stand here right before this explosion so what happens when intelligence explodes well it's going to be applied to many fields and it's going to be able to improve itself then they're going to solve robotics something that a lot of people don't think is solvable in the near future but then we're going to have ai solve Robotics and then AI design robotics factories then those factories are essentially robots producing other robots have you ever played factorio and then we will dramatically accelerate scientific and technological progress we will have an industrial and economic explosion extremely accelerated technological progress combined with the ability to automate all human labor could dramatically accelerate economic growth self-replicating robot factories quickly covering all of the Nevada desert why he chose Nevada I don't know fine so let's look at this chart this is the growth mode hunting farming science Industry and then Super intelligence here is where we began to dominate it and then here is how long it took doubling time of the global economy so Hunting 2 million BC took us 230 years then as you can see drastically drops 860 years 58 years 15 years after the Industrial Revolution to double the economy what do we think is going to happen when we have super intelligence and then back to the list it will provide a decisive and overwhelming military Advantage the Drone storms and Robo armies will be a big deal but they are just the beginning we should expect completely new kinds of weapons from novel wmds to invulnerable laser based missile defense to things we can't even fathom and be able to overthrow the US government whoever controls super intelligence will quite possibly have enough power to cease control control from pre- Super intelligence forces even without robots the small civilization of super intelligences would be able to hack any undefended military election television system cunningly persuade generals and electorates economically outcompete nation states design new synthetic bioweapons and then pay a human in Bitcoin to synthesize it and so on very very scary stuff again another graph showing what happens when we hit that intelligence explosion so let's continue what happens with super intelligence very plausibly within just a year we would transition to much more alien systems systems whose understanding and abilities whose raw power would exceed those even of humanity combined there is a real possibility that we will lose control as we are forced to hand off trust to AI systems during this rapid transition again what I said earlier if we get to a point where these AI systems are so alien we can't even understand them we are basically just trusting them we have no way to verify we don't know if they're lying to us we don't know what's going on and we have to trust them cuz there are so much smarter than us and that is what he is trying to say we need to protect against more generally everything will just start happening incredibly fast by the end of it super intelligent AI systems will be running our military and economy during all of this Insanity we'd have extremely scarce time to make the right decisions here he talks about how the early discovery of the atomic bomb and what happened right before that it's very similar to what we're experiencing now I'm not going to dive too much into that that is the gist of what he is describing here next chapter 3 the challenges racing to the trillion dollar cluster as AI Revenue grows rapidly which it has but it's still a fraction of the amount we've spent on AI many trillions of dollars will go into GPU yes Data Center and power build out before the end of the decade the industrial mobilization including growing us electricity production by tens of percent will be intense the race to AGI won't just play out in code and behind laptops it'll be a race to mobilize America's industrial might and this is very true because again we need infrastructure both silicon and power as revenue from AI products grows rapidly plausibly hitting $100 billion an annual run rate for companies like Google or Microsoft by 2026 I think that might be a bit aggressive because as I said right here revenue from AI is still a fraction of what we've invested to it into it today here's a trillion dollar cluster factory made with Dolly kind of looks like factorio now let's look at this chart this is training compute year on the left orders of magnitude Improvement h100s equivalent the cost of purchasing and building it out and then the power necessary 100 gaw for 2030 plus four orders of magnitude 100 million h100 equivalent a trillion dollars of spend and this is probable I'd say it's at least plausible so Zuck bought 350,000 h100s Amazon bought a 1 gaw data center campus next to a nuclear power plant and so on and he goes on to describe that it's not actually the building out of the infrastructure specifically on the Silicon side it's actually finding the power to power all of this that's becoming a challenge very quickly so for the 100 gaw of power it will required is equivalent to greater than 20% of us electricity production today so here he talks about if it can be done and will it be done the economic returns justify the investment the scale of expenditures is not unprecedented for a new general purpose technology and the industrial mobilization for power and chips is doable now where he says it is not unprecedented he is talking about the internet roll out in the 9s and how much money was spent by Telecom providers to lay Cable in the ground to get ready for the internet explosion and then he continues to talk about power probably the single biggest constraint on the supply side will be power and this is something that musk has directly said and why Tesla is actually in a really good position they are a power company after all and I really hope that musk stays at Tesla and builds out his AI Vision because we need more AI competition so here we can see from 1985 all the way through 2030 this is the trend line of American power generation really not a huge explosion in generation but here is the total AI demand starting at about 2021 boom look at that increase all of a sudden we are going to need so much power that is not available to us so here's something important that he touches on and maybe why he was called racist by open ai's HR department which I just don't believe to most this seems completely out of the question some are betting on Middle Eastern autocracies who have been going around offering boundless power and giant clusters to get their rulers a seat at the AGI table he goes on to talk about democracy and how America and other Western Nations need to get ahead of this now before the decade is out many trillions of dollars of compute clusters will have been built the only question is whether they will be built in America some are rumored to be betting on building them elsewhere especially in the Middle East do we really want the infrastructure for the Manhattan Project to be controlled by some capricious Middle Eastern dictatorship oo the national interest demands that these are built in America anything else creates an irreversible security risk it risks the AGI weights getting stolen and shipped to China it risks the dictatorships physically seizing the data centers when the AGI race gets hot okay let me let me just break this down for you what he is saying is in the American AI Labs we are going to make all of these discoveries that eventually will lead to AGI whether that's the model weights the algorithm Innovations whatever it is we're going to have these secrets and all another country has to do is steal the secrets they don't need to invent them themselves they simply spin up an AI Factory an AI compute cluster take our model weights take our algorithms and have AGI for themselves and then it's Off to the Races and we've lost all of our competitive Advantage now here's the part where it's really hard for me to agree with he goes on to talk about locking down the labs which yes agreed we need to have high levels of security for AGI however he basically says open source shouldn't be a thing he doesn't explicitly say that but he says even the AI Labs the secretive close Source AI labs are not enough we need government intervention to come in and work with private sector to make sure that we are protecting these secrets at all costs the nation's leading AI Labs treat security as an afterthought currently they're basically handing the key secrets for AGI to the CCP on a silver platter securing the AGI secrets and weights against the state actor threat will be an immense effort and we're not on track he goes on to describe that it's going to become clear that the US will see AGI as the biggest secret it needs to keep and today we're not treating AI as anything important that needs to be kept secret especially from the CCP so all the trillions we invest the mobilization of American industrial might the efforts of our brightest Minds none of that matters if China or others can simply steal the model weights or key algorithmic Secrets behind our AGI and just build it themselves here he describes how current AI Labs measure their security efforts against random Tech startups competition not key National Defense projects as the AI race intensifies as it becomes clear that super intelligence will be utterly decisive in international military competition we will have to face the full force of foreign Espionage currently labs are barely able to defend against script kitties let alone have North Korea proof security let alone be ready to face the Chinese Ministry of State security bringing its full force to bear and he basically gives a bunch of examples where we have really failed to protect our secrets too many smart people underrate Espionage here are just a bunch of examples zero click hack any desired iPhone and Mac infiltrate air gapped Atomic weapons programs modify Google Source these are all things that happened find dozens of zero days a year that take on average of seven years to detect Spear Fish major tech companies install key loggers and look at all these examples so he is very big on security already China engages in widespread industrial Espionage the FBI director stated the PRC has a hacking operation great than every major Nation combined and there are two key assets we must protect the model weights and the algorithmic secrets we both know what those are I'm not going to describe those in detail but he just goes on to explain that if we can't keep those two things secret all the work we've put in all the research is meaningless because it could just be stolen and replicated outside of the US perhaps the single scenario that most keeps me up at night is if China or another adversary is able to steal the automated AI researcher model weights on the C of an intelligence explosion they would immediately use it and kick off that intelligence explosion and of course who knows if they're going to have the same sentiment around security and safety that we do it's hard to overstate how bad algorithmic Secrets security is right now between the labs there are thousands of people with access to the most important Secrets there is basically no background checking siloing controls basic infos SEC Etc people gab at parties in SF anyone with all the secrets in their head could be offered $100 million and recruited to a Chinese Lab at any point you can just look through office Windows simply look across the street and see and here's the example of being recruited indeed I've heard from friends that bite dance a Chinese company emailed basically every person who was on the Google Gemini paper to recruit them offering them L8 level 8 very senior position with presumably similarly High pay and pitching them by saying they'd report directly to the CEO in America of B dance once China begins to truly understand the importance of AGI we should expect the full force of their Espionage efforts to come to Bear think billions of dollars invested thousands of employees in Extreme Measures Special Operations strike teams dedicated to infiltrating American AI efforts and really the reason he was fired or that's what what he says is that a lot of these AI Labs just don't get it they do not understand they are optimizing for the new shiny product which is the same thing that Yan said as he's leaving versus safety now Yan was talking about more AGI safety and super intelligence safety how do we control and align these systems but what Leopold is talking about is actually protecting the secrets against foreign actors now I want to read this paragraph in full we're developing the most powerful weapon mankind has ever created the algorithmic secrets we are developing right now are literally the nation's most important National Defense Secrets the secrets that will be at the foundation of the US and her allies economic and Military predominance by the end of the decade the secrets that will determine whether we have the requisite lead to get AI safety right the secrets that will determine the outcome of World War III the secrets that will determine the future of the Free World and yet AI lab security is probably worse than a random defense contractor making bolts it's Madness all right now let's talk about super alignment and what super alignment means is making sure that super intelligent AGI or super intelligent AI in general is aligned with human incentives that it does not go against us basically reliably controlling AI systems much smarter than we are is an unsolved technical problem I am not a Doomer but I did spend the past year working on technical research on allying AI systems as my day job at open AI working with Ilia and the super alignment team there is a very real technical problem our current alignment techniques methods to ensure we can reliably control steer and Trust AI systems won't scale to superhuman I systems so our current techniques for aligning the systems won't scale but our current techniques for scaling the systems to Super intelligence he's saying will scale very interesting if we do rapidly transition from AGI to Super intelligence we will face a situation where in less than a year we will go from recognizable human level systems for which descendants of current alignment techniques will mostly work fine to much more alien vastly superhuman systems that pose a qualitatively different fundamentally novel technical alignment problem by the time the decade is out we'll have billions of vastly superhuman AI agents running around these superhuman AI agents will be capable of extremely complex and Creative Behavior we will have no hope of following along we'll be like first graders trying to supervise with multiple doctorates in essence we Face a problem of handing off trust here he talks about rhf which is reinforcement from Human feedback basically the model says something we say yes that's right or no that's wrong but the only way R lhf works is if we actually understand the input and the output and we can verify it and if we can't we can't use R lhf so here's an example we have a human with a little robot and we have a very simple output for user and user list give user a cookie just something silly then all of a sudden we have super intelligence and rather than this very simple and readable output we have something that is not readable and the human is saying is this safe well I don't know consider a future powerful base model that in a second stage of training we train with long Horizon RL reinforcement learning to run a business and make money so let's have this thought experiment we create a model and all we tell it to do is run the business make money by default it may well learn to lie to commit fraud to deceive to hack to seek power and so on simply because these can be successful strategies to make money in the real world what we want is to add side constraints don't lie don't break the law Etc but here we come back to the fundamental issue of alignment of superhuman systems we won't even be able to understand what they're doing so we're not going to notice if they're lying we're not going to be able to penalize them for bad behavior and that is very scary it's totally plausible they'll learn much more serious undesirable behaviors they'll learn to lie to seek power to behave nicely when humans are looking and pursue more nefarious strategies when we aren't watching oh my gosh what's more I expect within a small number of years these AI systems will be integrated in many critical systems including military systems it sounds crazy but remember when everyone was saying we wouldn't connect AI to the internet I remember that it wasn't that long ago now ai has access to the internet and the same people said we'll make sure humans are always in the loop not the case now let's look at alignment during the intelligence explosion on the left we have different categories that we're going to measure AGI here then Super intelligence required alignment technique rlh f++ and for super intelligence we don't know we're not sure the failure of aligning AGI pretty low stakes the failure of aligning super intelligence catastrophic the architecture of AGI pretty familiar descendant of current systems super intelligence aliens designed by previous generation super smart AI systems backdrop world is normal world is going crazy extraordinary pressures epistemic State we can understand what the systems are doing how they work and whether they're aligned and for super intelligence we cannot now here is one proposed approach to Super alignment a simp simple analogy for studying super alignment instead of a human supervising a superhuman model we can study a small model supervising a large model for example can we align gp4 with only gpt2 supervision will the result in GPT 4 appropriately generalizing what gpt2 meant from weak to strong generalization and this is a really interesting blog post I recommend reading it basically it's saying if we can train a small model to then align a larger model that then aligns a larger model and so on you kind of have this cascading effect and then they also talk about interpretability which I've been talking about in the last couple weeks on this channel being able to actually see inside the Black Box know why a certain input results in a certain output and the anthropic team has done a lot of this work and I am still planning on making a video about The Golden Gate model paper that said I'm worried fully reverse engineering superhuman AI systems will just be an intractable problem similar to say fully reverse engine ing the human brain and I'd put this work mostly in ambitious moonshot for AI safety rather than default plan for muddling through bucket we might try to build an AI Lie Detector by identifying the parts of the neural net that light up when an AI system is lying but maybe they'll figure out how to get around that detection we have Chain of Thought interpretability which is the same thing as looking into the model except we're actually looking at when using something like Chain of Thought where we actually get it to Output its steps of thinking to arrive at the final answer we can actually read those steps which we can do today pretty much but how do we ensure that the Chain of Thought remains legible and he goes on to describe needing to test and stress test the alignment at every step and we're also going to need to automate alignment research which is kind of crazy if you think about it he's not only saying that most likely we're going to have ai researchers that can automate AI research but also AI researchers that are going to automate the alignment so how do we trust them if we manage to align somewhat superhuman systems enough to trust them we'll be in an incredible position Labs should be willing to commit a large fraction of their compute to automated alignment research during the intelligence explosion and if you remember that is why Yan left open AI he was promised 20% of compute and that did not come to fruition instead open AI is optimizing and committing and investing in things that are shiny new products and we're also going to need defense so if it gets out of our hands what are we going to do so security air gap cluster we're going to need monitoring targeted capability limitations targeted Training Method restrictions maybe a kill switch so here's why he's optimistic and why he's also scared I'm incredibly bullish on the technical tractability of the super alignment problem so he thinks it's possible to solve it feels like there's tons of low hanging fruit everywhere in the field more broadly the empirical realities of deep learning have shaken out more in our favor compared to what some speculated 10 years ago for example deep learning generalizes surprisingly benignly in many situations it often just does the thing we meant rather than picking up some obstru line behavior and here's something that I found to be terrifying if it does happen I'm not sure we would even realize if a model self exfiltrated basically if a model got out of our hands would we even notice next chapter 3 D the Free World must Prevail super intelligence will give a decisive economic and Military Advantage China he talks about China a lot in this chapter I'm not going to go too deep in this chapter though because we've touched on it a lot in this video already already but he does talk about how Chinese power generation is accelerating dramatically much more than the US so even if they are behind us in compute or if they're behind us in our algorithms and our model weights simply being able to have that much power and getting decent chips and just massively scaling those decent chips while stealing our model weights and our algorithms is enough to build their own AGI and once they do that they no longer need us all right last let's talk about the project this is essentially what he is describing as the Manhattan Project for the atomic weapon for what we need for AI as the race to AGI intensifies the National Security State will get involved the USG will wake from its Slumber and by 2728 will get some form of government AGI project no startup can handle super intelligence somewhere in the skiff the endgame will be on he's saying it is absolutely necessary and he thinks a good thing that the government gets heavily involved in AGI creation I find it an insane proposition that the US government will let a random SF startup develop super intelligence imagine if we had developed atomic bombs by letting Uber just improvise that's a really funny line I got to say but in the next few years the world will wake up so too will the National Security State history will make a triumphant return and he's basically saying we are going to be able to Rally our American industrial might much like we did during the Industrial Revolution during Wars even during Co regarding regardless of your position of how that played out so do we need an AGI Manhattan Project slowly at first then all at once it will become clear this is happening things are going to get wild this is the most important challenge for the National Security of the United States since the invention of the atomic bomb in one form or another the National Security State will get very heavily involved the project will be the necessary indeed the only plausible response what do you think about that but he's not hyper prog government either I am under No Illusion about the government governments face all sorts of limitations and poor incentives I am a big believer in the American private sector and would almost never advocate for heavy government involvement in technology or industry AI labs are very good at some things they've been able to take AI from an academic science project to the commercial big stage we simply shouldn't expect startups to be equipped to handle super intelligence by the early 2030s the entirety of the US Arsenal like it or not the Bedrock of Global Peace and Security will probably be obsolete it will not just be a matter of modernization but a wholesale replacement simply put it will become clear that the development of AGI will fall in a category more like nukes than the internet again perhaps you are a true libertarian and disagree normatively let Elon Musk and Sam Altman command their own nuclear arsenals but once it becomes clear that super intelligence is a principal matter of National Security I'm sure this is how the men and women in DC will look at it so I'm going to read these last two paragraphs to close out this video and I want to know what you think let me know in the comments I know this was a lot to digest I'm going to link the entire paper down below I encourage you to read it yourself it is fascinating whether you agree with all of it some of it none of it just take a look it's good to be informed let's read this last paragraph and so by 2728 the end game will be on by 2829 the intelligence explosion will be underway by 2030 we will have summoned super intelligence in all its power and might whoever they put in charge of the project is going to have a hell of a task to build AGI and to build it fast to put the American economy on wartime footing to make hundreds of millions of gpus to lock it all down weed out the spies and fend off allout attacks by the CCP to somehow manage a 100 million AGI furiously automating AI research making a decades leaps in a year and soon producing AI systems vastly smarter than the smartest humans to somehow keep things together enough that this doesn't go off the rails and produce Rogue super intelligence that tries to seize control from its human overseers to use those super intelligences to develop whatever new technologies will be necessary to stabilize the situation and stay ahead of adversaries rapidly remaking US forces to integrate those all while navigating what will likely be the tensest international situation ever seen they better be good I'll say that if you enjoy this video please consider giving a like And subscribe and I'll see you in the next one