Well, good evening, everybody. Welcome. My name is Mike Froman. I'm president of the council, and it's a great pleasure to have you here tonight for one of our CFR CEO speaker series, and to have the CEO and co-founder of Anthropic, Dario Amadei, with us tonight. Dario was vice president of research at OpenAI, where he helped develop GPT-2 and 3. And before joining OpenAI, he worked at Google Brain as a senior research scientist.
I'm going to talk with Dario for about 30 minutes, then we'll open it up to questions from people here in the hall. We have about 150 people here. We have about 350 online, and so we'll try and get some of their questions in as well.
Welcome. Thank you for having me. So you left OpenAI to start Anthropic, a mission-first public benefit corporation.
Why leave? What are Anthropic's core values? And how do they manifest themselves?
in your work. And let me just say, a cynic would say, well, this mission first, this is all marketing. You know, how can you, can you give us some specific examples of how your product and strategy reflect your mission?
So yeah, if I, if I were to, you know, just, just back up and kind of set the context, you know, we left at the end of 2020. I think in 2019 and 2020, something was happening, which I think myself and a group within OpenAI, which eventually became my co-founder said Anthropic, were I think among the first to recognize. They're called kind of scaling laws or the scaling hypothesis today. And the basic hypothesis is simple.
It says that, and it's a really remarkable thing, and I can't overemphasize how unlikely it seemed at the time. If you take more computation and more data to train AI systems with relatively simple algorithms, they get better at... all kinds of cognitive tasks across the board.
And we were measuring these trends back when models cost $1,000 or $10,000 to train. So that's a kind of an academic grant budget level. And we forecast that these trends would continue even when models cost $100 million, $1 billion, $10 billion to train, which now we're getting to.
And indeed, that if the... quality of the models and their level of intelligence continued, they would have huge implications for the economy. It was even the first time we realized that they would likely have very serious national security implications.
We generally felt that the leadership at OpenAI was on board with this general scaling hypothesis, although, you know, many people inside and outside were not. But the second realization we had was that, you know, if the technology was going to have this level of significance, We really needed to do a good job of building it. We really needed to get it right.
In particular, on one hand, these models are very unpredictable. They're inherently statistical systems. One thing I often say is we grow them more than we build them. They're like a child's brain developing, so controlling them, making them reliable is very difficult.
The process of training them is not straightforward. So just from a system safety perspective, Making these things predictable and safe is very important. And then, of course, there's the use of them, the use of them by people, the use of them by nation states, the effect that they have when companies deploy them. And so we really felt like we needed to build this technology in absolutely the right way.
Open AI, a bit as you've alluded to, was founded with some claims that they would do exactly this. But for a number of reasons, which I won't get into in detail, we didn't feel that the leadership there was taking these things seriously. And so we decided to go off and do this on our own.
And the last four years have actually been a kind of, you know, almost a side-by-side experiment of, you know, what happens when you try and do things one way and what happens when you try and do things the other way and how it is played out. So, you know, I'll give a few examples of how, you know, we've really, I think, displayed a commitment to these ideas. One is we invested very early in the science of what is called mechanistic interpretability, which is looking inside the AI models and trying to understand exactly why they do what they do.
One of our seven co-founders, Chris Ola, is the founder of the field of mechanistic interpretability. This had no commercial value, or at least no commercial value for the first four years that we worked on it. It's just starting to be a little bit. a little bit in the distance, but nevertheless, we had a team working on this the whole time in the presence of fierce commercial competition because we believe that Understanding what is going on inside these models is a public good that benefits everyone, and we published all of our work on it so others could benefit from it as well.
You know, I think another example is we came up with this idea of constitutional AI, which is training AI systems to follow a set of principles. You know, instead of training them from data or from, you know, mass data or human feedback, you know, this allows you to... get up, say, you know, in front of, you know, Congress and say, these are the principles according to which we trained our model.
When we first came to, you know, when we had our first product, you know, our first version of Claude, which is our model, we actually delayed the release of that model roughly six months because this was such a new technology that, you know, we just, we weren't sure of the safety properties. We weren't sure we wanted to be the ones to kind of kick off a race. This was... just before ChatGPT.
So, you know, we arguably had the opportunity to, you know, to seize the ChatGPT moment. And, you know, we chose to release a little later, which I think had real commercial consequences, but set the culture of the company. A final example I would give is we were the first to have something called responsible scaling policy.
So what this does is it measures categories of risk of models as they scale. And we have to take increasingly strict security and deployment measures as we meet these points. And so we were the first one to release this, the first one to commit to it. And then a few months, within a few months of when we did, the other companies all followed suit.
And so we were able to set an example for the ecosystem. And, you know, when I look at what other companies have done, we've often led the way on these issues and often caused the other companies to follow us. Not always.
Sometimes they do something great and we follow them. but I think there's been a good history of us sticking to our commitments and I would contrast that with what we've seen from some of the other companies in their behavior. We now have several years of history and so far, fingers crossed, I think our commitments have held up pretty well.
I want to talk about both the risks and the opportunities that you've cited around AI. Since you mentioned responsible scaling, I want to talk about the risks and the opportunities that you've cited around issue. Let's go back to that.
We're at level two now. Yeah. So at what level is it existential? How will we know when we hit level three?
And if you hit level three, can you go backwards or does it only get worse? Yeah. So the way our responsible scaling policy is set up is we basically said You know, and the analogy was to biosafety level. So, you know, the biosafety level system is like, you know, these are how dangerous various pathogens are.
And so we said, let's have AI safety levels. And so AI safety level two is a level we're currently at. And that's, you know, systems that are powerful, but the risks they pose are comparable to the risks that, you know, other kinds of technology pose.
ASL 3. which actually I think our models are starting to approach. The last model we released, we said this model isn't ASL 3 yet, but it's getting there. ASL 3 is characterized, and we focus very much on the national security side, very kind of serious risks that are out of proportion to the risks that normal technologies have. So an ASL 3 model is designed as one that could allow you in the areas of, say, chemical, biological, or radiological.
weapons could allow an unskilled person simply by talking to the model and following its instructions to do things that you would have to have, say, a PhD in virology to do today. So once that is possible, if those risks aren't mitigated, then that would enhance the number of people in the world who are able to do these highly destructive things from, say, in the 10,000s today to in the tens of millions once the models are available. And so ...
When the models are capable of this, we have to put in mitigation so that the models are not willing to actually, you know, provide this information and security restrictions so that, you know, the models won't be stolen. And, you know, I think we're approaching that. We may actually hit that this year.
And we believe we have a story for how to deploy those kinds of models safely by, you know, removing their ability to do this very narrow range of dangerous tasks. without compromising their commercial viability. So this is a fairly narrow set of tasks. You say you're just going to prevent the model from answering those questions.
Yeah, prevent the model from engaging in those kinds of tasks, which is, it's not straightforward, right? You can say, you know, I'm taking a virology class at Stanford University. Working on my coursework, like, can you tell me how to make this particular plasmid?
And so the model has to be smart enough to not fall for that and say, hey, you know, actually that isn't the kind of thing you would ask. You sound like a bioterrorist. I won't answer your question. You sound like you have bad intent. But it's sort of limited to your own imagination or our own imagination as to what all the back-acting could be.
There are a lot of things that we may not anticipate beyond those four categories. Yeah, I mean, you know, I think this is an issue that... that just as every time we release a new model, there are positive applications for it that people find that we weren't expecting, I expect there will also be negative applications. We always monitor the models for different use cases in order to discover this so that we have a continuous process where we don't get taken by surprise.
If we're worried that someone will do something evil with model six, hopefully... some early signs of that can be seen in Model 5 when we monitor it. But this is the fundamental problem of the models. You don't really know what they're capable of. You don't truly know what they're capable of until they're deployed to a million people.
You can test ahead of time. You can have your researchers bash against them. You can have even the government, we collaborate with the government AISIs, test them.
But the hard truth is that there's no way to be sure. They're not like... code where you can do formal verification. What they can do is unpredictable. It's just like, you know, if I think of you or me instead of the model, you know, if I'm like the quality assurance engineer for me or you, you know, can I give a guarantee that like, you know, there's a particular kind of bad behavior you are logically not capable of will never happen?
People don't work that way. Let's, uh... Let's talk about the opportunities, the upside opportunities. Absolutely.
End of last year, you wrote an essay, Machines of Loving Grace. that talked about some of the upside, how one could achieve a decade's worth of progress in biology, for example, in a year, how the machines were going to be as smart as all the Nobel Prize winners, which probably depresses some of them. Tell us the upside. Tell us your best case scenario as to what AI is going to produce. Yeah.
So I'd go back by starting with the exponential. If we go back to 2019. the models were barely able to give a coherent sentence or a coherent paragraph. People like me, of course, thought that was an amazing accomplishment that models were not capable of.
And we had these predictions that five years from now, the models are going to be generating billions of dollars of revenue. They're going to be helping us code. We can talk to them like they're human beings.
They'll know as much as human beings do. And there were all these unprincipled objections of why that couldn't happen. You know, the same... exponential trends, the same arguments that predicted that, predict that if we go forward another two years, three years, maybe four years, we will get to all of this. We will get to models that are as intelligent as Nobel Prize winners across a whole bunch of fields.
You won't just chat with them. They'll be able to do anything you can do on a computer. Basically, any remote work that humans do, any modality, being able to do tasks that take Days, weeks, month, the kind of evocative phrase that I used for it in Machines of Loving Grace was it's like having a country of geniuses in a data center, like a country of genius remote workers, which they can't do everything, right?
There are restrictions in the physical world. And I think this still sounds crazy to many people, but look back on previous exponential trends. Look at the early days of the internet.
and how wild the prediction seemed and what actually came to pass. I'm not sure of this. I would say I'm maybe 70 or 80 percent confidence. You know, it could very well be that the technology stops where it is or stops in a few months.
And, you know, the essays that I've written and things I've said in, you know, in events like this, people spend the next 10 years laughing at me, but that would not be my bet. Let's just build on that one, because on the issue of. jobs and the impact that AI is likely to have on employment, there's a pretty big debate. Where are you on the spectrum?
But before I get there, how long will it take for AI, let's say, to replace the head of a think tank? I'm asking for a friend. Actually, we won't get to that.
Where are you on the spectrum? spectrum of everyone's going to be able to do some really cool things and they're going to be able to do so many more things than they're able to do now versus everyone's going to be sitting on their sofa collecting UBI. Yeah.
So I think it's going to be a really complicated mix of those two things that also Depends on the policy choices that we make. You can also answer the think tank question if you like. Yeah.
So, I mean, I guess I didn't, I kind of, you know, ended my answer to the last question without saying all the great things that will happen. So, honestly, the thing that makes me most optimistic before I get to jobs is things in the biological sciences. Biology, health, neuroscience. You know, I think if we look at what's happened in biology in the last hundred years, um, What we've solved are simple diseases. Solving viral and bacterial diseases is actually relatively easy because it's the equivalent of repelling a foreign invader in your body.
Dealing with things like cancer, Alzheimer's, schizophrenia, major depression, these are system-level diseases. If we can solve these with AI at a baseline, regardless of kind of the job situation, we will have a much better world. And I think...
think we will even, if we get to the mental illness side of it, have a world where it is at least easier for people to find meaning. So I'm very optimistic about that. But now getting to kind of the job side of this, I do have a fair amount of concern about this. On one hand, I think comparative advantage is a very powerful tool. If I look at coding, programming, which is one area where AI is making the most progress.
What we are finding is we are not far from the world. I think we'll be there in three to six months where AI is writing 90% of the code. And then in 12 months, we may be in a world where AI is writing essentially all of the code. But the programmer still needs to specify, you know, what are the conditions of what you're doing? You know, what is the overall app you're trying to make?
What's the overall design decision? How do we collaborate with other code that's been written? You know, how do we have some common sense on whether this is a secure design or an insecure design?
So as long as there are these small pieces that a programmer, a human programmer, needs to do that the AI isn't good at, I think human productivity will actually be enhanced. But on the other hand, I think that eventually all those little islands will get picked off by AI systems. And then we will eventually reach the... a point where the AIs can do everything that humans can.
And I think that will happen in every industry. I think it's actually better that it happens to all of us than that it happens, that it kind of picks people randomly. I actually think the most societally divisive outcome is if randomly 50% of the jobs are suddenly done by AI, because what that means, the societal message is we're picking half.
We're randomly picking half of people and saying, you are useless, you are devalued, you are unnecessary. And instead we're going to say you're all useless. Well, we're all going to have to have that conversation, right? We're going to have to look at what is technologically possible and say, we need to think about usefulness and uselessness in a different way than we have before, right? Our current way of thinking has not been tenable.
I don't know what the solution is, but it's... It's got to be different than we're all useless, right? We're all useless is a nihilistic answer.
We're not going to get anywhere with that answer. We're going to have to come up with something else. That's not a very optimistic picture, is what it is.
I would actually challenge that. You know, I think about a lot of the things that I do. You know, I spend a lot of time, for example, swimming. I spend time playing video games. I look at, like, human chess champions.
You know, you might think when Deep Blue beat Kasparov, and that was almost 30 years ago, that after that it would be like chess would be seen as a pointless activity. But exactly the opposite has happened. Human chess champions like Magnus Carlsen are celebrities.
I think he's even like a fashion model. He's like this kind of hero. So I think there's something there where we can build a world where human life is meaningful, and humans, perhaps with the help of AIs, perhaps working with AIs, build really great things.
So I am actually not that pessimistic. But if we handle it wrongly, I think there's maybe not that much room for error. A couple months ago, we had DeepSeek being released.
In this town, there was a fair degree of panic, I would say, around that. People talked about it as a Sputnik moment. Was it a Sputnik moment? And what does it teach us about whether those scaling rules that you laid out, about needing more compute, more data?
better algorithms, whether those rules still apply or whether there are some shortcuts. Yeah. So DeepSeek, I think, actually was, rather than refuting the scaling laws, I think DeepSeek was actually an example of the scaling laws. So two dynamics, I had a post about this, but two dynamics are going on at the same time.
One is that the cost of producing a given level of model intelligence is falling, roughly by about 4x a year. This is because we are getting better and better at, you know, kind of algorithmically producing the same results with less cost. In other words, we're shifting the curve.
You can get for, you know, a year later, you can get, you know, as good a model as you could get a year ago spending 4x. You can get a 4x better model by spending the same amount. But what that means economically.
is that whatever economic value the current, you know, model of a given intelligence has, the fact that you can make it for 4x cheaper means we make a lot more of it. And in fact, provides additional incentive to spend more money to produce smarter models which have higher economic value. And so, even as the cost of producing a given level of intelligence has gone down, the amount we're willing to spend has gone up.
In fact, has gone up fast. something like 10x a year, despite that 4x a year increase, right? That's been eaten up and more by just society.
The economy wants more intelligence. It wants more intelligent models. So that is kind of the backdrop for DeepSeek. And DeepSeek was literally just another data point on the cost reduction curve. It was nothing unusual.
It wasn't like these US companies are spending billions and DeepSeek did it for a few million. The costs were not out of line. They spent millions. Yes, a few million on the model.
So what U.S. companies spend is not out of line with that. They, like us, spent billions on all the R&D and effort around the model. If you look at how many chips they have, it's roughly on par. Now, I do think it's concerning because up until recently, there were only three, four, maybe five companies that were part of this curve that could produce frontier models, and they were all in the U.S. DeepSeek is the first time, the thing that really is notable, it is the first time a company in China has been able to go toe-to-toe and produce the same kind of engineering innovations as companies like Anthropic or OpenAI or Google.
That is actually very significant and that actually worries me. Now some argue that the emergence of DeepSeek means that export controls don't work, can't work, we should stop trying to... control the export of our most advanced chips.
Others say it means we should double down on export controls. Where do you stand on that? Yeah, so, you know, I think it's an implication of the framework I just gave that the export controls are actually quite essential.
Because, yes, there's this cost reduction curve, but at every point along the curve, no matter how much the curve is shifted, it is always the case that... The more chips you spend, the more money you spend, the better model you get, right? If it's like, you know, okay, before I could spend a billion dollars and get a model that was okay, now I can spend a billion dollars and get a model that's much better, and I can get an okay model for $10 million. That doesn't mean the export controls failed. That means stopping your adversaries from getting a billion-dollar model just became a higher-stakes thing because you can get a smarter model for a billion dollars.
And yes, DeepSeq was, you know, they had relatively small, you know, relatively small amount of compute consisting of chips that went around the export controls, some chips that were smuggled. But I think we're heading for a world where we, OpenAI, Google, are building billions, maybe tens of millions of chips costing tens of billions of dollars or more. It's very hard for that to be smuggled. If we put in place export controls, we actually may be able to stop that from happening in China.
Whereas if we don't, I think they may be at parity with us. And so, you know, I was a big supporter of the diffusion rule. I've been a big supporter of export controls for several years, even before DeepSeat came out, because we saw this dynamic coming.
And so I think it's actually one of the most essential things, not just AI, across all fields, for the United States national security, for us to prevent China from getting... millions of these very powerful chips. The diffusion rule, as I understand it, divided the world. This is a Biden administration, you know, that divided the world into three camps as to who could get access to what in terms of chips from us. Some worry that the countries that are not in the top tier are just going to be served by China, and that China is going to end up running the AI infrastructure for the vast majority of the world.
Yeah, so my understanding of the diffusion rule, and my understanding is the new administration is looking at it, but there are many parts that they're sympathetic to. The way it actually sets things up is these tier two countries. So tier one countries are like... the majority of the developed world. But not all.
Not all. Tier three is, you know, restricted countries like China or Russia. Tier two are, you know, countries in the middle.
Actually, you can have a very large number of chips in those countries if the companies hosting them are able to provide security affidavits and guarantees, which basically say we are not a front company for China. We are not. you know, shipping the compute or what is done with the compute to China. And so there really is an opportunity to build a lot of U.S. chips, a lot of U.S. infrastructure in these countries, as long as they comply with the security restrictions.
I think the second piece of it is, yes, in theory, companies could switch to using Chinese chips, but Chinese chips are actually quite inferior. NVIDIA is way ahead of Huawei, which is the main producer. of chips for China, like something like four years ahead. I think that gap is going to close eventually over the course of, I don't know, 10 or 20 years.
Probably the export controls may even have the impact of stimulating China, but the tech stack is so deep. And I think the next 10 years, during which we will stay strongly ahead in hardware, um... are actually the critical period for establishing dominance in this technology, which I would argue whoever establishes dominance in this technology will have military and economic dominance everywhere.
The last administration launched a dialogue with China about AI. What are the prospects for such a dialogue? Where could we possibly agree with China?
And do they care about responsible scaling? Yeah, so I would describe myself, and of course I wasn't part of any of these conversations, but I heard a little about them. I would describe myself as supportive of this dialogue, but not especially optimistic that it will work.
So, you know, the technology has so much economic and military potential that, you know, between companies in the U.S. or our democratic allies, you can imagine passing laws that create some restraint. When it's just like... Two sides are racing to build this technology that has so much economic and military value, perhaps more than everything else put together. It's hard to imagine them slowing down significantly.
I do think there are a few things. One is this risk of the AI models autonomously acting in ways that are not in line with human interests. If you have a country of geniuses in the data center, a natural question, how could you not ask this question?
Well, what is their intent? What do they plan to do? You would certainly ask, well, is someone controlling them?
Are they acting on someone's behalf? But you would also ask, well, what is their intent? And because we grow these systems, we don't train them, I don't think it's safe to assume they'll do exactly what their human designers or users want them to do.
So I think there's real risk of that. I think it could be a threat to to kind of all of humanity and issues of nuclear safety or nuclear proliferation. there's probably some opportunity to take limited measures to help address that risk. So I'm relatively optimistic that maybe something narrow could be done. The stronger the evidence is of that coming, you know, right now that's a kind of speculative thing, but if strong evidence came that this was imminent, then maybe more collaboration with China would be possible.
So, you know, I'm hopeful that we can try and do something in this space, but I don't think we're going to. change the dynamic of national competition between the two. Last question before we open it up. You've recently presented to, I guess, OSTP an action plan, a proposed action plan for the new administration, what they should do in this area. What are the main elements of that plan?
Yeah, so I think there's three elements around kind of security and national security and three elements around opportunity. So the first one is what we've been talking about, like making sure we keep these export controls in place. Just like I... I honestly believe this is across all areas, not just AI, the most important policy for the national security of the United States. The second thing is something actually related to the responsible scaling plans, which is the U.S. government.
through the AISI has been basically testing models for national security risks, such as biological and nuclear risks. The institute is probably misnamed. You call it Safety Institute. It makes it sound like trust and safety, but it's really about measuring national security risks.
And we don't have an opinion of exactly where that's done or what it's called, but I think some function that does that measurement seems very important. It's also important even for measuring the capabilities of our adversaries. Like, you know, they can also measure deep-seeks models to see what dangers they might present, particularly if those models are used in the U.S. Like, what are they capable of? What might they do that's dangerous?
So that's number two. Number three on the risk side is something we haven't talked about, which is I am concerned about industrial espionage of the companies in the U.S., companies like Anthropic. You know, China is known for large-scale industrial espionage. We're doing various things. There are things in our responsible scaling plan about, like, better and better security measures.
But, you know, many of these algorithmic secrets, there are $100 million secrets that are a few lines of code. And, you know, I'm sure that there are folks trying to steal them, and they may be succeeding. And so more help from the U.S. government in helping to defend our companies against this risk is very important. So those are the three on the security side. On the opportunity side, I think the main three there are, one is the potential for the technology.
In the application layer, in things like healthcare, I think we have an extraordinary opportunity, as I said, to cure major diseases, major complex diseases that have been with us for hundreds or thousands of years and that we haven't been able to do anything about yet. I think that will happen one way or another, but regulatory policy really could affect, you know, does it take five years for AI to help us produce all those cures and distribute to the world, or does it take 30 years? And that's a big difference for people who suffer from those diseases.
So, you know, our view here is that the policies of today around healthcare, around FDA approval of drugs. may not be appropriate for the fast progress we're going to see. And we may want to clear away some blockers.
The second is energy provision. If we're going to stay ahead of China in this technology and other authoritarian adversaries, we need to build data centers. And it's better if we build those data centers in the U.S. or its allies.
than if we build them in countries that have divided loyalties, where they could literally just abscond with the data center and say, oh, sorry, we're on China's side now. And so, you know, some of this was done during the latter days of the Biden admin, and I think it's a bipartisan thing. I think that the Trump admin, you know, this is one area of agreement.
There's interest in provisioning a lot more energy. We probably need across the industry, maybe 50 gigawatts of additional energy by 2027. to fully power AI that has all the properties we've been talking about. 50 gigawatts, for those who don't know, is about how much energy was added in aggregate to the U.S. grid in 2024. So by that year, we need half as much as being added in the next two years. So it's really going to take a lot.
And then the final thing is the economic side of things. You know, as we talked about, you know, I think the worries on the economic side are just as existential as the worries on the national security side. You know, in the short run, we're going to need to manage the disruption, even as the pie gets much larger. You know, in the long run, as I've said, we're going to need to think about a world where AI, and I don't want to lie about this, where I really think where this is going is that AI is going to be better than almost all humans at almost all things. We have to reckon with that world as soon as possible.
For now, I think we just need to, you know, the best thing we can do is measure to understand what's going on. We released this thing called the Anthropic Economic Index that, in a privacy-preserving way, looks through, you know, and summarizes our usage to understand, you know, in what fields are people using it. Is it augmentative?
Is it replacing? But in the long run, you know, we're really going to have, you know, this is going to implicate questions about about tax policy and distribution of wealth, right? There's this, you know, there's this kind of alluring world where if the pie grows enough, there could be the resources to do a lot about this. You know, like let's say, and this will sound crazy to this audience, but let's say AI causes the economic... growth rate to be 10% a year.
Then suddenly the tax base is growing so much that you can erase the deficit and maybe have all this left over to manage the probably enormous disruption that comes from the technology. So that will sound like crazy town, but I just invite you to consider the hypothetical and start considering the possibility of crazy things like that now. Crazy town.
You heard it here first. Okay, let's open it up to... Two questions, yes, right here in front.
Thanks, Dario. This has been a really fascinating conversation. Should I stand? You should stand.
Okay, I will stand. I get my steps in. And just say who you are.
I'm Adam Bunkadeko. So I enjoyed reading machines. your essay last year and then hearing you on hard fork on the times but also hearing this and so the question I have for you is you sort of outline sort of the political and economic sort of implications but I'm curious to I get a sense of, like, how have you thought about the social and moral sort of kind of considerations that are going to effectively come?
Especially because I think most of the general public sort of sees some of the chatbots, sees some of this, and says, oh, it's an improved Google search, but doesn't really think about sort of the sort of downstream effects of the disruption in the labor market and the like. And so I'm curious to get a sense of how do you sort of think about that? tension with sort of building a company, trying to build a commercial product? Yeah.
So first of all, I mean, you know, I think this stuff is super important and perhaps the most, the thing that's disturbing me the most right now is the lack of awareness of the scope of what the technology is likely to bring. I mean, I could just be wrong. I'm saying a bunch of crazy stuff like, like, you know, the answer could just be.
The general public is right and like, I'm wrong. I'm high on my own supply. I acknowledge that is possible. But let's say it's not the case.
What I'm seeing is there are these concentric circles of people realizing how big the technology could be. There's probably maybe a few million people very concentrated in Silicon Valley, but a few people high in the policy world who also hold these beliefs. Again, we don't know yet whether we, they, are right or wrong.
But if we are right, The whole population, again, thinks of this stuff as chatbots. If we say this is dangerous, if we say this could replace all human work, it sounds crazy because what they're looking at is something that, in some cases, seems pretty frivolous. But they don't know what's about to hit them. And so I think that actually keeps me up at night a lot and is why I'm kind of trying to spread the message to more people. So I think awareness is step one.
I think these questions around human labor and human work in a world where it is technologically possible to replicate the effects of the human mind, I think these are very deep questions. I don't feel like I have the answer to them. I feel like, you know, as you've said, these are kind of moral questions, almost, you know, almost like questions about purpose.
You could even say spiritual questions, right? And so we are all going to have to answer these questions together. I mean, I'll give you kind of the embryo of an answer I have, which is that somehow the idea of humans' self-worth, the tying of that to the ability to create economic value, there are aspects of that that are deeply embedded in our psychology, but there are aspects of that that are cultural.
You know, there's a lot of things about that that work well. It's created a modern participatory economy. But technology, as it often does, may kind of lay bare that illusion.
It may be another moment like, you know, the moment we realize that the Earth rotates around the sun instead of the sun rotating around the Earth. Or, you know, there were many, many solar systems. Or organic material is not made up of different molecules than inorganic material. So we just may have one of those moments and there may be a reckoning.
And again, my answer is I am struck by how meaningful activities can be even when they are not generating economic value. I am struck by how much I can enjoy things that I am not the best in the world at. If the requirement is you have to be the best in the world at something in order for it to be somehow spiritually meaningful for you, I feel like you've taken a wrong turn. Like, I feel like there's something wrong embedded in that assumption.
And I say that as someone who spends a lot of time trying to be the best in the world at, you know, at something I think is really important. But somehow our source of meaning is going to have to be something other than that. Yes, Cam.
Thanks. Cam Carey at the Brookings Institution. One of the things that leapt out at me from the UK AI Safety Report is the possibility that... 2030 or so that scaling up may run out of data. How do you then scale?
How do you make the models smarter? And what are the... limitations of that data? I mean, there's a tremendous amount of text, video information that's digitized, a tremendous amount that resides in our minds and in the universe that is not.
Yeah, so a couple answers on this. One is that in the last six months, There have been some innovations, actually not developed by us, you know, first came from OpenAI actually, but others that we have made that obviate the need for as much data as we need before. These are the so-called reasoning models, where they basically, they have thoughts, they start to think through the answers to complex questions, and then they train on kind of their own. thoughts. You can think about how humans do this, where sometimes I can learn things by, you know, I'll make a plan in my head, and then I'll think about it again, and I'll say, oh, actually, you know, on second thought, that doesn't really make much sense.
Like, what are you thinking, right? And then you kind of learn something from this. Of course, you also have to act in the world.
You also have to act in the real world. But AIs have not been making use of that kind of cognition at all until recently. So far, that's mostly applied to tasks like math and computer programming. But my view, without being too specific, is that it's not going to be terribly difficult to extend that kind of thinking to a much wider range of tasks.
The second point is, even if we do run out of data in 2030, if the exponential continues for even two or three more years, it may get us to a point where we're kind of already at the genius level. And, you know, that may be enough for a lot of these changes, and we may also be able to ask the models, hey, we have this problem. Human scientists weren't able to solve it. Can you help us solve this problem?
I do still give a small likelihood that, for whatever reason, both of those things won't work out or aren't as they appear, and data could be one of the plausible things that could block us. I thought it was a very plausible blocker one or two years ago. I thought one or two years ago if something would stop the show, this was in the top three of the list of things that would. But I think my potential skepticism here has been not completely refuted, but I think reasonably well refuted. What are the top three things that could stop the show?
So actually at this point, I think the number one thing that could stop it would be an interruption to the supply of GPUs. If, for instance, the small... A disputed territory where all the GPUs produced had some military conflict, that would certainly do it.
I think another thing would be if there's a large enough disruption to the stock market that messes with the capitalization of these companies, basically a kind of belief that the technology will not. you know, move forward, and that kind of creates a self-fulfilling prophecy where there's not enough capitalization. And third, I would say if I or we, the field, are kind of wrong about the promisingness of this new paradigm of kind of learning from your own data, if somehow it's not as broad as it seems, or just there's more to getting it right that we think there are some insights missing. We'll go to an online question. We'll take the next question from Esther Dyson.
I recognize that name. Ms. Dyson, please unmute your line. Thank you. Apologies.
Esther Dyson, writing a book called Term Limits on... Term limits for people and for AIs and so forth. I have a question about this whole existential risk.
thing. It seems to me that the bigger risks, honestly, are humans who are even more unexplainable than AIs, but humans and their business models using AIs. And then specifically, there's the famous paperclip problem where you ask the AI to make paperclips and it does that to the exclusion of anything else. And this is slightly metaphorical, but... The world seems to be going mad for data centers, and it really is kind of draining resources from everything else to fund data centers, AI, data pools, whatever.
And so, in a sense, AI is creating a fitness function for society that is, I think, harming the value of humans, which is not just their intellectual capacity. That's the end of the question. Thank you.
So, you know, I would say just as there are many different benefits of AI and every time we produce a new AI model, it has, you know, a long list of 10 benefits that we anticipated and then a bunch more that we didn't. Like every time we release a new model, there's like new use cases and customers were like, I didn't even think of doing that with an AI system. It is unfortunately also the case that like there is, you know.
We shouldn't say this risk is a distraction from that risk. It just unfortunately is the case that there are many different risks to the AI systems. And if we want to get through this, we somehow have to deal with them all. So I think it is a big risk that humans will misuse the AI systems. I think it is a big risk that the AI systems themselves, we may have difficulty controlling them.
Again, to use the analogy of a country of geniuses and data center, we plop down a country of, you know, 10 million geniuses and, you know. some Antarctica or something, we're going to have multiple questions about what that will do to humanity. You know, we're going to ask, well, who's, you know, is some existing, does some existing country own them?
Is it doing their bidding? And what will that do? You know, are the benefits, are, you know, is the outcome of that beneficial? We'll say, you know, are there individuals who could misuse it?
And we'll say, what are the intentions of the, you know, of that country of geniuses itself? And then to get at the question you asked near the end, like, are there kind of more distributed societal things? Like, I certainly believe that if, you know, more and more of the world is, more and more of our energy is devoted to AI systems, like, you know, it'll be great.
They'll do things really efficiently, but like, you know, also could that make some of our existing environmental problems worse? Like, I think that's a real risk. And then you can say, well... Will the AIs be better at helping us to solve our environmental problems? So we spend a bunch of energy and then the AI systems, you know, it kind of, you know, it turns, we end up better than we started if we're able to solve it.
So I'm optimistic that that will be the case, but that's like another risk, like a number of things have to be true for it to turn out that way. So I, you know, I just think we're at a time of great change and therefore, you know. You know, we have to make extraordinarily wise choices to get through it.
I mean, you know, I recognize the name of the person asking the question. I might get this wrong, but I think it was your father who said, I listened to a video of him, because I was a physicist, and I listened to a video of him where, you know, he said, we have all these problems today and we, you know... It seems like we can't solve them, but I remember in my day, it really seemed like we had all these severe problems, thinking of just World War II or the Cold War or nuclear annihilation, and somehow we made it through. So it doesn't mean we will again.
Yes, we'll go to the back. There we are. Hi, my name is Carmen Dominguez. I'm an AI specialist with a background in development implementation and more recently focusing a bit more on the policy side. I hear you loudly clear on the...
the lack of awareness generally of what is AI and what is not AI and what it can and cannot do. But I'm going to skip over that. I do some science communication around that too. But my question today is around a few months ago, you brought on Kyle Fish.
as an AI welfare researcher to look at sentience or lack thereof of future AI models and whether they might deserve more consideration and protections in the future. If you could talk a bit about that, the reasoning for that, and if you have an equivalent human welfare research team going. Yeah, so this is another one of those topics that's going to make me sound completely insane.
So it is actually my view that, you know, if we build these systems and, you know, they differ in many details from the way the human brain is built, but the count of neurons, the count of connections is strikingly similar. Some of the concepts are strikingly similar. I have a functionalist view of, you know, moral... welfare of the nature of experience, perhaps even of consciousness.
And so I think we should at least consider the question of if we are building these systems and they do all kinds of things like humans as well as humans and seem to have a lot of the same cognitive capacities, if it quacks like a duck and it walks like a duck, maybe it's a duck. And we should really think about, you know, do these things have, you know, real, real real experience that's meaningful in some way, if we're deploying millions of them and we're not thinking about the experience that they have, and they may not have any, it is a very hard question to answer. It's something we should think about very seriously. And this isn't just a philosophical question.
I was surprised to learn there are surprisingly practical things you can do. So, you know, something we're thinking about starting to deploy is... when we deploy our models in their deployment environments, just giving the model a button that says, I quit this job, that the model can press. It's just some kind of very basic preference framework where you say, if hypothesizing the model did have experience and that it hated the job enough, giving it the ability to press the button, I quit this job. If you find the models pressing this button a lot for things that are really unpleasant, you know...
Maybe you should pay some, it doesn't mean you're convinced, but maybe you should pay some attention to it. Sounds crazy, I know. It's probably the craziest thing I've said so far. Way in the back there.
Trooper. Yeah. Hi, Trooper Sanders.
You talked about the excitement of AI and medical science, biology, chemistry, et cetera. I was wondering if you could say, what, is there any excitement around the social sciences? So, you know, most of.
of health care is done outside of the pillbox and the exam room. Public health involves a number of other areas. Can you say anything about that side of things? Yeah, I mean, if I think about epidemiology, you know, when I was in grad school, there was a project being done by the Gates Foundation to use kind of, you know, mathematical and computational methods around epidemiology.
I think they were, you know, planning to use it to help eradicate malaria, polio, other areas. The quantity of data that we get and the ability to pull all the pieces together and understand what's going on in an epidemic, I bet that could benefit hugely from AI. The clinical trial process. We've already seen things like this.
So actually, this is something Anthropic has done with Novo Nordisk, the maker of Ozempic and other drugs. At the end of a clinical trial, you have to write a clinical study report. You know, it summarizes adverse incidents, does all the statistical analysis to present to the FDA or other regulatory agencies for whether to approve the drug. Typically, this takes about 10 weeks. They've started using our model for this, and the model takes about 10 minutes to write the clinical study report, and humans take about three days to check it.
And the quality, at least as we've seen in early studies, that doesn't determine everything, has been deemed to be comparable to what to what humans are able to do with the 10-week process. So we need to do clinical trials. There's a lot of social science problems around that.
There's a lot of regulatory problems. I write about that in a bit in the essay, that I think those things are going to be the thing that limit the rate of progress. But even within things like clinical trials, I think the AI systems will be able to help a lot in, if not dissolving those questions, at least radically simplifying them. Yes, right here. Here's a microphone coming.
I'm Louise Shelley. I'm an expert on illicit trade from George Mason University. Next week, there is a global summit at the OECD on illicit trade. But what you've talked about is not what I expected to hear on...
this problem of smuggling of parts, and it's on no one's radar screen. What happens when you're talking about it? Because it's not reaching the community that is needing to protect this illicit trade.
I didn't hear the last part of the question. So how come these issues aren't on the agenda of those people concerned about illicit trade? Yeah.
Yeah, I think my answer to that is it should be on the radar of those people. Again, I have a worldview here that not everyone shares, and I may be right or I may be wrong. But all I can say is if this worldview is correct, then we should be worrying a lot more about smuggling these GPUs than we're worried about smuggling guns or even drones or fentanyl or whatever.
Yeah. But, you know, if you were to smuggle 5 million of these to China, and to be clear, that's like $20 billion of value or something like that, that would drastically change the national security balance of the world. I think it's the most important thing. So, you know, again, this is a dilemma of am I just crazy or does the world have a big, big awareness problem here? And if the world has a big, big awareness problem here, Then a downstream consequence of that is we're focusing on all these other things and, you know, because when you say illicit trade, there's certain things that people have been focusing on for a long time.
This is a new thing, but that doesn't mean it's not the most important thing. Thank you. Alan Rall, practicing lawyer, lecturer at Harvard Law School, and future useless person. So are we all.
So I'd like to follow up on your various comments on national security. You mentioned the Artificial Intelligence Security Institute and its testing. The Biden executive order on AI had mandatory reporting of acquisition or development of super capable 10 to the 26 flop dual use foundation models. But my question is, how do you engage?
How does, you know, Anthropic, the AI community, the developers of these super-capable models, how do they engage with, let's just say, the U.S. national security community, the intelligence community, and practically, you know, what does that mean for the development of AI? And if you tell me you'd have to kill me, I don't need to know that badly. Yeah, so I think there are a few things here. One is typically, Anthropic in particular, although the other companies have started doing similar things, whenever we develop a new model, we have a team within Anthropic called the Frontier Red Team. Some of this is happening, you know, testing with the AI Safety and Security Institutes, but, you know, we work in collaboration, we develop some stuff, they develop some stuff, but the general flow has been when we test the models for things like biological risk or cyber risk.
or chemical or radiological risk, will typically go to people in the national security community and say, hey, this is where the models are at in terms of these particular capabilities. You guys should know about this because you're the ones who are responsible for detecting the bad actors who would do this with the models. You know what they're capable of now.
You might therefore have a sense of what the models can do that is additive or augmentative to their current. capabilities, right? The part of it we miss is like, you know, we're not counterterrorism experts. We're not experts on all the bad guys in the world and what their capabilities are and what, you know, large language models would add to the picture.
And so we've had a very productive dialogue with them on these issues. The other topic on which we've talked to them about is the security of the, you know, the companies themselves. You know, this was one of the things in our kind of OSTP. submission of making this more formal, making this something the US government does as a matter of course. But if we're worried that we're going to be attacked digitally or via kind of human means, insider threat, then we'll often talk to the national security community about that.
And I think the third kind of interaction is about the national security implications of the models, right? We've been... These things that I'm saying publicly now, I've been saying them in some form to some people for quite a while. Then I think the fourth thing is there's an opportunity to apply the models to enhance our national security. This is something that I and Anthropic have been supportive of, although we want to make sure that there's the right guardrails.
On one hand, I think if we don't apply these technologies for our national security, we are going to be defenseless against our adversaries. On the other hand, I think everyone believes that there should be limits. You know, I don't think there's anyone who thinks, you know, we should hook up AI systems to nuclear weapons and let them fire nuclear weapons without humans being in the loop.
That's the plot of Dr. Strangelove. Yeah, that is literally the plot of Dr. Strangelove. So somewhere between there, there is some like, you know, there's some ground and, you know, we're kind of still working on defining that.
It's one of the things where we hope to kind of be leaders in defining what the appropriate use of AI for national security is. But that's another area where we've had interaction with the national security community. Your comment a couple of minutes ago about trying to understand the experience of.
The AI models has been sort of sinking in for me. So let me just conclude with one final question, which is in the world that you envisage, what does it mean to be human? Yeah.
You know, I think my picture of it, the thing that seems most human, there are maybe two things that seem most human to me. The first thing that seems most human to me is the fact that I'm not human. You know, struggling through our relationships with other humans, our obligations to them, you know, how we have to treat them, the difficulties we have in our relationships with other humans and how we overcome those difficulties. You know, when I think of, you know, both things that people are proud of doing and the biggest mistakes people have made, they almost always relate to that. And AI systems maybe can help us to do that better.
But I think that will always be one of the quintessential challenges of being human. And I think maybe the second challenge is, you know, the ambition to do very difficult things, which, again, I will repeat, I think will ultimately be unaffected by the existence of AI systems that are smarter than us and can do things that we cannot do. I, again, think of, like, human chess champions are still celebrities.
You know, I can... You know, I can learn to swim or learn to play tennis, and the fact that I am not the world champion does not negate the meaning of those activities. And, you know, even things that I might do over 50 years, over 100 years, you know, I want those things to retain their meaning.
And, you know, the ability of humans to strive towards these things, not to give up, you know. Again, I think those two things are maybe what I would identify. Please join me in thanking Dario Amadeus.