Understanding and Building Effective AI Agents

I feel like agents for consumers are like fairly overhyped right now. Okay, here we go, hot take. Trying to have a agent like fully book a vacation for you, almost just as hard as just going and booking it yourself. Today we're going behind the scenes on one of our recent blog posts, Building Effective Agents. I'm Alex, I lead Cloud Relations here at Anthropic. I'm Eric, I'm on the research team at Anthropic. I'm Barry, I'm on the Applied AI team. I'm going to kick us off here. For viewers just jumping in, what's a quick version of... what an agent actually is. I mean, there's a million definitions of it. And why should a developer or somebody that's actually building with AI care about these things? Eric, maybe we can start with you. Sure. Yeah, so I think something we explored in the blog post is that, first of all, a lot of people have been saying everything is an agent, referring to almost anything more than just a single LLM call. One of the things we tried to do in the blog post is really kind of separate this out of like, hey, there's workflows, which is where you have a few LLM calls chained together. And really what we think an agent is, is where you're letting the LLM decide sort of how many times to run. You're having it continuing to loop until it's found a resolution. And that could be, you know, talking to a customer for customer support. That could be iterating on code changes. But something where like you don't know how many steps it's going to take to complete, that's really sort of what we consider an agent. Interesting. So in the definition of an agent, we are letting the LLM kind of pick its own fate and decide what it wants to do, what actions to take, instead of us predefining a path for it. Exactly. It's more autonomous, whereas a workflow, you can kind of think of it as like, you know, yeah, a workflow or sort of like it's on rails through a fixed number of steps. I see. So this distinction, I assume this was the result of many, many conversations with customers and working with different teams and even trying things ourself. Barry, can you speak? more to maybe what that looks like as we got to create this divide between a workflow and agent and what sort of patterns surprised you the most as you were going through this? Sure. Honestly, I think all of this kind of evolved as like model got better and like teams got more sophisticated. We both worked with a large number of like customers were very sophisticated and we kind of went from like having a single LLM to having a lot of LLMs and like eventually having like LLMs orchestrating themselves. So, you know, like one of the reasons why we decided to create this. distinction is because we started to see these two distinct patterns where you have workflows that's pre-orchestrated by code. And then you also have agent, which is a simpler but complex in other sense, like different shape that we're starting to see. Really, I think as the models and all of the tools start to get better, agents are becoming more and more prevalent and more and more capable. And that's when we decided, hey, this is probably a good time for us to give a formal definition. So in practice, if you're a developer implementing one of these things, what would that actually look like in your code as you're starting to build this? Like the differences between, like maybe we actually go down to like the prompt level here. What does an agent prompt to look like or flow? And what does like a workflow look like? Yeah, so I think a workflow prompt looks like you have one prompt, you take the output of it, you feed it into prompt B, take the output of that, feed it into prompt C, and then you're done. Kind of there's this straight line, fixed number of steps, you know exactly what's going to happen. And maybe you have some extra code that sort of checks the intermediate results of these and make sure they're okay. But you kind of know exactly what's going to happen in one of these paths. And each of those prompts is sort of a very specific prompt, just sort of taking one input and transforming it into another output. For instance, maybe one of these prompts is taking in the user question. and categorizing it into one of five categories so that then the next prompt can be more specific for that. Okay. In contrast, an agent prompt will be sort of much more open-ended and usually give the model tools or multiple things to check and say, hey, here's the question, and you can do web searches or you can edit these code files or run code and keep doing this until you have the answer. I see. So there's a few different use cases there. That makes sense as we start to arrive at these different conclusions. I'm curious, as we've now kind of covered at a high level how we're thinking about these workflows and agents and talking about the blog post, I want to dive even further behind the scenes. Were there any funny stories, Barry, of wild things that you saw from customers that were interesting or are just kind of far out there in terms of how people are starting to actually use these things in production? Yeah, this is actually from my own experience. building agents. I joined about a month before the Sonnet V2 refresh, and one of my onboarding tasks was to run OS World, which was a computer use benchmark. And for a whole week, me and this other engineer were just staring at these agent trajectories that were counterintuitive to us. And then, we weren't sure why the model was making the decision it was, given the instructions that we would give it. So we decided we're going to act like cloud and put ourselves in that environment. So we would do this really silly thing where we close our eyes for a whole minute, and then we blink at the screen for a second, and we close our eyes again and just think, well, I have to write Python code to operate in this environment. What would I do? And suddenly it made a lot more sense. And I feel like a lot of agent design comes down to that. There's a lot of context and a lot of knowledge that the model maybe does not have. And we have to be empathetic to the model. And we have to make a lot of that clear in the prompt, in the to description, and in the environment. I see. So a tip here for developers is almost like to act as if you are. Looking through the lens of the model itself in terms of like, okay, what would be the most applicable instructions here? How is the model like seeing the world, which is very different than how like we operate as a human, I guess, with additional context. Eric, I'm curious if you have any other stories that you've seen. Yeah, I think actually my like in a very similar vein, I think a lot of people really forget to do this. And I think maybe the funniest things I see is that people will... put a lot of effort into creating these really beautiful detailed prompts. And then the tools that they make to give the model are sort of these incredibly bare bones, like, you know, no documentation function, like the parameters are named A and B. Yeah. And it's kind of like, oh, like an engineer wouldn't be able to like, you know, work with this as a, you know, work with this as if this was a function they had to use, because there's no documentation. Like how can Claude expect, how can you expect Claude to use this as well? So kind of it's like that lack of like putting yourself in the model shoes. And I think a lot of people, when they start trying to use tool use and function calling, they kind of forget that they have to prompt as well. And they think about the model just as this, as a more classical programming system. But it is still a model. And you need to be prompt engineering in the descriptions of your tools themselves. Yeah, I've noticed that. It's like people forget that it's all part of the same prompt. Like it's all getting fed into the same. prompt in the context window and writing a good tool description influences other parts of the prompt as well so that is like one aspect to consider agents is this kind of all the hype term right now a lot of people are talking about it and there's been plenty of articles written and videos made on the subject what made you guys think that now is the right time to write something ourselves and talk a little bit more about like the details of agents you Sure, yeah. I think one of the most important things for us is to be able to explain things well. I think that's a big part of our motivation, which is we walk into customer meetings and everything is referred to as a different term, even though they share the same shape. So we thought it would be really useful if we can just have a set of definitions and a set of diagrams and code to explain these things to our customers. And we are getting to the point where the model is capable of doing a lot of the the agentic workflows that we're seeing. And that seems like the right time for us to have some definitions or just to make these conversations easier. I think for me, I saw that there was a lot of excitement around agents, but also a lot of people really didn't know what it meant in practice. And so they were trying to bring agents to sort of any problem they had, even when much simpler systems would work. And so I saw that as like one of the reasons that we should write this is like... guide people about how to do agents, but also where agents are appropriate. And that you shouldn't go after a fly with a bazooka. I see. I see. That was a perfect parlay into my next question here. There's a lot of talk about the potential of agents. And every developer out there and every startup and business is trying to think about how they can build their own version of an agent for their company or product. But you guys are starting to see what actually works in production. So we're going to play a little game here. I want to know one thing that's overhyped about agents right now and also one thing that's underhyped, just in terms of implementations or actual uses in production or like potentials here as well. So Eric, let's start with you first. I feel like underhyped is like things that save people time, even if it's a very small amount of time. I think a lot of times if you just look at that on the surface, it's like, oh, this is something that takes me a minute. And even if you can fully automate it, it's only a minute. Like what help is that? But really... That changes the dynamics of now you can do that thing a hundred times more than you previously would. So I think I'm most excited about things that, if they were easier, could be really scaled up. Yeah, I don't know if this is necessarily related to hype, but I think it's really difficult to calibrate right now where agents are really needed. I think there's this intersection that's a sweet spot for using agent, and that's a set of tasks that's valuable and complex. but also like maybe the cost of error or cost of monitoring error is relatively low. That set of tasks is like not not super clear and obvious unless we actually look into the existing processes. I think coding and search are two pretty canonical examples where agents are very useful. Take search as an example. It's a really valuable task. It's very hard to do deep iterative search, but you can always trade off some precision for recall and then just get a little bit more documents or a little bit more information than is needed and filter it down. So we've seen a lot of success there with agent-taked search. What does a coding agent look like right now? Coding agents, I think, are super exciting because they're verifiable, at least partially. Code has this great property that you can write tests for it, and then you edit the code, and either the tests pass or they don't pass. Now that assumes that you have good unit tests, which I think every engineer in the world can say, like, we don't. But at least it's better than a lot of things. There's no equivalent way to do that for many other fields. So this at least gives a coding agent some way that it can get more signal every time it goes through a loop. So if every time it's running the tests again, it's seeing what the error of the output is, that makes me think that the model can kind of converge on the right answer by getting this feedback. And if you don't have some mechanism to get feedback as you're iterating, you're not injecting any more signal. You're just going to have noise. And so there's no reason without something like this that an agent will converge to the right answer. I see. So what's the biggest blockers then in terms of improving agentic performance on coding at the moment? Yeah. So I think for coding, we've seen over the last year on Sweebench, results have gone really from very, very low to I think over 50% now, which is really incredible. So the models are getting really good at writing. code to solve these issues. I feel like I have a slightly controversial take here that I think the next limiting factor is going to come back to that verification. It's great for these cases where we do have perfect unit tests and that's starting to work. But for the real world cases, we usually don't have. perfect unit tests for them. And so that's what I'm thinking now, like finding ways that we can verify and we can add tests for the things that you really care about so that the model itself can test this and know whether it's right or wrong before it goes back to the human. I see. So making sure that we can embed some sort of feedback loop into the processes itself, the right or wrong. Okay. What's the future of agents look like in 2025? Barry, we're going to start with you. Yeah, I think that's a really difficult question. This is probably not a practical thing, but one thing I've been really interested in is how... a multi-agent environment will look like. I think I've already shown Eric this. I've built an environment where a bunch of cloud can spin up other clouds and play werewolf together. And it's like a completely different... What is werewolf? Werewolf is a social deduction game where all of the players are trying to figure out what each other's role is. It's very similar to Mafia. It's entirely text-based, which is great for cloud to play in. I see. So we have clods... multiple different clods playing different roles within this game all communicating with each other. Yeah, exactly. And then you see a lot of interesting interaction in there that you just haven't seen before. And that's something I'm really excited about. It's very similar to how we went from single LLM to multi-LLM. I think by the end of the year we could potentially see us going from agent to multi-agent. And there are some interesting research questions to figure out in that domain. In terms of how the agents interact with each other, what is this kind of... emergent behavior look like in that front as you coordinate between agents doing different things? Exactly. And just like whether this is actually like going to be useful or better than like a single agent with access to a lot more resources. Do we see any multi-agent approaches right now that are actually working in production? I feel like in production we haven't even seen a lot of like successful single agents. Okay, interesting. But like you know this is kind of like a potential extension of like successful agents with the I guess like improved capabilities of the next couple of generations of models. Yeah, so this is not advice that everyone should go explore the multi-agent environment. It's just, I think, to understand the model's behavior. This provides us with a better way to understand model behaviors. I see. Okay, Eric, what's the future of agents in 2025? I feel like in 2025, we're going to see a lot of business adoption of agents starting to automate a lot of repetitive tasks and really scale up a lot of things that People wanted to do more before but were too expensive. You could now have 10x or 100x how much you do of these things. I'm imagining things like every single pull request in triggers a coding agent to come and update all of your documentation. Things like that would be cost prohibitive to do before. But once you think of agents as sort of almost free, you can start doing these, adding these bells and whistles everywhere. I think maybe something that's not going to happen yet, going back to what's overhyped. I feel like agents for consumers are fairly overhyped right now. Okay, here we go. Hot take. Because I think that we talked about verifiability. I think that for a lot of consumer tasks... It's almost as much work to sort of fully specify your preferences and what the task is as to just do it yourself. And it's very expensive to verify. So like trying to have a agent like fully book a vacation for you, describing exactly what you want your vacation to be and your preferences is like almost just as hard as just going and booking it yourself. Interesting. And it's like very high risk. You don't want the agent to go like actually go book a plane flight without you first accepting it. Is there a matter of... Maybe context that we're missing here too, from like the models being able to infer this information about somebody without having to explicitly go ask and learn the preference over time. Yeah, so I think that these things will get there. But first, you need to build up this context so that the model already knows your preferences. And I think that takes time. I see. And we'll need some stepping stones to get to bigger tasks like planning a whole vacation. I see. Okay, very interesting. Last question, any advice that you give to a developer that's exploring this right now in terms of starting to build this or just thinking about it from a general future-proofing perspective that you can give? I feel like my best advice is make sure that you have a way to measure your results. Because I've seen a lot of people will go and... and sort of build in a vacuum without any way to get feedback about whether their building is working or not. And you can end up building a lot sort of without realizing that it's either it's not working or maybe something much simpler would have actually done just as good a job. Yeah, I think very similarly, like, you know, starting as simple as possible and like having that measurable result as you are, you know, like building more complexity into it. One thing I've been really impressed by is like I work with some really resourceful startups and they just... they can do everything within one LLM call. And the orchestration around the code, which will persist even as the model gets better, is kind of their niche. And I always get very happy when I see one of those because I think they can reap the benefit of future capability improvements. And yeah, I think realistically, we don't know what use case will be great for agents and the landscape's gonna shift. It's probably a good time to start building up some of that muscle to think in the agent land, just to understand that capability a little bit. Yeah, I think I want to double click on something you said of being excited for the models to get better. I think that if you look at your startup or your product and think, oh man, if the models get smarter, all of our moat's going to disappear. That means you're building the wrong thing. Instead, you should be building something so that as the models get smarter, your product gets better and better. Right. That's great advice. Eric, Barry, thank you guys. This is Building Effective Agents. Thank you. Thanks.

I'm Alex, I lead Cloud Relations here at Anthropic. I'm Eric, I'm on the research team at Anthropic. I'm Barry, I'm on the Applied AI team. I'm going to kick us off here. For viewers just jumping in, what's a quick version of...

what an agent actually is. I mean, there's a million definitions of it. And why should a developer or somebody that's actually building with AI care about these things? Eric, maybe we can start with you.

Sure. Yeah, so I think something we explored in the blog post is that, first of all, a lot of people have been saying everything is an agent, referring to almost anything more than just a single LLM call. One of the things we tried to do in the blog post is really kind of separate this out of like, hey, there's workflows, which is where you have a few LLM calls chained together. And really what we think an agent is, is where you're letting the LLM decide sort of how many times to run. You're having it continuing to loop until it's found a resolution.

And that could be, you know, talking to a customer for customer support. That could be iterating on code changes. But something where like you don't know how many steps it's going to take to complete, that's really sort of what we consider an agent. Interesting.

So in the definition of an agent, we are letting the LLM kind of pick its own fate and decide what it wants to do, what actions to take, instead of us predefining a path for it. Exactly. It's more autonomous, whereas a workflow, you can kind of think of it as like, you know, yeah, a workflow or sort of like it's on rails through a fixed number of steps.

I see. So this distinction, I assume this was the result of many, many conversations with customers and working with different teams and even trying things ourself. Barry, can you speak? more to maybe what that looks like as we got to create this divide between a workflow and agent and what sort of patterns surprised you the most as you were going through this?

Sure. Honestly, I think all of this kind of evolved as like model got better and like teams got more sophisticated. We both worked with a large number of like customers were very sophisticated and we kind of went from like having a single LLM to having a lot of LLMs and like eventually having like LLMs orchestrating themselves.

So, you know, like one of the reasons why we decided to create this. distinction is because we started to see these two distinct patterns where you have workflows that's pre-orchestrated by code. And then you also have agent, which is a simpler but complex in other sense, like different shape that we're starting to see. Really, I think as the models and all of the tools start to get better, agents are becoming more and more prevalent and more and more capable. And that's when we decided, hey, this is probably a good time for us to give a formal definition.

So in practice, if you're a developer implementing one of these things, what would that actually look like in your code as you're starting to build this? Like the differences between, like maybe we actually go down to like the prompt level here. What does an agent prompt to look like or flow?

And what does like a workflow look like? Yeah, so I think a workflow prompt looks like you have one prompt, you take the output of it, you feed it into prompt B, take the output of that, feed it into prompt C, and then you're done. Kind of there's this straight line, fixed number of steps, you know exactly what's going to happen. And maybe you have some extra code that sort of checks the intermediate results of these and make sure they're okay.

But you kind of know exactly what's going to happen in one of these paths. And each of those prompts is sort of a very specific prompt, just sort of taking one input and transforming it into another output. For instance, maybe one of these prompts is taking in the user question.

and categorizing it into one of five categories so that then the next prompt can be more specific for that. Okay. In contrast, an agent prompt will be sort of much more open-ended and usually give the model tools or multiple things to check and say, hey, here's the question, and you can do web searches or you can edit these code files or run code and keep doing this until you have the answer.

I see. So there's a few different use cases there. That makes sense as we start to arrive at these different conclusions.

I'm curious, as we've now kind of covered at a high level how we're thinking about these workflows and agents and talking about the blog post, I want to dive even further behind the scenes. Were there any funny stories, Barry, of wild things that you saw from customers that were interesting or are just kind of far out there in terms of how people are starting to actually use these things in production? Yeah, this is actually from my own experience.

building agents. I joined about a month before the Sonnet V2 refresh, and one of my onboarding tasks was to run OS World, which was a computer use benchmark. And for a whole week, me and this other engineer were just staring at these agent trajectories that were counterintuitive to us. And then, we weren't sure why the model was making the decision it was, given the instructions that we would give it. So we decided we're going to act like cloud and put ourselves in that environment.

So we would do this really silly thing where we close our eyes for a whole minute, and then we blink at the screen for a second, and we close our eyes again and just think, well, I have to write Python code to operate in this environment. What would I do? And suddenly it made a lot more sense.

And I feel like a lot of agent design comes down to that. There's a lot of context and a lot of knowledge that the model maybe does not have. And we have to be empathetic to the model.

And we have to make a lot of that clear in the prompt, in the to description, and in the environment. I see. So a tip here for developers is almost like to act as if you are. Looking through the lens of the model itself in terms of like, okay, what would be the most applicable instructions here? How is the model like seeing the world, which is very different than how like we operate as a human, I guess, with additional context.

Eric, I'm curious if you have any other stories that you've seen. Yeah, I think actually my like in a very similar vein, I think a lot of people really forget to do this. And I think maybe the funniest things I see is that people will... put a lot of effort into creating these really beautiful detailed prompts.

And then the tools that they make to give the model are sort of these incredibly bare bones, like, you know, no documentation function, like the parameters are named A and B. Yeah. And it's kind of like, oh, like an engineer wouldn't be able to like, you know, work with this as a, you know, work with this as if this was a function they had to use, because there's no documentation.

Like how can Claude expect, how can you expect Claude to use this as well? So kind of it's like that lack of like putting yourself in the model shoes. And I think a lot of people, when they start trying to use tool use and function calling, they kind of forget that they have to prompt as well.

And they think about the model just as this, as a more classical programming system. But it is still a model. And you need to be prompt engineering in the descriptions of your tools themselves. Yeah, I've noticed that.

It's like people forget that it's all part of the same prompt. Like it's all getting fed into the same. prompt in the context window and writing a good tool description influences other parts of the prompt as well so that is like one aspect to consider agents is this kind of all the hype term right now a lot of people are talking about it and there's been plenty of articles written and videos made on the subject what made you guys think that now is the right time to write something ourselves and talk a little bit more about like the details of agents you Sure, yeah.

I think one of the most important things for us is to be able to explain things well. I think that's a big part of our motivation, which is we walk into customer meetings and everything is referred to as a different term, even though they share the same shape. So we thought it would be really useful if we can just have a set of definitions and a set of diagrams and code to explain these things to our customers. And we are getting to the point where the model is capable of doing a lot of the the agentic workflows that we're seeing. And that seems like the right time for us to have some definitions or just to make these conversations easier.

I think for me, I saw that there was a lot of excitement around agents, but also a lot of people really didn't know what it meant in practice. And so they were trying to bring agents to sort of any problem they had, even when much simpler systems would work. And so I saw that as like one of the reasons that we should write this is like...

guide people about how to do agents, but also where agents are appropriate. And that you shouldn't go after a fly with a bazooka. I see. I see. That was a perfect parlay into my next question here.

There's a lot of talk about the potential of agents. And every developer out there and every startup and business is trying to think about how they can build their own version of an agent for their company or product. But you guys are starting to see what actually works in production.

So we're going to play a little game here. I want to know one thing that's overhyped about agents right now and also one thing that's underhyped, just in terms of implementations or actual uses in production or like potentials here as well. So Eric, let's start with you first.

I feel like underhyped is like things that save people time, even if it's a very small amount of time. I think a lot of times if you just look at that on the surface, it's like, oh, this is something that takes me a minute. And even if you can fully automate it, it's only a minute.

Like what help is that? But really... That changes the dynamics of now you can do that thing a hundred times more than you previously would. So I think I'm most excited about things that, if they were easier, could be really scaled up.

Yeah, I don't know if this is necessarily related to hype, but I think it's really difficult to calibrate right now where agents are really needed. I think there's this intersection that's a sweet spot for using agent, and that's a set of tasks that's valuable and complex. but also like maybe the cost of error or cost of monitoring error is relatively low.

That set of tasks is like not not super clear and obvious unless we actually look into the existing processes. I think coding and search are two pretty canonical examples where agents are very useful. Take search as an example.

It's a really valuable task. It's very hard to do deep iterative search, but you can always trade off some precision for recall and then just get a little bit more documents or a little bit more information than is needed and filter it down. So we've seen a lot of success there with agent-taked search. What does a coding agent look like right now? Coding agents, I think, are super exciting because they're verifiable, at least partially.

Code has this great property that you can write tests for it, and then you edit the code, and either the tests pass or they don't pass. Now that assumes that you have good unit tests, which I think every engineer in the world can say, like, we don't. But at least it's better than a lot of things.

There's no equivalent way to do that for many other fields. So this at least gives a coding agent some way that it can get more signal every time it goes through a loop. So if every time it's running the tests again, it's seeing what the error of the output is, that makes me think that the model can kind of converge on the right answer by getting this feedback.

And if you don't have some mechanism to get feedback as you're iterating, you're not injecting any more signal. You're just going to have noise. And so there's no reason without something like this that an agent will converge to the right answer.

I see. So what's the biggest blockers then in terms of improving agentic performance on coding at the moment? Yeah.

So I think for coding, we've seen over the last year on Sweebench, results have gone really from very, very low to I think over 50% now, which is really incredible. So the models are getting really good at writing. code to solve these issues.

I feel like I have a slightly controversial take here that I think the next limiting factor is going to come back to that verification. It's great for these cases where we do have perfect unit tests and that's starting to work. But for the real world cases, we usually don't have. perfect unit tests for them.

And so that's what I'm thinking now, like finding ways that we can verify and we can add tests for the things that you really care about so that the model itself can test this and know whether it's right or wrong before it goes back to the human. I see. So making sure that we can embed some sort of feedback loop into the processes itself, the right or wrong. Okay. What's the future of agents look like in 2025?

Barry, we're going to start with you. Yeah, I think that's a really difficult question. This is probably not a practical thing, but one thing I've been really interested in is how... a multi-agent environment will look like.

I think I've already shown Eric this. I've built an environment where a bunch of cloud can spin up other clouds and play werewolf together. And it's like a completely different...

What is werewolf? Werewolf is a social deduction game where all of the players are trying to figure out what each other's role is. It's very similar to Mafia. It's entirely text-based, which is great for cloud to play in.

I see. So we have clods... multiple different clods playing different roles within this game all communicating with each other.

Yeah, exactly. And then you see a lot of interesting interaction in there that you just haven't seen before. And that's something I'm really excited about. It's very similar to how we went from single LLM to multi-LLM.

I think by the end of the year we could potentially see us going from agent to multi-agent. And there are some interesting research questions to figure out in that domain. In terms of how the agents interact with each other, what is this kind of...

emergent behavior look like in that front as you coordinate between agents doing different things? Exactly. And just like whether this is actually like going to be useful or better than like a single agent with access to a lot more resources.

Do we see any multi-agent approaches right now that are actually working in production? I feel like in production we haven't even seen a lot of like successful single agents. Okay, interesting. But like you know this is kind of like a potential extension of like successful agents with the I guess like improved capabilities of the next couple of generations of models.

Yeah, so this is not advice that everyone should go explore the multi-agent environment. It's just, I think, to understand the model's behavior. This provides us with a better way to understand model behaviors. I see. Okay, Eric, what's the future of agents in 2025?

I feel like in 2025, we're going to see a lot of business adoption of agents starting to automate a lot of repetitive tasks and really scale up a lot of things that People wanted to do more before but were too expensive. You could now have 10x or 100x how much you do of these things. I'm imagining things like every single pull request in triggers a coding agent to come and update all of your documentation. Things like that would be cost prohibitive to do before.

But once you think of agents as sort of almost free, you can start doing these, adding these bells and whistles everywhere. I think maybe something that's not going to happen yet, going back to what's overhyped. I feel like agents for consumers are fairly overhyped right now. Okay, here we go. Hot take.

Because I think that we talked about verifiability. I think that for a lot of consumer tasks... It's almost as much work to sort of fully specify your preferences and what the task is as to just do it yourself.

And it's very expensive to verify. So like trying to have a agent like fully book a vacation for you, describing exactly what you want your vacation to be and your preferences is like almost just as hard as just going and booking it yourself. Interesting.

And it's like very high risk. You don't want the agent to go like actually go book a plane flight without you first accepting it. Is there a matter of...

Maybe context that we're missing here too, from like the models being able to infer this information about somebody without having to explicitly go ask and learn the preference over time. Yeah, so I think that these things will get there. But first, you need to build up this context so that the model already knows your preferences. And I think that takes time.

I see. And we'll need some stepping stones to get to bigger tasks like planning a whole vacation. I see. Okay, very interesting. Last question, any advice that you give to a developer that's exploring this right now in terms of starting to build this or just thinking about it from a general future-proofing perspective that you can give?

I feel like my best advice is make sure that you have a way to measure your results. Because I've seen a lot of people will go and... and sort of build in a vacuum without any way to get feedback about whether their building is working or not.

And you can end up building a lot sort of without realizing that it's either it's not working or maybe something much simpler would have actually done just as good a job. Yeah, I think very similarly, like, you know, starting as simple as possible and like having that measurable result as you are, you know, like building more complexity into it. One thing I've been really impressed by is like I work with some really resourceful startups and they just...

they can do everything within one LLM call. And the orchestration around the code, which will persist even as the model gets better, is kind of their niche. And I always get very happy when I see one of those because I think they can reap the benefit of future capability improvements.

And yeah, I think realistically, we don't know what use case will be great for agents and the landscape's gonna shift. It's probably a good time to start building up some of that muscle to think in the agent land, just to understand that capability a little bit. Yeah, I think I want to double click on something you said of being excited for the models to get better.

I think that if you look at your startup or your product and think, oh man, if the models get smarter, all of our moat's going to disappear. That means you're building the wrong thing. Instead, you should be building something so that as the models get smarter, your product gets better and better. Right.

That's great advice. Eric, Barry, thank you guys. This is Building Effective Agents.

Thank you. Thanks.

Transcript for:Understanding and Building Effective AI Agents

Transcript for:
Understanding and Building Effective AI Agents