Building with Claude Code — Anthropic's Erik Schluntz on AI Coding Agents & Developer Tools | AI Rabbit Hole 2025

For this, we're gonna welcome on stage Henry Yin, CTO at AGI House and Erik Schluntz, member of technical stuff at Anthropic. Thank you very much, Nick. Hey, everyone. Thank you. Should we take a seat? Yeah. Adjusting my M hat a little bit. Good morning, everyone. How's everyone doing? so, hey, my name is Henry from AGI House. and super excited to be here today in conversation with Eric from Anthropic. Eric's one of the most thoughtful people I've come across when it comes to I agents in terms of how they are built, how they work, and how they are used in practice. So, Eric, before we dive in, could you share a bit about your journey? You were CTO and co-founder at Cobalt Robotics for a while, and now you're doing product research and anthropic. So, how did the transition happen and what put you into this world of coding agents?

Yeah, sure. Thank you. yeah, I'm Eric. nice to meet everyone. so I started Cobalt Robotics, right after graduating from college. and we built security guard robots, very friendly, no tasers or anything, just a lot of computer vision. and I ran that business for about eight years. and I think by the end of it, you know, eight years is a long time, I was starting to get more excited about pure AI, outside of the hardware space. so I took some time off and basically did a lot of exploration and kind of very much to my surprise I I was expecting to go do another startup after my time off. but I kind of realized you know what I really wanted was to be at the center of the current AI revolution and that felt much easier to do at one of the core foundation model companies like Anthropic.

and so you know after a lot of soulsearching I decided to go join Anthropic on the research team. and in particular I'm in a subgroup called product research which is sort of a very entrepreneurial sub team where we spend 20% of our time with customers with startups with enterprises and 80% of our times doing real research on LLMs and it's that combination is really designed to help us feel the real problems that are happening in the industry and what we need to make these models better at so that they can be more useful and add more value to everyone. so at Anthropic I mainly work on agents and coding agents in particular largely within sort of by doing reinforcement learning and creating new agent architectures. Awesome. Thank you for sharing that. I know you now spend a lot of time working on cloud code. So before we get dive in, I actually want to get a quick sense of where the room's at.

Can we do a really quick show of hands of how many of you have used cloud code or something similar coding agents living in your terminal? Awesome. Thank you. a lot more hands that can be upset. And how many of you have used any AI assisted coding tools like cursor or copilot? All right, that's good to see everyone. Cool. So, Eric, for folks who haven't used cloud code before, can you share a bit about like what's the core idea behind it and how do you see it fitting into how developers work today? Yeah, thank you. So, Claude Code is one of our newest products. it is a coding agent that runs right in your terminal. And by coding agent I mean a it's an LLM that can take multiple steps of actions. You give it a highle prompt like I want to implement this feature in my codebase and it is going to agentically go explore your codebase, read files, edit them, maybe even run the code, make sure the tests pass and do all of this sort of multiplestep work right in your codebase and then come back to you rather than using something like cloud AI or chat GBT where you're kind of needing to do every step one at a time.

So, I'm really excited about sort of using agents in the future with coding because they can do this multiplestep work and it saves you all of that back and forth of copy and pasting things in and out of it. And the other thing I'll say is that we designed clawed code very intentionally to be in a terminal because we see developers every developer they use an IDE but they also still use the terminal and we see this as a very complimentary relationship where there are things that you know exactly what you want to change and how you want to change it and we don't want to take that away from people. We want people to still use an IDE and be deep in their code, but we also want them to have sort of a agent pair programmer that's there in their terminal with them and kind of have this combination of tools.

Awesome. That gives a good sense of the vision. So let's get a bit more concrete. So you mentioned that you are using both cursor and along with you know cloud code. So how do you determine what task you want to do it you know manually maybe using cursor and what task to delegate to you know cloud code. What's the framework for determining this? Yeah. I'd say if there is an a small change or something where I'm very picky, I know exactly what I want to change, I'm going to do that myself in cursor because I, you know, want to have that craftsmanship and kind of see exactly what I'm going to do. if it's something that is a bigger piece and it's or if it's something where I don't care or know as much about how the details are going to shake out, that's a really good task for a coding agent like Claude Code where I can say, "Hey, I have this API endpoint, you know, that does X.

I need to write a similar one. It's going to take me an hour to go and it's pretty similar." that is a perfect task for a coding agent because it has all of the context that it needs by looking at sort of the starting point and the similar one that you're trying to make a new ver version of. and it's something that's not just a few lines where I'd be able to do it quickly, but it's going to be a lot of lines and back and forth and writing tests. So, that's something where it it has the information that it needs to succeed already in the codebase and it's saving me a lot of time. So, I think kind of these bigger these bigger tasks are really good for coding agents. another great thing is if you already have an implementation, tell it to go write tests and then run them and like loop until it gets all the bugs out.

is a really good use case where it saves you a lot of this back and forth time. When we chatted earlier, I think one thing really stuck with me is that you have this u rule of thumb of the ratio between you know the amount of work that actually you have to do versus the amount of work you need to verify the work correctness. Can you talk more a little bit more about that? Yeah, sure. I think this is true for sort of all AI, not just agents, but it gets even bigger with agents. I think a lot of people are excited about automating with something with AI if it's a lot of work. But I think that's only half of the story. You need to look at the back end of okay, once the agent or AI has done that hour of work for you, how long does it take you to verify that it's done?

And I think there are some cases where even though the work product is, you know, very intense, it's almost just as hard to verify it. and so it actually you don't end up saving much time at the end. If it spends an hour, generating, you know, a complex legal document for you, it's very hard to verify that and you're going to need to maybe spend that hour anyway just carefully double-checking everything. And so in the end, you haven't saved any time. I think the flip side of this is you know, generating very visual code. it can write a bunch of front-end code that would maybe have taken you an hour or two to write and you just look at it and you can immediately see this is correct. and I think that's sort of like an example of this ratio is amazing. and so I think anywhere where you can make verification easier and faster that is like a really important complement to using AI.

If you can verify it, then suddenly if you can generate it with AI now, like you've closed the loop and you're going to be able to get value way faster out of this. Awesome. So you're saying when you're when it's easy to actually verify the result and it's actually better for AI to AI is more valuable for cases that can be verified. So I imagine cloud code is pretty good at front-end tasks. Yes, it's fantastic at front end. It's fantastic at like writing visualization code or data science code where you need to graph things or make plots. It's also really good if you have a lot of tests because you get some sort of verification from those tests. Of course, you know how much everyone trusts their tests is is always sort of case by case. Awesome. And besides front end tasks, you know, software engineering has a very long life cycle, right?

Like planning, you know, coding and then debugging, testing, deployment. Is there any other thing that's you know cloud code is particularly good at handling besides know front end? Yeah, I really like also using Claude code for exploring a new codebase. So if if there's an area that you're unfamiliar with, there's, you know, thousands of files and you're trying to figure out how something works, Claude is really good at answering questions like that because it's going to agentically search and it's going to go do GPS and finds and sort of look and explore these files and come back to you with an answer. and so I think that's also like a really really great use case for a product like this. and I think there are many hidden gem features in cloud code. I think one of the features you privilege share with me is you actually can pay screenshots into code which wasn't obvious to me because it's like all everything living in terminal.

Is there any other you know under underused features or you know you want to share with us in cloud code? there is a hidden tool where if you ask claude a code to send you stickers it will actually like open up a web form for you can request us to send you stickers. that's probably the funniest Easter egg. I don't know if that's actually still in there but it was there when we launched. Yeah. Awesome. Great. so I want to zoom out a little bit. You know we have talked about how to you know get the most out of clot code. and cloud code itself is a coding agent. I know you have a lot of thoughts on how to build more effective agents in general. So maybe we can touch a little bit on that. so maybe we can start with a terminology clarification.

So what is an agent to you? What is the difference between agent with traditional workflows? Yeah. so this agents have been sort of a super hot topic for the last year and sort of everything is being described as an agent. Every company is trying to rebrand whatever they've made as some sort of agent to get sort of hype around it. and I think it it's become very unclear of like wait okay what even is an agent? and I think a lot of people were describing even like a single LLM call of oh now it's an agent. so at Enthropic we have kind of this this framework between just a single LLM call, a workflow and an agent. single LLM call that's very kind of obvious that just you know one call. A workflow is something that we describe that has no loops. So you might have an one query that then feeds into another that feeds into another.

but it's very deterministic or you know there can be branches but there's kind of this very set graph of what's going to happen that is a workflow I think what something for something to be an agent there needs to be a loop where the agent's output is fed back into it and it might run for three loops or it might run for 30 loops and you don't know it's it's kind of like it's like the difference between a car and a train. the car you can go anywhere and it's kind of like up to the driver. A train is on rails and there's only so many different paths for it to take. So I think that's that's the biggest difference there. Awesome. And for the last the the third kind of you know agent where there's a loop going on and you're really unleashing the power of intelligence of large language models.

How do we ensure that it's going to do what we want them to do and then have good reliability for example like in coding agent cases? Yeah. so reliability of agents I think is a really interesting and hard point. I think you know evaluating the work of these of agents is even harder than evaluating the work of a single LLM call. which has already been sort of a huge challenge for the industry and everyone in software is so used to having deterministic systems. LLMs are inherently non-deterministic and agents bring that even a step further of they become a chaotic system where the you know slight differences in input keep compounding. So I think I think again this comes back to verification of agents are going to be more valuable in an area that is where you have some way to verify the work product. I think there's also great ways that you can build out test sets and be able to like while you're building an agent have some inputs that you have you know known good outputs or correct answers.

So that way you can run you know run these and make sure that you're getting the correct answers you expect. That's super interesting. Any chance you can give us example maybe how you are evaluating cloth code internally? What are the lessons or learnings from evaluating coding agents? Yeah, it's it's super tough but I think one of the best ways to evaluate coding agents is a benchmark called SWEBench. And this is an open source benchmark that takes real GitHub issues from the internet and it looks at the actual pull requests that close them and looks at the unit tests. And the idea of this benchmark is you can give that same GitHub issue to a coding agent and then test whatever code it writes against the original unit tests that were written for that issue. and if it passes those unit tests, then you know, you know that this agent did equivalent work.

and of course, you know, there's there's edge cases and there's there might be other solutions that, you know, work but didn't pass those specific unit tests, but I think it's a really good benchmark. and Sweepbench verified is this set of 500 of these tasks where you have you know a GitHub issue and you have the tests and yeah our our latest model sonnet 3.7 gets about 60% of them right and then 70% if you do more test time compute on top of that which is state-of-the-art great now we have talked about evaluation I also want to talk about the other aspect which is the design of agents so in the traditional self-engineering world there's this concept called design patterns which is something I think distinguish some of the more senior engineers from the junior ones. So how do you see the design patterns in the arrow of agents?

What are some common commonly used or most effective ones that you would recommend? Yeah. So, I think maybe the thing I'll start off with first before going into the some specific design patterns is I think the big mistake that I see lots of people is they over complicate and they start off they have some idea of I want to make an agent to do my taxes and they start off with just the architecture like oh we're going to have five different agents that talk to each other and kind of it's like hey hold on like did you try just like one agent first and I think a lot lot of times like people get very excited about this idea of agents and multi- aent and they kind of start looking they have like a tool and they're looking for a problem to solve with it and I really really encourage people like start simple because if you start with some system that's five agents talking to each other and you haven't just tried one agent doing the task you know it's going to be very hard to debug and much harder to debug and improve than if you just had one simpler agent so I think maybe that's like the guiding principle I'd say is like simplicity and that's more important important than any particular pattern.

Awesome. So starting simple and then for teams just starting out so when so they probably want to start stinking with you know just coding up you know everything from scratch instead of using some tool but when would be a good timing to let's say start using tooling like langraph you know to get to implement more complicated you know agents. Yeah. So I think yeah maybe like the two biggest pieces or one one thing I'll answer first is that I think the most common architecture pattern that I want to see people adopt is separation of concerns. So if you have an agent that needs to produce some output that is follows three different rules. A lot of people will try to just generate you know give those three rules to the agent and have it go. but it can actually be much more effective to have those be three separate LLM calls.

Each one is responsible for judging one of those rules and only if all three of those outputs together match, then you know you've you've succeeded. and the reason for that is that it lets you operate and test each one of those sort of mini agents or prompts separately without any fear of, you know, messing up the other ones versus if it's all crammed into one prompt. You could make improvement for one rule and it might actually hurt performance on the other rule. Great. This has been super insightful and now I want to go even one level higher not just tool and design but also culture because today we have many founders and also engineering leaders here. So how I wanted to get your advice on how can people you know get the most out of AI and AI agents for their organization. Yeah, thank you.

I think this is super important and I see a huge difference between companies that are really trying to be AI first and make the best of these tools and companies that are trying to do that but kind of in name only and people that are just trying to stay away from this at all. I think the biggest thing that companies can do to make sure that their comp that all of their employ and now other people see that and they want to try it too. I think that a lot of more traditional companies using AI almost feels like cheating and like people want to hide it and I think that's a really dangerous attitude. I think for your company to succeed over the next 5 years, you need people to feel like not using AI is wasteful. and that you need to really celebrate using AI in your organization and have ways that people that are good at this and find ways to do it make sure that they can share their work and are celebrated internally.

Awesome. Be proud of using AI in your work. Yeah. Awesome. So, right, I think we have covered a lot you know from how to get the most of cloud code and also you know how to build more effective agents. anything you know last final thoughts you want to leave the audience with? I think as agents do more and more work in software engineering and other parts of everyone's work life, I think that all of our jobs are going to become sort of a little bit more like managing and like more strategic where you're not doing the day-to-day work as much anymore, but you're responsible for understanding and really owning the goals and understanding the context that's needed to solve this. And so I think everyone's going to become like a manager of a fleet of agents. And I think that sounds like a pretty fun future to me.

Amazing. Thank you so much for joining us today and sharing so openly. Thank you everyone. All right. Thanks. All the fun.