Diffusion LLMs vs. Autoregressive AI: Stefano Ermon | The AI Rabbit Hole 2026 | DVC

How's everyone doing? Oh, that come on. You go louder than that. How's everyone doing?

Right. Well, before I start, I I do want to give my I want to give some props. Let's give it up for the Mad Hats conference team. Nick, Marina, Koosma, Alexa, everyone who's put this on. This is like my second year coming and it gets better and better. I don't know about you, but that that magic tea definitely put me on a on a trip. So, I I can definitely confirm that. No, I'm just having fun. But no, seriously, the the spread is so nice outside and it's really great to be here. So, I have the pleasure of interviewing Stfano. I'm going to read his bio just so we're all on the same page because this is a very very impressive individual and we're going to have a really deep dive talk on where LLMs are going, what he's doing differently with Inception and of course where he sees the future because we can't not talk about future at something like Mad Hats.

So, Stefano Arman is a prominent AI researcher, entrepreneur, and he was a professor of computer science at Stanford University. Best known, this is why you all want to definitely take a seat here. He's definitely known for pioneering work in generative AI and right now he's the co-founder of Inception Labs, also known as Inception AI. So, let's get started. So Stephano talk to us about your journey from academia like we talked offline about you had a lab you were doing this cutting edge work why did you want to go and co-ound inception AI and do that

yeah excited to be here and yeah so I'm Stephano I've been a researcher in AI studying generative models for a long time like over a decade I started building yeah generative models before it was cool. I still remember you know it was hard to publish papers on generative models. It was hard to get funding for research on generative models. People did not understand why you would want to train a machine learning model that can create images or text. So those were different times. I've been lucky to be working at the frontier of you know how these models are built and and served. my group at Stanford is mostly famous for having invented diffusion models which is kind of like the main approach that is used today to generate images video music if you've used the stable diffusion or midjourney or Sora they're all based on on diffusion models that work you know initially started out focusing on images and video but since 2019 I've been thinking about how to extend that kind of approach to text and code generation.

Initially things didn't work. but then in 2024, a couple of years ago, we had a breakthrough in my lab and we figured out a way to basically train diffusionbased language models that were competitive with the kind of approach that everybody else was taking, which is the standard GPT style auto reggressive model where you generate left to right, one token at a time. And so we published a paper showing that we could train a GPT2 sized model. So something with less than a billion parameters which was kind of like the largest model we could train given the compute that we had access to in the lab. But it was very exciting because for the first time we were able to match the quality of an auto reggressive model using a diffusion-based approach and these diffusionbased LLMs were like 10x faster than the traditional auto reggressive model.

And the reason that is possible is because existing models like the you know the cloud models or GPT models or the Gemini models they are all auto reagive and what that means is that they generate you know you ask a question and they will generate an answer left to right one token at a time. Instead the diffusion models that we're training at inception are essentially more parallel. So they work like a traditional diffusion model for images or video where you start with noise and then you gradually refine it until you get a crisp answer, a crisp image at the end. And because these models are able to modify all the tokens, all the pixels in the image, they are significantly faster than the traditional auto reggressive models. And so that's the kind of technology that we've been building at inception. we've trained you know after the research at Stanford we started a company called inception to scale up and train commercial scale diffusionbased language models and we've been successful in training large scale models that are competitive in quality with some of the best speed optimized models from Frontier Labs but they are significantly faster we are able to get to over a thousand tokens per second with these diffusion based LLMs and So you can kind of get the speeds that you would get using AI inference chips on cerebras gro able to serve our models on GPUs and so they are also significantly more costefficient compared to traditional autogressive models.

So along those lines you mentioned you have the secret sauce you know you said that you can do something better and faster and how was this developed? Was this developed post Stanford? like is this why you needed to do inception and not stay at Stanford? Walk us through that journey.

Yeah, so the the core of the research started out at Stanford. We had to figure out a lot of new mathematics, a lot of new science to figure out how to extend a diffusionbased model to that was originally created for images and video to more discrete kind of settings like text and code. So a lot of the initial algorithmic idea, a lot of the math was developed at Stanford. then when we started scaling we started trying to figure out how to train bigger models as to how to make them more efficient. we had to keep you know inventing new things. We had to kind of eventually change a lot. and so at inception we had to develop new methods for training the models for serving the models. we had to build our own inference stack and so there's a lot of IP that was developed at the company but a lot of the core initial ideas came from from my lab at Stanford.

walk us through kind of, you know, because you talked to me offline, you know, that, you know, 2024 is when you had to go all in. Like what's it like now at this moment with all this attention on LLMs and inference and like how is it different from when you started it?

It's you know it's getting more and more exciting by by every day. I think we were early in the sense that we realized that the trend was shifting from training towards inference. you know everyone was focused on trying to find the most efficient solution at for training the largest model in the most efficient way possible. And so if you think about historically the big shift happened in 2017 when people figure out that it you know if you transition from a recursive neural network an RNN to a transformer-based model you can get much better scaling when you train large language models. Why? Because a transformer-based model can process many tokens at the same time. It's more parallel and so the scaling laws are significantly better. Now the shift was starting to happen towards spending more and more resources, more and more compute for inference, serving these models to more and more people, enabling more and more applications, agents.

And so it became obvious that we needed something more scalable for inference, not for training. And so we had to think about a way to make the model more efficient at inference time. And the way you make things more efficient is always through parallelism. The bitter lesson is that the more parallel solution is the one that is going to win. And so we had to figure out a way to move away from the traditional auto reggressive generation where you generate one token at a time and move towards a paradigm where you are able to modify to output many tokens at the same time with the neural network and that's what diffusion models can do.

Okay, I want to ask a question to the crowd. Is anyone here a stage startup founder? raise your hand. And I have a very specific reason why I'm asking. Okay, we have a couple here. Anyone here in the process of raising money? Raise your hand. Okay. Wow, you guys are all funded by DVC. I guess there's like not many hands there. I was kidding. question for you on building the company. I want to talk about that for a sec. Like what was the transition like from going from the research world to building a company? And the second question if you can answer it is how do you stay focused with all the attention and all the things happening like walk us through that please.

Yeah so it was there was definitely some learnings switching from from the lab to to to the company but fundamentally still a lot of the work especially in the first few years of the company was still mostly R&D kind of like figuring out how to train the models how to make them as efficient as possible. So the work was not significantly different from the you know not too different from the one I was used to be doing at Stanford. So the focus was heavy on on research and so the day-to-day was not too different from the from the the lab at Stanford. I think what changed is the scale the fact that we finally had access to a lot of compute a lot of engineering resources the ability to buy data and so that's really the thing that changed and that's why I started the company.

if we wanted to compete with the with the you know with the with the labs the big labs and Google and all the other players we needed more resources to really scale up the size of the model and all the the kind of like dirty work that you need to do to actually get good performance in practice and so that's why you know I had to to to raise venture capital to to to get the resources to to really scale up and demonstrate and go from a research idea to a prototype to a product which is where we are right now.

Okay. And I have one more question for you and if anyone has a question specifically for Sfano, I'm going to we're going to do it right after this. Where do you see now the world of Frontier Labs and LLMs now that you've been in it for so long? I mean, you were a pioneer to a certain degree with what you've did at Stanford to the work you've been doing the last few years. Like what's your prediction as we're entering 2027? Yeah, what we're seeing is that inference time scaling is becoming more and more important you know agents and the the ability of having LLMs that can reason that can think through many possible solutions that can revise their solutions that can interact with tools with environments to to get better and ultimately achieve more and more complex tasks. That's where things are going. eventually they might even get towards self-improvement where maybe the LLMs are actually able to improve themselves.

it's all part of the same pattern. It's all about kind of like shifting from training towards inference. and that's why it's such an exciting time for inception because we have a solution that it's more scalable at inference time which is the axis that is going to matter more and more as we you know deploy these models to to solve more and more complex tasks and in more applications and you know just people are are using AI more and more to solve all kinds of different things.

Okay. Does anyone have any questions for Stefano? Yes, you sir.

Yeah. So our model is still like the the neural network under the hood is still a transformer. So although it's trained differently, it's used differently at inference time, but it's still based on attention. And so we still have like the kind of quadratic scaling that you would have in a traditional auto reggressive model. So it's not better, it's not worse. The models that we have in production, they have about 130,000 tokens of context length maximum. and so it's not the largest but there is no fundamental bottleneck. We we can get to essentially as large of a context as we want. Yeah. The main value proposition right now is is speed and cost. Those are the two axis where we dominate auto aggressive models. and that's the value proposition. a lot of our customers they you know when you think about an LLM you always care about three things.

You care about quality, you care about speed and you care about cost. And you see that different LLMs, they kind of like land differently across this this spectrum, right? And depending on the application, you might care more about quality, you might care about speed, you might care about cost. there is a fundamental tradeoff that cannot be avoided if you're using an autogressive model. And so if you want the highest possible quality, you need a very large model that is going to be slow. It's going to be expensive. if you want a fast model then you typically need something smaller which is not going to be as accurate as maybe what you need. with a diffusionbased LLM we're shifting that paro frontier and so what that means is that for the same you can pick whatever latency you know is okay for your application and then you can get higher quality compared to what's possible with an autogressive model or for the same speed you can reduce costs.

So you're basically able to achieve different kinds of trade-offs which might make make sense depending on the application that you're that you're thinking about and depending on what you're optimizing for.

Before we go, could you tell people where to find you, how they want to use your product?

Yeah, please I know visit our website inceptionlabs.ai. we have a an API platform where you can use our models today. it's all open compatible, so it's very easy to try out our models. you know, it's just like one or two lines of code that you need to change to swap out the traditional autogressive LLMs with a diffusion based one. you're going to get high quality results, very fast answers. Our models are very costefficient. It's like 25 cents per million tokens input, 75 cents per million tokens output. So, it's very economical and it's really really fast. So if you're working on real-time agents, voice agents, coding agents where you really care about interactivity and latency, give it a try. it's very easy to to to you know give it take them for a spin. we have a pretty generous kind of like free tokens package when you sign up and so please give it a try.

Right. Thank you very much. Give it up for Stfano. Thank you.