/running AI on Kubernetes: what actually works

Coining a term you cannot take back

James Carman: Welcome to the Forward Slash Podcast, where we lean into the future of IT by inviting fellow thought leaders, innovators, and problem solvers to slash through its complexity. I am your host, James Carman, and today we're talking to William Morgan. William is the co-founder and CEO of Buoyant, the creators of Linkerd. Before Buoyant, he was an infrastructure engineer at Twitter, where he helped move the company from monolith to microservices, an experience that led him to co-create the term service mesh, the architecture that powers today's cloud native applications. He has been a passionate contributor to the open source community for over two decades. Welcome to the Forward Slash, William. So you helped coin that phrase, service mesh. I did not know where that came from. That's cool.

William Morgan: Howdy, James, thanks for having me. Yeah, well, now I regret it, because that's going to be like the thing written on my tombstone, but yes.

James Carman: Well, it's pretty important. The concept and the pattern have contributed tremendously, especially in the Kubernetes world, but in broader terms too. Maybe explain what a service mesh is, because there may be some folks listening who don't know.

William Morgan: Sure. If you're in the Kubernetes space, you're probably sick to death of this term, and if you're not, then it has no meaning whatsoever. But the idea is that it's basically a layer of infrastructure that sits in between where the application is and the underlying network, and it does what I would roughly classify as application level networking. So it handles things like how do I do retries and timeouts, how do I secure a connection, how do I load balance individual HTTP requests instead of connections. The goal is that you should be able to add a whole host of reliability features and observability features and security features to the platform without having to change application code.

William Morgan: It is this collection of features, but they all sit at one specific layer of the stack, which is the in-between of the Kubernetes cluster and the application. The goal is that the developers don't have to care about this or even know that it's there, but they still benefit from it, because they don't have to write this complicated distributed systems code around how do I do retries correctly in a big multi-step microservices environment, or how do I do TLS correctly. You can offload all of that to the platform. So that's a service mesh.

James Carman: When I think of service mesh, I think of what folks characterize as east-west traffic, within the cluster. Is ingress also part of the service mesh's responsibility, or is that a different thing?

William Morgan: It can go either way. Some service meshes have a default ingress. Ours, which is called Linkerd, which is the first service mesh, the one to coin the term, and the first one to achieve graduated status in the CNCF and all the other cool stuff, we did not ship an ingress layer by default, because we were going to follow the Unix philosophy. We wanted to be small and self-contained and do only one thing, so we'll play nicely with whatever ingress you have.

William Morgan: But you're right about that east-west traffic. You think of north and south as coming into the cluster and going out to some database. East-west is all the communication that's happening between the services. That's the change that happened maybe 10 years ago, and it was a big change in cloud native services. Prior to that you'd run these three tier applications where you'd have a big proxy in front, then the application, one monolithic code base, and then the database at the bottom. That application got split out into tons of different services, and there was this new kind of communication happening. That's when the service mesh suddenly became relevant.

Twitter, Mesos, and the libraries that became Linkerd

James Carman: In your bio you mentioned your time at Twitter. Was Linkerd born out of Twitter? Is that where it came from?

William Morgan: Yeah, exactly right. When I started at Twitter, and this was a long time ago, 2010 was when I started there, it was a monolithic Ruby on Rails application. By the time I left four years later, a lot of what I had done was help turn that monolithic Ruby on Rails app into this big distributed system, what we would now call cloud native, even though the components were very different. So we split it up into a bunch of different services. Now instead of having function calls, method A calls method B calls method C, you've got network calls happening. Service A is calling service B calling service C. That was written on the JVM and in languages like Scala. We didn't have Docker, so the JVM was kind of our packaging mechanism.

William Morgan: It was running on Mesos, which was a big distributed systems orchestration platform, in some ways a predecessor of Kubernetes. A lot of the lessons we learned collectively during that transformation, especially around the management of this new type of communication, went into the thought and, early on, the actual code behind Linkerd. When we started that project, we literally took a bunch of open source Twitter libraries that navigated that east-west communication, bundled them up as a proxy, and that was Linkerd. We later had to rewrite it for a variety of reasons, but that was the impetus for all of it.

The monolith backlash, and what microservices actually buy you

James Carman: You mentioned monolith to microservices, and Linkerd and technologies like it and the CNCF ecosystem have been big on that trend. But now there's this counterculture, this back to the monolith thing. What do you think about that? Is it well founded, or is it just FUD?

William Morgan: I think it's like any engineering decision, there's a bunch of tradeoffs involved. It's not like you move from monolith to microservices and life is suddenly better. You've introduced a whole new set of complexities. Now you have a service mesh that you have to think about. You never had that. You had function calls where A calls B calls C. What do we know about function calls? They work all the time, basically, and they're very, very fast. And now you've replaced them with network calls that fail half the time and are slow.

William Morgan: So you've introduced a whole bunch of complexity. What have you gained in exchange for that? You've gained these separate units that are deployable, that are owned by individual teams. Now you can have a team that owns service A and can deploy it on their own schedule. You've earned this autonomy that you didn't have before, when everything had to merge into one big code base. Is that good? It depends. At the scale Twitter was, it certainly was good. There were other reasons why Twitter went down this route. One was pure scalability. We were pushing the limits of what Ruby could do and what the hardware could do, so we had to split things out.

William Morgan: So it gains you this autonomy of teams, and it gains you the ability to scale different parts independently. One part of your code base might be very simple, and another part might be really load-bearing, and if you can turn that into a separate service, now you can scale up to a thousand replicas of that without having to scale everything out. But you're adding all this complexity. So it depends on the situation you're in. Like anything, when there's a new shiny tool or way of doing things, people go overboard and you end up pushing it to an extreme, and you're like, okay, I've got a hundred developer teams and 10,000 services, that probably doesn't make a lot of sense.

William Morgan: I don't want to say that pushback was unwarranted. I think it's a natural course correction. I will say when you look at how software is built today at a non-trivial scale, the majority of it is built in this multi-service way because it makes sense. A lot of the friction around doing things that way has been paid down pretty dramatically by things like Docker and Kubernetes. But it's not the only way, and if I was starting something, I wouldn't start that way.

James Carman: It was funny when the whole microservices thing happened, people would look at me funny, and I'll be curious about your take on this. To me, what I saw as the major benefits, there are technical benefits of course, but I think the major benefits were more organizational in nature. As you were saying, you can split up teams, they can be more autonomous, they can do their own thing, and you don't have that massive coordination. To me that was the biggest benefit. People always looked at me like, what are you talking about, this is a technical thing, this is not organizational. But it always felt that way to me.

William Morgan: Well, the whole software engineering discipline is about the intersection of humans and computers, or teams and computers. So much of what you do when you're working as a professional software engineer, a lot of the decisions you're making are around how other human beings are going to interact with what you're doing. They're not purely technical decisions. And I'll be a little critical here: I think that's a pretty junior point of view, to say these are technical opinions, this is not a thing where we care about the people. The moment you have more than one person involved, now you're caring about the people. So you have to take that into account.

James Carman: Until AI takes over everything and then we don't care about people anymore. Then who cares, right?

William Morgan: Well, that's right. We get a chance to reevaluate some of these things in a world where AI is taking over.

It rhymes with Cardi B

James Carman: I want to make sure I'm saying it right, because I've said it a couple of times already. Linkerd. Am I pronouncing that right? I've heard other people say it differently.

William Morgan: Linkerd. Yeah, rhymes with Cardi B. It's in the FAQs. Says it right there.

James Carman: Cardi B, okay. All right, good. I'm glad I was saying it right. Somebody corrected me.

William Morgan: And the etymology there, if you are a deep Unix nerd, it'll be very natural for you. We add a d on the end of anything that's supposed to be daemonized, which means it's running in the background. It's not a thing that's waiting for your input. The analogy we had in our brain was that in operating systems, there's loaders and there's linkers. When a binary gets pulled into memory and it has these dynamic libraries it depends on, you have a linker that pulls in the right library code and does it seamlessly, without you as the user having to do anything.

William Morgan: So we had that analogy in mind. We're linking these microservices together. We're making it so that if I'm A and I want to talk to B, I just say, hey, give me a connection to B, and I don't have to care about where B is or how many instances there are or whether it's busy or whether it's secure. I just say, give me that thing, and Linkerd will figure out the details. So that's the origin of that name.

A series of happy accidents

James Carman: Service mesh is one of those things where when you look back on how it came to be, you're like, wow, that totally makes sense, this is just obvious. What was that journey like? How did you get to the point where you understood that if we put these proxies in here, we can create this substrate and plug things in dynamically and people don't have to worry about it? Was it an aha moment, or an evolution over time?

William Morgan: There were some geniuses involved. Real visionary people. No, it's a series of happy accidents, I think, is what it came down to. We started out, at Twitter, we didn't have Linkerd or a service mesh. We had a bunch of libraries that we engineered to do this. So the first version of Linkerd, our only moment of inspiration, was to say, hey, Twitter's doing this, but it's doing it with a Scala library. The intersection of people who care about both Scala and distributed systems at that point in time was zero, or maybe the engineers at Twitter and no one else.

William Morgan: So we said, okay, let's pull this into a proxy. We'll forget about all the cool programming nicenesses and functional programming and flat maps over RPC calls, we're just going to give you a proxy, and you stick that thing next to whatever application you have, and the devs are going to be unaware of it. That made a lot of sense. And we did that at a time where Docker was starting to really come into play and Kubernetes was just getting started, and suddenly all these other conditions were ripe for people to write multi-service systems. Once we had that realization, we were like, Kubernetes is a thing we should target.

William Morgan: And then we were like, man, running the JVM on Kubernetes sucks. People at that point were writing these Go microservices, and they would take like 50 megs of memory. We were taking the JVM and turning this thing into a proxy, and it's really transparent and seamless, but it was running at like 150 megs or more. We had to work hard to get that down, because it wasn't just a JVM, it was Scala libraries and all that. The JVM, love it or hate it, is great at scaling up, but it's not good at scaling down. And we were trying to encourage this model where you would distribute these proxies next to every application pod, so you couldn't quadruple your memory cost.

William Morgan: So we ended up having to rewrite it. We rewrote it in Rust for the core proxy, and the control plane, which is the management piece, we rewrote in Go, because it was talking to the rest of Kubernetes. It was one step after the next. And then we also had to pick a name for it, because when we started out we were calling this thing an application proxy, because we didn't really know how to frame what we were making. We knew it worked, we knew it was a good idea, but people would be like, I already have a proxy, what are you talking about, I use Nginx or I use Apache.

William Morgan: So we had to invent a term. We invented service mesh because it was a meaningless term, but it sounded cool, and it gave it a name, and it was this white space where we could now say, what is Linkerd? It is a service mesh. What does that mean? Well, here, let me tell you. Once you got that idea into people's heads, it started to make sense, and it was easier to distinguish it from their existing proxies, which were, in fairness, used for totally different things. In retrospect, of course, it all looks pretty easy and straightforward, but it was not. A lot of the plan changed as we went along.

James Carman: It's funny when you look at those things and they're very elegant and make things seamless. You look at it and you're like, yeah, I could have figured that out. No, you couldn't. No, I wouldn't have figured that out.

William Morgan: I mean, maybe. A lot of it was the product of the environment, and us just being in this situation at this point in time, and trying to, if we had any skill it was just trying to understand and listen to and observe what was happening with this thing we were saying, and then being willing to change it in response.

Scala, Ruby, and it all sucks

James Carman: You brought up Scala. I'm a Java guy, long time on the JVM. I love Scala as a language, but at least in the circles where I've had experience, it just didn't take off like I thought it would. I feel like the community around Scala was very cerebral. As you said, they were talking about flat map and monads, very functional programming, very almost academic sounding in nature. How did Twitter arrive at Scala?

William Morgan: Some influential engineer was really into that stuff and was like, hey, this is what we're going to use, and you can get on the bus or get off the bus. That's really all it came down to. I was a Ruby guy. I came to Twitter because I love Ruby, this thing's all written in Ruby. And then they said, no, we're going to move on to the JVM. I think I sent an email to the entire engineering department being like, don't let them do this. I got in a little trouble for doing that. And then finally I was like, okay, whatever.

William Morgan: You go through phases as your engineering career progresses. You start out being very wedded to individual tools and ideas, and then at the end you're like, you know what, it all sucks, and it's just a question of which way. So why should I care about that? Let me think about things at a higher level.

James Carman: I so hope that's the pull quote for this episode. At some point you realize it just all sucks.

William Morgan: It's just a question of where you're putting the sucking.

James Carman: I love Ruby too. I thought Ruby was a fantastic language. Whenever you would hear people talking about how you shouldn't use Ruby because it doesn't scale, Twitter was always the counterexample. Well, they're using it at Twitter, so it must scale. And then there was a further conversation that they were doing something different, some sort of binarization of Ruby going on there at Twitter. Is that true? People said they were compiling it, or there was an optimized interpreter, something.

William Morgan: I'm trying to remember. I think we were doing a bunch of stuff like that. We had folks getting deep with the Ruby interpreter. There was a kernel team. There was some serious systems programming going on in there. But I don't think the scaling factor that was really killing us was the choice of language. At that point in time, we were really hardware constrained, at the data center level. We were locked into the situation where, yeah, we could scale this more by adding more machines, but they literally didn't have more machines to give us.

William Morgan: And the nature of the application, you had these very asymmetric situations. You would tweet and you'd have 10 followers, who cares, zero impact. But then Oprah would tweet and 500 million timelines would have to be updated. The many orders of magnitude of difference between different activities, or different people really, put a lot of stress on the system and the way it was designed. There were some design choices early on that weren't really expecting people to have hundreds of millions of followers, and that was really the stress point. So I wouldn't really blame Ruby. It's not the fastest language in the world, but that wasn't the thing that was killing Twitter at that point.

James Carman: That is an interesting problem. You think about posting a tweet as just posting a tweet, but it's not. Me posting a tweet, Twitter's not going to sweat at all. Nobody's hearing it. That's like me going out in the woods and yelling. But Oprah or somebody big like that, that's really an interesting problem.

William Morgan: And there were all these famous moments. This was a fun time to be there. There was, gosh, what was it, the World Cup was happening in twenty, I don't know, when was that? Twenty ten sometime, between twenty ten and twenty four, maybe it's twenty ten there was a World Cup. And every time someone would score, the site would fall over, because everyone was tweeting goal. There were so many tweets, and we were just running on the edge at every point.

William Morgan: I came in there being like, hey, I want to write Ruby and everything else sucks. And after a little bit of that, I was like, okay, we've got to figure out how to keep the site alive. That's level one of Maslow's hierarchy of SaaS applications.

Is Kubernetes settled science?

James Carman: The world of Kubernetes. If you're doing modern software engineering, I kind of feel like Kubernetes is a done deal, settled science kind of level. Are we there? Are there alternatives right now that are in any way going to knock it off its pedestal?

William Morgan: That's a great question. Kubernetes just celebrated its ten year anniversary, and for many technologies that would indicate it's time to put you in the retirement home. And yet that community continues to grow and the usage continues to grow, so it seems like whether you like it or not, it's kind of here to stay.

William Morgan: I cut my teeth on Mesos, so when I came to Kubernetes I was like, this is not as good as Mesos. But now, what do I care? It's the thing, it's there. What does it solve? It solves an important problem, which is, I don't want to have to care about individual machines. Just give me a thing where I can say, okay, I want six of these and I want 12 of these, and this one's a database and this one's an LLM or whatever, and you just figure out the rest of that. That's what it does.

William Morgan: At this point, I hate the term de facto standard, because I almost always see that in a marketing context, when people are trying to say our solution is a de facto standard. But in this case Kubernetes is kind of the de facto standard. What else would you use? It's not zero cost. It introduces a bunch of abstractions and things you have to learn. What's the difference between a pod and a deployment and a replica set and a rollout? You have to have that ontology mapped out in your brain, so there's a cost you pay up front to being able to use it.

William Morgan: But in Kubernetes's defense, people always say Kubernetes is complex in a pejorative way. My sense is that Kubernetes is pretty well designed. Yes, there's things you have to learn, but they all make sense and they compose nicely with each other and they've been thought through, and you can assemble the pieces in a way where it feels coherent at the end. So I don't think it's complex. I don't think it's trivial, and I don't think it's zero cost to get started by any means. You have to learn some stuff, but it kind of makes sense.

William Morgan: To answer your question, I haven't seen a big desire to move off of it, because it just kind of works. And at this point the ecosystem has absorbed enough of it that it's everywhere. Every piece of software probably comes with a helm file. Is that what you want to innovate on, or do you want to move on to the more interesting stuff?

James Carman: Do you want to write your business's software, or do you want to try to write the glue when the glue's there already for you?

William Morgan: Again, if you were starting something new, would you start with Kubernetes day one? I think there's an argument that no, you wouldn't. You would start with something simpler. But I would expect us to hit Kubernetes at some point. By the time you have two or more people trying to use the same set of machines, I think you're pretty close to wanting Kubernetes.

James Carman: It's funny, in simpler situations, we're running one of our internal things as a container app in Azure. But underneath the covers, all of those technologies are usually just a wrapper around Kubernetes anyway. So it's Kubernetes all the way down. I'm with you on the complexity thing. I came from a Cloud Foundry background, that's where I started in the container world, and moving to Kubernetes it did seem complex at first. But once you wrap your head around it, it's kind of elegantly simple. The whole nature of declaring these resources and then the cluster takes care of converging on the state you say you want it to be in, that is a really liberating approach. And the tooling around Kubernetes is so advanced now that there's no excuse.

What Kubernetes actually gives you for LLMs

James Carman: You threw something in there, LLM. Where is the Kubernetes picture now? You do hear people bringing up Kubernetes as almost being the substrate or the foundation that we're going to run all these crazy AI workloads on. What's your experience there? Does that have legs?

William Morgan: Actually, I think that's a great example of why Kubernetes is kind of the de facto standard. Kubernetes doesn't actually give you a lot for LLMs. Forget about training, because training is a whole different activity. But if you just want to run inference, you want to boot up vLLM or SGLang or TensorRT-LLM or any of these things, the way those things all work is they want full access to however many GPUs you have on your node. That's what they want, and they want to take the whole thing and just max it out. They want to stay on that machine. They're using every bit of the GPU memory that's possible. So Kubernetes gives you basically nothing in that situation.

William Morgan: And yet it kind of is the place where everything is happening, because what's the alternative? There are alternatives in the machine learning world, but by the time you're trying to run inference, it's all happening on Kubernetes, because where else would you go? So there's not a compelling reason other than, what's the alternative? As far as I can tell, that's kind of why it's happening there. All of the inference stuff that I'm working on, we're targeting Kubernetes, because that's where it's going to happen. Is it for good reasons, or is it just because that's how the world has evolved? I don't know. I run around all day with an appendix in my body that has no purpose, and I'm not trying to innovate on that.

James Carman: I just leave it there. I haven't really messed with it at all. So from the inference side, actually running an LLM, sending queries to it, not a lot of gain there. Now, what about AI-enabled workloads? Let's say the LLM is running somewhere else, but I want to run a workload that leverages that LLM. Is Kubernetes particularly adept at those workloads, or is it again just, where else are you going to go?

William Morgan: There's some interesting stuff people have been digging into around sandboxes and being able to provide agents with a much more secure environment than what a container is. A lot of that is happening on Kubernetes. So that's one aspect where I think, yes, there is some innovation happening. But apart from that, a workload that uses inference is just a workload. It's some code that runs and talks to some service and gets something back and then does something else. So Kubernetes is valuable to the extent that it's valuable for any workload, with that sandbox thing being something I think is pretty interesting.

William Morgan: On the one hand it's just a regular workload. On the other hand, it's one that you don't really trust, and it's one that's almost by definition doing non-deterministic things, and eventually, if there's a way to delete your database, it's going to find it and it's going to call it.

James Carman: There's been a lot of work around policy as code and integrating policies into the cluster. Is that what they're leveraging? Are they standing on the shoulders of those giants to control and sandbox these things?

William Morgan: I think it's more than that. And I'll quickly admit I have dug about one percent into this, so I don't really know the details, but I think it's a much stronger set of guarantees than that, around how do we truly prevent a non-deterministic workload from being able to access anything else in the cluster.

James Carman: You're highly involved in the Kubernetes world, in the CNCF. What's the next big thing for Kubernetes? What's on the horizon? What's that next era?

William Morgan: There's only one answer to that, and you've already said it, and it is spelled with two vowels.

James Carman: Just AI.

William Morgan: It has eclipsed everything else in the universe, other than peptides, I guess.

James Carman: If you could combine a business for both, you'd just print money.

William Morgan: I'm sure it's out there. I'm sure it's doing super interesting stuff. Peptide design. DeepMind, some of the early successes were around protein folding and stuff. So we're using it for customer success chatbots or whatever, but there's way more interesting stuff we could be doing with the same core technology.

James Carman: Is quantum anything that the Kubernetes world is whispering about at this point, or is that kind of a pipe dream still?

William Morgan: Not that I have seen. Maybe it's out there and I'm just not exposed to it. Certainly from the perspective of Linkerd, which does TLS, we've updated the TLS components to be quantum proof, to use algorithms that we expect to be still valid cryptographically after some quantum improvements happen, which may or may not ever happen. But other than that, I haven't seen a lot of that.

James Carman: I found it funny, the standards around the post quantum cryptography stuff. You would think that cryptographers and people in that world are kind of geeky folks. But then they went and chose Kyber and Dilithium, the two crystals, as what they named their standards after. How could you get any more nerdy? You're giving us a bad name here.

William Morgan: Not doing yourselves any favors with that.

Ship it or skip it: is AI coming for our jobs?

James Carman: We have to move on to our ship it or skip it segment. The idea is that we come up with a topic and we say, yeah, go with that, ship it, that's good, or skip it, that's bad. It's our hot or not. First up, you hear a lot of talk about AI taking our jobs. Do you think AI is going to take people's jobs like yours and mine, or is that just FUD?

William Morgan: If I say ship it, am I endorsing that idea?

James Carman: Let's say ship it means you think it is going to take our jobs, and skip it means no, that's BS.

William Morgan: I have a more nuanced answer than a binary one.

James Carman: Most architects say it depends. That's what we always say. So I'm totally on board with that.

William Morgan: I think there's two levels to this conversation. The first is what's happening today. If you spend time as a software engineer with something like Claude, or any kind of AI development tooling, you will quickly realize that the writing of code is now a thing that you can hand off pretty effectively, but there's a lot that you cannot hand off right now. There's a lot of decision making that you have to do.

William Morgan: As someone who's driving Claude, there's a lot of thinking, in some ways harder thinking, because you have to do it more frequently than you had to in the past, around what am I actually trying to do here and what tradeoffs am I willing to make. So from that perspective, no, AI isn't taking our jobs. If anything, it's this amazing tool that allows us to operate at a higher level. We've all shifted up one level of abstraction. I think that part is cool.

William Morgan: Now, the other layer of the conversation is, if you look at the pace of advancement from what AI has been able to do over the past couple of years, it really is astonishing. And if you play that forward, is there anything that's really going to stop AI from being able to do this next level of stuff? The things that I'm doing today with Claude, where I'm thinking through the design constraints and trying to figure out what is important and what isn't and what's the interface that I want. Is there anything that stops AI from being able to do that? Okay, maybe not.

William Morgan: And what about the higher order decision making? What about me as CEO of Buoyant? I'm making decisions around how does Buoyant invest in this opportunity versus that one, which part of AI is good for us. Is there a reason why AI couldn't do that as well? Okay, well, maybe not. Things are advancing so fast, who can say? Well, then what's the point of any of these jobs? What's the point of having a CEO? Why isn't this company just a bunch of AI things? And that's going to happen to our competitors and to our customers. So the logical conclusion is we're all just going to be AI bots talking to AI bots, and then what's the point of any of this stuff? How does the economy work? What value do humans have at all here? That's the train of thought.

William Morgan: Which is scary, and also it's really hard to navigate, because you've kind of taken off any kind of constraints or rules here. I guess property is still a thing we have to defend, so maybe guns are useful. But what else? That part I don't have a great answer for, other than an optimistic vision that it's not that we have imminent societal collapse, or at least the probability is not at a hundred percent. I think that's a harder thing to answer. Is AI coming for our jobs? I don't know. One answer is all society is doomed, and so yes.

James Carman: There was a Harvard study or somebody like that not too long ago. They talked about, like you're saying, if you keep pulling that thread, we're going to replace jobs, keep pulling that thread, okay, fine, nobody's working. Well, who's going to buy your service? Who's going to buy your product? Nobody's going to have money. This just devolves into chaos and gets to a bad place. Obviously that's not the way we want to go, but people keep pulling on that thread.

James Carman: I don't think we've hit it yet. I feel like we're like monkeys who have stumbled across a hammer. We've realized there's a tool here and we're just floundering around with it. We haven't realized what we can really do with this tool yet. I don't think we've hit that aha moment.

William Morgan: Or we've stumbled across a chainsaw and we're like, it makes a fun sound over here when I use it to answer customer success questions. What happens when I put it over here?

James Carman: Yeah, that's probably a better analogy.

William Morgan: And I do understand the angst, and I share some of that around this question, because in a world where you have taken off all constraints and you don't know what's possible, all you know is that progress is happening incredibly quickly, and then it is hard to have any kind of mental framework around what's going to happen.

James Carman: It could be a scary conversation for sure.

William Morgan: Yep. So stock up on ammo and Bordeaux. That's my advice.

James Carman: That's a true Texan right there. Get some Lone Star beer and some ammo. We do that in Kentucky too. I'm from Kentucky.

Ship it or skip it: loop engineering

James Carman: You mentioned working with Claude, using these tools as your pair programming partner. There's been a lot lately about loop engineering being the next phase of the aha moment we've had with using AI to help with software engineering. Setting up loops and these things just keep on moving. I've said for a long time that the superpower of the future is how are you going to keep the agents constantly running. What are your thoughts? Is this the future, or is this just a stepping stone to the next thing?

William Morgan: I think it's an interesting thing to explore. I think it's a stepping stone, maybe. At its core, it's not really any different from how you're interacting with something like Claude today. You tell Claude something and it goes around and does a bunch of stuff and then comes back, and the value of what it does really comes down to how well have you specified the problem and how accurately can you give it a framework by which it knows, yes, this is good, and no, that's not a good solution.

William Morgan: When you look at what people are doing with loop programming, that really is the deciding factor as to whether this loop is actually useful or not. How well can you specify, this is the goal and this is how to know whether you've hit it or not. For some problems it's very clear how to do that, and you can be pretty successful, and you're just like, okay, keep going over and over again until you can't do it anymore. If you can't do that, well, now you've got a human in the loop and you're kind of in the same situation you were in anyway.

William Morgan: The hard parts of all this stuff are figuring out what you want and being able to describe it, which by the way is the same problem that programming has always had, or any kind of engineering has had. Figuring out what you want and being able to describe it in a way that's sufficiently detailed that whatever system you're using can deliver a solution. So I see it as an interesting thing. I don't think it's any more transformative than the rest of AI has been.

James Carman: I think that's fair. Anytime we try to remove feedback loops, we start to get into dangerous territory, because there's got to be some feedback in that loop of the AI going off and doing things. If you just leave it to its own devices for hours and hours on end, it can get off the rails. It can do stupid things much more quickly than we could. So that human in the loop still has to be involved. That judgment still has to be involved.

James Carman: One thing that happened a couple of days ago, Elon Musk always says these silly things, but one in particular that he said was that programming will be dead in 2026. What he meant is that literally writing computer language or software development language is gone. The AI will write computer binary code directly and we won't have these programming languages anymore. So programming itself is gone. I've said that we could get there, but I can't imagine that's happening in twenty twenty six, given how much I have to correct the AI when it is doing software programming language work.

William Morgan: If the meaning of that is literally we're just going to have the AI write the CPU bytecodes directly, I don't think that really makes a lot of sense, unless you take a very specific interpretation of it. Because right now, the way these LLMs work, they really benefit from as much structure as you can put in place. So when I write code with Claude, I use Rust, because I want the most difficult, strict language involved. And I see it fail the borrow check and go, I've got to fix that, and I've got to fix that. Great, that's exactly what I want. I want as much structure as I can around this thing, so that when it's going through its non-deterministic process, it can be constrained to code that I know is going to be memory safe.

William Morgan: If you remove all those constraints, then you're going to get something that doesn't really work. So the more constraints you can put in place, the better. Now you could say, they'll just do all that stuff as part of the harness, the harness will just validate all of it. Sure, maybe there's a definition where you can do that, but to me that doesn't seem very practical.

William Morgan: The more interesting thing to consider is, is there any point to looking at code anymore? That's one where I feel like we should expect to see a world where you're not really looking at the code. What you're looking at is the plan files, the documents that describe how the system works, the interfaces. But the actual code, who cares? If it compiles, if it passes a borrow checker, it doesn't really matter. I don't think we're quite there yet. I think there's a couple of other things that have to be in place, but that's the level I would expect to get to, which is pretty astonishing.

James Carman: Rust is a particularly nice language for this because it has those extra checks. A compiler is one level of thing, and for compiled languages it's very helpful, but Rust is a bit more extreme in its restrictions about what you can and cannot do. I like that those guardrails are built into the language and it wouldn't even compile. And I'm with you on the reading code. I do find myself reading code less and less. I'll look at the interface level, are the contracts right, those sorts of things. But the down and dirty stuff, is it really that important when I have the observable outcome? I don't know that I'm quite ready to say just go straight to machine code that I can't understand if I needed to. I'm not at that level of trust yet.

William Morgan: Well, right. That's the other thing you've got to keep in mind. Two years ago I would have said, there's no way this stuff is going to work, this is stupid, why are you even bothering.

James Carman: The thing can't even do math correctly.

William Morgan: So I wouldn't listen to any of my opinions. I'm objectively wrong about all this stuff.

James Carman: We have a conference in town, Cincinnati AI Week. I'm from Cincinnati, Ohio. We were talking about something on stage and I said, you know what would be interesting, every year when we do this conference we should have a that's cute section where we look back on predictions and things we were talking about and worried about the previous year, and just look back on it and be like, that's cute. In one year's time it changes that dramatically. This is how wrong we were just one year ago.

Ship it or skip it: token optimization

James Carman: Token optimization. We went through this era of token maxing, everybody was bragging about how many tokens they were using. And then they were like, wait a minute, these things cost money, and now the pendulum's swinging back the other way to this token optimization thing. Should you be token optimizing, or is this premature optimization?

William Morgan: I think it's totally normal. I would think of inference as a resource like CPU or memory. You start out being like, my gosh, I have gigs and gigs of memory now, I can write my application to do whatever it wants. And then pretty soon you're like, everyone else is doing that, and now I actually have to care about keeping my footprint down. It's just a thing that happens. The motivations behind token maxing I think are pretty silly, and so the correction against that maybe is extreme, but it's a resource that only exists in finite quantities, and so if you're going to use it, you should be efficient in your usage about it.

James Carman: The only thing I hope doesn't happen is the extreme. We went from token maxing, where everybody thought that was a huge flex, and then our industry tends to do these massive corrections back the other way. It's this pendulum thing. So I don't want the pendulum to swing so far that we're token optimizing down to the minutest level, and it's like, I only used three tokens to write this whole program. It loses its utility. There's a balance in everything, and I think we need to quickly converge on a balance instead of these extremes with this one especially. We work for clients, and I don't want to go through the time where I'm having to say, wait a minute, you could have used 500 tokens instead of 700 on this to write this function. We've got bigger fish to fry. And if it costs that daggone much, we shouldn't even be using it anyway.

William Morgan: There's so many interesting things to talk about here. This is like a ninety-minute conversation we could have, but I'll just say quickly, we've only scratched the surface of different models, different levels of quality, different levels of expense. You can run open weight models very cheaply, and you can send a lot of tokens to them. If you're running them yourself, you're not paying per token. You're renting hardware from AWS. You can run it on third generation hardware and not pay that much to AWS. So what would you serve with those? There's probably a whole class of workloads where you don't need the latest and greatest model. Have your agent talk to the cheap thing. Now that price has really gone down.

William Morgan: But if I've got a human who's using Claude and is deep in thought and advancing, I want them to have the absolute best experience. So we'll pay the big bucks for that. Various QoS levels, levels of quality. We're just scratching the surface of that, and I think it's going to be super interesting to see how that evolves.

James Carman: You brought up open weight models. I think that conversation changed, and I think it changed within the last few weeks, when we went dark on Fable for a little bit and then GLM came out with the open weight models.

William Morgan: What great timing for that. Like, okay, here you go.

James Carman: How fortuitous. You can't use Fable? Well, let me show you this open weight model that's almost as good. I think the conversation has changed dramatically, and I'm with you, there are going to be situations where the open weight stuff, as you said, it's not trivial to set up, but it's going to get easier and easier. It wasn't easy to run Kubernetes when it first came out either, but now I can spin one up in Azure in like five minutes. I think it's going to get easier and easier to run your own inference with these open weight models. And they're good enough for a lot of stuff. I don't need to send what's the weather in San Francisco going to be to Fable. I can ask an open weight model those sorts of things.

James Carman: I'm with you on routing to the right thing. I was talking to somebody last week about this. I use the Claude superpowers plugin, and I did notice that it's consciously choosing models based on the task at hand, which is really cool. If you throw an open weight model into that mix as an option, where it can offload to a cheap model, I think we're going to be doing a lot better on this token thing.

William Morgan: Part of the reason I'm so excited about this is also that I think this is a big opportunity for Buoyant. A lot of the stuff that we're starting to work on, which by the way is not released at all, is around how do you run inference really efficiently and economically in Kubernetes, and how do you do it for the lower constraints. If I want an agent running and I don't care about sending it to the most powerful model, how do I do that? You can do it pretty easily.

James Carman: That's what I'm saying, it's getting easier. The only thing that came up in my research was that with GLM, their default way of getting it stood up was that you would download Python code from a Chinese company, and there's a little bit of weirdness there that you've got to be careful of. But if you're using a standard harness, just a way to run these open weight models where you say, here's the architecture of the network, here's all the weights and files, next year we're going to be talking about this like it's nothing. When Buoyant releases their awesomeness, we're all going to be like, yeah, just spin this up. That's cute, worried about my inference. Buoyant made it so easy.

The lightning round

James Carman: If you're a friend of the show, you know that this is the most important segment of the show. This is what we call the lightning round. All the things we've talked about before, nobody really cares about that stuff. This is what people tune in for. This is where the answers start to matter. There is a score involved here. We do send it through an algorithm to grade your answers. There are right and wrong answers to these questions, and the stakes are much higher in this round.

William Morgan: What do I get with a high score? I get like a prize?

James Carman: To be determined. It's like when you go to Dave and Buster's and you hit the jackpot, you can buy a Rubik's Cube. But if you don't have a great night, you're walking out with a spider ring. It's one of those kinds of things. Now if you come on a bunch of times, you can stock up your tickets and maybe get that Hello Kitty doll that's sitting up at the top. I'm making this up. I haven't been in a Dave and Buster's in forever. But suffice it to say, this is important. Are you mentally prepared for the lightning round? Do you snore?

William Morgan: Would you like a demonstration? If you give me twenty minutes, just keep talking and I'll gently drift into a slumber, James.

James Carman: I have a list here. The loaded question that I won't ask you was, who has it easier, men or women? I'm not doing that to you. How about, have you ever worn socks with sandals?

William Morgan: No. God.

James Carman: Good. What's your favorite type of tea?

William Morgan: Oolong. Formosan oolong. Specialist stuff, you grow it on the high mountains. Get the little monkeys to pick it. It's called monkey picked tea. It's transcendent.

James Carman: Is that how it works? I've got to get into this. I would feel very important if that's what I drank a lot of. Are you a Godfather or Star Wars kind of guy?

William Morgan: I'm just buying the tea, I'm not leasing the monkeys. I am a Star Wars guy with a very specific restriction, which is I only go up to nineteen eighty three. Anything produced after that, I don't look at it.

James Carman: So that doesn't include Jedi?

William Morgan: No, maybe it's eighty four. I do want to include Jedi.

James Carman: Okay, so you do include the original trilogy, you draw the line after that. I always tell people my favorite is definitely Empire. Do you like the smell of gasoline?

William Morgan: No, not really. With maybe a minor exception. If you get a good Riesling, Riesling the type of wine, and you get one with some age on it, the classic description is you'll have notes of petrol, what they very poetically call notes of petrol. Actually I do like that. But it's not like smelling a gasoline can.

James Carman: Notes of petrol in a Riesling. I did not know that. Have you ever seen a kangaroo in person?

William Morgan: I don't think so. Maybe at a zoo when I was a kid.

James Carman: I think a zoo would qualify as in person. I've never fought one, like you see those videos of guys punching them. They get buff. I don't think I'd mess with one of them. Polka dots or stripes?

William Morgan: They look muscular as heck too. Probably stripes.

James Carman: Most embarrassing store you might be seen shopping at?

William Morgan: I don't know. There's not a lot that embarrasses me.

James Carman: I have three daughters. My answer would probably be something like Sephora. If somebody saw me in Sephora, they would understand, because I have three daughters, but it just feels weird for me to be there.

William Morgan: That's not weird. I think that's sweet. Shopping for your daughters, who would be embarrassed about that? I'm not embarrassed by anything I do. Other than this podcast.

James Carman: That's fair. Scale of one to ten, how good are you at trivia?

William Morgan: I'm really bad. Like three. A lot of trivia is what's happening in the world, or what has happened since the eighties, and I'm oblivious to so much of it. Unless it's Kubernetes. I'd be good at Kubernetes trivia.

James Carman: If they had a bar night and it was Kubernetes trivia, you'd be all in.

William Morgan: I'd be the king of that thing. I'd be the only one there.

James Carman: How long can you hold your breath?

William Morgan: Could probably go fifty plus seconds.

James Carman: That feels respectable. One more. How many times do you think you have sneezed in the last seven days?

William Morgan: I had a little allergy attack a couple of days ago, so probably like forty. I think you could go three days back and it'd be like thirty-eight.

James Carman: Forty. All right, that's our final round question.

William Morgan: Should the sneezing be higher or lower?

James Carman: The algorithm will tell. We'll see how you did when the algorithm parses through all of it. I don't know how many times I've sneezed in the past seven days. I'd have to guess like a dozen. But I haven't had any allergy problems in the last seven days, so maybe that's the thing. That is it for the lightning round.

Closing out

James Carman: Anything you've got coming up? You mentioned something with Buoyant coming up, but are you speaking anywhere? Any books coming out? A one man show on Broadway? What is Buoyant working on that you'd like to share?

William Morgan: For one, if anyone out there is running Kubernetes, I'd love to talk to you about Linkerd or service meshes. We have a new release out that has all sorts of exciting features, just a week or two old, including support for Windows. So if you're on both Kubernetes and have legacy Windows applications, we've got some cool stuff for you. We're heavily invested in inference serving on Kubernetes, and we'll be talking a lot about that coming up, but I don't have anything to point you at other than, I guess you could follow me on LinkedIn. What an embarrassing thing to say. And that's probably where you'll hear about it. Other than that, we'll be at KubeCon. We're always at KubeCon.

James Carman: Well, it's not Facebook. At least you're not saying Facebook. My daughters cringe if I ever say anything about Facebook. I'm in the group of people that did communicate over Facebook, so you still have these groups and parent things and all that. But my daughters are like, my gosh, that's old people.

William Morgan: I was on Twitter for a long time. I used to say follow me on Twitter, but I'm not there anymore.

James Carman: It's not even called Twitter anymore. X. But we still say tweeting. That blows my mind.

William Morgan: I still call it Twitter, and I shed a little tear.

James Carman: It'll always be Twitter to me. Any books or speaking? You said you're going to KubeCon, anything like that?

William Morgan: I'll be at DevOps Days in New York, I think, next month. Other than that, just working away on fun service mesh stuff.

James Carman: Well, it has been a joy. It was a lot of fun, we had a lot of laughs, it was very informative, really cool stories about Twitter and the birth of service meshes and Linkerd. I really enjoyed this conversation. Thank you very much for coming on.

William Morgan: Absolutely, I enjoyed it as well. Thanks for having me, James.

James Carman: If you'd like to get in touch with us, drop us a line at theforwardslash@callibrity.com. See you next time.