Episode 7: Dan Bentley, CEO and co-founder of Tilt

Transcript

This transcript was generated using auto-transcription software and the source can be edited here.

Beyang Liu: Alright. I'm here with Dan Bentley, the very first full time developer tools engineer at Google and now CEO of Tilt, a tool for multi-service development environments. Dan, welcome to the podcast.

Dan Bentley: Yeah. Great to be here.

Beyang: So to kick things off, can you tell us about how you first got into programming? You know, how, what has been your journey from programming newbie all the way to founding a developer tools company?

Dan: Yeah, so I had just an incredibly privileged background where my dad is a computer scientist. And so I started programming in college, but actually growing up, uh, we talk about, um, Big O or Quicksort, uh, over dinner. Uh, and so I think that really shaped, uh, How I thought. And so when I got to college, I took a class computer science classes, and I learned how to program, uh, originally, uh, at first in, see, over winter break from the book, uh, the C programming language by Kernaghan and Richie Kandar. Uh, and as I took computer science classes, I realized that I was learning lots of interesting things, but it felt like my brain had kind of already been prepared or, or molded. Uh, to think that way. So actually went and was an English major, uh, for my undergrad degree. Yeah, because I was like, Hey, this is really teaching a new way to think, as opposed to just kind of sponging up, uh, things that already fit into how my brain worked. Um, And so I did that. And then I was very fortunate to get a job at Google, uh, in, as I was, uh, coming out of college in 2004. Uh, and so I, uh, showed up there and I think we'll talk about this a little more, uh, in a bit, um, But yeah, I just kind of thought, you know, Hey, I'm going to figure out what kind of product that really matters to end users. I want to work on at some point, but until then, I'll just kind of do things that help, uh, other engineers. And I feel like my career has just kind of been a continuing descent into, uh, the bit mines, so to speak. Um, and that's yeah, kind of where I'm at now.

Beyang: Awesome. And now you've, uh, started a company, uh, it's called tilt. It's a, it's a tool that makes it easier to develop multi-service applications. Um, but in, in the development environment, uh, can you tell us a little bit about that? And in particular, how would you describe tilt to an engineer who has never heard of it before?

Dan: Yeah, I think that tilt is a tool that you might not realize you need because you can do everything you already do without it, it just requires you. Taking care of a lot of things that really should be someone else's problem. And so what we want is that when you come in, you're able to just run, tilt up and then you have your whole app running. You can save a file and, and, uh, the app will be updated and you won't have to think about so much. And so in that sense, it's kind of like a build tool. In that it goes from your code to your app. And I think, yeah, we can talk more about how it's more than a build tool, but really we want to let you focus on programming and let tilt focus on what steps need to happen in your workflow for you to be able to solve the problems you have.

Beyang: Got it. So it's kind of the pain that you're trying to solve. Uh, this pain that a lot of the developers have where. You know, I want to go in, I want to write some code, uh, try it out in my development environment, but you know, every time I save a file, if I want it to automatically rebuild and redeploy, I have to set that up manually. Uh, if there's like some tests I want to run, uh, after making some change, I have to set that up manually. Is that kinda like, um, that kind of, uh, yeah, like write code, what do they call it? Right. Test, uh, deploy that kind of loop. The thing that you're optimizing.

Dan: Yeah, edit compiled the bug. Um, it seems like lots of people have different names for the cycle, but they almost always have, uh, three pieces. there's this tweet I really love, uh, that says debugging is like being the detective in a murder mystery. You're both the victim and the killer. And so we want to let you focus on figuring out why something's happening or how to fix it. So that. You can be the detective. We can be the CSI lab tech. Who's shaking the beakers at the right times so that you can really use all of your brain for the important stuff.

Beyang: That makes sense. Uh, you know, there, there are a couple of. You know, existing, uh, tools that fit into this, that, that come to mind, um, you know, on one end of the spectrum, there's, you know, make, uh, there's things like, you know, proc files, foreman, uh, things like that. How does tilt fit into that picture? Is it a replacement for those tools or is it, uh, somehow adjacent.

Dan: I think it sits above them a little. Yeah. Um, you for a tool, like make. Uh, or Bazell, it's really thinking about how do I go from ELLs too? Another file, a binary or even a unit test output, whereas, uh, tilts really about multi-service development that now my app, isn't just one binary and for decades, it was the case that your app was one binary. Now it's all of these different services that you have to run. Uh, make or even Docker build is about how you go from files to one binary or one image, a foreman or Docker, compose or Gorman is about how you start up a bunch of different things, tilts about how you start them up, how you keep them up, how you iterate on them. Uh, when you change one file, what are the, what's the fewest number of things in the fastest to get the right code running also the feedback. Hey, how do you go from there being 20 different services with 20 different logs to you, knowing where a problem it is. It's too easy in lots of cases to have. Uh, to miss an air because it's flying by, or because they're all scattered throughout to have to play 20 questions flipping between terminal tabs or log files, just to see where a problem is. And so we want you to have a peripheral vision of having a UI that gives you a holistic sense. Tilt. Understands your app and helps you understand it across these different surfaces.

Beyang: That sounds awesome. It sounds, uh, really featureful um, one question I have is, you know, if, if a developer is trying out tilt, what is it? You know, what is it? Is it a CLI? Is it. The GUI, is it a conf like a config file format? Like what, what are the different things that, you know, one would have to set up, um, and use in using tilt?

Dan: Yeah. So the tilt UI, uh, is in a web UI. So it's served on local hosts. So tilt is a single binary. You download it, you run, tilt up and it, uh, opens a tab in your browser where you can see the tilt UI. Until it's figuring out what to do based on the configuration in a tilt file. So like, make file, but for tilt and in it, you tell, tilt about the different pieces of your app. Now that might be. Coobernetti's Yamhill's and Docker builds and the port forwards you want, it might be because your dev isn't in Coobernetti's that it's, uh, this process to run, uh, to compile and this other process to run, to run the service for each of your. And different services. Uh, so you configure it in a tilt file, which is a subset of Python and you see it in web browser. And I'm really proud that we make tilt respond. To what you're doing as you do it. So as you change a file, tilt can recompile it and you'll see it kick off in the browser. As you change the tilt file itself. Tilt will reexecute the tilt file. So just from starting to set it up, you aren't dealing with this. Um, well let me change the file then rerun the tool. Telstra's always there responding to the edits you're making in either your source code or the configuration itself.

Beyang: Cool. And, uh, so diving into the gooey part. Um, can you talk a little bit more about that? Because you know, I've tried out the tool and one of the things that really stuck out to me was this experience of being able to run multiple processes in my dev environment, and then having this nice GUI interface to, you know, kind of see the state and the logs of each process separately, as opposed to just like, you know, dumping everything to standard out, which I think is kind of the. Um, you know, standard practice today in development.

Dan: Yeah, I think it is the standard practice. And it's what you can get when you're building a tool kind of in a hurry because you have a real product to deliver. Uh, for us tilt is our product and we're excited to make it as functional and as beautiful as a consumer product. Uh, my co founder, Nick, uh, worked at medium on their editor, which I think is, uh, pretty featureful and, uh, Oh, maybe less feature for sure. More minimal and really focused. We want to be focused on development. And so standard out is great when you have one standard out because when there's a problem, boom, the problem is for the last thing in the log. Now that we have so many things we just realized it was too easy to miss. The problem was. And so building a UI versus just dumping everything to standard out is an order of magnitude or more, more effort. But we think it's worth it in this new world where you have so many services to have that, uh, peripheral vision where you can see problems, even if they're not the latest thing across the whole system, because you're able to slice and dice and dig into each service.

Beyang: Yeah, that makes a lot of sense. It's it almost feels like the development environment is kind of this orphan in the overall developer experience. Uh, you know, we have so many tools these days that have kind of made. Our lives easier in testing in deployment, but the development environment is still like someone wrote a bunch of bash scripts and they kind of magically spin up the processes and it all dumps the standard out. And, uh, you know, God help you if you ever have to go in and, um, debug anything in, in that piece of orchestration.

Dan: I mean, kind of what we say is that every team has a half-assed version of tilt, whereas we're, uh, putting our whole ass into it and we really think it's a, it's an improvement.

Beyang: You mentioned something earlier that I want to dive into, which is that tilt kind of knows. Uh, you know, it's watching your files, so it knows when those files change and it figures out somehow which processes need to be restarted. Uh, can you go into how it, does that? Is it something that you have to configure and set up or is it something that's able to kind of infer magically?

Dan: yeah. Tilts, able to figure it out in common cases, if you're using Docker build, uh, because you're using Coubernetties or Docker, compose tilts, able to figure that out from where the context goes. What we really want with tilt is that, uh, Simple setups that match, uh, best practices or at least the introductory best practices just work out of the box, but also that whatever your team does, even if it's not what we recommend you do, even if it's not what you're happy you're doing. We want you to be able to do it in tilt because, cause that's what you're doing and you shouldn't have to change everything or throw it out just to start using tilt. And so we can try and figure it out, but you can always tell us more about it. And if you find a situation where you can't, where it's too limited, please let us know because we want to be adding more, those hooks for more complex projects.

Beyang: That makes it, that makes a lot of sense. Um, so you've been talking about tilt. You've been mentioning Docker and Kubernetes a lot and totally makes sense. Cause these are kind of the. Ways that, um, you know, modern service oriented architectures are microservices, architectures, uh, you, uh, deploy applications. But what if, you know, I don't know, use Docker in my development environment or what if I deploy, uh, many services, but I don't actually containerize them is tilt still applicable in that case.

Dan: Yes, it certainly is. We are working on it and we may still have some rough edges that we want to fix. And that we'd love your feedback on how to fix what we think is that there. Our smarts that are Coobernetti's specific until, and that's maybe 20% of it. The other 80% is just dealing with, as soon as your app is more than one binary, how do you develop? And I think what we've really seen is the. That having more systems, like you mentioned to operate multiple services has led more teams down that route that as you have more CIC, D more tracing and observability, and there's a lot of reasons to prefer multiple services. You can have different teams release on different schedules, but. But for development there hasn't been the same kind of innovation and tools. And so I think kind of the tail has wagged the dev dog because we've been so able to do with an operations and in production and staging. We haven't done it in dev that time from when you. Create a new branch until you have a pull request ready to go, where you want feedback in less than a second, you want your compiles or your unit tests to be super fast. That's where tilt can fit in. And so absolutely. If you're. Uh, just running processes because you have some shell script or some Wiki page that tells you what different servers to start. Absolutely. Absolutely. Tilt can, uh, still work with that. Using a feature called local resource.

Beyang: That's a that's very cool. Um, you know, comment earlier about the ops tail whacking wagging the dev dog? Uh, yeah, definitely. Uh, strikes home. Um, you know, there's some developers I talked to who have just had enough of the whole, you know, microservices trend and Docker and Kubernetes. Uh, w what would you say to the folks who, you know, say, you know, we've gone too far. It's time to go back to model is, um, most applications, especially if their crowd can be run as a single binary. Uh, do you think that a they're completely off base or, uh, you know, w what is your, what are your thoughts on that?

Dan: Yeah, I think I talked to that. Say that the more I dig in. The more, it's not quite true. Yeah. We have a monolith, but we also have, we have a database. We also have redness. We also have this binary and go for search or that elastic search. Um, there's, there's some exceptions that don't appeal to. Most people just set it up once and you don't think about it, but it hurts your onboarding. It hurts, uh, developers who do edit that code. It hurts when you try and make changes that don't fit into the monolith. And so I think people who say we should only run one. Are like, yes, you should only have one within the incentives that we've had. And so you should give tilt or some of the other tools, uh, you know, I'm happy to, to mention, to try, because being able to do all of this. It's what we've had within the Malo with. Right. We've had libraries you've been able to add functionality that you didn't write into your, by way of a library. Yeah. And so tilt is kind of taking that to a slight, the higher level, uh, which is why it kind of sits on top of a make or a Bazell to say now the way to add functionality. Isn't another library that you have to fit into a binary using something kind of weird FFI or like C you know, include something. And I, you can just say these are slightly different pieces and I can still develop with them. Without having to know that to me, that's the key that the cost of having multiple services has been. You've had to think about them, update them to bug them if we can automate that. And so that to you, it's something you don't think about the same way you don't think about the library is included in your binary until you're trying to improve them. Well, then why do you care?

Beyang: you're, you're kind of saying that, um, a lot of the people who want to go back to the monolith it's because the monolith is kind of this, uh, established model and the, the tools and. Practices were kind of mature around it. And the multi-service model is something that's newer to a lot of people. And, um, it's going to be painful at first, but if we make that experience better, there's going to be more, more and more people are opting for that for kind of the right reasons.

Dan: Yeah. If, if I think what I'm even saying is if you can not feel the pain without giving up the benefits that attracted you to it, that's the best of both worlds. And so we're trying to reduce the pain you feel with tilt.

Beyang: cool. So taking a step back a bit. Um, can you tell us, uh, about how you got started on the road to tilt and, you know, maybe go back all the way to, to when you first started your career, uh, you know, in conversations before you, you know, you mentioned that you were the first engineer inside, Google working full time on developer tools, um, back in, you know, kind of the 2004, 2005, uh, time period. What, what was that like?

Dan: Yeah. Uh, when I joined Google, Everyone had a day job of shipping some product. And there was this notion of, of intergroup splits. There was your group where you were doing your work and the intergroup split. And so one was about documentation. One was about, uh, testing. One was about, uh, Developer tools and there were a bunch. And at that time, Google was realizing we need to actually vest in these things that are crucial to engineering to actually staff them up with full time engineers as the org was growing. And so I was working on developer tools and I'll be honest, there was. A cost to that of not working on production systems full time means that you lose some of what the means our day to day. You get this kind of stilted view. I think that notion of there's your work doing app dev. Where your, uh, actually program, what your users use that run in the data center versus being developer experience engineer, where you're working on the developer experience that every other developer has on their laptop. At the time desktop, uh, it was, it's really key to invest in as well at a certain size. Um, it was amazing to see how quickly things were growing. It wasn't enough just to make, you know, the build tool faster, uh, when. The whole code base was crowing so quickly that you had to not just make it faster at the rate that the code base was growing, but even faster than that in order to actually have users see an improvement. So it was stressful, but, uh, exciting. I'm really, uh, fortunate, I think, to have been there at that time.

Beyang: That's that's really fascinating. Uh, cause you know, today Google, uh, from an engineering engineer's perspective is known for having just this wonderful internal environment. Uh, and. You know, all sorts of developer infrastructure that make them kind of best in class in software engineering. But what you're saying is, you know, back in 2004, that wasn't yet the case is that right? It was all just kind of informal grassroots. Like these, these group let's, um, uh, sprung out of what engineers felt was important. And they started working in their 20% time on it. Is that right?

Dan: Yeah. I mean, I think a lot of the, the tools that you now think of as really great came about a bit later, maybe in the 2005 to 2010 timeframe.

Beyang: that's fascinating. So were you around, so a couple of tools, uh, that I'm aware of? Um, just because I, I did a brief stint at Google and they've also talked publicly about, these are, uh, you know, Google code search, which. Searches across, uh, almost all of Google's, uh, code base and critique, uh, their code review tool. Uh, were you around when these were first created?

Dan: I forget exactly dates. Um, but certainly as they spread out, um, code search is amazing. And, um, making that public, uh, led to a lot of Russ Cox's work on, um, Are you to a great library for regular expressions, which is just fascinating to read about, but I think really the idea that you can just see everything in your, in your scope is so important when you also had this notion. Of being able to change anything in your scope. Uh, that part of what made build so difficult at Google was you could go and build, uh, anything you could change down to kind of the common libraries that had the hash map that you use in, in most other places you're using so much. Library code that you just consume as a prepackaged, uh, Executable files or jar files, but at Google you could really change the RPC library, which you didn't need to that often. But when you did it, you really did. Or when you were working on it. Yeah. You could also go build the deck Gmail binary that was having the problem. And so it just led. Gave you this freedom to understand things across your whole app, which I think is really, really powerful. So code, search. Yeah. Uh, amazing. And I'm sure, uh, you'd say so too. Cause you are dedicating a lot of effort. I think there's code search as well as deeper understanding of the code. Um, At Google that was called Gras, G R O K, which has been open sourced. I forget the name of it now. yeah.

Beyang: is opensource name.

Dan: Um, there's a lot of different things of a language server protocol that's trying to get at. How do you, you know, make a UI that doesn't just let you search for tokens, but actually search for, you know, just this function here where all the places it could have been defined because of, uh, Inheritance. Um, so that's even more than just the code search. That's some more understanding of it, a code review. I mean, I think some of the ways that you really want to do code review interactions, Google was fantastic at, um, you can see it now in Garrett, which is one, uh, open source. Um, Tool, uh, that's similar. I think even GitHub pull requests, still quite aren't at the level of, you know, I want to, uh, switch between multiple different versions that were uploaded. And how did the review progress, which is something that, again, I don't care about that often, but when I do, it's amazing.

Beyang: Can you, can you talk a little bit more about that? Because that's something I hear from a lot of people inside Google, or who've worked at Google that, you know, when they compare it to, um, you know, publicly available tools like get hub PR is, uh, it just, they really miss, uh, the internal code review tool that they had inside Google. What, what was so special about it?

Dan: Yeah, I think, uh, So one piece is going back and looking two months later, when you're trying to chase down a bug. And you're trying to understand how did this happen with a gift and pull requests. We often squash and kind of lose the history of what were the intermediate steps. Whereas sometimes being able to really dig in to why did this change happen? If I just look at consecutive. Commits in the actual history, it seems like, you know, you wouldn't do this well, Hey, there's another, you know, that's kind of the chapter by chapter, but the page by page of the code review, helping you understand how it happened can help you understand how to avoid it or where else it might be happening. Um, because you can understand that, you know, Hey, this was a long code review, and this was getting kind of a realize that the last minute in a smaller scope than you need to think about this larger, or, you know, this, uh, issue of interaction between different systems. I think it helps helps you get empathy, frankly, to understand how things can occur.

Beyang: So it, does it really come down to that kind of get flow like with GitHub PR is you're kind of pushing up to a branch and because of that, you're often, you know, force pushing or squashing changes, whereas with a Garrett or Google's internal tool, um, if I recall correctly, you're not really pushing to a branch at all, you kind of have this like. Fake, um, uh, head that you're pushing too. And then Garrett kind of keeps track of like, Oh, this is the first version of the patch said, Oh, you push, push another. And then it, it is kind of smart enough. It's not just showing you the diff between, uh, the, um, head commit of your branch and the base of master, but it's actually keeping track of like each subsequent revision you're pushing. Is that kind of the, the crux.

Dan: Yeah, I think just, um, GitHub. Frankly has been amazing, getting more people to do open source development. And I think you go where your users are. Google's internal tools are built for what's it like to work on huge systems, you know, with teams that are around that you can count on people, you know, I think, um, A lot of what get hub really got. Right. Was you can fire off a pull request. And if someone doesn't reply to it, it's no skin off your back because you have a fork. The flip side of it is they built things that really encouraged the simple cases too, you know, get you more of a sense of community. I think they didn't have as much of that. What's it like, you know, working 40 hours a week working with so many code reviews at a time that you need more. Uh, accountability, more dashboards just to know what all of the different things in flight are. Maybe I'm missing stuff. I, it's still easy for me to miss all of the different pull requests that are, uh, in different repos on kit hub. Yeah.

Beyang: can you tell us about some of the developer tools that you worked on directly at Google?

Dan: Yeah. Um, I worked on a tool called, uh, Mark M a C H uh, cause it was a build tool that was trying to be as fast as the speed of sound as the speed of sound. Um, and that was for Java developers and I'm really proud of some things I learned there in terms of. Tool Smith thing where, uh, It's easy to blame a tool for slowness if it's a black box. But what I realized is that lots of the reasons, um, that MOC seemed slow was because of how people had configured there. Clients and they weren't including enough code. And so this is a little trick of the trade I just had. The first time Mark was going to read, build, say, it's because of this file that changed and that little, uh, change. Let people know, Hey, I'm depending on that file. Why am I depending on that pile and either go change it. So they weren't depending on it or kind of going, and, and, uh, they, they knew what to do. They were able to take that and make some decision off it. Uh, that's one tool I worked on, um, Another tool that, uh, did unit tests, I think today we'd call it CGI, but that wasn't a, a huge term back in 2004, uh, at the time Google's nightly build was starting to take more than 24 hours. So it wasn't really nightly. I did it in parallel across a bunch of computers in the data center and it would take 40 minutes and that was. Technically super impressive. And I realized not as useful as it could have been because it was too much focused on bandwidth across all of the, uh, system, all of the repo, as opposed to really answering questions people had about, Hey, for what I'm working on. Is it green or red? And so there was another effort, uh, internally that was, um, uh, Hmm. More impactful that I really learned from, um, I ended up working on the open source team. How do you get a bunch of the great code that Google has made available in, uh, the world outside Google and tried to figure that out. And for awhile, it was kind of consulting, uh, with teams that wanted to open source later made tools that helped you open source repeatedly. Uh, That led to a Google code, the project hosting, uh, on code.google.com, which

Beyang: wow. So were you kind of the instigator for Google code?

Dan: Oh no, no. Uh, Google code, um, uh, Krista bona and Brian Fitzpatrick and Ben Collins, Sussman. Um, The latter two were very involved with, uh, uh, subversion and Apache. And so they set that up. By the time I came to Google code, it had been doing project hosting for, uh, years and had both a sub version and mercurial support. Um, I joined it in, I don't know, 2011 or so. Um, and you know, I thought it was really great because, uh, Uh, at the time SourceForge was falling over and it's important that Oakland course code has a place to, uh, live. And so Google code supported, um, you know, repos and big repos and had issue tracking and a descendant of that issue. Tracker is still what, uh, Google Chrome uses for issue tracking, uh, today. Um, So it really helped teams work on software. There were ways to copy stuff around, but it didn't encourage forking. And that's really what get hub I think was just fantastic at was saying fork isn't a four letter word up until then it had been a, you know, um, you know, taking my toys and going over to this other place. Is it really a community desolving whereas GitHub was about how it makes the community stronger that everyone can work to gather without having to ask them permission all the time, till then you kind of had a committers and they could do whatever. And non project members were stuck waiting. Um, and get hub, you know, with the abilities that get offered, um, made a model of, uh, being able to do what you want. And that was fantastic.

Beyang: Kind of. Comparing the two, you know, Google code. I used it heavily during my college years for things like, you know, working on projects with friends and sharing a code for assignments, group assignments. Um, comparing that to the GitHub experience. Are there things that you like about one that you wish the other had had.

Dan: there are specific things about the issue tracker, uh, especially as they relate to, um, Privacy, uh, and, and being able to hide things that were incredibly powerful, um,

Beyang: This is in a

Dan: in Google code. Yeah. Um, I mean, I think gift hub, uh, has just done so much now that it's, and it's hard to remember exactly when they did what I think. I think GitHub still could be more focused on, uh, You know, teams where like you are working together and you kind of are sitting side by side. Um, but you're also doing stuff on, for instance, an open source project where it kind of has that attitude of it's volume tier and like, no, we're, we're working together. Like yeah, help me know that. Um, That I want more from tools. Uh, I joined Google code. I think a lot of what I was excited about was getting more internal Google tools out into the world. And, uh, for various reasons, I don't think we did a great job of that. I'm excited that there are now companies, for instance, source graph. They're trying to bring a lot of, uh, cool tools, uh, That massive companies have had internally to make it available to anyone who wants it.

Beyang: Are there any other tools that you had inside Google?

Dan: yeah, I think, I mean, I think there's distributed build. It's always nice to have your bill be faster. People kind of think about that. But to me, one sea change Google that I saw was, uh, there was a tool called spot internally and it had all of your builds and tests. And so the sea change was when you went to ask for help. You didn't just describe the problem. Hey, is anyone else seeing a problem with a backend number three? Uh, instead you'd say here's a link that has the exact test failure, um, and a lot of different context for it. And it just saved you from having to play this back and forth game of telephone. Using a coworker as, uh, the world's kind of, uh, highest latency and most error prone shell to like give you data. You instead had all of that bundled up. I think that's a lot of what we're excited to do with, uh, the tilt cloud aspect of letting people share context. So that. Getting people, unstuck and productive again is a matter of sharing a link, not a matter of trying to describe everything again. So I miss I miss sponge in that way.

Beyang: That would actually be super cool. So what you're describing is, you know, if I have some bug in development, instead of copying a bunch of logs and, you know, sending someone a chat message, I could just. Say, Hey, here's the link to my development context and all the services are running. You have all the context that I have. Can you help me figure this out? Is that

Dan: yeah. Yeah. In fact, it's just unfair, right? Like when you need help is kind of when you're often the least able to know what's helpful. So, uh, what we kind of already have and is coming out soon for teams is this notion of a snapshot where when your Intel. You can click a button to get a link that instead of being served off local hosts, that only you can see because it's running on your laptop lives on cloud dot, tilt.dev, and it's the same tilt UI that your team knows with exactly what you were seeing. And so you can just paste that link into Slack. Um, so instead of saying, Hey, is anyone else seeing import errors in the shopping cart service? Uh, Someone else on the team can just click that link, see the import air and what it is and go, Oh yeah. That's because, uh, Jeff merged a new change that changes, you know, packaged shot, Jason, uh, run this command or click this button in the tilt UI and you'll be good to go.

Beyang: So it's kind of hard to talk about build systems at Google without mentioning blaze. And it's a open source counterpart, uh, basil or basil. Um, can you talk about how the tools that you've discussed so far, Mach sponge, how did those relate to blades?

Dan: so Mark was a predecessor to blaze. Um, And blaze replaced it and is fantastic and has all kinds of distributed build goodies, um, that, that MOC never had sponge, uh, Was a, almost a backend for blaze in that. How do you know what test failures are? Well it's whatever blase says a test failure is. And so one of Google's great strengths is just, uh, there was so much respect for. The engineering leadership that the top down of you will use blaze, or you will use a, what became GRPC were all right. Cause they were good decisions. And so there was so much commonality that you took build a thing like sponge didn't have to build integrations with any individual sure. Build tool. You just. Built it into blaze and it worked for everyone at Google. Um, and so, uh, blaze, uh, I think, I think it is fantastic. I think that it is like the culmination of, you know, 40 years of development with make that Stu Feldman wrote, you know, a kind of mildly complete version of in one weekend. And. You know, it's gotten better and better. And now blaze has had like sweet millennia invested in it. I think of it as like CERN, like it's a, it's a $15 billion particle accelerator of like smashing files, like dots and Don each files into, you know, the carbon atom of a library. Smashing those. Uh, to become like an iron atom of a binary, uh, putting them together with others to become a uranium atom of a container image. Um, I think, yeah, and it does it at huge scale and, you know, really effectively.

Beyang: I guess it does. It really lives up to his name. Doesn't it.

Dan: Yeah. Yeah. Um, whereas like, Tilt on the other hand, like we can't compete with P ISIL in terms of doing that. We're kind of saying, Hey, we take you, you know, the different atoms that your build tool built each for a different service. And we combine them into your app, which isn't one atom anymore. It's like a molecule. And so we're not a $15 billion particle accelerator. We're a $15 chemistry set where you can put these things together in a fundamentally different way. Not by like building them or linking them together, uh, at compile time, but of making them one, living, breathing, distributed system.

Beyang: Yeah. Before, before you can test your grand experiment on, you know, the world's largest particle accelerator, you got to go through many iterations with the home chemistry set. Right.

Dan: Yeah.

Beyang: So then do you think that so basil is the open source version of blaze. And, you know, one question people always have is, are they essentially the same or are there differences? And, um, you know, if they are kind of the same thing, do you foresee that basil will eventually kind of take over the world? You know, in like five years, 10 years, is everyone going to be using basil is their, their build system.

Dan: Yeah, I'm honestly not sure. I think they. Are pretty much the same, uh, in terms of how you configure them, what they're able to do. I think one difference is that you probably don't have as much of a distributed build cluster outside Google. And there are efforts to change that Google has a remote build execution, uh, offering. I've heard of some other startups trying to do it. I think the larger question of does it take everything over is, well, your constraints are different. Uh, Bazell requires a lot of, uh, strictness and it offers a lot of benefit of being able to parallelize everything. That's great. If your problem is slow builds, if your problem is I'm trying to get something up and running. I think it's very intolerant of how, uh, w whereas as, you know, lots of times, right. If you could, uh, make something 10 X faster, but you know, one out of a thousand times you have to, it set it afterwards. That's okay. Probably worth it on most teams. And so I think, um, Baseball is fantastic for what it is and what it is, is a crystallization of the constraints and a specific set of trade offs that made sense at Google in that 2005 to 2010 timeframe. Whereas now, Builds are in main cases getting faster because binaries are getting smaller as they're becoming, you know, if not microservices, at least, you know, Mila services. So I think there's a lot of value to standardizing on a tool across a company, but the trade offs that you have to make, uh, I, I don't think it makes sense for every org to adopt Bazell. It still might be the case that everyone does because nobody ever got fired for buying IBM, who gets fired for copying Google.

Beyang: So after you left Google, you were at Twitter for a time. Can you talk about how the transition from Google to Twitter changed your perspective? Uh, if it did so on developer tools,

Dan: Yeah, I think Google has developer tools kind of figured. Out or at least they think they have them figured out they have a perspective, like Baisil is a hammer. And so things look like nails. They knew what they wanted development to look like. And in order to get closer to that, they needed to have a vision and to be able to bring down into, you know, roadmaps of years that teams of dozens can work on. And the result has again, has been incredibly effective. Towards that model. I think I kind of joined Twitter to think about different models. Um, I feel like we've siloed into like source control versus build versus test and, uh, not gotten to the same, uh, Point of, Hey, what do I do as I'm editing? I just want it to be fast in this new world. I can't join Twitter to try some different stuff and I wasn't there that long, but I think I learned a lot just technically in other ways, uh, I was really impressed with how Twitter thought about functional programming in. A distributed system, like not at the level of what does this definition of the factorial function look like, but actually, how do we think about it? The paper, your server function is just amazing. And, and I think their models are so different. If you've been at one place for a long time, I highly encourage you to think about what you can learn from other places that just have other, uh, dogma. That was, that was cool.

Beyang: Can you talk about how tilt began? Uh, so where did the idea and inspiration come from and when did you decide, Hey, you know, let's go all in on this.

Dan: Yeah, my then friend now, cofounder, Nick and I had always, uh, talked about developer tools and he was looking for a new adventure. And so we kind of set out originally just the two of us, no company, just renting desks in a coworking space and exploring how cloud. Can help that inner loop developer experience. And there were a lot of ways, uh, we think it can, and we built a. An early iteration and MVP of something. We called live integration. So like continuous integration, it runs your builds and tests, but it's live. So actually each time you save we'd upload just the diff and run your builds and tests in the cloud in parallel in seconds. And luckily I'm super proud of it, but when we went out to market and, uh, To talk to people about it. The feedback we got was, yeah. I've had times where this has really been an issue for me, but nowadays I'm more dealing with a container Jay's and Coopernetti's. And so we kind of realized that our vision. That cloud can help the developer experience. What was running smack dab into the reality that cloud has already hurt the developer experience, uh, in all the ways we talked about. So that really, uh, Let us to throw that out and we still have a repo with all the code. I think there's some really impressive technical work there that I hope we'll get back to at some day. But for now, we're really just trying to get to this, uh, how do we help people develop and, uh, on an individual cloud app. And as we were talking about some of the multiplayer, how do you share that state and get better?

Beyang: That makes sense. You, you talked a little bit about, uh, how the development environment got worse, uh, due to the, the shift to. Cloud, um, you know, from where you stand, you've thought about this a lot. Um, the dev environment, what things have gotten better and what things have gone awry over the past decade?

Dan: Oh, yeah. I love it. That you ask the hard questions. Um, I think IDs have in lots of ways gotten so much better. Um, there's, uh, you know, I think just much better knowledge of how to jump to. References, um, how to use that, I think, um, Oh, and really, it can depend what you mean by, um, development environment. Uh, specifically, I think people's use of gift has gotten a lot better, um, either because they have some rappers that are companies use, but like people are more able to share things. Um, and. You're able to take a look at things more easily in web UI. Um, where have things gone awry? I think it's, uh, I mean, I think, I think one blog post that we maybe don't talk about that much anymore is the jolt test. Do you remember this at all? Um, Joel.

Beyang: test, like the 10 things that your software organization needs to do in order to be a good place to work

Dan: Yeah, yeah, exactly. Um, you know, that, that was posted in 2000, so 20 years ago. Um, and I think if you go back and you revisit it, lots of things, uh, are really, um, Taken for granted now, like, do you use source control, but I think can you make a build in one step? Well,

Beyang: answer is no, it's always never no. Or never. Yes.

Dan: Yeah. And so I think like that's, that's part of it is how do you just come in one day, you know, with a monolith, you didn't have to know what all libraries there were now, but now. You hire someone to be a front end engineer and suddenly they're having to also go to Kubernetes. Um, that's not great. I think, um, What's team configuration and what's individual configuration can there's something there that we've lost at least a little of, in terms of, you know, how do different people, uh, set things up? How do they use it? IDs, I saw a Twitter debate about is vs code an IDE or not. Um, because kind of IDs originally were like, they're opinionated about everything. Um, I'm not sure, completely sure how I, how I feel about it. Um, I think, you know, how do we make it, that people are able to develop? They know. What they can do, you know how often, and have you been looking over the shoulder of someone and you've been surprised by what they're able to do, or you're surprised by what they're not doing that you know how to do? I think, um, That almost is like a Metta response to your question of like it's so individual what the developer experience is like. And that itself is a problem that it's not easy to see what others do or learn from them. Except when you happen to pair

Beyang: yeah. Yeah. One, one thing that people are talking about now is cloud IDs, um, especially as a way to standardize the development environment. And, uh, if I could be, uh, a little bit provocative here, I'd like to get your take on that trend. So, you know, from your perspective are cloud IDs, a para I am shift, uh, you know, are we all going to be using them in five to 10 years or they a passing fad?

Dan: Yeah. So I think, um, One, I certainly have at least a bit of a conflict of interest here, which might make it, you know, more interesting, uh, might make it, you know, uh, less correct. Might at least make it, um, more provocative. Um, you know, we've,

Beyang: the conflict? Is it that tilt is for local development. Okay.

Dan: uh, I think it's like tilt, uh Hmm. I think the conflict is that we say we want to help move your development to the cloud. And we aren't starting with an IDE. And why is that? And I think it's cause to us, there's. The text that you edit and how do you edit that? We worked on Google sheets and Google docs. And so we kind of know what it's like to build an editor, uh, in web technologies. We instead started not with the text that you're editing. But the workflows, once you have, you know, bytes saved in some file system on disc, what do you actually want to do with that? How do you mean no, that they're the right words in some programming language. We think that unlike with, uh, Documents like Google docs, it's way less, uh, in that inner loop about what do other people think, which is what web collaboration makes easy and way more about what are the workflows. And so till we. Uh, want you to be able to use it and to use that, to standardize your developer environment, whether you run it, um, on your laptop, talking to a cloud cluster on your laptop, talking to a cluster like mini cube, or like we said, the foreman, whether you're running it. Um, in a cloud VM talking to a cloud cluster, there are all of these kinds of things. Uh, and I think almost a de lamination occurring between web IDs, between all IDs, really, and how they, uh, effect with an E how they. How they operate is maybe less ambiguous the workflows that you want to do. You're seeing with language server protocol, the IDE, he knows less about the language, uh, because it's farming that out to this language server. I think more of that has to happen with IDs saying, Hey, it's not about running this one, binary or debugging, this one binary. Um, it's about what are the different workflows and what's the menu that's available. And we kind of think that kilts a great compliment to that and that, uh, I don't know. I think about Gmail, Gmail really changed what, what mail was like, but, you know, nowadays, most people access Gmail through mobile apps. So how often do you really use the fact that you can get it from anywhere? And so like, what's. What ends up being, you know, the key part of an idea or a change, um, can reveal itself differently 10 years later. So I think that what's important. He is less the, you know, I believe that Pete, we're going to figure out how to edit in a browser and it's more, once you've edited in the browser. How do you know that's right. And I think the answer isn't just, cause that's too slow, you know, YouTube has for years had this, you can go in and proposed small edits just in the web, but it's really, how do you make that fluent? And I think that the answer isn't. Going to be just, I happened to have a shell in some cloud VM, because you still have so much to get up to speed with, to actually do interesting things, like start your app, or we see the database. And so we want to fit into that. Tilt, because we think that otherwise it's just a bunch of, uh, using Wiki pages to know which shelf scripts to invoke with which arcs. We want to be a workflow in that teams find so useful that it's the obvious answer for how to help new contributors and what kind of stuff. See what the popular UI is, are to make the. Edits to the text, which is a huge problem. And which I think is why you're kind of seeing this separation between the IDE what's the Chrome of it. And what, and where do the workflows themselves happen?

Beyang: in 10 years, do you think that you'll be using a cloud ID as your primary editor?

Dan: Oh, I'm embarrassed. I'm scared to admit this. I still use,

Beyang: my answer too, if that would help.

Dan: I still use Emacs. Um,

Beyang: Me too.

Dan: yeah. Um, uh, Which can be, you know, quite the IDE and, you know, just has a level of, you know, not just configurability, but customizability of how you can, you know, put, put in your own workflows in a way that I haven't seen in other editors, it's so much to make an extension in vs code. Whereas an Emacs is just. A list, but, you know, will I even be programming 10 years? I think of that quote of like, you know, I don't know what, uh, the high performance programming language of 20 years from now will be like, but I know it will be called Fortran. You know, there's kind of things that are always constant and, you know, even as they change, uh, I don't know. And I'm frankly spending all my time thinking about. Wherever you're typing something. How do you know it works? How do you know what works for others? How do you know what the problems are as quickly as possible? Um, that's, that's where I'm at in the stack.

Beyang: Yeah, that makes sense. Okay. Uh, another, uh, kind of trend or buzz word that we've seen lately is, um, serverless, uh, . So AWS Lambda, uh, Google cloud functions, CloudFlare workers, and, um, a bunch of other, um, Uh, infrastructure providers are now offering people the ability to run code, uh, with minimal deployment configuration. Um, first of all, you know, do you, you see that becoming a, a big thing in terms of how people deploy software and if so, how does tilt fit into that picture?

Dan: Yeah, I think it's definitely becoming a big thing. The question is, does big mean a 0.1%, which already is huge or one or 10 or 50 or 90 or 99.9. Right. Um, I think that, uh, really tilt. Uh, about being an engine that understands distributed systems and giving you a holistic view of it with the UI. Uh, You definitely want it in a situation with server lists, where there are more pieces to think about where, you know, changing one thing, there's more nooks and crannies that can go wrong. If anything, it makes our job. Uh, more essential and harder as you actually, you know, think about, I'm not just changing this one, function, my credit app to save it, but what actually happens when I save it and then my next serverless function reads that value. When there's problems across that, how do you deal with it? I think, uh, people don't know what's going to happen with Coobernetti's. Um, will people use Coobernetti's will they use their company's layer atop Coobernetti's will they all use one common layer on top of Kubernetes and, uh, I think that's a really interesting question that I don't know. And almost like with tilt what's important to us is that we're, um, you know, Help writing the freak Treme of Coobernetti's not trying to lay track in front of it, right. That we're not trying to do something that Coobernetti's could. And, uh, I think, uh, thankfully for us, the Kubernetes project is kind of clear about what the sounds are serverless. What do you want? Difficult to figure out, I think of get, get has this idea of plumbing versus porcelain that you could put different UIs and by you, or I might mean a command line interface, but with different, you know, verbs and, uh, on top of the get, um, internals and none of those have taken off. Like, there's just something about, uh, having some interface and having it be common that makes it hard to build something on top of it. I think of worst is better. Um, this amazing paper by Richard Gabriel, uh, about how kind of C and C plus plus and Unix are good enough. And he was this old common lisp packer. Uh, Emacs, uh, person, uh, to call back to our shared, uh, love apparently. And he's kind of the moaning, like why aren't people rebelling that all they have is C and Unix and he gets at like, kind of what makes it such an effective virus to spread and be impossible to replace because it's wormed its way in. I don't know. And I think I'm kind of hedging my bets by trying to be in places where that doesn't matter, no matter what you use to deploy, you're going to have to think about the different functions you're writing, whether they're one binary or 10 or 1500 to figure out what the overall app is. That's what. That's what I believe. That's all I have to believe to make something that really helps a bunch of people. And so what you're asking is kind of a over the horizon for me, which might make me a less interesting podcast guest, but I hope makes me a more effective, uh, tool Smith.

Beyang: No, I think it makes a lot of sense. It sounds like what you're saying is, you know, whatever happens on the ops side of things, you know, whatever crazy new thing pops up the next five to 10 years, you know, like a microservices nano services, you know, millions of cloud functions, whatever, uh, you know, that's all gonna happen. The people who make those are going to be really focused on the deployment environment and. Tilt, it sounds like is going to be the company that takes all that and makes it so that developers don't go crazy with their development environment because Lord knows, uh, things might get really bad if, uh, you know, we continue down the road that we've been on.

Dan: I certainly hope so. That's what we're setting out to do. Uh, yeah.

Beyang: awesome. And as kind of a final parting thought, you know, if someone is listening to this and they're thinking to themselves, tilt is something that sounds really neat. I want to try it out. How would one get started with it?

Dan: yeah, go to tilt.dev. Um, and you can download it. Uh, there's lots of different. Uh, content there for how to get it set up. Also throw out that, uh, I'm on Twitter at the Bentley. You can always reach out to me. You can go to, uh, calendly.com/d Bentley, uh, to, uh, schedule some time if you want to investigate it. I think, uh, if you're an app dev tilt can be a really good experience. Really the most value is when. Teams are able to adopt it. And then they're able to take advantage of all of the, uh, hooks we have built in to help it, uh, proliferate across a team. and be more. Effective. So if you're in that developer roll, uh, definitely reach out. We have a channel in the Cooper, these Slack for tilt, it's all open source. You can download it, you can download the source, you can use it, uh, always, uh, for free and we're there to help you figure out how.

Beyang: My guest today has been Dan Bentley. Dan, thank you for being on the podcast.

Dan: Thank you for having me.