Rijnard van Tonder is the creator of Comby, a pattern-matching syntax and command-line tool that offers a more expressive and more user-friendly alternative to regular expressions for many common patterns in code.
Rijnard earned his PhD from Carnegie Mellon University in 2019. In this podcast, we chat about the state of the art in static analysis and automated bug-fixing, new tools made in industry like Pyre and SapFix, and what place machine learning has in the world of developer tools.
Comby pattern matching syntax: https://comby.dev
Comby Gitter channel: https://gitter.im/comby-tools/community
Program synthesis: https://en.wikipedia.org/wiki/Program_synthesis
Tree sitter: https://github.com/tree-sitter/tree-sitter
Codemod, from Facebook: https://github.com/facebook/codemod
Automated bug fixing and program repair: https://en.wikipedia.org/wiki/Automatic_bug_fixing
Chomsky's hierarchy (regular, context-free, and Turing-complete languages): https://en.wikipedia.org/wiki/Chomsky_hierarchy
Regular expressions, or regex (pronounced "REG-ex" or "REJ-ex"): https://en.wikipedia.org/wiki/Regular_expression
Spoofax Language Workbench: http://www.metaborg.org/en/latest
Infer static analyzer, from Facebook: https://fbinfer.com
Rice's theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem
SapFix, Sapienz, Mark Harman from Facebook (note: in the recording, we mixed up Sapienz with SapFix. SapFix is the end-user tool, Sapienz is an underlying technology which can be used to surface issues to SapFix): https://engineering.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/, https://engineering.fb.com/developer-tools/sapienz-intelligent-automated-software-testing-at-scale, https://research.fb.com/blog/2019/05/spotlight-session-with-mark-harman
Pyre, from Facebook: https://pyre-check.org
This transcript was generated using auto-transcription software and the source can be edited here.
Beyang Liu: Alright. I'm here with Rijnard van Tonder, an engineer at Sourcegraph who created the Comby pattern matching syntax, which is kind of an alternative to regular expressions, but much easier to use and designed and optimized for use in a code. Rijnard, welcome to the show.
Rijnard van Tonder: Hi Beyang. Yeah. Thanks for, uh, thanks for having me on.
Beyang: So I guess before we dive into, you know, all the technical stuff, uh, if that'd be interesting to go over kind of like the backstory of, of how we met each other, uh, do you, do you remember how, how we first met.
Rijnard: Yeah. So I think you might have to fill in some gaps here. I remember my side of things. Um, basically, you know, I was, I was in grad school, um, at CMU and working on in our research problems, taking classes, but at the same time, you know, curious to kind of explore other things, uh, as, as you do in grad school. So, um, at one point I, I, uh, Sourcegraph came up on my radar and I kind of, uh, Uh, I started playing with it. I thought, you know, code search, this is cool. Um, but also is kind of poking at security holes, um, trying to try to find some fun things. Uh, th there wasn't, there wasn't much, I think, uh, you know, the basic stuff, uh, was covered. Uh, but, but at some point I found like, uh, you know, you can have like an information leak, uh, for, for public and private repos. And that's when I reached out. And I think you were the person who replied to that email. I think.
Beyang: Yeah, I think he said it to security ad Sourcegraph, uh, which is our kind of a catch all for any sort of security bug report. And yeah, I did reply to that.
Rijnard: Yeah. Uh, and I think, uh, I think that that kicked off like, Oh, you know, do you want to talk about, you know, what Sourcegraph does? Do you have any feedback about the product and, and so on? And I, I remember hopping on a call with, uh, with a couple of folks back then. Um, And, uh, and I thought it was interesting, you know, you're talking about kind of a hard problem that, uh, uh, there's a bit of a gap, right? Um, the reason I came across, so Scott is kind of, well, you know, cut search that, that works, you know, a little bit more ubiquitous ubiquitously than, than anything else out there. So
Beyang: Yeah. Yeah. I remember that. Cause uh, I remember looking you up when you reported the bug. Cause it was, uh, it was a very nice, uh, bug report. Uh, it was for, you know, like an issue that was indeed, uh, an issue and it was like, well kind of described and uh, Uh, it was a good report. And so I looked you up and saw that you did, uh, research into software engineering, uh, developer tools, static analysis, like that kind of general area. And, uh, afterwards we kind of struck up a, an E we were kinda like email, um, uh, buddies for a while. And then I think at some point you came out to California cause you were doing an internship at Facebook or something like that and you dropped by the office.
Rijnard: Yeah, that's right. Uh, that was a, that was a couple of years after I think that was at least two years, two years after we had the like email exchange. Um, and, uh, yeah, I was just there, you know, and I was like, Oh, I remember that company. Uh, so our scraps, so I'm going to go pay a visit. Um, and, uh, yeah, at the time, uh, I think subscriber was doing a lot of things in the LSP space, right? So mine were server protocol stuff and, uh, And, uh, I was, I was also like becoming more aware of, of those efforts. And so that kind of tied into some of the things that, that I ended up, uh, working on during a Facebook internship there, um, uh, working on, uh, like Python and kind of surfacing, you know, Python related code intelligence, or code editor, uh, uh, functionality. So, yeah.
Beyang: Yeah, it was it's all, at least from my point of view, it was all very serendipitous. And, uh, I knew very early on that I would love to work with you someday. And just so happy that you ended up joining the team and, uh, bringing all your Thoughts and insights into, what's on the research frontier to what we're doing at Sourcegraph. Cause I think, uh, there's a lot of, uh, interesting, uh, kind of product features and capabilities, uh, that can be built on top of that, which, you know, I suspect we'll get into, uh, over the course of this conversation.
Rijnard: Yeah. So thinking about it now, right? Um, What I remember as well. Like everybody I talked to , we're very keen to get feedback right on like, No, not just the product or, you know, the, the, the ideas that, that social graph was kind of working on. But, but also just like, you know, what are your thoughts of, you know, having, for example, code search or code intelligence, that's kind of ubiquitous and on these sorts of things. And so it was always, there was never a dead end to the conversation to anyone I've had at search graph. Uh, you know, when an initially talk it's always like, what do you think of this? What do you think? And that stood out to me. Um, I think the. Yeah. Basically identified like a deep desire to kind of, you know, go after those goals go after kind of the things that the company is setting out to do, which is, you know, uh, this idea of, you know, code search, everywhere, code intelligence, everywhere, every editor, every, you know, in our face. Um, and, uh, I think, you know, it was also a question of, you know, I guess, mutual interest and, and, and really commitment to this idea. So, um, that's, that's why I ended up here.
Beyang: Awesome. Well, I guess, you know, before we get into , uh, research that you've done, um, and Coney in particular, uh, I was like to kick off the show by asking people how they got into programming, uh, originally. Uh, so you know, what was kind of your backstory.
Rijnard: I think I got into, uh, my, my first real kind of programming experience during high school. So, uh, my, my final two years in high school learned to cut in Java. Um, I still remember, I think. My first programming experience was looking at a textbook and there was this program in the textbook. I mean, it's like, copy this right character for character into your editor. And I like type it out. I'm like, I can do this. Right. And we're going to run this thing and I'm like copying. And then. Now after I think I've copied the whole, the whole thing in the book I click run or compiler was compiled. Right. And so, uh, this thing just tells me like, all these syntax errors happening everywhere. I'm like, what? Like how, how did I make this mistake? I looked very carefully at what the book was telling me to do. And it's like, you know, you missed this. I mean, Colani minister, a brace or something. And I was so surprised back then because I was like, wow, it's so easy to make a mistake during this. Right. Um, so, so that's one of the first recollect trends I had, uh, during programming and, uh, uh, from there it became, you know, became more interested kind of in, in, uh, understanding, you know, the whole, the whole kind of theory around it, computer theory, but I just no ideas like, you know, algorithms, data, structures, that sort of thing. Um, and then, you know, and university kind of focused in, uh, And learning more of computer science. I think I was only a university that I kind of got exposed to, you know, Lennox for the first time and, and like, uh, becoming like a command line, power user and all that sort of thing happened having over the course of my varsity years. Um, and I can go into more detail, but that's basically the, the early beginning, um, for me.
Beyang: I think that, that makes sense. Like it's, I think a little bit similar to how I got into programming. Like I, my first programming language was also Java in that high school environment.
Rijnard: yeah, my condolences.
Beyang: I mean, there are, I think, worst, first experiences.
Rijnard: true. Yeah. I'm just poking fun.
Beyang: Yeah. Um, what, when, where along the, uh, line of your studies, did you decide to, that you wanted to get a PhD or was that something that you always felt that you wanted to do?
Rijnard: Um, definitely not something that, that I had my heart set on doing. Um, I. I think my, my story aligns with a lot of kind of other, uh, graduate students or people who end up in academia, at least for computer sciences, you know, you try and shoot for a technical job, maybe Israel, too many technical interviews. And you're like, okay, well, I'm going to go to grad school now. Um, I definitely have a couple of data points. Um, you know, of people who shared that. That experience and I, and to some extent I am one of them, I, I think I did a couple of Google interviews and it didn't work out. So I was like, well, I'm not, well, let's try this other Avenue. Right? Like you send out applications for, for various things. So I'm like, I'm open to this, to this whole PhD idea. Um, and, uh, I mean, that ended up working out. So, uh, I went for it.
Beyang: Yeah, I got rejected from Facebook myself, so I can definitely sympathize with getting rejected from those big tech companies. Um, cool. How did you pick a, your research area? Was that something that kind of developed over the course of your, like an undergraduate, uh, career or, um, was there some other influence or inspiration.
Rijnard: I decided to focus, essentially. I went for, uh, the reasoning to my mind was. I'm only interested in going through a whole like research experience or doing academic research. If it's, if it's a topic that I really care about. Right. Um, um, because I mean, all the, all the kind of advice out there is, you know, and they do this. If you really are committed to the idea or you're really, uh, interested or passionate, if you will. Um, I don't think that's a prerequisite. Um, For, you know, going into research and my perspective on that has changed a bit afterwards, but, um, certainly going in, it was a question of like, I know that I want to focus on these areas. This is interesting to me. I'm not going to kind of put myself through a lot of pain to learn things that I'm not interested in. So, um, the, the, the kind of stuff that stood out to me back then was, uh, you know, software security research, but, but also just, you know, automated. Um, techniques, automated program analysis for, um, finding bugs. For example, that was, that was the biggest kind of, uh, uh, piece of attraction. Right. Was. Um, can we use tools to automatically find bugs? Um, uh, bugs and security bugs specifically are, are super interesting because, you know, they have such severe consequences and, uh, you know, it's amazing what, what, what, what ends up happening if you exploit, you know, one of these super severe bugs and an important system. Um, and so for me, it was a question of, well, you know, how far can we kind of push like automated techniques to find these really complex. Really interesting books. Um, and from there it kind of, uh, it changed a bit of direction, but, but that kind of interest has always been there.
Beyang: yeah, that makes sense. It's kind of like, um, I guess the general category, I put all that under is like, it's almost like meta, meta programming, like programming tools that like. Right. Programs or help you debug programs? Um, uh, yeah.
Rijnard: there's definitely a lot of, I think, meta to it, for sure. Um, and, uh, uh, people have a love kind of throwing that term around. Um, I think Mehta programming as a term is pretty loaded. So I won't, I won't try to put too much definition to that, but, um, I know, you know, people who are in the program synthesis community, right. Programs, generating programs to do things. Um, this is all also very meta, but also quite different from, you know, uh, writing programs to find bugs and other programs. Um, but definitely related.
Beyang: yeah. So one of the things that you covered in your PhD research was this new pattern matching syntax, uh, called, uh, Comby. Uh, can you tell us a bit about what Comby is and what was the motivation for creating it?
Rijnard: so Comey is a tool I wrote, um, that. Essentially fit the bill for the kind of research problem I was tackling at the time. Um, I didn't really find other tools that could do quite the thing that I, that I wanted out of a code transformation tool. Um, and basically that came down to, you know, I want it to selectively change. Parts parts of a program, parts of programs, syntax without, you know, having to compile the whole program without needing to know that the whole program, a scope or, you know, across calls across files or anything like that. Just give me one file. Even if you know, this file contains like some, some malformed syntax or it contains macros or things like this, uh, I didn't need a full compilation or even a full parse of this thing online. All I wanted to be. Able to do was change like an if statement, right. Or, or, or a four loop or things like this. And, uh, uh, to make it more concrete, what ended up happening was I wanted to change something and Java and C um, it was a simple if statement, um, and, uh, The conventional way that I would have had to go was find tools that could change particular. If statement, you know, matching whatever conditions I was interested in changing in these two separate languages. Now there are tools out there that can do. Uh, you know, you can express a transformation that covers multiple languages. Um, they were a little bit more heavyweight than I was kind of willing to entertain. So, uh, I ended up going, can I just find some like simple Java and some simple C thing. Maybe client can do this for me in Java. Uh, some Java person things. So I, so I ended up picking out some Java parser project as well. And then, um, I, uh, I wanted to change this particular statement, but I couldn't do it without kind of parsing the whole program or even, even I couldn't even parse like a single file. And so then I ended up. I ended up going to, that's a, that's a slacker. I don't quite remember. And I asked, you know, asked them kind of, can I, can I like change this one Just like, without doing anything super complex, you know, and the response was kind of like, Oh, well, why would you want to do that? Uh, uh, you know, why, why not just, uh, you know, the whole thing and do the thing that you want to do. And so, um, I mean, I was like, you know, that's fine because you know, the, the, at the end of the day, these tools are made for, for particular reasons. And, and the reason I wanted what wasn't kind of aligned with each, right. So I wanted to change these, these fragments. So, so call me is as kind of an answer to that, which is basically like part and parcel program to, to a general parse tree that, you know, preserves some of the fundamental, you know, syntax that, that, that. Corresponds to a tree structure, right. Which is typically like braces and parentheses and these things. But you also have to take into account things like comments, strings, because you know, if you have parentheses or, or braces in a comment or in a string, you don't want that to be considered part of the tree or the syntax that you're trying to part. So, uh, it's not just a question of, Oh, you're gonna detect a couple important characters. It's like you actually sort of have to parse the whole thing. Um, but the, the guy is that, that, that notion generalizes over a lot of languages, right? Because at the end of the day, trees are useful in programs. Um, it turns out like this is a useful representation, um, that, that, that basically all programs use. Um, so. So, if we can kind of approximate that in some way, then, then it's easy to manipulate. Now, there are tons of tools out there, right. That, that, go along this thinking. And, you know, I can name a lot off the top of my head nowadays, you know, there's like tree sitter, there's all various kinds of, you know. Yeah. Um, And so the way I see it is, you know, there's a lot of, uh, there's a lot of tooling in this space that, that addresses. Parts of the problems and parts of it, the kind of design space of, of co transformation. And so it's a, it's a matter of picking, picking the right tool for the job. And if that tool that doesn't exist, you kind of have to engineer it. So call me is my response to engineering. So, um, for this like purpose fit role of. You know, manipulating syntax, uh, uh, at, uh, at, uh, smallish scale at the, at the scale of, you know, simple link checks or, or linear transformations, auto fixes, this sort of thing. And at the end of the day, I needed that to do, uh, you know, to integrate with a more, more sophisticated approach of, of, uh, Program repair or what others might call, uh, automated program texting or automated bug fixing. So that's the context.
Beyang: That makes sense. So just to lay out kind of the spectrum of, um, like pattern matching and replacement, um, like tools and syntaxes, I guess, on, on the far end of the spectrum, there's like the full on compiler based, uh, static analysis based tools. Like, you know, that Java library that you were mentioning that actually hook into, you know, Java C or whatever the compiler is, and they compile the code, they build up the AST and then this probably like. I in order to express your transformation, you probably have to express it in terms of like programmatic changes to the AST or something like that. Right. Or maybe the, so that's like one end of the spectrum. And then on the other end of the spectrum, I would say like, probably things like regular expressions or tools that make use of regular expressions. Do you think that's kind of like an accurate, um, book-ending of, of the spectrum?
Rijnard: I think that's a good way to look at it. There's to my mind, there are multiple kind of spectrum, let's say. Um, I think, I think you can actually, you know, uh, Th th the way I think of tooling, uh, you know, to categorize the vast tooling landscape of things that match and change code, it's more of a, kind of like a, uh, a two D landscape. Um, where are you at various, uh, axes. Right? And so along these axes, you have data points. So, um, the one that you point out, is a very good one to think about in terms of expressivity, right? What, what is the set of transformations that I can express? So regular expressions is, is on one end of this. And you can map some of the, this notion of expressivity to a more formal way of thinking in terms of, you know, Chomsky's hierarchy and that. That sort of thing. I don't think you have to go that far because we're at the end of the day talking about tools here. Um, and so the tools, you know, correspond to in some senses Trump's desire to keep it, but I found that China, you know, fit to link into that formal notion. Isn't it very good because there's a lot of overlapping and fuzziness,
Beyang: Chelsea's hierarchy is like this hierarchy of like languages from like your regular languages all the way up to like turn complete languages. Is
Rijnard: Right. It's, it's, it's a way to describe kind of a distinct, uh, capabilities and computation. Right? So, so, uh, something that is expressive at, at, at, uh, you know, a context free level, um, Kind of can recognize languages in a, in a regular expression, uh, language and so on. Um, so it's just a breakdown basically, of, of, of at farm level recognizing languages, but it corresponds to, so to the ability to compute, right. Um, and, um, As far as tooling goes, you know, regular expressions have, or regular expression tooling, like grip, you know, I have certain operators that, that break out of the regular expression, the formal motion of regular expressions. Right. So, um, so in terms of what you were saying about, you know, the spectrum of tooling yeah. There's this idea of exposivity right. So regular expressions and then compilers are, are, are, are, uh, at least. Able to recognize, you know, context free, um, constructs, which I think is like balanced parentheses, but, you know, typically you need, uh, a context sensitive, uh, notion to recognize things like that. Yeah. Def Cincy, for example. So this is like, uh, you know, you need to know about things that were defined, uh, at the point that they were used in order to interpret our pars something correctly. Um, So there are all these notions and, and, you know, tools take that into account to varying degrees. Um, if you use it, it's an actual compiler tool. It knows about all of these things. Right. And so, Mmm. That's a useful way to think of it in terms of expressivity and that, that, that translates into performance as well. Right? Like regular expressions can be extremely fast and, you know, uh, compiling can be comparatively soil. Um, so. So that's one way, the other way that I think of it is in terms of how you, how you specify things, right? How you express what you can match, um, or transform. And this is, this is something where, you know, with coma, it was like, well, I want something simple and lightweight. I don't, I don't want to go and write up a program or a script or, you know, a clang plugin. This is just too much effort for what I'm trying to achieve. Um, And, uh, you know, similarly, you know, with regular expressions, there are some great things about it, but you know, a lot of people also gripe about the syntax. And so, um, there's a lot to take into account that that is not about exposivity or, or, or performance. It's also how, how you describe what you're trying to do. And I think that's really where the, a lot of. Design choices and tooling choices come into play. As far as you know, how important it is for a tool to succeed, uh, or, or be, be effective in the domain that it's intended for.
Beyang: So, so Comby, if I understand correctly, what you're saying is it's, it's the syntax and this tool that kind of hits a sweet spot in this, um, Two dimensional space. Like it's, it's expressive enough that you can describe basically, you know, like 98% of what you want to do in, um, like every day kind of finding our place, uh, instances in code. But at the same time, it's also, it has this ease of use or like, Ergonomics developer ergonomics, uh, aspect to it that makes it a lot easier to kind of write patterns in it than something like rejects, which as far as I know, like every developer that uses red jacks has kind of a love, hate relationship with it.
Rijnard: Yeah. Um, I mean, the goal is definitely like to make, to make it lightweight, to make it as declarative as possible. So, you know, I've kind of made the choice of, you know, not allowing medicine talks or escape characters and things that, things like this, um, just as a usability thing, but over time, I've also come to recognize that, you know, uh, it's very useful in this world to, to match. You know, lexical constructs, just, just things that regular expressions are good at and why reinvent another language. So, um, that, uh, you know, especially one that's so popular that people are, are already to some extent familiar with. So, um, you know, Com is now kind of, uh, advancing to the state where, you know, you can optionally embed regular expression syntax into it, um, to, to match those things. You know, things like certain character classes. Um, so there's definitely, definitely, you know, uh, parts of syntax that you can improve on. Uh, and, and certainly I've strived to make it. Uh, you know, declarative, so it's not, not, not so much dependent on, uh, you know, writing a program or, or, or expressing exactly. Oh, this is a function, you know, that I want to match, or this is a variable. Uh, none of that is really incorporated. It's all pure, purely syntactic and purely reliance kind of on, you know, just course syntactic structures, um, that. That certain languages have. So if, uh, if you're in, you know, Earline and you have certain, uh, keywords that, uh, delineate blocks or Ruby, you know, um, Uh, then, then it would recognize those things. Um, so it's language specific at some level, but it's not, not at the, not at the kind of semantic level where you necessarily know that, Oh, you know, I want to match a function block. Um, so basically there's a way of, of, of identifying syntax that corresponds to the underlying, you know, language construct that you might be interested in.
Beyang: yeah, it's, it's kinda like it, it handles the balanced delimiter, uh, like patterns that are fairly common in code, like, you know, string quotes, parenths brackets, that sort of thing. It handles really well. Whereas like, those are always super annoying to express him rejects cause re regular expressions have. I have no concept of like memory so that they can't keep track of like how many, like nested layers, deep you're into. Whereas like Cody kind of understands these at a, like a foundational level and the syntax makes it easy to cause like I've tried it myself and it's like, if you want to switch the order of like arguments in, in, in a function call, for instance, it's just like, Uh, I guess maybe you could get to the notion of holes. Um, but the, you express like these holes, which are just like in, in the syntax, it's just like colon bracket and then a number. And then that becomes an argument and you can type it out. It looks like actual code, the pattern that you wrote. Um, and then that's a pattern. And then you express the, the replacement pattern, just referencing the numbers that you entered into those like holes later and super kind of intuitive. I think.
Rijnard: Yeah. Um, you can compare a whole sense, exa I mean, you can compare those just to, you know, name named identifiers are named groups in regular expressions. Uh, people rarely end up, I think, in practice using named identifiers for regular expressions because they, they're not using the context of changing code. Right. It's just, I just want to match this thing. Maybe I'll group some expressions, but I'm not going to attach a name to it. Um, and I think that also limits like, Readability, right. Like if I wrote something and I'm going to match a group of something, maybe it corresponds to, you know, a telephone number. I give that to you beyond. And I'm like, here's the reg X. Maybe you don't even know that this like group expression corresponds to, you know, uh, the, the, the numbers that are care about. Um, so, so it's kinda, it's kind of like, you tend to ex explicitly, you know, match that, identify to, to the, to the texts that, the whole like contextually corresponds to, and so you can reference that when you're rewriting the code, which is, you know, with your swap arguments example, it's like, okay, well now you already have the identifiers that you can use to swap, you know, um, Uh, arguments wanting to, um, so yeah, uh, as you said, it's kind of this, it's a way to kind of identify and rewrite, you know, structural pieces of your code. So maybe you want to identify calls inside of a loop or inside, inside a double four loops, nested for loops. And there are certain calls you want to match or, or check. Uh, so one example is, uh, you know, your, your compile, it's funny. So you can find code right where you. Where you compile regular expressions, right? So you can use Comey, for example, to find where you're maybe compiling and recompiling, you know, a regular expression inside a loop. And that's redundant. Typically what you want to do is you want to compile the regular expression once outside of the loop. If it doesn't change, you know, it doesn't affect the regular expression. And then. And then just run that regular expression on whatever's inside the loop. Um, so there are certainly cases like that that I found, uh, you know, in Java or go or, or pick your language. Right. Um, this is an easy mistake to make. Um, so, uh, so that's kind of, uh, a very kind of concrete and, and, and, uh, the accessible, uh, example of, of how you might use it, where regular expressions are just going to be hard to identify, you know,
Beyang: and we should mention to folks listening that, uh, co Comby is not just, um, uh, a syntax. It's actually like a command line tool. And if you go to komi.dev, or you can look up the documentation, like actually just download the tool. I think it supports Mac iOS, Linux, and. I don't know if it's on windows yet, but,
Rijnard: Well, everything's on windows nowadays with a window.
Beyang: Oh yeah, the subsystem.
Rijnard: Yeah. That's, that's a recommended way to go. Um, for sure. Yeah. Uh, yeah, I mean, and uh, always happy to, to, to help out I'm hopping at the, at the Gitter channel or, yeah, I'm
Beyang: Yeah, there's a, there's a guitar channel and you're, you're super responsive on it. I noticed.
Rijnard: Yeah. Um, it's a, it's a valuable way for me to also collect feedback and understand what people people want, the teaser tool for. Um,
Beyang: What sort of people have reached out, um, so far. And what sort of use cases have you kind of seen in the wild.
Rijnard: you know, it's interesting. I think, I think there's a, there's a wide, there's a wide array of things that. people want to do once, once you tell them, Hey, you know, you can, you can change your code and in certain ways, and you know, it can, it can be more sophisticated than regular expressions. And I think what's interesting is that you end up with a lot of kind of interesting and unexpected ideas. So, uh, one example of just, you know, usage people wanting a way to review, you know, Code change by code change. So I want to go through these because they're not all gonna match exactly the thing I'm interested in, or maybe it changes like some test code that I don't want to have effected. And so, uh, I think code mod, the Facebook variety of code mod, you know, the, the actual tool on GitHub, um, kind of pioneered that idea, which is this like review Patrick coaching. Um, and that's a, that's something I integrated kind of after the fact, right. In response to, ,
Beyang: This, this is
Rijnard: want the workflow.
Beyang: this is the interactive mode. So like you can go through and it'll actually show you each place that the pattern match and ask you, like, do you want to change these? Yes or no?
Rijnard: Yes, exactly. So, um, that, that was something that people wanted just to, to integrate into their workflow. Um, what, what I've seen other people want to do is changing things inside of HTML tags. Uh, you know, Casa trivia is names, that sort of thing crops up often. Um, and that's again, another place where, you know, That kind of pointed out to me that, uh, to some extent you want to support regular expressions, um, to, to let people like match their arbitrary kind of classes of characters, but within, you know, angle bracket tags, or, or within strings, for example,
Beyang: How much, how much do you find yourself reaching for Comby and day to day pattern matching, find replace tasks versus, you know, something like grap or said, uh, or like defining replace in your editor.
Rijnard: it's an interesting question right now. I am most, I'm most drawn to using comedy for, for kind of high level applications, rather than, you know, Oh, I want to change a couple of things in my code. I've I've done it a couple of times, you know, working with source craft. And so it's popped up a couple of times where it's like, Oh, I want to change. Um, you know, particularly things in tests, for example, you know, you have like a bunch of test cases and these test cases are very similar, like maybe of, you know, 20 or 30 or 40 test cases. And each of these test cases is like a record or, you know, struct and you have a field and a value. Maybe you want to rename. Those fields are you went to reading, uh, or you want to add something to the structure or whatever, and then it can be used because, uh, then you can just, you know, match on that each independent struct and then add some fields that maybe it's dependent on some of their input. So I recall a couple of times needing that maybe, I dunno, maybe, you know, once a week or once every couple of weeks, I I've run into a case like that. So not, not super often, but you know, I don't, I don't end up using said for, for things, uh, Much either. Uh, but the, the, the kind of appeal I see right now for using comma is more and yeah. In a sense of a, you know, manipulating code for other purposes. So one is, uh, mutation testing. Uh, so, you know, for the unfamiliar mutation testing yeah. Um, had you transferred your program and then running a test suite, and then if, um, if your test suite doesn't detect anything. Bad. You've essentially created a mutant program that, that, you know, if you've changed it in some, uh, some way that, that you're wanting it to fail, but don't, then, then that's a, that's a gap in your, in your test suite. Right? And so, uh, you can imagine that, you know, easily changing programs and that scenarios is very useful. Um, Uh, there's a lot of research in that space, like academic research and a lot of tooling associated with that research as well. Um, and, and usually it does come down to a more heavyweight tool, or maybe you're just like, Hey, I'm just going to like focus on Java or, or C to do, to do, to evaluate some interesting new approach. Um, and so, uh, it was coming. That could be something that you could do at the, at the most general level. Right. Um, not restricted to any particular language. Um, as far as the tooling goes,
Beyang: I myself have tried to use some of those more kind of, um, heavyweight, uh, refactoring tools that actually hook into the AST. And my experience is that it's, this is quite a, like a learning curve. Um, cause you, you kind of have to like learn the ins and outs of the EST and in order to ensure that you can actually like express the pattern that you're trying to, um, describe, and it's, it's almost, um, There's a greater mental barrier, because like, when I think about the pattern that I want to, to match, oftentimes it's like at a kind of textual or a syntactic level, like, Oh, you know, switch these arguments or, Oh, you know, add another argument at the end of this function. Or, you know, move this struck, rename this struck field or something like that. And. Uh, I'm not really thinking about like which AST node specifically I have to modify and like a w which kind of like layer in the tree that might occur and what are all the other like hidden AST nodes that I don't even know about, but are kind of like implicit, implicitly constructed, uh, when, when the language gets parsed, um,
Rijnard: for sure. That's part of it, right? Like, um, clang has an excellent like query matching or way to match. Kind of C, C or C plus plus syntax with a query language. So, you know, they, they recognize that, okay, visitors, you know, writing a plugin or. Uh, you know, that does a programmatic thing using a visitor to face those kind of heavyweights sometimes. So we're gonna introduce this queer language, but in order to use this career language, you, you have to be familiar with all of the grammar constructs to make sure that you're not you're matching on this, this, this specific thing in the C plus plus grammar that you're actually interested in. And many of the time you might not even know that, you know, the. The L value for assignment corresponds to the left hand side. It's it's, it's very heavyweight. So, um, yeah, as a, as a, what I would say a declarative technique allows you to do is kind of what you see is what you get, right. Um, And, uh, that's kind of the aim here. Now. Of course, there are many tools again, out there that do declarative, like transformation and matching, um, especially on the academic side. So, uh, you can find tools like a rascal and, um, uh, spoof is like a language workbench. Uh, Uh, that, that, that covers a lot of languages. And you can kind of define all of these declarative ways to, to match a syntax. But, uh, at the end of the day, either have to like dig up the thing that you want or, um, or kind of learn a little bit more context about those tools and, and with come, is this thing that basically I wanted to just, just to be you a brew install, and then run thing that I care about and not. Pay too much attention to, you know, is this like a, a, you know, a fully fledged parser? Is this, did someone go and make the effort to turn this into a, you know, a declarative way to, to specify things for Java or whatnot?
Beyang: I want to return to kind of, uh, an earlier use. Okay. So like, you know, we've talked a bit about how this can make the lives of people who are doing kind of like serious refactors, um, easier. Um, so like people who, who would use more, more of a heavyweight tool, um, we didn't exist, but I kind of want to return to that, like every day yeah. Use case that kind of like rejects replacement. Use case, um, cause you know, you said you don't, you don't use it every day. Um, I, I actually find myself using it more and more often. And I do wonder whether like it, in my view, there's kind of this hurdle that like any new everyday tool has to, um, Overcome, which is kind of like the familiarity, a hurdle, right? Like the people who have already invested, uh, many hours of our lives into learning, uh, rejects, uh, syntax and all its, you know, special variants. Uh, that's kind of like a thing that, you know, we already have and now along comes like a new syntax and it's like, Oh, you know, do I really want to learn this a thing that I have? You know, it's not perfect, but it works reasonably well. Um, but I do wonder, because I know a lot of other programmers, both, um, new and, uh, some fairly experienced who actually have avoided rejects, uh, so far. And it's like something that's kind like scary to them. And I almost feel that like Komi would be. In, in many ways, like a better starting pattern, matching syntax. Cause it's just more intuitive to use. You can kind of like parse it with your eyeballs. You don't have to go and like upload it to one of those, like rejects visualizers to figure out what the heck is going on. Do you think that Comby like will evolve into a tool like that, especially for beginners where they kind of use it, in more of everyday setting.
Rijnard: I am very happy to, to see that happen, but it's not, not, I wouldn't say that that's kind of an end goal. Um, at least to the extent of, you know, I'm not, I'm not pushing for the tool to be adopted by, let's say novice or unfamiliar, um, uh, you know, programmers, it's, it's, it's more a question of, you know, The tool is designed to do something in a simple way, something more complex than reg X allows in a, in a simpler way. And I think you can, you can absolutely hone in on, you know, how do I make, how do I make it minimally simple for people to do X, um, as far as matching or transforming code goes. Uh, and I, and I think that would, that would, you know, at some point you're going to come up to you. Yeah. Your, your tool's gonna sacrifice something in the interest of simplicity. So, um, my, the kind of thing that's top of mind for me for designing, you know, syntax for comas, make it dead, easy to kind of, you know, match code and make things correspond to code. Um, uh, but at the same time, you know, that there've been more and more kind of examples where. You know, once you start using the tool, you want something more out of it. You want to, maybe here's a, here's a concrete example, right? It's like, Hey, I want to, I want to sort my, my list of imports in my program alphabetically. Right. Um, and so com can make it super easy for you to like match all the import statements, you know, within some like import, uh, group. Parenthesize in per group and go for example. Um, but it doesn't, it doesn't have any native capability to go and say, okay, I have a list or a set of lines that were matching. Now. I want to sort these less lexical graphically, right? Like that capability doesn't exist. And, and you could pipe that stuff into another program and then, you know, rewrite and, and that that's something you could do. But, uh, the point kind of here is, uh, at the end of the day, It's good at doing one thing really well, which is , match, you know, syntax that corresponds to code structures, and then whatever comes on top of that, um, you know, I'd love to support things. And branch out capabilities, but you have to kind of draw the line somewhere. And I think the same thing happens essentially, when you say, Hey, I want to make this thing super accessible, right. To people who don't, who don't know regular expressions. And I, and I don't know what the, the shape of that solution looks like, but, um, certainly some of the designs and decisions around the syntax of Kobe, you know, uh, advocate for, for, for a solution like that,
Beyang: Yeah, that makes sense. Um, I want to kind of take a step back and like chat about some of the other research that you've done, but before we move on to that, I think it would be remiss of us if we were not to mention that, uh, the commies index is actually afforded in Sourcegraph, uh, currently. So if folks want to try that out, you just go to dot com and in the search bar, it's kind of like the, um, bracket icon, uh, called structural search. And so if folks want to try that out, we'd love to hear feedback on that.
Rijnard: Yeah. And I'm looking to put out more examples of that, uh, as well. Um, so, uh, yeah, go find interesting things. Uh, try it out, give us, give us feedback and, uh, And, uh, the, the benefit is there that you don't even have to clone your repos. You just, you know, we have repos up there and, uh, you can search, you know, for example, some of the more, more popular get hub repositories, uh, Java go of hyphen, all these things. Um, so yeah, it works out of the book.
Beyang: Yeah, actually, just a quick question on that, like in implementing that syntax for Sourcegraph, did you have to do anything special to get it to scale?
Rijnard: Um, well, we're doing some we're, we're doing some fancy things, uh, leveraging, you know, the fact that we, we have indexed, um, a lot of source code already. Um, I can, I can look up, you know, in certain files, whether, whether a fall at all com contains, you know, strings that would match. So, uh, There are various kind of optimizations that I worked on. Um, and they'll tend to, to kind of identify that, that sort of thing. Right. Um, and it's not, it's not any kind of, it's not any different than, you know, whipping out like rip grip and then, you know, finding files that you want to search. And then, uh, you know, finding that in Tacoma. But, but the benefit is on, on search gas.com of course, that, you know, we do trigram indexing, which is even faster than, than rip rap. So. Um, uh, so that, that's why it works really well out of the box.
Beyang: Cool. Um, yeah, but, uh, so with respect to, you know, the other research that you've done, uh, my understanding is that you, you wrote Comby because, uh, you were having trouble with, um, Like a pattern matching and replacement because of another aspect of, uh, of your, your research. Um, can you tell us about like that, uh, I guess initial motivating problem and, or maybe just like about your, your thesis in general and what were kind of the themes there?
Rijnard: Yeah. So my, my high level focus was to. Take take existing tools that we have out there that are fairly popular and use fun practice, uh, to, uh, to find bugs. Right? So to find bugs that are a little bit more sophisticated at a, at a semantic level, right? So not, not your, Oh, I'm missing a semi-colon or, um, you know, I. I have some, you know, Lindt check that that's failing, but, um, something more in the line of, Oh, you forgot to close this file resource. And maybe you've, you've opened up a follower, a socket and you know, one function and then three. Function calls deep. You're done with this thing, or you've turned up the stack and you forget to close it. Right. Um, so, so from a program analysis perspective, this gets pretty complex because now you have to consider, you know, um, calls across multiple functions, uh, different contexts in which a function is called. Um, you know, whether, whether there any logic inside of, you know, if conditionals are four loops change, whether you. Close the file later or not because you don't want false positive. You don't want the analyzer telling you, Hey, you didn't close this file, but then you did. And they analyze there's too dumb to know the difference. Right. Um, so, uh, That's the flavor of maybe the sort of bug that you want to fix. And so, so, uh, I went and dove in and said, okay, given that an analysis run, it knows things. It has reasoned that, you know, there's, there's this bug that happened. How can we go about fixing that automatically? So we want to make some change to the program and, and fix this bug in a way that's that we can have a reasonable amount of confidence, uh, that, that it fixed, uh, that it is fixed. And so, um, Don't have a lot of detail to that. Uh, and I, and I won't necessarily go into all of it, but basically you leverage the fact that, you know, a static analysis. And so in this case, I used a tool called Infor, um, which is also open source on GitHub. Um, uh, and that's maintained by Facebook. Uh, I think it's a great tool to have out there. And, um, you know, it finds this sort of bar Gregg resource leaks and, and. And, uh, uh, memory leaks and C C programs and so on. And so. Um, I w I was using that as like one tool as a basis of, you know, trying to automatically fix the bugs that reports. And so at the end of the day, you can use some of the things that the analysis inferred about your code to also inform a fixed, right? You can, for example, identify places in the code that also close the file on some condition. Um, and you can identify that, uh, you know, in that, in the analysis output, But, but once you have that, right, it's like, okay, I'm ready to make a change. I'm ready to change the program. Uh, and that means you have to make a syntactic change, right? Like at some level, this stuff has to translate into some tactics, change the program. Okay. So you're like, I'm ready for this. I just need to add this like close, close socket inside this if condition. Right. This is what I need to do. And it's like, okay, how can I, how can I match on the body of this if condition, um, you know, reliably? So I can just insert, and this is, this is kind of, uh, around, into this problem, which is like, okay, I know everything that I need to know what I want to do. It's just. It's just not easy to just do this one thing, right? Like I just got on matches this if conditions, body inside these braces, but I can't because they're there they're too many parentheses in the, and they have conditioner too many bracelets inside that confused rug. Like I tried regular expressions at first was like, maybe I can get around those. Maybe I can just, so it's a research problem. This isn't like part of a novel or a significant contribution. I just, I just wanna, I just wanna. Do this change. Right? And so this is where I got
Beyang: you thought finding the issue would be the hard part and then actually making the change would be easy, but it turns out making the change was a lot more annoying.
Rijnard: Yeah, absolutely. Um, and so, and it's so big, it became, this is kind of thrown in my side. Right. Um, and, and that, it was at that point where, you know, That I ended up using, I think a, you know, a specific Java thing at that point. But, uh, but I came out of that thinking, you know, this is ridiculous. Um, I'm not doing that again. And this is actively hindering, like other things I want to do in my research. So I'm going to go build this tool. Um, and that's kind of how it, how it sprung out and, uh, uh, yeah, it was, it was kind of a built around that purpose. Um,
Beyang: that Mo motivating problem is really interesting cause like the kind of magical, uh, uh, like the, the thing that I've used is probably closest to that. It would be like find bugs in, in the Java world. But like, when you get suggestions like that, where it finds like actual bugs in your code, uh, and flags them to you, that is almost like. Uh, the first time you see that it's like a wow moment. You're like, wow, this is magical. Uh, it knows what's going on in the code. Uh, and I think like these days you see more and more kind of, uh, stronger claims being made about what actually can be done. I think you and I have chatted length about, uh, like machine learning for instance, is, is, uh, A topic or buzzword, that's thrown about a lot these days and I've seen it. Yeah. Used, um, a number of times in, in the specific domain. Yeah. And like code transformations and, you know, automatically writing programs. Um, which admittedly, you know, would take it a step. Yeah. Even just the like kind of semantic analysis and transformation. Uh, we've chatted about this at some length. I I'd be curious to, if you could share your thoughts with, um, the podcast audience about like, um, the intersection of machine learning and, uh, like, uh, language analysis, um, and, and, uh, programming automation.
Rijnard: Yeah, I'll scope it a little bit more to, I'll only kind of comment on, I think, you know, the, the interesting aspects around, you know, can we use, for example, machine learning to automatically generate or fix programs? I think I'm more familiar with the kind of, uh, effort around automatically trying to fix programs, uh, using, for example, machine learning. Um, And I think there's also a distinction between, you know, uh, applying machine learning in industry, you know, uh, scenarios versus academic. Yeah. It's around machine learning. So, uh, to me it's a, it's a, it's a matter of, well, there are many, uh, approaches that say, well, okay, we, you know, programs are just. Data, and we're going to feed it into, you know, some supervised learning, um, algorithm. We're going to get some outputs and we can use that. Right. And show that the results is useful for some context. And you can absolutely do that for something like, you know, uh, uh, automated program, repair, automated book fixing. Um, but, but, but I think the, the challenge here is that like, if you treated it like a black box, right, you. A, the stuff you're going to get out is going to be kind of a function of, you know, the fidelity of things that you fit into it. And, uh, uh, you know, if you're just treating it as, as text, then you're, then, then there's only, you know, there's an extent to how good you can do. Um, but when you go to the office, um, and so, uh, this, despite the tendency to, you know, just, just throw things at, at. A machine learning algorithm w which happens right in research. And I think it's not, it's not, it's not totally unreasonable to report on those results and say, Hey, you know, uh, uh, observe this. And it's sort of, it's full maybe. Um, Oh, we're not really sure. Right. But, uh, I think it's important to caveat that and say, well, um, Programs are a lot more structured. That's why they're so interesting. And we have a wealth of, you know, history on, on, on program analysis and research that, that really dives into, you know, what's, what's the complexity, what's the underlying, uh, complexity of the thing where we're dealing with this problem that we're trying to solve. And, um, I mean, it's, it's already been proven at a, at a formal level, um, that, you know, uh, any sufficiently interesting property is impossible to, to. To, to detect in general. Right? And so there's this corresponds to rice as theorem. And I don't know if you're interested in that. You can go, go dive into that. But the point is like, uh, neither machine learning or anything else can actually solve this at a very general level, but we can do pre pretty dang good. If, uh, If we reason about what we can and cannot do in certain contexts, right. And, and what approximations we're going to make. Um, and so I think machine learning as, as a, as a tool to solve a wrong, and there's a very course, uh, unless you encode all of this stuff that we know. And so one of these things that we know about programs is they have structure. They have a grammar, they have, uh, you know, uh, various attributes and semantics around certain buggy properties. And so, um, I think kind of the research space is split between those two lines of thought, right? It's like, uh, do we, do we. Do we come from the perspective of, okay, we're sharing this as a program, uh, as a foundation that's, you know, logical or able to reason about and model maybe, you know, just great sense, uh, versus, okay. We have a very powerful kind of inference engine, but it's, it's, you know, it is, uh, it is based on kind of, you know, uh, Elements that, that, that we're not going to reason about or necessarily yeah. Incorporate, you know, ex or explain how we arrived at a certain result. Um, and so my hope is, you know, these, these two things converted, um, and get, get closer to each other. They're a bit detached. And so, um, of these two camps, right. I fall very much into the like, you know, study and research programs as a, as a. As a, as a structured or logical concept. Um, and then, uh, maybe on top of that, you know, implement various, uh, ways to, to do something like machine learning or, or other AI approaches to, to, um, to, to kind of reveal interesting properties or, or, or, uh, ways of fixing programs. Right.
Beyang: Yeah. Are there any efforts, either an industry or academia that you're aware of that, um, in your view, are, are taking kind of the right approach of synthesizing these two worlds?
Rijnard: I don't know so much about synthesis. I think an inch, I don't know of any good examples right now, at least that, that I can point to. I will say that, you know, In terms of, you know, where we are going with automated reasoning and things like, you know, uh, being able to do generate programs automatically or fixed programs automatically, you know, are we, are we going to put software developers out of jobs or are they gonna, you know, at some point, uh, you know, uh, we're, we're not gonna need certain types of software developers or engineers, uh, in terms of that, I think it's clear that, you know, we're. Where we're not quite. And I don't see us getting there anytime soon, you know, uh, automating a way, any substantive, uh, engineering, uh, ability. Um, what we are doing is getting closer to kind of, uh, you know, removing the TDM around certain bug fixes and reasoning about some. Um, uh, pieces of code and also just, you know, engineering oversight. So it's like, you know, you're coding, you're trying to, you're just trying to, you know, implement some new interface to call out some other code that, you know, Or removing a feature flag or something like this, uh, and then bugs crop up. And it's like, I don't want to deal with this right now. Are you, maybe you're not thinking about that right now, or it's not in your mind right now. And then an analysis tool picks that up. Right. And that's really where the value is. And that's really the gap I see, kind of the tooling and an automated nature of, of software analysis going is really to, to make us more effective at, at. Working on the core of the problem. Right. And the core of our day jobs, we don't want to deal with all the TDM and the, the, you know, do I have to check this thing? Whether it's all again.
Beyang: the knowing parts of the job, we want to focus on the creative and fun parts.
Rijnard: Yeah. So as far as, you know, successful efforts in industry, I think, um, Since I was at Facebook and then I kind of follow along the research. That's adjacent to some of their stuff. Um, I think they're really doing well at, you know, Doing kind of automated program, air, automated bug fixing for these more sophisticated, uh, classes of bugs that I mentioned. Um, and so, you know, they're, they're looking at things like, know the references, you know, for, for Java and so on and automatically fixing them, incorporating that into. Uh, their CIA and, uh, you can go look up on the internet. There's a project called sapiens. So Mark Harmon and his team behind that, um, you know, they, they do some pretty interesting stuff where it's really like, kind of at this frontier of, okay. We're at the point where we can kind of change programs and they result in kind of. Mostly reliable or at least feasible fixes to bugs that we are willing to show to developers and say, Hey, does this fix your issue? And you know, it becomes this kind of push, push button approach, um, to, to automating these fixes. And that's really what I also kind of went after in my research. It's like, can we, can we achieve this idea of like push button program pair where we have. Enough confidence about a buck fix that we're willing to show it to developers, uh, and, and say, Hey, we actually, we actually tested this, right? Like this, this fix, um, Stopped this test from failing or this fixed stops this analysis from reporting a bug and we have high confidence that this is an actual issue. Um, and so I'm excited to kind of see that expand. And I think it's a question of, you know, when it's, uh, it's going to happen eventually where we see more of this sort of. Tooling, um, kind of crop up and it's very much going to be, I think, a function of, you know, integrating with a developer's workflow, whether that's CGI or in their editor, or, you know, a tool that they use to search a review code. Um, that's really going to be the. The interface to this sort of interaction, um, where, where you, where you interact with an automated tool and what did, um, there's going to have to be yeah, yeah. A human in the loop for a lot of it. And I see that, uh, factoring into, especially things like your CGI workflow, um, and editors.
Beyang: You mentioned that you worked, uh, at Facebook, uh, forbid I believe he also did stints at, uh, MSR and, uh, Google, if I'm not mistaken, um, of those three, uh, companies, uh, you know, w w we'll have a short time left, but like, are there any like, projects that you think are worth calling out there that were particularly interesting to you in, in the space of developer tools?
Rijnard: Yeah. So my focus became more specialized, I think, as I, as I did more of these internships. Right? So, um, I think the, the most recent and most kind of focus to, to, to my, to my interest in speciality was a Facebook, right? So I worked on, you know, uh, Pyre, which is a static type checker for Python. Um, and it's really interesting to me that, you know, over. Over time. Facebook has kind of gone and been very, very much, uh, investing in tooling, right. Um, uh, for software qualities, things like bug detection type checking for Python and this sort of thing. And, uh, I th my impression is, you know, Facebook is kind of at the forefront of doing this a Google, a Microsoft, for sure. You know, they invest in their tooling and, uh, and a lot of it's great, but, um, I think a lot of, you know, Talented researchers and, and, uh, You know, engineers, uh, are working on these tools that have really impressive results. So finding, uh, anything from super sophisticated bugs and, and hack that, you know, could lead to, you know, Facebook servers being compromised or, or, you know, just daily developer, a I missed the, uh, no, the reference check. Um, they've, they've really kind of covered a broad space of it. Um, I think Amazon is also now ramping up a lot of, of their dev tools and program analysis efforts. Um, but, but certainly, uh, it's been interesting to see, uh, you know, Facebook as a, as a social media company at the end of the day, really investing so heavily into software quality dev tools. And so that to me just says, you know, Uh, this is, uh, this is, if it isn't already, it will be a ubiquitous concern for any software company, right? As you grow, as you, as you understand the complexity of, of, you know, what you're, what you're doing and what your software and, and the, the activities that your engineers are engaged in, um, it becomes a critical piece that has to integrate with, with what you're doing. And if you're a company that. That isn't aware of that yet. Right? It's a bit of a, um, and I think this problem becomes more important as you grow, obviously like the size of your company matters. Right? And so that's really where the value of these automated tools like automated bug fix is automated by finding come into play is as soon as you reach a scale where you want to cover a lot of, a lot of ground and a lot of the complexity, um, of like, you know, millions, billions of lines of code.
Beyang: If a folks listening, want to learn more about, um, Comby or any of the research that you've done, um, how would you recommend they go about doing that?
Rijnard: Uh, the easiest way is probably to just find me on Twitter or just DME on Twitter. Uh, and then we can, uh, fire off an email exchange. If that's, that's something that, you know, you're open to, or you want to chat more in depth. Um, So you can find me on Twitter at RV T O N D.
Beyang: Cool. And we'll put that in the show notes as well.
Beyang: My guest today has been Rijnard van Tonder. Rijnard, thanks for being on the show.
Rijnard: Thanks Beyang.