Deterministic debugging with Delve And the state of Delve

all right hello everybody um uh yes thank you very much uh my name is Derek Parker um I am once again going to be talking about delve this is true um uh so this time the the predominant theme of this talk will be deterministic debugging with delve uh this is a talk that I'm particularly excited about um as I think that this this style of debugging is um not only not very well known but as a result of that underutilized um I think a lot of people think of debugging in the traditional sense of you're debugging a live process um a lot of different things can happen a lot of things can happen between debugging sessions and between runs of the process and things like that so uh how do we how do we eliminate a lot of that and especially uh if you have a bug that say only uh only exposes itself maybe one out of a 100 runs how do you sanely debug that that's the kind of things that I'm going to be talking about during this talk um just to introduce myself a little bit more again my name is Derek Parker I am a software engineer at Red Hat um I work on go tooling Upstream um and I work on delve as well and some of the go packages that we ship through uh real um I uh am the original author of delve um and have been working on that for a long time um among among a few other things okay so I want to give kind of a run down of what we're going to go through in this talk so as mentioned um I've I've started to steal the uh the idea for the state of go and um I've spoken here uh for a few years so I like to kind of give an update on the state of delve what's happened uh what new features and what things have changed since last year um because in a year a lot of things can change a lot of new features uh can be can be completed um um and I think it it gives kind of an idea of the pace of the project and the types of things that we're working on and focused on also allows a little bit bit of feedback from from just users in general everybody here about maybe what they would like to see in the future uh after that I'll go into what deterministic debugging is and explain uh how it works first of all just explain the concept for for those who may not be familiar with it and then I'll dig into a little bit of how it works under the hood so actually um how how you can get rid of all of these deterministic or un non-deterministic things that happen in every process run um and have a a repeatable uh debugging session so how that actually works from a relatively low level following that I'll talk about how delve leverages this to uh be able to debug go program uh and then finally I'll do a live demo just kind of showing off uh showing off the concept A little bit showing it in action and giving a sense of how just how easy it is to do this yourself in your day-to-day uh debugging and and work and programming workflow so that hopefully this is something that you'll reach for or at least know to reach for if you come into a problem where uh this style of debugging might come in handy okay so first of all let's just jump right into the state of delve let's talk about what's what's happened in the last year um some of the new features some of the new changes and then we'll briefly talk about some things that we're that we're going to work on going forward um so the project is now 6 years old uh which is crazy to me um it's uh you know they grw up so fast right um uh I'm I'm very excited to to have been working on it and um that uh so many people have found it to be useful it's is very exciting um we've had 175 commits over the last year uh not a ton but for a more esoteric project as a debugger um that's that's really really good and and we've had a ton of new contributors um which is great we love first-time contributors and try to encourage that as much as possible um and folks building in some really huge features which I'll I'll talk about next so what's uh what's changed since last fom well there's been a couple go releases right so we've we've released uh uh some new versions of delve that support that so specifically 113 and 114 which which will be released uh another big big big change that's happened uh over the last year is arm 64 support so uh before this we only really supported x86 64 um we've just gotten a a a great patch in recently to add uh kind of initial arm 64 support there's still some more work to be done there but that's that's still a huge uh a huge addition to the project we obviously aim to support every architecture and operating system that go supports natively uh but you know it just takes some time so contribu are always welcome uh another big thing is free BSD support again we want to uh we want to include as many operating systems and and everything as we possibly can so that was another big big addition um we already support you know various linuxes um windows and OSX uh so this just adds one more operating system to the repertoire um we've added uh the ability to script delve via a language called star which was um originally uh created for the build system basil um it's Bas it's a pythonesque dialect uh and that adds a lot of powerful scripting features uh uh into delve so you can you can Define new commands within delve and other things uh just by writing this this starlark language um and those those commands will be available to you in your debug session uh We've also added support for debugging go plugins um So for anybody using that feature that's Now supported um Windows mini dump support which is basically Windows core dumps um and improved support for debugging pies uh position independent executables uh some other notable just changes some internal refactoring um just ongoing code maintenance cleanup I think that's an important thing um several performance improvements uh es especially on um breakpoint handling and a lot of other things um improved handling of optimized binaries I think this is an important one and this took a lot of coordination with the go team uh you if you have a if you have a binary that's actually running in production or something like that or you build it just just normally with go build it's going to be optimized so we want to be able to to debug those ideally as well as you can debug uh uh binaries that you've built with optimizations explicitly turned off um we've improved function call support and just in general uh uh We've we've included a bunch of other improvements and Bug fixes okay so now let's go on to part two the uh the real the real kind of me and subject of this talk so first off um I'm up here speaking about it but I would like to thank my co-maintainer who happens to be here uh for doing all of the hard work of actually Plumbing up delve to the uh the record replay back end that we used to to achieve this um he did a lot of the hard work I just reviewed it and now I'm up here bragging about it so I just wanted to give credit where credits to and say thank you okay now let's talk a little bit about uh what is determinate stick de plugging um and let's also talk about what's that weird little character down at the at the bottom of the screen uh so um my my son is four and a half years old watches a lot of cartoons this is a new Toy Story cartoon who starts off a lot of his sentences asking what is something so I thought it would be funny to include him in the slide saying what is deterministic debugging it's more for me than it is for you uh um anyway so just to kind of gauge the room a little bit how many people have heard of deterministic debugging very few that is perfect that is that is actually exactly what I wanted to see and it kind of proves my point that I think this is an an a relatively unknown and very underutilized uh way to debug your programs so I want to talk a little bit uh about what it is just to give some context and then as I as I mentioned I'll I'll go into uh like a pretty deep dive of how it works and and the the different kinds of things that that is done to uh to actually achieve this so um one more one more quick question how many people have seen the movie Groundhog Day a good amount of people okay so for those who haven't seen it um it's a movie where Bill Murray goes to This Town patani Pennsylvania where uh every year actually today on February 2nd it it turns out um a groundhog comes out of the out of the ground and if it sees its shadow or doesn't it has some sort of mystical bearing on on how long the the winner is going to last but the the connection here is that every day uh in this movie that he wakes up it's Groundhog Day over and over over and over again he's repeating the same day over and over and over again every single day every time he wakes up for a really long time um and the the no matter what he does no matter what he changes in in his environment when he starts the day over again the next day he goes through the same exact series of events the same exact people talk to him and say the same exact things over and over and over again and this happens forever well at least for a really long time um and and the connection here is that that's essentially what deterministic debugging is that's what it's trying to achieve so the point of it is that every single time you you start this session debugging what we're going to get to know as a recording everything is exactly the same um and this includes memory layout uh signal delivery uh you know a lot of a lot of the different variables that may change and may slightly change how a bug is is reproduced are always is exactly the same so as I mentioned before if this if your bug only shows up maybe once uh once in 100 runs if you capture it just that one time you can replay it indefinitely to try to figure out where the bug is so how is this possible uh as we know every time you run a program there's a lot of things that are different uh uh memory layout uh threads um uh where where variables are stored in memory what happens as a result of of certain CIS calls when signals are delivered Etc there's a lot of things that are different how can we eliminate all of this non-determinism and Faithfully reproduce the execution of a process and as a result Faithfully reproduce the events leading up to the bug um that you're trying to fix so the solution turns out to be record and repl uh and what this what this actually means is we we essentially record the execution of a process um record all uh the different types of the result of any kind of non-deterministic operation and are able to replay that back so anytime one of those operations is hit again instead of actually going through and doing the operation or making the actual assist call or anything like that um the the pre-recorded output is just replayed back to the program now this turns out to be pretty difficult but also very very powerful um now there's a lot of different types to do this some are are kind of better than others I'll I'll go through a little bit of of of uh existing implementations so there's VM recording which isn't actually very widely supported um VMware actually even dropped it from their uh their virtual machine offering um there's there's a couple things but they're really clunky really heavyweight and really typically Al you only want to record a process not the entire execution of the kernel and operating system and everything else so it tends to be bloated and not not very useful uh there's also some user space recording um undo DB uh replay engine and mazilla RR we'll talk about RR uh a lot more in the end of this talk there's pros and cons to each so some of them do code instrumentation which I personally am not a huge fan of um uh I think leaving leaving the code AS unchanged as possible possible is kind of in the spirit of debugging you're not you're not trying to change a person's code you're just trying to show them what happened so that they can find the issue so let's go through some of the pros and cons real quick of of some of the existing implementations as I mentioned with VM recording uh you have to debug within within a virtual machine that supports it that adds a lot of bloat um and not a lot of virtual machines actually support this uh so it's it's really not portable and doesn't doesn't t to work very well with your workflow um especially if you're not using a virtual machine or you're on bare metal or just kind of writing code on your laptop you're probably not running it within a virtual machine um it produces large traces a lot of bloat um it's not widely supported as I mentioned um and so let's let's move on to user space recording uh it doesn't some problems with that um it it doesn't record kernel execution uh we'll I'll explain later why this actually turns out to not be that big of a deal as I mentioned some require code instrumentation the the the backend that we use um does not uh so um we get rid of that and some require some require kernel extensions which again uh destroys portability so there's a lot of cons sounds terrible why am I even talking about it what's the solution uh so what we use is Milla RR and I I'll talk about how how it works a little bit um but it it solves a lot of those problems by not doing not doing any kind of code instrumentation it's it's pretty lightweight and and relatively performant there are some drawbacks that I'll discuss a little bit uh especially with um with certain go programs but it's still I think the best solution that we have available right now and it's a start right I I think as as kind of shown by the polling in the room not a lot of people know about or are using this technology so I think as awareness grows more people can work on it improve it and we'll get better Solutions in the future but right now uh uh RR is is um is the preferred solution so eliminating sources of non-determinism how can we do that and what are some of the sources of non-determinism uh so here's a couple that that we can control right uh certain CPU instructions so most CPU instructions are deterministic you feed them you know the the input or whatever they do an operation they put that the results of that operation in memory or register or whatever and uh it's going to be the same every single time you call it but there are some that that aren't so like Rd Rand a uh instruction to Generate random numbers hopefully that's not deterministic um so uh the the way to get around this the way that RR does it is it actually it turns out it's not used very much that actual instruction most people prefer Dev random so it can just be patched out of anything that uses it uh there's also like time stamp counter uh instructions um those can actually be trapped and recorded um and CPU ID that's mostly deterministic if you're running in the same the same Hardware uh but it does return which uh which core the process is running on that can be different each time so that's also trapped and recorded um on newer kernels and Intel CPUs uh thread scheduling so this is important especially for go uh so um what what RR does and and one of the the kind of performance hits that it takes is um it runs everything single threaded so it handles thread scheduling um it does do preemption and things like that for Via signals and everything but it runs your program single-threaded so for large go programs and programs that are are really really highly parallel this could incur some slowdown um but unfortunately that's that's a penalty in in a price that we have to pay uh for right now um so system calls uh the result of a system call could potentially be different uh we can we can trap uh the um whenever whenever a system call is executed and in and record inputs and outputs so we can just replay that back after uh when we're when we're replaying the the uh recording memory layout uh again uh we want to ensure that memories are that variables are at the same address in memory cuz potentially that leads to a certain bug or exhibits certain symptoms so we want to again we want to recreate the process as Faithfully as we can as it was executing exactly when it was first run um shared memory um this is a big a big one actually for non-determinism um uh but for for RR again it runs single-threaded so you don't have to worry about different threads or whatever within your process messing with shared memory um there there are some things where like the X server and pulse audio and things like that communicate back and forth using shared memory but we there's ways of disabling that signal handling um so RR is also able to handle this um we can we can uh replay signal execution uh and delivery um by using uh Hardware performance counters so basically we can count and know where we are in the program and what's happening when that's when we're at that specific spot so when a signal signal is delivered at a certain point in the execution of the process we can record when that happens and replay it um so some sources that we that we can't control just really quick uh we don't control anything that actually happens outside of user space but as I mentioned it turns out we don't really need to because we just we just record the result of that anyways um so Hardware failures I mean if that's happening you're not going to have a lot of luck anyways um so I'm kind of getting a little bit low on time so I'll kind of go through some of this pretty quickly to show a bit of a demo um so some pros and cons pros relatively low overhead um except for single threaded uh you can Faithfully reproduce bugs every single time um you can record a trace and replay it anywhere so you can record a trace on your server and replay it locally um for easier debugging uh can execute programs in Reverse so you can go to the where a bug exists itself and kind of work backwards instead of trying to work forwards towards the bug and replaying that every time which I think turns out to be pretty powerful um you can replay from the beginning execute the same sequence of events it's always recorded always the same um cons there's a performance hit for highly parallel programs uh it only works on Linux and uh only works with um uh performance counters uh enabled uh so let's talk about how to use it with delve first use Linux uh right now it's only supported on Linux um unfortunately OSX is not supported um install install RR mazil RR um that's not installed for you automatically when you install delve and there's three options for using the RR backend with delve which I'll talk about quickly now and then I'll I'll save a little bit of time still for a demo so option one is a little bit more manual um so the the first thing that you're going to want to do is build your program again ideally you're building it with optimizations disabled that's what those flags are uh you use RR and P pass it the path to your binary it's going to record it and save a trace somewhere and then you can use delve to replay that Trace um once it once it's completed uh option two uh is a little is a lot easier there's one command that just kind of does this whole process for you so you can use the delve debug uh subc command pass it the backend RR and then uh it'll compile the program run the trace record it everything and then open it up so that you can start debugging um and then last you can use the scripting feature that I mentioned earlier so um this this is actually included in the documentation on our GitHub so you don't have to memorize this command but essentially what it does is uh It intro introduces a command called flaky where it'll rerun the process over and over and over and over and over again until it uh until it hits a bug so it it exits um you know with uh with a bad error code um and then uh once it once it does that it'll stop running the process over and over again and you'll be debugging that Trace so if you don't want to do this manually you know uh make the computer do it right and uh once it's done and once it stops you'll be you'll be recording uh the uh the bug so let me do a quick little demo just to show um how how some of this stuff works um one minute okay I'll be really quick okay uh so um as mentioned you can just um debug with backend RR um it's recorded you can see the output of the run of the program um and uh um now we're actually we're actually debugging the trace so if we set a breakpoint at Maine um continue we can see some of the things that it's doing right so it's saying it's asking what date it is printing that out and it's also opening Dev random um which should be different every time right so if we if we next a little bit we can see um we read we read from Dev random uh and if we print the result of that so like the bites that we have have um you can see kind of roughly just the output that we get right so just remember the first two 180 183 uh so let's set a breakpoint here on our T.O 24 um so we'll restart we're restarting um all the way from the beginning again continue continue again to our second breakpoint and if we print the value of that variable again we see it's exactly the same um so even even complete sources of Randomness will be replayed Faithfully every single time um and just to show that it's not a trick uh you can see when I run it for real um every single time the output is actually different so we are recording and replaying faithfully and that's time thank you very much