Postmodern strace

hello everybody uh thank you for coming my name is dimitri levin i am the chief software architect at result where we do genuinely gnoll linux separating system but i'm also the maintainer of stress for the last slightly more than last 10 years so today i'll be talking about postmodern estres are what is postmodernist race i used to talk about modernist race last year so i understood that i can name it modern stress any longer if i'm talking about variation features so where where traditional stress ends and modern stress begins and when modern stress ends well more stress modernist never ends so when it turns into postmodern uh well it's kind of subjective so well my definition is very simple that the stress that was before i started maintaining it it's traditional and all the rest is modern and yeah so here it goes and uh well postmodern is all new features since the last talk at first them so i'll be covering mostly what have changed for the last two years but i'll remind you briefly about additional features just to to refresh these switches in your memory so stress is a mostly linux system called tracer it also can not just trace but trumpet the system calls since like several years ago but it has a lot of options to control its behavior in different ways like whether it prints instruction pointers whether it prints timestamps or not how it prints strings what system calls are printed and which way they are printed what's abbreviated and what's not there are also options to control what signals are printed it can also dump the data that goes through descriptors it can print its output in different ways so you can for example redirect it into a pipe or collect a collect output for each process separately uh yeah a lot of features that control or what ciscos would be printed it can also print statistics on system calling vacations it can attach to already existing processes it can follow forks and it can don't follow works depending on whether you specify the option well that was traditional there were also quite a few options added for the last 10 years like you can print a lot of details about uh about descriptors like what paths are associated with them or what socket information is behind circuits when these descriptors are circuits we can bring a stack of user function calls yeah you can filter filter system calls by path names we've finally got support for regular expressions for filtering system calls so you can specify which ciscos are printed using regular expressions and so on yeah more ways to control how statistics is printed uh what is being how it's traced so you can for example attach too many processes you can run this trace as a detachable process and so on and so on well and there's also this big feature which change it is trace i mean change not just this trace but the way how people look at it it's a system called tampering so you cannot just trace system calls but also inject uh various things like starting with return code also you can inject signals and delays but this all was more or less covered in the previous talk so in the last two years we got pictures gates is calling for support he it went both into the kernel and in into stress we got system called returned status filtering we have com comp assisted cisco filters nowadays there are also a lot of new system calls in the kernel that are supported and we have more and more elaborated system called parsers we also finally have long options yeah we had no choice we will soon see why and finally a bit more than a year ago we changed our busy style lenses to a copyleft license so let's start with the first feature uh well the story itself started very very long like i think it was 2001 then this new architecture x8664 appeared so the way it was uh added in linux kernel obviously was to to support both 64 and 32-bit processes for obvious reasons because it was the main like feature this architecture compared to its competitor that it could run legacy code and in early years of this there are a lot of legacy code and very little native 64-bit code but the way it wasn't implemented in the kernel it allowed not just to mix instructions but also mix system call invocations so you could actually invoke from a native code both native 64 bit system calls but also legacy 32-bit scores and it was very poorly documented if at all and it was very surprising to many people and it wasn't really exposed in the kernel api so yeah what user space jason debuggers could do they could fetch the system called number uh they could like fetch this sys register that describes the business of process and then they would just guess do the wild guess and say well if the process is 64 bit then probably the cisco is also 64 bit right it's mostly the case and if it's a 32-bit process then cisco is definitely like fit to beat and all the logic dependent on this wild guess and it's it mostly works because in most cases it's it's exactly what what happens but sometimes it's not the case and back in 2008 there was a bug report against strays in debenberger there is a very simple example that looks uh you can see are very similar to it it's somewhat simplified compared to the one reported in that dark report so the program does a very simple thing it just prints a line of output so and then it invokes a 32-bit system call and then it prints another line of output but these 32 bits is called is actually a fork so what happens is that there are two processes the and each of them prints the line so if you compile uh lincoln run this program you'll see an output similar to this well maybe p numbers will change but all the rest will be just very simple but if you run this very simple program under stress you will see something very strange so you will see this line is being printed and then suddenly process attaches and then you see this ridiculous open system call with very very odd very impossible i would say our arguments but all you can say about this is what and all the rest looks very usual and regular making the whole picture completely ridiculous like this ridiculous open among all nice expected system calls so yeah if you run this program several times you will see that all these old open flags are different you will never see the same combination or probably never see the same combination flux because nowadays thanks to colonel address syndemization uh all these registers contain garbage that changes and this reminds me of a of a toy i had a in my childhood a kaleidoscope you turn it slightly and you see a different different nice picture so you can use this simple program uh as a kaleidoscope if you like yeah so this problem was i approached it several times but until until 2018 there was no progress and finally well thanks to two people who contributed this api in the kernel and two yeah there are two authors and 20 more people who reviewed this and i could buy silent bias it took us uh almost nine months to get this into the kennel and like don't remember how many iterations but it was two digit number of iterations so finally we have it in the in the kernel and for all architectures that support jstrokes which are like all supported architectures or almost all i would say and some that are not supported but get it for free we we have this and yeah the api looks this way there is a structure you can request from the kernel it contains this crucial architecture field and in other ways it looks similar to comp data so you can obtain in one go both the architecture this is called number cisco arguments also instruction point or step pointer and this is this makes uh traces that use this api reliable in this respect in respect to the original problem so the same program now looks uh if the if linux is fresh enough and this trace is fresh enough you see this is like as expected so process attaches you see a proper for call and not this ridiculous open and all looks good so i think other traces and debuggers that have something to do with system calls should switch to this api by the way it also allows uh to to find out what kind of picture stop is uh the the current stop otherwise or up to this time kernel provided no way to find out uh so they used to think that they alternate so first your entrance is called and exits is called but it's not always the case so you actually can use this nice api to find out and the what is the actually the picture stop you're dealing with okay so it was a very like major feature first race uh and yeah as i said some other other choices are welcome to use this of course let's speak about system called filtering there is a new option to filter system calls by written status it had a very like unusual history first race so it was actually introduced like in 2002 but it was broken from the beginning and it was never announced you couldn't find it exists unless you accidentally type it in or look into the source code because it it was broken so it what it did uh it printed the beginning of system call and when it failed it just didn't print the ending it wasn't useful but now you can filter system calls by return status so you can print only those system calls that are succeeded or those that are failed so in this very simple example you can see the difference well if you run a very simple program like cat uh with a yeah with a modified ld library path it it makes a dynamic linker to look into different in different places i wonder whether whether you expect dynamic link to look into so many different places but well you can see the difference uh uh as a very useful side effect of this uh option you can have an aggregation for free so for example if you trace several processes that are running asynchronously and then you will see a lot of this unfinished and resume stuff and sometimes it's not very convenient we used to implement special aggregators to collect this data so it would look like this but now thanks you can use this uh option also to aggregate the only the only need i would say is that it might change the order of invocations so in this example it looks like if nano slipsy skulls were invoked sequentially which is definitely not the case they were invoked simultaneously but because they were printed at the moment they finished uh this system calls it looks not the way you are used to but then you when you're aggregating it doesn't really matter in which order they are printed so the yeah there is also the another option uh that um where is the funniest order connected with it so when i try to come up with a with something useful as an example i started um i started invoking all programs i had in my small small route and i found i found out a few programs that were not printing uh not printing their arguments correctly then they couldn't find them i just invoked programs with a non-existing file yeah and i found two of these programs in infusions and i fixed them but you can get an idea when this could be useful for example when program doesn't print what's going on you can like chase and have a look when you are filtering this when you are filtering system calls you probably don't want to if you don't want to print all the rest you'll probably want to make the both system calls you're not printing execute faster and now we have a very nice feature which we planned for several years but couldn't get until well we had two just so google summer course students in the year before last the student made a prototype and yeah and last year we had a student who is going to talk about this feature very soon i hope so he will describe how it works but from user perspective it looks like space is no longer no longer delays everything by two orders of magnitude on those system calls that are not traced so in this yeah it's a famous example because uh the this is exact this is a modification for example a bpf people uh use to to describe how how slow stress is and now we use the bpf stuff to show how fast the stress is [Laughter] so you can see that second bf itself slows down things about 10 percent which is nothing compared to what uh all these picture stops do with the speed of running programs uh yeah you can see this is a long option and it was actually the first option that we couldn't find a good uh short analog so we had quite a few not as many as ls program has but quite a few options and some of them are not obvious and we had a i think what is dash n is in our prototype but we could couldn't find an explanation what y should be called dash n so we decided that it's time to introduce long options and now we are started adding irises for not so obvious names are yeah so second people was the first one another option which should probably have a a long option analog and is the option that has named k i don't know why it's called hk it prints a stacks stack of user user calls at the time of system convocation yeah it's very useful thing because you can see the logic that behind the program if you don't know what's going on you can just apply this it will produce a lot of output but it makes makes a straight somewhat kind of debugger more than a tracer so in this example you can see why why for example cat closes is the doubt from names of these functions you can see that it does some kind of it exit handling and it closes down to ensure that everything is written otherwise it should return non-zero exit code yeah another option uh you can attach you can use this trace as a detached process so because this trace affects traces in different ways ah it's not always desirable for example if these processes interact with their parents and they know they want to know their bid numbers so you can run a stress or this way and be more transparent yeah there is also a relatively new option that that says how all these symbolic constants should be printed uh so you can print as usual uh like translate these numbers into symbolic names so you can print both symbolic numbers and and draw numbers or just raw numbers it has like various useful implications you can uh you can debug programs that you suspect you pass arguments uh to system calls in a wrong way which is not very surprising because on different architectures there are different system called different apis or the different number and order of passing system system called arguments and this also can be used i think it's used in cisco project so yeah we added support for all new system calls that were added into the linux kernel and nowadays they started adding system calls again so yeah like bunch of few bunch of new system calls that work with mount points but well i can't describe them all there are too many you should ever look into man pages probably but we have support for them uh we also have a lot of very sophisticated sophisticated uh system called parses and i'll show you an example which looks them very monstrous but you will get the idea how sophisticated system passes could be so we support decoding of net networking protocol and you can see this output this is the output of a very simple routing table and here you can see what's going behind yep so you you see this knitting protocol is very structured it has some structures substructures sub sub structures and everything is printed color coloring is mine all the rest is made by stress yeah and the last but not least is that in december of 2018 we changed the license so stress used to be released since the very beginning under berkeley style license uh so he it was a uh yeah by a request of paul kreinburg i don't know this man it was too early so when we added support for petraeus gets his calling for api it was kind of crucial point uh most contributors to space didn't want to contribute on the permissive license any longer so we decided we will drive a change to a copyleft so test suit is released under new gpa v2 plus and all the rest is is the license that allow us to release this as a library someday in the future if we manage to make a library out of space so this is more or less what i wanted to talk about and if you have some questions or ideas or something related to this choice to discuss we have some time yeah should i repeat the question oh so please be because i uh should i pick yeah from the back to the front yeah so the question was uh why this combat system system called filtering is not by default and why you you have to type this very long option to use this feature first of all you can abbreviate long options so uh you don't have to type that much i think i think two letters usually enough if not and type three letters and some some shells allow expansions of program arguments so i don't think this is a problem but well there is a there are two important points about this way of filtering is first it it generates and attaches a bpf program i'm not going to dive into details but it makes this program and you can attach it but you can't get rid of it unless you're privileged so this implies that you have to follow forks you have to follow all purchases that are forked by the process you are chasing and this kind of change behavior and one of important points of stress is that this trace is backwards compatible so we can't enable follow forks by default because people are not used to this uh yeah and if you specify this option and do not specify follow forks it says that i am enabling follow folks hey so this is one point another point is that unless you are privileged and stress is used as a previous program you can't attach a bpf program to another process so you can attach attach to a process using pcs or p3s attach but you can't attach a bpf program to another process you all can only attach our program to yourself so uh one of the important features of stress is to trace already existing ports wouldn't work with this unless you're privileged but if you're privileged you can use a lot of kernel chasing nowadays it's not really a big deal although they don't have so elaborate choices yeah yeah please another question yes so you mentioned that on the last slide that the color was your your own uh would you consider adding color output to s-res so the question was that on this slide the color was my own and would i consider coloring by strays it's kind of this is a difficult question because we had actually a plan to generate a structured output from a stress and if you generate for example some json output you would apply already existing software to do all this fancy stuff like coloring so we decided we will make structured output first and then other people will do whatever coloring they like but as you can see there is no structured output yet and i have to do all the coloring yes please me i think i can so the question was whether i can pretty print this i think this is pretty enough whether i can bring this in blocks so it'll be easy to read yeah it's getting closer and closer to our idea of structured output so yeah you can see why we decided to go the simple way but it was not so simple is it over ok thank you for coming you