Python & multiprocessing: Fork'ed ! by Teijo Holzer

[applause] Okay. Um, yes, welcome everybody. Thanks for joining my talk. Um, Python multi multi Python and multiprocessing forks. This is the official title there. My name is Teo Holtz. I've been working for W FX as a senior u uh software engineer for now almost 20 years and I've been working with uh the Unix operating system for more than 30 years, Linux as well 20 years and Python close to 20 years as well. So that's kind of my background and um today I'm going to take you through the through a ride of Python and multipprocessing and the problems that you might encounter. So this is the uh structure of my talk today that I'll be going through. So it's just a quick introduction. That's where we are now. And then I'm sort of going to um set the scene for the fundamentals. You know, what is the process? What what is a thread? You know, what are multi-threaded processes and so forth. And then I'm going to be explaining um different ways of how you can spawn processes in Python and why you want to do that, right? So there's also um you know, why are we doing this? You know, what's what's the motivation behind it? So I'm going to cover that. Then I'm going to explain the most of the fundamental um ways of spawning process on a Unix operating system. Linux belongs to that. Mac OS as well. Windows is a bit separate. I'm going to touch on that as well. Um, and then I'm going to talk about the forked part, you know, what are the problems when spawning subprocesses on a Unix operating system like Linux and Mac OS. And we're going to investigate or was going to have a live demo there as well. So I'm going to show what can happen if you spawn processes and I'm going to show how you can um diagnose and debug uh the underlying causes for those deadlocks, crashes, all those kind of things, right? And then um hopefully I'm going to leave you with some uh proposed fixes and workarounds so that when you go home um you have a better understanding of the um underlying issues that can happen when you are dealing with processes in general, you know. So that's the structure of the talk. So to start off with What is a process? Okay, so a process is an isolated container that runs on a computer and u this runs your code basically, right? And it's identified by an ID that's usually referred to as a process ID and it owns resources. Resources in that sense are memories, you know, the memory it can access the open files. It has sockets, pipes, you know, they're files as well. And so these are the resources that a process uses and interacts with, you know, and and each process can contain one or more threads of execution. And that's what what is a thread you know. So so you can have a single threaded process there's only one threaded for execution and it can stop and do do another task and then come back to the original task but that's just a single thread. So what is a thread then? Um so that's an exe execution strand within a process to run code and the the idea here is that you you can have multiple threads with inside a single process right and each of those threads is identified by a thread ID. You have the process ID for the whole process and then you have each individual thread thread and they identified by thread ID their unique thread ID and they can execute concurrently inside a single process right so you have in a multipprocessing a multi-core machine like I have here 12 cores I can have 12 things literally running in parallel within a single process right and that's a really effective usage of the resources available namely the 12 cores that I have and at one single process can max out all the 12 cores I can show I can show that in one of the live demos as well. So you can have one or more threads uh per process. So usually you refer to those entities that becomes important later on either as a single threaded process or a multi-threaded process. So it doesn't matter if it has two or more if it has more than one thread. It's a multi-threaded process and that's really important later on. So the um the key uh difference there is that processes are usually not allowed to share resources like memory or open files but threads are. So that's the main difference there. So the threads that are running inside the process, they have full access to the whole memory that the process can access and the all the resources like the files and the sockets and the pipes and all the graphics card and they can access that concurrently. Not always a good idea without proper synchronization mechanisms, but that's what threads allow you to do and it's really lightweight. Creating a thread is extremely lightweight for the operating system compared to creating a new process. We'll see that later on as well. And then yeah obviously the um threads should use synchronization mechanisms like mut mutxes or locks if they have content you know contentious access to the resource concurrently. There's lock algorithms as well that can help you there. So you don't need to use locks absolutely but they always help if you have these race conditions. So this is just like a visualization that you can see there. So on the left you can see the uh single threaded process and there's just the files and the socket and the pipes and the resources that you deal with in your memory and there's nothing special you have to do right but on the right hand side you can see there's multiple threads of execution that are running concurrently indicated by these um green loops uh and they access they have access to all these same files and the same memory but they would normally use locks or mutual access to access these resources so they don't trample over each other and go in there at the same time causing problems. So that's actually a a really important uh concept to remember that you have these different entities single threaded multi-threaded process. So that's the um the difference between those two. So now we're sort of coming to the APIs of dealing with processes you know how can you um spoon processes or create processes in Python. So this kind of um API stack that you see here, it's sort of top down, right? So the highest level API that the standard Python library brings with it, you know, like the batteries included type of thing is the um concurrent um uh standard library and specifically the futures inside that we're going to cover those a little bit more. So then underneath that is the multipprocessing library and that's where all the heavy lifting happens. Concurrent is just like a really uh simple abstraction layer of the multipprocessing library. All the heavy lifting happens in the multipprocessing library. We're going to talk a little bit about that in detail as well. Uh subprocess that's the original that's always been around. Um and that was just a simple way of spawning a single process and then capturing the output and the return code. That that is really what subprocess is used for. Not so much for managing a group of processes. This is more the domain of the multipprocessing module. And when you look at the code, you find that concurrent is there's virtually no code in there. All it does, it just delegates either to the threading library. Concurrent can do threading and as well as uh multipprocessing, multi- threading. It delegates all the work to multiprocessing. And in multipprocessing, you find there's tens of thousands of lines of code, both Python and C code that actually does all the heavy lifting of being able to spawn all these processes, managing them all, all the interprocess communication via sockets or pipes or shared memory or whatever it does, right? And collecting all the results and all that kind of stuff. So multipprocessing is really where all the complex work happens. And then on the lowest level, you have um the standard Python API. It's called the OS API. And that's where you find things like fork for example, right? Or exec. So those are the the actual underlying operating system primitives or like the C library primitives that you can call directly. There's also pix spawns and other ones that we could talk talk about a little bit later. Windows is the odd one out here. That's why I listed that one as well. Windows never really knew about fork and exec at all. That was always a Linux concept and uh also Unix concept I guess and then therefore Linux and Mac OS as well. But uh Windows always went down this separate path which uh is the create process path. And basically create process is very similar to project spawn. It spawns a completely new process. It's it's nothing like fork and exit. So what we're going to leave that aside a little bit. This is the main focus of this talk is mostly on the Unix based operating systems like Linux, Mac OS, Android and all those kind of things. So Windows is a little bit the odd one out. So we're just going to leave that aside for now. Okay. So concurrent futures the top level entry point a standard Python library module as I mentioned you can execute concurrent code right that's that's really what you want you've got code and you want to make it faster you make it run on multiple cores so it's usual will work that's CPU IO bound what does it mean so you have to compute something which is computationally intensive or you have to wait for something to happen like reading from the network reading from a file system something like that so and then you have multiple workers and they're all lined up and then they all run concurren concurrently, right? So, concurrent futures allows you to abstract away with a thread pool executor. So, that uses multiple threads within one single process. Remember the picture before one process, multiple threads or you can use the process pool executor that uses multiple processes, single thread each, right? That's the kind of idea behind it, right? And then they return you a future object. It's basically like a promise. So, the future object says, "Oh, are you finished?" No. Okay, not okay. Come back later. Oh, you finished. Okay. What's your result? Oh, no, you don't have a result. you have an exception because something went wrong. So that's the future object is just like a a promise of what's going to happen in the future. Either you get the result back or the exception from the worker basically, right? So that's concurrent futures. So let's look at some code. I mean this is some some of the stuff that we do at work quite often. We do image processing like a standard movie um you know has million frames in it you know million images in it. You know if it's a 48 frames and stereo yeah it's a million frames for three hours. So all this code does we have a a list of uh 50 images you can see in the line three there and then we do some processing. So obviously the function that does uh that is called process file in line number five. It loops through all the pixels. This is 512 x 512. So this is really a thumbnail. Normally we deal with 4K images but this is just 512 x 512 pixels. And then the line 8 really is a standin. So always do you random basically means give me 100 random numbers. And this is extremely expensive to do for an operating system to give you proper random numbers. So that's just simulates CPU work on the pixel. So that's that's all there is, right? So we're we're looping through 50 images and then for every image we loop through every pixel and then we perform an operation on it. And the the OS uran normally would do apply gamma or something like that. But the OS urandom is just a standin. So we can actually generate some work for the for the CPU, right? So then this is this runs sequentially and I I'm going to give a live demo in in a minute about that one and so every single file is processed in sequence on one single core and it just runs that program just runs now let's just make it faster you know how can we make it faster with concurrent futures I mean that's what we have this library for so first coming back to the slide with the multiple threads within a single process it's exactly the same code but you can see in line 10 now we do with concurrent futures and using the thread pool executor. We say like for every file that you have in this list of 50 file names in line three, simply call that process file but call it concurrently with as many threads as you can find or as many cores as you can find. Right? So that's what that library does for you. So the library goes out creates all these threads, calls these things in parallel and then gives you back the results, right? Yeah. I mean in this case it doesn't even return anything but it just processes the image in place. Yeah. But that that is the idea behind it. Now, how can you make this? How can you turn this from a multi-threaded single process execution into a multipprocess uh framework or solution? Trivial. See, like there's almost the same code. All you have to do in line 10 is replace one single keyword. First of all, it was a thread pool executor. Now, it's a process pool executor. Suddenly, you have now you have 12 processes each with a single thread and they're all running in parallel. Right? So what we're going to look at what we're going to have a look at now is we're going to compare the the way that these three programs run and how how uh how much time they take. Right? So let's see if I can um bring this up. Um so we got this concurrent here. So we got the three examples exactly like we saw them before, right? So we start with the first one and it's um it's really simple. We'll just um we'll just run it and see how long it takes. Um so remember this just runs them all in sequence. Oops. Python 3 of course. So as you can see these images are getting processed right one at a time. And now one single core is spinning at 100%. And that's it. That took us 8 seconds. So now let's go to the thread pool executor. Remember the uh the second slide that we had there. This is now using concurrent threads, you know. So, let's have a look at how that runs. You can instantly see it already started up 16 threads at once. But then what do you notice? It's just stuck. But if you look at the CPU usage now, you'll find multiple cores. They're both packing out. Oh, the next 16 coming in, right? So now the next 16 threads are running. And um but this is already taking much longer than the single threaded example beforehand. This is actually far worse, you know. So, we're just going to uh let this uh finish through there. I think this is now coming up to the end. But you can already tell that that this is u this much worse. So, even though we've paralyzed the work within a single process, this now runs much much slower. There you go. 39 seconds and the other one was eight. This is almost a 4x slowdown. But by us trying to make it faster is completely ridiculous, of course, you know. So now what we do we'll use the the exactly the same example that's so this was threads so pull up the thread pull executor here right so that was the extremely slow one and now we run it with a process pool executor same same code exactly the same code watch what happens bang finished so this is now less than two seconds so now you can see there's a real gain there moving from a single threaded you remember the first example just doing one process one thread one one file at a time we're doing it with threads four times slower. I'm going to explain to in the future the of this talk a little bit about how we can make this faster with threads as well. But multipprocessing gives us this kind of immense win, you know. So now suddenly it's four times faster. So that's actually what we want. So that's we're going to continue down that route now. And um the reason for that being four times slower that when you're using threads is the um Python interpreter the the global interpreter lock the gill. And we're going to talk a little bit about that um later on. Remember when I showed you the picture with the multiple threads and they all have to have this concurrent access to the memory and the files and and the locks that they use and that's exactly where that slowdown comes from. The contention comes from having to lock the global interpreter lock and I'll show a future example of how we can fix that. But for now, let's just concentrate it on the multipprocessing makes it extremely fast, four times faster. So that's good. We don't want it four times slower. We want it four times faster. So we use multiprocessing, right? So that's why we're doing it. So that's what I touched a little bit about before. So the multiprocessing mod module is um where all the heavy lifting is done, right? Standard Python library module. So the concurrent futures with a process pool executor goes into multiprocessing and it it tries to emulate the threading API. It's got the cues for the interprocess communication. The cues they use sockets, pipe, shape memory, whatever they have and then they they have support for fork or fork server spawn. These are different ways of spawning processes and we're going to talk a little bit about that now. So multipprocessing module obviously now as we've seen in in this example is responsible for spawning 16 processes right you know yeah however many it it deems fit based on the hardware that found there right so there's different ways that it can go about the spawning these process or creating these process for you and then collecting results and tearing down again at the end so we're going to look a little bit um into the start methods right so the original start method of again for Unix platforms there was fork right create process on Windows and it just forks a process when needed. So your Python code encounters this multipprocessing or concurrent statement says okay the the user now needs 10 processes let's just create them with fork one after the other. Um it can also use spawn right this is uh this using fork fork exit or spawn pix spawn which is a new implementation of creating a process. Uh and the third one that is relevant in this context is the fork server. So the idea of a fork server is actually quite simple. You throw the server out at the very beginning of your program and it just sits there and it's single threaded and all it does it waits for the process to say oh give me 10 processes I need five more okay and then it forks out these processes on behalf of the main application so that's the only difference the fork server is so fork server is basically a second process single threaded that just sits there waits for you to ask it to spawn processes and we we'll see a little bit later why that is useful but so you can see that the defaults have changed here as well across the different Python versions This is quite important. So, Windows had always was born, you know, there's no fork in Windows, right? You have it in Windows Subsystem for Linux if you want the Siguin as well and then MinGW or whatever they have some kind of implementation that emulates fork, you know, but natively Windows doesn't know anything about fork. So, that's why it's just great process and it's a spawn context. Now, Mac OS, we'll talk about that a little bit later. In 3.8, Python 3.8, They changed from fork to spawn and Linux has now followed suit as well. Um and we talk a bit why why fork is broken. I mean if this is really because fork is broken. So they changed from fork to fork server in the 314 uh because uh that was just released a few weeks ago um because of the deadlocks that we'll see now as well. Okay. So yeah sub process I briefly touched on that one. This is the original standard library. It's more like for in spawning individual programs collecting the output and with a timeout you know so that's that's what that is for. So just to for completeness sake I'll just mention it here subprocess you can say run me this this command on the you know on the shell collect standard out standard and then after 30 seconds if it hasn't finished just kill it you know and then give me back the results that's all there is right and the implementation again can use v fork instead of fork or pos spawn instead of fork exec is the same kind of concept there but that's that was sort of the original way of just spawning a single process manually so that was subprocess okay so what's this fork business all about right that goes back to the 70s. I mean, as I was said, I worked with Unix for 30 years. So then, um, this is much older than I am. Um, you know, probably as old as I am. And it duplicates the calling process. It's really important to understand what's actually happening here. So when you call fork in a process, OS.fork in Python, it makes a full copy of the whole process. That includes your memory, your open files, sockets, all that kind of stuff, mutx, conditions, everything gets duplicated. But every other thread but the one I called fork gets killed wherever they are. And that's that's the fundamental problem with fork because fork was there way before threads you know. So that was like there 50 years ago. That was the only way of creating a process on on Unix. But the threads came much later. They came only in the early 2000s for example in Linux with the the native posics thread library or threading library in Linux. And as I said all threads die in the child process except the one they called fork. So that's now obviously causing problems from other threaded application because if you think about it I think I have actually have a slide here. There you go. So this is now what happens in a fork. So now we'll look into it right we look what's actually happening. Okay so you can see we have a multiro a multi-threaded process on the left has three execution threads the green circles there but only the third one the third thread the the green uh the third green arrow there calls fork. Now what happens in the copy? So there's two processes now. One on the left, one on the right. The one on the left just continues running as if nothing had happened, right? The one on the right is a full copy of all the files and memories and the logs and everything except for the two threads at the beginning, thread one and two, and they just died wherever that red access, you know, and that's where the problem comes from. You can see thread number one happened to h happened to have a hold on a lock, you know, just because it was trying to access some shared resources. And then thread number three called fork and it was just killed off. Bang was dead. And then suddenly there's no time to clean up. Nothing to release the locks. That's the end of that thread. Same for thread two. It was not in a critical section where where it was. It was just killed. The only thread that's running after fork is that thread they called fork in the parent process. And it's a single threaded process now, right? And all the resources and everything was copied. So the parent process doesn't have to worry about anything because it just continues running as before. But the child process now has this problem. Right? So you can clearly see that these locks are being held and resources are not being cleaned up because that the thread set just got killed off. Okay. So this is now the fork part of the talk. So let's have a look at quickly look at this um simple program and we can have figure out what the uh bugs are in this program. So line number three again this is all standard libraries. Line number three uh just simply creates a thread and what are we saying here? Okay, we're saying we want to we want to run print and we just want to print the word thread. It's easy, right? Okay, and then we start it. That's it. So that's a multi-threaded program right from the start. We have created a second thread and it runs. It just prints thread and then exits. But that's fine. Then in line six, we call os.fork. So we're forking like in the in the in the diagram before. So now when pit returns, basically when you get back zero, you know that you're the child process and the parent gets back the pit of the child process. So you can see line eight, that's the child process, and line 11, that's the parent process. So the parent process in line 11 just simply says, I'll wait for the child process to finish and then print finish. That's it. Uh the child process says, oh, all I want to do is just say print process. Yeah, I'm the child process. And then exit. Okay, so um there's two bugs on the code. Um can anybody spot those bugs? Yes. Oh yes, you are you are correct. The the I should have put a a thread t. join. Yes, that's that is third bug. Very good spotting. That's a correct slide. That's actually good spotting. Um I have it in the in the and I'm going to run that code live. I think in the live code I have a t join but you are correct. That's that's um a third bug. But there's two two more. Exactly right. Yep. Yep. Exactly right. That is the correct answer. So the first bug is a race condition between line three and line eight on the print statement. That's correct. Like uh the audience member has correctly um pointed out the thread in line three that was spawned could still be running. We don't know. I mean, it's only printing something, but the the by the time the fork happens, it could still be running. And it was just trying to print the word thread, but it got halfway through and then it got killed off. So now in line eight, if if the child process now tries to print process, the thread in line three in the child process is dead. It's still running in the parent process, but not in the child process. And and thus the uh the print statement fails. That's correct. Um so this will actually deadlock. So I'm going to do a live demo now. Um there's there's a third bug. That's exactly right. Systems exit is suffers from exactly the same problem. Yeah. So what and don't take this for as a gospel. We're just going to run that now. So then you see what happens. So this is now what I call um so we'll just have a look at the code. And so we got the deadlock here. See if that I think I might actually have a join in here. Oh yeah, it has the join. So yeah, thanks for pointing that out. [laughter] It's made it in here, but otherwise it's exactly the same code. So we're printing and we're forking. So now the the interesting part about this one, this is now undefined behavior as Oscar pointed out very nicely. And undefined behavior is that I don't even know what's going to happen. So let's just run the program and see what happens. So that deadlock we'll just run that first. Okay, already there we have it. The deadlock has deadlocked, right? So a couple of things have happened now. So if you look at the output, you can see thread. Okay, so the thread has printed. Wow, but what happened now in line seven where we called fork the deprecation warning from the Python interpreter actually telling us this process is multi-threaded use of fork mal to deadlock and child and you can clearly see the weight p is being called as well and uh we're deadlock there's there's nothing happening anymore right so now what you what what we're going to do um we're just going to do the split terminal here and let's close this off a little bit so you can see that this process is just hung it should have just returned I did put the join in here. So I should have just returned and exited, right? So let's just um have a look inside that child process. Right? This is the one that's hung, right? So we just use GDB and um let's just have a look where some Python were hanging. There you have it. Look at that. That is it. The print is deadlocked. That's our deadlock. Right? So now you can clearly see line number nine in the stack trace is deadlocked. Right? I mean if you're interested in the uh in the back trace the Python back trace as well you can see where they call come from and if you're interested in the C++ or sorry the CPython side of things right you can look at the um the C back trace as well and here you can see the lock there right as as you pointed out there's a buffer writer text IO writer so that's pretty much what has happened right so the important thing is the print statement has deadlocked us right so that's what we wanted to So um and the message from the Python interpreter actually tells us about that. It tells us my deadlock and I go like yes we see that right now the print statement has deadlock. So now let's just run that a couple of times. Oh that did locks every single time. Like when I was testing it before it wasn't. So obviously something has changed but as like the race condition triggers every single time. I I thought I could get it to to to run through but now you can see that right. So that that is basically what we're saying here. So that is that print that print statement here that kills us. And yes, this exit suffers from exactly the same problem. So now that's really concerning of course because what do you do if you can't even do print? No, then we'll find out. So that we we know now that this happens and this is real. You know, this was just a this not a contrived example. It's really simple example. just a thread uh two processes and a couple of print statements. Right? So what we've seen we can use GDB like pi bt or pi back trace pi list shows you exactly where in the python code you're dead logged you can dive into the c python layer if you want the print function dead logs after fork we found that the logging library did logs oh that's actually the other one I wanted to show let's just have a look at there because we got let's say oh this is the problem with print I've got exactly the same code here so now instead of printing all I'm doing it's logging do warning right Let's run that. So there's no print statements in that code. It's just using login.warning. So it's got the basic config there and it just calls login. Otherwise the same code. Now watch this. See this has succeeded. Let's run it again. Succeeded again. Oh there it is. So this is the kind of intermittent behavior that I was talking about undefined behavior. I run the same code exactly the same code multiple times and I get different results. It's not insanity. No, this is experience. you run exactly the same do exactly the same thing over and over and over again and you get different results and that's exactly what's happening here and so now you can see this this is the same deal if you want to attach the debugger to it you can see the loginwarning call is stuck you know but it's exactly the same problem right so it's nothing to do with the print statement um so that was just basically showing that this this is not just limited to the print statement let's have a look okay cool exit as well that we pointed that out that that locks as well go to the exit handlers. Okay, so why is that happening? After a fork in a multi-threaded program, the child, what can the child actually do? You can't print. You can't call can't log. What can he actually do? He can't call exit. You can only call async signal safe functions. And that's the crux of this talk. Async signal safety is the key. Print is not async signal. Can't call it. Logging is not as signal safe. Can't call it. So exiting is not asyn signal safe. You can't call it. Is there anything we can do at all? That's a good question. Yes, there are. Okay. Okay, so as signal safety quick, you know, kind of degression here into the Unix world. Again, this comes originally from signal handlers because remember the original Unix concept didn't know about threads. It was just a single process. But sometimes you needed to interrupt a process and tell it, oh, there's something you need to do really quickly and then you can go back to what you were doing. So that's where the signals came in. You could send a signal to a process like, oh, there's an interrupt. Pick up these network packets otherwise they disappear. Okay, cool. I'll do that now quickly and then go back to what I was doing. So async signal safety means that these are functions you can call even in inside a signal handler because there's a lot of restrictions on what you can do inside that and there's only there's there's there's an absolute list you can look that up there's a man page for that on Linux and Unix there's 200 safe C functions only nothing else so definitely not melo or free you cannot allocate memory it's bad idea but you can call open read write really good you know so those kind of things you can do like open read write you and close you know receive send on sockets underscoreexit as compared to the exit which doesn't call any of the um exit handlers and doesn't tear down all the global destructors so underscore exit is safe to call and nothing else especially no print definitely not allocating memory that's a bad idea allocating Python objects on the heap not good not good so now okay so this is the the fix right so you can see it in line eight I've taken the print away and the logging you know and I I'm using o write why why am doing that because always write is the direct operating system in uh interface into the right call and that's asynch number two means standard error so zero is standard in one is standard out two standard standard error and I'm printing the word process just like I'm printing before and I'm not calling sisexit I'm calling oscoreexit because that's async signal safe doesn't run any of the um tear down code so again the um proof is in the pudding let's have a look if this actually works So this is the fix now. So we're just going to I think the uh here we go. We're just going to close this shell. So yeah, this this has killed it. So that's fine. So this is now the deadlock fix, right? Oops. So I'm just running the the fixed version. So you can see it prints threads, it prints process, and it finishes. It still gives us a warning. Let's run that again. Yeah, it still gives us a warning, but that's fine. But that's obviously that doesn't tell us anything because I just ran it three times, you know. Let's let's do this properly now, right? So remember this is now using async signal safe functions only, right? Uh and underscoreexit. Oops, it's not helpful. Um running it once doesn't tell us anything as as we've seen before. We want to run this forever like that. So this means just run it forever. And now now we'll see what happens. There you go. So that's running hundreds of times, thousands of times. I can run that overnight, million times, no problem. Right? So this works, right? So again, the proof is in the pudding. This is running. I can leave that running until the end of the talk. It'll still be running, right? So obviously this is completely impractical. [laughter] But yeah, what's going on here? What's What are you you can't use print, you can't use exit. What's obvious? It doesn't make any sense, right? I mean, nobody's going to write code like that. One in a thousand programmers write code like that. You know, I sometimes have to write code like that, but I tend to avoid it. It's not pleasant. It's not it's not very pleasant. Now, I can tell you it's not very pleasant. So, okay. So, we know that now. Okay. So, we we understand that that this is the problem and how we can fix it, but this is not practical. This is what's going on. So, people have thought, okay, let's clean up that mess. you know, we know that we got ourselves and you know, on the twist and it's all kind of broken and all these locks are hanging around from these threads that got killed off. Let's just clean them up. And they go like, "Oh, great. I always register at fork." There's a there's um a Python call that you can do. And it basically says, I want to register something that calls gets called when fork when anybody calls fork and then I can clean up clean up my mess, you know. So, you can give it three arguments. One gets called in the parent before the fork actually triggers. Then the other one gets called after the fork returns and the parent and the other one gets the third one gets called in the child. So the idea is in the child you can go a it's all this mess here let's just try to clean up that stuff and the problem is they tried doing that right and the standard login library is riddled with this to this day you know just all these registered fork and trying to clean up all these locks and but the problem is after child is not async single save either you know you cannot clean up a lock it's not async signal save they tried and they failed okay yeah there you go it's not asyn it's not async single save there's nothing you can do you can register something to be called in a child but you can't even what what good is that? Let alone trying to clean up any locks or freeing any memory or something like that. It doesn't work, right? So, this is garden path. I I recommend staying away from that. We just said fork. Um P3 fork is the C equivalent. And um so that's the underlying one. It's not fit for purpose and the Austin Pix group has finally given in and said [laughter] like no, just deprecate it. This was such a bad idea to even introduce this you know that like this kind of cleanup functions after fork and he goes like nope nope don't do that. So there there's a proposal in from the posit exhausting group to deprecate all that mess. So if a fork calls call in a multi thread process leads to a child fork handler calling any function that's not as single safe then the behavior is undefined. And these are the crashes and we'll show a crash later on as well. Deadlock logs that we've seen and Oscar pointed that out as well. Undefined behavior not good. Don't want that. It's basically unpredictable. You don't know what's happening. You run it once works. Run it again doesn't. So that's bad. So the this is kind of garden path territory, you know. Yes, they tried that for many years and trying to clean that stuff up. Didn't work and then they walked away from it. It's just still in the logging library. There's still stuff there that's trying to mitigate that problem. Okay. So what what can we actually do? So this is what I've used uh myself, you know, vfork. So it's similar to fork. It doesn't duplicate the RAM. So it's exactly the same RAM with a parent process. So you have to be extremely careful what you do there. Other resources are duplicated. But it doesn't call any at fork handlers. That's the main that's the key point there because the problem was especially if you have a really complex system like Maya with dozens of plugins and then another dozen of your third party plugins our own plugins and this all gets loaded in and people registering at forkanders left right center is such a mess. So you just really want to bypass that. Vfor does allow you to do that and but you can virtually do nothing in there. It's still the same restrictions apply and you can just only call exit or exit you know. So because you're sharing the same memory so it's even more restrictions apply but it doesn't call the add fork handler. So that's a that's a good thing you know I don't want those and then pix spawn came along that's actually comes from the pix group again and that's a replacement for fourth basically right so it's safe to call for multi thread applications at any time and it spawns a new process with a completely new executable so there's none of that duplication and killing off threads it runs a new executable and it uses vforoke internally they figured that out pretty quickly as well so as soon as the posics group have implemented that and they go like let's use fork they go like no this is broken let's use vog yes good idea So even they agree with that. So pix spawn with vfork is the solution, right? Okay. So uh it's a bit bit of a whirlwind there. Um so don't use at fork handlers. They're broken inherently because remember in the child after a fork there's nothing you can do if your program was multi-threaded. If it was single threaded nothing f there's no problem whatsoever. Remember all that stuff I've discussed all these problems only when you have multiple threads in a process when you call fork and then I show you later as well one when that's not inherently obvious immediately. Um do not assume your program is single thread simply because there's nobody creating a thread. Sometimes threads get created for you by third party libraries underneath the hood you don't even know about. Right? Use posics spawn for a clean alternative. And I wanted to briefly touch on that Python gill environment variable. Now this is a new addition to uh 314 now as well. So we can turn on and off the gill. That's amazing. You remember the at the very beginning of the talk I showed that it was four times slower was with with all these threads. You know the program run four times slower. Let's just go back and uh and turn off the guild the global interpreter lock and see what happens. So let's um let's just go back to that. Um okay. Oh cool. I [laughter] have to to go go have to go quickly. Uh what one thing I wanted do wanted to show so I'll just boot up um let's boot up my Mac in the meantime but um so um with that Python gill I just wanted to briefly show how quickly that can run but I also I do have some more bonus slides but um let's just yeah let's just do that. So, I've got a virtual environment here. Um, no, it's a way to have it. I think I've got this. Was it 314? I think, isn't it? And then no, I don't. But the idea is Z. I can now run this uh concurrent thing. Let's just have a look. Concurrent. So remember this one here that we ran before the threads one. So you can immediately see the threads getting created but then it gets stuck. So it's really slow. So let's just turn off um let's just turn off the gill. So how does it work? So let's just turn off the gill. Oh and the time as well. Oops. Oh, okay. Oh, sorry. I I know what I've I've done wrong. No gill. I want the nogill. Sorry. Does that work? Huh? Oh, there you go. Yeah. So, now you can see this is without the gill and we're actually a bit bit better. So, you say I don't need. Okay, cool. And then just Does that work? Yeah. See, that's much quicker. So, now I've turned on the global turned off the global interpreter lock. So, this is much quicker now. So, that's what I wanted to show you. Oh, and like last thing very quick. Um, this is now a Mac and I wanted to show you a quick crash because uh that's always good. Hard crash, you know. Yeah, of course. So, this was just deadlocks, you know. This is just for beginners. Now, we're on hard crash. So, you can see here again, this is using the request library quickly just to request something from the um um from the uh from a web server. So, let's just bring up a web server here as well. Python actually has a web server built in which is really good. So you can just run it like so. So then web server running and let's just see if we can crash this guy. Um what is it? Yeah. And normally come Oh yeah, there you go. Magic. Magic crash. There you go. And what does it tell us? It crashed hard. Sick kill. You know that's good. There was a sick kill that it got and the reason said look at that multif process forked. I've killed you. There you have it. Okay, that's me. Thank you very much. [applause] Thank you very much. Uh, I think we have time for one question. Oh, up the back. Making me work for it. I'm just wondering with all the kind of jumping through hopes to get Python more performance and all these kind of traps that you have there. At what point do you say well Python's maybe not the best language for performance anyway and is there a point where you go I'm giving up and I'm going to try something else? >> Um no I think Python can be extremely performant especially if you're using things like numpy or the xarray that was pointed out in one of the previous talks. um you just have to know how to use your resources appropriately. For example, with a the with the with a gill turned off or with a multiprocessing module used in a correct manner, all the cores were picked out at 100%. So you can use all the cores like 12 cores on that machine and they're all sitting at 100%. So you can use you can write performant code in Python. Um you know it's it is possible but you just have to be aware of the performance there. And yeah, go and see avatar that comes out very [laughter] FIRE AND ASH. AND I recommend watching that on the big screen. There's the embassy theater just around the corner and IMX lower hut. Of course, IMX lower hut. [laughter] Yeah. Cool. Cool. All good. >> Thank you very much. [applause]