How many spoons does your environment cost: Broken demos & human element - Dawn Wages

[applause] >> Hello everyone. It is a pleasure to be here. This is the second time I'm giving this presentation. And so, what we're in in for is going to require some audience participation. You're going to see some failing code five times exactly. And then when it fails, I'll let you know when it happens, but I would love for you all to cheer for me. Is that okay? Okay. All right, so let's do a practice. Yeah? Okay, let's do it one more time. 1 2 3 Oh, okay. I like this. Okay, this is really great. All right. Uh so yes, my name is Dawn Wages, she/her pronouns. Um and it is an honor to speak with you all today. This is an experimental version of a talk that I've given about a half a dozen times. Um and I'm adding in a human element to the abstraction and process considerations of your Python environment. This is going to cover why it's so hard to build and maintain software in any environment, but especially specifically a Python environment. I'll have a moment of vulnerability. Feel free to laugh at me, the last audience did. Um when I came into product aligned work a number of years ago and I was working on VS Code with some of the brightest and most thoughtful Pythonistas I've ever met. Um I said, "Well, are Python environments even that hard?" Yeah, yes. That's where the laugh goes. Yes. Um and it was a resounding yes. Um and almost a duh. And I've thought a ton um over the last um maybe five or six years that I've been doing this on why I thought it could have been easy despite it being incredibly hard. One, I internalize failure. When my environments broke, it was me. It wasn't hard. I must have been dumb or lazy or didn't read the right docs, or tired, or something. Uh two, I came into programming in around like 2016-ish area, and it's a time period where we were aggressively solving for some of those considerations. Really trying to make the the environment and getting started that 101 level experience of development much easier. I mean, I my first uh deployed Python site was through Django Girls, and the if you've ever been through the Django Girls documentation, they really step you through the process. And also, number three, I've always viewed programming as a team sport. I came through Django Girls, I've always worked in lovely teams. I was lucky enough to start programming before the um remote um options of in of development, and so I had a a team member on the next cubicle to talk me through and help me through processes. So, game time. As I mentioned, in this talk, we'll have a small game. I'll be running those scripts that are designed to break on one side of the screen, and when you see it break, I'd love for you all to cheer. So, we're going to hack our brains. Let's give it another try. I already know you did it before, but let's do a woo. >> Woo! >> [cheering] >> Yay! Thank you. Um I lead community and developer relations at Anaconda. I also co-host a podcast with Luciana Bud, and it will be returning from hiatus this summer. Um and you also may find me on the internet experimenting with local-first AI and ML. So, who all, and I guess a raise of hands, it's a bit dark, but I can see the hands. Who's familiar with spoon theory? Okay, cool. But, it's enough of the audience for me to kind of go through it. It first was coined by Christine Misia Mis Ren Misia Randino, uh a real originally about chronic illness, and it maps disturbingly well to the hidden attacks of cogni- cognitive works, and so that's what I'm going to be uh referencing today. The core insight isn't just energy is finite, which is what it covers, but it's that different tasks cost different amounts, and the costs aren't always visible, and you can't always choose when and how you spend it. There is no connection to why a spoon versus another object measure. I got a hallway track conversation last time of like, why spoons? And they were thinking about it the whole presentation. Just to close that loop, spoons are what was on the table when Christine was describing what it was, but it describes a finite number of what you of your energy in any given time period. And from this terminology, spoony grew into an identity term for self-identifying as someone with a chronic illness. Every task costs spoons, and you can't push on willpower alone. So, we're managing with limited energy, and your environment setup is almost entirely miscategorized as free overhead. When a developer sits down to work and hits these conflicts, an ambiguous path, a broken environment, they're spending spoons on something that was supposed to already be done, and that cognitive cost isn't just a technical challenge. It's an interruption in attention and intention, and that is shared amongst your team members. So, the goal of this in the audience that's for this presentation is for team leads who are managing environments, for individuals who may internalize failure like I do, and for people who are builders, who are maintaining some of these systems that are interruptible, and they have these pending tickets. They know that there are some bugs, and to prioritize or understand the and have empathy for the bugs that they're fixing. So, sometimes you're a developer, and you're starting your day already with a limited amount of spoons. Sometimes we're not lucky enough to wake up fully refreshed. You spend the cognitive effort at the beginning of the day to write down your to-do's. Maybe you understandably forget some personal to-do's in your first draft. And then you have your tax task that will be ultimately need to be expanded or adapted. When you fill your cup and you eat breakfast, you stretch before you sit down, maybe your spoons go a little bit back up. And then you start your day. Let's start with our first broken code and then we'll continue back on to our personal experience. So, All right, this is going to be a first one. Let's get bigger. Okay. All right. So, what's happening here? Uh we have the application we're running into conflicts in our package dependencies. So, CuPy is a NumPy compatible array library in uh CUDA GPUs and it does not support Apple Silicon, which is what I'm running this on. If your code base uses CuPy for GPU accelerated NumPy operations and the Apple equivalent is going to be the Apple ML framework NumPy compatible MLX or torch with MPS backend or JAX with metal plugin. Regardless of the in-depth scenarios of this, none of these are drop-in replacements. There are migrations required auditing CuPy call CuPy call. All right. And so this is going to be hardware understanding the hardware assumptions versus hardware realities. These are going to be failures at import time and device check time and the code is going to be written assuming one thing but actually finding another. So, let's return back to our our your You knocked out an issue between this and the last uh last slide. And your list gets heavier, and you try to remind yourself to take care of yourself. Drink water. Mentally prepa- prep- uh prep. Don't forget lunch. And your day just somehow continues to get even more hectic. And without reprieve, you just make it through. But even what you have on paper isn't all of it. The things that require feedback. Uh sometimes you have to stop and change gears and restart. And again, more context switching. And you may ask for help on on your on your teammates' terms, and by the 2:00 p.m. slump, you're almost empty. So, here's what we're going to cover today. We're going to talk about why environments fail, and then we're going to talk about why we fail. Uh let's do another demo. All right. Okay, what's happening in this one? So, this one's about dlopen. The extension was compiled expecting to find libgiomp.so.1. These are .so extensions in the pip lib directory, which is exactly where pip would put it on Linux, but I'm on a Mac. So, conda doesn't ship uh this extension because Mac OS doesn't use that name, and so we have what is a naming mismatch. The package installed, the pip was reported success, and the wheel landed in site-packages, but the failure only import um appears at import time. We've missed the first cheer because I was too deep into my um into my code, but here we go. Failing code. >> Woo! Woo! >> Thank you. I really should have done that the first time. It really helps. So, thank you. All right. So, let's go keep going back to the Prezi. So, why did it fail? This was a naming mismatch. No amount of searching can fix a naming mismatch and an extension compiled on Linux with GCC hard codes its package name at its run time dependency, but when you're running on Mac, which is what I'm doing, uh the dynamic linker searches every known path and can't find it. So, if it finds a problem at run time, not before, and this is just a factor of different conventions and operating systems. All right, so let's talk about the two different ways that you are inter- interacting with your two different paradigms that you're interacting with your with your environment. First, I'll cover and this is going to be the majority of it. I'll talk about the layers of abstraction in your environment and then I'll also talk about process. So, here are the seven layers that I've articulated as the the process. So, you start really low level, physical firmware. You have your hardware abstraction. You'll have your process and your operating system interface, some of which I've already mentioned before and you've already seen this kind of abstraction layer. Environment isolation, then your package dependency, your run time interpreter, and then your application and your user intent. This serves as a model for creating resilient environments that can traverse this stack. And then the process, right? So, this may look really familiar to just any iterative ideation process as you're building, you're planning, you're designing, you have your build sprint, you're testing, and then hopefully you're releasing. But, abstraction can be messy. Extensions can bypass run time, layers can be substituted for each other, and there's also novel developer hardware. And your process can be messy. This is more like what my process looks like. You plan and then you plan again and then you design and then you turn it up on its head. Maybe you start building and then you stop and you test all the way through or something and then you got to start again. And the process is not all cut and dry. We have false start starts and stops, designs and redesigns, testing in imperfect ways, and ecosystem considerations and releases. So, let's go through some of these abstract the layers of abstraction. The application user intent is what the developer is meaning to do. The layer where the correct algorithm fails because the environment was written for that environment and it could have silently diverged. The one time interpreter, this is where you have that translation layer between the Python syntax and your machine instructions. So, this is CPython, PyPy, Py 313T, this is all different kind of flavors and variations of what's going on in your runtime, your language in your runtime. Um but each make different promises about your global interpreter lock, for example, uh your garbage collection, or what C extensions that can say that can be safely assumed. And if you want to go deeper into this, we'll just do a pretty quickly. I'll run through some of these, but you also have what's happening during that runtime interpreter layer. And this is how Python works. CPython does seven several things in sequence. It'll parse your code into an abstract abstract syntax tree that compiles tree into byte code, and then have lower level sequence of instructions stored stored in your PyWI PyC files. Uh then you'll execute that byte code in the Python interpreter, uh excuse me, and the Python virtual machine in a stack-based interpreter loop written in C. And that pipe byte code is not machine code, it never runs directly on your CPU. And that CPython uh reads each byte code instructions and executes the corresponding C code for it. And then if you're looking at where that C API is, that's going to be where your source level API for if you are using other languages and then you have underneath that the ABI which is your binary level which will define function signatures and macros, headers and stuff and in more. Okay. Um next we'll talk about package dependencies. So this will be that social contract between your library authors and the versions that can coexist um in your solving. So this breaks when the contract is implicit or under documented or written for hardware targets that are not yours. In your environment isolation, so this would be your virtual environment, your venv or your conda environment. That bounder boundary says this project's dependencies are not that project's dependencies. venv, conda, docker all attempts to draw lines around uh that OS it doesn't naturally uh that the OS doesn't naturally enforce. So let's go to another breaking code. No, not yet. Almost, almost. >> [laughter] >> But thank you. I love how you're ready. So we're talking about mounting in this one as well. So this one's going to be about a docker fail. Um your platform is going to be mismatching in this one. We're unable to allocate memory. So at one time when you are uh um defining how much memory you have and you're running it on an emulation, um it may run slower and you may run out of memory. Also docker does not inherit your shell environments and in this particular one, timezones are hard. Woo! [cheering] Thank you. Thank you. Y'all are on it. I love it. Okay. So this one's more about emulation. So the failures that are discovered either after the fact which would be like wrong data or external observations or a downstream issue like time zone like set in setting US UTC for example uh will create problems later and not at the installation installation time. Let's keep going. So, we'll talk about process and OS interface. So, this will be more about file descriptors, local, signal handling, memory limits, mount points. Um the OS enforces constraints that the developer didn't know existed until they're violated. And we showed an example of that before. In your hardware extraction, we're getting really low level now. This is a layer that pretends all of the compute is equivalent. So, if you're using CUDA, MPS, like I've mentioned before, it's metal, and CPU are all not the same thing and they have different names. Uh but they have different and they have different D type support. So, this means that there is different precision guarantees and different failure modes at the same API surface. And then finally, you have the lowest level, which would be your actual hardware or your firmware. The layer that doesn't negotiate. This is where you have I have my Apple silicon that doesn't have any CUDA on this on this particular machine. Uh the disk is full, for example, or you have network cable that is unplugged somewhere, or um or some other hardware-related uh integration. And no amount of correct Python is going to resolve resolve the fact that you have differences in the real world in the physical world. And this is your example test environment. This is a process of what you're going to build your widget. And my build process, as I mentioned, is a lot different. I'll have an ideation phase. I'll have a plan phase, a test phase that are written against environments that may or may not exist or uh release flow that I'm assuming, but does not always match reality. Uh this will be the last demo. And no almost done. Almost done. Still. >> All right, so what's happening here? So, we're seeing some dependency conflicts and the wrong interpreter. Uh we also see conda and pip are package uh clobbering our packages. For example, conda and pip are putting the packages in different locations of your machine when you're when when installation. And so, when it goes to look for it, it doesn't know which version is supposed to be uh which. Although, there are some tooling that's coming. Find me in the hallway track. I'll talk to you about conda PyPI. We also have in here a broken editable install. Uh so, if you're have a mono repo, for example, and you're um pushing your code and the editable package is not there, it's not going to know until runtime. Um and then also in this particular one, we all have an architecture mismatch. Uh cheers. Woo! >> [cheering] >> I really like that. I need to build this into every one of my presentations. It's so good. Um Great. Thank you so much. So, that's the last of the demos. But, I want to finish with uh So, this is what happened. When it's installed, what installed and when. But, now I want to go into just the human focus. Why do we fail? Well, uh here's an example of where our memory model may be. So, there you have cognitive limits and I want to explain a little bit more about them. So, the critical point is capacity. The central executive uh has a fixed ceiling. You also have your phonological loop. Uh these mean that there is a structural uh constraint on your cognitive system. It holds sound based off of energy information before it decays. Right? So, you're going to have uh a point where you first learn and it's going to be most crisp in your mind and then it will continue to decay over time. You also have your visual space um spatial sketch pad. So, this is visual and spatial memories respectively. So, if you are one of those mind palace type of people, this is also a hack into your brain on how you uh can build your memory models. And then we also have the episodic buff um buffer. So, when you have limited capacity, uh you can have a temporary store that integrates across memory and components. And at the bottom of the screen, I also have uh research that uh describes some of these. The spoon is not running low because you're tired or frustrated. Uh it's running low because the working memory architecture that enables complex thought is occupied with infrastructure that was supposed to be invisible. And this memory model kind of helps us kind of go through some of these cognitive limits we have. One of which I also want to talk about switching tasks. So, every time an environment breaks, it forces a task switch uh switch, which is a goal of reconfiguration cost and a rule activation cost also load, which means loading the procedural rules for that new task. Um I'm calling out Meyer and Rubenstein's 2001 paper specifically, um but I also have Gloria Mark's uh paper at the at the bottom just to kind of uh notate where the references are. Meyer and Rubenstein's paper demonstrated the costs were not reproducible to a simple surprise or confusion. They persisted even when participants knew exactly when and what they would switch to. Gloria Mark says, which is the most cited finding, that it takes on average 23 minutes to fully return to a task after interruption. An average task that includes the entire chain of secondary tasks and interruption triggers. So, just kind of giving yourself grace that interruptions actually do and research-wise um really slow you down. And then the last is miscategorization. So, as I alluded to in the very beginning, this work is not free. There is invisible labor in the environment tasks that are um showing up in your development workflow. So, setting up a Python environment reads as I've done this before and gets budgeted as maybe 2 hours. But, the fact remains that the environment has different OS constraints or different package ecosystem or different hardware target. None of that invisible planning time is factored in and the estimate for that categorization environment setup is not specific for whatever your use case is. So, giving yourself grace and understanding that your team is also going to have to traverse and traverse this as well, good documentation helps, but understanding that there is mental load and categorizing it as not invisible until it's too late. And we're rounding out the end of this presentation, so I really want to kind of guide you to what the ecosystem is doing about some of these issues, but really you'll find a lot of the decisions that are happening to mitigate these considerations in the issue logs of a lot of these open source projects. I want to guide you to uh PEP 751, which is lock files. Um there's wheel next that's happening in the ecosystem. There's conversations of having a new wheel format and also the infrastructure that supports it. Much of that is happening through the uh new packaging council and the Python Software Foundation and PyPI. Um there's also container works that's happening as well. Docker and OCI containers address environment reproducibility as well um and they are um solving some of these problems. And then also PEP 723, which is uh metadata that's also going to show up in the packaging council work. So, last but not least, I want to talk about uh the sustainable autonomy through deliberate ownership. So, by taking control of your environment, I want to give you the agency to be able to uh understand at the beginning of the task or while you're in the middle middle of it and you're like me internalizing some of the failure that you may see on your console, that you're able to build strength and it comes from balancing that efficiency and iteration while maintaining ownership in the systems that you're depending on. Uh thank you so much for your time. I appreciate it. >> [applause] [applause] >> Okay. I think I used all my time, so I don't know if I have time for questions. >> Um >> But I can probably do them out in the hallway. Which is >> Okay. Uh thank you, Dawn. It was an amazing presentation. Thank you so much. Uh we have some time for a couple of questions, so if anyone wants please proceed to the microphone. >> Hi Dawn, I'm Vlad. Thanks so much for your talk which I really enjoyed. Um I really liked how you appointed out all the different levels of software abstraction and I think it's interesting how, you know, usually when we're building the apps that we're building, we're rather focused on the higher levels of abstraction, but sometimes things can go wrong in the lower levels of abstraction and we're not prepared. However, familiarizing yourself with those lower levels can also take up a lot of spoons, like you said. And so I was wondering what your thoughts are, what your approach is in terms of educating yourself about those lower levels so that you can be prepared without uh you know, in a realistic way that is not like a tremendous amount of effort. >> Yeah. Um so I'll start with maybe my enterprise context if like I'm doing it at work. Many times they issue the same hardware to you and your peers, so you can ask them what they're running on and if something like looks familiar, right? So I moved over from Windows to Mac recently and I've been a decades-long Windows developer and you know, got really I really enjoyed WSL 2. You can complain in the hallway with me if you want, but yes and and so it was really a learning curve to figure out what tools mapped from one operating system to another and it's a big jump to going from Mac to Windows. Uh tools on how to do it. I just kind of doc I've continued to document, right? I have this running wiki page on all of the equivalent Oh, I can get this in tmux and I had this in here and maybe I can do this to hack some new thing. Um I don't have an easy description or like easy path, but documentation asking my peers and leveling in context or connecting in context where you have the shared resources is really helpful. I don't know if I answered that well, but it's hard. >> No, it's good to hear your experience with it. Thank you. >> I think there's time for one more question if anyone wants. >> [snorts] >> Okay, cool. Okay, thanks again Don. >> Thank you.