The Ironies of Automation in the "Age of AI" - amanda casari - 2026

Our next presenter is a roller derby enthusiast from Vermont. Uh, but he's going to be talking about, as far as I know, neither of those things. >> Yes. Uh, this is one of the the two talks that we expected to be a bit more explicitly about uh, some of the things going on in our industry. Though, of course, in this event, everything touches on that. Uh, but this is one of the ones that we expected. Uh, so I'm interested to see where that goes. Uh, please welcome Amanda Casari. Thank you so much. Thank you. Uh, just to check the room, does anybody here watch or listen to things at 1.25 speed or above? Fantastic. This talk is for you. Um, so hello. Thank you, everybody. Um, I'd like to thank you first of all for everyone who's come before me as well in the conference with content warnings and questions and discussions that have already started here about the impact of AI. I'm not going to repeat those. Uh, I am going to be talking about AI as well. Um, if it's not obvious from the title, but hopefully this is a complimentary lens uh, to folks who came before me this weekend. Um, and I dearly appreciate the environment that North Bay Python and the Python environment generally creates where everyone belongs and no one needs to explain their right to an opinion, technical or otherwise. Um, I do feel like it might be helpful for this group to hear more about my background in a way that I actually usually don't talk about. Um, so this is an example of the bio and an intro slide that I usually use for tech talks. So, these are the highlights I present. Um, it helps to explain where I am now at this very weird intersection of engineering at scale, working in open source, and focusing on global social technical communities. And what I don't usually talk about is where I came from before 2013. So, my undergrad uh, usually gets short handed into systems engineering or control systems engineering with a focus on robotics. Uh and that's both true at a macro level. Um the reality is that my undergraduate major is uniquely focused on building control systems for the highest risk full stack technology systems an engineer can possibly design. And including where and how autonomy of humans and autonomous systems meet. Um and that major included a minimum of six professional legal and ethical courses. Um which I've since also learned is unique compared to some other areas of study for my peers and colleagues. Um I've actually never been taught that technology would save the world. Um I was taught that any technology can be used for harm and that my job was to always believe and prioritize people over machines. Uh so why am I still here? Might be a good question for this. What am I passionate about? What takes up my brain space? Um the reality is is I am endlessly fascinated by the difference between systems we create and the real systems which emerge. Um I talk a lot about where things come from, where intent design was original in problem scoping, and how that's actually working out for people it impacts. Um you might have heard the phrase the purpose of a system is what it does. Um this is also known as known as Pazziwood. Um I'm not always big on acronyms, but I like that one. Um this comes from systems engineering and an operational researcher Anthony Beer. Not a great guy. Um but first wrote about this heuristic in a 1979 book The Heart of the Enterprise. Um this is also kind of one of the cybernetics folks who worked down in South America. Again, I don't recommend them as a hero, but it's good to know where this comes from. I really hate this phrase. Um I know we throw it around a lot. It feels a lot like a shrug and a scapegoat uh for people in power to push the blame of systems onto people who are responsible for maintaining them. Uh back onto whoever first built something that didn't work out the way that it might have been intended to. Um so I'd like to give you the Caseri Pazziwood corollary, which is the impact of a system is what we continue to allow. Um because the reality is is that we all have a part to play in the continued impact of systems we design, build, deploy, maintain, and eventually deprecate. Uh the choices we make, the power we grant will live far beyond initial launch date of any system that we're building. And right now, uh we're going to start talking explicitly about AI. Um should you choose to leave the room at this point, that's totally fine. I don't blame you. Um I'm going to go through these very specific terms because I want to make sure that you know what I mean in this setting. Um and when I say I for the rest of this talk, I want everyone to be very clear about where this sits in technology. Um so, a lot of these days AI is really used as a shorthand uh when folks are talking about generative AI or gen AI. Uh and as contributors and maintainers of open source, it's important to be precise in the kinds of language we're using for the algorithms that they imply and the kinds of applications these algorithms can be used well or poorly in. Um so, just again, gen AI is part of a larger field uh neural networks, which is still only one part of the field of deep learning, which is only one part of the field of machine learning, which itself is a subset of AI. So again, I hear a lot of folks talking about AI again as a very shorthand for like one class of models. The reality is is that these models have been around for a very long time. This field of study has been around for a very long time. Some of them are very beneficial and helpful when they come into places and systems that have been built for that. Other times they are not. Uh further context, basics of any artificial intelligence system distills into a few simple components. Um systems have a kind of input from their environment which are applied to different kinds of learning algorithms in order to take an action. The actions taken are evaluated using optimization formula, which is outlined to have a set of predetermined goals defined by the creators of the system. Again, this is not magic, this is math. Uh most recently, gen AI has advanced problem solving for a few said like solid problem sets including image generation, text summarization, and text generation, two different kinds of problems. Um extending context windows into semantic search and improving Q&A design patterns. These are called chatbots. Again, this is math. This is science. It gets applied in different ways, but at its heart, this is a bunch of pile of math that we stir. You might have caught that I switched from algorithms to systems at one point, um which is placing the algorithm in the context of a larger technology system it's a part of. Models don't do things in and of themselves. They're part of the systems that we build. Uh I've been long using Monica Rogati's ARGE hierarchy hierarchy of needs to talk about the relationship between data and algorithms since 2017. This does not get me visa VC funding and nobody invites me to parties because of this. She makes it very clear that getting to the part where you can integrate algorithmic outputs into your larger system is only the top of a much larger system to create these models in the first place. So, everybody I work with in technology, if someone asks you, "Do you work in AI?" probably there's part of this tech tech the tech stack you work in and the answer is yes. Yes, I do. Uh the stacks can be overwhelming to navigate, especially uh in the last few years when all of a sudden there's a lot of attention around these specific kinds of uh information. So, I distill this again down to you into three parts. There are There is data, there's software, and there's models and weights. I should warn you again, this pragmatic approach will not sell you the AI to get VC funding or angel investments. Nobody likes when I break it down this way. So, I also want to bring up um this is important for the rest of the talk. Why agentic and agent AI uh needs to be talked about now. We've talked a lot about models, talked a lot about the impact of models and harm. I have not heard a lot of folks actually talking lately about how agents actually are differentiated from that. Um so, what's different now? Uh what's not new is computers talking to computers. Computers have been talking to computers for a very long time. Um computer computers and humans have interactions. This has also been studied and understood and examined at scale and in different systems and how processes work for very long time. Uh what else is not new? Learning from history, um understanding how things in the past should guide us towards the future. Something we should continue to do. Um what else is not new? Uh reinforcing inequity? Yes, we keep doing that. We have not yet stopped. Um chaining models together or ensemble modeling techniques. Bringing models and systems together in a way that they build on each other and create whole new models and systems. Sometimes which create whole new harms as was pointed out in the last talk. This is also not new. Probabilistic algorithmic modeling. You may not be familiar with the fact that probabilities have been around for a long time and they've existed in computers and mathematics and we keep adding them into the systems that we build in. So when somebody tries to remind me about repeat about reproducibility, I like to talk to them about whether or not it's a non-deterministic a non-deterministic system or a probabilistic system, how that combines with their deterministic expectations, and whether or not they should keep moving that forward when they try to recreate something. Okay. Uh so now I guess like is the kind of the concept of what's normal. Um so at the intersection of many fields, but at the heart of all of these is where and how we grant autonomy and design technology for systems at scale and intervene in human decision-making. Um again, we've done these for a while, but there are some new things that we do need to discuss as these continue to move forward. AI is not a machine that's fundamentally changed the time-space continuum, but it is a technology paradigm that's changing the way that humans interact with technology, what they expect from it, and what they expect from us. People are changing the way that they grant consent and who they understand and expect to have consent as part of technical system that they interact with. Um what's actually new as well is the concept is the concept of automation, autonomy, and productivity for whom? And I think there's a lot of discussion. This is one of my uh favorite quotes actually and I I realize it's sadly a year old now. But when first conversation started happening about units of labor and how we were going to sell the concept of agents, Hillary Mason pointed out that depending on who you talk to, this could either be honestly a chain of API calls or it can be a bunch of software that does stuff. And all dependent on who was trying to buy it. And because naming is hard in programming and in marketing, I do want to acknowledge that there are real updates in the kind of probabilistic mechanisms that we are building. If anyone here has been familiar with multi-agent architectures, these are also something yeah, these have been around for a while. Right? We've worked with these for quite some time. So the idea that agents is coming out now to mean something that might be similar but also squinty something different. There are true fundamental differences between how agents and agent architectures are being talked about and used now which had a different flavor of autonomy, control, and who is being granted access and where trust is being built into the system that is fundamentally different than the systems that we built previously. So it is important to understand the difference between when you hear agents, assistants, and bots being talked about in terms of the autonomy that is granted, the complexity which emerges from these systems, and what types of machine learning can be integrated into each step for programmatic design. So what's actually new as well, there is new different kinds of patterns that are being built and designed with multi-agent adversarial training and chaining. There are some new interesting architectures that are coming out. These are These are old diagrams but to be able to demonstrate that there is actually new changes that are being made into how these probabilistic and non-deterministic systems are not only operating with each other but being granted access to each other. Um so, what's actually new? Um this is the lowest barrier to entry for advanced AI tools that we've ever been at. This is the lowest barrier entry of for global deployment to billions of people. Uh and this is also the lowest barrier of entry to infinite computed scale. I mean, it's not really infinite. It's finite, but it's the it the appearance of near infinite computed scale. When we add all those things together, it's now possible to deploy an AI agent-controlled application to billions of people around the world in minutes to hours. And that time is decreasing. The more autonomy and control that's being given over to the systems to be able to have advanced kind types of deployment and advanced types of CD CICD systems, uh we'll continue to decrease that, and pretty soon it could be seconds to minutes. So, what do we do? How do you maintain autonomy in a world of accelerated automation at global scale? Uh when an automated system can be weaponized, you have to fundamentally change how you approach control, and who are the controllers? And thankfully, uh as we pointed out yesterday, what are software engineers for anyways? So, if you're not familiar with uh Christopher, uh uh I mean, thankfully, uh engineers are not born fully formed from the head of an incident response manager. Um uh if they were, uh maybe we wouldn't be asking these questions, but we're not. Um I I I present this model frequently when I talk to folks and talk to folks on my team about kind of development and changes over time, and I appreciate Christopher's talk leading off into this yesterday about thinking about problem scoping. And how and what our role is in problem scoping and growing to the point that we can identify not just the right scope for the problem but the constraints that bring that down to a level that can then be solved with the right solution. So, when we talk and I talk with folks about not necessarily um where you are and where you should go next, but in as we move through different kinds of experience my baseline expectation of working with someone that is new is going to be have to give them the proper guidance and outlines and constraints to be able to identify the problem, the process, and the solution. Right? And as we go up over time, some of these things can become abstracted away because they can understand that for themselves. And at some point that first piece that I really want to see folks start to understand is what is the right process to solve a very clearly scoped problem for a very clearly scoped solution. Right? Getting from A to B is actually work in meaningful in and of itself. And having that muscle built over that time and that understanding built over time means that eventually your ambiguity and your complexity can increase to a point where the most experienced pieces or people that we work with are really choosing their own defined problem spaces, finding something meaningful within there identifying the best way to move forward, and helping everybody else get to that place. And that means that continued nav like continuing continuing to navigate the process is the essential friction necessary for both effective problem scoping and for solution finding. Thank you, Christopher. But in order to do this you need someone who knows the processes to know how to fix the system when it inevitably fails. Thank you, Benno. Uh which is very fascinating because um when we want to take this back, um really what we're talking about is the irony that the more advanced a control system is, the more crucial may be the contributor, the human operator. You may have noticed this in the title of this paper in the flyby earlier. Or maybe it stood out to you in a non-subliminal way in an earlier portion of this talk. Because this is what I'd like to talk to you about today is bringing this into the lens of a paper by um Lisanne Bainbridge from 1983. Um so for you for not familiar with her work, Lisanne Bainbridge is a cognitive psychologist uh who was active in human factors research between the 1960s and 1998. Her specialties included mental load and process operations. You can find much of her research on her personal site including her musing her musing through her research and modern applications with some absolute banger callouts. Uh I did not realize she was still updating this on her site called complex cognition which is found following here. I can share this afterwards. So Bainbridge's focus of this work was examining process controls including online operation, system failure, and recovery for man-machine systems. Check out all these keywords cuz I think I got a bingo. In particular, this paper, Ironies of Automation, discusses the ways which automation and industrial processes expand rather than eliminate problems with the human operator. Does this feel familiar to anybody else has been working anything in technology anytime in the past few years? Yeah. No idea. Um and so as whoops. One day we're going to have microphones that don't with people who fuss with their hair while they're talking. Sorry, Sam. Okay. Uh okay. So um I love the way that this this starts off with the definition of um irony and paradox um because we're going to be talking ironies and paradoxes as they move through. Did I just bork everything up? Are we good? Thank you. Um, and so when we first start going through I'm going to kind of walk through through the paper. We're going to talk about a few different pieces and then bring that back. Um, and so one of the um, uh, uh, Bainbridge talks about in the very beginning of this paper um, and I can remember this is again 1983. What's happened before will happen again. Um, one of the ironies that we should be talking about and addressing is that as the classic approach to automation uh, lies in the expectation of system designers and that the nature of the tasks left for human operators to carry out. So in general designers usually view a human operator uh, as unreliable and inefficient uh, and should be eliminated from the system at all possible. And not to get biblical here uh, but I'm willing to assume in this case we have all found a point where we thought we knew better than the person who came after us or the person who is interacting with whatever we built. Right? Um, and so I do think it's it's helpful for us to apply again the uh, maybe for us it's good for us as well as good for um, the people who are designing for us is that um, when a designer tries to eliminate uh, the operator basically leaves the operator with the tasks of everything they didn't know how to automate away. Which is a whole collection of stuff that might or might not be the best parts of the job or the things that keep you either entertained or skilled. Um, and when I mean entertained I mean with your attention and your focus and the ability to interact with the work that you're doing. Um, and that approach actually causes the problems that means that um, when you're left with this arbitrary collection of tasks little thought is actually given to providing support for those as you design the rest of the system because all the value's been given into the automation and it's just assumed that the operator's going to adjust. Um, and so I think my question there is as we think about within our own work and within the own systems we design and the designed for us um, how do you know what the boring stuff is if you've never gotten bored while building or maintaining it? So, when we think back to those places of where we're talking about um building and gaining experience uh for engineers, for people working in technology, um and where we're starting to design systems to skip over some levels of knowledge and skip over some levels of friction, um some of that's because people don't like doing it, but what happens when that skill set goes completely away from the workforce and the systems that we're designing? Um next I think that uh starts to Bainbridge starts to address that. So, what's left after automation for operators, for people working with these systems? And that's when what's really left as part of this work for process control is monitoring and reviewing for intended behavior to identify when intervention is necessary. Anyone working in large-scale systems, does this sound familiar to you? We might need to have humans take over and re-correct things. Yeah. Um the challenge here is the irony of necessary intervention means that you have to have deep expertise of the working system as well as the diagnostic expertise to recover from the fault. How do you gain that when you've turned away and automated all the processes that actually allow you to gain that level of expertise and diagnostic expertise? So, uh Bainbridge calls us out that when manual takeover's necessary, um unusual actions will be needed to control it. And one can argue that you need to be more rather than less skilled uh and less rather than more loaded than average, which means you need to have a lot of experience, you need to have a deep understanding of the system, and you need to not be so tapped out and stressed out that you can't understand and respond to something as it's the as the stress increases. And the challenge with this is is that experienced operators will make the minimum number of actions. Uh the process output moves more smoothly and quickly to the next level. While inexperienced operators oscillate around the target value and approach the final answer in a much less in a much more inefficient manner. And developing and maintaining that deep expertise of a working system is essential to smooth and efficient recovery from a failure state. Again, this is from 1983. The problem with this is that we are continued to invest in experienced operators. They don't just gain this from trying things out and from expecting that you're going to have some level of knowledge without actually doing things repeatedly over time. So again, when we're thinking about automation, we're thinking about the kind of trade-offs we're making and how we spend our time. Um there is value in having not just the raw data of what's happening, not having access to a dashboard, but having the level of experience and knowledge and work with a system that you can make predictions and decisions about the process which are useful in future situations, including how your future actions will affect that. And that information takes time to be built up and it cannot be replaced by automation. We build up those novel intervention strategies and the knowledge of what maintains a steady state through multiple cycles of learning of what didn't work last time. And the challenge, at least in 1983, was that there was a concern that the present generation of automated systems are monitored by former people who had gained all of this level of experience and that the current generation did not have that same level of building and skill building over time. And so the same systems will not be able to be maintained by them cuz they will not have the same skills that allow them to operate at that high state. And the lesson from this is that we can't plan for future systems to adapt to different skill sets then cascade ins will continue to the cascading failures will continue until morale improves. Or until we retire. I'm going to I'm going to kind of go through a few more sections so the deeper into the the section on monitoring Bainbridge talks more about the paradox and ironies of humans monitoring sufficiently complex systems with well-defined criteria making decisions at machine speed. And this is many challenges and and the ironies that go into whether or not that's feasible and what tax that puts on for both the trade-offs of like human attention as well as skill building. For operator attitudes the paradox and ironies of automated systems deskilling experienced workers into system monitors the challenges this creates for both identity and pay. And where this leaves very experienced operators into the worst type of job which is very boring but also very responsible. And there's no opportunity to acquire maintain qualities required to handle the responsibility afterwards. Approaches to solutions thankfully there's also some very fascinating pieces here which I feel like should be familiar to anybody who works in systems at scale. Recommendations include that low probability events require more automation assistance and alarms should be designed to prevent attention fatigue. But automatic systems should fail obviously. There is actually a a delightful piece as well around high critical systems so where high critical systems should fail into a safe state. Which I think is very deeply related to the robotic stuff that we had yesterday. Humans must monitor details of computer decision-making but also it must be at a rate which the operator can follow. Even if this is not the most efficient method technically. And the reason for that is that whether an operator believes or agrees with the computer's decision, at least you can go back and trace the decision sequence to see when you stop agreeing. This means that you have to intentionally build critical processes to operate at human speed. Rather than attempt to accelerate people to make a critical decision that meets machine speed. Bainbridge goes on to go through other aspects of human-computer collaboration. And again, for anyone working at scale, there's a lot of human in the loop decision processes and designs, which are also will sound extremely familiar to you. Addressing where humans and computers work in different kinds of patterns for instructions and advice. Where mitigating human error actually can be beneficial in systems at scale. Software-generated displays. Fantastic idea. This was like, obviously, very different place in terms of the being able to show important information for making decisions. And relieving human workload. Again, the point here for all of this was really to point out the fact that technology was and remains a multi-dimensional design process. And process autonomy is a critical dimension of systems design. But that automation itself is not inevitable. Something that we either choose or we don't. And when it's placed upon us, then we also have a decision to make. Because equal autonomy has never been granted. But it can be collectively won and defended. Thank you.