An SRE's Guide to Camping in Extreme Conditions - EMF 2026

[applause] Cool. Hello. Welcome to an SR's guide to camping in extreme conditions. When I prepared this, I very much had winter in mind and minus 21°. So, I didn't quite expect to be presenting to a camp in 30° heat where people are probably hoping for advice and to survive the heat. This isn't really that, but bear with me. Uh, so who am I? My name is Donna. I'm a staff site reliability engineer. And what that basically means is it's my job to make sure that the platforms stay alive and people don't break it. So part of what my job involves is educating other people at the company on how to break things safely. So one way I've done this is by throwing in a calendar invite for the entire company on a Friday afternoon called you're going to break production. Yes, you. uh mostly just to get their attention uh and then using that to share availability best practices, talk about incidents that we've learned from and basically where we went from there. Uh and I do this managing a lot of incidents, but also I love camping. I spend roughly 60 nights a year in our camper van from hacker camps in the field in England to the Austrian Alps and around Ireland and mostly in the highlands of Scotland. and I'm a dog person, so there will be dog photos throughout this presentation. So, what am I going to talk about? First, a little bit about fan life and the logistics of taking a camper van from Scotland to the Austrian Alps in January with two dogs. Uh, then I'm going to cover a little bit around SRE concepts. What does SLI, SLO, what do I mean when I say incident management, uh, escalations and all of the highlevel pieces. I don't have long enough to go into all the SRE concepts and then I'm calling it the incident. There are many incidents that we have had when camping. So you can find me in the bar or the QA tent afterwards if you want to hear more stories. So first of all, winter in mainland Europe. One of those ideas like, oh, wouldn't it be really nice to take a camper van to Europe in winter and we can have our dogs with us and it'll be really cozy and nice in the van. And we we prepared. We were ready for this. I'm one of those like narrators like no, they were not ready. Uh things happened. So what did we do to prepare? We insulated all of the things. So they we took apart parts of the door filled with insulation. All of the water tanks covered in insulation. We had heating pads. We were ready. Or so we thought. We even have an insulated windscreen cover which has actually come in handy for the heat as well. So it does work both ways. Uh spare resources. Uh, we have a gas heater in our van and if you're familiar with camper vans, gas heaters are a bit temperamental. You learn all the runbooks and things that go wrong with them very quickly. So, we had redundancy. So, we had a backup heater. We also have multiple power cables just in case one cable stops working. And also, you can daisy chain like some people have probably done at camp today. Also, snow chains. It's a legal requirement in Switzerland and Austria on some roads, but also you don't want to get stuck in the snow or ice. Uh lots and lots of layers, which is the opposite of what I think people need today, but you really just preparing so that you can stay warm is the priority of everything. Uh and then also dog passports cuz a common question is but how did you take the dogs with you? And I'm from Northern Ireland and and my dogs actually have had Irish passports for the longest time. Uh recently legislation's changed and I could spend 20 minutes talking about that alone. Uh so if you want to know more about taking dogs specifically, uh come ask questions later. Uh and I just wanted to quickly touch on some of the fun stuff cuz some of my friends who are here today often ask like why do you keep going to do these things when there's so many things that go wrong? Like it's very chaotic. And I'm like, but look at all the cute pictures we get with our dogs in a blizzard in Switzerland. And you make all these really lovely memories. So before I talk about some of the chaos, I just wanted to share some really cute pictures on why we put ourselves through some of these things. Uh, and then SR concepts. So site reliability concepts. And if you know an SRE are an SRE, you'll be familiar with the this is fine fire dog background thing where you're like yeah everything's fine as everything burns down around you. So very much that vibe is what I bring to SRE. Uh so I want to talk about some of the terms that people generally use when talking about SRE. Uh so that's SLI and SLOs's. So service level indicators, it's what is the data are you measuring? Like how do you know something's working? What do you care about? Is it latency? Is it error rates? What do your users of your system care about? And then the objectives is what is the target for those systems? Uh like you'll often hear people talk about 59s availability. What sort of availability are you aiming for? Uh so how do you know your users are happy and what's the error budget to go with that? These often come with a third term that I'll not go into here because it's not particularly relevant for my example, but it's service level agreements where basically people companies sign documents that say if we don't meet our objectives, we will pay you money because we didn't meet your needs. Uh but I didn't have to pay anyone for our van breaking down. So it's not relevant for this talk. Uh so I want to then touch on incident timelines because incidents are the one interesting thing to talk about and also a big part of an SRE is trying to fix things. So what does the timeline of an instant look like? So first piece uh identification. How do you know that something is wrong? In an ideal world an alert will fire and tell you that something is going to break or like you've got some symptoms that something is a bit unhappy. Uh on alternative things, people message you like, "Hey, something's on fire. This isn't working." And your customers start yelling at you. Uh also a signal, not a great signal, but still a signal. The next piece is once you have that signal is assessing the severity. How bad is this situation? Is this a we need to drop everything and we need all hands on deck or is this uh it's a little bug with this one part of the system and we'll get to it later? like that determines the next path forward. Uh and then you have to triage what is actually happening like what do I need to do? Who do we need to get involved? What is actually broken under the hood and causing this problem? Fourth is escalating. So you've identified what the problem is and what you need to fix. Uh a lot of the times that may not be you in a widely distributed system. So you need to call in experts like oh I need to talk to the person that knows more about the databases. I need to talk to someone who knows more about very specific part of the system. So you escalate and depending on the severity that's am I going to wake someone up or am I going to wait until tomorrow? Like the severity feeds into all of that. And then fix it. How do you get back to a stable state? And I emphasize stable state because I mean how do your customers start using your system again in a way that they are happy? That doesn't necessarily mean the problem is fully resolved under the hood, but it's back to a state where customers can keep going and keep working. And then you have the follow-ups. So, this is the tidying up, making sure you've got everything back into the perfectly stable state that you need to be fully resolved. And then you consider how do we do better next time. You have those retrospectives. And these are often the overlooked parts of an instant timeline because it's admin. It's boring. it's, oh, I really don't want to bother doing that. It's fixed now. Not my problem. But actually, these are the important parts because it allows you to do better next time. And this all feeds into where I'm going next. So, the incident uh again there were many. I focused on one because of time limitations, but I'll go through uh all of these concepts for this incident. So, what was the SLI and SLO that we cared about? SLI temperature. It was cold and we wanted to stay not cold. We wanted to be warm. So the SLI, the indicator we cared about was the temperature. The SLO, we wanted it to be over 15°. Like in an ideal world, 20°. We want to be nice and cozy in the van, but we're in a metal box in winter. I'll accept 15° in the van. And the goal is not to freeze to death. Uh so just to set the scene of what was happening, it was 11:00 at night. Uh our water pipes had frozen days ago because as much as we had insulated all the things, we did not insulate the water pipes. So we had water tanks full of water, but we did not have pipes that would sufficiently transport said water. Uh thankfully when they unfroze, nothing broke, which was just pure luck. So that's a near miss incident on itself. Uh on this day we had also spent our day skiing. Uh I had not skied in years and I could not walk at this time. I was just sore. Everything hurt. I cried cuz I kept falling. My partner very kindly looked after me. I was like, "There, there, you'll get there." And your legs weren't working. And the temperature I'm very glad I took a screenshot of this was real feelous 21°. You walked outside and the insides of your nose were freezing. the dogs are unhappy and you're like, "Okay, yep, this is fine, but at least the van is cozy because the heater's working." And then we have identification. So, this is the start of our incident. So, I'm just casually brushing my teeth. We have a little toilet, bathroom, cubicle in the van, brushing my teeth, and then the lights go off, and I kind of mumble through the door to to Elliot saying, "Hey, did you turn the power off?" And he's like, "No." Oh, So that was our signal. That was our identification. Something's gone wrong. So this at this time the temperature was still fine because the heater had only died. But at that point we're like the heater is no longer working. That's a signal. So then we needed to assess the severity of the situation. So whenever we talk about incident severity, uh we in in my company we talk about severity four right up through one. So4 is kind of like it's inconvenient, but like it's fine. You can have a coffee or a hot chocolate and you will get to eventually solving the problem. Uh but it's not urgent. You don't need all hands on deck to fix it. Severity 3 is like, "Oh yeah, it's it's getting a bit cold, but like we we're probably fine till the morning or we can like wait a couple of hours. It's late. We'll wrap up warm extra blankets. It'll be fine." Severity too is when you're like, "Oh, there's actual customer impact." Uh, so for us, we're like, we're at risk of pneumonia. If the heater is actually dead, it's 11:00 at night. It's going to be minus 21 down to minus 24 all through the night. Without a heater, we're probably in danger. So, for this purposes, we say this is a severity 2 incident we're dealing with. Uh, and included severity one there as well with my little skeleton that would be we are in real danger. like this is threat to life danger. Uh thankfully we did not get there but it was very much one of those depending on how the instant played out this could become an S1. Um so what did we do to triage the issue? And as any good computer person would do, we turned everything off and on again and we unplugged and replugged all the cables again and again and again. And we're like it could not possibly be our van. This is not our problem. Like it's it's definitely the campsite's problem. Maybe we need a different power source. Maybe we need to plug in somewhere else. Oh, and we have a fuse box in the van. Like, let's go iteratively one by one through all of the fuses and see what's going wrong. Which narrowed it down to there is a problem between the fuse box and the shore power going into the van. So somewhere in that cable, uh, something's gone wrong and we cannot get power into the van and something with that's broken and our leisure battery is probably not going to survive the night in the cold either. So that's not great. So we escalated. Uh, so while uh while my partner is very good with electronics and things, uh, we're like we maybe need to call in some experts. So, to start with, I went and woke up the campsite manager at 11:00 at night saying, "Hello, this must be your problem with your site. Please help us." Which, fun fact, blaming someone else means they're not really wanting to help you. So, approaching it with a, "Hello, please help us. We think there's a problem. We'll get you much further." But I was in a little bit of a panic and common sense goes out the window. So, he was rather unhappy with us. Came down and checked. He's like, "No, it is your van. It is the problem." And just walked away. We're like, "Okay, that was unsuccessful. We need to escalate to someone else." So, we called our dads because you can always rely on dads. So, uh, so we were on the phone and he's like, "Okay, we walk through all the same troubleshooting again cuz that's what you do. You're like, oh, just confirming we've done this and this and this. Okay, you've done that. Okay, it must be the fuse box." So, now we narrowed it down. We know what we're doing. How do we fix it? So our fix in this incident is that we basically hardwired that spare cable that we had that I told you about. Uh we hardwired that direct. I say we Elliot did and I was the incident manager managing the situation which is why I'm talking about it and not him. Uh but he hardwired the cable directly into the fuse box and we basically use the other end of that cable as the inlet the shore power into the van coming in through the window. And the solution at that time was it was a hack. It worked. We got and we had little lights in the van that only work when you're plugged into the mains. So when they turned back on, we had a little dance party because we're like, "Yes, okay. We have electricity." But the heater wouldn't turn on. Like, "Oh, it still has a red warning light." Uh, so then I turned to the runbooks, went back and was like, "Okay, we need to reset the heater. Where's the heater?" and you're going through all the processes to find, oh, I need to find a very specific red button hidden under under the seeds, which one of these photos is me very happy when realized that resetting the button actually worked and we survived. Uh, so then the follow-ups. So, I said the retrospective like what can we do better next time? Uh, immediately the next day was tidying up our hack solution so that we can continue our trip. we still had another two weeks in winter in mainland Europe and we really didn't want to come home. Uh so it was very much okay we we need to work out how to do this. So it was rewired in a way that meant we can continue working. It's just a little bit awkward. And then buy better tools cuz what I didn't mention before is that we use a small pedager duty uh Stanley knife or uh Swiss Army knife kind of thing to uh to fix things which is irony cuz it's pager duty and I got it from a conference one time but it came in handy. And then at the time it was like future us problem. We need to fix the power store socket eventually but it can wait till we get home. So that was our future us problem. And because I promised you dog pictures, uh there are always lurkers and observers in incidents. People that join the incidents but don't really contribute. They're just there to see what's happening. So the dogs enjoyed hanging out on the bed while we continue to debug and fix the problem. And they're not usually allowed on the bed. So they were delighted. They were having a great time. Uh, so I promised you a guide and I realized on the way down I'm like, "Oh, nothing about this is a guide. I'm just telling a story." So I wanted to throw in three final things before I finish up. One is about life imitates art uh or work in this case where at the time we're like, "Oh, it feels like I'm in an incident and I'm doing troubleshooting as if I'm doing my day job. This would be a cool talk sometime. I should tell I should do this." Uh so it's really that whole real life can be seen to mimic art or work and you can always see those parallels. Uh and hope is not a strategy. Uh I've been in instance before like oh we'll just wait and see if it fixes itself. That will be fine. Uh you need to actually do something about the problem before it gets worse. And finally iterate, iterate, iterate. Go exploring, get it wrong, fix something, learn, do better next time. And then you can iterate and you can be slightly more prepared. Uh for example, we'll probably do additional justific adjustments to our van because of the heat because it's very warm. Uh so with that, that's me. Uh thank you very much for your time. If you have any questions, you can find me run camp. I'm on mastadon. Uh all the links here. Um yeah, thank you. [applause]