Network State Checking: A Case Study in Automation using Python by Craig Henderson

[applause] Hello everyone. Thanks very much for having me today. Uh I must confess from the outset I'm something of an intruder in this room. Uh you would have to squint quite hard I think actually to see my contributions to our codebase. I I didn't really become an expert in Python until we got AI. Uh um if you have questions about the Python involved in this system, please direct them in fact to your conference organizer Chelsea Finny who made much more major contributions than I did. Uh I can speak however uh as a user of these systems uh how we've used uh automation with Python in order to ensure the accuracy of some changes we've made on the reins network recently when we refreshed our entire domestic network. Now, if I'm going to represent Rian's, the first question I need to answer as always is what actually is that even? Um, the last guest that we heard was the Recreational Aircraft Association of New Zealand. And as cool as that sounds, unfortunately that is not what I do. Uh, we are a New Zealand network carrier like a commercial telpo. Uh we offer you private WAN services and internet uh network security and uh as part of our public service remitt um also a federated uh authentication system for wireless uh connectivity we call edgerome uh recently we have also taken on board the national e- research infrastructure platform uh to offer high performance computing solutions as well. Uh however I won't speak so much to that subject because we will have Cal speaking later today and I'm sure that he will do a terrific job. Uh now the interesting thing about Rian's the only reason that we can exist as we do is because we are part of a global indeed network uh that is national research and education network. Uh there are about 120 uh such organizations around the world and we have very tight-knit relationships uh that result in a lot of tangible exchange of um uh of information and uh connectivity that we all possess. Um yeah um shout out to all of these organizations. In fact they are what makes it possible for us to do what we do with their providing of um transit free connectivity and remote hands and all sorts of useful things. Now, I'm going to be talking about the changes we've completed, so it makes sense to start from the beginning. Uh, Rians was pretty late to the Gen Ren game. We only came about in 2006. And this is the earliest record I have of the Rian's network. Um, this was 2008 when we were known as Karen. And uh, this measurement system here, our weather map operating under the University of Wicked, shows what our network then looked like. It was a ring switching topology. And you can see that we were established all around the country to provide in ren services only. Uh so we didn't do much networking actually at this stage. We offered you we offered a very specific set of uh customers connectivity into research resources only. Uh we wouldn't even offer an internet connection uh even as something as simple as that. Um even for those members that were within our remit at this time. Uh we grew considerably in 2013 with the joint operating agreement. We uh signed with FX networks which went on to become Vocus. Um through this arrangement uh we would join together in a public partner a public private partnership um where we would take a stake of ownership in their optical network. And this is what our network then looks like. It is still a ring switching topology at this point and you can see the first signs of standardization have creeped in now that we have these um codes for all of these different nodes for our domestic connectivity. Um we settled on airport codes to try to identify these things and that system immediately broke down as you can see. Um because when we established a second pop in it had to become simply dud 2 uh which is not quite ideal but we came back to that and uh unfortunately made it a bit worse. In 2018 we uh upgraded to 100 gig. You can see we've been expanding quite dramatically and we fixed that uh dud 2 problem in a way. Uh as you can see we now had the to pop instead. Um now interestingly recently there was a a mass network outage in Indonesian you might recall and and that got people discussing on the internet um looking at this weapon map application of what exactly to might possibly mean and as you know people on the internet are quite clever and came up with a lot of very uh interesting and creative theories to explain that. Um and no one I saw was successful I think because the answer is actually not at all clever in the slightest. Uh to stands for the other Deneden and I'll take you now to our present day. This is what we looked like uh right after we had finished uh refreshing all of the hardware that we have in our domestic network. Um the design is a sort of ladder. We have east west paths all up and down the country. Um so if uh any side of uh this topology breaks uh hopefully we have another route to keep everybody connected. Now, simultaneously with our network change journey, uh we also saw some changes to our network automation. And I've divided this into a few different periods. From 2006 to 2014, uh it's quite easy to describe. There was no automation. Everything was handbuilt uh entirely bespoke uh entirely uh down to the creativity of whatever engineer was um building a service. Um now, I'm sure they would defend that system uh as as [clears throat] the ideal of course. Um it it's not just uh you know poorly planned hazard system. It's more like a you know a luxury car handbuilt to perfection is is what I would have said in those days. Uh however those things notoriously don't really work very well. And uh because of that we we needed to move on from that. 2014 to 2019 is when we started to see the introduction of some one-off scripts uh to help with some specific purposes we might be able to leverage this automation for and get some easy wins. uh given some examples here of um various network meshing that we require and firewalling and routing policy that we might come up with. And these things um were the first targets for a few reasons. Um they're they're repeatable and very formulaic. Um to create a mesh of all our different devices, they all have to talk um you know, one device has to talk to every other device on the network and they all require configuration pointing at each other to accomplish that. That is very simple. It's very boring. Uh but it's a very long process and one that's very errorprone if you try to do it manually. Um so it's very easy to implement in automation of course and we get an easy win there. Uh firewalling is a bit more complicated to manage. Uh however the the win in that sense is just the blast radius of getting that wrong is so enormous that it's worth uh investing the resources in order to in order to try to build a system uh to enforce some accuracy in that as well. Um, now this this system of just designing stuff on a one-off basis every time that you need it has some problems of its own. Uh, the main one being that you have to reinvent how you're going to pass information out of the network every time that you do this. If you're just beginning from scratch every time, uh, then every different uh, piece of code that you come up with has to figure out a different way to do this. It has to figure out a different way to interact with all of our various APIs. And then when somebody like me comes along and makes a change on the network that uh, one or more of these scripts can't understand, things start to explode. And now you have this difficult troubleshooting exercise to go and find out why. So we wanted to you know introduce some libraryization um just try to introduce as much standardization as possible. And the problem we had to solve before we could do that is simply to choose a mascot that was appropriate for an IT professional. Uh you know somebody decisive somebody with a crippling dependency to alcohol. And we came up with this. Bender is the Rian's network automation monor repo. Um and it has two functions really. It began as a paraser of the network as a uniform way of understanding everything that was out there. Um which first of all helps us to understand what's wrong on the network. If we have certain rules certain architectures we're expected to abide by when something doesn't conform to that we can have bender uh recognize that that uh that desync and push it out into our alerting system to be fixed. Uh over time uh we also started to generate config through the system as well and the model looks a bit like this. So from the top we have our network device configuration as our source of truth. Uh we do of course have resource allocation databases of various kinds where we try to maintain our um understanding of various systems. Uh but the network device configuration is always the source of truth um until you have a way to enforce what is in those databases. um you can keep all the best records with all the best intention in the world if you like, but if uh one person comes along without following it and they have the ability to push config out to the network that's not in sync with that, then that's um that's the reality that you have to deal with. So for now um the network is our source of truth. Uh Bender passes the network to produce a current state um of what's out there. And as I've uh just mentioned, we can link through that in order to push alerting out um in order to fix things that are wrong by interacting with the CLI tooling that's integrated into the into the system as well. We can generate new config and every time that we can do that, we can predict the future state of the network so that we have something to compare against what it was. And then we have our apply config tool to complete the cycle and push out the config that's been generated to all of the relevant devices. Now, that brings us on to our big hardware refresh project where we were very much in need of such a system. You can see here the delivery in progress. I had a truck outside that was uh over the course of perhaps an hour delivering pallets upon pallets of boxes into our our room, which I've been slowly piling up into the hallway here. Uh you might wonder why I didn't put them into the store room instead of forming a barricade in the hallway. Uh and the reason is because I want to see I I want everybody to see how hard I'm working. I'm I'm still milking this even to this day. uh refresh looked a bit like this. We started on the left with the uh MX480 uh carrier grade chassis that Juniper puts out. And our model was going to look a bit different going forward. Um rather than having a a big chassis based model with various line cards we can install depending on our needs at a certain location. Uh we are instead now stacking individual one or two rack unit devices um as they are required. We've got a couple of opt options for access routers on the bottom. We've got something that will give us connectivity up to 25 gig and on the top something up to 400 gig and we have a couple of options for border routers where we need uh need something a bit fancier in terms of resources and the system software that runs on these things. Uh those you'll find anywhere that we have external handovers to upstream providers or uh anybody that we might have a private interconnect with and need to do something fancy to manage that traffic. Uh physically it looks a bit like this. That was our old system with the 480 chassis. And when we were done, we had uh settled on a system that looked a bit like this where everything's been moved across onto this new hardware that you've seen. Um and our server infrastructure pulled along with it. Um that was a bit of an undertaking. Um we did get to the point where the configurations were about 97% generated by code um and therefore uh immune from any human error, which was great. Um there was a remaining set of edge cases we would have to look for. So we'd have two engineers go through and audit that code to make sure that it makes sense as well. When it came time to do the cut over, we'd have an engineer on site with the devices um repatching all the fiber and we'd have an engineer leading the change remotely from his computer. We'd have three-hour change windows and we would migrate 8 to 12 member handovers per night depending on the complexity. Um if that meant the change took place over multiple nights, that's fine. Uh this is just the system that we uh proceeded under. [sighs] Now the problem when we do something like this is to decide when we have moved a network service whether it is still working on the other side and indeed if it's not working on the other side um was it broken when we got there um we need to have a way of determining that and the first answer you might think um to answer that is well can you refer to your monitoring um and unfortunately you can't as it is currently implemented by rans um we don't package all of the metadata associated with a service into one parent structure that we can understand that service by if we pick something up and move it. Um, it now exists as a new service in a new place and our monitoring understands it with the parameters it's seen for that new service. It can't tell us that there's something wrong with the old service because the old service just doesn't exist in its uh mind anymore. Our monitoring also lacks context. There's some things I'd like to be aware of in a cutover like this that I don't want to be aware of under normal operation. Uh, for instance, if I lose some routes because of something I've done, that's a problem I might want to know about and uh want to investigate. If uh somebody in Seattle for instance withdraws a route on a on a on a route server in the middle of the night on an average Tuesday where I've not been working at all, I don't want to hear about that. I don't want to be woken up by a monitoring. Um so monitoring is not the appropriate way to do that. Uh now unfortunately even really basic handovers have enough complexity in them that it's quite difficult and timeconuming to check everything associated with a service and find out anything that's wrong. Um I've got some examples of that here. Um these are the kind of things our state checking looks for um when we're trying to determine if a service is uh working as intended and whether we need an engineer to intervene and look at things. Uh in terms of physical interface, I've got these various attributes we might concern ourselves with. Is the interface even connected? Is it up or down? Um do we see throughput over it? Um are the lasers that are lighting these services operating within spec? We're looking for things like that. and I canii continue with the services we're building on top of that layer 2 services like this again we're looking for frames passed over the over the top are we foring um in the way that we expect without loops and such and then when we get into our routing um protocols as well do we have all the routes that we expect do we have all the neighbors that are sharing these routes with us and things such as that for this audience um you're perhaps not as interested as uh somebody as sad as me and knowing what these things actually mean but um you will be interested at least that we have something to begin with and something at the end that we'd like to be the same and there are interesting ways that we can accomplish that with Python and we had a simple test in every case fortunately because as part of our public service remit we have uh test nodes installed all over the country uh as part of the Perf Sonar project uh which is something that originated within the EN community um and it is just a suite of test software that's installed the servers around various um EN carriers and universities and some private operators around the world. Um so that if you're interested in testing network connectivity over long distances and very um you know opaque use cases um you have a system for doing that. And fortunately for our cases in terms of their network connectivity all they need is a simple internet up link. It is as basic as you could hope. Um and there is even a public platform by which you can see if it is working as intended. So this is everything that we could hope for in this context. Now when we generate the config for this basic incident handover um a YAML file gets produced something like this that simply says what did we have at the beginning and what are we expecting to have at the end and you can see in this case I've got a physical interface I've got a you know a VLAN layer 2 network operating over the top of that and I can continue building this file with as many objects as I need depending on the complexity of the service. Uh now based on what is in this YAML file, I'm going to go to those devices and I'm going to collect some information that I know describe those things. Um I'm going to begin by collecting information off the original device. Um in this case, for instance, I want to know about the physical interfaces. So I'm going to run a quick show interface command and I'm going to dump that out to a text file which we're going to maintain for the purpose of this stateeing system and for any later auditing that would might be required if we need to iterate upon this process over time. So that's um the kind of information we need about a physical interface and I can continue in this fashion. I can get information about the you know the routing interface that's on top of that and the kind of addressing it has. Um I can get information about the lasers that are lighting this service um you know it's transmit and receive um signal that we're seeing. And then I can do the same process again um as I go through the new device when it's moved over. And all I'm looking for is something I can take from the beginning that I can compare to what I have at the end. Um, now happily as well as an operational engineer like me, this isn't actually a very abstract process at all. This is exactly the process that I would follow as a human being as just a computer can do it much more accurately and much quicker than I can. Um, so the the goal with the system effectively is to make me as redundant as possible. Um, we go through this process and then ultimately we can answer based on this information we've gathered, is this service working? In the case of a persona node like this, I want to be able to run the tool that says compare the current state to that uh previous state that you captured earlier and I want to see an empty diff. I want to see everything is the same as it was before. Um if if I don't have that, even in the case of my persona node, something's gone terribly wrong and we might not be able to proceed with the migration. Um now that's something I can do quite easily myself. Um there are only a few things there you saw I had to look at in order to determine if a simple service like that is working. However, I'd also like to come to an example like this. This is a bit more challenging. Um, this is an anonymized recreation of one of our our member networks. This is all real. And you can see here I've got 24 different separate routing instances routed through geographically redundant firewalls onto multiple border instances. They have handovers to different cloud environments. They have a public internet connection and access to private uh research networks as well. uh and they have uh local area networks that we participate in that uh I couldn't have fit on this diagram when I was creating it. Um it's uh fairly nightmarish as it is. Um now you can imagine verifying all of this information when you've picked it up and moved it is uh quite the undertaking. Um again we have our YAML file produced. This is a bit what it looks like. And like before I've got an interface and I've got a VLAN. That's fine. I keep scrolling. I've got a you know a border VRF now. Fine. I've got a BGP neighbor, more VLANs, I've got an EVPN that will have some kind of multihoming I'll have to check on. Then I've got a a VLAN. I've got more VRFs. I've got an OPF neighbor as well. So there's a different kind of routing I have to worry about. And I can scroll for quite a long time with this. Actually, this uh this YAML file that was produced totaled 1,525 lines just describing all of the various services that were on this single member handover. Um if you look at the config that's um that was set aside for any potential audit as well. Um just looking at the bottom of my Vim session there you can see that they were over 50,000 lines deep in edge case. So the before state and the after state. So if we were to go through a checklist and try to manually verify all of these services are working exactly as expected. Uh you can imagine that would take quite a long time. uh probably in the case of even the single service it would um exceed the freehour window that we've set aside to do these migrations. So that's uh not great and fortunately we had the um the automation step in to help out that 50,000 lines of uh comparison that was required was reduced to this that was my diff that I got from uh having migrated this service and there's a few things it's pointed out that it thinks I might be interested in. And I can triage a lot of this quite quickly. You can see the first couple of options there. It got um it sees a a noteworthy difference in the the output throughput and the input throughput on a couple of different VLANs on this interface and I understand these services fortunately enough to know that that's not a problem. There's a little bit of variability here. Uh but I've got birectional traffic. I'm happy with what I'm seeing. In the third case, it's a bit different because I have the number zero and the number zero is always something of a red flag. Uh this VLAN was not particularly noisy in the first place. It's a bit shy. It only had uh nine input packets recorded and 10 output. Uh but zero means nothing. That is potentially something that's not working and I'll have to come back to that. Uh moving on, I've got zero packets going to two in the next example. Um that's potentially a problem. If you have a network that's not expected to uh to to push traffic, and it is. Uh there are some contexts where that might be a problem. Fortunately, not the case here. Um which I'm glad of because if that was a problem, it's likely to be a very big one. Um then we have a uh a route that's changed. We've got a host route for some management on this device. um it's got a different address than it had to begin with. That could be a problem as well. However, because of the new architecture we were uh we were delivering as part of this refresh, I I know that's to be expected and I'm not worried about that at all either. And then another example where an interface is just a bit quieter on ingress than it was to begin with. And that's another one that I'm quite happy with. So the only problem I actually identified that worried me here uh was that single VLAN that had no input or output traffic. And um my approach to this of course was to run the tool again and hope that I got a different answer and one that I actually wanted to hear. And a couple of minutes later that was missing from the diff. It was happy with what it saw. So this uh this turned out was just a bit shy. Um and in 2 minutes we had managed to uh verify all of the services working on this handover. So this was an enormous success story for this um for this tooling. That was a really worrying topology. I will show it to you one more time just so you can see how cursed all of this is. Uh this is not something I was looking forward to moving but with the help of this Python automation we had delivered um two minutes of work. It was no problem at all. We just uh plugged it in where it belonged and and it all uh and we could verify everything worked as expected. It was great. Uh so I have been a little bit disingenuous you might think in how I presented this. Uh obviously a human being wouldn't go through those 50,000 lines of uh output delivered from those show commands. a human being is capable of rational thought and can see what is broken and investigate um using their um your capacity for complex thinking. Uh so I've got a couple examples here of that um uh cases where we saw things pointed out I probably didn't need pointed out to me. So I've got an interface that was up and now it's down. A BGP handover that was established now it's in the uh connect state instead. I think that I could have noticed these things. My terminals even color code them. Um so I can all I have to know is that the green the thing that used to be green is not green anymore. Um, that's uh that bears some similarity to the uh the homework my niece got from kindergarten quite quite recently. I don't want to brag or anything, but I got 100% on it. It was pretty easy. Um, so yeah, we probably didn't need to go to the effort of uh investing in all this tooling in order to uh help me with those problems. Uh, but here's another example. Uh, again, we've got zero packets on ingress and I've mentioned when I see zero, I'm a little bit wary about that and uh with good cause. It turned out in this case uh there was a problem with interoperability on some Ethernet interfaces. um connecting to one of our junipers and some other vendors and in this case that meant um ENET auto negotiation didn't work such that um uh we we could broadcast packets to them fine all our output was working uh but the ingress wasn't um wasn't working as intended. So that was not actually a working service and it's hard to tell when these are just um you layer 2 broadcast domains. if you don't have any kind of um routing protocol running over the top to make give you some obvious sign that things are down maybe this is just a quiet VLAN it can be hard to tell um but fortunately since the tooling was able to direct my attention to this and I could look a bit deeper into it I saw actually this is not the behavior I'm expecting to see and I can do something to remediate that uh here's another such example I mentioned before I do not want to be woken up in the middle of the night just because some routes have been withdrawn from a uh from a route server however uh in this case we lost four routes immediately following something that I did and that's slightly of interest to me. And in this case, um it did actually represent a um a an a routing policy that we applied on import that was set incorrectly. So, we weren't accepting for routes that we were supposed to. Um again, big thank you to the tool for pointing that out. And here's another example. We've got the receive optical level on one of our services um that has taken a massive dip. So, that laser has become a lot dimmer than it was in the beginning. Um that's a very annoying problem to have. um they can operate um at lower levels, even out of spec sometimes, and still bring an interface up and and have a working service across it and give the engineer the impression that everything's been fine. And then later you just have the odd error that accumulates on this interface. And we have to report back to the government and say that we're doing a good job as a network carrier with public funding and we're, you know, we we we don't deliver packets with errors. And then if we have interfaces that just say randomly conjure one out of 10,000 packets um that aren't working, it's a bit embarrassing for us. So that's something we don't want to see. Um, and in this case, fortunately, the tooling was able to tell us, hey, um, maybe maybe try to clean this fiber or replace a hash league or try a new transceiver. Just do something to make sure that doesn't happen because it's very very annoying. Um, and the last point I want to cover is, um, how effectively did this work? Um, did anything go wrong in spite of all of this tooling that we had and can we blame the tooling for any of those things? So, let let's put it on trial a bit. First problem that we had there was a a fairly obscure BGP attribute we set for uh reasons of Rian's architecture in some cases. Uh independent domain attribute that was required from a private WAN VRF which caused BGP to um use its loop detection mechanism to suppress some active routes. Uh this occurred during the first migration that was completed by rans and at this time the tooling wasn't actually yet available. So uh we can't blame the tooling for this one. Uh and it does serve as a useful benchmark. We had um we put our two best engineers on the case for this uh this first go. They spent a couple of weeks doing their best to verify all of the config and um come up with the the the plan of attack to make all of this work. And I think they did a they accounted for themselves very well. They did a great job in a on a on a diffic difficult piece of work. Um but I'm here now talking to you like it was a failure because they had no Python automation. And I guess that's the take away from that uh that particular incident. Number two, we emitted the MTU from one of our core backhole links. Um, which is not a great thing because it meant an entire remote region of New Zealand was isolated from the rehance network. Um, now that uh the the the ability to generate the config that would have dealt with that was implemented from the tooling, but it wasn't used. So, we can't unfortunately blame the tooling for that either. Uh, now as Python people, that's something you need to be aware of. Half of your job even isn't the technical implementation of what you're trying to do. It is uh also the advocacy of uh using these tools because you know the operational engineers are stressed and they just want to use what they know. They don't necessarily want to use their tools if it has the potential to spit out some error they don't understand and they'll have to troubleshoot. Uh so sometimes you'll uh you'll invent the best tools with the best will in the world and then they just won't use them. So uh be aware of that problem. And then lastly we had uh BGP configuration emitted for a longlined internet service downstream of device. That is a lot of words to say that somebody forgot to turn something on. Um, we didn't implement this functionality in the tooling because it was a bit of an edge case. It would take a little bit of work to implement and we thought we just don't have anybody on the staff that's dumb enough to um to uh to to do something like this to make this error. Uh, but I did uh in my defense I I was tired at the time. I was on call that week and it was a bit busy. But um yeah, the the plan of course was um we would just be smart and not do that. And uh unfortunately we learned uh I'm not smart. So we iterated on the process and later introduced that functionality to the tooling. Um so where are we at now? Um we've never been able to make use of this tooling actually in the case of our firmware upgrades on all of our devices because we have about three times as many now that we've uh moved off to all of those individual devices. Um it's become kind of a pain and it's helped us to be able to use the same kind of system of verification for that. where we'd like to go is to have a new source of truth that's based on an actual orchestration platform and not just scraping information off the network itself. Um the problem with that is it doesn't really scale very well. Um any of our service enriching kind of metadata has to be encoded onto the network which results in increasingly bloated kinds of config um which becomes increasingly difficult to manage and it doesn't offer a very good story for the RANs members. They want to tell about all the nice things that you're doing. And if we have to go back to them and tell them, well, you've pushed 100 gig of traffic on this uh this VLAN that has this routing protocol running over the top of it. They they just say, I don't know what that means, and nobody's going to be interested if that if I tell them. They won't talk to me anymore. Just leave me alone. Uh so we want to have u these things understood in the context of a service that we're providing and what it actually does. Um and an orchestration platform is going to help us with that. um that will be able to manage the life cycle of the service from its deployment all the way through its requirement relying on Python all the way uh we will always allow custom configuration from the engineers but we would at some point like to contain that to a system where we have visibility and accountability over those things and they can't just push config out and directly to devices and ruin all all of our good intentions. Uh that's all I have for you in fact. Uh thank you very much for bearing with me today. Um, do we have time for questions or have I uh overrun? >> Uh, we we do not have time for questions, but uh thank you uh to Carl. [applause]