Validate HA Before It Fails You: A 5-Minute Checklist | Alexus Gore, SIOS Technology

Chances are that your failover is not as ready as you or your teams may think. A failover solution being configured is not the same as knowing it will work when it's needed. Most teams figure and find that out mid-outage when downtime costs money and their customers lose trust. By that time, it's already too late. SIOS Technology helps organizations prove high availability works before they need it. They have built a 5-minute checklist to validate your failover setup. And today, we have with us Alexis Core, senior customer experience software engineer at SIOS, to walk us through that. First of all, Alexis, it's great to have you on the show. >> Thanks for having me. >> It's my pleasure to have you here. Uh as you know, as you know, most organizations, they assume that they're protected just because they have failover solutions in place. And from their perspective, there's nothing wrong. That's what they should assume. But can you talk about what is the biggest difference between having failover configured and actually knowing that it will work when they actually need it? >> So, the biggest difference in determining how your failover environment is going to work when you need it is largely based in the testing that is that has been completed for it. With testing, you need to kind of like check the ins and outs with how your environment is going to react in a situation like should a network outage occur. And if there is like an unknown answer into whether or not you know, you have your failover configured, into whether or not that failover solution, you know it's going to react appropriately in the event that a failover outage occurs. If there any unknowns there, then that is typically not the best-case scenario. You want to be able to know what's going to happen, how your solution is going to react in the event that a network outage occurs, and the event that any type of disruptance occurs, ideally. >> And let's assume that you have just 5 minutes to assess the health of a production failover environment. What are the first few things you would check? >> Um, first few things I'd check, I think mainly I'd start with the logs. Logs are generally going to tell you everything you need to know that's happening in your failover environment. Um You have your failover solution logs, and your failover solution logs are typically going to cover what's going on with how your failover solution is handling your environment. And you can also check your system logs. Your system logs can usually tell you, I mean, both will usually tell you if there are any like network issues or communication issues that are happening within a cluster um or in your environment. But with your system logs, you can also kind of determine if there are any issues with your system having any network problems or just like lack of, I guess, communication or just like mishaps going on between your system and your failover solution. >> You talked a lot of customers, you talked to teams, and I'm sure you have heard a lot of stories. What are the most common failover configuration issues that you have seen, the ones that don't show up until an actual outage happens? >> So, most common, I would say happens when sometimes when customers are using, let's say, SAP applications with Data Keeper, and a lot of the times there are we have documentation that cover the steps that are required to configure your SAP application properly to be used with Data Keeper, but sometimes the things that are like missed in the cracks, maybe like small parameters that are not updated to be used with DataKeeper and are not usually caught until an outage occurs because it's it's something that's so small, you know, if this customer we haven't done a health check with, then you're not going to really see it until an outage occurs that there may be a problem happening. And a lot of the times it does get caught. It it can get missed, but it will get caught by just checking the logs because we do update our product to make sure that we're catching these things beforehand. Um but yeah, I think the most like the most common issue is definitely not having your application set up in respect to the failover solution that you're using. >> Now, the fact is that a lot of IT teams they hesitate to test failover because they worry about disrupting production and that's also a valid concern. How should organizations approach failover testing so that uh they build that confidence that things will not break, they get to test so they're not taking unnecessary risk? >> So, two things. Um to avoid any unnecessary risks, you would ideally you'd want to create a QA cluster beforehand before you even go into production. And a QA cluster is typically going to be a one-to-one exact copy of a production cluster that's primarily going to be used for testing. So, that way anything that you change or that you plan to change for your production cluster, you can do it in the QA cluster first. Um second thing is creating a run book. A run book is usually going to have uh the steps that you ran during your testing, what happened, what occurred, what you expected to happen, um maybe why it even didn't happen. So, this is good to keep track of everything that happened up to the moment where maybe a disruption occurred that you really didn't expect to happen there. Um but yeah, to avoid any unnecessary risks, create a QA cluster and create a run book to keep track of everything that you did. >> Now, the fact is that organizations are modernizing. We are already in the journey of AI, but a lot of organizations are you know hybrid cloud, Kubernetes. Of course, AI driven workloads are also here. How does a failover validation also need to evolve with these changing times? What new risks are there that IT teams should be aware of and watching for? >> So, when it comes to failover validation, um it's you're now going to be testing for predictability or at least testing more for predictability and adaptability. How well can your environment predict when a outage is going to occur and whether or not your your solution is able to adapt to what your environment needs at that time. Um when it comes to AI AI large is largely based on learning from its learning from its failures. So, when it comes to integrating with AI, a lot of more HA solutions and failover solutions, it leaves more room for error, which you know, it's not like the best thing to hear, but it's bound to happen, right? When you're introducing so many more products to create almost like a bulletproof highly available solution. There's just going to be more room for error. Um and the good thing about AI I guess is that when it's learning from its failures, it's also succeeding and learning to improve again and again. So, when you take that risk, um there is a high reward at the end of it. Uh but yeah, I mean, ideally you want to you just want to test more, cover more ground, do as much as you can. Um but those are yeah, that that's those are the things that IT teams should be on the lookout for. >> Is it possible for you to share real world example where a company believe we don't have to name a company that it's disaster recovery plan was ready only to discover a critical gap during testing or worse during an actual outage and through that there there was a lesson that was learned and that lesson can be shared with others as well. >> Yeah, so there are actually three occurrences that I can think of where this happened. Um Both of these customers or all three of these customers um have experienced using DataKeeper. And all three of these customers also wanted to use another failover solution or another replication solution as well alongside DataKeeper. And with the systems that they provided us with the logs that they provided us it looked like it worked. And so everything was in place, both solutions were working fine, there were no issues. The problem happened when a network outage occurred and the failover solution did not work. Um and it's something that could have been caught if the customer had taken the steps, the precautionary steps uh into testing their solutions together. Um technical incompatibility is I one of the themes here when I'm talking about these three customers. And the lesson learned here is that it's important to test, but I do want to mention that oh man it's like their configuration did look like it worked because they used both products independently. So putting them together was where they thought there would be success, but unfortunately because there wasn't any testing involved, it just was a catastrophe. And it was missed in the beginning. Like I think looking through the logs, you can't really see where the issue lied. You would have to go way deep into the kernel code to really figure out what's happening. So yeah, making sure that you test is the is the biggest aspect of this and making sure that you test before you go into production as well. >> How often should teams actually run through these the 5-minute checklist? And how does that frequency changes as that environment becomes more dynamic or as we're talking about, you know, new use cases, AI and everything is entering, so it's becoming more dynamic as well. So frequency of these tests and check, you know, >> I say you should test as as often as possible, right? When you're introducing this new environment, you want to test as much as you can. Um when it comes to introducing more software in there is when you want to keep maintaining that testing. But as much as you can, as much as possible. I know it's like if you're running through a 5-minute checklist, sure. But testing is like the crucial thing here to make sure that you're not missing anything. So as much time as you can put into testing your environment, your environment changes is the way to go. >> We live in this world of AI, we talk a lot about automation. Where does automation fit into all of this? Can any of this validation happen continuously behind the scene instead of only when somebody remembers to actually check for it? >> With the QA cluster that you can build um before you go into production, automation can be used to automate that testing for QA cluster continuously in the background to catch things beforehand, before they're even caught in a production cluster. So, you can use automation to kind of like catch what's going to happen before it happens. >> Can you also share your thoughts on the what are some of the really crucial things when it comes to testing, you know, this checklist that they should not miss? What are those? >> The biggest thing that you should not miss when it comes to testing is first of all, with creating a run book, which [snorts] is going to have the list of steps that you ran during your testing to kind of like keep track of everything that happened. You want to make sure that everything that you tested was documented. In the event that a situation happens, so you can kind of trace back your steps to see what could have been in place to prevent something from happening again. Um run books, I think alongside QA clusters, run books, I feel are the the most crucial thing a lot of the times. We'll have events or situations that have happened, and we're asking customers, "Oh, like well, what happened when when this occurred?" And it's like, "I don't really know what happened when this occurred." Or "I don't really remember what I did." Um but having those set of steps in place, having documented information is the most crucial when it comes to failover validation and failover testing. >> What kind of culture organizations should build? It could be from top-down or within the engineering community that this is not seen as as we used to say someone else's problem. It is not It should not be an afterthought. It should be the core part the moment you write your application and deploy it. High availability should be part of that. So, what is your advice there? >> I would say to definitely build trust with customers. Um and making sure that there is a lot of communication that happens between you and the customers with making sure that or at least showing the customer how you are building the trust with improvements that you are going to make to your product. Um, because obviously no one no one likes when failovers happen and you know, you you lose money. So, making sure that you build trust with customers, making sure that you improve your product upon failures that have happened with your product is the biggest the biggest way to improve the culture um within your organization. >> Alexis, thank you so much for joining us and sharing these insights. And for uh anyone who is watching, their failover actually works when it really matters, please check out Scythe Technology and the solution that they're offering. Once again, Alexis, thanks for joining me and I look forward to chat with you again. >> Hi, thank you for having me.