TFiR

Automate HA Failover Testing Before Your Next Patch Cycle | Cassius Rhue, SIOS Technology

Duration: 3:43 579 transcript words EN summary

Video summary

For organizations beginning to establish a regular high availability testing practice, the most critical first step is aligning their testing cadence with their actual business update cycles. Teams should start by ensuring they have a lightweight copy of their production environment and then determine how frequently their business requires updates or patches. If an organization performs monthly patching, for instance, their HA testing schedule must match this rhythm to ensure that the test environment is ready and validated before any changes are rolled out to production. Similarly, companies with quarterly or bi-annual disaster recovery exercises should use those regulatory or certification requirements as the framework for their testing frequency, ensuring that both routine updates and major DR scenarios are adequately covered without disrupting operations. To make continuous HA validation practical without overburdening already stretched IT teams, automation plays a pivotal role in reducing friction, lowering costs, and increasing the frequency of tests. Rather than relying on manual processes, teams should look for frameworks that integrate key test functionalities directly into their HA environment to simulate node failures and verify that applications and services restart correctly. Automation tools allow IT staff to script these scenarios or use APIs to interface with the high availability solution, effectively removing the heavy lifting from manual execution and enabling more rigorous testing schedules that were previously too costly or time-consuming to maintain. SIOS Technology specifically addresses these challenges by providing an extensive command-line interface that can be leveraged for automation through scripting or dedicated automated tools. This capability allows administrators to define specific commands essential for their test environment, covering the entire lifecycle of a maintenance window from deploying patches to restarting applications and validating failover, failback, switchover, and switchback operations. By utilizing built-in operating system commands to crash services or stop databases within this automated framework, teams can efficiently validate that their recovery mechanisms function properly, ensuring that their infrastructure remains resilient even as they navigate complex patch cycles and disaster recovery requirements.

Read the full video transcript

Let's assume that there are teams who are just starting to build a regular HA testing practice. Where should they start and what does a realistic cadence look like for them? >> Yeah, that's a good question. If you're if you're just starting by building out an HA test infrastructure, right? So, let's assume that you've made sure you're doing a light for light copy of production. Um what you want to look at is what we tend to to to see with a lot of customers and recommend is what is your business cadence? Right? So, there are a lot of organizations that are doing monthly patching. And so, you want to make sure you're matching your HA testing to the cadence of the business. If you're doing patching every 3 weeks, uh then you want to make sure your test environment is set up and that your plan is um your project plans include time for you to test and validate in that QA uh environment before you start doing the rollout on your production system. Uh there are also organizations that in in addition to their monthly patching or update strategy, they're also doing like a quarterly or bi-annual disaster recovery exercise. And so, I would say start with looking at your business requirements. Uh how often do you need to update your production nodes? How often do you need to do disaster recovery scenarios either uh to satisfy a business requirement, a regulatory requirement, or a certification requirement for your particular industry, and use that as the the groundwork or framework for how often you do the testing. >> Are there specific tooling or automation approach that makes continuous HA validation more practical without burdening already stretched IT teams, and how does Scios help there? >> That is a that is a really good question and um great great thing for our teams that are thinking about uh testing or trying to make that burden lighter, automation can play a huge role in uh reducing the friction, reducing the cost, and increasing the frequency of doing that testing. Um a lot of tools exist out there, so I won't name specific ones, but what you want to look for is a framework that allows you to uh integrate uh key test functionality for your applications or services in your HA environment. Uh we talked about um tests that do hard [snorts] failures and then making sure that applications and services restart. So, you want to look for automation tools that allow you to have a framework that can uh simulate node failures, that can activate the HA solution functionality. A lot of HA solutions like SIOS, uh we have a SIOS has an extensive uh command line interface that you can use for automating either through scripting um or through automated tools that simply interface with a CLI or API, and that allows you to say which particular commands are most important for your your test environment. Um if you are, for example, doing patch maintenance, you can automate everything from deploying the patches, uh restarting the application, testing the failover, testing a failback, testing a switchover, switchback. Uh there are lots of commands um in each operating system, whether it's Linux or Windows, that allow you to do things to crash application services or stop databases, and then you can use those automated HA tools or HA CLIs to validate that things recover properly.