Video summary
The video highlights the critical importance of routine High Availability (HA) testing by presenting real-world examples where such tests uncovered hidden configuration errors before they could cause significant downtime. In one recent instance, a partner's system was found to have a protected service misconfigured to start automatically outside of the HA solution's control. This setting would have served stale data or blocked access entirely during a failover, but early detection allowed the team to correct the configuration and ensure the system always pointed to the latest data. Similarly, another case revealed that client applications were hardcoded to connect to the physical server IP address rather than the virtual IP used during failovers; without testing, this would have prevented clients from connecting once services moved to the backup site. A third example involved a primary system upgrade where a configuration file and necessary packages were updated on the primary node but forgotten on the backup, an issue that routine validation would have caught and resolved before a disaster occurred.
Conversely, the transcript details a stark scenario where the absence of regular testing led to a major business-critical outage despite the HA environment appearing healthy. A client declined external validation services and skipped internal testing during a maintenance window, while also lacking a dedicated test environment for updates. Consequently, when a power outage forced a failover, the backup server failed because it was missing updated libraries and packages that existed only on the primary system. Furthermore, configuration parameters stored locally on the primary node had not been replicated to the secondary server. This combination of missing dependencies and unsynced settings turned what should have been a seamless transition into a complex emergency requiring multiple teams to troubleshoot database errors and application startup failures simultaneously.
The conclusion drawn from these contrasting experiences is that routine testing acts as a vital preventative measure that saves businesses from chaos, panic, and financial loss during actual disasters. When an outage occurs without prior validation, organizations are forced into reactive troubleshooting modes where they scramble to identify root causes, coordinate with multiple vendors, and often rely on on-call engineers who may not be fully familiar with the system's nuances. This delay allows executives to watch their business lose money by the second while teams work through issues that could have been resolved in minutes during a scheduled test window. Ultimately, the video emphasizes that proactive validation ensures seamless failovers, maintains data integrity, and prevents the costly confusion that arises when critical infrastructure fails without warning.
Read the full video transcript
Is it possible for you to share a real
world example where routine HA testing
uncovered a hidden issue that would have
only surfaced during an actual outage
and would have actually caused downtime
and chaos?
>> Yeah, we actually have um I have two um
in fact one is very recent as recent as
earlier this week. Um so in the first
one we were actually working with a
partner who was setting up a system for
an end customer and they engaged with us
to do some testing to make sure that
after their installation and updates
that the system was working correctly.
And our very first validation test uh we
discovered that one of the protected
services was uh misconfigured. In this
particular uh case the service was set
to what's called automatic start. So it
actually would start outside of the HA
solution and when that occurs uh the the
the data that it would serve up would be
stale data um because it was not you
know a part of the HA monitoring and
recovery. By detecting that early, we
were able to go in change the
configuration so that it was would not
start outside of the HA solutions
control and make sure that it was always
going to be pointing to the the latest
set of data. You know, if that had been
a found during a realworld scenario,
just think of how long it may have been
before the end customer discovered that
they were receiving stale data. um or in
in some examples uh related to this
particular environment, the client
application can start in in a way in
which it no longer allows anyone else to
access the data. And so finding that
prevents an outage. Um it prevents an
error. It prevents data integrity uh
issues because um you've detected it
early. you've corrected the
configuration and then you went on to
further validate that things are working
properly with the HA solution you know
um a second one related to that uh to a
client realizing that they had
configured all of their client software
so our end customer had an HA solution
in place uh using our SCIOS lifekeeper
high availability solution they were
protecting several databases and
applications uh their clients however
ever were configured originally to point
to the physical server IP address. And
so once the applications, services, and
databases failed over to the backup,
those clients could not connect because
they had been hardcoded. Testing and
validating in the the U customer's
environment helped us find that early,
make that a change and adjustment and
then get things corrected and retested
in a way in which the failover and
client reconnectivity became seamless.
And that's something you don't want to
find during an actual outage because of
course uh when there's an outage you
you're having to troubleshoot. You're
having to do a lot more root cause
analysis. You're opening tickets with
multiple vendors. You're trying to
coordinate understanding and and what
are the symptoms and signs and problems
and one thing that often gets overlooked
if the applications or databases are up
and running uh then then you're having
to go look and see okay what happened
with clients and that that and that
panic and chaos and that moment where
you're scrambling is something that can
be missed. So testing in real world
scenarios helped us find that before it
became an issue in in an actual
disaster. And then one recently we had a
client that did an upgrade of their
primary system but forgot to actually do
the upgrade on the backup system. And
they called us in because they wanted to
do some testing. And what we found is
that there was a configuration file that
needed to be um updated on both servers.
And that was something that we needed to
make sure that they were able to run
through and that allowed us to to find
the issue, update the configuration file
and also update their runbook. So now
going forward they know uh the
procedures and steps they have to do for
updates of the operating system, updates
of their critical database, updates of
their client application and they know
how to test it in a proper manner and
where to validate that things were done
uh correctly and so that saves them a
lot of time. Of course, if you're in an
actual disaster, uh you're you're
looking at, okay, well, why didn't this
work? And you're starting with
troubleshooting steps. And often these
are the little uh nuances that you might
miss while there are 21 executives
wondering why their business is losing
money by the second. Now, let's flip
this coin. Can you walk us through a
scenario where an HA environment look
perfectly healthy but the absence of
routine testing led to real problems
when a failover event actually happened?
That's a great question and it's a very
vivid example in my mind because of uh
how how many um different organizations
were involved in this particular
incident and how long um it took to
resolve and remediate the problem and
the fact that it was a business critical
system in a live production down event.
Uh so we did have a client who um set up
their system with our software and they
went through some uh what they
considered to be routine maintenance. It
was actually turned out to be more than
routine maintenance. Uh but for time
constraint pressures or business
pressures or stakeholder involved
pressure uh this particular client opted
you know first uh to decline services
that we offer for validation and and uh
testing and then second they also
declined to do that testing themselves
to make sure during their maintenance
window that they um that the failover
would work or that the applications
would run on the backup. A third thing
that we've kind of touched on earlier,
they also did not have a test
environment where they could run these
updates uh prior to rolling them out to
production. And so all of the updates
were made on their production system.
Things appeared to be healthy on the
production node. Um they had an outage,
a power outage in their data center. uh
things uh attempted to fail over to the
backup server and that's when they
discovered that there were during the
upgrade of the primary there were some
configuration changes that were made. Uh
there were also some libraries and
packages that had been updated to a
newer version that did not exist on
their uh target system or their
secondary. And so when it came time to
start the the database, uh the database
failed with some errors related to those
updated packages that were missing. Uh
the the application when they were able
to troubleshoot the database and get it
to a running and working state. uh the
actual application failed because there
were some configuration parameters that
had been made on the primary um and
stored locally in a in a config file had
not been replicated or synced to the
backup server. And so in a scramble you
you have a a team member uh from their
organization from the application team
trying to troubleshoot what's going on,
why the application won't start. Um, a
lot of times you you have in an
emergency like that, you may not have
the architect of the application
available. And so you're dealing with an
on call engineer from the app team, uh,
an on call engineer from the database
team. Uh, you may be dealing with folks
that are not as familiar with the
system. And in that emergency situation,
uh, this particular customer was trying
to figure all of these things out at one
time. you know uh ultimately their
resolution was to update the packages on
that target system, update the
configuration file and update the
database so that it would all start and
run. That outage um instead of it being
a simple failover, you know, power
failed in the primary data center,
everything starts up seamlessly on the
backup without clients noticing it.
instead that turned into a major outage
event and cost them uh significantly.
Whereas a routine test uh would have
identified all of these issues and would
have saved the business a lot of money.