A habit that is easy yet dangerous to fall into is to build recovery plans and disaster plans and then shove them in a drawer and ignore them until they are needed.
If you do that, it is almost guaranteed that by the time you need the recovery/disaster plans, they will be incorrect or out of date. In addition, if you do not keep them up to date, you open up the possibility for a number of other problems to be introduced, making the plans impossible or impractical to implement successfully.
As such, you should plan to test your recovery/disaster plans on a regular basis. It should become part of your company culture to regularly test these plans and other risk mitigations.
One model for testing these plans is to run Game Days. A Game Day is when you test invoking a specific failure mode into your system and watch to see how your operators and engineers respond to it, including how they implement any recovery/disaster plans. After the Game Day, a postmortem review will uncover issues with your plans and changes that need to be made. These changes will keep your plans fresh and updated and ready to be used when a real problem occurs.
Staging Versus Production Environments
You might be wondering whether you should test recovery plans on a staging environment or on your live production application. This is a tough question and does not have a simple answer. Let’s take a closer look at each of these options.
Staging/Test Environments
Testing recovery plans in a staging/test environment is the safest option. Using a staging or test environment allows you to perform invasive testing that would normally disrupt production environments in inappropriate ways. In addition, you can perform those tests without fear of mistakes that could cause production outages. If you decide to use a staging/test environment to test your recovery plan, keep the following information in mind:
· Make sure the staging/test environment is completely independent from your production environment. The testing environment should not depend on any production resources, and the production resources should not depend on any resources from the testing environment. See Figure 10-1.
· Make sure the staging and test environment mimics your production environment as closely as possible. Using a staging/test environment to test your recovery plan can be effective, and you can use these types of environments for testing a wide variety of destructive failure scenarios. However, they cannot guarantee the same results that would occur in a production system.
Figure 10-1. Dangers of not separating environments for testing
This is because production systems are almost always scaled to more and larger servers, contain a larger dataset, and manage significantly more traffic in real time.
These differences make certain types of testing in nonproduction environments unuseful.
Ideally, the test environment should be scaled to the same size as your production environment and be seeded with the same data used in your production environment; however, this is usually not financially viable and can be difficult logistically.
If you believe your testing requires a system scaled to the same level as your production environment, you might want to consider production environment testing instead.
Production Environments
Testing risk and recovery plans in a production environment seems illogical. Why would you force a failure mode in production just to make sure that your production systems don’t fail? The answer is simple: if you test recovery on your production environment at a time when your team is available and sharp (in other words, not in the middle of the night) and at a time of day that has the least impact on your customers, and you carefully consider the steps your tests will take, you can safely perform testing on your production environments under real-world situations and get valuable data on how your efforts would really perform in real failure conditions.
If you decide to use a production environment to test your recovery plan, keep the following information in mind:
· Be aware of the impact of your injected failure on your live customers.
· Consider the business aspects of the testing. What’s the trade-off between adding additional production risk to your customers’ use of the system and the reduced long-term risk of learning from the results of those tests?
· Perform the tests at a time when your staff members are at their sharpest (during the workday, when your staff is normally in the office), but also at a time that will minimize impact to your customers (such as at potentially slower traffic times, not during critical time periods like end-of-month or end-of-quarter sales pushes).
· Make sure you have the processes in place that can implement necessary fixes, and roll back failed fixes, quickly and easily.
Concerns with Running Game Days in Production
Game Days on your production environment should be planned and monitored very carefully. Planned appropriately, a production Game Day can be quite revealing of problems within your production environment. Here are some example production Game Days that you could run:
Server failing
What happens if a single server in your system fails? Try taking one out of service. If your system has a proper amount of redundancy built in, this should not have any impact on your production system. But use the removal as a test of your systems to detect such a problem, and of your recovery plan for replacing the lost server.
Network partition
What happens if a network outage or partition occurs? Planned carefully, you can simulate a network partition without significantly endangering your production application. But this can be a solid test for the notification and follow-through actions of your system and your teams in responding to the event.
Data center failure
What happens if an entire data center goes down? Planned carefully, your application should be able to handle such an event. How do you respond to such an outage?
Random failures
What happens if you introduce smaller, random errors into your system? Does your application recover from these errors reasonably?
The last item in this list in many ways feels the most threatening. Why? Because you can imagine what will happen when a server goes down or a data center goes down. You probably already have plans in place for dealing with that (if you don’t, then you should). But a “random” problem, even if only small in scale, feels like something out of control. It is out of control. But it is these random events that cause you the most problems in building highly available, risk-mitigated systems.
CHAOS MONKEY
Netflix takes the random failures problem to a new level. The company has a system called Chaos Monkey built into its application. This system randomly and regularly introduces random faults into the application, in the production environment, with live running customers. Exactly what Chaos Monkey does and how it does it are not known to the engineers and operators managing the application. Instead, it is assumed that engineers have put the proper recovery and mitigation processes in place so that the problems that Chaos Monkey introduces can be resolved or worked around without affecting customers at all.
Chaos Monkey runs only during business hours when engineers are around and available to respond to any problems that don’t self-correct. The philosophy around Chaos Monkey is to encourage, and actually require, the building of highly available, self-reliant services and applications that can survive and recover without human intervention. This is tested during the day when humans are around, with the hope that the problems won’t occur at night when the application is busier (more customers) and engineers must be paged into work. It is a novel approach that works well for Netflix.
Chaos Monkey is a great example of a best practice for Game Day testing, and Netflix has done some miraculous things with its Game Day infrastructure.
However, it took significant effort, significant resources, and a significant commitment before Netflix could get to the point where running Chaos Monkey in production could be done in a safe and effective way.
Chaos Monkey should not be your first step into production Game Day testing. But it can be a reasonable goal to work toward if your company has the commitment to making it happen.
Summary
Game Day testing is an important avenue of testing that can help assure your production environment will operate fully at a systemic level. It allows validating your support plans and processes in a safe manner so that when you really need to use them, they will work without issue.
Done properly, Game Day testing can dramatically improve your system availability at scale and reduce your risk of serious problems or failures in your production environment.