When Plan B doesn't work!

Every disaster is quickly followed with news reports of things not working. Infrastructure being the first to be highlighted with power, transit systems, and cellular service all leading the list. Plan B, that is, your disaster recovery plan, does not always work. The New York Stock Exchange closed for two days during Hurricane Sandy due to a lack of confidence in its back-up plan. That is a huge "Blink" in my opinion.

The question is: have we really learned from Katrina, Irene, and now Sandy? Do we have a plan if our current Plan B doesn't work? From my point of view, disaster recovery planning means having more than one script and one plan. Disaster recovery means having several options and an extensive understanding of the technology that is being used and is available.
 

The power of the question
If cellular service and internet access are central to your disaster recovery plan (or business continuity plan), you should be certain that you have more than one internet provider and more than one way to access them. A case in point: cell service is tower based and represents the last or first mile. It is, in essence, the distance from your cell phone to the cell tower. Once you connect with the cell tower your call is routed via land line. The opposite is also true if the person that you are trying to reach uses a cell phone. The person being called is located and the call is routed to the nearest cell tower to them. If they are not available, or if the tower does not have any capacity (due to heavy usage) then the call is routed to their voicemail where you can leave a message providing the mailbox is set up and not full. This little known fact has caused numerous arguments because the person making the call is frustrated because the person receiving the call did not answer and their cell phone did not ring nor does it indicate a missed call! Hmm....

 
Now imagine a disaster and a huge surge of call traffic assaulting the available cell towers. Important item to note: when a cell tower is busy or overloaded, it doesn't tell you. Another complication pertains to power. If the generator for the cell tower is in the basement of the building that tower is on, and the basement floods, then you have no power and no tower! The result is a cascading calamity of unfortunate events. The same can be said about the internet. If both of your internet providers, which you thought were totally separate, are being delivered using the same infrastructure, then you have a resiliency problem you are not even aware of until it fails.

 
Know your infrastructure (KYI)
A sound disaster recovery or business continuity plan should be built by those that know and understand the infrastructure that you are relying on. The plan should also include the assumption that mission critical components of the plan do not work. The irony of Hurricane Sandy is how popular pay phones suddenly became in addition to wood burning stoves that were set up in public areas that boiled water and generated electricity for area residents to recharge their cell phones.

 
Plan B doesn't always work. We should apply the lessons learned from events such as Sandy, study our infrastructure, assume at some point we will not have all the components of our infrastructure and plan accordingly. "What if" scenarios are a very important part of planning and preparation process. Looking back to Y2K, one of the items we purchased was a hand crank radio that did not need a battery. We were laughed at, at the time, but residents of New Jersey and lower Manhattan could of have used one, in addition to a hand crank generator to recharge batteries. Maybe simplicity is best when it comes to managing complexity and failure.