Hurricane Katrina´s unprecedented devastation holds significant lessons for IT that should prompt CIOs and managers to thoroughly review existing Disaster Recovery Plans (DRPs). The sheer scale and scope of the disaster highlight how much more vulnerable small- to mid-sized businesses (SMEs) are. Review the enterprise DRP now to identify and address any lingering vulnerabilities.

A Different Kind of Disaster

Until now, the September 11th, 2001 terrorist attacks were viewed as the watershed events that reinforced the importance of having a DRP for enterprises of all sizes. Hurricane Katrina has cast additional light on just how inadequate many of those DRPs are.

Katrina differs from the September 11th terrorist attacks in a number of ways:

• It covers a far wider geographic region.
• The duration of impact will be significantly longer due to the massive and prolonged flooding.
• It took down virtually every element of physical and technological infrastructure in the impact zone.
• It affected a much broader range of the population, raising the potential for large numbers of employees to be unavailable to the enterprise for extended periods.
• Even intact organizations were compromised by vendors, suppliers, and consumers that were not so lucky.

Unlike terrorist attacks, earthquakes, and some types of virus and malware incursions, hurricanes typically offer at least a day or two of notice. Info-Tech strongly recommends updating the DRP to account for both predictable and unpredictable types of disasters.

SMEs Are Especially Vulnerable

Although the risks of widespread, long-term disasters apply to all sizes and types of businesses, SMEs are particularly vulnerable. Unlike larger enterprises, SMEs are less likely to have their own remote data centers. This leaves their primary sites more exposed to loss, and their business operations less likely to survive a direct blow.

Enterprises focused around a single location are inherently less able to continue operations in the event of a disaster, increasing the risks of significant profitability impacts or bankruptcy in worst-case scenarios.

Recommendations for Tightening the DRP

Info-Tech recommends implementing change to the DRP on two key fronts: long-term and immediate-term. Longer-term changes must focus on elements of planning that directly address latent vulnerabilities in the existing DRP. Immediate-term actions should help define and refine the action plan that the enterprise will follow in the days leading up to a future predictable event.

Long-term

1. Calculate long-term and widespread infrastructure damage in DRP assessments. Many DRPs assume only the particular enterprise is affected. This fails to recognize the risks associated with large-scale natural disasters like hurricanes.

• Assess risk based on vulnerabilities of all supply chain and market constituents. Challenge suppliers and vendors to engage in similar assessments of their own DRPs.

2. Identify low-bandwidth messaging alternatives. Immediately following Katrina´s rampage, text messages got through with a greater degree of success than voice or e-mail. Text messaging has lower bandwidth requirements and is more tolerant of limited or compromised network connectivity. Although this is not a viable long-term communication strategy, consider implementing this type of infrastructure in the immediate aftermath of a major disaster to bridge the enterprise´s needs until more capable communication infrastructure can be rebuilt.

• Blogs and wikis are other possible interim communication media that can be quickly implemented, then left behind as recovery progresses.

3. Consider alternative infrastructure. Wireless networks can quickly replace destroyed landline infrastructure after disaster strikes. Do not rely on only one type of infrastructure. Select at least two to ensure continued connectivity. For voice-based communication, for example, look into traditional wireline, Voice over IP (VoIP), cellular, shortwave radio, and satellite.

• A number of vendors specialize in managing messaging infrastructure on an emergency basis. MessageOne´s Emergency Mail System (EMS) allows IT to failover its e-mail servers to the service. EMS will then manage all messaging needs for the duration of the disaster. SunGard provides similar capability for basic messaging environments as well as complete data centers.

4. Document and distribute a disaster-related messaging hierarchy. Ensure all employees are aware of and trained in messaging protocols if primary means of communication are unavailable. Include a media response plan as well.

5. Build more detailed timelines. Disaster is often viewed as an immediate event, followed fairly closely by an extended period of recovery. SMEs have learned from Katrina that relief may be weeks or months away. Build multiple scenarios for both near-term and long-term outage scenarios.

• For example, if at least partial business operations cannot be restored within one week, leadership will need to determine whether relocation is warranted. Details should reflect all potential scenarios that apply to the particular enterprise.

6. Research offsite alternatives. Although building a standby data center facility in another state may be beyond the resources of most SMEs, managed service providers could provide an affordable remote data center alternative. Remote data replication services use Continuous Data Protection (CDP) to routinely back up data in near-real-time. Info-Tech is also aware of some enterprises that have signed agreements with like-sized organizations to serve as each other´s hot sites.

7. Define offsite work processes for employees. Remote or telework policies can often allow post-disaster work to continue from wherever employees might find themselves. Determine where work can - and cannot - be moved, and how it will get done afterward.

• It is important that this functionality be regularly tested by all employees involved, even if they don´t normally work from home.

Immediate-term

8. Accelerate the rate of backups. Weekly backups, for instance, should become daily backups if a major storm approaches. If backup media are not already being taken offsite, start doing so now. Also consider taking backed up media further away from the potential impact zone. More frequent backup schedules also reduce data loss potential in advance of unplanned disasters like earthquakes.

• Conduct a review of the backup sets. Changes implemented in data sets since the last restoration test may not be reflected in the current backup set definition.

9. Confirm fuel supply for generators. Speak with fuel vendors to ensure backup generators will have an uninterrupted supply. Identify key employees and accountabilities to ensure the flow continues throughout the disaster period.

10. Contact vendors and suppliers. All key constituents should be aware of the enterprise´s plans in advance of a disaster. The more they know beforehand - before communication links are potentially cut - the better able they will be afterward to respond in a timely and appropriate manner. This reduces business continuity risk.

• Remember to confirm alternative means of communication with key constituents. If a supplier´s main phone line is down, for example, having the CEO´s home number might help. Reciprocate in all cases to maximize goodwill.
• Challenge vendors to share their plans as well. Verizon Wireless, for example, has rolled out Cells On Wheels (COWs).

These self-powered remote cell sites are providing interim coverage in areas of New Orleans. BellSouth is implementing network regeneration techniques and an extensive network of generators and alternative fuel supplies to ensure network availability.

Bottom Line

CIOs and IT managers everywhere will be studying Katrina´s lessons for years to come. But the message is already clear: most DRPs were sorely lacking. IT must aggressively identify and address all weaknesses in the plan before the next big disaster hits.