Hurricane Lessons Learned
What worked and what didn't for business hit by hurricanes this year? We went straight to the source and asked a variety of businesses to share their stories.
Tracking down employees, activating redundant systems, and gaining site access for initial damage reports was just the beginning.Many companies affected by the 2005 hurricane season are still in the cleanup and recovery phase. However, some business continuity managers are beginning to examine what worked, what didn't, and the lessons they learned. Leaders from a wide variety of organizations agree that lessons learned from past hurricanes provided the foundation for the monumental recovery efforts after hurricanes Katrina, Rita, and Wilma. For David White, Director of Emergency Preparedness for Tulane University in New Orleans, Katrina threatened not only the school and its employees but also the 1,500 freshmen who had arrived on campus that week. Having a detailed reaction plan based on the hurricane's forecasted category and speed, assembling university administration at a Houston site far from the area of impact, and pre-arranging student housing helped the university evacuate smoothly and recover quickly .In the aftermath of the 2005 hurricane season, employers and employees alike are shouldering the task of re-building their personal lives and professional organizations. For many ravaged by hurricanes Katrina and Rita, it took weeks of waiting for the water to dry before they could even evaluate the damage. Overcoming the New Orleans infrastructure collapse is an ongoing exercise in ingenuity and the utilization of disaster recovery plans. The Friday before Katrina hit, "It [the BCP] was thrown into full swing," White explains. "We met several times a day as time progressed to look at the threat and how it may have changed." After meeting again early on the morning of August 27, "the decision was made with very little hesitation - we needed to close the campus and evacuate the students," White says. They held campus meetings and distributed fliers to get the word out. "Most folks had been traveling so they were instructed to put their belongings in their dorm and either get back in the car with their parents or leave and get out of the city," White said. "We had 335 freshmen who could not leave for various reasons who had flown down alone or couldn't get a flight so they spent their first day of college getting buses to be evacuated to Mississippi. They handled it quite well … They understood the severity of the situation and once we got them there and they were watching the news reports they understood the necessity."
Plans in Action
While students were evacuating, Tulane's interdepart- mental emergency operations group assembled in Houston, TX to continue operations and wait for word from President Scott Cowen. Cowen, along with Senior Vice President of Operations and Chief Executive Officer Anthony P. Lorino, Director of Public Relations Debbie Grant, and Chief of Police Ken Dupaquier remained on campus to ride out the storm in a command center set up on the 150-acre campus. "There were some very harrowing moments from what they could hear outside, White explains. "The building [The Reily Recreation Center] was completely shattered, they would try to look outside but couldn't really see, they lost power, and they lost drinkable water (other than water bottles) … At one point the only source of water was the swimming pool."After executing the detailed evacuation portion of its plan, which included specific plans depending on the category and speed of the forecasted hurricane, the management team was ready to "start working to renovate the campus facilities and to get the contractors and companies lined up to start bringing in materials as fast a possible," White says.After they convinced Cowen to evacuate the campus, response and recovery went into full swing. The president is "as dedicated as they come, certainly a captain-go-down with-the-ship type of guy who did not want to leave campus," White notes. The university began the recovery process quickly, and "the president was there to hit the ground running," White says. After taking a boat to a dump truck to a helicopter, Cowen met the team in Houston where they started mapping out "what [they] needed to do and how [they] needed to get there." During the initial recovery phase Cowen's positive attitude helped get the team through the difficult moments.
While the president of Tulane University was riding out the storm on campus, text messaging proved invaluable. "We learned a lot and the most important key to survival was text messages," White explains. "They worked when the phones did not and that kept us alive. The president even commented 'I didn't know what text was before this and much less know how to send one,' and now he is very adept at it … We really did survive in the early days being able to send text back and forth." Along with most of the city, Tulane experienced severe damage and half of the 150- acre campus was under four to six feet of water. "Every building had roof damage, broken windows, and rain issues. When the levees broke half the campus was flooded," White says. "We started recovery on campus almost immediately. We contacted [a restoration services company with which the university had previously contracted]who started clearing debris, repairing roofs, and repairing windows." Having cleared the major debris, restored power, and provided drinkable water, the University is slated to open for classes on January 17 and is now working to dry flooded buildings, restore library materials, and replace building materials. "It looks like an island in the middle of an ocean," White observes. "The campus itself is farther ahead in its recovery than the surrounding neighborhood in New Orleans.We are occupying buildings, the lawns are cut, bushes are trimmed, the flowers are planted… We are approaching normality much faster than the rest of city. The primary reason for the quick recovery is our leadership."
Leadership combined with team work and quick-thinking were key to the New Orleans- based Whitney National Bank's survival. After evacuating essential equipment and 175 disaster recovery team employees to Houston two days before Katrina, Contingency Planning Manager David Lott was faced with a new challenge - housing enough employees to keep the bank running for an extended period of time. "We had a place to go not only for servers and the mainframe and items processing and printing and we also had a work area recovery space and pre-arranged employee housing," Lott explains. Throughout the storm and in the days following, the bank remained at least 75 percent functional. "At no point did we ever limit the amount of funds customers could get, and we always knew what customer balances were." Lott says. "I can't say enough about what vendors did for us. Our vendors came through for us in every instance. We would ask for things and we would have them the next day, had it all in 24 hours… I would call those guys up and say hey, I need 100 more PCs and they would have them here the next day." While a BCP based on past hurricanes combined with round-the-clock work and a focus on customer needs kept Whitney Bank on its feet, New Orleans-based Hibernia Bank's alternate site in Shreveport, LA enabled the company to relocate without interruption before the storm arrived.
According to Vice President and Manager of Business Continuity Greg Stelly, "The things that we had best planned for worked. Of course we had prioritized our computer systems for recovery and we had backup systems despite the power outage, communication was not a problem for Lydian Trust Company, the parent company of Lydian Private Bank, says Bob Alsan, the bank's corporate business continuity planner.
Having a corporate BCP manager on site at the data center throughout any hurricanes.
The disaster recovery team sent updates to the incident management team (IMT), which received live updates from the company headquarters, the data center site itself and from the disaster recovery site.
Pre-scheduled telephone conferences before, during, and after the storm
An 800 call-in line for employees to get info updates
The company supplied (wireless devices/Blackberries) to all team members
Air-cards (they are PC cards similar to a computer modem) were supplied so all team members had mobile Internet access by using their Air-cards from the disaster recovery site, the main office, or at home via a mobile computer with an Air-card.
A replication of the corporate network environment. "Our e-mail and calendar is all on the Internet through the Air-cards. If you are mobile you can actually call in and access the network email through Blackberries and through mobile computers," Alsan says.
Voice over Internet phones were connected to the network,"which never went down," Alsan says.
Katrina's devastating effects on Louisiana were echoed in other
areas of the country as this year's exceptionally harsh hurricane
season progressed. Southeastern Florida faced prolonged power outages
from hurricane Wilma and businesses faced many obstacles, though much
less comprehensive and systemic than New Orleans', in their
For Alpha Staff, a Boca Raton-based human resources outsourcing firm, the key to survival was geographical diversity. With three operational offices in Atlanta, Tampa and Boca Raton and three sales offices in New York, Atlanta, and Boca Raton, the 109-member company sent a team of 17 people to their Atlanta office, where their servers and systems were located. Jack Rahner, Director of IT Operations, said they were 90 percent operational during the storm. During the 12-day power outage that followed at their Boca Raton headquarters, the company's disaster recovery team completed 100 percent of their nationwide clients' payroll and benefits processing from Atlanta. "The key issues from a technology standpoint were covered by having everything in Atlanta" says Chief Information Officer Ralph Labarta. With e-mail failover capabilities, Blackberries and backup phone service the company had communications covered. "However," he added, "all the technology in the world does no good if people can't access it. In Atlanta we were able to expand quickly based on the need," Labarta says. "The provide payroll and benefits services for our companies," Labarta explains. "We did get 100 percent of payroll out the door and service customers just like we would under normal circumstances."Continuing operations without interruption becomes easier with practice. After weathering four hurricanes in two years, Palm Beach Gardens-based Lydian Trust Company, comprised of seven business lines and 14 offices and banks nationwide, was well positioned to implement their BCP. From pre-arranged generator fuel and janitorial services to employee meals and child care they have a detailed BCP and continuously improve it.
Before Katrina and Wilma hit and during the aftermath, Lydian's business continuity plan worked smoothly. "Lydian's data center is always up" says Corporate Business Continuity Planner Bob Alsan. The company's impending disaster timeline, which has become more comprehensive each year, worked as planned when activated for Katrina and Wilma. "You have a granularity to [the plan] showing what needs to be done at what point in time in an impending disaster," Alsan notes. This year when Lydian's BCP was activated for both storms team members went to Lydian's backup site in Atlanta, GA. "The administrative projects that worked within the BCP included contacting armed guards to work at the banks, the offices had dedicated teleconference lines, the postal re-direct services worked and we engaged Crisis Link to our telephones, as well as [distributing] emergency cash to key personnel," Alsan says.
What Worked? What Didn't? And What Did We Learn?
Contacting vital vendors, detailed in Lydian's BCP, was conducted before the storm to secure generator fuel delivery, a cleaning and drying crew, and a handyman. "We were also in touch with the property management throughout the storm… All those components of the BCP prior to an impending disaster worked as planned," Alsan says. "We actually had vendors such as diesel fuel for generator service the same day as the storm… Just several hours later they were all here on site."
Many companies, while used to working around hurricanes, had BCPs that are optimized for three to five days and did not include procedures for a long-term relocation or complete infrastructure collapse. "We anticipated being out of town for maybe a week, with some damage or disruption for a month as most," David White of Tulane University says. "On Tuesday afternoon we knew that was not the case. "We are now overcoming hurdles as they come up and plowing forward,"White reports. "Our biggest hurdle was assembling everybody because so many were so dispersed and because communications were devastated. We lost Web, cell, and land lines.
The mirror site from within city lost the ability to communicate via Web site until we got servers and systems back up and running," White explains. "The main page was working one week afterwards to post messages but we were not able to send e-mail for over a month because of the infrastructure of the city." With power still out in parts of New Orleans, communication was and is an obstacle. While many organizations have redundant communication capabilities, contact between disaster recovery sites, headquarters, employees, and vendors working to restore damaged buildings remains a problem for some companies.
After Procter and Gamble's Folgers plant in New Orleans was hit by Katrina, "Communication was a major challenge," says Public Relations Representative Jen Becker. "For [the] first few weeks, it is was especially difficult… And even from time to time today reaching cell phones is a challenge. Getting computers back up was also a challenge... E-mail was intermittent at best and with phone the lines we have had major challenges because of the loss of infrastructure."While Hibernia had several types of back-up communications, the employees, scattered throughout Alabama, Florida, Mississippi, Louisiana, and Texas also relied on old-fashioned phone chains. They were also able to call an 800 number, set up for each state, to receive company updates and essential information. "We constantly updated the online portal to provide the numbers employees needed to call for human resources and we constantly updated messages on the 800 number," Stelly says. "That's how we communicated because cell and land lines didn't work." While most companies who have experienced hurricanes along the Gulf Coast and in the Southern states are used to dealing with communications disruption they were not prepared for long-term evacuations or relocating large numbers of employees for an extended period of time. "[It] was challenging … to relocate such large numbers," Greg Stelly of Hibernia Bank emphasizes. "We planned for 200 people with vacant work space … 200 people that would get us easily through the first few days. In this longer-term outage, we really in the end, relocated 1,400 people from New Orleans area to Shreveport, Baton Rouge, Houston, and Dallas… Primarily that was a big challenge and we got it done pretty quickly and had folks up and working in the first few weeks."
On the Move
Hibernia put the plan into motion immediately after the hurricane
forecast changed from the Florida panhandle to the New Orleans area.
"Within eight hours we went from pretty comfortable to knowing that we
had to react," Stelly explains. "We sent the technology team to prepare
for recovery systems and placement. We went from original planning,
which takes care of relocating a smaller number of people, to having
approximately 3,000 of 6,200 employees displaced by Katrina."Before
they were able to secure corporate housing complete with furniture and
cable, David Lott and a team of five people from Whitney Bank spent
countless hours looking for employee housing and employees spent
their free time moving from hotel to hotel. "Everybody in New Orleans
relocated this way. [Houston] hotels were a big problem." Lott says.
"Although we had deals in place with hotels beforehand … we would call
and automatically block off a certain amount of rooms. Initially they
were able to accommodate us… However we kept having more and more
people arrive and some hotels could only accommodate us for so long."
Eventually the company needed to relocate more than 400 employees and
their pre-set hotel arrangements began to run out. "Housing was the
biggest challenge we had … and we were extremely lucky… We had a place
to relocate our workforce," Lott observes. After a month of shuffling
employees from hotel to hotel, Whitney Bank secured corporate housing.
"It wasn't a smooth process but it turned out okay. I don't think
anybody is wanting for anything anymore over here. I don't know what we
would have done without that group of people working together, working
long hours walking to apartments and signing a lot of leases," Lott
says.Every company's first priority after the storm was to locate
employees. However providing shelter was not the only challenge when
faced with a large-scale relocation. Employees faced having to relocate
to start business
activities while worrying about their families at home. "It's quite traumatic for people to relocate, some employees were worried about their families and had left with broken roofs on their houses," explained AlphaStaff's Rahner.
"Five days into this thing with people relocated in Atlanta and there is a lot of stress and worry those employees are carrying around," Labarta says Stelly agrees that it's important to be mindful of what employees are going through. "Checking in with employees to make sure they are OK was our first priority; you want to be able to know where people are and to account for everybody… Taking care of employee's is essential."
For companies in New Orleans, the infrastructure collapse after Katrina also affected the availability of resources for rebuilding and site access during the recovery phase. Whitney Bank had trouble accessing some branch locations to conduct damage assessments and provide customer's access to their safety deposit boxes. "Site access was severely limited wherever FEMA and the National Guard were," Lott explains. "They blocked all of the roads going in. You had to have a pass to get around and even with a pass you had to go through checkpoint after checkpoint to get to where you were going. Luckily enough, we had all the contacts we needed and we were able to secure passes to go wherever we needed to go. We were able to make several runs into our operations center to get equipment we needed. "Because of the flooding, Whitney Bank had trouble gaining site access to 17 of its 56 damaged branches."Until [the water] receded, there wasn't a way to do any kind of damage assessment there was a lot that got water damage because of the wind damage that came first," Lott says. Greg Stelly says site access was also a major obstacle for Hibernia Bank. "The parishes and local government closed down the area for quite some time until they felt it was safe for companies like us and individuals. It took two to three weeks before we were allowed access back into those areas … We were able to get special passes from authorities to go in and check things out." While most companies in areas susceptible to hurricanes incorporated hurricane-specific plans into their BCP, the flooding was a new obstacle. For Tulane University, not only were buildings damaged but also the special collections and archives sections of the library had spent weeks underwater."Four to six feet of water was a major obstacle that prevented access for internal assessments," said White. "The library special collections and archives, the law school and the business school had flood damage … about 50 percent of the campus was flooded with obstacles in the water."
Not far from Tulane, Hibernia Bank had to shut down their New Orleans data center Wednesday morning when water service failed because of the flooding. Loss of water to the building meant that they could no longer cool the computer rooms, and the mainframe went offline. "We worked to get backup tapes out of New Orleans and to the disaster recovery facility where we'd reestablished mainframe operations. Our restored disaster recovery mainframe was back online when our branches opened on Saturday, September 3.The branches were open and worked in offline mode late Wednesday, Thursday, and Friday as the data was restored. So, customers did see a difference in service on those days. Branches could not access account balances or provide other online services, but did their best to serve our customers without those services, " Stelly explains.While flooding and site access was a concern for those watching from afar they were also worried about the rapidly deteriorating conditions and lawlessness in New Orleans, which posed an immediate danger for those who remained. "One of things that was an obstacle in doing the assessment were conditions in the city," White comments. "There were looting and safety concerns and downed lines; the group on campus was hesitant to venture out into areas of the campus … especially at night. At that time the [National] Guard had not been able to get in and the police department … wasn't concentrating on crime. That created a serious safety concern for those on campus. It was also an obstacle to getting assessments done to give an idea of what direction to go in. Once we got the president and others to Houston, they could give us a better idea by informing us of what was going on. Then we had a clear idea of what we were facing and where we needed to go with the recovery process … because the city was not recovering in the initial days."White emphasizes, "We knew our recovery was up to us. We could not rely on federal or local city and state [departments] to help us recover, because of many other things going on. If we were gonna survive, we were gonna have to do it ourselves … We could not afford to sit back and wait."Most companies that routinely experience hurricanes are used to dealing with power outages. However, the extended outage caused by Katrina, and for some companies that were hit by Wilma, resulted in no telephone or computer access and rationed fuel."Since we have had so much experience year after year, we pretty much nailed it down," says Lydian's Alsan. "What didn't work for Southeast Florida is fuel. Florida had plenty of fuel … However, since there were major power outages there was no power to pump fuel from the ground. All of Southeast Florida lost power so we are considering new service agreements with a delivery company to ship it to our employees via a dedicated truck," Alsan says. An officer for the Association of Contingency Planners (ACP) for Southeast Florida, Alsan regularly discusses local BCP issues with business leaders throughout the region. He maintains that providing fuel for company employees, particularly hospital employees, is essential. "Some BCP managers work in hospitals for example … They could get fuel for the hospitals but the doctors and nurses that worked at the hospital were not allowed to get fuel … The police cut them off due to the mandate from the state. We are trying to offset this with a subscription in advance before the storm. A dedicated truck can come down and pump [fuel] up to employee cars if needed."
While it is too early for after- action reviews for most companies, many say that the planned and tested aspects of their BCP worked as expected. However, many BCP managers also mentioned that having a disaster recovery site or duplicate operations far from the area of impact and having a business network of vendors surrounding the disaster recovery site are important BCP additions. Some other important lessons learned from this hurricane season include: securing enough pre-arranged employee housing and having enough redundant communications systems, data systems, and servers in place to sustain a company throughout long-term power outages. "We are very accustomed to the threat of hurricanes, we understand that threat very well because it's had a heavy impact on business with Katrina," Stelly says. "Now it's working to mitigate the level of risk we have in the New Orleans area … Perhaps moving some of the computer systems and more important business units out of the area or setting up duplicate operations so we don't have the level of risk we do now." While Whitney Bank's business units are still somewhat scattered, they are preliminarily considering relocating their essential operations. "We are looking at different places now … places that are unaffected by hurricanes," Lott notes. "The main office will still be in New Orleans but it does not matter where you process your work. We'll definitely move the processing site out of harm's way so that we won't get affected like this again. Even compared to places we compete with we fared extremely well because we had a place to go. We tested that site numerous times… Everyone knew what to do, where to go and what their job was. I know a lot of companies that their backup site is 30 miles from their main … In this case that does not work. Even 100 miles would not have been far enough … We would have been down for weeks … so when you consider the scope of a disaster like this you've got to have a way to get out of that disaster cone."
Plans Are ChangingWhitney Bank plans to have more robust data communications backup and redundant communications in place in the future. Lott is also considering establishing an automatic company roll-call and training employees to automatically call the emergency 800 number once a storm or other event has passed. "When going through these things there is a lot out of your control … You can call the phone company over again but they are not on your time schedule," Lott acknowledges. "Everyone is in the same boat and you are at the mercy of the communication companies. We hope to change that in the future and have everything already in place and not have to depend on those guys quite as much as we did this time." When the power fails, sometimes hours before storm hits, BCP managers need to be kept abreast of any changes in the storm's path, category, and wind speed. According to Lydian's Bob Alsan "Everyone got caught off guard when Wilma turned out to be a Category 3 instead of the predicted 1 or 2 We relied too much on weather predictions and by the time it was overhead it was a 3. We may have a secondary BCP contact out of the region to update me because Wilma increased dramatically through the time period and we were unaware of it." AlphaStaff is planning to continue to diversify geographically in order to remain resilient and still retain offices in hurricane-afflicted areas. "The number one hardest lesson to figure out is how long can you really go on your own in an environment where many people are impacted. Whether it is a power outage, a lack of fuel, or debris in the road preventing people from traveling … If you are a medium-sized company there are only so many resources you can devote just to disaster recovery efforts. Diversify geographically and make accommodations so at any point in time you can move people from office to office and operate in a different environment depending on the length of the outage and impact in the general area,"
Labarta says.Maintaining several sites will allow companies like AlphaStaff to develop business networks surrounding their offices, mitigating some of the risk they face when relocating to a disaster recovery site. Setting up a network of business vendors and contacts in the same city as a company's alternate or disaster recovery site will help to speed up the recovery process after relocating company operations. Lott agrees that working to create business networks surrounding disaster recovery sites is worth the effort. "It's a totally different communications provider here in Houston than in New Orleans. "We had some pull in New Orleans but when you come to Houston and deal with Southwestern Bell as opposed to Bell South, you are just another customer with no priority at all. We plan to add more hotel contacts and robust redundant communications, to have everything already in place," he says.
Leadership, ingenuity, flexibility, and round-the-clock work combined with a BCP built on lessons learned from past hurricanes propelled many damaged companies out of the disaster zone back towards normalcy. Whether adding more redundant communications, backup generators, or working to move the mainframe to an alternate site, BCP managers are planning for the future while still rebuilding from this year's hurricane season. Companies are becoming more self-reliant while also working closely with other organizations to plan and test BCP. Their goal is to create a BCP that can encompass the consequences of long- term power outages, employee evacuations and extensive flood damage while protecting employees and returning to regular operations as soon as possible.
Plans for more extensive disaster recovery networks and more comprehensive hurricane-specific additions to BCP are just beginning to take shape, Procter and Gamble's Jen Becker agrees that forming alliances is the fastest way to recovery and will lay the groundwork for a quicker response in the future. "There is a great need for collaboration on all fronts," she said. "We worked very closely with local landscape and government officials in the aftermath and are working towards securing preventative measures. The biggest lesson learned was that the more the different entities can work together to get everything up and moving again the more benefit it is to everyone."