Power Plans

By Mark Hagland
April 2005

Article URL: http://www.healthcare-informatics.com/issues/2005/04_05/cover.htm

At the top of a road that dates back to the Civil War era sits 1,100-bed Montefiore Medical Center, Bronx, N.Y. In August 2003, the entire northeastern section of the United States was darkened by a massive power-grid outage. But "we weren't affected at all because we have so many power and IT capabilities built in," says Jack Wolf, CIO of Montefiore and CEO of its subsidiary, Emerging Health Information Technology (EHIT), which manages IT operations for Montefiore and nearby 854-bed Bronx-Lebanon Hospital Center. "Montefiore Medical Center was absolutely a beacon on the hill in that blackout."

How did Montefiore achieve its standing as a lighthouse in the gloom of the worst outage in U.S. history? The organization has "complete power redundancy with multiple power grids/feeds, multiple central office connections, and multiple service providers on our communications network, as well as a full UPS [uninterruptible power source] generator covering backup on power," Wolf reports.

Equally important, Montefiore and Bronx-Lebanon have made comprehensive preparations for disaster recovery (restoration of normalcy after a crisis or service interruption) and business continuity (keeping operations running during a disruption). Through EHIT, they have contracts with disaster recovery companies, including GoldenGate Software, San Francisco, Calif., chosen for its ability to "duplicate every aspect of every transaction on the production box onto the mirror box," Wolf says. After testing, the system will go live this summer.

A realization nationwide

Implementation of comprehensive clinical information systems, especially electronic medical records (EMRs), is altering executives' assumptions about disaster recovery and business continuity (DR/BC) preparedness, say industry experts. When hospitals had only paper records and few clinical systems, DR/BC planning focused primarily on financial systems. Crashes were inconvenient but not disastrous. That situation has changed.

Business continuity preparedness for mission-critical functions now means securing the EMR and clinical applications, plus maintaining 24/7/365 availability and usage. Most patient care organizations haven't spent the funds necessary to protect their burgeoning clinical systems. And it's not just catastrophic disasters like fires, floods, hurricanes, tornadoes, earthquakes and terrorist acts that could disrupt IT-facilitated patient care. A host of possibilities are less cataclysmic yet more likely, such as power outages, human error and small-scale sabotage.

DR/BC planning, say experts, must be comprehensive and robust for any organization with broad clinical information systems and must factor in facilities, infrastructure and hardware, software, data, personnel and processes. And plans must be drilled out and kept active; an unpracticed plan has far less chance of succeeding. Few hospitals and large medical groups are as prepared as they need to be, given the current proliferation of clinical systems, say those in the know. EMR and clinical information system adoption is far outstripping disaster preparedness.

"The No. 1 concern is patient safety," says Jonathan Thompson, a senior manager with the StoneBridge Group, a Minneapolis-based consulting firm, and as organizations become more dependent on EMRs, they must have "contingencies for processes and systems." Thirty percent of disasters are related to simple power outages, Thompson says. "What will you do if you're looking up a patient record and looking up the last 10 lab results to make a decision on care for a patient and the power goes out?" Continuity planning is "absolutely critical," he says.

Money remains the dominant constraint. The cost to implement a DR/BC plan is very high compared with other initiatives, Thompson concedes, but "there are some very quick hits that you can perform that do not cost a lot of money and that deliver immediate benefits." Among them are purchasing a UPS attached to a local PC not on your network, or at least defining what your current process is and doing basic planning. Most hospitals still have not done effective basic planning, Thompson notes.

Of particular concern are small clinics and physician groups, says Scott Jarr, director of product marketing at LiveVault Corp., Marlborough, Mass. They tend to be badly underfunded for DR/BC planning and lack the IT professionals needed to help them continue operations and recover from disasters.

Plan, prioritize, practice

DR/BC planning might be challenging but shouldn't be seen as overwhelming. In fact, it has distinct components that can be tackled in an orderly way. "I've always broken it down into process, technology and people," says John Medaska, vice president of business development at consulting firm Relational Technology Services, headquartered in Columbus, Ohio. It's all about scenario planning and practical preparedness, he emphasizes.

For example, when the fall 2004 hurricanes hit Florida, hospitals' ability to operate "depended to some extent on the power function and the facility function" at individual organizations. Plus, many key people had trouble getting to their hospital. Thus, Medaska says, two key elements for business continuity planning are building in remote operating capability for IT professionals and cross-training individuals across the breadth of the organization for functions they wouldn't normally handle but might have to in an emergency.

Another key point, says Pat McAnally, senior director, professional services marketing at SunGard Availability Services, Wayne, Pa., is rubber-meets-the-road prioritization. From the standpoint of business continuity, every IT application in a patient care organization simply cannot have equal standing, she says.

"We recommend that in prioritizing one's system, people understand and decide what the dependencies are and that they make sure that folks from both the medical side and from the IT side agree on what the strategy is to stay up and running." For example, if the decision is that the ICU can never experience downtime, then "CIOs and their team must architect their system to account for that," McAnally says. "And then accounts payable might be a slightly lower priority."

When robust DR/BC plans are not fully implemented, even minor problems can have major consequences. John Donovan, director of global healthcare marketing at data continuity vendor American Power Conversion (APC), West Kingston, R.I., cites the recent example of a New England hospital that "had a power outage that was extended to over three hours."

The data center support staff, located off-site, arrived at the facility within a half-hour of the outage, but the IT staff hadn't built in enough battery power backup. More than 100 servers had to be shut down, a lot of things crashed, and rebooting the hospital's picture archiving and communications system was difficult and prolonged. And this was a "simple" problem involving lack of battery and generator backup, Donovan says.

Six-pronged preparations

At 338-bed St. Vincent's Hospital, Birmingham, Ala., CIO Timothy Stettheimer, Ph.D., has a detailed DR/BC plan in place as well as a prioritized rationale for each step in keeping operations going during an interruption. Six components are built into the hospital's plan.

• Information overlap: location of mission-critical data in multiple databases (e.g., EMR and clinical data repository)
• Asset distribution: constant mirroring of mission-critical data between two data centers in different locations
• Offsite storage of tapes and other storage components
• Caching for remote (outside campus) storage
• Redundant connectivity for both wired and wireless infrastructures
• Power source/infrastructure protection: two data centers tied to separate power generators in different locations

It's not surprising that St. Vincent's has preparations in place, since it has had an EMR for more than 12 years. But the extent of its planning and the degree of its continuity preparedness are exceptional compared with the vast majority of facilities. That's because the idea of downtime has simply become unacceptable, with hospital operations and staff totally dependent on the EMR and related clinical applications, say St. Vincent's executives.

"It's about our patients," Stettheimer emphasizes. Their well-being and lives depend on the system. And physicians totally rely on computerized physician order entry and other systems to be working 24/7. It's clear that "the practice of medicine itself is inherently going to depend on these systems," he says.

The success of St. Vincent's planning was illustrated in real time last fall when the hurricanes that hit Florida unleashed monsoon-like storms across Alabama. The region endured extensive power outages, but there was no power loss at St. Vincent's.

Payer strategies

Some health insurance executives have been equally insistent on DR/BC planning. "I would say we're among the best prepared" of health plans "because it's been a constant focus for us," says Joe Fraser, the client delivery executive for Blue Cross Blue Shield of Massachusetts, Boston. Fraser and his colleagues are in charge of ensuring business continuity for the insurer as outsourced professionals affiliated with Plano, Texas-based EDS.

"Sept. 11, 2001, raised the consciousness of a lot of people, but in this environment, we've been conscious of it for a long time," Fraser says. "We have call trees, we run periodic disaster drills, and we have processes in place," including remote mirroring to a second site in Rockland, Mass., 20 minutes outside Boston. Protecting plan member eligibility and other information is absolutely crucial, so that data has top priority in DR/BC planning.

Ramon Chen, senior director of product marketing at GoldenGate Software, says that transactional data management, with its built-in, continuous, 24/7 synchronization of critical data, is the next advance. It will be essential for healthcare organizations, awash in clinical data, to be able to continuously move transactional data from one database to another, he says.

Blast Sasser

Unfortunately, protection against deliberate sabotage is becoming a prominent aspect of DR/BC planning. Worms, viruses and other attacks have been taking down systems in healthcare, as in all industries, and the degree of possible harm is immense. A key strategy is proactivity, says Oliver Schmelzle, a senior production manager at Austin, Texas-based WholeSecurity Inc.

Worms such as Blaster and Sasser have "hopped inside internal networks" in the past couple of years, Schmelzle says. IT managers must understand that "reactive, signature-based systems are not effective against these attacks and that it takes several hours to create new signatures" to disable the worms. IT staff "need to obtain a copy of the worm, analyze it and push it out to others. And in the case of Blaster, your network could already be down before you could push this out," he says.

J.T. Keating, WholeSecurity's vice president of marketing, says that one aspect of DR/BC planning is to have both "an established plan for patch management, such as the patches Microsoft creates, as well as SSL-based VPNs." These virtual private networks based on secure sockets layer technology allow "anyone with a computer to help" if an organization's entire system goes down, Keating says.

Preparations pick up

Fortunately, preparedness in healthcare is changing rapidly, industry observers say, largely because of the surge in clinical IT development. Tony Asaro, a senior analyst with the Enterprise Strategy Group in Milford, Mass., follows DR/BC vendors across industries. He sees hospitals, medical groups and health insurers quickly catching up to industries like financial services that have long had robust DR/BC plans in place.

Few organizations, in healthcare or in other industries, have the optimal degree of remote mirroring, Asaro says. But, he adds, "of the healthcare organizations I have spoken to, it is on their list to implement a more comprehensive disaster recovery plan using remote mirroring in the next 12 to 24 months."

Mark Hagland is a contributing writer based in Chicago.

[back to In The News] | top of page