When Hurricane Sandy plowed into the U.S. East Coast in 2012, its effects were felt up and down much of the seaboard, perhaps hitting New Jersey the hardest. Even now, the impact of the damage can be seen, and it might be years before the area is truly up to speed again.

Residents bore the brunt of the high winds and tidal surges, enduring days and even weeks of gas shortages, empty food shelves and power outages. And that wasn’t the worst of it— homes and other buildings were destroyed by the high waters and ferocious winds, forcing many into makeshift evacuation centers.

Businesses were also impacted, with lost power being the most obvious consequence. International Fidelity Insurance Company (IFIC), based in Newark, N.J., was among the many businesses to lose electricity, leaving the company reliant on a co-location center. Thankfully, however, it had a sound disaster recovery (DR) plan in place, which included DR software from Maxava.

As a result, even a storm as wicked and widespread as Sandy couldn’t keep IFIC down. Working around the clock with the assistance of Maxava, it continued to operate despite darkened offices and silent hallways.

A Simple View

​IFIC is one of the biggest independent and privately owned surety companies in its industry. Barbara Kissner, IFIC CIO and senior vice president, puts surety in context by saying that it’s essentially a three-way agreement. She explains: “For example, let’s say you own a large piece of property on which you want to build a shopping center. Per the usual, you would get a contractor who would hire subcontractors. You want to ensure, however, that the contractor is going to be able to deliver the job and pay those subcontractors because, as sometimes happens, the contractor might go bankrupt or run into problems, leaving you with a hole in the ground.​

“To ensure this doesn’t happen, you would ask the contractor to get a bond, which the surety would underwrite based on the contractor’s financial credibility, past work experience and any other germane factors. If the contractor does run into problems, we would then have several options, such as honoring the amount of the bond or hiring another contractor to finish the job. That is a simple view of surety.”​

It extends beyond contractors and job sites, however, to commercial surety bonds. These are specific to other transactions, such as estate trustees, who, in the state of New York, must get surety bonds. The same also applies to notaries, who often get surety bonds. In addition to offering these types of bonds, IFIC also offers bail bonds and is the largest private writer of bail bonds in the U.S., according to Kissner.​

The company is registered in all 50 states, Guam and Puerto Rico, and has some 250 employees, about two-thirds of whom work in IFIC’s regional offices. Supporting them is a production IBM Power Systems* 520 Express* server, which runs a general ledger package from DAPREX Inc. and a number of surety-specific customized applications, and an older IBM System i5* 520 running at its DR hot site.​

Without that off-site 520 and the Maxava HA Enterprise solution, which was suggested to IFIC by United Computer Group, the company’s operations could have come to a screeching halt when Sandy hit in October 2012. “We work in all U.S. time zones, so even a temporary outage would cost us money,” Kissner says. “That’s totally unacceptable to everyone, from upper management to field employees.”

Necessary Calls

​When the storm started battering the shore with driving rain and high winds, IFIC began taking precautions; beginning on Sunday when Kissner and her network manager, afraid they might lose power, started remote testing the server to ensure it was still operational. IFIC’s Newark office closed on Monday due to the weather, and staff continued testing until around 8:30 p.m., when the lights finally went out.

“I was going downstairs at my house to make popcorn when I received the message I had been dreading on my phone: Newark was without energy. Having seen the city lose power before, I knew this wasn’t going to be a small problem, so I called our four key disaster recovery staff members and declared a disaster at a quarter to nine or so. From there, we just went into complete failover mode,” Kissner recalls.

Thankfully, the company had a business-continuity email system, which allows users to switch from local access to cloud-based Web access. That was the easy part. The somewhat more difficult aspect had to do with the Power Systems server—and not simply because it was now dependent on an uninterruptable power supply system. Rather, it had to do with the widespread outage.

As Kissner explains, “I always looked at a disaster as a building-centric event, such as a localized fire or flood. One thing I didn’t consider is that my IT staff would also be without power. I had it, but they didn’t. Because of this, I had people sitting in their cars with their phones plugged into a power source so we could begin making the necessary calls.”

One of those calls was to a Maxava technician IFIC worked with during implementation and testing. “He was an absolute angel,” Kissner adds. Because her employees didn’t have power or access to the system, they gave the technician the password and he began the failover process, which took between two to three hours to complete.

This rapid rollover was due in part to the Maxava software, which replicated IFIC’s everyday processing in real time to the backup box. If the company had relied on tape-only backups, as many other companies do, this process would have been much more arduous, with someone having to pick up the tapes, take them to the co-location site and repopulate the server from scratch. Given the gas shortages during and after Sandy, this would have been no easy feat.

​As the backup system was coming online, Kissner called every IFIC office to tell them the company was in disaster mode and services would likely be interrupted. She also informed them that, because the backup system was older, response times wouldn’t be as quick.​

On Thursday, Kissner and her core staff returned to the office despite the continued power outage. It wasn’t until Friday that power returned to a steady state. At that time, the company—again using Maxava HA Enterprise— reverted back to its production system at the Newark office. But it had been a harrowing few days, with Kissner noting that she and her staff probably had only five hours of sleep over the course of those four days.​

However, that was a small price to pay, Kissner says, estimating that IFIC would have lost three full days of work if the failover hadn’t worked, which would have equated to around $2 million in revenue. That the company continued to operate in the face of the storm wasn’t overlooked, as Kissner explains.​

“We were getting notes from people around the company telling us how pleased they were that in the face of this enormous storm, that all of our systems continued to run. I personally couldn’t believe it either. It was the first large-scale disaster I had lived through where everything had failed and nobody had any creature comforts. But there we were, still operating. It was quite remarkable,” she says.

Good Faith

​Although things ran rather smoothly during this disaster, Kissner notes she learned several lessons along the way. One was ensuring that the company had not only a redundant backup box, but also redundant communication lines, which it didn’t at the time.

​She also discovered an auto-dialer that could broadcast messages to IFIC’s remote offices to indicate potential problems with the production system. This negates her having to make the 200 or so phone calls she had to make during the Sandy disaster.

​And because so many members of her IT staff had lost power and had to rely on car power sources to make phone calls, the company also put together emergency kits for home use that include batteries and water. “If there’s another disaster, they at least have two or three days of batteries they can use,” Kissner says.

​Perhaps more importantly, however, the company also learned the value of having a dedicated frontline IT staff that sacrificed hours of sleep to keep things up and running. “Sure, it’s ultimately about dollars and cents,” Kissner says, “but there’s also this less tangible, but no less important, aspect of good faith. My team took such pride in themselves for keeping the company—despite all of the personal hardships they were experiencing—up and running.”

​Kissner also had a similar feeling of good faith regarding Maxava. Without that replicating software and the support IFIC received during and after the storm, things could have gone horribly awry. “Their tech support was absolutely top-notch, and they were on the phone with us night and day, even at 2 or 3 in the morning. They were absolutely terrific.”