Worst-case scenarios

As the banking world relies more heavily on technology to deliver round-the-clock services, so business interruption risks increase. business continuity (BC) and disaster recovery (DR) are increasingly important aspects of the banking world’s business plans, and embrace a wide range of banking applications, including ATMs. The challenge is to weigh the financial and PR costs of maintaining continuity against the costs of losing it.

The banking sector is now susceptible to a number of BC problems, ranging from risks associated with IT upgrades to more serious problems resulting from major disasters such as hurricanes and terrorist attacks. And, as events in the US and parts of Europe proved last year, power failures can have a severe impact on an organisation’s ability to perform. Finally, security failures – particularly in a self-service environment – can bring a network or channel down, just like a natural disaster.

Including ATMs

While such threats may require significant effort at the back-office and head office level, some banks are also looking at how self-service customer systems, such as ATMs, can be included in their DR plans. How important ATM recovery is depends on the importance of this channel as a means of getting cash. In countries such as the UK and US, this is a critical part of business.

A BC plan requires a diversity of skills if risks are to be handled correctly. Technical solutions are a key element of any risk management process, and in this context, data back-up at a second site is critical. However, as John Hickman, business continuity manager, at Hitachi Data Systems, EMEA, says: “As a key part of the ATM’s security is driven by credit scoring and other behavioural systems, it’s essential that credit-worthiness validation systems are backed up and are as up-to-date as possible so as to avoid fraud.”

Keeping back-up data at a separate location may also make good sense, especially if there is a risk that it could take days, weeks or even months to move back into the bank’s permanent site.

The Emirates Bank International (EBI) uses GoldenGate’s Global Data Synchronization platform for its BC activity. This system is designed to capture, transform and deliver data continuously from EBI’s primary BASE24/HP NonStop systems to its secondary back-up system. Transactions are replicated immediately after entry on the primary servers, so up-to-the-second critical data is maintained on the back-up site to ensure against data loss in case of disaster.

Uninterrupted service

When EBI upgraded the hardware and operating systems in its primary NonStop site, the company used GoldenGate to manage the system migration. This enabled EBI to load the secondary system while BASE24 remained active on the old system. The result was that ATM and point-of-sale (POS) operations continued uninterrupted as the upgrade occurred. When the migration activity was complete, the backlog was applied to the migrated system and brought into synchronisation.

Technical developments are also helping data transition. For example, Dense Wavelength Division Multiplexing (DWDM) optical networking can make the transfer of data even more seamless and affordable and so reduce disaster recovery times considerably.

Banks can also learn from the casino industry. According to Safwan Shah, president of Infonox: “The ATM transaction volume in this industry is roughly three times the average monthly volume seen on bank ATMs.” Solutions for this sector are, therefore, designed to provide a robust response to BC.

Technical solutions represent only part of the answer to BC problems. Success is more likely if a banking organisation takes a holistic approach to the problem. In other words, all employees, technical personnel, suppliers and management should be trained to understand their role, should a disaster occur. Such training needs to take into account issues such as the variety of risks likely to occur as well as the different business effects that may be experienced.

The important thing is to ensure there is continuity in maintaining revenue as you deal with the disaster. For example, a major financial organisation was severely damaged by a terrorist bomb in the 1990s, but the network infrastructure remained largely undamaged and enabled the bank to move a lot of staff to another location and transfer data very quickly. Also, the effect on its branch network was minimised because its computing function was distributed in two locations outside the main disaster area.

Two options for BC

BC at ATMs may be delivered via either a standby system or a dual live site. “In standby environments, there is a replacement back-office system at an alternative site to the main live site that can take over ATM operations,” explains Andy Brown, senior product manager at ACI Worldwide. There are several different levels of standby:

• cold standby – the second system has to be loaded, started from scratch and will not have the latest data;

• warm standby – the system is in some state of readiness but may have been last updated on the previous night. It may need to be updated before it can be made operational;

• hot standby – the second system is operationally ready and the data from the live system is being fed on a regular basis so it is being kept up to date with changes (for instance, to balance files or security keys).

A dual live site, meanwhile, enables two sites to share the workload and keep each other informed of data changes. As such, a failure at one site will not interrupt operations, because the second site is already operational and can handle transactions instantly.

BC may be provided either in-house or through an outsourced solution. Organisations taking the in-house approach may set up complementary or duplicate systems in other offices so that if a disaster occurs in one location, the operation is managed from a sister site. Staff can be temporarily relocated or agency employees used in the alternative location on a temporary basis.

Organisations taking the outsourced approach may set up a contingency model with a professional DR company. They replicate systems and processes and effectively keep the DR company on permanent standby in the event of a disaster. The advantage of in-house solutions is that a DR plan is cheaper and usually easier to prepare and maintain, especially if the same process, operation and systems are used in multiple locations. Ultimately, if nothing ever happens, DR costs are proportionally lower. However, internal resources have to be applied to a DR plan and the plan must be consistently reviewed, maintained and also simulated on a frequent basis, as business changes so fast and processes rarely stay constant for even six months. Added to this, a permanent and significant DR fund must be available or insured to cover the cost of a disaster.

Going the in-house route

Bank of America has adopted an in-house approach to BC. Its primary criterion for its disaster recovery system was to have a standardised solution that would eliminate the need for dedicated, internal support personnel. It also wanted to migrate from the HP NonStop Himalaya K-Series to the new S-Series. The initial migration had to take place without system degradation or downtime. The bank adopted GoldenGate’s Data Synchronization platform, which enabled it to keep its ATM network up and running during the upgrade.

“We chose a phased approach to the migration and also decided to install GoldenGate ourselves,” says Michele Schwappach, vice-president and senior systems manager for ATM/debit applications at BoA. Today, the platform is a standardised part of the bank’s internal disaster recovery solution.

Whether an ATM BC system is operated in-house or outsourced, it is essential that it can meet the latest challenges. The success or failure of a bank to cope in a time of crisis could be critical to its ongoing success. And, with the focus of the regulatory bodies on risk management, most banks are thinking carefully about how they handle disasters in all areas of their business – including ATMs.

Asking the right questions

When implementing a disaster recovery solution that meets the needs of your ATM system properly, you need to consider the following questions:

• What is the cost of implementing a DR solution relative to the cost of suffering losses associated with a disaster?

• What is the cost (per hour/per day etc) to your business in the event of a total shutdown?

• Can you forecast this cost both in terms of customer impact/cost and also the operational cost to your business to restore operations?

• How often do you review your DR plans?

• When did you last simulate your DR operation?

• How often do you practise a disaster with your internal teams?

• How will you communicate contingency plans within your business?

• How will you communicate contingency plans to your customers?

• Do you have a communications/marketing/PR strategy plan to manage a positive result?