Thinking the unthinkable± some lessons from september 11 by Paul Bodart

Information Security Identification: Confidential

The Bank of New York Mellon and the Impact of

September 11, 2001

Presented by Paul Bodart

March 22, 2011

Information Security Identification: Confidential 2

Agenda

I. Impact of WTC Attacks on New York City

II. Implementing Disaster Recovery

III. Steps taken

IV. Where are we now

V. Lessons learned


Section I.

Impact of WTC Attacks on New York City


Impact of WTC Attacks on New York City Estimated loss of life = over 5,000 people

Estimated total property losses = $60 billion, 15 million square feet

More than 50,000 employees displaced

Implosion of the towers knocked out more than 400 megawatts of

electricity to lower Manhattan (enough to power 400,000 homes)

Total amount of time needed for clean-up = approx. 1 year

Estimated cost of clean-up = $40-$60 billion

Estimated cost to replace the technology infrastructure, which was

obliterated inside the towers and adjacent buildings = approx. $3.2

billion

NYSE closed for almost 4 consecutive days, the longest period

since the stock market crash of 1929

Sources: The Daily News and The New York Times


All of Lower Manhattan Affected


Aerial view of lower Manhattan

after attacks

6


One Wall Street Corporate Headquarters, Custody Staff, Broker Dealer

Services staff, Relationship Management

32 Old Slip Global Foreign Exchange, Securities Lending and

Treasury Operations

101 Barclay Street Stock Transfer, Funds Transfer systems, Corporate Trust,

Broker/Dealer Services systems, Unit Investment Trusts

and Depositary Receipts

(One of BNYM’s two Primary Data Centres, along with Teaneck, NJ)

100 Church Street Mutual Fund Custody Administration and Fund

Accounting

Business Primary Locations in Lower Manhattan

for The Bank of New York


Prior to September 11th, BNY’s US-based processing and recovery centers were concentrated in the NY/NJ area

8

Two primary production facilities

101 Barclay

Teaneck

Three recovery centers

Maywood, One Wall Street and Comdisco

Verizon connected all centers through a campus-type network

Customers and external agencies connected through 101B, TNK and OWS

Teaneck

Primary Production

101 Barclay

Primary Production

One Wall Street

Recovery Center

Maywood, NJ

Recovery Center

Comdisco/N. Bergen, NJ

Recovery Center


Section I.

Implementing Disaster Recovery


Implementing Disaster Recovery

10

What Happened to BNY on September 11?

Evacuation of 8,300 BNY staff, including:

1 Wall Street -- Broker Dealer Services Operations Center

101 Barclay -- Broker Dealer Services Data Center

Corporate Headquarters moved from One Wall Street to 1290 Sixth

Avenue

Contingency plans immediately implemented

101 Barclay Street and 100 Church Street most impacted due to

proximity to World Trade Centre


Primary and Backup Business Locations (Staff)

Primary

Business Site

Back-up

Business Site

Broker/Dealer Services One Wall Street Maywood, NJ

Custody One Wall Street Pleasantville, NY

Mutual Fund Custody & Acctg 100 Church Street Pleasantville, NY

Funds Transfer Utica Utica (no relocation necessary)

Depositary Receipts 101 Barclay Street West Paterson, NJ

Corporate Trust 101 Barclay Street West Paterson, NJ

Stock Transfer 101 Barclay Street West Paterson, NJ

Securities Lending 32 Old Slip Greenwich, CT

Trading 32 Old Slip Greenwich, CT

Treasury 32 Old Slip Greenwich, CT

Foreign Exchange 32 Old Slip Greenwich, CT

Additional warm sites for operations/business staff were at West Paterson, NJ; Teaneck, NJ; Lodi, NJ and Secaucus, NJ.


After September 11th, and the collapse of WTC 1, 2 and 7, 101 Barclay was unusable

12

Processing from Barclay was distributed to recovery centers and Teaneck

Maywood, One Wall Street, and Comdisco/SunGard (vendor)

Rolling telecommunications outages delayed recovery

Personnel had difficulty reaching recovery centers

BNY was at risk until new recovery centers were established

Teaneck

Primary Production

101 Barclay

Primary Production

One Wall Street

Recovery Center

Maywood, NJ

Recovery Center

Comdisco/N. Bergen, NJ

Recovery Center

X


Telecommunications Infrastructure Damage

Loss of switching stations crippled service to downtown Manhattan. As a result:

Client connectivity with several critical Bank’s systems like Government

Securities Clearing System (GSCS) and collateral management system

(RepoEdge) interrupted

Communications lines between the Bank and key providers (like Federal

Reserve or DTC) interrupted

Connectivity among different Bank’s systems (located in different

locations) interrupted

Voice communications in Metropolitan area interrupted


Client/Broker Dealers The Bank of New York Providers

Morgan Stanley

Downtown Manhattan

Goldman Sachs

Downtown Manhattan

Lehman Brothers

Downtown Manhattan

US Government Securities Clearing

Operations

Downtown

Manhattan

Systems

Downtown

Manhattan

US Money Transfer

Operations

Utica (NY)

Systems

Downtown

Manhattan

Federal Reserve

Bank of New York

(“The Fed”)

Downtown

Manhattan

Depositary Trust

and

Clearance Cy

(“DTCC”)

New Jersey

An Example The Bank of New York US Government Clearing Services

Situation before September 11.



Morgan Stanley

Downtown Manhattan

Goldman Sachs

Downtown Manhattan

Lehman Brothers

Downtown Manhattan


Operations

Downtown

Manhattan

Systems

Downtown

Manhattan

US Money Transfer

Operations

Utica (NY)

Systems

Downtown

Manhattan

Federal Reserve

Bank of New York

(“The Fed”)

Downtown

Manhattan

Depositary Trust

and

Clearance Cy

(“DTCC”)

New Jersey

An Example

Have migrated to a Back-up Center

On or after September 11, a majority of the players located in

Down town Manhattan migrated to back-up Centers.

Operational Back-up are not always in the same building as System back-up



Morgan Stanley

Downtown Manhattan

Goldman Sachs

Downtown Manhattan

Lehman Brothers

Downtown Manhattan


Operations

Downtown

Manhattan

Systems

Downtown

Manhattan

US Money Transfer

Operations

Utica (NY)

Systems

Downtown

Manhattan

Federal Reserve

Bank of New York

(“The Fed”)

Downtown

Manhattan

Depositary Trust

and

Clearance Cy

(“DTCC”)

New Jersey

An Example

Have migrated to a Back-up Center Communication Lines impacted

On the telecommunications front, two things happened :

1. Loss of switching stations crippled services to/from Downtown Manhattan

2. Communications between Back-up Centers was not sufficiently robust or tested.


Section III.

Steps taken


Steps Taken

18

1. Operations Centers Relocation from One Wall Street

Operations personnel began establishing operations in Back-up Centers, NJ 9-11

Immediately began client contact 9-11

Facility expansion initiated 9-12 - 150 workstations operational by 9-13

2. Computer Centers Relocation

Personnel relocated 9-11

Systems successfully switched to back-up for processing

All data integrity maintained - 100% capture of all transactions

3. Restoration of Communications Lines Between Banks and Providers

Lines monitored on continuous basis

Replaced ISDN with dedicated dual high speed lines


Steps Taken

19

4. Restoration of Client Connectivity

Switched to client back-up communications where available

Worked with clients to establish back-up communications where necessary

Implemented manual solutions for clients where no communications back-up was available

Clients relocated to BNY back-up site

Established special disaster recovery e-mail location for bulk file to be received for transaction processing

Established e-mail delivery process for extracts and client reports

E-mail fax service established

5. Re-establishing Voice Communications with Clients/Business Partners


Section IV.

Where are we today


Where are we today

21

Two of the four buildings in downtown Manhattan were re-occupied within two

weeks of September 11.

The two other buildings that were closed to WTC have started to be reoccupied and will be fully occupied during the summer.

Interim measures:

Leased 100.000 square meters in 9 new Manhattan locations;

most staff were moved to these 9 buildings by the end of Q3 2001

Data Centres

Established a new primary data center in Sterling Forest

moved out of contingency sites by November 30.

Established geographical diversity :

400 staff permanently relocated to New Jersey

accelerating occupancy of existing “growth center” in Orkado and Syracuse

The organization has been battle-tested and emerged stronger


Section V.

Lessons learned


Lessons learned

23

Be flexible. The disaster planned for is not the disaster that occurs.

Planning is critical. Have a strong plan, and practice it.

Telecommunications is a resource which is not entirely under your control.

Route diversity planning via multiple central offices may be undone by the

plans or actions of your service provider. Know the complete end-to-end

routes of your mission critical circuits.

At the precise time that accurate and timely communication with clients is

most essential, it becomes the most difficult.

Geographic diversity of operating units enhances recovery. The most

successful relocations occurred when there was a geographically separate

group with the same functions as the group being moved, permitting

uninterrupted processing.


Lessons learned

24

Systems integration is a changing landscape. Every critical system has

supporting applications without which the base system cannot function. The

support systems change over time, and periodic reevaluation is important.

Right size back up capabilities. Recovery may involve processing volumes

that exceed normal. When in doubt, err on the side of too much backup

capacity.

Processing the day’s work is the key objective, but do not overlook

accounting, financial, and proof and control systems. Getting the work

accomplished is important, but is only part of the job – you still have to be

able to reconcile internally and with external systems and vendors.


Lessons learned

25

Individual client CPU links become a liability in a disaster recovery scenario,

as each must be individually recovered. In a major disaster, clients may also

be operating out of contingency sites, which further complicates recovery. It

is far easier to restore a single link to an industry utility such as SWIFT than to

50 or 100 clients.

Going forward, we all need to create a state-of-the-art data centre,

telecommunications, and business environment with full redundancy, real-

time backup, and adequate dispersal of staff and systems to ensure

continued operation through any disaster event, regardless of severity or

breadth.


Site Selection Considerations

26

Geographic diversity. Adequate separation to insure survivability, close

enough to facilitate communications.

Exposure to terrorist attack.

Exposure to natural disasters - flood, hurricane, tornado, volcanic activity.


Cost

27

The price of safety and assurance of business continuation is going to rise.

Fully redundant data centres with multiple dedicated links from multiple

processing centres to multiple client sites and utilities will add significant cost.

These cost will have to be recovered by all of us – which means higher prices

in all industries, especially those heavily dependent upon data processing,

such as the financial services industry.


For more information

28

Paul BODART : Executive Vice President

Head of EMEA Global Operations for Asset Servicing

Phone : (32-2) 545.82.37

E-mail : [email protected]


Documents

Thinking the unthinkable± some lessons from september 11 by Paul Bodart