Message-ID: <4507567.1075845442746.JavaMail.evans@thyme>
Date: Mon, 11 Jun 2001 02:17:00 -0700 (PDT)
From: philippe.bibi@enron.com
To: mark.frevert@enron.com, greg.whalley@enron.com
Subject: FW: Enron Center North/Enron Center South Damage & Restoration
 Overview
Cc: mike.mcconnell@enron.com, jeffrey.shankman@enron.com, 
	jeffrey.mcmahon@enron.com, raymond.bowen@enron.com, 
	john.lavorato@enron.com, louise.kitchen@enron.com, 
	rebecca.mcdonald@enron.com, kevin.hannon@enron.com, 
	ken.rice@enron.com
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Bcc: mike.mcconnell@enron.com, jeffrey.shankman@enron.com, 
	jeffrey.mcmahon@enron.com, raymond.bowen@enron.com, 
	john.lavorato@enron.com, louise.kitchen@enron.com, 
	rebecca.mcdonald@enron.com, kevin.hannon@enron.com, 
	ken.rice@enron.com
X-From: Philippe A Bibi
X-To: Mark Frevert, Greg Whalley
X-cc: Mike McConnell, Jeffrey A Shankman, Jeffrey McMahon, Raymond Bowen, John J Lavorato, Louise Kitchen, Rebecca McDonald, Kevin Hannon, Ken Rice
X-bcc: 
X-Folder: \Greg_Whalley_Oct2001\Notes Folders\All documents
X-Origin: WHALLEY-G
X-FileName: gwhalley.nsf

Mark/Greg,

Please find attached a post-mortem of our weekend problems...

Summary is last paragraph in red.

Regards


 -----Original Message-----
From:  Terech, Henry  
Sent: Sunday, June 10, 2001 11:23 PM
To: Bibi, Philippe A.; Donovan, Bill
Cc: Kean, Steven
Subject: Enron Center North/Enron Center South Damage & Restoration Overview

Bill/Philippe-

Below is a summary of the major problems which occurred the past 96 hours and 
the steps taken to mitigate the damage and risk.

Enron Center North (1400 Smith)

As a result of the flooding downtown, approximately six skyscrapers had their 
electrical vaults flooded and began to cause cascading problemd in the 
electrical grid downtown. The net result was the loss of one of the two 35kv 
circuits which serve 1400 Smith. We (engineers, EES Maint, electrical subs)  
believe at the time the circuits blew we shifted from three phase to single 
phase power, which caused an imbalance in the load to the building. 

Consequently, UPS "C" which serves a portion of the Lv 34 Data Ctr, the Rolm 
phone switch, and a protion of the Traders floors lost a circuitboard, and 
blew two internal fuses. This power degradation caused PDU "S" on Lv 34 to 
trip its main breaker and shutdown power load to servers and equipment on 
this PDU. This occurred shortly after Noon on Saturday.

Response/Restoration

> UPS "D" transferred load from UPS "C" as the N+1 back up and carried the 
full load of the UPS loss as designed
> EPSC/electrical sub (Henderson Electric); EES Maint; IT worked together to 
power down systems to begin repair work by 4:00pm Saturday
> EPSC/EES Maint coordinated with Pillar/Henderson Elec to obtain replacement 
parts and begin the repairs. By 9:30pm all repairs to UPS"C" had been made 
and the system bought back on line. As a measure of extra protection the back 
up genset for Enron Ctr North  was in operation
> Reliant Energy restored the circuit at approximately 10:30pm Saturday 
evening
> EPSC/EES Maint and subs conducted a thorough review of all electrical and 
back up systems Saturday evening
> Enron Network IT Operations began restoration of all affected systems 
Saturday evening
> EPSC/EES Maint obtained diesel fuel and topped off all gensets Saturday in 
case of additional problems and due to fuel consumption during the outage

Other Building damage was limited to flooding in the tunnels, and water 
damage/flooding to portions of the Body Shop on Lv B-1

Enron Center South (1500 Louisiana)

Impact/Damage Summary

> Levels 8 & 9  received very limited amounts of water from the core area 
elevator shafts and exterior curtainwall. There was no exposure to water on 
any systems or network cabling during the entire period. (Response) Crews 
contained and cleaned all water as it traced into the building.
> ALL IDF closets on Lvs 3-5 Traders Floors remained dry throughout the 
duration of the rains/flooding on Friday & Saturday
> Lv 6 received some water from Lv 7. (Response) The area was contained and 
cleaned. No additional damage beyond Tuesday evenings flooding occurred
> Due to extreme street flooding and water pressure, a 10" sanitary sewer 
line tied into ECS burst on Friday evening causing extensive flooding of Lv  
B-1 area (approximately 4-6 inches of water in the basement. (Response)  
Clark & Way Engr was able to install a temporary plug in the line to stop the 
on going flood waters from penetrating the basement. Clean up has continued 
throughout the weekend   
> Due to the extreme flooding and water pressure, the water seal of the 
Reliant Vault burst early Saturday and flooded the electrical vault room with 
24-30" of water. This resulted in a complete loss of power at approximately 
1:30pm on Staurday afternoon. The back up systems engaged and operated per 
design to support the Data Ctr and Traders Floors. The temporary cooling 
tower went off line. Prior to the flooding Clark had been in the process of 
tieing in the temporary feed into the back up system by 7-1-01 in prep for 
the migration (Response) 
 * Clark used back up water pumps to pump out and clean the Reliant 
electrical vault, as well as maintained operation of the back up systems
 * The Data Ctr AHU's were kept on line with the dehumidifiers/blowers to 
modulate the airflow in the Data Ctr. EPSC had Way Engr techs use 
temp/humidity probes
    to monitor the areas. The recordings indicated temps on Lvs 8 & 9 did not 
exceed 78 degrees and relative humidity did not exceed 68% during the outage
 * Clark/Fisk Elec/EPSC coordinated desing efforts and obtained gear on an 
emergency basis and completed the tie in off the temporary coolong tower to 
the 
    generators by 10:30pm
 * Reliant Energy is still cleaning and replacing gear in th vault room. 
Presntly (AT 11:00PM Sunday) we have one circuit operational in the building 
and 50% of the
              Reliant  gear on line). We are still operating all systems 
safely on the back up sytems for the past 36 hours. It is expected Relaint 
will be complete by 7:00am.
   Clark; the engineers and MEP subs shall be conducting a thorough system 
review and repirs as required of all systems throughout the building during 
the coming
   week.

In summary, the back up systems in both facilities did engage and operate as 
designed; however, due to the extreme conditons and stress placed upon the 
systems the facility held up extremely well. I would also like to note 
throughout the ordeal beginning Friday evening through Sunday, everyone 
(Enron NetWorks;EES FacilitiesClark Const;Way Engring;Fisk Electric;Henderson 
Electric; KW-Pillar;Hines) did an outstanding job under enourmous pressure 
and the most severe conditions I have seen in downtown Houston. 
Communication, around the clock work  and cool heads managed to assess damage 
and develop action/restoration plans to facilitate repairs for Monday business

Let me know if you need further details on specific issues or items.

regards-

Henry