I apologise for the problems with the San-Franciso servers today. The problem has now been fixed and the servers should be functioning normally again.
The following is an overview of events leading to the outage and subsequent restoration.
- On August 24, 2003 at approximately 1:30 PM PG&E experienced a power failure on the lines servicing the San Franciso hosting facilitiy
- At this time the facility went on UPS Power and then subsequently generator backup without incident.
- This backup continued to run until August 25, 2003 at 2:47 AM when a component in the generator itself failed. The component that failed is known as a CT and monitors power within the generator. Due to a manufacturers design problem, the CT was not physically restrained to prevent possible movement within the generator.
- At some point after running for 14 plus hours the CT physically spun inside the generator causing a physical short between two of the electrical phases, which in-turn caused several of the control boards to fail. With built in safeties, the generator shut itself down.
- The Building Engineering staff was notified that the generator failed and they started repair efforts. The generator technicians were called and dispatched to the facility.
- At 7:30 AM PG&E dispatched a truck to reconnect the facility to the power grid.
- At 8:03 AM PG&E reconnected the power grid and power was restored.