There's A Fire In The Hold

Yesterday, I was working, at 9am, there was a routine UPS service taking place .. soon, there is a funny smell from the computer room, I figure he is soldering something .. I have no fear ..

About 45 mins later, he tells me there is a burning smell coming from the electric box on the wall where the mains power enters the computer room. He thinks that the 3 phase supply has overloaded and is about to catch fire, we have to get an electrician right now and we have to power down the load (which means every single computer system in DAS) .. now there is fear .. and there are no principles .. I am on my own and this is something I have no wish to deal with - various senarios run through my head as I try to gather my thoughts and think what to do.

From here on in, I ride the entire day on stress and adrenalin .. I am out of control and I have the situation under complete control .. I am full of fear and I do not give a thought at any point to the principles.

There is maybe £3millon of business critical equipment in the computer room and I do not wish to be solely responsible for this and I must be, as there is no other representative from IT on site.

The UPS guy explains that there are three 10mm cables in the three phase supply and that the first phase pulls 16amps under normal load .. when he bypasses the UPS, it is pulling 50amps and there must be a fault because the cable should be up to 70amp tolerant and the circuit breaker up to 100amps .. he thinks a connection must be loose and is probably arcing .. great !

If the alarms go off, I am leaving the building and I am not passing GO to collect my £200 !

Honeywells are on site doing fan coil maintenance .. they are in the computer room talking to the UPS guy, between them they take the cover off of the electrical box and the problem becomes immediately apparent. The main circuit breaker has overloaded and is badly scorched, the first phase is melted about 5cm down the cable, all three cables are still extremely hot.

I am given the go ahead to shutdown the systems, methodically I start to do so .. eventually someone comes in to help, he stays less than half an hour and then is gone, I am on my own again.

Once everything is powered down, they cut mains power (I go to another floor while they do this), replace the circuit breaker and restore mains power to the room.

B has come in now at my request to help me bring all the systems back up, I feel a bit better now he is here. Between us we bring all the systems back into normal operation and let the users know .. I finally stand down from high alert and relax a bit .. I still manage to get away by the end of my shift.

I go home, I am physically and emotionally exhausted and I go virtually straight to bed for a nap, which turns into 2 hours ! I get up, clean my teeth and go back until the morning, when my alarm wakes me for the next shift !

This time, nothing catches fire ..