What a day

[John Meissner]

We lost external power to our data center today... the ups's picked up but the generators didn't start... we were told to shut EVERYTHING down ... what fun.... after about 10 out of 100 servers were down they got the generator running. So we get to power our systems back up now. What an exciting friday....

[Umapathy S]

John,
I think, better than consuming the weekend. Always optimistic :).

Wish you a happy weekend.

cheers
Umapathy

ps: spend some time with your family. I dont think, it will be solved if you raise the issue with Dan :)).

[Paula J Frazer-Campbell]

Cheer Up

The weekends here


LOL
Paula

[Marco Santerre]

It always has to happen on a Friday, huh? Usually it's at 3pm though :-)

So your Friday will go by quickly that's all.

[John Collier]

John,

Keep your chin up. From the sounds of it you are going to survive this.

It just gives you more reason to party this weekend!!

Don???t waste the excuse??? ;-)

[Pete Randall]

John,

A month or so ago, we lost A/C in the computer room at 6:00PM on Friday. Spent Friday night shutting things down, Saturday morning acquiring and setting up temporary, portable A/C, Saturday afternoon bringing everything back up, and the rest of the weekend doing periodic checks to make sure things were still "cool".

I can sympathise.


Pete

[Dave La Mar]

Just went through similar in last two months.
Power goes out, UPS takes over, generator fails to start.

Power goes out, comes back on, spikes UPS and literally blows UPS.

Operate with no UPS for two weeks until fixed.

All equipment, Mainframe, HP-UX, NT servers, PBX, etc., etc., go down.

Thank God I have a good SA who handled the HP-UX side (XP512 etc.) so I could handle the mainframe side for our inexperienced operators.

Ah, life's experiences ....
that seem to arise on a Friday.

Enjoy the weekend and don't look back.

No points please, this is just an empathy note.

Best regards,

dl

[Michael Steele_2]

Remember once when someone forgot to check the gas tanks in both backup generators in the central office. In California in the middle of Silicon Valley on a nice warm day when everyone needed A/C. Most embarrassing.

[curt larson_1]

Looks like you've had a very busy day. unfortunately, I'm sure many of our more experienced members can tell you of interface cards being blown by oops, i mean, ups problems.

being your in a data center, you probably have diagnostics running on every machine and ito or similar product checking for hardware failures.

But, I'd still check my insurance policy before leaving for the weekend, ie.
all my mirrors are still mirroring, pvlinks haven't switched, back up networks are still operating and in standby mode. no nodes have switched over. and the tape drives for backups can be read and written to.

[Steven E. Protter]

John,

I hope your future is smoother without such fun.

We have an advantage over some.

We are connected to a City plant that provides air conditioning via a pipe system installed downtown. The system is on three power grids and has its own backup power.

Our issue is keeping the computer room WARM ENOUGH so that humans can work in it.

Good Luck,

Pleasant weekend to all.

SEP

[John Poff]

Hi John,

We went through the same scenario a few months ago. We actually got to the point of powering down our two big EMC arrays. Bad idea. We lost a couple of drives in the process. Ouch. We fared better than our Windows/Intel cousins. They lost a couple of machines.

That's why when we take one Saturday a month for system maintenance, the first thing we test each time is our generators. Things have gone much smoother since we started testing the generators, and the maintenance folk are happy because the generators work right now and they don't get yelled at for them failing during production times.

JP

P.S. Keep the candy cool! ;)

[Jeff Schussele]

Hi John,

I can sympathise...I've had the privilege to work in a very wide variety of "data centers". And I have to say - and brag somewhat simultaneously - that I currently work in a facility that I would humbly offer could be considered a world-class caliber data center.

Now, I have servers in three physical locations & collectively 8 separate DCs.
But the facility where I work & the one I admire has 4 & an empty 5th (make that 9) - by far it shines. We have very strict rules about how things are done. NOT intrusive - just rigid. We have redundant redundancy to the point where we have a total of 10 megawatts of CAT generators out back coupled with battery banks that I swear outdistance that DCs in square footage. Hell, when the megawatt $ hits certain points, we fire 'em up & sell it to the grid. They HAVE to buy it - it's the law.

We have strict rules about why, where & what type & even colors CAT 5 we run - rulesets are extended to ALL cable types. Mandatory labeling, detailed engineering designs & docs all add up to order & admin-ease.

But I've SA'd systems w/no UPS at all. No order - fly-by-the-seat types. YOU know 'em - we've all adminned 'em.
But I'm not lookin back....except in gratitude & thankfulness. I learned a LOT back then.

The moral is keep the faith, keep giving 110% & be the best SA around & all good things will come.
But more importantly - enjoy the ride & take no one or no thing TOO seriously. Most people forget that SA is *much* more than purely technical skills - you MUST have people and mgmnt skills - it's arguably more important.

My $0.02,
Jeff

[Khalid A. Al-Tayaran]

Hi,

Schedule a PM (preventive maintenance) to check your UPSs, generators and ACs at least twice a year.

It is good to see this for yourself and write down notes of what happened to be prepared for the actual thing.

We schedule this on regular basis to check all equipment and servers.

[T G Manikandan]

A good experience to stop happening again.


Great day indeed!