- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: complete power failure in data center
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 04:41 AM
тАО09-15-2003 04:41 AM
complete power failure in data center
this is just a true story that had happened a few weeks back. No questions to answer. Just sit back and enjoy....
We recently had a power failure of the complete data center. There are mainframes, lot's of unix and nt servers along with a few storage systems running. not a single device was running ! except for the light everything else was without power. the reason was overvoltage causing one of the huge uninteruptable power supplies to fail. this again causing overload on the other one and triping just every circuit breaker.
now for the downtimes.
- the old ncr mainframe was down for almost two days. main reason were the disk drives. some of them are almost 20 years old. simply would not start. some network multiplexers haven't been shutdown for more than 20 years.....
- of the countless windows servers a few have been fried completely. unfortunately primary and secondary dns.... along with the network devices this part of the datacenter was not 100% working for 8 hours.
- ncr unix had some issues with filesystems. downtime 5 hours.
- hp unix (13 servers, 3 va74x0, 1 FC60, 1 Autoraid, couple of fc switches, MCSG clusters) was up and running 42minutes after the 'big bang' ! None of the 40+ oracle databases was seriously damaged. Only one power supply of a va7400 died. It was replaced within less than 1 hour and this includes delivery time ! (Unfortunately none of the HP-UX based applications could be accessed due to network problems)
What did we learn out of it ?
- really good admins are priceless
- keep any users out of the data center, they just make you nervous
- hp support is amongst the best available
- hp products don't fail as often as ncr, dell, compag
- enjoy the situation. one of the rare chances to proof you're worth your money....
Hope this never happens to anyone of you. It's worse than hell.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 05:15 AM
тАО09-15-2003 05:15 AM
Re: complete power failure in data center
I agree with all your conclucsions and have only one comment to add concerning "- keep any users out of the data center, they just make you nervous": *most* managers are users!
Pete
Pete
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 05:34 AM
тАО09-15-2003 05:34 AM
Re: complete power failure in data center
Out of curiosity, what models of NCR are you running?
It's not surprising that older disk drives and power supplies failed after running for years. We used to experience that quite often, which is why our new data center models use SAN disk (EMC), redundant power paths (3 of them), dual UPS's, diesel generators, failover capability to the other data center and of course security which prevents "users" from even visiting the Datacenters.
live free or die
harry
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 05:39 AM
тАО09-15-2003 05:39 AM
Re: complete power failure in data center
ncr mainframe is a 9844 with ap/dsp4. some of the disks are 6099 the other i'd have to check. largest disk is 700mb...
the mentioned multiplexers are of type ncr 721
veeeeeeeery old but mostly reliable equipment
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 08:03 AM
тАО09-15-2003 08:03 AM
Re: complete power failure in data center
We built our DR site (w/EMC technology & new RP8400's.) Well on the day we had everyone slated to come in and test our wonderful new DR plan with our SRDF sync'd disks and our big new Continental Cluster...we had a disaster.
No .. honest... a real disaster. At our Disaster Recovery site no less.
Seems with all the flooding, that the power company decided to just drop all power in that area to fix something or another....and poof...no power. Mgmt had gone with a mammoth UPS unit, that apparently held everything up for a several hours, but then died. Servers, disk arrays, the whole SAN on that side...DEAD. (Yes they are looking into putting in a generator out back now hopefully). So even though power came back on, nothing came back up because the UPS hadn't been 'reset'.
My Director thought I was playing a joke when I called him at home.
Fortunately, the power was back up before folks got in and after a few nervous moments with the disk array everything went off without a hitch.
What a way to prove to mgmt that they need to remember that the guy with the pliers on the utility pole down the block can stop a system too !!! tee hee
Lesson learned:
Always come in early....always !!!
Rita
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-15-2003 08:30 AM
тАО09-15-2003 08:30 AM
Re: complete power failure in data center
Each center also has 4 diesel generators, 3 are needed to run the load. These generators start up immediately when the center goes on UPS and take 3 minutes to stabilize. They are then cut into the power feed.
We have one week of diesel fuel in the tanks, some few thousands of gallons.
We are sometimes notified by our power company during the summer that we need to run the generators as they need the extra capacity for utility customers. Because of this arrangement, we get a special price for the electricity we use from them.