topic Re: hsz70 ctl, lost pwr to storage box in Operating System - OpenVMS

hsz70 ctl, lost pwr to storage box

Dean McGorrill — Wed, 13 Jun 2007 22:53:23 GMT

I have ds20's connected to storage works boxes
2 in a 6' rack, one other was added on the side. each have 24 disks scsi, model don't know
and too po'd right now to remember.

lab folks decided to move the systems a couple
of feet and thought it could be moved power on.

problem, during move the side box power got
removed for a while. when I finally got the
hsz70 console back connected, all the side storage disks were failed out into the failedset.

The Question, all the failed disks are blinking
like they are bad. Does the hsz70 controller make them blink bad if they are in the failedset?? If so, if I remove them from the
failed set then I can rebuild all my mirrors etc. tx, all posts pointed - sigh tx dean

Re: hsz70 ctl, lost pwr to storage box

Hein van den Heuvel — Wed, 13 Jun 2007 23:43:47 GMT

I think they are blinking for want of some TLC. Sounds like you are not ready for that just now. Take a deep breath and give them the attention they bleeb for. Given the scenario you know that the drives are likely just fine, even though moving spinning disks around is not a great idea. The HSZ however just saw them dissapear and miracoulesly come back. Thus they are labelled suspect. I think you can just remove from faileset and reassign to spareset. CHeck out the HSZ70 - HSOF doc, for example page 4-28 in:
ftp://ftp.compaq.com/pub/products/storageworks/techdoc/controllers/EK-HSZ70-CG-B01.pdf

hth,
Hein.

Re: hsz70 ctl, lost pwr to storage box

Hoff — Thu, 14 Jun 2007 09:17:47 GMT

I'd start with a serious and careful system upgrade to the lab servants. This potentially involves replacement of all memory contents, and potentially also a cold reboot to clear existing all existing errant processes within the servants. Even servant replacement. I'd also check the backup and related processing, as cases such as this one tend to have additional and parallel secondary manifestations within the BACKUP processing; these sorts of servant failures tend to involve multiple systems within the environment.

As for the HSZ70, you are likely going to end up rebuilding the RAID sets. The blinking disks failed, after all, secondary to the operational failures of the servants.

The documentation for the controller has the sequence for clearing out and re-adding disks, assuming your data still exists.

The documentation on recovering from servant failure and -- if deemed necessary -- servant replacement is generally available separately, and not included with the OpenVMS nor the HSZ70 documentation.

:-)

Re: hsz70 ctl, lost pwr to storage box

Dean McGorrill — Thu, 14 Jun 2007 11:27:22 GMT

Gentlemen,
I was able to clear the
blinking lights by removing all the drives
from the failed set. fortunately this storage box I had configured for only for mirrors, thus my system is up for use. I now must rebuild all my mirrorsets. Thank you! Dean

Re: hsz70 ctl, lost pwr to storage box

Dean McGorrill — Thu, 21 Jun 2007 16:01:28 GMT

theres some foot notes to the power loss story. it turns out the storage ba370 box was not
moved but its power had to be cut to unentangle the wiring maze. power was removed from one side only (redundant supplies) returned then the other side was
removed and returned. power swiches were
on but no juice after that, why? flipping both switchs reset the power. next I did
lose 2 disks from the power supply maneuver. that surprised me! ? they were
under warrenty, their replacesments came
today. power hits are common, anyone else
lose disks this way? I would not think
they would be that fragile. Dean

Re: hsz70 ctl, lost pwr to storage box

Ian Miller. — Fri, 22 Jun 2007 05:40:18 GMT

power blips can often lead to hardware failure especially for systems which have been on a long time.

Re: hsz70 ctl, lost pwr to storage box

Rob Leadbeater — Fri, 22 Jun 2007 17:03:01 GMT

If they're in a BA370 shelf, chances are the drives have been spinning and their bearings wearing out for a *long* time.

It's not at all surprising that some didn't spin back up after a power outage. Pulling the drives and giving them a slight knock may well have got them spinning again...

Cheers,

Rob

Re: hsz70 ctl, lost pwr to storage box

Dean McGorrill — Fri, 22 Jun 2007 17:31:01 GMT

Rob,
these were new drives out of the box.
they could have been sitting a few years
at the vendor but they had not been spinning long. It really surprized me
how fragile they were! -Dean

Re: hsz70 ctl, lost pwr to storage box

Wim Van den Wyngaert — Sat, 23 Jun 2007 13:26:57 GMT

I'm already 22 hours on site (with a short sleep in the middle). We had FDDI problems. The card was to blame. We replace the card. PCI power has gone dead. We replace the power. New card is malfunctioning. Old card again. Dead. The neighbour GS160 member also want to fail. Just as the first node is functioning again, the neigbour node starts with a hang. Power cycle. Node boots but give memory ecc errors after a while. And power problems in error log. And we're still busy.

Was power on for a few years.

Do not touch the nodes if possible !

Wim

Re: hsz70 ctl, lost pwr to storage box

Ian Miller. — Sun, 24 Jun 2007 15:56:14 GMT

Dean,
what model of drives where they?
They may have been made a while ago and they may have been repaired (recycled).

Re: hsz70 ctl, lost pwr to storage box

Dean McGorrill — Mon, 25 Jun 2007 11:11:09 GMT

Wim,
man what a mess!! my hats off to you and
get rest in between boughts with that hardware.

Ian,
They are RZ1FC 36g drives. they came in
what appeared to be new boxes, not marked, clean labels etc. all sealed up pretty.

I'm a little more confident today. last friday,
at 106 degrees in az, we had a power hit.
ac failed, UPS shunted the lab. I expected
a mess similar to Wim's. After ac fixed, systems came up AOK! Dean

Re: hsz70 ctl, lost pwr to storage box

Wim Van den Wyngaert — Tue, 26 Jun 2007 01:50:59 GMT

But it finished at that. The power supply was not replaced but the machine is running again. But I will not tell what went wrong on the ING bank side.

And no application damage so far.

Wim