R5500 Electronic Module Not really Hot Swappable?

[Luca]

We have several R5500s installed mainly to protect c3000 bladesystems at our clients. We've never had any major problems apart from a couple of Management Modules acting up and dieing - but we can live with this. Now the manual states that the 5500 is completely "user hot swappable": Electronics Module and battery packs. Well this apparently isn't so!
One of our 5500 died (after some storm or power glitch, we don't know but there was a big black hole in the motherboard pcb with heavy carbonised epoxy smell) and the UPS's fail safe relays set it on BYPASS so that even with the module dead, the 5500 fed power to the servers - so this worked fine. Removing the Electronic Module was no problem and everything still stayed online. However, when inserting the replacement one, instead of staying online, the whole thing powered off killing the power to the whole c3000 (it "glitched"). The funny thing is, that the C3000 was in AC redundant mode with 4 power supplies, 2 were connected to the UPS and 2 were connected to a preferential filtered power outlet and the power usage was far below the threshold (only 3 servers powered on with around 500Watts total use - max is 1200 per PSU line). When inserting the Electronic Module, the UPS powered off and the whole c3000 "glitched" and reset. Now I can understand if we connected the whole C3000 to the UPS, but this was AC redundant with seperate power lines so there must be something not working correctly with this system apart from the fact that the Electronic Module is NOT hot swappable because if it were, it would not power off the whole unit upon insertion! We tried this with all the blades off and just the OA on and it reset again upon inserting the module.
I'd like to have some feedback on this R5500 issue: is it or is it not Hot Swappable? It appears it is hot removable, but not hot insertable meaning that if you do insert it, it will disrupt power to the segments (we can clearly hear the segment relays quickly clicking off and on again upon insertion).

[gregersenj]

It's Hotswap.

In general.
Eventhough things are hotswap, and redundant. There's allways a little risk, that it may fail.
It could be due to:
A more severe problem
Bad spare
Incorrect replacement procedure.
Just bad luck.

In You case.
Did you hold the alarm button, untill the module was properly seated?
There's a big hole in the BCB. Could be more severe.
Is this due to a incomming spike, and could it affect the C3000?

I cn't remember if I have replaced parts on a R5500. But I have done it on the 3000 models and the 12000. Hot, without any problems.

BR
/jag

[Luca]

Greg,
thanks for the feedback. Well, first of all, if it's hotswap, it's hotswap - of course there is always some risk involved, thats why we had the AC redundant setup. The C3000 appears fine, all is green. While replacing the module the test button was pressed. Of course it could be possible that while inserting the button may have not made contact well - this is why we tried it again but apparently the same thing happened. We will try all this again on a new setup as soon as we have the chance as we need to verify what really failed.

Luke

[Trygve Henriksen]

Which PSUs in the c3000 were connected to the UPS and which to the filtered line?

3 servers in use?
does this mean that you have more servers in the c3000?
What's the position of the 3 that was powered up?

[gregersenj]

Yes, get a service window

Do recheck, every thing.
Are the C3000 really setup correctly.
Is all PSU's OK.
Is there sufficient power to run the C3000 on the non UPS power feed.

Check if there's more servere damage to the UPS.

BR
/jag

[Luca]

In the c3000 the servers were in bays 1,2 and 4 and in bay 5 we have the adjacent Ultrium Blade tape drive for server 4. 4PSUs installed in slots 1,2 and 4,5 in AC redundant. Load with everything on was about 500Watts (2 PSUs were off in Dynamic power savings mode) so there was no overload problem. 2 UPS powered AC leads go to PSU 1 and 4 and 2 preferential AC leads go to PSU 2 and 5.

We think that by cutting the power off BOTH PSUs (blue tagged) in slot 1 and 4 at the same time the PSUs in slot 2 and 5 don't come online fast enough apparently.

We will run some tests in a non production environment as soon as we can and let you know.

[Luca]

Correction:

In the C3000 the servers are in bay 1,2 and 5 (not 4) and in the adjacent bay 6 the Ultrium drive (properly across the 2 bay power bus-es).

[gregersenj]

It doesn't matter how the servers are palced, everything is powered from all PSU's.

I'm trying to find the documentation on the C3000 regarding AC redundency with 4 PSU's.

BR
/jag

[gregersenj]

I haven't found what I was looking for yet.

But I believe, your power cabling is wrong.

PSU 1, 2 and 3 should be the A side
(Left side of the C3000 it is rack mounted)
PSU 4, 5 and 6 should be the B side.
Thats how it is on the C7000.

So you should put PSU 1 and 2 on the UPS
and 4 and 5 on the preferential powre.

I would like some confermation, from other users or a manual.
But I believe they wouldn't change that on the C3000 vs the C7000.

A/C redundency = loss of 1 side (A orB).

BR
/jag

[Trygve Henriksen]

That's how we do it with our c7000, too.

Except we have 1+2+3 on one 25A fused outlet and 4+5+6 on another 25A outlet.
(Our entire server room is on a BIG UPS. Very big... )

and I'm pretty certain that's how it's supposed to be done on the c3000, too, as it's mostly the same design, only fewer BLade slots and interconnect bays.