EVA4400 both controllers halted

[Mike Texter]

Hi, i have an EVA4400 (HSV300) with 4 expansion shelves, that has been working flawlessly for some time, but today when i went to boot it up, it failed to do so and spat out the following stream in the "last controller fault" logs

Controller 1:

Fault sequence number is 0
Termination code is 0x403047f
    0x20378c1c
    0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfda000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2

Fault history:-
Fault sequence number is 193
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfde000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Fault sequence number is 194
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfda000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Fault sequence number is 189
Termination code is 0xb080100

Fault sequence number is 190
Termination code is 0x4e3096a
    0xb
    0x52b9550c
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0

Fault sequence number is 191
Termination code is 0x47b0022
    0xf
    0x0

Fault sequence number is 192
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfd0000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Controller 2:

Fault sequence number is 0
Termination code is 0x403047f
    0x20378c1c
    0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfcc000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2

Fault history:-
Fault sequence number is 125
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfd2000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Fault sequence number is 126
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfcc000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Fault sequence number is 121
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfdc000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

Fault sequence number is 122
Termination code is 0x4e2096a
    0x2
    0x52b2d31b
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0

Fault sequence number is 123
Termination code is 0x478393f
    0x400
    0x0
    0x0
    0x4e2096a
    0x4de011f
    0x4de011f
    0x4e2096a
    0x4de011f
    0x403047f
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0
    0x0

Fault sequence number is 124
Termination code is 0x4de011f
    0x1020
    0x0
    0x29000
    0x20378b9c
    0x20dbcd80
    0x38f13cb0
    0x21fd1254
    0x21078014
    0x28180225
    0x402
    0x0
    0xbcfd0000
    0x0
    0x0
    0x0
    0x38f13cb0
    0x0
    0x0
    0xffffffff
    0x2
    0x402
    0x101
    0x38f11b44
    0x0
    0x19000
    0x2a
    0x2a
    0x21fd1288
    0x2
    0x0
    0x101c

If anyone can make any sense of this, please let me know.  I should add that my shelf ID's go like this:  01, 02, 12, 03 in that order.  I can't find a way to set them to anything else, but not sure that that is causing this since it's worked fine forever until now :(

[Mike Texter]

I've taken the system down to a minimal configuration, just 1 disk shelf attached with 8 disks installed, still no luck on getting either controller to boot.  A thought i had was that one of my 8 system disks has failed, but all the LEDs are green so that seems unlikely.  Would uninitializing the system from the management module do anything?  Would i be able to re-initialize it afterwards?  I don't have the WWNs or checksums for this system.