Storage Server Status = Down alert

Jonathan Hemphill — Thu, 05 Nov 2015 20:47:14 GMT

Hi all,
I have a P4300G1, and a P4300G2 running in a SAN Cluster along with a FOM.
I am getting alerts that are saying that the Manager is Down, but I've not been able to figure out how to solve this. The alerts will also show that several of my volumes are down too, yet they remain operational.
Here are the alerts for today from the two SANs and FOM. Based on the timestamps, I feel like they are all reporting the same problem, just slightly differently.

FOM:
Nov 5 01:38:09 fom hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 01:39:09 fom hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed. (E00080100)
Nov 5 01:40:09 fom hydra: Volume Status = normal. Value changed. (E00080102)
Nov 5 01:48:09 fom hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 05:04:09 fom hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 05:05:10 fom hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed. (E00080100)
Nov 5 05:08:09 fom hydra: Volume Status = normal. Value changed. (E00080102)

LHSAN:
Nov 5 00:00:01 LHSAN logger: MARK
Nov 5 01:33:41 LHSAN hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed.
Nov 5 01:33:41 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = storage not ready. Value does not equal up
Nov 5 01:34:41 LHSAN hydra: Volume Status = normal. Value changed.
Nov 5 01:37:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server Status = no quorum. Value does not equal up
Nov 5 01:37:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'FOM IP Addr' Status = manager down. Value does not equal up
Nov 5 01:37:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN2' Status = manager down. Value does not equal up
Nov 5 01:47:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server Status = no quorum. Value does not equal up
Nov 5 01:47:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'FOM IP Addr' Status = manager down. Value does not equal up
Nov 5 01:47:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN2' Status = manager down. Value does not equal up
Nov 5 05:03:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server Status = no quorum. Value does not equal up
Nov 5 05:03:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'FOM IP Addr' Status = manager down. Value does not equal up
Nov 5 05:03:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN2' Status = manager down. Value does not equal up
Nov 5 05:05:14 LHSAN hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed.
Nov 5 05:05:14 LHSAN hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = storage not ready. Value does not equal up
Nov 5 05:06:15 LHSAN hydra: Volume Status = normal. Value changed.

LHSAN2:
Nov 5 00:00:01 LHSAN2 logger: MARK
Nov 5 01:26:31 LHSAN2 hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 01:37:32 LHSAN2 hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 01:38:30 LHSAN2 hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed. (E00080100)
Nov 5 01:39:31 LHSAN2 hydra: Volume Status = normal. Value changed. (E00080102)
Nov 5 01:47:32 LHSAN2 hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = manager down. Value does not equal up. (E00030100)
Nov 5 05:03:31 LHSAN2 hydra: Volume Status = offline. In management group MySAN, the following volumes are offline: VMStor1, VMStor3, VMStor5, VMStor6, VMStor7, VMStor8, VMStor9. Value changed. (E00080100)
Nov 5 05:05:57 LHSAN2 hydra: Management Group: 'MySAN'; Storage Server: 'LHSAN' Status = storage not ready. Value does not equal up. (E00060202)
Nov 5 05:06:32 LHSAN2 hydra: Volume Status = normal. Value changed. (E00080102)

There are no alarms being reported, and all diagnostics run with a score of PASS.
I do have a Flow Consistancey splat, as LHSAN has NIC Flow Control "off" and LHSAN2 has it "on".

Any insight into how to resolve this is greatly appreciated.

Thank you.

Re: Storage Server Status = Down alert

Ravi_K — Thu, 30 Jun 2016 06:31:26 GMT

Am not sure if am too late to reply, you yourself have revealed the cause of the issue => Flow control inconsistency.

This leads to communication drops as seen in your environment , the way forward is to completely enable flow control end to end ( server - relevant switch ports - Storage) or completely turn it off end to end.

Also make sure to have same frame transfer settings end to end . Example make sure not to have jumbo frames ( 9000 Bytes on 1 node and default setting of 1500 bytes on the other ) . This again must be same end to end ( server - relevant switch ports - Storage)

topic Storage Server Status = Down alert in StoreVirtual Storage

Storage Server Status = Down alert

Re: Storage Server Status = Down alert