CIM Server' server is currently 'Down'. On P4500 G2

[adrews]

Hi everyone.

I have a cluster of P4500 G2 Storage.
Few days ago, we turn off two nodes, when turn up, one node alert:

Event: EID_SERVER_DOWN E02020201
Severity: Critical
Component: Storage System
User: System
Object Type: Service
Object Name: CIM Server

Message: 'CIM Server' server is currently 'Down'.

Storage System Software Version 12.5.00.0563.0

Add info:

boot.log say:

hpasrd 1
[ OK ] Starting HP Advanced Server Recovery Daemon
Starting Health agent (cmahealthd): [ OK ]
Starting Standard Equipment agent (cmastdeqd): [ OK ]
Starting Host agent (cmahostd): [ OK ]
Starting Threshold agent (cmathreshd): [ OK ]
Starting RIB agent (cmasm2d): [ OK ]
Starting Performance agent (cmaperfd): [ OK ]
Starting SNMP Peer (cmapeerd): [ OK ]
Starting Storage Event Logger (cmaeventd): [ OK ]
Starting FCA agent (cmafcad): [ OK ]
Starting SAS agent (cmasasd): [ OK ]
Starting IDA agent (cmaidad): [ OK ]
Starting IDE agent (cmaided): [ OK ]
Starting SCSI agent (cmascsid): [ OK ]
Starting NIC Agent Daemon (cmanicd): tail: inotify cannot be used, reverting to polling
[ OK ]

Starting snmpd: [ OK ] touch: cannot touch `/var/lock/subsys/snmpd': No space left on device
[ OK ]
Checking snmpd status: [FAILED]
Starting SAN/iQ services: touch: cannot touch `/var/lock/subsys/saniq': No space left on device
[FAILED]
Starting cliqd: [ OK ] touch: cannot touch `/var/lock/subsys/cliqd': No space left on device

Any idea?
Thanks!!

[Mukesh2]

Hi,

It seems to be an issue with SNMP agents & service.

We will need to restart the following services by logging into the node via SSH :

hp-snmp-agents 

hp-snmp-agents 

mgmt-gw 

snmpd

SSH login requires root password which is generated by HPE Support, hence please raise a support case with us to resolve this.

[Serverparken]

Hi,

I have the exact same problem with a P4500G2 node (in a two-node cluster), running StoreVirtual OS 12.5.00.0563.0.  The storage is still up though, luckily.

Only difference is that I have two services/servers that do not start on the node - SNMP server and CIM server.

I guess this is the result of a filled root-partition. I think you need to log in as root to be able to access it, unfortunately.
I think the OS disk size is to small to be running LeftHand/StoreVirtual OS 10.0 and later without problems. 
Old versions of the LeftHand/StoreVirtual VSA's have the same problem.

More on this can be found here:
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c05309216

[Assen]

Hi adrews,

it seems the root partition is full. You will need to open a case with HPE, so we can free up space.

Assen Vassilev

[Serverparken]

Can this be fixed free of charge? As the units no longer have valid support contracts.
Logs filling the root partition is a time ticking bomb on all old StoreVirtual appliances I suppose. 
With no way to free up the space, the nodes are becoming useless.  But that is maybe what you want? So that we buy new stuff instead? Else you would have bulit in a job that deletes old logs, to prevent this, before they fill up the space.

[Alvaro-Drews]

Hi! any news for this?

Thanks!!

[Mukesh2]

Hi,

We are sorry but it would be chargeable service.

Regards,

Mukesh

[BC2]

Hi

Is there any way to log into this secret account without involving HP support?

We are having a similar problem with random SNMP warnings, (snmp isn't even enabled).

These are old devices that are not under any sort of HP Warranty.

I'm also now concerned that there is a random account on our SAN's with a potentially hardcoded password that we have no control over.

Regards Brad

[Serverparken]

Maybe you can try (at your own risk) booting from CentOS installation media (as StoreVirtual OS is based on CentOS), in Rescue Mode, and try to free some space up from the root partition.
It should be safe to remove all .gz files from /var/log/ and its sub-folders.
Also check if there are any very large .log files there, probably they can be safely removed also (depending on what it is).
I have not tried this myself, and I don't know if it's possible on StoreVirtual OS, but maybe it can be worth a try? If it's a non-critical, non-production environment...

https://www.thegeekdiary.com/centos-rhel-7-how-to-boot-into-rescue-mode-from-installation-dvdiso/

[Serverparken]

I had to try myself, as I had the problem on two P4500 G2 systems.

I booted from CentOS install media, choosed Rescue/Troubleshooting mode, and searched for folders holding a lot of data.

I then found that the folder "/var/spool/clientmqueue/"  ( /mnt/sysimage/var/spool/clientmqueue/ ) was huge, holding millions of useless files.
I removed them and rebooted the system. 
Now it's running fine.
Probably the folder will fill again, but now I know how to solve it.