Virtual Manager starting/stopping and Manager starting/stopping

[cu4imbv]

Dear,

The system we use is 4x P4500 G2 into one cluster.
This system has been running for years without issues.
The Nodes are:
SAN01-01
SAN01-02
SAN01-03
SAN01-04
All nodes are updated to version 12.5.00.0563.0
CMC version is 12.7

SAN01-02 stated the cache battery has to be replace, so we shutdown this node, replace the battery and start it back in the cluster, but the node status keeps showing Manager Starting/Stopping and never go to Normal state.
I can't restart the node since it says: the Manager has to be stopping. I can't stop it at al, because it then says: the Manager has to be started.
So this node look like to hangs between Starting and Stopping state.

Also, I see on another Node (SAN01-01) the state Virtual Manager is hanging in Starting/Stopping state.
Since the cluster doen't have phycical issues and have 4 nodes, the virtual manager has to be stopped, but I can't stop these manager (It's complaining that the Virtual Manager has to be stopped, but it is in Starting/Stopped state).

Due to the "hanging" Manager and Virtual Manager no events are visible, the mail setting won't show up and the DNS gave me "Unable to get DNS settings because of a CIM and/or Management gateway issue", so I can't locate the problem.

Also, when I start the HP CMC, the pop-up immidately says:
The following systems report that they are running managers in a management group, but the management group thinks they are not running managers:
SAN01-02
- AND -
The following management groups report that this system is not running a manager, but the system does not agree:
SAN01-02

The Data are still accessible from our servers to the SAN, so no data loss.

Summary:
SAN01-01 reports: RAID: OK, Manager Normal, Virtual Manager Starting/Stopping.
SAN01-02 reports: RAID: OK, Manager Staring/Stopping
SAN01-03 reports: RAID: OK, Manager Normal 
SAN01-04 reports: RAID: OK, Manager Normal

It is nearly impossible to stop the Virtual Manager and the "haning" Manager from CMC,.
So how can I stop these managers without data loss AND without downtime?

Since our support contract has been ended, we can't dial the HP support number and do any chat, so we hope there is someone in this community who can help us out!

[cu4imbv]

Our SAN look like have no more space more on the OS partition, as described here: https://community.hpe.com/t5/StoreVirtual-Storage/CIM-Server-server-is-currently-Down-On-P4500-G2/td-p/7017150#.XBuZ1S1x_ys

Since we have no support contract, how is it possible to free up the OS partition on our own?

Please kindly help and/or advice.

[cu4imbv · Accepted solution]

After dialing HP they have checked the SAN in the "chrooted" enivromnent and they found out that the spooler is full.
Unfortunately we can't gain access to chrooted since it is only accessible with a special key that changed every time.
HP have this special key.

After cleaning up the spooler by HP, the SAN came up and all error's went away.