LVM Commands Hang on SLES 12 SP 1 HAE Cluster

[cjhsa]

All LVM commands hang similar to this:

# vgscan -vvvv
#lvmcmdline.c:1466 DEGRADED MODE. Incomplete RAID LVs will be processed.
#libdm-config.c:997 Setting activation/monitoring to 1
#lvmcmdline.c:1472 Processing: vgscan -vvvv
#lvmcmdline.c:1473 system ID:
#lvmcmdline.c:1476 O_DIRECT will be used
#libdm-config.c:933 Setting global/locking_type to 3
#libdm-config.c:997 Setting global/wait_for_locks to 1
#locking/locking.c:155 Cluster locking selected.

I noticed that in /etc/lvm/lvm.conf, that "locking_type=3" was set.   I read elsewhere on this forum that changing that to 1 can resolve the problem, however, I think that type is required for HAE (corosync) clustering.  I don't want to bring down the cluster.  

Help!

[cjhsa]

Also, in the output of dmesg, I see this, over and over:

floppy: error -5 while reading block 0

[simplylinuxfaq]

This indicates that your system is unable to get cluster locking from all of the clustered member nodes. Check out the cluster status on all cluster member nodes and make sure the "clvmd" (in Suse 12 this may be different) service and related services are up. You may need to look into the clustered logs to see what is happening.. this is just a hint..

[cjhsa]

They LOOK fine.  clvmd is running on both.  And:

hostname1 # crm_mon -1
Last updated: Thu Aug 17 12:25:08 2017 Last change: Mon May 22 14:27:58 2017 by hacluster via crmd on hostname1
Stack: corosync
Current DC: hostname1 (version 1.1.13-17.2-6f22ad7) - partition with quorum
2 nodes and 40 resources configured

Online: [ hostname1 hostname2 ]

....  (packages look fine)

hostname2 # crm_mon -1
Last updated: Thu Aug 17 12:28:47 2017 Last change: Mon May 22 14:27:58 2017 by hacluster via crmd on hostname1
Stack: corosync
Current DC: hostname1 (version 1.1.13-17.2-6f22ad7) - partition with quorum
2 nodes and 40 resources configured

Online: [ hostname1 hostname2 ]

[Matti_Kurkela]

It is possible that the clvmd process exists but is hanging for some reason.

You might try running "clvmd -R": it tells all the clvmd processes within the cluster to reread the LVM configuration file and reload their device cache. That should be safe to run at any time. If that command produces any errors, the errors might indicate which cluster node has clvmd in a not-sane state.

On that cluster node, you might try running "clvmd -S" to tell the clvmd to exit and re-execute itself, maintaining any locks it held before the restart. But it is possible that you might have to shutdown any clustered applications on that node, then perhaps reboot the node to clear the hang and to ensure that all the cluster software components will again be running in a correct state.

You might also consider checking the patch notes for any clvmd updates available; I think this resembles a clvmd bug that we used to have on RedHat. That bug was fixed quite a while ago.