CFS Mount point packages not automatically starting in A.02

[Devin Lucas]

We are doing some testing of the SGSMS A.02 T2775CA package and our mount point packages are not starting up upon reboot. The CVM SMNP comes up fine, as does the disk group packages, but the mount point packages do not. They have to be started manually, which is not desirable for obvious reasons. Has anyone seen this behavior yet? I never saw this happen in A.01 clusters before. Is there something new in A.02 that needs to be done? I used the cfsmntadm commands that I have always used to setup a CFS. We ignited the systems with the June 07 MCOE, and applied all of the required patches listed in the attached document. Any insights would be greatly appreciated.

[Mike Chisholm]

I am not aware of anything additionally that has to be done. When you say "have to be started manually", what do you mean? Are you talking about the mount points being mounted when your first run the cluster via cmruncl or cmrunnode? Do you do a cmrunpkg command on the CFS mount packages and this starts or do you just have to manually run the "cfsmount /" command? Is AUTO_RUN enabled for the mount point packages?

I think it would be helpful for you to describe the problem in a bit more detail and perhaps provide a relevant excerpt from syslog.log and/or the CFS package logs along with a timeline of what you tried to do when and what the results were of each action to assist in troubleshooting.

[Devin Lucas]

Thanks Mike. AUTOSTART is enabled on the mount point packages, but when the cluster starts up, the mount point packages are all down and need to be started up with a cfsmount. I'll peruse the latest syslogs to see if I can see anything that might give us a clue and post it here. We do have another package that is dependent on the mount point package being up, but that shouldn't affect anything, and we observed this behavior when first creating the mount point package using he cfsmntadm command, before adding any additional packages.

[Devin Lucas]

Sorry, I meant AUTO_RUN, not AUTOSTART. The sequence of steps was the way I normally setup a CFS cluster.

# cfscluster config
# cfscluster start
# cfscluster status
# vxdg -s import configa01dg
# cfsdgadm add configa01dg all=sw
# cfsdgadm activate config01a01dg
# cfsdgadm display -v
Node Name : hail01 (MASTER)
DISK GROUP ACTIVATION MODE
configa01dg sw (sw)
MOUNT POINT SHARED VOLUME TYPE


Node Name : hail02
DISK GROUP ACTIVATION MODE
configa01dg sw (sw)
MOUNT POINT SHARED VOLUME TYPE

# cfsmntadm add config01adg a01dg test1vol /test1 all=rw
Package name "SG-CFS-MP-1" is generated to control the resource
Mount point "/test1" is associated with the cluster

# cfsmount /test1
Executing "/usr/sbin/cmrunpkg -n hail01 -n hail02 -v SG-CFS-DG-1 SG-CFS-MP-1"
Package SG-CFS-DG-1 is already running on all requested nodes
Running package SG-CFS-MP-1 on node hail01
The package script for SG-CFS-MP-1 failed with no restart. SG-CFS-MP-1 should not be restarted
Unable to run package SG-CFS-MP-1 on node hail01
Check the syslog and pkg log files for more detailed information
Running package SG-CFS-MP-1 on node hail02
The package script for SG-CFS-MP-1 failed with no restart. SG-CFS-MP-1 should not be restarted
Unable to run package SG-CFS-MP-1 on node hail02
Check the syslog and pkg log files for more detailed information
cmrunpkg: Unable to start some package or package instances
Error: Failed to mount mount point "/test1" on "hail01 hail02"

The package log yields 2 errors that could be the cause of our problems:

UX:vxfs mount: ERROR: V-3-20005: read of super-block on /dev/vx/dsk/configa01dg/test1vol failed: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version

Does this help? Each of those errors are listed in the VxFS File system admin doc, but neither has any fises listed.

[Sameer_Nirmal]

UX:vxfs mount: ERROR: V-3-20005: read of super-block on /dev/vx/dsk/configa01dg/test1vol failed: No such device or address

This error indicates the lvol is not accessible. It looks like VXVM problem somewhere. Just wondering how manual mount would work as I see the steps you performed shows the failure?

I would suggest to check VxVM setup using vxprint command to make sure it is okay.

[Mike Chisholm]

I don't see how this could have worked given the output you posted. It doesn't look like cfsmount actually mounted the filesystem. What does the following command show?

# vxprint -h -g configa01dg

[Devin Lucas]

Hi Mike,

Sorry I didn't provide all of the information. We are replicating the disk group to another system, and the replication link is not coming up in time, and that appears to be preventing the file system to mount up at startup. Once the system is up and the link is recovered and no longer stale, you can mount it manually. This looks like perhaps it is a timing issue with VVR and CVM. We don't have access to the lab systems anymore, but we will take it up again next week. Here is the vxprint output that I grabbed from the putty logs from one of our sessions. Thanks for your help. I'll let you know if we make any headway on it next week.

hail01:/etc/cmcluster/rvg# vxprint -ht g configa01dg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME RVG KSTATE STATE NVOLUME
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
EX NAME ASSOC VC PERMS MODE STATE
SR NAME KSTATE

dg configa01dg default default 4000 1191425064.38.hail01

dm configa01dg01 c4t0d1 auto 1024 10468224 -
dm configa01dg02 c4t0d2 auto 1024 10468224 -
dm configa01dg03 c4t0d3 auto 1024 10468224 -

v test1srl2 - ENABLED ACTIVE 5242880 SELECT - fsgen
pl test1srl2-01 test1srl2 ENABLED ACTIVE 5242880 CONCAT - RW
sd configa01dg02-03 test1srl2-01 configa01dg02 9455248 17800 0 c4t0d2 ENA
sd configa01dg03-03 test1srl2-01 configa01dg03 5243144 5225080 17800 c4t0d3 ENA
pl test1srl2-02 test1srl2 ENABLED ACTIVE LOGONLY CONCAT - RW
sd configa01dg02-04 test1srl2-02 configa01dg02 9454984 264 LOG c4t0d2 ENA

rv test1rvg 1 ENABLED ACTIVE primary 1 test1srl
rl rlk_192.168.1.4_test1rvg test1rvg RECOVER STALE 192.168.1.4 configa01dg rlk_192.168.1.160_test1rvg
v test1vol test1rvg ENABLED ACTIVE 19922944 SELECT - fsgen
pl test1vol-01 test1vol ENABLED ACTIVE 19922944 CONCAT - RW
sd configa01dg01-01 test1vol-01 configa01dg01 0 10468224 0 c4t0d1 ENA
sd configa01dg02-01 test1vol-01 configa01dg02 264 9454720 10468224 c4t0d2 ENA
pl test1vol-02 test1vol ENABLED ACTIVE LOGONLY CONCAT - RW
sd configa01dg03-02 test1vol-02 configa01dg03 0 256 LOG c4t0d3 ENA
pl test1vol-03 test1vol ENABLED ACTIVE LOGONLY CONCAT - RW
sd configa01dg02-02 test1vol-03 configa01dg02 0 256 LOG c4t0d2 ENA
v test1srl test1rvg ENABLED ACTIVE 5242880 SELECT - SRL
pl test1srl-01 test1srl ENABLED ACTIVE 5242880 CONCAT - RW
sd configa01dg03-01 test1srl-01 configa01dg03 264 5242880 0 c4t0d3 ENA