vxfs, largefiles, and ADSM hickups

[Ralph Grothe]

Hello,

back from holiday again I've been confronted with a couple of ADSM backup failures of Oracle DB "hot" backups that appeared during my absence.

For these regular backups we use a fairly simple shell script that was inherited by my sysadmin ancestor, and where I never felt much inclination to do major modifications to because of lingering intentions to change from the up till now crude ADSM backup strategy to sophisticated SAN/Omniback storage technology also for this database.

So far the backup is initiated by the ADSM server that executes the mentioned shell script on the db server that collects all t-spaces, and loops over them while setting each t-space in backup mode, doing an interim copy of each dbf file therein to some other filesystem of the same db server (I was told this became necessary since the dsmc client couldn't cope backing up from the t-space filesystem directly, for whatever reason?).
Finally a sync is issued, and after a short sleep the picked dbf file gets backed up via "dsmc archive ..." on the ADSM server.
After all dbf files of a t-space were looped over the t-space is reset to end of backup mode again, and the interim copies to the transfer filesystem are removed from there again.

Now it happened that despite the sync call and the following sleep period to the cp of some dbf files the dsmc client obviously wasn't yet able to stat the copy in the transfer filesystem (how come? the sync manpage explains that all buffers including meta data are flushed), and it came up with errors like the following:

# cat /var/adm/adsm/dsmerror.log
11/01/03 22:16:07 ANS1076E *** Directory path not found ***

This matches with the following log entry from the script's log:

22:16-01.11 ADSM: /lola/data1/oradata/LOLA/vbzlb_index03.dbf NICHT gesichert

This particular t-space file (as most others) is larger that 2 GB, and currently haevily used by db procs

# ll /lola/data1/oradata/LOLA/vbzlb_index03.dbf
-rw-r----- 1 oracle dba 3565166592 Nov 3 10:40 /lola/data1/oradata
/LOLA/vbzlb_index03.dbf

# fuser -u /lola/data1/oradata/LOLA/vbzlb_index03.dbf 2>/dev/null|wc -w
435

The filesystem is vxfs, and was created with the largefiles option, though not mounted explicitly with it as it seems.
(I guess HP-UX 11.00 is taking care of this implicitly)
Btw, how can I check it is mounted with this option because the mnttab wouldn't reveal it?


# grep /lola/data1 /etc/mnttab
/dev/vgdat1/lv_data1 /lola/data1 vxfs log 0 0 1056912310

I suspect that filesystem which is on an exlusively activated, cluster shared VG (as being part of a MC/SG package)...

# grep ^LV\\[0 /etc/cmcluster/lola/lola.cntl
LV[0]=/dev/vgdat1/lv_data1; FS[0]=/lola/data1; FS_MOUNT_OPT[0]="-o rw"

...probably isn't mounted correctly (just as rw as can be seen above).

Do you think that intstead I should rather specify other additional mount options in the package control script?
What about options such as convosync, mincache, largefiles, delaylog, blkclear?
Could some of those prevent the ADSM client stat() failure?

Since I cannot restart the package would it be possible to remount the mounted LV online (the server has the OnlineJFS software installed) through the remount option of "mount -F vxfs -o remount,..." manually meanwhile?

However, if all this won't help I think about modifying the script by including stat calls (e.g. ls) or comparing checksums (e.g. cksum) in a while loop in order to make sure the the copied dbf files will be seen by the ADSM client.
Although it will take some time to have cksum executed on a 3.5 GB file will it do any harm to a dbf file (meaning rendering it corrupt for the db, although I can't hardly believe this)?

[RNY]

Hello,
Without using the cluster package,you should add in /etc/fstab the following:

/dev/vgxx/lvolx /mountpoint vxfs delaylog,largefiles 0 2

So for this reason i think you should add also the largefiles option.

Regards

[Robert-Jan Goossens]

Hi Ralph,

check this doc,

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000063235977

Hope it helps,
Robert-Jan.

[Ralph Grothe]

Robert,

thanks for the link.

But I have to stress that all the involved filesystems aren't NFS, and since the ADSM client is running as a daemon in socalled schedule mode itself (i.e. to be triggered by the ADSM server) I don't even think that it uses any RPC calls but rather standard BSD socket IPC.

Anyway the document answers some of my questions, like the detection of largefiles option.
I was simply anxious to tamper with the fsadm command on such important a filesystem ;-)

# fsadm -F vxfs /lola/data1
nomultifsets
largefiles

As you can see it carries the largefiles bit.
Otherwise there wouldn't have appeared any files larger than 2 GB I guess.

Also the doc confirmed my assumption that while you have OnlineJFS installed a "live" remount is possible.
Hope that MC/SG has the same notion of online remounts ;-)

[Bernhard Mueller]

Ralf

guess by now you found it out yourself, if not: you *can* use the remount option on filesystems mounted by an MC/SG package.

At least I have had no problems converting a couple of package filesytems to largefiles with the package up and running.

Regards,
Bernhard