2nd BCV split fails fsck

[Stuart Abramson_2]

We are implementing a 2nd BCV off of the primary STDs. The 2nd split volumes failed fsck. I got a message:

Checking logical volumes in vbkp01.
log replay in progress
vxfs fsck: file system does not contain a valid log
vxfs fsck: cannot perform log replay
file system is not clean, full fsck required
full file system check required, exiting ...
Mounting /dev/vbkp01/lvol01 to /u30.
vxfs mount: /dev/vbkp01/lvol01 is corrupted. needs checking

Here is the time sequence:

2300: First BCV syncs
0115: Shut down Oracle
0030: split 1st BCV
0100: 2nd BCV syncs
0118: 2nd BCV splits
0119: fsck fails
0300: restart Oracle

The code is the same. The commands look like this:

symmir -f ${BCVFILE} -sid ${SID} -instant split -nopromt

fsck -F vxfs /dev/${VOLUME}/${LV}

When I do a "full" fsck with the "-y" option, the fsck is successful, and the filesystems mount:

fsck -y -F vxfs /dev/vbkp03/lvol05

And, the filesystems are identical, and Oracle starts on the 2nd BCV.

So, what should I do?

1. Is an 'fsck -y' legitimate in this case? 2. Should I do
# 'fsck -y'
- or:
# fsck -o full,nolog -F vxfs /dev/vbkp03/lvol05
3. What makes an fsck fail like that?
4. Can the fsck fail if someone (else) is READING my down Oracle database.

[James Murtagh · Accepted solution]

Hi Stuart,

This sort of problem normally occurs when you perform a hardware split of a volume and try and mount it on another server which doesn't have the same version of JFS installed. For example, if your original system is 11i using the default JFS 3.3 and the system used to mount the filesystem is 11.00 using JFS 3.1 then you would have this problem. I'd imagine you are going the other way, from an older version to a newer version as the fsck fixes it, hence it knows about that version. Here is the table:

JFS OS Disk Layout Intent Log
Versions Versions Version Versions
-------------- -------------- -------------- -----------
3.0 10.20 2,3 5
3.1 11.00 2,3 6
3.3 11.00 2,3,4 9
11.11 2,3,4 9
11.22 2,3,4 9
3.5 11.11 2,3,4 9

To identify the intent log version for a filesystem use:

# echo "8192 B; p S" | fsdb -F vxfs /dev/vgXX/lvolYY | grep log_version

At the very least make sure the filesystem is unmounted before the split. If it is still mounted it is marked "Dirty" and an fsck will always be required.

Regards,

James.

[Geoff Wild]

On our backup server, when we mount bcv's, we always do a fsck with the -y option. As long as the bcv's are split, then it doesn't matter what anyone does to the "downed db". Chalk this up as one of those "EMC" features...

Rgds...Geoff

[James Murtagh]

Hi again,

Didn't think that table would come out ok, here is the list:

JFS Version 3.0
HPUX 10.20
Intent log version 5

JFS Version 3.1
HPUX 11.00
Intent log version 6

JFS Version 3.3
HPUX 11.00, 11.11, 11.22
Intent log version 9

JFS Version 3.5
HPUX 11.11
Intent log version 9

The intent log version for each JFS release is the key. If you give me some more information on the OS and JFS versions you are working on I'll try and help more.

Regards,

James.

[Stuart Abramson_2]

You are exactly right. Look at this:

saroot@acshps05:/home/saroot# db -F vxfs /dev/vbkp01/lvol03 | grep log_versio*
log_version 9 logstart 0 logend 0
saroot@acshps05:/home/saroot# db -F vxfs /dev/vbkp01/lvol05 | grep log_versio*
log_version 6 logstart 0 logend 0
saroot@acshps05:/home/saroot# vxfs /dev/05vg89/lvol01 | grep log_version <
log_version 9 logstart 0 logend 0

My new system is v 9. My old system is v 6.

Would that cause this problem?

Also look at swlists:
# swlist | grep -i jfs
B3929BA B.11.00 HP OnLineJFS (Advanced VxFS)
B3929CA B.03.03 JFS 3.3 Filesystem For 11.00

My OLD system doesn't have this 2nd product, B3929CA!

Thanks a million.

[James Murtagh]

Hi Stuart,

The default JFS product on HPUX 11.00 is 3.1, which has a log version 6 from the table. You are trying to mount this on a system using JFS 3.3 with log version 9. A full fsck will fix this as you are seeing.

An easy solution is to ensure the filesystem is unmounted before the split. As long as the filesystem version is supported this shouldn't require a full fsck. Check the filesystem version with:

# fstyp -v /dev/vgxx/lvoly|grep version

or

# echo "8192 B; p S" | fsdb -F vxfs /dev/vgxx/lvoly | grep " version"

Another solution is to upgrade the original server to JFS 3.3, I think it should be a free upgrade if you have 3.1 but you can check this out.

Just be carful though - you won't be able to split volumes from the 3.3 server and mount them on the 3.1 server - it won't have a clue about a later release.

Regards,

James.

[Tim Sanko]

Stuart,

We found for oracle on HP that HFS is a simpler solution.

The symm handles hardware reliability, and the HFS allows for a simpler FS with higher performance with our Oracle database.

Regards

Tim