URGENT! vxfs filesystem hangs

[Kay Liesenfeld]

Hi,
after installing the Support Plus 12/02 the server (L-class, HP-UX 11.11) hanged during shutdown. After reseting the machine, one filesystem (vxfs) was broken: access was not possible, ll hang.
I exluded it from /etc/fstab, did a reboot and then fsck ???F vxfs ???o full,nolog [device] and mounted it again. Now access was possible. After startup of Oracle, database recovery hangs on accessing this filesystem. Now ll hangs again. I did some fsck's and reboots, but nothing helps.

Seems that heavy traffic kills this filesystem, other filesystems are not affected.

What more can I do?

Thanks,
Kay.

[Dietmar Konermann]

Check if you are able to access the lvol w/o problems; run a dd if= of=/dev/null bs=256k... maybe the underlying disk(s) make(s) trouble.

Anything in syslog/dmesg?

Best regards...
Dietmar.

[Kay Liesenfeld]

Dietmar,
the underlying disks are part of RAID-5 on VA-7100. I don't think it is a hardware problem.
Everything was fine till I installed the patches...
Kay.

[Dietmar Konermann]

Hmm, you're wrote that your system was hanging during shutdown _before_ being booted with the new kernel. I don't believe that a non-kernel patch could cause your problems (IMHO, I don't believe that a SP 1202 patch would cause them either).

Best regards...
Dietmar.

[Patrick Wallek]

I would agree with Dietmar. I doubt that your problems are due to the patches since you say the machine hung during shutdown.

I would take a serious look at the VA, the controllers, ALL the disks, you FC cards. Since and 'll' hangs as well, this really does smell like a hardware problem of some sort.

I would open up a call with HP as soon as you can and tell them that the machine is down and not usable. That will get the response center moving on this.

[Michael Steele_2]

More then likely the support plus upgrade is unreleated.

Although not mestion I'm assuming you have fibre chanel adapters going into that VA7100, could you confirm this? If true then lets check them with fcmsutil, if not then lets look at LOGTOOL.

"...ioscan -fknC fc..."
note the /dev/td? devices

fcmsutil /dev/td?
all | more

Loss of signal?
Topology?
driver stat?

fcmsutil /dev/td? devstat all > /tmp/file

please attach

If SCSI connected,

STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > VIEW > RAW SUMMARY.

Note the first and last dates of transactions and calculate the difference. If the difference is short, like 4 hours, then this is important to note. Now read down the report of hardware addresses and observe the integer numbers in parenthesis. Anything over 150 in this 4 hour period should be called into HP for replacement.

[Kay Liesenfeld]

Thanks for your answers.

Everything is ok with the fa's. This is no hardware problem. Other filesystems on this machine are ok.

Is there any possibility to uninstall the support plus pack?

[Michael Steele_2]

export DISPLAY=xxx.xxx.xxx.xxx:0.0
swremove &

This is the mouse point and shoot and drop down menu driven GUI version of swremove. Your last installed STM product is probably the last line in the list.

[Michael Steele_2]

By the way, without STM you won't be able to properly isolate your HW problems. I'd really leave it on.

[Steven E. Protter]

It is possible to remove the patch set.

But it won't help.

Your system shutdown problems prior to the patch installation are the real issue.

Sysadmin advice: You don't install huge patch sets on a system thats already flakey. You identify the problem and correct it, then patch it. Sometimes a patch helps, somethimes there are other issues.

SEP

[Kiran Kumar Aekabote · Accepted solution]

Hi Kay,

You just list the patches on the system for VxFS keyword, note down the latest and next latest patches (whether they are from 12/02 patchset) from the list and search for the patches whether they have warnings issued by hp after they released. If warning are there, remove the patch(es), and go ahead with the new recomended patch.

Find patches like below:
swlist -l product | grep -i "vxfs"

Good Luck
Kiran

[Kay Liesenfeld]

Hi all,

with help of HP support I removed the new installed patches.

After rebooting everything is normal as before patch installation.

I have the idea that this problem has something to do with Veritas QuickIO files which are located in the affected file systems. (Veritas Database Edition 2.1.2 is installed.) I was wondered that only these filesystems hung on accessing, all other did not show this behaviour, even they are on the same VA-hardware.

However, neither Oracle nor HP know that problem, so I have to ask Veritas support now.

Thanks to anyone for help!

Kay.

[Michael Steele_2]

Points to award?