HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

[Vince Pooley]

Hi All,

We have an RX7640 running HP-UX 11.23, the main storage for this server is split between two SANs, each SAN allocated a single volume group, the main logical volumes on each SAN have vxfs filesystems mounted as /data and /data2 on the server.
This server runs my company's main core investment system and stability and performance of this server is key to the running of the company, and has about 500 simultaneous users logged on each day, this brings me to a severe problem we encountered last week which has left me wondering what happened as I can find no evidence of what caused the issue in any system logs I can think of.

In the middle of the day I got some reports of multiple database applications running slowly on our server.
On investigation GPM showed me that the processes that were running slowly were waiting on disk I/O operations on databases residing on /data2 however the disc and fibre utilisation was low with no bottlenecks.
The effect was severe, the server simply could not read and write to /data2 in a timely fashion. An 'ioscan -funC disk' also hung for around 5-10 seconds.

No hardware issues was logged by the SANs, the switches, the server or the OS so this had to be a problem relating to the OS.
We eventually got everyone out of the server's applications, tried to shut down our database which hung trying to commit some it's final journals, once it was down I remounted the volumes but the problems remained.
The only things going on at the time other than the usual day to day user sessions was that I was copying a large file to another server via an NFS mount, however killing the copy and unmounting the share had no effect on the problem at the time.

Only a reboot resolved the issue but as the temporary loss of service had a financial impact on the business I am now left with a requirement to find a cause to the problem.

Any helpers?

[Torsten.]

From a very first look this sounds like a problem with connection to the array or the array itself, but without all the details probably nobody can say much more.

[Vince Pooley]

As I said though there were no hardware issues including connectivity to the SAN. I have access to all the hardware consoles for the SAN controllers, brocade switches and the server but no hardware issues where logged and gpm displayed no warnings.
Also if the cause was hardware connectivity a reboot wouldn't have made any difference.

The symptoms suggest that the OS had a problem, maybe a problem with VXFS? ANyway to trace this kind of thing?

[Patrick Wallek]

Anything in /var/adm/syslog/OLDsyslog.log?

I would also go ahead and open a call with HP support.

[Vince Pooley]

Nothing that suggests a cause no.

[Vince Pooley]

BTW I've logged this with HP, just thought I'd throw it out there in case any gurus have any ideas.

Chz

[Duncan Edmonstone]

The fact that "ioscan -FunC disk" took a while makes me suspicious (an "ioscan -fnC disk" could certainly take that long or longer and wouldn't be strange). Are you sure there wasn't any heavy sequential IO going to your system disks at the time of the issue? Heavy sequential IO on vg00 will often cause a severe performance issue, which usually shows up as commands appaeraing to run slowly... did other commands appear to be running slowly?

HTH

Duncan

[Vince Pooley]

Only processes reading from or writing to the /data2 volume which is mounted in the fstab as below:

/dev/vgmsa01/lvoldata2msa /data2 vxfs rw,suid,largefiles,delaylog,datainlog 0 2

[Wim Rombauts]

OK, this is a classic remark, but since your story also makes me suspect something in the HP-UX kernel, are you running recent storage IO patches and drivers ? And if not, have you chacked the fixed issues in the patch and driver versions you have not yet installed ?

[Vince Pooley]

2 weeks previous to the incident I had installed the latest quarterly quality pack.

[Vince Pooley]

... however I do agree that the problem sounds like a kernel level problem.

[Vince Pooley]

Well nor I or HP couldn't find the cause of this problem, just looked like the kernel threw a wobbler which is concerning for a stable HP-UX system.

Thanks for your suggestions, points awarded.