General
cancel
Showing results for 
Search instead for 
Did you mean: 

HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Vince Pooley
Advisor

HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Hi All,

We have an RX7640 running HP-UX 11.23, the main storage for this server is split between two SANs, each SAN allocated a single volume group, the main logical volumes on each SAN have vxfs filesystems mounted as /data and /data2 on the server.
This server runs my company's main core investment system and stability and performance of this server is key to the running of the company, and has about 500 simultaneous users logged on each day, this brings me to a severe problem we encountered last week which has left me wondering what happened as I can find no evidence of what caused the issue in any system logs I can think of.

In the middle of the day I got some reports of multiple database applications running slowly on our server.
On investigation GPM showed me that the processes that were running slowly were waiting on disk I/O operations on databases residing on /data2 however the disc and fibre utilisation was low with no bottlenecks.
The effect was severe, the server simply could not read and write to /data2 in a timely fashion. An 'ioscan -funC disk' also hung for around 5-10 seconds.

No hardware issues was logged by the SANs, the switches, the server or the OS so this had to be a problem relating to the OS.
We eventually got everyone out of the server's applications, tried to shut down our database which hung trying to commit some it's final journals, once it was down I remounted the volumes but the problems remained.
The only things going on at the time other than the usual day to day user sessions was that I was copying a large file to another server via an NFS mount, however killing the copy and unmounting the share had no effect on the problem at the time.

Only a reboot resolved the issue but as the temporary loss of service had a financial impact on the business I am now left with a requirement to find a cause to the problem.

Any helpers?
11 REPLIES
Torsten.
Acclaimed Contributor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

From a very first look this sounds like a problem with connection to the array or the array itself, but without all the details probably nobody can say much more.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

As I said though there were no hardware issues including connectivity to the SAN. I have access to all the hardware consoles for the SAN controllers, brocade switches and the server but no hardware issues where logged and gpm displayed no warnings.
Also if the cause was hardware connectivity a reboot wouldn't have made any difference.

The symptoms suggest that the OS had a problem, maybe a problem with VXFS? ANyway to trace this kind of thing?
Patrick Wallek
Honored Contributor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Anything in /var/adm/syslog/OLDsyslog.log?

I would also go ahead and open a call with HP support.
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Nothing that suggests a cause no.
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

BTW I've logged this with HP, just thought I'd throw it out there in case any gurus have any ideas.

Chz
Duncan Edmonstone
Honored Contributor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

The fact that "ioscan -FunC disk" took a while makes me suspicious (an "ioscan -fnC disk" could certainly take that long or longer and wouldn't be strange). Are you sure there wasn't any heavy sequential IO going to your system disks at the time of the issue? Heavy sequential IO on vg00 will often cause a severe performance issue, which usually shows up as commands appaeraing to run slowly... did other commands appear to be running slowly?

HTH

Duncan

HTH

Duncan
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Only processes reading from or writing to the /data2 volume which is mounted in the fstab as below:

/dev/vgmsa01/lvoldata2msa /data2 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
Wim Rombauts
Honored Contributor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

OK, this is a classic remark, but since your story also makes me suspect something in the HP-UX kernel, are you running recent storage IO patches and drivers ? And if not, have you chacked the fixed issues in the patch and driver versions you have not yet installed ?
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

2 weeks previous to the incident I had installed the latest quarterly quality pack.
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

... however I do agree that the problem sounds like a kernel level problem.
Vince Pooley
Advisor

Re: HP-UX 11.23: Worrying filesystem issue. Required reboot to resolve, unknown cause.

Well nor I or HP couldn't find the cause of this problem, just looked like the kernel threw a wobbler which is concerning for a stable HP-UX system.

Thanks for your suggestions, points awarded.