NAS 1200s NFS Hang

[Michael Kidd]

I have a 1200s that hangs when multiple files are copied or deleted via NFS mounted directories. The box has had SR3 installed along with the following hotfixes: 828214, 838238. By searching the forums, I see that other people have experienced this problem, but claim that either one of those two hotfixes, or SR3 mitigated their problem. Does anyone have any other suggestions?

[peyrache]

is it a gigabit link ?

[Michael Kidd]

No, it is on a 100Mb link.

Thanks,
Michael

[Monty Phillips]

Michael,

I have seen issues like this in the past and using the mount options really helped both performance issues and even hang issues.

mount -t nfs -o tcp,rsize=32768,wsize=32768 host:/share /mount-point

example: mount -t nfs -o tcp,rsize=32768,wsize=32768 192.168.1.150:/testshare /testshare

- Monty

[Michael Kidd]

Thanks for the mounting suggestions. I have tried changing the packet size to 32768, and still no difference in the the hanging behavior.

I would try using NFS over TCP, but it is not supported under Solaris 8, the OS of the NFS clients.

Thanks.

Michael Kidd

[Monty Phillips]

Hi Michael,

Hmm. Ok. The TCP is not as inmportant as the 32k block size. I assume you are using UDP then and that is faster than TCP is anyway.

After re-reading the failure description, it sounds like the entire server locks up. I am wondering if the NFS shares is just a side symptom of some other issue like non-paged or paged memory running out. One of these being depleted would certainly result in a hung, non-responsive system that would require a reboot to clear everything out.

Are the event logs showing any errors? The non-page and paged memory errors are 2020 and 2019. I think some other 20xx errors are also possible.

Do you have any AV software installed on the 1200s? If so, what version? I know there are some issues with certain versions of Symantec AV software causing 2020 errors (they have a patch for it).

Do you have applications like OVSM or any other type of data replication software installed? If so, what version?

- Monty

[Monty Phillips]

One more thing:

Have you run poolmon before? If you have, run it while you perform the copy operation and see if any memory tag is increasing in size dramatically and note what the tag name is and whether it is non-paged or paged.

If you have not run it before, log on the the nas console or remote desktop and go to c:\hpnas\support tools. From this directory, run poolmon -b (sorts it by byte size).

- Monty

[Greg Carlson]

Michael,

They have made some NFS updates in the SR 5.4 kit for the NAS 1200s running WSS 2k3.

It can be ordered here:
http://software.hp.com/portal/swdepot/displayProductsList.do?category=NAS

Here are the release notes for SR 5.4 as well:
http://h200002.www2.hp.com/bc/docs/support/SupportManual/c00305784/c00305784.pdf

Large I/O on an NFS mount point hangs NFS server
If you experience a problem with the NFS server not responding, increase the read/write buffer size.
Increase the buffer size by changing the mount options to:
mount -o rsize=32768,wsize=32768 server:/share /mount
If the NFS client is in a hung state, the client may need to be rebooted and the NFS server service may
need to be restarted on the Storage Server. To restart the NFS server service on the Storage Server, do the
following:
1. Open a command prompt.
2. Type net stop nfssvc.
3. Type net start nfssvc.

The SR5.4 cd is not available for download and must be ordered currently. It is free to order.

Ciao,
Greg

[Michael Kidd]

Thank you for notifying me of the latest Service Release. As suggested in the Release Notes, I have tried mounting the filesystems with the increased read/write buffer of 32K, but it does not prevent the hanging problem associated with large I/O transfers.

I have ordered SR5.4, and will install on the 1200s as soon as I receive it.

I also used poolmon.exe to monitor the NAS during a large I/O transfer, and did not notice anything out of the ordinary during the transfer, but as soon as the NFS service hung, any reference to it dropped off of poolmon. This time, the console was still responsive on the NAS, but I was unable to stop/restart the NFS service and had to resort to rebooting the NAS.

I appreciate everyone's help with this problem. I am currently working HP support as well, but the engineers don't seem to have an answer either.

Michael Kidd

[Michael Kidd]

I have now installed SR5.4 and two additional hotfixes as suggested by the HP engineers. These hotfixes are KB888618 and KB890018. I am still in the testing phase with this latest iteration, but so far I have not experienced any hanging on the NAS. I am currently still copying/removing large amounts of files as a test, but if it continues to work tomorrow (usually hangs within 1-2 hours of this test) I am going to finally declare victory.

Thanks for everyone's advice.

Michael Kidd