NFS Performance HP Unix 11.31

[ManojK_1]

HI,

We have recently upgraded our NFS Server OS version from HP Unix 11.23 to 11.31.

NFS Clients are still running in HP Unix 11.23.

After upgrading the OS in NFS Server, we are getting the following error continually in NFS clients.

"READ: No locks available" and the application which is running on NFS giving slow performance during peak load.

What will be the reason for this. In HP Unix 11.23 we are having the option to change the Number of NFS & BIOD deamons. But it is not available in HP Unix 11.31.

Can somebody explain?

[Dave Olker]

Sounds like you're seeing multiple problems. When do you get the "no locks" message? Is it after your applications have been running a long time or the first time you try to get a lock?

You can certainly configure the number of nfsds and asynchronous I/O threads on 11.31. The nfsds are configured via the NFSD_SERVERS variable in the /etc/default/nfs file, and the "biods" are replaced with pools of asynchronous I/O threads that are sized via the nfs<2/3/4>_max_threads kctune parameters. Since this is an NFS server you probably don't care about tuning the client-side thread pools, but tuning NFSD_SERVERS is helpful for NFS servers.

If you can reproduce the file locking issue I can try to help with that. I also recommend you install the latest ONC patch bundle for 11.31 available from here:

https://software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=ONCplus

Hope this helps,

Dave

[ManojK_1]

Hi Dave,

We are using HP ServiceGuard NFS tool kit also on Servers for the High availabilty of NFS.

Version Details of ONC and NFS Tool kit given below.
ONCplus B.11.31.06.01 ONC+ 2.3

B5140BA A.11.31.05 Serviceguard NFS Toolkit

The problem is repeating when the load on the server increases.

I have changed the value NFSD_SERVERS=64. How can i verify 64 nfsd daemons are running in NFS server .In 11.23 we can do it by ps -eaf | grep nfsd.
Whenever i am trying to failover the NFS package to the other node in the cluster it is throwing an error given below.
"Unable to run the Package.The node is not eligible."
and also not starting on the original node.

[Dave Olker]

> How can i verify 64 nfsd daemons are
> running in NFS server .In 11.23 we
> can do it by ps -eaf | grep nfsd.

On 11i v3 the nfsds are kernel threads so you'd need to use a tool like Glance that displays kernel threads.


> Whenever i am trying to failover the NFS
> package to the other node in the cluster
> it is throwing an error given below.
> "Unable to run the Package. The node is
> not eligible." and also not starting on
> the original node.

You get this message when you first start the package or only when you try to migrate the package? Can you start the package on either cluster node manually? Does the package start and run correctly on all the cluster nodes?

Can you cut/paste the contents of the nfs.cntl.log file used by this package so I can see how far it gets before failing?

Thanks,

Dave

[ManojK_1]

Hi Dave,

Please find the attached syslog details during the package failover.

There is nothing written in the nfs control log file during this time.


Manoj

[Dave Olker]

Can you start the package on each cluster node manually? Does the package start and run correctly on each of the cluster nodes individually?

[ManojK_1]

Hi Dave,

Now it is not possible to stop or start nfs package as the server and application is live.

At the first time when we created the package
i have done a failover of the nfs package from one node to the second node by removing the N/W cable and it was successfull.

The issue was happen last time, when the network failed on the node where the NFS package runs.
The NFS package not started in the second node. It has just thrown an error "Unable to run the Package.The node is not eligible".

After making the network up on the first node we tried to start the package, there also we got the same error "Unable to run the Package.The node is not eligible".

At last i have stopped the node from cluster and started the node then NFS package came up automatically..

[Dave Olker]

Ok, well what you're describing doesn't sound anything like an NFS performance problem to me. It sounds like a Serviceguard cluster configuration problem. Perhaps you should move this question over to the Serviceguard forum and see if anyone there can examine your cluster configuration or Serviceguard log files to determine why the package won't start.

The phrase "node is not eligible" to me says the node does not have the required resources to start the package. Perhaps there are not enough network interfaces available for the package to run? Perhaps there are not sufficient heartbeat interfaces for the package to run? Perhaps you've specified a capacity for the systems and there is no capacity left to allow the system to be eligible to run the package?

My suggestion would be to try turning on Serviceguard debug logging (i.e. increasing the log level) and see if the log files give you a better indication of which resource is missing and causing the node to be deemed ineligible to run the package.

Perhaps someone more versed in Serviceguard can offer better suggestions.

Regards,

Dave

[ManojK_1]

Hi dave,

ServiGuard issue i will take it up separately.

As i mentioned before, NFS Server is now HP Unix 11.31 and NFS client is HP Unix 11.23

Is there is any special settings has to be done for releasing the NFS lock automatically by the appalication (from NFS Client).

We are facing the locking issue after upgrading the NFS Server OS.Before that the application was working fine.

Waiting for your valuable suggestion.

Manoj K