topic Re: CIFS Client - RPC Timeouts in Operating System - HP-UX

CIFS Client - RPC Timeouts

David Connolly — Thu, 24 Aug 2006 05:33:16 GMT

Hello, We are using cifs client v2.02 on two HPUX 11.11 system pathed to June 2005.

Both systems mount the same Windows share. On one of the systems (server1), the share is in frequent use (read/write) but on the other (server2), the share is just mounted.

In the syslog on system1 I see frequent errors of the following nature:

Aug 24 11:22:41 system1 vmunix: NFS lookup failed for server : RPC: Timed out
Aug 24 11:22:41 system1 vmunix: NFS getattr failed for server : RPC: Timed out
Aug 24 11:22:43 system1 vmunix: NFS getattr failed for server : RPC: Unable to receive
Aug 24 11:23:37 system1 vmunix: NFS lookup failed for server : RPC: Unable to receive
Aug 24 11:22:41 system1 vmunix: NFS lookup failed for server : RPC: Timed out
Aug 24 11:22:41 system1 vmunix: NFS getattr failed for server : RPC: Timed out
Aug 24 11:23:37 system1 above message repeats 2 times
Aug 24 11:23:13 system1 vmunix: NFS getattr failed for server : RPC: Unable to receive
Aug 24 11:23:38 system1 above message repeats 2 times

I have checked out this thread http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=1012892
but it seems to suggest an upgrade, and I am using the latest version at this time.

One difference I noticed between system1 and system2 is that NFS services are disabled on system1 and not on system2. What NFS services are required for the CIFS client? I tried starting NFS Core (which also starts rpcbind) but I still get the error.

Thanks in advance

Re: CIFS Client - RPC Timeouts

David Connolly — Thu, 24 Aug 2006 05:47:28 GMT

PS: nfs client and nfs server are disabled on both systems, but START_MOUNTD=0 on the affected system whereas the other has START_MOUNTD=1

Re: CIFS Client - RPC Timeouts

Peter Nikitka — Thu, 24 Aug 2006 10:53:25 GMT

Hi,

did you enable and start the cifsclient daemon?

Look at ar /etc/rc.config.d/cifsclient and then call

/sbin/init.d/cifsclient start

mfG Peter

Re: CIFS Client - RPC Timeouts

Eric Raeburn — Thu, 24 Aug 2006 14:21:31 GMT

David,

This is Eric from HP's CIFS Client lab. What exactly do you mean by "nfs...is disabled"? How have you disabled it, and why?

The HP CIFS Client depends on the NFS client--it is an integral part of its operation. The CIFS client uses NFS as a frontend, a means to "intercept" user-space commands and system calls, so it can convert them to CIFS protocol requests, and send them to the CIFS server.

I would like to see what your answers are to my initial questions, so I can try that myself and look at the results, but my suggestion is that you fully enable NFS, restart netowrking or reboot, and try the CIFS Client again.

-Eric

Re: CIFS Client - RPC Timeouts

Eric Raeburn — Thu, 24 Aug 2006 16:46:46 GMT

David,

I just re-read your original post and noticed you stated that "the share is in frequent use" on system1. Other than the messages in syslog, do you notice any abnormal behavior with respect to activity against the cifs mounts? In general, does it seem to work properly? If it does, my suggestion regarding NFS in my previous post is probably not correct.

The "timeout" messages are usually indicitave of a slow response from the server. We have a fix for this, but it is only available on HP-UX 11.23 and later; more specifically, it depends on an enhancement in NFS that is available only on 11.23 and later.

Please post your replies to my questions. Depending on those replies, we (you) may need to open a call with HP support. They will advise you which log levels to enable and request a CIFS Client logfile to send us for analysis.

-Eric

Re: CIFS Client - RPC Timeouts

David Connolly — Fri, 25 Aug 2006 06:07:27 GMT

Hi folks, thanks for the response.
@Peter - yes, cifsclient is enabled.

@Eric,
NFS core was disabled on system1 as a result of server hardening according to the Centre for Internet Security guidelines. The server was hardened prior to CIFS installation. I did not restart the system or networking after enabling nfs.core, so this might be what causes the problem to persist.

Aside from the syslog messages, I notice no abnormal behaviour w.r.t. CIFS, the shares work fine. However I do (occasionally) notice a "freeze" of the system where, for e.g., my ssh session will pause for 30 to 60 seconds. This roughly coincides with the syslog errors. Of course this might be something else and the errors are caused by this freeze instead.

Re: CIFS Client - RPC Timeouts

Eric Raeburn — Fri, 25 Aug 2006 17:22:49 GMT

David,

If the CIFS Client is in general behaving, it is extremely unlikely that any component of NFS which it needs could be the culprit. If a needed NFS component were down, I doubt the CIFS Client could even mount the share, much less operate on it. I think we can ignore NFS as a cause of the syslog messages.

I suggest one or both of the following:

(1) The system "freeze" you describe could be significant (especially if your sense that the cifs syslog messages are coincident with it is correct), and should be pursued as an HPUX performance or system administration issue with HP support. A delay of thirty to sixty seconds is very long.

(2) The syslog messages for system1 could also suggest that the CIFS Client is itself in a bad state. To diagnose this, I would suggest, first of all, starting an HP support call--this type of issue is too complex to pursue in this forum. We will need a cifsclient logfile (with appropriate log levels enabled in the cifsclient configuration file), at the time the syslog entries are created. Support can help you with this.

It is critical that you catch it at the right time, in order for the cifsclient log to contain useful data. In this case, I don't think it would be difficult to automate this in a shell script. You can create a 'while' loop that sleeps for 1 second, looks for the required pattern in syslog, if not found loop again, else make a copy of the cifsclient logfile and break out of the loop.

If you call support, tell them we have already "spoken" in the ITRC. Let me know here if you decide to call, or if you have further questions.

Good luck,
-Eric

Re: CIFS Client - RPC Timeouts

David Connolly — Mon, 28 Aug 2006 02:52:12 GMT

Thanks Eric

I'll open a call.

Dave

Re: CIFS Client - RPC Timeouts

David Connolly — Thu, 31 Aug 2006 10:28:54 GMT

Just for info - not confirmed yet but it looks like dbc_max_pct kernel parameter has an impact on the NFS client. My setting on the affected server is 10% (of 2GB physical). On the unaffected server it is 10% (of 4GB physical RAM).

Re: CIFS Client - RPC Timeouts

David Connolly — Fri, 15 Sep 2006 03:23:05 GMT

Latest update. It would appear my system is missing PHNE_33498. This patch is "ONC/NFS General Release/Performance Patch" and is included in the "June 06: GOLDAPPS11i"

I've installed the bundle and had only one NFS error in the week since (I was getting about 4 per hour)

The error is not the usual getattr or lookup failure, but it is still an RPC timeout.

Sep 12 13:00:16 ilife622 vmunix: NFS fsstat failed for server : RPC: Timed out

So, on the face of it, patching to the latest goldpack sorted the problem :)

Re: CIFS Client - RPC Timeouts

Eric Raeburn — Fri, 15 Sep 2006 11:55:56 GMT

David,

That's good news; thanks for the updates. If you are interested in pursuing the timeouts further, please infrom your support representative. As I mentioned earlier, we can set appropriate log levels in the CIFS Client and examine the logfile. This can help determine whether the NFS timeouts are due to a slow response from the server, or some (inappropriate) delay internal to the CIFS Client.

-Eric