NFS client fails to start on HPUX 11.11

[Robert Loureiro]

Hello,

I have recently upgraded my system to HPUX 11.11 with the latest patches. My nfsconf file has the following flags set:

NFS_CLIENT=1
NFS_SERVER=0
NUM_NFSD=16
NUM_NFSIOD=16
PCNFS_SERVER=0
LOCKD_OPTIONS=""
STATD_OPTIONS=""
MOUNTD_OPTIONS=""
AUTOMOUNT=0
AUTO_MASTER="/etc/auto_master"
AUTO_OPTIONS="-f $AUTO_MASTER"
START_MOUNTD=1
AUTOFS=1
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""

When I reboot the system, my syslog.log file contains the following errors:

Nov 4 12:43:02 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 12:43:08 gatling vmunix: NFS server ak47 not responding still trying
Nov 4 12:44:56 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 12:44:56 gatling above message repeats 19 times
Nov 4 12:44:56 gatling vmunix:
Nov 4 12:44:56 gatling above message repeats 19 times
Nov 4 12:45:02 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 12:45:02 gatling vmunix:
Nov 4 13:04:56 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 13:04:56 gatling above message repeats 199 times
Nov 4 13:04:56 gatling vmunix:
Nov 4 13:04:56 gatling above message repeats 199 times
Nov 4 13:05:02 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 13:05:02 gatling vmunix:
Nov 4 13:24:56 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 13:24:57 gatling above message repeats 199 times
Nov 4 13:24:56 gatling vmunix:
Nov 4 13:24:57 gatling above message repeats 199 times
Nov 4 13:25:02 gatling vmunix: WARNING: STR_PUSH rpcmod failed, error = 22, ret
val = -1
Nov 4 13:25:02 gatling vmunix:

If I try to manually mount any remote filesystem, I will receive the following:

mount ak47:/data/os /mnt
NFS server ak47 not responding still trying

This NFS message will continue to repeat continuously, even if I try to kill -9 the mount command. The only way to terminate the mount request is to reboot the machine.

I cannot find any info on the web on why I am getting this error. I have 3 other HPUX systems that have similar settings in the nfsconf file and have no problems with NFS and remote mounts.

Can anyone shed some light on my NFS problem?

[Robert Loureiro]

Let me also add that the rc.log file contains the following lines:

Start NFS client subsystem
Output from "/sbin/rc2.d/S430nfs.client start":
----------------------------
starting NFS CLIENT networking

starting up the rpcbind
rpcbind already started, using pid: 592
starting up the BIO daemons
/usr/sbin/biod 16
starting up the Status Monitor daemon
/usr/sbin/rpc.statd
starting up the Lock Manager daemon
/usr/sbin/rpc.lockd
mounting remote NFS file systems ...
mounting CacheFS file systems ...
mount: cachefs : Invalid argument
usage: mount [-l][-v|-p]
mount [-F FStype][-eQ] -a
mount [-F FStype][-eQrV][-o specific_options]
{ special | directory }
mount [-F FStype][-eQrV][-o specific_options]
special directory
FAILURE CODE: 1
Mount failed! Possible local filesystem mount.
If local filesystem, then nfs.server will mount it.

If I manually attempt the command:
mount -F cachefs
I will receive the following:

gatling# mount -F cachefs
usage: mount [-l][-v|-p]
mount [-F FStype][-eQ] -a
mount [-F FStype][-eQrV][-o specific_options]
{ special | directory }
mount [-F FStype][-eQrV][-o specific_options]
special directory

And if I manually attempt the command:
mount -F cachefs -a
I will receive the following:

gatling# mount -F cachefs -a
mount: cachefs : Invalid argument
usage: mount [-l][-v|-p]
mount [-F FStype][-eQ] -a
mount [-F FStype][-eQrV][-o specific_options]
{ special | directory }
mount [-F FStype][-eQrV][-o specific_options]
special directory

Why is the mount command invalid?

- Rob

[TTr]

Most probably the patching upgraded the NFS version to v3 and it is no longer compatible with the ak47 NFS server. Check the NFS version on both servers using the rpcinfo command "rpcinfo -p | grep -i NFS" and see if there is a version mismatch and add the nfsvers option in /etc/fstab. See man pages

For the cacheFS error what is the cacheFS configuration? Are you even using cacheFS or did that come in with the patching as well?

[Robert Loureiro]

TTr,

Here is the result of issuing the rpcinfo command on ak47:
rpcinfo -p | grep -i NFS
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs

When I ran the same rpcinfo on gatling, I had no matches!

As for the cachefs, I am not setting this up specifically: it is being mounted as part of the startup. I have noticed that on my other PA-RISC system (11.23) I have the same mounting taking place it that startup without error.

- Rob

[Suraj K Sankari]

Hi,

Did you check your NFS server is it working for otheres ?
If yes then did you tried from other client to mount the server ak47 ?

Suraj

[Peter Nikitka]

Hi,

first check, if there are any backup files left in directory /etc/rc.config.d, like
xxx.old ...
Delete them, or move them to .xxx.old or to a different directory.

If you did an upgrade or the installation of patches, look into the directory /etc/newconfig/...
There may be startup files having an updated structure:
- modification concerning your site have to be included
- activate them in copying to the final location

mfG Peter

[Steven E. Protter]

Shalom,

The usual suspect is an error in /etc/rc.config.d/netconf or trying to maintain a backup copy of netconf in /etc/rc.config.d, only certain names will work.

Boot the system to single user mode.

mount /var
mount /usr

configure nfs.server nfs.core nfs.client to not start.

Boot the system normally.

See what the state of networking is. Correct any errors and try starting nfs manually, core, server then client.

SEP

[TTr]

Check that status of the installed patches with "swlist -l fileset -a state" and see that all the patches are in the configured state. If any are in teh installed state something went wrong during the patching. Also check all the NFS related patches that got installed and if any additional patches are required.
The error "mount: cachefs : Invalid argument" indicates that the mount command is NOT aware of the cachefs option. Maybe more patches are needed.

[Robert Loureiro]

I have applied every patch I can find. No difference.

There are no /etc/rc.config.d/old* files.

There is no /etc/newconfig/ directory.

I was trying to determine why the "mount -AQF cachefs" command was failing during startup. I found in the man pages the ability to query the kernel to find out all of the filesystem types that are available. The source code is below:

#include

int main( argc, argv )
int argc;
char** argv;
{
int max_type, error, i;
char name[FSTYPSZ];
max_type = sysfs(GETNFSTYP);
for (i -0; i< max_type; i++)
{
error = sysfs(GETFSTYP, i, name);
if (error == 0)
printf("Name is %s\n",name);
}
return 0;
}

I built and ran this on two systems, both HPUX 11.11 systems. Here are my results:

Older machine with no NFS problems:
Name is hfs
Name is nfs
Name is cdfs
Name is autofs
Name is ffs
Name is cachefs
Name is vxfs
Name is nfs3
Name is lofs
Name is pipefs

My troubled machine (gatling):
Name is hfs
Name is nfs
Name is cdfs
Name is nfs3
Name is ffs
Name is vxfs
Name is lofs
Name is pipefs

As you can see, the kernel on gatling does not have filesystems cachefs nor autofs installed. How did this happen and how do I correct it?

Regards, Rob

[Peter Nikitka]

Hi,

it may be, that there is a products called 'Enhanced AutoFS' is in play. It put significant changes of stability to the NFS implementation at 11i. I do not have any host running 11i without it.

Look at the 'swlist' of both servers to identfy this; get ENHAUTOFS from software.hp.com (free of charge) and install it.
Dave Olker would recommend the same, IMHO.

mfG Peter

[Robert Loureiro]

I installed ENHAUTOFS as well as reinstalled the NFS product from my base HPUX 11.11 cd. No improvements.

I have also run kmasdmin -k and found out that I do not have autofsc and cachefsc configured!! If I run this same command on my other, working HPUX 11.11 system, they are listed. Why are these not configured in the kernal?