NFS Shares can not create files

[Ken Martin_3]

I am having a problem with an old HP/UX 10.20 machine.

I was able to share exports with PC clients with no problems before. Now when I 'net use' a share off the HP/UX box from Windows 2000 and XP Pro machines I can see the files and directories but not add any new ones. Even though the directory is set to rwxrwxrwx.

The PC client software shows the correct userid and group. I tried changing exports to make the shares -anon=202 which is a valid user who has access to same files via telnet. No luck. Even with exports entries set to -anon=555 I can not add files even to a directory marked as rwxrwxrwxrwx.

Is it possible someone altered the services or settings on the HP/UX box related to NFS server or PCNFSD or rpc.lockd? I ask this because I can share OK with Linux boxes.

And yes, I know it's old software. It's a legacy file server that dosen't get used much. Actually not at all now!

Any ideas? Thanks...

[Patrick Wallek]

What does your /etc/exports file look like on the 10.20 machine?

[Ken Martin_3]

Originally it was
/users -anon=555
/var -anon=555

I changed it to:
/users -anon=202
/var -anon=202

I also tried:
/users
/var

After each change I did: exportfs -a
Then from the PC client re-connected.


Ken

[Nagashankarp]

since it is only write that is not working, how about /usr -rw

cheers
nagashankar

[Ken Martin_3]

changing /usr to /usr -rw did not work.
exportfs would not allow it.
So I made it /usr -rw=

exportfs accepted that however I still only have read and update access no write access.

Any attempt to write or create a directory from the PC client results in 'Access Denied'.


Ken

[Dave Olker]

Hi Ken,

My first suggestion would be to collect a network trace of the problem so we can see what packets are being sent by the PC to the HP-UX server and the reply being sent back. That might help better understand why the request is being rejected.

Here's what I'd suggest:

1) Unmount any shares from the PC client
2) Start a network trace on the server:

# /etc/nettl -tn pduin pduout loopback -e NS_LS_IP -s 1024 -tm 99999 -f

3) Mount the share from the PC
4) Try to create a file in the NFS directory
5) Stop the trace

# /etc/nettl -tf -e all

Once you have the trace (it will be called whatever file name you gave with an extension of .TRC00 and possibly .TRC01) make this trace available to us along with the IP address of the HP-UX NFS server so that the trace can be filtered. I'd be happy to give the trace a quick look.

Regards,

Dave

[Ken Martin_3]

Steve,

Find attached (I hope) a file called 1206.TRC0

I did a quick 'strings' of the file and just after 'Reflections Network Series' (Which is the NFS client software) I see the text 'No access via pcnfsd to this account'!

Any ideas? Thanks...

P.S. How do you delete stale (old) entries in showmount -a? Stopping and restarting (NFS Server) does not clear them out.

[Dave Olker]

Ken,

> P.S. How do you delete stale (old) entries in showmount -a?
> Stopping and restarting (NFS Server) does not clear them out.

You have to stop rpc.mountd on the server, remove the /etc/rmtab file, and restart rpc.mountd on the server.

Dave

[Dave Olker]

Ken,

I looked at the trace. Here's what I see:

1) The client successfully mounts the /users/kmartin directory using NFS V2

2) The client retrieves the directory contents with a series of READDIR/LOOKUP calls

3) The client sends a LOOKUP call for the file "jjj"

4) The server replies that the file doesn't exist, which is true

5) The client sends in an UNMOUNT request for the /users/kmartin directory


Bottom line, I never see the client even attempt to create a file in this trace. It looks to see if "jjj" exists but I don't see any attempt to create the file. The directory permissions appear to be 777 and owned by the appropriate user/group.

Are you certain the PC-NFS client is working properly? I don't see any CREATE call in the trace getting denied.

The only other things I see in the trace that give me any concern is the client is sending UDP broadcast packets for a program called "epmap". Not sure why, but the client is doing it.

Also, the HP-UX server appears to be configured to use DNS for hostname resolution and it is trying to contact the named/bind daemon on the local HP-UX system, but since the server is not running named/bind the requests fail.

You might want to verify the /etc/nsswitch.conf file and /etc/resolv.conf file settings on the HP-UX system to ensure the server is configured correctly for whatever hostname resolution method your site uses.

Regards,

Dave

[Ken Martin_3]

Dave,

The server did not have an
/etc/nsswitch.conf file.

It is a stand alone file server on a LAN.

It has a working /etc/hosts file so I created an /etc/nsswitch.conf as follows:

# cat /etc/nsswitch.conf
hosts: files[NOTFOUND=return UNAVAIL=continue]

/etc/resolv.conf not created, no NIS running.

I cleaned out /etc/rmtab and rebooted the HP.

I then fired up a Windows 2000 machine.
Enabled nettl on the HP.
Loged in via NFS to the HP.
Mounted the K drive to /users/kmartin.
Did a Directory.
Attempted to create file jjj using the
command: dir >jjj
Access still failed.
Disabled nettl logging.
Attached new log file.

The 'epmap' you mentioned before. Isn't port 135 (End Point Map) a Windows function?

Any ideas?

Thanks.

[Dave Olker]

Hi Ken,

This trace is similar to the last one but there are some important differences. Just like last time, the client mounts the remote filesystem, reads the directory, looks for the file "jjj", gets back a reply that the file doesn't exist. So far, same as last time.

Then things get a bit more interesting.

The client sends in a SHARE request to the server's rpc.lockd daemon on port 1037. This should be the correct port number because it's the number the HP-UX system provided in response to a GETPORT call from the client. In other words, the client asked the server "What port number is rpc.lockd listening on for NLM Version 3 requests?" and the server replied "1037". In any case, this SHARE request is rejected with an ICMP Port Unreachable.

The only reason I can think of why this would happen is if the rpc.lockd is dead on the HP-UX system but for some reason the portmapper still has the old port number cached in its table from the last time rpc.lockd ran. The client then sends another GETPORT asking again for the server's NLM V3 port number. This time the portmapper replies back with PROG_NOT_REGISTERED - meaning there is no rpc.lockd daemon registered on the HP-UX system.

So I would be looking at the server's rpc.lockd daemon. Is it running? Even if it is, I'd try the following:

1) Look for rpc.lockd and rpc.statd

# ps -ef | grep rpc

2) Terminate any running rpc.lockd and rpc.statd

# kill

3) Restart rpc.statd and rpc.lockd

# /usr/sbin/rpc.statd
# /usr/sbin/rpc.lockd

4) Wait 60 seconds for rpc.lockd's grace period to expire

5) Verify that rpc.lockd and rpc.statd are running

# rpcinfo -u localhost nlockmgr
# rpcinfo -u localhost status

Once you've verified that rpc.lockd and rpc.statd are running and responding to rpcinfo requests, try the test from the PC-NFS client again.

Regards,

Dave

[Ken Martin_3]

Dave,

OK. The lockd was stopped. For some reason on the W2K PC after starting and ending an NFS session the HP if a new session is attempted I get a message from the PC that no lockd is available, continue YES/NO. lockd stops with a signal 15 in the log?

# /sbin/init.d/nfs.server start
starting NFS SERVER networking
starting up the portmapper
portmap already started, using pid: 354
Reading in /etc/exports
starting up the mount daemon
/usr/sbin/rpc.mountd -l /tmp/mountd.log -t2
starting up the NFS daemons
/usr/sbin/nfsd 4
starting up the Status Monitor daemon
/usr/sbin/rpc.statd -l /tmp/statd.log
starting up the Lock Manager daemon
/usr/sbin/rpc.lockd -l /tmp/lockd.log
starting up the PC-NFS daemon
/usr/sbin/rpc.pcnfsd

Why is rpcinfo: RPC_UNKNOWN_PROCEDURE ?
and why is rpcinfo: version 1 not available?

# rpcinfo -u localhost nlockmgr
program 100021 version 1 ready and waiting
rpcinfo: RPC_UNKNOWN_PROCEDURE
program 100021 version 2 is not available
program 100021 version 3 ready and waiting
# rpcinfo -u localhost status
program 100024 version 1 ready and waiting
#

I really appreciate your help. This is getting kind of deep!

[Dave Olker]

Hi Ken,

> lockd stops with a signal 15 in the log?

A signal 15 is a SIGTERM, which is the normal kill signal sent to processes if someone types "kill ". It appears something is terminating rpc.lockd.

Are you saying rpc.lockd gets terminated on the server every time you unmount a filesystem from the PC client? Obviously that shouldn't happen. Is that something you can reproduce - where rpc.lockd gets terminated when you unmount a filesystem from the PC?

If you can reproduce the problem then you might want to check with the vendor of the PC-NFS software to see if they know about this problem and can offer a solution.

Another possibility would be to disable file locking for these PC-NFS filesystems. Do you really need file locking on these PC-NFS filesystems or is the file locking just a byproduct of how Windows wants to grab a SHARE lock on any file it opens? If you don't have an explicit need for file locking, for whatever applications you're running on the PC against these filesystems, then you might want to see if your PC-NFS software offers a way to disable file locking. In many cases this is done with a command-line flag when the filesystem is mounted (/NS) or a checkbox item in the configuration screens.

The final suggestion would be - update to a supported release of HP-UX and test again. 10.20 has been out of support for years. If you're going to stay on 10.20 the only thing I could suggest would be to install patch PHNE_28886, as this is the last NFS patch we released for 10.20. That would at least guarantee that you have the last round of fixes before we stopped changing the 10.20 NFS code.


> Why is rpcinfo: RPC_UNKNOWN_PROCEDURE ?
> and why is rpcinfo: version 1 not available?

Version 1 is available. It's version 2 that we don't support/register. That's why you get the RPC_UNKNOWN_PROCEDURE message. This is expected behavior. Don't worry about it.

In any case, your PC-NFS client is sending in NLM Version 3 requests, which we do support and which your server is responding to appropriately.

The question is - when rpc.lockd is running and responding to requests does your PC-NFS client work as expected?

Regards,

Dave

[Ken Martin_3]

Dave,

Thanks for all your help!

Ever heard the term when it rains it pours?

The latest patch CD I have is:

CD 5012-7958
Kit 5064-11.34
September 2001
HP/UX 10.20

Well as luck would have it the HP can not read this CD. The CD drive LED keeps flashing. Swinstall returns the message 'Internal Error 7 in function 'IC_SourceGet''

I tried a much older patch CD from June 2000 and could read it except swinstall complained about the depot being empty!

You may be on to something about patches. This machine ony has the following installed:

XSW700CR1020 B.10.20.32 Extension Software Patch Bundle
XSW700GR1020 B.10.20.32 Extension Software Patch Bundle
XSW700HW1020 B.10.20.32 Extension Software Patch Bundle
Y2K-1020S700 B.10.20.B0315 HP-UX Core OS Year 2000 Patch Bun

AND

PHCO_10048 B.10.00.00.AA LVM commands cumulative patch
PHCO_10144 B.10.00.00.AA HP-UX grep/egrep/fgrep cumulative patch.
PHCO_10274 B.10.00.00.AA passwd(1) cumulative patch
PHCO_10428 B.10.00.00.AA Cumulative login patch
PHCO_10432 B.10.00.00.AA awk(1) cumulative patch
PHCO_10589 B.10.00.00.AA sort(1) cumulative patch
PHCO_10663 B.10.00.00.AA HP-UX cut (1) cumulative patch.
PHCO_10718 B.10.00.00.AA POSIX shell cumulative patch
PHCO_10719 B.10.00.00.AA ksh(1) cumulative patch

Last the patch you gave me PHNE_28886 does not appear on the HP/UX patch database? From what I could see that patch also had other required patches.

Something in the back of my mind is telling me to use 'GR' patches. Can't remember. It's been too long. Can't read CD's anyway...

Maybe it's finally time to haul this thing off to the dumpster unless I can find patch CD's that work and were the most current for HP/UX 10.20 on a B132L. Maybe on eBay? This machine just dosen't wanna die. If nothing else I have a very heavy door stop. It's a shame I have spare cards, drives, and memory for it...

Let me know if you have and final thoughts...

[Dave Olker]

Ken,

> Last the patch you gave me PHNE_28886
> does not appear on the HP/UX patch
> database? From what I could see that
> patch also had other required patches.

PHNE_28886 is the correct patch. The problem is the patch database no longer contains information about 10.20 patches because that release has been out of support for so long. Yes, there were dependent patches required along with 28886, but it doesn't really matter if you can't download them.

> Something in the back of my mind is
> telling me to use 'GR' patches.

"GR" just means General Release. In other words, official patch - not a pre-release or GR0 patch. Most customers prefer GR patches for their production environment unless they're experiencing a problem that is only fixed by a GR0 patch and they're willing to test the patch during the pre-release period.

> Let me know if you have and final
> thoughts...

Well, I'm curious about some of the points I raised in my last post. Namely:

1) Does the PC-NFS client work when rpc.lockd is running on the HP-UX system? At least for the initial mount before you unmount the share?

2) Can you reproduce the problem where rpc.lockd is terminated when you unmount the share?

3) Can you configure the PC-NFS client to not use file locking? If so, would this work for you or do your applications require file locking for some reason? In the test you've been doing (dir > file) there really isn't a need for locking the file, it's just a side-effect of how Windows opens files. If you don't need locking, can you disable it on the client?

Regards,

Dave

[Dennis Handly]

I can get it from:
ftp://hpatlse.atl.hp.com/hp-ux_patches/s700_800/10.X/PHNE_28886

The dependencies are (and a B132L is a s700?):
s700: 10.20: PHCO_26158 PHCO_20061 PHKL_16750 PHNE_19937
s800: 10.20: PHCO_26158 PHCO_20061 PHKL_16751 PHNE_19936

PHNE_19937 was replaced by PHNE_22506:
ftp://hpatlse.atl.hp.com/hp-ux_patches/s700/10.X/PHNE_22506

PHKL_16750 is there too.

ftp://hpatlse.atl.hp.com/hp-ux_patches/s700_800/10.X/PHCO_20061

PHCO_26158 was replaced by PHCO_31920:
ftp://hpatlse.atl.hp.com/hp-ux_patches/s700_800/10.X/PHCO_31920

[Ken Martin_3]

Dave, Dennis,

I was able to down the last workstation jumbo patch HP released for Dec 2001:

XSW700GR1020_10.20_700.depot

It's a monster. Got to free up disk space.

I will let you know how it works.

P.S. I don't think the NFS client understands no locking. When I tell the NFS session to ignore locking I can't access anything. Read or write.

I tried the same thing on a Linux box. Stopped the lock daemon and the NFS client did not work correctly.

I'll let you guys know if the jumbo patch fixes the problems.

Thanks...

[Ken Martin_3]

Dave, Dennis,

I did the depot install to B.10.20.55.3.

Didn't fix the 'create' problem.

Further testing revealed the following.

WRQ is now part of Attachmate.

Both WRQ NFS clients can update, read and delete files.

WRQ NFS client 7.x on NT 4.0 'can' create new files.

WRQ NFS client 10.x on XP Pro SP-2 or Win2K Pro 'can not' create files. It can update, read and delete files??

WRQ NFS client 7.x won't work on XP. Wonder if there is some 'free' NFS client that will run on XP?

Wondering through the WRQ support web site did not return any usefull information. Sounds like a bug in WRQ NFS client 10.x?

[Dave Olker]

Hi Ken,

I have no idea what "B.10.20.55.3" is. Can you tell me what NFS patch is installed with that bundle?

# swlist -l product | grep -i nfs

Depending on the bundle, you still might be running a comparatively old NFS patch.

As for comparing the two WRQ clients (aside from contacting WRQ and seeing if this is a known problem that they have a fix for) my only suggestion is to collect two network traces: one of the working client creating a file and one of the failing client trying to create a file. By comparing the traces we can try to understand why one client works and the other client fails.

Regards,

Dave

[Ken Martin_3]

Last available from HP downloads XSW700GR1020_10.20_700.depot
dated Dec. 2001 version B.10.20.55.3?
Product number: B3782XSW1

Heres what got installed:

# swlist -l product | grep -i nfs

NFS B.10.20 ONC/NFS; Network-File System,Information Services,Utilities

PHCO_15339 B.10.00.00.AA fsclean(1M) support for NFSv3

PHCO_15340 B.10.00.00.AA /sbin/is_local_root support for NFSv3

PHCO_15341 B.10.00.00.AA mountall(1M) support for NFSv3 and CacheFS

PHCO_15343 B.10.00.00.AA umountall(1M) support for NFSv3

PHKL_17573 B.10.00.00.AA ki/nfs vasusage, ACE 2 bundle cumulative patch

PHKL_20126 B.10.00.00.AA NFS diskless client kernel patch (ACE 199912

PHNE_24143 B.10.00.00.AA NFS/NIS General Release/Performance Patch

[Dave Olker]

You're running PHNE_24143. That is two patches back from PHNE_28886. There is only one rpc.lockd change between those patches but it is one where rpc.lockd dumps core.

Is rpc.lockd still dying on your 10.20 server? If so, getting the rpc.lockd from PHNE_28886 might still be a good idea. If lockd is no longer dying then I'd focus on the WRQ client.

Can you confirm whether rpc.lockd is still dying on the server during these tests from the WRQ clients?

[Ken Martin_3]

Dave,

Yes the 'lockd' is dying but only when the HP is accessed from the W2K machine running WRQ NFS client 10.x

It is configured the same as the client on a Win Pro machine.

However, the strange part is lockd appears to die when the W2K machine tries to create OR update a file? But, 'not' every time. When updating a file under W2K you get a message 'not able to create a temporary file'. This does not occur under XP Pro?

Looks like there might be a problem with WRQ NFS client 10.x on XP Pro and W2K. The latest is version 14 I think. To upgrade however you need to buy a new license $$.

Have you ever tried the NFS client from Microsoft's free Services For Unix 3.5? Uh, I think that won't work on W2K Pro.

I am beginning to think the HP is doing what it is supposed to be doing aside from lockd dieing at times when accessed by a W2K machine. As I said before an NT machine with WRQ NFS 7.x works as expected.

I wonder how NFS clients will work with Vista?

[Dave Olker]

Hi Ken,

If rpc.lockd is dying then there's not much we can ask from WRQ unless they're sending in a lock request that is somehow violating the NLM protocol.

As I said, your current rpc.lockd is 2 patches back from the last one we released for 10.20, and one of the fixes in the newer patch addresses cases where rpc.lockd core dumps with certain lock requests.

Since you have the ability to pull patches (from somewhere), I'd recommend the following patches:

PHCO_20061 mount(1M) cumulative patch
PHCO_23437 LVM commands cumulative patch
PHCO_23844 fsck_vxfs(1M) cumulative patch
PHCO_31920 libc cumulative patch
PHKL_16750 SIG_IGN/SIGCLD,LVM,JFS,PCI/SCSI cumulative patch
PHKL_16959 Physical dump devices configuration patch
PHKL_17857 Fix for mount/access of disc sections
PHKL_20610 Correct process stop recording on ufs inodes
PHKL_21660 lo_realvfs panic fix, Cumulative LOFS patch
PHKL_23418 VxFS mount(2) cumulative patch
PHKL_25438 LVM cumulative patch, array controller hot-swap
PHKL_26980 VxFS (JFS) mount, fsck, vx_real_iget deadlock
PHNE_22506 cumulative ARPA Transport patch
PHNE_28886 NFS/NIS General Release/Performance Patch

This is the latest NFS patch and all of the dependent patches, and any patches dependent on them, and so on.

Once you get those patches installed, if you are still able to reproduce the rpc.lockd core dump then I can look at a network trace to see what the offending SHARE request looks like to see if it's legal, but if anything needs to be done on the HP side you'd have to update to a supported OS release.

Regards,

Dave

[Ken Martin_3]

Dave,

If I can get these patches is there a certain order in which they have to be installed?

Thanks...

[Dave Olker]

Ken,

When I need to install a bunch of patches that have interdependencies, I typically swcopy all the patches into a single depot and do one big install.

Dave