autofs/automount issues when changing IP of NFS server

[Mark van Hassel]

Hi,

We have the follwing situation:

A 2-node cluster setup consisting of serverA and serverB and 1 package pkg. This package exports 2 file systems: /export/usr/sap/trans/SID and /export/sapmnt/SID. DNS is used for resolving and we use fully qualified names in the NFS and automount configuration.

Before going live we have to change the package IP address and we do the following:
- halt package
- remove automount entries from /etc/auto.direct on both systems and run automount
- change IP in DNS
- change IP in package control script
- start package

After this when I try to access the automount file systems I get the error "unreadable". I am able to mount the exported file system using "pkg:/export/sapmnt/SID /dir" so I have the feeling it's an automount issue. Also, when I kill and restart the automount deamon it works too, but I dont want to do this because it creates loopback mounts and because other automounts are active on the systems. I know there's a product "enhanced autofs" to force NFS mounts instead of loopback, but a reboot is required.

To make matters worse: when I change back the package IP address to it's original IP and do not change the DNS entry the systems are able to access the automount file systems ! Which for me means that the automounter keeps track of resolving itself instead of using DNS.

Does anyone know how to resolve this issue without rebooting or killing the autmounter ?

Thanks.

[Massimo Bianchi]

Looks like SGeSAP...

I would check the following:

- check /etc/hosts, and check if any of the hosts i declared there

- check the /etc/cmcluster/SID/xnfs.cntl of its name, to see of there you specified the IP or the fully qualified name

- check the arp entry with "arp -a", maybe they have been cahed on one of the server

- check if all the clients (SAP application) resolve the name in the proper way

- check you can stop/start the automount itself, issuing /sbin/init.d/nfs.client stop|start . Before rebooting, it's a good chance. KEEP IN MIND that nfs.client stop/start MUST BE DONE non on the exporting server, or with with the package up, becuse it will restart the nfs.server part.

And , when you stop the package, SAP will be down in any case, since you are tearing away it's executables, so a reboot could be the right way in any case.

HTH,
Massimo

[Mark van Hassel]

Thanks for the reply. All lookups/resolving are ok. I'm using FQN everywhere, no entries in /etc/hosts.

Like I said: I can NFS mount the exported file systems without the automounter.

I'm pretty sure automount or rpc keeps a map somewhere, since when I change the package IP back to it's original, without changing DNS I can still automount the file system. This also works when I use an IP instead of FQN in /etc/auto.direct.

I cannot stop the automounter because loopback mounts will be created (enhance autofs is not installed) and I can't reboot the system since more packages are active).

As a bypass I can use the IP address instead of the FQN in /etc/auto.direct and it will automount with no problem.

[Mark van Hassel]

Also, I get the following error when I turn automount tracing on:
duplicate request, xid 1073746576vers=3 NO RESPONSE


When I use rpcinfo to ping the remote mount deamon I have no problems:

program 100005 version 1 ready and waiting
rpcinfo: RPC: Program/version mismatch; low version = 1, high version = 3
program 100005 version 2 is not available
program 100005 version 3 ready and waiting

[Massimo Bianchi]

May be the server is a DNS cliant the same..

Was the DNS slave refreshed ?

Did you checked the arp cache ?

Otherwise, you could use the IP, like you said, but i think that you may encounter problem when it comes to authorizations....

Let's summary:


mount IP:/export... /mount_point WORKS
mount LOGICAL:/export... /mount_point ERROR

Is this right ?


Did you try flushing the automount ?

Once i had a similar problem, and i used a manual flush procedure:


mount -v
mount -v # refresh state in /etc/mnttan
> /etc/auto.direct # zeroes the automounts
automount
automount
automount # three time, because some kernel structures are not updated properly in time

After this operation, check the content of the /etc/mnttab:

cat /etc/mnttab

cp /myfile /etc/auto.direct # set back my mountpoints...

aumount # let's use them

After this operation, check the content of the /etc/mnttab:

cat /etc/mnttab

It may worth a try..

HTH,
Massimo

[Mark van Hassel]

-Did you checked the arp cache ?

Yes, I make sure it has no entry for the package


- Let's summary:

mount IP:/export... /mount_point WORKS
mount LOGICAL:/export... /mount_point ERROR
Is this right ?

YES, also: I can mount it using the FQN and using the normal mount command instead of the autmounter.


- Did you try flushing the automount ?

mount -v
mount -v # refresh state in /etc/mnttan
> /etc/auto.direct # zeroes the automounts
automount
automount
automount # three time, because some kernel structures are not updated properly in time

DONE

-After this operation, check the content of the /etc/mnttab:

cat /etc/mnttab

NO ENTRIES of the automounts

-cp /myfile /etc/auto.direct # set back my mountpoints...

-aumount # let's use them

After this operation, check the content of the /etc/mnttab:

cat /etc/mnttab

ENTRIES back in mnttab

It may worth a try..

Does not work, same issue:

After a minute or so I get the error "unreadable".
Messages in var/adm/automount.log :
Dec 14 15:05:34 (thread 356) MOUNT REQUEST : name=/sapmnt/SID map=/etc/auto.direct opts= path=/sapmnt/SID
Dec 14 15:05:34 (thread 356) PUSH /etc/auto.direct
Dec 14 15:05:34 (thread 356) POP /etc/auto.direct
Dec 14 15:05:34 (thread 356) nfsmount: pkg.xxx.zzz:/export/sapmnt/SID /sapmnt/SID proto=udp,bg
Dec 14 15:05:34 (thread 356) ping: pkg.xxx.zzz timeout=15 Dec 14 15:06:34 (thread 1) ignored.
: duplicate request, xid 1073746576vers=3 NO RESPONSE
Dec 14 15:06:49 (thread 356) Couldn't mount pkg.xxx.zzz:/export/sapmnt/SID
Dec 14 15:06:49 (thread 356) MOUNT REPLY : status=2

rpc ping:
program 100005 version 1 ready and waiting
rpcinfo: RPC: Program/version mismatch; low version = 1, high version = 3
program 100005 version 2 is not available
program 100005 version 3 ready and waiting

[Massimo Bianchi]

Hi,
unfortunatly looks like you are in the case decribed by path PHNE_30380, and going towards enhanced autofs might help...

One last thing you could try: new mount options ?

These are the options on one of my server:

-retry=5,rsize=32768,wsize=32768,retrans=10

Just a last stab...

HTH,
Massimo

[Mark van Hassel]

Installing the patch did not help, I am still having the same problem.

I am testing the enhanced autofs product at the moment. This seems to be workinh, however it introduces other issues.
In all documentation about the automounter I read that you should never kill the automountd, but always use /sbin/init.d/nfs.client stop and start. But when you're systems is NFS server and client, it will stop/start the NFS server too, which means your exports from your package are lost.
This is not a big issue when running a single package on a system, but it is when your running multiple packages with exports.

[Dave Olker]

Hi Mark,

I think you should definitely migrate to Enhanced AutoFS. There are so many benefits to this new version that I'd recommend it to anyone and everyone. In fact, as time goes on you'll find the lab less and less willing to even investigate problems with the older AutoFS on 11.11. The first response will undoubtedly be "if you can reproduce the same problem on Enhanced AutoFS then we'll look at it."

As for stopping AutoFS, I agree that the whole NFS client/server startup/shutdown logic is flawed. We're redesiging it in a future release and I've been on the review committee for the new design.

The plan for 11i v3 (if not sooner) is to break AutoFS out into it's own startup/shutdown script that can be run independently. All that script will do is unmount any AutoFS filesystems (i.e. /usr/sbin/umountall -F autofs >/dev/null 2>&1) and then terminate automountd.

My guess is the current documentation requests you to use the startup/shutdown scripts in order to avoid any problems associated with forgetting to unmount the AutoFS filesystems prior to terminating automountd. However, I don't think it would be a problem to manually unmount all AutoFS filesystems (using the above command) and then manually terminate automountd if you desire to do so.

Let me know if you have any questions about this.

Regards,

Dave

[Dave Olker]

One other thing - since it sounds like you're using AutoFS on your HA/NFS servers, be sure to use the new -L option to avoid any LOFS mounts that could potentially tie up your HA/NFS packages and cause a package migration to fail.

Regards,

Dave

[Mark van Hassel]

Thanks for the reply Dave.

When I said "Enhanced autofs" seems to be working, I meant it solved the problem only by stopping and starting the nfs client (I already knew this would help since I tested it before). Enhanced autofs only solves the lofs issue.

Another interesting issue: My HP support engineer did a test to reproduce the issue. He used 2 Itanium A and B(11.23) and 1 pa-risc system(11.11). Changing IP addresses of the exporting system was no problem on the Itanium systems (it mounted the correct file system), but he encountered the same problem as I (mounting the "old" file system when IP changed to new).