NFS Shares from Package no longer getting mounted

[Brad Marshall_2]

Hi

This problem cropped up about 3 months ago and even though I recently recreated my cluster it is still happening.

1) On the secondary node the shares are not getting mounted by automounter. Primary is fine.
2) The hanfs file is working and the exports are all there and look legitimate.
3) When moving to where the mount point is I get a message permission denied.
4) Other nodes that use these shares have no problem mounting them.
5) When the primary is sharing the secondary can mount them no problem.
6) The minor numbers on the vg for NFS shares were the same on both systems. In my new cluster they were not and I will change this,but the problem is the same.

As stated this was working fine and about 3 months ago stopped working. No known changes to do with the vg's etc that I am aware of.

Your help is appreciated.
thanks
brad

[Steve Lewis]

Is this SAP? If so, there is a technote on how to set-up the Automounter.

I am not totally clear about what is mounting and what is exporting.

My preference for the automounter with MC/SG is actually to not use it - just manually exportfs, exportfs -u, mount and umount the filesystems in your package scripts. It saves a lot of trouble.
The problem with my method is that you can theoretically get into difficulty if the exporting server node TOCs and the clients lose their connections. But this may also happen with the automounter if a file transfer is in progress when the node TOCs. I can foresee some confusion regardless.

Why are you talking about minor numbers if its NFS? As long as the VG can be activated on all nodes then you should be OK as long as the clients know to disconnect when the package goes down. This is another argument for not using the automounter. It sounds like the node taking over a VG thinks it already has an NFS filesystem on the mountpoint (mounted the other way) and therefore refuses to mount the real filesystem onto the same point. You need to control this situation.

Try mounting it normally first to prove the network and that you are using the correct lan interface. Incidentally check the routing table, then the exports files and make sure that you are using the package hostnames, not the real ones.

[Brad Marshall_2]

HI

1) Yes I can get it to work by restarting nfs.server and nfs.client. This requires that the appropriate shares get added to the exports file, as nfs.server loses the HANFS shares.
2) I would rather not change anything by not using auto.mounter as this is how it was working and should continue to work. It works this way on the primary node no problem. This is production for a large company.
3) The Service guard NFS pdf B5140-90011 states the minor numbers must be the same especially for NFS. This was true in the past but the NFS permission error came up while this was still true.
4) This is for an SAP system, and the failover server is trying to mount its own mount points. Other systems mount the same shares no problem.

thanks

[Geoff Wild]

You should never stop/start NFS on a MC/SG node that has a NFS package - just causes problems.

You have 2 nodes in cluster and you want to mount shares on node NOT running the package?

Okay, here's what I do:

# cat auto_master
/- /etc/auto.direct proto=tcp

# cat auto.direct
/usr/sap/trans pctransf:/export/usr/sap/trans
/sapmnt/IPR prodbf:/export/sapmnt/IPR
/data/sap/IPR/ARCHIVE prodbf:/export/data/sap/IPR/ARCHIVE

In hanfs.sh, make sure access is granted, for example:

XFS[0]="-o access=node1f:node2f:svr110f:svr111f:svr112f:prodbf:procif:qadbcif:pctransf:svr1000:svr1005,root=node1f:node2f
:svr110f:svr111f:svr112f:prodbf:procif:qadbcif:pctransf:svr1000:svr1005 /export/usr/sap/trans"

Rgds...Geoff

[Brad Marshall_2]

Hi
1) It was only stopped and started for trouble shooting while the app was not running.
2) No I am trying to mount the shares on the server running the package. All other servers can mount no problem.
3) My setup is pretty much the way you have it. auto_master is /- /etc/auto.direct and hanfs is XFS[0]="-o root=hpusam06ci:hpusam06:pa2ms:pa2ci:
4) Both node names, the alias and the real one are in the hanfs which is probably not a good thing.

thanks

[Geoff Wild]

Okay - looks good so far, for #3, mounting on server that has package - make sure the server (real host name) does the mounting from the floating ip...

IE, I have svr0003, floating name for nfs package is pctransf, different subnets:

# mount |grep trans
/usr/sap/trans on /etc/auto.direct ignore,direct on Mon Dec 6 19:15:17 2004
/export/usr/sap/trans on /dev/vg55/lvtrans log on Sat Dec 11 19:05:28 2004
/usr/sap/trans on pctransf:/export/usr/sap/trans nodevs,rsize=32768,wsize=32768,NFSv3 on Sat Jan 8 11:36:40 2005

pctransf: 192.168.59.34
svr003: 192.168.176.16

Have you done any NFS patching lately?

Rgds...Geoff

[Brad Marshall_2]

1) Yes auto.direct is mounting the NFS filesystem to the alias name.
2) I am also wondering if patching broke something but have yet to figure out what if anything changed.
3) We did have a project to tighten up some security and again I have not figured out if this caused some problems.
thanks

[RAC_1]

Send a -usr2 signal to autmountd. This will put it in debug mode and will create a log file under /var/adm/syslog. Looking at debuf file, will give some information on what to do.

Anil

[Brad Marshall_2]

Yes tried that the other night while troubleshooting. Here is the automount log, but I dont understand what it means.

Jan 8 20:23:25 READLINK call(fh=[4528, 1103479600, 7])
Jan 8 20:23:25 lookup() dir->dirname /sapmnt/PA2, name /sapmnt/PA2
Jan 8 20:23:25 "/sapmnt/PA2 pa2ms:/export/sapmnt/PA2"
Jan 8 20:23:25 /sapmnt/PA2/ /sapmnt/PA2 ()
Jan 8 20:23:25 / -rw pa2ms:/export/sapmnt/PA2
Jan 8 20:23:25 ping 172.17.20.244 Jan 8 20:23:25 OK
Jan 8 20:23:25 READLINK return(stat=NFSERR_ACCES)

Seems like an access problem, but why?

[Geoff Wild]

Sounds like it is trying to mount read write - does it have access?

Is AUTOFS=1 in /etc/rc.config.d/nfsconf

Can you post your auto.direct?

Rgds...Geoff

[Brad Marshall_2]

HI
1)nfsconf on both primary and secondary nodes have AUTOFS=0, everything works on the primary server.

2) auto.direct is
hpusam06$ more auto.direct
/sapmnt/PA2 pa2ms:/export/sapmnt/PA2
/sapmnt/CA2 hpusam06:/export/sapmnt/CA2
/usr/sap/trans pa2ms:/export/usr/sap/trans
/var/archive/PA2 pa2ms:/export/var/archive/PA2
/var/archive/CTS pa2ms:/export/var/archive/CTS

3) It is these pa2ms mount points that are not working. pa2ms is the floating hostname and hpusam06 is the failover node with the problem.
4) The package is now running on a box called hpusam10 and these shares are available no problem. It just cant access them when hpusam06 is the owner.

Current mount
hpusam06$ bdf .
Filesystem kbytes used avail %used Mounted on
pa2ms:/export/sapmnt/PA2
8388608 5319094 2946194 64% /tmp_mnt/sapmnt/PA2
hpusam06$

[Geoff Wild]

Okay, at a later date, you should move to AUTOFS=1 = beter reliability - but not now - wait til your issue is fixed first...

Let me get this straight - when running on hpusam10, all nodes (including hpusam10) can access the shares, but when running on hpusam06, hpusam06 can not mount them...

This can cause you some grief - mounting back through the same interface:
/sapmnt/CA2 hpusam06:/export/sapmnt/CA2

Make sure auto.direct is same on all nodes, as well /etc/hosts and your /etc/cmcluster files.

Do you run CCMON?

http://h71028.www7.hp.com/enterprise/cache/10973-0-0-225-121.aspx

Rgds...Geoff

[Brad Marshall_2]

Hi

Yes when the package is running on hpusam06 it cannot mount the hanfs shares. They get mounted from the pa2ms failover hostname. Which is itself. These are the PA2 mountpoints the CA2 ones are what normally run on hpusam06, when the package is not there.

Autodirect and hosts looks consistent. I can only guess that somehow hpusam06 has not forgotten the old nfs shares and gets confused when the new ones are presented?

[Geoff Wild]

Okay - I can't believe it worked before - but NFS on HP-UX works in mysterious ways...

I think what you have to do is, when you move the package to hpusam06, you need to un-export the CA2 ones first - prior to starting up the package.

In your "customer defined run commands"

exportfs -u /sapmnt/CA2

That should stop hpusam06 from being confused...

It should be more eloquent then that - check to make sure it un exports...etc...

NFS is definately confused - unfortunately, from experience, only way I have seen it clean up is with a reboot :(

Rgds...Geoff

[Brad Marshall_2]

Hi Geoff
I dont think the CA2 mounts have anything to do with the problem. There is no reason why hpusam06 cannot continue to have these mounted as well as the newer and previosly mounted pa2ms:/export/sapmnt/PA2.

I think the problem is that when the package stops on hpusam10 and the pa2ms:/export/sapmnt/Pa2 mounts are no longer available and then when they do become availabe again when pa2ms is now hpusam06 (package failover)automounter cant figure out what happened e.g. why is pa2ms now somewhere else, it is holding onto something related back to the old hpusam10 server?

As mentioned above what I did do was stop nfs.server and nfs.client and once the entries were included in the exports file,all worked. I am really curious if just restarting nfs.client would have solved the problem.

This being a production box I cannot test except in a controlled window. I would like to have some idea what to try in the next window. e.g. if restarting nfs.client does no good, but restarting nfs.server fixes everything (or exportfs -a) then I know re exporting the shares solves the problem, but why does hanfs.sh not accompish the same thing.

thanks
brad

[Steve Lewis]

(long shot) arp ? I think that HANFS would be thoroughly tested for this sort of thing.

The fact that bouncing NFS.server fixes it makes me think you should check your NFS versions. Its better to use TCP than UDP and I would urge you to upgrade it to the latest version of NFS if possible (having read Dave Olkers postings).
Does this server have the same patch level as the others? This includes MC/SG patches and version as well as NFS versions.

You mentioned the no access error message and said you had tightened up security. Are you blocking a port on that server somehow?

You said that other nodes can mount it (but not itself), plus this node can mount to the package when it running on the primary node, but when both packages run on another node can those other nodes mount back to themselves?

[Todd Whitcher]

Hello Brad,

The errors definately indicate a permission or access issue.

Jan 8 20:23:25 READLINK return(stat=NFSERR_ACCES


What version of HPUX are you running?
What version of NFS toolkit are you running?
What is your ONC cumulative patch level?

There were some older fixes in NFS code for a looping problem win automount when using replicated servers (Serviceguard)

As far as t-shooting what I would suggest is while this fails you get debug from mountd and automount and run a quick nettl trace to see what/where your mount requests are doing/going.

Also, check the permissions on the underlying mount point your using for that filesystem locally on the server. I mention this because you said you had issues w/ the major and minor numbers that needed to be corrected. I've seen in the past where the permissions on the actual underlying mount point are incorrect and cause this type of problem.


You crank up mountd debug the same way you do automount debug, toggle it on/off with a kill -17 to the running PID. It logs to /var/adm/mountd.log

Here is a quick nettl trace command you can use.

# nettl -tn pduin pduout -e ns_ls_ip -s 1024 -tm 99999 -f trace

Reproduce your problem

# nettl -tf -e all

You will have a trace.TRCXXX that you can format out.

You would want to use a filter file so you only get interesting trafic.

# cat /filter
filter rpcdirection call
filter rpcdirection reply

Then format it out, you can also use ethereal to read it which is much nicer to look at.

netfmt:
# netfmt -Nnlf trace.XXXX -c filter > trace.out

Or download ethereal if you dont have it and read it. www.ethereal.com.

Let me know your version information etc. and I'll do a quick check on known defects for you.

Hope this helps,

Todd

[Brad Marshall_2]

Yes the primary node can mount back to itself no problem, using exactly the same method and hanfs script etc.

I am trying to check the NFS patches etc and so far nothing has changed for well over a year. It stopped working a few months back.

Any suggestions on security areas to check and what types, that may cause this kind of problem?

[Brad Marshall_2]

HI Todd

Hopefully I am giving you the right information.

HP UX- 11.00
NFS 11.00
ONC PHNE_28567

thanks
brad

[Todd Whitcher]

That patch is from 2003/04/29 The Patch PHNE_31096 is the replacement. There are a lot of defects fixed but I dont see anything directly related to your problems.

Hopefully the debug and network trace may help.

I would generally recomend and update from the patch you have installed since it has some warning associated with it and later patches fix several panic situations.

Todd

[Brad Marshall_2]

HI Todd
The server that works has a newer patch on it
PHNE_28982. But I am going to go to the recomended patch PHNE_30660 on both. I believe they have both been at this state for a while, but it is worth a try.

thanks
brad

[Brad Marshall_2]

HI

I had a chance to test the package again and it worked. I did two things.
1) Used the same minor number on the NFS VG.
This was the case before my new disk and vgs were created and I had the NFS problem.
2) Added the floating ip name and address to the host file on the failover node. For what ever reason this was never there and used DNS. I did not detect any problem with DNS when troubleshooting but added entry to host file and problem is gone for at least two tests. Maybe DNS had trouble re-pointing itself?

thanks for your help
brad

[Brad Marshall_2]

Host file entry instead of using DNS on the failover server.