Recommendation on NFS in a SAP/MC/SG Environment

[Geoff Wild]

Don't know how to say this - but NFS on HP-UX is a pain.

Here's my environment:

3 SAP Landscapes - DEV, QA, Prod.

Prod is running on a 2 node 11.14 MC/SG cluster and 3 App servers.
DEV and QA on their own servers.
All servers are 11i.

All servers NFS mount:
/export/usr/sap/trans

APP servers and PROD servers also mount:
/export/sapmnt/PROD
/export/APPL
/export/usr/sap/xfer
/export/data/sap/PROD/ARCHIVE

This is all done with auto maps:

# cat /etc/auto.direct
/usr/sap/trans prtrans:/export/usr/sap/trans
/sapmnt/PROD prdbf:/export/sapmnt/PROD
/data/sap/PROD/ARCHIVE prdbf:/export/data/sap/PROD/ARCHIVE
/APPL prcif:/export/APPL
/usr/sap/xfer prcif:/export/usr/sap/xfer


AUTOFS=1 in all /etc/rc.config.d/nfsconf

The trans export is in it's own package. The other exports happen in the prod package.

Here's the output of exportfs on the trans:
# exportfs
/export/usr/sap/trans -
access=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:qadbcif:pransf:pc1000:pc1005:srv031:srv032:srv041:srv042,root=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:pransf:qadbcif:srv1000:srv1005:srv031:srv032:srv041:srv042

Every now and then - every couple of months or so, NFS hangs...

NFS server prcif not responding still trying

No matter what I do, stop/start nfs client, etc....the final solution is to reboot all the servers...this happend last Wed night - had to reboot 3 App servers, 1 QA server, 1 Dev server, and the first node of the cluster (I moved the PROD packages to the alternate node).

Can anybody tell me a better way to configure NFS? This is Unix after all - not Windows....I should NOT have to reboot....

Thanks...Geoff

[melvyn burnard]

So are you saying you have Serviceguard and are using the SGeSAP add on product? If so, are you using the HA NFS Toolkit as well, as this is the correct method of configuring and using the NFS setup for SGeSAP.
Take a look at
http://docs.hp.com/hpux/pdf/B5140-90012.pdf
and
http://docs.hp.com/hpux/pdf/B7885-90018.pdf

for further information.

[Jeff Schussele]

Hi Geoff,

I hear what you're saying - in my book NFS stands for Not Friggin Stable.

What you can do depends on what the cause of the hangs usually is. If someone's booting the NFS server w/o umounting the NFS mounts, then that's an admin education issue.

If it's because rps.lockd hangs periodically then I'd say stay *very* current on NFS cumulative patches & even consider running NFS over TCP as opposed to UDP. May be a slight bit slower - but it'll be more stable.

Rgds,
Jeff

[Marco Santerre]

I know what you mean Geoff.. been having similar trouble with MC/SG/SAP/NFS..

Jeff, the problem is, you cannot unmount all NFS mounted directory, cause the package itself depends on some of them as well. And once the package is down.. well now you're stuck with a NFS hang until the package is up again.

I know I digress from Geoff's problem as it seems more of a flaky NFS because it seem sto be happening even while the package is up.

So I'd have to agree.. keep the patches up to date.. and make sure you use HANFS

[Volker Borowski · Accepted solution]

Hi,

had the same problem on Solaris, and as already suggested, switching from udp to tcp helped a lot.

Unfortunately it was no real solution. Still trouble from time to time, but less often (two, three times a year).

Hope this helps
Volker

[Geoff Wild]

Ran this on one of the APP servers - which was rebooted 5 days ago - quite a lot of retrans and badcalls...

# nfsstat -c

Client rpc:
Connection oriented:
calls badcalls badxids
44129 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
2157561 0 4206
badxids timeouts waits
16 4205 0
newcreds badverfs timers
0 0 11265
toobig nomem cantsend
0 0 0
bufulocks
0

Client nfs:
calls badcalls clgets
2201474 0 2201473
cltoomany
0
Version 2: (0 calls)
null getattr setattr
0 0% 0 0% 0 0%
root lookup readlink
0 0% 0 0% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 0 0% 0 0%
Version 3: (2201474 calls)
null getattr setattr
0 0% 117838 5% 9812 0%
lookup access readlink
1141059 51% 184603 8% 0 0%
read write create
227700 10% 239337 10% 30602 1%
mkdir symlink mknod
1 0% 2 0% 0 0%
remove rmdir rename
44540 2% 1 0% 17910 0%
link readdir readdir+
0 0% 14712 0% 34704 1%
fsstat fsinfo pathconf
6528 0% 25 0% 599 0%
commit
131501 5%


Just so you know, we do have ha-nfs and the SAP SG extentions...

Read the above docs - nothing new there...

Currently reading:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,952,00.html

As far as udp/tcp - I thought NFS on 11i uses tcp by default - yet when I run:

# rpcinfo -p
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100024 1 tcp 61801 status
100024 1 udp 63019 status
100021 1 tcp 61802 nlockmgr
100021 1 udp 63024 nlockmgr
100021 3 tcp 61803 nlockmgr
100021 3 udp 63025 nlockmgr
100021 4 tcp 61804 nlockmgr
100021 4 udp 63026 nlockmgr
100020 1 udp 4045 llockmgr
100020 1 tcp 4045 llockmgr
100021 2 tcp 61805 nlockmgr
100005 1 udp 63030 mountd
100005 3 udp 63030 mountd
100005 1 tcp 61806 mountd
100005 3 tcp 61806 mountd
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs

- looks like udp and tcp....

Points will be assigned later - once I determine how much your answers have helped me...

Olympians and NFS guru's - feel free to post :)


Rgds...Geoff
Rgds...Geoff

[Geoff Wild]

So, it looks like I need to modify my hanfs.sh files from:

XFS[0]="-o access=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:qadbcif:pransf:srv1000:srv1005:srv031:srv032:srv041:srv042,root=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:pransf:qadbcif:srv1000:srv1005:srv031:srv032:srv041:srv042 /export/usr/sap/trans"

to:

XFS[0]="-o access=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:qadbcif:pransf:srv1000:srv1005:srv031:srv032:srv041:srv042,root=srv003f:srv004f:srv110f:srv111f:srv112f:prdbf:prcif:pransf:qadbcif:srv1000:srv1005:srv031:srv032:srv041:srv042,proto=tcp /export/usr/sap/trans"

Rgds...Geoff

[Jeff Schussele]

Hi Geoff,

It should be noted that *both* ends need to be using TCP or it will fall back to UDP.

Rgds,
Jeff

[Brian Hackley]

Geoff,

To find out how the NFS mounts are being done (proto=tcp or proto=udp), and which version 2 or 3), run "nfsstat -m", which will give you the infos from the perspective of an NFS Client. Note the command does not give statistics for access times on TCP mounts, only for UDP. For example a TCP NFS PV3 Mount:
/net/vbox/tmp/nfs_tcp from vbox:/tmp/nfs_tcp (Addr 15.17.185.250)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,rsize=32768,wsize=32
768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

You should check the NFS kernel tunings per Dave Olker's book, "optimizing NFS performance", http://www.hp.com/hpbooks ISBN #0-13-042816-7. This is VERY important!

Finally, ensure that your NFS patching is the LATEST NFS patches! PHNE_29211 plus all dependencies. Contact HP Support with ITRC or phone for the latest list.

Hope that helps,

-> Brian Hackley

[Marco Santerre]

I may be wrong but I was under the impression that under 11i, NFS defaulted to proto=tcp
Am I wrong?