NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

[Bishwajit Kumar]

Hi,

I am using HP-DP 6.11 MA/DA on HP-UX 11.31 IA.

I have a NFS Export (from DeDup Appliance) mounted on my HP-UX system (over 10G NIC). Configured File Library with 8 Writers (for 8 Streams); now when I start my backup it works fine at least for 5 minutes (i.e. AutoFS idle time) then suddenly I see no progress at all. And finally my backup fails after timeout (120/140Min):


What i notice is:
while backup is going I can see my HP-UX box as active client on my NFS Server (DeDup Appliance). As soon as my backup hangs i see HP-UX Box (Client) becomes inactive on NFS Server.


This clearly indicates that, mount point is becoming inactive after 5 min (after manually doing any operation on the NFS Mount on client side). What is surprising me is that - HP DP is writing data to the NFS shares actively (i am monitoring the traffic and Disk IO all the time) and NFS mount becomes inactive. !!!! Why???
#netstat -I lan2 1
#iostat -t 1



--Environment details--


<>

---------
# bdf
Filesystem kbytes used avail %used Mounted on
orca45:/data/col1/qaia48
45438690816 10076493312 35362197504 22% /orca45
#
# mount
/orca45 on orca45:/data/col1/qaia48 llock,rsize=32768,wsize=32768,NFSv3,dev=4000002 on Mon Sep 20 17:03:28 2010
#
---------
# ndd -get /dev/tcp tcp_recv_hiwater_def
262144
# ndd -get /dev/tcp tcp_xmit_hiwater_def
262144
#

Modified /etc/rc.config.d/nfsconf
AUTOFS=0
NFS_TCP=1

-------------
DP Eror:

[Major] From: BSM@qaiacm1.chaos.local "QAIA48_HP-UX11.31_DD890_FileLib2" Time: 9/18/2010 5:46:31 PM
[61:1002] The BMA named "DD:890_qaia48_FileLib2_Writer8" on host qaia48.chaos.local
reached its inactivity timeout of 8400 seconds.
The agent on host will be shutdown.

[Critical] From: BSM@qaiacm1.chaos.local "QAIA48_HP-UX11.31_DD890_FileLib2" Time: 9/18/2010 5:47:02 PM
None of the Disk Agents completed successfully.
Session has failed.

-------------

[Hakki Aydin Ucar · Accepted solution]

Hi,
in generally, NFS timeouts on a file system, increase the number of nfsd processes daemons that handle file system requests--
in /etc/rc.config.d/nfsconf ;
like NUM_NFSD=16

[Bishwajit Kumar]

Thanks "Hakki Aydin",

Unfortunately this did not help.

Anyone: Please provide me some solution?

Thanks,

[Patrick Wallek]

If this mount point is this important, take it out of the AUTOFS configuration and mount it manually. That way you will not be subject to the autofs timeouts.

[Bishwajit Kumar]

Hi Patrick Wallek,

I am not using AutoFS and I don't want to. I am mounting the export manually. and that's why i have disabled the AUTOFS (set it to 0/disable)

But this it doesn't help.

Please let me know the steps - you want me to follow?

Thanks,

[Dave Olker]

Can you try accessing this remote device outside of your backup application? In other words, can the 11.31 NFS client write data to the appliance using cp, iozone, etc. for longer than 5 minutes?

Dave

[Bishwajit Kumar]

Very good point Dave,

YES It's more then 15 minutes and i am copying big data set (of 280G) to my NFS Mount is still in progress. I do see my NFS Client is still active on my DeDup Appliance. Annd iostat ane netstat is showing IO/trafic.

So sounds like it problem with HP Data Protector? What that could be? May be some kind of locking issue by HP-DP???

Any insight?

Thanks for your suggestion...

[Bishwajit Kumar]

BTW, right now "/etc/rc.config.d/nfsconf" has following entries:

-------------------
NFS_CORE=1
RPCBIND_OPTIONS=""

LOCKMGR=1
LOCKD_OPTIONS=""
STATD_OPTIONS=""

NFS_CLIENT=1

NFS_SERVER=1
PCNFS_SERVER=0
START_NFSLOGD=0
START_MOUNTD=1
MOUNTD_OPTIONS=""

AUTOFS=0
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""
AUTO_MASTER="/etc/auto_master"

NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4 <>
-------------------------------

[Dave Olker]

A couple of things:

1) There are some lines in your nfsconf file that don't belong there for an 11i v3 box:

NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4

None of those lines should be there. They are all obsolete. The number of nfsd threads is set in the /etc/default/nfs file via the NFSD_SERVERS variable, the number of async I/O threads is now a per-mount tunable set via the kctune parameter nfs3_max_threads, and the NFS_TCP parameter is just plain gone. We've supported NFS over TCP for many releases now so that tunable doesn't belong there. I'd suggest removing all of these from your nfsconf file.

2) If you think the problem is related to file locking, I noticed you're mounting the filesystem with the "llock" option. That parameter causes the NFS client to not send lock requests to the server. Perhaps that is confusing the appliance, as it might be expecting to receive lock requests and gives up waiting after 5 minutes. Just a guess because I don't know anything about Data Protector or your appliance, but I thought it was worth mentioning since you suspect a file locking issue.

You might try re-mounting the filesystem without the "llock" option and see if you get different behavior.

Regards,

Dave

[Bishwajit Kumar]

Thanks Dave,

Thanks for the advice;

With referance to your comment;
--------------------------------------
1) There are some lines in your nfsconf file that don't belong there for an 11i v3 box:
NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4
None of those lines should be there.
--------------------------------------

I tried to eleminate these lines from my "/etc/rc.config.d/nfsconf" file but it did not get any better. Look like these variables are still having impact on the NFS Demon. So i decided to verify all these variables by eliminating one by one.


My findings are here:
"/etc/rc.config.d/nfsconf"
-------------------
NFS_CORE=1
RPCBIND_OPTIONS=""

LOCKMGR=1
LOCKD_OPTIONS=""
STATD_OPTIONS=""

NFS_CLIENT=1

NFS_SERVER=1
PCNFS_SERVER=0
START_NFSLOGD=0
START_MOUNTD=1
MOUNTD_OPTIONS=""

AUTOFS=0
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""
AUTO_MASTER="/etc/auto_master"

NFS_TCP=1
---------------------------------

Scenario#1
With the above entris: (including AUTOFS=0 or AUTOFS=1)
Backup *Hangs* every time after 5 minute (aprox) from the time it starts.

-------------------------------

Scenario#2
With: NUM_NFSD=16 # Added this line

Backup started at: Time: 9/22/2010 7:48:27 AM
Backup hanged at : Time: 9/22/2010 8:32:41 AM
Backup hannged about in 44 Minutes later

------------------------------
Scenario#3
With: NUM_NFSD=16 and # Added this line
NUM_NFSIOD=4 # Added this line

Backup Started at : Time: 9/22/2010 12:10:18 AM
Backup completed at: Time: 9/22/2010 4:42:51 AM

Backup and CP -r command never hanged; I was able to backup or/and copy 1TB data
------------------------------

BTW what does NUM_NFSIOD=4 and NUM_NFSD=16 do?

Thanks,