System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

SOLVED
Go to solution
Bishwajit Kumar
Frequent Advisor

NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi,

I am using HP-DP 6.11 MA/DA on HP-UX 11.31 IA.

I have a NFS Export (from DeDup Appliance) mounted on my HP-UX system (over 10G NIC). Configured File Library with 8 Writers (for 8 Streams); now when I start my backup it works fine at least for 5 minutes (i.e. AutoFS idle time) then suddenly I see no progress at all. And finally my backup fails after timeout (120/140Min):


What i notice is:
while backup is going I can see my HP-UX box as active client on my NFS Server (DeDup Appliance). As soon as my backup hangs i see HP-UX Box (Client) becomes inactive on NFS Server.


This clearly indicates that, mount point is becoming inactive after 5 min (after manually doing any operation on the NFS Mount on client side). What is surprising me is that - HP DP is writing data to the NFS shares actively (i am monitoring the traffic and Disk IO all the time) and NFS mount becomes inactive. !!!! Why???
#netstat -I lan2 1
#iostat -t 1



--Environment details--


<>

---------
# bdf
Filesystem kbytes used avail %used Mounted on
orca45:/data/col1/qaia48
45438690816 10076493312 35362197504 22% /orca45
#
# mount
/orca45 on orca45:/data/col1/qaia48 llock,rsize=32768,wsize=32768,NFSv3,dev=4000002 on Mon Sep 20 17:03:28 2010
#
---------
# ndd -get /dev/tcp tcp_recv_hiwater_def
262144
# ndd -get /dev/tcp tcp_xmit_hiwater_def
262144
#

Modified /etc/rc.config.d/nfsconf
AUTOFS=0
NFS_TCP=1

-------------
DP Eror:

[Major] From: BSM@qaiacm1.chaos.local "QAIA48_HP-UX11.31_DD890_FileLib2" Time: 9/18/2010 5:46:31 PM
[61:1002] The BMA named "DD:890_qaia48_FileLib2_Writer8" on host qaia48.chaos.local
reached its inactivity timeout of 8400 seconds.
The agent on host will be shutdown.

[Critical] From: BSM@qaiacm1.chaos.local "QAIA48_HP-UX11.31_DD890_FileLib2" Time: 9/18/2010 5:47:02 PM
None of the Disk Agents completed successfully.
Session has failed.

-------------
45 REPLIES
Hakki Aydin Ucar
Honored Contributor
Solution

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi,
in generally, NFS timeouts on a file system, increase the number of nfsd processes daemons that handle file system requests--
in /etc/rc.config.d/nfsconf ;
like NUM_NFSD=16
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Thanks "Hakki Aydin",

Unfortunately this did not help.

Anyone: Please provide me some solution?

Thanks,


Patrick Wallek
Honored Contributor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

If this mount point is this important, take it out of the AUTOFS configuration and mount it manually. That way you will not be subject to the autofs timeouts.
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Patrick Wallek,

I am not using AutoFS and I don't want to. I am mounting the export manually. and that's why i have disabled the AUTOFS (set it to 0/disable)

But this it doesn't help.

Please let me know the steps - you want me to follow?

Thanks,
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Can you try accessing this remote device outside of your backup application? In other words, can the 11.31 NFS client write data to the appliance using cp, iozone, etc. for longer than 5 minutes?

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Very good point Dave,

YES It's more then 15 minutes and i am copying big data set (of 280G) to my NFS Mount is still in progress. I do see my NFS Client is still active on my DeDup Appliance. Annd iostat ane netstat is showing IO/trafic.

So sounds like it problem with HP Data Protector? What that could be? May be some kind of locking issue by HP-DP???

Any insight?

Thanks for your suggestion...
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

BTW, right now "/etc/rc.config.d/nfsconf" has following entries:

-------------------
NFS_CORE=1
RPCBIND_OPTIONS=""

LOCKMGR=1
LOCKD_OPTIONS=""
STATD_OPTIONS=""

NFS_CLIENT=1

NFS_SERVER=1
PCNFS_SERVER=0
START_NFSLOGD=0
START_MOUNTD=1
MOUNTD_OPTIONS=""

AUTOFS=0
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""
AUTO_MASTER="/etc/auto_master"

NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4 <>
-------------------------------
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

A couple of things:

1) There are some lines in your nfsconf file that don't belong there for an 11i v3 box:

NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4

None of those lines should be there. They are all obsolete. The number of nfsd threads is set in the /etc/default/nfs file via the NFSD_SERVERS variable, the number of async I/O threads is now a per-mount tunable set via the kctune parameter nfs3_max_threads, and the NFS_TCP parameter is just plain gone. We've supported NFS over TCP for many releases now so that tunable doesn't belong there. I'd suggest removing all of these from your nfsconf file.

2) If you think the problem is related to file locking, I noticed you're mounting the filesystem with the "llock" option. That parameter causes the NFS client to not send lock requests to the server. Perhaps that is confusing the appliance, as it might be expecting to receive lock requests and gives up waiting after 5 minutes. Just a guess because I don't know anything about Data Protector or your appliance, but I thought it was worth mentioning since you suspect a file locking issue.

You might try re-mounting the filesystem without the "llock" option and see if you get different behavior.

Regards,

Dave

Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???


Thanks Dave,

Thanks for the advice;

With referance to your comment;
--------------------------------------
1) There are some lines in your nfsconf file that don't belong there for an 11i v3 box:
NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4
None of those lines should be there.
--------------------------------------

I tried to eleminate these lines from my "/etc/rc.config.d/nfsconf" file but it did not get any better. Look like these variables are still having impact on the NFS Demon. So i decided to verify all these variables by eliminating one by one.


My findings are here:
"/etc/rc.config.d/nfsconf"
-------------------
NFS_CORE=1
RPCBIND_OPTIONS=""

LOCKMGR=1
LOCKD_OPTIONS=""
STATD_OPTIONS=""

NFS_CLIENT=1

NFS_SERVER=1
PCNFS_SERVER=0
START_NFSLOGD=0
START_MOUNTD=1
MOUNTD_OPTIONS=""

AUTOFS=0
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""
AUTO_MASTER="/etc/auto_master"

NFS_TCP=1
---------------------------------

Scenario#1
With the above entris: (including AUTOFS=0 or AUTOFS=1)
Backup *Hangs* every time after 5 minute (aprox) from the time it starts.

-------------------------------

Scenario#2
With: NUM_NFSD=16 # Added this line

Backup started at: Time: 9/22/2010 7:48:27 AM
Backup hanged at : Time: 9/22/2010 8:32:41 AM
Backup hannged about in 44 Minutes later

------------------------------
Scenario#3
With: NUM_NFSD=16 and # Added this line
NUM_NFSIOD=4 # Added this line

Backup Started at : Time: 9/22/2010 12:10:18 AM
Backup completed at: Time: 9/22/2010 4:42:51 AM

Backup and CP -r command never hanged; I was able to backup or/and copy 1TB data
------------------------------

BTW what does NUM_NFSIOD=4 and NUM_NFSD=16 do?

Thanks,
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

> I tried to eleminate these lines from
> my "/etc/rc.config.d/nfsconf" file but it
> did not get any better.

I didn't claim these paramters did anything or would change the behavior. I said they're obsolete - meaning they are no longer used.


> Look like these variables are still having
> impact on the NFS Demon. So i decided to
> verify all these variables by eliminating
> one by one.

Nope. I searched the entire kernel source code and there isn't a single place where we reference the NUM_NFSD or NUM_NFSIOD variables any more. If you're getting different behavior I think it has more to do with other factors.

I don't know if you rebooted in between tests, stopped and started things between tests, etc. Those variables, and NFS_TCP are not used by the HP-UX kernel on 11i v3.

Regards,

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Dave,

Thanks a lot for the clarification...

> I tried to eleminate these lines from
> my "/etc/rc.config.d/nfsconf" file but it
> did not get any better.

>> I didn't claim these paramters did
>> anything or would change the behavior.
>> I said they're obsolete - meaning they
>> are no longer used.

I

> Look like these variables are still having
> impact on the NFS Demon. So i decided to
> verify all these variables by eliminating
> one by one.

>> Nope. I searched the entire kernel source
>> code and there isn't a single place where
>> we reference the NUM_NFSD or NUM_NFSIOD
>> variables any more. If you're getting
>> different behavior I think it has more to
>> do with other factors.

>> I don't know if you rebooted in between
>> tests, stopped and started things between
>> tests, etc. Those variables, and NFS_TCP
>> are not used by the HP-UX kernel on 11i
>> v3.

Yes I did reboot my system every time i made any changes (added/removed any parameters); then ran my test.

What that could this problem be? This is so strange - when have NUM_NFSD and NUM_NFSIOD my test passes if i remove NUM_NFSIOD my test fails.

Please guide me.

Please let me know if you want some info logs from the system?

Regards,
~Bish
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Please clarify something for me.

The only difference in behavior you're seeing with all your configurations is with Data Protector, correct? Regardless of the way you set these obsolete variables, you can always copy data to the appliance outside of DP without issue, correct?

If that's the case then I'd be investigating this issue purely from a DP perspective. I know nothing about DP. Maybe they do something really screwy and look for the NUM_NFSIOD parameter to change their internal algorithms. I have no idea.

Have you opened a case with HP and asked someone on the DP team to investigate the problem?

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Dave,

With

AUTOFS=0
NFS_TCP=1
NUM_NFSD=16

I started copying data using ("cp -r") command; and i was monitoring it with "iostat -t 1" and "netstat -I lan2 1" it was fine for about 3 minute then i all the became 0 (Zeros). And still on my DDR I could the see the connection was active for the hp-ux client. but later (about 10 min) when i check the client status on DDR it was inactive.

but my "copy -r" command was still running it hasn't time out yet.

At least with HP-DP my backup worked for 40min.

Does this help?

Thanks,
~Bish

initially looked like it was going fine.

#NUM_NFSIOD=4
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

When the copy job appears to hang, does nfsstat report any traffic? Try this:

# nfsstat -z

Start the copy job and wait for it to "hang" then type:

# nfsstat -c
# nfsstat -z (zero out the counters)

Wait a few minutes

# nfsstat -c

Did the counters increase while the job was hung? If so, which ones?

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

This is so weird,

Last time IO stopped in 5-10 min and "cp -r" command never completed i had to reboot the system...

So after I rebooted the system: i started copying the data to the NFS Mount. 2 & Half hours past and I don't see any issue so far.

----->
output files are attached

<-----
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Like I said, I do not believe what you're seeing has anything to do with the values of the NUM_NFSDS, NUM_NFSIODS, NFS_TCP, or AUTOFS tunables (assuming you're not using AutoFS). NUM_NFSDS, NUM_NFSIODS and NFS_TCP tunables are obsolete - we don't use them anymore.

There's likely something else in your environment causing the hangs to occur, which is why some of your tests failed while others succeeded when the changes you made in between should have no effect.

The nfsstat counters should be the indication of whether outbound over-the-wire NFS calls are being made, or at least attempted, but the 11i v3 client.

Out of curiosity, what version of ONCplus are you running on this system?

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Dave,

This is so strange. once the "cp -r" operation completes i'll run my DP Backup session without rebooting or making any change to my environment:

I'll post the result.

>Out of curiosity, what version of
> ONCplus are you running on this system?

----------
# swlist ONCplus
# Initializing...
# Contacting target "qaia48"...
#
# Target: qaia48:/
#

# ONCplus B.11.31.09.01 ONC+ 2.3
ONCplus.NFS B.11.31.09.01 ONC/NFS; Network-File System,Information Services,Utilities
#
--------



Thanks,
~Bish
Hakki Aydin Ucar
Honored Contributor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi,
11iv3 is different from earlier versions, as Dave stated here, ( I did nor know that )

When it comes to DP side, last time I installed last release of DP ,I remember it needed couple of patches highly recommended by HP Support. I am talking about release 11iv1 but maybe 11iv3 have some problems like this ,you can ask your local HP with a case.
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Hakki Aydin,

Thanks for insight,

BTW, my HP-UX 11i v3 Media is September 2010 edition, I have installed required HP-UX patched recomended by HP-DP 6.11.

I have all latest HP-DP 6.11 Patch installed on Windows 2003 R3 x64 cell manager, RHEL 5.5 x86_64 Installation Server, so my HP-UX DA & MA client is fully patched (i mean DP Patch and OS Patch).

Thanks,
~Bish
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Dave,

>> This is so strange. once the "cp -r"
>> operation completes i'll run my DP Backup
>> session without rebooting or making any
>> change to my environment:

After "cp-r" completed with not issue i started HP DP Backup as well and I did not see any issue this time as well.

With
------------
AUTOFS=0
NFS_TCP=1
NUM_NFSD=16
------------
Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Then I guess the prudent thing would be to leave things alone. If it ain't broke...

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???


Thanks Dave,

>> Then I guess the prudent thing would be to
>> leave things alone. If it ain't broke...


BTW do you recommend
Scenario#1 Adding all of these?

NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4


OR
Scenario#2 : leaving as is:
NFS_TCP=1
NUM_NFSD=16


OR
Scenario#3
removing NUM_NFSD=16
not adding NUM_NFSIOD=4

OR
Scenario#4
not having none of these entry as you described that they as not used any more:
NFS_TCP=1
NUM_NFSD=16
NUM_NFSIOD=4


How about AutoFS?
Do i need to keep AUTOFS=1 or AUTOFS=0 (in my case where i hard mount my nfs share)

Really appreciate your help...

Thanks,
~Bish

Dave Olker
HPE Pro

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Hi Bish,

My preference would be scenario 4. NFS doesn't use those variables any more so they can only add confusion to any troubleshooting efforts. If you experience client hangs with these tunables removed then the hang needs to be investigated.

As for AutoFS, if you do not use AutoFS to mount filesystems then set AUTOFS=0 since there's no reason to run it.

Regards,

Dave
Bishwajit Kumar
Frequent Advisor

Re: NFS Mount becomes inactive on HP-UX11.31 while DP6.11 Backup is writing to it???

Perfect!!
Thanks Dave, I'll go with Scenario#4 as you suggested (i.e. they not being used by NFS any more in HP-UX 11i v3(B.11.31) IA.

I'll post you the results.

~Bish