NFS - GB FE / 100FE failover

[rmueller58]

I have a question on setting up a failover for a remote mount NFS ..

We have a NAS with a mount point I dump to from my HP/UX box.

on my UX box I have three NICs, 1 GB and 2 100FEs..

The primary for this particular mount is over the 1000-Base-T interface..
the addressing scheme
is 192.xxx.xxx.XXY on the GB
and
10.xxx.xxx.XXY on the FE

I have the mount point defined in the /etc/fstab as nashostname:/

in the host table I have
192.xxx.xxx.XXZ defined as the nashostname

as this NAS is an NT hack it periodically fails.. I would like to be able to failover to the 10.xxx.xxx.XXZ as a mount point in the event the 192.xxx.xxx.000 network goes south. (It has periodically done that, and our ops guy is looking into it. )

Prior to getting the GB NIC I was running through the 10.xxx.xxx.XXZ mount it is commented out in the /etc/hosts


I do have a script that checks mounting, but I was thinking I could add the failover in the script somehow..

#!/bin/sh
nasmount=`mount |grep nas |awk '{print $3}'`
if [ "$nasmount" != "nashostname:/NAS" ]
then
echo "NAS is NOT MOUNTED ATTEMPTING MOUNT" > /nas.log
date +%Y%m%d:%H%M >>/nas.log
mount /DIRNAME/DIR
else
echo "NAS is MOUNTED" >/nas.log
fi

It's my guess I would have to umount /DIRNAME/DIR then mount the 10.xxx.xxx.xxZ:/NAS /DIRNAME/DIR

Rather then using a name coded in the /etc/hosts,

Any NFS gurus, let me know what their thoughts are as to a better, best optimum method to do this would be.

Thanks in Advance..

Rex M. - ESU#3 Omaha

[Steven E. Protter]

I would suggest pulling the IP address from a file,perhaps a special one you set up for these purposes.

Your methodology seems okay, but I'm not sure how you are going to be sure the NFS server is actually down. Long delays are typical in such a setup.

Seems to me a setup using Serviceguard and the NFS toolkit to provide nfs services to your network instead of NT would be better. I guess the NAS device is an obstacle to that.

SEP

[rmueller58]

Steven,

Our MCSE and manager are MS people, my HP/UX has to find ways to play with their box(ses)
The NAS is an obstacle to using more more then the NFS Client mounts.

This has worked fairly well, but I would like to have the mount point failover to the 10.xxx address in the event the 192.xxx bugs out.

It should work to set up a secondary mount point in the /etc/fstab and leave it mounted, in the event the primary fails the writes would failover in my naswrite script.

If something more comes to mind Steven, let me know..

[rmueller58]

Oh,

In the event the mount point fails,

ie, through a mount TEST


nasmount=`mount |grep nas |awk '{print $3}'`
if [ "$nasmount" != "nashostname:/NAS" ]

But I got to thinking this portion only sees if the point is mounted not if it is inactive, and unavailable.

Good point,

Maybe if I do a umount then mount and if the mount fails on primary then the script should roll to secondary, if secondary fails then the script writes and error to my logging.
What do you think?

[RAC_1 · Accepted solution]

That can not be done with some scripting or something easy. Do you mount this NFS mount all the time or whenever required??

If you mount it whenever you require it, then you can do some scripting and mount through choosen ip address-through FEs or through GB NIC. - This part is easy.

But you mount it at boot time and it is mounted all the time, then you will have unmount and mount again with required NIC.

Other option is to use Auto Port Aggregation-a $$ product. With this you can joing nics and load balanace etc. If one failes, no problem. (but under this also, I do think you can club nics from different subnets)

Anil

[rmueller58]

RAC,

By default it is mounted on the GB card at all times right now. However, I have one script that creates a secondary tarball of my DB exports (to assuage a director's ill ease on weekends)..

The MS and Dell guys can't figure out why the dang NAS goes south.. So I am just looking for a way to CMA.. I've got enough redundancy built into my backups and exports but this is like an extra place my director wanted a set in the event of a tape failure or other disk failure.

[harry d brown jr]

May I also suggest you use jumbo frames for your NFS network:

/etc/rc.config.d/hpigelanconf
HP_IGELAN_MTU[XXX]=9000

where XXX matches the right subscript.

HP's documentation on GigE:
http://docs.hp.com/en/netcom.html#1000Base-T%20and%201000Base-SX%20%28Gigabit%20Ethernet%29

Make sure you have at least the latest patches installed:

http://www2.itrc.hp.com/service/patch/releaseIndexPage.do?BC=patch.breadcrumb.main|patch.breadcrumb.bundle|

live free or die
harry d brown jr

[rmueller58]

Thanks Harry,

reset MTU to 9000 from 1500

[harry d brown jr]

Make sure you do that on all systems on the gig-E that are talking to each other using NFS. I'm sure your network switch can hanle it, but you might want to check with your switch vendor.

The cool thing is that an ENTIRE NFS status header will fit into a single packet.

live free or die
harry d brown jr

[rmueller58]

Harry,

the only HP box on that subnet is the one in question, the rest of the stuff on that net is MS servers, and that Dell running a hack of MS2003 server. Not quite sure about this particular hack.


RM (flying the Gadsden at home)

[harry d brown jr]

You can configure the M$ servers as well!!!

Microsoft benchmark on Gig-e stated their numbers:
http://download.microsoft.com/download/9/d/0/9d0f411d-640d-47cd-8bdb-326c7fd6078f/perf.doc

SUN's white paper on Understanding Gigabit Ethernet Performance on Sun Fireâ ¢ Systems
http://www.sun.com/blueprints/0203/817-1657.pdf


live free or die
harry d brown j

[rmueller58]

Harry,

I am mentioning that to the MS types in house, we don't have that many devices on that particular segment so it shouldn't tax them too bad.. :D

[rmueller58]

Harry,

Our M$ guy was slobbering to see if he could change it on a couple of their High utilization boxes.. Sounds like throughput is and has been a problem for sometime.. gracias..

[harry d brown jr]

Throughput will only work if these conditions are met:

* PointA to PointB to ... to PointC must have jumbo Frames enabled. Most GigE switches have this ability. When serverA talks to serverB through a gigE jumbo frames enabled switch, then both servers can be set to jumbo frames.

* The application might have to be "tuned" to utilize the jumbo frames. A lot of small packets will just waste bandwidth. Backing up the server over a jumbo frame enabled gig-e network will scream at close to 100megabytes a sec (we call that near wire speeds).

live free or die
harry d brown jr

[rick jones]

If you plan on failing over from Gigabit to 100BT you may not want to become dependent on JumboFrame. The performance change at failover is going to be large enough as it is :)

The MS boxes may already be taking advantage of "poor man's JumboFrame" as it is - a feature of some GbE NICs called Large Send or TSO. It is like JumboFrame in that the local TCP stack can pretend the MTU is larger, but the NIC resegments the traffic to fit on a 1500 byte MTU network. So, you get most of the sending side benefit, but not the receiving side benefit of JF.

LargeSend/TSO is available on certain 11i patches/releases.

WRT APA, I'm not sure if it supports failover between different NIC types. Indeed with APA, you would trunk/bond/link the links together into a failover group, and assign IP addresses to the virtual NIC created by APA - the transport would believe there is just the one physical NIC presnet.

[rmueller58]

Rick,

With multiple nics in each box the framing capability if on a card to card basis.. We've had problems on our GB network with the M$ boxes in question anyway. I believe partially or wholly because they aren't configured to meet the network..
Our M$ staff met with my director this AM because they had a 10GB SQL dump that was "taking a dump", acting more like it was on a 10mb segment then a 1000T.. :D Their 10gb flatfile has been taking about 10 hours to move( copy), my 3gb informix exports are taking about 30 minutes, to the same destination box. I am using TAR and GZIPping directly to the NFS mount point.

[rmueller58]

We have completed this.. thanks all.