NFS mount not responding

[Timothy P. Jackson]

Hello Everyone,

I have been reading everything I can find on NFS issues and I can not seem to find any thing that may help me with my issue.

So I am looking for some input from any one who may have an idea as to why this is happening.

Here is my issue.....

I have two HP-UX 11i Version 1 machines. One is a production machine and the other is basically a Disaster Recovery machine. Every night I stop my data base, nfs mount the drives from the DR machine to the production machine and using tar, I transfer all production data to the DR machine.

All of a sudden, in the last couple of days, the nfs mounts have been getting all messed up. The transfer of data will start ok but after maybe 5-10 minutes of tranfering data the transfer slows to a crawl. If I do a bdf it comes back and tells me that the mounts are not responding.

Any thoughts on this issue would be greatly appreciated.

Thanks,
Tim

[RAC_1]

You starting point should be a network check. Check network logs on both systems, also in background you can run ping to check if machines have network problem or not.

netfmt -f /var/adm/nettl.LOGxx on both machines. Also check with network administrator to know if there were network problem between these machines.

[Hemmetter]

Hi Timothy,

Have there been some changes on the network between the maschines? New Hardware? Did you check interface settings on your maschines an d on the network components?

Are there Syslog.log messages?

Are you using nfs over UDP or TCP? How are your machines connected? Crossover-cable, switched or a network with lots of network componets between.

Some network components may prioritize tcp traffic and limit ucp. So if there is a high tcp load on your network simultaniously your udp may suffer a deceleration.

Is there a new service running? Maybe a network backup scheduled since that couple of days?


rgds
HGH

[Darrel Louis]

Hi,

Are the servers on the same subnet or vlan?

Can you check if you see any Collision errors or packet loss, netstat -I ?

Is there a system backup running during your syncronize process?

Have you already tried stopping the NFS-server & -Client?

Darrel

[Timothy P. Jackson]

Thanks for the input everyone! Here is what I do know about the physical network.

The two HP machines in question have been move to a new facility so the network between them is new. We are using Cisco switches also on this network. I belive that the problem is with the network but I am not sure where. I am able to ping the two machines at any time during the NFS mount problems. I can also telnet and do anything else between the two machines.

I did not look at the network logs but I will do that and let you know what I find.

Tim

[Steven E. Protter]

Shalom Tim,

Have networking check everything, including wiring and switches.

Use lanadmin (ex lanadmin -x 0) the see that speed and duplex is correct, but the key factor here is the move of facilities.

I suspect the physical networking situation.

SEP

[Anil C. Sedha]

Tim,

Run the following now and then later tonight through a script during the time when your DB backup is supposed to kick in (possibly 5 - 10 mins after that)

traceroute DESTIP > /home/yourname/somefile

This will show you where the congestion was, how many hops were involved in data transfer, and what might be the problem.

In addition to this you can run the netstat and netfmt commands also to pipe to seperate log files for you to review in the morning. I believe the switches may be having some issues. Try asking your network folks to connect your systems to another switch and also ask them to check for collisions on their network devices involved.

Should be able to resolve in a day or two with everyone's involvement.

-Anil

[Timothy P. Jackson]

I agree with you Steven.

This is probably a physical network problem seening that is is all new. If nothing has really changed on the HP hardware or OS side that it must be some where between them.

I check the nettl.LOGXXX's on both machines and they have not been updated in a couple of days so I know that they are not seeing any errors. I have also checked the syslog.log before, during and after the attempt to transfer the DB, and it also is not logging errors.

I will check lanadmin and traceroute to see if anything shows up there.

Thanks again for all the help!

Tim

[Timothy P. Jackson]

Through everyone help I have some new information that I am not sure if it is the problem. As I mentioned before nothing changed on the HP machines or should have changed but.......

On one machine the NIC is running at FULL Duplex and on the other machine it is runnign at HALF Duplex. Is this enough to be causing my problem?

I also checked netstat -I on both machines. Because I rebooted this morning after my final attempt to do my data transfer, I do not have a full scope of what was happening through out the night. Right now on the machine that is running FULL duples there are no collisions or errors. On the machine running HALF duplex it is logging both Oerrs and Collisions.

Should I change the machine from HALF duplex to FULL? I can reboot this machine during the day because it is not the production machine.

Thanks,
Tim

[Victor Fridyev]

Hi,

Could you please check whether NICs and appropriate ports are configured equally. A difference in speed and negotiation configuration may cause performance problems.
You can check NICs setting by
#>lanadmin -x and set by
#>lanadmin -X 100FD or whatever.

HTH

[Bill Hassell]

The duplex mismatch is definitely a big problem. For 100 Mbit networks, the switches and NICs can negotiate both speed and duplex. Unfortunately, there is a specific length of network cable (something like 30-50 meters) which confuses the circuitry and when duplex negotiation fails, the standard specifies half duplex. Thus, you would expect to see a LOT of colisions and other errors, especially if the switch is stuck at full duplex. This may have worked fine initially but someone may have replaced the cable with one that is now in the problem length area.

You can use lanadmin to change the NIC temporarily to full duplex, or you can shorten or lengthen the cable from the NIC to the switch. To permanently change the negotiation, you'll need to edit the appropriate LAN card file in /etc/rc.config.d to specify FULL duplex. Make sure you turn off autonegotiation in the switch port and set it to match your NIC full duplex setting.

[Timothy P. Jackson]

Thanks again to everyone for all the help!

The issue was the duplex. After setting it in config file in /etc/rc.config.d, I rebooted the system and started my DB transfer. It went with no problem. I also had no Oerrs or collisions.

I believe that Bill hit on why this changed. the actuall machine is at least 200 feet from the switch. I will see about getting the switch moved closer to the machine.

Tim