Intermittent NFS incident

[organicko]

Hello everybody!

I have a HP rx-5670 box with HP-UX 11i version 1.6 installed on it.
I also have a NetApp FAS 250 working as a NFS server. I installed oracle 9i engine in my HP box while the database on the FAS 250.

During these few months, my oracle server randomly got offline due to weird error i found in my /var/adm/syslog. The error looks like this:

Jul 1 06:40:56 mysvr01 vmunix: NFS server fas25001 not responding still trying
Jul 1 06:41:01 mysvr01 vmunix: NFS server fas25001 ok

Below is the error from oracle trace file:

Fri Jul 1 06:41:01 2011
Errors in file /ora1/oracle9i/rdbms/log/mysvr_ora_11091.trc:
ORA-00345: redo log write error block 2552 count 1
ORA-00312: online log 6 thread 1: '/database/oradata/SPKBR5/redo06.log'
ORA-27072: skgfdisp: I/O error
HPUX-ia64 Error: 238: Connection timed out
Additional information: 2552

I did everything i could to avoid this intermittent from happening again. But it kept coming back with random pattern (unpredictible time of occurences).

Could anyone here help me out from this misery. Your help will be truly appreciated

[Dave Olker]

What is 11i version 1.6?

> I did everything i could to avoid this intermittent

> from happening again. But it kept coming back

> with random pattern (unpredictible time of occurences).

What steps did you take?  How exactly is the filesystem mounted? 

Dave

[organicko]

HP-UX 11i version 1.6 is the operating system a.k.a HP-UX B 11.22

Here's a bit setting info on my HP box:

# swlist -l product | grep -i nfs
  NFS                   B.11.22        ONC/NFS; Network-File System,Information
Services,Utilities
  PHNE_30084            1.0            ONC/NFS General Release/Performance Patch

# more /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs delaylog 0 1
/dev/vg00/lvol4 /home vxfs delaylog 0 2
/dev/vg00/lvol5 /nsr vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /tmp vxfs delaylog 0 2
/dev/vg00/lvol8 /usr vxfs delaylog 0 2
/dev/vg00/lvol9 /var vxfs delaylog 0 2
fas25001:/vol/vol1 /database nfs rw,suid 0 0
/dev/vg02/lvol1 /data1 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg01/lvol1 /ora1 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg03/lvol1 /data2 vxfs rw,suid,largefiles,delaylog,datainlog 0 2

Some steps i have taken:

1. Enable the automounter

2. increase the number of NFSD to 16 instead of 4

[Dave Olker]

11.22 hasn't been supported for a very long time.  If this is a problem requiring a patch, you'll need to update to 11i v2 or 11i v3.

As for the NFS mount, "nfsstat -m" will give me the mount options.

Dave

[Torsten.]

11.22 is out of support for a long time now.

>> I/O error

Looks like a bad disk.

Check the disk and replace if needed.

When_Good_Disks_Go_Bad_WP
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

[organicko]

# nfsstat -m
/database from fas25001:/vol/vol1  (Addr PP.LL.YY.XX)
 Flags:   vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsi
ze=32768,retrans=5
 All:     srtt=  0 (  0ms), dev=  0 (  0ms), cur=  0 (  0ms)

FYI, the HP box never gave me problem like this before. This mess started when re-installed the OS due to hard disk damage.

My aim is just to bring this server back to its stable condition before it was re-installed. Upgrading it to 11i version 2 or 3 is not an option to me for this moment.

[Dennis Handly]

>I have a HP rx5670 box with HP-UX 11i version 1.6 installed on it.

11.22 stopped being supported when 11.23 was shipped.  Probably a Merced box.

It is supported on 11.23 but you can't buy that any more.  So the box is near useless.

>due to weird error I found in my /var/adm/syslog. The error looks like this:
Jul 1 06:40:56 mysvr01 vmunix: NFS server fas25001 not responding still trying
Jul 1 06:41:01 mysvr01 vmunix: NFS server fas25001 ok

This is not a weird error, this is what you learn to expect for NFS.

>What is 11i version 1.6?

Yep, pretty old.

[organicko]

>due to weird error I found in my /var/adm/syslog. The error looks like this:
Jul 1 06:40:56 mysvr01 vmunix: NFS server fas25001 not responding still trying
Jul 1 06:41:01 mysvr01 vmunix: NFS server fas25001 ok

This is not a weird error, this is what you learn to expect for NFS.

Yep. You are right Mr Dennis. However, as far as i know, this error never happen before we re-installed the OS due to hard disk damage.

We never experience database down before during those days.

[organicko]

Is there other way to find out what actually causing the NFS intermittent?

syslog is not helping me alot. Is it possible i left any kind of parameter?

Hope someone here can enlight me.....really appreciate that..

[Dennis Handly]

I assume that fas25001 didn't just go down.  Did it hang for 5 seconds?

Any networking problems?