fsck fixes corruption but it comes back

[Ted Pierson]

HPUX 11 on L3000 server. New install 2 months old. Last couple of weeks we started getting file corruption bad nodes in vg00 disks /var /usr. fsck fixes the files but problem will come back in a couple of days. Problem seems to appear during peak usage times.

HP says to reinstall the system. I'm leaning twords hardware problems or memory issues.

thanks trpi

[Michael Tully]

Hi,

To determine if it a bad disk do
you have any error messages being
report to /var/adm/syslog.log or
from EMS (if you have it installed)
into /var/opt/resmon/log/event.log
If there is nothing being reported
what happens when you run something
like
'sar -d 5 5' during the peak periods
'glance' and view the disk activity?
The output of the latter will determine
more of diskIO bottlenecks.

-Michael

[Roger Baptiste · Accepted solution]

hi,

<>

Doesn't sound good. Few things to check:

*) /var/adm/syslog/syslog.log file for
disk or PV errors; If you have ems setup,
it will send useful info to the syslog file
*) dmesg output
*) xstm -> run xstm and look at the logs.
(the tool is easy to use , it will guide
you through)

*) if you have additional space in the system,
create temporary filesystem for /usr and /tmp;
move the files; newfs the old /usr and /tmp
and copy it back. (may sound tedious, but depends on your desperation)

and finally to confirm if it is a disk error:

*) If VG00 is mirrored, do a lvdisplay -v on
the Logical volumes and see whether any extent is stale. Then, to isolate whether the problem is with the disk, break secondary mirror ;If problem persists, add the secondary disk back as a mirror and break the
primary. This way you can confirm the
disk issue without disturbing anything.
(break mirrors by using lvreduce and vgreduce commands).

HTH
raj

[Ted Pierson]

thanks for the response.
I do get bad node errors in the syslog.log. I can cause an error when I do a find (get cannot stat error)
I'll check the other logs.

We've done dd and the hardware checks out good.
The disks are mirrored (we do plan to break them) and test