topic Re: filesystem hanging in Operating System - HP-UX

filesystem hanging

jedd — Tue, 04 Mar 2003 17:52:07 GMT

hello all
i need help with diagnosing this problem. i'm attaching a txt file where i get it from the syslog. it seems like about once a week, the filesystem hangs for some reason. i look at the syslog and this is what i get. it seems like i have a user that is increasing the nfs sever daemons. will that cause the filesystem to hang. i don't know how to diagnose all the scsi error. i have (4) 17Gb drives in a single channel, with oracle running on it. i have vg00 for all my root filesystem. i have vgDB for all my oracle stuff including RDBMS, and all the datafiles. nothing on vg00. i striped (3) 17GB drives for all the oracle. so i don't know if that's causing it to fail. i installed the latest release HP-UX 11.11 09/2002. i would be appreciate if somebody can help me decipher the errors on the txt file.

Re: filesystem hanging

Sridhar Bhaskarla — Tue, 04 Mar 2003 18:01:01 GMT

Hi

You have a problem with your c0t5d0 disk.

Do a ioscan -fnC disk > disk.out

Look at disk.out and see if this disk is claimed. It could be a hardware issue.

-Sri

Re: filesystem hanging

James R. Ferguson — Tue, 04 Mar 2003 18:13:53 GMT

Hi:

Assuming that the disk is claimed by 'ioscan' you could look for errors thusly:

# dd if=/dev/rdsk/c0t5d0 of=/dev/null bs=256k

Regards!

...JRF...

Re: filesystem hanging

Rita C Workman — Tue, 04 Mar 2003 18:14:14 GMT

I agree that it looks like you have a disk problem. But I think running ioscan may still show the disk as claimed.
It looks like there are some bad blocks on the disk that can no longer be read.

Determine for this disk what it associates to...then you will know best how to proceed and what you may need to address.
Fixing vg00 is one thing, but if you have bad blocks (that HP did not re-allocate) and it involves your Oracle database..that is another can of worms.

Just a thought,
Rita

Re: filesystem hanging

Eugeny Brychkov — Tue, 04 Mar 2003 18:38:38 GMT

I agree with Rita and James. It's highly possible that it's hardware problem. dd will show if disk surface is readable, and if not then as soon as this disk is a part of root and has contiguous allocation policy this can be the cause of hangs. The best choice is to replace it. Make ignite tape and replace disk (call HP if you have warranty/contract)
Eugeny

Re: filesystem hanging

Helen French — Tue, 04 Mar 2003 18:43:59 GMT

I would suggest you to check your disks and other hardware with STM tools. You can do 'dd' on the disks as specified above. Also, install any patches if you haven't updated your system lately.

Re: filesystem hanging

Steven E. Protter — Tue, 04 Mar 2003 18:48:03 GMT

I chime in.

Bad disk.

Time to shut the database cold back it while HP support comes out with a new disk. Every lbolt error I ever got led to disk replacement.

It is conceivable that stm will point you to a scsi problem, and if the problem repeats itself, look into cabling scsi card or drive cage(if applicable).

Not an optimal Oracle setup either. You'd be better off with mirror/us and raid 10 mirroring, but it is more costly in terms of disk space.

At least for data, having a full mirror copy is much more important for performance and reliablity than striping.

SEP

Re: filesystem hanging

jedd — Tue, 04 Mar 2003 21:24:32 GMT

hello all thanks for the reply. i am not too good at reading dd output. can you tell me what format the output will be.

Re: filesystem hanging

Helen French — Tue, 04 Mar 2003 21:39:31 GMT

If you execute the dd command as specified by James and if you have a problem with your disk, it will report an I/O error and will come out of the process. If no error found, the task will be completed with total number of blocks in and number of blocks out.

Re: filesystem hanging

Michael Steele_2 — Tue, 04 Mar 2003 21:39:54 GMT

Jedd:

dd if=/dev/rdsk/c0t6d0 of=/dev/null count=1000000

At this point the cursor will hang for several seconds. Let it. You want to read beyond the disk cache, note count=1000000 to do this.

Then, "...control c..." to break out and read the following

6930+0 records in
6930+0 records out

If the disk is bad the cursor will not return until the disk is swapped out. So you'll have to open up another window or another session.

You'll occasionally see the I/O error.

dd read error: I/O error
6930+0 records in
6930+0 records out

Re: filesystem hanging

Michael Steele_2 — Tue, 04 Mar 2003 21:50:26 GMT

As an alternative you can run LOGTOOL.

STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > VIEW > RAW SUMMARY.

Note the first and last date, calculate the period between both first and last date, i.e., 4 days 20 hours and then read down the list of HW addresses. Observe the integer numbers in parenthesis. Are there any numbers over 150 or 200? If so open up a HW case with HP. Cross reference the HW address in ioscan.

Re: filesystem hanging

jedd — Wed, 05 Mar 2003 00:16:24 GMT

i'm not getting any errors out of the dd. so what else can i do? nothing in dmesg and syslog. i'm stumped. and it only happens after about 2 weeks or so. but everything else is okay.

Re: filesystem hanging

jedd — Wed, 05 Mar 2003 00:57:30 GMT

hello all, another question for you experts.
i am attaching another txt file. i get a lvlnboot after my initial boot. why is that?

Re: filesystem hanging

jedd — Wed, 05 Mar 2003 01:00:32 GMT

sorry i forgot to attach the txt file. i don't understand why i will need a lvlnboot, if all my configuration are working. i am not recreating/creating any logical volumes

Re: filesystem hanging

Michael Tully — Wed, 05 Mar 2003 01:00:47 GMT

Even though the errors only show up once in a while, I would get the disk replaced. As long as you have a support contract with the disk vendor, then insist on them replacing the disk. Make sure that you have a backup of your system first.

Re: filesystem hanging

Michael Steele_2 — Wed, 05 Mar 2003 17:05:30 GMT

Execution of lvlnboot always shows up in syslog.log and can be called by other commands, SAM for example. It???s normal. The -v option is a report listing option and not an update option.

Re: filesystem hanging

Steve Post — Fri, 07 Mar 2003 20:05:57 GMT

A different (and probably remote) possibility: how is it mounted?

I had a filesystem mounted to
/var/opt/omni/db40

That was fine EXCEPT THAT
/var/opt/omni is a link to
/bigdisk/var/opt/omni.

I should have had it mounted to
/bigdisk/var/opt/omni/db40 instead of
/var/opt/omni/db40.

So......I had errors in my syslog.log file too.

It's just a thought.