filesystem hanging

[jedd]

hello all
i need help with diagnosing this problem. i'm attaching a txt file where i get it from the syslog. it seems like about once a week, the filesystem hangs for some reason. i look at the syslog and this is what i get. it seems like i have a user that is increasing the nfs sever daemons. will that cause the filesystem to hang. i don't know how to diagnose all the scsi error. i have (4) 17Gb drives in a single channel, with oracle running on it. i have vg00 for all my root filesystem. i have vgDB for all my oracle stuff including RDBMS, and all the datafiles. nothing on vg00. i striped (3) 17GB drives for all the oracle. so i don't know if that's causing it to fail. i installed the latest release HP-UX 11.11 09/2002. i would be appreciate if somebody can help me decipher the errors on the txt file.

[Sridhar Bhaskarla]

Hi

You have a problem with your c0t5d0 disk.

Do a ioscan -fnC disk > disk.out

Look at disk.out and see if this disk is claimed. It could be a hardware issue.

-Sri

[James R. Ferguson]

Hi:

Assuming that the disk is claimed by 'ioscan' you could look for errors thusly:

# dd if=/dev/rdsk/c0t5d0 of=/dev/null bs=256k

Regards!

...JRF...

[Rita C Workman]

I agree that it looks like you have a disk problem. But I think running ioscan may still show the disk as claimed.
It looks like there are some bad blocks on the disk that can no longer be read.

Determine for this disk what it associates to...then you will know best how to proceed and what you may need to address.
Fixing vg00 is one thing, but if you have bad blocks (that HP did not re-allocate) and it involves your Oracle database..that is another can of worms.

Just a thought,
Rita

[Eugeny Brychkov]

I agree with Rita and James. It's highly possible that it's hardware problem. dd will show if disk surface is readable, and if not then as soon as this disk is a part of root and has contiguous allocation policy this can be the cause of hangs. The best choice is to replace it. Make ignite tape and replace disk (call HP if you have warranty/contract)
Eugeny

[Helen French]

I would suggest you to check your disks and other hardware with STM tools. You can do 'dd' on the disks as specified above. Also, install any patches if you haven't updated your system lately.

[Steven E. Protter]

I chime in.

Bad disk.

Time to shut the database cold back it while HP support comes out with a new disk. Every lbolt error I ever got led to disk replacement.

It is conceivable that stm will point you to a scsi problem, and if the problem repeats itself, look into cabling scsi card or drive cage(if applicable).

Not an optimal Oracle setup either. You'd be better off with mirror/us and raid 10 mirroring, but it is more costly in terms of disk space.

At least for data, having a full mirror copy is much more important for performance and reliablity than striping.


SEP

[jedd]

hello all thanks for the reply. i am not too good at reading dd output. can you tell me what format the output will be.

[Helen French]

If you execute the dd command as specified by James and if you have a problem with your disk, it will report an I/O error and will come out of the process. If no error found, the task will be completed with total number of blocks in and number of blocks out.

[Michael Steele_2]

Jedd:

dd if=/dev/rdsk/c0t6d0 of=/dev/null count=1000000

At this point the cursor will hang for several seconds. Let it. You want to read beyond the disk cache, note count=1000000 to do this.

Then, "...control c..." to break out and read the following

6930+0 records in
6930+0 records out

If the disk is bad the cursor will not return until the disk is swapped out. So you'll have to open up another window or another session.

You'll occasionally see the I/O error.

dd read error: I/O error
6930+0 records in
6930+0 records out

[Michael Steele_2]

As an alternative you can run LOGTOOL.

STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > VIEW > RAW SUMMARY.

Note the first and last date, calculate the period between both first and last date, i.e., 4 days 20 hours and then read down the list of HW addresses. Observe the integer numbers in parenthesis. Are there any numbers over 150 or 200? If so open up a HW case with HP. Cross reference the HW address in ioscan.

[jedd]

i'm not getting any errors out of the dd. so what else can i do? nothing in dmesg and syslog. i'm stumped. and it only happens after about 2 weeks or so. but everything else is okay.

[jedd]

hello all, another question for you experts.
i am attaching another txt file. i get a lvlnboot after my initial boot. why is that?

[jedd]

sorry i forgot to attach the txt file. i don't understand why i will need a lvlnboot, if all my configuration are working. i am not recreating/creating any logical volumes

[Michael Tully]

Even though the errors only show up once in a while, I would get the disk replaced. As long as you have a support contract with the disk vendor, then insist on them replacing the disk. Make sure that you have a backup of your system first.

[Michael Steele_2 · Accepted solution]

Execution of lvlnboot always shows up in syslog.log and can be called by other commands, SAM for example. It???s normal. The -v option is a report listing option and not an update option.

[Steve Post]

A different (and probably remote) possibility: how is it mounted?

I had a filesystem mounted to
/var/opt/omni/db40

That was fine EXCEPT THAT
/var/opt/omni is a link to
/bigdisk/var/opt/omni.

I should have had it mounted to
/bigdisk/var/opt/omni/db40 instead of
/var/opt/omni/db40.

So......I had errors in my syslog.log file too.

It's just a thought.