High local mirror utilization

[Blaz Podrzaj_1]

Hi guys,

Recently I encountered weirdly high utilization of local mirrored disks on newly installed HP-UX 11.11 (L2000 server).

I have four local disks from which first two are in vg00 (c1t2d0, c2t2d0) and second two in vg01 (c1t0d0, c2t0d0) volume group. Both second disks in mirrors (c2t2d0 and c2t0d0) have strangely high utilization on practically idle system and I don't think that this is a normal situation. Or is it?

Output from sar (every 3 seconds for 5 minutes) says:

Average c1t2d0 0.77 0.50 1 12 0.00 8.57
Average c2t2d0 87.79 0.50 1 11 7.52 1118.92
Average c1t0d0 0.11 1.27 0 8 3.69 13.31
Average c2t0d0 12.45 0.83 0 8 202.48 1254.95

Regards,

Blaz

[Keith Bryson]

Hi Blaz

Can you post the output from the following commands:

ioscan -fnC disk
vgdisplay -v
swapinfo -t

Regards - Keith

[Blaz Podrzaj_1]

Hi Keith,

No problem, here you go:

bash-3.00# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 878 -878
memory 1472 686 786 47%
total 5568 1564 4004 28% - 0 -

bash-3.00# ioscan -fnCdisk
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 1 0/0/1/1.0.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 2 0/0/1/1.2.0 sdisk CLAIMED DEVICE FUJITSU MAJ3182MC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 3 0/0/2/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 4 0/0/2/0.2.0 sdisk CLAIMED DEVICE FUJITSU MAJ3182MC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0

Regards,

Blaz

[Keith Bryson]

Blaz

I think we can safely say that this isn't normal.

If you have glance, run it up (text version), hit 'o' then enter and change the "Sort Key" to disk (you will then be prompted to save settings). Now when you hit 'g', what process is mainly using disk (look under "Disk IO Rate")?

Let me know if you haven't got glance!!

Keith

[Keith Bryson]

Can we also just make sure you don't have any syslog errors:

egrep "EMS|vmunix" /var/adm/syslog/syslog.log

and the root VG mirror is configured OK:

lvlnboot -v

Keith

[Blaz Podrzaj_1]

Hmm, nothing unusual:

bash-3.00# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Root: lvol3 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Swap: lvol2 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Dump: lvol2 on: /dev/dsk/c1t2d0, 0

Glance is showing normal utilization in process view. Oracle DB is more or less idle right now (several users) but that doesn't count because DB is on SAN disks.

I suspect SCSI controller, but I will not turn off this server until I drain all sw options.

I can also try to disengage mirrors, exclude second disks from volume groups and put c2t0d0 into vg00 volume group making it mirror to see if this behaviour persist, but doing this on production machine in first days of month is rather suicidal :)

Regards,

Blaz

[Keith Bryson]

There are some sar patches (although these don't specifically point to your sort of problem - but others using the -d option). There are also the usual SCSI driver patches associated with performance issues.

One last thing in glance, does the Disk Util bar echo the sar averages? If you press 'u' and find the disk in question, does the glance figure echo the sar figure for this device? If so, press 'i' (IO BY FILE SYSTEM) which filesystem is heavily used?

KB

[Blaz Podrzaj_1]

This disk utilization is definitely not fs dependent because if I do lvsplit, utilization should jump to the only remaining disk but it doesn't.

Only second disks in mirrors which happen to be on the same controller are 100% busy, the first ones averages are normal. I'm calling hw support :)

Regards,

Blaz

[Keith Bryson]

Ah - OK. I didn't deduce from the sar stats that both disks on that bus were heavily loaded (that's not clear). Anyway, good luck with HP. Maybe you should swap all cabling and/or the terminator too.

Keith

[Blaz Podrzaj_1]

Yeah :) Don't get me wrong, utilisation on 2nd and 4th disk is not 100% all the time, I never said that, but if one calculate from the above sar data one wold come to a conclusion that 2nd and 4th disk are about 110x more busy in average than 1st and 3rd.

Thanks for your quick response and help.

Regards, B

[Florian Heigl (new acc)]

This sounds really strange, maybe have a look if there still is a stale PE, but then the lvsplit should've failed.

Next thing I'd look at would be the EMS logs available through STMs logtool. Some low-level errors will be recovered by the driver and You won't see them anywhere else.