Severe I/O performance issues on server

[Adam Noble]

All,

Hope somebody can give me their opinions on this and see if they agree with my analysis. Users started complaining today that the server was running extremely slowly. I could see via Glance and perf that disk utilisation for 2/3 disks was 100%, disk queue at aroun 9/10 and average service time a rediculous 240 milliseconds. What is strange is that these individual disks were part of a stripeset of 10 disks so I would have expected to see them all hammered. I have perceived the issue to relate to the EMC symmetrix due to the service times but any opinions would be appreciated. EMC are currently investigating problem.

[Fred Ruffet]

It may happen that some disks start running excessivly when a disk is broken on a RAID LUN. Have you checked the disks state in the array ?

Regards,

Fred

[Alzhy]

Adam,

Are you using LVM or VxVM? And are you using LVM/VxVM to stripe your storage ? Can you give us the LVM/VxVM config and a "sar -d 5 10" output?

Also, I suspect your EMC may be severely loaded. What's your qdepth settings on HP-UX?

[Adam Noble]

Nelson the problem is mysteriously no longer there all I have is historical data, what do you mean by disk q depth on the emc???

[Alzhy · Accepted solution]

I meant: scsi_max_qdepth
Depending on your EMC model and how you build your storage volumes (LVOL or VOL) on HP-UX .. you will need to adjust q depth so you do not overrun the frontends of the EMC.

Is your EMC connected to more than one HP-UX machine? On a SAN?

sar -d would have shown you this if it is a loading issue on the EMC or 'something' else. If your filesystems/volumes/lvols are stripes accross EMC LUNs.. then if "sar -d" shows ALL the component LUNs with the same access/service times and q length - then it means your EMC is severely loaded. If only one or a few show signs or severe access/service times and queuing -- then it is 'somehting' else.

[Adam Noble]

Nelson as explained previously it was only 2 disks that were showing utiliation of 100% and average service times in excessive of 200 milliseconds. We use standard LVM for all our filesystems. Thanks for your comments though

[Alzhy]

I understand but how do you build your filesystems/storage? DO you stripe accross several LUNs or use a LUN for each filesystem?

[Adam Noble]

Nelson in this instance we have 10 LUNS (obviously we see them as physical disks) within one volume group. I cannot tell you the back end configuration, however I believe striping takes place there aswell. One particular disk was being hammered in this stripe set and one in another 10 LUN/disk set. Historically as you would expect I can see that the disk utilisation for the disks within the stripe sets are almost identical as you would expect. This morning however these 2 disks one from each stripe set were being hammered independantly of the others.

[Alzhy]

Okay so you do stripe accross the LUNS (aka disks or PVs)? Do you stripe accross the 10 disks (aka LUNs)? Also, are the LUNs presented on different HBAs?

Ideally (and this is what I have been doing with EMC's) .. I present the LUNs equally on however many number of HBA's the server has. If say the server has 4 HBA's - I build my striped LVOL or VOL (on the host side using LVM/VxVM) accross the 4 LUNs with a LUN on each HBA. If I build an 8 way stripe - I build it accrss 8 LUNs with 2 LUNS from each HBA...