Disk utilization problem

[Yazan Yacoub]

Hello,

when I do sar -d 5 5 | grep (diskx) i got the following result:

Diskx 100 31.3 828 332849 0.0 37.8
Diskx 100 27.2 1133 270135 0.0 24.0
Diskx 100 28.1 929 317121 0.0 30.2
Diskx 100 31.9 1031 314083 0.0 30.9
Diskx 100 26.1 826 271291 0.0 31.6
Diskx 100 28.9 949 301095 0.0 30.4

1-System is connected to two EVA's with 2HBAs,
2-this server is running oracle database.

sar -u 5 5 result:
14:54:16 %usr %sys %wio %idle
14:54:21 46 17 28 9
14:54:26 46 15 29 11
14:54:31 60 17 17 6
14:54:36 51 15 23 10
14:54:41 41 14 25 19

Average 49 15 25 11

sar -b 5 5 result:
14:55:06 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
14:55:11 2 19435 100 3 181 98 0 0
14:55:16 0 19614 100 4 932 100 0 0
14:55:21 1 34755 100 3 12764 100 0 0
14:55:26 0 41725 100 18 13709 100 0 0
14:55:31 0 41150 100 5 14143 100 0 0

Average 1 31340 100 7 8348 100 0 0

I have checked the HBA's and no bottleneck found, and I checked Storage controller, and no singe of bottlenecks.

Please advice,

[Duncan Edmonstone]

Don't worry about the 100% utilisation number - that just means the disk was doing something for the entire interval of measurement (a good thing!)

WHat is concerning is the q length (queued IOs) and the service time (30ms is *not* good for an EVA). Also note that for the blk/s column were' seeing up to 332849 blocks per second (a block being 512 bytes). That works out at about 160MB/s - quite a high IO rate.

QUestions:

1) WHat OS version?
2) What is on that LUN? What are you doing to generate this amount of IO?
3) What generation of EVA? Any MPIO software?
4) Any logical volume configuration in the mix?
5) IS the EVA LUN replicated using Continuous Acess?

HTH

Duncan

[Yazan Yacoub]

Many thanks for your reply,

But I am little bit confused, since I am not able to find bottleneck from HBA side, and thats mean no high IO rate on HBA, is my problem on OS level ?

I will send the the info as soon as possible since I am not the system admin, but what I know its running Oracle Database and we have 2 EVAs5000 hosting the LUNs,
EVA is not replicated.
Many thanks,

[likid0]

Yes, please give us some info, hp-ux version, release, virtual machine?


Have a look at what it's on that disk,is it a oracle .dbf? :

pvdisplay /dev/dsk/diskx

and have a look what lvols are on the disk.

Then go and talk to your dba admins and check what tablespaces they have on those .dbf and why they are burning them to death.

[Yazan Yacoub]

Hello,

I got required information, but you will not be happy HP gurus :)

Its Solaris 9 machine
with Solaris volume manager running on it.

Any ideas ?

[yulianto piyut]

from your output sar -d. the average service time is high and busy is 100%. it's mean that all disks used by database. maybe you can add hba to server or add port in storage.

[Eric SAUBIGNAC]

Bonjour,

You said "and I checked Storage controller, and no singe of bottlenecks.".

Though I don't know anything about Solaris, I guess that sar outputs have about the same meanings than in other unix. So I don't agree with your postulate that there is no bottleneck at EVA side : queue permanently above 26, large sustained amount of IO around 1000, 150 Mo/s which is a heavy load, and a delay at around 30ms from storage and 0 from unix side ...

You should go back to EVA management team and ask them to analyze some things like % of cpu used by controlers, efficiency of cache, loadbalance between controllers, IO rate related to number of physical disks, etc ...

And what are you doing to this poor storage array ? Backup (large bloc size around 150 Ko) ? If yes maybe the situation is not so serious.

Eric

[Yazan Yacoub]

Hello,

Sorry I was mistaken, I/O's were not distributed equally on my case to the HBA's, I have distributed them equally, problem solved.

Thanks for all

[Yazan Yacoub]

see above