disk I/O slowness on HPUX 11.23

[wangjs5]

Hi experts,

We have two clustered rx6600 servers running HPUX 11.23. Same EMC Symmetrix disks are assigned to both servers. Hardware and OS setup are identical. They both run fully licensed Powerpath with policy SymmOpt without any error.

When I run "dd" command to test the same lun from both servers like this:

timex dd if=/dev/rdsk/cXtXdX of=/dev/null bs=2048k count=1000

Server A only takes 25 seconds, Server B takes 150 seconds. I have tried several disks, some are activated on server A, some are activated on server B, but the results are similar.

By looking at glance, server A can push I/O to over 50000 KB/sec with percentage over 80%, but server B can only push I/O to 3000 KB/sec with percentage like 25%. Both systems are pretty much idle while testing.

It seems something on server B blocks the I/O.

Does anyone have any ideas why one server is significantly slower than the other server when accessing the same lun?

Thanks in advance,
Jason

[Ron Irving]

Hey Jason!!

connection between server and disks perhaps? Fiber, right? Try swapping out the fiber cables.

My 5c worth.

Ron

[wangjs5]

Every server has 4 fiber connections through different switches and FAs to the same lun, so I doubt it's the physical connection problem.

Thanks.

[Steven E. Protter]

Shalom,

I would make sure both machines have the same hardware enablement depot from HP, this has an impact.

It would then check with SAN utilities to check the speed of the connection.

I'd try swapping out fiber cables to see how what works with the server B I/O issues.

Next I'd run mstm cstm or xstm and test the hardware for flaws.

SEP

[Tim Nelson]

There could be a million reasons best to start disecting it one by one..

-is everything really the same ? it rarely is. firmware, ports, switches, I was told once by EMC that depending on the switch model there are performance differences where you plug hosts and FA into. e.g. the backplane is divided, hence a host in port 1 and an FA in port2 will talk faster than a host in port 1 talking to an FA in port 12.

-are you absolutely positively sure that both servers use the same paths during your test ? perhaps there is contention on one path or another. remove all but one path on each, leave the same path, with 1 hba, via the same switch, to the same FA. ( you probably cannot do that so find another way )

how are you reviewing the io stats ? via powermt or something else. powermt may at least tell you if the same path is being used.

-have the tests been repeated from each server to average out the abnormalities, maybe you got lucky on one one.


-have you looked at the stats during both tests from the EMC side to look for abnormalities during one iteration vs another ?

just shooting out ideas here... maybe it will lead to something.

best of luck.

[chris huys_4]

Hi Jason,

Run sar -d 1 200, while doing the timex tests and post the results, of both the timex output and the sar -d output.

What model of EMC is connected ?

And do the rx6600 have any internal disks, were you can do similar tests, to exclude the OS ?

Check also the /var/stm/logs/os directory of the 2 servers. If a lot of logX "have filled up" in the last few days, ls -ltr log*, there might be a hardware problem, get then the "last logX, i.e. logX.raw.cur, examined. (via cstm)

Check also /var/adm/syslog/syslog.log of the 2 servers, for any strange powerpath/lvm/scsi errors.

Greetz,
Chris