Strange I/O performance issue

[Joe Short]

I have a client that is experiencing a very peculiar performance issue. They have an rp7410 running HP-UX 11.11 The storage is a SAN attached Clarrion CX3-40 array. There are 2 HP AB678-60001 4GB HBA's connected to the SAN.
The problem is this, when copying data into file systems created on this array, the copy will start out moving quite quickly. Then as it nears about 95% completion, the performance drops to a near halt.
I was able to repeat this result using a small 2GB file as a test.
I ran an fsck on the file system, and found no errors at all. But the problem remained.
So, I removed the entire volume group (vgexport) and completely rebuilt it from scratch. I ran pvcreate -f on all of the LUN's in the volume group, the vgcreate -p 255 -s 32 to create it. The file system is comprised of 4 50GB LUNs.
After rebuilding the file system, the problem seemed to be resolved. So I turned it back over to the DBA who then began copying a large quantity of data inot the file system to clone a database. He was able to copy 53GB into the file system in about 2.5 hours. Then it throughput nearly completely stopped.
I again ran my 2GB test, and after 40 minutes, it did not complete. Glance shows negligable IO of any type during the test.
We are not using PowerPath, just PVLinks.
I cannot figure out why this would work well for a short time, then stop functioning.
Any advise or clues as to where to look would be most welcome.

[Jeeshan]

Hi Joe

There may be many things you need to checkup.

Coz this is a performance related issue. may be happened both in HP and EMC.

In EMC part do the checks of
read cache value and its usage
write cache value and its usage
watermark values and throughputs of storage group.


and also check in HP that sar -b outputs
and also check the pv time out value.

If possible please ask EMC for through checkup in their system.

[Joe Short]

The read and write cache are activated on the array. There are no other systems connected to this SAN or array yet, so benchmarking elsewhere is not a possibilty.

The pv timeout is set to a value of 90, as prescribed by EMC.
I am now seeing the same issue on other file systems within this array as well.

[chris huys_4]

Hi Joe,

> They have an rp7410 running HP-UX 11.11 The
> storage is a SAN attached Clarrion CX3-40
> array.
There is youre problem. You have a (emc) clariion diskarray. ;)

> The problem is this, when copying data into > file systems created on this array, the
> copy will start out moving quite quickly.
> Then as it nears about 95% completion, the
> performance drops to a near halt.
> I was able to repeat this result using a
> small 2GB file as a test.
What was the exact command you used to copy the file ?

2Gbyte is very close to the largefiles attribute that begins with 2 Gbyte. Is this attribute set on the filesystem you copy to ?

I suppose you also have 2 internal disks. Do you have the problem if you copy the 2 gbyte file from a filesystem that is created on 1 internal disk to a filesystem that is created on the other internal disk.

During the copy issue, the commands.
#sar -d 1 1000
#sar -b 1 1000

And if all else fails, log a call with support.

wtec loves to run their tools, to get this sort of problems sorted out. ;)

And ps, for the oracle thing, get oracle (consultant) to comment on the performance issues first, via statspack or whatever they use.

Greetz,
Chris

[Joe Short]

The largefile setting is set on these file systems. Further testing revealed other file systems on the SAN that were exhibiting the same behavior.
A supprt call resulted in me being diverted to EMC. Support says it's a hardware issues with the array. But the array does not show any hardware issues.

[Ken Metrunec]

Hi,

The site:
-------------
HP-UX 11.11, 2006 Gold Bundle, Mirror/UX and OnlineJFS
A. 2xn4000s => va7410, 2Gbit FC direct attached, with LUNS ranging 50-200GB
B. 1xn4000 => FC 2Gbit JBODS (3) chained config with 1 disk per filesystem.
-------------

This may not be your problem root cause; however...

I found similar tail-end bad performance problem during large file (2-4GB) transfers in vxfs file systems. Some Copies (cp, tar, cpio) & netbackup took 2 GB in 2 hours instead of 2-4 minutes.

It was most noticeable with netbackup. The KnowledgeDB gave a solution that worked for me: increasing the vxfs parameter discovered_direct_iosz from default 256K, and in my case, I had to go to 1GB to get my netbackup performance back.

The cause was HP-UX poor performance mixing buffered and direct I/Os, which is determined by the vxfs paramter above.



Not sure if this OK, but I put in the excerpt from doc in case you can't to it:
.............
Submitted Date: Thu Dec 14 06:00:00 GMT 2006
Title: HP-UX 11i File System and Storage Software - Slow Restore Performance with Symantec Netbackup
Document ID: emr_na-c00821409-2
Last Modified Date: Thu Dec 21 06:00:00 GMT 2006
..............


=================================
Ken
+++

[Joe Short]

It turned out to be a faulty fibre channel cable. Everything has been fine since replacing it.

Thank you to everyone for all of the help.