Performance Issue on 11.31

[Gireesh]

Hi All,

The problem is reported by Application team.This application is intensively use sequential IO activity and file will be big like 10,25 or even up to 50gb files.

[Gireesh]

Hi All,

The problem is reported by Application team.This application is intensively use sequential IO activity and file will be big like 10,25 or even up to 50gb files.

HP engineer suspect this could be because of file cache setting.I am going to change it to check.

[Dennis Handly]

>Michael: You mean the high end superdome that starts with 1 to 16 cores

HP's marketing blurb says rp7420-16 is only midrange. It's not a super computer.
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c00337654

>Read the test material Dennis.

Yes for that case but if you read my reply, I changed things.

>if you don't know what 100% busy, or 0.00% avwait time

I was agreeing with your sar(1m) interpretation, even though it didn't make sense for "simple unix copy of 15g".

[Duncan Edmonstone]

Gireesh...

I think possibly our questions and your responses have been a little bit hit and miss on this thread so far. If you really want to solve this issue we probably have to tackle it from first principles and then understand everything that is differemt between the 2 systems.

Lets start with what actually runs slowly.. you say it is a "simple unix copy of a 15g file".

Can you actually share the command line that you have been using to run your tests on. I refuse to beleive a "simple unix copy" can be CPU bound, and as such I'm going to continue to maintain we have an IO related issue here even if we see no avwait on sar outputs.

So I'd like to see for both the rp7420 and the bl860c all of the following:

1. the command you run to do the copy (just a cp?)
2. if you have tusc installed on these systems it would be nice to see the length of time spent in read/write system calls... e.g. if your copy command was usually "cp /dir1/file1 /dir2/file1" then "tusc -Cs read, write cp /dir1/file1 /dir2/file1" would give that data. You should try and collect this data on both systems immediately after the filesystem is mounted, and then see what differemce you get when you repeat the operation with a warm cache. I'm interested in this beacuse from the sar outputs you posted, all the long service timnes appear to be associated with read operations rather than write operations.
3. The filesystem mount options for both the source and target filesystem in the copy - also if you have anything in the /etc/vx/tunefstab
4. The LV and VG configurations for the source and target filesystems (lvdisplay -v / vgdisplay -v etc.)
5. the amount of physical memory and the kernel parms for buffer cache (dbc_min_poct and dbc_max_pct on 11.11 and filecache_max and filecache_min on 11.31)
6. the SCSI queue depth on all LUNs involved in the IO operations (use "scsictl -m queue_depth " on 11.11 and "scsimgr get_attr -D -a max_q_depth" on 11.31)
7. You mentioned you changed to preferred path for the 11.31 load balancing - can you show me how you did that (it's easy to get wrong...)

there's probably more, but that should get us going...

HTH

Duncan

[Michael Steele_2]

Hi

"...application is intensively use sequential IO ..."

Please post 'sar -b' reports.

Question: Is this a database or a series of flat files being written?

You should also attach all the regular performance analysis commands like vmstat, sar -d,u, etc.

[James Munroe]

Hi Gireesh,
Were you able to find an answer to this issue? We are noticing similar i/o performance issues with the BL860C running 11.31.

Thank you,
James

[chindi]

Hi after changing max queue depth parameter to 128 all was fine for us .