1824172
Members
2664
Online
109669
Solutions
First things first are to check you have all the correct director bits set on the DMX for 11.31. They are _not_ the same as for 11.11. Speak to your storage team or EMC to check this out.
So what sort of MPIO were you using on the 11.11 host? PowerPath? LVM PVlinks? nothing at all?
I ask because some disk arrays (and I can't comment on whether the DMX4 is one of those), don't do very well at handling sequential IO when those IOs arrive via differemt ports on the array. One big copy operation will generate just sequential IO.
If the 11.11 system is using PVlinks then actually all the IO will be going to just the one port on the DMX, whereas the default with the 11.31 MPIO will I expect be round-robin. If the DMX can't detect and optimise when sequential IO arrives down different ports, it could well end up being slower.
If this is the case you might want to change the load balancing policy - see this WP for how:
http://docs.hp.com/en/native-multi-pathing/native_multipathing_wp.pdf
I'd imagine that "preferred path" or "weighted round robin" might give better performance for sequential IO (if what you really want is better sequential IO performance - this might make random IO worse)
HTH
Duncan
I am an HPE Employee
So what sort of MPIO were you using on the 11.11 host? PowerPath? LVM PVlinks? nothing at all?
I ask because some disk arrays (and I can't comment on whether the DMX4 is one of those), don't do very well at handling sequential IO when those IOs arrive via differemt ports on the array. One big copy operation will generate just sequential IO.
If the 11.11 system is using PVlinks then actually all the IO will be going to just the one port on the DMX, whereas the default with the 11.31 MPIO will I expect be round-robin. If the DMX can't detect and optimise when sequential IO arrives down different ports, it could well end up being slower.
If this is the case you might want to change the load balancing policy - see this WP for how:
http://docs.hp.com/en/native-multi-pathing/native_multipathing_wp.pdf
I'd imagine that "preferred path" or "weighted round robin" might give better performance for sequential IO (if what you really want is better sequential IO performance - this might make random IO worse)
HTH
Duncan
I am an HPE Employee
>> If I monitor my FC t-put I am getting 46MB/s for the first time copy but if I do the same copy second time it will shootup to 80MB/s.
So probably a result of your filesystem buffer cache - maybe you have different settings on the filesystems you are using on 11.11 vs. 11.31 - compare using "mount -p" on both systems.
>> I have noticed one more thing ,while I am using dd command ,server is giving good thru put of 70MB/s.I am not sure how it is.
If you are using dd to the raw device you are again avoiding biffer cache, so this backs up that the effect you are seeing may not be disk related. What performance do you get if you run the raw dd on the 11.11 system?
HTH
Duncan
I am an HPE Employee
So probably a result of your filesystem buffer cache - maybe you have different settings on the filesystems you are using on 11.11 vs. 11.31 - compare using "mount -p" on both systems.
>> I have noticed one more thing ,while I am using dd command ,server is giving good thru put of 70MB/s.I am not sure how it is.
If you are using dd to the raw device you are again avoiding biffer cache, so this backs up that the effect you are seeing may not be disk related. What performance do you get if you run the raw dd on the 11.11 system?
HTH
Duncan
I am an HPE Employee
>> dd if=/dev/rdsk/c14t9d7 of=/dev/rdsk/c16t9d6 bs=1024k count=500000.
Also be awrae that unless you have set the load balancing to something different, then using the legacy device files will still cause round robin'ing between all paths to the disk.
You should figure out which agile DSF this legacy DSF is assocaited using "ioscan -m dsf", then post the output of:
scsimgr get_info -D /dev/rdisk/diskNN
replacing diskNN with the agile DSF you indentifed.
HTH
Duncan
I am an HPE Employee
Also be awrae that unless you have set the load balancing to something different, then using the legacy device files will still cause round robin'ing between all paths to the disk.
You should figure out which agile DSF this legacy DSF is assocaited using "ioscan -m dsf", then post the output of:
scsimgr get_info -D /dev/rdisk/diskNN
replacing diskNN with the agile DSF you indentifed.
HTH
Duncan
I am an HPE Employee
Gireesh...
I think possibly our questions and your responses have been a little bit hit and miss on this thread so far. If you really want to solve this issue we probably have to tackle it from first principles and then understand everything that is differemt between the 2 systems.
Lets start with what actually runs slowly.. you say it is a "simple unix copy of a 15g file".
Can you actually share the command line that you have been using to run your tests on. I refuse to beleive a "simple unix copy" can be CPU bound, and as such I'm going to continue to maintain we have an IO related issue here even if we see no avwait on sar outputs.
So I'd like to see for both the rp7420 and the bl860c all of the following:
1. the command you run to do the copy (just a cp?)
2. if you have tusc installed on these systems it would be nice to see the length of time spent in read/write system calls... e.g. if your copy command was usually "cp /dir1/file1 /dir2/file1" then "tusc -Cs read, write cp /dir1/file1 /dir2/file1" would give that data. You should try and collect this data on both systems immediately after the filesystem is mounted, and then see what differemce you get when you repeat the operation with a warm cache. I'm interested in this beacuse from the sar outputs you posted, all the long service timnes appear to be associated with read operations rather than write operations.
3. The filesystem mount options for both the source and target filesystem in the copy - also if you have anything in the /etc/vx/tunefstab
4. The LV and VG configurations for the source and target filesystems (lvdisplay -v / vgdisplay -v etc.)
5. the amount of physical memory and the kernel parms for buffer cache (dbc_min_poct and dbc_max_pct on 11.11 and filecache_max and filecache_min on 11.31)
6. the SCSI queue depth on all LUNs involved in the IO operations (use "scsictl -m queue_depth" on 11.11 and "scsimgr get_attr -D -a max_q_depth" on 11.31)
7. You mentioned you changed to preferred path for the 11.31 load balancing - can you show me how you did that (it's easy to get wrong...)
there's probably more, but that should get us going...
HTH
Duncan
I am an HPE Employee
I think possibly our questions and your responses have been a little bit hit and miss on this thread so far. If you really want to solve this issue we probably have to tackle it from first principles and then understand everything that is differemt between the 2 systems.
Lets start with what actually runs slowly.. you say it is a "simple unix copy of a 15g file".
Can you actually share the command line that you have been using to run your tests on. I refuse to beleive a "simple unix copy" can be CPU bound, and as such I'm going to continue to maintain we have an IO related issue here even if we see no avwait on sar outputs.
So I'd like to see for both the rp7420 and the bl860c all of the following:
1. the command you run to do the copy (just a cp?)
2. if you have tusc installed on these systems it would be nice to see the length of time spent in read/write system calls... e.g. if your copy command was usually "cp /dir1/file1 /dir2/file1" then "tusc -Cs read, write cp /dir1/file1 /dir2/file1" would give that data. You should try and collect this data on both systems immediately after the filesystem is mounted, and then see what differemce you get when you repeat the operation with a warm cache. I'm interested in this beacuse from the sar outputs you posted, all the long service timnes appear to be associated with read operations rather than write operations.
3. The filesystem mount options for both the source and target filesystem in the copy - also if you have anything in the /etc/vx/tunefstab
4. The LV and VG configurations for the source and target filesystems (lvdisplay -v / vgdisplay -v etc.)
5. the amount of physical memory and the kernel parms for buffer cache (dbc_min_poct and dbc_max_pct on 11.11 and filecache_max and filecache_min on 11.31)
6. the SCSI queue depth on all LUNs involved in the IO operations (use "scsictl -m queue_depth
7. You mentioned you changed to preferred path for the 11.31 load balancing - can you show me how you did that (it's easy to get wrong...)
there's probably more, but that should get us going...
HTH
Duncan
I am an HPE Employee
