Disk Arrays
cancel
Showing results for 
Search instead for 
Did you mean: 

VA7410 sudden reduction of I/O throughput

SOLVED
Go to solution
karza167
Occasional Advisor

VA7410 sudden reduction of I/O throughput


While in the middle of copying data from a filesystem holding Oracle datafiles into another filesystem on a VA7410 array, the I/O throughput was dramatically reduced.



The server is rp4440 server with multiple CPUs. SG-cluster and applications were offline, the system load was minimal and there is no significant other I/O in the arrays.

SOURCE /d/db filesystem exists in a EMC CLARiion CX-700 array (it holds about 540 GB data)
TARGET /d/backup filesystem exists in a VA7410 array. The filesystem usage can be seen below:

Before starting:
Filesystem kbytes used avail %used Mounted on
/dev/vgbackup/lvol1
1611792384 692004256 912873936 43% /d/backup
/dev/vgdb/lvol1 816054272 568495080 245698640 70% /d/db

After:
/dev/vgbackup/lvol1
1611792384 989401624 617799920 62% /d/backup
/dev/vgdb/lvol1 816054272 568495112 245698608 70% /d/db


The purpose was to take an offline backup. For this activity 'cp' command was used for the backup.

After 305 GB of the data was copied the I/O throughput was reduced into one tenth of the original speed.
(From 6GB/min into 0.5 GB/min). Because of the reduced I/O, the activity had to aborted.

Before posting more technical details, I'd like to know has anyone experienced similar problems before?

armdsp does not show any HW faults or Warnings.

ARRAY INFORMATION

Array Status:_________________________Ready
Firmware Revision:____________________38370A120P1111031050
Product Revision:_____________________A120
Local Controller Product Revision:____A120
Remote Controller Product Revision:___A120
Last Event Log Entry for Page 1:______37416146
Last Event Log Entry for Page 2:______37415811
Last Event Log Entry for Page 5:______36983827

CAPACITY USAGE

Total Disk Enclosures:________________4

Redundancy Group:_____________________1
Total Disks:________________________28
Total Physical Size:________________934.6 GB
Allocated to Regular LUNs:__________782.024 GB
Allocated as Business Copies:_______0 bytes
Used as Active Hot Spare:___________0 bytes
Used for Redundancy:________________152.315 GB
Unallocated (Available for LUNs):___267 MB

Redundancy Group:_____________________2
Total Disks:________________________28
Total Physical Size:________________934.6 GB
Allocated to Regular LUNs:__________798 GB
Allocated as Business Copies:_______0 bytes
Used as Active Hot Spare:___________0 bytes
Used for Redundancy:________________135.991 GB
Unallocated (Available for LUNs):___624 MB

WARNINGS

None
12 REPLIES
Torsten.
Acclaimed Contributor

Re: VA7410 sudden reduction of I/O throughput

Can you post a full "armdsp -a"?

I noticed already the old firmware.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

I tested the copying again. It started with a throughput between 3-5GB/min but then was reduced back to 0.5GB.

I run sar -d 1 10 while the copying was in progress.

The vgbackup volume group consists of two LUNs which device files in OS are:

c6t0d1 and c6t0d2

In the sar output for some reason only the second device file shows busy% close to 100%.

I find this strange because the logical volume is LVM striped.

Attached:
-armdsp -a , full output


karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

attached:
-sar -d 1 10 output
-vgdisplay -v vgbackup
tkc
Esteemed Contributor

Re: VA7410 sudden reduction of I/O throughput

your source is a VA7410 array, and are you running on autoraid mode? in order to investigate the slowness, you are suppose to collect the armperf data.
karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

It's the other way around, the VA7410 is the target. Anyway, I will try to collect the armperf data(haven't used that utility before.)
tkc
Esteemed Contributor

Re: VA7410 sudden reduction of I/O throughput

since the va7410 is the target, there will be a write penalty to va7410 due to parity calculation, hence the slowness.

the armperf command to run are following :

#armperf -c OPAQUE -x COMMA -s mmddhhMM[yyyy]
#armperf -c ARRAY -x COMMA -s mmddhhMM[yyyy]

the output can be displayed from ms-excel.
karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

What about the fact that the throughput was ten times better at the beginning of the copying. Ie. half of the data was copied with 'normal' rate first and the rate dropped suddenly. Shouldn't the parity penalty apply right from the start.
tkc
Esteemed Contributor

Re: VA7410 sudden reduction of I/O throughput

the va7410 could be busy modifying the data in the disks from one raid 5DP to another raid 1/0. that's how autoraid works.
karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

Here's an excel sheet attached with the output of the armperf command using the options you mentioned.

The data is gathered from the time of the incident. You can see big drop in the performance in the 11.02 sample. Maybe this will give additional clues what happened.
tkc
Esteemed Contributor
Solution

Re: VA7410 sudden reduction of I/O throughput

your RAID 5 DP Allocation has dropped tremendously during that time. you have used up the free space in the array that made the array to start busy modifying existing data from RAID 1/0 to RAID 5DP. do you have some spare disks for this array? try to add more disk space to help the writing process perform better.
tkc
Esteemed Contributor

Re: VA7410 sudden reduction of I/O throughput

your 'RAID 5 DP Writes in Place' was also high during that time. Write in place means you are writing onto existing RAID 1/0 space as RAID 5DP. So these writes are basically read, modify before actual write which is why the whole process of writing is that slow. This will help preserving the much space needed now in the array because of more writes. i can't suggest much other than adding more disk space or you can write to the array for abt 20 mins, let it stop for a while for the array to optimise its space and do the writing again at a later time.
karza167
Occasional Advisor

Re: VA7410 sudden reduction of I/O throughput

Thank you for the explanation! Now that I have some understanding how the Autoraid works, your reasoning makes perfect sense.

In this case we don't have a possibility to add any HW to the array. Also because of the intended usage of this array, it is not an option to write to the array the way you suggested. On the other hand, it is possible to reformat the array and choose the RAID 1+0 option which should have a more consistent throughput.

I was able to download the documentation and also found some posts in this forum how this can be done. While at it, I'll probably have the firmware upgraded as well.