EVA 3000 Performance

[Victor_101]

We suspect that our EVA3000 is facing some performance issues in the fact that in the since last month we have been evaluating a SATA Disk Array for Intermediate Backup purposes. As such, investigation of the EVA 3000 performance (attached) suggests that its maximum sustained throughput is not more than 80Mb/sec (actual data transferred) while reading and not more than 120 MB/sec while writing.(not speeds but Megabytes transferred)
We are primarily worried about the Read speeds.The 80 MB is reached after several repetition of the same process ( can be attributed to Cache?)
As such , we are looking for an alternate means of monitoring the actual throughput to validate if these speeds are true and acceptable since the Data Sheet of the EVA 3000 speculates the Maximum Sustained Throughput to be 335MB/sec and we are currently just at 1/5th those speeds. Are there any kernel paramaters in place that are limiting these speeds since we even monitored the Port Utilization on the SAN Fabric switches and those too are not peaking to maximum.

The read speeds are our main concern , the 80 MB/sec read is achieved by repeating DDs , hence probably attributed to Cache. The 1st time a read is performed on a 5 GB file we get not more than 40 MB/sec. if the DD is for a larger file, in our case 350GB of Raw data files as one backup, we get worse figures and the cache will not help



We have an Oracle RAC 2 node CLuster of Rp5470s connected via 2GB FC to an EVA 3000 with 26*73GB disks (15k rpm) with Vraid1.

Is the Raid level a problem ?

What tools can I use if I need a better measurement of performance?
We are currently just using DD to measure throughtput.

Cheers
Victor

[Ivan Ferreira]

You can download and use the EVAPERF utility from the EVA download section.

There are some filesystem parameters that you should tune to get best performance, this parameters are the journaling options, the block size, the sparse super, stripe size. Use the mkfs and tune2fs options to adjust these parameters.

You can use dd on the raw devices to check the impact that the filesystem is having in your tests.

Use iostat to check the bps and tps on the disks, you need to have installed systat tools.

[Bill Costigan]

The real trick is to figure out where the bottleneck is. In this case you are sequencially reading through the files so you rarely read the same block twice. Thus you can't expect the block to be sitting around in cache.

So if the array is doing read-ahead to try to get the blocks into cache before you get around to asking for them, the array can only fill the cache as fast as the disk can provide data.

Assume a 15K disk can do 200 reads per second. You have 26 disks doing 200 reads per second and 8196 bytes per read.

26 * 200 * 8196 = 42.5MB/sec.

That means the data is coming into cache at 42.5MB /sec so if cannot go out any faster.

Note: writes would be half of that because with raid 0/1. each write has to go to 2 disks.

When HP [or anyone else] publishes I/O rates they typically use an array with the maximum number of disks. With enough disks (assuming the load is balanced) the next bottleneck may be the fiber channel or controllers.

FATA drives will be slower than the 15K
I don't know what drive technology you were testing. The I/O rate per disk will depend on how localized the data is on the disk. I've seen some 15K drives approach 500 I/Os per second and drop to about 120 when the data is scattered.

In any case, the numbers you are seeing are diectly related to the number and speed of the disks.

[Victor_101]

thanks Bill & evan
were running 26 FC disks in 2 enclosures that can hold a maximum of 28 disks. were running them over 2 GB FC. and while monitoring the port utilisation on the SAN switches (brocade) ,we see that the traffic is getting evenly distributed across the ports.. hence load balancing is working.. but to what effect..
main question is .. is this the maximum speeds?? or does the eva perform faster?
if it should. then i will log a call with HP and ask them solve it..

[Victor_101]

Hi Guys,
I am attaching the EVAPerf statistics from the HOST PORT UTILISATION. Can anybody interpret?

Thanks

Attachment is Excel sheet

[Ivan Ferreira]

The maximum speed you will get using the raw devices. The operating system overhead and the server speed are also performance factors, even more if they are clusterized and using a cluster filesystem. OCFS is very slow in this type of tests (if you use it).

"The 80 MB is reached after several repetition of the same process ( can be attributed to Cache?) "

Yes, maybe is filesystem cache.

[Bill Costigan]

You might get more speed by increasing the blocksize. But basically you are being limited by the speed of the 28 disks.

Arrays try to maximize the efficiency of each I/O by trying to transfer the most data per I/O (concatenating I/Os) or doing them at times when the host isn't waiting for them. (read ahead, and write caching)

The bottom line is 28 disks can only do so many I/Os and I'm sure you are hitting this limit. Adding more disks should increase your throughput.

It's not that the eva is slow, It's just sitting around waiting for the disks to complete their I/Os.

[generic_1]

I would call HP, because they sold you this product with the expectation that it will work.
I suspect you may need something with a little more horse power. An XP with more disks would probably be a better match.
It will have more cache, more I/0, more fibers,real active/active controllers, holds allot of disks.
If you do go this route carve small luns and make say a meta 3 that is 13 gigs in size. Have hp preaddress the luns so they are spread evenly accross the controllers for performance, not down the fibers. Allocate as many disks in the correct lun order to your vg. If you use bcvs put em inside the spindle.



On your eva. Talk to oracle and make sure your memory/swap and kernel parameters, and system resources are not an issue.

I would create multiple luns the same size. Do a Lvcreate -L size with -i -I
use a stripesize of 1024 oracle likes that
adn the number of disks you create

This should help you by allowing more cache per lun and lowering the probability that the same lun is being hit by to different IO requests at the same time which creates a random search. This is bad because your caching and prefetch goes to hell. Hoepfully the EVA is smart enough to do this at lun level vs disk. The other problem you are going to be fighting with this design is it decides where its putting the data in the disk group and you have no control over it whether the built in intelligence is right or wrong.

[Tom O'Toole]

Are these figures for single threaded performance? If so, I don't think they are all that bad. Also, the 335MB/s number is a maximum, so probably assumes all cache hits.
As others are saying, performance (specifically, multithreaded total throughput) will go way up when you add disks.

Can anybody actually verify (or comment on) Jeff's idea of using many little LUNS and striping them at the host level with an EVA?
I'm skeptical, but interested in discussion.

[Leif Halvarsson_2]

Hi,
I have a similar, strange performance problem with one of our EVA3000 array.

- General performance is poor.
- Write (restore) is faster then Read (backup).
- A EVA 3000 with 8 disks performs better then a system with 56 disks.

You can find more details about my problem here:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=888255

I have logged a support case to HP for this problem but, not yet got any useful feedback.


About striping LUNs:

Assuming you are using only 1 disk group, it is no idea (for performance reasons) to stripe LUNS. As I have found in the EVA documentation, it is possible to get the specifyed throughput, using just one single LUN.

[Virgilio Santiago]

Hi to all

I am currently experiencing a similar problem as described in the following thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=774143

Unfortunately, I discovered this discussion thread when my 2 evas are already in production. Now I find myself with worse performance than the obsolete hardware I just replaced.

[Victor_101]

Hi guys .
Any updates ??? I have already contacted HP ie. through the guys who sold it to me.. but to no avail...

Please help.. ASAP

[Dave La Mar]

Victor -
We just went through an analysis on our 5000. One of the recommendations was changing the kernel parm scsi_max_qdepth from 8 to 12.
This can also be changed on a device level rather than a global kernel parm. As a matter of fact we are tracking the queueing today for a 24 hour period. Then next week we will change the value using scsictl command and track on the that Tuesday. In onther words, our bottle neck was in disk queueing. Since this type of change can affect service time as well, we are tracking service time over those two days as well.
In looking at your output, it does seem a bit low. I would guess you need to turn on other metrix for your report. i.e. (DISK):BYDSK_AVG_SERVICE_TIME and
(DISK):BYDSK_REQUEST_QUEUE
At least you can eliminate these two factors.

Will be interested to hear the outcome as it seems you have company on this issue.

Regards,

dl

[Victor_101]

Hi Dave,
I did get this recommendation from HP last month to Change SCSI_MAX_DEPTH parameter from the default 8 to 64. I did get an increase in performance by 30% , but not really the figures I was looking for...
reading a 5 GB file completes in 45 secs .. but writing which originaly was taking 2min15 secs came down to 1min50secs... to u see.. still a vast difference in speeds...
is HP looking into the matter as a probable bug?? maybe we need to get their attention with regard to this ..
i will do so at my end.. at the same time.. with regard to the matter above.. i did ask them if i increase the above Kernel parameter more ( max allowed =255) would if get better.. ??? no reply yet..

with regard to "
In looking at your output, it does seem a bit low. I would guess you need to turn on other metrix for your report. i.e. (DISK):BYDSK_AVG_SERVICE_TIME and
(DISK):BYDSK_REQUEST_QUEUE "
what do you mean ? where can i monitor these values ? i added all the available options under the Evaperf tools to be monitored... am I missing something ??

how are ur tests goin for u now ?? must be a couple of hours since u started i guess

victor

[Dave La Mar]

Victor -
The monitoring I am using is with Glance in Adviser mode.
I am not sure of the equivalent metric syntax in EVAPerf, but seems like you have the user guide.
In any event, to check the queueing and service time using glance, the metrics are:
bydsk_request_queue
bydsk_avg_service_time
Neither is case sensitive and can be all caps.
The report runs from the HP server.
I'm attaching a down and dirty script for capturing the values by disk, at 10 second intervals.
1. We started the script yesterday at 14:00 and it will end at 13:00 today.
2. We will then look at the data and adjust the queueing by disk using the scsictl command.
3. On Tuesday fo next week we will run the script for the same time period and compare the results.

Excuse the roughness of the script, but is is only for this temporary task.

Will follow your thread.
I was afraid to raise the kernel parm as high as you did. A 30% increase is significant, though in the context of poor performance, this is not always true. By raising it that much, you might ask HP what else was affected.
We were told to watch and monitor the service time because increasing the queue depth too much would likely result in higher service time.

In the sample output attached you will see disk 255/0.0.5 doing queueing at that given snapshot.
My initial look at our current data capture seems to indicate service time variation is more prevalent than queueing. I'll be running all this by HP.

Best of luck with this.

-dl

[Dave La Mar]

Victor -
Thought I would pass on the update of our data collection for the queue depth change. As it turns out, HP has agreed that no change in our environment is necessary. Our periods of disk queueing were small and varied over different disks.
Wish I had more to offer, but it appears our environments must differ too much from yours.
As before, best of luck on your performance challenge.

Regards,

dl