VA7400 Performance

[Thomas Maddox]

I have a possible performance on a VA7400. When disk queues are high, users connecting to our Oracle 8i database through a web based Java Application get disconnected. Also, jobs running on other systems are slow.

I have 2.1TB usable on my VA7400 and over 3TB of data is read and written to and from the SAN most any day. We get sustained disk queuing on LUNS that are primarily writing. The disk queues show a queue legnth of over 100 for days in Galnce, Perfview, and SAR. My Oracle database is 120 GB.

We have a VA7400 that is connected via fibre to 2 Brocade Switches. Two
rp7400s, four rp2470's, one L2000, one Windows 2000 server, and an 4/40 LTO
tape library are connected to the two Brocade Switches.

It seems odd that the queue legnth stays consistant over a long time frame. It
also seems odd that the queue legnth does not clear.

We upgraded the VA7400 to firmware 17 and installed the latest patches in December 2002.

When are reads and writes excessive in a VA7400?
How can we determine it this is a true measure of this LUN's performance?

[Eugeny Brychkov]

Tomas,
what to check:
- everything is in fabric (looking to brocades' switchshow output you see onyl F-ports);
- you assigned VA LUNs PVs primary paths through their perfomance path. For LUNs created within RG1 perfomance path goes through VA controller C1 and for LUNs within RG2 through C2;
- you have no hardware problems with VA (armlods -e);
- you have VA in ready state (armdsp -a);
- tune queue depths for VA LUNs 'scsictl -m queue_depth=16 /dev/rdsk/c5t0d0', this value should be 3*number of disks in RG. Max 240 per array.
Attach these here zipped (supportshow from both brocades, armlog -e and armdsp -a)
Eugeny

[Thomas Maddox]

In the armlog I get several messages:
Event Type: Controller Event
Event Code: 348
Severity: 1
Event Description = FRONTEND_FC_ABTS_EVENT_EH This error code indicates that the Host sent a Fibre Channel ABTS (Abort Sequence) BLS frame to the abort an IO. The array will log this event for informational and debug purposes only. It does not necessarily indicate a problem with the array.

[Brian M Rawlings]

Thomas: the VA7400 is a fine array, who's strengths are easy administration and easy expansion, but which is not well suited to the massive I/O performance you appear to be hitting it with.

I suggest that users trying to connect are actually timing out, since this array can't "drop users" itself.

My suggestion would be that, for the kind of multi-server load and extreme I/O you are talking about, you need a much faster array, such as the EMC Clariion CX600 (current hot box this week). Not what you wanted to hear, but I can't imagine how else your array is causing users to drop or to fail to connect.

The VA7400 performs best with extra unused storage, and several drawers full of disks to spread the I/Os across. If you only have one disk tray, I'd see about getting another. When you have lots of free space, the VA7400 will try to store everything as RAID-1/0, mirrored, for fastest access. But it can only do so much. In the design, a tradeoff was made for easy self-management of data, at a sacrifice of performance (for most data patterns). It sounds like you've buried it, now all you need is a headstone.

Good Luck, I'm afraid you'll need it until you can upgrade to something that can handle this large workload.

--bmr

[Roger_22]

attach the output from the command. I look at it ....

In general, you have a lot of cpu power on a single array.

[Eugeny Brychkov]

Thomas,
in addition to what Roger asked you to attach please post driver your windows2000 server is running for FC HBA. Old Agilent driver 3.0.4107 had a timeout bug, and it's recommended to upgrade driver to the current Adaptec 2.0.25.44 downloadable from http://h20004.www2.hp.com/keeper_rnotes/bsdmatrix/matrix213991.html
Eugeny

[Thomas Maddox]

The armdiag command hangs on vgdisplay -v.
The old Agilent driver is not relevant for this server, because we do not use that card.

[Roger_22]

try this script. copy to the command view/client/bin directory. getvalogs.

[Roger_22]

if you have problems with that, i just need the output of the following commands:

armdsp -a
logprn -t all -v -s -a
armperf -c ARRAY -x COMMA -s
armperf -c OPAQUE -x COMMA -s
armperf -c LUN -x COMMA -s
armperf -c DISK -x COMMA -s


see the manpages for the format of the start time. give me 2 weeks of data. sorry for the hassle, but you'll like what i send back.

[Roger_22]

one more thing, zip/tar them up, they can be large.....

[Vincent Fleming]

Your attachment isn't coming through.

How many drives do you have in your VA7400? What size are they? (73GB? 36GB?)

Can you attach a "sar -d" output so we can see how much I/O you're doing?

Thanks,

Vince

[Thomas Maddox]

Getvalogs created a 7 MB file compressed on this system so I sent this file. I picked the time frame we experienced the worse performance.
Yesterday, we began freeing up space for RAID 0+1 space by removing LUNS we do not need. We currently have 734 GB of free space in the array. Of that free space 267 GB is for active spare space. We will gree up more space over the next 2 weekends.
By doing this we drastically reduced the number of aborts listed in the "armperf -c OPAQUE" file. We did have 11 aborts listed at 10:32 this morning.
Things are better, but not perfect.

[Eugeny Brychkov]

Can not get attachment. From my experience you can not attach files bigger than 256K, and ITRC only accepts files with extensions txt and zip (not gz, tgz, tar etc).
Attach at least 'armdsp -a' and 'armlog -e' zipped (.zip, in using winzip)
Eugeny

[Thomas Maddox]

I had to use to zip files.

[Thomas Maddox]

and again

[Vincent Fleming]

I get the .zip, but it's corrupt. How about emailing it to me at vince.fleming@hp.com

I can forward to Eugeny as well.

-Vince

[Thomas Maddox]

I have 45 73GB drives. Attached are sar logs from tha main database system.

[Vincent Fleming]

Your second try was still not good.

[Eugeny Brychkov]

Thomas,
no way. Both attachments say 'file is not a valid archive'. During download I see that only first 256K are available to download.
So please attach files/archives less than 256K. Sorry :o((
Eugeny

[Vincent Fleming]

Dude!

You're hammering the snot out of that array!

I'm taking a closer look, but at first glance, it looks like it's too small of an array for the load you're putting on it.

Vince

[Eugeny Brychkov]

Fully agree with Vince. Many LUNs, high concurrency. Recommendations:
- tune per LUN queue depth (see my first reply);
- increase LUN PV timeout value in hpux (pvchange -t) to, let's say, 180. At least hosts stop abotring requests;
- make sure you use perfomance paths for LUNs: RG1 LUNs -> VA controller C1, RG2 LUNs -> VA controller C2.
Could you please attach supportshow from both brocades zipped to your next reply (believe they will be under 256K :o) )
Eugeny

[Thomas Maddox]

I tried to state from the beginning of this that I have an over active array. How do I get better performance is my question.

[Eugeny Brychkov]

Thomas, please attach supportshows to check if they're clear.
You can uprade to VA7410, it has more powerful controllers and more cache, twice more frontend and backend ports.
Why did you create so many LUNs (I counted 53). Do you assemble these LUNs into volume groups on LVM level? If yes then I think it would be better to have one big LUN in vg than vg consisting of many little LUNs. This will eliminate high concurrency
Eugeny

[Vincent Fleming]

Thomas,

So, what you are saying is that you cannot purchase any new hardware, and have to make due with what you've got (which happens a lot)...

Don't expect it to double in performance. We may be able to get 10-20% better performance via tuning, but don't expect anything dramatic. I don't think we're going to make all your problems go away.

Now, for some recommendations:

1. Tune your OS' according to Eugeny's suggestions. Queue depth, timout values, etc.
2. If there is any way you can migrate this to RAID 0+1, do it.
Once a VA is in AutoRAID mode, you really need to backup and restore all the data to change to RAID 0+1. Moving to AutoRAID is easy, but the reverse is not true.

3. If you can add more drives to the array, it will help the performance - possibly dramatically. I would add 15 or more 36GB 15K drives.

4. If you can't add more drives, and can't afford the downtime to backup/restore to get to RAID 0+1, then clean up as best you can. The more free space on the array, the more will be RAID 0+1, and not RAID 5 (which is slower than 0+1).

To give you an idea of what I think you need... If I had to make a recommendation for a replacement subsystem for you (which is my day job), I would suggest 2 VA7410's with 45 36GB 15K drives each (minimum), and RAID 0+1. Maybe even an XP array.

Look back for some more recommendations from myself, Euegny, and others.

Vince

[Thomas Maddox]

We will do the following:
Changed queue depth threshold to 72.
Continuing take back and to delete unnecessary LUNS to increase the space for raid 0+1.
Give the array a rest day Saturday to allow it to convert space to raid 0+1.
Set prefetch to on.
Change performance from high to normal.
Continue to monitor performance.

We cannot do the following:
Buy more hardware.
Get the downtime to backup and restore all the data to all the systems on the array.
Stop beating the snot out of the array, because we have an application that is full of snot.
THANKS