topic Re: VA7400 Performance in Disk Enclosures

VA7400 Performance

Thomas Maddox — Mon, 03 Feb 2003 20:18:34 GMT

I have a possible performance on a VA7400. When disk queues are high, users connecting to our Oracle 8i database through a web based Java Application get disconnected. Also, jobs running on other systems are slow.

I have 2.1TB usable on my VA7400 and over 3TB of data is read and written to and from the SAN most any day. We get sustained disk queuing on LUNS that are primarily writing. The disk queues show a queue legnth of over 100 for days in Galnce, Perfview, and SAR. My Oracle database is 120 GB.

We have a VA7400 that is connected via fibre to 2 Brocade Switches. Two
rp7400s, four rp2470's, one L2000, one Windows 2000 server, and an 4/40 LTO
tape library are connected to the two Brocade Switches.

It seems odd that the queue legnth stays consistant over a long time frame. It
also seems odd that the queue legnth does not clear.

We upgraded the VA7400 to firmware 17 and installed the latest patches in December 2002.

When are reads and writes excessive in a VA7400?
How can we determine it this is a true measure of this LUN's performance?

Re: VA7400 Performance

Eugeny Brychkov — Mon, 03 Feb 2003 20:34:07 GMT

Tomas,
what to check:
- everything is in fabric (looking to brocades' switchshow output you see onyl F-ports);
- you assigned VA LUNs PVs primary paths through their perfomance path. For LUNs created within RG1 perfomance path goes through VA controller C1 and for LUNs within RG2 through C2;
- you have no hardware problems with VA (armlods -e);
- you have VA in ready state (armdsp -a);
- tune queue depths for VA LUNs 'scsictl -m queue_depth=16 /dev/rdsk/c5t0d0', this value should be 3*number of disks in RG. Max 240 per array.
Attach these here zipped (supportshow from both brocades, armlog -e and armdsp -a)
Eugeny

Re: VA7400 Performance

Thomas Maddox — Mon, 03 Feb 2003 22:57:43 GMT

In the armlog I get several messages:
Event Type: Controller Event
Event Code: 348
Severity: 1
Event Description = FRONTEND_FC_ABTS_EVENT_EH This error code indicates that the Host sent a Fibre Channel ABTS (Abort Sequence) BLS frame to the abort an IO. The array will log this event for informational and debug purposes only. It does not necessarily indicate a problem with the array.

Re: VA7400 Performance

Brian M Rawlings — Tue, 04 Feb 2003 02:46:16 GMT

Thomas: the VA7400 is a fine array, who's strengths are easy administration and easy expansion, but which is not well suited to the massive I/O performance you appear to be hitting it with.

I suggest that users trying to connect are actually timing out, since this array can't "drop users" itself.

My suggestion would be that, for the kind of multi-server load and extreme I/O you are talking about, you need a much faster array, such as the EMC Clariion CX600 (current hot box this week). Not what you wanted to hear, but I can't imagine how else your array is causing users to drop or to fail to connect.

The VA7400 performs best with extra unused storage, and several drawers full of disks to spread the I/Os across. If you only have one disk tray, I'd see about getting another. When you have lots of free space, the VA7400 will try to store everything as RAID-1/0, mirrored, for fastest access. But it can only do so much. In the design, a tradeoff was made for easy self-management of data, at a sacrifice of performance (for most data patterns). It sounds like you've buried it, now all you need is a headstone.

Good Luck, I'm afraid you'll need it until you can upgrade to something that can handle this large workload.

--bmr

Re: VA7400 Performance

Roger_22 — Tue, 04 Feb 2003 02:59:07 GMT

attach the output from the command. I look at it ....

In general, you have a lot of cpu power on a single array.

Re: VA7400 Performance

Eugeny Brychkov — Tue, 04 Feb 2003 08:15:42 GMT

Thomas,
in addition to what Roger asked you to attach please post driver your windows2000 server is running for FC HBA. Old Agilent driver 3.0.4107 had a timeout bug, and it's recommended to upgrade driver to the current Adaptec 2.0.25.44 downloadable from http://h20004.www2.hp.com/keeper_rnotes/bsdmatrix/matrix213991.html
Eugeny

Re: VA7400 Performance

Thomas Maddox — Tue, 04 Feb 2003 22:51:14 GMT

The armdiag command hangs on vgdisplay -v.
The old Agilent driver is not relevant for this server, because we do not use that card.

Re: VA7400 Performance

Roger_22 — Wed, 05 Feb 2003 04:34:28 GMT

try this script. copy to the command view/client/bin directory. getvalogs.

Re: VA7400 Performance

Roger_22 — Wed, 05 Feb 2003 04:43:54 GMT

if you have problems with that, i just need the output of the following commands:

armdsp -a
logprn -t all -v -s -a
armperf -c ARRAY -x COMMA -s
armperf -c OPAQUE -x COMMA -s
armperf -c LUN -x COMMA -s
armperf -c DISK -x COMMA -s

see the manpages for the format of the start time. give me 2 weeks of data. sorry for the hassle, but you'll like what i send back.

Re: VA7400 Performance

Roger_22 — Wed, 05 Feb 2003 04:46:07 GMT

one more thing, zip/tar them up, they can be large.....

Re: VA7400 Performance

Vincent Fleming — Wed, 05 Feb 2003 18:04:15 GMT

Your attachment isn't coming through.

How many drives do you have in your VA7400? What size are they? (73GB? 36GB?)

Can you attach a "sar -d" output so we can see how much I/O you're doing?

Thanks,

Vince

Re: VA7400 Performance

Thomas Maddox — Wed, 05 Feb 2003 18:06:48 GMT

Getvalogs created a 7 MB file compressed on this system so I sent this file. I picked the time frame we experienced the worse performance.
Yesterday, we began freeing up space for RAID 0+1 space by removing LUNS we do not need. We currently have 734 GB of free space in the array. Of that free space 267 GB is for active spare space. We will gree up more space over the next 2 weekends.
By doing this we drastically reduced the number of aborts listed in the "armperf -c OPAQUE" file. We did have 11 aborts listed at 10:32 this morning.
Things are better, but not perfect.

Re: VA7400 Performance

Eugeny Brychkov — Wed, 05 Feb 2003 18:11:56 GMT

Can not get attachment. From my experience you can not attach files bigger than 256K, and ITRC only accepts files with extensions txt and zip (not gz, tgz, tar etc).
Attach at least 'armdsp -a' and 'armlog -e' zipped (.zip, in using winzip)
Eugeny

Re: VA7400 Performance

Thomas Maddox — Wed, 05 Feb 2003 18:15:18 GMT

I had to use to zip files.

Re: VA7400 Performance

Thomas Maddox — Wed, 05 Feb 2003 18:15:53 GMT

and again

Re: VA7400 Performance

Vincent Fleming — Wed, 05 Feb 2003 18:20:31 GMT

I get the .zip, but it's corrupt. How about emailing it to me at vince.fleming@hp.com

I can forward to Eugeny as well.

-Vince

Re: VA7400 Performance

Thomas Maddox — Wed, 05 Feb 2003 18:22:05 GMT

I have 45 73GB drives. Attached are sar logs from tha main database system.

Re: VA7400 Performance

Vincent Fleming — Wed, 05 Feb 2003 18:31:39 GMT

Your second try was still not good.

Re: VA7400 Performance

Eugeny Brychkov — Wed, 05 Feb 2003 18:33:42 GMT

Thomas,
no way. Both attachments say 'file is not a valid archive'. During download I see that only first 256K are available to download.
So please attach files/archives less than 256K. Sorry :o((
Eugeny

Re: VA7400 Performance

Vincent Fleming — Wed, 05 Feb 2003 20:47:08 GMT

Dude!

You're hammering the snot out of that array!

I'm taking a closer look, but at first glance, it looks like it's too small of an array for the load you're putting on it.

Vince