topic Re: D2600 with 36TB 10k rpm performance in Operating System - HP-UX

D2600 with 36TB 10k rpm performance

chindi — Thu, 02 Apr 2015 12:56:05 GMT

Hi ,

We are facing performance issues when we are trying to take tar or export backup on rx2800i4 11.31 with 256gb RAM, 16Cpu .

Its connected to D2600 with 36TB 10k rpm directly.

What option are we left now to increase throughput .

Can we look at SSD option ??

If yes where do we look at Host end or Storage end ?

Re: D2600 with 36TB 10k rpm performance

Bill Hassell — Fri, 03 Apr 2015 00:15:44 GMT

Before you look at the disk performance, we'll need a bit more information.

tar simply copies data to some destination, so the first question is are you copying disk-to-disk?

What do you mean by "export backup".

Or perhaps ou are sending the data over a network? Your disks will be much faster than any network connection.

This is especially true for NFS where there is a *lot* of overhead.

As far as the disk array, what RAID level are you using? RAID 1, 5, 6, ??

How do you know that the backup is too slow? 36 TB is a lot of data to backup.

Re: D2600 with 36TB 10k rpm performance

chindi — Mon, 06 Apr 2015 06:49:36 GMT

Hi Bill,

My fault , question is not very clear .

Re-framing it ;

We have a rx2800i4 host with 16c / 256 GB RAM , 11iv3 directly connected to DS2600 36TB , 12 disks each of 3 TB each .

We have Cobol 5.1 & Oracle 11g environment in our setup .

Raid 5 configured using Smart Array controller .

When we are trying to import Database impdp of 600GB on this host ; its taking almost 3 days :(

We do understand its JBOD and not meant for performance , but now what are the options that we can look at if we need to increase overall Disk / I/O performance ?

Internal SSD one option , only 6 can be added with 400GB each .

No SSD can be added in D2600 right ?

Our total DB size is approx 10TB .

Any other option left ?

Re: D2600 with 36TB 10k rpm performance

Bill Hassell — Wed, 08 Apr 2015 01:31:34 GMT

>> We have a rx2800i4 host with 16c / 256 GB RAM , 11iv3 directly connected to DS2600 36TB , 12 disks each of 3 TB each .

Very nice system and very fast.

>> Raid 5 configured using Smart Array controller .

Oops, not the best choice for performance. RAID 5 is great for read-mostly access, lousy for writing (as in 'importing').

>> We do understand its JBOD and not meant for performance...

Actually, that's not true at all. The Smart Array controller can setup several different RAID levels and with striping it can run at several hundred MB/sec.

>> When we are trying to import Database impdp of 600GB on this host ; its taking almost 3 days :(

This is not a disk issue at all.

What you left out of the description is how you are importing.

This system can copy between internal disks at more than 100 MB/sec (which is 6 GB/min or 360 GB/hour)

I have attached a script called diskperf.

This will report the serial reading rate of specified disks or mount points.

Usage: diskperf 1000 /var /tmp

where: 1000 is 1000 MB and /var and /tmp are mountpoints. Use diskperf -? for details.

You'll probably see well over 100 MB/sec read rate.

So I'll take a wild guess: you are using a network to import your data.

So the math for networks is somewhat easy to overlook. A Gigabit link has a wire speed of about 100 MB/sec, but even with a pure data stream (*NO* database overhead) and using jumbo frames from end-to-end, 70-80 MB/sec is about all you can expect. But a pure data stream is much, much faster than an NFS link -- figure about 20 MB/sec if the other end isn't slowing things down. And if your network interface has dropped to half-duplex, figure less than 1 MB/sec. Use lanadmin -g 0 to check the lan0 interface.

You should have Glance on this system. Use it to look at the LAN and disk performance.

But even with a very slow network connection, 3 days indicates a much more complex issue.

So exactly how is the data being 'imported'?

Re: D2600 with 36TB 10k rpm performance

chindi — Wed, 22 Apr 2015 10:51:13 GMT

Hi Bill ,

We have database size of 500Gb , Oracle 11gR2 .

We are using below import command ;

impdp directory=dump
dumpfile=gb_POST_eod_backup_compressed_25032015_033145.dmp
logfile=AL_10042015.log REMAP_SCHEMA=olc:jhc TABLE_EXISTS_ACTION=REPLACE include=table,index

root #/tmp >lanadmin -g 0

LAN INTERFACE STATUS DISPLAY
Wed, Apr 22,2015 15:53:42

PPA Number = 0
Description = lan0 HP PCIe 1000Base-T Release B.11.31.1307
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 1000000000
Station Address = 0xd89d67f554f8
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 813500551
Inbound Octets = 1031542579
Inbound Unicast Packets = 1243750298
Inbound Non-Unicast Packets = 34044390
Inbound Discards = 0
Inbound Errors = 2
Inbound Unknown Protocols = 701272
Outbound Octets = 31483550
Outbound Unicast Packets = 1015008193
Outbound Non-Unicast Packets = 197558
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367

Ethernet-like Statistics Group

Index = 1
Alignment Errors = 0
FCS Errors = 1
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 1

root #/tmp >netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lo0 32808 127.0.0.0 127.0.0.1 18299529 0 18309435 0 0
lan0 1500 10.1.21.0 10.1.21.15 21218059813 2 20970940747 0 0

===========================================================

root #/tmp >sar -d 5 5

HP-UX dev B.11.31 U ia64 04/22/15

16:02:35 device %busy avque r+w/s blks/s avwait avserv
16:02:40 disk2 0.20 0.50 8 133 0.00 0.09
disk6 99.88 0.51 185 7821 0.05 16.50
disk7 22.77 2.66 33 2644 21.02 27.71
16:02:45 disk2 0.20 0.50 9 141 0.00 0.46
disk6 99.79 0.67 177 10225 1.10 31.22
disk7 66.60 3.12 114 19828 25.46 20.27
16:02:50 disk2 0.40 0.50 10 144 0.00 0.49
disk6 100.00 0.76 202 9213 1.67 24.67
disk7 31.40 3.66 33 818 26.65 42.54
16:02:55 disk2 0.60 0.50 10 233 0.00 0.89
disk6 99.79 0.50 185 8963 0.12 21.03
disk7 22.40 1.73 32 952 11.84 25.05
16:03:00 disk2 0.40 0.50 35 224 0.00 0.10
disk6 99.99 0.50 206 8729 0.05 16.80
disk7 46.20 4.39 76 19920 27.24 18.83

Average disk2 0.36 0.50 15 175 0.00 0.31
Average disk6 99.93 0.59 191 8990 0.60 21.89
Average disk7 37.87 3.31 58 8831 24.04 23.84

Re: D2600 with 36TB 10k rpm performance

Bill Hassell — Wed, 22 Apr 2015 11:39:33 GMT

>> Oracle ... import command

So this isn't basic HP-UX, this is using Oracle to retrieve the data and then to write it into the database.This means that Oracle overhead at the source and at the destination will be a limiting factor.

As mentioned before, RAID 5 is the slowest layout for heavy writes to disk.

Your network connection is OK (no errors) but it will limit the maximum throughput.

The real issue is seen in the sar listing. disk6 and disk7 have terrible service time averages, with disk7 having a large backlog (queue) of waiting requests and a long average wait time to complete requests.

So a solid state disk will improve performance but will be quite expensive. Your least expensive option is to start over by removing the RAID 5 layout and rebuilding the Oracle data disks as RAID 1 (simple mirroring). The Smart Array will load balance between disks for reading, and writing will run at full disk speeds. You can verify the serial read throughput with the diskperf script.

Re: D2600 with 36TB 10k rpm performance

chindi — Wed, 22 Apr 2015 12:40:05 GMT

Hi Bill,

You mean i will have to rebuild my Raid from Raid 5 to Raid 1 .

We have 12 disks each of 3TB , so we can use at the max 18TB right ? if we configure Raid 1 .

With Raid 1 how much write speed we can achieve ??

The other challenge would be restoring data 20TB .

As of now we have assigned two vg's from this DAS .

vg01 & vg02 .

Each vg contains approx 10TB of data in it .

How can i migrate this data if we dont have Tape in our environment ??

Re: D2600 with 36TB 10k rpm performance

Torsten. — Wed, 22 Apr 2015 12:59:10 GMT

Let's have a look.

Please do

# ioscan -fnCciss

to list the smart array controllers. You will find a P410i for internal disks and probably a P411 for external disks.

Please do

# saconfig /dev/ciss_

with the device file of the P411.

Re: D2600 with 36TB 10k rpm performance

chindi — Wed, 22 Apr 2015 13:26:12 GMT

Hi Torsten,

root #/ >ioscan -fnCciss
root #/ >ioscan |grep -i arra
0/0/0/1/0/0/0 ext_bus PCIe SAS SmartArray P410i RAID Controller
0/0/0/9/0/0/0 ext_bus PCIe SAS SmartArray P411 RAID Controller
root #/ >saconfig /dev/ciss1

******************** SmartArray RAID Controller /dev/ciss1 ********************

Auto-Fail Missing Disks at Boot = disabled
Cache Configuration Status = cache enabled
Cache Ratio = 100% Read / 0% Write

---------- PHYSICAL DRIVES ----------

Location Ct Enc Bay WWID Size Status

External 1E 1 1 0x5000c50057cb0d59 3000.6 GB OK
External 1E 1 2 0x5000c50057cb57d5 3000.6 GB OK
External 1E 1 3 0x5000c50057cb2279 3000.6 GB OK
External 1E 1 4 0x5000c50057ca7d11 3000.6 GB OK
External 1E 1 5 0x5000c50057cc73d5 3000.6 GB OK
External 1E 1 6 0x5000c50057cc7001 3000.6 GB OK
External 1E 1 7 0x5000c50057ca6321 3000.6 GB OK
External 1E 1 8 0x5000c50057cc75d5 3000.6 GB OK
External 1E 1 9 0x5000c50057cae411 3000.6 GB OK
External 1E 1 10 0x5000c50057cc6451 3000.6 GB OK
External 1E 1 11 0x5000c50057cb5a45 3000.6 GB OK
External 1E 1 12 0x5000c50057cb5eed 3000.6 GB OK

---------- LOGICAL DRIVE 0 ----------

Device File = /dev/dsk/c4t0d0
RAID Level = 5
Size = 14307780 MB
Stripe Size = 16 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1E 1 1 0x5000c50057cb0d59
1E 1 2 0x5000c50057cb57d5
1E 1 3 0x5000c50057cb2279
1E 1 4 0x5000c50057ca7d11
1E 1 5 0x5000c50057cc73d5
1E 1 6 0x5000c50057cc7001

Participating Spare Drive(s):

None

---------- LOGICAL DRIVE 1 ----------

Device File = /dev/dsk/c4t0d1
RAID Level = 5
Size = 14307780 MB
Stripe Size = 16 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1E 1 7 0x5000c50057ca6321
1E 1 8 0x5000c50057cc75d5
1E 1 9 0x5000c50057cae411
1E 1 10 0x5000c50057cc6451
1E 1 11 0x5000c50057cb5a45
1E 1 12 0x5000c50057cb5eed

Participating Spare Drive(s):

None

Re: D2600 with 36TB 10k rpm performance

Bill Hassell — Thu, 23 Apr 2015 12:47:51 GMT

You mean i will have to rebuild my Raid from Raid 5 to Raid 1 .

That's correct.

We have 12 disks each of 3TB , so we can use at the max 18TB right ? if we configure Raid 1 .

correct: 12*3 = 36TB, or 18TB mirrored.

With Raid 1 how much write speed we can achieve ??

You'll get the full speed of the disk with no RAID5 parity computation and distribution overhead.

The other challenge would be restoring data 20TB .

You'll have to look at adding more disks from your storage, perhaps another tray or two.

What external storage are you using currently?

How can i migrate this data if we dont have Tape in our environment ??

Not sure what tape has to do with this task. If you need 20TB of data, you'll need more disks.

Re: D2600 with 36TB 10k rpm performance

Torsten. — Thu, 23 Apr 2015 15:29:19 GMT

Maybe the poor little P411 is not powerful enough to serve a heavy load. A "real" disk array would do a better job.

However, if you need more disk space with the current setup, consider to connect another D2600 to a new P411 at least.

Re: D2600 with 36TB 10k rpm performance

chindi — Tue, 28 Apr 2015 09:59:46 GMT

We are going for MSA 2040 , since MSA 1040 doesnt support HPUX .

Re: D2600 with 36TB 10k rpm performance

Torsten. — Tue, 28 Apr 2015 12:08:55 GMT

IMHO a good choice. I would recommend FC SFPs. Make sure it is running firmware GL200 for much more features.