topic Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device! in Operating System - OpenVMS

AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Tue, 16 Nov 2010 23:17:56 GMT

I want to find a proved solution (compatible disk and interface) using a big
1TB or 2TB disk (like new ones used with PCs) recognized by OpenVMS, to perform image/backup to it, instead of tape-drive, and then later-on from there to tape.

The main purpose is to speed up the backup process of data-bases, which is taking an average of 5.5 hrs every night.

Please, I had done all kind of things to improve the backup process with quotas, system parameters, etc, etc.

I want the hardware solution.

Does anybody been able to use something like and/or a tape drive been capable of backing-up 1TB let say in 1 to 2 hr time frame, and compatible with OpenVMS?

Thank you.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Andy Bustamante — Wed, 17 Nov 2010 08:09:03 GMT

You might also consider using host based volume shadowing.

Quiesce the database, break out a a set of database disks from the logical shadow volume, resume the database application on production disks while reading the backup disks. Minimal user impact.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Vladimir Fabecic — Wed, 17 Nov 2010 08:35:55 GMT

Hello Edmundo

Can you give us more informations:
1. What is your OS version?
2. How much data do you backup every day?
3. What type of tape device you have?
3. What is your storage subsystem? Is it external storage (EVA, MSA or some other storage or some DAS)?

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Jan van den Ende — Wed, 17 Nov 2010 10:16:34 GMT

Edmundo,

I have to agree with Andy!

It takes:
1 extra device (set) such as you are using
1 shadowing license
a little modification of bootstrap procedure (modify MOUNT command(s) to use shadow set(s) i.o. disk(s)
modification of Backup procedure (quiesce database; dismount 1 member of shadow set(s); reactivate database).
Your database interruption period will be seconds, and, depending on the particular database, can be just reduced performance of your database has online checkpointing capability.

Now you have (nearly) until your next backup schedule to copy the dissolved member(s) to external, and MOUNT the dissolved member(s) again to the set(s) - for which NO database actions are desired.

>>>
I want the hardware solution.
<<<

Ah well -- this may be stressing a little, but you can consider this to be some extra hardware, and hardware manipulation...

Success.

Proost.

Have one on me.

jpe

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Robert Gezelter — Wed, 17 Nov 2010 11:07:05 GMT

Edmundo,

In general, I would agree with Andy and Jan. The optimum way to collapse a backup window is to use host-based shadowing to expand a shadow set; momentarily quiesce the database; disconnect the extra shadow set members; and restart the database. Done within a procedure, this can be accomplished in seconds.

In terms of increasing hardware, the OP did not mention the details of the current mass storage configuration. Details always matter.

- Bob Gezelter, http://www.rlgsc.com

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Hoff — Wed, 17 Nov 2010 15:32:54 GMT

Fewer disk spindles will be slower than configurations with more disk spindles, as disks are glacial devices.

Newer storage is SAS and SATA, and those are not the easiest devices to connect onto an Alpha.

If you want faster storage, you want SSD. Again, disks are glacial-speed rotating rust devices. HP offers SSD as an option on BladeSystem, though apparently isn't making a particular push to get SSD into other configurations.

To add to the difficulty, you have slow I/O buses on this VMS box, which means you're also limited to some older storage I/O controllers. Newer stuff tends to be PCIe based, and the PCIe storage is massively faster than the parallel SCSI possible on this box. (I don't know off-hand if Alpha has any possible 8 Gb FC HBA options with the PCI-X, that's typical on PCIe on other platforms.)

Others have discussed RAID-based archival processing and the split-RAID and that's certainly functional. You must have a way to quiesce the environment.

An alternative approach is to use the features of a replication-capable database, and run multiple boxes and online backup. Various database packages offer this, and can perform continuous backups. Most places now use continuous backups; the classic "backup window" is becoming a rarity.

Various x86 boxes routinely obliterate the performance of the Alpha boxes, too. You may be in line for an upgrade, depending on your requirements. Possibly a replacement server, or potentially a dedicated database server, and connecting to that from the VMS box and the clients. (This box generation is around seven years old, and it's going to be showing its age, in terms of performance and available options and I/O buses.)

If you're waving some money around for some new storage (or a new server), then call up an HP reseller or sales rep and make them do some work.

If you're looking to get speed by connecting an off-the-shelf SAS or SATA drives by yourself, then you're looking at buying off on your own debug and integration and testing and support effort. And at just getting to a SAS or SATA bus. Or buying off on somebody that will do that work for you.

But I'd look at the distributed database replication or on-line backups first, then at quiescing and splitting volumes, then at storage or more serious upgrades.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 15:44:03 GMT

Thank you all for your interest!

Seems like I forgot one thing when I exposed
my problem, clarify the system side OS/hardware:

This is a 24*7 Operation

OpenVMS v7.3-2 with Volume shadowing
14 80GB disk/volumes (2 memberâ s shadow-sets) SAN (fiber line)

The tape drive is part of MSL5026 Library

The backup process is managed by a very sophisticated DCL script which consider database freeze, dismount/mount of disk-volumes, etc, etc, etc.

I wouldnâ t had dare to expose such a problem without having tried volume-shadowing.

Now, would you please provide a solution?

For example: Would a PCI interface will work with one of these big disk and/or something like a Ultrium tape drive in these configuration?

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Hoff — Wed, 17 Nov 2010 15:58:16 GMT

Figure out where you're spending your 5.5 hours. What's the bottleneck. Could be getting stuff out of the database. Could be the I/O bus or the controller or the SAN giblets, could be the disks, or the target archival media, or could be the processor or memory.

The following is a generic answer...

Your SAN is probably using 1 or 2 Gb HBAs, which are slow. 4 Gb is around, and 8 Gb HBAs are around, but I don't know off-hand if VMS has support for those.

Your disks and your SAN controller are probably also slow. There are differing 14-slot I/O connections around.

While tape bandwidth is faster than disk, tapes are not usually the primary archival medium chosen if you're looking for speed.

AFAIK, there is no way to upgrade the existing PCI-X slots of an AlphaServer ES47 box to a more modern PCIe bus, short of wholesale replacement of the box. PCIe (which is now widespread in the industry) only appeared in some of the more recent Integrity boxes.

Look at your online archival processing for the database and at quiescing and splitting RAID (shadowing, mirroring) volume sets.

Then start looking at whether you should move forward with an effort to upgrade your existing I/O, or start looking at a port to a faster box. If your applications are local or available on Itanium, moving to a newer box might be the most feasible approach.

If you have backup windows, then your database archives are not being run as a 24x7 operation; you're working with an older system management model and older archival design. Whether you're willing to take a hit of a day on your activities, or if there are secondary and shorter-term archives available here is an open question. I'd definitely look to see if there's a continuous-archive option in the database package(s) involved here.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 16:45:12 GMT

Sometimes is difficult for some people to get a broad expectrum if they donâ t see details!

Again â ¦ more for you, nothing tangible for me yet

The disk-volumes are mounted in a IBM8100 SAN with a farm switches connected to the Alpha ES47 BY 2 GB FCA2684 PCI HBA

This is a 24*7 operation. The only time the application environment could freeze
Databases and generate a journal is at 8:00 PM every night and last a little bit more than 6 minutes. The, only then the process dismount each one member of the shadow-sets in order to produce a image/backup save-set of each it to tape.

This application environment is running an instance of 72 databases with very complex
structure were some of them have over 1000 Globals. We need more disk-volumes in the SAN, in order to have a better distribution of the databases across more spindles and reduce the actual per/volume IO, and I know that moving into a much modern box like a
Itanium will make a difference, but I can not do that at this point.

Important: There are multiple background jobs that run after 6:00 PM every day (besides the interactive access) and they compete with the backup. Due to time scheduling and compliance there is no way to change this. Take a look at the attachement
to obtain an idea of the disk IO rate in a normal week day!

So, yea I want to improve the overall present IO and reduce the backup time.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

RBrown_1 — Wed, 17 Nov 2010 18:24:56 GMT

Your attachment is one of the funniest things I have seen in a long time. "I'm from Microsoft and I'm here to help you!"

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

RBrown_1 — Wed, 17 Nov 2010 18:27:59 GMT

Sorry about that, Microsoft is helping me even less than I thought it was helping you.

My first attempt to view your attachment showed me a picture of a paperclip!

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 18:49:09 GMT

...

0 for the comment + 1 point for the joke

...

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Robert Gezelter — Wed, 17 Nov 2010 18:58:03 GMT

Edmundo,

Thank you for the attachment. It clarifies some things and leaves a great many questions unanswered.

1TB in an hour translates to 284KB/sec (1,024MB/(60*60). Converted to bps, that translates to 2.27Mbps. Granted this is a sustained rate, which must be discounted significantly, but offhand, I am not sure I would be so fast to attribute the problem to the SAN.

I would recommend a deep examination of that backup procedure. I have seen more than my share of backup and other end-of-day procedures that were effectively fratricidal in one way or another.

If this is the case, adding a disk drive may very well make the problem worse, not better. I would recommend a careful, in-depth review of the backup/end-of-day procedures, either in-house or out-of house (Disclosure: We provide services in this area, as do others who regularly contribute to this forum).

As a side note, I would also be interested in seeing the CPU utilization data corresponding to the same time period as the IO queue length chart.

There is a good chance that working the problem without looking at the script in detail is a futile exercise.

- Bob Gezelter, http://www.rlgsc.com

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Andy Bustamante — Wed, 17 Nov 2010 19:13:06 GMT

As Bob (Robert G) points out, it's time to collect and review system metrics. Before deciding on any approach, identify what the system is really doing, what bottlenecks may exist and potential solutions.

Look at CPU load, processes in the COM state, memory used including XFC numbers.

Some potential options including adding a third set of shadow set members or clustering another server to manage the backup operation while the production server continues to service user and report loads. Before deciding an an approach, identify the current system metrics.

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 20:01:59 GMT

Trust me Robert (and all others), you donâ t want to see (less analyze) a 3rd party procedure of 3334 lines, where you can see all kind of routines dealing not only with OpenVMS DCL commands but also MUMP, SQL.

Sorry to say like this, but there is nothing to be improved here.

This is a fault proven procedure running here and others places for 7 years.

Attached you will find more graphics snap-shots of the AlphaServer ES47 for the same period.

You will notice (if you can rationale: performance vs. load) that the only things bugging a little bit (during only the high picks of day) is physical memory, which is not the problem during the backup time from 8:00 PM to 1:15 AM

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 20:04:06 GMT

CPU vs IProcesses

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 20:04:58 GMT

CPU Process Types

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 20:05:56 GMT

Page Faults Rate vs IProcesses

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Edmundo T Rodriguez — Wed, 17 Nov 2010 20:06:13 GMT

Page Faults Rate vs IProcesses

Re: AlphaServer ES47 7/1000 - Need a SUPER fast (big) io device!

Jon Pinkley — Wed, 17 Nov 2010 20:16:15 GMT

>>1TB in an hour translates to 284KB/sec (1,024MB/(60*60).

This was a mentagraphical error off by a factor of 1024.

1TB in an hour is sustained 291.3MB/sec. (1024*1024)MB/(60*60)sec ( if TB = 2^40 bytes)

So a 2MB SAN can be a limiting factor if 1TB/hour is your goal.

A single spindle mechanical PC drive won't cut it, even if you could connect it to your ES47.

Edmundo,

You said you are using shadowing, but it isn't clear that you are using it for backups.

If you are backing up 1TB in 5.5 hours, that is about 53MB/sec, which is reasonably good.

Why is 5.5 hours too slow? If you are using volume shadowing with split members for backup, that is essentially using your storage a snapshot. If you are going to use shadowning, make sure you have the HBVS patch that supports mini-copy on 7.3-2, and make sure you ensure that minicopy gets used when you reintroduce your shadowset members.

Jon