best option between hardware and software mirroring

[Markus Waldorf_1]

Hello,

I'm trying to figure out the best option to configure a combination of hard and software mirroring. The hardware is an AS4100 5/533 connected via DSSI to a HSD50 controller with 3 BA356 storage shelves providing 16 x 4 GB scsi-2 harddrives. The controller has easy to configure options to setup mirroring, stripping and partitioning.

From what I know, volume shadowing makes it relatively easy to do image backups by simply breaking a shadow set and mounting a device separately in read-only mode, without having to shut the system down. Although the HSD50 controller is pretty nippy, its data throughput is not state of the art anymore. I would assume that hardware mirroring is faster and has less i/o impact on the system compared to software volume shadowing.

Since I plan to partition the drives by the HSD50 controller I cannot use its disk cloning feature, and beside, I would not have enough spare disks available anyway. So I was wondering about using Volume Shadowing, which would allow me to break a shadow set and do an image backup of a volume to tape. I think that an image backup for anything else than the system disk is not necessary, and probably nonsense for a data disk, unless the applications using the files are properly shutdown, e.g. Oracle.

So far I have come to the following conclusion:

Configure 2 stripsets with 3 partitions for best performance, to be used for the system, swap and pagefiles and queue manager, and use one stripset for volume shadowing to mirror the partitions on separate disks. For the rest I will use the HSD50 to setup mirrors among different storage shelves, and use stripping and partitioning to have maximum performance.

I'm happy for any comments.

Best regards!

[Hein van den Heuvel · Accepted solution]

What version of VMS are your running (better than 7.2? Fibre storage ready?)

What is the main reason you are using (KFPSA)-DSSI-HSD storage?
- because it was there?
- cluster requirements?

As you indicate, while the controller is decent still, the DSSI bus performance is really poor by todays standards. From the spec: "Highest performance DSSI controller with 1700 QIOs/sec and 3.4 MB/sec bandwidth."
Recent fibre controllers run at 100 times the MB/sec and 30 times the IO/sec. Disks all run at well over 10MB/sec.

Given that MB/sec speed, I think you are right in minimizing host based shadowing (requiring to transfer the data twice) and to focus on controller based shadowing. Plus, witha single controller, you are not increasing availability with software shadowing.

Any chance to go towards a Mylex PCI raid controller? Fibre + MSA1000?
Are those 4GB drives the 7200 rpm RZ29s?
Any chance to go to a 18GB or 36GB 10Krpm drives?

How about that partitioning? Are you sure you want to do that? Why not keep it as once 'large' striped disk? I can see management advantages for partitioning, but I can not see performance andvantage, only performance risks (maximizing seek distances for no good reasons).

As you remark, you would have to shutdown a database like Oracle anyway before making a useful/useable shadow clone split. Do you have the time window for backups? Do you have access to online DB backup tools (RMU) to get the bulk of the work done online?

In my mind, a system disk rarely needs backups. Just a few selected files that do not need to live on the system disk (sysuaf, rightslist,...).... and which need special treatment as they are always open (I like convert/share). Page files need no backup (but can use hardware shadowing to protect against drive failure).

Oops, bed time. Gotta go.
Sorry for all the questions, hopefully they get you thinking in the the right directions.

Hope this helps some,
Hein.

[Michelle Popejoy]

Markus,

Since you're only concerned with image backup for the system disk and have only one system and one controller, I don't see any reason to use Volume Shadowing.

I find it useful to create a controller-based Mirrorset for the system disk. I then simply remove one member of the mirrorset prior to an upgrade and replace it with a spare as my "image backup".

I too wonder if partitioning is the way you want to go.

It's been awhile since I was on an HSD but isn't RAID available? If improving fault tolerance is one of your goals you might consider placing your data on controller-based RAID sets and having one or more disks in your spare set that the RAID sets and the system Mirrorset can use. You'll have to enable autosparing on the *sets. Having the RAIDsets gives you the combination of performance and some fault tolerance, even if you don't use a spareset. And, even with the spare disks you're likely to consume fewer physical disks overall than shadows of stripsets.

If you do decide to use volume shadowing for some reason, then I would recommend using VMS 7.3-2 with the HBMM kit if at all possible for the enhanced features including priorities.

[Markus Waldorf_1]

Hello Hein,

many thanks for taking the time to answer my question! The reason for the stripeset idea is to increase performance by having 3 drives participate in one i/o request. The idea to partition the space is only to move queue files or other things that may fill up the disk to another partition to keep the main system available. I would move applications to other disks to avoid disk i/o bottlenecks.

After all, I guess nothing will bypass the DSSI performance issue, and it probably doesn't even matter much from a disk layout point of view. Moving to more modern storage hardware would be nice, but involves my money, and I doubt that the purpose of a mailserver justifies it. I would consider that reliability is more important.

You mentioned that I will not increase availability by volume shadowing, simply because I have only one storage controller. That's certainly true! For that reason I would use volume shadowing simply for the purpose of doing on-line image backups of the system disk. The question is now how much it is going to slow down disk performance.

For the main part I would agree that I could rebuild a system disk easily from CD in conjunction with normal and special backups. And I'm better off with this strategy unless I can rely on regular and up-to-date image backups. On the other side, if I have an image backup and I loose the system disk, or if I move to new hardware, it will give me the safest and shortest restore time.

Well, I should not go too fancy with configuring the hardware, nor software, or backup - keeping things simple. I think I will take advantage of shadowing the system disk only and benefit from easier management and restore at the cost of some performance.

Again many thanks for your reply! It sometimes boils down to what real requirements and priorities are. I feel more comfortable with my plan now.

Best regards!

[Markus Waldorf_1]

After going to implement the plan I ran into more concerns:

shelf 1 [disk130] [disk120] [disk110] [disk100]
shelf 2 [disk230] [disk220] [disk210] [disk200]
shelf 3 [disk330] [disk320] [disk310] [disk300]

Since I have only 3 storage shelves I cannot create a mirrored stripe (raid 1 + 0) of 3 disks without loosing the complete stripe if one storage shelf fails. I can however create mirrorsets of 2 disks, each on different ports, that will continue to work if only one disk or storage shelf fails.
But I cannot stripe 2 or more mirrors without loosing performance and availability as one of the members will always reside on the same port. e.g.

Creating mirrors like:

disk100 + disk210
disk200 + disk310
disk300 + disk110
disk120 + disk230
disk220 + disk330
disk320 + disk130

will give me protection if either one disk of a mirrorset or if 1 port fails, since no pair resides on one and the same port.

If I was going to stripe 2 or more mirrors than a port failure will jeopardize the complete stripe.

If I need larger drives, than the only option I can see is to create stripes of 2 disks and mirror them using software volume shadowing. e.g.

Creating strips like:

disk100 + disk200
disk300 + disk110

and use volume shadowing to mirror the 2 stripes. The problem can be that volume shadowing will decrease performance as it need to write to two drives on the same port, but at least I have a larger single drive and protection from failure.

Can anyone see how I could do this with a stripe of 3 drives without having a 4th shelf?

[Markus Waldorf_1]

Another idea. Maybe this will do the trick, but I noticed something. What I don't understand here is why the 2nd block
of commands does not produce the warning, although to my understanding should be the same thing. What is the difference?

HSD50> add mirror mirror1 disk100 disk210
HSD50> add mirror mirror2 disk200 disk310
HSD50> add mirror mirror3 disk300 disk110
HSD50> add stripe stripe1 mirror1 mirror2 mirror3
Warning 3000: This storageset is configured with more than one disk per port.
This causes a degradation in performance


But I don't get the performance warning when using:

HSD50> add stripe stripe1 disk100 disk200 disk300
HSD50> init stripe1
HSD50> add unit d1 stripe1

HSD50> mirror disk100 mirror1
HSD50> set mirror1 membership=2
HSD50> set mirror1 replace disk210

HSD50> mirror disk200 mirror2
HSD50> set mirror2 membership=2
HSD50> set mirror2 replace=disk310

HSD50> mirror disk300 mirror2
HSD50> set mirror2 membership=2
HSD50> set mirror2 replace=disk110

[Michelle Popejoy]

Marcus said:
shelf 1 [disk130] [disk120] [disk110] [disk100]
shelf 2 [disk230] [disk220] [disk210] [disk200]
shelf 3 [disk330] [disk320] [disk310] [disk300]
:
:
Can anyone see how I could do this with a stripe of 3 drives without having a 4th shelf?
-----------------

How about using RAID3/5 instead of striping (see my previous post)?
RAIDSET3 = [disk130] [disk230] [disk330]
RAIDSET2 = [disk120] [disk220] [disk320]
Mirrored if you like?
MIRROR1 = RAIDSET2 RAIDSET3

[Michelle Popejoy]

Oops! Sorry, Markus, I didn't mean to misspell your name on that last post.

[Markus Waldorf_1]

Hi Michelle,

I wonder what would happen with the Raid 3/5 solution if one of the shelf dies. I think I need to make sure that no drive on the same port gets mirrored.

I think I have figured it out, hopefully. I will continue to do some tests by removing multiple drives from a shelf to see if it still works. If it breaks up, I' will give an update.

Below an extract of what I have done with the the HSD50 controller so far.
I use it for my own reference, but maybe someone else may find it usefull too.


HSD> set this_controller initial_configuration
HSD> set this scs_nodename="HSD50"
HSD50 >set this prompt="HSD50> "
HSD> restart this
HSD> set this path
HSD>run config
HSD>run HSUTIL
-> format all drives
HSD>run CFMENU
-> add the drives

(front view)
shelf 1 [disk130] [disk120] [disk110] [disk100]
shelf 2 [disk230] [disk220] [disk210] [disk200]
shelf 3 [disk330] [disk320] [disk310] [disk300]


HSD50> add stripe stripe1 disk100 disk200
HSD50> add stripe stripe2 disk300 disk110
HSD50> init stripe1 save_configuration
HSD50> init stripe2 save_configuration
HSD50> create_partition stripe1 size=50
HSD50> create_partition stripe1 size=50
HSD50> create_partition stripe2 size=50
HSD50> create_partition stripe2 size=50

HSD50> add unit D1 stripe1 partition=1
HSD50> add unit D2 stripe1 partition=2
HSD50> add unit D3 stripe2 partition=1
HSD50> add unit D4 stripe2 partition=2
HSD50> set unit D1 writeback_cache
HSD50> set unit D2 writeback_cache
HSD50> set unit D3 writeback_cache
HSD50> set unit D4 writeback_cache

OVMS Volume Shadowing for unit D1 and D3 - 4 GB.
-> disk$alphasys, static system files

OVMS Volume Shadowing for unit D2 and D4 - 4 GB.
-> disk$scratch, queue files, virtual memory files.

The stripsets should help to overcome the performance issues with
volume shadowing.

HSD50> add stripe stripe3 disk120 disk220 disk320
HSD50> init stripe3
HSD50> add unit D5 stripe3
HSD50> set unit D4 writeback_cache
HSD50> mirror disk120 mirror1
HSD50> mirror disk220 mirror2
HSD50> mirror disk320 mirror3
HSD50> set mirror1 membership=2
HSD50> set mirror2 membership=2
HSD50> set mirror3 membership=2
HSD50> set mirror1 replace=disk230
HSD50> set mirror2 replace=disk330
HSD50> set mirror3 replace=disk130

-> D5 for disk$data - 12 GB

HSD50> add mirror mirror4 disk210 disk310
HSD50> init mirror4
HSD50> add unit D6 mirror4
HSD50> set unit d6 writeback_cache

-> D6 for disk$install - 4 GB

HSD50> add tape tz887 4 5 0
HSD50> add unit T887 tz887
HSD50> add loader tz887L 4 5 1
HSD50> add passthrough pass451 tz887L
HSD50> add unit D887 pass451

-> TZ887 tape-library support and loader for MRU



HSD50> sho tape full
Name Type Port Targ Lun Used by
------------------------------------------------------------------------------

TZ887 tape 4 5 0 T887
DEC TZ887 (C) DEC D473
Switches:
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
HSD50> sho loader full
Name Type Port Targ Lun Used by
------------------------------------------------------------------------------

TZ887L loader 4 5 1 PASS451
DEC TZ Media Changer D473
HSD50> sho pass full
Name Storageset Uses Used by
------------------------------------------------------------------------------

PASS451 passthrough TZ887L D887
HSD50>


HSD50> sho mirror full
Name Storageset Uses Used by
------------------------------------------------------------------------------

MIRROR1 mirrorset DISK120 STRIPE3
DISK230
Switches:
NOPOLICY (for replacement)
COPY (priority) = NORMAL
READ_SOURCE = LEAST_BUSY
MEMBERSHIP = 2, 2 members present
State:
NORMAL
DISK120 (member 0) is NORMAL
DISK230 (member 1) is NORMAL
Size: 8378028 blocks

MIRROR2 mirrorset DISK220 STRIPE3
DISK330
Switches:
NOPOLICY (for replacement)
COPY (priority) = NORMAL
READ_SOURCE = LEAST_BUSY
MEMBERSHIP = 2, 2 members present
State:
NORMAL
DISK220 (member 0) is NORMAL
DISK330 (member 1) is NORMAL
Size: 8378028 blocks

MIRROR3 mirrorset DISK130 STRIPE3
DISK320
Switches:
NOPOLICY (for replacement)
COPY (priority) = NORMAL
READ_SOURCE = LEAST_BUSY
MEMBERSHIP = 2, 2 members present
State:
NORMAL
DISK320 (member 0) is NORMAL
DISK130 (member 1) is NORMAL
Size: 8378028 blocks

MIRROR4 mirrorset DISK210 D6
DISK310
Switches:
POLICY (for replacement) = BEST_PERFORMANCE
COPY (priority) = NORMAL
READ_SOURCE = LEAST_BUSY
MEMBERSHIP = 2, 2 members present
State:
RECONSTRUCTING
DISK210 (member 0) is NORMAL
DISK310 (member 1) is NORMALIZING 50% complete
Size: 8378028 blocks


HSD50> sho disks full
Name Type Port Targ Lun Used by
------------------------------------------------------------------------------

DISK100 disk 1 0 0 STRIPE1
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8377528 blocks
Configuration being backed up on this container
DISK110 disk 1 1 0 STRIPE2
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8377528 blocks
Configuration being backed up on this container
DISK120 disk 1 2 0 MIRROR1
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK130 disk 1 3 0 MIRROR3
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK200 disk 2 0 0 STRIPE1
DEC RZ1CB-CS (C) DEC 0656
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8377528 blocks
Configuration being backed up on this container
DISK210 disk 2 1 0 MIRROR4
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK220 disk 2 2 0 MIRROR2
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK230 disk 2 3 0 MIRROR1
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK300 disk 3 0 0 STRIPE2
DEC RZ1CB-CS (C) DEC 0844
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8377528 blocks
Configuration being backed up on this container
DISK310 disk 3 1 0 MIRROR4
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK320 disk 3 2 0 MIRROR3
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
DISK330 disk 3 3 0 MIRROR2
DEC RZ1CB-CA (C) DEC LYJ0
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 10MHZ (synchronous 10 MHZ negotiated)
Size: 8378028 blocks
HSD50>

HSD50> sho this full
Controller:
HSD50-AX ZG65201006 Firmware V57D-2, Hardware A01
Not configured for dual-redundancy
SCSI address 7
Time: 07-MAR-2005 23:39:57
Host port:
Node name: HSD50, valid DSSI node 1
Host path is ON
MSCP allocation class 0
TMSCP allocation class 0
MAXIMUM_HOSTS = 4
NOREMOTE_DISK_COPY_DATA
Cache:
32 megabyte write cache, version 3
Cache is GOOD
Battery is GOOD
No unflushed data in cache
CACHE_FLUSH_TIMER = DEFAULT (10 seconds)
CACHE_POLICY = A
NOCACHE_UPS
Licensing information:
RAID (RAID Option) is ENABLED, license key is VALID
WBCA (Writeback Cache Option) is ENABLED, license key is VALID
MIRR (Disk Mirroring Option) is ENABLED, license key is VALID
Extended information:
Terminal speed 9600 baud, eight bit, no parity, 1 stop bit
Operation control: 00000000 Security state code: 26713
Configuration backup enabled on 4 devices
HSD50>

Power On Time: 5. Years, 227. Days, 23. Hours, 55. Minutes, 38. Seconds
Controller Model: HSD50-AX