Get Expert Advice: How to Accelerate Oracle Backups Using HPE StoreOnce

[StorageExperts]

Use these 10 silver bullets to tune an entire solution stack – Oracle, Oracle Recovery Manager (RMAN) and StoreOnce Catalyst – to meet the RPO/RTO SLA for backups and restores, plus a brief discussion on the newly released Recovery Manager Central for Oracle (RMC-O).

As a part of my job, I interact with Oracle DBAs, backup admins and IT management teams from a wide variety of customers to discuss data protection needs for their enterprise Oracle environments. We might give demonstrations of HPE solutions (OR) run proof-of-concepts in their environment. The common question that I face from almost every other customer is “How do I reliably complete my backups fast as well as be 100% sure about seamless recovery capability later on when needed?” Or in some instances simply “I have a 20TB+ database and a backup window of 4 hrs. How can I meet my SLA? Any recommendations from HPE?”

In a previous blog titled, Protect your Oracle Database backup with HPE StoreOnce systems and StoreOnce Catalyst, we introduced the StoreOnce Catalyst plug-in for Oracle Recovery Manager. Today, we will do a deep-dive on how to tune the entire solution stack – Oracle, Oracle Recovery Manager (RMAN) and StoreOnce Catalyst – to achieve the bottom-line: meeting the RPO/RTO SLA for backups and restores. Also, we will discuss briefly about the newly released Recovery Manager Central for Oracle (RMC-O), which enables fast data protection for Oracle using 3PAR StoreServ systems as data storage platform.

First things first, I would like to take you through the 10 silver bullets that I recommend to increase performance for Oracle backup/restore to meet your SLAs:

1. Enable ArchiveLog mode: Check if the database is in the ArchiveLog (OR) non-ArchiveLog mode. If online backup of the database is desired, i.e. backup of the database without shutting it down while applications are using the database continuously, enabling ArchiveLog mode is recommended. Once this is done, periodic backup of the archived Redologs is recommended to achieve the desired RPO

2. Analyze the physical structure of the database. How many datafiles are present and what are their sizes? Are datafiles in all tablespaces evenly loaded with data (used space)? Is there a mix of bigfile and smallfile tablespaces in the database? Is storage using a filesystem or ASM? The reason to check these aspects is, if there are unevenly loaded datafiles (OR) if there is a mix of bigfile and smallfile tablespaces multi-section backup is recommended. This enables evenly load-balanced backup across all RMAN channels allocated and prevents stranded running channels that get allocated with the particular big datafiles and thus elongating the backup window. Remember, even though datafiles’ allocated size would be similar to one another, with Oracle null-block compression only the used pages are backed up and the pages that never contained any data are ignored. Hence just looking at the Allocated Size is a skewed approach.

3. Backup performance depends on your DB server + storage + networking infrastructure, mainly: I have had many proof-of-concept situations where the customer demanded 10xTB/hr when reality was just xTB/hr. Closer investigation has always revealed that the problem is elsewhere such as an absence of 10GigE on hosts, storage arrays not able to pump enough data at the desired rate, networking issues, etc. Once the bottleneck was resolved, expected performance was achieved. The best way to verify this in your particular environment is to do a “BACKUP VALIDATE DATABASE” with similar settings, number of channels, etc. as your normal backup operation. Backup Validate reads all data but doesn’t backup to disk/tape. Hence you will come to know what the data pumping capacity of the server/storage/network infrastructure is, before proceeding to set expectations around the backup SLAs. If the backup validate performance is satisfactory, then the goal would be to achieve the same with actual backups.

4. Understanding RMAN Input and Output buffering: When allocating RMAN channels, it is always suggested to use larger block-sizes for I/Os such as 4MB. To that effect, the parameter BLKSIZE can be used while allocating the RMAN channels to tune the Output Buffer size. The number and size of the RMAN Input buffers are decided and allocated automatically by Oracle depending on two factors: 1) based on whether the DB is on Filesystem or on ASM and 2) based on the multiplexing level The best performance is always seen for the StoreOnce Catalyst plug-in for RMAN with multiplexing level =1 (i.e. MAXOPENFILES = 1) for DBs on filesystems and multiplexing level=2 (i.e. MAXOPENFILES =2) for DBs on ASM with default values of FILESPERSET.

5. Enabling BACKUP_IO_TAPE_SLAVES and LARGE_POOL: When using any SBT_TAPE device such as StoreOnce Catalyst plug-in for backing up Oracle, it is important to enable the BACKUP_IO_TAPE_SLAVES option for 3 key reasons. Firstly, enabling it will make way for DBAs to use RMAN duplexed backups option which will come in handy during disaster recovery scenarios for recovering data from secondary copies. Secondly, all I/O processes to the SBT_TAPE devices are synchronous by default. Enabling the tape secondary option makes the I/O to be asynchronous. Thirdly, the benefit is that the DBA can manage memory allocation to RMAN channels manually when enabling the tape secondary. By default, when tape secondary are disabled, memory for Oracle RMAN comes from the Oracle Programmable Global Area (PGA) but when tape secondary are enabled, the memory allocation comes in from SGA’s shared_pool area. If the DBA wants to be assured of memory being granted to the RMAN channels 100%, then he can configure the LARGE_POOL which guarantees memory allocation for the RMAN channels. The formula is:

LARGE_POOL_SIZE = (No. of RMAN channels * (16 + 4*(output_buffer_block_size)) +16 Mb

6. Enable BCT(Block Change Tracking): One of the major complaints that I usually get from customer’s DBAs is that their incremental level 1 backups take as much time as a FULL backup (an incremental level 0 backup) even though the size of the incremental level 1 backup would be a very small percentage of the Full backup. This is expected as RMAN, when it is asked to do an incremental level 1 backup, scans through the entire database to find the changed blocks and then back it up. This can be worked around by enabling the Oracle RMAN Block Change Tracking feature. Once enabled, this feature keeps track of any changes to any block in the DB. Hence, in the event of RMAN doing an incremental level 1 backup, it immediately knows from the block change tracking file which blocks have changed and then backs it up very quickly. For example, in one setup, it took 1200 seconds for a non-BCT incremental level 1 backup to complete vs. 20 seconds for a BCT-enabled incremental level 1 backup. Note that there will be a small performance penalty when you turn on the BCT feature but it is definitely worth it considering the time savings.

7. Load-balance backups across all RAC nodes, if possible: Backups complete faster if ALL nodes in a RAC cluster are used as “data-movers” for backing up data to StoreOnce. This can be achieved by allocating channels to each of the RAC instances present. Use the “connect user/password@instance_number” to explicitly mention RMAN to allocate channel to a particular instance. The upside to this approach is that backups complete faster as the load is shared by all nodes. The downside to this approach is that the increased load on every node in the cluster. Hence based on your requirements, use any number of nodes to parallelize your backup and restore operations.

8. Use BACKUP OPTIMIZATION and RESTORE OPTIMIZATION: Whenever Backup Optimization option is used, it skips backup of datafiles that have no changes that have already been backed up certain number of times until the specified retention time is met. This comes handy especially when backing up Archive RedoLogs regularly in short intervals during the day. Hence backup can complete faster if there aren’t a lot of changes in a lot of datafiles. One of the other good options RMAN provides is Restore Optimization which does the same as backup optimization during the restore process.

9. Enable “controlfile autobackup” and also “validate” your backups once in a while: Controlfile is the brain for any Oracle database and has to be backed-up continuously so that you have the latest representation of the Database structures represented in your controlfile backup, if required to be restored. Configuring controlfile autobackup is an easy way to accomplish the same so that the control file can be made as a part of every backup done. Also, it is a good idea to test your restore, as well as DB recovery, once in a while to make sure your backups are consistent through the Restore Validate

10. StoreOnce performance tuning: Once you have completed all tuning on the Oracle, RMAN side and the rest of the ecosystem, tune your StoreOnce settings. Always check to be on the latest firmware version for StoreOnce and the latest version of the StoreOnce Catalyst plug-in for Oracle RMAN. Always parallelize your backups and restores to achieve best performance. For more in-depth performance tuning guides for StoreOnce Catalyst plug-in for Oracle, reach out to your HPE contact, who will be happy to provide you with a copy.

Here are the results of the above step-by-step tuning benefits that we saw:

Introducing Recovery Manager Central for Oracle – HPE Recovery Manager Central for Oracle (RMC-O) is a brand new addition to the HPE data protection portfolio to protect Oracle DBs constructed on 3PAR StoreServ primary storage. RMC-O allows Oracle database administrators to create, schedule and manage Oracle application-consistent snapshots on a 3PAR array (for Oracle environments running Red Hat Enterprise Linux or Oracle Enterprise Linux). The RMC for Oracle Express Protect feature enables automatic backup of Oracle snapshot data from HPE 3PAR to StoreOnce, independent of any backup software. RMC for Oracle Express Protect delivers up to 20X faster backups and up to 5x faster restores than traditional backup/recovery methods at an attractive price point. This provides simple, rapid and affordable recovery of Oracle mission-critical databases as well as flexible RPO and RTO. Here is the summary of the holistic storage solutions from HPE Storage for your Oracle environment right from the 3PAR StoreServ platform plus associated software for primary storage/DR requirements and StoreOnce Systems plus data protection software components. 

Meet Around the Storage Block blogger Bharath Ram Ramanathan, Worldwide Solutions Technical Marketing Specialist, HPE Storage.