The Cloud Experience Everywhere
Jim Turner

DPTIPS: Catalyst Housekeeping and Other Chores

Man In Cafe.jpg

I was recently asked to describe how Micro Focus Data Protector and HPE StoreOnce Catalyst manage deletion of expired media.  Specifically, what happens when protection expires in DP for one or more Catalyst media in a Backup to Disk device, and how does that relate to recovering storage space in StoreOnce?  Should I see more available space immediately?  What exactly is "housekeeping"?  And what happens if a DP admin manually exports expired Catalyst media in an attempt to free StoreOnce space?  Prior to clicking "Send", it occurred to me that the information in my email response could benefit a wider audience if I share it here.  And so we have today's DPTIPS article.

Housekeeping comes in two forms with StoreOnce Catalyst and Data Protector.

At the DP application layer, there is a process that runs at noon each day (more often if you wish) that does two things:

  1. Identifies and exports all expired Catalyst media from every media pool.
  2. Makes a Catalyst Client API call to StoreOnce to delete from each Catalyst store the items that were just exported from DP’s MMDB.

The Catalyst item deletion resulting from #2 above triggers the queueing of housekeeping (HK) jobs for all affected stores within StoreOnce. There are two main phases to HK in StoreOnce.

  1. If a block of deleted data is a pointer, the stale reference is simply removed from the actual block of data being referenced.
    1. If said block is still referenced by other pointers, nothing else happens for that block.
    2. If said block is now completely dereferenced, it is marked as a candidate for reuse.
    3. This activity takes place whether or not the Catalyst store is busy.
  2. When greater than 10% of blocks in a container are dereferenced, container compaction is queued.
    1. Container compaction is a highly IO intensive operation and would adversely impact running sessions.
    2. Thus container compaction is held off until the store is idle (no ingest, no reads, no replication).
    3. Physical space within StoreOnce is only recovered once container compaction completes.
    4. Container compaction for a Catalyst store will be suspended if host IO resumes with that store.

Some things to keep in mind.

  1. New data being ingested by StoreOnce is always written to free physical space. Read that again – it’s important.
  2. Keeping a Catalyst store busy 24x7x365 is a bad thing. It means that container compaction will never run, and you will have ever-increasing physical space consumption.
  3. Size and space are different depending upon who you ask.
    1. DP measures data in logical terms based upon what the host sees.
    2. Catalyst measures data based upon physical StoreOnce storage after dedupe and compression.
    3. The difference between a and b is your deduplication ratio.
  4. Deduplication is a double-edged sword.
    1. Great for cramming a ton of logical data into a much smaller physical space.
    2. Can be confusing when you delete a large amount of logical data within the DP application.
      1. Loads of logical data could be mostly pointer info which results in only modest physical space recovery and a mathematical reduction in dedupe ratio.
      2. Even when physical blocks are dereferenced, an increase in free physical space is only realized in StoreOnce after container compaction completes.

There are certain scenarios in which older, obsolete Catalyst items can be abandoned in a store with no corresponding media in DP. If a DP admin manually recycles Catalyst media in DP thus giving it an expired status then manually exports the expired Catalyst media from DP afterward, that removes DP’s ability to sense the presence of expired media during the daily job and carry out the normal process of export from DP / delete from Catalyst store.

If an admin needs to manually remove Catalyst media from DP, it is critically important to do these two specific things – nothing more, nothing less – in this order.

  1. Recycle the Catalyst media that needs to be removed.
  2. Execute “omnimm -delete_unprotected_media -all -force”

Step 2 ensures that the expired media is exported from DP’s MMDB and that the corresponding Catalyst items are deleted from StoreOnce.

But none of this addresses existing abandoned Catalyst items. The only real option is to use the StoreOnce web interface to identify and manually delete the Catalyst items that have no corresponding media in DP. Then going forward, refrain from manually exporting Catalyst media in DP. Only recycle Catalyst media and let DP eliminate it organically via daily automated process or interactive execution of “omnimm -delete_unprotected_media -all -force”.

With Data Protector and StoreOnce Catalyst, you enjoy the automatic creation, management, and deletion of backup target media along with the benefit of inline data deduplication.  I'm part of Advisory and Professional Services within HPE Pointnext.  As a Data Protector and StoreOnce subject matter expert, I can address most any situation you may face with DP and StoreOnce.  You can use your contract time to engage my services for anything from upgrades to platform migrations, application and storage integrations, troubleshooting, optimization, and deep dive knowledge transfer.  Contact your HPE Account Support Manager (ASM) or reach out to me directly for more information on how we can work as partners to help you succeed!

Jim Turner
Master Technologist
Hewlett Packard Enterprise

twitter.gif @_Vejete_
linkedin.gif dataprotection 

About the Author

Jim Turner

Jim is a multi-disciplined engineering professional with 32 years of electronic and systems experience. For the past 19 years, Jim's primary focus has been enterprise backup, recovery, and archiving (BURA). As an HPE Master Technologist and Veeam Certified Engineer, he is recognized as a global authority on data availability and protection. Jim's consulting has stretched over 425k miles and 156 unique locations in North America during his 10 years with HP(E). When not traveling, Jim resides in Edmond, OK with his wife and three dogs.