The data dilemma: Can it stay, or should it go?

[Eric_J_Brown]

Keep more data for extended periods using the Elasticsearch frozen tier.

The combination of Elasticsearch and Scality RING on HPE solutions ensures that enterprise organizations can have confidence that their data will be managed throughout its lifecycle based on index lifecycle management policies.

What would we do without our mobile phones? They serve myriad purposes to help us accomplish various daily tasks while offering many forms of entertainment. In fact, for many, making calls is no longer primarily what we use our phones for most.

So how much and what we store on them before it impacts their usefulness? Think about, for example, downloading entertainment options to occupy the time for long flights. You spend all that time spent finding videos, games, music, and e-books to enjoy at the airport and on the plane, only to discover that your phone has no free space.

It’s the hassle of storage management: What can or cannot be deleted? (As you constantly check how much free space was gained.) Wouldn’t it be great if all that data could be automatically managed?

This is significantly more challenging for companies due to the tremendous amounts of data generated. You must effectively manage all that data while balancing keeping it with the risk of storing unnecessary data. In addition, your digital exhaust is used to gain insights to drive better business outcomes. That’s why you need to optimize your storage resources – without having to overthink resource management or costs.

The frozen tier for Elasticsearch

Elastic offers solutions that help companies solve these data dilemmas. Elasticsearch simplifies storing and accessing structured or unstructured data that companies create, enabling them to search and analyze large volumes of data in real-time. In addition, Elasticsearch supports data tiers to optimize storage resources based on different needs and usage patterns. Data can be automatically migrated between tiers based on index lifecycle management (ILM) policies, ensuring it’s stored on the most appropriate media based on value and access frequency.

Elasticsearch searchable snapshots transform how data is retained while remaining searchable for the long term. As data ages and is rarely queried, snapshots of indexes can be performed and then migrated to the frozen tier for Elasticsearch, which can be an Amazon S3-compatible object store such as Scality RING. This optimizes storage resources by eliminating the local copies because searches can be performed directly on the snapshots in the object store.

Efficient long-term data retention in action

To demonstrate how the Elasticsearch frozen tier and index lifecycle management policies help companies optimize long-term data retention, I deployed an Elasticsearch cluster to utilize ILM policies to migrate data between tiers automatically. Enabling the frozen tier for Elasticsearch required registering a Scality RING bucket as a snapshot repository. Then various logs were ingested into Elasticsearch using Beats. Once the logs were in Elasticsearch, it was confirmed that they remained searchable throughout the data retention lifecycle:

Read more about this solution: Implement efficient long-term data retention with the Elasticsearch frozen tier on HPE Solutions 

Combining Elasticsearch and Scality RING on HPE Solutions

HPE offers extensive computing, networking, and storage options to give companies flexibility when deploying Elasticsearch and Scality RING.  HPE has the experience and expertise to enable companies to connect, protect, analyze, and act on all their data. Overall, using HPE Solutions provides reliable, secure, innovative, and customizable infrastructure backed by excellent support services.

The combination of Elasticsearch and Scality RING on HPE solutions gives you the confidence that your data will be managed throughout its lifecycle based on index lifecycle management policies. The Elasticsearch frozen tier lets you keep data for extended periods on a cost-effective object store while keeping that data searchable.

Learn more about Storage Solutions for Scality.

---

Eric Brown
Hewlett Packard Enterprise

twitter.com/HPE_Storage
linkedin.com/showcase/hpestorage/
hpe.com/storage