HPE Storage Solutions for Splunk: A critical enabler for optimal data management

[StorageExperts]

Machine-generated data holds valuable insights, critical to both the security and profitability of the enterprise. Are you making the most of your data assets?

Every IT application, system, and piece of infrastructure in a company, right down to the light switches, generate data at millisecond intervals. This machine-generated data is as complex as it is invaluable. It is also one of the fastest growing areas of Big Data.

Machine-generated data contains a detailed record of all user transactions and individual customer and component behavior. It further captures sensor activity, machine performance, security threats, and fraudulent activity, to name a few. The data holds valuable insights, critical to both the security and the profitability of the enterprise.

Splunk, a leader in the collection and analysis of machine-generated data, helps simplify and modernize IT thru its operational analytics capabilities, and expands into business analytics, security, and IoT use cases. Storage is a critical enabler for a successful Splunk software deployment.

The storage layer faces a serious set of requirements in operational analytics environments. These include:

• Performance

  The ability to store incoming events (ingest), while at the same time accessing the data for reporting & historical analysis, results in demanding and spiky IO profiles.

• Scalability

  The storage architecture must provide near limitless scalability as the number of ingest devices increases, or data retention periods are extended to support additional use cases.

• Economics

  IoT and operational analytics environments tend to see data grow at dramatic rates, hence the cost aspect of the underlying storage layer is critical for a positive ROI.

HPE Storage provides a range of differentiated solution architectures that fulfill the most demanding Splunk requirements, while at the same time providing an optimal price-point. Hewlett Packard Enterprise brings an advantage to the table by offering a wide variety of storage choices. This allows customers to choose the optimal solution based on their specific needs.

Architectural option #1: Direct-attach storage (DAS) with HPE Apollo 4000 or HPE ProLiant servers with an optional S3/Object

An architecture based on a scale-out approach, leveraging x86-based servers with an optional Splunk SmartStore tier, on S3-compatible object storage. This option is optimal for massive-scale deployments, environments with an existing object store, or for those interested in building a multi-use case object store. HPE Apollo 4000 servers provide unmatched capacity-density as well as a combination of Persistent Memory, NVMe, SSDs, and high-capacity HDDs.

Architectural option #2: High-performance data services with rich shared all-flash or hybrid storage arrays based on HPE Primera/Nimble

Shared array-based architectures provide a number of advantages in addition to the ability to independently scale compute from storage (aka disaggregation):

• High performance through the use of all-flash and hybrid deployments
• Multi-workload and multi-tenant consolidation onto same platform; ability to consolidate Splunk as an additional application onto an already existing primary array
• Mission-critical reliability
• Efficient use of storage capacity through compression, deduplication and system-inherent redundancies
• Advanced data protection options through the use of snapshots and data mobility via built-in HPE Recovery Manager Central (RMC)
• Pain-free management of infrastructure through HPE Infosight predictive analytics
• Cloud-like, consumption-based acquisition.

HPE GreenLake Flex Capacity allows cloud-like procurement through a pay-as-you-go, consumption basis model. HPE GreenLake allows Splunk users to scale storage and compute capacity up and down while paying only for the capacity consumed.

As retained data capacity increases, a second tier of storage based on S3/object assists in keeping costs under control. HPE provides architectural choices where arrays are combined with an S3/object tier leveraging the Splunk bucket tiering capability to transparently move data to the most appropriate storage tier. The private cloud on-premise S3/object capabilities from HPE are provided through HPE Scality RING Scalable Storage.

Want to dive deeper? HPE has recently completed a solution engineering test effort around varying daily Splunk ingest rates. The corresponding whitepaper describes the impact on the storage arrays, and investigates the effect of moving frozen buckets out to an S3 on-prem private cloud. The usefulness of rich data services is also highlighted in the paper. Watch out for additional test results being published in this space in the near future.

For additional information, please check out these resources:

• HPE Storage for Splunk architectures and validation paper from Evaluator Group 
• HPE Solution Engineering technical whitepaper featuring primary array (HPE Primera and HPE Nimble) along with S3/Object test results
• HPE Storage for Splunk solution brief 

---

Meet Around the Storage Block blogger Tilman Walker, Manager, WW Storage Solutions.

Storage Experts
Hewlett Packard Enterprise

twitter.com/HPE_Storage
linkedin.com/showcase/hpestorage/
hpe.com/storage