Around the Storage Block
Showing results for 
Search instead for 
Did you mean: 

The Critical Role of Storage, Servers and Networking in Big Data Solutions



When building a Big Data solution, much of the development focus is on the servers and storage infrastructure. But keep in mind what a critical role the network also plays in delivering scalable end-to-end performance. 

From the beginning, one of the key design considerations for Hadoop was the ability to scale-out processing across hundreds to thousands of nodes built using commodity hardware – usually with two-socket Intel® processors, 10-to-12 1TB SATA drives and 1Gb networking.

As technology has evolved, modern computing now offers larger drives, faster processors and faster networking – at almost the same cost. With the explosion of data over the last five-to-10 years, along with the need to ingest and process this data in a timely manner, organizations are looking to accelerate analytics applications and data processing pipelines by exploiting these advances in modern computing for real-time analytics and faster query times. The importance of delivering this data quickly and efficiently is higher than ever.

Factoring in servers, storage and networking for Big Data

When building a Big Data solution, much of the development focus is on the servers and storage infrastructure. However, the network plays a critical role in delivering scalable end-to-end performance. big data_networking_storage_blog.jpg

Big Data clusters are now built using SSD, NVMe and other solid-state storage devices to reduce the storage device latencies associated with spinning media by configuring hot data and logs on the faster storage tiers. These changes are shifting the bottleneck of the clusters from CPU, memory and storage to network. 

More and more frequently, 1Gb and now 10Gb network infrastructures are becoming the weakest link in the chain. Higher performance network fabrics are a critical component to reducing network latencies and improve overall performance for the Big Data cluster—and that has typically come at a high cost premium.

Ethernet-over-high-speed Infiniband and 40Gb network fabrics have been price prohibitive for a lot of customers looking to build new big data clusters. Yet the importance of a high-speed fabric has never been more critical. With the introduction of 25/100Gb network fabrics, you can now have a more modular approach to building the critical network layer to increase bandwidth and reducing network latencies, while maintaining the same cost as previous 40Gb network solutions.

As Big Data matures and the number of workloads you want to run increases, the need for optimized compute and storage resources is becoming more prevalent. This shifts the design focus from traditional symmetric designs towards disaggregated compute and storage models with optimized compute resources for running multiple types of compute.

The elastic platform for analytics workload and density optimized design

We at HPE refers to this as the Elastic Platform for Analytics Workload and Density Optimized (WDO) design. In this design paradigm, we are seeing flash storage in both the compute (e.g. for Apache Cassandra) and storage blocks (e.g. tiered storage). 

Compute nodes with SSDs require connections faster than 10GbE. Likewise, storage systems with 20+ spinning disks or 3+ SSDs (even the slower SAS/SATA SSDs) often need 20Gb/s or more of bandwidth to ensure that network is not a bottleneck. This makes 25GbE ideal for new server deployments, 40Gb or 50Gb (2x25Gb links) ideal for storage, and 100Gb the most efficient link to aggregate switch traffic or bridge data center racks and rows. 25 Gb and 100 offer not only faster performance but lower cost and easier scaling for large big data configurations than 40Gb network fabrics.

The importance of the network

Let’s look at an example to highlight the importance of the network. If we say the average throughout of a HDD is 100MB/sec, then with a single 10G link the bandwidth is saturated with 10 HDDs and dual 10G links are saturated with 20 HDDs. 

With density optimized symmetric designs or storage blocks in an EPA WDO you will need more than 2x10 Gb Ethernet to full realize the throughput capabilities of the server with >20 HDD. For example, an HPE Apollo 4200 Gen9 server with 28LFF, each doing ~100MB/sec in Hadoop, that gives you a total of 2800MB/sec per node, so in order to not have a network bottleneck you need to have network pipe >=25G (25Gb = 3125MB/sec). SSD exacerbates this issue even further.  Taking a throughput rate of ~500MB/sec, with 2 SSDs you will already saturate a 10G link, and 4 SSDs are enough to saturate dual-port 10G cards.

25GbE is a new standard that leverages technology defined for 100 Gigabit Ethernet implemented as four 25-Gbit/s lanes running on four fiber or copper pairs. The 25GbE and 50GbE solutions are backward and forward compatible with 10, 40/100 Ethernet products, since they use the same IEEE 802.3 frame format. The 25 Gbps links used in 100 GbE are attractive alternatives to 10 GbE, being faster for a relatively small increment of cost.big data_networking.jpg

What’s next with network bandwidth requirements?

The Dell’Oro Group predicts that our customers’ data centers are on the verge of experiencing a major speed upgrade cycle, driven by server migration to 25GbE. This is driven initially by large scale data centers, with the rest of the market following suit quite rapidly, as the price premium of 25GbE over 10GbE collapses on the switch side. The 25/50/100GbE network will comprise over a third of server access ports, within just three years of shipments. 

25_100GbE market momentum_big data.jpg


Many organizations currently deploy top-of-rack architectures that utilize 10GbE. To keep up with the required network bandwidth, they would currently need to deploy twice as many 10GbE switches, along with additional cables, space, power, and cooling.

Utilizing 25GbE results in a single-lane connection similar to existing 10GbE technology—but it delivers 2.5 times more data. Compared to 40GbE solutions, 25GbE technology provides superior switch port density by requiring just one lane (vs. four with 40GbE), along with lower costs and power requirements. The 25GbE specification will enable network bandwidth to be cost-effectively scaled in support of next-generation server and storage solutions residing in cloud and web-scale data center environments.ProLiant_networking_big data.jpg

A rack of 18 HPE ProLiant DL380 Gen9 servers with 25GbE network costs about 2-3% more than 10GbE network, but would increase networking throughput by more than double. A rack of 18 DL380 Gen9 servers with 100GbE network costs about 2% more than 40GbE network, but would increase the network bandwidth by more than double.

If you want to reduce the number of cables and switches with 25% more network bandwidth, you can use a single port 100GbE NIC and 100GbE switch instead of two 40GbE switches the saving is about 4-5%. The HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter costs are comparable or about 1-2% more than HPE Ethernet 10Gb 2-port 560SFP+ Adapter network adapter but will get more than double the throughput.

When it comes to network switches, the HPE FlexFabric 5950 48SFP28 8QSFP28 Switch 48 port 25GbE switch costs the same as HPE FlexFabric 5940 32QSFP+ Switch. HPE FlexFabric 5940 48SFP+ 6QSFP+ is about 30%, and HPE FlexFabric 5940 48SFP+ 6QSFP28 Switch is about 19% less than 25GbE switch whereas 25 GbE switch provides 48% and 25% better performance respectively compared to the 10GbE switches.

Dig deeper

The following papers offer more details information on the components:

HPE Reference Configuration for networking best practices on the HPE Elastic Platform for Big Data Analytics (EPA) Hadoop ecosystem

HPE Reference Configuration for the Elastic Platform for Big Data Analytics

Get the latest sizing-and-build information using the HPE EPA Sizing Tool.


Ramesh Peddakotla-HPE Big Data.JPG

Meet Around the Storage Block blogger Ramesh Peddakotla, Sr. Software Engineer, HPE Big Data Solutions Engineering.





0 Kudos
About the Author


Our team of Hewlett Packard Enterprise storage experts helps you to dive deep into relevant infrastructure topics.

Christopher Smith

Your statement, "With the introduction of 25/100Gb network fabrics, you can now have a more modular approach to building the critical network layer to increase bandwidth and reducing network latencies, while maintaining the same cost as previous 40Gb network solutions." reads wrong. This should be compared to "previous 10Gb network solutions" to make sense as most organisation are deploying 10Gb as opposed to 40Gb networks.

I would also like to see a discussion on the impact of TCPIP Incast on Hadoop and hos HPE Networks can mitigate this issue, see:




Online Expert Days - 2020
Visit this forum and get the schedules for online Expert Days where you can talk to HPE product experts, R&D and support team members and get answers...
Read more
HPE at 2020 Technology Events
Learn about the technology events where Hewlett Packard Enterprise will have a presence in 2020
Read more
View all