Sharing knowledge and empowering others through open source projects

[DevComExperts]

-- How KubeDirector empowers data-intensive apps --

Robert Noyce, co-founder of Intel Corporation, once said “Knowledge shared is power multiplied.” I believe open source projects are the biggest testament to the power of knowledge sharing. It gives me immense pleasure to learn and collaborate with people worldwide, which is why I am part of the open source community.

Experience with open source and KubeDirector

First, let me tell you a little background on my experience with open source technologies. In 2013, I graduated from Indiana University Bloomington with a MS in computer science and a research focus on parallel computing and distributed systems, I started my professional career at AMD, then worked at several technology start-up companies, including BlueData, which was acquired by HPE in 2019.

I’m currently a Master Technologist at HPE, leading the MLOps initiatives on the HPE Ezmeral Software Platform. I specialize in container orchestration for large scale data processing clusters, and I’m a major contributor to the KubeDirector open source project.

I first started contributing to open source for a project that lets developers query Cassandra tables using Spark SQL. Since that time, almost any piece of software I’ve written or consumed has been based on open source projects or libraries in some capacity. In my role at BlueData, I first used KubeDirector to orchestrate Big Data and ML pipelines, but then I started contributing as well. I found this project so engaging, I’m currently the leading contributor and maintainer for the KubeDirector project.

What’s so special about KubeDirector?

KubeDirector is empowering because it allows application developers to deploy their applications as a custom resource without having to implement a full-blown Kubernetes Operator. It decouples the operator boiler plate using application intelligence.

Basically, KubeDirector works as a custom controller for generic applications. It uses standard Kubernetes (K8s) facilities of custom resources and API extensions to implement stateful, scale out application clusters. This approach enables the transparent integration with K8s user/resource management, as well as existing K8s clients and tools.

KubeDirector is unique since it has a rich catalog of complex stateful applications as part of the open-source codebase. And we are constantly adding more and more applications that developers can use as a template/example to onboard their application of choice on Kubernetes. Without KubeDirector, this task would be intimidating with a huge learning curve, especially for implementing Day 2 operations for applications, such as scaling in and out.

How HPE customers benefit from KubeDirector

Over the last few years, Kubernetes has become the de-facto standard for orchestrating containerized applications. It’s worked quite well for stateless applications, but it is less effective for stateful applications, such as those focused on AI, machine learning, and big data analytics.

When containers were first introduced as a simple way to package microservices, they were designed to be entirely stateless and ephemeral. A container would spin up, do its job, and then disappear--without leaving any record of what happened while it was running.

In contrast, stateful applications save data to persistent disk storage for use by the server, clients, and other applications. An example of a stateful application is a database or key-value store to which data is saved and retrieved by other applications. Tools, such as stateful set and persistent volumes help developers build stateful applications on Kubernetes, but the entire process is quite difficult to manage as an application scales.

Since KubeDirector provides an application-agnostic deployment pattern, it enables developers to run non-cloud native stateful applications on Kubernetes without modifying any code. KubeDirector makes it easier to deploy data-intensive distributed applications for AI and analytics use cases, such as Hadoop, Spark, Kafka, TensorFlow on Kubernetes. HPE Ezmeral customers automatically receive the benefits of KubeDirector because it’s integrated into the HPE Ezmeral Software Platform, enabling them to develop AI and ML applications more easily.

To learn more about the open source projects HPE is involved with, please visit the HPE DEV website. Interested in exploring what HPE offers for developers and data scientists? Check out our the site for a ton of articles, workshops, tutorials, and other resources.

About the author:

Kartik Mathur is an engineering manager for the HPE MLOps team.

Hewlett Packard Enterprise

https://twitter.com/HPE_DevCom

 https://developer.hpe.com/