Managing chaos in a containerized environment

by | Sep 30, 2019 | Hybrid Cloud

Quick, name some weird stuff that’s happened to your production machines.

Accidentally dropping a production database table? Rolling out a patch that enabled any user to log in with any password? Disabling a load balancer? Using a dictionary to physically keep keyboard keys depressed so “terminals [could] repeatedly [hit] the enter key in order for the logins and print jobs of about 40,000 people to work”?

It’s happened to Alex Corvin, a senior engineer at Red Hat. Well, not that last one. But Corvin has been around long enough in his career to have met Mr. Murphy and his Law: if it can go wrong, it will.

Corvin, knowing we live in an entropic universe, cheerfully calls such occurrences “chaos.” And those were just a few of the human-created issues he’s seen. Sometimes it’s a cloud region going down, malicious hackers DDoSing your systems, or a straight-up hardware failure. Whatever the cause, and whatever the effect, chaos is going to be a part of any IT professional’s life at some point or another.

Sometimes that chaos starts off as a benefit. An app is deployed across container space; traffic climbs as it becomes popular, revenue flows in, promotions seem guaranteed… until a database server overloads and takes a header. Tale as old as time, right?

In a world before OpenShift, this sort of incident could be prevented by proper load balancing and configuration management. But, Corvin emphasized in a recent interview, these operations would need to be calibrated manually. Nowadays, OpenShift can step in and automatically help manage chaos.

For instance, monitoring in any Kubernetes environment is critical, and Prometheus has already emerged as a clear preference for monitoring in Kubernetes and OpenShift. High availability management becomes a lot more straightforward in OpenShift, too, since in traditional deployments, adding redundancy meant you needed to deploy another instance of the app and configure custom load balancing. OpenShift provides this feature out of the box.

Looking down the road

Beyond what OpenShift can provide, Corvin and his teammates are also working on a new innovation that could prove to be a game-changer for containers and data management. Known as the Data Catalog, this prototype is being built with the specific goal of managing the data side of deployments. 

Specifically, the Data Catalog would control access, ownership, and exploration rights to any data lake, Corvin explained. This collection of open source tools, still in its early days of development, would also enable data governance policies to be implemented, making it a potential one-stop shop for data management in container environments. This new way of using existing tools is one more innovation that will eventually find its way into Open Data Hub.

No one tool can take away all of the chaos in IT, but existing tools in the OpenShift toolkit, and developing ones from Red Hat, are going to do their best to mitigate what chaos they can.