Based on the platform nature of many Red Hat products, our engineering teams are in continuous collaboration with many of our ecosystem partners. Recent collaborations in the solution testing have resulted in noticeable improvements to our core partner continuous integration/continuous delivery (CI/CD) tools known as Distributed CI (DCI).
As software systems continue to grow more complex, distributed in development, and dependent upon services and virtualization components from different vendors, CI/CD has become critical to maintaining software stability. While applicable to all customers in general, the individual requirements of the telco customer make working together ahead of multi-vendor solution testing and delivery critical. More and more, customers expect that marketed partnerships translate into multi-vendor solutions that operate as if they are delivered from one company.
Red Hat’s now+Next blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.
Continuous collaboration
Red Hat offers DCI to our partners to enable them to test their solution component(s) with select Red Hat products (Red Hat OpenShift Container Platform, Red Hat OpenStack Platform, and Red Hat Enterprise Linux) before they are generally available. At its core, DCI delivers daily development life cycle stage versions of these products. It also provides a web portal view of remote (partner) CI/CD job results that is shared and monitored by Red Hat and the partner.
Over the past year, Red Hat has explored various ways to improve DCI ease of use, as well as ways to add more value to the service. The focus of this work has been on how the partner integrates and uses DCI, and has resulted in collaboration with a few select partners to roll out a new DCI service consumption model. This model was identified to improve partner DCI deployment stability, as well as to enable a more seamless use of native partner testing and resources within the DCI monitoring framework.
CI/CD without borders
While it is historically common to perform CI/CD within organizations (whether corporate or community), there is no reason that it can’t be done between vendor ecosystems or tactical partners. Open source software development is a great example of how software development and testing can be done across vendors and even individuals. The difference being referred to here as “CI/CD without borders” is specifically that CI/CD becomes increasingly more common between vendors. The ultimate goal is that the customer’s exposure to multi-vendor solution issues must be minimal in order for all vendors and customers to succeed. In fact, there are many economic benefits for investing in CI/CD among solution stack vendors.
- Solution stack stability (proactive process).
- Broader deployment configuration and environment validation (hardware, skill set).
- Knowledge transfer of partner stack component (collaboration).
- Regular cadence of solution quality monitoring at the engineering level (continuous collaboration).
Red Hat’s fundamental goals for providing access to pre-release versions of our products are:
- To enable our ecosystem partners to release their latest code alongside Red Hat’s latest releases.
- To engage at an engineering level to collaborate on aligned features.
- To squash integration issues early and ease troubleshooting.
- To cross-train at integration points to improve deployment and field collaboration.
- To enable testing of integrated solutions on Red Hat partner-focused hardware configurations.
So how can we “distribute” CI between partners?
DCI was born with roots in the OpenStack community project, with the goal of addressing platform deployment and upgrades. It evolved into a vehicle for enabling remote platform distribution and integration by implementing a cloud-based control server along with a customer premise agent. The DCI agent both hides some of the service configuration from the consumer as well as provides templates for partner customization. The figure “DCI Agent Deployment Model” below represents the architecture and flow of the current, most popular way to access DCI.
At the core of DCI is the control server that sits in the Red Hat service cloud. DCI Control Server securely connects to vendor DCI Agents, presents DCI remote CI job status, and serves Red Hat product releases as they come off Red Hat internal CI/CD pipelines. For each Red Hat product supported by DCI, a new feeder and agent are developed and supported by Red Hat. DCI tools are primarily written in Ansible with configuration in YAML manifest files.
DCI Agent overview
Before we move on to the new DCI updates, let’s take a more detailed look at the OpenStack DCI Agent. This overview will help communicate what DCI offers, either through the agent or, as we will see next, via direct DCI application programming interface (API) calls without the agent.
The first section in the DCI Agent performs the initial setup, which schedules a new job specified by a product release version (target) and a previously generated session ID (remoteCI). DCI returns all the data associated with the job that enables subsequent testing and reporting.
The next three sections of the DCI Agent specify the pre-run, run, and post-run phases of testing. These sections are all provided to the user as examples that ultimately can be refined or redefined by the partner. The Agent is delivered with the following sample configurations.
- Pre-run boots the undercloud without provisioning it.
- The run phase then provisions the undercloud as well as the overcloud.
- The post-run phase runs the tests, organized as blackbox API testing (for the Openstack Agent, this would be the Tempest tests), and hooks to incorporate partner functional tests if desired.
Red Hat has found that functional testing adds significant value to the overall quality and readiness to the solution. For instance, upgrade functional testing alone can identify performance and operations issues proactively, furthering customer confidence in the solution during the sales cycle. Another example of the value of functional testing beyond API/smoke testing is in regular focus on targeted solution components, such as a network function’s kernel dependencies.
In the final section, the deployment is torn down and all files are cleaned up. Within all of these steps, job status reporting back to the DCI Control Server is done and subsequently posted to the DCI Dashboard for Red Hat and partner monitoring. The DCI Dashboard (GUI) provides Red Hat and partners with authentication-based (i.e. private, secure) monitoring of DCI job results.
The following “dci-openstack-agent” pseudocode further provides a view into the agent workflow.
Recent DCI updates
As introduced in the opening, the main update is in offering a new DCI service consumption model. This model is based on DCI API calls that can be directly embedded into the partner CI/CD pipelines.
This new model was motivated by a partner’s desire to create a hybrid CI/CD environment that allowed it to leverage the same lab resources and tests used in internal testing, with its Red Hat solution testing. The model also provides our partners with the opportunity to more seamlessly add their own functional tests into the DCI framework so that both companies can test and monitor collaboratively.
The API consumption model removes the DCI Agent deployment and management requirements, as shown in the following “DCI API Deployment Model” diagram. This eliminates the need to deploy DCI in the partner DMZ, which has typically been required in order to provide direct access to Red Hat DCI support teams to help when issues arise. It also removes the necessity of resource duplication otherwise required to implement a separate CI/CD environment.
Additional DCI features were leveraged, added, and enhanced during this project. These include using another new utility called dci-downloader, which streams bits from the DCI Control Server to the partner environment, as well as integration of that utility with the Red Hat Satellite server. This way, the partner can source Red Hat platform development releases from the same source as from where they do for general releases.
Oh, the places we can go!
Today’s Telcos depend upon multi-vendor solutions to continue rolling out competitive, cost-effective, and innovative services. With no shortage of new and evolving technologies, we as vendors can and must rise to the challenge of delivering on our ecosystem benefit claims. Red Hat’s DCI offers a way to address this in an open source and collaborative way.