Deploying Crossplane Providers with the Operator Lifecycle Manager

by | May 20, 2021 | Hybrid Cloud

Crossplane is a project that strives to bring cloud infrastructure, services, and applications closer to your Kubernetes cluster in order to create a hybrid control plane. This goal is primarily achieved through the use of providers, which are standalone controllers for a specific API group. The Crossplane project itself manages the lifecycle of these providers, from installation to cleanup. In this post, we will briefly discuss what providers are, the Crossplane architecture as well as how we can repackage a provider installation via the Operator Lifecycle Manager (OLM).

We’ll show you how to repackage a provider in this post, but it’s important that we discuss the various components of a Crossplane provider first. We will also briefly talk about different ways you can install a Crossplane provider and challenges you might encounter during this process. Lastly, we’ll go over a demonstration of the workflow for repackaging and then consuming a Crossplane provider through Operator Lifecycle Manager (OLM). 

Let’s dive right into what providers are and how they work within the Crossplane architecture.

Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

Crossplane Providers

Providers aim to add additional resources and extend Crossplane to actually handle the provisioning of infrastructure resources. This is done through the use of Custom Resource Definitions (CRDs), which need to be registered in your Kubernetes cluster. Additionally, each provider contains at least one controller that watches all of the Custom Resources defined by the CRDs. A single provider may contain many CRDs and their controllers.

Within the Crossplane architecture, the main Crossplane operator is responsible for pulling, unpacking, and parsing the opinionated Open Container Image (OCI). From there, it applies the CRDs, creates the service account with appropriate Role-based access control (RBAC), and generates the Deployment for the provider. During this process, it also handles cleaning up older versions of the provider if they exist. 

How to Install Crossplane Providers

Crossplane v1.Provider Resource

One of the most popular ways to install a Crossplane provider by the end user is to create a pkg.crossplane.io/v1 Provider resource. With this resource, the end user must provide a reference to the image. For example, crossplane/provider-aws:master will install the most recent version of the provider-aws.

Internally, the OCI image contains a generated meta.pkg.crossplane.io/v1 Provider, which describes:

  • Dependencies on other providers and their specific versions.
  • Valid Crossplane versions that the provider will work with.
  • The image reference for the controller.
  • Additional RBAC requirements for this provider to function.

Crossplane v1.Configuration Resource

The second most common way to install a Crossplane provider is through the use of pkg.crossplane.io/v1 Configuration resources. Configurations can be installed the same way as providers, but the configuration exposes CompositeResourceDefinitions (XRDs) instead of a controller. The main way that users install providers through a configuration is through the use of dependencies. 

This works the same way internally, and an end user could install a catalog of XRDs, which then has dependencies on several providers. 

Challenges

Within the Crossplane package management model, there are several challenges that can occur from the cluster administrator perspective.

  1. Proxies: When your Kubernetes cluster is behind a corporate proxy, one initial roadblock is setting up the proxy for Crossplane. However, this can be done fairly easily by setting the HTTP_PROXY environment variable. The user can also set this environment variable through the use of a ControllerConfig resource for providers. Alternatively, some organizations opt to use an internal registry, and manually mirror or sync versions from a public registry such as Docker.
  1. Custom Authorities (CAs): Another challenge can occur if your private registry makes use of a Custom Authority root certificate. There is currently no method within Crossplane to make use of client certificates natively. 
  1. Sharing credentials: Crossplane allows you to supply secrets that can be used to pull images from private repositories. For Kubernetes images in general, some cluster admins opt to configure credentials on the node instead of utilizing secrets. This would mean that, within your cluster, you do not need to specify the pullSecret for every deployment. It’s possible that credentials will become stale or inconsistent over time.

We can get around many of these issues by utilizing the OLM to deploy our providers. Let’s get into what that process looks like. 

Demo

To begin with, we need to examine the key components of an OLM operator and the key steps in the Crossplane workflow, we can then find the delta and work backward. 

OLM Structure

For OLM we can take a look at the Memcached Operator. Here, we can see there are a few main folders:

  • api: This folder contains the API definitions for the Memcached Operator
  • controllers: This folder contains the logic and tests for the controllers
  • config: This folder contains the manifests for the deployment of the operator and the related RBAC. Additionally, the generated CRDs are created here.

Provider Structure

For the provider-aws, we will notice the structure is somewhat similar:

  • The apis folder is analogous to the api folder in Memcached Operator. However, this difference can be handled with command-line flags.
  • The pkg folder contains the controllers and clients used by the provider, and this is similar to the controller folder.

The Delta

One notable difference in the provider is the lack of a clear Dockerfile for the provider. Similarly, it is also unclear where the different manifests for deployment and RBAC are. With some further investigation, we can find the Dockerfiles for the provider-aws-controller and the provider-aws under cluster/images. As previously mentioned, the provider-aws image contains a package.yaml file which has a reference to the controller, as well as some metadata about the provider.

Nonetheless, we still do not have any deployment or RBAC-related resources. If we move over to take a look at the Crossplane Operator, we can see that there are several controllers that monitor the v1.ProviderRevision resource, which is created when Crossplane sees that a new Provider was created. These controllers are responsible for creating the Deployment, ClusterRole, and ClusterRoleBinding resources. Essentially, Crossplane opts to define these various resources at runtime as opposed to build time. 

We have established what we are missing with providers, and now we can start filling in this delta. The resources required for this can be found in the olm-repackage repository. Namely:

  • gen_rbac.sh: This script generates a rbac.go file, which contains kubebuilder annotations for generating the RBAC resources for the provider. With Crossplane, this responsibility is shifted to the installing Operator, but for OLM we need to generate these at build time.
  • gen_project.sh: This script is similar to the gen_rbac, however, it instead generates our top-level PROJECT file. This file contains metadata about the provider.
  • PROJECT.boilerplate.txt: This is used by the gen_project script for the initial boilerplate of the PROJECT file.
  • Dockerfile: We use a custom, minimal Dockerfile to build and run the provider. 
  • Makefile: We also utilize a minimal Makefile to handle the code-generation, build, and deployment steps. Testing steps from the Makefile have been removed.
  • config/: this directory contains all the required manifests for deploying the provider and setting up the RBAC. This is done through a series of kustomize configurations.

Repackaging

To go through the steps of the repackaging process, we have developed a GitHub Action script, which can be found in the provider-ci repository. This sample action accepts three inputs:

  • The GitHub repository for the provider, e.g., crossplane/provider-aws
  • The commit reference targeted for repackaging, e.g., v0.17.0
  • The quay.io user, which is the destination for the Docker images, e.g., krishchow

Then, we take the following steps:

  1. Checkout: We first checkout the target provider Github repository. 
  2. Setup: We then want to set up all of our dependencies, namely:
    1. Go (v1.15)
    2. Other dependencies (yq and operator-sdk)
    3. Logging into quay.io with our service account.
  3. Pull olm-repackage: We will clone, and then copy all of the previously mentioned files into the top-level of our repository.
  4. Build Operator: Then, all we need to do is run the appropriate steps in our Makefile to build the Operator. Additionally, we need to supply the correct environment variables. During this process we run the code generation, build the Docker image and then push the image. 
  5. Build Bundle: The final step involves building the bundle for our provider. During this step we:
    1. Generate our PROJECT file
    2. Generate our RBAC manifests
    3. Template values into our manifests
    4. Create our ClusterServiceVersion
    5. Build and push the Bundle image

Running the Operator

The easiest way to get started with running the operator is through the Operator-SDK CLI tool. Additionally, if you are not using an OpenShift cluster, then you will need to install the Operator Lifecycle Manager

If both of these steps are met, then you can simply run: operator-sdk run bundle $BUNDLE_IMG. This will start the process of spinning up the operator, installing the CRDs and setting up all the RBAC resources.

Conclusion

Deploying Crossplane providers using the OLM allows cluster administrators another avenue to manage providers. OLM can assist with each step of the process, from deployment to day two operations. The repackaging process is seamless, with zero changes to the code base, while maintaining feature parity with the de facto method for installing providers through the Crossplane operator.

Future work may focus on repackaging multiple versions of existing providers into bundles that can be separately deployed into a Kubernetes cluster. This can provide an opportunity to utilize provider resources separately from the main Crossplane operator. OLM repackaging can also be integrated into the existing deployment pipeline for providers, representing another target for images. An example of this process can be seen in the aforementioned provider-ci repository, and minor changes could integrate this action into the workflow for an arbitrary provider. 

I would also encourage you to examine the README in the olm-repackage repository for more detailed technical information. Feel free to open up an issue with any questions, or bugs you run into, also feel free to reach out on the Crossplane slack for any general discussion.

For even more Crossplane demos, you can check out previous posts, in which we have explored Crossplane for provisioning, managing, configuring, and consuming cloud services to deploy an instance of Red Hat Quay from scratch. We also recently discussed using Crossplane to manage and deploy Operators from OperatorHub.io via the OLM.