Crossplane as an abstraction platform to manage and deploy service Operators

by | Mar 31, 2021 | Hybrid Cloud

How do you manage and deploy Kubernetes Operators in the open hybrid cloud to take advantage of multiple clouds? One approach is Crossplane, a Kubernetes add-on that extends any cluster with the ability to provision and manage cloud infrastructure, services, and applications. In this post, learn how the new in-cluster provider enables the use of any Operator Lifecycle Manager (OLM) Operator.

Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

In a previous post, we explored Crossplane as an alternative for the Open Service Broker API, which focused on creating a custom catalog of Kubernetes resources that reflected a set of cloud resources. Similarly, we also utilized Crossplane for provisioning, managing, configuring, and consuming cloud services to deploy an instance of Red Hat Quay from scratch. 

To understand the new in-cluster provider for Crossplane, we’ll look at what goes into creating a new Crossplane provider. In addition, we’ll explore the new Crossplane packages, which are opinionated Open Container Initiative (OCI) images acting as a new abstraction over the CompositeResourceDefinitions (XRDs). This all comes together at the end of this post in a demo utilizing a Crossplane package, in-cluster provider, AWS provider, and Helm provider.

Red Hat’s now+Next blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

In-cluster provider

In understanding why the in-cluster provider is needed, we need to examine the current landscape of the providers in the Crossplane ecosystem. The provider extends Crossplane to enable provisioning infrastructure resources. Providers achieve this by creating Custom Resource Definitions (CRDs) that map 1-to-1 to external infrastructure resources. These Crossplane providers also manage the on-going life cycle of resources, which differentiates them from Terraform providers since those cannot dynamically react to failures or make automatic changes.

Crossplane providers offer an opinionated solution to provision infrastructure using Kubernetes resources. The Crossplane community offers several key base providers, such as AWS, GCP, and Azure, but there has also been a significant uptick in interest from other public cloud providers. Some key examples include:

  • Provider IBM: A new addition to the Crossplane Ecosystem, Provider IBM exposes more than 85 IBM Cloud resources. The project was bootstrapped by Provider Template.
  • Provider Alibaba: Provider Alibaba was created by the Crossplane team and engineers from Alibaba.
  • Provider AWS: Although Provider AWS was primarily created and driven by the community, code generation from the ACK project was recently used to accelerate the development of resources for the provider.

The design of Crossplane providers makes them flexible enough to describe and configure application resources, not just infrastructure. Some notable examples of providers that are more focused on configuring services and application as opposed to infrastructure are:

  • Provider SQL: Orchestrates MySQL and PostgreSQL servers by creating databases, users, access grants, etc.
  • Provider Helm: Manage and deploy Helm chart releases.
  • Provider Rook: Reflects storage services exposed by Rook such as CockroachDB and YugabyteDB.

Why do we need the provider in-cluster?

One critical piece of functionality currently missing from Crossplane is the ability to natively provision common services within your Kubernetes cluster. This functionality would allow us to create interchangeable compositions for testing, QA, and deployment purposes since we adhere to the standard interface. 

The in-cluster provider is flexible enough to support non-infrastructure resources as well. There is currently support for provisioning Operators created with Operator SDK, which are exposed through OLM Catalogs. This allows us to create additional arbitrary Operators that can be used to create other resources in the Kubernetes cluster. There are tradeoffs for using the provider in-cluster—we will not be able to replicate all cloud resources in-cluster. The main limitations are proprietary cloud resources, for example:

  • Amazon Simple Storage Service (Amazon S3)/Azure Blob Store/GCS.
  • AWS SNS.
  • AWS DynamoDB.

Creating a custom Crossplane provider

There are several reasons we may want to create a custom Crossplane provider. We may want a custom `Update`, `Create`, or `Delete` logic for infrastructure resources or configuration, or we may need to call external services through network calls. Custom providers expose CRDs that represent some other resource, these external resources can exist on a local network, in the cluster, or on a public cloud.

How do you create your own provider?

The easiest way to create your own provider is to bootstrap development using the provider-template. This template has the basic structure of a provider, and the key directories are:

  1. `apis/`
    1. The `template.go` file lists all GroupVersionKinds under each of the `api` folders. By registering the types with the Scheme, we enable the components to map to GroupVersionKinds and vice versa.
    2. All subfolders here represent separate groups of resources, with the `v1alpha1` folder including the ProviderConfig, which is required for each provider.
      1. The resource folders are organized by version (e.g., `v1alpha1`, `v1beta1`, etc.). 
      2. The `register.go` file defines the Kind, GroupKind, KindAPIVersion, and GroupVersionKind.
      3. The `doc.go` file defines the kuberbuilder parameters for the versionName and the package.
      4. `types.go` is the root for all the API types exposed in this GroupVersionKind.
      5. Lastly, all files starting with `zz_` are generated files.
      6. Optionally, you may utilize references, which are conventionally defined in the `referencer.go` file. This file usually exposes a public method on the API struct called `ResolveReferences`. 
  2. `pkg/`
    1. The `controller/` subfolder typically defines the controllers for all of the different API resources. 
      1. The controllers defined under the subfolders contain the reconciler, this is typically done using a connector struct and an external struct. The connector is responsible for the logic related to initialization for any clients related to the resource. The external struct defines Observe, Create, Delete, and Update functions, which are called at different stages of the reconciler.
    2. We often define a `client/` subfolder to wrap methods on a client library. This allows us to define a separate interface for testing purposes.
  3. `examples/`
    1. The `examples` subdirectory hosts example manifests for the different resources exposed by the provider. 

Demo

Components

In-cluster provider/Quay Operator

The in-cluster provider allows us to create instances of Operators in our Kubernetes cluster. Namely, we use the in-cluster provider to run a version of the Quay Operator, and we achieve that by first creating a CatalogSource, which uses an opinionated OSI image as the source. 

After creating our Quay Operator, we have the ability to create the QuayEcosystem resource. The QuayEcosystem expects credentials for Amazon S3, Postgres, and Redis. 

AWS Provider

The AWS Provider is used to create the previously mentioned dependencies for Quay using Amazon RDS, Amazon ElastiCache, and Amazon S3. This requires Amazon RDS, Amazon ElastiCache and our Kubernetes cluster to all be in the same Virtual Private Cloud (VPC). We also create various other AWS resources in our compositions:

  • Amazon S3: IAM User, IAM Access Key, Bucket Policy, Bucket
  • Amazon ElastiCache: Cache Subnet Group
  • Amazon RDS: DB Subnet Group
  • Networking: Subnets (Multi-AZ), Route Table, Security Group

The Amazon S3, Amazon ElastiCache, and Amazon RDS compositions all output secrets with connection details for the Quay composition. 

Helm Provider/Helm Chart

The Helm Provider is a powerful tool, which allows you to run arbitrary Helm charts in your Kubernetes cluster. More specifically, we run the quay-chart, which is deployed on Github pages. This chart is responsible for: 

  1. Running a pre-install hook to block the creation of all other resources until the dependencies are complete.
  2. Running a job to prepare the Postgres database.
  3. Creating the QuayEcosystem resource for the Quay Operator.

Crossplane Package

The Crossplane Package is the final component of the demo, as it allows us to package the compositions into one OCI image that can be installed through the Crossplane CLI. We expose five compositions:

  1. Bucket.
  2. RedisCluster.
  3. PostgreSQLInstance.
  4. NetworkGroup.
  5. Component.

The first four should be self-explanatory, but Component composition embeds the other compositions, and it also creates the Quay Operator and the Helm Chart. 

Running the demo

The demo can be found in the repository here. The process for running the demo is divided into six steps:

  1. Prepare credentials for an AWS account and a Kubeconfig for a Kubernetes Cluster hosted on AWS. These can be placed in the `root` directory of the repository. 
  2. You can then run make crossplane, which installs the Crossplane Operator into your Kubernetes cluster.  
  3. Next, run make provider, which installs the Helm, AWS, and in-cluster providers. It also creates ProviderConfigs for each of the providers. This allows AWS to authenticate for the various resources it will create, and the Helm and in-cluster provider will utilize the Kubeconfig to authorize the Kubernetes cluster.
  4. Run make configuration to install the Crossplane Package. 
  5. Next, run make catalog to create the custom CatalogSource, along with the pull secret for the Quay image.
  6. Last, you can configure the parameters in the Kubernetes manifest under `manifests/requirements.yaml`. You will need to update the `vpcId` and `igwId` fields to reflect the state of your cluster. You may need to update the subnets CIDR blocks or the region. You can then run make quay. This will start the process of spinning up the dependencies, the Operator, and then the QuayEcosystem resource. You can watch everything spin up by running `make watch`.
  7. Everything will be provisioned after about 10 minutes, and you can access the Quay registry by going to the `Route` exposed by the Quay Operator.
  8. You can clean up all the resources simply by deleting the requirement, or running `make clean`.

Crossplane can be used for powerful abstractions

In conclusion, Crossplane and the in-cluster provider can be used to create powerful abstractions to wrap and dynamically create operators in local and remote clusters. Take a look at the GitHub repository for this demo and give it a try. Feel free to open an issue or a pull request if you run into any problems. If you want to talk about the Crossplane project, or get started with creating your own provider, join the Crossplane Slack and check out the provider-template.

view posts