Signing Images

by | Oct 6, 2022 | Hybrid Cloud, Trust

The release of Kubernetes 1.24 includes signed images, which highlights the importance of delivering secure images. Whether container images are being distributed to customers or run within your own datacenters, you must ensure that the assets within the software have not been compromised. The following steps walk through the process of building and signing an image, then enabling verification of that image within a Kubernetes cluster.

Note: Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

Why sign images?

Image signing ensures that the assets you created and packaged are from a trusted resource. This asset can then be verified via a public key. The best part? The signing framework only needs to be set up once. Once signing verification is in place, the verification can serve as a gatekeeper to ensure that only images matching the image signing key can be run.

“My image registry is private, so I don’t need this”—wrong!

Image signing is more than just ensuring you are using a resource you trust. It is only one of many safeguards that can be employed against container images. Think of image signing as yet another safeguard for the environment. Credentials can be misplaced or leaked, allowing access to the image registry. As with attacks in the past, someone could push a malicious container image into the repository, where it would then be pulled onto clusters and systems. When using image signing, an attacker would need both the private key and the credentials for the repository in order to implement such an attack.

Prerequisites

There are many tools available to sign images at creation—this blog will focus on implementing Tetkon Chains. Tetkon is a CI/CD tool that runs within a Kubernetes environment that allows for a step-by-step process to build, test and deploy assets. Tekton can be installed either via an operator from Operator Hub or by deploying the latest version using kubectl. If the Kubernetes cluster is an OpenShift cluster, the installation can be performed using the Red Hat OpenShift Pipelines operator. Once Tekton has been deployed, Tekton Chains must also be added to the cluster. This demo uses the v0.9.0 release of Tekton Chains using the kubectl command.

kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/previous/v0.9.0/release.yaml

After Tekton and Tekton Chains are installed, the next step is setting up Tekton Chains. If you are running an OpenShift cluster, it is possible to install both Tekton Chains and Tekton with the Pipelines Operator.

When setting up Tekton Chains, the cosign binary is required to create the public and private keys that will be used for signing and verification of images. Cosign, created by the Sigstore project, has the ability to sign, verify and publish artifacts (like binaries or configuration files) to OCI(Open Container Initiative) registries.

Install the binary on the workstation that will be used to interact with the Kubernetes cluster. Once installed, generate the keys.

cosign generate-key-pair k8s://tekton-chains/signing-secrets

The cosign binary generated a secret containing the public and private keypair directly on the Kubernetes cluster in the tekton-chains namespace, or when using OpenShift the openshift-pipelines namespace, then saved a public key on the workstation in a file called cosign.pub.

Creating the pipeline and secrets

For the demo, create a namespace called signing-test. Before beginning, it is important to ensure that the registry supports image signing. The registry will store both the signature for the image as well as the corresponding container image. Check the list to verify that signed images are supported.

A service account must have access to the registry credentials. Tekton Chains uses this secret to push the signature.

kubectl create ns signing-test

kubectl create sa -n signing-test pipeline-signing
export USER=myQUAY-registry-user
export PASSWORD=myQUAY-password

export EMAIL=myquayemail.com

kubectl create secret docker-registry regcred -n signing-test \
--docker-server=quay.io --docker-username=$USER \
--docker-password=$PASSWORD --docker-email=$EMAIL

kubectl patch serviceaccount pipeline-signing -n signing-test -p '{"imagePullSecrets": [{"name": "regcred"}]}'

kubectl patch serviceaccount pipeline-signing -n signing-test -p '{"secrets": [{"name": "regcred"}]}'

An example pipeline has been included in this GitHub repository. To test this out on your own, pull down the file either using git clone or curl. The only required modification for this test is to change the IMAGE variable. Also included in this repository is a task that tags the built image with the commit id from Git, then pushes the newly created image to the repository.

Note: Using Tekton and Tekton Chains does not require using the pipeline and tasks presented here. They just serve as a starting point. It is also possible to use the internal OpenShift registry.

curl -o pipeline.yaml https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/build-tag.yaml
export MYREPO=quay.io/rcook/signing
sed "s|quay.io/rcook/signing|${MYREPO}|g" pipeline.yaml
kubectl apply -f pipeline.yaml
kubectl apply -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/task/buildah.yaml
kubectl apply -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/perms/binding.yaml

At this point, the framework has been established to allow for builds of our application within our repository. When a container image is created, the image will be tagged using the commit ID. Once the build is complete, Tekton Chains will automatically sign the image and update the labels within the Tekton build. This framework can be used over and over again to generate images that have been signed.

Generating the image and signing

To test this workflow, a Tekton pipeline run has been provided. This pipeline run will clone the repository used in this blog, push the image to your repository and then Tekton Chains will sign the newly created image.

kubectl create -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/pipelinerun/image-build-demo.yaml

With the pipeline now running, there are multiple ways to follow the job’s output, such as using the Tekton UI, following the pipeline run within the OpenShift console, or monitoring the output of kubectl get taskrun.

Ensure the run has been successful.

kubectl get tr

NAME                         SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
image-build-demo-buildah     True        Succeeded   4m31s       3m36s
image-build-demo-git-clone   True        Succeeded   4m52s       4m31s

To verify that Tekton Chains has signed the image, view the labels of the image-build-demo-buildah Task Run.

kubectl describe tr image-build-demo-buildah | head -n 16
Name:         image-build-demo-buildah
Namespace:    signing-test
Labels:       app.kubernetes.io/managed-by=tekton-pipelines
              tekton.dev/memberOf=tasks
              tekton.dev/pipeline=image-build
              tekton.dev/pipelineRun=image-build-demo
              tekton.dev/pipelineTask=buildah
              tekton.dev/task=buildah-commit-tag
Annotations:  chains.tekton.dev/cert-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5: 
              chains.tekton.dev/chain-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5: 
              chains.tekton.dev/payload-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
                eyJjb25kaXRpb25zIjpbeyJ0eXBlIjoiU3VjY2VlZGVkIiwic3RhdHVzIjoiVHJ1ZSIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjItMDUtMDlUMjA6MDI6NTVaIiwicmVhc29uIj...
              chains.tekton.dev/signature-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
                MEYCIQC1YTrPasaEY7pJ/G6MaycyCEtZem6ysJlkuNIfnsS6aQIhAOpz0eJF5NBedf3xJZd44J/fTliM7G2YwLzP5SRvRUV/
              chains.tekton.dev/signed: true
              pipeline.tekton.dev/release: 6b5710c

The important item to see here is that chains.tekton.dev/signed is true. Next, within Quay, verify you can see the signature.

Verifying the signature prerequisites

For this demonstration, I chose Cosigned, which provides a lightweight webhook to verify that the image was signed with the expected key. Using the cosign.pub from earlier, we will verify the signing. The examples below were performed on a separate Kubernetes cluster, but the steps could be performed on the image build cluster if desired.

kubectl create namespace cosign-system

kubectl create secret generic pub-sign -n cosign-system --from-file=cosign.pub=./cosign.pub

kubectl create namespace cosign-system

kubectl create secret generic pub-sign -n cosign-system –from-file=cosign.pub=./cosign.pub

Cosigned is provided via a Helm chart.

helm repo add sigstore https://sigstore.github.io/helm-charts

helm repo update

helm install cosigned -n cosign-system sigstore/cosigned --devel --set cosign.secretKeyRef.name=pub-sign

If OpenShift is used, the following permissions must be added for the service accounts.

oc adm policy -n cosign-system add-scc-to-user anyuid -z cosigned-policy-webhook
oc adm policy -n cosign-system add-scc-to-user anyuid -z cosigned-webhook

Deploying our signed app

Now that Cosigned is running within the Kubernetes environment, it is time to create and label a specific namespace to verify the image has been signed and the application is deployed.

NOTE: If this is being performed on OpenShift the following command must be run. oc adm policy -n signed-deploy add-scc-to-user anyuid -z default.

kubectl create ns signed-deploy
kubectl label ns signed-deploy cosigned.sigstore.dev/include="true"

kubectl create deployment -n signed-deploy apache –image quay.io/rcook/signing:15858f9363946c4931c557c5190edb0c2590d576

Testing Cosign capabilities

Now that the signed image is running in the environment, what happens when an image is not signed? The Cosigned webhook stops the container from being deployed.

kubectl patch -n signed-deploy  deployment apache -p '{"spec":{"template":{"spec":{"containers":[{"name":"signing","image":"quay.io/rcook/signing:hackers"}]}}}}'

Error from server (BadRequest): admission webhook "cosigned.sigstore.dev" denied the request: validation failed: no matching signatures:
: spec.template.spec.containers[0].image
quay.io/rcook/signing@sha256:5451b297db3a65aba1ec227d55ec52fc5b9495453b81084df63dec8193a2dd71

Conclusion

No matter where you are running your containers, image security is incredibly important. With the help of Sigstore projects and various container registries supporting storing signatures, the process of image signing and verification is easier than ever. Keep an eye out for all of the amazing work in the community by following https://sigstore.dev. If you would like to try this out, you can run all of the steps above on MicroShift or Kind.