The release of Kubernetes 1.24 includes signed images, which highlights the importance of delivering secure images. Whether container images are being distributed to customers or run within your own datacenters, you must ensure that the assets within the software have not been compromised. The following steps walk through the process of building and signing an image, then enabling verification of that image within a Kubernetes cluster.
Note: Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.
Why sign images?
Image signing ensures that the assets you created and packaged are from a trusted resource. This asset can then be verified via a public key. The best part? The signing framework only needs to be set up once. Once signing verification is in place, the verification can serve as a gatekeeper to ensure that only images matching the image signing key can be run.
“My image registry is private, so I don’t need this”—wrong!
Image signing is more than just ensuring you are using a resource you trust. It is only one of many safeguards that can be employed against container images. Think of image signing as yet another safeguard for the environment. Credentials can be misplaced or leaked, allowing access to the image registry. As with attacks in the past, someone could push a malicious container image into the repository, where it would then be pulled onto clusters and systems. When using image signing, an attacker would need both the private key and the credentials for the repository in order to implement such an attack.
Prerequisites
There are many tools available to sign images at creation—this blog will focus on implementing Tetkon Chains. Tetkon is a CI/CD tool that runs within a Kubernetes environment that allows for a step-by-step process to build, test and deploy assets. Tekton can be installed either via an operator from Operator Hub or by deploying the latest version using kubectl. If the Kubernetes cluster is an OpenShift cluster, the installation can be performed using the Red Hat OpenShift Pipelines operator. Once Tekton has been deployed, Tekton Chains must also be added to the cluster. This demo uses the v0.9.0 release of Tekton Chains using the kubectl command.
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/previous/v0.9.0/release.yaml
After Tekton and Tekton Chains are installed, the next step is setting up Tekton Chains. If you are running an OpenShift cluster, it is possible to install both Tekton Chains and Tekton with the Pipelines Operator.
When setting up Tekton Chains, the cosign binary is required to create the public and private keys that will be used for signing and verification of images. Cosign, created by the Sigstore project, has the ability to sign, verify and publish artifacts (like binaries or configuration files) to OCI(Open Container Initiative) registries.
Install the binary on the workstation that will be used to interact with the Kubernetes cluster. Once installed, generate the keys.
cosign generate-key-pair k8s://tekton-chains/signing-secrets
The cosign binary generated a secret containing the public and private keypair directly on the Kubernetes cluster in the tekton-chains namespace, or when using OpenShift the openshift-pipelines namespace, then saved a public key on the workstation in a file called cosign.pub.
Creating the pipeline and secrets
For the demo, create a namespace called signing-test. Before beginning, it is important to ensure that the registry supports image signing. The registry will store both the signature for the image as well as the corresponding container image. Check the list to verify that signed images are supported.
A service account must have access to the registry credentials. Tekton Chains uses this secret to push the signature.
kubectl create ns signing-test
kubectl create sa -n signing-test pipeline-signing
export USER=myQUAY-registry-user
export PASSWORD=myQUAY-password
export EMAIL=myquayemail.com
kubectl create secret docker-registry regcred -n signing-test \
--docker-server=quay.io --docker-username=$USER \
--docker-password=$PASSWORD --docker-email=$EMAIL
kubectl patch serviceaccount pipeline-signing -n signing-test -p '{"imagePullSecrets": [{"name": "regcred"}]}'
kubectl patch serviceaccount pipeline-signing -n signing-test -p '{"secrets": [{"name": "regcred"}]}'
An example pipeline has been included in this GitHub repository. To test this out on your own, pull down the file either using git clone or curl. The only required modification for this test is to change the IMAGE variable. Also included in this repository is a task that tags the built image with the commit id from Git, then pushes the newly created image to the repository.
Note: Using Tekton and Tekton Chains does not require using the pipeline and tasks presented here. They just serve as a starting point. It is also possible to use the internal OpenShift registry.
curl -o pipeline.yaml https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/build-tag.yaml
export MYREPO=quay.io/rcook/signing
sed "s|quay.io/rcook/signing|${MYREPO}|g" pipeline.yaml
kubectl apply -f pipeline.yaml
kubectl apply -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/task/buildah.yaml
kubectl apply -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/perms/binding.yaml
At this point, the framework has been established to allow for builds of our application within our repository. When a container image is created, the image will be tagged using the commit ID. Once the build is complete, Tekton Chains will automatically sign the image and update the labels within the Tekton build. This framework can be used over and over again to generate images that have been signed.
Generating the image and signing
To test this workflow, a Tekton pipeline run has been provided. This pipeline run will clone the repository used in this blog, push the image to your repository and then Tekton Chains will sign the newly created image.
kubectl create -f https://raw.githubusercontent.com/cooktheryan/signing-blog/main/pipeline/pipelinerun/image-build-demo.yaml
With the pipeline now running, there are multiple ways to follow the job’s output, such as using the Tekton UI, following the pipeline run within the OpenShift console, or monitoring the output of kubectl get taskrun.
Ensure the run has been successful.
kubectl get tr NAME SUCCEEDED REASON STARTTIME COMPLETIONTIME image-build-demo-buildah True Succeeded 4m31s 3m36s image-build-demo-git-clone True Succeeded 4m52s 4m31s
To verify that Tekton Chains has signed the image, view the labels of the image-build-demo-buildah Task Run.
kubectl describe tr image-build-demo-buildah | head -n 16
Name: image-build-demo-buildah
Namespace: signing-test
Labels: app.kubernetes.io/managed-by=tekton-pipelines
tekton.dev/memberOf=tasks
tekton.dev/pipeline=image-build
tekton.dev/pipelineRun=image-build-demo
tekton.dev/pipelineTask=buildah
tekton.dev/task=buildah-commit-tag
Annotations: chains.tekton.dev/cert-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
chains.tekton.dev/chain-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
chains.tekton.dev/payload-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
eyJjb25kaXRpb25zIjpbeyJ0eXBlIjoiU3VjY2VlZGVkIiwic3RhdHVzIjoiVHJ1ZSIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjItMDUtMDlUMjA6MDI6NTVaIiwicmVhc29uIj...
chains.tekton.dev/signature-taskrun-c03e237e-f512-4ade-9dbe-33a23d9667c5:
MEYCIQC1YTrPasaEY7pJ/G6MaycyCEtZem6ysJlkuNIfnsS6aQIhAOpz0eJF5NBedf3xJZd44J/fTliM7G2YwLzP5SRvRUV/
chains.tekton.dev/signed: true
pipeline.tekton.dev/release: 6b5710c
The important item to see here is that chains.tekton.dev/signed is true. Next, within Quay, verify you can see the signature.
Verifying the signature prerequisites
For this demonstration, I chose Cosigned, which provides a lightweight webhook to verify that the image was signed with the expected key. Using the cosign.pub from earlier, we will verify the signing. The examples below were performed on a separate Kubernetes cluster, but the steps could be performed on the image build cluster if desired.
kubectl create namespace cosign-system kubectl create secret generic pub-sign -n cosign-system --from-file=cosign.pub=./cosign.pub
kubectl create namespace cosign-system
kubectl create secret generic pub-sign -n cosign-system –from-file=cosign.pub=./cosign.pub
Cosigned is provided via a Helm chart.
helm repo add sigstore https://sigstore.github.io/helm-charts helm repo update helm install cosigned -n cosign-system sigstore/cosigned --devel --set cosign.secretKeyRef.name=pub-sign
If OpenShift is used, the following permissions must be added for the service accounts.
oc adm policy -n cosign-system add-scc-to-user anyuid -z cosigned-policy-webhook oc adm policy -n cosign-system add-scc-to-user anyuid -z cosigned-webhook
Deploying our signed app
Now that Cosigned is running within the Kubernetes environment, it is time to create and label a specific namespace to verify the image has been signed and the application is deployed.
NOTE: If this is being performed on OpenShift the following command must be run. oc adm policy -n signed-deploy add-scc-to-user anyuid -z default.
kubectl create ns signed-deploy kubectl label ns signed-deploy cosigned.sigstore.dev/include="true" kubectl create deployment -n signed-deploy apache –image quay.io/rcook/signing:15858f9363946c4931c557c5190edb0c2590d576
Testing Cosign capabilities
Now that the signed image is running in the environment, what happens when an image is not signed? The Cosigned webhook stops the container from being deployed.
kubectl patch -n signed-deploy deployment apache -p '{"spec":{"template":{"spec":{"containers":[{"name":"signing","image":"quay.io/rcook/signing:hackers"}]}}}}'
Error from server (BadRequest): admission webhook "cosigned.sigstore.dev" denied the request: validation failed: no matching signatures:
: spec.template.spec.containers[0].image
quay.io/rcook/signing@sha256:5451b297db3a65aba1ec227d55ec52fc5b9495453b81084df63dec8193a2dd71
Conclusion
No matter where you are running your containers, image security is incredibly important. With the help of Sigstore projects and various container registries supporting storing signatures, the process of image signing and verification is easier than ever. Keep an eye out for all of the amazing work in the community by following https://sigstore.dev. If you would like to try this out, you can run all of the steps above on MicroShift or Kind.