In a recent post we walked through using Tekton and Cosign to build, sign and verify container images within Kubernetes. Red Hat utilizes OpenShift, signing and scanning in its own build systems to deliver release artifacts. Red Hat’s secure software delivery is evident with Red Hat’s Certified Container Images registry, where various security-related artifacts are publicly available for all container images. Red Hat was founded upon the principle of delivering trusted open source content. This post describes how you can fortify your own build pipelines, and also how to consume software more responsibly. It will demonstrate how to incorporate signing, Software Bill of Materials, scanning and attestations using the open source tools Podman, Cosign, Syft and Grype.
Note: Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.
Universal Base Images
A prerequisite to securing software pipelines is to build upon trusted content. When the industry shifted to delivering containerized applications, Red Hat began providing minimal images from trusted sources run with a non-root user. These security features are an essential first step when producing trusted software. Take for example Universal Base Images based on RHEL 9 (UBI9). Varying use-cases demand a broad set of base images, such as UBI9, UBI9-minimal and UBI9-micro. These include a base set of trusted packages, from the same pipeline as RHEL9, with which users can build their applications. The registry console includes a list of packages along with versions within an image, its signature, the results of vulnerability scanning via Clair, the image version, its Dockerfile, how to download the image and other technical information. The following screenshot shows the security profile of a UBI9 image from the registry console. More information can be found here regarding UBI’s risk based approach to vulnerability management.
UBI base images lend toward multi-stage builds. With multi-stage builds, resulting images include only what they need to execute. The reasoning is that the fewer packages included in an image the smaller its attack surface. UBI-micro base images are the most minimal since there’s no package manager included (in order to add packages you have to rely on Buildah and the package manager on the container host). UBI-micro images are also referred to as distroless container images. Consider including Red Hat’s UBI images in your software build systems. These minimal, certified and trusted base images provide the groundwork for a secure software pipeline.
What’s a Software Bill of Materials (SBOM)?
The specifics of what constitutes a Software Bill of Materials (SBOM) are currently being determined and will eventually be standardized. How can you build and consume images with risk management top-of-mind today? If you do not currently include a manifest of components with your images, consider adding it to your build process. This allows you to build up a process to work from while the community works out what exactly is a Software Bill of Materials. Generally “SBOM” describes the packages and components along with versions included in a piece of software. There are several open source tools that generate such manifests. Syft is used in the following example. Syft works seamlessly with Sigstore/Cosign and the container registry Quay.io.
SBOM doesn’t hold much value without an attestation or assurance that the code you received is the code that was released. An attestation is cryptographically-signed metadata used to verify the integrity of an event or artifact, known as a predicate. In this case, the SBOM is the predicate, and the attestation is the metadata that verifies the code within an SBOM. An attestation with an SBOM should be generated as part of the build process, to assure that the SBOM hasn’t been tampered with before being attached to an image. In the following example, an in-toto attestation is generated along with an SBOM. In-toto is an open metadata standard and a framework to verify the legitimacy and the integrity of software supply chains. Fortunately, tools like Podman, Cosign, Syft, Grype and Quay.io make it easy to attach SBOMs and attestations to your images.
What are you waiting for? Let’s try it out!
Red Hat is partnering with others in the SBOM ecosystem across industries to shape standards for delivering SBOMs at scale. Tools and standards are evolving. Outlined here are actions to incorporate into your software pipelines today to improve the security and integrity of the software you produce and consume.
Prerequisites
Install Podman, Cosign and Syft, and create an account with Quay.io.
The quay.io/sallyom/hello-go:test image was built and pushed using podman. Notice the Dockerfile specifies a multi-stage build with the result based on UBI9-micro and the unprivileged user. We’ll use Cosign to generate a Sigstore ECDSA-P256 image signing key pair. For details on Sigstore’s key pair format, read here.
$ cosign generate-key-pair # add Password if desired Private key written to cosign.key Public key written to cosign.pub
Sign with Sigstore key pair
Sign an image. Cosign is used here, but Podman can also sign with a Sigstore/Cosign key-pair. Podman recently added podman push –sign-with-sigstore-private-key. As of Podman version 4.2, users can opt-in to sign, attach and push Sigstore signatures.
$ cosign sign --key ~/cosign.key quay.io/sallyom/hello-go:test Enter password for private key: Pushing signature to: quay.io/sallyom/hello-go
Generate SBOM
Create a Software Bill of Materials for an image. The SBOM, as defined by Syft, can be viewed here.
$ syft -o spdx podman:quay.io/sallyom/hello-go:test > hello-go.sbom ✔ Loaded image ✔ Parsed image ✔ Cataloged packages [23 packages]
Sign, attach and attest SBOM
Sign and attach an attestation for the SBOM predicate. With this, the SBOM is signed (and therefore tamper-proof) as it is within an attestation, and consumers can validate its authenticity. This issue describes the difference between the attach and attest commands. The –verbose flag added to Cosign commands reveals more detailed information than the output given here.
$ cosign sign --key ~/cosign.key --attachment sbom quay.io/sallyom/hello-go:test Enter password for private key: Pushing signature to: quay.io/sallyom/hello-go $ cosign attach sbom --sbom hello-go.sbom quay.io/sallyom/hello-go:test Uploading SBOM file for [quay.io/sallyom/hello-go:test] to [quay.io/sallyom/hello-go:sha256-a66f3923d0ea7890bce8586690f1fbfbcf3ae1d0935acc372299b29c99f92e50.sbom] with mediaType [text/spdx]. $ cosign attest --key ~/cosign.key --predicate hello-go.sbom quay.io/sallyom/hello-go:test Enter password for private key: Using payload from: hello-go.sbom
Attaching signatures, SBOMs, and attestations to images is as simple as a few commands. As a consumer of software, Podman, Cosign and Quay can ensure you are running exactly what you have intended. Here are some suggestions to ensure the integrity and security of your containerized workloads.
Verify signed images before downloading
With Podman, it’s possible to set a policy such that images can’t be pulled if they are not signed, and can also specify they must be signed with Sigstore. Add the following to Podman’s policy.json file
# for root podman the file is /etc/containers/policy.json
$ cat ~/.config/containers/policy.json
{
---
"transports": {
"docker": {
"quay.io/sallyom": [
{
"type": "sigstoreSigned",
"keyPath": "/home/somalley/cosign.pub",
"signedIdentity": {"type": "matchRepository"}
}
],
With this in place, any quay.io/sallyom/image pull will fail if there is no signature attached.
$ podman pull quay.io/sallyom/cli:test Trying to pull quay.io/sallyom/cli:test... Error: Source image rejected: A signature was required, but no signature exists $ podman pull quay.io/sallyom/hello-go:test Trying to pull quay.io/sallyom/hello-go:test... Getting image source signatures Checking if image destination supports signatures ----- Copying config 93e70746a6 done Writing manifest to image destination Storing signatures 93e70746a69c780df7a1961487c7b77726eacadd3f70f6f960d2320580968eab
Verify signed SBOM for image
$ cosign verify --key ~/cosign.pub --attachment sbom quay.io/sallyom/hello-go:test Verification for quay.io/sallyom/hello-go:sha256-a66f3923d0ea7890bce8586690f1fbfbcf3ae1d0935acc372299b29c99f92e50.sbom -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key
Verify attestation for SBOM authenticity
$ cosign verify-attestation --key ~/cosign.pub quay.io/sallyom/hello-go:test > /tmp/att Verification for quay.io/sallyom/hello-go:test -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key
Scan image for vulnerabilities
Grype is a tool to scan images for known vulnerabilities. The tool can be configured to add data sources, although it has reasonable defaults for where it obtains its data. You do not need to install Grype as you can run the Grype image with Podman. If you do not have the Podman socket running (the service is installed with Podman but not enabled), you can start it with:
$ systemctl --user start podman.socket
$ podman run --rm --volume /run/user/1000/podman/podman.sock:/var/run/docker.sock --name Grype anchore/grype:latest quay.io/sallyom/hello-go:test NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY libgcc 11.2.1-9.4.el9 rpm CVE-2021-46195 Low libgcc 11.2.1-9.4.el9 (won't fix) rpm CVE-2022-27943 Low ncurses-base 6.2-8.20210508.el9 (won't fix) rpm CVE-2022-29458 Low ncurses-libs 6.2-8.20210508.el9 (won't fix) rpm CVE-2022-29458 Low
Lastly, Cosign has an all-in-one command to list known security related artifacts of an image. The command is cosign tree.
$ cosign tree quay.io/sallyom/hello-go:test 📦 Supply Chain Security Related artifacts for an image: quay.io/sallyom/hello-go:test └── 💾 Attestations for an image tag: quay.io/sallyom/hello-go:sha256-a66f3923d0ea7890bce8586690f1fbfbcf3ae1d0935acc372299b29c99f92e50.att ├── 🍒 sha256:06a8bd80b513e9f98009cddd4856f9e54c22cba3df2a9714b457e8c10b1c0b5c ├── 🍒 sha256:06a8bd80b513e9f98009cddd4856f9e54c22cba3df2a9714b457e8c10b1c0b5c └── 🍒 sha256:22dfc2808a9008d76c79ef0b80cc06a569c367fe880fbf155600385125e18b4a └── 🔐 Signatures for an image tag: quay.io/sallyom/hello-go:sha256-a66f3923d0ea7890bce8586690f1fbfbcf3ae1d0935acc372299b29c99f92e50.sig └── 🍒 sha256:94683d70b9542b4e5a1d9be2e27923f9f37e3a7fc37c3900b0ff90a6be3c8e9c └── 📦 SBOMs for an image tag: quay.io/sallyom/hello-go:sha256-a66f3923d0ea7890bce8586690f1fbfbcf3ae1d0935acc372299b29c99f92e50.sbom └── 🍒 sha256:87ee673638aaf1c194241b65794a508fa02f93503e1113d4e6eb8278dd28bfdf
In closing
Core aspects of our society depend on open source and community-driven code. Enhancing the security of software supply chains is essential and a responsibility of every open source software citizen. As supply chains become more complex and interwoven, inclusion of signing, Software Bill of Materials (SBOMs) and attestations must become standard. The tools we walked through in this post help to achieve these goals. Give them a try and let us know what you think!