Establishing a Secure Pipeline

by | Oct 27, 2022 | Containers, Hybrid Cloud, Security

In a recent post we walked through using Tekton and Cosign to build, sign and verify container images within Kubernetes. Red Hat utilizes OpenShift, signing and scanning in its own build systems to deliver release artifacts. Red Hat’s secure software delivery is evident with Red Hat’s Certified Container Images registry, where various security-related artifacts are publicly available for all container images. Red Hat was founded upon the principle of delivering trusted open source content. This post describes how you can fortify your own build pipelines, and also how to consume software more responsibly. It will demonstrate how to incorporate signing, Software Bill of Materials, scanning and attestations using the open source tools Podman, Cosign, Syft and Grype.

Note: Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.

Universal Base Images

A prerequisite to securing software pipelines is to build upon trusted content. When the industry shifted to delivering containerized applications, Red Hat began providing minimal images from trusted sources run with a non-root user. These security features are an essential first step when producing trusted software. Take for example Universal Base Images based on RHEL 9 (UBI9). Varying use-cases demand a broad set of base images, such as UBI9, UBI9-minimal and UBI9-micro. These include a base set of trusted packages, from the same pipeline as RHEL9, with which users can build their applications. The registry console includes a list of packages along with versions within an image, its signature, the results of vulnerability scanning via Clair, the image version, its Dockerfile, how to download the image and other technical information. The following screenshot shows the security profile of a UBI9 image from the registry console. More information can be found here regarding UBI’s risk based approach to vulnerability management.

Red Hat catalog page entry for UBI 9 minimal - the foundational container image for a secure build pipeline

UBI base images lend toward multi-stage builds. With multi-stage builds, resulting images include only what they need to execute. The reasoning is that the fewer packages included in an image the smaller its attack surface. UBI-micro base images are the most minimal since there’s no package manager included (in order to add packages you have to rely on Buildah and the package manager on the container host). UBI-micro images are also referred to as distroless container images. Consider including Red Hat’s UBI images in your software build systems. These minimal, certified and trusted base images provide the groundwork for a secure software pipeline.

What’s a Software Bill of Materials (SBOM)?

The specifics of what constitutes a Software Bill of Materials (SBOM) are currently being determined and will eventually be standardized. How can you build and consume images with risk management top-of-mind today? If you do not currently include a manifest of components with your images, consider adding it to your build process. This allows you to build up a process to work from while the community works out what exactly is a Software Bill of Materials. Generally “SBOM” describes the packages and components along with versions included in a piece of software. There are several open source tools that generate such manifests. Syft is used in the following example. Syft works seamlessly with Sigstore/Cosign and the container registry

SBOM doesn’t hold much value without an attestation or assurance that the code you received is the code that was released. An attestation is cryptographically-signed metadata used to verify the integrity of an event or artifact, known as a predicate. In this case, the SBOM is the predicate, and the attestation is the metadata that verifies the code within an SBOM. An attestation with an SBOM should be generated as part of the build process, to assure that the SBOM hasn’t been tampered with before being attached to an image. In the following example, an in-toto attestation is generated along with an SBOM. In-toto is an open metadata standard and a framework to verify the legitimacy and the integrity of software supply chains. Fortunately, tools like Podman, Cosign, Syft, Grype and make it easy to attach SBOMs and attestations to your images.

What are you waiting for? Let’s try it out!

Red Hat is partnering with others in the SBOM ecosystem across industries to shape standards for delivering SBOMs at scale. Tools and standards are evolving. Outlined here are actions to incorporate into your software pipelines today to improve the security and integrity of the software you produce and consume.


Install Podman, Cosign and Syft, and create an account with

The image was built and pushed using podman. Notice the Dockerfile specifies a multi-stage build with the result based on UBI9-micro and the unprivileged user. We’ll use Cosign to generate a Sigstore ECDSA-P256 image signing key pair. For details on Sigstore’s key pair format, read here.

$ cosign generate-key-pair
# add Password if desired
Private key written to cosign.key
Public key written to

Sign with Sigstore key pair

Sign an image. Cosign is used here, but Podman can also sign with a Sigstore/Cosign key-pair. Podman recently added podman push –sign-with-sigstore-private-key. As of Podman version 4.2, users can opt-in to sign, attach and push Sigstore signatures.

$ cosign sign --key ~/cosign.key
Enter password for private key:
Pushing signature to:

Generate SBOM

Create a Software Bill of Materials for an image. The SBOM, as defined by Syft, can be viewed here.

$ syft -o spdx > hello-go.sbom
 ✔ Loaded image            
 ✔ Parsed image            
 ✔ Cataloged packages      [23 packages]

Sign, attach and attest SBOM

Sign and attach an attestation for the SBOM predicate. With this, the SBOM is signed (and therefore tamper-proof) as it is within an attestation, and consumers can validate its authenticity. This issue describes the difference between the attach and attest commands. The –verbose flag added to Cosign commands reveals more detailed information than the output given here.

$ cosign sign --key ~/cosign.key --attachment sbom
Enter password for private key: 
Pushing signature to:

$ cosign attach sbom --sbom hello-go.sbom
Uploading SBOM file for [] to [] with mediaType [text/spdx].

$ cosign attest --key ~/cosign.key --predicate hello-go.sbom
Enter password for private key: 
Using payload from: hello-go.sbom

Attaching signatures, SBOMs, and attestations to images is as simple as a few commands. As a consumer of software, Podman, Cosign and Quay can ensure you are running exactly what you have intended. Here are some suggestions to ensure the integrity and security of your containerized workloads.

Verify signed images before downloading

With Podman, it’s possible to set a policy such that images can’t be pulled if they are not signed, and can also specify they must be signed with Sigstore. Add the following to Podman’s policy.json file

# for root podman the file is /etc/containers/policy.json
$ cat ~/.config/containers/policy.json 
    "transports": {
        "docker": {
            "": [
                    "type": "sigstoreSigned",
                    "keyPath": "/home/somalley/",
                    "signedIdentity": {"type": "matchRepository"}

With this in place, any pull will fail if there is no signature attached.

$ podman pull
Trying to pull
Error: Source image rejected: A signature was required, but no signature exists

$ podman pull
Trying to pull
Getting image source signatures
Checking if image destination supports signatures
Copying config 93e70746a6 done  
Writing manifest to image destination
Storing signatures

Verify signed SBOM for image

$ cosign verify --key ~/ --attachment sbom

Verification for --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Verify attestation for SBOM authenticity

$ cosign verify-attestation --key ~/ > /tmp/att

Verification for --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Scan image for vulnerabilities

Grype is a tool to scan images for known vulnerabilities. The tool can be configured to add data sources, although it has reasonable defaults for where it obtains its data. You do not need to install Grype as you can run the Grype image with Podman. If you do not have the Podman socket running (the service is installed with Podman but not enabled), you can start it with:

$ systemctl --user start podman.socket
$ podman run --rm --volume /run/user/1000/podman/podman.sock:/var/run/docker.sock --name Grype anchore/grype:latest
libgcc        11.2.1-9.4.el9                   rpm   CVE-2021-46195  Low       
libgcc        11.2.1-9.4.el9      (won't fix)  rpm   CVE-2022-27943  Low       
ncurses-base  6.2-8.20210508.el9  (won't fix)  rpm   CVE-2022-29458  Low       
ncurses-libs  6.2-8.20210508.el9  (won't fix)  rpm   CVE-2022-29458  Low

Lastly, Cosign has an all-in-one command to list known security related artifacts of an image. The command is cosign tree.

$ cosign tree
📦 Supply Chain Security Related artifacts for an image:
└── 💾 Attestations for an image tag:
   ├── 🍒 sha256:06a8bd80b513e9f98009cddd4856f9e54c22cba3df2a9714b457e8c10b1c0b5c
   ├── 🍒 sha256:06a8bd80b513e9f98009cddd4856f9e54c22cba3df2a9714b457e8c10b1c0b5c
   └── 🍒 sha256:22dfc2808a9008d76c79ef0b80cc06a569c367fe880fbf155600385125e18b4a
└── 🔐 Signatures for an image tag:
   └── 🍒 sha256:94683d70b9542b4e5a1d9be2e27923f9f37e3a7fc37c3900b0ff90a6be3c8e9c
└── 📦 SBOMs for an image tag:
   └── 🍒 sha256:87ee673638aaf1c194241b65794a508fa02f93503e1113d4e6eb8278dd28bfdf

In closing

Core aspects of our society depend on open source and community-driven code. Enhancing the security of software supply chains is essential and a responsibility of every open source software citizen. As supply chains become more complex and interwoven, inclusion of signing, Software Bill of Materials (SBOMs) and attestations must become standard. The tools we walked through in this post help to achieve these goals. Give them a try and let us know what you think!