Open source software supply chains are exposed to multiple hacking risks, often with potentially disastrous outcomes considering the proliferation of open source in all sectors of industry. This is undeniable, the writing is on the wall, and so has been for a while. Even a cursory search of news sites brings up swathes of attacks happening on a weekly basis. Any software is a possible target, from small open source projects to large software vendors experiencing coordinated nation state orchestrated attacks.
Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.
Trust in open source can all too often be implicit in the flawed notion of a community ethos of good intent. We incorrectly assume if a project is open source, it’s somehow safer to use.
This assumption is somewhat credible when it comes to software vulnerabilities. It’s been demonstrated that “the more eyes” on code can result in a well-reviewed code base. However this is far from the case in respect to how software is built and distributed, and this is where a number of attacks now occur.
One common method to protect projects is software signing. Cryptographic keys are used to “sign” a software artifact (tarball, container, binary, package). The signature and public key then allows users to establish a premise of “this package has been signed by a certain person or organization” and so I can “trust” it.
However, at present it’s not a solution that is leveraged much at all. The fact is, very few open source projects actually sign code and even fewer bother to verify. Those that do sign, often store the public keys needed to verify the cryptographic signatures on an insecure medium such as a website or a markdown file on a git repository.
There is also no way to be confident that the person whose key you are presented with is actually the person they purport to be. Add to this, there is no way to be certain the keys have not been compromised and are also in someone else’s possession.
The current signing tool sets are quite frankly starting to show their age. The only way to really establish trust of a key pair is to meet with others in person and sign each other’s keys, by looking at each other’s government IDs. I don’t believe I need to belabor the point much here, that this old system no longer scales, even more so in the post-COVID world.
I should be clear here, there is still very much a place for signature tooling such as GPG, when in a controlled environment (such as Red Hat’s RPM packaging signature systems), but this does not carry over to upstream open source ecosystems with no central body to provide the needed security controls and specialist hardware to store cryptographic keys.
So we know the following statements to be true:
- Very few projects sign
- Very few users verify
- Keys are risky (compromise, revocation)
- Key storage is difficult (hardware solutions required)
- Trust in keys is difficult (in person signing / trust on first use (TOFU))
With these factors in mind, we asked ourselves what if:
- …Keys require no revocation / storage?
- …All signing events are publicly available on a secure medium?
- …Software signing is made easy with minimal setup?
- …Current tools / security systems can be leveraged?
With that in place, I created a proposal and prototype. Then I shared this work with Dan Lorenc, a security software engineer at Google, and we started to do what engineers do best, write code and try to solve problems. We decided early on that we would take a code fast / fail fast approach and not become bogged down in standards.
Shortly after this Red Hat’s Office of the CTO engineer Dr. Bob Callaway took an interest and started to collaborate and things then started to really take off… thus was born sigstore.
Sigstore is a project that provides the infrastructure for developers / software maintainers to sign code with no need to manage keys.
Users generate ephemeral short-lived cryptographic keys with the sigstore client tooling and use the keys to sign software.
A sigstore PKI service will provide an X.509 signing certificate generated upon a successful OpenID Connect grant. All certificates are then recorded into a certificate transparency log and software signing materials are sent to a signature transparency log.
The use of transparency logs introduces a trust root to the user’s OpenID account. We can then have guarantees that the claimed user was in control of an Identity Service Provider account at the time of signing. Once the signing operation is complete, the keys can be discarded, removing any need for further key management or need to revoke or rotate. The keys do not even need to touch the disks.
Using OpenID Connect identities allows users to then take advantage of existing security controls such as 2FA, OTP and hardware token generators.
Sigstore’s transparency logs can act as a source of provenance, integrity, and discoverability. Being public and open, anyone can monitor sigtore’s transparency logs for occurrences of their software namespace being used, perform queries using an artifacts digest, return entries signed by a specific email address, public key etc. Further to this, security researchers can monitor the log to seek out possible nefarious patterns or questionable behavior.
The project recently joined the Linux Foundation and will become a public good / non-profit service to improve the open source software supply chain by easing the adoption of cryptographic software signing, backed by transparency log technologies. The service is being modeled after the success of the Let’s Encrypt project. The code, both software and operational, will be open source and developed by the community. This will allow us to run a public service, with coding and tools developed by a community. Individuals could also elect to run their own signing infrastructure. In turn this benefits Red Hat and our customers, by helping to improve the overall security posture of open source software.
You may also consider further reading on Project Thoth, another project related to secure supply chains from Red Hat Emerging Technologies, part of the Office of the CTO. Anyone with an interest in finding out more about sigstore, can visit the sigstore project site.