This talk was about the work we're doing to keep Sigstore cosign interoperable with client libraries and things like npm provenance, Homebrew attestations, and PyPI attestations. It was given at SigstoreCon 2024.
Cosign, of course, is the popular CLI that many people use to interact with Sigstore. The client libraries are things like sigstore-python and sigstore-js. What we've learned is that as language ecosystems adopt Sigstore (like PyPI or npm) they prefer to have implementations in their language. That means the client libraries have had lots of great features added to them, but cosign has fallen behind a bit.
It's helpful to remember how old cosign is, relative to the Sigstore project. In 2021, Sigstore behavior was "whatever cosign does". That changed a year in, when the first client library (sigstore-python) had its first release. Now that there's more than one implementation, we need a way to make sure they are compatible.
We made progress towards that compatability with the [conformance tests](https://github.com/sigstore/sigstore-conformance), first released in late 2022, which codified a verification API and what scenarios verification should succeed or fail. Early 2023 saw the first release of the protobuf-spec, which standardized file formats for signed content and verification content, to help Sigstore components be interoperable. The client libraries all adopted conformance testing and the protobuf-specs, but cosign was never updated to make use of these specifications.
We want Sigstore tooling to remain interoperable, so the community doesn't fragment. But there's other reasons why it makes sense to update as well. Today if we rotate key material on the public good instance, cosign users have to update the version of cosign they're using to make use of that key material. We want key rotation to not require client updates.
Additionally, today cosign verification of Rekor content requires network calls to Rekor. For availability, we don't want the Rekor server to be in the verification path, but also for security users want to be able to verify content in production environments with limited or no access to the public internet. This also means that Rekor won't be needed as an attestation store, simplifying its operation.
This talk mostly focuses on verification, and as a reminder there's several inputs to the verification process. First you have your signed material, which could refer to a container image or a file on disk. Then you have your verification material with which to check the signed material. This is a combination of public keys and certificate chains for things like Fulcio certificates, transparency log entries, and timestamp authorities.
There's also your verification policy: what keys do you trust to have signed certain content, or for keyless flows what identities (user or workflow) do you trust? Policy is quite a complex topic, and in this talk we're going to focus on signed material and verification material.
Here's an example of the confusingly-named cosign bundle, not to be confused with the newer protobuf bundle. If you don't use the cosign bundle, you have a bunch of files on disk you have to keep track of that make up your signed material. But even if you use the cosign bundle, it doesn't include signed timestamps, so if you're using those you still have another file to keep track of outside the bundle. The Rekor entry in the bundle has an inclusion promise, which is nice, but no inclusion proof, which is needed for verification. This means that verification requires a network call to Rekor, which we're trying to avoid.
The protobuf bundle was developed to address these shortcomings. It has space for signed timestamps, and it has the Rekor inclusion proof. It's one file that has all the information you need for offline verification.
Similarly, in cosign there isn't a single file you can use for verification material, which is especially inconvenient for private deployments. Here's an example of verifying a file on disk where you have to specify certificate chains for Fulcio and a timestamp authority. Not only that, but there isn't a way to specify Rekor keys out-of-band: this verification path queries Rekor for its key. This is not great, as a compromise of Rekor could both serve out malicious content alongside a malicious key to verify it.
Protobuf-specs include a trusted root file specification to address this shortcomings. Rekor transparency log keys are included in it, and crucially each section of verification material is a list, not a single element, with start and end times. This makes it much easier to rotate key material, but it does mean that for a given time there might be more than one set of verification material that's valid - that'll come up again in a few slides.
There's one more problem with verification content - how to distribute it. Shown here are the TUF targets of the public good instance, and the current implementation fetches different files for Fulcio or Rekor verification. You can see this has already lead to a problem, as when we rotated Fulcio verification material we went from fulcio.crt.pem filename from fulcio_v1.crt.pem. Clients had to be updated to use this new filename, or they would fail to verify newly signed content.
This problem has an easy solution - trusted_root.json is one of the TUF targets, so just use that instead. It has room to handle rotated key material, and clients don't have to be updated to handle key rotation.
So those are the problems we want to fix, and the new behavior we want to get to. But there's years of content signed by the existing versions of cosign, how do we help existing users move to this new behavior?
Here's an oversimplification of the cosign verification path. This example is from cosign verify-blob, but, if you aren't using OCI, all the commands use pkg/cosign/verify.go and call through verifyInternal(). There are many steps to verification, so here we're only showing Fulcio verification functions.
The first thing we did is add an if statement at the top of non-OCI verification commands, so if you're using a protobuf bundle (and optionally a trusted root) we just call the top-level sigstore-go verification API instead of using the existing cosign verification path. Branching the verification path isn't great, but sigstore-go has gone through external audits and runs conformance testing, where cosign does not.
This was expedient, and it allowed us to write a blog post about how to use cosign with npm provenance, GitHub Artifact Attestations, and Homebrew attestations (this was before the PyPI attestation public beta). This is great as it closes the interoperability gap between cosign and the client libraries, but what about people who aren't yet using the protobuf bundles for their signed content?
cmurphy is working on updating cosign to use the TUFv2 client, and part of that is moving to fetching the trusted_root.json target instead of disparate verification material. That's great, but remember how that means there might be multiple valid verification material for a given time? cosign doesn't handle that, and we didn't really want to re-implement that capability from sigstore-go, so cmurphy came up with a clever solution. In cosign's verification path, we check if we have a trusted root, and if so we call the sigstore-go function to do that specific verification step.
We also wrote some helper functions to migrate to the new file formats. cosign bundle create will take a cosign bundle, or files on disk, and provide you with a protobuf bundle. This might involve an online call to Rekor, but afterwards you can use the protobuf bundle to perform offline verification.
Similarly cosign trusted-root create helps you assemble a trusted_root.json file from certificate chains and transparency log keys. dmitris has used this successfully for a private deployment he manages.
We didn't talk much about signing, but it's very easy to output a protobuf bundle at the end of a cosign signing operation, when you have all the signed content available. Signing, along with verifying with protobuf bundles (as long as you aren't using OCI) landed in cosign v2.4.0.
[codysoyland](https://github.com/codysoyland) is working on OCI support for signing in #3888 and for verifying in #3889.
cmurphy is working on updating TUF and trusted_root.json support in #3844.
The protobuf bundle helper and trusted root helper have landed in main, and will be in the next cosign release.
If you use cosign and aren't working with OCI, we're ready for you! You can convert your old signed content with cosign bundle create and start verifying protobuf bundles. If you run a private deployment of Sigstore, you can move to using the trusted_root.json file, and we strongly recommend distributing it via TUF for easy key rotation. We're still working on this transition, so if there's something missing or something else you'd like to see, let us know!