This talk was about how package managers have implemented two related, but different, attestations: build provenance (which maps a hash of an artifact to its source code and build instructions) as well as release attestations (which lists out the artifacts to expect for a given package at a specific version). Together, these two capabilities help with incident response as well as preventing the package repository index from tampering.
We'll start off by defining build provenance and release attestations, then we'll show how to get this information from various ecosystems and talk about the minor differences in implementation, and we'll finish by talking about some of the open questions and what the near-term future might hold.
First is build provenance, which links the hash of an artifact to its exact source code, build instructions, and build logs in a non-falsifiable way. This is a new security capability that ecosystems have been deploying in the past few years, and as we'll see, it's very useful for things like incident response. The specification for build provenance that package managers use comes from the SLSA project.
Similar, but different, is the release attestation, which links a given package name and version to a list of artifacts (and their hash) that it contains. This is one of the key responsibilities of a package manager - mapping package names and version strings to a list of artifacts reliably. The release attestation specification comes from the in-toto project
I want to emphasize how these attestations are related, but different, and link to one another. If you want to know what hashes a package should contain, that's a release attestation. If you want to take that hash and get information about it, like its exact source commit and build instructions, that's build provenance. They link together through the hash of the artifact that you're interested in.
Since build provenance is relatively new, I think it's fair to ask if it's actually useful. In December 2024, the Ultralytics package on PyPI saw a strange pull request name that looked like an injection attack. Soon after, versions 8.3.41-42 were published, with build provenance. This raises an important point: just because something has build provenance doesn't mean it's secure, and a lack of build provenance doesn't mean something is insecure. Build provenance provides links with useful information, but it's up to you to follow those links and make your own determination.
That's exactly what the Ultralytics team did: they followed the build provenance links and found malicious code in the new release. They kicked the attacker out of their source repository, but unfortunately the attacker also exfiltrated API keys to PyPI, and so the attacker published version 43 and 44 directly to PyPI without build provenance.
Now again, the absence of build provenance doesn't mean something is insecure, but at the very least it's a yellow flag when something was using it and then stopped. These releases were also found to contain malicious code, the maintainers rotate their PyPI API keys, and they published version 45 which was free of malware.
This is an excellent example of how build provenance can be useful in incident response, and you can read more about the incident on the PyPI blog.
Before we dive into how to get this information from package managers, let's talk about how build provenance works so we understand what the attestations APIs are returning to us. Several years ago, build systems began adding the ability to request a signed workload identity, usually in the form of an OIDC token. This token contains all the build provenance properties we're looking for - links back to the source code and build information.
So our goal is to take those properties in the OIDC token and integrate that into the package signature. Here's an example X.509 signing certificate, and you can see we've mapped the OIDC token properties we care about into well-known OIDs in the certificate.
We do this by using the open source project Sigstore, which includes public good infrastructure services that package managers can make use of. Specifically Sigstore's Fulcio component will take an OIDC token, check to make sure the signature is valid and from a system Fulcio trusts, and return a X.509 signing certificate with the build provenance properties burned in as OIDs
Your build process can then send the signing certificate from Fulcio to the package manager, along with the package that was built.
To help us make sense of what the package managers return, it's helpful to know that Sigstore signatures are contained in bundles, and the signing certificate is just one part of the bundle. There's also transparency log entries, so you can monitor when your identity is being used to sign as well as ensure the honest operation of the Sigstore public good service. There's also room for either a plain signature or an attestation with metadata about what was being signed.
One last thing before we look at the package managers; it's important to verify the Sigstore bundle before you use the information it contains. This prevents the bundle from being tampered with, and also makes sure the bundle references the artifact you have and not something else. Here we're downloading a package from Maven Central and its associated Sigstore bundle, and using Cosign (a Sigstore CLI tool) to make sure the bundle verifies. I won't be showing the verification step on the following slides, but you can use a similar command in each case.
There's lots of good content on the Sigstore blog about how to verify bundles and what to think about when verifying bundles.
We're finally ready to talk about a package manager! Let's start with npm, which implemented a new attestation API. It returns multiple attestation types, so we're using jq to select the SLSA provenance, and then another jq command to look inside the bundle to pull out the certificate bytes and pass them through openssl. And we can see the OIDs for build provenance! We can see what platform this package was built on, the repository, and the exact source commit, as well as additional information that doesn't fit on the slide.
npm calls its release attestation a publish attestation, for the very good reason that npm implemented it before the in-toto release attestation specification existed! Here we're querying the same attestation API endpoint, but instead filtering for the npm publish attestation. Despite the different name, it serves the same purpose: mapping a package name and version (in this case formatted as a purl) to the artifacts that version contains.
PyPI build provenance is very similar to npm; they also implemented a new API endpoint, and there's a slightly different jq selection, but the end result is the same: we can see in the X.509 extentions the build provenance properties for this package.
PyPI also has the concept of a publish attestation, but it isn't currently served out via an API. It could be in the future, but one question here is when would these attestations be verified.
Next up is RubyGems, which also implemented an attestation API. But there's a problem in the content it serves out: the certificate field has the base64 encoded PEM encoding of the X.509 certificate instead of the raw bytes. The good news is no information is lost, but this issue needs to be fixed in order for these Sigstore bundles to be verifiable with Cosign or Sigstore client libraries.
Last on our tour is Maven Central, which has opted to serve the Sigstore bundle alongside the packages, instead of via a separate attestation API. This is totally fine! The goal is to have signed content that is interoperable, not to have every implementation detail be uniform across ecosystem.
So that's a quick tour of attestations through several ecosystems. The implementation of build provenance is pretty uniform, but we haven't quite figured out how we want to protect the registry index. npm signs its publish attestations by maintaining a public/private keypair and having the public key trusted by Sigstore. We wouldn't recommend this approach to other ecosystems, as if we need to rotate the npm publish attestation signing key we would need to coordinate that with the Sigstore keyholders.
PyPI is able to create its publish attestation via Trusted Publishing, where instead of an API key in your release process you can set up a trust relationship between PyPI and your CI/CD system and use the signed workload identity from that CI/CD system to authenticate for the publish.
But also, maybe per-packager-version attestations aren't the right way to solve this problem. The RSTUF project instead periodically signs and publishes the entire index, and is an interesting alternative approach. Content from RSTUF is not yet verified in a normal production workflow.