We're going to talk about what we mean by signing builds, why it's a good idea to sign them, and crucially show how to verify those signatures as well.
This talk was about using Artifact Attestations to sign builds, link them back to source code and build instructions, and verify those properties before deploying to production. It was given at GitHub Universe.
We're going to talk about what we mean by signing builds, why it's a good idea to sign them, and crucially show how to verify those signatures as well.
What do we mean by signing software? On the left is something you built on GitHub Actions. It could be a Java .jar file, a .zip file, a container image - anything. It's probably several megabytes in size, or maybe even a few gigabytes. How would we detect if someone made a small change somewhere in there to add malicious behavior?
On the right is a signature over that build. We won't get into the details of asymmetric cryptography here, but essentially you make a public / private keypair and produce a signature that's hundreds of bytes or a few kilobytes. If any of the bytes of the build change, the signature will no longer verify. And you can only produce a valid signature if you have access to the private key.
Unfortunately, attackers are always looking for new, clever ways to get malicious code running in production. In the past the main focus was the source repository, but with features like multifactor authentication and required code reviews for the main branch, this has become less common. So attackers have turned to other parts of what we call the software supply chain.
Inside the Linux Foundation is the OpenSSF, an industry consortium working to secure open source software. One of their projects is SLSA which describes the steps of the software supply chain, the attacks that can occur at each point, and how to mitigate them. Attackers might try to tamper with your build instructions, or modify your package when it's stored in a package repository like Azure Container Registry or PyPI, or in your organization's cache of open source dependencies.
Signing software allows you to detect these modifications after the build process finishes. But of course that doesn't cover the entire software supply chain. GitHub Artifact Attestations also includes build provenance, which ties a build back to its exact source code, build instructions, and build logs, which can be very helpful in a variety of situations, including when you're doing incident response.
There have been different approaches to signing software over the years, but many of them are quite complicated and require a lot of manual steps. You have to provision a public / private keypair, and then keep the private key as protected as possible (since that's what generates the signatures) while simultaneously distributing the public key as widely as possible so people can verify the signature.
If someone leaves your team, or if you accidentally leak the private key, you have to generate a new keypair, distribute the new public key, and somehow communicate to people which key they should use when. Operationally, this can get quite complicated.
We built Artifact Attestations to streamline this process. To sign something you built in Actions you add a step to the end of the workflow, and then you can easily verify that signature by using the gh CLI. Let's see what that looks like.
Here we have a GitHub Actions workflow building a container image, and the bold lines are what we need to add for signing. The build provenance comes from the Actions workload id token, so we're requesting the write permission so we can get that token. We also are requesting the attestation write permission so we can save the signature to GitHub's attestation API.
Then we add an id to our existing docker/build-push-action so we can reference its output later, and then we add a call to actions/attest-build-provenance to do the signing. We pass it the name of our container image, the digest (or hash) of the image from the docker/build-push-action step, and we also optionally tell the step to also push the signature to our container registry, in addition to GitHub's attestation API. Signatures can be verified offline, so having a copy in your container registry is great for air-gapped deployments, or if you don't want your deploy process to depend on the attestation API's availability.
You might be thinking "that was too easy" - and you're right! Signing is just the first step, you also need to verify those signatures in order to get the security benefit. Just because something has a signature from Artifact Attestations does not mean it's secure. Anyone can create a GitHub account, build whatever they want in Actions, and sign it this way. We need to verify the signature came from where we were expecting.
To do that, we can use the gh attestation verify command. We use the -o flag to tell it what organization we're expecting the attestation came from. Not only does this tell the command the attestation API endpoint for your organization to call, it also creates a very basic verification policy where we're checking to make sure the Action that built this artifact ran in our organization. And lastly we tell the command what container image or file on disk we want to verify.
You can see that the verification was successful (that's a relief!) The command also outputs some of the build provenance information that we were talking about earlier: what repository built this artifact and what build instructions were used. You'll noticed in this case the information is repeated twice. If we were using reusable workflows, the second stanza would have information about which reusable workflow we used; we'll get back to this in a second.
Here we're taking that same verification command, except we're asking it to output all of the build provenance in JSON format. You can see there's a lot of additional information in the attestation: the exact commit the build came from, if that commit was the head of a branch at the time, a link to the build logs, and even that the build took place on a GitHub-hosted runner instead of a user-hosted one.
All of these properties are things that you can require with a more detailed policy, based on your organizational policies or external audit frameworks you're subject to.
Here's what it looks like to write a more detailed policy. We're using a Open Policy Agent with the command opa, which uses a policy language called Rego.
The box on the top shows our rego policy. We're using the attestation.slsa1 package, we want to default to not allow the policy to verify, and here we're doing the same check as before, that our container image was built in our GitHub organization. But you can add as detailed a verification policy as you want.
Then below, we're taking the same verification command as before, but we're piping the JSON output to opa and checking to see if our policy passed or not.
That's great, but if you're working with container images, you probably don't want to shell out to verify the image. You're probably running that container image in Kubernetes, which already has a system for vetting if a newly requested container image should run or not: the admission controller.
Open Policy Agent has Gatekeeper which is a popular way to enforce admission policy. We built an integration between Artifact Attestations and Gatekeeper. You can easily install it with helm, and in the repository are various policy templates you can modify to meet your needs and then use kubectl apply to run them in your cluster.
So how does that help your organization get a handle on your build process? This is not required, but if you want to go further and write a policy about what build instructions were used, it doesn't help if every team has their own version. This is where reusable workflows can help.
On the left we have our same GitHub organization, but now the myapp repository is using a vetted build process maintained in the build-team repository. On the right we show what the new myapp build instructions look like - it calls the reusable workflow, and can pass arguments in using the with: keyword.
Let's look at the resulting build provenance from a build that uses a reusable workflow. We still have the same commit information and path to the initiating workflow, but we also have that information for the reusable workflow that the initiating workflow calls.
Here's an example of a policy that checks that our build process used the reusable workflow we were expecting. Again, these policies can be as simple or complex as you want, and you can refer to any items in the JSON output of gh attestations verify.
And that's it! As you think about using Artifact Attestations in your organization, it can be overwhelming thinking about all the builds you have to update and how to coordinate teams to use reusable workflows. The good news is you can start to get value even if you are signing and verifying some of what you're running in production.
We have great resources that you can share with people in your organization, including our documentation on Artifact Attestations and a 5 minute video focusing on cloud-native deployments with Artifact Attestations.