Academy Software Foundation - Project Requirements Office Hours - August 28, 2024

Join the meeting at https://zoom-lfx.platform.linuxfoundation.org/meeting/97880950229?password=81d2940e-c055-43b9-9b5a-6cd7d7090feb

Attendees

Cary Philips - OpenEXR
Jean-Francois Panisset - WG CI
John Mertic - LF
Yarille Ortiz - LF
Stephen Mackenzie - Rez
Jean-Christophe Morin - Rez
Gary Oberburnner - OpenFX
Eric Reinecke - OTIO

Meeting Minutes

Scope: an informal meeting where Cary Phillips from OpenEXR shares the work done on OpenEXR to help meet the OpenSSF Best Practices Requirements for ASWF project graduation.
Confluence Page
Based on experience with OpenEXR, not an expert! It’s not rocket science, you can work your way through every step without being an expert. If ayone knows more, happy to hear about it.
Thought sharing experience was worthwhile
Most important is fuzz testing. We’ve been using Google Fuzz testing since 4-5 years. Great service, but requires a fair amount of configuration. Will schedule a later session to focus on that since it’s non trivial. OpenEXR had a fuzz test as part of test suite, so hooking into Google service was reasonable straightforward, but ended up rewriting it (Peter Hilman did this).
Requirement for “security policy”: certain security researchers want vulnerability made public (CVE). Thought it was a demerit and to be avoided, but actually it’s a “good thing” to show publicly resolved CVEs. So when someone reports a vulnerability, fix it. We don’t file CVEs ourselves, but if someone else wants to file one, once the fix has been made, we go through the NIST site and update the info for which releases contain the bug and which fix it. We list them at OpenEXR Security Document
Feel free to copy this security document
Every project should have a security email address, LF can help set up. security@openexr.com forwards to the TSC private email. A few years ago someone sent an email to security@openexr.com but didn’t get a response for a few weeks / months, then filed a public CVE. But turns out their email had been blocked by Google Groups. So no guarantee that an email will land in the right place.
So also enable the GitHub mechanism for filing security advisories. Now that I’ve configured the repo and turned this feature on, the Security tab will now take you directly to the SECURITY.md file. If you don’t have this file, I forget what this looks like. But you should see a Report a Vulnerability button.
Statement of commitment: OpenEXR says acknowledge in 48h, patch in 14 days. Our obligations as project maintainers are serious, 48h / 14 days seems like a reasonable commitment.
For the range of supported versions, borrowed statement from OIIO. Will commit to patching the latest version, and a set of previous releases which may only be patched on request, and some older releases we can’t / won’t patch.
Security Expectations: explaining the nature of the library / project, what it involves, what it does / doesn’t do. A lot of security issues don’t apply to a project like OpenEXR: it has no interface, no web component. So this is a place to make these statements. And here are the steps we’ve taken to make sure it is “legit”. Expected audience is someone who stumbles across the library, and needs to know if there’s some attention to security. We want to show that we are taking these issues seriously.
A lot of issues around security have to do around vulnerabilities from dependencies. OpenEXR has very limited external dependencies, but that’s important to outline.
In general, it would make sense if all ASWF projects followed the same template. If anyone wants to improve on the template, by all means, but feel free to copy and tailor to your project.
Signed releases: I had little to do with OpenEXR before 2018 / ASWF foundation, there was a long history of maintenance of OpenEXR that went on at ILM / Dreamworks / Weta that I was only somewhat aware of. The software was originally hosted on GNU Savannah instead of GitHub. We signed the release tarballs using GnuGPG. On the GitHub release page, what you get “out of the box” on GitHub is the tarball in .zip and .tar.gz format, I don’t know enough about the GitHub implementation, but if you click on the link, you get a download, not sure if it’s created on the fly or pre-created, but that’s the out of the box GitHub feature. Potential vulnerability is that a malicious actor could inject something into the process, so the link gives you something you don’t think it is. So you can not use this feature and instead build the tar file explicitly and generating a key based on the content you know to be good, and then uploading that key so that when someone downloads the specific release artefact that you built, and also downloads the GPGkey, they can use to verify that what you downloaded is what you intended. But you have to manage these GPG keys, which is a pain. The process for OpenEXR sort of fell by the wayside since I didn’t understand it so well.
When it was decided to implement the policy, found the Sigstore service which manages holding the keys. It’s a service you can sign up for which will automatically generate the key file and upload it for you. In openexr/.github/workflows/release-sign.yml it uses the Sigstore service with pre-built GitHub Actions. This workflow is triggered when a releases is published: it downloads the tarball, generates the key, and uploads the key.
In the SECURITY.md, I added a paragraph on how to use sigstore through Python: pip install sigstore ; sigstore verify github --cert.... This seems like a proper implementation of release artefact signing. When there are issues, it comes from Linux distro managers who are very concerned that the code they build and distribute with their OS distribution has been verified. So unlikely to get concern from anyone but distro maintainers. sigstore is a project of the LF / OpenSSF, so it’s the mechanism of choice.
JC: I contributed a change in https://github.com/sigstore/gh-action-sigstore-python for your exact use case so that you can use it without having to manually create the archive. https://github.com/sigstore/gh-action-sigstore-python/blob/main/CHANGELOG.md#changed
Stephen Mackenzie: PyPI may also support some kind of signed release, so could be used for Python projects. So maybe there’s a secure way to use GPG keys for PyPI releases. Cary: that’s a good point. OpenEXR’s Python support is 15 years ago someone implemented Python bindings for OpenEXR, published them on PyPI, so if you did pip install openexr it wasn’t our project. It was then abandoned, but someone took it over and turned it over to us. So we’ve only “owned” the OpenEXR package recently. The state of the Python bindings for OpenEXR is “useful enough that some people rely on it”, but no one in film industry uses it (that I know of). So my perspective is that OpenEXR is primarily a C++ library, but that’s in the process of changing it.
We’re using the command line github tool to upload the release artefact.
Signed release tags is another requirement. On the releases page, the green circle with a check, as well as green Verified tags in the tags page. There’s a mechanism to embed a signature in a tag. This is also an option you can enable for the repo, not clear where we enabled that in the repo settings. The individual who will make the release will reference the GPG key in their GitHub local setup. Last step before creating a release is to create the tag for that release. When you draft a release, I don’t let the “publish release” process to create the tag, I create the tag myself with tag -s to associate the signature. I will then publish the release from the pre-created tag.
Gary: Usually you create the key manually, offline, and upload the private key as a secret. Then use the secret in the gh action And upload the pub key to your site for confirmation
Dependabot: I have developed a love / hate relationship with dependabot. It’s a service which mostly keeps your GitHub Actions up to date.
StepSecurity. Enter your GitHub repo and click Analyze, it will file a pull request that enhances your configuration in certain ways. There’s about a half dozen things it does, when I created the Pull Request, it will have a lot of “stuff” in it, better to step through the difference parameters one by one so you can see what it’s suggesting in isolation rather than all at once.
You can pin actions to a SHA instead of a version tag. This will transform your workflow file where actions reference a specific SHA-1 hash, and will put the version in a comment. If your action just says “use this action at this version”, it’s conceivable that a malicious actor could intercept this and run a different action than what you think. Specifying the SHA makes sure you are using the version you think. The workaround that Dependabot uses is the comment after the SHA-1. It makes things a bit harder to read. But this is the stated principle of hardening that’s worth instituting.
When you configure this, Dependabot will send PRs to upgrade to new versions, for instance yesterday got a PR to bump a SHA for codeql, and somehow only the last of 3 changes as the version as a comment.
What I don’t like about dependabot is that’s it’s a constant stream of updates, maybe that’s good, “cost of doing business”, but it feels like a lot of change / churn. Seems we’re getting more dependabot PRs rather than real code updates, it clutters your PR list. But upside is that it’s important to stay on top of your GHA dependencies.
When the PR comes through, provides a description of what changed, but typically don’t have a deep understanding of what it means. In one case accepted a change which broke something in our CI. codeql started failing, was a regression in one of the releases, notified them and they are working on it. You will get a deluge of updates, you will have to decide how to deal with it. Constant updates / stay on bleeding edge, or more intermittently. You can respond in a comment with a special @dependabot comment that can shut it up about certain things. For instance our website doesn’t build with latest version of Sphinx, so you can tell dependabot to ignore major versions, but still provide updates to version in use.
JC: you can also configure it via the .github folder, there’s a dependabot config file, there’s a parameter where you can specify a version range for your dependencies. Cary: yes, we have a .github/dependabot.yml as well as using the PR comments. https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#ignore
I set up Dependabot SHA pinning through stepsecurity.io, which was my introduction to it. Also workflows should have write permissions only where explicitly requested. Every workflow should have a permissions: contents: read and only add write for sections where you upload a build artefact. Someone contacted me to show that this is the proper configuration to make sure that by default workflows shouldn’t be able to modify the repo. Restrict permissions for GITHUB_TOKEN in stepsecurity.io does this.
John: would it make sense to scope one or two of these items that would be good to start with? What would be 2 things that would be good to have everyone do? Cary: maybe do a sweep through the projects to see which ones are doing these things and which are not. All of this is “drop in” easy to do, just takes a bit of someone’s time to read through the documentation and figure out what to do. When the idea of this session came up, I could take the release signing workflow and submit a PR to projects that don’t have it, but I think it’s better for someone on the project to do it. Stephen: yes, should be better to distribute the knowledge / work. I wonder if maybe there’s a more granular way to handle it. Maybe we can pick one item per month, for instance OSS-Fuzz, next month it’s signed releases, and so on. John: signed releases are a good example of something that’s fairly “drop in” to do, not a tremendous amount of work, especially since there’s a template. Maybe that’s a good starting point? Could we get everyone to do that, and at the next meeting figure out if there are any issues? Cary: currently approach is that every project has the badge requirements, and it’s up to them to work through the requirements according to their resources. What if we flipped this around and said “let’s get every project set up with signed releases?”. John: that makes sense, try what you have done here, see if it works for them, come back at next call, see if there are pain points, and figure out how to get projects not on the call to do this. Cary: would be good to look at the badge requirements spreadsheet, get everyone to step through the issues together.
That kind of thing may align with Dev Days, except we don’t want newbies doing this. John: could be something we push through the TAC. Most of the projects have struggled getting signed releases going, so could be a good one to get started? JC: each project releases differently, have talked to most of the projects as to how they release they Python bindings, everyone does it slightly differently, which is normal. So signing could be tricky for some projects like MaterialX. Would like to build some resources for this.
John: #1 thing that every project misses is signed releases, so that sounds like a good one to start. We may not hit 100% of the use cases, maybe find the oddballs, next new project that comes in, here’s how to do signed releases, here are exceptions… Seems like a common problem everyone has. Not trivial, but not hard.
Cary: OpenEXR is simplest from a structure point of view since we just build C++ libraries. What I’ve done applies to the simplest projects, may be more involved for other projects. John: for now we’re trying to sign the source releases, not the binaries. JC: there are 2 aspects to signing: you can sign like what Cary did on OpenEXR, and then there’s attestation / provenance, “SLSA framework”: you sign your releases, but also provide a trace / proof of how it was generated, and prove it wasn’t tampered in any way. That’s the next level, what OpenEXR is doing is already a good step. John: yes, this is where everyone needs to go over time. JC: GitHub supports SLSA natively, it’s built in in some ways. You can now have an attestation tab. This Git entry is for Go, there are others as well.
Cary: based on the spreadsheet, will file issues with projects on how to address what we’ve talked about here.
John: will make a call out to technical project leads, point them to your instructions, and follow up at next call.