ASWF TAC Meeting - November 17, 2021

Video Conference Link

Voting member attendance

Other Attendees

John Mertic, Linux Foundation
Andrew Grimberg, LF Release Engineering
Shannon Deoul, RazPR
Rachel Rose, ILM / D&I WG
JT Nelson, Pasadena Open Source consortium / SoCal Blender group
Scott Wilson, Rust WG
Matthew Low, DreamWorks Animation
Ashley Whetter, Python 3 WG
Deke Kincaid, Digital Domain
Morgan Prygrocki, Adobe
Sergio Rojas, Arena World
Alex Gerveshi, AWS
Patrick Hodoul, Autodesk / OCIO
Milan Kolar, Pype.Club
Allen Rose, Madison Square Garden Entertainment
Richard Shaw, Fedora Project
Nick Cannon, Walt Disney Animation Studios / VFX Reference Platform

Apologies

Michael Dolan, OpenColorIO Representative
Sean Looper, Amazon Web Services
Roy C Anthony, DNEG

Agenda

Richard Shaw, packaging (ASWF) projects for Fedora
- Fedora: supported by RedHat, but community driven, only one person paid by RedHat
- RedHat provides build servers and infrastructure
- 6 month release cycle target, currently at version 35
- 2 versions supported, 1 year support cycle, slight overlap, for instance 35 was recently released, 33 will EOL in a month
- No clean correlation between RHEL and Fedora versions, but RHEL picks technologies from Fedora. RHEL 8 is roughly based on Fedora 24
- Not one of the larger distributions such as Ubuntu, but everyone benefits from it, lots of people working on gcc, kernel, systemd contribute to Fedora. Before gcc 10 was released, rebuilt every package against gcc release candidate, find a lot of issues this way.
- Tries to be leading but not leading edge. Jump on opportunity with building new versions of packages.
- Not release a rolling release, in development release is called “RawHide”, will turn into Fedora 36 next. Most people don’t run RawHide on bare metal, mostly VMs for testing.
- Packaging quality of life: must adhere to guidelines as packagers.
  - All artifacts must be provided from archive that was provided, plus patches and scm tree. No Internet access during build, so for instance no ffmpeg in source tree due to legal / US-based company. Everything must be in git.
  - Can’t pull in non-free/OSS code which would violate principles. Rpmfusion helps for binary blobs such as NVIDIA.
  - Everything built from source, even Java (which have become difficult to maintain).
- Good to make things that are optional truly optional, sounds basic, but:
  - Parts of build system that may try to download stuff, must be able to turn fof
  - Bundled libraries: common in Linux world, Fedora used to have a hard stance against this, required committee approval. Now a bit easier, but if the library already exists in Fedora, encouraged to build against system version instead of bundled copy. But if the build system tries to check for existence of system library, may have to remove it during build.
  - Documentation generation and installation should be optional, especially if complex tooling is needed. Typically in /usr/shared/doc/package_name, so if the build installs documentation in the wrong place, will need ad hoc bash script to move it.
  - Would like to keep manual configuration to a minimum.
  - Large changes need to be well documented, this was an issue for OpenEXR 3 structural changes. Know your main consumers, such as Blender. OCIO 1->2 change caused issues since Blender went with 2 unilaterally, but some of its dependencies were still at 1.
- Personally maintain around 150 packages on a volunteer basis.
- Wasn’t the maintainer of older OpenEXR + IlmBase, but took over OpenEXR + Imath.
- Porting guides are important, Larry wrote the one for OpenEXR, would have been stuck without it. Ended up doing the porting myself, and submitting to the list or as PRs to upstream projects for inclusion. May have spent 100s of hours to move everything hour.
- Kimball: main issues were downstream users such as OpenVDB?
  - Yes, OpenVDB not being compatible yet (part of Version 9).
  - Would be helpful if release notes mentioned timeline for adoption
- Ashley: how often do you find issues and need to create a patch for it? Sometimes those patches just exist in an RPM, is that preferred to upstream that patch to the project, do you wait for that patch to make it back to the project?
  - Richard: all cases happen. Very few packages I maintain have zero patches. Larry / OIIO does a good job, but often you need a dirty hack to get things to build that can’t be upstream, but if the patch is clean, will try to upstream is. Some of these patches have been around for a long time.
- Cary: point about “knowing your consumers” is a good one, which we struggle with. The ASWF as a forum brings together representatives of our community, a forum for projects to talk to each other. Adding package managers to the mix is a big step forward. How could OpenEXR 3 experience could benefit decisions in the future. We have no upcoming changes that are as disruptive, if we had to do it again, we would have reached out earlier. In ideal world, would have coordinated with OpenVDB to upgrade to new system to narrow the gap.
- Kimball: we talk to Blender every SIGGRAPH, but they don’t come to TAC meetings (often). In our community, we need to lock software for a long time for multi-year project, but another project starting up may new version. So we need to support multiple versions at the same time. So projects have version number in the namespace. Maybe it’s time to change this behavior.
- Richard: not necessarily a bad thing, soversions exist for this version to have multiple versions installed at the same time. Unless Fedora needs a compatibility package (for openssl for instance), typically have a single package. Only thing that creates grief is lack of ABI compatibility. OIIO is good about it, some upstreams aren’t that disciplined, won’t bump up soversion even when ABI changed, or change it needlessly. Tools such as ABI package diff, ABI compliance checker. Kimball: when redoing build files for OpenEXR, struggled to find a good reference for what is common / best practice. Richard: common practice isn’t necessarily best practice! Libtool has a strict policy, but hard to remember / understand how it works. But in general, if you make an API / ABI breaking change, increment soversion. Cary: yes, libtool is not simple, would like to move forward to something simpler, but haven’t found an alternative yet. Is there a better reference than libtool? Never fully understood the “age” it encodes in the soversion. Would love to retire that part of the process. Richard: would love to see an example of how this is all useful, from distro perspective, have to rebuild everything anyway.
- JF: could look at VFX Platform to get set of interoperable ASWF projects.
- Scott: what’s the main difference between being a Fedora package vs a Debian / Ubuntu packager, or are all the packaging processes similar?
  - Richard: same but different, RPM vs DEB, serve same function purpose. In RPM world, single spec file controls everything. Typically don’t repackage, use the tar / archive provided (unless legal issues requiring code to be excluded) and check that into repo. Debian tends to repackage the archive.
  - Kimball: do you cross reference / share patches with other distros? Richard: sometime yes or no, may look at Debian packages for instance. Sometimes there is formal coordination, but often it’s just “who fixed it first” and use their work.
- Larry: you maintain 150 packages, is there a thread that ties these together, or is it just historical?
  - Richard: at beginning, just wanted to be a packaged, started with OpenCollada, then OCIO… Dependencies are a big part of it. Have personal interest in ham radio, so take care of those packages. People come in and out as package maintainers, so “Fedora Old timers” collect more and more packages to maintain. So an organic process, especially with an all volunteer team.
  - Not clear how many maintainers there are, hard to determine who is active, and what would be a cutoff for activity. But likely around 150 really active packagers.
- Larry: there are lots of distributions out there, most of our projects have no idea how many distributions there are around. Would be interesting for us to try to upstream any patches, but we’re probably unaware of the vast majority of them.
- Kimball: we’re redoing the CI infrastructure to make sure we can compile against 3 years of the VFX platform. Is there a good source of Docker containers for RawHIde so we can build an RPM package that’s good for Fedora? Larry: GHA runners a natively Ubuntu, but we run our CI inside CentOS 7 Docker containers, so we have the ability to test under different (containerized) distributions. Richard: build system runs some RHEL and some Fedora, use a program called mock that creates a chroot environment that allows building under different environments (except for the kernel). Andrew: mock will read the SPEC file, fill a cache with all the dependencies it finds, executes the build steps in a chroot environment with that cache, and will install those into that environment. Mock does it completely offline after it has filled its cache, no external dependency / download during your actual build. And your build will break if you haven’t declared a dependency. Kimball: but that doesn’t test people who depend on you. Andrew: yes, they would need to be building packages as well that are pull in your packages. Richard: there’s a way to test packages after they are built, creates a new chroot environment, installs your package, and runs any command you want. Have tested packages using X11 forwarding.
- Cary: how does Security affect your work? Richard: upload sources to a lookaside card to generate SHA-1 sum, also a RPM signing key is required (using Sigil). No Internet access during build process. Kimball: we get emails about potential flaws / CVE reports, how does that affect Fedora? Richard: RedHat controls these, for instance a bunch of CVEs against OpenEXR were already fixed in Fedora. For serious issues, get help from RedHat, especially if it’s a product also in RHEL. Cary: process on how to respond to CVEs is educational process we went through, unclear what are our obligations as a widely distributed package. We now have a valid process / doing the right thing. Richard: was surprised recent OpenEXR issues were already fixed.
- Larry: Fedora stays close to latest releases of all packages, in the more stable RHEL (and other distros), sometimes they are “annoyingly behind”, how do they determine what versions to use, anything we can do to help them stay closer to our releases? Do we need to communicate better? We want to lighten our support load of how many versions back we have to support. Richard: once RHEL releases, they stay mostly “stagnant”, they don’t do disruptive releases within a point release. On Fedora, when a maintainer has dropped out, sometimes a package will stagnate. There’s a process for dealing with unresponsive maintainers / orphaned packages. Also at some point some packages aren’t needed anymore. Can always submit a bugzilla issue to get someone’s attention.
- Cary: about internet access during builds, OpenEXR uses the CMake fetch content mechanism to download Imath, that’s not a problem for you since you’ve already installed Imath as a separate project, right? Several people have mentioned not being fond of this Fetch Content feature. Richard: maybe should be a CMake flag to explicitly turn it on or off, so that if it can’t find it, it should just fail the build instead of falling back to trying to fetch (which would fail in turn). Cary: wanted to make things simpler for people who want to build OpenEXR and have it fetch its dependencies.
Carol: Google Summer of Code 2022
- Rachel: talking with Carol about internships for next summer, something important in WG to bring new talent to ASWF. Wanted to plant seed in people’s mind, it may seem early, but we need to figure out something by end of January. Two questions that need to be answered: what sorts of projects would we have for projects for each of our ASWF projects, thinking about what would someone do, and not necessarily easy to answer, might be better done in individual TSCs. And whether we want to work with GSoC again, apply as an organization, end of January date tends to be the time we would need to put in our application.
- Have the option of doing something we grow ourselves, last year we discussed different opinions on this topic.
- Larry: 2 years ago we got 3 students, was considered successful. We didn’t apply last year, instead of task sizes spanning the whole summer, it didn’t seem like we had tasks that wouldn’t span the summer. Now they’ve changed the rules again, allowing for “full size” or “half size” tasks.
- Rachel: this year, also opened up eligibility requirements, so non ‘traditional’ students may be eligible and can apply, could be interesting from D&I perspective. Kimball: do they have to be enrolled in some kind of online training? Rachel: “all newcomers to open source 18 years or older, not just university students or recent grads”, so no specific requirements stated now, but may be specific requirements in the application form (working full time?). Larry: open possibility of people who are experienced developers, but not in this area / existing open source developers. This may open up wider range of tasks we could do. Kimball: do we want to do this, or do our own. Rachel: we are happy to contribute to GSoC application, or help spin up our own.
- John: have lot of projects that will do both, GSoC and LFX mentorship, get wider reach / different set of constituents. Have to have larger audience of people who have interest.
- Kimball: if we have 7 students, we won’t get 7 students… Larry: last time, new orgs doing it for the first time got maximum 3 students. We may get higher limit as a returning organization, but no guarantee we’ll get number of applications related to number of projects in ASWF. We don’t want to hard constraint that, we may get 3 great applicants for one project and non for another. Last time we looked at top students, had coordination between the projects, ended up with 3 that were spread across 3 projects. It’s up to us to decide how to use them across the organization. Kimball: seems like we should hedge our bets and try to do GSoC, but also try to do our own thing so we can open up to more people. John: when you do your own, a project might want 2-3 mentees, gives more flexibility on what they want to work on. Cary: GSoC requires code writing, as opposed to documentation, test case writing, asset gathering…
- Kimball: any thoughts on how to broaden scope? Rachel: we’ve talked about being able to bring in students who haven’t made it far enough to contribute code, but maybe code, assets. Would open up a wider range of people who could contribute. Wave: Asset WG talked about getting Intel cloud VDB assets, and get a student who would want to try these assets into various packages, classify them. Has to be a valid production level project, could be good for someone who is more on the artist side. An opportunity that’s different but related to writing code. Larry: we should do GSoC, it’s a no brainer. But we should strategize on how to evangelize the GSoC to targeted universities that have programs relevant to us, but also ones that we would not normally target. Try to encourage applications from people who would be good fits for our projects. Student on OpenEXR 2 years ago came from a university that had a Motion Picture Sciences program, might want to “pave the road” for some of these programs to encourage applicants, rather than relying on the somewhat random nature of GSoC applicants. Wave: could reach out to profs at some of the art schools or relevant programs to find students who might already know some of the DCC tools.
- Rachel: started making a list of schools where we wanted to have focussed “advertising” of our programs, timeframe was too tight, but could go back to that. Wave: college students only have limited opportunities to “lift up their head and look around”, January might be a good time for that. Reach out to profs who could recommend students?
- Rachel: we should plan to apply for GSoC, and should come up with list of projects that either apply to GSoC.
- Kimball: should we have a brainstorming session at next TAC? Rachel: yes. Shannon: will want to blog post about it as well.
Pipeline Working Group Proposal: https://github.com/pypeclub/aswf-wg-pipeline/blob/main/README.md
Python 3 Working Group Closure request (https://docs.google.com/document/d/1RmvZPwwhLu0KmCXbn4XgS4uzVuFCT2K21eQdQfnWYLE/edit?usp=sharing)