ASWF TAC Meeting - December 15, 2021

Video Conference Link

Voting member attendance

Other Attendees

John Mertic, Linux Foundation
Andrew Grimberg, LF Release Engineering
Shannon Deoul, RazPR
Alix Robinson, LF Marketing
JT Nelson, Pasadena Open Source consortium / SoCal Blender group
Scott Wilson, Rust WG
Matthew Low, DreamWorks Animation
Deke Kincaid, Digital Domain
Allen Rose, Madison Square Garden Entertainment
Brent Villalobos, Dreamworks
Allen Johns, Methods Studios / Rez
Lee Kerley, Sony Imageworks
Stephen Mackenzie, Method Studios

Apologies

Christina Tempelaar-Lietz, Epic Games, Inc.
Joshua Minor, OpenTimelineIO Representative

Agenda

Google Summer of Code deadline
- Larry: reminder of deadline for different projects groups to get their lists of projects together
New marketing manager for LF, Alix Robinson
- Plan to attend some of the calls to get an idea of what the foundations are up to
- Based on West Coast, will be at Open Source Forum
Assets (Eric)
- Talking to ASC about StEM 2 project
- It’s a large project, around 20TB, if that’s downloaded 30 times, could end up rakeing up large bandwidth costs (over $50K, hosting server egress fees, above total budget)
- Wanted to add a clause to the license: if someone redistributes asset in a different form, does it match the artistic intent of the director / DP (recompression, extreme color correction).
- “Redistributions of these digital assets or any part of them must include ^a description of how the redistributed version differs from the original,^ the above copyright notice, this list of conditions and the disclaimer below.”
- Will talk to LF legal about this.
Allan Johns: Rez proposal as ASWF project
- Started at MPC London, one of the major topics being discussed was package management, packet version context, system in place at the time was somewhat crude, a hierarchy of shell scripts. Moved back to Australia, started working on solving the version problem, first version was called drd-config.
- Let’s have a repository of version definitions defining the packages and their requirements, and as solver algorithm that satisfies user requests. From this create the config runtime.
- Early version was a Python solver with bash around it.
- Drd-config / Rez was open sourced when DRD Studios closed.
- Worked for Method / moved to LA, worked on second generation of Rez, complete rewrite and became a Python library.
- Been developed on and off since then, over the last year there’s been consistent progress.
- Kimball: what do you see as major functionality left to implement? Allan: Rez is not a build system, but facilitates running a build system. It has a “build” command to create an appropriate runtime to build a package, detecting CMake files, package.py tells which build command to use. But does not do the build itself, it is not designed to facilitate the build chain. To get there, major piece of work to be done is to introduce concept of “features”, having dependencies of the package that describe a characteristic of the package, not another package. For instance the build type (debug vs release), this “features” concept is require to enable building an entire chain of packages, and thus have the “safe” installation of packages.
- Brent: ASWS is currently using Docker images, has good features and benefits, are there plans to have “Rez in a Docker environment” or “Rez playing nicely with Docker”? Allan: at Method, beginning to use Rez-based software in containers, “Rez Bundle” feature will take a Rez context (full definition of a resolved runtime) and bundle it, copy all of the required packages into a relocatable directory, on Linux will do all the required ELF patching (fix up any dependencies on rpath) and copy the bundle directory into your container. Seems to work fine. Method doesn’t have an internal fork of Rez, always working on the open source version, so this functionality is available in the publicly available version.
- Brent: would there be desire to distribute the VFX Reference Platform as Rez Libraries? Allan: interesting question, right now we don’t have a public repository of Rez packages, so that could be useful. There are various forms of this around from various studios, but no formal public repo. But will be more useful once Rez is able to act as a full packager. An obvious goal would be to have a repository that satisfies the Reference Platform. Brent: publishing the canonical package.py for OpenEXR / Rez for VFX Platform 2022 would be useful. Allan: if we’re going to have a public repository of packages, have to agree on what the dependencies look like. JF: starting to produce Conan packages. Allan: Rez is definitely specific to industry, so multiple package formats makes sense.
- No CLAs or code of conduct yet, but should be straightforward to put in place.
- Rez is GitHub based
- Have full list of dependencies and licenses, in process of switching licenses and verifying
- Dependencies are embedded in the project itself, “chicken and egg” problem. Want to use the Rez API as a Rez package, so need to be able to install the Rez API as a Rez package. But if it pulls dependencies from GitHub, need a bridge (RezKit) to turn these dependencies into Rez packages. But it can be a barrier to entry to install Rez as a Rez package. So that’s why packages are embedded in Rez currently.
- Role of the project: Allan has been sole maintainer, managing releases and PRs, although there are a lot of external contributions. Kimball: do you know how many studios are using Rez? Allan: I wish, no way to know for sure. Looking at GitHub stats, should give a rough idea. 234 forks currently. Seems like a lot of smaller studios are using it, maybe 200, and a few larger studios. Have had lists in the past, but not up to date anymore.
- There’s a Slack channel and a Google forum, but typically people create GitHub issues directly. Issue tracking in GitHub.
- CII: summarized points that will need to be addressed. Have integrated SonarCloud, which was reasonably straightforward for static analysis.
- Rez has a pretty good suite of GitHub Actions workflows.
- Lack of expertise on designing secure software. Kimball: all projects have been struggling with that. Allan: package definitions have to be Python source packages, so runtime code evaluation is a vulnerability.
- Website is GitHub hosted, and a reasonably extensive Wiki (on GitHub).
- Release methodology and cadence: release is simple / linear, will do a new semantic versioned release based on content. No formalized process to mark major releases. Cadence is pretty good, over last year about one release per week on average.
- Kimball: what do you see as a benefit to having Rez becoming an ASWF project? Allan: primary goal is to facilitate confidence in the project to have more people involved from a governance point of view. Should not remain the sole “benevolent dictator”, wanted to write Rez in the first place to solve the packaging problem in VFX, would like to have more people involved to help solve this problem.
- Kimball: have you talked to people about joining TSC? Allan: will do this once project is accepted.
- Stephen Mackenzie (work with Allen): from someone who works at a studio using Rez, and thinks about other studios that use Rez. When interacting with Rez as a project, Allan is the one who understands the project inside out, when considering new features or external contributions. ASWF could not only bring legitimacy to the project, but ability to put it in front of other developers in the industry, build a community of people who can learn how the project works, and “externalize” the knowledge that is only held by Allan right now.
- Allan: I can be the bottleneck, especially with complex PRs. Sometimes can take quite a while to look at a complex PR, can sometimes take weeks / months, consider the implications. The project is big enough that I become a significant bottleneck.
- Michael Min: are there considerations about licensing issues and conflicts once packages are created. Also MPC packages was a bit of a tangled web, and difficult to archive and bring back later, have you considered package archiving and retrieval. Allan: in process of Rez-ifying, haven’t looked at licensing issues since for now it’s been internal, so no redistribution yet. What if you have an external repository? Rez doesn’t have a facility for hosting artifacts, would just be the Rez package definitions. For archival, now have more support for package archival, it’s not complex enough to take into considerations of the requirements of the package itself, for now mostly moving into a repository marked as archival repository. Primary motivation is that over time you end up with more and more Rez packages in your repository, which affects your resolve time, so archival can be an optimization step. There’s a feature to integrate with RabbitMQ, send data to Elastic so we know when and where packages are used, to identify packages that haven’t been used in a while.
- Larry: there is another project with similar goals being worked on by Imageworks/ILM/Weta/Disney, doesn’t seem like there’s a mutually exclusive issue with having two similar efforts. These projects can influence each other, move in parallel directions. Not bringing this up as a point against Rez, but would be good to understand what the relationship would be, considering that several of the larger ASWF member companies are involved in the other project. Don’t want to set up a situation where we accept Rez and some of the companies go in another direction.
- Allan: watched the talk on spk, there were some misunderstandings about Rez in that talk, would have been good to reach out. One thing that came up was the fact that the build environment was based on package runtime requirements, but could be fixed fairly easily. More communication would have been good. Is it dependent on overlayfs? Larry: yes, big difference, Rez is based more on central storage, whereas spk wants to make packages are mountable layers, pros and cons to both approaches. Don’t want to turn this into “Rez vs Spk”, solve slightly different problems. But could they influence / cross-pollinate each other. There’s no ASWF rule that we should have only one solution to a specific problem, and spk is still somewhat far away. Allan: talking about different approaches, overlayfs vs centralized artifacts, I think it would be better to separate the concept of how to create the runtime environment. With Rez can end with very long PYTHONPATH, can create metadata load on shared storage. Could be better to create symlink-based runtime environments, so better to abstract away / more modular, could have different ways to create a runtime environment separate from the original solver. It’s where I see Rez going, have to solve the problem of overly long PYTHONPATH
- Larry: having both projects coexisting not a problem. For instance when accepting OpenCue, we knew that many of the major ASWF companies already had their own solutions, it was still felt there was value to OpenCue.
- Gordon: is there a comparison of “Rez vs spk”? Would be good to understand those differences. Allan / Larry: don’t know that there’s a good one, given current form of both projects. Gordon: how we handle multiplicity is a good question, there’s a “support cost” to having multiple solutions.
- Allan: if spk gains enough ground and is shown to be easy enough to be used by smaller studios, that would be good as well. Main goal is to solve the VFX packaging problem. Gordon: maybe there two very different packaging problems / philosophies and we should support both.
- Kimball: maybe there’s room for a common resolver?
- Bill: is there a rough timeline for spk to become open source? Larry: thought we would be there by now, have a list of issues to be resolved before can have a public 1.0, repository is private for now, add users who show interest, but it’s not a “secret”, but it’s not ready / out of box experience isn’t there yet. Still figuring stuff out, have shrinking number of critical issues before we can let anyone try it out. Don’t want to create bad initial impressions. Don’t have a specific timelines, but feel like we’re closing in on it. Happy to invite anyone to join meetings / repositories. No secret / NDA required.
- Bill: Dreamworks uses Rez routinely / across the board, so we would like to ass Rez join ASWF. Larry: I’m also supportive of Rez being admitted.
- Allan: not having compatibility in spk with existing system such as Rez could be a lost opportunity? Larry: not sure what you mean by compatibility? Allan: yes, spk being able to consume Rez package definitions? Larry: not sure how close that is to be feasible. Could be that the two share package specs / build recipes, and difference would be different backends / solvers, otherwise they can be quite different, but can import package spec from the other system.
- Kimball: we’re unfortunately out of time, great discussion, thank you Allan for presenting. We can continue discussion over email / Slack, we will have a vote over email and can move forward.
Pipeline working group proposal from 17-Nov: Invite Milan to present at next TAC meeting

Rez Project Proposal

Name of the project (existing or proposed): Rez
Requested project maturity level (select one): Adopted or Incubation Incubation
Project description (please describe the purpose and function of the project, its origin and its significance to the ecosystem): Rez is a cross-platform package manager. Using Rez you can create runtimes configured for a given set of packages. However, unlike many other package managers, packages are not installed into these runtimes. Instead, all package artifacts are installed into a central repository, and runtimes reference these existing artifacts. As a consequence, runtimes have a minimal footprint can be constructed quickly, often within a few seconds. The first version of rez was implemented at Dr.D studios in Sydney, and was called drd-config (it was initially close-sourced, but was made open source around the same time the company was shelved). The original motivation for the project came from having worked at MPC London previously, where we’d started running up against significant issues related to package management, that at the time we did not have a suitable solution for. Rez is used widely in the VFX industry as its ability to configure runtimes with minimal overhead makes it useful for managing a large number of different configurations at once (corresponding to the large number of active shows and tools common within many studios in our industry).
Please explain how this project is aligned with the mission of ASWF? The intent of rez from the beginning was to provide a common platform for package management in the VFX industry, which I think quite closely aligns with the ASWF’s mission to “provide a common build [and test] infrastructure” in VFX. Rez aims to solve a problem common to virtually every studio, and unsurprisingly we do see a number of similar, closed-source solutions in place in various studios. I believe that by collaborating on a common project instead, we can minimise redundant efforts at solving this problem; help share knowledge of one system across studios so that expertise is more easily accessible; help share software interstudio if and when that’s required; and overall minimise the internal resources a given studio needs to put towards managing their software.
What is the project’s license for code contributions and methodology for code contributions. ASWF maintains recommendations for contribution and licensing for hosted projects.
- License is (as of very recently) Apache 2.0.
- LICENSE file is included in root of project
- Copyright and license header is included in every source file
- CONTRIBUTING.md present
- DCO signoffs are not present
- CLA is not in use
- No code of conduct provided
What tool or platform is utilized for source control (GitHub, etc.) and what is the location (e.g. URL)? Rez GitHub Repo
What are the external dependencies of the project, and what are the licenses of those dependencies? See Rez project README.md
What roles does the project have (e.g. maintainers, committers?) Who are the current core committers of the project, or which can a list of committers be found? I would have to describe myself as the sole maintainer (I’m the only one who merges to master and manages releases), however there are significant contributions from other members. Committers
What mailing lists are currently used by the project?
- Slack (rez-talk)
- Mailing list
What tool or platform is leveraged by the project for issue tracking? GitHub.
Does the project have a Core Infrastructure Initiative security best practices badge? Do you foresee any challenges obtaining one? (See: CII Best Practices: No it does not have the badge. I don’t see challenges in obtaining one, as it seems that most requirements are already met. However points that may need to be addressed are summarized below (taken from CII Criteria:
- The project website MUST provide information on how to: obtain, provide feedback (as bug reports or enhancements), and contribute to the software. (minor updates to CONTRIBUTING.md required)
- The project MUST provide reference documentation that describes the external interface […] (Sphinx docgen is provided, however public API is not well defined, this will need some work)
- To enable collaborative review, the project’s source repository MUST include interim versions for review between releases (Currently we do not have release candidates, only standard releases)
- The project MUST publish the process for reporting vulnerabilities on the project site (honestly I’m not clear on what vulnerabilities mean in the context of rez presently)
- It is SUGGESTED that the test suite cover most (or ideally all) the code branches, input fields, and functionality (testing is good in general but more could be done to establish actual coverage)
- The project MUST have at least one primary developer who knows how to design secure software (I don’t know what this means in practice tbh)
- At least one of the project’s primary developers MUST know of common kinds of errors that lead to vulnerabilities in this kind of software, as well as at least one method to counter or mitigate each of them (see above)
- At least one static code analysis tool (beyond compiler warnings and “safe” language modes) MUST be applied to any proposed major production release of the software [..] (we do not currently have any form of static analysis in place)
- It is SUGGESTED that at least one dynamic analysis tool be applied […] (what is dynamic analysis and how is that different to a test suite?)
What is the project’s website? Is there a wiki? Rez Wiki
What social media accounts are used by the project? None.
What is the project’s release methodology and cadence? Release methodology is basic and should be updated. Currently we simply perform a linear set of semver releases (patch or minor - major version changes are extremely uncommon). We do not have formal releases where some amount of production testing has been quantifiably done beforehand. Cadence: 66 releases over the past year, so just over one release a week on average. All releases are summarized here: Rez ChangeLog
Are any trademarks, registered or unregistered, leveraged by the project? Have any trademark registrations been filed by the project or any third party anywhere in the world? No.