Project sandboxing and how do we scope to our principles
Notes
OpenCue, Brian Cipriano, TSC chair, working on project since its inception for the open source version. Slides link:
OpenCue History
OpenCue is an open source render management system
Can submit jobs to the system, have a complex structure with multiple layers of stages, dependencies between jobs, layers, frames, anything in the system.
Job submission, scheduling, monitoring progress
Supports arbitrary commands as part of job submission
Single or multiple frames
Aimed at rendering frames, but supports any kind of work.
Developers originally at Sony Imageworks, open sourced in partnership with Google
Cue3 starts in 2003 at SPI
2018: OSS development starts
2019: OSS release, joined ASWF
2021: Graduated to “adopted”
Past milestones
Oss release
Project governance
Proposed -> incubation -> adopted in ASWF
Docker support, initial packaging
Documentation / sandbox
Limits system for fixed license pools
Python 3 support for all components
Windows support / cross-platform support
Scheduler improvements
Past Year
TSC / developer turnover
Some churn, new developers
M1 support
Complete dependency refresh: Postgres, gRPC, Docker, PySide 6 (existing dependencies were older and not compatible). That came with some challenges.
User by several of the developers
Affected us quite a bit, put some items on backburner
Developers on M1 MacBooks could not run OpenCue until we did that.
Had to go for a while without an official release.
Change affected release schedule, how can we improve this in the future, continue to do releases decoupled from developer env.
Improve release process
Revamped Docker images and CI workflows
Don’t use CentOS since EOL
Integration testing w/ sandbox setup and test job
Latest PyQt version is not supported until next year (VFX Platform), added compatibility layer to support PySide 5/6
Still trying to maintain Python 2 compatibility, getting harder over time, less testable, fewer packages available.
Starting to get back on a regular release schedule.
Increase confidence in build / test / releases
Use GHA, uses our sandbox setup, stands up whole OpenCue setup on a single machine, increases confidence in each commit.
Improve install and configuration process
Configuration refactor of all client-side component
Some was hard coded, some scattered in config files.
Now standardized, much easier to create config files, env vars pointing to config files. Don’t have to dig into the code to change parameters.
Configuration guide
Written, about to be released with new release
Packaging script cleanup
setup.py, improve install process
Initial sdists, wheels in internal testing
Want to get PyPI packages published, made good progress.
Hoping to get those out to users soon.
We need to expand our testing setup to cover Windows / Mac to help verify these wheels. Some of our dependencies (PySide, gRPC) are not pure Python, so wheels are important.
Biggest long term project, especially for new users
It’s been a problem area for a long time, lots of moving parts. Someone who just wants to try the system without a multi day deployment process. Starting to get back to this.
Our core dev team doesn’t use the sandbox setup, we all know how the system works, sandbox setup could fall out of data. Now that it’s part of the CI process, can make sure it stays up to date.
Contributions:
Around 2 merged PRs per week, a bit down, work on M1 took time and affected those numbers.
Increased total and active contributors: total contributors 185 (up 15%), active contributors 53 (up 26%). We noticed this in GitHub and email traffic, unaffiliated people from SPI or Google, a positive sign, looks like improved project health and project interest.
Net Year
Continue to improve install / config process
Get PyPI packages out to users
More deb/rpm packages for server-side components
Expand testing
Windows, Mac environments
GUI testing
Our automated tests use our Python API, job description library, but not doing a lot of testing of GUI components. We have unit tests, but not the same as the full environment
Blender plugin
A couple of developers working on this plugin, design and build. Long term project to add more official plugins for major VFX packages.
Challenges
Core team is consistent, others rotate in and out: 3-4 consistent TSC members, another 3-4 positions are moving in and out.
Consistent enhancements, less progress on the big stuff. But expanded knowledge sharing, developer diversity.
Maintenance tasks like the M1 issues can derail progress
New minor features, what people want to use in their own environment. It has made it more challenging to work on big new features / architectural features. Current architecture works well now, can be harder to focus on long term issues.
Questions
JF: any uses in non-M&E applications? Brian: we had someone present to TSC, they were doing work on holographic displays, a but outside the realm of VFX, it involved rendering, they were using OpenCue to help prep models for real-time physical holographic displays. But in a “adjacent” area. We might look at expanding to non media use cases.
Kimball: Python 2 support: through end of Avatar 2 we were still supporting VFX Platform 2015, but now we are saying Python 2 is dead. Are people still asking for Python 2 support? Brian: mostly trying to “be nice”, not sure we’ve heard a lot of people clamoring for it. Some fear of dropping that support altogether. It gets harder over time, we drop some pieces of testing, hasn’t yet gotten so painful that we’ve felt to cut it off. Nick: having to support gcc 6 uses up resources, we should conserve our hours for more forward looking work? Larry: We still have shows using older DCC versions… Nick: those should be support platforms, not development platforms. There’s no need for us to modernize features on old platforms. Larry: not everyone has a complete locked pipeline, some of the old shows are expected fixes / new features. It’s a hard decision, we may be able to tear ourselves away from older platforms some time this year. Nick: if we want to put a new feature in openexr, if a show wants it in an older pipeline, they can bring that requirement to us. But lets focus our attention to maximiize our ability to move forward.
Brian: we’re trying to support a wide range of features. VFX Platform helps, but some pieces where it doesn’t help, especially on Windows / Mac where it’s not specified, Java version for instance. Nick: having a version of long term support vs development can help. Brian: having useful guidelines of “we are supporting this for 3 years” can help. Carol: there’s a delay between a VFX Platform release and its adoption. OCIO counts on this approach, we tend to be a bit more aggressive in releasing new features with new requirements with the knowledge that it will take 1-2 years for anyone to pick it up, gives projects the time to finish out and update their dependencies. Nick: I really need OCIO 2, but stuck on 1 right now because shows that need compatibility back to VFX Platform 2019 and Python 2.7. We could have been integrating OCIO 2 for a long time, but haven’t been able to. Larry: all our apps are on OCIO 2, but we have 1.0 configs everywhere since there’s a stray 1.0 only third party tool we can’t trust to read 2.0 config. Carrol: target has been to get libraries updated to 2.0 so that gets integrated and tested, would like if people used the new features, but that’s a longer term goal.
Eric R: the fact that newer versions exist that can’t be used, how does that drive the desire for studios to move to newer VFX Platform years? Larry: we have a show still on Maya 2020, we would have to have specific versions made for that show. Lots of #ifdefs, multiple compiles… Eric: decisions we make as individual projects, how much does that influence studios? Larry: it all flows from the DCCs, we would stay more current if we could, lots of work to Q&A a new DCC major release, which we can’t do during a show. New shows want to pick up new DCC versions, but internal tools need to support all those releases, have to stay compatible with older shows. Kimball: gets even more complicated when collaborating with other studios. Teasing studios with new features / performance improvements, you can incite movement. Getting off topic, maybe gives some guidance on Python 2 deprecation?
Eric R (chat): The opportunity cost of supporting Python 2 is high too - there are a lot of great language features in Python 3.7+ that help velocity a lot
David: follow up on a previous TAC conversation, how do we consider new projects, Gordon framed the issue well. Once we accept a project in the foundation, feels like a one way street. The question is about refining our criteria for projects when they come from the outside when they are not “obvious” projects, projects that are a bit further aways, from other industries, not too many users. (WrangleBot)
We should strengthen our criteria. We didn’t finish that conversation at that time. We may see pictures not directly related to Motion Pictures, we want to be mindful when they come in to accept those that are “worthy”, and what is the definition of “worth”?
John: Early-stage projects that the ASWF TAC believes warrant experimentation.
New projects that are designed to extend one or more TAC projects with functionality or interoperability libraries.
Independent projects that fit the ASWF mission/vision and provide the potential for a novel approach to existing functional areas (or are an attempt to meet an unfulfilled need).
Projects commissioned or sanctioned by the ASWF, including initial code for ASWF Working Group collaborations, and “experimental” projects.
Any project that realistically intends to join ASWF Incubation in the future and wishes to lay the foundations for that.
David: for WrangleBot, feels like we didn’t end up having enough time to conclude the discussion. What do we do with it?
Larry: feel like a sandbox project is something we’ve already deemed aligned in a way that it should be an ASWF project. What we’re debating is what’s the filter that’s applied before that, what’s allowed to take focus from the organization.
Kimball: we have finite resources in terms of support bandwidth, how do we gauge projects. Carol: what is a sandbox stage project, what are the requirements and expectations. Currently the requirements are pretty well defined, but maybe we should add more details, all the big projects that were “easy” are already in the foundation (not all, but most). Need to add some qualifications as to how we need to look at these projects.
David: also maybe giving ourselves more tools to say respectfully “no”. Sometimes we don’t know how to say “no”. Larry: often the “no” is “not yet”. David: refining those set of answers (“not yet”), need to define those better.
JF: also the issue of hard costs (CI, Slack…). Kimball: not the sole consideration.
Eric E: initially wanted to make sure that key projects crucial to studios didn’t go away. Larry: I agree with that 100%, one exception is that there is a certain kind of project where the neutral forum is crucial to find out if it can get there. The ORI project is like that: if it goes away we would be OK, but it could be great for the future, and the major project members are major members of ASWF, and this is a good match. For other projects, the “risk management for the industry” criterion makes a lot of sense. Kimball: if there are projects where an open standard would benefit the studios that would be a good candidate. OpenAssetIO for instance. Eric R: what does become an ASWF project do for that project, we don’t want to be an ASWF project to be a “stamp of approval”. It creates a neutral ground where we can collaborate. A lot of projects may not benefit from the legal framework / centralization. Not sure if we want to write that down?
Matthew: have had similar discussions in DPEL, how to accept assets, sometimes we have “gut reactions”. Some proposed assets are interesting, for instance an anatomy body, was really interesting, compelling as a data source, but didn’t feel assigned with DPEL mission of a production asset for film & TV, helped to communicate decision back. Helps to drive the decision. Wave: we have an emergency release valve, the USD WG assets repo if it doesn’t fit the DPEL criteria. We don’t have this for software project, a place for project to go. If you get to sandbox stage, doesn’t guarantee that you make it to the next stage.
Larry: but a project can bootstrap as before, just create a GitHub project, it works fine. Wave: right, once it moves into the sandbox, there is no notion of moving on to the next stage.
Eric Enderton: are we talking about which projects need to be let into the sandbox, or leaving the sandbox? Carol: probably both, but we’re looking for more rigor on what projects to let into the sandbox.
Wave: for DPEL, it’s helpful to have something we can point to as to why we didn’t accept the submission. Having the “if this went away, we’d be in trouble” criteria would be useful.
Ben Giles (chat): I think there was a similar “maybe one day” regarding Wayland versus X for the VFX RP. And the reason was something like “need an understanding of colour” and there was a lot of experience with X and colour, while Wayland needs a lot of understanding (and it’s a protocol, not a particular window system).
Eric R: would be helpful to write down these evaluation criteria. “How much is it actually used in the world”, is it something people are already relying on.
Kimball: yes, the next step, codify a set of parameters. Here’s how we judge critical libraries. Eric E: two kinds of “not used”: we may already have an in house tool for that, or we don’t need that.
John: this is a good conversation to have. A lot of foundations get to this point of “we have a lot of projects in, we need to make sense of where we go from here”. CNCF did an exercise in its early days, they were at crossroad where they had Kubernetes project that took a lot of the investment that was happening. Cloud Native world had a much wider range of potential projects, lots of back and forth to define itself, are we a “Kubernetes” foundation? They get 20 projects presented by TAC meeting! (TODO: add CNCF link). What are we trying to do, what’s our role in trying to shepherd this forward. This is our role, did this in conjunction with their Governing Board. How do we know if a project is a “bad idea”? How can we tell that upfront? Maybe we need to define “this is what we are here to do”, is it being used out there is a valid one, but what if it’s a new idea which is not used yet? Also comes down to what is the role of the foundation, it’s fair to say that the initial goal of the foundation was that some important projects needed a home, that was accomplished in first couple of years, what should be the focus next? Is this a forum for new ideas, or for existing, widely used projects? Once you answer that, it becomes easier to answer project proposals. CNFC PRINCIPLES.md
Kimball: in guiding principle of “is this crucial”, sometimes we struggle in reviews for Sci-Tech investigations, if a piece of tech comes not from VFX but film making, for instance a piece of software that tracks on set equipment. How do we write the requirements to look at a piece of software like that, doesn’t help VFX, but helps the motion picture industry. We should continue this discussion on the Slack. John: we can also take some time at next meeting.