Academy Software Foundation Technical Advisory Council (TAC) Meeting - October 2, 2024

Join the meeting at https://zoom-lfx.platform.linuxfoundation.org/meeting/97880950229?password=81d2940e-c055-43b9-9b5a-6cd7d7090feb

Voting Representative Attendees

Premier Member Representatives

Project Representatives

Carol Payne - OpenColorIO Representative
Cary Phillips - OpenEXR Representative
Chris Kulla - Open Shading Language Representative
Diego Tavares Da Silva - OpenCue Representative
Jonathan Stone - MaterialX Representative
Ken Museth - OpenVDB Representative

Industry Representatives

Jean-Francois Panisset - Visual Effects Society

Non-Voting Attendees

Non-Voting Project and Working Group Representatives

LF Staff

David Morin - Academy Software Foundation
Emily Olin - Academy Software Foundation
John Mertic - The Linux Foundation
Yarille Ortiz - The Linux Foundation
Andrew Grimberg - LF Release Engineering

Other Attendees

Lee Kerley, Apple
Sam Richards, ORI
Robin Rowe, Cinepaint
Victor Lu
Doug Walker, Autodesk / OCIO
Spencer Stephens, Zero Trust WG
Lorna Dumba, Framestore
Jonathan Schwartz, NVIDIA / OpenVDB

Antitrust Policy Notice

Linux Foundation meetings involve participation by industry competitors, and it is the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

Examples of types of actions that are prohibited at Linux Foundation meetings and in connection with Linux Foundation activities are described in the Linux Foundation Antitrust Policy available at linuxfoundation.org/antitrust-policy. If you have questions about these matters, please contact your company counsel, or if you are a member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of Gesmer Updegrove LLP, which provides legal counsel to the Linux Foundation.

Agenda

General Updates
- Move Technical Charter and license scan requirements to the Sandbox s #824
- Security Threat model analysis for ASWF projects #615
- OpenQMC #434
- Dev Days 2024 #662
- Project Leads Office Hours #760
- Create voting guidelines #855
- upgrade GPUs of CI runners #863
OpenAPV #803
- Commentary on OpenAPV
Evolving our working groups program #798

Notes

General Updates
- Move Technical Charter and license scan requirements to the Sandbox s #824
  - Reflected the need to address all housekeeping tasks before project proposal to TAC, for instance had issue with project naming
  - Also license scanning
  - Before we get to sandbox stage
  - Eric: for license scan, what if a process wants to get into sandbox to get access to resources for a license scan? John: if they are needing that, LF can do that ahead of time, and help isolate potential issues. So they can have good guidance when they get to the TAC. We’re not expecting 100% perfect, but want to check major items. Catch anything that says “proprietary” for instance. Started doing that with many projects, sometimes TAC doesn’t catch it and that delays the process.
- OpenQMC #434
  - No updates
- Create voting guidelines #855
  - Misalignment on voting expectations for OpenAPV
  - PR should align with our process we tend to follow, but wanted to codify it. Specific to the TAC, but could be adopted by TSC.
  - JF: will add comment to PR about determining quorum
- Dev Days 2024 #662
  - John: chart posted to TSC leads
  - Carol: we will want to reserve time in future TAC meeting for feedback. We want to get the ‘non tangible’ stuff in, get feedback from projects. Overall seems to have gone OK, different projects had different experiences, which is fine. No major problems from what I could tell.
  - Larry: still evolving, sometimes people don’t finish the work on the day, but may still be a few cycles of comments / reviews.
  - John: also a few PRs didn’t get tagged for dev days. We doubled number of PRs from last year, 33% increase on closed PRs. Good overall, will work on final report.
- Project Leads Office Hours #760
  - Trying to get second time slot
- upgrade GPUs of CI runners #863
  - Ken: adopted new project that can do learning on sparse VDBs. Jonathan: hardware available as CI runner has a T4 GPU, lacks the data format we need to run our tests, generation before Ampere generation that has Tensor Flow 32 format that is needed to validate some of our kernels. Also FP8 came out with Hopper generation. Have other projects needed similar requirements?
  - Ken: GPUs on CI runners have old GPUs, we need newer GPUs, as NVIDIA we should be part of the solution. Is there someone on the TAC I can work with so we can potentially work with to resolve this? Also if there are other projects with similar issues, we would like to talk to you.
  - Jonathan: T4 are Tesla generation cards from 2018, there have been newer data formats on later GPUs that we need for testing.
  - Larry: my experience is on using GPUs for graphics, for years and years NVIDIA has had a good track record of forward and backward compatibility, is this a performance issue, or you can’t run at all on the T4s? Jonathan: it’s code specifically written for these data formats, but FP32 is a 19 bit float format that only exists on newer generation cards. We can test validity of operator on other formats, but not these ones.
  - Andy: we’re using GitHub hosted runners, unless NVIDIA were to work with Microsoft / Azure / GitHub (GitHub hosted runners are hosted on Azure). There’s also the option of AWS CodeBuild (used by OCIO), so if CodeBuild offers newer GPUs, that’s an option.
  - Eric R: you can self host a GitHub runner if they have hardware sitting in a rack. Andy: yes, this is also an option. Erik: they would also have to manage those runners.
  - JF: self hosted may be the most realistic option. John: should this go to CI WG? Jonathan: bringing this to TAC, the idea was to see if other projects had this requirement. We’ll work with CI WG for an implementation.
  - Eric E (chat): Wonder if other OSS PyTorch projects have this issue. Andrew: PyTorch is 100% on AWS. They do not currently use GH hosted runners at all.
- Security Threat model analysis for ASWF projects #615
  - Talked to two project leads who seem interested
  - Need to talk with the board to budget this
  - If this go well and the projects get good value, we could have recurring annual budget, and TAC can work on prioritization
  - Comment on GH issue if you have thoughts
  - Would be useful to bring in outside professional resources
OpenAPV #803
- Pre-Read: Commentary on OpenAPV
- Sam: can talk in context of OpenRV and xStudio. An issue we have when building executables is that we need open source codecs we can build into it, have been trying to cheerlead AV1 / VP9 which are excellent codecs but designed for web playback. What we’re missing is equivalent of ProRes / DNxHD which are intra-frame codecs, designed for acquisition or editorial. Not inter-frame compression, very fast to compress / decompress, gave examples of other open source codecs which exist but don’t have a lot of development.
- Potential of Samsung delivering a ProRes equivalent that we could be using that is open source, actively under development is important not just for acquisition / editorial, but also lots of studios do all their in house reviews with codecs like ProRes.
- If there were variants of ProRes that were open source we would use them, but that doesn’t currently exist. OpenAPV has a lot of interesting features, not having licensing issues. As a roadmap issue, we’re mostly missing 4:4:4 encoding, extended bit depth, RGB, those are interesting roadmap items. Could be used during reviews and other in-house uses.
- If ASWF could work with Samsung to take better advantage by embedding OCIO, or work with SMPTE RIS committee to standardize camera metadata. Could capture focal length, focus distance on a per frame basis, could be really interesting. We have a lot of experience on this within ASWF, having Samsung joining ASWF and being active in this project, this could turn into an extremely interesting codec. Our core need of having an open source substitute for ProRes would be very helpful.
- Erik R: as OpenAPV is proposed, it’s just a codec, not a container format, right? Is there an example within a container format? Youngkwon: working on ISO BMF (sp?). Erik: container format would dictate ability to carry metadata, ISO BMF may not have solved this. Want to highlight that metadata is more about container format. Youngkwon: we’ve been working mostly on camera side, our metadata design is more frame based, embedded in the bitstream, data generated by the device. When doing editing / processing, there’s other types of metadata, for instance in MXF. We add to ISO BMF, which is continuously evolving. I’ve been working on that for many years, if it’s not sufficient, we can improve ISO BMF, or look at another binding.
- Erik R: have internal conversations at Netflix about what it would look like to add per-frame metadata to ISO BMFF. Would like to have conversations with you about that. You said there’s medata in the bitstream, is it usually about how to interpret pixel information, like if you have a multi-view source, dolby vision, HDR 10+, what source of data is in the bitstream? Youngkwon: we have compiled list of metadata that can be generated by the camera: what kind of device, lens, characteristics, location / time, we can add HDR related information, it’s that kind of data. Somewhat like EXIF data.
- Carol: from OCIO perspective, would be excited to have a partner to embed OCIO data in a codec. But it would need to work in other codecs as well. It’s more about the container than the codec.
- Sam: colorspace wouldn’t be per frame, but it’s not something that’s well defined within container. So if we can collaborate and establish a precedent that we can use with other codesc, would be a win.
- Erik R: BMFF has the COLR tag, encodes colorspace metadata, but not clear on how.
- Does APV support random access to a frame? Sam: I’ve mostly used it for individual frames, should work to merge into a single file. Erik R: supports tiled decoding. HTJ2K is a codec, not a container format, can put it in .j2c files, but usually put it in a MXF file which provides frame indicies. Can also do tiled decoding.
- John: where is every one at with respect to this? Erik R: there’s a landscape of other codecs in this realm, don’t know that we’ve seen a “bake off”, or specific shortcomings. Some of the specific features stated in this document haven’t been put in a matrix. Do we want to put effort behind OpenAPV, or do we want to try to push other codecs over the line. From my perspective, when I look at it with AV1 developers, haven’t seen really good analysis using accepted metrics in the codec space, and see where it lies on the chart.
- John: proposal is to have it within Sandbox, lots of foundations use that stage to “shake out” alignment issues. If it looks like it doesn’t get there, the sandbox can be the exit stage as well.
- Sam: would be a good use of the sandbox so we can “kick the tires”. There aren’t a lot of alternatives in this area: Cineform and J2K are the only 2 I know about, I’ve been trying to push AV1, but it isn’t great for this particular use case due to slower encoding (and sometimes decoding). Worth testing, have been doing same benchmarks as other codecs, but haven’t generated numbers yet.
- Erik R: curious why we can’t have these metrics as part of the proposal. Shouldn’t be too hard to run “competitive test set”, shouldn’t require foundation resources to do?
- Larry: we may be over scrutinizing for a sandbox project, in the past we haven’t put this threshold of “is it better than all other projects”. It seems like Sam’s report says it could fill an important role, we can give it “shelter” as it grows up.
- John: we could add a 6 month check in for a sandbox stage project.
- Erik R: my main point is that I don’t know where this codec lands at all, and it should be possible to answer the question. The previous projects we’ve adopted were already widely used, this is a bit different, something we would be doing more “zero to one” work, which can be a big lift in the codec space.
- Michael Min: we’ve asked the question of Rez for instance. For instance the OpenAssetIO repo hasn’t been updated in a while, maybe we need to be more rigorous with sandbox projects, but I think this is a reasonable sandbox project.
- John: also something can’t live indefinitely in sandbox. I’ve seen other TACs to use sandbox stage for early collaboration.
- Carol: sandbox is different than incubating, closest analogy might be OpenAssetIO, a brand new project, they are currently building it, and we see the need for it.
- Youngkwon: we brought this to ASWF since we aren’t experts in content creation industry. We know how to evaluate acquisition codecs, but not in the content creation space. But we have something to start with, and would like to work together to improve it. A good exercise for us is to figure out how to compare it. So the concept of sandbox seems appropriate for this. We are interested in having an opportunity to work with ASWF, maybe later on we decide it’s not worth adoption, but would be good to have opportunity to learn and contribute.
- John: this group hasn’t done a lot of sandbox projects, but trying to apply experience from other LF foundations.
- Cary: this isn’t coming out from “out in the weeds”, Sam / Jay / Erik are well respected in the industry, so if they think this is a good idea, it likely is.
- Erik R: from ORI perspective, if the thought is that we need a codec to do “X” (an intra-frame codec that meets licensing requirements and meets requirements), what if we talk a WG from that perspective to figure out what’s the codec that makes the most sense, if we find that it’s OpenAPV (with work), we can use that?
- Sam: I’m hoping that Samsung will be doing bulk of development, but what we’re bringing is the expertise on cine camera and vfx production side. So this isn’t a project from us in terms of coding, but we can mutually gain in a better product through collaboration. Youngkwon: yes, we have lots of expertise on codec development, encoding methods, preserving color, complexity tradeoffs. Sam: the idea that we should be continuing to vet all codecs, that’s a lot of what I’ve been doing with Encoding Guidelines, want to continue doing this and don’t think we should be dissuading people from using ProRes / DNxHD, and they may end up open sourcing? We don’t want to have lots of codecs in ASWF, but it doesn’t hurt to have at least one that we have reasonable development effort behind and that we can embed. Also apps such as Maya could embed it. We need to be looking beyond just RV / xStudio: how does our industry create content, and what’s required to get DCC vendors to include codecs. Maya currently really only has MotionJPEG. Erik R: there’s a gap, I agree. HTJ2K has a lead, already integrated into a set of applications, there’s been development effort behind creating accelerated libraries to get even more performance. Theoretically fills the same gap. Since we don’t have metrics, not sure what it would take to “catching up” OpenAPV to HTJ2K, we might be taking on something that works perfectly fine (HTJ2K) and switching courses from it and fracturing our efforts. HTJ2K already in Resolve and other apps. Also part of IMF spec, some of us will start delivering HTJ2K for distribution. The lead time it takes to proliferate into products, silicon implementations… Maybe that’s something we can do in sandbox phase, but I think it should be possible to generate data for submission.
- John: we see different TACs deal different when projects take competitive position. We’ve seen more and more TACs taking the perspective that if there’s alignment, people behind that can make it happen, then use sandbox mechanism to nurture it and “let the market decide”. Some TACs try to be a bit more forward and say “there’s already an established leader”. But we see TACs being a bit broader and use sandbox as a mechanism to determine this.
- Youngkwon: is the idea to select the “best possible codec”, or work with us on APV in this context? We’ve been struggling on what’s the best way to compare, content to compare, coding conditions to use… So many variables, we’re also interested in learning about this so we can have a correct comparison. We know how to compare consumer codecs, but may not apply for professional codecs. That’s an exercise that we are interested in, and could be first mandate for the project.
- Sam: we can pick a milestone 6 months or a year out to do an evaluation, to really see how APV compares to HTJ2K. Will take time to do both, we may find there’s value in both, particularly with the multi-layer datasets, that’s not in HTJ2K, and could be nice to have. So may have value to have both.
- John: so we could have a specification that would need a review in 6 months? Erik R: we should lay out expectations of sandbox period, that could make sense. After an amount of time, we want to understand how this fits in a landscape of existing codecs.
- John: should we vote on this? Erik: we should define expectations before we take a vote. John: Erik says we should have structure, there’s time sensitivity, do you think it’s OK if the TAC can come up with 6 months vs 1 year, is that OK? Youngkwon: yes, that’s OK, takes a long time to do evaluations, can be interesting exercise to define how to compare with other codecs, if that can be done in 6 months.
- John: OK to do offline vote with 6 month stipulation?
- John: we’ll propose a vote with specific 6 month review requirements.
- David: Erik R, if you can help define that goal, what you want to see, that would be very helpful. Erik R: have dropped a comment asking for 1 specific set of metrics. Sam’s document talks more about other dimensions than just image quality. I can work with Sam on this stuff, we can work together. Sam: yes, we can add to this document or start a new one. We’ll chat and get brief verbiage together. Erik R: as we identify dimensions we think are important, I can reach out to AV1 and VMAF teams and ask for evaluation criteria we are interested in. The AV1 and VMAF team is keep on what’s important to content creators. Youngkwon: that sounds very interesting.
- John: we’ll get the vote rolling, Erik and Sam will collaborate on 6 month review point. Is that right? OK?
- Kimball (chat): am keen to hear how the group thinks we should balance requests like OpenAPV vs. the (current) request to add jp2k as a compression type into OpenEXR… love seeing more open standards…
- Eric E (chat): Does HTJ2K support random access to a frame?
- Eric R (chat): Personally - I think this metadata stuff all should be in the container, not the codec.
- Rob (chat): I just built openapv for the first time and have a tech question. Who do I contact to follow up?
- Spencer (chat): re licensing/patent claims an issue. AV1 has two patent pools that want to collect licensing fees.
- Carol (chat): The closest analogy is probably OpenAssetIO. Sandbox is different than incubating. Eric R: Oh, good point Carol
- Michael Min (chat): HTJ2K is open-sourced under Aous Naman? Erik R: That is one of the open implementations - however the spec is widely published. Michael M: Yup! Aous’ link is on that page, under an example implementation
- Michael M (chat): DPEL’s ASC StEM2 asset has reference media w/ ProRes/DNxHR, and IMF/HTJ2K that could be used in the comparison