How far can you take OpenFX?

Amos Wenger: So I've been up all night. No, that's not true. Trying to figure out what to bring to show the class. And I thought it might be fun if we did the freaking news, but I didn't. I don't know. It kept me up. So I opened my iPhone Notes app just to list all the things I could talk about. And I decided to use the research that I did yesterday to kind of close the arc of me going deeper and deeper into rabbit holes when it comes to making videos the silliest possible way. James, are you excited?

James Munns: I am. I'm glad I was using Bluetooth headphones because I did wander away to get my fidget toy, but I did hear the whole intro.

Amos Wenger: Good. Good, good, good. So today's episode is called How Far Can You Take OpenFX? Subtitle, can you embed a full motion design suite with it? But I would like to manage everyone's expectations, including my own. And it's that I only put like one day of research into this.

James Munns: For folks who haven't listened to your last episode where we were introduced to what OpenFX, do you want to remind the audience what OpenFX is and where you start this rabbit hole dive in?

Amos Wenger: James, ye of little faith.

James Munns: Well, I've got to ask for the audience

Amos Wenger: Previously, on Self-Directed Research.

James Munns: Oh, love it. Love it.

Amos Wenger: So first of all, as always, you can get the slides on SDR dash podcast dot com slash episodes or by watching the episode on YouTube or Spotify. We have video on Spotify. And for those of you who don't know, I make YouTube videos. And on the slide, we can currently see what my videos look like. It's mostly a talking head, mine, in front of code. Sometimes there's a green screen involved, but we don't need to get into that. The important part is how do we render that code and how do we like animated and highlight parts of it and whatnot?

And I've gone through several different techniques to do that. It all started with screenshots because the code blocks are already on my website. Right. I my website is also all custom. It does syntax highlighting. So they're already rendered exactly the way I would like to present them on my website. So ideally, I just take that and put that in the videos. And that's what I used to do is just use the Mac OS screenshot tool and do a rectangle screenshot and save that somewhere. Rename that to something meaningful. Make sure I don't lose it because that's the thing with DaVinci Resolve projects The cloud is a relative concept.

And yeah, it just took a lot of time and it was hard to position them properly. If you go from one slide to the next, if you if you did the rectangle screenshot slightly different from the one before, it's funky. You can definitely see it in the transition from one to the other. The controls in the venture resolve to set the position and size of the text are very weird. I don't even know what they're based on, honestly. And, you know, it was not good. So I started looking for solutions on how to make this easier because I usually write an entire article first and then make a video out of it. So I have everything. So I can just open the article in a browser like Safari.

And I noticed that in Safari in the dev tools, well, they call it the Web Inspector. But you can make transparent screenshots. And then I in the automating macOS episode of the season, we discovered that we can use Apple scripts to automate Safari, which means you can make those transparent screenshots automatically. And for one of the videos I had, like, I don't know, like 200 of them, blocks of code and diagrams and whatnot. So, you know, it gets very long to do it by hand. Which one was that? I forget. Honestly, I have. OK, I'm lost in my pipeline.

James Munns: To be honest, you probably could have said anything and it wouldn't surprise me.

Amos Wenger: Yeah, I started making shorter pieces because it's just not nice to have a 50 minute video to edit, I think some people here can relate.

James Munns: Now you're considering the niceness of your editors time. And so that just being your time, you go, well, I would I wouldn't inflict that on anyone, let alone someone who works with me.

Amanda Majorowicz: Yeah. I appreciate it. The past I don't know how many episodes have been like,

James Munns: said Amanda, who's been editing 50 minute episodes. Don't worry, Amanda, this one will be a short.

Amanda Majorowicz: Yeah, exactly. Only 44 slides or what is it?

Amos Wenger: Yeah. Yeah. Yeah.

Amanda Majorowicz: Nice.

Amos Wenger: I've aligned incentives for my videos because I paid my editor by minute of edited video.

James Munns: Nice.

Amos Wenger: So I have an incentive to make them short. And then we discussed the rate and evolve and evolve it over time. But so that's what we discussed in automating Mac OS. You just use Apple script, which was very fun. There's on the slide. If you're watching the slide, you can see that there is not a mono space font, but a serif font. Actually, is it serif? I don't see serif. Is it sans? Whatever. It's that it's proportional for sure, which is the opposite of monospace.

Amanda Majorowicz: It's sans serif. There are no serifs on this font.

Amos Wenger: I thought maybe the the lack of sleep made me the sans disappear. The serifs disappear. And another as the episode named Blackmagic Fusion is weird. I discussed my new and current setup. I've already reached a video with it. I forget which one, which involved reversing the Lua based formats that Blackmagic Fusion uses for node graphs and just kind of generating what it expects and generating a Text+ node with the character level styling modifier and doing syntax highlighting that way.

And I even vibe coded a little preview on my website. So you press a button and then it opens this editor that uses "contenteditable" So you can remove the parts that you're we're not talking about in the video right now, you can like select a bit of text and press space. And it actually highlight this and dims everything else. It's really, really nice. It has a stand in for my face. I look super goofy again. Really, you should check the slides. That's half the fun.

James Munns: This is disappointing because this is one of the few website plugins that only works locally and you can't trigger it remotely. Because otherwise I'd love to like edit your page and then slap your face in on examples and be like you are making your own meme templates except for it's.

Amos Wenger: I need a new patrons here for like like green screen pictures of me. Yeah, sure. Yeah, exactly. For evil people. I just saw yesterday on Reddit a post where some burger place lets you pay. The Veruca Salt special. You can pay a thousand dollars to get in front of the line and just. I want it now, Daddy. I think that's what it said.

James Munns: Exactly. Yeah.

Amos Wenger: And I think that's something people should consider, like making tiers that are higher. So yeah, that reminds me of also someone said like on Twitter a long time ago, you know, when Twitter was still Twitter, they said like if someone were to pay me like 5000 a month, I would just stop posting indefinitely. And that's true.

James Munns: I do. Who was I'm going to have to go back and find that because I definitely remember that of the like it's someone we know.

Amos Wenger: I don't I forget whom.

James Munns: Yeah, it was just I'll shut up, you know, for that check keeps clear. And I exactly I'll just shut the hell up.

Amos Wenger: And it worked out beautifully on my blog. I have my articles that are exclusive for six months. Someone was like, hey, it's bad that exclusive for six months. And I was like, OK, you want to you want to pay to change that? I was like, OK, that was Depot, the sponsor of this episode. So a big thanks to them.

James Munns: Yeah, I was going to say it was our lovely sponsor, Depot, who is I know, who is sponsored this entire season and that article.

Amos Wenger: Very generous and very cool platform. Check them out.

James Munns: Yeah. Listen to the end of the episode.

Amos Wenger: Yeah. Back to my short presentation in the Black Magic Fusion is Weird episode. It's kind of a misnomer because it has this whole nice solution of generating. And I also talked about the other ways in which fusion can be extended, including fuses, which we learned come from the same word as fusion. We weren't sure during the episode, but in the in the show notes that Amanda put together, I saw that the etymology says that they are from the same word.

James Munns: Were they consistent? I saw one was like a bundle of twine, like "fuset" I think it was a French word. And then I don't remember where fusion came from, but

Amos Wenger: "fusée" is a spaceship like a rocket ship.

James Munns: Oh, yeah. Maybe I'm probably pronouncing the French word very, very wrong.

Amos Wenger: But yeah, my brain is too mushy to figure out which which French word you mean.

James Munns: We'll leave etymology for another day. Yeah.

Amos Wenger: So fuses, which are a bits of Lua that you can hot reload and everything. It's a very nice development environment, which are fast enough because it's using a jit It's actually a LuaJIT Whenever people say Lua, they usually mean LuaJIT not always, but usually. And and OpenFX, I briefly touched on OpenFX. I was like, OK, it's a standard way to do video plug ins a little bit like AU and what was the one VST for audio? But this one is for video. And I said, you probably could bring in your own renderer and do exactly what you wanted.

Yeah, one of the limitations we ran into by using back magic fusion's own text renderer is that it does not support color emoji, for example. And it does not support font fallbacks. So part of the difficulty for me in generating the graph is that if, for example, there is a nerd font symbol, it has to specify that this pan is using the nerd font symbol font. Everything else is in Berkeley Mono except for emojis, which are using Noto, Noto without color.

James Munns: monospace or the monochrome.

Amos Wenger: Yeah, the monochrome.

James Munns: Yeah. And I went back and looked it up. And the word I thought was toast is actually tofu. And that's where the Noto Sans name comes from is "no tofu". Because it defines all of the symbols for all of the emojis. So you'll never get the tofu, which is that little rectangle that's either an exclamation point or has like the hex characters in there. I don't remember why it's called tofu, but that was the word I was trying to think of in that episode.

Amos Wenger: The Unicode replacement character.

James Munns: Yeah. Yeah. Well, there's maybe two. There's one. Wow.

Amos Wenger: No tofu.

James Munns: Yeah. But tofu was what I was thinking of when it shows you the like, I don't have the glyph. Here's some hex codes. Do you like that?

Amanda Majorowicz: I like learning about all this stuff, like going through editing the episode, collecting all the links and then being like, I don't know what you were talking about. And now next episode, learning about last week's episode. Very cool.

Amos Wenger: I think you should feel free to cut in like add editors notes in the middle of episodes. Like I checked it. It's not true or something. They're still wrong.

Amanda Majorowicz: I was actually thinking about that. Yeah. But you know, who knows? Maybe in a future episode. We'll see.

Amos Wenger: Yeah. So yeah, I mentioned OpenFX saying, yeah, you can you can make native plugins. You know, it's clearly overkill. Obviously, I'm not going to touch that. That would be crazy. So that's what I did next.

It's time... for self-directed research

James Munns: Yeah, of course.

Amos Wenger: Yeah, of course. Because because it's

James Munns: stuck in the back of my mind. There were some hints in there where you're like, oh, there's a crate. You know, I definitely exactly don't need that. I'll I'll definitely use the Lua for that. And it was one of those like, oh, no, it's this is Chekhov's gun in the last episode.

Amos Wenger: Check Chekhov's crate. Yeah. That's what I was writing the slides. I was actually thinking of that line. Yeah, there was a crate for it. So how could I not look into it? Of course. I think the audience deserves that. I think that's what we're all here for. It's to watch me suffer. I was kind of immediately disappointed because the crates ofx-rs is written against nightly rust from six years ago. I didn't realize that I thought it was stable rust from six years ago, which would have been totally fine.

I think maybe maybe a couple of fixes, but like unless something was wrong in terms of soundness, I think we would have been fine. But nightly from six years ago, I was using two features that just have changed or not there anymore, whatever. I got over 200 errors trying to compile the example code. So I thought that would be the end of it. That I told you it would be short.

James Munns: Yeah, I'm trying to like 2019 rust. It's not as far as it could be. We're like post 2018, which is a pretty big step. But yeah. Yeah. Six years of nightly is a stretch. Yeah.

Amos Wenger: But then I found NTSC-RS, which is a port of NTSC-QT to rust and SIMD by Valadaptive. And among other things, it's 20 times faster. Yeah, it uses SIMD and concurrency and stuff. It's an analog.

James Munns: They have a very pretty website.

Amos Wenger: It does. It's a VHS effect for your video filter, essentially. And it uses what does the website say? Uses algorithms that model how NTSC transmission and VHS encoding actually work. I imagine that is the next word.

James Munns: Ooh, someone was asking me about this on BlueSky because there's shaders that I've seen for terminals that give you that like CRT glow and stuff like that. But I'm not surprised that especially with all of the back rooms and like cassette futurism stuff that's going on, that there's not like super shaded CRTs and VHS things going on these days.

Amos Wenger: True. True, true, true, true. That one is all CPU as far as I can tell, but it's I don't know. I think it's an old effect. Yeah, ported to rust. And the code place is very versatile because it has a standalone application with egui controls. I don't know if you recognize the egui-ness of the of the GUI.

James Munns: Definitely.

Amos Wenger: Yeah, it could be. What's the other one? What's the C++ one? The one used in all the video games. Imgui. Oh, yeah, Dear Imgui. Yeah, Dear Dear Imgui. Exactly. I could be that, but I spend too much time with egui making demos. So it's available as a standalone. It's available as a CLI. It's available as an After Effects and Premiere plugin. And as an OpenFX plugin.

James Munns: And are we getting back into those plug-in world like we were talking about with the audio ones where it's like you have purchased a DLL. Congratulations. It runs in your environment. Kind of. Is it something much more specific to this platform?

Amos Wenger: Kind of. Actually, so NTSC RS is open source. Okay. So actually they use cargo-xtask to build the plugin, but it's just so it can generate that little bundle, which looks like a Mac OS app bundle. It has a contents directory with an Info.plist in there, a MacOS directory with a .ofx, which is a shared library, resources directory with an icon of the plugin. And the info.plist is just XML. There's one of the plist formats. There's a binary one and there's a third one, I forget. Like text but not XML. I don't know James, does that ring a bell?

James Munns: Yeah, I gotcha. It's exactly the same kind of thing that I had to put together for the Postation bundle and for when you're shipping a binary on Mac OS. So it definitely rings true.

Amos Wenger: Yeah, it looks a lot like it, but it's not .app. It's .ofx.bundle. It's weird.

James Munns: It's a folder that's pretending not to be a folder.

Amos Wenger: And yeah, so the reason there's an xtask, which is kind of turning Cargo into make, you know, you can have different targets and it's "cargo xtask build-ofx-plugin" is because they want to make a universal build. I don't know if that's built into rustc yet. Nowadays, you make a binary for x86-64 and for aarch64, but in the past you did it for like, well, Intel 32-bit and Intel 64-bit, you put that in the same binary or a PowerPC plus until 32-bit, even further back. So like any kind of transition that Apple has done, like Mac OS has done, you could have a transition period where you've shipped both binaries in the same thing.

Visual Studio Code shenanigans

Amos Wenger: And it's still happening today because not everyone has Apple Silicon Mac, an ARM Mac. If you go on Visual Studio Code webpage, for example, there's a single download for Mac OS button. But if you download that and you use lipo-info, then you see that there's two architectures in there, x86-64 and arm64.

James Munns: I think even, I can't remember if it was Android or iOS, like for the iPhones and things like that, where they were doing that as well, because there's different generations of the processors that they use instead of doing the like, hey, we're compiling for the oldest compatible X86 version and things like that. There's the option to do multiple compilations for different target platforms so that you can get either like the operating system platform level or like the CPU for Apple where there's much more control over like, hey, this works on these versions of phones and later, which means you have all of these processor extensions available for more performance and stuff like that. I wouldn't be surprised if both of them were doing that, but I guess it's just how you do the packaging and the app store handles it.

Amos Wenger: It's possible, but given that these would be mobile apps. This is an aside, I will, yeah. I think the equation changes a little bit because who cares if VS Code is a few megabytes larger, but on the phone, I don't know. I know there's like SDK target levels for sure, and I know those are hard to handle properly, and those are also a thing on macOS, but yeah.

James Munns: But yeah, universal binaries. I remember them, like you said, from X86 and ARM, but they continue to be relevant.

Amos Wenger: If you want to save the bytes, you can go to that detailed download page and have like the Intel or Apple Silicon build separately and all the, oh yeah, for Linux, you get the ARM32 builds for like, what? What's ARM32 these days?

James Munns: Original Raspberry Pis. Raspberry Pi was 32-bit for a long while, and then it did a very weird thing where the kernel was 64-bit, but the user space was 32-bit, which was like a compat thing so that they didn't

Amos Wenger: have to-- Because of RAM usage? Like 32-bit pointers or, okay.

James Munns: For compat, like if people had, yeah, so if people had compiled some binary application for 32-bit, you need a 32-bit user space to continue to run that, because otherwise you'd have to recompile to run on a 64-bit user space, but you might want your operating system to be able to handle more than the two, or be able to handle more RAM, or at least have extended RAM range, even if it's virtual memory range. So for a while they were doing that. I think they're all 64-bit now, but yeah, I know Raspberry Pi was like the 32-bit ARM that everyone had and was popular for quite a while.

Amos Wenger: Yeah, it's funny. If you look at Windows or Mac, Mac is just Intel chip, right? It doesn't even think about 32-bits, and Windows, it's x64, x64, Arm64. They're using the terminology for each platform, which is nice. Anyway, that bundle structure that I just showed is the same on other platforms, because it's a very macOS thing to do to distribute an app as a folder and call it a bundle, and have it show differently in the finder, but actually it's just a folder if you look in the command line.

But as far as I've been able to find, because I've been looking around the OFXRS directory, repository, even if I wasn't able to run it, I saw, I looked at how they were doing things, and they are making a content slash Linux dash x86 dash 64 directory, which is, it's a Franken bundle. Like it's weird, because it's macOS semantics on Linux. I don't know. It's really weird to me. Anyway, NTSC RS does work. I was able to build it from source and import it in DaVinci Resolve, and it works everywhere. It works as an effect. You can drag onto a clip in the edit page. It works as an effect in the color page, and it works as a node in fusion, which is absolutely lovely. You can see on the slide is a me on a green screen with a NTSC effect on a 480p timeline, because that's what they recommend doing. Even though it can run at higher resolutions, it's not supposed to have more than 480 lines.

James Munns: You get sort of that like anachronism of like, here's my 4k image with scan lines, and you're like, that I don't think anyone was doing that. There might've been raw captures of that, but you certainly weren't broadcasting it more than 480 lines. It's got the vibe of it. It's very like found footage or like someone sent in a VHS of themselves in 1990 to some technology show to it. You've definitely gotten that vibe out of the effect.

Look at all them parameters! NTSC-rs

Amos Wenger: And there's a ton of the parameters you can, a ton of controls you can change. I've put side by side the parameters as seen from the venture resolve and the parameters as seen from the standalone E-GUI interface. And the code base is just really well-designed because they have a single way to describe all the parameters and then it's mapped to whichever interface because they have essentially three different GUIs, right? They have egui and standalone, they have Premiere and After Effects, I think are roughly the same. I have a lot of commonalities.

And then OpenFX. And OpenFX has different kinds of hosts like DaVinci Resolve or Natron or like any sort of VFX program that can load OpenFX plugins is called the host. It provides functions and provides these, the UI, the parameters like loading settings, saving settings, et cetera. And also you can see on every parameter here, there's a little diamond that's for keyframes. So OpenFX plugins don't need to know about keyframes. They just need to, for every render request, they just need to get the parameter for that time. And then the host is in charge of making the values change over time in the timeline and like interpolating things and stuff.

James Munns: Hello, Amos. Yes, as someone who does not edit a lot of video, what distinction does keyframes make when it comes to applying an effect? Is the effect not just a stream of images that it's modifying or is there intelligence going on?

Amos Wenger: Sure, but you see intensity here. You can make it change over time. So keyframe is just like, it's this value at that frame and then you wait 30 frames and then change it. And then it changes linearly or like maybe there's a little tweening happening between the two values. It's just animation basically. Gotcha. So yeah, let's keep walking through like what is actually in an OpenFX plugin. There's a well-known struct. It's exactly what you would expect from a plugin API. Essentially there's well-known structs. You export two important functions.

One is, ofxGetNumberOfPlugins which in this case returns one and it's not shown on the slide. And ofxGetPlugin, which returns a struct. What's interesting in the OpenFX API is that they use a lot of string constants everywhere. Sorry, the first thing in the struct that you return is just a magic string. It's a "kOfxImageEffectPluginApi" as a string and that's it, well, without the K. And then there's API version, which is good. That means they can actually evolve it. Well, it's set to one in this case, but they can just like add or--

James Munns: Which means it's never changed.

Amos Wenger: Yeah, but they could like change the struct. They could interpret the struct as having this header that never changes except for the API version field. And then based on if it's one or two or three, like read different fields after that, like sort of like a rust enum.

James Munns: Yeah, these are fun things for like binary compat, like, cause you run into it in protocols as well as like ABI things of, you need to just read the first two fields and then you promise that those never change. But that second API field, like you said, it acts like an enum where it's like, ah, do I actually know how to interpret the variant that it's claiming? Because if it says two and I have no idea what two means, I should stop reading because things are about to get very, very weird.

Amos Wenger: Just throw an error.

James Munns: Yeah.

Amos Wenger: Yeah, instead of getting memory corruption. Next up, we have main entry, which is directly one of the fields of that struct that we return. And main entry is kind of like the main, kind of like DLLs on Windows also have a main entry point. It's just funny to me because it gets a string, a C string and you compare it against a bunch of constants and then you know if the action you're supposed to do is load or describe or get regions of interest or whatever or render.

So that means every frame you render, every tile you render, there's a multi-threading thing that you can ask the host to do. You can ask the host to like divide the image in different tiles and ask you to render tiles in different threads and whatnot. You're just a poor shared library. So your function can be called from different threads. The first thing you do is just compare against like do a chain of ifs. It's like, okay, it's not load. It's not described. It's not describing context, blah, blah, blah, blah, blah. Finally, I was like, oh, it is render. So either string comparisons are a lot faster than I imagine or that does not that worried about performance. I don't know, James, how do you feel about this?

James Munns: I mean, computers are fast. Like, yeah. True. I mean, a match, like it looks grosser than a match, but a match is essentially just doing the same thing. Like you can do much smarter string compares of like making either like a regex style automata or what are they called? They're not just finite statements.

Amos Wenger: Sure, but why not just export constants from the library or something? Like define, I don't know, the U64 or whatever in file formats, like MP4, we have those atoms which are just 32 bits, like four. Yeah, four times eight is 32, just four characters. So you have like MOOV, you have MP4A, whatever, that's what all those things are four characters long. It's so that you get a nice 32 bit constant out of it.

James Munns: I assume 32 str comps are way cheaper than whatever video effects you're about to be doing, honestly. So like order of magnitude, they might go, "Eh, it's so we can debug it easier." Cause then you could just print instead of having to do a lookup or whatever when you print logs of like, "We were told to do this." True. I don't know. I think you're right. It would definitely be much smaller, but I think probably one of those order of magnitude things where it goes, "Eh, whatever, we're about to do a bunch of DSP, so whatever."

Amos Wenger: Well, what I also like is that in this, in NTSC-RS specifically, it calls set hook, set panic hook. And I have no idea what the cost of that is. Like I have not measured it. I've measured other parts because the plugin was slow at some point, I did some performance measurement. I did not measure how long it takes to set the panic hook, but I imagine, I don't know.

James Munns: I don't think a huge amount of time. No, but- Yeah, I have no idea either off the top of my head. This is running in std, which I don't do most of the time. Or at least I don't think too hard when I'm running in std. Like I think hard when I'm running in bare metal, but yeah.

Amos Wenger: So next up, if the action that we're supposed to do is a render, then we get the address of the functions from the host that we're supposed to call, like propGetString or propGetDouble, et cetera. Then we call that and we get all the properties that we want. Again, mostly with string constants, like the propGetDouble, kOfxPropTime.as_ptr() That's just the string. That's just "OfxPropTime". And also we're getting it as a double, which is an F64, but it's actually just in DaVinci Resolve, that's the frame number. So it's fully an integer.

James Munns: speed, summer games done quick was just happened, which means everyone was talking about basically all like speed running and glitching and stuff like that. And there's a whole thing of like, what goes wrong when you use a floating point number for your frame counter. And it was basically like 16 million is where just add one breaks down because it rounds down instead of up, which means you just hard lock at 16 million frames.

Amos Wenger: So 16 millions at 25 FPS.

James Munns: Oh, it's like months. Like this was a Mario 64 glitch that was like, look, this portal stops rippling after four and a half months, but this one stops after a month and a half because it increments by two or something like that.

Amos Wenger: It reminds me of the Windows 95 thing where if it stays up for 91 days or something,

James Munns: didn't do this. You get this in planes and stuff. When you have-

Amos Wenger: It overflows.

James Munns: It's either a millisecond or a 10th of a millisecond. Whenever you see something that says it needs to be restarted every 50 days, or it crashes every 50 days, that's like, yeah, it's either every millisecond or a 10th of a millisecond in a U32 counter. Or like, yeah, you start to notice those numbers. Like this is one millisecond, 10 millisecond, a hundred millisecond, one microsecond of like those times where it's like, yeah, it just crashes if you run that long and it means that they didn't handle overflows right.

Amos Wenger: Well, to be fair, it can be hard to handle that properly. 16 million frames is 25, according to o3, which I trust with reservations, it's seven days, nine hours, 46 minutes and 40 seconds. 16 So if you make a student film, that's just really boring in DaVinci Resolve. You might get into issues at some point. So you get all your, you get the addresses of the methods you need to call from the host, and then you call those and you get all the properties, all the parameters that I've shown earlier, like intensity, frequency, detail, snow, in the case of NTSC RS, but it could be anything, right? You can have anything in the UI, it's pretty flexible.

You also get the input signal, the output signal, output buffer, where you're supposed to write the thing, the result of your effect being applied. The host tells you which part of the output buffer you're supposed to touch. It tells you the buffer format, if it's RGB, RGBA, if it's eight, 16, 32 bits, if it's floating point integer. I was curious what code path actually got used because NTSC RS handles all possible combinations.

How is developing an OpenFX plug-in?

Amos Wenger: So how is developing an open effects plugin? I wanted to find out what was the actual buffer format. I wasn't planning on supporting everything. This is research level, we're just trying to get it to work for one specific configuration, like only on Mac OS, only on one host. And the first thing that hit me is how do you log? James, do you have any idea?

James Munns: How do you log?

Amos Wenger: How do you log? How do you print things?

James Munns: Okay, so if I had to guess, there's...

Amos Wenger: You send a postcard-rpc messag—no.

James Munns: It's probably not far off. I mean, what I would assume is in the plugin, you get passed in some vtabl that has methods that you're allowed to call. Or there's some like global or something like that.

Amos Wenger: But no.

James Munns: Yes, but no.

Amos Wenger: There's absolutely nothing standard.

James Munns: Oh, love it, love it.

Amos Wenger: There's no OpenFX blessed way to do logs. You can write to standard error, but then you need to start your host from the terminal, which is annoying. Resolver has its own log, which you can't touch. I'm assuming you can write it from fuses and they also have their own SDK to make plugins that are not open effects.

James Munns: Are you gonna tell me that you spawned a Lua interpreter just to print messages using some kind of Lua trampoline or something like that?

Amos Wenger: No, I only had a day, James. I didn't do that. No, no, no, you're seeing solution in front of you, which I just opened the file and just appended to it.

James Munns: Yeah, okay, yeah, just no sandboxing, just straight to temp. Yeah, of course.

Amos Wenger: Sandboxing, James, your concerns are very far from the concerns of VFX people, I think.

James Munns: Oh, it's fair, it's fair, yeah.

Amos Wenger: Another question, how do you reload your plugin? Any ideas? You panic. No, if you panic, I'm pretty sure it takes down the host.

James Munns: Okay.

Amos Wenger: Well, yeah.

James Munns: Oh, great, yeah, that's great. Write no sandboxing.

Amos Wenger: Yeah, this is not WebAssembly, this is not, you know.

James Munns: How do you hotload or reload your bundle? Exactly, well, you don't,

Amos Wenger: you can't. I'm showing you now the video plugin settings pane and you can do, "pain" is right, and you can disable all, enable all, but all it does is just show you the plugin and the list of things you can use or not. But once it's loaded, it's loaded, right? "dlclose" is a lie in pretty much all platforms. It always leaks things. There's no cleanly unloading something. In Rust, if you load a library, it returns to static lifetime and for good reasons because you really shouldn't unload things that you've loaded.

Even if you could like cleanly do it, apparently in C++ you can cleanly do it. Even if you could do all that and you're initializing metal kernels and like shaders and whatever and those can't necessarily be torn down pretty like, I don't know, I don't know the details, but essentially you cannot. So the way to reload it is to close the Vintra Resolve and then reopen the Vintra Resolve and then reload the project. Do you have any concept of how long that takes? Uh, like? I've timed it. How long to shut it down?

James Munns: Okay, I know vaguely what it is for the self-directed research, but we do no video effects or any, well, I mean like we have slides, I'm sure, minutes?

Amos Wenger: No, no, thankfully, no, I have a very nice M4. Wait, is it taking minutes on your end to open the project?

James Munns: No, it takes like 15 or 20 seconds for this, but we also aren't loading any video.

Amos Wenger: Yeah, exactly, yeah, yeah, that's more in the ballpark. What gets me is that it takes 10 seconds to shut down before you can even start it up again. That's annoying.

James Munns: You know someone just has a sleep that they're like waiting for something to do and they're like, whatever, it's a bunch of sleep(1) or sleep(3) in there and it just, it becomes serialized and it takes 10 seconds.

Amos Wenger: I think it's all the shaders and stuff, whatever, being unloaded or something. Actually, oh, Jesus, that's another nerd snipe. But Instruments could probably tell us what it's spending its time on.

James Munns: Come back next season for debugging the plug-in interface of this, but yeah.

Amos Wenger: No, this is the end of the arc.

James Munns: That is one of those fun things of the thundering herd of destructors is a not trivial performance problem in a lot of places of just, oh, we're gonna run all the destructors now and it turns out that takes a long time, but actually just killing the program would have achieved the same thing 99% of the time, except for now it takes us 30 seconds to close the program.

Amos Wenger: So because it takes a long time to shut down DaVinci Resolve and open it up again what other people do is that they use another host like Natron, I don't know how to say it in English. I know it's a French software. It's scriptable with Python and you just write a little script that sets up a node graph and renders everything and saves an image.

James Munns: Now, Amos, I can excuse a Lua, but a Python?

Amos Wenger: I told you, it's also doing Python in DaVinci Resolve.

James Munns: Sure, I remember what episode it was. There was definitely some episode where I said something about Python and Amos was like, I can excuse a Forth, but a Python? Oh, I think it was talking about bootstrapping.

Amos Wenger: But there's no aarch64 version of Natron or Natron or something. When I launch it on my computer, it just immediately crashes. Well, not immediately, that's what's puzzling. Like most things immediately crash. It opens, it shows a little window, waits a few seconds, it's loading more stuff and then it crashes.

What about render?

Amos Wenger: At that point, we have talked about OpenFX, but we haven't really talked about how do we actually do the rendering, right? We wanted to improve on the text rendering with Blackmagic Fusion. How do we actually do that in Rust? James, I don't know if you're familiar with the font rendering ecosystem in Rust.

James Munns: No, no is my answer. I know the people who are doing it a lot of the times, but I am not familiar with it.

Amos Wenger: You do? Well, okay.

(Laughs)

Amos Wenger: That was a hell of a reaction. Okay, I wanted, what I was thinking when I blurted out was okay, was like, you choose to thank them, they're doing great work. But what came out was okay.

James Munns: Oh. Oh. Well, I know some of the people. I don't recognize Mooman219. If it wasn't Rafe, then I probably wouldn't have recognized who it was.

Amos Wenger: The one that I found that is pretty simple and that kind of replaces rusttype type and ab-glyph and glyphbrush and glyphbrush-layout, rusttype being like, I think an early port of free type to Rust. Pure Rust alternative. I'm not actually sure if it's a port.

James Munns: I'm just excited that it's no_std because now I need to, I want to go mess with it and see if I can use that for font rendering because that's a fun thing in embedded where a lot of the times you pre-bake your fonts. Like you rasterize them to specific font sizes and then you just--

Amos Wenger: Yeah, a lot of this stuff is actually no_std

James Munns: Take the glyphs and stamp them in.

Amos Wenger: Yeah. Nice. This is how far I got with my research. I integrated fontdue, like I kind of vampirized the NTSC-RS plugin and I got it to render the number of the frame in the output. At first upside down and then finally right side up.

James Munns: Is this one of those like is zero top left or bottom left or bottom right? Who is it that has the excellent someone we both follow on Twitter and then BlueSky has all of the graphic, are all like the game engines and stuff like that?

Amos Wenger: I think it's Freya Holmer.

James Munns: Oh yeah, exactly. Exactly, it's Freya of like, what is X, Y and Z positive, negative and where does the zero index start on all of those on 2D and 3D modes? Cause everything's different.

Amos Wenger: But that's the fun thing is if there's several ways to do a thing, if there's several possible conventions, all of them are in use somewhere. But Unreal is actually changing that I think. Oh really?

James Munns: Okay.

Amos Wenger: Yeah, they're switching to something else. I don't know, they're joining it or whatever. I saw the announcement, I was like, what, why? It's gonna break everything. I'm pretty happy with that and it runs in almost real time. Apparently getting all the parameters that I wasn't using was what was taking the most time. And I got it down to like for a 4K frame rendering between three and four milliseconds, which I thought was pretty nice. I thought we had like a, for 60 FPS, you get a 16 millisecond budget.

James Munns: Yeah, 16.6 milliseconds of frame, yeah.

Amos Wenger: Exactly and I only need to do 25 FPS. So I figured we had plenty of time but it's not running smoothly.

James Munns: You know what that means? You know what that means, Amos, right? What does that mean? You were using the NTSC plugin at a PAL framerate. Because NTSC is 30 or 59.97 and PAL is 25 or 50.

Amos Wenger: But my footage is, I live in Europe, okay? Leave me alone.

James Munns: But yeah. Yeah, it's like the slow-mo guys episode where they explain why they want the lower frame rate and it's just why more frames doesn't actually help.

Amos Wenger: Yeah, for some reason, even with it taking three or four milliseconds by frame, it's still not running smoothly. Like playback is not smooth. Fortunately, you have a playback resolution kind of fixed. You can run at half resolution while you do the editing and then while you export it full resolution. But it's still kind of not great because this is just one line of very basic text. This is why I was wondering about this, setting the panic hook because maybe that's taking a few milliseconds here and there, you know? Probably not much but...

James Munns: Yeah, I was just saying, if you don't have logging, then doing timestamps of things is gonna be egregious. And then, like you said, trying to do profiling for a DLL that you didn't start, that someone else is starting becomes dot, dot, dot fun. Especially if the Egooey, like if that raw binary setup does things that are just intrinsically different than the inside of DaVinci Resolve kind of things or inside of Fusion or whatever. Huh, that sounds fun.

Amos Wenger: Also, the other issue is that this doesn't actually improve on the renderer from Fusion because it does not support color emojis. It just supports like the basic free type stuff. Look basic, I don't know. Look at me dismissing their hard work. It's still like an amazing feat. It's only basic, yeah. But it supports a ton of stuff but not color emojis. Swash supports more things as far as I can tell but it doesn't do layouting. If you wanna do layouting, you need something like cosmic text. And cosmic text actually works really nice but I tried the demo and it does render color emoji. I was impressed. Like, oh, does anything in the Rust ecosystem support color emoji or is it only a thing that browsers can do? No, no, no, you can render emojis with cosmic text, the Mac OS color ones. Apparently the color emojis format for Mac OS, Windows and Linux are completely different because yeah, that's why we can't have nice things.

But the demo of cosmic text was actually pretty slow. Like it's rendering a bunch of text and you can select it. It's kind of an editor. But yeah, on this pretty good computer, I expected it to be really fast, like running at 60FPS or something and it did not. So I don't know who's to blame here. I don't know what's happening. Cosmic text is probably great and fine and this is not its fault. But text rendering is just more expensive than I thought. I guess the trick is to not do it every frame if you don't need to and like cache as much as possible. And I don't know, like, you know, at-laces and whatnot. And the demo is just about correctness instead. I have no idea. But at this point, I started wondering, well, I have demos on my website which use more than just text. Like what if I could just render a whole browser in there?

Instead of messing around and having just text and text renderers and Rust. Okay. When it comes to embedding browsers, there are several options. One of them is UltraLite, which is a stripped down build of, I think, WebKit, which means not Chrome, but like Safari with like JavaScript core and whatnot. It's this pure GPU HTML renderer. I'm pretty sure that's, I don't know how accurate that is. It's GPU accelerated, I'll acknowledge that. There's SDKs for Rust, non-official ones, but feasible. But I took a look at the pricing. James, what do you think this costs? Wait, browsers cost money? It's not a browser. It's an embeddable pure GPU HTML renderer. HTML renderer?

James Munns: Contact us for pricing?

Amos Wenger: No, it's a little better than that. So there's a free tier up until 100K of revenue per year. And then there's a pro tier. Okay. You can pay something yearly. Okay. And it's 3K. So some years flirt with 100K.

James Munns: Wait, so where does the browser come into this? Where browser? We were talking about video editing. Where browser?

Amos Wenger: Where browser? UltraLite lets you open a webpage pretty much and render it in a GPU buffer.

James Munns: You'd have a plugin that does browser rendering, so you'd write JavaScript. So you're running a scripting engine inside of your plugin with a rendering target, and then you're going splat into the image and then like, you know, rasterizing that into the green screen or something?

Amos Wenger: Not sure I follow your explanation, but yes, the idea would be to-- It's probably wrong. To have, instead of rendering text directly, you would render web contents. And that way, those motion design tools are available. I forgot their names, but you can write TypeScript code essentially instead of doing things on a timeline and say, okay, this element comes in, this element rotates, this element's opacity does this.

Yeah, code your own visualizations. And then the idea would be, instead of exporting that, like rendering all that to video, you could have an OpenFX plugin where you're just, hey, here's the file for that scene or whatever. It would render frames as requested by the host. Interesting. Using an HTML engine. Yeah. But HTML engine is, you know. That's how the question has become, how do we embed HTML content? So how do we render that onto a canvas? So UltraLite, easy to embed in everything, but I'm not paying 3K a year, the year that I break 100K of revenue. I looked into, it was a nice, like.

Servo?

James Munns: That was my question. Servo was my next question.

Amos Wenger: Yeah, Servo. Servo has a gazillion dependencies. It has SpiderMonkey, it has its own build system called "mach". I tried to build it and then immediately complain about GStreamer. I freaking hate slash love GStreamer.

James Munns: GStreamer is one of those words that I've seen it install logs and stuff like that. I have no idea how it works and it would probably hurt my head to learn.

Amos Wenger: (Laughing) It's not that bad. I should do an episode about it. But it has a gazillion dependencies and libraries spread all over the place because they lean into dynamic linking a lot. The Servo embedding story is before it was abandoned, someone was keeping a repository on GitHub with examples, Paul Rouget, I think. And that hasn't been touched in six years. In the meantime, it's been unabandoned and they have collaborated with Tauri in January of 2024. Tauri, they're like Electron but in Rust. And they've done a proof of concept integration.

James Munns: It got picked up by that collective, what are they called?

Amos Wenger: Igalia.

James Munns: Igalia, yeah, yeah, yeah. It got picked up by Igalia because a lot of people want a little browser engine. I guess what you were saying makes a lot of sense. I've seen people wanting to embed it in either VR, XR type stuff, or even in game engines. Because like you said, you want someone to be able to write, use web tech to make your heads up display in a VR setting or something like that. So actually it does kind of track.

Amos Wenger: A lot of people do, but they use CEF, which I'm just getting to. I just wanted to mention that like the Tauri embedding proof of concept thingy happened in January of 2024. And then in February of 2025, there's a new blog article with a tagline of "Servo is becoming truly embeddable this year." So, you know, we're halfway through the year. I'm excited to see what's to come. But for now in the documentation, there's nothing about how to embed or anything.

James Munns: The year of servo on the embedded.

CEF?

Amos Wenger: Speaking of embedded, what most games and like, for example, the League of Legends clients, Riot Games client and all that, all the applications that are mostly Electron, but actually not, they're using the Chromium embedded framework, CEF. There's a crate for that, which entire page just says, use the Chromium embedded framework in Rust. And that's it.

James Munns: I love it. Straight into the point. Yes. With a version number that definitely makes sense.

Amos Wenger: It matches the Chromium version. You can tell it's automated because it was less published three days ago. So I'm like, and it works, right? I tried their example and it works. You can have your own little browser with no Chrome. Chrome is the browser interface, like the address bar, next previous button, the menus and everything. That's the Chrome. Firefox has a Chrome. It's written in CSS and JavaScript. But I don't know much about CEF and that simple demo worked, but there's an offscreen rendering demo and that one didn't work. So I'm assuming it was coded against Windows or something.

And you need this whole kind of app bundle structure with all the libraries in the right places. It seems like it's a lot of effort. I just, I gave up on that avenue. I'm sure it's probably the right way to go at this point because when I look at Servo, for example, I load it, my voice out in it to see it. And it supports a lot of things. But for example, I have SVG diagrams that use HTML text formatting and it only showed the shapes and it didn't show the text because it didn't support like foreign objects, nodes in SVG, which is kind of a bummer.

Electron?

Amos Wenger: Finally, I figured the easiest way to experiment with all this would be to use Electron because Electron has an offscreen rendering API and you can just write a bit of JavaScript code. And the way I started experimenting with this is just listening on some TCP port for some RPC requests, which is just a line of JSON. There's a bunch of ways to capture the contents of the webpage and one of them is CPU, the other is GPU. I only tried the CPU because GPU is essentially you get a shared texture handle and then you need to open it and like copy from the, or try not to copy from the OpenFX plugin.

I'm pretty sure there would be a way to, instead of filling out the buffer, say, hey, here's the shared texture handle, go take that, go use that, I don't know. Yeah, it gives those nice images. It's, this is not different from all the other slides, but that one was generated with the whole Electron pipeline. And I figured, yeah, okay, the plugin could like start Electron in the background, send those RPC requests, get the images back. But do you have any idea how long it takes to generate one frame, one 4K frame?

James Munns: I bet it's not gonna be three to four milliseconds.

Amos Wenger: It's not three to four milliseconds. Okay. 650 milliseconds.

James Munns: Okay, love that, love that.

Amos Wenger: So in that there's like Node.js startup time connecting over TCP socket, this is not a good test. And also it's encoding everything to PNG. So you can definitely shave off a bunch of times here.

Next steps?

Amos Wenger: There's next steps, more research is required. And next season is required for this for sure. Because I think the shared texture handle thing is the way to go if doing Electron. I think embedding servo is probably a lot better, but then I started thinking bigger picture, even if everything worked well and was low latency, like you get real time playback and everything.

You now have some interface inside the host, inside DaVinci Resolve. And ideally you'd also want to be able to, if you're doing motion design, if you have like bits of code, like you can zoom in and highlight and whatnot, that's something you need to do inside of your web contents. Right, you can't do it in the venture resolve because if you zoom in, if you have a clip, you put it in your scene and then you zoom that in, it's gonna be pixelated. All right, does that make sense?

James Munns: Yeah, yeah, yeah. You're doing like pan and scan on a rasterized image, which means either the rasterized image needs to be huge, so you can zoom into whatever fidelity you want, or you want it rendered on the fly, which is more like a camera flying around.

Amos Wenger: Exactly. So that means now that all your positioning, your zooming, everything needs to be a part, that needs to be a parameter, that you can keyframe inside the host, or it needs to be part of a scene that you can kind of open in a separate editor, a web-based editor, and then the OpenFX plugin is just a renderer, and it's just like giving you a frame of a predetermined scene.

And you still have the problem that I often have, which is you usually want to do the animation synchronized with some audio, right, someone's talking, and then you want the code to come in as they talk about different things. And if your editor is outside of resolve, you're not gonna get the audio from the resolve timeline. At that point, the only reasonable course of action is to make your own nonlinear editor, right?

James Munns: I was wondering if it was gonna come to this.

Amos Wenger: Well, I certainly haven't done it in a day. I'm not planning on doing it because how do you hire editors for your own nonlinear editor software? But that would be the way to do it.

James Munns: You sound like you're standing on the ledge, and I'm not sure whether to slowly talk you down or not, or if you're excited for this jump into the unknown.

Amos Wenger: On top of Gear Acquisition Syndrome, I have Software Replacement Syndrome. I use software too much. DaVinci Resolve used to be this really intimidating, big piece of software that I didn't know how to deal with. And now I kind of see the seams of the different parts of the things that they pulled together. And I think you could do better if you started from scratch. The problem is, of course, all the unknown unknowns, all the things that they got absolutely right, and I have no idea how to do high-performance image processing using shaders. No idea about that.

James Munns: Or just 30 years of paper cut shaving. There's a huge amount of, with Second System Syndrome, where you're like, "I could make it so much easier." And you realize that so much of the things that make it painful are like, "Oh, we decided it was worth that pain, accumulated over 30 years." You know what I mean? But yeah. You cheat by doing less, so if you could figure out how to do something that only matters for you, hell yeah. But yeah, I imagine it's gonna be a thing.

Amos Wenger: Well, I'm gonna try not to go any further down that road. Wish me luck.

Amanda Majorowicz: Famous last words. I think we both know.

James Munns: That road does not end. I was gonna say, we are both the epitome of if you give a mouse a cookie. We figured out scripting to, okay, well, I guess I need to replace DaVinci Resolve.

(Laughs)

Amos Wenger: But I, no, no, I-- I will be strong.

James Munns: All right, I believe in you. Thank you. I'm looking forward to next season.

(Laughs)

James Munns: Me too.