This is Self-Directed Research. Our
hosts, James and Amos,
entertain each other with
weekly hyper-focused technical deep
dives. Amos continues
on the arc this week with
"How Far Can You Take OpenFX?" Make sure
to like, follow, and
subscribe wherever you find us,
and visit sdr-podcast.com/episodes
for all the
presentations, videos, show notes,
and transcripts. Also, mark your
calendars! Self-Directed
Research will be recording live
at EuroRust in Paris and online October
9th and 10th, 2025.
Don't miss it! And as usual,
stay tuned to the end to hear more about
Depot, the sponsor of this episode.
Amos Wenger: So I've been up all night.
No, that's not true.
Trying to figure out what
to bring to show the class.
And I thought it might be fun if we did
the freaking news, but I didn't.
I don't know.
It kept me up.
So I opened my iPhone Notes app just to
list all the things I could talk about.
And I decided to use the research that I
did yesterday to kind of close the arc of
me going deeper and deeper into rabbit
holes when it comes to making videos the
silliest possible way.
James, are you excited?
James Munns: I am.
I'm glad I was using Bluetooth headphones
because I did wander away to get my
fidget toy, but I did
hear the whole intro.
Amos Wenger: Good. Good, good, good.
So today's episode is called
How Far Can You Take OpenFX?
Subtitle, can you embed a full motion
design suite with it?
But I would like to manage everyone's
expectations, including my own.
And it's that I only put like
one day of research into this.
James Munns: For folks who haven't listened to your
last episode where we were introduced to
what OpenFX, do you want to remind the
audience what OpenFX is and where you
start this rabbit hole dive in?
Amos Wenger: James, ye of little faith.
James Munns: Well, I've got to ask for the audience
Amos Wenger: Previously, on Self-Directed Research.
James Munns: Oh, love it.
Love it.
Amos Wenger: So first of all, as always, you can get
the slides on SDR dash podcast dot com
slash episodes or by watching the episode
on YouTube or Spotify.
We have video on Spotify.
And for those of you who don't know, I
make YouTube videos.
And on the slide, we can currently see
what my videos look like.
It's mostly a talking
head, mine, in front of code.
Sometimes there's a green screen
involved, but we don't
need to get into that.
The important part is how do we render
that code and how do we like animated and
highlight parts of it and whatnot?
And I've gone through several different
techniques to do that.
It all started with screenshots because
the code blocks are
already on my website.
Right.
I my website is also all custom.
It does syntax highlighting.
So they're already rendered exactly the
way I would like to
present them on my website.
So ideally, I just take that
and put that in the videos.
And that's what I used to do is just use
the Mac OS screenshot tool and do a
rectangle screenshot
and save that somewhere.
Rename that to something meaningful.
Make sure I don't lose it because that's
the thing with
DaVinci Resolve projects
The cloud is a relative concept.
And yeah, it just took a lot of time and
it was hard to position them properly.
If you go from one slide to the next, if
you if you did the rectangle screenshot
slightly different from
the one before, it's funky.
You can definitely see it in the
transition from one to the other.
The controls in the venture resolve to
set the position and size
of the text are very weird.
I don't even know what
they're based on, honestly.
And, you know, it was not good.
So I started looking for solutions on how
to make this easier because I usually
write an entire article first and then
make a video out of it.
So I have everything.
So I can just open the article in a
browser like Safari.
And I noticed that in Safari in the dev
tools, well, they call
it the Web Inspector.
But you can make transparent screenshots.
And then I in the automating macOS
episode of the season, we discovered that
we can use Apple scripts to automate
Safari, which means you can make those
transparent screenshots automatically.
And for one of the videos I had, like, I
don't know, like 200 of them, blocks of
code and diagrams and whatnot.
So, you know, it gets
very long to do it by hand.
Which one was that?
I forget.
Honestly, I have.
OK, I'm lost in my pipeline.
James Munns: To be honest, you probably could have
said anything and it
wouldn't surprise me.
Amos Wenger: Yeah, I started making shorter pieces
because it's just not nice to have a 50
minute video to edit, I think
some people here can relate.
James Munns: Now you're considering the
niceness of your editors time.
And so that just being your time, you go,
well, I would I wouldn't inflict that on
anyone, let alone
someone who works with me.
Amanda Majorowicz: Yeah.
I appreciate it.
The past I don't know how
many episodes have been like,
James Munns: said Amanda, who's been
editing 50 minute episodes.
Don't worry, Amanda,
this one will be a short.
Amanda Majorowicz: Yeah, exactly.
Only 44 slides or what is it?
Amos Wenger: Yeah.
Yeah.
Amanda Majorowicz: Nice.
Amos Wenger: I've aligned incentives for my videos
because I paid my editor
by minute of edited video.
James Munns: Nice.
Amos Wenger: So I have an
incentive to make them short.
And then we discussed the rate and
evolve and evolve it over time.
But so that's what we
discussed in automating Mac OS.
You just use Apple
script, which was very fun.
There's on the slide.
If you're watching the
slide, you can see that there is
not a mono space font, but a serif font.
Actually, is it serif?
I don't see serif.
Is it sans?
Whatever.
It's that it's proportional for sure,
which is the opposite of monospace.
Amanda Majorowicz: It's sans serif.
There are no serifs on this font.
Amos Wenger: I thought maybe the the lack of sleep
made me the sans disappear.
The serifs disappear.
And another as the episode named
Blackmagic Fusion is weird.
I discussed my new and current setup.
I've already reached a video with it.
I forget which one, which involved
reversing the Lua based formats that
Blackmagic Fusion uses for node graphs
and just kind of generating what it
expects and generating a Text+ node
with the character level styling modifier
and doing syntax highlighting that way.
And I even vibe coded a
little preview on my website.
So you press a button and then it opens
this editor that uses "contenteditable"
So you can remove the parts that you're
we're not talking about in the video
right now, you can like select a bit of
text and press space.
And it actually highlight
this and dims everything else.
It's really, really nice.
It has a stand in for my face.
I look super goofy again.
Really, you should check the slides.
That's half the fun.
James Munns: This is disappointing because this is one
of the few website plugins that only
works locally and you
can't trigger it remotely.
Because otherwise I'd love to like edit
your page and then slap your face in on
examples and be like you are making your
own meme templates except for it's.
Amos Wenger: I need a new patrons here for like like
green screen pictures of me.
Yeah, sure.
Yeah, exactly.
For evil people.
I just saw yesterday on Reddit a post
where some burger place lets you pay.
The Veruca Salt special.
You can pay a thousand dollars to
get in front of the line and just.
I want it now, Daddy.
I think that's what it said.
James Munns: Exactly.
Yeah.
Amos Wenger: And I think that's something people
should consider, like
making tiers that are higher.
So yeah, that reminds me of also someone
said like on Twitter a long time ago, you
know, when Twitter was still Twitter,
they said like if someone were to pay me
like 5000 a month, I would just stop
posting indefinitely.
And that's true.
James Munns: I do.
Who was I'm going to have to go back and
find that because I definitely remember
that of the like it's someone we know.
Amos Wenger: I don't I forget whom.
James Munns: Yeah, it was just I'll shut up, you know,
for that check keeps clear.
And I exactly I'll just shut the hell up.
Amos Wenger: And it worked out beautifully on my blog.
I have my articles that
are exclusive for six months.
Someone was like, hey, it's bad that
exclusive for six months.
And I was like, OK, you want to you want
to pay to change that?
I was like, OK,
that was Depot, the sponsor of this episode.
So a big thanks to them.
James Munns: Yeah, I was going to say it was our
lovely sponsor, Depot, who is I know, who
is sponsored this entire
season and that article.
Amos Wenger: Very generous and very cool platform.
Check them out.
James Munns: Yeah.
Listen to the end of the episode.
Amos Wenger: Yeah.
Back to my short presentation in the
Black Magic Fusion is Weird episode.
It's kind of a misnomer because it has
this whole nice solution of generating.
And I also talked about the other ways in
which fusion can be extended, including
fuses, which we learned come from the
same word as fusion.
We weren't sure during the episode, but
in the in the show notes that Amanda put
together, I saw that the etymology says
that they are from the same word.
James Munns: Were they consistent?
I saw one was like a
bundle of twine, like "fuset"
I think it was a French word.
And then I don't remember
where fusion came from, but
Amos Wenger: "fusée" is a spaceship like a rocket ship.
James Munns: Oh, yeah.
Maybe I'm probably pronouncing the French
word very, very wrong.
Amos Wenger: But yeah, my brain is too mushy to figure
out which which French word you mean.
James Munns: We'll leave etymology for another day.
Yeah.
Amos Wenger: So fuses, which are a bits of Lua that you
can hot reload and everything.
It's a very nice development environment,
which are fast enough
because it's using a jit
It's actually a LuaJIT
Whenever people say Lua, they usually
mean LuaJIT not always, but usually.
And and OpenFX, I
briefly touched on OpenFX.
I was like, OK, it's a standard way to do
video plug ins a little bit like AU
and what was the one VST for audio?
But this one is for video.
And I said, you probably could bring in
your own renderer and do
exactly what you wanted.
Yeah, one of the limitations we ran into
by using back magic fusion's own text
renderer is that it does not support
color emoji, for example.
And it does not support font fallbacks.
So part of the difficulty for me in
generating the graph is that if, for
example, there is a nerd font symbol, it
has to specify that this pan is using the
nerd font symbol font.
Everything else is in Berkeley Mono
except for emojis, which are using Noto,
Noto without color.
James Munns: monospace or the monochrome.
Amos Wenger: Yeah, the monochrome.
James Munns: Yeah.
And I went back and looked it up.
And the word I thought
was toast is actually tofu.
And that's where the Noto Sans name
comes from is "no tofu".
Because it defines all of the
symbols for all of the emojis.
So you'll never get the tofu, which is
that little rectangle that's either an
exclamation point or has like the hex
characters in there.
I don't remember why it's called tofu,
but that was the word I was trying to
think of in that episode.
Amos Wenger: The Unicode replacement character.
James Munns: Yeah.
Yeah.
Well, there's maybe two.
There's one.
Wow.
Amos Wenger: No tofu.
James Munns: Yeah.
But tofu was what I was thinking of when
it shows you the like,
I don't have the glyph.
Here's some hex codes.
Do you like that?
Amanda Majorowicz: I like learning about all this stuff,
like going through editing the episode,
collecting all the links and then being
like, I don't know what
you were talking about.
And now next episode,
learning about last week's episode.
Very cool.
Amos Wenger: I think you should feel free to cut in
like add editors notes
in the middle of episodes.
Like I checked it.
It's not true or something.
They're still wrong.
Amanda Majorowicz: I was actually thinking about that.
Yeah.
But you know, who knows?
Maybe in a future episode.
We'll see.
Amos Wenger: Yeah.
So yeah, I mentioned OpenFX saying, yeah,
you can you can make native plugins.
You know, it's clearly overkill.
Obviously, I'm not going to touch that.
That would be crazy.
So that's what I did next.
James Munns: Yeah, of course.
Amos Wenger: Yeah, of course.
Because because it's
James Munns: stuck in the back of my mind.
There were some hints in there where
you're like, oh, there's a crate.
You know, I definitely
exactly don't need that.
I'll I'll definitely
use the Lua for that.
And it was one of those like, oh, no,
it's this is Chekhov's
gun in the last episode.
Amos Wenger: Check Chekhov's crate.
Yeah.
That's what I was writing the slides.
I was actually thinking of that line.
Yeah, there was a crate for it.
So how could I not look into it?
Of course.
I think the audience deserves that.
I think that's what we're all here for.
It's to watch me suffer.
I was kind of immediately disappointed
because the crates ofx-rs is written
against nightly rust from six years ago.
I didn't realize that I thought it was
stable rust from six years ago, which
would have been totally fine.
I think maybe maybe a couple of fixes,
but like unless something was wrong in
terms of soundness, I
think we would have been fine.
But nightly from six years ago, I was
using two features that just have changed
or not there anymore, whatever.
I got over 200 errors trying
to compile the example code.
So I thought that would be the end of it.
That I told you it would be short.
James Munns: Yeah, I'm trying to like 2019 rust.
It's not as far as it could be.
We're like post 2018,
which is a pretty big step.
But yeah.
Yeah.
Six years of nightly is a stretch.
Yeah.
Amos Wenger: But then I found NTSC-RS, which is a port
of NTSC-QT to rust
and SIMD by Valadaptive.
And among other
things, it's 20 times faster.
Yeah, it uses SIMD and
concurrency and stuff.
It's an analog.
James Munns: They have a very pretty website.
Amos Wenger: It does.
It's a VHS effect for your
video filter, essentially.
And it uses what does the website say?
Uses algorithms that model how NTSC
transmission and VHS
encoding actually work.
I imagine that is the next word.
James Munns: Ooh, someone was asking me about this on
BlueSky because there's shaders that
I've seen for terminals that give you
that like CRT glow and stuff like that.
But I'm not surprised that especially
with all of the back rooms and like
cassette futurism stuff that's going on,
that there's not like super shaded CRTs
and VHS things going on these days.
Amos Wenger: True.
True, true, true, true.
That one is all CPU as far as I can tell,
but it's I don't know.
I think it's an old effect.
Yeah, ported to rust.
And the code place is very versatile
because it has a standalone
application with egui controls.
I don't know if you recognize the
egui-ness of the of the GUI.
James Munns: Definitely.
Amos Wenger: Yeah, it could be.
What's the other one?
What's the C++ one?
The one used in all the video games.
Imgui.
Oh, yeah, Dear Imgui.
Yeah, Dear Dear Imgui.
Exactly.
I could be that, but I spend too much
time with egui making demos.
So it's available as a standalone.
It's available as a CLI.
It's available as an After
Effects and Premiere plugin.
And as an OpenFX plugin.
James Munns: And are we getting back into those
plug-in world like we were talking about
with the audio ones where it's like you
have purchased a DLL.
Congratulations.
It runs in your environment.
Kind of.
Is it something much more
specific to this platform?
Amos Wenger: Kind of.
Actually, so NTSC RS is open source.
Okay.
So actually they use cargo-xtask to
build the plugin, but it's just so it can
generate that little bundle, which looks
like a Mac OS app bundle.
It has a contents directory with an
Info.plist in there, a MacOS directory
with a .ofx, which is a shared library,
resources directory
with an icon of the plugin.
And the info.plist is just XML.
There's one of the plist formats.
There's a binary one and
there's a third one, I forget.
Like text but not XML.
I don't know James,
does that ring a bell?
James Munns: Yeah, I gotcha.
It's exactly the same kind of thing that
I had to put together for the Postation
bundle and for when you're
shipping a binary on Mac OS.
So it definitely rings true.
Amos Wenger: Yeah, it looks a lot
like it, but it's not .app.
It's .ofx.bundle.
It's weird.
James Munns: It's a folder that's
pretending not to be a folder.
Amos Wenger: And yeah, so the reason there's an
xtask, which is kind of turning Cargo into
make, you know, you can have different
targets and it's "cargo xtask
build-ofx-plugin" is because they want to
make a universal build.
I don't know if that's
built into rustc yet.
Nowadays, you make a binary for x86-64
and for aarch64, but in the past you did it
for like, well, Intel 32-bit and Intel
64-bit, you put that in the same binary
or a PowerPC plus until
32-bit, even further back.
So like any kind of transition that Apple
has done, like Mac OS has done, you could
have a transition period where you've
shipped both binaries in the same thing.
And it's still happening today because
not everyone has Apple
Silicon Mac, an ARM Mac.
If you go on Visual Studio Code webpage,
for example, there's a
single download for Mac OS button.
But if you download that and you use
lipo-info, then you see that there's two
architectures in there, x86-64 and arm64.
James Munns: I think even, I can't remember if it was
Android or iOS, like for the iPhones and
things like that, where they were doing
that as well, because there's different
generations of the processors that they
use instead of doing the like, hey, we're
compiling for the oldest compatible X86
version and things like that.
There's the option to do multiple
compilations for different target
platforms so that you can get either like
the operating system platform level or
like the CPU for Apple where there's much
more control over like, hey, this works
on these versions of phones and later,
which means you have all of these
processor extensions available for more
performance and stuff like that.
I wouldn't be surprised if both of them
were doing that, but I guess it's just
how you do the packaging
and the app store handles it.
Amos Wenger: It's possible, but given
that these would be mobile apps.
This is an aside, I will, yeah.
I think the equation changes a little bit
because who cares if VS Code is a few
megabytes larger, but on
the phone, I don't know.
I know there's like SDK target levels for
sure, and I know those are hard to handle
properly, and those are also
a thing on macOS, but yeah.
James Munns: But yeah, universal binaries.
I remember them, like you said, from X86
and ARM, but they
continue to be relevant.
Amos Wenger: If you want to save the bytes, you can go
to that detailed download page and have
like the Intel or Apple Silicon build
separately and all the, oh yeah, for
Linux, you get the
ARM32 builds for like, what?
What's ARM32 these days?
James Munns: Original Raspberry Pis.
Raspberry Pi was 32-bit for a long while,
and then it did a very weird thing where
the kernel was 64-bit, but the user space
was 32-bit, which was like a
compat thing so that they didn't
Amos Wenger: have to-- Because of RAM usage?
Like 32-bit pointers or, okay.
James Munns: For compat, like if people had, yeah, so
if people had compiled some binary
application for 32-bit, you need a 32-bit
user space to continue to run that,
because otherwise you'd have to recompile
to run on a 64-bit user space, but you
might want your operating system to be
able to handle more than the two, or be
able to handle more RAM, or at least have
extended RAM range, even if
it's virtual memory range.
So for a while they were doing that.
I think they're all 64-bit now, but yeah,
I know Raspberry Pi was like the 32-bit
ARM that everyone had and
was popular for quite a while.
Amos Wenger: Yeah, it's funny.
If you look at Windows or Mac, Mac is
just Intel chip, right?
It doesn't even think about 32-bits, and
Windows, it's x64, x64, Arm64.
They're using the terminology for each
platform, which is nice.
Anyway, that bundle structure that I just
showed is the same on other platforms,
because it's a very macOS thing to do to
distribute an app as a folder and call it
a bundle, and have it show differently in
the finder, but actually it's just a
folder if you look in the command line.
But as far as I've been able to find,
because I've been looking around the
OFXRS directory, repository, even if I
wasn't able to run it, I saw, I looked at
how they were doing things, and they are
making a content slash Linux dash x86
dash 64 directory, which
is, it's a Franken bundle.
Like it's weird, because
it's macOS semantics on Linux.
I don't know.
It's really weird to me.
Anyway, NTSC RS does work.
I was able to build it from source and
import it in DaVinci
Resolve, and it works everywhere.
It works as an effect.
You can drag onto a
clip in the edit page.
It works as an effect in the color page,
and it works as a node in fusion, which
is absolutely lovely.
You can see on the slide is a me on a
green screen with a NTSC effect on a 480p
timeline, because that's
what they recommend doing.
Even though it can run at higher
resolutions, it's not supposed to have
more than 480 lines.
James Munns: You get sort of that like anachronism of
like, here's my 4k image with scan lines,
and you're like, that I don't think
anyone was doing that.
There might've been raw captures of that,
but you certainly weren't broadcasting it
more than 480 lines.
It's got the vibe of it.
It's very like found footage or like
someone sent in a VHS of themselves in
1990 to some technology show to it.
You've definitely gotten
that vibe out of the effect.
Amos Wenger: And there's a ton of the parameters you
can, a ton of controls you can change.
I've put side by side the parameters as
seen from the venture resolve and the
parameters as seen from the
standalone E-GUI interface.
And the code base is just really
well-designed because they have a single
way to describe all the parameters and
then it's mapped to whichever interface
because they have essentially three
different GUIs, right?
They have egui and standalone, they
have Premiere and After Effects, I think
are roughly the same.
I have a lot of commonalities.
And then OpenFX.
And OpenFX has different kinds of hosts
like DaVinci Resolve
or Natron or like any sort of VFX program
that can load OpenFX
plugins is called the host.
It provides functions and provides these,
the UI, the parameters like loading
settings, saving settings, et cetera.
And also you can see on every parameter
here, there's a little
diamond that's for keyframes.
So OpenFX plugins don't
need to know about keyframes.
They just need to, for every render
request, they just need to get the
parameter for that time.
And then the host is in charge of making
the values change over time in the
timeline and like
interpolating things and stuff.
James Munns: Hello, Amos.
Yes, as someone who does not edit a lot
of video, what distinction does keyframes
make when it comes to applying an effect?
Is the effect not just a stream of images
that it's modifying or is there
intelligence going on?
Amos Wenger: Sure, but you see intensity here.
You can make it change over time.
So keyframe is just like, it's this value
at that frame and then you
wait 30 frames and then change it.
And then it changes linearly or like
maybe there's a little tweening happening
between the two values.
It's just animation basically.
Gotcha.
So yeah, let's keep walking through like
what is actually in an OpenFX plugin.
There's a well-known struct.
It's exactly what you
would expect from a plugin API.
Essentially there's well-known structs.
You export two important functions.
One is, ofxGetNumberOfPlugins which
in this case returns one and
it's not shown on the slide.
And ofxGetPlugin, which returns a struct.
What's interesting in the OpenFX API is
that they use a lot of
string constants everywhere.
Sorry, the first thing in the struct that
you return is just a magic string.
It's a "kOfxImageEffectPluginApi" as a
string and that's it,
well, without the K.
And then there's API
version, which is good.
That means they can actually evolve it.
Well, it's set to one in this case, but
they can just like add or--
James Munns: Which means it's never changed.
Amos Wenger: Yeah,
but they could like change the struct.
They could interpret the struct as having
this header that never changes except for
the API version field.
And then based on if it's one or two or
three, like read different fields after
that, like sort of like a rust enum.
James Munns: Yeah, these are fun things for like
binary compat, like, cause you run into
it in protocols as well as like ABI
things of, you need to just read the
first two fields and then you promise
that those never change.
But that second API field, like you said,
it acts like an enum where it's like, ah,
do I actually know how to interpret the
variant that it's claiming?
Because if it says two and I have no idea
what two means, I should stop reading
because things are about
to get very, very weird.
Amos Wenger: Just throw an error.
James Munns: Yeah.
Amos Wenger: Yeah, instead of
getting memory corruption.
Next up, we have main entry, which is
directly one of the fields
of that struct that we return.
And main entry is kind of like the main,
kind of like DLLs on Windows
also have a main entry point.
It's just funny to me because it gets a
string, a C string and you compare it
against a bunch of constants and then you
know if the action you're supposed to do
is load or describe or get regions of
interest or whatever or render.
So that means every frame you render,
every tile you render, there's a
multi-threading thing that
you can ask the host to do.
You can ask the host to like divide the
image in different tiles and ask you to
render tiles in
different threads and whatnot.
You're just a poor shared library.
So your function can be
called from different threads.
The first thing you do is just compare
against like do a chain of ifs.
It's like, okay, it's not load.
It's not described.
It's not describing context,
blah, blah, blah, blah, blah.
Finally, I was like, oh, it is render.
So either string comparisons are a lot
faster than I imagine or that does not
that worried about performance.
I don't know, James, how
do you feel about this?
James Munns: I mean, computers are fast.
Like, yeah.
True.
I mean, a match, like it looks grosser
than a match, but a match is essentially
just doing the same thing.
Like you can do much smarter string
compares of like making either like a
regex style automata
or what are they called?
They're not just finite statements.
Amos Wenger: Sure, but why not just export constants
from the library or something?
Like define, I don't know, the U64 or
whatever in file formats, like MP4, we
have those atoms which
are just 32 bits, like four.
Yeah, four times
eight is 32, just four characters.
So you have like MOOV, you have MP4A,
whatever, that's what all those things
are four characters long.
It's so that you get a nice
32 bit constant out of it.
James Munns: I assume 32 str comps are way cheaper
than whatever video effects you're about
to be doing, honestly.
So like order of magnitude, they might
go, "Eh, it's so we can debug it easier."
Cause then you could just print instead
of having to do a lookup or whatever when
you print logs of like,
"We were told to do this."
True.
I don't know.
I think you're right.
It would definitely be much smaller, but
I think probably one of those order of
magnitude things where it goes, "Eh,
whatever, we're about to do
a bunch of DSP, so whatever."
Amos Wenger: Well, what I also like is that in this,
in NTSC-RS specifically, it calls set
hook, set panic hook.
And I have no idea
what the cost of that is.
Like I have not measured it.
I've measured other parts because the
plugin was slow at some point, I did some
performance measurement.
I did not measure how long it takes to
set the panic hook, but
I imagine, I don't know.
James Munns: I don't think a huge amount of time.
No, but- Yeah, I have no idea
either off the top of my head.
This is running in std, which
I don't do most of the time.
Or at least I don't think too
hard when I'm running in std.
Like I think hard when I'm running in
bare metal, but yeah.
Amos Wenger: So next up, if the action that we're
supposed to do is a render, then we get
the address of the functions from the
host that we're supposed to call, like
propGetString or
propGetDouble, et cetera.
Then we call that and we get all the
properties that we want.
Again, mostly with string constants, like
the propGetDouble,
kOfxPropTime.as_ptr()
That's just the string.
That's just "OfxPropTime".
And also we're getting it as a double,
which is an F64, but it's actually just
in DaVinci Resolve,
that's the frame number.
So it's fully an integer.
James Munns: speed, summer games done quick was just
happened, which means everyone was
talking about basically all like speed
running and glitching
and stuff like that.
And there's a whole thing of like, what
goes wrong when you use a floating point
number for your frame counter.
And it was basically like 16 million is
where just add one breaks down because it
rounds down instead of up, which means
you just hard lock at 16 million frames.
Amos Wenger: So 16 millions at 25 FPS.
James Munns: Oh, it's like months.
Like this was a Mario 64 glitch that was
like, look, this portal stops rippling
after four and a half months, but this
one stops after a month and a half
because it increments by
two or something like that.
Amos Wenger: It reminds me of the Windows 95 thing
where if it stays up
for 91 days or something,
James Munns: didn't do this.
You get this in planes and stuff.
When you have-
Amos Wenger: It overflows.
James Munns: It's either a millisecond
or a 10th of a millisecond.
Whenever you see something that says it
needs to be restarted every 50 days, or
it crashes every 50 days, that's like,
yeah, it's either every millisecond or a
10th of a millisecond in a U32 counter.
Or like, yeah, you start
to notice those numbers.
Like this is one millisecond, 10
millisecond, a hundred millisecond, one
microsecond of like those times where
it's like, yeah, it just crashes if you
run that long and it means that they
didn't handle overflows right.
Amos Wenger: Well, to be fair, it can be hard to
handle that properly. 16 million frames is
25, according to o3, which
I trust with reservations,
it's seven days, nine hours, 46 minutes
and 40 seconds. 16
So if you make a student film, that's
just really boring in
DaVinci Resolve.
You might get into issues at some point.
So you get all your, you get the
addresses of the methods you need to call
from the host, and then you call those
and you get all the properties, all the
parameters that I've shown earlier, like
intensity, frequency, detail, snow, in
the case of NTSC RS, but
it could be anything, right?
You can have anything in
the UI, it's pretty flexible.
You also get the input signal, the output
signal, output buffer, where you're
supposed to write the thing, the result
of your effect being applied.
The host tells you which part of the
output buffer you're supposed to touch.
It tells you the buffer format, if it's
RGB, RGBA, if it's eight, 16, 32 bits, if
it's floating point integer.
I was curious what code path actually got
used because NTSC RS
handles all possible combinations.
So how is developing
an open effects plugin?
I wanted to find out what
was the actual buffer format.
I wasn't planning on
supporting everything.
This is research level, we're just trying
to get it to work for one specific
configuration, like only
on Mac OS, only on one host.
And the first thing that
hit me is how do you log?
James, do you have any idea?
James Munns: How do you log?
Amos Wenger: How do you log?
How do you print things?
James Munns: Okay, so if I had to guess, there's...
Amos Wenger: You send a
postcard-rpc messag—no.
James Munns: It's probably not far off.
I mean, what I would assume is in the
plugin, you get passed in some vtabl
that has methods that
you're allowed to call.
Or there's some like
global or something like that.
Amos Wenger: But no.
James Munns: Yes, but no.
Amos Wenger: There's absolutely nothing standard.
James Munns: Oh, love it, love it.
Amos Wenger: There's no OpenFX
blessed way to do logs.
You can write to standard error, but then
you need to start your host from the
terminal, which is annoying.
Resolver has its own
log, which you can't touch.
I'm assuming you can write it from fuses
and they also have their own SDK to make
plugins that are not open effects.
James Munns: Are you gonna tell me that you spawned a
Lua interpreter just to print messages
using some kind of Lua trampoline or
something like that?
Amos Wenger: No, I only had a day, James.
I didn't do that.
No, no, no, you're seeing solution in
front of you, which I just opened the
file and just appended to it.
James Munns: Yeah, okay, yeah, just no sandboxing,
just straight to temp.
Yeah, of course.
Amos Wenger: Sandboxing, James, your concerns are very
far from the concerns
of VFX people, I think.
James Munns: Oh, it's fair, it's fair, yeah.
Amos Wenger: Another question, how do
you reload your plugin?
Any ideas?
You panic.
No, if you panic, I'm pretty sure it
takes down the host.
James Munns: Okay.
Amos Wenger: Well, yeah.
James Munns: Oh, great, yeah, that's great.
Write no sandboxing.
Amos Wenger: Yeah, this is not
WebAssembly, this is not, you know.
James Munns: How do you hotload or reload your bundle?
Exactly, well, you don't,
Amos Wenger: you can't.
I'm showing you now the video plugin
settings pane and you can do, "pain" is
right, and you can disable all, enable
all, but all it does is just show you the
plugin and the list of
things you can use or not.
But once it's loaded, it's loaded, right?
"dlclose" is a lie in
pretty much all platforms.
It always leaks things.
There's no cleanly unloading something.
In Rust, if you load a library, it
returns to static lifetime and for good
reasons because you really shouldn't
unload things that you've loaded.
Even if you could like cleanly do it,
apparently in C++ you can cleanly do it.
Even if you could do all that and you're
initializing metal kernels and like
shaders and whatever and those can't
necessarily be torn down pretty like, I
don't know, I don't know the details, but
essentially you cannot.
So the way to reload it is to close the
Vintra Resolve and then reopen the Vintra
Resolve and then reload the project.
Do you have any concept
of how long that takes?
Uh, like?
I've timed it.
How long to shut it down?
James Munns: Okay, I know vaguely what it is for the
self-directed research, but we do no
video effects or any, well, I mean like
we have slides, I'm sure, minutes?
Amos Wenger: No, no, thankfully, no,
I have a very nice M4.
Wait, is it taking minutes on
your end to open the project?
James Munns: No, it takes like 15 or 20 seconds for
this, but we also
aren't loading any video.
Amos Wenger: Yeah, exactly, yeah, yeah,
that's more in the ballpark.
What gets me is that it takes 10 seconds
to shut down before you
can even start it up again.
That's annoying.
James Munns: You know someone just has a sleep that
they're like waiting for something to do
and they're like, whatever, it's a bunch
of sleep(1) or sleep(3) in there and
it just, it becomes
serialized and it takes 10 seconds.
Amos Wenger: I think it's all the shaders and stuff,
whatever, being unloaded or something.
Actually, oh, Jesus,
that's another nerd snipe.
But Instruments could probably tell
us what it's spending its time on.
James Munns: Come back next season for debugging the
plug-in interface of this, but yeah.
Amos Wenger: No, this is the end of the arc.
James Munns: That is one of those fun things of the
thundering herd of destructors is a not
trivial performance problem in a lot of
places of just, oh, we're gonna run all
the destructors now and it turns out that
takes a long time, but actually just
killing the program would have achieved
the same thing 99% of the time, except
for now it takes us 30
seconds to close the program.
Amos Wenger: So because it takes a long time to shut
down DaVinci Resolve and open it up again
what other people do is that they use
another host like Natron, I don't know
how to say it in English.
I know it's a French software.
It's scriptable with Python and you just
write a little script that sets up a node
graph and renders
everything and saves an image.
James Munns: Now, Amos, I can
excuse a Lua, but a Python?
Amos Wenger: I told you, it's also doing
Python in DaVinci Resolve.
James Munns: Sure, I remember what episode it was.
There was definitely some episode where I
said something about Python and Amos was
like, I can excuse a
Forth, but a Python?
Oh, I think it was
talking about bootstrapping.
Amos Wenger: But there's no aarch64 version of Natron or
Natron or something.
When I launch it on my computer, it just
immediately crashes.
Well, not immediately,
that's what's puzzling.
Like most things immediately crash.
It opens, it shows a little window, waits
a few seconds, it's loading more stuff
and then it crashes.
At that point, we have talked about
OpenFX, but we haven't really talked
about how do we actually
do the rendering, right?
We wanted to improve on the text
rendering with Blackmagic Fusion.
How do we actually do that in Rust?
James, I don't know if you're familiar
with the font
rendering ecosystem in Rust.
James Munns: No, no is my answer.
I know the people who are doing it a lot
of the times, but I am
not familiar with it.
Amos Wenger: You do?
Well, okay.
(Laughs)
Amos Wenger: That was a hell of a reaction.
Okay, I wanted, what I was thinking when
I blurted out was okay, was like, you
choose to thank them,
they're doing great work.
But what came out was okay.
James Munns: Oh.
Oh.
Well, I know some of the people.
I don't recognize Mooman219.
If it wasn't Rafe, then I probably
wouldn't have recognized who it was.
Amos Wenger: The one that I found that is pretty
simple and that kind of replaces rusttype
type and ab-glyph and glyphbrush and
glyphbrush-layout, rusttype being like,
I think an early port
of free type to Rust.
Pure Rust alternative.
I'm not actually sure if it's a port.
James Munns: I'm just excited that it's no_std
because now I need to, I want to go mess
with it and see if I can use that for
font rendering because that's a fun thing
in embedded where a lot of the times you
pre-bake your fonts.
Like you rasterize them to specific font
sizes and then you just--
Amos Wenger: Yeah, a lot of this
stuff is actually no_std
James Munns: Take the glyphs and stamp them in.
Amos Wenger: Yeah.
Nice.
This is how far I got with my research.
I integrated fontdue, like I kind of
vampirized the NTSC-RS plugin and I got it
to render the number of
the frame in the output.
At first upside down and
then finally right side up.
James Munns: Is this one of those like is zero top
left or bottom left or bottom right?
Who is it that has the excellent someone
we both follow on Twitter and then
BlueSky has all of the graphic, are all like
the game engines and stuff like that?
Amos Wenger: I think it's Freya Holmer.
James Munns: Oh yeah, exactly.
Exactly, it's Freya of like, what is X, Y
and Z positive, negative and where does
the zero index start on all
of those on 2D and 3D modes?
Cause everything's different.
Amos Wenger: But that's the fun thing is if there's
several ways to do a thing, if there's
several possible conventions, all of them
are in use somewhere.
But Unreal is actually
changing that I think.
Oh really?
James Munns: Okay.
Amos Wenger: Yeah, they're
switching to something else.
I don't know, they're
joining it or whatever.
I saw the announcement,
I was like, what, why?
It's gonna break everything.
I'm pretty happy with that
and it runs in almost real time.
Apparently getting all the parameters
that I wasn't using was
what was taking the most time.
And I got it down to like for a 4K frame
rendering between three and four
milliseconds, which I
thought was pretty nice.
I thought we had like a, for 60 FPS, you
get a 16 millisecond budget.
James Munns: Yeah, 16.6 milliseconds of frame, yeah.
Amos Wenger: Exactly and I only need to do 25 FPS.
So I figured we had plenty of time but
it's not running smoothly.
James Munns: You know what that means?
You know what that means, Amos, right?
What does that mean?
You were using the NTSC
plugin at a PAL framerate.
Because NTSC is 30 or
59.97 and PAL is 25 or 50.
Amos Wenger: But my footage is, I
live in Europe, okay?
Leave me alone.
James Munns: But yeah.
Yeah, it's like the slow-mo guys episode
where they explain why they want the
lower frame rate and it's just why more
frames doesn't actually help.
Amos Wenger: Yeah, for some reason, even with it
taking three or four milliseconds by
frame, it's still not running smoothly.
Like playback is not smooth.
Fortunately, you have a playback
resolution kind of fixed.
You can run at half resolution while you
do the editing and then while
you export it full resolution.
But it's still kind of not great because
this is just one line of very basic text.
This is why I was wondering about this,
setting the panic hook because maybe
that's taking a few
milliseconds here and there, you know?
Probably not much but...
James Munns: Yeah, I was just saying, if you don't
have logging, then doing timestamps of
things is gonna be egregious.
And then, like you said, trying to do
profiling for a DLL that you didn't
start, that someone else is starting
becomes dot, dot, dot fun.
Especially if the Egooey, like if that
raw binary setup does things that are
just intrinsically different than the
inside of DaVinci Resolve kind of things
or inside of Fusion or whatever.
Huh, that sounds fun.
Amos Wenger: Also, the other issue is that this
doesn't actually improve on the renderer
from Fusion because it
does not support color emojis.
It just supports like
the basic free type stuff.
Look basic, I don't know.
Look at me dismissing their hard work.
It's still like an amazing feat.
It's only basic, yeah.
But it supports a ton of
stuff but not color emojis.
Swash supports more things as far as I
can tell but it doesn't do layouting.
If you wanna do layouting, you need
something like cosmic text.
And cosmic text actually works really
nice but I tried the demo
and it does render color emoji.
I was impressed.
Like, oh, does anything in the Rust
ecosystem support color emoji or is it
only a thing that browsers can do?
No, no, no, you can render emojis with
cosmic text, the Mac OS color ones.
Apparently the color emojis format for
Mac OS, Windows and Linux are completely
different because yeah,
that's why we can't have nice things.
But the demo of cosmic
text was actually pretty slow.
Like it's rendering a bunch
of text and you can select it.
It's kind of an editor.
But yeah, on this pretty good computer, I
expected it to be really fast, like running
at 60FPS or something and it did not.
So I don't know who's to blame here.
I don't know what's happening.
Cosmic text is probably great and fine
and this is not its fault.
But text rendering is just
more expensive than I thought.
I guess the trick is to not do it every
frame if you don't need to and like cache
as much as possible.
And I don't know, like, you know,
at-laces and whatnot.
And the demo is just
about correctness instead.
I have no idea.
But at this point, I started wondering,
well, I have demos on my website which
use more than just text.
Like what if I could just
render a whole browser in there?
Instead of messing around and having just
text and text renderers and Rust.
Okay.
When it comes to embedding browsers,
there are several options.
One of them is UltraLite, which is a
stripped down build of, I think, WebKit,
which means not Chrome, but like Safari
with like JavaScript core and whatnot.
It's this pure GPU HTML renderer.
I'm pretty sure that's, I
don't know how accurate that is.
It's GPU accelerated,
I'll acknowledge that.
There's SDKs for Rust,
non-official ones, but feasible.
But I took a look at the pricing.
James, what do you think this costs?
Wait, browsers cost money?
It's not a browser.
It's an embeddable
pure GPU HTML renderer.
HTML renderer?
James Munns: Contact us for pricing?
Amos Wenger: No, it's a little better than that.
So there's a free tier up
until 100K of revenue per year.
And then there's a pro tier.
Okay.
You can pay something yearly.
Okay.
And it's 3K.
So some years flirt with 100K.
James Munns: Wait, so where does the
browser come into this?
Where browser?
We were talking about video editing.
Where browser?
Amos Wenger: Where browser?
UltraLite lets you open a webpage pretty
much and render it in a GPU buffer.
James Munns: You'd have a plugin that does browser
rendering, so you'd write JavaScript.
So you're running a scripting engine
inside of your plugin with a rendering
target, and then you're going splat into
the image and then like, you know,
rasterizing that into the
green screen or something?
Amos Wenger: Not sure I follow your explanation, but
yes, the idea would be
to-- It's probably wrong.
To have, instead of rendering text
directly, you would render web contents.
And that way, those motion
design tools are available.
I forgot their names, but you can write
TypeScript code essentially instead of
doing things on a timeline and say, okay,
this element comes in, this element
rotates, this
element's opacity does this.
Yeah, code your own visualizations.
And then the idea would be, instead of
exporting that, like rendering all that
to video, you could have an OpenFX plugin
where you're just, hey, here's the file
for that scene or whatever.
It would render frames
as requested by the host.
Interesting.
Using an HTML engine.
Yeah.
But HTML engine is, you know.
That's how the question has become, how
do we embed HTML content?
So how do we render that onto a canvas?
So UltraLite, easy to embed in
everything, but I'm not paying 3K a year,
the year that I break 100K of revenue.
I looked into, it was a nice, like.
James Munns: That was my question.
Servo was my next question.
Amos Wenger: Yeah, Servo.
Servo has a gazillion dependencies.
It has SpiderMonkey, it has
its own build system called "mach".
I tried to build it and then immediately
complain about GStreamer.
I freaking hate slash love GStreamer.
James Munns: GStreamer is one of those words that I've
seen it install logs and stuff like that.
I have no idea how it works and it would
probably hurt my head to learn.
Amos Wenger: (Laughing) It's not that bad.
I should do an episode about it.
But it has a gazillion dependencies and
libraries spread all over the place
because they lean into
dynamic linking a lot.
The Servo embedding story is before it
was abandoned, someone was keeping a
repository on GitHub with
examples, Paul Rouget, I think.
And that hasn't been
touched in six years.
In the meantime, it's been unabandoned
and they have collaborated
with Tauri in January of 2024.
Tauri, they're like Electron but in Rust.
And they've done a proof
of concept integration.
James Munns: It got picked up by that
collective, what are they called?
Amos Wenger: Igalia.
James Munns: Igalia, yeah, yeah, yeah.
It got picked up by Igalia because a lot
of people want a little browser engine.
I guess what you were
saying makes a lot of sense.
I've seen people wanting to embed it in
either VR, XR type stuff,
or even in game engines.
Because like you said, you want someone
to be able to write, use web tech to make
your heads up display in a VR setting or
something like that.
So actually it does kind of track.
Amos Wenger: A lot of people do, but they use CEF,
which I'm just getting to.
I just wanted to mention that like the
Tauri embedding proof of concept thingy
happened in January of 2024.
And then in February of 2025, there's a
new blog article with a tagline
of "Servo is becoming
truly embeddable this year."
So, you know, we're
halfway through the year.
I'm excited to see what's to come.
But for now in the documentation, there's
nothing about how to embed or anything.
James Munns: The year of servo on the embedded.
Amos Wenger: Speaking of embedded, what most games and
like, for example, the League of Legends
clients, Riot Games client and all that,
all the applications that are mostly
Electron, but actually not, they're using
the Chromium embedded framework, CEF.
There's a crate for that, which entire
page just says, use the
Chromium embedded framework in Rust.
And that's it.
James Munns: I love it.
Straight into the point.
Yes.
With a version number
that definitely makes sense.
Amos Wenger: It matches the Chromium version.
You can tell it's automated because it
was less published three days ago.
So I'm like, and it works, right?
I tried their example and it works.
You can have your own
little browser with no Chrome.
Chrome is the browser interface, like the
address bar, next previous button, the
menus and everything.
That's the Chrome.
Firefox has a Chrome.
It's written in CSS and JavaScript.
But I don't know much about CEF and that
simple demo worked, but there's an
offscreen rendering demo
and that one didn't work.
So I'm assuming it was coded against
Windows or something.
And you need this whole kind of app
bundle structure with all the libraries
in the right places.
It seems like it's a lot of effort.
I just, I gave up on that avenue.
I'm sure it's probably the right way to
go at this point because when I look at
Servo, for example, I load it, my voice
out in it to see it.
And it supports a lot of things.
But for example, I have SVG diagrams that
use HTML text formatting and it only
showed the shapes and it didn't show the
text because it didn't support like
foreign objects, nodes in
SVG, which is kind of a bummer.
Finally, I figured the easiest way to
experiment with all this would be to use
Electron because Electron has an
offscreen rendering API and you can just
write a bit of JavaScript code.
And the way I started experimenting with
this is just listening on some TCP port
for some RPC requests,
which is just a line of JSON.
There's a bunch of ways to capture the
contents of the webpage and one of them
is CPU, the other is GPU.
I only tried the CPU because GPU is
essentially you get a shared texture
handle and then you need to open it and
like copy from the, or try not to copy
from the OpenFX plugin.
I'm pretty sure there would be a way to,
instead of filling out the buffer, say,
hey, here's the shared texture handle, go
take that, go use that, I don't know.
Yeah, it gives those nice images.
It's, this is not different from all the
other slides, but that one was generated
with the whole Electron pipeline.
And I figured, yeah, okay, the plugin
could like start Electron in the
background, send those RPC
requests, get the images back.
But do you have any idea how long it
takes to generate one
frame, one 4K frame?
James Munns: I bet it's not gonna be
three to four milliseconds.
Amos Wenger: It's not three to four milliseconds.
Okay.
650 milliseconds.
James Munns: Okay, love that, love that.
Amos Wenger: So in that there's like Node.js startup
time connecting over TCP
socket, this is not a good test.
And also it's encoding everything to PNG.
So you can definitely shave
off a bunch of times here.
There's next steps,
more research is required.
And next season is
required for this for sure.
Because I think the shared texture handle
thing is the way to go if doing Electron.
I think embedding servo is probably a lot
better, but then I started thinking
bigger picture, even if everything worked
well and was low latency, like you get
real time playback and everything.
You now have some interface inside the
host, inside DaVinci Resolve.
And ideally you'd also want to be able
to, if you're doing motion design, if you
have like bits of code, like you can zoom
in and highlight and whatnot, that's
something you need to do
inside of your web contents.
Right, you can't do it in the venture
resolve because if you zoom in, if you
have a clip, you put it in your scene and
then you zoom that in,
it's gonna be pixelated.
All right, does that make sense?
James Munns: Yeah, yeah, yeah.
You're doing like pan and scan on a
rasterized image, which means either the
rasterized image needs to be huge, so you
can zoom into whatever fidelity you want,
or you want it rendered on the fly, which
is more like a camera flying around.
Amos Wenger: Exactly.
So that means now that all your
positioning, your zooming, everything
needs to be a part, that needs to be a
parameter, that you can keyframe inside
the host, or it needs to be part of a
scene that you can kind of open in a
separate editor, a web-based editor, and
then the OpenFX plugin is just a
renderer, and it's just like giving you a
frame of a predetermined scene.
And you still have the problem that I
often have, which is you usually want to
do the animation synchronized with some
audio, right, someone's talking, and then
you want the code to come in as they talk
about different things.
And if your editor is outside of resolve,
you're not gonna get the
audio from the resolve timeline.
At that point, the only reasonable course
of action is to make your
own nonlinear editor, right?
James Munns: I was wondering if it
was gonna come to this.
Amos Wenger: Well, I certainly
haven't done it in a day.
I'm not planning on doing it because how
do you hire editors for your own
nonlinear editor software?
But that would be the way to do it.
James Munns: You sound like you're standing on the
ledge, and I'm not sure whether to slowly
talk you down or not, or if you're
excited for this jump into the unknown.
Amos Wenger: On top of Gear Acquisition Syndrome, I
have Software Replacement Syndrome.
I use software too much.
DaVinci Resolve used to be this
really intimidating, big piece of
software that I didn't
know how to deal with.
And now I kind of see the seams of the
different parts of the
things that they pulled together.
And I think you could do better if you
started from scratch.
The problem is, of course, all the
unknown unknowns, all the things that
they got absolutely right, and I have no
idea how to do high-performance image
processing using shaders.
No idea about that.
James Munns: Or just 30 years of paper cut shaving.
There's a huge amount of, with Second
System Syndrome, where you're like, "I
could make it so much easier."
And you realize that so much of the
things that make it painful are like,
"Oh, we decided it was worth that pain,
accumulated over 30 years."
You know what I mean?
But yeah.
You cheat by doing less, so if you could
figure out how to do something that only
matters for you, hell yeah.
But yeah, I imagine
it's gonna be a thing.
Amos Wenger: Well, I'm gonna try not to
go any further down that road.
Wish me luck.
Amanda Majorowicz: Famous last words.
I think we both know.
James Munns: That road does not end.
I was gonna say, we are both the epitome
of if you give a mouse a cookie.
We figured out scripting to, okay, well,
I guess I need to
replace DaVinci Resolve.
(Laughs)
Amos Wenger: But I, no, no, I-- I will be strong.
James Munns: All right, I believe in you.
Thank you.
I'm looking forward to next season.
(Laughs)
James Munns: Me too.
This episode is sponsored by Depot,
the build acceleration
platform that's on a mission
to make all builds near instant.
If you're tired of watching your builds
and GitHub actions crawl
like the modern day equivalent
of paint drying, give Depot's GitHub
Actions runners a try.
They're up to 10 times faster with
unlimited concurrency,
faster caching, support for
Linux, macOS and Windows,
and they plug right into
other Depot optimizations
like accelerated container image builds
and remote caching for
Bazel, Turborepo, Gradle and more.
Depot is built by developers who are
tired of wasting time
waiting on builds instead of shipping.
It's made for teams
that wanna move faster
and stay focused on
what actually matters.
That's why companies
like PostHog use Depot
to cut build times from over three hours
to just three minutes,
saving tens of thousands
of build hours every week.
Start your free seven
day trial at depot.dev
and let them know we sent you.