WEBVTT NOTE This file was generated by Descript 00:00:14.226 --> 00:00:15.896 This is Self Directed Research. 00:00:16.136 --> 00:00:19.466 James and Amos, our hosts, get really excited about different topics, 00:00:19.476 --> 00:00:22.446 and each week, they take turns presenting their ideas to each other. 00:00:23.056 --> 00:00:25.456 We have the video version ready for this episode, too. 00:00:25.915 --> 00:00:29.749 Check out the website, YouTube, or Spotify to see the presentation in action. 00:00:30.019 --> 00:00:35.339 You can visit sdr-podcast.com/episodes for previous episodes, more presentations, 00:00:35.389 --> 00:00:36.839 show notes, and transcripts. 00:00:37.169 --> 00:00:39.029 New episodes are published every Wednesday. 00:00:39.629 --> 00:00:43.459 For this week's episode, after a short message from James, Amos presents 00:00:43.599 --> 00:00:45.019 "Fixing build times with rubicon." 00:00:46.130 --> 00:00:48.760 The Self-Directed Research Podcast is looking for sponsors. 00:00:48.800 --> 00:00:52.190 If you would like to promote your company, project, conference, or open 00:00:52.190 --> 00:00:55.910 job positions, stay tuned to the end of the episode and send us an email to 00:00:55.910 --> 00:00:59.540 contact@sdr-podcast.com for more info. 00:01:06.356 --> 00:01:09.254 Amazingly, the podcast thing that we use generates file 00:01:09.274 --> 00:01:12.474 named `full-name--pronouns.mp4`. 00:01:13.844 --> 00:01:15.724 So it pops all those notifications with your pronouns. 00:01:16.192 --> 00:01:18.655 And as we discussed earlier, I just want to have this on the record: it 00:01:18.655 --> 00:01:21.461 shows that I'm recording at 19 kilohertz, which is not a thing. 00:01:21.471 --> 00:01:25.394 What's actually happening is that we're recording at 192 kilohertz and that 00:01:25.404 --> 00:01:28.314 they only tested for 48- and 41,000. 00:01:28.334 --> 00:01:31.714 And so they only take the first two characters of the full string. 00:01:32.044 --> 00:01:34.382 And so it chose 48 for James and 19 for me. 00:01:34.559 --> 00:01:36.269 Which it took me a while to wrap my head around 00:01:36.269 --> 00:01:42.569 because in serial ports, you might have a baud rate of 19200 or 19.2. 00:01:42.839 --> 00:01:46.819 So I was like, "It's weird that they pick like a serial baud rate for your audio 00:01:46.819 --> 00:01:53.352 interface," until I realized that you were talking about 192,000 and not 19,200. 00:01:54.592 --> 00:01:56.289 I mean, it's, it's a Scarlett 2i2. 00:01:56.289 --> 00:02:00.519 I was kind of surprised if it was going worse than whatever USB mic. 00:02:00.909 --> 00:02:01.159 Yeah. 00:02:02.674 --> 00:02:07.103 Okay, so today's presentation is called "Fixing 00:02:07.103 --> 00:02:10.653 build times with rubicon: a somewhat reasonable use of dynamic linking." 00:02:10.673 --> 00:02:13.733 So in Self-Directed Research tradition, we have a nice subtitle. 00:02:14.223 --> 00:02:17.593 This is the follow-up to "I Was Wrong About Rust Build Times." 00:02:18.053 --> 00:02:19.093 Our very first episode. 00:02:19.663 --> 00:02:21.764 Yes, I was wrong to say you should just split 00:02:21.764 --> 00:02:23.114 things into different crates. 00:02:23.394 --> 00:02:26.558 Actually, you should split things into different dynamic libraries. 00:02:26.862 --> 00:02:31.242 And immediately, if you've tried that, you might be thinking, "Oh, but rustc 00:02:31.262 --> 00:02:32.722 lets you do that, cargo lets you do that." 00:02:33.092 --> 00:02:35.872 And it kind of does, but also not in a way that makes me happy. 00:02:35.872 --> 00:02:38.191 So we're going to get into that, this is why I made rubicon. 00:02:38.233 --> 00:02:39.223 Let's just follow the slides along. 00:02:40.328 --> 00:02:42.078 So building large Rust projects is slow. 00:02:42.798 --> 00:02:46.228 And by large Rust projects, I mean, like anything that's trying to do webby 00:02:46.238 --> 00:02:50.751 stuff, like HTTP and whatnot serving endpoints- is going to have anywhere 00:02:50.751 --> 00:02:56.705 from a hundred crates at the beginning to like, it can easily go up to 300, 400. 00:02:56.705 --> 00:02:58.451 It's easy to rack up dependencies. 00:02:58.538 --> 00:02:59.858 And it's really hard to shed them. 00:02:59.878 --> 00:03:04.650 I have tried for my own website, but, uh, sometimes you just need to transcode 00:03:04.710 --> 00:03:08.469 images or videos or something, minify CSS. 00:03:08.469 --> 00:03:09.596 There's always a lot of things to do. 00:03:09.596 --> 00:03:13.256 So you can either call an external tool or you can link to something. 00:03:13.256 --> 00:03:15.186 And if you link to something, then that's more build time. 00:03:15.741 --> 00:03:18.201 Why is it that slow to build these projects? 00:03:18.221 --> 00:03:20.531 Well, because simply, there's just a lot of code. 00:03:20.531 --> 00:03:21.411 There's just lots to do. 00:03:21.611 --> 00:03:24.471 There's lots of code to parse, and then to typecheck, and then to 00:03:24.471 --> 00:03:28.751 borrowcheck, lots to codegen, lots to optimize, and then lots to link. 00:03:29.821 --> 00:03:33.806 And rustc already has incremental builds, which are enabled by 00:03:33.806 --> 00:03:37.676 default for development, I believe, but not for release builds? 00:03:38.176 --> 00:03:38.986 Does that sound true? 00:03:39.121 --> 00:03:41.861 I'd have to check the- I know on default profiles incremental's 00:03:41.881 --> 00:03:43.861 turned on, I can't remember. 00:03:43.861 --> 00:03:46.741 And a lot of people turn 'em off in CI too, because when you're doing a clean 00:03:46.741 --> 00:03:50.131 build in CI, you want them off because they just add overhead, if you're 00:03:50.131 --> 00:03:52.741 never- if you're gonna throw away the environment as soon as you're done. 00:03:52.741 --> 00:03:56.703 So I know a lot of people in their CI like production builds 00:03:56.703 --> 00:04:00.243 turn them off completely, even if the release profile has them on. 00:04:00.243 --> 00:04:04.523 Just because you go: well, it adds 10% to the build time for context, 00:04:04.523 --> 00:04:06.143 I'm gonna throw away immediately. 00:04:07.178 --> 00:04:10.437 Yeah, even people who actually store cache with like the 00:04:10.437 --> 00:04:14.257 rust-cache action on GitHub Actions tend to disable it for some reason. 00:04:14.257 --> 00:04:17.447 I know for sccache, it doesn't really work for the incremental builds. 00:04:17.757 --> 00:04:21.372 So incremental builds are sort of unfinished, let's say. 00:04:21.922 --> 00:04:25.542 Also even when they do work for you, they're not incremental enough. 00:04:25.542 --> 00:04:28.767 Not everything has been incrementalized, if that makes sense. 00:04:28.767 --> 00:04:32.097 It's something that you, as a rustc developer, you need to manually do for all 00:04:32.097 --> 00:04:35.907 sorts of different tasks and just not all of them have been done due to lack of time 00:04:35.907 --> 00:04:38.447 and funding and mental health, as always. 00:04:39.122 --> 00:04:42.622 Cache invalidation is not the fun problem to solve most of the time. 00:04:43.322 --> 00:04:44.182 No, it's not. 00:04:44.331 --> 00:04:47.791 It's, it's pretty- well, we have crater, so when, when they make a 00:04:47.791 --> 00:04:51.311 change to rustc, it's actually tested against, I want to say all the crates, 00:04:51.311 --> 00:04:52.781 but is that actually all the crates? 00:04:52.931 --> 00:04:54.221 I'm not actually sure it's all the crates. 00:04:54.221 --> 00:04:56.289 I mean, there's not all the publicly available ones, right? 00:04:56.289 --> 00:04:58.705 It's all the one on crates.io and then some 00:04:58.705 --> 00:05:00.638 smattering of GitHub projects. 00:05:00.638 --> 00:05:05.725 But if you have a private infrastructure at your large tech company, then it's 00:05:05.765 --> 00:05:10.155 obviously not going to be touching all of your private, you know, "only tracked 00:05:10.155 --> 00:05:11.765 in your internal repo" type stuff. 00:05:12.595 --> 00:05:15.685 Well, chances are large companies don't use cargo anyway. 00:05:15.764 --> 00:05:16.774 They have their own build system. 00:05:17.199 --> 00:05:19.289 But yeah, ? Why is incremental not enough? 00:05:19.319 --> 00:05:20.679 Because it's not incremental enough. 00:05:20.802 --> 00:05:23.352 Proc macros aren't cached because they can have side effects. 00:05:23.392 --> 00:05:25.422 So you just simply cannot cache them. 00:05:25.832 --> 00:05:28.382 So you have to build them and run them. 00:05:28.712 --> 00:05:32.392 And then if they then produce the same code I guess you can... 00:05:32.912 --> 00:05:33.722 not do anything? 00:05:33.722 --> 00:05:36.592 Just reuse the artifact from last time, but you still had to run them. 00:05:36.936 --> 00:05:40.616 So if you use a lot of proc macros, it might make sense to build them with 00:05:40.616 --> 00:05:46.066 optimizations, which is a bad situation to be in, but that's how it is sometimes. 00:05:46.440 --> 00:05:48.900 Static linking takes time because you're generating files that can 00:05:48.900 --> 00:05:52.030 be hundreds of megabytes with debug information and everything, and 00:05:52.080 --> 00:05:55.120 you're just copying a lot of things in different places and doing relocations. 00:05:55.120 --> 00:05:56.140 It's still, it's a lot of work. 00:05:56.140 --> 00:05:58.580 We've done, we've made faster linkers, but it's still a lot of work. 00:05:58.960 --> 00:06:00.330 Link time optimization takes time. 00:06:00.330 --> 00:06:04.441 You're, you're delaying optimizations from compile time to link time, which, 00:06:04.539 --> 00:06:08.032 all happens within the same cargo invocation, but still: it's work. 00:06:08.042 --> 00:06:09.542 There's just a lot of work to do. 00:06:10.015 --> 00:06:13.342 And like I said, there's a chance that other big companies, use something 00:06:13.342 --> 00:06:16.702 that isn't cargo, they use Bazel, they use Buck, they use something 00:06:16.702 --> 00:06:18.024 else we haven't heard of yet. 00:06:18.351 --> 00:06:24.471 And those build systems are typically designed to avoid doing any work 00:06:24.881 --> 00:06:28.841 that doesn't need to be done by making everything content-addressed. 00:06:28.911 --> 00:06:31.184 And yeah, they're aiming for perfect caching. 00:06:31.194 --> 00:06:34.324 And they think about remote building as a first class citizen. 00:06:34.324 --> 00:06:38.055 Constraints that, yeah, weren't there when cargo was designed. 00:06:38.597 --> 00:06:43.266 And it's worth mentioning that if, I don't know, I tend to think that 00:06:43.266 --> 00:06:46.766 rustc needs more resources, but I think that's even more true of cargo. 00:06:47.417 --> 00:06:48.377 I know it's changed recently. 00:06:48.481 --> 00:06:53.174 It's gone from being like just the Alex Crichton show to now there's quite 00:06:53.174 --> 00:06:54.704 a few more full-time contributors. 00:06:54.714 --> 00:06:57.744 So I think that trend has been changing and I've started seeing a lot of 00:06:57.754 --> 00:07:01.624 motion on stuff that had been stalled for a long time and I think other 00:07:01.634 --> 00:07:02.854 people are seeing the same thing. 00:07:02.864 --> 00:07:05.774 Because cargo is one of those- like you mentioned all these other build tools- 00:07:06.024 --> 00:07:09.574 you'll have these big companies that have internal tooling teams and a consistent 00:07:09.574 --> 00:07:13.942 environment and consistent resources for build servers and things like that. 00:07:14.166 --> 00:07:17.128 Cargo is kind of hard because it's gotta fit for everyone. 00:07:17.168 --> 00:07:23.478 It's an amazing tool for like, the center 95 percent of mass of the problem, but... 00:07:24.338 --> 00:07:27.494 like you said, when you are designing something that ends up having 800 00:07:27.864 --> 00:07:32.544 dependencies that change, that all have proc macros, like, just extenuating 00:07:32.544 --> 00:07:37.114 circumstances to death, it becomes hard to have a one size fit all tool 00:07:37.534 --> 00:07:41.344 if you aren't willing to constrain the problem, like: hey, we always build 00:07:41.364 --> 00:07:45.604 with these build machines and we always have these profiles or whatever and 00:07:45.604 --> 00:07:49.184 whatever that companies can get away with when you control the environment. 00:07:49.364 --> 00:07:53.045 Yeah, there's only so much you can do without completely 00:07:53.045 --> 00:07:56.065 breaking compatibility and saying, "Okay, everyone who uses cargo now 00:07:56.065 --> 00:07:58.005 has to migrate over to cargo 2." 00:07:58.005 --> 00:08:00.405 And then that's, I don't think that's something that anybody wants. 00:08:00.415 --> 00:08:03.775 So anytime you introduce something new in cargo, it has to be opt-in or 00:08:03.775 --> 00:08:06.865 it has to be compatible, support both versions, like the edition system. 00:08:07.295 --> 00:08:08.875 It's, it's a hard job. 00:08:09.065 --> 00:08:13.685 I have respect for the maintainers of cargo, even though I'm the first one to 00:08:13.685 --> 00:08:14.955 complain when things take too much time. 00:08:15.495 --> 00:08:18.505 So building big projects take time. 00:08:18.795 --> 00:08:22.975 And like I said, the Rust compiler already knows how to use dynamic libraries. 00:08:22.975 --> 00:08:23.785 There's an attribute. 00:08:23.785 --> 00:08:27.105 You can put it into a cargo manifest called `crate-type` under `[lib]`. 00:08:27.575 --> 00:08:30.075 You can set it by default for libraries. 00:08:30.075 --> 00:08:31.197 I think it's rlib? 00:08:31.322 --> 00:08:34.302 And then for executables, what is it for executables? 00:08:34.312 --> 00:08:35.422 Just static or something? 00:08:35.472 --> 00:08:35.922 I don't know. 00:08:35.922 --> 00:08:36.902 `staticlib`? 00:08:36.922 --> 00:08:37.132 No. 00:08:37.422 --> 00:08:38.802 It's just, uh, binary. 00:08:38.958 --> 00:08:42.588 Yeah, `staticlib` is one, but I think it's just `bin` for binaries 00:08:42.608 --> 00:08:45.938 when you're doing a fully linked, statically linked executable. 00:08:46.163 --> 00:08:46.733 That makes sense. 00:08:47.103 --> 00:08:47.443 Yeah. 00:08:47.693 --> 00:08:50.900 So, but because the, the only ones you really said by hand 00:08:51.230 --> 00:08:56.267 are like, well, `dylib`, or "die lib" as you call it or `cdylib`. 00:08:56.317 --> 00:08:59.897 So it has that capability, but it doesn't help much because it's still 00:08:59.917 --> 00:09:04.137 one big graph when you do `cargo build`, if you have this big project 00:09:04.137 --> 00:09:08.027 with 700 dependencies and all sorts of features enabled and disabled, it has 00:09:08.027 --> 00:09:09.803 to do a lot of work to just figure out: 00:09:09.803 --> 00:09:10.697 what is in that graph? 00:09:10.697 --> 00:09:11.257 What is enabled? 00:09:11.257 --> 00:09:11.977 What is disabled? 00:09:12.168 --> 00:09:15.495 And then load a bunch of information that it's already 00:09:15.495 --> 00:09:18.086 computed from last time from disk. 00:09:18.606 --> 00:09:21.348 And it's actually really cool how they do that. 00:09:21.358 --> 00:09:26.761 They have a package called `odht` on-disk hash table for hash tables 00:09:26.761 --> 00:09:31.654 that do not fit in memory and/or that you do want to persist to the disk. 00:09:31.722 --> 00:09:35.212 So it's, it's a really, really cool trick, but it also means, 00:09:35.212 --> 00:09:36.632 uh, usually lots of I/O. 00:09:36.632 --> 00:09:41.847 I feel like if you measure the amount of syscalls that cargo has to do for 00:09:41.847 --> 00:09:44.667 a no-op build you will realize that. 00:09:45.917 --> 00:09:46.280 Yeah, 00:09:47.180 --> 00:09:51.040 . So, not only with d-y-lib or dy-lib is it still one big graph. 00:09:51.103 --> 00:09:53.323 Cargo has a lot of work to do, to come to the conclusion that there 00:09:53.413 --> 00:09:54.773 is actually no work to be done. 00:09:55.143 --> 00:09:57.580 But it's also kind of, informational. 00:09:58.020 --> 00:10:00.760 It's "prefer-dynamic linking". 00:10:00.800 --> 00:10:03.020 It's not "do dynamic linking". 00:10:03.030 --> 00:10:05.284 You're, you're asking nicely, just like `#[inline]`. 00:10:05.340 --> 00:10:07.645 I was gonna say like `#[inline]` versus `#[inline(always)]`. 00:10:07.665 --> 00:10:08.335 Exactly. 00:10:08.436 --> 00:10:12.486 It's a global setting saying, "Hey compiler, if you can find any boundaries 00:10:12.506 --> 00:10:15.580 where you can do dynamic linking instead of static linking, please do that. 00:10:15.890 --> 00:10:19.820 And then I guess you find out the result by just looking at your target folder. 00:10:19.820 --> 00:10:23.530 And if there's a bunch of `.so`s or `.dylib`s or dlls, 00:10:23.560 --> 00:10:24.720 then I guess it found some. 00:10:25.050 --> 00:10:29.390 It's a very active area of development of rustc because the docs for this were 00:10:29.390 --> 00:10:32.190 wrong for four years and nobody noticed until I submitted a pull request. 00:10:32.220 --> 00:10:34.020 I love bragging about my little pull requests. 00:10:34.030 --> 00:10:35.530 I'm like, "I read the docs, nobody did! 00:10:35.550 --> 00:10:37.310 Or: noticed the mistake, nobody noticed." 00:10:37.860 --> 00:10:38.890 and now it's correct. 00:10:39.120 --> 00:10:39.760 So, yay!!! 00:10:40.150 --> 00:10:40.720 For me!! 00:10:41.047 --> 00:10:45.027 So because building big projects is hard, let's build smaller projects instead. 00:10:45.387 --> 00:10:48.787 Let's compose smaller projects together so you can pick your own 00:10:48.787 --> 00:10:51.107 boundaries, if that makes sense, because maybe you don't want libtokio. 00:10:51.117 --> 00:10:53.997 Maybe you just want like lib download files from somewhere, 00:10:53.997 --> 00:10:55.799 or lib use object storage 00:10:55.826 --> 00:10:58.326 . And, this isn't so weird, it's already kind of like that today: 00:10:58.326 --> 00:11:01.926 as a maintainer, you have to pick where your crate boundaries are, and when do 00:11:01.926 --> 00:11:05.606 I split this up into multiple crates, versus when is the orphan rule so annoying 00:11:05.606 --> 00:11:07.796 it's worth to have one really big crate, 00:11:08.130 --> 00:11:08.590 . Yes. 00:11:09.150 --> 00:11:13.010 Unfortunately, making that boundary happen is trivial for 00:11:13.270 --> 00:11:14.394 command line interface binaries. 00:11:14.442 --> 00:11:18.745 It's trivial for HTTP servers trivial for GRPC, but it's real tricky 00:11:18.865 --> 00:11:21.075 with dlopen Why is it tricky? 00:11:21.115 --> 00:11:22.835 Because Rust has no stable ABI. 00:11:23.188 --> 00:11:24.678 That's not a bug, that's a feature. 00:11:24.778 --> 00:11:27.872 It lets compiler developers experiment with optimizations. 00:11:28.642 --> 00:11:31.222 They're not, you know, bound to a specific way of doing things. 00:11:31.242 --> 00:11:34.792 They can, if they need to, change things from one version to the other. 00:11:34.913 --> 00:11:37.753 Because in the README for rubicon, I said, "Okay, you have to respect a few rules. 00:11:37.753 --> 00:11:39.683 One of them is build everything with the same rustc version." 00:11:39.703 --> 00:11:43.033 And then someone was like, "Well, if you enable the option that randomizes all 00:11:43.043 --> 00:11:44.673 struct layouts, it would still break." 00:11:44.673 --> 00:11:46.053 I was like, "Yes, but if you do that... 00:11:46.193 --> 00:11:48.173 I mean, you're kind of asking for it, literally." 00:11:48.173 --> 00:11:50.683 And then the other problem is that globals get duplicated. 00:11:51.048 --> 00:11:54.863 Because, well, if you trust rustc to do the dynamic linking, you 00:11:54.863 --> 00:11:57.133 will be disappointed because you have to patch everything. 00:11:57.293 --> 00:12:00.003 It's going to pick its own boundaries that are not necessarily going to 00:12:00.003 --> 00:12:04.153 match what you want, and then it's going to rebuild more than you want. 00:12:04.183 --> 00:12:05.087 Like, it's surprising. 00:12:05.087 --> 00:12:07.377 You change the call site and it rebuilds libtokio. 00:12:07.717 --> 00:12:11.037 But if you do it yourself, by design- like if you, I don't know- you 00:12:11.037 --> 00:12:14.268 have your main program and then you have a module, dynamically loaded 00:12:14.278 --> 00:12:16.753 module that talks to object storage. 00:12:16.753 --> 00:12:18.373 So something like Amazon S3. 00:12:18.993 --> 00:12:23.008 Well, they both have tokio, so you have two copies of tokio now, and the 00:12:23.008 --> 00:12:25.628 problem is tokio uses globals to work. 00:12:25.648 --> 00:12:27.418 For example the current runtime. 00:12:27.418 --> 00:12:30.228 I think we've already talked about that at some point I don't know 00:12:30.808 --> 00:12:32.668 when or if it will be published... 00:12:33.438 --> 00:12:37.342 But you have that problem where the application starts the tokio runtime, 00:12:37.372 --> 00:12:41.212 it calls into the module and the module says, "There's no ambient runtime. 00:12:41.212 --> 00:12:45.187 This is thread does not currently have a runtime set up," and they're both right. 00:12:45.577 --> 00:12:48.437 As A thinks there's a runtime, B thinks there's no runtime because there's two 00:12:48.437 --> 00:12:53.717 copies of the thread-local variable that stores what runtime is currently active- 00:12:53.767 --> 00:12:55.557 the ambient runtime, as I like to call it. 00:12:55.716 --> 00:12:57.896 You've now opened yourself to a multiverse. 00:12:57.896 --> 00:13:02.046 You've gone full Marvel, and now you have a multiverse of worlds where one 00:13:02.056 --> 00:13:07.131 multiverse is a Globals exist and the other one doesn't and they get very 00:13:07.131 --> 00:13:10.851 confused when you try and pretend that there's only one true universe. 00:13:10.901 --> 00:13:11.281 Yes.. 00:13:11.341 --> 00:13:12.661 But it gets worse than that. 00:13:12.751 --> 00:13:15.881 Because having several copies of tokio, okay, find a way to 00:13:15.891 --> 00:13:17.151 make them use the same global. 00:13:17.291 --> 00:13:19.311 You can pass, the, the runtime handle explicitly. 00:13:19.311 --> 00:13:21.084 That's fine, . Here's another scenario. 00:13:21.084 --> 00:13:23.354 The app installs a tracing subscriber, which is what you usually 00:13:23.354 --> 00:13:24.254 do at the beginning of an app. 00:13:24.274 --> 00:13:27.868 You say like: Okay, uh, tracing events should go to the console or they 00:13:27.868 --> 00:13:32.128 should go to that file as JSON or pipe to somewhere, my log pipeline. 00:13:32.770 --> 00:13:36.611 And then if you have some other modules that also use tracing... 00:13:37.020 --> 00:13:42.180 again, their version of tracing has a different copy of the global that 00:13:42.180 --> 00:13:45.270 says: here's what the subscriber is for that process of that thread. 00:13:45.587 --> 00:13:47.728 And so their events silently go nowhere. 00:13:48.038 --> 00:13:50.687 So if you're trying to debug why they don't work and you're like, "Hey, we can't 00:13:50.687 --> 00:13:51.997 enable logging for them, that's weird..." 00:13:51.997 --> 00:13:54.243 and you're like, "Are we using the wrong syntax? 00:13:54.243 --> 00:13:54.813 What's happening? 00:13:54.813 --> 00:13:56.653 Are we not setting the environment variables in the right place?" 00:13:56.653 --> 00:13:57.603 And you're like, "Oh, okay. 00:13:57.653 --> 00:13:59.291 It's just different copies of the global. 00:13:59.609 --> 00:14:00.979 Even worse: panic handlers. 00:14:01.009 --> 00:14:01.699 Again, those are process-wide... 00:14:02.862 --> 00:14:05.832 This one is tricky because , even if you use rubicon, you actually have to use 00:14:05.832 --> 00:14:10.532 prefer-dynamic because they need to use the same libstd because otherwise you 00:14:10.532 --> 00:14:16.346 have one panic handler per shared object and anything like `color-backtrace`, 00:14:16.366 --> 00:14:19.726 whatever, anything that would record crashes from the application, like 00:14:19.736 --> 00:14:24.193 Sentry to some sort of web service wouldn't work if it happened in a module. 00:14:24.724 --> 00:14:26.174 And something even more fun. 00:14:26.234 --> 00:14:29.688 So if you try to work around the fact that modules think that there's no, 00:14:29.708 --> 00:14:33.848 there's no running or ambient tokio runtime and you pass a handle explicitly- 00:14:33.938 --> 00:14:39.408 well, because tokio uses parking_lot and parking_lot also has globals it has a 00:14:39.408 --> 00:14:44.048 global table of threads that need to be woken up in the case of certain events. 00:14:44.883 --> 00:14:50.743 And so what happens is that not only can it just hang forever sometimes, 00:14:50.773 --> 00:14:54.617 because it's, it gets an event from the wrong copy of the parking_lot 00:14:54.637 --> 00:14:57.387 code and it puts it in wrong global and something's never checked. 00:14:57.507 --> 00:15:01.137 It can also sometimes wake up the wrong thread and I've had memory corruption 00:15:01.157 --> 00:15:05.697 because there's some unsafe code in there that assumes that there's only one copy of 00:15:05.697 --> 00:15:08.887 globals, which is a perfectly reasonable assumption to make, to be fair... 00:15:08.917 --> 00:15:10.297 but it's just not true anymore. 00:15:10.577 --> 00:15:13.135 So yeah, uh, that was fun. 00:15:13.160 --> 00:15:15.530 You're really selling me on dynamic linking right now 00:15:16.280 --> 00:15:18.937 It's just- it's if you thought you like Rust, but 00:15:18.937 --> 00:15:21.547 you missed all the memory and safety and then random crashing. 00:15:22.071 --> 00:15:27.100 So, it was supposed to be a quick hack, all right. 00:15:27.450 --> 00:15:30.830 I thought I would do just enough crimes and just be careful, which 00:15:30.830 --> 00:15:32.180 is something I say never works... 00:15:32.250 --> 00:15:35.860 it's a typical case of do as I say, not as I do, 00:15:36.230 --> 00:15:37.850 You should use a memory safe language. 00:15:38.320 --> 00:15:40.866 You can't just be careful because everybody makes mistakes. 00:15:40.899 --> 00:15:42.459 "Just be careful" is not a plan. 00:15:42.809 --> 00:15:43.923 It's wishful thinking. 00:15:44.173 --> 00:15:47.303 But it was going to be enough for me because I'm special, right? 00:15:47.902 --> 00:15:50.472 And, uh, no, I just spent eight weeks debugging this. 00:15:50.582 --> 00:15:52.262 Mostly memory corruption and hangs... 00:15:52.262 --> 00:15:53.367 Think of all the learning! 00:15:53.397 --> 00:15:55.207 There was so much learning going on! 00:15:55.347 --> 00:15:58.507 I had to relearn to use a debugger. 00:15:58.747 --> 00:16:02.371 I'm on macOS now, so GDB is kind of, a 5th class citizen. 00:16:02.851 --> 00:16:03.981 So I have to use LLDB. 00:16:04.611 --> 00:16:06.851 And yeah you- you learn about internals. 00:16:07.044 --> 00:16:10.959 And then the last two weeks of that, were actually a tokio bug. 00:16:10.982 --> 00:16:15.692 I was getting weird crashes around intrusive linked lists for two weeks 00:16:15.692 --> 00:16:18.409 and I posted about it on Discord because that's what I do, all right? 00:16:18.439 --> 00:16:23.383 I post completely incomprehensible screenshots and then people just kind 00:16:23.383 --> 00:16:24.720 of go, "Oh he's cooking something!" 00:16:24.902 --> 00:16:27.523 It's one of those things though, when you get used to Rust, 00:16:27.523 --> 00:16:30.653 especially when you're writing unsafe code: if something goes wrong, you go, "It 00:16:30.653 --> 00:16:34.983 must be me," like, "Other people surely would have done the right thing, and..." 00:16:35.423 --> 00:16:39.163 so it's always such a surprise when you go, "Oh no, there was just a weird 00:16:39.193 --> 00:16:42.583 edge case-" which is such a- that's almost the default assumption for my 00:16:42.583 --> 00:16:44.743 days in like C and things like that. 00:16:44.743 --> 00:16:47.493 You go, "Well, someone must have screwed up," but in Rust, it 00:16:47.493 --> 00:16:48.403 feels like it's the opposite. 00:16:48.413 --> 00:16:50.453 You go, "I must have screwed up somewhere." 00:16:50.650 --> 00:16:54.620 Well, not only that, but also I was doing pretty explicit crimes, right? 00:16:54.650 --> 00:16:57.220 Anything that went wrong at this point, especially memory corruption 00:16:57.220 --> 00:17:01.320 related, must have been because I was like trying to deduplicate globals, 00:17:01.365 --> 00:17:06.315 but no, what happened was: when I forked tokio to add support for rubicon , there 00:17:06.315 --> 00:17:07.875 was a bug in the default branch. 00:17:07.925 --> 00:17:10.248 . At some point, I was just checking the issue tracker... 00:17:10.838 --> 00:17:13.978 And I was like, "Wait- wait- wait a fucking second, when was 00:17:13.978 --> 00:17:15.808 anyone gonna ping me about this?" 00:17:16.368 --> 00:17:17.178 I tweeted about it. 00:17:17.178 --> 00:17:17.958 This is how it works, right? 00:17:17.958 --> 00:17:20.808 You just tweet a screenshot and then people ping you when it's fixed. 00:17:20.838 --> 00:17:21.108 Right? 00:17:21.168 --> 00:17:22.938 That, that's, that's my normal experience. 00:17:23.422 --> 00:17:26.120 So now that it actually works, I'm pretty confident about it. 00:17:26.120 --> 00:17:29.602 I've deployed the version of my website that uses that, maybe a month ago, 00:17:29.696 --> 00:17:33.186 Uh, I present rubicon, which is just a way to import and export globals. 00:17:33.186 --> 00:17:35.426 You just wrap things in macros. 00:17:35.446 --> 00:17:38.176 You wrap your thread-locals and your process-locals. 00:17:38.229 --> 00:17:39.329 , nobody talks like this. 00:17:39.402 --> 00:17:41.084 I just like the symmetry, you know? 00:17:41.134 --> 00:17:41.334 You're 00:17:41.725 --> 00:17:41.925 inventing 00:17:42.058 --> 00:17:45.968 precise vocabulary for a problem that you are the explorer of. 00:17:46.918 --> 00:17:48.578 A problem I made for myself, yes. 00:17:48.740 --> 00:17:52.191 So rubicon works through source-level patching, I have patches ready 00:17:52.191 --> 00:17:56.531 for parking_lot, tokio, tracing, and eyre- that's e y r e. 00:17:56.821 --> 00:17:58.531 Those are very small patch sets, 00:17:58.531 --> 00:18:01.411 The rubicon README goes into the specific setup you have 00:18:01.411 --> 00:18:02.661 to do if you want to use this. 00:18:02.701 --> 00:18:05.711 I'm not aware of anyone else using this, but everyone's missing out 00:18:05.761 --> 00:18:07.541 because my link times are great now. 00:18:07.571 --> 00:18:08.781 I used to have to wait like... 00:18:09.211 --> 00:18:09.921 I don't know... 00:18:09.991 --> 00:18:14.448 12, 20 seconds just for the damn thing to assemble itself so I could run it. 00:18:14.498 --> 00:18:19.302 And now, some modules, yeah, they just link in less than a second and, uh, I 00:18:19.302 --> 00:18:20.812 can restart the app and it works great. 00:18:20.812 --> 00:18:24.552 I could do actually dynamic hot reloading, but I'm, I'm not actually doing it. 00:18:24.652 --> 00:18:25.212 That's, uh... 00:18:25.452 --> 00:18:26.782 That's spooky for another day. 00:18:27.032 --> 00:18:29.941 Yes, but the end result is: I'm having fun working on my site again. 00:18:29.991 --> 00:18:32.772 I've discovered so many things that I actually did in the 00:18:32.832 --> 00:18:34.712 templating language, like liquid... 00:18:34.812 --> 00:18:36.022 I migrated to `minijinja`. 00:18:36.042 --> 00:18:37.249 That's another episode we can do. 00:18:37.328 --> 00:18:41.208 There's so many weird hacks I did as either client-side JavaScript or in 00:18:41.208 --> 00:18:44.516 the templating language, because I didn't want to touch the Rust code 00:18:44.591 --> 00:18:48.011 again, because it was so slow to compile and deploy and everything. 00:18:48.214 --> 00:18:52.513 But now I'm able to ship lots of small changes because testing a new change 00:18:52.596 --> 00:18:56.518 in debug- I think, the app itself- if I change the binary, the executable 00:18:56.518 --> 00:19:01.718 crate, it takes, I think, three seconds to compile and link and then reload. 00:19:02.148 --> 00:19:06.252 And then it's building and loading modules at runtime in 00:19:06.262 --> 00:19:08.092 the order that it needs them. 00:19:08.747 --> 00:19:11.067 And also preemptively before it actually needs them. 00:19:11.067 --> 00:19:13.647 So it kind of warms up the cache as it does that. 00:19:13.767 --> 00:19:17.600 And the idea was that in CI, I could upload the built version of those 00:19:17.620 --> 00:19:20.923 modules somewhere, and then just download them, because it's not like the image 00:19:20.923 --> 00:19:22.604 encoder changes every week, right? 00:19:22.632 --> 00:19:24.982 So you still have your three or four hundred dependencies, they're 00:19:24.982 --> 00:19:31.542 just sharded into 7 or 8 reasonable blocks, and it's rare that anything 00:19:31.562 --> 00:19:35.592 other than the leaf block that is your actual site is what actually changes. 00:19:36.192 --> 00:19:37.092 That is exactly it. 00:19:37.202 --> 00:19:41.502 And it also means because each module is its own layer in the container 00:19:41.522 --> 00:19:44.512 image that you deploy, when they don't change, they don't change. 00:19:45.332 --> 00:19:48.864 You don't get like one layer with all the big several hundred 00:19:48.864 --> 00:19:50.504 megabyte executable that changes. 00:19:50.808 --> 00:19:55.048 You got like blobs between 2 and 12 megabytes that just stay 00:19:55.048 --> 00:19:57.008 exactly the same across deploys. 00:19:57.008 --> 00:20:00.583 And deploying is now really fast I have a server in Singapore, which 00:20:00.593 --> 00:20:02.163 is usually pretty slow to deploy to. 00:20:03.113 --> 00:20:05.811 Real quick future steps: future steps is to kill this crate 00:20:05.861 --> 00:20:07.161 because it shouldn't exist. 00:20:07.181 --> 00:20:11.775 It should be a single flag called `globals-linkage` import or export. 00:20:12.265 --> 00:20:18.223 And if it was a single flag, then it could keep name mangling, which rubicon 00:20:18.333 --> 00:20:22.229 completely disables because I've tried my best, but I don't think there is a way 00:20:22.229 --> 00:20:27.519 to abuse current Rust syntax to export and import globals with name mangling. 00:20:28.009 --> 00:20:28.739 I may be wrong. 00:20:28.769 --> 00:20:30.491 But, I want to get people more interested in it because 00:20:30.491 --> 00:20:31.971 the benefits are really cool. 00:20:31.971 --> 00:20:34.689 And it could also be something you only use in development, but then in 00:20:34.689 --> 00:20:35.819 production, you do a static build. 00:20:35.819 --> 00:20:38.288 I think this should be a built-in Rust feature because 00:20:38.298 --> 00:20:39.388 it would be really, really cool. 00:20:40.248 --> 00:20:41.528 I really like the trailblazing. 00:20:41.548 --> 00:20:44.968 Like, the trailblazing is super fun because you go: look, these are the 00:20:44.968 --> 00:20:49.638 biggest problems, and other people can come up with even better solutions, but 00:20:49.638 --> 00:20:53.688 at least I've walked down the road far enough to go, "That's a problem, that's 00:20:53.688 --> 00:20:56.608 a problem, that's a problem," ... even if the solution's incomplete or 00:20:56.608 --> 00:21:00.668 requires you to source-patch things- which isn't unreasonable, honestly, 00:21:00.678 --> 00:21:02.808 for people maintaining an application. 00:21:02.818 --> 00:21:07.844 Like, I sometimes carry dependency source patches for a fairly long time 00:21:07.844 --> 00:21:11.234 because it- you know, my solution that works great for me isn't reasonable 00:21:11.234 --> 00:21:14.334 for other people, but it's something that cargo already has handles for. 00:21:14.334 --> 00:21:19.634 So I like the idea of: let's play around with this and then kill it and get it 00:21:19.634 --> 00:21:24.264 upstreamed because you're certainly not the only one who wants or needs this. 00:21:24.264 --> 00:21:27.144 So I think being able to show it off and getting people to 00:21:27.144 --> 00:21:28.294 be able to take it for a spin. 00:21:28.314 --> 00:21:31.194 I've seen some of your builds where you just go: save! 00:21:31.324 --> 00:21:32.804 And it's deployed three seconds later. 00:21:32.804 --> 00:21:36.094 Which- I have a customer project right now that's in a similar size, 00:21:36.094 --> 00:21:40.354 like three four hundred deps and our main crate with all the business 00:21:40.354 --> 00:21:42.334 logic is still one large crate. 00:21:42.344 --> 00:21:44.244 And even just running tests. 00:21:44.284 --> 00:21:48.534 I changed one line of a test and it took like nine seconds to 00:21:48.534 --> 00:21:50.604 build on my silly fast MacBook. 00:21:50.864 --> 00:21:52.504 And I'm like: Oh no, this is... 00:21:52.858 --> 00:21:55.208 Coming from mostly embedded stuff, this is the first time where I 00:21:55.208 --> 00:21:57.288 go: Oh yeah, I see that problem. 00:21:57.368 --> 00:22:00.188 And I could definitely use something like this. 00:22:00.518 --> 00:22:02.518 Yeah, so contributing to rustc is probably going to 00:22:02.518 --> 00:22:04.068 take a hot minute just because... 00:22:04.408 --> 00:22:05.888 well, you need to get it right. 00:22:05.898 --> 00:22:09.988 It needs to be useful to everyone, not just like your weird thing in a basement. 00:22:10.338 --> 00:22:13.568 So in the meantime, I think I'm on good terms- with the tokio 00:22:13.618 --> 00:22:15.548 developers and others and tracing. 00:22:15.568 --> 00:22:18.665 So I'm going to try to get those merged because it's completely opt-in if 00:22:18.665 --> 00:22:22.763 you don't enable the import-globals or export-globals features, it's 00:22:22.763 --> 00:22:24.083 not gonna mean extra dependencies. 00:22:24.093 --> 00:22:25.523 Gonna be the exact same code as before. 00:22:25.882 --> 00:22:28.549 Other people, mostly in game development, have looked into that. 00:22:28.599 --> 00:22:30.904 I just want to encourage people to write about what you're doing, 00:22:30.914 --> 00:22:34.186 because then when others try it, you go a little further each time. 00:22:34.186 --> 00:22:37.087 You just on the shoulders of giants, except giants are like... 00:22:37.612 --> 00:22:39.600 someone who blogged about it two years ago. 00:22:40.242 --> 00:22:40.762 Very cool. 00:22:46.197 --> 00:22:49.497 The Self-Directed Research podcast is made possible by our sponsors. 00:22:49.737 --> 00:22:53.787 We offer 30 second host-read ads- like this one- at the end of every episode. 00:22:54.207 --> 00:22:56.277 Not sure how to get your message out, or what to say? 00:22:56.607 --> 00:22:57.117 Let us help! 00:22:57.473 --> 00:23:01.383 If you'd like to promote your company, project, conference, or open job positions 00:23:01.383 --> 00:23:05.413 to an audience interested in programming and technical deep dives, send us an 00:23:05.413 --> 00:23:11.053 email to contact@sdr-podcast.com for more information about sponsorship. 00:23:11.703 --> 00:23:13.523 Thanks to all of our sponsors for their support.