WEBVTT NOTE This file was generated by Descript 00:00:13.702 --> 00:00:15.442 This is Self-Directed Research! 00:00:15.547 --> 00:00:18.884 James and Amos get hyped about different topics and take turns each week, 00:00:18.914 --> 00:00:20.414 presenting their ideas to each other. 00:00:20.655 --> 00:00:22.485 This episode wraps up season one. 00:00:22.608 --> 00:00:23.898 We hope you enjoyed it so far. 00:00:24.204 --> 00:00:27.084 We'll be taking a little break over the holidays, but we'll be back next year. 00:00:27.330 --> 00:00:30.780 Make sure to like follow and subscribe wherever you find us to stay up to date. 00:00:31.050 --> 00:00:35.339 And as usual visit sdr-podcast.com/episodes for all 00:00:35.339 --> 00:00:38.129 of the presentations, videos, show notes and transcripts. 00:00:38.374 --> 00:00:41.974 This week, James makes his pitch for "The Embedded Buddy System." 00:00:42.359 --> 00:00:44.989 This episode is sponsored by Poststation, a new tool for 00:00:44.989 --> 00:00:46.579 embedded systems from OneVariable. 00:00:47.069 --> 00:00:50.294 Check out the link in our show notes or listen at the end for more information. 00:00:55.544 --> 00:00:59.734 This week I invented a whole new term for something that I've been doing for years. 00:00:59.809 --> 00:01:00.939 It seemed like a good name. 00:01:00.939 --> 00:01:03.639 So, this week we're going to talk about "The Embedded Buddy System." 00:01:03.939 --> 00:01:04.269 Ooh... 00:01:04.419 --> 00:01:07.049 The alternative title here is actually 00:01:07.359 --> 00:01:11.369 "James' Cheat Codes for Low to Mid Volume and Rapid Embedded Development" because 00:01:11.369 --> 00:01:16.739 it's basically how I've done almost every customer project for certainly the last 00:01:16.989 --> 00:01:20.869 three years but probably more likely the last five or six years because I 00:01:20.869 --> 00:01:24.089 end up helping a lot of companies that are either startups so they're making 00:01:24.089 --> 00:01:26.887 their first batch of hardware where they're shipping the first thousand 00:01:26.887 --> 00:01:29.939 units or they're shipping the first 10 units so they can get feedback 00:01:29.939 --> 00:01:31.599 from people and things like that. 00:01:31.599 --> 00:01:35.705 So they're not necessarily at the point where they're making a million items yet. 00:01:35.705 --> 00:01:39.525 And a lot of what they're doing is either trying to figure out what their device 00:01:39.525 --> 00:01:43.045 should do, or for more specialized people, they're never going to ship a million 00:01:43.045 --> 00:01:46.867 units because it's some huge piece of scientific or industrial equipment. 00:01:46.867 --> 00:01:49.737 And if they sell 20 a year, they're going to be pretty pumped because 00:01:49.757 --> 00:01:51.667 they cost six or seven digits a piece. 00:01:51.987 --> 00:01:55.357 I help out a lot with those kinds of problems. 00:01:55.781 --> 00:01:58.991 If you're building something embedded today and you're 00:01:58.991 --> 00:02:01.161 designing it from scratch, you're not iterating on something. 00:02:01.161 --> 00:02:05.551 You don't have some really ridiculous constraint on size or anything like that, 00:02:05.841 --> 00:02:09.621 you should probably consider this buddy system because I know you're 00:02:09.621 --> 00:02:12.241 going to immediately have some- I'm talking to the embedded developers 00:02:12.241 --> 00:02:15.261 out there- you're going to immediately have some pushback like, "Ah, this 00:02:15.261 --> 00:02:16.601 certainly doesn't address my needs." 00:02:16.601 --> 00:02:20.361 And for a very few of you, you will be very, very right. 00:02:20.631 --> 00:02:22.531 But more of you than maybe you'd think. 00:02:22.531 --> 00:02:26.541 So keep an open mind on this one, embedded developers and stick with me on this. 00:02:27.539 --> 00:02:32.989 What I mean by the buddy system is we have some more powerful embedded Linux SoC. 00:02:33.019 --> 00:02:37.449 This is like Raspberry Pi class, something, you know, ARM Cortex-A that 00:02:37.459 --> 00:02:39.589 has a fair amount of processing power. 00:02:39.854 --> 00:02:42.578 These days they're getting faster and they're getting more powerful 00:02:42.578 --> 00:02:44.678 and get them in fairly tiny sizes. 00:02:44.908 --> 00:02:45.698 But they're running Linux. 00:02:45.718 --> 00:02:48.278 They are more like a computer than you would think. 00:02:48.798 --> 00:02:51.628 And on the other side, we've got our bare metal microcontroller. 00:02:51.638 --> 00:02:55.859 This is more Raspberry Pi Pico or RP2040 class. 00:02:55.869 --> 00:02:59.389 You know, we're talking ARM Cortex-M or other 32 bit microcontrollers 00:02:59.399 --> 00:03:03.089 these days and having them linked up by some interface. 00:03:03.119 --> 00:03:07.089 I have USB in this one, but it could be whatever you happen to have available, 00:03:07.129 --> 00:03:10.239 but have them working together as buddies. 00:03:10.999 --> 00:03:13.616 There's a real push to try and pick one. 00:03:13.616 --> 00:03:17.526 You go: well, is this a microcontroller sized project, or is this an 00:03:17.526 --> 00:03:19.596 embedded Linux SoC sized problem? 00:03:19.596 --> 00:03:21.636 You go: well, obviously I want it to be cheap. 00:03:21.636 --> 00:03:23.466 I don't wanna have a bunch of things going on. 00:03:23.676 --> 00:03:27.846 I wanna try and do it all in one chip, and I think that is a foot gun. 00:03:27.906 --> 00:03:31.846 Maybe not on day one, but certainly as the project grows, it can 00:03:31.846 --> 00:03:33.436 very quickly become a foot gun. 00:03:33.616 --> 00:03:35.376 James, I think this is going to be a very short episode 00:03:35.406 --> 00:03:37.106 because I wholeheartedly agree with you. 00:03:37.106 --> 00:03:39.786 I mean, I've never done this professionally, unlike you, but 00:03:39.786 --> 00:03:41.106 it does seem to make sense to me. 00:03:41.116 --> 00:03:45.126 Like, don't try to do everything on a system that is only suited for half of it. 00:03:45.126 --> 00:03:46.076 Just have both. 00:03:46.106 --> 00:03:47.840 It's actually affordable nowadays. 00:03:48.070 --> 00:03:50.868 You talk about Raspberry Pi, the computer like ones? 00:03:50.868 --> 00:03:53.238 The, like, Raspberry Pi 4, 5. 00:03:53.788 --> 00:03:55.098 Aren't they always out of stock though? 00:03:55.098 --> 00:03:55.878 Isn't that an issue? 00:03:56.198 --> 00:03:58.148 Well, if you pick the newest model, yeah, but this is 00:03:58.148 --> 00:04:01.918 sort of my placeholder for, it's not just Raspberry Pi that makes SoCs. 00:04:01.918 --> 00:04:04.838 And in fact, they're usually the more high end, low volume kind of thing. 00:04:05.088 --> 00:04:09.088 There's a lot of very cheap SoCs, especially now that RISC-V is a thing. 00:04:09.288 --> 00:04:14.948 You can get like, surprisingly powerful quad core 64 bit RISC-V processors 00:04:15.108 --> 00:04:16.838 for basically nothing at this point. 00:04:17.048 --> 00:04:19.758 So RISC-V is actually getting adoption right now... 00:04:20.058 --> 00:04:22.833 They're selling a lot of chips, so I think, especially coming out 00:04:22.833 --> 00:04:28.068 of China, you see a ton of Allwinner parts that are getting used for set-top boxes. 00:04:28.147 --> 00:04:30.797 Depending on when you will listen to this, it will either be different or not. 00:04:30.807 --> 00:04:34.527 But especially for things that are not super cutting edge, like high end phones, 00:04:34.537 --> 00:04:39.697 things like set-top boxes or signage or appliance type stuff where, yeah, it's 00:04:39.697 --> 00:04:44.703 like 40 percent slower than the equivalent ARM device or something like that. 00:04:44.703 --> 00:04:48.583 But if it's a set-top box and it has the DSP that you need, and you know, you've 00:04:48.583 --> 00:04:52.153 hit the minimum threshold, you might be able to hit that minimum threshold for way 00:04:52.153 --> 00:04:54.403 cheaper, but it could be an ARM Cortex-A- 00:04:54.423 --> 00:04:58.673 So that is why so many lower end TVs have UI 00:04:58.683 --> 00:05:01.543 that freaking lags so much. 00:05:02.108 --> 00:05:04.488 Minimum viable product for the price point. 00:05:04.606 --> 00:05:07.206 They have hardware H.264 decoding. 00:05:07.476 --> 00:05:11.076 So that works, but everything else is so slow. 00:05:11.076 --> 00:05:13.223 It takes seconds to get to where you want to watch. 00:05:13.223 --> 00:05:14.243 It's incredible. 00:05:14.285 --> 00:05:16.045 You can't just blame that on RISC-V because that's 00:05:16.045 --> 00:05:17.205 been the case for 20 years. 00:05:17.940 --> 00:05:18.292 That's true... 00:05:18.292 --> 00:05:20.038 That's just one of those, like we're hitting a price 00:05:20.038 --> 00:05:22.898 point with the compute that we can afford to put in all of these 00:05:22.898 --> 00:05:24.398 set-top boxes and stuff like that. 00:05:24.533 --> 00:05:26.243 I mean, meanwhile, people are complaining about Apple 00:05:26.243 --> 00:05:29.193 TVs and saying you should just use the newer Mac Mini M4 instead. 00:05:30.473 --> 00:05:31.513 Because reasons? 00:05:31.573 --> 00:05:32.173 I don't know. 00:05:32.563 --> 00:05:34.797 I follow too many weird people on BlueSky 00:05:34.797 --> 00:05:34.857 yeah. 00:05:35.113 --> 00:05:37.623 But the thing is, embedded Linux is great for this big stuff. 00:05:37.653 --> 00:05:40.873 If we're talking about video encoding or decoding, if we're talking about 00:05:40.927 --> 00:05:44.445 crunching numbers, Linux is just easier for doing these things, especially when 00:05:44.445 --> 00:05:46.355 you want to have an operating system. 00:05:46.575 --> 00:05:49.765 You want networking, you want file systems, you want databases, you 00:05:49.765 --> 00:05:52.455 want to be able to do over the air updates without having to really 00:05:52.455 --> 00:05:54.245 dangerously re flash the whole device. 00:05:54.415 --> 00:05:58.545 You just want, like apt-get update or whatever BitBake tool you're using. 00:05:59.115 --> 00:06:01.705 And the other thing that's not even a technical thing is hiring. 00:06:02.145 --> 00:06:05.375 You would probably be comfortable, Amos, sitting down on an embedded 00:06:05.375 --> 00:06:08.385 Linux device because it's still Linux and it feels very comfortable. 00:06:08.385 --> 00:06:11.605 So even if you don't necessarily fully consider yourself an embedded 00:06:11.615 --> 00:06:15.705 developer, it's still a pool you're much more comfortable in, especially 00:06:15.705 --> 00:06:18.205 if someone else has done the setup and you're like, "Okay, now I can SSH 00:06:18.205 --> 00:06:20.501 in," and it's not markedly different. 00:06:20.501 --> 00:06:24.171 It might even be faster than a super cheap VPS instance, which is 00:06:24.181 --> 00:06:27.935 also, you know, itty bitty in terms of performance and capabilities. 00:06:28.208 --> 00:06:30.468 Yeah, I was thinking if you have some sort of web UI, 00:06:30.488 --> 00:06:34.138 which a lot of connected gadgets do: it would definitely be served 00:06:34.138 --> 00:06:35.638 from like the big, the big buddy. 00:06:35.922 --> 00:06:38.648 So these kinds of things are all things that are super off 00:06:38.658 --> 00:06:41.828 the shelf and easy, both in terms of like actually doing it and what 00:06:41.838 --> 00:06:43.228 exists out there for doing it. 00:06:43.228 --> 00:06:47.288 So these are all things that are very easy to do on Linux, even if 00:06:47.288 --> 00:06:49.138 it's a relatively small Linux system. 00:06:49.353 --> 00:06:51.093 Actually, isn't that what Tesla does? 00:06:51.233 --> 00:06:52.893 Like they run Ubuntu in their cars? 00:06:52.923 --> 00:06:55.108 A lot of gauge clusters these days are running 00:06:55.138 --> 00:06:56.408 embedded Linux and things like that. 00:06:56.438 --> 00:07:00.318 They have some ways of like hyper visoring away the safety critical stuff and the 00:07:00.318 --> 00:07:03.764 non safety critical stuff and you have- exactly- that kind of stuff of 'not 00:07:03.784 --> 00:07:07.954 on the critical path' things because it just makes iteration way easier. 00:07:07.954 --> 00:07:11.059 And if you can segment your product in a way where a little bit of lag 00:07:11.059 --> 00:07:14.349 on the user interface isn't going to cause problems with your brakes, then 00:07:14.349 --> 00:07:17.779 it's way easier to not do everything in the super safety critical world. 00:07:17.779 --> 00:07:19.049 So exactly this kind of thing. 00:07:19.252 --> 00:07:21.409 And this is why I like that you're my co host James! 00:07:21.409 --> 00:07:25.039 Because- because people get scared, Oh, they hear about Kubernetes and the cars 00:07:25.039 --> 00:07:27.589 or whatever, and they go like, "No, I don't want my brakes to stop working!" 00:07:27.589 --> 00:07:28.159 Because- whatever. 00:07:28.169 --> 00:07:31.714 Because they have their own experience of like, someone tried to roll out Kubernetes 00:07:31.724 --> 00:07:35.554 in their internet startup, and it took eight months and everyone was lost. 00:07:35.588 --> 00:07:37.798 But actually, yeah, it is just for some stuff. 00:07:37.838 --> 00:07:39.828 And the rest of the stuff can still keep working. 00:07:39.828 --> 00:07:42.368 It's very important, like isolating- reducing the blast radius is 00:07:42.368 --> 00:07:44.413 also a thing in web services. 00:07:45.498 --> 00:07:47.168 Then you can do bare metal. 00:07:47.188 --> 00:07:49.128 And when I say bare metal, I mean, like this is stuff that 00:07:49.128 --> 00:07:50.078 runs on microcontrollers. 00:07:50.118 --> 00:07:52.568 It could be on a real-time operating system. 00:07:52.758 --> 00:07:55.716 But they're still fairly limited, it's not the full capabilities 00:07:56.036 --> 00:07:56.886 of something like Linux. 00:07:56.896 --> 00:08:01.066 Or it could be a totally bare metal program, or just an async executor, 00:08:01.076 --> 00:08:03.666 which is very common in like embassy and things like that today. 00:08:04.006 --> 00:08:06.309 But we're using bare metal for the little stuff, because it's 00:08:06.369 --> 00:08:07.649 good at a lot of things too. 00:08:08.009 --> 00:08:12.104 If you're building up custom hardware, a microcontroller is usually pretty well 00:08:12.104 --> 00:08:16.114 self contained, they're cheap to put on a circuit board, you don't have to put 00:08:16.184 --> 00:08:22.234 DRAM and an SSD and all of cooling and all of these kind of things- usually you 00:08:22.234 --> 00:08:27.299 can drop a single chip, or a chip and its storage on a circuit board, which means 00:08:27.299 --> 00:08:31.461 that board is very easy to design and it also means it's very cheap to make. 00:08:31.461 --> 00:08:35.160 So if you go, "I just want something that's a little custom," you don't 00:08:35.170 --> 00:08:37.922 have to build a motherboard, basically. 00:08:37.922 --> 00:08:41.076 You can just build like a little accessory card fairly quickly. 00:08:41.163 --> 00:08:44.633 It lowers the threshold of how skilled of an electrical designer 00:08:44.733 --> 00:08:46.983 you either need to hire or pay for. 00:08:47.033 --> 00:08:51.453 Right, just for the rest of us: so DRAM would be actually chips 00:08:51.463 --> 00:08:53.343 that you would solder onto the board? 00:08:53.403 --> 00:08:54.033 It can be. 00:08:54.203 --> 00:08:57.803 the same way, like Apple solders, their memory onto the board, 00:08:57.803 --> 00:09:02.263 you'll have like DDR2 or DDR3 is fairly common for embedded devices. 00:09:02.263 --> 00:09:05.303 They're behind what your top tier Android phone would be. 00:09:05.843 --> 00:09:09.318 But they're still more complicated to have external memory. 00:09:09.578 --> 00:09:12.998 There are some chips that have built in like, on the same chip as 00:09:12.998 --> 00:09:17.242 the CPU they have their memory, but usually DRAM takes a lot of space. 00:09:17.362 --> 00:09:20.696 So that's usually why it's outside of the main CPU even though you want it 00:09:20.696 --> 00:09:24.736 to be as close as possible but just physically the transistors and everything 00:09:24.736 --> 00:09:26.686 that you need, you need space for that. 00:09:26.807 --> 00:09:27.077 Right. 00:09:27.157 --> 00:09:29.847 And the D stands for dynamic because it's refreshed... 00:09:29.847 --> 00:09:30.227 Exactly. 00:09:30.252 --> 00:09:30.462 Yeah. 00:09:30.462 --> 00:09:33.622 So SRAM is usually what you have in microcontrollers 00:09:33.772 --> 00:09:35.332 and it's also fairly large. 00:09:35.552 --> 00:09:39.853 But it's comparatively like electrically and space-wise, expensive compared to 00:09:39.853 --> 00:09:46.303 DRAM, but main CPUs use SRAM as cache because it's the fastest thing and 00:09:46.303 --> 00:09:47.983 you put it right next to your CPU. 00:09:48.263 --> 00:09:52.571 But usually when you want a big bulk DRAM, which is your DDR memory- like 00:09:52.571 --> 00:09:56.141 you have on your desktop or your laptop or whatever- that's a separate chip. 00:09:56.191 --> 00:10:00.001 On desktops it might be on a card that you plug in just for convenience, but on 00:10:00.001 --> 00:10:03.741 your laptop or your phone it's going to be soldered on right next to your chip. 00:10:03.973 --> 00:10:06.505 I remember upgrading old laptops, they still had 00:10:06.505 --> 00:10:07.875 the, whatever it's called... 00:10:07.949 --> 00:10:08.699 you slot them in. 00:10:08.729 --> 00:10:10.913 I think ThinkPads might still have it like the bigger ones, 00:10:10.913 --> 00:10:13.283 but not the like ultrabook style ones. 00:10:13.368 --> 00:10:15.778 I don't know, because ThinkPads is no longer IBM, so 00:10:15.778 --> 00:10:16.908 I just assume they're crap now. 00:10:16.938 --> 00:10:17.278 I don't know. 00:10:17.983 --> 00:10:19.353 So the other stuff that microcontrollers 00:10:19.353 --> 00:10:20.703 are great for is real-time. 00:10:20.723 --> 00:10:23.493 When you don't have an operating system or you have a really purpose built 00:10:23.523 --> 00:10:26.123 operating system, you can make guarantees. 00:10:26.163 --> 00:10:30.943 Like: if this signal comes in, I will respond to it and finish responding 00:10:30.943 --> 00:10:32.823 to it within 10 microseconds. 00:10:32.823 --> 00:10:36.777 Where that might be something that you can start to get real-time or you can 00:10:36.807 --> 00:10:40.127 overcome the problem with power because if your CPU is running fast enough, 00:10:40.157 --> 00:10:44.077 you know, probably you'll get to it quickly, but it can be challenging 00:10:44.077 --> 00:10:46.237 to really like objectively guarantee. 00:10:46.237 --> 00:10:46.337 And- 00:10:46.337 --> 00:10:49.047 It's this kind of thinking that leads to audio glitches 00:10:49.047 --> 00:10:51.850 in consumer hardware all the time because they're designed with like 00:10:51.850 --> 00:10:55.100 it's probably gonna be fine and then sometimes like *glitching sound*. 00:10:55.262 --> 00:10:58.472 So audio is usually called soft real-time because no one's day's 00:10:58.472 --> 00:11:00.292 really messed up if it goes wrong. 00:11:00.312 --> 00:11:00.962 Except mine. 00:11:01.015 --> 00:11:03.127 Hard real-time is usually, like, you stick your arm 00:11:03.127 --> 00:11:07.359 into a machine and the sensor detects that and kills the motor within 10 00:11:07.359 --> 00:11:10.439 microseconds Like the SawStop, for example, where there's that table saw. 00:11:10.479 --> 00:11:12.799 It's basically a cap touch sensor, like your phone. 00:11:13.069 --> 00:11:16.089 And if you touch it within a certain amount of time, it fires an 00:11:16.089 --> 00:11:18.059 explosive, which drops the blade down. 00:11:18.069 --> 00:11:20.959 Like you don't want that to lag by like 30 milliseconds, because 00:11:20.959 --> 00:11:23.769 that might be the difference between having your finger and not. 00:11:23.819 --> 00:11:25.089 So that one's pretty important. 00:11:25.539 --> 00:11:25.799 True. 00:11:26.219 --> 00:11:30.854 I wanted to mention that the real-time patch set for the Linux 00:11:30.884 --> 00:11:33.344 kernel has been merged recently. 00:11:33.794 --> 00:11:36.364 They had a whole celebration of that and I didn't know. 00:11:36.414 --> 00:11:39.404 So to give you an idea, right now I'm using all Apple hardware because 00:11:39.454 --> 00:11:42.064 I unified everything and it makes my life so much simpler and I'm 00:11:42.064 --> 00:11:43.804 able to get so much more work done. 00:11:44.029 --> 00:11:46.669 But back in the days, I was an early Linux adopter. 00:11:46.730 --> 00:11:50.732 I had like Mandrake, I used Red Hat before it was renamed to Fedora, I 00:11:50.732 --> 00:11:53.812 installed Slackware from floppies, I don't know, I'm older than I look, people 00:11:53.852 --> 00:11:57.775 assume I'm a zoomer when they look at my YouTube videos, but I'm 34, and I grew 00:11:57.775 --> 00:11:59.838 up with 10 years of lag on technology. 00:11:59.856 --> 00:12:00.746 Why was I saying all that? 00:12:00.956 --> 00:12:06.323 Because I remember building my own version of the Ubuntu kernel with 00:12:06.343 --> 00:12:10.903 the real-time patch set to be able to run a digital audio workstation 00:12:10.903 --> 00:12:13.537 and do audio work on Linux because that was the only way to do it. 00:12:13.618 --> 00:12:17.388 It's crazy to me that it took, I don't know, two decades? 00:12:17.798 --> 00:12:20.361 I don't know exactly how long to get this merged into the main kernel. 00:12:20.361 --> 00:12:20.921 I have no idea. 00:12:20.921 --> 00:12:22.181 This is probably a very good reason. 00:12:22.766 --> 00:12:27.026 So my question to you James is, is it still as relevant now that real-time 00:12:27.046 --> 00:12:30.166 Linux is in the main line kernel? 00:12:30.291 --> 00:12:34.347 Yes, because there's still limitations on how granular 00:12:34.347 --> 00:12:36.637 you want your real-time guarantees to be, because that's one of those 00:12:36.637 --> 00:12:39.517 things where real-time doesn't mean faster, it means deterministic. 00:12:39.807 --> 00:12:42.867 So the scheduler might be way less efficient, but you have the 00:12:42.867 --> 00:12:44.947 chance to swap tasks often enough. 00:12:44.997 --> 00:12:48.107 But there's still some limit where a microcontroller is likely 00:12:48.107 --> 00:12:49.937 to be able to respond faster. 00:12:50.207 --> 00:12:53.027 Sometimes that's down to the CPU architecture. 00:12:53.297 --> 00:12:56.917 For example, on Cortex-M there's a bounded guaranteed amount of time when 00:12:56.917 --> 00:13:01.197 an interrupt occurs, how many cycles it takes to when you're starting to do 00:13:01.197 --> 00:13:03.467 that versus on Linux, it might be... 00:13:03.719 --> 00:13:06.809 because Cortex-A processors as an architecture don't have the same... 00:13:07.219 --> 00:13:08.992 it might not even be at the operating system. 00:13:09.022 --> 00:13:11.672 It might be down to the, like, architectural level of the CPU, 00:13:11.672 --> 00:13:15.562 because these CPUs are designed to be real-time and if we're talking 00:13:15.562 --> 00:13:18.512 about like a millisecond, both of them will be able to do that fine. 00:13:18.702 --> 00:13:21.212 A hundred microseconds, probably fine? 00:13:21.512 --> 00:13:25.222 But when we're talking like one microsecond or 10 microseconds, you're 00:13:25.262 --> 00:13:29.202 really counting CPU cycles and you want the actual electrical hardware 00:13:29.462 --> 00:13:32.782 to guarantee how long it's going to take before you start executing 00:13:32.782 --> 00:13:34.842 those cycles that turn the motor off. 00:13:35.117 --> 00:13:37.959 Yeah, love the way you phrased it, like 'real-time 00:13:37.979 --> 00:13:41.224 doesn't mean faster' because I'm thinking about tokio as well. 00:13:41.324 --> 00:13:45.184 A bunch of people, you know, benchmark reading a large file from synchronous 00:13:45.184 --> 00:13:47.996 Rust and then they do it with tokio and like, "tokio is slower." 00:13:48.116 --> 00:13:51.076 Well, first of all, yes, for file system IO, yes, because it's actually using 00:13:51.076 --> 00:13:52.676 blocking sys calls in a thread pool. 00:13:52.826 --> 00:13:56.496 But even if it wasn't, it would still probably be slower because there's 00:13:56.516 --> 00:14:00.586 overhead involved in making sure you can actually multiplex that on a single thread 00:14:00.596 --> 00:14:05.046 and that costs CPU cycles, but then you can service multiple requests at once. 00:14:05.046 --> 00:14:07.832 It's better to have like all the customers get their response in 00:14:07.882 --> 00:14:11.407 some time then be blocked on only handling one customer, one response. 00:14:11.426 --> 00:14:11.656 Yeah. 00:14:11.706 --> 00:14:14.456 I know I've talked about this with Eliza, but in the same way that you care about 00:14:14.506 --> 00:14:18.806 tail latency on a server where you say: well, yeah, okay, 99 percent are handled. 00:14:18.856 --> 00:14:22.141 But what about the next nine or the next nine, if that becomes ridiculous? 00:14:22.332 --> 00:14:25.922 In hard real-time, you want to say it will never literally never, no matter 00:14:25.922 --> 00:14:29.672 what's going on- I don't care if I'm in the middle of a database transaction- it 00:14:29.672 --> 00:14:35.030 will never take more than two and a half microseconds from 'we noticed a touch' 00:14:35.180 --> 00:14:40.240 to 'we're firing the explosive' and the explosive takes 20 microseconds to fire. 00:14:40.370 --> 00:14:44.500 Which means we know that from when your finger touches that, it will never be 00:14:44.500 --> 00:14:48.217 more than like whatever I said, 12 and a half microseconds before that blade is 00:14:48.217 --> 00:14:50.137 dropping and no longer touching your hand. 00:14:50.357 --> 00:14:52.485 It's one of those things where engineering is all about 'good 00:14:52.485 --> 00:14:55.215 enough for what I'm doing.' But sometimes you really care about that. 00:14:55.245 --> 00:14:59.875 And honestly- and this is what I'll get into later- sometimes it could be possible 00:14:59.875 --> 00:15:02.735 with Linux, but it's not necessarily easy. 00:15:02.745 --> 00:15:03.765 We're on a microcontroller. 00:15:03.765 --> 00:15:05.855 It's like, "Okay, yeah, I analyzed it. 00:15:05.865 --> 00:15:06.595 It's fine." 00:15:06.825 --> 00:15:07.545 Versus on Linux. 00:15:07.545 --> 00:15:08.855 You're like, "I'm... 00:15:08.935 --> 00:15:11.335 I'm pretty sure it's probably fine-" you know... 00:15:12.195 --> 00:15:14.075 And all of this stuff that I'm recommending is not saying that 00:15:14.085 --> 00:15:15.525 you couldn't do it another way. 00:15:15.970 --> 00:15:19.690 But if you use the tools for what they're good for, you're 00:15:19.690 --> 00:15:20.960 going to have a better time. 00:15:20.971 --> 00:15:24.277 So that's why you're saying it's not for very large productions. 00:15:24.677 --> 00:15:25.527 You don't say productions... 00:15:25.557 --> 00:15:26.147 runs? 00:15:26.281 --> 00:15:26.622 Yeah. 00:15:26.622 --> 00:15:27.852 We'll get into that a little bit later. 00:15:28.092 --> 00:15:30.462 But then there's other things like the amount of IO you have. 00:15:30.642 --> 00:15:35.198 And this can be just like, "Hey, I need to talk to 600 switches cause I'm a big 00:15:35.228 --> 00:15:40.291 panel," Or I need to talk to 27 different weird serial ports and things like that. 00:15:40.341 --> 00:15:43.711 These embedded Linux devices, especially like the Raspberry Pi has popularized it. 00:15:43.761 --> 00:15:47.141 You see people plugging them straight into sensors and buttons and stuff like that. 00:15:47.141 --> 00:15:47.971 And they do work. 00:15:48.221 --> 00:15:52.021 They just usually have a fairly limited palette of things that they can do, or 00:15:52.021 --> 00:15:55.451 a limited number of pin counts where it's challenging to get, you know, 00:15:55.451 --> 00:15:58.298 super high pin counts on one chip. 00:15:58.298 --> 00:16:00.858 And so you end up having to have like expanders or 00:16:00.858 --> 00:16:02.188 adapters and things like that. 00:16:02.358 --> 00:16:04.118 And it ends up complicating your design. 00:16:04.128 --> 00:16:08.108 And you end up having to like, "Oh, well, I can only scan these things so fast 00:16:08.138 --> 00:16:11.188 because I have to talk to 10 of them," and you can get into real trouble when 00:16:11.188 --> 00:16:13.758 it comes to doing what you'd like to do. 00:16:14.018 --> 00:16:16.198 Well, and then if you use an adapter, aren't you using the buddy 00:16:16.198 --> 00:16:19.523 system anyway, because you have big Raspberry Pi and then something over USB 00:16:19.533 --> 00:16:21.463 that has all the IO pins that you need? 00:16:21.633 --> 00:16:24.737 There's something to be said about paying for less 00:16:24.737 --> 00:16:26.467 software, if that makes sense. 00:16:26.477 --> 00:16:28.915 In that hardware it acts the way it acts and you don't 00:16:28.915 --> 00:16:29.845 have to think about software. 00:16:29.845 --> 00:16:31.305 So there is something to be said about that. 00:16:31.322 --> 00:16:34.087 By the time you've added a couple adapters and things like that, 00:16:34.097 --> 00:16:37.257 it ends up costing more than a microcontroller in a lot of cases. 00:16:38.107 --> 00:16:39.517 And the other thing is low power. 00:16:39.627 --> 00:16:42.987 Your phone is relatively low power, but you are charging it every day. 00:16:43.351 --> 00:16:46.891 A lot of these, especially the low cost embedded Linux SoCs, 00:16:46.921 --> 00:16:50.761 don't have nearly as good power management as a nice phone would. 00:16:51.111 --> 00:16:53.901 And if you're trying to deploy something that's going to last 00:16:53.911 --> 00:16:57.781 for a year with no solar or no landline or anything like that. 00:16:57.974 --> 00:17:01.709 If you have like a Raspberry Pi, unless you have a car battery it's going 00:17:01.709 --> 00:17:06.759 to drain your battery in a day or a week or maybe a month, but you want 00:17:06.759 --> 00:17:08.759 something that's going to last years. 00:17:09.039 --> 00:17:11.559 What you really need to be able to do is not just sleep, but 00:17:11.559 --> 00:17:13.519 shut down the power hungry... 00:17:13.959 --> 00:17:15.752 it's one of those things where it's not even the CPU like... 00:17:15.802 --> 00:17:19.762 One of the differences between SRAM- static RAM- and DRAM, which is 00:17:19.762 --> 00:17:23.292 dynamic RAM is you have to refresh dynamic RAM constantly which takes 00:17:23.292 --> 00:17:27.282 electricity to basically recharge the capacitors in the memory so that 00:17:27.292 --> 00:17:28.872 your memory doesn't get corrupted. 00:17:29.132 --> 00:17:32.492 And that takes energy, and so if you can't shut off your memory, if you 00:17:32.492 --> 00:17:35.792 can't shut off your CPU, you just have this sort of baseline power 00:17:35.792 --> 00:17:39.153 usage that most, not all of them, most microcontrollers will be able to 00:17:39.163 --> 00:17:41.733 go down to like micro or nano amps. 00:17:42.173 --> 00:17:45.743 Whereas a lot of these embedded Linux SoCs are way, like, orders of magnitude 00:17:45.743 --> 00:17:49.743 higher in milliamps, even when they're in sort of like low power mode. 00:17:50.540 --> 00:17:53.520 And I'm going to tell you to use Rust for both because I am who I am, but 00:17:53.660 --> 00:17:56.010 it allows you to do a lot of very cool things because we don't have 00:17:56.010 --> 00:17:59.672 to pretend that our microcontrollers are archaic little systems anymore. 00:17:59.982 --> 00:18:03.242 They are 32 bit processors, and that's not the same as the 64 bit 00:18:03.242 --> 00:18:06.782 processor you're using, but it's much closer than the little 8 bit 00:18:06.782 --> 00:18:08.672 microcontrollers with weird architectures, 00:18:08.912 --> 00:18:12.912 which means you can share code, tools, workflows, and even developers. 00:18:12.932 --> 00:18:16.712 It gets to the point where if I sat you down and showed you an embassy project 00:18:16.712 --> 00:18:20.332 that's still using Rust async, there might be some learning curve of like, "Oh, 00:18:20.332 --> 00:18:21.672 I need to learn some different habits. 00:18:21.672 --> 00:18:23.132 Cause not everything's great here." 00:18:23.537 --> 00:18:25.047 But the big picture would make sense. 00:18:25.581 --> 00:18:27.137 You'd go, "Yeah, I'm going to put a timeout on it. 00:18:27.137 --> 00:18:27.777 I'm going to await." 00:18:28.017 --> 00:18:30.137 It's all going to make sense fairly quickly. 00:18:30.177 --> 00:18:33.292 And that's super valuable because a lot of embedded teams that I work 00:18:33.292 --> 00:18:37.343 with, there's usually like five or 10 backend and front end people. 00:18:37.603 --> 00:18:41.473 There's like one embedded developer and one hardware person, and that's 00:18:41.473 --> 00:18:44.513 like the whole team, which means if you ever need peer review or getting 00:18:44.513 --> 00:18:47.393 people to work together, if you're not sharing tools and language, it 00:18:47.393 --> 00:18:51.963 becomes very challenging to ever get good code review or just a good second 00:18:51.963 --> 00:18:53.753 set of eyes on what you're building. 00:18:54.223 --> 00:18:59.253 Yeah, and I mean, this is context for the entire first season of 00:18:59.263 --> 00:19:03.673 SDR, because it's like: why do you care about serialization/deserialization, why 00:19:03.673 --> 00:19:05.293 do you care about it running on embedded? 00:19:05.513 --> 00:19:05.833 That's true! 00:19:06.263 --> 00:19:08.513 The nice thing of like, if it runs on embedded, it will definitely 00:19:08.513 --> 00:19:14.083 for sure work on desktop or like a Raspberry Pi 4 or 5 Cortex-A, I guess. 00:19:14.159 --> 00:19:14.789 Yeah, exactly. 00:19:14.789 --> 00:19:17.679 And that's what I'm going to recommend is you tie it together with a library like 00:19:17.679 --> 00:19:22.049 Postcard RPC or a tool that I'm working on called Poststation, which is sort of 00:19:22.049 --> 00:19:24.259 like a reverse proxy for embedded devices. 00:19:24.259 --> 00:19:27.309 But the thing is it's a way of communicating where we treat 00:19:27.309 --> 00:19:31.054 them like a real set of peers that communicate with each other. 00:19:31.054 --> 00:19:34.094 In the same way that we treat a browser and a server as peers, 00:19:34.284 --> 00:19:37.482 like the client and the server, they are equivalent to each other. 00:19:37.482 --> 00:19:40.242 We're not just treating them like totally different machines. 00:19:40.242 --> 00:19:41.272 They are both computers. 00:19:41.602 --> 00:19:44.922 Using something like Postcard RPC also lets you use whatever communication 00:19:44.922 --> 00:19:46.742 link you have between these devices. 00:19:46.822 --> 00:19:51.697 Like the Raspberry Pi, it's got a USB port, it's got UART or a serial 00:19:51.697 --> 00:19:55.627 port, it's got SPI or I2C, all these different kinds of serial ports that 00:19:55.627 --> 00:19:59.571 are better or worse at certain things, but you can use whatever links you have, 00:19:59.631 --> 00:20:03.581 because Postcard RPC is very simple, and Poststation is very flexible in 00:20:03.581 --> 00:20:07.391 what it will proxy to, instead of just over the network, you can throw it over 00:20:07.676 --> 00:20:12.136 essentially any interface that you can make frames on, which is every serial 00:20:12.136 --> 00:20:13.986 interface, every one that I have here. 00:20:14.256 --> 00:20:18.516 So it becomes very easy to proxy over whatever link you have, even 00:20:18.516 --> 00:20:19.926 if it doesn't look like ethernet. 00:20:20.199 --> 00:20:22.719 And really my goal here is to have the best of both worlds. 00:20:22.739 --> 00:20:25.699 You don't have to sort of pick and choose and be awkward on some things and 00:20:25.699 --> 00:20:29.889 just lean into it, design your system in a way where you go: look, all the 00:20:29.889 --> 00:20:32.859 real-time stuff- the microcontroller is going to be in charge of that. 00:20:33.089 --> 00:20:36.979 And I'm going to make it sort of the authoritative one on anything safety 00:20:36.979 --> 00:20:40.979 or real-time or signal processing or IO expansion and things like that. 00:20:41.349 --> 00:20:45.924 And for all the big number crunching and networking and deciding user 00:20:45.924 --> 00:20:48.654 interface things, I'm going to put that on an embedded Linux and it's 00:20:48.654 --> 00:20:49.934 going to be in charge of that. 00:20:50.204 --> 00:20:52.854 And when you think of your system, you're going to go, who's in charge of what? 00:20:52.916 --> 00:20:56.254 In a network cluster, you're like: that's my database server and that's my front 00:20:56.254 --> 00:20:57.794 end server and that's my caching server. 00:20:57.974 --> 00:21:01.094 They're specialized to different tasks, but I still treat them as 00:21:01.104 --> 00:21:04.894 peers that are working together on like the bigger common goal. 00:21:04.952 --> 00:21:07.729 I keep thinking of how you said you can use Rust on the big 00:21:07.729 --> 00:21:12.699 one and on the little one, and the equivalent in my head I have is using 00:21:12.699 --> 00:21:15.089 JavaScript in the browser and on server. 00:21:15.579 --> 00:21:17.708 And it's much more awkward to do that. 00:21:17.723 --> 00:21:18.433 What's the word for that? 00:21:18.463 --> 00:21:19.313 Isomorphic. 00:21:19.378 --> 00:21:20.468 Yeah, I think so. 00:21:20.648 --> 00:21:23.793 But with Rust, it's so interesting because I've been mostly in touch with people 00:21:23.793 --> 00:21:28.313 who write Rust not on embedded and they look at async Rust and they're like, "Ah, 00:21:28.313 --> 00:21:31.303 it's, it's too complicated," and they don't understand the design constraints. 00:21:31.303 --> 00:21:36.348 But then if you actually are working on both sides, you see why it was 00:21:36.348 --> 00:21:40.338 designed that way, because then you just get to write async Rust code on 00:21:40.368 --> 00:21:43.228 those tiny, well, tiny ish devices. 00:21:43.543 --> 00:21:44.343 Yeah, for sure. 00:21:44.438 --> 00:21:45.223 And it's the same language. 00:21:45.227 --> 00:21:47.967 It's something I really appreciate with Rust, is that you have so much range. 00:21:48.027 --> 00:21:52.597 You can build abstractions, you're not stuck to the ground, like with 00:21:52.597 --> 00:21:53.637 C, where it's like, Nowhere safe. 00:21:53.742 --> 00:21:55.722 Truck is all you get and that's it. 00:21:55.992 --> 00:21:57.312 You don't get anything else. 00:21:57.312 --> 00:22:00.052 Or C++ where I'm not sure what's happening with rustc. 00:22:00.072 --> 00:22:03.162 You get like this whole range and you can go down into the library. 00:22:03.282 --> 00:22:04.842 And, yeah, there's some unsafe code. 00:22:04.842 --> 00:22:05.142 Okay. 00:22:05.142 --> 00:22:08.652 Unsafe is- it's- it needs to be there, but it's all the same language. 00:22:08.652 --> 00:22:12.132 You can dig, you can understand there's no VM boundary, like high 00:22:12.132 --> 00:22:13.662 level code versus low level code. 00:22:13.842 --> 00:22:14.922 It's all the same language. 00:22:15.252 --> 00:22:15.482 Yeah. 00:22:15.482 --> 00:22:18.452 And I think the isomorphic JavaScript speaks to that of like, there's 00:22:18.452 --> 00:22:23.272 a tangible benefit when you have multiple parts in your system, having 00:22:23.282 --> 00:22:25.712 common tooling and common language and things like that, because 00:22:25.712 --> 00:22:27.772 there's developer benefits to that. 00:22:27.827 --> 00:22:30.107 Because you're able to jump between the two of them. 00:22:30.117 --> 00:22:31.957 This is something that I recommend to people where they're like, 00:22:31.957 --> 00:22:33.317 "Rust is really good at FFI!" 00:22:33.627 --> 00:22:37.497 and I go, "Yes, it is, if you have to," but if you can, sometimes like, 00:22:37.527 --> 00:22:41.457 there's a real benefit to only having one language because anytime you jump 00:22:41.487 --> 00:22:45.057 that gap, you need to be an expert in the semantics of both of them. 00:22:45.367 --> 00:22:49.637 That's possible, but it's way easier to get someone to be really understanding 00:22:49.637 --> 00:22:53.177 of Rust and where things start and stop there, or even just someone to be familiar 00:22:53.177 --> 00:22:56.907 and C where things start and stop there, whereas whenever you transfer over 00:22:56.907 --> 00:22:58.897 that boundary, you have to think both. 00:22:59.002 --> 00:23:03.215 And that's true over the network as well, because if you have some Go server that 00:23:03.215 --> 00:23:07.095 does verification of data in one way, but JavaScript or Node does it in a different 00:23:07.095 --> 00:23:11.435 way, you get something that mostly works until that one day when they don't agree. 00:23:11.725 --> 00:23:15.225 And then you need someone who knows enough about both of them to really 00:23:15.225 --> 00:23:18.365 figure out where that just like tiny little slip in the seam is. 00:23:18.695 --> 00:23:19.055 Yeah. 00:23:19.108 --> 00:23:22.928 Again, that's also a place where Rust shines, is that you- specifically you 00:23:22.928 --> 00:23:26.048 in Postcard RPC, are leveraging the type system a lot, so there's just 00:23:26.048 --> 00:23:27.658 a lot of mistakes you cannot make. 00:23:27.738 --> 00:23:31.518 When I try to teach Rust, I don't really try to advocate for it, I just try to 00:23:31.518 --> 00:23:34.468 teach it, because I learned it, I thought it was neat, and so I've been teaching 00:23:34.468 --> 00:23:36.398 it, and people are advocating for it. 00:23:36.478 --> 00:23:39.988 What you're taking away from them is the pride of being in very dangerous 00:23:40.028 --> 00:23:43.578 waters and not dying, like having all those mistakes you can make, 00:23:43.728 --> 00:23:47.310 but being smart and clever enough that you don't actually make them. 00:23:47.740 --> 00:23:50.300 And then switching to something like Rust is like: okay, occasionally 00:23:50.310 --> 00:23:52.650 you're going to have the compiler be mad at you and it's going to 00:23:52.660 --> 00:23:53.680 take some time to understand why. 00:23:53.710 --> 00:23:56.325 But most of the time, there's so many mistakes you can never 00:23:56.325 --> 00:23:58.215 make and it's so relaxing. 00:23:58.225 --> 00:24:01.651 You just kind of: if it compiles it probably works, if you design your 00:24:01.651 --> 00:24:03.141 types correctly which is a big if. 00:24:03.161 --> 00:24:05.091 But you know you get a sense for it after a while. 00:24:05.261 --> 00:24:08.291 That in the embedded world, where you don't have all the niceties of 00:24:08.291 --> 00:24:10.953 the operating system which is going to have your back and like just it's 00:24:10.953 --> 00:24:13.679 no big deal to blow up the stacks no big deal to do a lot of things, you 00:24:13.689 --> 00:24:14.839 don't even have to free your memory. 00:24:15.159 --> 00:24:17.299 If you exit the process you're just going to reclaim everything. 00:24:17.299 --> 00:24:18.509 Virtual memory is really nice. 00:24:18.539 --> 00:24:19.782 I don't know if you get that on MCUs. 00:24:19.942 --> 00:24:20.552 No, nope. 00:24:20.576 --> 00:24:22.446 We have no MMU, so there's no virtual memory. 00:24:22.866 --> 00:24:23.266 Exactly. 00:24:23.316 --> 00:24:26.546 But my threshold for this is usually around one to 10,000 units. 00:24:26.639 --> 00:24:30.199 This number seems higher than most people, where I'm like: below that, 00:24:30.226 --> 00:24:34.226 what you do in terms of the hardware you design surprisingly doesn't matter. 00:24:34.446 --> 00:24:37.286 That's probably more like one to ten thousand units per year. 00:24:37.536 --> 00:24:38.996 If you're below that threshold... 00:24:39.176 --> 00:24:43.472 you mentioned Kubernetes- embedded sort of has the same problem as some 00:24:43.472 --> 00:24:47.162 people complain about Kubernetes- you're designing for a scale that you don't have. 00:24:47.362 --> 00:24:49.712 It's not that it's a bad solution, but it's a solution 00:24:49.712 --> 00:24:51.672 to a problem you don't have yet. 00:24:51.702 --> 00:24:56.402 And so when I see people like really optimizing the cost of per unit If 00:24:56.402 --> 00:25:00.222 you're not making 10,000 units, yeah, don't do something super egregious, 00:25:00.232 --> 00:25:01.852 but I'll get to this in a little bit. 00:25:01.852 --> 00:25:02.122 Like... 00:25:02.462 --> 00:25:06.667 the buddy system is probably going to be cheaper in the big 00:25:06.667 --> 00:25:08.083 picture for what you're doing. 00:25:08.083 --> 00:25:10.443 I see what you're doing with the Kubernetes comparison and 00:25:10.483 --> 00:25:15.503 I agree with your point, but I think Kubernetes makes sense for a lot more 00:25:15.503 --> 00:25:17.033 people just because it's standardized. 00:25:17.033 --> 00:25:18.803 I think we'll have to save that chat for another day. 00:25:18.928 --> 00:25:20.498 It breaks down at this point! 00:25:21.084 --> 00:25:23.890 Because you brought up Kubernetes: there is a nice article that I 00:25:23.890 --> 00:25:28.060 read recently called, "Dear friend, I'm afraid you've just written a 00:25:28.060 --> 00:25:29.210 Kubernetes," or something like that? 00:25:29.230 --> 00:25:32.214 We'll have it in the show notes, so that we don't have to discuss it right now. 00:25:32.284 --> 00:25:32.454 Sure. 00:25:33.292 --> 00:25:37.622 And so my pitch is if you're under this sort of like one to 10,000 unit threshold, 00:25:37.882 --> 00:25:39.752 the buddy system is probably cheaper. 00:25:39.752 --> 00:25:42.342 Even though I'm telling you to use two chips, you know, a slightly 00:25:42.342 --> 00:25:45.702 more complex design, because you've got two systems that you're writing 00:25:45.702 --> 00:25:47.142 software for and things like that. 00:25:47.372 --> 00:25:49.592 It will likely end up being cheaper because 00:25:49.782 --> 00:25:56.227 development time is expensive, and being able to do two easy things is 00:25:56.237 --> 00:26:00.401 often much faster as a developer for trying to pack both of those easy 00:26:00.401 --> 00:26:04.161 things into one thing, which makes it end up being a very complex problem. 00:26:04.171 --> 00:26:07.721 It's not that you can't, it's that: you might have to write kernel patches, you 00:26:07.721 --> 00:26:10.311 might have to figure out a lot of external hardware that you need as adapters, 00:26:10.311 --> 00:26:13.811 you might have to really pack things in and optimize because you're like, "I 00:26:13.811 --> 00:26:15.891 really have to hit that timing thing." 00:26:16.151 --> 00:26:20.191 If you just use two systems, then you can let them both play on their 00:26:20.191 --> 00:26:24.241 strengths and you can even pick versions of each of those two buddies 00:26:24.431 --> 00:26:26.111 that are really specialized for that. 00:26:26.111 --> 00:26:31.341 And usually, things that are specialized for one feature is much cheaper than if 00:26:31.341 --> 00:26:35.491 you need one chip that is specialized on three different things because 00:26:35.491 --> 00:26:36.921 you need to do that all in one chip. 00:26:37.091 --> 00:26:40.051 So either the per unit cost or just the amount of time you spend 00:26:40.051 --> 00:26:43.701 on development because developers are very expensive per hour. 00:26:43.747 --> 00:26:44.547 I agree. 00:26:44.615 --> 00:26:47.405 And yeah, like I said, doing microcontroller things on Linux sucks. 00:26:47.445 --> 00:26:48.155 You can do it. 00:26:48.155 --> 00:26:49.335 There's the real-time patches. 00:26:49.565 --> 00:26:51.205 Down to some threshold it will be fine. 00:26:51.225 --> 00:26:53.315 If you're building something that needs one serial port, 00:26:53.565 --> 00:26:54.765 ignore me, you'll be fine. 00:26:54.965 --> 00:26:58.365 As soon as that starts chafing, as soon as it starts being uncomfortable, 00:26:58.565 --> 00:27:01.925 that should be your sign of like, instead of trying harder and committing 00:27:02.028 --> 00:27:03.558 harder and harder to optimize things. 00:27:03.608 --> 00:27:06.278 Just give up, use the buddy system because you know, for 00:27:06.278 --> 00:27:07.488 one or two things, it's fine. 00:27:07.668 --> 00:27:09.068 When you get to five, it's uncomfortable. 00:27:09.068 --> 00:27:10.518 When you get to 10, it's egregious. 00:27:10.568 --> 00:27:12.998 It's one of those things where you don't have to necessarily do it on day one, 00:27:13.058 --> 00:27:14.278 unless you think you might need it. 00:27:14.518 --> 00:27:18.565 But just trying to make the big system do little system things is painful. 00:27:18.995 --> 00:27:20.915 I'm imagining the Floppotron? 00:27:21.885 --> 00:27:22.095 Okay. 00:27:22.095 --> 00:27:24.865 The thing that plays music with like a hundred floppy drives or 00:27:24.865 --> 00:27:27.145 whatever, and it's taking MIDI files? 00:27:27.295 --> 00:27:29.305 That's something you would need an MCU for, right? 00:27:29.333 --> 00:27:30.429 To drive all of those. 00:27:30.480 --> 00:27:31.665 Or like a lot of expanders. 00:27:31.777 --> 00:27:34.987 You need stepper motor controller drivers times a hundred. 00:27:35.045 --> 00:27:36.785 The timing has to be very precise. 00:27:36.785 --> 00:27:37.175 Yeah. 00:27:37.175 --> 00:27:37.176 Yeah. 00:27:37.220 --> 00:27:37.480 Yeah. 00:27:37.480 --> 00:27:40.199 You won't find any embedded Linux system that can run a hundred of those. 00:27:40.199 --> 00:27:43.979 So it needs to be a special driver IC, or you need some way of managing that. 00:27:43.979 --> 00:27:46.435 And the same way doing Linux things on a microcontroller sucks. 00:27:46.455 --> 00:27:49.825 Like there are libraries that are nice these days where you can have a 00:27:49.825 --> 00:27:52.145 TCP/IP stack on an embedded device. 00:27:52.145 --> 00:27:56.315 You can have HTTPS on an embedded device and it's async and it's 00:27:56.315 --> 00:27:59.455 nice, but it still means you need a really big microcontroller. 00:27:59.675 --> 00:28:01.815 It still means that you're running a network stack. 00:28:01.865 --> 00:28:04.205 What happens when you want to update your certificate chain? 00:28:04.265 --> 00:28:06.510 On Linux it's just an apt get update. 00:28:06.550 --> 00:28:09.490 And on the microcontroller, you go: we need to be really careful because 00:28:09.490 --> 00:28:11.130 we want to do our update over SSL. 00:28:11.150 --> 00:28:14.410 If we ever send a corrupted set of SSL certificates or TLS 00:28:14.410 --> 00:28:17.420 certificates- oops, now our firmware updates don't work anymore! 00:28:17.590 --> 00:28:20.220 I was thinking Linux people get uppity when you do static 00:28:20.220 --> 00:28:21.770 linking, like bring your own libraries. 00:28:21.770 --> 00:28:25.470 Cause they're like, "No, I want you to use my distribution's version of LibSSL!" 00:28:25.470 --> 00:28:27.080 So that they can patch vulnerabilities. 00:28:27.080 --> 00:28:31.000 But now imagine your entire TCP stack is baked into- 00:28:31.010 --> 00:28:32.160 Baked into the firmware. 00:28:32.160 --> 00:28:33.140 Yep, exactly. 00:28:33.220 --> 00:28:33.490 Yeah. 00:28:33.490 --> 00:28:34.990 A firmware is just one application. 00:28:35.040 --> 00:28:37.270 Like it is an application that does operating system 00:28:37.270 --> 00:28:38.960 jobs is all a firmware is. 00:28:39.140 --> 00:28:41.760 And the thing is, off the shelf boards are cheap, like I said. 00:28:41.850 --> 00:28:48.700 It's way easier to buy a high end off the shelf board than to design a custom 00:28:48.819 --> 00:28:53.599 optimized low cost board, because just economies of scale are what they are, and 00:28:53.599 --> 00:28:57.789 it's cheaper, in a lot of cases, to get a Raspberry Pi than a correctly sized thing. 00:28:57.789 --> 00:28:59.299 And this is one of those things that people on the 00:28:59.299 --> 00:29:00.489 internet lose their mind over. 00:29:00.669 --> 00:29:01.249 They're like, "Ah! 00:29:01.249 --> 00:29:03.519 Why are you using a whole Raspberry Pi to do that?!" 00:29:03.519 --> 00:29:07.849 And it's like, because my time is expensive and that costs 40 bucks. 00:29:07.879 --> 00:29:13.049 And by the time that I have to buy three cheap things and wire them together, even 00:29:13.049 --> 00:29:14.699 just in parts, I've probably paid more. 00:29:14.699 --> 00:29:18.779 But then when I factored my time, we're way, way over these things. 00:29:18.839 --> 00:29:21.629 Do your hobbies as you like, if you like retrocomputing, yes. 00:29:21.819 --> 00:29:26.012 But you should not approach a company project as this, unless you have to. 00:29:26.694 --> 00:29:29.884 That's the other thing: when you do want to design something of your own, the 00:29:29.884 --> 00:29:31.744 simpler it can be, the cheaper it is. 00:29:31.774 --> 00:29:35.616 If you can have someone throw together a two layer circuit board, or use off 00:29:35.616 --> 00:29:39.066 the shelf parts that have fairly big pins, and you can throw them in on a 00:29:39.066 --> 00:29:42.562 board, I can send that design over to JLC, and in two weeks I'll have the 00:29:42.562 --> 00:29:44.312 board for, like, five bucks a board. 00:29:44.522 --> 00:29:47.072 Or I can get it made locally for a bit more, but I'll have it in, 00:29:47.072 --> 00:29:50.937 like, two days, because I don't need an eight layer graphics card 00:29:50.947 --> 00:29:52.927 design sort of thing going on. 00:29:52.997 --> 00:29:55.467 So even when you do design something custom, cause you want 00:29:55.527 --> 00:29:58.977 it to be smaller or fit whatever you'd like, that's cheaper too. 00:29:59.007 --> 00:30:02.877 As long as you can have two simple things instead of one really complex thing. 00:30:02.877 --> 00:30:07.607 Cause if you need the 800 pin count CPU to get all of your IO, that's 00:30:07.607 --> 00:30:08.817 going to be an expensive board. 00:30:08.847 --> 00:30:10.257 And every time you get it wrong... 00:30:10.717 --> 00:30:14.659 Circuit board design is like when you hit compile, it takes two weeks to figure 00:30:14.659 --> 00:30:18.749 out if the compile worked and also like a couple thousand dollars every time 00:30:18.749 --> 00:30:21.399 you compile because you have to get these board made, you have to bring 00:30:21.399 --> 00:30:24.849 them in, someone either has to do the assembly or you have to do the assembly 00:30:25.049 --> 00:30:27.259 and then you might find you shorted two pins together and the whole power 00:30:27.259 --> 00:30:30.279 supply fries and you go, "Well, I guess we're going to wait two more weeks." 00:30:30.689 --> 00:30:33.439 Versus a simple design, you go: yeah, we'll just slap it on. 00:30:33.439 --> 00:30:35.532 It's probably going to work more likely in the first thing. 00:30:35.532 --> 00:30:39.659 And even if it does not work, it's faster to cycle and cheaper to cycle. 00:30:39.743 --> 00:30:43.573 If you go even deeper in designing your own hardware and you're 00:30:43.573 --> 00:30:48.703 writing VHDL or whatever the other one is, you can simulate those kind 00:30:48.703 --> 00:30:50.813 of, but synthesizing takes forever. 00:30:50.823 --> 00:30:53.493 The whole tooling around that is horrible. 00:30:53.773 --> 00:30:56.273 I've done a little bit of it at university, so it dates back. 00:30:56.313 --> 00:30:58.910 I hope things are improved, but I'm not holding my breath. 00:30:59.360 --> 00:31:02.142 FPGAs are a whole conversation for another day. 00:31:02.463 --> 00:31:05.283 And that's the other thing about Rust is you get a lot of stuff off the shelf. 00:31:05.313 --> 00:31:08.683 Like I said, I talked about postcard, but we also get libraries for stuff. 00:31:08.723 --> 00:31:12.233 And this is both on the embedded Linux side and the microcontroller side. 00:31:12.473 --> 00:31:16.623 Sometimes the same library, like in Postcard, but if not, when 00:31:16.623 --> 00:31:18.723 you're doing your microcontroller stuff, we have embassy. 00:31:18.883 --> 00:31:21.303 You don't have to build your own OS or scheduler or whatever. 00:31:21.573 --> 00:31:23.333 You know, there's drivers for all of these things. 00:31:23.333 --> 00:31:24.463 They're fairly portable. 00:31:24.633 --> 00:31:27.713 We have the whole trait system, which allows you to do portable drivers and 00:31:27.713 --> 00:31:31.973 stuff, even to the point where you could port it from the Raspberry Pi to 00:31:31.973 --> 00:31:36.343 the Raspberry Pi Pico and use the exact same display driver on both of them. 00:31:36.643 --> 00:31:41.198 So like being able to use pieces off the shelf, either public ones or ones 00:31:41.198 --> 00:31:43.998 that you've built at your company where you go: well, sometimes we just have 00:31:43.998 --> 00:31:47.468 the Raspberry Pi, but sometimes we have the Raspberry Pi with a buddy system. 00:31:47.898 --> 00:31:51.328 You just write the driver once and use it in house and reuse things. 00:31:51.648 --> 00:31:53.388 Yeah, I guess a buddy system with a language 00:31:53.388 --> 00:31:55.798 like C, an ecosystem like C- 00:31:55.867 --> 00:31:56.378 It's hard. 00:31:56.418 --> 00:31:58.298 -without a package manager or a standardized thing, 00:31:58.298 --> 00:31:59.438 yeah, it would be a lot more work. 00:31:59.569 --> 00:32:00.528 Yeah, it's possible. 00:32:00.528 --> 00:32:01.228 And people have done it. 00:32:01.228 --> 00:32:02.328 It's just, it's challenging. 00:32:02.368 --> 00:32:04.738 Like the language doesn't fight you, but it doesn't help you. 00:32:04.988 --> 00:32:05.768 It can be done. 00:32:05.989 --> 00:32:09.536 Yeah, you have to enforce convention inside of the project... 00:32:09.688 --> 00:32:11.056 And be really careful, yeah. 00:32:11.916 --> 00:32:13.796 So Postcard RPC is an off the shelf comm stack. 00:32:13.806 --> 00:32:18.076 You just plug stuff together, whether it's over USB or serial or whatever. 00:32:18.296 --> 00:32:20.936 And you just have client and server libraries for both of them. 00:32:21.436 --> 00:32:24.076 And Poststation, the tool that I'm building now also has a bunch of 00:32:24.456 --> 00:32:26.446 tools and APIs and SDKs for it. 00:32:26.466 --> 00:32:29.980 So when you plug your device in, you get a GUI for it, or you get 00:32:29.980 --> 00:32:31.541 history data and things like that. 00:32:31.591 --> 00:32:35.281 Whether you're plugging it into your laptop or the actual embedded SoC, you 00:32:35.281 --> 00:32:38.751 can just mess around, because there's different stages to designing hardware. 00:32:39.031 --> 00:32:42.041 First, you design the circuit boards and usually you're sitting there making 00:32:42.041 --> 00:32:44.841 sure all the motors spin and you can read raw sensor data where you're not 00:32:44.851 --> 00:32:47.291 really doing business logic stuff yet. 00:32:47.301 --> 00:32:50.961 You're just poking it slowly to make sure it does all work. 00:32:51.371 --> 00:32:53.431 And then you sort of iterate from there where you go: okay, now I'm 00:32:53.431 --> 00:32:57.451 going to make it automatically do this at 100 hertz or whatever, and 00:32:57.461 --> 00:32:59.081 then aggregate or filter the data. 00:32:59.091 --> 00:33:01.321 And then I get to sort of the shape that I want. 00:33:01.904 --> 00:33:06.247 Being able to not have to build custom stuff, not having to build jigs yourself 00:33:06.267 --> 00:33:08.677 where you can just go: okay, I'm just gonna, can I poke, poke, poke? 00:33:08.757 --> 00:33:09.067 Cool. 00:33:09.107 --> 00:33:09.447 Done. 00:33:09.527 --> 00:33:10.007 Move on. 00:33:10.177 --> 00:33:13.454 And then I can hand that tool to the factory people to make sure that whenever 00:33:13.454 --> 00:33:18.145 we assemble one, that it wasn't miswired or not calibrated or whatever you need. 00:33:18.889 --> 00:33:23.140 We've seen you do that a bit on your YouTube live streaming channel? 00:33:23.150 --> 00:33:23.430 You've been 00:33:23.665 --> 00:33:24.115 - Yeah. 00:33:24.410 --> 00:33:26.923 -iterating on the real life version... 00:33:27.003 --> 00:33:27.593 Of our logo? 00:33:27.593 --> 00:33:27.963 Yeah. 00:33:27.985 --> 00:33:28.965 of the SDR logo. 00:33:29.120 --> 00:33:29.360 Yeah. 00:33:29.360 --> 00:33:32.120 That was a fun one because a lot of that is like the first stage of any 00:33:32.120 --> 00:33:35.500 project is just wiring it up and then poking it to make sure that you 00:33:35.510 --> 00:33:38.690 wired it up right before you start spending a lot of time writing a lot 00:33:38.690 --> 00:33:41.114 of software that adds more confusion. 00:33:41.114 --> 00:33:41.724 to the pile. 00:33:41.879 --> 00:33:44.575 And the other thing it lets you do is scale horizontally This is another one 00:33:44.575 --> 00:33:47.845 of those server terms: scaling vertically means you just buy a bigger server. 00:33:47.875 --> 00:33:52.689 You get the 64 with 96 gigs of ram instead of a bunch of little machines, 00:33:52.689 --> 00:33:55.753 because for some things that's just easier because then you're not 00:33:55.783 --> 00:33:57.303 worrying about a bunch of machines, but 00:33:57.573 --> 00:34:00.148 James, they make so much bigger servers. 00:34:00.193 --> 00:34:00.894 Yeah, they do. 00:34:00.933 --> 00:34:01.503 Yeah, they do. 00:34:01.503 --> 00:34:02.603 You can have terabytes of RAM. 00:34:02.873 --> 00:34:03.313 Hell yeah. 00:34:03.493 --> 00:34:05.363 But scaling horizontally lets you say: look... 00:34:05.573 --> 00:34:09.823 if I'm doing something really weird and I need a hundred input pins, instead of 00:34:09.823 --> 00:34:13.403 trying to find a chip that has a hundred input pins, you go: I'm just going to buy 00:34:13.823 --> 00:34:16.763 10 chips that have a hundred input pins. 00:34:16.773 --> 00:34:20.803 So then I get my thousand input pins and that's 10 USB connections. 00:34:20.803 --> 00:34:22.703 Linux is like, whatever, that's a USB hub. 00:34:22.743 --> 00:34:23.643 It's not a big deal. 00:34:24.363 --> 00:34:27.763 So we can go from our embedded Linux SoC with one microcontroller. 00:34:28.078 --> 00:34:29.848 We stick a little USB hub in front of it. 00:34:29.848 --> 00:34:32.988 And all of a sudden we now have triple the capacity for the thing that we're doing. 00:34:32.988 --> 00:34:35.458 We can have triple the motors, triple the sensors, whatever. 00:34:35.808 --> 00:34:37.598 And this was cheap to do. 00:34:37.608 --> 00:34:43.338 You can buy a single chip USB hub from WCH for like 60 cents and 00:34:43.338 --> 00:34:45.168 it's four port USB high speed. 00:34:45.178 --> 00:34:48.568 And all of a sudden you can talk to four devices now for an extra buck. 00:34:48.638 --> 00:34:53.055 What is the maximum number of devices you can hook onto a USB hub? 00:34:53.221 --> 00:34:55.655 There's, uh- I don't remember either limit off the top of my head. 00:34:55.665 --> 00:34:58.265 The answer is you'll run into some limit in your operating system and 00:34:58.265 --> 00:35:02.225 you'll run into some limit in the version of USB that you are using. 00:35:02.225 --> 00:35:04.995 So like full speed and high speed, but it's in the hundreds. 00:35:05.085 --> 00:35:08.777 Up to like a hundred or so everything will probably work if you have enough 00:35:08.777 --> 00:35:13.099 power and things like that, that you might have some degradation of how 00:35:13.099 --> 00:35:14.659 fast you can talk to all of them. 00:35:14.659 --> 00:35:17.859 You'll hit some other thresholds first, but like tens: no problem. 00:35:18.109 --> 00:35:20.699 Fifties: probably fine to clear. 00:35:20.849 --> 00:35:21.479 Hundreds... 00:35:21.489 --> 00:35:22.919 well, you'll probably start having problems. 00:35:22.919 --> 00:35:25.609 James, I need to pitch you a video idea to your company. 00:35:25.629 --> 00:35:29.559 It's how many USB devices can we hook up to a single computer before 00:35:29.679 --> 00:35:31.049 things start going very wrong? 00:35:31.336 --> 00:35:31.636 What was it? 00:35:31.726 --> 00:35:32.666 LTT did that. 00:35:33.077 --> 00:35:35.551 Yeah, they did it with like keyboards or, flash drives or something. 00:35:35.581 --> 00:35:37.491 We can put that in the show notes, but they did exactly that. 00:35:37.681 --> 00:35:41.111 They even talk about like, there's the theoretical USB protocol limit. 00:35:41.461 --> 00:35:43.261 And then there's when Windows just falls over. 00:35:43.321 --> 00:35:44.341 They all have power. 00:35:44.341 --> 00:35:45.171 It should be fine. 00:35:45.171 --> 00:35:48.241 But just at some point, the operating system's like, "I just can't anymore." 00:35:48.241 --> 00:35:50.681 And it drops like 60 of them immediately. 00:35:50.681 --> 00:35:52.261 Like things just timeout probably. 00:35:52.291 --> 00:35:55.151 Then the next obvious step is to write a driver so that you can have 00:35:55.161 --> 00:35:59.611 virtual USB devices that are actually proxied over ethernet to a mega hub. 00:35:59.611 --> 00:35:59.701 Yeah. 00:35:59.891 --> 00:36:01.981 And I'm not sure Linus did that one, James. 00:36:02.021 --> 00:36:02.431 Did he? 00:36:03.071 --> 00:36:05.351 By the time that you just have that- probably Ethernet's the answer. 00:36:05.386 --> 00:36:05.906 Yeah. 00:36:06.001 --> 00:36:08.471 But that's again, another conversation... 00:36:08.506 --> 00:36:10.096 Mac Minis are going the other way, you're replacing 00:36:10.116 --> 00:36:12.796 Ethernet with USB4, then you get, 00:36:12.981 --> 00:36:14.121 or Thunderbolt specifically. 00:36:14.121 --> 00:36:14.471 yeah. 00:36:15.456 --> 00:36:17.732 Isn't USB4 and Thunderbolt basically the same thing now? 00:36:17.799 --> 00:36:18.786 Uh, kinda. 00:36:18.786 --> 00:36:19.996 They're on the same connector. 00:36:19.996 --> 00:36:21.006 They have similar capabilities. 00:36:21.026 --> 00:36:23.097 Thunderbolt is slightly different. 00:36:23.153 --> 00:36:24.207 I don't know off the top of my head. 00:36:24.442 --> 00:36:25.432 That's very confusing. 00:36:26.197 --> 00:36:29.457 My opinion is more buddies, more better, because if you 00:36:29.457 --> 00:36:32.827 can make a firmware that does one thing and it's a hundred lines of 00:36:32.827 --> 00:36:36.857 code and it is obviously correct, you will never need to touch that. 00:36:37.077 --> 00:36:40.857 If you write a firmware that does 10 things and it's 10,000 lines of 00:36:40.857 --> 00:36:43.947 code and you have to worry about interactions between the different 00:36:43.947 --> 00:36:48.527 ones, that's not 10 times harder, it's a hundred times harder versus having 00:36:48.527 --> 00:36:52.527 10 really simple things where you go: yeah, I can see it all on one page. 00:36:52.737 --> 00:36:56.597 It's obviously just going to receive a message, set the motor value. 00:36:56.867 --> 00:36:58.497 I don't have to worry about synchronization. 00:36:58.507 --> 00:36:59.227 It's all fine. 00:36:59.237 --> 00:37:01.407 You don't want to talk about Kubernetes, cause you're trying 00:37:01.407 --> 00:37:05.127 to keep it tight, but you're just selling us a microservice architecture. 00:37:05.527 --> 00:37:05.797 Yeah. 00:37:05.877 --> 00:37:07.117 Oh, I absolutely am. 00:37:07.187 --> 00:37:08.407 I absolutely am. 00:37:08.427 --> 00:37:09.627 It's the same kind of vibe. 00:37:09.677 --> 00:37:11.587 It's just learning lessons from other people. 00:37:11.697 --> 00:37:15.217 Sometimes, not all the time, but sometimes it's easier to have one 00:37:15.217 --> 00:37:18.977 thought in your brain at a time, if you can design it in a way where it's 00:37:18.977 --> 00:37:22.047 clear how you separate those and how they interact with each other and 00:37:22.047 --> 00:37:23.467 you don't run into any other limits. 00:37:23.577 --> 00:37:27.423 Much like in real life, it is all about choosing boundaries. 00:37:27.728 --> 00:37:28.808 Yeah, it's engineering! 00:37:29.318 --> 00:37:30.968 And the other thing is you can iterate faster. 00:37:31.018 --> 00:37:34.338 When you have these big complex circuit board designs, iteration takes a while. 00:37:34.338 --> 00:37:37.028 Like I said, designing the board, updating the board, making any fixes... 00:37:37.508 --> 00:37:40.168 different people have to synchronize on it because you know, 00:37:40.168 --> 00:37:41.368 different people have opinions. 00:37:41.578 --> 00:37:44.728 Versus if you have like relatively siloed functionality, it's like, 00:37:44.978 --> 00:37:48.038 "Okay, we switched to a higher power motor driver," or "Hey, we switched 00:37:48.038 --> 00:37:50.718 to three sensors instead of one sensor," or something like that. 00:37:51.148 --> 00:37:53.638 Because, in the beginning, when you're bringing up, like I was 00:37:53.638 --> 00:37:57.208 doing on my stream, you don't even need embedded Linux to be working. 00:37:57.208 --> 00:37:58.748 You don't need to worry about the image for that. 00:37:58.748 --> 00:38:01.324 You don't worry, you have flashing that, updating it, provisioning it, 00:38:01.344 --> 00:38:03.254 whatever your laptop is the SoC. 00:38:03.504 --> 00:38:06.264 And at least with Rust, you can run Rust on Windows, Mac, and Linux. 00:38:06.694 --> 00:38:08.214 If it's over USB, who cares? 00:38:08.244 --> 00:38:12.914 And if you don't have USB, you buy a little USB to serial adapter or whatever. 00:38:12.914 --> 00:38:15.671 And all of a sudden you're testing not that far from things. 00:38:15.721 --> 00:38:17.971 The first prototypes might be thousands of dollars because you're 00:38:17.971 --> 00:38:19.241 making them in small batches. 00:38:19.521 --> 00:38:21.661 If you have five devs on your team, you go, "I don't really 00:38:21.661 --> 00:38:23.071 want to make 10 boards..." 00:38:23.291 --> 00:38:25.671 and then someone fried one of them because they plugged it in backwards. 00:38:25.721 --> 00:38:29.611 It's way easier to just have this collection of cheap things and everyone 00:38:29.611 --> 00:38:31.021 gets a version that they can plug in. 00:38:31.241 --> 00:38:34.073 You can hack on that quickly because you can go from your embedded 00:38:34.073 --> 00:38:36.793 Linux SoC to a laptop: whatever! 00:38:36.913 --> 00:38:37.263 No big deal. 00:38:37.263 --> 00:38:38.539 I was thinking about the other way around. 00:38:38.539 --> 00:38:40.809 You can also mock the microcontroller. 00:38:40.839 --> 00:38:42.569 Like, it's just a bunch of messages. 00:38:42.609 --> 00:38:42.919 Yeah, yeah. 00:38:44.728 --> 00:38:45.413 We're going to get there. 00:38:46.453 --> 00:38:47.023 We're going to get there. 00:38:47.144 --> 00:38:49.174 You can replace either half as the design evolves. 00:38:49.264 --> 00:38:51.794 You don't necessarily have to worry about breaking your motor 00:38:51.794 --> 00:38:55.024 control circuit because you updated your embedded Linux SoC. 00:38:55.054 --> 00:38:58.174 You can sort of tick tock between the two pieces. 00:38:58.564 --> 00:39:00.964 And like you were saying, you can test either half in isolation. 00:39:01.264 --> 00:39:05.804 If we have our normal system, which is our SoC talking to the microcontroller, if we 00:39:05.804 --> 00:39:07.124 want to test the microcontroller system, 00:39:07.134 --> 00:39:10.234 we can have a hardware test rack that is just exercising this 00:39:10.244 --> 00:39:11.394 with a bunch of sensors on it. 00:39:11.614 --> 00:39:12.944 And there's no embedded Linux. 00:39:12.944 --> 00:39:15.175 It's just CI server that's just chugging on that. 00:39:15.585 --> 00:39:17.655 And if we want to test just our application. 00:39:17.885 --> 00:39:18.935 Like you said, we can mock that. 00:39:18.935 --> 00:39:23.138 Postcard RPC even has the ability to say like, it has a generic interface and 00:39:23.138 --> 00:39:27.028 I have a version that instead of using the USB port, it uses tokio channels. 00:39:27.068 --> 00:39:31.188 So you can just have another task that's pretending to be something like that. 00:39:31.438 --> 00:39:34.798 Poststation actually does that built in where it will simulate devices for you. 00:39:35.028 --> 00:39:36.538 That just like ping and show data. 00:39:36.538 --> 00:39:38.526 So you can test that you can poke them. 00:39:38.819 --> 00:39:40.039 This is the same in software. 00:39:40.199 --> 00:39:43.369 You can buy yourself time with things that work enough. 00:39:43.419 --> 00:39:47.009 Even if this design is too big to be shipping to your customers, you can 00:39:47.009 --> 00:39:49.549 make a hundred units for internal usage. 00:39:49.719 --> 00:39:51.429 You can take them to demos. 00:39:51.439 --> 00:39:55.859 You can take them to user testing where you're buying yourself time where 00:39:55.859 --> 00:39:59.249 you don't have to do the waterfally kind of design, where you have to 00:39:59.249 --> 00:40:03.089 wait until everything's in its final plastics before people can touch it. 00:40:03.519 --> 00:40:06.789 You can start getting that feedback early and iterating on that design. 00:40:07.199 --> 00:40:11.029 And you can buy yourself a lot of development time and unsticking that 00:40:11.039 --> 00:40:15.936 critical path in the design and worry about optimizing for cost or these 00:40:15.936 --> 00:40:17.946 kinds of things after shipping V1, 00:40:18.246 --> 00:40:22.176 because a lot of the customers I work with don't need V2 or they 00:40:22.176 --> 00:40:24.906 have the V1 that they shipped, which is based on this buddy system. 00:40:25.106 --> 00:40:26.436 And they went, "You know, it's fine. 00:40:26.436 --> 00:40:27.026 It's okay. 00:40:27.026 --> 00:40:28.426 It's like 10 cents more a unit. 00:40:28.826 --> 00:40:29.956 A dollar more a unit..." 00:40:29.996 --> 00:40:33.996 usually the way you translate from hardware to retail is: every dollar it 00:40:33.996 --> 00:40:37.706 costs you more to make something, you should be charging three more at retail. 00:40:37.716 --> 00:40:40.106 So if it costs three bucks more, it's going to increase your 00:40:40.106 --> 00:40:42.059 product probably by like 10 bucks. 00:40:42.517 --> 00:40:45.618 But in a lot of these low volume cases or initial prototypes and 00:40:45.618 --> 00:40:49.498 things like that, just having something that works well today is 00:40:49.508 --> 00:40:51.638 worth charging an extra 10 bucks for. 00:40:51.875 --> 00:40:53.315 And I'm going to use a couple terms here. 00:40:53.315 --> 00:40:54.605 There's BOM and NRE. 00:40:54.615 --> 00:40:56.615 So BOM is our bill of material. 00:40:56.835 --> 00:40:58.000 Byte order mark... 00:40:58.160 --> 00:40:58.770 Byte order mark. 00:40:58.790 --> 00:40:59.380 Exactly. 00:40:59.780 --> 00:41:02.460 Our bill of material, which is what it costs us to make everything. 00:41:02.520 --> 00:41:05.580 It's the checklist of every bit that gets assembled into our product. 00:41:05.860 --> 00:41:09.060 So our bill of materials, it's a whole list and what it costs 00:41:09.060 --> 00:41:10.520 us per unit to make that. 00:41:10.930 --> 00:41:14.620 And the NREs, which are non recurring expenses, which is you and me. 00:41:14.690 --> 00:41:18.030 That's engineering, design time, that stuff you pay once. 00:41:18.370 --> 00:41:22.210 You have to include that cost in what you're selling, but it's not a cost you 00:41:22.210 --> 00:41:25.923 have to keep paying when you're going from a thousand units to a million units. 00:41:26.313 --> 00:41:29.403 But it's one of those things where if you're only making 10 units and you have 00:41:29.403 --> 00:41:32.713 to pay 10 developers for a year, each of those units needs to have a whole 00:41:32.713 --> 00:41:35.683 developer salary in the cost of the unit. 00:41:35.733 --> 00:41:40.443 Which is why this industrial stuff or the experimental stuff: you buy one 00:41:40.443 --> 00:41:43.069 unit where you go, it's like a Windows computer in a couple of things. 00:41:43.069 --> 00:41:44.609 Why are you charging me a million for it? 00:41:44.869 --> 00:41:47.639 And it's like, because I only make 10 a year and you need this and I'm the 00:41:47.639 --> 00:41:48.979 only one that makes this kind of thing. 00:41:49.542 --> 00:41:52.937 Yeah, so this is our per unit cost versus our design and dev time, that's 00:41:52.937 --> 00:41:54.197 where that threshold comes from. 00:41:54.477 --> 00:41:58.567 When you have more than 10,000, more than a hundred thousand units, that fraction 00:41:58.577 --> 00:42:00.807 gets divided really significantly. 00:42:01.217 --> 00:42:04.047 And it only adds pennies, dollars to your bill of material. 00:42:04.247 --> 00:42:07.897 When you're making a thousand, if you have one dev for one year and they're 00:42:07.897 --> 00:42:12.027 making 50, a hundred thousand dollars a year, like not even San Francisco 00:42:12.027 --> 00:42:14.027 salaries, you divided that by a thousand. 00:42:14.027 --> 00:42:19.127 That is a non trivial amount of what you have to charge per unit for one developer, 00:42:19.127 --> 00:42:20.821 and you probably have more than a couple. 00:42:20.932 --> 00:42:24.302 The whole goal here is you should be treating your buddy as a partner 00:42:24.672 --> 00:42:26.132 instead of as a black box. 00:42:26.142 --> 00:42:28.522 Cause that's the history of embedded is you go: well, I make my little 00:42:28.522 --> 00:42:31.442 embedded system and then we pretend that it's not running firmware. 00:42:31.452 --> 00:42:34.372 It is just a thing that works and it's a card. 00:42:34.402 --> 00:42:36.622 Whoever writes the driver has to think about it, but then 00:42:36.622 --> 00:42:37.862 we never think about it again. 00:42:38.282 --> 00:42:42.186 My pitch is to treat it more like a partner in the same way on servers where 00:42:42.186 --> 00:42:45.436 you go: look, I'm going to consciously acknowledge these are all computers 00:42:45.446 --> 00:42:46.466 and they can talk to each other. 00:42:46.466 --> 00:42:49.606 And we can treat them not as equals, but as like reasonable 00:42:49.606 --> 00:42:50.856 partners in this whole thing. 00:42:56.066 --> 00:42:59.456 This episode is sponsored by Postation, a tool from OneVariable that makes 00:42:59.456 --> 00:43:02.806 it easy to set up communication between your desktop, laptop, or 00:43:02.876 --> 00:43:06.276 an embedded Linux system to as many connected microcontrollers as you need. 00:43:06.708 --> 00:43:09.458 If you're a company building a product around multiple devices, and would 00:43:09.458 --> 00:43:12.538 like to have all of the plumbing, tooling, and device management handled 00:43:12.538 --> 00:43:17.038 out of the box, send us an email to contact@onevariable.com for early access. 00:43:17.405 --> 00:43:21.385 Check out onevariable.com/poststation for more information. 00:43:21.775 --> 00:43:25.205 That's onevariable.com/poststation.