Episode 774 transcript

FLOSS-774

Jonathan: This is Floss Weekly, episode 774, recorded Wednesday, March 13th. Let's get Rusty.

Hey, this week we sit down with Herbert Wolverson and talk about Rust. He's sort of like one of the Randall Schwartzes of that language. He's written, he teaches, and he sat down with us for an hour and let us pick his brain about Rust. You don't want to miss it, so stay tuned.

Well, hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we have a bit of a different show. It's, it's just me. There's not a co host, and that is because there's this little thing going on this week called Scale.

Scale 21. All of the people that you normally see in the co host slot are doing something with scale, either on the way there, or they're at it, or something maybe unrelated, but everybody for the co host slot was tied up for today, and that means I get to do it, well, not by myself, I have a guest. In fact, I've got, I've got a really neat guest Herbert Wolverson is going to be the guest today.

And I told him in the, you know, in the, in the chat before the show that I was going to refer to him as the Randall Schwartz of the Rust World. For those of you that know, Randall, a wonderful friend of the show is, is known for being the guy that kind of wrote the books on Pearl, and that is one of the things that Herbert has done.

He has written some of the books on Pearl, and he said, well, I'm not sure that I'm quite the guy for that. And so we decided he is One of the Randall Schwartz of the Rust world, and that, that's fine, that, that works but written some books on Rust and doing some teaching on it and something of a Rust evangelist, which I am fairly excited about because Rust, Rust is one of those things that I've been excited about for a long time.

And sort of had it on my to do list for the longest time that I need to actually write some Rust code. And I've sort of always had this feeling, until recently, had this feeling that it's just another programming language. Surely it'll be easy. I, I know C, I know C I do PHP, I've done a little bit of Perl, I do Python, all of these languages.

And so it's, To the point to where most of the time I could just pick up a new language and spend a couple of minutes googling for you know exactly what the syntax is supposed to look like and be on my feet in a way. About a, about a month ago I had a conversation with someone and it was said, well it could be a problem that we were adding Rust to the Linux kernel because you have all these kernel developers that know C and don't know how to read Rust.

And the thing that I immediately thought was, it's just another language, surely you can't, it can't be that hard. And I decided before I would put my foot in my mouth that I really ought to go and check and take a look at some Rust code and make sure that it was as easy to read as I thought. And I pull up the the, the, the sysutils, kind of those, those basic Linux system utils that historically have been GNU projects.

And it's, it's the Rust implementation of them. And I go to look at the Rust code. And the first thing I see is if let sum, like, why is there a let statement inside an if statement and what does sum mean? And I just kind of, okay, it's going to take more time than I thought to learn Rust. I don't have that time right now.

All of that said, let's bring onto the show, the guy that can help. We've got, like I said, it's Herbert Wolverson, and we've got him today and excited to chat about Rust and maybe get some tips as a first time. Herbert, welcome,

Herbert: sir. Ah, thank you for having me.

Jonathan: So let's start with what you're up to. Let's, so what is, what is sort of, how do you fit into this, this brave new world of Rust?

What, what are you up to? And like I said, sort of the one of the Randall Schwartz, unpack that for us. What, what are you, what are you doing?

Herbert: Okay. So right now I have like five jobs because I'm an independent contractor, I'm the Rust series expert for Pragmatic Publishers. The people who.

Published the pragmatic programmer years ago made enough money doing that they became their own publishing house so that means I read other people's manuscripts and if anyone wants to send me one I'd love to and Say whether say yay or nay on publishing it or hey, this is great. You should maybe change this or things like that and That's, you know, that's the job I love.

If I could get paid to sit and read other people's manuscripts all day, I would be the happiest man on earth. My second job is, you know, 15 years ago, I co founded a wireless internet service provider in central Missouri. I'm less connected to the day to day running of the thing now. But we have a remarkably Rust based backend for managing everything.

The third job is I write books. Now it's a weird conflict of interest because technically speaking, I approve my own manuscripts. So we made sure somebody else got to do that. But my third book will actually be coming out into beta in the next couple of months, so advanced hands on rest.

So I'm hard at work on that. And that's another job I really love. My, that's the problem with all these jobs is I love all of them and So I'm going to keep doing them until I keel over. Yes, I understand that. Now, my books did well enough that a fellow named Naguel contacted me out of the blue and said, you know, Hey, would you like to teach Rust?

And so I signed up with Arden Labs who are mostly a Go shop. And they had been getting more and more companies asking them about using Rust as part of their ecosystem. And so they brought me on board to start teaching Rust and started out with a few webinars, turned into full week long classes where I teach you remotely or personally.

via recorded video from Hello World all the way up to building big multi threaded network servers. And you can get there in a week if you put every day into it. It's as long as you've already got to the point that you know, you know what a for loop is, know, know what an if statement does. So that's turned into this crazy job where I, I, I'm at REST Nations UK.

In two weeks time I was at RustConf. I was at GoConf of all things. They had me go to GoConference and give a class on Rust, which was weird, but also kind of exciting. They fly me around to teach people. I was in Silicon Valley a while back teaching a C company that's interested in Rust. So that keeps me Really busy and I also love most of that job.

So Keeping that one too. And finally, there's Libre QoS, which is the real passion project I know you had Dave Todd on here a while back. Dave is the chief science officer of Libre QoS I am the I think I'm technically the chief product officer because I wanted 3PO to be my to be my job title. In practice, I crank out Rasp that, and also some C that handles the task of reading you know, 40, 50, 000 packets a second in kernel space, parsing them, sending them off to the right queues, and then feeding them into this just enormous Rust infrastructure that gives you.

Vision visibility into what your network is doing. And labor, when we started a couple of years ago, labor QOS was the biggest rest project I'd ever tackled. And I'm kind of proud of it because we've not had one customer yet. Tell us that the core system has crashed and

Jonathan: that's that's saying something

Herbert: that's, you know, that I'm going to say that's more the language than it is me because, you know, I'm a, I'm, I'm a decent programmer.

I'm, but I'm certainly not. I'm certainly not amazing, you know, that's That

Jonathan: that sort of strikes me as being the the biggest difference between The Rust language and let's say C or C and I'll put some caveats on this in a moment, but with C the foot guns abound And with Rust, you don't have to be a wizard level programmer to not crash your program because you did something silly with memory management.

Herbert: Yeah, that's, that's about right. And, you know, the history of Rust is, um, Grayson Hoare and his team And a team at Mozilla had been deep, deep, deep in the C world, making Firefox work and other Mozilla products. And so they sort of had a laundry list of what is making my life suck. And it's also why the Rust syntax is a little strange.

They loved OCaml, so they wanted to borrow as much of OCaml and the bits of C that they liked all at once. Mix them together in a pot and out comes the syntax. Yes. So, you know, the core idea of Rust was that you never, ever, ever want to have a buffer ever run. It's where you read past the end or even the beginning of an array, and suddenly you're looking at something you shouldn't.

And it's far better to crash than to crash with an error message telling you what you did than it is to allow that. And then they Sort of went pretty crazy and just started listing off all the things that made their lives suck like race conditions So we'll stop those use after free Well, we won't do that.

And they also looked at other languages that were around at the time, you know, go was appearing java was very popular The whole dotnet infrastructure, but all of those did did do garbage collection And the problem with that is when you're trying to do something Like a browser, you don't want the browser to pause periodically to clean up your memory.

Yes. So you've got to do memory management and C and C are wonderful at that, but they give you all sorts of ways to accidentally shoot yourself. So Rust was basically a collection of let's try and have the compiler fix as much of this as we can and then have Runtime safety that stops us from, you know, accidentally sharing the corporate password database with the world.

Jonathan: So one of the things that made the news recently sort of in my orbit is the U S the executive branch, the white house put out a document, basically encouraging businesses and people to start using memory safe languages for writing important code. And rust was one of the ones I believe got a specific call out.

And there, there's, there's been sort of a, well, with anything that touches politics, as one might imagine, there is a multitude of responses to this. One, one, in fact, it was the fellow Hackaday writer I mentioned earlier made the point, well, hasn't, hasn't Ada had all of this for the longest time? And I think that's kind of a, a more interesting question.

Like Rust is not necessarily the first language to try to do some of these things, is it?

Herbert: No, it's not. Ada, particularly with the Spark extensions, gives you kind of a limited language in some ways, but it's deliberately limited in that, and you check preconditions everywhere. And a lot of bugs become compile time bugs, and that's what you want, but the DoD even adopted issued a mandate years ago that everything was going to be written in Ada, and it didn't, didn't go well because Ada Ada's a, An interesting language.

It's a great language in many ways. At the same time, it's really verbose and you get it, you tend to wind up with these incredible you know, 30 level deep nested statements that look, just look like a nice little diagonal line down your screen. And there wasn't a lot of Ada talent out there to use it.

And so this time around, you know, they listed a whole bunch of languages. They listed Go, they listed Java. Rust was the exception there that it didn't. That it doesn't give you a runtime that manages your memory. And you know, I thought, Hey, that that's encouraging. And I saw that rust recent through a company called oxide recently met the vehicle safety standards as well.

I just blanked on the name of them. I think there's going to be the same sort of headache that Ada had that now you've got to find a whole bunch of rust developers and it's a newish language. So there's not many of them out there. At the same time the thing that really made me happy when I So that announcement was going to reddit slash r slash CPP and looking at the reaction there You've got the and I'm not saying this in a schadenfreude kind of way because you've you've got the old school C coders Which was me five years ago Saying well, we don't have a problem and you know, they're absolutely right that C If you opt in, it gives you a huge amount of safety, like never use the index operator on a vector, always use at, always check the result.

You just got rid of a lot of the problems. And the problem is you have to remember to opt in to all of these. But then, you know Strassrup and Herb Sutter from Microsoft popped into the thread. And we're saying, well, you know, what we really need is C with a profile where we switch all the defaul all the defaults to the exact opposite, and now you've got 90 percent of the safety of Rust, and it's like, yes!

I've been asking for that for 20 years now! Yeah. Yeah, it's it's one thing to give people the option of wearing seatbelts. I prefer you wear a seatbelt by default and can take it off for the scary part where you have to do something dangerous. And that's one of the things I love about BuzzGo and Rust, is the the unsafe tag.

You're not saying I shouldn't do this, You're definitely not shouting, hold my beer, you're labeling. I've done something here that operates at a lower level than the static analyzer can verify. So please accept this as a big warning label that if this program crashes, look here.

Jonathan: Yes, and that that unsafe tag, that's particularly useful for doing things like writing drivers, right?

Is, and I guess I'm curious, is there another scenario that you have to use unsafe outside of we're working with real hardware, therefore we need to talk to this memory address that is uninitialized because it's not actually a memory address, there's a device

Herbert: there. Ed? That's a complicated one, but so unsafe means that for whatever reason I am opting out of most of the Rust checks.

There are still some that you can't avoid, like you can't, you can't have aliases where two references point to the same thing. Item in memory. The Rust compiler just doesn't let you do that. It's the same as strict, enabling strict aliasing on your C compiler. So if you're talking to hardware, you absolutely have to have unsafe at some level.

Because quite literally you're saying this blob, this blob of memory isn't even mine. If you're interacting through FFI with say a C library, your actual boundary calls are going to wind up being marked as unsafe because Rust Isn't going to make any promises that it's going to step into that C library and verify that it's correct.

Then you run into the unsafe that I discourage which I sometimes when I'm teaching call the YOLO unsafe. Because just like C has, you know, you can access a vector entry by index, or you can use dot at to see if it's there. Rust actually has get unchecked. Which is the same as the index operator on C Right down to it doesn't do a bounce check.

And if you call that with the wrong number, you will read out of your vector. And you will, you know, best case, crash your program. Worst case, do something you really didn't want to do. Now in Rust, that's unsafe. But, if you're writing something that's super performance critical, maybe you don't want to do a million bounce checks because you're blasting data into a vector.

from some hardware source. So that unsafe tag there is you saying that, you know, yes, this is on me, I've done the checks. And another programmer who comes along may look at it suspiciously. And if you the reason I discourage it is that please, you know, start with a profiler and find out if that, that is actually why your code is slow because the compiler LLVM is absolutely amazing at realizing I've got a million bounds checks here but I'm going to have one upper bound.

So let's check that first. It doesn't always pull it off, but compilers are way smarter than me. Yeah, to give you a real world example, some code I was working on last night. I've got a, I'm reading buffered data coming in from an eBPF driver running in kernel. And I've got a hard time limit on how, how long it can take to read this.

And I was missing my time limit, and I was all frustrated, and I'm like, Okay, well, here comes the unsafe. So I unsafe, start you know, get, accessing the data directly, and You know, I didn't blow anything up, because I did, actually did the math. But the speed improved by 0. 01%. And then I noticed that I had a mutex lock in the wrong place and hoisted the mutex lock out of the loop and instead of calling 18, 000 mutex locks per second, I'm now calling one and all of a sudden I'm at point back to point one seconds and everything's happy.

He's happy. And so it's like, you know, run the profiler. It's it it's gonna tell you it usually tells you something that you missed

Jonathan: Yeah, so i'm i'm trying to remember the exact quote, but somebody somebody said something like the root of all evil is Trying to optimize optimization before profiling or something like that premature optimization.

Herbert: Yes. Yes. I think that I think that's attributed to Donald Nutt But I'm not not a hundred percent

sure

Jonathan: there that is that is possible It was it was one of the one of the old timers one of the one of the greats So, let's see. Let's maybe let's talk about rust in the kernel because that's something that's definitely going on My problem with this is I've got so many different things that are rust adjacent, but I want to ask you about that So we may just kind of bounce around from topic to topic for a while and then slow down.

But let's do that. Let's talk about RESTing the kernel. This is obviously something you're watching, I'm sure.

Herbert: Oh, yes. And I'm actually happy to report that my first kernel module that prints hello world to a log works. It's not a great kernel module, but that's as far as I, that's as far as I got with, hey, how do I do this?

I was Pretty excited to see it though. Partly because the Linux kernel has been one language for so long. For years and years, I was following the efforts to get C into the Linux kernel. And I kind of agreed with last week that the problem with C is that it gives the, it tends to allocate behind your back.

It's so easy to use types that allocate memory and don't, and sometimes have side effects that you weren't quite expecting. And I honestly think that could be fixed, but it didn't seem like the C standards committee was agile enough to make that happen. And so quite a few years ago long before Rust was officially in the kernel, I ran into a few people who were.

Hacking away at rust until they could get a kernel module to load. Anyway, and they were they could do that because your rust outputs see ABI code that is effectively indistinguishable from C from the compiled binary point of view and so they hacked at it until it worked and finally started talking actually to the People who maintain Linux and eventually word got up to Linus and his comment was Hey, go for it.

Give it a try come back to me in a few months and We'll see what happened with your first try and everybody was like, wow, he didn't just say no and set the room on fire So there was a lot of effort for that and it's been a really positive experience Bidirectional process too with once you get into the Linux kernel, a lot, some of the Rust safety assumptions don't necessarily apply because you're dealing with a whole lot of things that happen in kernel land that you don't necessarily want to do in user land, especially with modern hardware being so incredibly complicated.

And so part of that was when, when they ran into some Rust does something that shouldn't be done in the kernel. The Rust. Core team have been really responsive in terms of changing Rust to meet, to match Linux. You know, a good example of that is that you've always been able to compile without the standard library.

You can now compile with no, with no memory allocator at all if you need to, which is what you want in a driver because you have to use the kernel allocator. And you can substitute. The memory allocator. So building, building those, just those bindings. And I think there was some compiler changes required that took months, but they did the work and now they've finally got to the point that I think it's 6.

8 is shipping with network driver written in Rust. And there's the full sort, they actually took one of the existing ones, ported it, it's the first time Linux has had duplicate drivers for something in there. It's one of the simple ones, and it's there partly as a proof that it can work, and then there's the crazy people over at I think it's Asahai Linux, I'm not sure if I just butchered the pronunciation.

I think they were pronounced that Asahi. That would make sense. I can say I've talked to them, but only ever by text. So they took the Mac, Mac and one chip output. And yeah, I, I really wish I had this much energy and enthusiasm, but they managed to feed that into. Tons and tons of Rust and get to the point that it boots.

You have X11 working OpenGL sound. I think they were having troubles with the webcam last I checked in. And they were making a lot of comments that they could have done this in C, but the Rust type system Stopped themselves, them from tying themselves in knots, and you know, I saw, I was snooping on a lot of the chat that went into how that was being done, and had a look at some of the code they've written, and some of it honestly looks like C.

The lower level in Rust you get, the more C like it starts to look. You start seeing pointers everywhere, pointer math everywhere, and the syntax is similar. Not quite the same. And you do always have to differentiate between whether a pointer is mutable or not. So is it read only or can you write to it, which honestly, I, I'm a big fan of the const tag in C in C anyway, to remind me that I shouldn't be changing this but it's, it's a remarkable.

Effort and it's turned into this big Rust on you know, Rust for Linux library that comes complete with instructions for how to bootstrap your own kernel module, compile it, it's becoming something that's pretty approachable. And I think that's can only be good long term because there's a growing pool of Rust programmers.

The People I know, the companies I've spoken to who hire C and C programmers, there's plenty of them around, but there's not very many of the really high level, good ones. And so I think allowing the wider pool is just going to be good for everyone. At the same time, I, I fully expect at some point.

One of the joking discussions you see on the kernel mailing lists where well, we just can't get this function to work and somebody says, well, we could write it in Rust and lots of people groan, and then they fix the problem. One of these days, somebody is going to rewrite something core and Rust, and then the real fight begins because right now Rust is in the place where you have to be able to build the kernel without it.

It's, it's at the edges. So it's at the edges? Yes. And if it ever makes it further into the core, then that's going to be a headache for the People who have been doing C for 30 years. But hopefully not because hopefully we can keep the boundaries well defined. And so at least you can interact with the Rust even if you're not changing it.

But we'll just, I think it's going to be an ongoing evolution where we have to just see where it goes.

Jonathan: Yeah, there's interesting things going on in the kernel right now. So as you were talking about that and you mentioned C I remembered back a couple of months ago now, there was a there was a a message sent in to the kernel, basically saying, let's do it in C Let's convert the kernel to C And the thing that was wild about it is C has come along to the point to where some of those problems have been either fixed or there's ways to work around them and the person that wrote this that wrote this message did not event did not immediately get set on fire so Personally, I think it might be kind of nice to be able to get some of kernel.

I don't know if it'll ever happen. But it is it's an interesting kind of new world that we're going into when the kernel is not just C code.

Herbert: Yeah, it's fascinating when you look in the kernel. Like to me the big C invention that everyone should Lord, forever is RAII. Resource acquisition is initialization.

Mm-Hmm. , which is a long-winded way of saying that you have a con you know that you have a destruct. So when this structure falls outta scope, something happens. It's not just gone. Mm-Hmm. . And that is amazing because then you can tie that distractor to cleaning up. Mm-Hmm. . And so if your bit of code is talking to a USB microphone and that USB microphone's disconnected, if the driver goes out of scope because it unloaded now it has a chance to run some cleanup call rust adopted that wholeheartedly. Much of you know, much of the easy memory management that you do in rust because honestly you can write thousands of lines of user user mode rust without ever touching ever touching an allocator Uses that wholesale It's right down to the, it's the same compiler mechanisms.

But if you dive into the Linux kernel, you'll find that there's a whole bunch of macros in there that basically added that into C to ensure that when something goes out of scope, it is destroyed because a lot of the time, you know, ownership is straightforward. I made this variable. It's my, my responsibility to clean it up.

And you'll see the lovely go to patent and see where something went wrong. So I'm going to jump to the end and clean up. And that works. RAII gives you the same effect because you free it up, the cleanup code runs, it's still deterministic. The problem comes when ownership starts to get fuzzy, like this This piece of data is now going to be used by five other things and it's unclear which of those five other things is going to finish first, finish last.

And I want to clean it up when none of them are using it anymore, but I don't want to clean it up before that because that's going to really mess up the day of one of those, of whichever ones are still going. And so tying, tying lifetime events like that at a language structure, you see C and Rust are both great at it.

I think Swift uses it. Garbage collected languages struggle with that because when the garbage collector will get around to deleting it is kind of indeterminate. But when you can do it, it's great. And, you know, it's lovely to see that there are structures in Linux for doing just that. And they've got this, you know, they've had the same problems we've had.

And they've converged on a kernel side approach that is basically the same thing. So I don't see there's anywhere near as much of a mismatch as there used to be. You just have to be really careful, like. You'll get some wonderful email if you start allocating in current regularly in kernel mode.

Jonathan: Yes, yes, that would not go over well. So one of the interesting things to me as well, Rust is, ah, there's several things here. So Rust uses what, generally speaking, the, the LLVM compiler.

Herbert: LLVM, although there is a GCC port that works. So that's

Jonathan: what I was going to ask about. What's the process or the progress on the GCC port?

And why do we need both?

Herbert: So, the progress is it just got published. Pulled into rust app, which is the tool that installs linux as an available backend the vast majority of stuff now works. There are still corner cases being ironed out but it wasn't too bad because they took the approach of when you compile rust rust c emits an intermediary That then gets read by llvm and so this one emits and then the intermediary that gcc likes and so You You just had to make sure that everything that was needed was there and I think gcc actually added a couple of facilities Also helped c because it uses the same thing.

So why do you need it ld? Llvm targets a lot of platforms, but not all of them whereas gcc Seriously targets everything. I mean, you can probably get GCC for your toaster. And while rust Is never going to work on 8 bit platforms according i'm told by the kernel team by sorry by the core team it'll work on 16 bit platforms you can Produce tiny binaries just like you can with C.

So it's a win there because once you've got GCC there's a well defined way to import your existing libc for a platform into rust and use that if you need to. And so most platforms out there ship with most of a libc and a GCC compiler. So now you've got the ability to bootstrap a whole bunch more platforms.

And,

Jonathan: and rust is low level enough that you can actually use it for embedded development, like on a little tiny device where the rust binary is. It is essentially the entire operating system, right?

Herbert: That's right. You can, there's operating systems written in Rust from the ground up, which on the embedded side you know, you, you enable what's called no STD to no standard library mode.

So all of a sudden you're missing some of the nice, the nice things. Okay. But you've got a very C like. Language with the Rust constructs that make people happy. Like some types, some types really strong typing and so on on top of that. And some, you know, some platforms also implement parts of the core.

So, you know, on some embedded devices, you can't, you don't even get to allocate on the heap because there isn't one on some embedded devices as a heap. So you can start using heap allocation, but Rust can go all the way down to I think this I, the smallest I've seen in person, if you remember those little pie badges that Adafruit used to sell, they're a tiny little liquid crystal display with a minuscule CPU on the back for an Arduino.

I've seen one of those in TinyGo. I've seen one of those in Rust. They all work. Yeah, it's cool. It's pretty cool. I personally have been going big more than small lately, but I'm, it's on my bucket list of things to try and fit in before all the jobs catch up with me. Yup.

Jonathan: Yup. So let's see, while we're on that topic let me ask you about, well, about cargo.

So this is, I guess this is a little bit different topic, but one of the, one of the things about Rust that sort of drives me crazy, and maybe I just don't understand the way it works is it, it reminds me of. It reminds me of JavaScript in that there are, there are all of these little tiny either JavaScript or maybe Java itself, all of these little tiny libraries that you grab at build time.

And, you know, coming from the C world, I'm, I'm much more used to, you've got, A handful of system libraries and it's it's very different and I'm not sure I'm not sure yet what I think about that and and one of the things I'm curious about is does rust have the problem that Java does that if one of those libraries has like a security vulnerability?

You can't update just the library on the system. You've got to rebuild the whole, the whole program.

Herbert: Okay. So first of all, I just want to say that I love cargo program because coming from C where I need I need either a make file or a C make to make a make file, right? I need a separate library for unit testing, another library of benchmarking.

All of that's in Cargo and then it's also got, you know, if you, if you like dependencies in C you need something like VC package, Conan, or all of those. That's built into, and Cargo is extensible. And so it's kind of got to the point that it's the Swiss army knife that does everything, it builds your program, it runs unit tests, it manages dependencies.

So to your question of lots of libraries. It is a difficult, it is a difficult one and when I first started using rust coming from c plus plus, I honestly looked at the build list and I was like, WTF how can I possibly trust that, you know, all of this other stuff. Is in some way good. Yeah, and then you I look at some of the games I like to write because I love writing games in c and I'm relying on glfw and everything else and You go in and look and they're actually depending on a lot of things too.

Yeah, just the c Approach often involves a directory where you've bundled some bundled the things you're depending on in so you at least know you've Got a set version like, you know you're using Zlib. So somebody's probably going to compile Zlib into your system at some point, or dynamically link to it.

Yeah. I was also a little shocked how big my executable was, but then I realized it was all statically linked by default. And you can, you can change that if you want to. So I went. Around as talking to people who do REST and working with it. And what I learned was that there's really a handful of crates that everybody depends upon.

And those are things like SERTI serialization and deserialization. I b I was CERD for years, and one of the authors came up to me at RESTconf and told me never to say that again, so now it's CERD y forever and other libraries like that, and also a lot of them make it, make their way into the standard library eventually.

But only once they're battle tested and in use. And so you can, so, you know, biggest thing I'm going to say against crates IO here is, you know, please, please, please give me some namespacing. Cause right now you can only have one crate. I named Herbert, you know, for example, so if somebody else wants to write Herbert, then that name is taken.

And so you end up with this big blob of not very searchable. You wind up searching by description. I'm told that's being fixed. I don't know a timeline for that. But in terms of figuring out what libraries to use I personally recommend going to. The rest playground it always has the 10 most used available and make sure you start with the ones that are popular and in use.

Then it make sure you install a program called cargo audit, which will download the current CVE database and check every one of your dependencies and transitive dependencies for current CVEs and can be built into your compile chain. So you can't, your CVE that gives you a lot of peace at night.

I usually put that in my code. You know, continuous integration pipeline. So I get nasty grams. And you can do, you can at any time type cargo update and all of your crates will jump up to the most recent version that you've allowed within, within your version number. You can also pin to specific versions.

And for shipping code, there's a built in command called vendor that actually downloads the source code for every single dependency you have. And then builds locally from those, never goes out to fetch them again. And so that's what you want to do if you're shipping something and you always want to have a repeatable build, because that way you've, you've guaranteed that it's, that you're not going to get the left pad problem of somebody yanking something and suddenly nothing works anymore because you've got your copy.

You're working on it. And part of it relies on Goodwill. The NPM problem is that. They didn't really think of the security problem until they had a million crate, a million packages and but there's also a philosophical thing. You know, I personally do not recommend getting down to the granularity where you download a crate for left pad.

Yeah, especially when it's built in try to, you know, just personally, I encourage people try to download. A crate that is big enough to justify being a crate. If it's three lines of code, just write three lines of code. Or go and cut and paste it if you have to If it's something like a massively optimized ace a star graft reversal Then yes, by all means go and use the one that's really good so I hope that kind of answers the question.

It is it is a culture shock. Yeah

Jonathan: It had have there been any problems with like typo squatting or dependency confusion?

Herbert: I haven't run into typo squatting. I have run into You I have no idea how it happened, but the maker of the mmap crate decided to make another one and called it mmap2. And the syntax is the same, but the number of times I've done cargo add mmap and I meant mmap2 is So it, if anything they've gone too far Down the it's hard to remove the old ones.

Jonathan: Hmm makes sense so I did when I when I threw the the fact that I was having this interview when I threw it out To kind of the people that I I interact with one of the questions that I got is what do you think about rust? This is from aj mcculkin. What do you think about rust as a high level language?

And aj says that his view is rust is particularly good when replacing c But he's not seeing consensus around whether it should try to replace languages like p Java and JS. I'll let Python into that.

Herbert: So you can actually write a fully functional web server that returns a JSON hello world in 21 lines of code with Rust.

So it's a very capable high level language. The question more become, isn't so much, should I, as or isn't, sorry, isn't so much, could I, as should I, because if you're, if you're So if you're writing you know, a blog and you just need to return JSON or whatever to the front end for your blog posts, you can do that in any language.

So pick the one you like. For structuring Rust programs, you know, there's usually going to be some nasty low level code somewhere that kind of justifies you using Rust, but you're very much encouraged to wrap that in a nice, safe, ergonomic interface so that the person who sits down to write a program doesn't actually have to worry about.

All of the really low level stuff I mean, one of the classes I just recorded was 18 hours of how to use how to use Rust to write REST, gRPC services, WebSocket servers, and similar, and it is really ergonomic. It's really nice on the other hand, you know, a lot of that, you know, If you don't have the low level latency requirements, you might as well use Go or Java or whatever currently fits your company.

So it's a matter of it's a matter of deciding I need Rust because I either want a really tiny binary. Like I did a webinar a few months back where I built up a little bookstore web service, Stuck it into Docker pound, pounded it with 5, 000 requests a second and watched it. It's still using 22 megs of RAM.

So sometimes you need that and Rust is great for that. And sometimes you really don't need that. Use, you know, maybe you want to use something that's more, I don't know, more comfortable to, to the domain problem. I'm a big believer in use the tool that fits, fits your problem.

Jonathan: Is there, is there a is there a mechanism to use Rust as like a scripting language?

Can you, can you write Rust scripts and then just have, you know, your your, your, your, your shebang at the top that says, hey, this is Rust, by the way? Thank you.

Herbert: Not a good one. There are plenty of toys that do it, but I've yet to see one that I would trust for anything important. Rust is very much compiled language.

And so at least one of them I saw quietly ran the compiler in the background and then ran the binary. And that's a little pointless.

Jonathan: Yeah, one of the, one of the things that I think it's the Dart language that, that Dart does that I thought was really interesting is they have a full fledged compiler.

And a full fledged, let's run it live as a scripting language. And the people that do Dart kind of had this guarantee that these two versions of the language are going to march on together and they're going to work exactly the same way. That was a little mind blowing when I, when I wrapped my head around that.

Herbert: And that is really cool. I think Rust is badly suited to that because so much of Rust's benefit comes from the static analysis that happens during compile. Yes. So the scripting versions just going to wind up being. Really slow because the it's gonna have to look at the whole thing to figure out how you start yourself in the

Jonathan: foot Yeah, yeah, and and the reason you know, the reason Dart did that is because they want well They're it's it's sort of built for doing mobile application Not just mobile applications But for user interface sort of applications and they want people to be able to write code real time and immediately see the update On on you know, they're they're gooey So it makes sense for that project Speaking of foot guns Something that I've kind of noticed over the years is a lot of languages have a, I don't know if I would call it a weakness, but A typical problem?

So, for example, when we talk about C, you get lots and lots of memory management errors, buffer overflows, that sort of thing. When you talk about Java, well, you're not getting memory management problems, but when it comes to security, one of the things you see often in Java are deserialization bugs.

Because it's just kind of endemic in the way java works, that unless you're extremely careful when you go to deserialize an object, it's easy to overwrite one of those important functions and suddenly arbitrary code. Does, does Rust have one of those endemic weaknesses somewhere? Or are we still waiting to find out if it does?

Herbert: Sort of. The first one is that the learning curve of Rust is much more vertical compared to most other languages. And so everybody who first comes to Rust, especially if they come from C, write C like Rust and then spend two weeks complaining about the borrow checker. Because a lot of, a lot of, Stuff that works in other languages, but is potentially not the safest is just going to refuse to compile.

But that digs all the way down that there are some, there are a tiny subset of programs that you can write and see that Rust wouldn't compile without dropping in some inline assembly, at which point all bets are off. The so okay, just occasionally you run into that and you can. Sorry. Big thumbs up, just appeared on my screen.

I think I, I think that was my Mac noticing my hand movement. . The, uh, so just occasionally you do run into that and, Mm-Hmm. you, because Rust puts so much emphasis on ownership, the idea that this is my pointer, it's my problem. It will live a certain amount of time, at some point you can kill it, and I will be the one to kill it.

Or I put it in a smart pointer and it will, you know, cease to exist at some time in the future. What you've, what you run into is newer Rust programmers realizing, Oh my goodness, there's this thing called arc that is the same as shared pointer in C it is a reference count. Stored atomically, which, what the R stands for, Atomic Reference Count.

And hey, this basically turns off a lot of the borrow checker. Because now the lifetime is indeterminate, it's going to go away on its own. I've got garbage collection. And the problem with that is once you start doing that everywhere, and now you're having atomic increments and decrements everywhere your program's a whole lot less understandable because you don't really know exactly when that's going to go away.

And so I've seen newer programmers to rest, tie themselves in knots because they want to not solve the problem. Not solve the bar checker issue. And just go for what looks like easy mode and C plus plus had the same problem back in 2013, I think it was when shared pointer appeared. I started seeing it absolutely everywhere, even in places where there was no sharing.

Because, because it feels like easy mode and neither language is that forgiving when it comes to, Hey, I want it, I want it. I want easy mode, you know, it's in both cases, it'll probably work in both cases, you're losing quite a bit of performance. And the other one I run into is. Just a hilarious side effect in a way, because Rust enumerations are a sum type, so only one of the possible variants actually exists at any given time.

So, for example, when you return an, when you return that example you were talking about where you said sum, That's what's called an option type, which is Rust's attempt to not have nulls. So it's a sum type that it's either equal to none or it's equal to sum and a value. And what you are doing with if let is a pattern match.

You're saying, if it matches the pattern of sum and something inside, please just give me what's inside. So the syntax for it's kind of messy, but it's super powered and then it collides headlong with desperate attempts to work with systems that aren't Rust. So I was building a demo program on error handling and the C standard library can give you literally thousands of potential error, error codes.

And somebody had gamely sat down and typed every one of them into this enormous enumeration of doom. And so. I, I call out to a C library, it goes through this translator, comes back, and I've got this type that's either going to be okay in my value, or an error. And it's like, okay, well, having an this is too important for me to just say, I've got an error, I'm going to bail.

Some of these errors I might want to handle. Right. So I type match you know, error type, sorry, error. kind, and allow my IDE to autocomplete and it entered 16, 000 possible line, possible errors that I might have to handle. And it's like, okay, well, that, that's not what I had in mind, but I'm going to spend a while because two of these were actually what I care about.

And so there, there's You know, Rust Rust prog the Rust core team and people who've been around it a long time tend to get very safety conscious. You know, you say safety way too much and take off everyone on Twitter by saying that wouldn't have happened in Rust. It might have done, you know, it's So you've got the weird tendency that PRs appear on GitHub in your repos saying that in some obscure circumstance, this function may not work.

And so Rust's biggest problem in some ways is that we really need to stop taking people off and be nice.

Jonathan: That's fair. That's fair. Let's see. One of the things we haven't talked much about is, is the Rust kind of concurrency guarantees. That's another one of those things that. tie C programmers up in knots. Concurrency can be difficult to work with. What's, what, what does that landscape look like in Rust?

Herbert: This was actually one of the things that, when I first started using Rust, I had a eureka moment, shouted, this is amazing, and realized I was going to keep using Rust. You can the compiler makes a very bold guarantee of you will not have data races. And there's actually now two languages, Swift, Swift has just offered the same guarantee.

And in both, both cases, it works by tracking whether or not ever any structure is safe to, A, send across threads. So you might have something that's pinned to a core, so it's not safe to move it to another core. And you might, but also whether or not it's what's called sync, which synchronized means I can safely read this value from another thread.

Right. And so what will happen is you use you create, so the old, you know, C demo where you create a variable called counter, spawn off three threads that count to a million and add one to counter, and then when the threads are all finished, you print out the result, you get a different number every time.

In Rust, that will not compile, and it will not compile with the error message that you're trying to mutably access a variable more than once, and that's the That's the underlying rule, is that only one thing at a time can ever have mutable access to a variable or memory location. And so, obviously, sometimes you need to do that.

So you implement atomics or mutexes. Now, if you're used to C or you're, you've probably used pthread create mutex, and now you've got a mutex variable, you have to remember to lock that. And you have to remember to unlock that. While C next level up, you make a mutex with a lock guard. As soon as it falls out of scope, RAII kicks in, the mutex is freed.

But you still have to remember to lock that. While Rust forces you to put the mutex around what you're protecting, it can either be on the outside, so a mutex protecting a whole structure, or a structure with mutexes, atomics, and safe structures inside, so you can access them independently. And that ties into the sync system.

You can now read that Mutex completely safely across threads because Mutex is a design for that. If you lock the Mutex, the Mutex then gives you safe access to what it contains and ensures that you don't get the concurrent rights to it. And so you can't forget to lock it because the way you Mutex protect an integer is Mutex and then the angle brackets and your integer type inside so that Mutex.

Contains the data, it's a wrapped template. And so, there is no sane way without dropping into really horrific, unsafe code to get to the contents of that mutex without locking it. So now you can't forget to lock it. You've got RAII, the lock goes away as soon as it falls out of scope. You can still deadlock.

So if you write mutex, if you write, you know, mutex. lock, mutex. lock twice next to each other, your program just stopped. Don't do that. But it means you can have a really big code base with lots of Atomic or mutex protected or read write lock and so on, you know, there's lots of lots of locks.

And the compiler will statically verify before it even finishes compiling that there is nowhere in your code that you have concurrently accessed this variable because you have these sync barriers are always in place and not having them as a compiler and you to give you a good example, you know, the Libre QoS monitor system is like 150, 000 lines of code.

Last I looked we have probably, well, we have one thread per core, plus about five more running at any given time. And there's all sorts, there's all manner of shared state. And we have in development, we had one deadlock because I did something dumb. But we've never had a race condition. And so I can attest from, you know, battle testing that you will not get race conditions in say for us.

Okay.

Jonathan: Yeah, that's, that's cool. Okay, so let's say that we have sold somebody on the idea of Rust, and actually, it sounds like various things are coming together that if someone wants a good programming job, one of the, one of the good languages to invest some time into learning would be Rust. Where, where do we start?

Where's a good place for someone to start learning Rust?

Herbert: Okay, there's lots of different learning styles. My personal one is, I pick a new language and I try and write a game in it. Okay. I like writing roguelike games, you know, the old NetHack style where the, where the little at moves around and kills the little g's.

That's actually how I got started was I took a seven day roguelike challenge. And decided to do it in Rust and made a really simple roguelike game that was some of the worst Rust I've ever written, but it worked. Then I decided to do it right, and that turned into an 80 chapter tutorial, the Rust roguelike tutorial that Is available free and that's a that is a great way if you already know some other languages If you're more of the study type go get the rust programming language by klapnick et al.

It's available free online or You can buy it if you want to support them if you like learning through games, I mean this Obviously, I'm going to plug Hands on Rust, my book, because you start at Hello World and that turns into Flappy, Flappy Dragon, which is Flappy Bird, and then a giant roguelike walks you through all of the basic, basics of the language and you have fun learning it.

If you're more into writing cool systems, Tim McNamara's Rust in Action is an excellent place to start. He, it's basically a book of cool little programs written in Rust that here's the program and here's how it works. So there's lots of different ways in, but I'd also encourage you, you know, find a, find a cool project and just pitch in.

And go on to the Rust discord. The official Rust discord is so ridiculously full of helpful people, it is almost intimidating. But I remember the first time I went in there with, with a really dumb newbie question. Nobody made me feel stupid. They were bending over backwards to help me. And I even woke up to PRs in my GitHub repo the next morning with suggestions.

And it is like that. It's, I mean, It'll be toxic if you come in saying, well, Rust sucks. But they're, they're really, really, really nice to people who genuinely want to learn. Also, you know, go out and look at some of the existing Rust stuff on GitHub. Often you'll stare at it. And think the syntax is alien and weird.

I've had that experience. Yes. Then you write some rust and then you come back to it. And once you get past the fact that if let is horrific syntax, honestly it should be called a one arm match. Cause It is a match statement, which is a pattern matching statement that can only match one thing.

And once you get, you know, once you get past a lot of that, you realize it is, hey, I know this, you know, because it is a lot, like a lot of other languages, the, but the syntax does make the learning curve a little steeper. Also chat GPT writes really, really bad Rust but it is fantastic for, here's a C function and here's a Rust function.

Are these equivalent? It does that amazingly well, right down, right down to telling you that on this line in the C and this line in the Rust, you're doing this non obvious thing different. Oh, interesting.

Jonathan: One of the, one of the few places where chat GPT is actually genuinely useful. I have opinions about that, we don't have to get into that.

Herbert: But what I don't recommend is, hey, chatGPT, I want to learn Rust.

Jonathan: Write me section one of chapter one of a book of how to learn Rust. Yeah, that might not go so

Herbert: well. Yeah, probably not. Although I think I've read that manuscript.

Jonathan: There you go. Oh, is that something, this is not really Rust related, but I'm just curious, is that something you guys have run into as a book publisher?

Have people tried sending you manuscripts written by LLM?

Herbert: So, Less Us, because Prague Prague is kind of a small boutique. Niche enough, yeah. I don't think any of them have made it past the first editor you talked to. But I know from talking to other publishers that the answer is, oh god, yes.

And you can, there's nothing wrong with having an LLM help you. Like, here's some text turn this into, You know, a 10th grade reading style or can you help me with this sentence? It's great for that but here's the outline. Write my introduction. Don't do that, please. Editors are getting really fed up with that.

Jonathan: Well, I mean, one of the other places we see it is people are starting to try it. People are coming after bug bounties with bugs, bugs discovered using LLMs. And I guess there might be something to be said for trying to find something that way, but only do that if you know enough to be able to actually confirm yourself whether it's a bug.

Because open source projects, I know the people that run them are getting sick of exactly the same thing. They're getting bug reports. Pay me a bug bounty for this bug that LLM found. No, it's not a bug. You don't understand what this C code

Herbert: does. You know, a funny unrelated one was while I was working on a chapter of my book, I forgot that I had turned on GitHub copilot to because I was curious to see what it did.

So I'm typing away and it suggests the next two paragraphs and I was like, no.

Jonathan: Yeah, thanks. Thanks Microsoft. That's yeah, that's real helpful.

Herbert: And there are, there are times it's good. It's good for I've written something that I now have to repeat in a similar vein 25 times. It's cheaper than asking, asking a junior programmer to fill that, fill out the blanks for you. On the other hand I've seen it suggest some really special code too.

So it's not a tool for, I don't know what I'm doing. It's a tool for, I do know what I'm doing, but I want a better autocomplete.

Jonathan: Yes. Yes. I think personally, I think LLM works well for me. for sort of like an autocomplete, maybe a glorified spell checker. It also seems to work okay if you're using it as a search engine.

And what I mean by that is you have to think of it the same way. If you type, you know, you type something into Google and you say, and how do you do this? And Google is going to give you several different websites where you can look for information. You can kind of do the same thing with, with chat GPT.

And rather than giving you a website, it'll give you kind of this condensed version of it. But as with a search engine, you can't trust, you can't trust it just because it's on the internet. You can't trust it just because chat GPT said it.

Herbert: I found it useful the other day. Cause I went a complete blank on I think they're pronounced chloropleth maps.

I, I knew I wanted a map where the country has changed color based on some data. And. So I'm trying to start type that into Google and I'm not getting very far because lots of things so I just asked cat GPT What's the what's the formal name of a map that changes color based on the value of some data associated with each country?

And it gave me the answer right away. Yeah, so it's great for that and I I was having a real senior moment, so chat GPT made me, made me feel better. So, yes,

Jonathan: I have those all the time and I've not started using chat GPT for that. Maybe I should tip of the tongue syndrome is I think one of the technical names for that.

All right. So we've covered a lot. We have a couple of minutes left. Is there anything that you really wanted to talk about that we didn't touch on?

Herbert: Obviously I'm going to say advanced hands on Rust is coming out in beta in the next couple of months. I am really excited about that. I'm going to be at rest nations UK.

So if any of you are in London I would love to say hello. I'm also going to be at rest conf in Montreal later in this year. Although exactly the exact details of that haven't been announced. So I, I don't know exactly when, but I know I'm going to be there. And so if anyone wants to say hi or talk about any of this, I'd love to hear from you.

And also the LibreQ OS guys wanted. me to thank you for having me, and the Arden Labs people wanted me to thank you for having me, and I want to thank you for having

Jonathan: me. It's been, it's been a lot of fun. So the advanced hands on Rust, is that also going to be kind of a let's build a game together?

It is

Herbert: indeed. The first section you take your basic Flappy Dragon, and you learn to make a library. And then fill it with generics and all sorts of cool stuff, while you add smooth animations, a really robust physics system that doesn't need to be that robust for Flappy Dragon. Parallax scrolling backgrounds, all sorts of cool stuff.

And then the second half of that book, you take your library, discover that you can now write a game and play it. 20 lines of code and start adding more stuff to allow you to learn about threading, async code, and some network things for high score tables. So it's, it's a continuation of learn the Rust language.

It picks up where the previous one finished.

Jonathan: All right. Where, where is the place to go to, to look for the hands on Rust book and when the second one comes out?

Herbert: All right, the publisher is pragprog. com. P R A G P R O G. com. They're also on the first two I wrote are on Amazon. Advanced Tens on Rust will be, but not until it finishes beta.

Because pragprog do a wonderful pre release system where you can buy, where you can buy the beta, you get every update as they come out, and a direct line to tell me what I did wrong.

Jonathan: I'm sure that's handy. That must be really

Herbert: helpful. It's actually a lot more helpful than you might think.

Jonathan: Well, I know, I've written a bit of technical stuff.

I understand how that would be nice. Maybe I should try to do that with my Hackaday

Herbert: articles. The first one, Hands on Rust, went out on beta. A couple of days later, I had a message from somebody, I'm not going to name, but they're on the Rust core team, telling me Your function on page 12 is a fantastic example of thinking around the problem, but here's the one line version.

Yes,

Jonathan: yes. That's great. Did you include both in the final work?

Herbert: I actually did because I used one to explain the other, so.

Jonathan: Yeah, I was going to say, that seems like a perfect example to include the more verbose version for the teaching tool, and then, oh, by the way, this is how you would actually do it.

Herbert: You know, and then you get, you get strange ones like I had a bug report at one point that I had a word in there that didn't make any sense in Ukrainian and, and I, I'm happy to change that, but I had not even considered that people were running this through a translator. Right.

Jonathan: Right. Yeah. That's always fun with bug reports when just generally speaking, when it's not, when English is not someone's first language, when you don't have a shared first language with someone, it could be, it could be a challenge in itself.

Yeah.

Herbert: Yeah. Actually, the funniest one was also that I, I was trying to demonstrate using Unicode to demonstrate and some of the pitfalls of that in Rust brain teasers. And I used an example with a word in Russian and got a message saying, you know, from, actually from another Ukrainian saying, you know, hey, the Russians really aren't very nice.

And. I was like, okay, I can understand that because, you know, they did, they did just invade, invade Crimea and Donetsk all that at the time, about the time that came out. And so I was like, okay, I'll change it. So I set out on a quest to find a language belonging to a country that doesn't have a hysterical beef with another country.

Good luck. I eventually settled on Iceland, just because they haven't had one in a few hundred years. Yeah.

Jonathan: Oh, I have no further comment on that. Not on the record at least. All right, so I've got to ask you two final questions before I let you go. I will get emails if I do not. And those are, what is your favorite scripting language? And what text editor do you spend all day in?

Herbert: Okay, I'm going to do this in reverse order because text editor if I'm teaching in front of a big audience, I'm in Visual Studio Code.

And the reason for that is that it is everywhere. And no matter what skill level you're at, you're comfortable. When I'm sitting at home working on something for fun, I'm usually in near them with The rust analyzer extension enabled. I like that because I have a big fat Linux machine at work and a tiny MacBook air, and I can sit in bed SSH into my fat machine, have it do the hard work.

Pause the TMX session, go work on it somewhere else. I also use rust rover from JetBrains. Partly because they handed it out free and I wanted to see how good it was. And it's getting better. Yeah. I don't use it all the time. Purely because my muscle memory hasn't caught up with the different keystrokes yet.

Yeah for a scripting language. I would say Python these days Even if it's overkill, I also write way too much stuff in just regular old bash But I

Jonathan: definitely counts nothing wrong with using bash We had the we had the creator of bash back several years ago, and I asked him like well What what's your favorite scripting language does bash count and he was almost offended.

He's like, yes, of course it counts That's just great for scripting Oh fun stuff Well, thank you sir for being here. I sure appreciate it. Thank you for letting me pick your brain about rust I I got a lot of my questions answered and honestly, I think I'm gonna have to go check out probably the Probably the free version first the, the, the, it's not Hands On Rust, what's the, the other tutorial, what's the title of it?

Rogue like! Rust, Rust Roguelike Tutorial. The Rust Roguelike Tutorial, yes, I've got it pulled up here, but I couldn't immediately see the title of it. I think I'm gonna have to start there, but I'm gonna have to look real close at Hands On Rust, because like I said, it's, it's kind of been on my to do list for the longest time to actually pick up and learn some Rust, and the idea of building a Roguelike with it, that, that sounds like fun.

I like it. Awesome!

Herbert: All right. Thank you for

Jonathan: having me. Yes, sir. I sure appreciate it. All right, folks. That was, that was Herbert Herbert Wolverson talking about Rust a lot of fun. I had a lot of fun with that. I appreciate that he was willing to talk, not necessarily just about his own work, but the, the wider Rust ecosystem.

And We get to, we get to plug his books. So good stuff. Next week we do not yet have a guest scheduled. So if you know of someone that is involved with an open source project give me a ring it's floss at hackaday. com or come into the discord and say, Hey, this is the person you need to talk to, or even better reach out to, you know, a programmer or the lead dev of your favorite project and have them get in touch with me directly.

But we don't have a guest yet for next week. We will scramble, we will make something happen, but we would love to have somebody to actually talk to. Alright, so let's see. The one thing that I want to plug for the end of the show here is, of course, the Untitled Linux Show. It is now, the audio version of it is available to everyone.

We have a lot of fun there just going over the week's news worth of Linux. And you know, funny thing? There's always enough news to cover. Whenever a week has gone by that we go, Man, there just wasn't anything to talk about. Nope. Pretty much always. Bunch of news going on in the, the Linux world that we all sort of inhabit together.

That is the, that is the big one for me. Thank you to everyone that caught us live in the Discord. We had a decent crew there today and those on the download as well. Make sure to share the show. We love letting folks know about it and we will see you next week on Floss Weekly.