Episode 829 transcript

FLOSS-829

Jonathan: This week I chat with Herbert Verson and Frank Bick about Buffer Bloat, Lipper, QOS, and remembering Dave Tot you don't wanna miss it. So stay tuned. This is Floss Weekly, episode 829, recorded Tuesday, April the 15th. This machine kills Gons.

It is Time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and today we've got something well, really fascinating that we're gonna dive into. But it's not one of the things that we really wanted to talk about today. We wanted to talk about something else.

And so I'm gonna bring our guests on and we've got Frank and Herbert. And these are a couple of the guys from Libre, QOS, the the, the saving the internet from Buffer Bloat organization basically. And today we're gonna talk about Lib, QOS, but we're also, we're, we're gonna remember Dave Todd, and.

If you don't know Dave did pass about, was it about a month ago? It's been, it's been a little while now, right guys?

Herbert: April 1st was the announcement I got, but

Jonathan: April 1st, and there are, so Dave lived in California. Dave was a huge space nerd, like, like I am. And I think like these guys are too and was actually in Brownsville, Texas and passed there.

And from what we can tell of natural causes, the, the weird thing about dying outside of your state is the paperwork is terrible. So like, we don't have all of the answers yet. We can tell you though, even though the announcement was made on April 1st, it's not an April Fool's joke. I wish it were. But it's not Dave, Dave Todd has passed.

And so we we don't wanna make this a downer of a show, but what we wanna do is, is talk about like the, the legacy, the buffer bloat legacy and all some of the other things that Dave has done. Also talk about Liberty QOS, where it's at now and where it's going, the, the neat things coming for the future.

Some of the exciting things that that project has done. And so that's, that's kind of what we're up to today. So like I said, we've got Herbert Verson and I, I guess technically it's, it's France sec Bik and you go by Frank.

Frank: Yeah, Frank is easier for, for Frank. Any Frank? Yes. Frank is easier. Ango Saxon world and even for India, basically, so Yeah.

Yeah, it works. I can, I can imagine. Did I, did I get it at least reasonably close? Absolutely. Okay. You know in the, like in the English Pro pronunciation, I think it's the best you can do.

Jonathan: I'll take that, I'll take that. I get, I get a lot of unusual to me names and trying to get through them sometimes is, is it's a challenge.

It's fun. Alright, so let's I don't know, where do we wanna start? Let's, let's talk about what, let's talk about what Liberty QOS is. Maybe that's a, a good starting point. And let's see, Frank, you told me in your notes that you are the, the gopher for Liberty QOS Let's start there. What, what does that mean and how did you get there and what things do you go for?

Frank: Right. So I go for everything basically. And I, I, I got this nickname when I was living in Iceland back then in 2013, 2014. And like besides anything else, what I was doing there. I was also at the fishing boat, and I was helping the maintenance engineers to do all the fixes and stuff. And, you know, the, the, basically the, the best thing that I was able to do was to fetch like all the things they needed.

And I was going for everything. And they said that You are really good, good, good in, in that. And basically, it, it, it stuck with me. And then I. Tried like everywhere. I was doing some work. I was trying to do the same. I was trying to help mm-hmm. You know, to do things that nobody liked to do and don't want to do and things like that.

So I guess that's a really helpful role to, to do. And you know, I have really good colleagues. They are you know, rock stars. Dave was a rockstar. Herbert is a rockstar. Robert is a rockstar even though like not many people know them yet, right? Mm-hmm. So I'm, I'm the, that one guy that want to help

Jonathan: them shine.

Yeah. I remember, I remember being like a young teenager working with mainly my grandfather at the time. And I didn't, I didn't know anything. And so that was my job. I was the gopher for a while. And, and the, the idea there was that you hang around long enough, you go for the right tools and you kind of pick up, you learn as you go.

And finally you, you get to the point to where. You have a younger cousin that gets to be the gopher and you get to actually work on this stuff. Right.

Frank: So maybe one day I will get my gopher as well,

Jonathan: graduate from Gopher School. Right. And then, and then Herbert, we know you as the rest guy. How do you, how do you fit in with Lu QOS?

Or, or do you, you are part of it, right?

Herbert: I am I'm the Chief product Officer. Yes. We picked entirely because I wanted to be called C3 po. The you know, the background for this is that on top of teaching rust and writing books about rust, I've been helping out with an internet service provider in Central Missouri for, I don't know, well over a decade now.

Mm-hmm. And we're kind of a weird ISP that we don't actually make any money. We, our mission is basically to reach out to the areas that nobody else can get to and get them the best service we can, we can manage 'cause the Potter, Missouri I'm in has some seriously swampy areas mm-hmm. That if you lay fiber, it just washes away areas that flood every year.

Areas that get hit by tornadoes every year. So it's really, so aerial fiber doesn't work because the tornadoes take it away. Mm-hmm. Buried fiber doesn't work because you're in a swamp and so you end up running a whole lot of wireless links. Mm-hmm. And that's what Ison does. And, you know, we fell afoul of serpentine Missouri tax low and we tried to actually be public benefit, but, so we're public benefit in what we do, but not on, not on the actual structure.

Jonathan: Mm-hmm.

Herbert: And one of the terrible things about things like 10 mile long wireless links is that buffer bloat gets absurd.

Jonathan: Mm-hmm.

Herbert: And we'll talk about what buffer bloat is in a bit, but I originally signed up at the com product called Proce that uses FQ cuddle. That's how I first heard about Dave and

Jonathan: mm-hmm.

Herbert: He actually didn't remember it, but I was lucky enough to run into him at a, at a convention about that year. Then we didn't meet again for a while. And anyway, Robert started up Liber, QOS around the time the pandemic was hitting and really wanted to tackle the buffer bloat. Situation to try and get even the people with not so good connections, good usable internet because suddenly everybody's at home.

Every network is suddenly massively over capacity because everybody's at home. Whereas previously you could reliably plan that you know, eight to five people go to work. In the evenings they watch videos do home things. Well now everybody's a business customer at home. Suddenly you have a lot less leeway.

And he started Libra QOS with Dave. They put together a giant script initially that set up FQ cuddle for everybody. And then Dave talked him into turning that into cake, which is basically the next version of FQ Cuddle. It adds a whole lot of bells, whistles, and works unbelievably well. And that's what got my interest was.

I saw them on GitHub. I said, Hey, I could, I could use this. And because I am, I dunno, probably autistic. I sat down and basically wrote a rest version in a week. Shared it with Robert and Robert showed me some of the pain points he had. Like IPV six wasn't working right. Mm-hmm. And he was having troubles scaling it up to multiple CPUs because of some bottlenecks in the Linux kernel.

And so I worked closely with Robert and Dave understanding what they needed. And Dave put me in touch with Taki whose name, last name I just forgot.

Frank: Taki ho Jorgenson.

Herbert: Thank you. Yeah. I'm actually really glad you said that because that will never make it out unscathed. I'm, I apologize. I but we.

Patched a project called XT, PS X-T-P-C-P-U MAP and managed to make the Linux Traffic Shaping System use multiple cars. And all of a sudden now we've got a system that can, you know, scale up well beyond a gig. You know, we, at the time we were celebrating having just hit 10 gigs. We patched in IPV six, which is quite important.

Mm-hmm. The original setup couldn't do that, which was a real problem because Robert has IPV six. So we got that in and then we were like, Hey, this needs a gooey. So I sat down and wrote one and Libra QoS 1.4 came out with bells whistles, gooey, and it. Really took off tar terrifyingly. So to the point that suddenly I was, we were getting messages from parts of the world that I had to look up where they were.

Mm-hmm. I think last time we checked, we're in what, 58 countries now and most US states and the most recent version we're working on has a whole new gooey because the first one was my was also me learning JavaScript. And this time around it's a, this time around it's a nice gooey we're pretty proud of it.

An optional paid stats backend system. We do, we do a lot, a lot of stuff with that, but we've gotta cover our hosting phase. Mm-hmm. So we're working on a self-hosted version, but at least for now that's, that's in the cloud because nobody wants to self-host something I'm updating three times a day.

Yeah. Which, and I, I don't blame them. But the scale has absolutely astounded me. We've gone from celebrating 10 gigs to talking to a guy who's pushing 80 to 90 gigs a second every night through one of these boxes. You know, it was one of Dave's real happy moments was, you know, wait, this code I wrote can do 90 gigs.

And it's like I mean we, we were doing a, basically a happy dance over Zoom on that one. Yeah. And so it's been an incredible ride. And, you know, Frank doesn't do himself justice. If Frank wasn't reminding me in the morning to drink my coffee and reply to support tickets, support tickets would never be replied to.

Frank: Yeah. So, so ba basically, you know you know, Jonathan and like everybody listening just this is the demonstration how how a rock star actually Herbert is and, and, and yeah. It's my pleasure, my, my pleasure to be part of it.

Jonathan: Yeah. You know, it's, i, I've gotten to be both of those roles throughout the years in various projects.

Right now I'm doing sort of an operations, or all, technically they're calling me CEO, but very much just taking care of operations where, where I'm at. And it is, it's, it's interesting. You're, you're, you call 'em a rockstar. You're your, your developers really, they really do need somebody to help handle that paperwork stuff.

It is extremely useful to not have to Yeah. Not have to deal with it. Yeah. I,

Frank: yes, I, I I, I like to take over anything that yeah, that's, you know, so your, your, your, your hourly rate is higher, you know, for, for such nonsense. Mm-hmm. So it's it's really it is really something like that. Yeah. And yeah, you know,

Herbert: that's not good.

Both Dave and I would forget to put socks on in the morning if someone didn't remind us.

Jonathan: So Frank, how did you come onto the, the Liberty Qs project? How did you find it and, and decide it was something you wanted to be involved with?

Frank: Right. So I, I, I knew Dave through the events, IP events in the US mm-hmm. And I was, I was working with the realtor company open, WRT based company from Republic tourists.

Oh, right, right. And yes. And, and, and they, they free, like all laughed, the, the, the product. So we, we started to talk, you know, I was one of the, the people that got to the US in 2021 mm-hmm. To osa. And it wa I, I was probably the only foreigner that managed to sneak in, in the US and we, we, we, we, we, we talked a lot with Dave and we find a lot of common ground.

Mm-hmm. And he, he offered me a help. With tourists and he tested the router. He wrote a, a really great book about how good tourist omnia is for, for the people you know from, especially from the security standpoint. I. Because the tourists or the parent company, CS Nick, they added lot of security features like firewall and things like that, that, that are not typical part of open WRT.

Jonathan: Mm-hmm.

Frank: And then in 2022, they've told me about Libra QoS and I understood that that's the missing piece that Dave was looking for, because most of the router companies, they will never accept or most probably never accept FCU coon cake for many reasons. You know, I want, I want to stay nice, so I will, don't name the reasons why big companies that like to brag.

That they have the biggest buffers in the industry. They, you know, they, they don't want to do that. They want to sell the new, the newest a OA eight oh 8 0 2 11 standard like every few years. They don't want you to have a router at home or, or in the office or in the stadium that will last 10, 10 years.

So

Jonathan: I remember, this is a bit of a rabbit trail, but I was actually in Ireland at an open WRT mini conference when the tourist Omnia, the first time I heard of it at least, was announced. And we all thought that it was just the coolest thing. Then we were excited about it. I don't know, I don't think I ever got my hands on one of them.

It, the, the price point wasn't quite what I was after for the, the small companies in Oklahoma that I was dealing with. But I remember it was just, it was the, it was the coolest thing that somebody was doing this making. Dedicated hardware, like powerful hardware for open WRT? Yes. It was exciting at the time.

Frank: Yes. So the, so the goal was to, to have it like, at least for 10 years, you know, the typical router won't stand, mm-hmm. Than much longer. Than much longer. So,

Jonathan: yeah. At, at the time I was taking TP link routers, little, little white UFOs from TP Link. We were flashing open WRT on 'em and selling them to companies for, I think.

$85 US installed, which was ridiculously cheap. Looking back on it now, it's like I was not charging enough for that, that I was doing. Yeah.

Frank: So, so, so ba basically, OO open source is really changing the world. And open WRT is one of the, the, the biggest examples because it's, it was just a three people that started open WRT.

Mm-hmm. One of them wa was from Hungary. One of them was from Canada. Canada. And the other was US guy, if I remember correctly. I remember only the name of the Hungarian guy because he's my friend. It's, his name is em. And he's still part of open WRT and he's doing a lot of cool stuff.

He's also a rockstar, but most of, most of the things that he's doing is under NDA and you can, you can say it out out loud, but So these, these are the people that are changing the world, right? Starlink was built on open WRT. Yep.

Jonathan: I've got, I've got the gut somewhere over there on my junk shelf.

I've got the guts of the first, the, the version one starlink router where we were taking it apart and trying to figure out if we could put a, a newer open WRT image on it. I worked a little bit, I worked a little bit with Dave on that, actually. Not, not a whole lot. I ran outta time to really do much with it, but I know we were, we were both poking at it at about the same time.

Yeah, I think, I think the first time I met Dave was when Doc Searles brought him on to a floss weekly to talk about buffer bloat. And that was, that was really interesting. He had his I think it was a funnel and a, you know, a picture, a picture of water and a funnel. Yes, yes. Pour he, he was pouring down and pour.

Yeah. We could see that. Oh yeah, you, you get a lot of water that, that stands up there. It doesn't make it through the funnel fast enough. And he, he shared that clip. We shared that clip all over the place. That was, that was a lot of fun. That was kind of when I, when I first met Dave, I realized that Yeah, he is, he is one of, one of my people, one of one of our people.

He is, he was a cool guy. We, we very much enjoyed having him on the show. Had he, had he back several times throughout the years. Talk about various things. So let's see, with with Liberty QOS it is it's really a, it's like a toolbox for ISPs. That's, that's really what it's made for, right?

Herbert: Yeah, we started out very ISP focused, 'cause Robert and I both have wireless networks.

Mm-hmm. We've been running into more and more people using it for different things. Apartment complexes. Mm-hmm. University campuses cruise ships cruise ships was the one that surprised me, but it does a great job of smoothing out satellite internet and giving the people on board a much better experience.

Yeah. It's, you know, honestly I'm surprised by all the, all the places that it keeps popping up. I've actually had people tell me they've heard of the project now, which is which, which is amazingly cool. You know, I've I'd got at Rust Con somebody actually remembered me as Herbert, the Libert Q os guy, not Herbert, the guy who wrote the book on Rust.

So I was like, Hey, that's cool.

And you know, from an ISP point of view, we were focused, we were tackling one of Dave's major frustrations, which is we can't get every home user out there to run a good router.

Jonathan: Right.

Herbert: And Dave's original plan was to make a nice router and basically give it to everybody on the planet so that we can stop buffer bloat that way.

So the discovery that you could put something in line and have everything in the ISP go through it and do a good job of fixing everybody's buffer bloat all at once.

Jonathan: Yeah.

Herbert: Was kind of, you know, Dave's eyes lit up. 'cause he figured out, Hey, I can do thousands of thousands of places at a time instead of having to go to everybody's house.

Because Dave was a man on O Mission. He Oh yeah. Dave was, Dave was absolutely determined the buffer bloat was going to going to be destroyed. And

Jonathan: let's, let's dig into that. Let's, let's dig into that part of this. First off, what, give us the, the quick, the, the quick definition. What is buffer bloat?

Herbert: Okay.

So when packets come into routers or whatever's forwarding them

Jonathan: mm-hmm.

Herbert: If there's any sort of congestion, so you. You can't immediately just pass it right through. Then packets get queued up in a buffer. And so traditionally the b when, when you've got more traffic than you have capacity, the buffer will eventually fill up and packets bounce off the back of it, just get randomly dropped.

Jonathan: Mm-hmm.

Herbert: And problem with this is that people solve that problem by, by getting ever bigger buffers bigger and bigger and bigger. And now the transit time of your packet all the way through the buffer gets ridiculous. And it gets unfair as well because Johnny downstairs downloading the Call of Duty update is going to be 99% of that buffer.

Jonathan: Mm-hmm.

Herbert: And Mary upstairs trying to make a video conference call for work is gonna be 1% of that buffer, but all her packets have to wait because Call of Duty's in the way. And so she gets that dreaded, you know, your video quality is un is unstable. Things stutter. It looks terrible. And so buffer bloat is, your buffer is much bigger than it needs to be.

And if you look at the round trip time of packets going through the system, there's actually an optimal amount of time for how long something should live in the buffer. And so cuddle, the first algorithm that they've worked on introduced the concept of sojourn time. So measuring the amount of time.

That this packet has been buffered. Putting it through an algorithm relative to approximately one round trip and preemptively dropping packets if they've been in the buffer too long. And that tends to, that as a result slows the call of duty download down a little bit, but makes room for everything else.

So FQ coddle was the follow up to that. In addition to, so on time it added fair queuing. So everybody's data gets basically put in buckets and you make sure that each bucket gets a chance to go first. So this ends up favoring the sparse, the small buckets. So your VoIP call no longer stutters because your wipe call doesn't have a lot in it.

The packets each get a fair chance to get out. The big bulky download that is designed to max out whatever it's got. But nobody's gonna notice that their download was 30 seconds longer when it, you know, when it's a huge multi-gig file.

Jonathan: Yep.

Herbert: That's okay. So slow that one down a little bit. But you don't, you're not trying to do, you know, DPI packet inspection.

You're not trying to say, Hey, this is YouTube, this is Netflix. You're just treating everything. The better behaved the protocol, the better FQ cuddle treats it. And so cake was FQ cuddle on steroids where Dave and a bunch of other people got together and thought of everything they could possibly come up with to make FQ cuddle better.

So the algorithm that handles so time in the queue improved substantially. Tuck's thesis Buff Bullet and beyond is great if you want to try and understand how that works. It's math heavy to the point that I have to read it twice every now and again. The fair queuing got extended out to have both have a better algorithm and also start looking at the SCP tags to tell you when traffic is willing to tell you what it is.

Allowing it to be divided up into tens with a little bit of preferential treatment for the real time and a little bit of slowing down for bulk traffic. Honestly, the tins, the tins are the first thing everybody goes to because it looks, hey, it's classifying my traffic. They're honestly the smallest, they're honestly the smallest component to smoothness.

They help. But the algorithm, the coddle algorithm the cobalt algorithm that replaced it and the fair queuing mm-hmm. Is the real magic. You can go to best. You can dump all the traffic into best effort and still see things a whole lot better. And so buffer bloat is problematic because it's not just your writer.

Your router is talking to the ISPs router mm-hmm. Which is talking to the internet exchange router, talking to another internet exchange writer and so on all the way to the hosting center. And so there's a good chance that somebody somewhere has buffer below whether and kind of the magic of K and FQ cuddle is that because they measure timings on packets as they go through.

Jonathan: Mm-hmm.

Herbert: It's got enough information to start making the streams self-pacing, even though the rest of the network kind of sucks. I mean, you can't make, you can't do something about a router in Chicago that's decided to drop half of its traffic that's outta your control.

Jonathan: Right. But if

Herbert: it's mostly working, then you know, fq coddle will make your experience a whole lot better just by working with what, what it sees going through your router.

Jonathan: Mm-hmm.

Herbert: And assuming that everybody else sucks. Now, if you can get, you know, that guy in Chicago to also install fq Coddle or Cake and maybe the hosting provider to provide. Some buffer improvements. Everything gets better for everybody, but you know, one, one bottle at a time. So buffer bloat. If I had, when I, when I give the elevator summary, you know, the summary to some guy in the, in the elevator who wants to know what I do, it's I make sure that you can still make a phone call.

When your daughter has decided that she wants to download an entire season of papa peg.

Jonathan: Yes. Very important. When, when did this become, when did this first become a problem? And I'm, I'm thinking about, I'm reminded of TCP slow start. It sounds like it's sort of the similar problem that got fixed back in the eighties. When, when did, when did it become that TCP slow start was not enough to, to solve this problem?

Herbert: Sorry, I'm trying to think of dates. TCP slow start was something that Dave liked to grumble about a lot. Yes.

Jonathan: Yes. That's probably why it came to mind. So,

Frank: so, but, so, but if, if I may, yes. Basically, they, Dave realized the problem when he went down to Nicaragua, which was he his favorite country.

And he, he was thinking about retiring after, like, everything, what he, what he was doing in the, in the, in the ISP world in the nineties before the dotcom bubble or telecom bubble rather. And he, he, there was, there was open WRT after 2000 and four like already on the market, right? So people were using flashing all, all the routers we did.

And so he, he, he, he saw a problem down there in Nicaragua. So he decided that he will help people. So he was flashing all the routers around him in hotels, in in the cafes in, in homes of people. But the problem was there. So there was a first like a video conferencing at the time. Video calls on Skype and he saw the issue like so like prior to 2010, so maybe 2008, he started to talk with it with Jim Gatis online.

And Jim was the he was the guy, if I remember correctly, that actually coined the term buffer boat. And so they started to talk and Dave decided to go after it and to, to help Jim to fix it.

Jonathan: Yeah. Interesting. So that was, that was kind of, that's when, that's when Dave got on this. I always think of Dave as the buffer boat guy.

But Jim, Jim was, Jim was working on it even before then, I suppose. Yes.

Frank: So he, he started early, but he, you know, Dave picked, picked it up later on. Mm-hmm. You know, Jim he, he did, did a lot of work and he was kind of burned out. So there was a, a space for more people to plug in and to actually continue, continue his his mission.

Mm-hmm. So, but they if I, if I get the date right, like buffer ball do net was start, started at 2010. Mm-hmm. And, and so that was the, that was the basically the, the starting, like the main starting point for Dave, and this Dave started to be phase of it. He was chasing like everybody right.

To, to change stuff for, for the good of people and, and you know, the or Right, right, right from the beginning of like, after Cordell that was FQ code and the, the uptake was really, really, really fast. Right. Linux opened WRT previous SD quite a lot of stuff. Right. And Microti as a, as a first probably the, the like mainstream closed source company.

Right. So the expectations where that, yeah, it'll be done in no time.

Speaker 5: Mm-hmm.

Jonathan: Yeah, it's interesting. I, I was just linked to RFC nine 70, which is on packet switches with infinite storage from 1985. Right? So people, people were, people were thinking about this even back then that this might not be a good thing.

Yeah, I, I'm not, I'm not immediately seeing when Buffer bloat was first uttered, when, when Jim put that word together. But it's such, it's such an interesting history and it's such an interesting problem too. The, the thing you do to try to fix your your router, fix your network switches, is introduces its own problem.

Interesting stuff. So what's what's, what's next in the buffer bloat story? Where, where are we? Is it a solved problem? If not, what's coming next?

Herbert: Let's see. They've got a bunch of enhancements to cake into production not very long ago at all. There in net next on Lennox added a good 10% to throughput and fixed a couple of things he called bugs that I honestly honestly, the math behind it is stretching my abilities.

But he I understood, I understood the performance improvements. I actually worked with him a little bit on some of that and pushed that up to Toki and Lennox is putting in, ePF based Q discs system in net next right now. Mm-hmm. I know there is some talk about getting cake pushed over to that to make it a little easier to customize because custom kernel modules have become a little problematic with everybody's having signed boot loaders now.

Yeah. So e that's the immediate focus. I am trying to fulfill what turned out to be Dave's last wish and fix a memory issue in cake. It, it's gonna be a while because it's it low level C and I'm more low level rust guy, but I, I mean, I've been using c since the nineties. It's just I'm rust.

I'm, you know, I don't wanna say I'm rusty, but you know what I mean. So that's, that's the immediate part of trying to carry on his legacy. But we're also talking to. Every yeah, everyone will listen to try and find next gen, next generation of researchers, see if we can feed them data. Mm-hmm. So they can look at the problem, maybe come up with something cool.

Like we're working with one PhD researcher right now, giving him vast amounts of TCP RET and round trip time estimates from various parts of the world. He's taking a geographic focus, see if there's anything you can do about long distance lengths that automatically give you some buffering.

Mm-hmm. Because it just takes so long to get there in packet terms. But yeah, it's also, Dave's really big shoes, you know, it's mm-hmm. I don't know if my feet are big enough to fill them, but we're, we're trying.

Frank: Yeah. And, and, and there is also like at least three groups. Of people that are working on, like something that it's called like Cake two Cake mq, MQ cake.

So like all the usual su suspects that that you can think about like Dave was, was scoping something that there was supposed to be like, at least the working title was cake mq. Then there is Stocky and his guys working on, on improvements and I heard from Dave that Jonathan Morton was also working on something to improve cake.

So there are things like that. And also we see like all the time new stuff coming toker, like I, I think it was last week, right, Herbert, that there was that BPF new stuff from, from Amari.

Herbert: Yeah, I'm, I'm actually excited to play with that. I. A BPF is what got me started on Libra QOS.

So not too bad for a fresh project.

Jonathan: Yeah. BBPF is something I find fascinating, I've never gotten a chance to play with yet. Maybe, maybe one of these days I'll have a project where I get to go in there and fiddle with it. You mentioned something Herbert, that, that really got me thinking, this idea of the, the buffering that's sort of built into long distance links and you mentioned geography and I I immediately in my mind, 'cause we talked about Dave being a space nut like I am, is the, the astronomy or, or space travel really implications of that.

And I don't know if this is something Dave was working on. I'm sure somebody is, but like when you start talking about, well, the ISS for one thing, but that's low earth orbit. What about when you start talking about putting people on the moon or on Mars? Mars what does. What does the buffer bloat situation look like?

What does the internet look like when you start talking about Mars and links between planets where just your, your round trip time of the speed of light is measured in seconds to minutes?

Herbert: Well, you're not gonna be doing interactive video unfortunately.

Frank: Right? For, for, for navo.

Herbert: You know, I, I actually read a paper yesterday about how I think it's toia proposing to put a 5G network on the moon.

Jonathan: Mm-hmm.

Herbert: I saw that. Which is clever because they're limit, they're going to be having, by having local nodes, you'll get good latency for at least talking to other people on the moon. I know the deep space network is something they've talked about a lot. It's not something I've ever worked on, but extensions to T-C-P-I-P where were made, especially for it.

So that. Packet latency to Jupiter would work at all. The, it, you know, it's a, it's a tough one. I, because it's the same problem that you have on earth, but on a, on a bigger scale, you, you often need a big buffer. I mean, you know, we laugh about, we've got the biggest buffers in the industry, but sometimes you need it because when there's packet loss going on, fragments being reassembled and so on, those fragments all have to get into a buffer somewhere so that you can reassemble 'em.

At the same time. There's, there's such a thing as too much buffer, hence buffer bloat.

Jonathan: Mm-hmm.

Herbert: So, I remember Dave telling me when we were chatting one night that in theory FQ cuddle would work in space if you changed the RTT parameter to assume. That really hideous latency is okay because you can't avoid it.

Right. I don't know if he I don't know quite how far he got. I know he was talking to some people putting satellites up. I know one of the first, actually starlink was one of the last things he worked on.

Jonathan: Mm-hmm.

Herbert: But starling's, you know, low hanging fruit because it's relatively low, relatively low orbit, it's low hanging the geo.

But we do know that FQ coddle helps with geostationary orbit.

Jonathan: Mm-hmm.

Herbert: So there's no reason it shouldn't help with the moon. It's just gonna be a matter of unfortunately, you know, experimentation requires that we get to the moon, put up a network and see what happens.

Jonathan: Mm-hmm.

Herbert: I will volunteer for that if anybody wants, wants me, by the way.

Yeah,

Jonathan: there you go. SCPS, the Space Communications Pro Protocol Specifications, that is at least one of the projects that people are working on to do this, to, to make extensions on things like FTP and TCP sec even. And, and what do you do when you're trying to talk between planets? I, I find this, I find this utterly fascinating.

I, it, this is one of these things that it really does make me, make me kind of, it makes me sad that Dave's not gonna be around to work on this. 'cause I know this is the kind of thing that he loved too. And we just eat it up.

Herbert: And can you imagine a three-way handshake with minute, one minute delays for each.

Just to establish, just to establish the TCP connection, that's mm-hmm. I mean, that's pretty nightmare-ish.

Frank: Yeah. So maybe, especially if all you want

Herbert: to do is send one packet saying hello,

Frank: right. Yeah. So may maybe UDP for, for such a situation.

Jonathan: Yeah. My, my thought on this was always that you would, you would have to have a, I mean, it would be storing forward of some sort, but you would almost have to have two, two discreet internets that would sink from time to time.

Just because doing anything live between here and there is just a total no-go. The, the, the round trip time to and from Mars is like 24 minutes. Or at least my quick Google suggests 24 minutes. And, you know, nothing, none of the internet that we, that we use works on that kind of a round trip time.

It just all completely breaks down. So, yeah, I find this, I find this super fascinating and I, I too, I look forward to, actually being able to run the experiment and see what happens. See if we can make it work.

Herbert: Do you remember a phyto net from the BBS days? It always, it always makes me think of that because you sent the process of sending email was you wrote it, you send it by modem to your local hop.

Jonathan: Mm-hmm.

Herbert: And it would, at some point during the night, send it off to the next exchange and mm-hmm. Your email might take three days to make its way across the world, but it would get there eventually.

Jonathan: Yep.

Herbert: And so I could see something like that working. Mm-hmm. And the old network news protocol too.

Jonathan: Yeah. Yep.

Yeah. Interesting stuff. But it's al it's not quite that extreme, but it's almost gonna go back to the days of days of sale when you didn't get any updates until the ship came into Harbor that had the latest news dispatches. Right. It's, it's. It's a fascinating thought that as we, you know, kind of expand the human race out it's gonna be a a in some, to some extent, a return to that sort of a, a paradigm.

Herbert: You know, at the same time, I still can't believe I wake up and see videos of a probe crashing into, you know, a, a, a moon off of satin. Mm-hmm. And and that's old tech, you know, had launched years ago.

Jonathan: Right. Yeah. Yeah. Fascinating stuff. What is speaking of all the state space stuff, what is the the, the current status of, of starlink with the open WRT version and Cake or kode built, built into it?

Frank: R Right, right. So, so in, in the beginning of, of March, they finally publicly announced that last year, I'm sorry beginning of March 2024, they publicly announced that they, they implemented FQ call. So they was ecstatic. That's great. And it, it really cut down the latency and, and the goal was 20 milliseconds.

And in, in certain instances, they are able to achieve that, which is incredible.

Speaker 5: Mm-hmm.

Frank: And they, they also have like air airtime, airtime, fairness, and some other, other stuff I was told mm-hmm. That Dave was fighting for. There, there there is one particular amazing guy that got into Starling and he's one of Dave.

Dave, Dave apostles Nathan os. So like, if we can credit someone, someone from, from that is on the inside right now, it's certainly him. Like he was really active for years on, on starlink mailing list and, and various buffer ball mailing lists. And like, I don't know for sure, but he, he might be the, probably the last drop to really make it happen.

Yeah. After like all those days runs and, and, and emails and, and songs and everything.

Jonathan: Yeah. I'm trying to, I'm trying to remember, I'm trying to find it actually. A apparently I, so I wrote a Hackaday article about Buffer bloat several years ago. And I think it was Dave that sent me an email and it was like who was it?

One of the, one of the guys from Doom, oh, I can't remember his name at the moment. Anyway John Carmack, I think it was, yeah, I think it was John Carmack forwarded my Hackaday article onto Elon Musk and he's just like, okay, that's cool. That's a good, that's a neat, deep bit of news to wake up to in the morning.

I'm gonna see if I can find that email. I've got it. I've got it here somewhere. I've just got, I've got get the right search term going. But you know, that's one of those things that. For, for one thing, the this space that I found myself in between doing hack a day and getting to talk with people like you guys and people like Dave, it's, it's really cool the, the connections I've been able to make, but also it's kind of a good feeling that like the fact that starlink is getting better, I would like to think I've had at least a tiny little part in making that happen.

Frank: Yes. Like pretty much everybody that was like tweeting to Eon talking to Starling, sending emails singing songs using Starling, you know, that was the, that was the push from, from all places. Yes. And it was really necessary for something like that to happen, so. Yeah. Yeah.

Herbert: You know, John, John Carmack was actually backing Dave on Patreon for a long time.

Mm-hmm. Big supporter of the Buffer BLO thing. You know, that that was one of the things I couldn't believe was after we posted the announcement I opened my inbox and I see email from Vince Surf. Jim Gdes. John Carmack. And it's like any other occasion I'd been jumping up and down Fanboying.

Jonathan: I've moved that.

A little bit of that. Yes. Yeah, I, I found the email Dave emailed me back in August of 2022. He wanted to know if I could see stats on Hackaday, and then he says, you'll be pleased to know John Carmack. Read your article and forwarded it to Musk. That was, that was one of those, that was one of those happy little dances for me.

I got, I got a lot of pleasure out of that. Well, well deserved. Yeah, it was fun. It was a lot of fun. Alright so Starlink is doing well. One of the other things I remember about Dave is he, would you call it flk? The, the music he was, he was definitely a, a, a music fan, but it was, it was very much his own sort of take on doing things like it.

Gpls me is one of the songs that I remember. He had a he had a sticker on his guitar case that said this, this machine

Frank: kill,

Kills Gans.

Jonathan: Yes, that was it. That was it. He, he said he was inspired by the, this machine kills fascists, but didn't want to quite take that I forget, I forget exactly what he said, but he went in Gons instead.

That was, that was sort of a quirky part of his personality, wasn't it? Do you know where that came from? Would you call that Fi? Is that, would we call that Fi Is it like a, a tech version of Fi?

Frank: Oh, yeah. Yeah. He, he was writing in the style, so it's like folk music but in sci-fi. Mm-hmm. So like everything tech tech wise people like Dave were like writing lyrics down and it was like always always a great success.

Yeah. You could always tell he

Jonathan: he really enjoyed it. He had fun with it. Yeah.

Frank: At the events, at the podcasts or as a, as an outro or as an intro at at, at the webinar even. Yeah.

Jonathan: Yeah. That, that's good. Mashed potato. One of our listeners has a suggestion that, that be the, that'd be the show title for today.

This machine kills Bogans. I like that. Absolutely. I think that would be, let's do that appropriate way to, to honor, to honor Dave.

Herbert: I think we've got pictures of Dave busking at at least 20 major conventions. Mm-hmm. Yeah.

Frank: And a also like close to the the actual SpaceX. Yep. Yep. That makes

Jonathan: sense. It's, that's, that's something that he would definitely do.

Alright, so what, what else Are there some, are there some memories that you guys have of Dave that you, we want to touch on that we haven't,

anything, anything comes to mind? Yeah, go, go ahead Bert.

Herbert: Oh, okay. You know, I just, a hu really a human memory. Mm-hmm. We were having a labor QOS staff meeting and my three-year-old ran upstairs, jumped onto my lap and introduced herself and Dave just lit up. Interacted with, with her, made her feel like the center of the room.

And we, we stopped talking for long enough that she ran over to her little toy piano and she's a really smart kid. She hit demo mode and Mimed playing for her leaves, and Dave is just egging her on, encouraging her. She goes down, tells me what a great kid I have. Meeting continues, but you know, I've, I've talked to a lot of people for whom the 3-year-old invading is cute for 10 seconds, but get rid of her.

Jonathan: Mm-hmm.

Herbert: Dave was not like that. Dave, Dave really just was a warm, kind person who would make time for you. We, you know, we've had, we've had emails from people saying, Hey, you know, I, I had a problem and Dave appeared outta the blue, fixed everything and swept away again. And that's just kind of who he was.

Jonathan: Yep, yep. Frank, did you have do you have something similar about Yeah.

Frank: Probably the time that I cherished the most that I was lucky to spend with Dave. Was was in Mexico after wifi now global Summit in Cancun in 2022. Like the rest of the tourist tourist guys, they already left, but I was still there.

Staying with Dave for a couple of days and we spend it you know playing his guitar, singing, sharing all of songs, stories, and also working on, on, on tourist, tourist routes because they was showing me all of stuff. He was measuring things with Flint and other tools. Mm-hmm. And he was showing me things that should be improved.

And we were just, you know, like enjoying time, walking around the, around the beaches. Talking about things, talk, talking about life, and you, you know, that these are, these were the things that like, he was able to pair, like, like he said, he was able to connect to, to, to to small three years old. Three years old kid.

Mm-hmm. And also like, you know, to the biggest names in the industry and also to like some strange guy from Europe that is working on, on routers and, and, you know, want to, want to learn stuff. So that was, that was the, like a bigger, bigger than life type of character. Like they, they, they was. And it, it's amazing to go to all, all the, all the media and to, to read comments from people.

And, and it's it's overwhelming. And, you know, I, I do believe that he's like up there you know, up there somewhere. Mm-hmm. And he, he, he sees stuff and like he, he likes to like, he like to set things like per aspera ra, right? Mm-hmm. So that's something that we can say to

Jonathan: him, for sure. I'm gonna start introducing myself as some strange guy from Oklahoma that works on tech and likes to learn stuff.

I think that's great, right? It's,

Frank: Ca can, can I, can I read a poem? Sure. Yeah. So Dave, like, you know, back then when, when, when the, when the work on, on Buffer board was going strong mm-hmm. Jim Gettis, John, John Morton, and many, many other people but they were working with Dave and there was a, like poem, poem from, from John God Godfrey Socks.

And the, the new version that John Mor, Jonathan Morton and Jim GTIs put together was Na was named The Blind Man. And and elephant and Dave read the poem at a couple of podcasts. And yeah, like I, you know, I would like to read it for him once again. Sure.

I am sorry. It's alright. It was six men of Stan to learn to learning. Much inclined who? So, so to fix the internet with market share in mind that evil of congestion might soon be left behind. Definitely the piman say why the link V view out for congestion occurs only when the bandwidth exceeds bout if only the earlier reverse were to, we would all live like gold, but count the cost.

If bandwidth had no end monument, say, but count the cost. If bandwidth had no end monument, say perpetual upgrade, cycle through madness. Right? Like that way. From the, this surf land, they, they're tried and true solution. Now, the parts, some packets are more equal, equal than they're the true lesser counterparts.

But choosing which is which presence new problems in all arts. Every pa every packet is sacred Christ. The data center clan demanding bigger buffers for, for their ever faster land. To them, a millisecond is eternity to plan. The end-to-end principles guides a certain band of man detecting. When congestion strikes inferring bandwidth, then aas or metals outcompete their algorithmic maven.

The is is o relation, is prefer explicit signaling, ensuring each and every flow is equally a king. If only all the other man would listen to them think. And so this man of industry disputed wild and wrong. Each in is in, is in each, in his own opinion, exceeding stiff and strong to each was partly in the right, yet mostly in the wrong.

So often in technology war, the disputants I been rail on in utter ignorance of which each other mean. And prayed about an internet. Not one of them has seen,

Jonathan: you know, I, I was thinking about it. My, my experience with Dave, I'm pretty sure I also ended up run it, rubbing shoulders with him. When the FCC was about to tell people they had to lock down routers that that actually happened right about the time the Tour Omnia came out. If I remember correctly. It was right in that same time period.

And I did, I didn't, you know, I didn't really get to know him then, but I'm pretty sure we did end up, you know, we were at least on some of the same mailing lists and cared about some of the same things. So yeah. And that I, I just, I went to my I, I went to my email address that I've had for the longest time.

I've never done this before. This is the first time I've done it. And I searched for Dave's name and now I'm, you know, I'm going back to like the oldest email that I've got that had him in it. And I see some from like 2011 when I was on the open. That's so cool. Open WRT mailing list. And he had something he was trying to add.

So, to it, you know, it just for, for so long, like we've just, we've been kind of in, in involved in the same things. And that's just, that's just neat to me. He's gonna be missed for sure. Right,

Frank: right. And and you, you, you have mentioned FCC, right? So Dave was picking up the fight. Mm-hmm. Like every time when, when it was really needed.

And, and Winder and others, like, we we're always ready to help. Right. Because like this, this kind of stuff like open source, it's really power to the people. And Dave, he was believe a strong believer in this principle in pretty much everything. And like the other, some of the other QOE solutions on the market, do they do DPI, right?

And we, we will never do that. Mm-hmm. And, and, and, and so like, you know, these are the things like guiding principles that Dave had. And like, I, I like to say that he never sold his soul, right? I, I don't have like anything against working for a big company. Right. And it's necessary, right? But there must be some people.

In the industry that are guiding you know, they are the guiding lights, basically lightning houses somewhere. And they are doing these like not, not so, not so well paid jobs, but they are, they are, they are actually changing the industry. And, and we all know that the, the meme of that one guy in Nebraska that's basically holding the, the weight of, of the whole stack, basically Dave was something like that.

For, for way way so many companies. Way too many companies.

Jonathan: Yeah. So that's funny. I use that meme all the time because we the XKCD comic, because we, we interview those people. That's what we're all about here on Floss Weekly. Dave was definitely one of those people. I, I kind of suspect that that was actually written about the NTP guy.

I. Because so many things work on NTP and for the longest time it was literally one guy. I think he may have been in Nebraska, I don't remember for sure. But yeah, it's, it is definitely true about Dave and, and you guys and, and a lot of other people that we talked to.

Herbert: Yeah. You know, it's funny, I went on the Den Labs, that's the company I teach Rust through Slack and me and mentioned that, you know, my friend David passed away.

Nobody had the faintest idea, who Dave taught was. I mentioned FQ Cuddle, they're all a bunch of Kubernetes nerds. And everybody's like, you knew that guy? You knew. And it's like it's like, wait, you know, I, I, I've told you that before, but you didn't pick the best time to be interested. But it's

Jonathan: oh,

Herbert: you know, it's fun.

You know, I've talked to a bunch of people, it's like. Not that many people knew Dave, but everybody used his stuff.

Jonathan: Yep, yep. He was, he was involved with other stuff too. I'm looking through my notes. He was, he worked for one laptop per child. He did obviously a lot of open WRT stuff. He worked on the, the C-R-W-R-T project make wifi fast.

As we said, he did some, some stuff with the FCC, trying to talk them out of their insane ideas all kinds of stuff. He was a, he was working for open access in academia. Lots, lots and lots of things that, you know, I think, I think most of us, sort of hacker types, we, we get that they're good ideas.

He was, he, he was, he was one of us. I don't know how, I don't know how else to put it. He's, he was our people. Right. And we'll be greatly missed. I. Alright. Is there anything that we didn't talk about either with, with memories of Dave or the future of Liberty, QOS or where it's at now that we, we should have, that I should have asked you all about?

And I know that's a hard problem 'cause you gotta think about all the things we did talk about and your list of things you wanted to talk about.

Frank: May, maybe, maybe a's AI stuff Hebert would, would you, would you think? Sure.

Herbert: You know, it's Dave was funny with AI when Che GPT-4 came out. He asked, asked it, the question we all ask, who is Dave Tu?

And it gave a pretty random answer. So we went over and tried Google Gemini and it said, Dave Tot is a fictional character from the Hit Jacket Guide to the Galaxy.

Jonathan: That's right. I remember seeing that.

Frank: Yeah. And he, he still has he still has that I think it's either like hero picture on the LinkedIn or Twitter x.

Yes. I remember when he tweeted about that, that that was, that was Chet

Jonathan: GT's understanding of him.

Frank: And, and, and basically today the former tourist CCEO asked about Dave Chet, GPT. And the answer was a really good, and he, he sent it over to me and I was like, oh, thank you. That's going well.

Jonathan: Yeah. Yeah. That's funny. Did, did Dave use any of the AI tools? I know some of us, so for example, myself, I am, I wouldn't say I'm morally against it, but like in practice, I'm pretty strictly against using AI tools for any of my stuff. I, I would like all of my work to actually be my work and I don't necessarily put that in anybody else.

That's just, that's where I'm at right now. Did, did Dave make use of any of 'em or was he kind of a, a purist as well? Do you guys know?

Herbert: A little bit. Now I. I use them because I love trying new stuff. Sure. I don't really let AI write anything more complicated than HTL form, but and that speak, because I could do it.

I know how to debug it, but it can do it faster.

Jonathan: Right.

Herbert: So Dave, I know Dave fired up Cursor, gave it a Linux kernel module, asked it to make some changes, and we spent the best part of a day laughing at what it was trying to do. That sounds about right. We struggled. We spent an afternoon once struggling to try and get emax to talk to AI code completion service that somebody, I think it may have been copilot, said that he, he could have for free.

So he was like, I wanna try this out. Mm-hmm. And his conclusion was not positive. We spent more time setting it up than we did throwing it away. It's kind, it's kind of funny. It. Autocompletes, HTML, really well, JavaScript and Python pretty well.

Jonathan: Mm-hmm.

Herbert: And it CSC the way I first did when I came from Pascal back in the nineties as a random stream of characters.

And that's what it autocompletes, you know, you just get this random barf of letters that might do something.

Jonathan: No, no, no. Ai. That's Pearl. Come on.

Herbert: All right. Once Pearl, I'm glad I don't have that job anymore.

Jonathan: Yeah. But fun. Alright, well that's actually a good segue. Let me ask each of you, as we wrap the show, what's your, what's your favorite text editor in scripting language with Frank?

Frank: I don't, I don't cult at all, which is really, really weird to thing to say on, on podcasts like this.

No, that's fine. And I, I, like, I, I, I. You know, the, the, the closest to did I ever get to something like that was, was that when I was trying to get my CCNA and there were some like, you know, command, non command line stuff. So I, I never actually did any programming yet. But I, I would like to try to understand stuff like more on, on a, on a deeper, deeper level.

However, like when I'm thinking about it, you know, some of those, the things that Robert Hubbard and Dave were doing, you know, I was not able, and I, I'm still not able to understand on that deep level. Mm-hmm. But I, I, I, I, I, I I consider it as some kind of advantage even in some parts of, of my, my job.

Mm-hmm. Because sometimes it's, it's a rabbit hole and it's easy to get stuck. Yeah. And, and to discuss. Things and you know, I will, it flies over me so I can focus on what, what it, what it delivers to the customer. Mm-hmm. So, you know that that's the story that I am, I, I'm always saying to myself.

And I hope that yeah. So basically I use like notes app on my, on my, on my Mac.

Jonathan: If you ever decide to learn rust, I know a guy that's pretty okay at teaching it.

Frank: Yeah, probably some, her, some Herbert, I guess.

Jonathan: Yeah. One of those Herbert guys. And, and you, sir, what is your favorite text editor in scripting language?

Herbert: Oh boy. I get asked this a lot and I learned the hard way. Never try and teach a major workshop with, with them because you'll spend more time answering questions about your text editor than you will actually getting to teach. So if I'm in public, I am using Rest Rover because JetBrains sponsor my Den Labs talks.

Jonathan: Ah, okay.

Herbert: If I'm teaching a private class, I'm using either Rest Rover or Visual Studio Code again, because I don't want the editor to be in the way.

Jonathan: Sure.

Herbert: When nobody's watching. I'm running Neo Vim with Rest Analyzer. Mm-hmm. And a ridiculously customized color scheme because I've finally found a color scheme that's easy on the eyes for eight hour sessions and mm-hmm.

I'm sticking with it. Yeah.

Jonathan: Yeah. Scripting language. And so here you have to ask yourself, does rust count as a scripting language?

Herbert: Not really. I mean, it, it compiles, it takes a while to compile. There is a crazy program there is a crazy project out there that lets you do shebang bin rust and have it work, but I wouldn't recommend it.

I actually, like, I actually like Python these days. Mm-hmm. I was a pearl guy for a long time. Robert has converted me to the Python gang, so,

Frank: yep. And, and, and, and I, I, I think I remember answer for, for, for Dave, and that was emax. That sounds right. And and, and Python, I guess. Yeah. Was Python, emac

Herbert: EMAX was his operating system.

He once told me he was on a little Chromebook and it took 20, it took two and a half hours to finish loading his customized EMAX setup.

Jonathan: That sounds about right. Oh goodness. All right. Lots of fun. Thank you guys both thank you guys both for being here. We appreciate it and it was good to chat with you both.

Herbert: Oh, thank you for having us.

Jonathan: Thanks for having us. Yeah. Glad to be able to do it. Alright. If you want more of me, oh, I didn't, I tell you what, we're gonna let, we're gonna let each of you plug something if you want to. I don't have a co-host. Normally I would have a co-host plug something and I, I don't have it.

So you guys are the co-host today too. So Frank, is there anything you wanted to plug before we let folks go?

Frank: AB absolutely. Liberty c io and there, there, there are all the links people need. There is a way how to contact contact us. Contact me. Mm-hmm. There is, I, I think there, there are our LinkedIns.

Yeah. So people can can reach me out. Yeah. And at Herbert,

Herbert: My most recent book, advanced Tons on Rest, went into final Beta yesterday. Should be actually in print and ebook print. I get paid a lot more for eBooks so interest Digital print. Digital print and is obviously what I should be pushing.

But I, I love physical books and if anybody wants to chat with me, I'll be at Rusko and this September in Seattle. Oh, cool.

Jonathan: Yep. Very good. Is there, and I didn't warn you guys, I was gonna ask this, but it just comes to mind. I, is there, is there some sort of nonprofit or like a a memorial fund?

If somebody wanted to give something in in memory of, of Dave, is there something you would recommend?

Frank: We don't know of any yet. Okay.

Herbert: We've been talking with Dave's family. Okay. They, they have a lot going on 'cause his mother, his mother also passed away recently.

Jonathan: Yeah.

Herbert: So as soon as we know, we'll, we'll make sure it's up on the memorial page.

Frank: Okay. Ca can, can I plug in Dave's mother? Sure. So Dave's, Dave's mother was a really good programmer at IBM.

She, yeah. And, and they also like to play a lot of music together. So they were like talking about technical stuff, but also about music. So she was, she was probably someone that really influenced his like development. Yeah. I'm sure that that leads, you know, him to, to this buffer boat world in the end.

Speaker 5: Mm-hmm.

Jonathan: Yeah. Interesting. Good stuff. Alright if we do get an answer on that, I will try to try to get the, the eventual article and such updated with that information. But we'll try to get it shared out. Again, thank you guys. Appreciate it.

Frank: Thank you, Jonathan. Yeah. Thank you.

Jonathan: All right. If you wanna find more of me, there is Hackaday.

We appreciate Hackaday being the home of Floss Weekly these days. I've got the security column that goes live every Friday morning and the articles in between things that I find interesting sometimes Buffer bloat, lots of times, other things. You can find that on Hacka Day. I've also got the Untitled Linux Show over at twit TWIT TV slash uls.

Make sure to check that out as well for all of your Linux news and tips and analysis. And we have a lot of fun there. We sure appreciate everybody that's here, both those that watch us live and get us on the download, and we will see you next week on Floss Weekly.