Start / Teknik / FLOSS Weekly

FLOSS Weekly

It's the OG podcast about Free, Libre, and Open Source Software, FLOSS Weekly! Join us each Wednesday as Jonathan Bennett and the posse of Co-hosts interview big names of Free Software, cover utterly ...

139 avsnitt • Längd: 35 min • Veckovis: Onsdag

Om podden

It’s the OG podcast about Free, Libre, and Open Source Software, FLOSS Weekly! Join us each Wednesday as Jonathan Bennett and the posse of Co-hosts interview big names of Free Software, cover utterly fascinating Open Source Projects you may have never heard of, and cover the news about software you use every day without even realizing it.

The podcast FLOSS Weekly is created by Hackaday. The podcast and the artwork on this page are embedded on this page using the public podcast feed (RSS).

Avsnitt

Episode 832 transcript

7 maj 2025 | min

FLOSS-832

Jonathan: Hey folks. This week I talk with Alexander Dunloy and Quentin Jerome about Kuna, which is an open source Linux threat monitoring tool and circle, which is the Computer Incident response group out of Luxembourg. It's a really fascinating conversation about the tool and also what All Circle is up to, and some talk about where the threat landscape for Linux is these days.

It's a lot of fun. You don't wanna miss it. This is Floss Weekly episode 832, recorded Tuesday, may the sixth. Give yourself a medal. It's Time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something very fun today. We're talking about, well, Linux.

Not to any great surprise for those that know me, but Linux security, again, not a great surprise. It's something I care a lot about. We're gonna be talking with Quentin Jerome and Alex Uhno about, well, a couple of things. The Kuna Project. And then circle C-I-R-C-L in general which is all about security, Linux security in particular, but lots of security things, trying to keep computers safe.

It's gonna be really interesting and I am gonna get to learn a lot about these projects as well, because I don't know a whole lot about them yet. We don't have a co-host today. That is fine. Again, it's on me. I just put putting things together at the last minute. I was, I was sometimes the do your homework on the bus, on the way to school type.

And it shows every once in a while, but that's all right. We're gonna have fun today. So we're gonna go ahead and bring them on. And so first off Quentin and Alex, welcome to the show. Thank you so much for being here. Welcome. Thank you. It's good. It's good to have you both. So who I, I know one of you is sort of representing Kunna and the other is sort of representing Circle, which, which is which.

Quentin: So I, I'm

Jonathan: representing Kuai

Alexandre: and I'm representing Circle. Yeah.

Jonathan: Okay.

Alexandre: Even if content is working at Circle,

Jonathan: that, that's how that tends to go sometimes. Yeah. And, and that's fine. So what is, what is Kuai? Why would somebody use it? And you know, what's the, what's the problem it's trying to solve?

Quentin: So, yeah, it's a problem that actually that showed up recently with the limitation that actually people are seeing with traditional antivirus for instance.

And basically the need to have security visibility on their, operating systems. So they're not like a, a lot of free tools for doing this on windows there was this project that has been kicked off by I don't remember Marinovich like a while ago. And basically they announced some time ago Linux version I don't remember exactly, maybe four years ago, something like this.

And I was basically waiting for this for very long time. And when the project like came out it was super weird because couple of weeks after, like the main developer left Microsoft, and I was also very disappointed by that project because they. It took like a lot of ideas from Sessman, which was really, really great.

But they, in my opinion, didn't adapt it properly to the UX world. And that's basically how I decided to start developing Koai. It is basically to have like the system one equivalent on Linux. And it brings a lot of visibility on the, on the Linux hosts and a bit more than what currently exists like ODD and and so on.

So probably you can do some stuff that could I does with ODD, but it's gonna be really, really hard to do it. And it, it's really focused on security visibility and basically threat hunting and yeah, threat monitoring. Basically on the, on the host, it's not for monitoring performance. It is not like for diagnosing C scores or thing like that.

It's really focused on threat hunting, threat monitoring, and all these let's say sphere.

Jonathan: Mm-hmm.

And, and Alexander what, what is circle and how does that fit into this?

Alexandre: Well, that's a long story, but Circle is like US cert. So we are the National C in Luxembourg. So we do incident response, but we have a small particularities compared to other C worldwide. We do open source from the early beginning of the CC, we decided to do open source.

We. Maintain and manage 17 open source project. Maybe the most well known one is misp which is a trade intelligence platform that you might know. And we do other open source project, but from the early beginning of setting up the third and giving services to cities and organizations and members in Luxembourg we decided from the early beginning to go open source and why?

The reason is very simple. We are funded by the state and it's public money, so we decided this must be public code. And for us it's, it's I would say kind of DNA of the third. So we do incident response, but next to that we do software engineering and software development, which are open source project.

And. Over the times, we did add additional tools and so on. And then we were missing EDA or ING platforms for Linux. And then content joined the team to work on this and to have a complete open source solutions for trailing on Linox Ker.

Jonathan: Hmm.

Interesting. So let's, let's let's dive into Kuna just a little bit first.

And I'm, I'm really, really curious, like what, so obviously it's for, it's for threat hunting, but there's a lot of things that could sort of fall under that sort of category. What, what are the, what are the tool, what are the techniques that Kuna uses to try to find something on a Linux system? So are we, you know, are we hashing binaries on the disc and comparing them to known good values?

Are we, you know, investigating what's running in memory? Is it all of the above? Is it trying to be sort of a, do all the things, all the bells and whistles, antivirus? What, where does it fit?

Quentin: So the idea is really to give actionable data to the, the, the person actually wanting to use Kuai. So it's, it's.

Really exposing the maximum of information we can so that people can actually customize their detection, they can create their own detection rule based on IOC. So IOCs indicator of compromised, this is usually something which is shared across. People who got infected maybe through mist, like the, the platform Alex just mentioned before.

Mm-hmm. Or from various sources open source intelligence for instance, or things like that. And basically people can let's say bend Kuna to their needs. So that, that's where it differs from all the, the other antivirus where basically you have a black box and you cannot do much with it.

And for what it does, of course it hashes, binaries not much memory investigation. But I mean, all this is a matter of implementing features. So Sure. If people are asking for these kind of features, maybe I can, I can implement those. Mm-hmm.

Jonathan: And, and so I, I've done some, I've done some research into this sort of thing.

I'm trying to remember the name of it. There was a, there was another sort of old school open source binary checker. And the main thing that it would do is it would hash binaries. It would store the binary hashes on the disc, and then it would come back later and do the same thing and compare the two.

Ah, I wish I could remember the name of that project, but it always seems with that one with what's that?

Alexandre: Or leads or Ed,

Jonathan: that might, might have been it.

Alexandre: There are many of them. There are several, yeah, several that we basically ask the Aries, keep that as a baseline, and if something deviate, then you have a new ashes and then you basically get a popup and think, eh,

Jonathan: the thing that always confused me about those was that it was hashing the binaries on the host, storing the hashes on the host, and then hashing the binaries again on the host and doing the comparison.

And you know, it's, it's, it's like one of those memes where you have a president of the United States giving himself the, you know, the Medal of Freedom. I did so great. It's like, yeah, of course this computer is clean. It, it always struck me as being a, a bit of a problem. Really it's a problem of establishing trust and it, doing something like Kuna, it's gotta have that same problem.

Is that an issue that you've thought through? Are there any solutions in Kuna or that, that you've thought of?

Quentin: So the goal of Kuna is okay, of course you can like run it on your host store, the logs on your host. Mm-hmm. But mainly the use case that people are going to, to implement is forwarding the logs directly mm-hmm.

To another log storage or something like this, where they can of course check later the logs. And what is also very important is depending on the log policy they have implemented in, they can. To do some sort of incident response because it gives like a lot of information and gives information about parent process, child process.

And basically you can reconstruct all the chain of execution happening on Host, which is very valuable. And yeah, basically that's, that's the use case. Most of the people using Kuai won't, won't just let it run on their laptop and not doing anything with it. They will very likely forward the logs.

Jonathan: Mm-hmm. Is, is, is Kuai also the piece that runs on the, the more trusted server and does the log checking? So like once we, once we get data from Kuai and I'm, I know I'm gonna ask about what other data sources you have, but I'm just curious about the whole, kinda the whole tool chain. Once you get Kuai, collects all this data and forwards it on somewhere, what do you do with it then?

Is that, is that part of the project as well?

Quentin: Not really. Why? Because, people tend to have, like, all this process that you're describing is kind of secret sauce of every organization. So it, it is very hard to provide one fit all solution for this. It's much more easier, like to say, okay, I give you the tool.

As I said earlier, this is something you can bend in a lot of ways. And that was the, the original goal of it. Probably it'll fit your use case and you can very likely adapt it to what happens in the, in the back in the backend. And yeah, I, I've got like several questions to talks and things like that.

Like why do you don't provide another format? Why do you don't provide yeah. A filtering format and so on. It's basically like everyone will come with another approach to store their logs and the information they want to keep. Mm-hmm. And yeah, this is not something I really want to go into because it's like endless and Yeah.

Basically doesn't bring much features to the tool. It's just like formatting. So,

Jonathan: yeah. Is this, is this actually something where circle fits in, where you've got sort of a database of known problems and you can do comparison with, this is the hash, you know, this is the hash that I found on my computer. Let me go, you know, give it to circle and say, is this a malicious binary that you're aware of?

Alexandre: Yeah, it, it is. And we have even a project and like Kan was mentioning a lot of projects are basically plug and plug and play and kind of of module base. So that means is fit in a complete framework where you'll do for example ING and so on. And for example, we have a project called Ash Lookup, which contain a huge database on all the, I would say, known binaries.

So that we, so if you know, the NSRL database from list is actually that extended with plenty of operating systems, especially Linux 'cause is, is really focusing on Linux. So that means when we have an incident and someone is providing an incident case with evidences like. Of Linux servers or some binaries and so on, we can automatically check, okay, do we know that binary is something that completely new?

Then we can even use Kuna because Kuna has this functionality that you can run in sandboxes. So Kuni can act as a kind of sandbox. Maybe a counter want to, to talk about it later on. But the idea is if you receive a binary that is suspicious and we receive it for a specific case, then we can go through the sandbox of koai, do the analysis, look at all the activities from the binary, so it's like dynamic malware reversing, and you just analyze the binary and then we can check if it, this one is known.

And we can even check if in the process of the binarys, if it's co comparable to, I would say one of the well-known binarys that we know and so on. And then the, the plan that we have too is to actually have rules that we can even share with the community to automatically detect I would say bad behaviors or things that are deviating from standard behaviors.

I would say.

Jonathan: Super interesting. What about, and this is, this is kind of a weird question, but it makes sense when you think about it. What about detecting known Windows, malware and that really, you know, it doesn't make sense in Linux until you start thinking about a, a Linux file server and then all suddenly maybe you do want to detect Windows malware.

Is that sort of in scope for, I guess with Kuna it's just a matter of your hashing files but with something like Circle, or is this where maybe we need a plugin to be able to connect Kuna to a virus total and do uploads there for new files?

Quentin: That could be possible. Look at that. He is like, oh, that's

Jonathan: an interesting idea.

Quentin: Yeah. Yeah. To be honest, I would instead of yeah, putting this on the hosts, I would maybe put this again on the, on the backend side where basically you can, again, do whatever you want with with your data. Mm-hmm. And why? Because, of course life checking hashes on vt. To be honest, would probably be the bottleneck of Kuna.

Yeah. Because like doing a lot of HTP requests while you are dealing with almost real, real, real time data, it's like super slow. Mm-hmm. So, yeah. This is something I would offload to, to a backend for instance, because there is no time criticality to this.

Alexandre: Mm-hmm. But it made me think of an idea. We, we talk about Windows, Linux, IES executions so maybe an execution with wine within Linux and wine is monitored to would work too.

So if you want to do active rehearsing, I remember some project. Back in the days where wine was used as a way to do windows monitoring of binary through Linux, maybe now we can have Kuna monitoring wine, which is basically executing a Windows binary.

Quentin: Yeah. Yeah. I don't know if it works.

Alexandre: Definitely an interesting idea after this, this, this stream, we can, we can just make a test, I think.

Jonathan: Yeah,

Alexandre: that's a good idea.

Jonathan: Yeah. So I, I am always fascinated by the way root kits work and a root kit for those that don't know, I think most, most of everybody does, but for those that don't, a root kit is basically malware that goes to some lengths to hide its existence.

So it, it's trying to hide from some anti-malware software. It's trying to hide from the cis admin and I have read about some crazy things that root kits will do. Where they're like live patching glib C to where when they find themselves in a list of processes, they just remove themselves from that list of processes.

And in some cases, these root kits are extremely difficult to find and root out. Does Kuna have any sort of anti root kit technology built into it? What is like, that's a difficult problem. What what, what is the solution to that look like?

Quentin: So I, I guess it depends on the type of root kit, because there is, which is called like user land root kits, which is basically a mm-hmm.

A malicious library that you manage to inject in all the processes. And that's probably what you were mentioning. Basically, they just redefined the CS calls or the function of the lips C and they filter when, when it's, related to the process the malicious process, for instance. So this is like completely caught by UNA because that's super easy.

Everything happens on new. Mm-hmm. And something we didn't mention is that una runs, I mean, all the interesting and all the events are coming from the kennel land using EBPF. So I don't know if you are comfortable with this, but it's basically the way the Linux kernel has found to allow developer to develop kernel components.

Mm-hmm. But basically without all the bugs we have good example of what happened to CrowdStrike, like one year ago. Yeah. How could we forget? Yeah. This would be hard to, to happen in EBPF because it comes with a nice code verification let's say step. So you cannot load any code in the, in the, in the kernel.

Everything gets verified and. Code is pretty safe let's say. And yeah, so this is how Kuai works thanks to BPF, and maybe there are ways where Kuai could be tricked, but I don't really see how. Mm-hmm. Because if, like the root kit is a kernel module kuai would see it and I'm not aware if it's like possible from another EBPF program to interfere on another.

So for sure this would be something really interesting to test. And yeah, if one of the guys in the community or one of the guys listening is. Would be happy to, to try this. I mean, let's go do it. Mm-hmm. And don't, don't hesitate to open issues after that.

Jonathan: Sounds like there's some, some really interesting sort of cases here where a, a security researcher that really is into Linux security could go heads down for a while and, and really explore all of this and see how that works.

I, you know, I've, I've read about some EBPF root kits where people are putting malicious code, running malicious code inside that, that extended Berkeley packet filter. I must admit, I've never actually messed with BPF at, at this point. I, I know what it does. I know it exists. I've never written any code and tried to make it work there.

So, you know, I'm, I'm not as familiar with it as I could be, but I, I, I know in a couple of places there was some, some real trepidation about the idea of malware running BPF code. But it's really interesting that, that it's a good place also to run anti-malware. Processes because it, I guess, I guess as BPF code, like you said, it's not going, if there's a problem, it's not going to crash the entire kernel.

Like someone, someone else's module did

Quentin: I know, to be honest. I'm like still wondering why, like, malware out. Yeah. Don't focus more on EBPF because Yeah. You can do like crazy stuff.

Alexandre: Yeah. Are you giving advice right now?

Quentin: It's just a matter of time, you know. Yeah. That's true. So what, but, but for sure it's not easy to develop, so Yeah.

Probably easier to develop a ker module.

Jonathan: True. Yeah. Right. Well, I. Probably what we're waiting for is for someone to, to release the first open source malicious EBPF example, and then all the script kitties will copy that into their own code and the floodgates will be open.

Quentin: I think

Jonathan: it already exists.

Yeah, it probably does. Something very basic. Mm-hmm. So what, what sorts of things you, you guys, obviously you have your pulse on sort of the, what's going on with malware in the Linux space. What, what are you seeing and what, what is sort of on the cutting edge? If it's not EBPF, what is, what's the things that people are actually doing?

Alexandre: I might disappoint you a bit, but a lot of of malware that we, we see are not so interesting. And the thing is, is quite interesting in the fact that what kind of things they target. We recently track botnet targeting only high performance computing infrastructures. So this one is well done.

We are tracking them for some months. We even exchange the C two server on a regular basis and so on. And we were wondering why they always target HPC server because it's a lot of investment, it's time and so on. And at the end it's only for doing crypto mining which is a bit disappointing. I was expecting things like that and you end up with that, it's like, oh, guys, I mean, do better.

But technically it's, it's, it's a, nevertheless we see some rare case which are, I would say more targeted or very targeted where it's more current modules or specific user and software that are well and used for one or two targets. Mm-hmm. Some of the. Blog is Wonder sometimes mention those one as mentioned, some of them on a regular basis.

Mm-hmm. So there are some Russian collectors, for example, that are very good at writing some exfiltration mechanism through ethernet frames, things like that, and that are running on Linux. And so there are some interesting case, unfortunately, I would say, compared to Windows. The malware space on Linux is really separated into like the one doing crypto mining.

The one that are doing kind of ransomware on Linux servers. Mm-hmm. So locking your servers and then you have to pay ransom, so classical ransomware and stuff. And then you have the espionage exfiltration cases, which are, I would say a minority. So that's three kind of space. And then you have all the tools for leaders.

So that's all those client for DDoS, DDoS management collaborative DDoS, because nowadays those DDoS activities are kind of collaborative. So you have to download the client for Linux, run on the VPS, and then you get points and those points give you back flip currencies. So it's kind of thing that we, we, we, we have seen.

So that's means the kind of malware for, for for line that, and, and content did an interesting repository which is available on the, on the gi, on the Git repository, which it takes some random samples of. Not always the same kind of malware, but different type of malware. Run it into the sandbox with Kuna.

And then we have we don't mention that, but with Kuna you can automatically generate with a QM setups pick up the Gson files with all the activities of the malware boundaries. A nice graph to an export for me and things like that. So if you want to. Investigate malware and so on, on Linux.

It's a perfect tool for doing that. And then you have a complete setup. You have a sandbox repository on, on the project explaining how it works. It's a blog post about it. Or to use it, it's super easy to do it. And if you have like suspicious banners or if you just want to reverse the Aries just for example, another thing that you can use it for, it's to create rule set.

So for example, you have Aries, you have to create rules for I know, something like that. Mm-hmm. Or you can use for doing the execution and so in the sandbox and see what are actually, the banner is doing that way of, of maybe another way of using ry.

Jonathan: Yeah. Interesting. I tend to play on the, the Red Hat side of the street.

For me, it's, so, it's all Sy Linux, not App Armor. But I could, I could definitely, I could definitely see that being useful because my, my usual workflow is try to run the program. Oh, it failed. Okay. Let's go see which thing failed first. Okay. Try to run the program again. It failed again. What's the next thing that failed?

It's a very tedious way of doing it. So this, this sounds like a definite upgrade. Now what about, what about, say if we had an arm binary and we wanted to try to use Kuna to look at that but, you know, maybe don't have an arm machine handy. Is there, is there some way to, to use Qai with cross-platform emulation?

Quentin: Yes. It is possible because QAI comes with, for the moment it can compile to two architectures. So it, it's 84 64 bits and a arch 64. Mm-hmm. Which is like a 64 bit arm. So yeah, if you have like tro binary, you can do it. Mm-hmm. I don't provide yet other binaries to test on let's say older, the arm architectures.

Yeah. Interesting.

Alexandre: You can even still run arm binaries on Intel one using QMU.

Quentin: Yeah. Yeah. Yeah. Makes sense. Yeah. But we would meet like the, the activity because we need to, to monitor the, the arm kernel and yeah. I, I, I need, I need like binary for this.

Jonathan: You need, you need QI to run inside the vm. No, not on the outside.

Would there be any way to, to do that? To, to run it? I, I don't even know if, if Linux and EDPF and the way virtual machines work right now I don't know if that's even possible, but would it be possible to monitor from outside the vm?

Quentin: Yes, this is probably possible. But I don't know exactly how it is done.

And basically many, sandbox platform are offering this approach, which is, which is called like hyper hypervisor based sandboxing. So there is no agent on the sandbox, and they just like monitor the, the instructions executed by the, the sandbox. Mm-hmm. And you, you can definitely do it. But yeah, that's a completely different topic.

It's more for, it's not the way Kuna is, is architected. Yeah. This is like for dynamic analysis or, and maybe like, yeah. Advanced dynamic analysis.

Jonathan: Mm-hmm. Have, have you discovered yet malware that looks for the presence of Kuna?

Quentin: Not yet.

Jonathan: Okay. It's, it's inevitably coming. Right. As the pro, as the tool gets more and more popular, it's inevitable that somebody's gonna write malware.

And one of the things it's gonna check for is can we see if there's a Kuna instance running anywhere? Yeah. It's properly, yeah.

Alexandre: It'll be a, a success factor I would say. Yes.

Jonathan: Yes. You really, there there's, depending upon what you're doing, there are different factors of success. So when you're running some sort of messaging platform, it's when people start using it to spin, to send spam, and I think maybe running an antivirus platform, it is you know, I say antivirus very, very broadly here, but it's when malware starts checking for the presence of your tool, that's when you know that you've really made it.

Yeah, I will be happy

Quentin: the day.

Jonathan: Happy and very grumpy, I'm sure.

So let's see. I'm curious more about what EBPF lets, lets you do again. I know, I know very little about it. Is it, does it basically give you all the same tools that you would get from writing a kernel module or is like a, is it a, a more limited set of things that you can do to the kernel?

Quentin: Yes.

It's more limited. It's more limited set because you have a limited number of instructions you can execute in each probe you are loading in the kernel.

Jonathan: Mm-hmm.

Quentin: So this is the first factor. And then you have, you don't have access to all the KL APIs. Mm-hmm. So you have to use one of the available APIs which is exposed by the EBPF framework.

And yeah, there are a lot of other limiting factors like the fact that you cannot allocate unlimited number of memory and so on, so that, yeah, sometime you really have to think a lot to write your EBPF programs. And that, what I was mentioning earlier with like malware author developing malware is that you really have to be motivated to write EBPF because I spend like countless hours debugging my BPF programs.

Jonathan: Mm-hmm.

Quentin: Because as I said, you have like the EBPF verifier which actually checks your program doesn't has doesn't have side effects. Mm-hmm. And yeah, this one sometime kicks in like for super strange reasons. And you definitely have to find it. And yeah, the additional difficulty I had, I guess is because I was I actually, I, I am using the IA project which is a compiling rust directly to EBPF Bytecode.

Oh,

Jonathan: interesting.

Quentin: And so this is of course very interesting because you can write both the user loan program and the EBPF program in the same language. Mm-hmm. But the other downside, let's say, is that it goes through an additional to the rest, compile, compiling, let's say chain. And sometimes it produce like weird stuff and you need to, to find, to find out, to find out why this happens and fix your code.

Compilers sometimes

Jonathan: are weird. Yes.

Quentin: Yeah. And I am was like kind of in early development when I first used it. And in the beginning was really, really hard to to make everything work. I mean, this was a combo of me being completely new to BPF IA, being like a new to and yeah, some features were not were not there.

And yeah, it was really, really hard in the beginning.

Jonathan: What is, what is debugging EBPF in particular with this kind of rust tool chain? What does debugging look like? You know, can you, can you run it in VS code and hit F five and single step through it to see what's going on or because it's running in kernel land, is that just not an option?

Quentin: No way. No. I mean, maybe there is something, something that exists. I guess I've seen some projects, some kind of emulating DBPF byte code and so on. Mm-hmm. But most of the time what you get is that you try to load your program in the kernel and then it like, provides you a huge it's not a stack trace.

It's basically the instructions that have been executed by the vm. Mm-hmm. And then you see where it fails. And then yeah, just you analyze this and you figure out where the issue is.

Jonathan: I, I imagine you, you do lots and lots of adding the, whatever the BPF equivalent of print fs do a lot of print f debugging with this.

You can, you can. That's what my debugging sessions tend to look like.

Quentin: Yeah. That's what that's what happens all the time. So it's called BP

Jonathan: fk?

Quentin: Yes.

Jonathan: Yes, I am. I am, I'm concluding that we need an EBPF debugger in the kernel. Let's get on that guys.

Fun stuff. Okay. So. What, what, one of the things that we've seen sort of in the security world that maybe ties into this that's interesting is security appliances being the target of, and this obviously is when we're talking more about nation state actors and not people just trying to mine for Bitcoin.

But you need actual security appliances being the target where someone will find a way to, you know, say a VPN box that's sitting on the edge of a network. Well, you know, there's an unauthenticated RCE and you can load code onto it. Do, do you guys kind of have insight into that and I'm also curious, is there anybody sort of in that space that's maybe looking at running Kuna as part of their security offering?

Alexandre: First I think they have really, really bad news for you. The kernel version used by those appliance are usually very, very old. I don't mention even t or Fortinet for some of the incident that we have on those devices we had kernel that are not actually supporting EBPF.

Jonathan: Mm.

Alexandre: So. Be, I would say for those kind of operators or vendors unrealistic to run even program.

Nevertheless, I would say, and if someone from the vendor space is listening to your stream, I would say it would be a good in advice because then you could stream the log back to a CM, for example, from the customer perspective, and they can have a better visibility of what's going on on those devices.

And I think it's a good point. We were always thinking as this says super hard appliance box, but if you look carefully, they are not super hard at all. They're running a pearl program somewhere or an open library from the, late nineties. So I think there's an opportunity there. No, the question is, are the vendors creative enough to, to do that?

I'm not sure.

Jonathan: Probably not yet. I, I very much enjoy reading the the watchtower writeups on the things that get found in those appliances. And to, to the vendor's credit. There are a few vendors in particular that have actually stepped up and are trying to fix things and turn on some hardening features and use code that was, you know, been, has been updated since the nineties.

But it is, it is quite hilarious to read the the things that, the cruft that they find in there. Sometimes the book post are incredibly

Alexandre: good. Yeah.

Jonathan: Yes, yes, it is. It is quite entertaining. So what, what is the oldest colonel that Kon I will run with then?

Quentin: Yeah, it's not very old. I'm sorry. It would be 5.4 for the moment.

Jonathan: Okay. Well, I mean, we're in the six dot, we're in the six dot series already, so it's not, it's not that big.

Quentin: LTSI guess is six point 12 something like this.

Jonathan: So nobody you know, you know what that means. Nobody should be running a kernel. Too old to run Kuna just shouldn't be doing it.

Quentin: You can do it, but you use ODD for doing such kind of monitoring. Hmm.

Jonathan: Are, are people taking the Kuna output and throwing it at LLMs to, to look for weirdness that, that you're aware of? Are you doing it? Are you doing it yourselves?

Quentin: I don't personally, I'm like super busy on the development, but I know like one friend of mine who is like throwing the logs into LLMs and generate detection rules for instance.

Mm-hmm. So, yeah, I guess it's something that people are doing.

Jonathan: Yeah. I always, I always found that to be an interesting idea of like, if you can get a baseline for a system and you know the normal things that are gonna be in the logs and the normal things the processes are gonna be doing. And if you have something just smart enough and we're at the point now where LLMs are just smart enough to do so, where they can look at it and say something is really different from the baseline.

I always thought that was a very interesting, like a useful thing. Even if it's not malware, it's still useful to know that my system is in a much different state than it's ever been before. Somebody might ought to take a look at this. Sounds like it's a service that somebody could offer. Maybe, and if they do it and it's not you, they should share some of their income with the Kuna project.

That's a good question. So you, do you get to do you get to work on Kuna professionally? Like is it your full-time job? Yes. How, how does, how did that come about?

Alexandre: Alex, do you want to Yeah, I can explain the Sure. Funding aspect. So as a search, we we basically develop open source tools, but the first reason why we develop the tool is for ourself.

So we develop open source tool. Will be used by us. And basically we use our own budget to develop those tools.

Jonathan: Mm-hmm.

Alexandre: The nice thing is more you develop tools more, you start to have users and those users want to contribute and form those projects. And for example, is funded through two different stream.

One is our own budget, but one is a project called T, which is an European project for soc for security operation center. Mm-hmm. To basically provide a complete tool set for operating a SOC with open source tooling. And we get funding from the European commission through the debt program to to work on it.

And it's a three year project and we. Circle. We do, we have three stream of of funding. We have those two streams, so that means internal national funding. We have European funding, and then we have private public partnership funding, which is basically customer. And for some of the open source project that we operate, we basically have organization that want to have extension, things like that, and we get funding out of it.

So that's how we found our project and we were quite successful on that because we're quite well known on the different tooling that we were doing. And people come to us and say, oh, by the way, we would like to have an extensions. Could we fund a part of it? Or sometimes even people provide in kind contributions.

So for some of the project that we have, we have even developers pay by companies. That are contributing to the open source project directly. So it's very common for the Linux kernel where you have, for example, people from, I know Intel, Google, whatever, working on the kernel, but it's the same for security space, for security open source tools.

We have organizations that are using our tools. One extensions. They sometimes are able to directly fund us, but sometimes they say, okay, we have good developers will you mind to join? Or this developer joining your core team for the next two years to work on the project. And so that's quite successful.

And for the past year that we were doing it, we get, I would say long term development and we get a different stream. So if one of the streams stop, we are able to continue to get funding from those different mechanism. Mm-hmm. And that's why we, for example K was working on this project because he joined for the European project, but at the same time he joined.

The group to develop all the tools too. So the team is quite smaller. So in we are, we are in, in my team we are 16 persons. So that means we do a lot of things, incident response and software engineering. Mm-hmm. But we are able to get funding through this different way of finding the project.

Jonathan: And so Circle is the incident response team based out of Luxembourg? Yes. Ah, very interesting. Okay. I'm, I'm curious how this, how, how, how part of this works and that is CVEs. Do you got how They're both laughing at me. Oh. How, how does it work now? There's some history here and that is that the, the, so the CVE system is sort of US-centric, at least it's based outta the us.

NIST works with it and MITRE works with it both. One is a part of the US government. The other is a weird entity that's halfway US government and halfway private. And you know, in the past couple of weeks there was, there was news that Mitre was going to lose its funding to do the CVEs. So I'm, I'm, I'm very curious, like, is, is Circle A, can you guys assign CVEs?

Do you do CVEs at all? Do you have your own system? Like, I don't, I honestly, I don't know very much about how that works outside of the us.

Alexandre: Okay. So, I dunno if you know the backstory, but so we develop a tool called Vulnerability look Cap. Mm-hmm. Which is an open source tool for vulnerability management and CVD process.

Jonathan: Mm-hmm.

Alexandre: According to vulnerability disclosure process. So we started at project one year ago, and this one is a backend software for the European vulnerability database. Mm-hmm. So the A UVD during that development process, we discovered that we were missing an autonomous way of assigning IDs. So we started a project called GCV, so it's Global cv.

So if you go to global cv.au it's a project where we basically assign id, so it's like e and r for assigning ID to organization that are able to get IDs. Why we did that, we did that some months ago. It was for the Vulnerability Luca project, but then with the recent news news, we had a lot of discussion at European level for it.

What should we do there? And then we say, okay, but the model that we developed is backward, compatible and, and forward compatible with the CV and CNA locations. Mm-hmm. So that's mean us as we can assign CV as CNA or we can as a GNA assign the cv, which is kind of global CVID and if later on the CVID are saying is fine.

So the GCV project that we started was just in the middle of this crisis from the CV program. So for us it was like, timing was like bit strange, but we was perfectly fine. Mm-hmm. And the European Union, and especially Anya, which is the agency, so you have to see Anya like Csar in in us which is the agency for the European Union regarding cybersecurity, did this European vulnerability database, which is no legal requirements for Europe networks.

And then we basically provide to them the software backend for that. So vulnerability lookup, which basically pluck to GCV. So something that we designed as an open source project, like more than one years ago. Like was like more for use case? No, is growing to a, I would say European use case. Mm-hmm. So if you're curious, it's vulnerability lookup.

It's vulnerability lookup.org. And you have, it's an open source project again. Again. And we have an online version. So you have all the vulnerability and then we can. Collide and correlate all the different together. Mm-hmm. So the timing was like, okay, and know the project is started. The UVD database was like announced just at the same time, just by accident, because technically it was foreseen to be released on February, but that's a long story.

So yeah no, the, we are circle, we basically operate that structure called GCV, where you can ask for an id and that's basically basically it. So even if it's CV continue to work as programming, that's great, but even if something is going crazy after the 11 months given to base C with the additional budget mm-hmm.

If something goes wrong, no, we are completely dependent and we can continue to operate as we wish.

Jonathan: Yeah. And that, that's only, I would, I would say that's only reasonable that the rest of the world would want to have a, a way to manage vulnerability to track vulnerabilities that's not quite so US centric.

And I don't say there's a political statement, but just as a general security statement I think certain news agencies made more out of the funding thing than probably should have been because apparently this is not the first time this has happened and not the first time. Exactly. Yeah. This guy, this guy wasn't falling the last time that Mitre had to figure out how to fund this.

But anyway, that's, that's sort of neither here nor there. Looking at the Circle website, I do also see something that really fascinates me, and that's the circle passive DNS What is, what is that I.

Alexandre: That's, that's, that's a long term project that we have. So the passive DNS is, is a way to build a directory of all the DNS response that you collect.

So the model is a following. You basically put sensors at various places. Could be internal sensors and so on. Mm-hmm. Or external sensors where you collect only the response, the DNS response. So when you query, I don't know, google.com, you get back a response. We see a records, CNA records, and so on.

What we do with the passive DNS, we basically have those sensor, we collect that in a ba in database, and then we have a historical database of all DNS record. What is useful for that? It's useful for threat intelligence. Mm-hmm. So we get an IP from a, I don't know, phishing website. We enter that IP and we can find back all domain associated that are related to that IP addresses.

We can find back the historical one, first time we see it. Last time we, we saw it. Mm-hmm. So it's giving a lot of, of meta information about the domains and so on. So we operate one for the past I think more than 10 years now. Our passive DNS is not commercial, so trusted partner can get access to it.

So if a soc a third want to have access, we can give you access. And, the passive d ns evolved a bit because as you know, for example, the TLS standard evolved. So the SNI record in the TLS start to disappear. DNS server note concentrate on, you know, main global DNS provider. So to start to do monitoring for DNS packet and so on is becoming more and more difficult for CODOing this, this kind of collections.

Nevertheless we found different way to do collections on, on passive DNS one, which is interesting. We operate different services online. Look Pandora and other things where people can test URLs and so on. But by doing that. This is doing the, the feeding of the passive DNS collections. Mm-hmm.

So if you have someone's looking for a phishing website and so on, they do automatically the the resolving and the resolving is going back into the passive DNS. So we have a pretty large database of that information. And for example, if we connect back to Kuna, if we have a malware binary running Kuna, doing a DNS request, then we can pivot from the DNS request from k.

Into the passive DNS finding back the infrastructure of the, maybe that is basically hosting different C two on the different domains. And the passive DNS is really an angular piece to be able to pivot from one place to the other place automatically. Yeah, absolutely. And no, nowadays, I think there's no, so many passive DNS database and passive D ns database are very localized because if you have a passive DNS collection in us, it'll be US centric.

If you have one in China, it'll be Chinese centric. So that's something to keep in mind when you do a passive DNS usage. Maybe you can call multiple one, and you have different, I would say, view of the world regarding a DNS request and so on.

Jonathan: Yeah. If a, if a security researcher decided that this would be something that would be useful to get access to and contact you guys, is that a possibility?

Alexandre: For sure. We, we give access to many research why we don't need put it public. It's, it's it's kind of per, per, per tool for even nce. Mm-hmm. And collector might use it for, for that. And that way we don't make it completely public.

Jonathan: Yeah. Yeah. That makes sense. There's been times that I've wished I had access to that, you know, you, you have a domain name and you're trying to figure out like, who are these guys and who did these guys used to be?

Well, I can, I can pull their IP address now and do a, who is on it. But I'm really curious on six months ago, what IP address did that resolve to? And let me do a, who is on that? And it's very, very useful. I've, I've literally wished for this service to exist before. Sure. So I'm glad that it does now, what, what is the SB Sanitizer? Yes, I'm, yes. I'm at Circle Do lu and I'm looking at the, your services. And I am, this is the other one that I'm not quite sure what is, what's a u SB sanitizer and how is it different from just running DD over A USB stick?

Alexandre: Whoa,

Jonathan: that's,

Alexandre: I can explain the background where, when it started.

So it's even a small story. So we had a representative from the government in Luxembourg going to some countries like China or whatever countries. Mm-hmm. And they receive USB key from partners and. When you are diplomat and you have a USB key, you somehow need to connect and see what's what there.

Well, at the same

Jonathan: time realizing that that thing is potentially nuclear radioactive waste that you don't want anywhere close to your computer. Exactly, yes.

Alexandre: And so what we did is it's a software, so we have two versions. One is a library, one is a, a software running on the Raspberry Pi. Although it works, you basically enter one USB key, which is a suspicious one.

You enter a second USB key and the software inside will start. So you have a little music running. When the music is stopping, the conversion is done. Super straightforward. That's great. Anyone can use it. So the, the idea is to convert content into some I would say payload disinfected content.

So for example, if you have a PDF, it's converted into images. You can quickly look into it. And so it's not running with antivirus and stuff like that. Can I, is not running on that one, for example, but could. But it's, it's actually converting the files into, I would say less risky files. And then people can have a look at it, plug, unplug the destination keys, and then start to to look and if.

This berry repair is compromised, then you can, you can trash it and that's it. But you basically don't use your computer for doing that. So that's the idea. Nevertheless, over the time, I would say the threat model change a bit. So it was, I think the USB key was usually a threat models, like I would say five or six years ago nowadays is not anymore a threat model.

So it's one of the open source project that we still have running. It's still operating, but we don't maintain it quite a lot, I would say because we have seen an evolution on the threat model regarding USB key and it's less and less common to actually use USB key in meetings and so on to exchange.

But it's still a trade, a trade model or trade vector mm-hmm. That you need to, to tackle in some way.

Jonathan: Yeah. If I find a USB key or get one from an untrusted source, I'm not plugging that into one of my computers. Not one connects you to the network. I do something like this, put it on a raspberry pie and, and spit it out as a text file or something, which is.

You guys basically just automated that idea. I, I imagine it's not so much a threat model anymore because it was, it became so well known as the threat model, particularly after things like stucks net and you have so solutions like this one, like the USB sanitizer that kind of chop the chop the real effectiveness out of it.

Yeah,

Alexandre: exactly. Yeah.

Jonathan: Yeah. Very. Yeah. Very interesting. Th this is this is some neat stuff that you guys are doing. Is there, is there anything, is there another one of these sort of projects that either you're doing that i, I haven't asked about? Or is there anything on the radar for circle?

Alexandre: Yeah, there are many.

So maybe one that would be interesting for you. It's, it's called a project, so it's a project.org. So this project is monitoring dark web. So it's basically a complete caller for tour. You have feeder for telegram channels and so on. We actively use it for monitoring tr it's automatically calling tour.

You can, for example, do correlations between websites so you can demise to and services. It's a project that we do a lot of work on. We do regular release and it's actually used by law enforcement c worldwide intelligence agencies, and it's completely open source. Then it's, it's, it could be a dangerous tool because you can actually extract credit card numbers cryptocurrency, stuff like that.

But if you have like good intentions like Security Operation Center is a very nice tool for getting insight about what actually trade actors are doing on, on forums and so on. It's, it's quite advanced tool, so I. I mean, Kuna is, is a recent tool into our different tool set. Mm-hmm. But El project is, is one of the project that we, we maintain and, and, and work on.

Then I will mention one, which is still not public, but will be released very soon. It's a rule set with a Z in between rule set do org. Mm-hmm. It's still not online. The idea behind is to basically publish rules from Tata Sigma Z and so on, and to basically use a green to get the existing one from the different repository, as you know.

They're on GitHub. Plenty of of repository with Sigma rules. Yeah. Rules. And sometimes it's very difficult to, to, to find does this one works well with this detection and so on. And the idea is to be able to have a called source version of all those different sources. And even author can claim the authorship ownership on the specific rules.

They can review it, they can update it, and then people can download from the website the rule set. And for example, the one that generating not that much false positive, the one that are efficient into this kind of stock operations and so on. And obviously one of the rules could be the one from obviously.

But that's the idea. And we, we want to do that because we have seen that in miss. A lot of people are sharing information about here are rules and so on. But we wanted to make it public. So the code base of the software will be, again, open source and web services will be online, but that's a.

Of something new that will be released in the next week. Very cool.

Jonathan: Yeah. That's neat. One more thing on the website I want to ask you about, and that is A BGP ranking. The, the, the entire idea of attacks against BGP is very fascinating to me, but also a little terrifying because there's not great solutions out there for it yet.

And what what, what, what is bgp, what is Circle trying to do to, to fix the BGP issue?

Alexandre: And first project of circle. So this one has more than 15 years. We still maintain it. So the project behind is very simple. So we actually get all the BGP announces. Mm-hmm. And we looked into the malicious list of ips from those different networks announce.

So that means we just collect all the A unknowns. Mm-hmm. And we collect all the malicious ip. Then what we do, we summarize to see how long those IP are still malicious in those subnets. And then we rank the ISN numbers. Based on the proportions of malicious ip, of the unknown ip. So for example, if you have like a small slash 24 and al for those ips, like 128, other ips are always having C two servers, phishing, and so on.

Mm-hmm. Your ranking will be very bad. Nevertheless, if you're very huge providers and you have like some ips that are malicious and so on, that makes sense. And then you clean up those one and so on and then you have a low ranking. And on a daily basis we can compute the ranking of all those license and see.

So if you go to BGP ranking, that's circle tell you, you get a top list of the most malicious one and obviously in the malicious list you get scanners. That makes sense. But you get a lot of, of bulletproof roster that are actually a thing thing that they should not host in various places. So it's a nice way to spot.

The malicious one.

Jonathan: Is anybody doing any automated work with this? Like one of these low ranked BGP numbers just announced that they're, they're, they're routing half the internet. Maybe we should ignore that.

Alexandre: That's, that's a good point. I've seen one or two papers that are actually using our dataset and they were looking at the outliers.

So there's some outliers that are interesting. Like they don't know that much. They're a bit like out of the radars. But why it's like that? Is it because they don't host anything? They're just giving access. For example, some obscure V provider tend to be like in the outliers.

Jonathan: Mm-hmm.

Alexandre: Because they just provide access.

But sometimes the visa provider should have like a lot, some malicious ips. You know, you have one compromised machines and so it makes sense. But you have visa provider with zero infections over like a period of five. Yes. Does it make sense? Hmm.

Jonathan: Yeah. That's interesting. Alright. And so, and then let's, let's turn back to Kuai.

I have a couple of questions about that before we wrap up. And one of the first ones is I run a Linux server. Let's say I want to deploy Kuai on it to keep track of things. What's, what's sort of the process to do that? Where, where can I go to figure out the steps to use Kuna on, on one of my servers?

Quentin: So there is a website which is why Kuna Rocks. So yeah, Y Koai Rock. And basically it it contains all the documentation for you to help you in the deployment of Koai Nice. In how you can use it for, yeah, doing really specific monitoring, how to design detection rules, how to load IOCs and so on.

So yeah, I try to keep this documentation up to date simple. And again this documentation is open source, so if anything is missing, just open an issue, I will fix it. And yeah, that's it. And yeah, I always try to do like the simplest things possible. So yeah. I hope, I hope the, I mean the, the setup is quite simple.

Jonathan: Yeah. Yeah. It looks really good. Is there, is there anything coming for Kuna that you want to let folks know about?

Quentin: Yes, there is something that we will put online soon, I guess, which is a platform where, where you can upload malware samples and it'll get directly analyzed with K nine virtual environment.

And why is this interesting? Is that because when you are doing most of the time incident response, you use some boxes to execute samples mm-hmm. Of any sort, but, the output of it is most of the time not directly compatible with your detect detection tool chain. So the idea was to have like directly nel logs out of malware monitoring so that you can directly create detection rules using Kuna.

So the platform will not aim at being an additional sandbox will not aim at defeating anti malware, sandboxing and so on. Mm-hmm. Will just be something where you throw your malware in and you get the Kuna logs. And yeah, that's, that's basically it. So it's using a, a project which is already on GitHub, which is Kuna Sandbox.

And basically it's front end on top of this. Mm-hmm. And yeah, I just like released the code past week, so it should be online,

Jonathan: Soon. Very cool. It save, saves me from having to stress about setting up an a, an absolutely secure sandbox to be able to do this testing. Very cool. Alright is there, is there anything that I didn't ask you guys about that you wanted to let folks know about?

I know this is a challenge. You've gotta think through all the things, all the things we talked about, and all the things you wanted to.

Alexandre: Just, just one thing that we have a conference in Luxembourg called ACT Tell You in October.

Jonathan: Oh, very cool.

Alexandre: And we are still have a CFP open for open source project. So if you go to act, tell you you can submit your paper there are still some day left before the closing of the CFP.

So and we welcome open source projects and especially open source security projects there.

Jonathan: Very cool. I, I'm curious, and again, this is, this is a question very much an American centric question. What about the language there? Is that gonna be a French conference or English? And if some, if it's in French and someone comes like me, that doesn't speak French, am I just gonna be completely lost?

Or is there helps? How, how does that work?

Alexandre: So Sandberg, we, we have like officially four languages. So Luxembourg is one French Germans. And one of the most spoken language in Luxembourg is Portuguese too. 'cause we have a huge Portuguese community. I believe it. But the conference, I don't tell you it's, it's obviously English or broken English depending of, of your origin.

But tech, technically it's, it's an English conference. So it's, we need to, to, to do a conference where everyone is welcome. So that's usually the universal language.

Jonathan: Yes. Yes. Very good. All right. Appreciate that answer. Okay. So, and I'm being reminded now in our chat room, I forgot this last week.

I want to ask each of you guys, and we'll start with Quentin. What, what is your favorite text editor in scripting language?

Quentin: A scripting language? Yeah. It's bash. It's bash and text editor. I would say VIM Bash and Vim

Jonathan: Makes sense. Alexander, I.

Alexandre: So obviously Veeam was a long time IMAX user, and then one day my had hit the toilets and then I changed back to Vim.

So, and then it's, I think it's, it's, and I think we are the only two Vim user at the office counter, I think. Really? That's funny. Yeah. I think the, the rest is more like more Id one scripting language. I'm a long term CA contributor, so I really like Pearl. Mm-hmm. For four years, even if I would say Pearl is not even a scripting language, it's much more than that philosophy.

But indeed for me, bash is still a great, great way to to do quick scripting and so on. But I would say. Favorite in skeptic language.

Jonathan: Yeah. Yep. Very good. Alright. Thank you guys so much for being here. It's been a blast learning about Kuna and all the things Circle is up to. I appreciate it. Once some of these things come to fruition after a while we'll have to have you guys back and talk about what's changed in the l landscape of Linux threats.

Alexandre: Thank you very much. Thank you.

Jonathan: Alright. That was a lot of fun diving in and talking about some of those things. You know, I cover a lot of these on the security column and it's neat to be able to talk to some of the people on the front line about things that are actually going on and very much enjoy that.

Alright, so I wanna let folks know, remind you that if you want more of me, there is of course, the security column goes live every Friday morning on Hack a Day. There's also the Untitled Linux Show over at twit. Make sure and check that out. We be, we will be back next week with another episode. I sure appreciate everybody that's here, those that listen, and those that watch both live and on the download, and we will see you next week on Floss Weekly.

Episode 831 transcript

30 april 2025 | min

FLOSS-831

Jonathan: Hey folks. This week Dan and I talk with Peter Von Dyke about power DNS. It's one of those building blocks of the internet that you use all the time, even though you probably don't realize it. It's open source and it's a really neat project. You don't want to miss it. This is Floss Weekly, episode 831, recorded Tuesday, April 29th.

Let's have lunch. Just a quick bit of a rattle that Peter let me know after we recorded this episode where we talked about power DNS admin and he complained that it was using the old pre API connection. It was pointed out that that is actually power admin. Empower. DNS Admin is a newer project that uses the API as it was intended.

Sorry for the confusion and enjoy the show. It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something really cool. Today we're gonna be talking about power DNS, and of course it's always DNS, right? It's not just me.

We've got the wonderful the original Linux Outlaw. Dan the Man Method. Dan, welcome, Dan. Hey, Jonathan. It's good to be back. It is good to have you. It seems like it's been a while since you've been here, which is unfortunate because we always have a lot of fun on the show together.

Dan: Yeah, it has been a little while, isn't it?

So it's, it's especially good to be back and on the streams. Mm-hmm.

Jonathan: Yeah. So today we've got something really fun. We've got Peter von Dyke of the Power DNS project, and I was, I was saying before the show, I actually use Power DNS on a couple of my, a couple of my Coho co-located, well, it's one co-located server with a couple of VMs on it.

For, for doing DNS, have you rubbed shoulders with the power DNS project before?

Dan: I haven't. I've never used it, so I'm a complete power DNS Virgin at the moment, so I'm looking forward to learning a lot more about it today. I know a bit about DNSI, I do know that it's always DNS. We all know that just to make the joke again, I know you've already made that joke, but it, it is even when you think it's not DNS, it is DNSI was told by, yeah.

In some way it is.

Jonathan: Have you, do you host your own DNS? Have you ever done that, gone down that rabbit hole?

Dan: Not much. No. I haven't really, so I, I'm really looking forward to learning more about it today. I've been reading up on power DNS during the day, so I, I know a little bit about all the different projects and stuff that they've got going, so I'm really looking forward to hearing more from Peter.

Jonathan: I, I will say it's not as painful as hosting your email, but there are some That's good. There are some real gotchas. I, I have gotten the call from the data center before, this may have been before I started using power DNS even, but the call from the data center saying you've got an open relay on your DNS and it's getting used in DDoS attacks.

Please stop that before we go unplug your cables.

Dan: Mm-hmm.

Jonathan: It's like, oh, that's not good. Lemme go fix that for you. That's,

Dan: yeah. If, if it's easier than hosting your own email, then that's good. 'cause I've come across a host in my own email before where I had a, a lot of spam sent through one of my servers, which was not great.

Jonathan: Yeah,

my, my, we'll get to power DNS here in a second, but this is just too much fun to pass by. I, I am currently struggling through an email problem. I had to change where my server was located, change ISPs for the server, and the static ips that they assigned me apparently somehow originally came out of Iran.

Like they got reallocated or something. So when you go and do a lookup on those IP addresses, it's like associated with the the Iranian, you know the, the, the, the, the nick from Iran. And so they get blocked by a lot of places and, mm-hmm. It's not good for putting an email server on. So we're still trying to fix that.

Wow. So DNS back to the actual topic of the show. We have the expert on power DNS at least. Let's go ahead and bring him on. Mr. Peter Von Dyke. Sir, welcome to the show. We are so glad to have you.

Peter: Good morning, good afternoon, good evening. In my case.

Jonathan: Good evening in your case. Yeah. Understandable. We have, we have all of the, just about all of the potentials covered here.

Yeah. It's morning time for me, afternoon for Dan, I think, and evening for you. That's perfect. Mm-hmm. So what, let's, let's start with the, the very basic questions. What is power DNS and why would somebody reach for it as a solution?

Peter: Okay. So I assume you know what DNS is, actually, you mentioned you do.

Power DNS is one of wide selection of software you can choose from if you want to run your own DNS, both as a resolver and as a hoster. Mm-hmm. There's others of course the original bind D-G-B-D-N-S, which almost nobody uses anymore as it hasn't seen a patch in 20 years. Not DNS the nano lab stuff.

We distinguish ourselves on the hosting site by allowing you to have your records stored in a database. Mm-hmm. Where things are served live from the database, you can easily update things. And on the resolver side, we distinguish ourselves in performance and in filtering or well content control features and scripting you can use to steer things this way or that way.

And then on top of all of that we have DNS dis, which is like a proxy or a load benefiter that is not power DNS specific. It also works with other name servers and people in fact, often run DNS dis in front of their resolver or authoritative name server from another brand as well.

Jonathan: Mm-hmm. How, how long has Powerd NS been around?

What's, what's like the history of the project?

Peter: Powerd NS is 25 years old now, I think maybe 26, but I think the first commit in the open source repo is from 99 or 2000. Mm-hmm. It, it was close source before that during the.com boom. Mm-hmm. But that did not last like many things at the time. Yes. So it was open sourced back then under the G GPL V two and it.

People just started using it. And we've been growing users, and our users have been growing users because a single user of our software might have a million customers at home. Mm-hmm. Passing all their DNS through power DNS.

Jonathan: Yeah. Does it do well with that kind of scale?

Peter: Yes. Yes.

Jonathan: And how long, how long have you been involved in Power DNSI guess, what's your position there now?

Are you, are you sort of the the head of the project now? And how did so I,

Peter: IJI joined like 14 years ago. Hmm. At that time there were zero employees. Mm. There was just the original founder who also had a different business to run.

Jonathan: Mm-hmm.

Peter: Who realized he was actually bringing in some support contract money from customers Hmm.

But did not have the time to actually support them. So he said, Peter, let's have lunch. I said, Bert, no, I have a job. He said, no, let's have lunch. So we had lunch and I quit my other job, which I did hate. Anyway, he knew that. Mm-hmm. So then I was the only employee, so I had to do support development be involved in sales a bit, not too much.

Then like what?

Jonathan: I know, I know how that goes. I am, I am a single man, single man shop, doing it here in town. So I get to do all of that stuff too.

Peter: Right. Yeah. Then, like two years later, we hired one other guy, and then we merged with a company called Open Exchange. Mm-hmm. Who do like exchange, but. Open, so on premise or now they also offered as a cloud thing or et cetera.

They also got acquired they also acquired Doco at the same time, the IMAP server.

Jonathan: Oh, okay. I is, are those projects still all sort of under the same umbrella?

Peter: Yes, yes. Still true. Today, anyway, we, we were thinking we could hire sales, hire hr, hire, finance, et cetera, or just find the company that already has all of those things.

Mm-hmm. And incidentally also wants to sell to the exact same customers we wanna sell to. So we joined them which gained me like 200 new coworkers overnight. Little culture

Jonathan: shock there. Yes.

Peter: Very, very much so. Yeah. And, and then over time power on that umbrella has grown to like 30 people and I'm not really in charge of, of anything, although I'm a product owner for one of the products.

Mm-hmm. But I do find that with my history, people come to me a lot because I, I've seen everything. I've had my hands in every project.

Jonathan: Yeah. Yeah. Interesting. I, that's always a nice problem to have where you have support contracts coming in and it's like, how, how do I deal with people giving me this money?

What what is the what is kind of the, the business model there? Is it, is it just that, is it other companies want to have someone to pay for support? Is that really how power DNS makes money?

Peter: So 40 years ago, that's how it started. And then by that time it got to the point that, that they could afford to hire one person for that.

Mm-hmm. But these days it's a little more, little bit more complicated. We do still sell support contracts. Mm-hmm. Quite a few of them effect. But we also realized that there are needs that only really big businesses have like big telcos. Mm-hmm. So we've written more software, which is not open source that we sell our license to some of those telcos, and that allows us to have this team and build all this software and have all of the bits that everybody would enjoy open source.

Mm-hmm. And then some of the bits that most people don't need are not open source.

Jonathan: Yeah. Interesting. Ha. Has there ever been kind of a movement to take some of those things that are not open source? Because this is what the, the original project, it started out as closed source and then it just, it sort of became apparent that it made more sense to open source.

It. Are, are some of those bits sort of on that same migration pattern?

Peter: Well, yes and no. If, if you, if you wanna know what we're up to in the closed source, it's always fun to watch our GitHub and look for new integration points appearing in the open source.

Jonathan: Yeah.

Peter: We have some libraries that we built internally where we realized, well, this has no, there's no value in keeping this exclusive.

We can publish this library, like we have a client library for Amazon S3 that we use in the product that we open sourced. Although the product that was for is also open source now, it's called Lightning Stream. It allows you to synchronize DNS data via a storage like Amazon S3. And that project is opensource as well.

No, but in general so let me explain what, what we're selling just briefly, because I'm not here to sell the biggest market for us is ISPs that need to offer content filtering or parental controls, like in the United Kingdom, ISPs have to offer this. Mm-hmm. And that's a big market for us, and I don't see us open sourcing the bits that do that.

Jonathan: Yeah, that makes sense. Is, is there, is there a challenge that you guys have run into about the, let's say the viral nature of GPL? Is, is that challenging to, to, to keep these closed source pieces? I, I don't know exactly how they integrate with the open source. But I know sometimes in some of the projects that I've worked in, even, it's a challenge to be able to do things like that without sort of inheriting the GPL unintentionally on your closed source stuff.

Or, or breaking That's exactly. Or breaking the license. Right. That's the other thing you don't want to do. Yeah,

Peter: yeah. No, that's exactly why you will sometimes see integration points, hooks appear in the open source that we then, on the closed source size can hook some Lewis scripts into which do, which can do logic on resolver queries.

And on the other end, there's APIs that software can talk to, to make the open source do things or not do things. So it's a, it's, it's, I wouldn't say it's been a challenge, but it's a, it's a very good question and we need to think about this a lot. Yeah.

Dan: Yeah.

So, Peter, do you have a, is is there a hosted product?

I'm thinking about the whole kind of business model that you've got there. Is there a hosted version of this that I could come to you or some, maybe a partner or somebody and say, I'd like, you know, as an off the shelf solution that you will host for me?

Peter: No, not really. There's places you can go to and give money that will run power Ds for you.

Of course, like a lot of web posters, if you let them do their dns, do your DNS, they might be doing that on power DNS. And if you use one of the open resolvers, you might end up also using Power DNS. But we don't sell any hosted products right now. On the email side we do, on the DNS side, we do not.

Dan: Right. Okay. Yeah, it was just something I was curious about. 'cause I know some companies use that as a revenue model as well. So you mentioned there's about 30 people working power DNS right now. Do you get a lot of in, I know you've got a lot of stuff going on through GitHub, 'cause I've been looking through all your GitHub stuff today.

Do you get a lot of patches from the community? Do you accept patches from the community? Do you build, do you pull them in?

Peter: Yeah. Yeah. We accept batches from the community. We get we get quite a few, we get more issues than batches, of course. Mm-hmm. But I do think we have a couple of pool requests from outside the company a week.

Jonathan: Yeah. That's

Peter: good. Wow. And lot of those will get accepted. So maybe after review and some back and forth.

Jonathan: Right. Yeah. But, you know, people

Peter: appear to enjoy working on software or at least

Jonathan: the, the willing and

Peter: able to fix the things they wanna fix.

Jonathan: The project has been around for 25 years. DNS has been around for, what, 40 years?

Isn't it done yet? What are actually still for?

Peter: That's a great question. People keep inventing new ways in which they want to change answers. They give out, ah, they keep inventing new transports. Their people keep having new performance needs. For example, Intel. Intel sell some hardware now that can offload TLS handshakes.

Mm-hmm. And then a customer came can support that because with D-O-T-D-N-S of T-L-S-A-D-O-H-D-N-S over A-C-T-P-S on the rise, people are finding that they're suddenly spending all their CPU doing TLS and not DNS. And then when Intel comes out with, with a little chip or a CPU add-on that can handle that in hardware, then people wanna use that hardware for that.

And that's code we have to write.

Jonathan: Yeah. I suppose that part of it is definitely not done.

Peter: No. No. And then there's just the ever evolving software landscape. Whenever a new DBN or red or whatever comes out we need, need to make sure we work with that, especially when it's a new open SSL.

Dan: Mm-hmm.

Peter: So

Jonathan: open SSL updates or glib C updates or any of those.

Yeah. Yeah. It's constant. Yeah.

Peter: Open SL breaks more stuff.

Jonathan: I can believe that. Yeah. That, that seems, that seems, yeah. That seems legit. Alright. So one of the things that we were talking about before the show that I mentioned briefly is this idea of DNS as a DDoS amplifier. I'm sure that is something that you guys have, have at least seen before.

I've not gotten, yes, I've not gotten the email while using power DNS that my server was being used to DDoS amplification attacks. And I guess I can give the background on this. The idea is you can send a UDP packet to a DNS server. You can send a really small UDP packet and trick the UD the DNS server into giving you a big response.

And because you can spoof UDP, you can get that server to send that response. Off to a target that you're trying to DDoS and it's, it's bad. It's not great. What does power DNS do to avoid that problem?

Peter: So a a as I mentioned before, we have the resolver and the product a hoster reduce, and then we have our proxy slash low er called DNS dis dis for distributor.

And that one is rate limiting and can check query rates, see where things are coming from. And the biggest trick we you can use in that area, which is not unique to us, is instead of responding with a big packet, you respond with a small packet that just tells the client, please try again over tcp.

That's a couple of extra round trips, but then you're not sending out more bites over UDP than you got in over UDP. So when you're. Under such an attack or actually being used for such an attack, you could decide to switch to that mode and incur a little performance penalty. But suddenly all the queries you're answering are legit because they came in, came in of TCP and not UDP

Jonathan: Power D Nest does some of this by default.

I, I, I hope, I assume,

Peter: Very little bit. We, we like to think our users are competent enough to decide what's good for them, configure things that way.

Jonathan: That may or may not be a safe assumption. Yeah. Is this sort of a I didn't,

Dan: go ahead, Dan. Sorry, Jonathan. I was just gonna say, I did notice when I was looking through your website today, you offer training as well, so you do offer, you know, you say you hope the users are smart enough to know what they're doing, but you also offer training as well.

That's another thing that, that you you do.

Yeah.

Peter: Interesting. That's true. Yeah. Usually when, when we do trainings, it's existing customers, but yeah, yeah, we do do have those.

Jonathan: Can I, can I pay you money and send you my setup files, my configuration files and an IP address and say, can you audit my power?

DNS install?

Peter: So we, we don't do that. We, we really want customers that will stick with us for a year, three years, five years, or whatever. I do have some friends I can refer you to, and we in fact do this sometimes when a request comes into sales and somebody's like, I have 5,000 bucks. I just wanted a couple of days of help.

We, we will find somebody for you. Except usually they have 500 bucks, in which case we will not.

Jonathan: We let, let it, we know a guy that might be able to help you out for $500, but that's below our threshold.

Peter: Exactly. Yeah.

Jonathan: And that may seem mean to folks, but when you're running a business particularly of a certain size, you just, you just have to have that sort of a cutoff or else you're, you, you spend all day chasing these little leads around and not making hardly any money.

Peter: Yeah. Yeah, exactly. And

Jonathan: it costs money to have 30 developers.

Peter: It, we don't have 30 developers. Sadly. There's a couple of other roles in there, but ah, yes, they, they do all cost money.

Jonathan: Yes. Yes. So when I use Power DNSI actually use it with something called Power DNS admin. And I have to admit, I'm not even sure is that official from you guys from Power DN do you have patches in power?

DNS Admin, did you write it?

Peter: We did not write it. It's not from us. I think we might have a patch or two in there. For a long time it was basically the only front end you could download for Power Ds, I think. So it got quite popular.

Jonathan: Mm-hmm.

Peter: But these days there's like 15 or 20 you could choose from.

We have a list on our Wiki. And I don't power DS admin is not one we would recommend today. That's not because it's bad but when it was written, all you could do was. Do SQL update, insert, delete, whatever. Over the last 15 years, we've developed a rest API right inside our product. Mm-hmm. Which does syntax validation and other sanity checks.

So we prefer that front ends use that interface and there's a bunch of front ends out there that, that do that. Which allows us to then make sure that whatever goes into the database is correct. And if you use Powers admin, you don't get that sanity check from our software. You just get the checks that are in Powers admin itself.

Jonathan: Mm-hmm.

One of the, one of the disadvantages of it being sort of the, the old, the oldest solution in the field.

Peter: Yes.

Jonathan: Yeah. Sort of sort. It's

Peter: quite battle tested. Of course.

Jonathan: Yeah. Right. So what, what do, what do you use, which of these front ends, graphical or web front ends do you use to administer power?

DNA servers.

Peter: So it, I spoiled this before the, before the show. I sometimes,

Jonathan: I sometimes tell guests that just because I ask a question, it doesn't mean that I don't know the answer.

Peter: Yeah, no,

Jonathan: that too.

Peter: So anyway inside ds, the, the products for hosts, the authoritative server, we have a tool called PD DNS util which has a feature called Edit Zone.

So you just go to your, shell go pd ns util edit zone example.com, and it fires up your editor. In my case, it's some form of vi. So that's what I do mm-hmm. For my private domains and therefore work domains, which, which I shared administration. I use a thing called zone control, which is a web front end, but not an opensource one.

Jonathan: Okay. Yeah. Interesting. Power DNS sort of baked into it. It has the ability to have. Multiple name servers that talk to each other and automatically stay synced.

Peter: So yeah, you can use zone transfer for that, which is very standard work with every brand and then cite that. Because we can be database backed.

You can use use mice cloud replication, Postgres replication or whatever. And then also, which is there's another opensource product, which I briefly mentioned before. It's called Lightning Stream. And then it'll take your DNS data serialize it into an S3 buckets, and another note will download it, merge that data, handle any conflicts, and that way you can have like a multi primary thing going on without an actual database server.

Jonathan: Yeah, it's, it's interesting every time you go to do anything with DNS and you know, you set up your glue records. Everything wants an NS one and an NS two. It's like you're required to have multiple DNS servers. And I must admit, for the longest time my NS one and my NS two pointed to the same server.

Peter: Yeah, why not? Because if you're actually, if you have a separate NS two and it was down, then your website was down anyway. I bet

Jonathan: Yes. When you're that small of a, of a, a, a hosting thing that yes, that is the way that tends to work. How, how big of deployments do you guys have? So on the other side of the coin, how many nsx, you know, do, do some of your big deployments have that you see

Peter: how many name servers?

Jonathan: Well, yeah, just like what kind of, what kind of scale are we talking about on the upper side of what power DNS does?

Peter: So I know one user, not actually a customer, but a user who has like one and a half million domains in one setup. My goodness. Wow. Replicated over a couple of name servers. And these people, they sent big patches, like really big.

Mm-hmm. Because they, they had a, a, a name serve. They wrote themselves, which worked for them, but they wanted to switch to powers because then they don't have to maintain it. Yeah. But they were unhappy about some of the aspects of it, including some of the scaling. So a guy there spent weeks typing away and we got big patches and we reviewed them, we merged those and everybody wins.

Dan: Yeah.

Peter: So there, there's that one and a half million is one I know of with a couple of name servers. I am aware of some deployments that do any cost where you can buy that as a service that must have hundreds or thousands of nodes with, again, probably hundreds, no, sorry, hundreds or thousands of nodes with probably hundreds of thousands of domains on them.

And then on the Resolver side, we, we, we know that we have users and customers that run entire countries on power DNS on a couple of servers. So might be millions of users.

Dan: Yeah.

Peter: When I think in Europe and then the US of course, even bigger.

Jonathan: Yeah. Yeah. Interesting. Has the project been battle tested enough that, like, that doesn't keep you up at night?

Peter: Well, that, that's, that, that's the beauty of being open source. I mean, we have a couple of customers and if we have had just those customers running this code that will be quite uneasy. Right. But I know there's a ton of users out there that run this stuff all day, every day without trouble. Or one other big user is Quad nine, the open resolver.

Jonathan: Hmm.

Peter: They split their load between power DNS and Unbound to resolve from an L net labs with RDNS, this, the front.

Jonathan: Hmm.

Peter: And they do that at hundreds of locations for I think millions of queries per second worldwide.

Jonathan: Yeah. Impressive. That's really cool. Yeah, I is is the problem actually always DNS as we joked a couple of times?

Peter: No, of course not. No, no. One of the reasons I think that comes up a lot is that whatever you want to do anything with a network, you gotta do DNS before you can do anything else. You can go to google.com without the IP of google.com. So if the network prevents you from reaching your resolver, it's your DNS that looks broke.

So I, I think actually we should just always blame the network.

Jonathan: Yeah. I. What about IPV six? Is that, is that supported in in power DNS? How long has that been supported?

Peter: So you can imagine, I have a sign in my office that says day since the dual stack incident. And the number on that sign is not a big number. No,

Jonathan: no, it's not.

Peter: But but generally fee six has been supported in our software for a long time and most other software as well.

Jonathan: It's kind of

Peter: sometimes we run into things.

Jonathan: It's kind of funny. On, on my ISPs, I have not been able to get IPB six until very recently. In fact, I, I talked about having to move my server. My, my previous ISP wouldn't give me IPV six. And that was actually one of the things I was excited about with the new one, is that I could get, you know, like a slash 56 of static IPV six addresses.

Yeah, I haven't done much with them yet, but I've got 'em, I've got a couple of posts up and, that's, it's, I think part of it is because I'm, you know, I'm in the middle of nowhere, I'm in flyover country and so like getting IPV six, my local ISP when I asked them the first time, they're like, you want what?

Dan: Yeah. Yeah.

So, yeah, I, PV six, that's, I remember about 15 odd years ago, maybe 16 years ago our local lug, we had a, one of the guys gave a talk about I about IPV six in which he said it was now gonna take over everything. And, and of course the standard joke since then is every year we say to him, is this the year that I PV six is gonna take over?

'cause it still doesn't seem to be doesn't seem to be quite doing that. Anyway, so so Peter, let's have a, a talk about some of the components that are involved in all this. 'cause I was, I think you can tell I've been reading the documentation. That's, that's where all my questions are going from. But I notice you have like three main products yeah.

That are used. So you've got Recursors. So what is recursors? That's the resolver.

Peter: Yeah, that's what a, an access ISP would run. It's what Quad Nine runs it. It's what works, stations, laptops, phones, talk to. Mm-hmm.

Dan: Right, right, right. And we've got authoritative server, which is the name server with the database in the background and all of that.

Which is what the recur

Peter: then

Dan: talks to on your behalf. Yeah. And then you've got DNS dis, which is the thing you were saying. A lot of companies run in front of other products even, not even in front of power DNS.

Peter: Yeah. Yeah, exactly. It's 'cause it's, it's just DNS, same way you could put an n engine, X as a proxy in front of some other web product.

You can put DNS this as, as a proxy in front of some of some other DNS product. Yeah. People are finding that the way you can intercept queries with that or look at queries with that, steer them in other directions also adds value to other name servers.

Dan: Excellent. So that, that's, was that part of the, I mean, I dunno if you were around in the, during the design process of, of this, but was that idea of being compatible with everything else, was that something that was important to you guys, do you think?

Peter: Yeah. To keep standards. I was, I was there, it's like 12 years ago, I think. Mm-hmm. And it seemed well, it seemed obvious. And also it's, if, if you're writing something for a protocol stack, it's harder to be incompatible than to just be compatible. Yeah. And there's, there's, there's, there's some nuance there because if you put a proxy in front of something than the thing in the back, no longer see who, who its clients are, and solving that within the DNS protocol in a way that's compatible with other products, that has been a, been a headache.

But other than details like that, it's always been obvious to us that you should just interoperate. Mm-hmm.

Jonathan: It, it probably helps that so much about DNS is kind of codified into the RFCs.

Peter: Yes.

Jonathan: I, yeah.

Peter: Of which there are many Yes.

Jonathan: I've, I, one of my other hats is security research and diving into things that have been around for a long time, like DNS and trying to go to the RFCs.

It's like there's about a dozen of them that you have to read through and you have to figure out, okay, which part of this still applies? Which part of this got replaced by some other RFC? It's, it's a lot of fun.

Peter: It's a lot. Yeah.

Jonathan: You've had your hand in some RFC work throughout the years, haven't you?

Peter: Yeah, I have. I, I've been like active in the ITF for quite some time part of which, just making sure that standards that we think are not good do not get out there. Mm-hmm. I have authored one RFC all by myself. Oh. Which is a very boring document, but it seemed important at the time.

Jonathan: What, what's, what is, what was your RFC about?

Peter: It's about the, the time to live on Ns second Nsec three records in DNS sec, ah, which is about a very specific case of caching when the nsec is in use. And I realized that some of the previous RFCs or in conflict with each other about what to do there. Mm-hmm. So it's kind of like a document review that then updates a few sentences to make sure things go right.

Jonathan: Yeah, those man, that's the sort of RFC that it, somebody that doesn't live in that world wouldn't appreciate, but those are important a lot of times because, like you, without that, you have a, you have a DNS response that is in a quantum state of both being RFC compliant and not RFC compliant, depending upon which one you, you, you look at, and, that's, that's sort of a problem. You could pet things from that. Yeah. Yeah.

Peter: Some, something, something like that. The other one I've been involved in is way cooler. It's called catalog zones, and it's a vendor independent provisioning mechanism. Mm. So what it does is say you have a primary, you have a hundred zones on there, and you want to provision a new secondary from the same or different brand or vendor.

You just tell the secondary, my primary is over here, IP 1, 2, 3, 4. My catalog zone is called catalog.example.com. The secondary transfers that one zone, which is called the catalog. And inside that zone is a list of all the zones you want to host. And this allows you to provision and deprovision zones from a primary to secondary without go getting on the phone or emailing.

Hello. I added the domain. Can you please edit on the secondary as well?

Jonathan: Yeah. Nice. It's very useful. What what language is power DNS written in? Is it, is it old school C still?

Peter: No, no, it was always c plus plus.

Jonathan: Okay.

Peter: Which is, you know, slightly less old school than C, but not much. Although c plus plus has been modernized quite well.

Yes. Over the last few years we've been adding bits of rust.

Dan: Oh, okay. I was gonna say, I noticed there's some rust in there from your GitHub repo. So is that something that's gonna expand, do you think? Is that I think so. Yeah.

Peter: It, it started with the, the, the configuration parser because we had a a, a hand rolled one, which was just barely adequate.

And as the, as the so grows, you want some hierarchy in there.

Jonathan: Mm-hmm.

Peter: So coworker said, well, I wanna turn this into yaml, which is mm-hmm. Not my favorite language, but it's also the least bad thing we have for the purpose. And you also wanted to try some rust? Yeah.

Jonathan: I was just, I like yaml configs is, it is basically my favorite.

It's a way to do config files, but to do continue it, it's, it's the least bad. But That's

Peter: fair. I, I, I get it. Yeah. So he, he said, well, I want to redo all this stuff in yamo, and also I want to try rust. And parsing yamo with rust is very easy. So he redid the configuration parser for the recursors in rust.

And that worked out quite well. Yeah. And then in DNS diss, so need to explain one more thing. The recurs or the iterative, they, they speak plain DNS of UDP and T ccp no TLS, no encryption.

Jonathan: Mm-hmm. We

Peter: handle that DNS dis as the frontend proxy. So DNS DIS has to speak all these protocols, T-L-S-H-P-S and quick.

Jonathan: Right.

Peter: And for quick, we use a library called Ki, terrible pun, that developed a CloudFlare, I think in Rust. Mm-hmm. So that's not a bit of rust that, that has made his, its way into our code base. And I think right now also is redoing the recurs web server in rust.

Jonathan: Is that, has that been a pretty good experience using rust and c plus plus together in the project?

Peter: Yeah. Yeah. There's there's some friction. 'cause not all data types fit Exactly. And if you do reference counting, you need to make sure things go well on both sides. But from what I can tell, 'cause I haven't been involved in that much it's, it's been going quite well. And the, the, the others appear to like rust.

And one day I will also have to learn, which I don't mind.

Jonathan: I'm reminded of the, the, the kernel, the Linux kernel and the, the, the, the hola around adding rust there and the they can't make us learn rust.

Peter: Yeah, yeah, I remember, yeah, a bunch of people left over that, didn't they?

Jonathan: A few people have left over that.

Yeah. I have, I have opinions on that, but this is probably not the show for that.

Peter: No, no.

Jonathan: Have there been, have there been any big CVEs in power DNS over the years? Have there been any real bad problems?

Peter: It, we, we have a page in the documentation which data has been looking at. It's not, it's not a big list over 25 years.

It's not a big list. Most of them have been crashes and not like remote execution. A couple of memory leaks. Mm-hmm. And, but I don't think we've had remote execution in quite some time, like maybe 10 years.

Dan: But I was gonna say on your website there, it does actually say on the front page, celebrating 10 years without a severity one.

So, oh,

Peter: that's, that's, that's a customer support severity one. Ah, okay. It's it's

Jonathan: not quite the same, but also worthy of celebration.

Peter: Yes, exactly. Yeah. That's something marketing wrote up, although it made me help.

Dan: Betty, how do people get involved if they, if they, if they want to, if they want to hack on, on this project and they wanna get involved, what sort of things of, I know you're on GitHub and people can go and clone the code and, and start working on it, but do you have any advice for people who might want to get involved and think, where do I start with this?

Peter: Well, the advice I would always give for anybody wanting to get, get involved with any, anything. Mm-hmm. It really helps if you have something you want, you see a piece of software, you, you, you say, this is great. However, then you have a problem to solve and having a direction. I. I feel makes it a lot easier to figure out how to get there.

Then we have a bunch of documentation on how to build how to develop, how to write safe and readable codes. And of course it's very important. I, I think that if you wanna do that, you join, join the IRC channel where we will help you and answer questions and, mm-hmm. Get you on your way to fixing things.

Jonathan: Old, old school IRC, that's yes, that's no, no discord, no matrix, none of that.

Peter: No, no. We have IRC and a mailing list. Although that's dying because email.

Jonathan: Yes. I had that conversation just earlier today. Email is terribly broken. Just

Peter: Yeah. Yeah, I heard.

Jonathan: Well, no, another, another one before that one. Oh, another.

Yeah. Yeah. I guess I've had that conversation twice. This makes it three times today. Yeah. Yeah.

Peter: No, we, we, we, we basically can't even send email to, to Microsoft anymore, and Gmail is hit and miss. So we have IRC and the mailing list, which are trouble, and we use GitHub discussions now, which is okay.

Dan: Yeah.

Yeah.

Jonathan: They're not, they're not terrible. They're not quite as good as full on issues, but it's not terrible. Now you've

Peter: and I, I like that they're not intermingled with the issues. It's true. Yeah.

Jonathan: That was a good, that was a good call that somebody made. Yeah. You, you've got your hands in some other open source stuff, not just not just power DNS.

What, what else do you dabble in?

Peter: So for work, I, I will dabble in some of our dependencies, some of our downstreams like DB and Ubuntu, Alpine, whatever. Mm-hmm. Ah I've, I've sent patches to Redis for things we ran into in some of our closed source. So out of a closed source do sometimes come patches for other open source.

Sure. Which, which is fun. And then outside of work, I, I, I don't automate my home that much, but I write a lot of software that allows people to do so. I, I have a bunch of components in E-S-P-E-S-P home. I have a couple of patches in home assistant in in the ZigBee quirks where the default stack doesn't handle device well, you can write a quirk that tells it well, this device has these endpoints that you need to use.

This way, I've, I've built a bunch of those. And also I work on an actual home assistant front end for low power, low power devices.

Jonathan: Hmm, interesting.

Dan: Excellent. So how did you come to get involved in open source software? Peter? Was there a moment where you were introduced to it or you had a, an epiphany and you just decided it was something that you were interested in?

Peter: Yes. Yes. I, I remember it vividly, it's like 30 years ago. I was inert with a PC with dos and Windows, and I got some shareware cd, which had a big list of software on it. And I found on this cd another directory, which was not listed on the rep or label or whatever, and it said Flagware.

Jonathan: Mm-hmm.

Peter: So somebody found some free space on the shareware CD they were selling and decided let's give the people some Linux.

So I, I installed that and it was magic was like, I, I was already familiar with the command line. I had the DOS command line. I had four dos on that, the command.com replacement, if you recall. Mm-hmm. It's been quite, quite a while ago. But then there was this Linux with this bash shell, which was a lot more powerful than I found Pearl, which ruined me forever.

I don't do a lot of pearl anymore. I, and then I realized I could just edit things and fix things. Mm-hmm. So this was 19 97, 98. We just got ISDN at home. If you remember, and I tried to call into an ice p with it, and the PPP demon would crash because it was written by a German and German ISDN in some specific bits was different from Dutch, ISDN.

But I had the sources so I could just find out what went wrong and fix that. And I think that's when I caught the opensource bug and that, well, it obviously has not gone away.

Jonathan: That's great. I, I read a little note here that you have apparently deployed ESP homes somewhere rather weird.

Peter: Yeah, I have one.

Well, I had one in a car. Okay. My, my wife had a, a Alta 2005 Toyota with, with a terrible radio, which, which in theory had navigation. We never figured out how to work it. And it did like this two line LCD thing anyway. Oh. So she always had her phone running Androids and a waste mounted to the dash.

Mm-hmm. But she learned about Android Auto. She says, well, can't I have Android Auto? So we looked at it, we bought an Android auto radio with a nice touchscreen and all. But then her steering wheel control, she has like this buttons, volume off, volume down next track. Those didn't work anymore. So I bought a conversion kit for that, which people sell for 70, 80 bucks.

They all contain the exact same little blue box and two cables, one that fits your car, one that fits your radio. Except did, it did not fit. So I had to go look around what. Went wrong, turned out Toyota changed these conductors a couple of times over the years. So I sent the seller of that conversion kit an email, and he said, well, I cannot sell you that cable.

So I decided, well, I have a multimeter, I'm sure I can find out. And that's when I learned that the steering wheel wheel controls they all sit together on one electrical circuit and just have different resistances running over them.

Jonathan: Oof

Peter: fun. So if you just have a multimeter and apply some voltage or, well, the multimeter does that, and you push the buttons, you see the different resistances.

Jonathan: Mm-hmm.

Peter: And then the radio on the other end had the exact same interface, but with different values. Okay. And I, I, so I looked online, people had been doing this with Arduinos. Ah one person designed a whole circuit without a microcontroller to do this conversion. Mm-hmm. That all seemed hard. And I was like, if you can do it with an AR Arduino with code, you can also do it with ESPO without code.

Mm-hmm. So ESPO has an a DC component which allows you to read voltages. Mm-hmm. Which would depend on those resistances. So a couple days later I was able to receive the steering wheel buttons, and then I used a bunch of resistors that I switched separately to send these different resistances to the radio.

And it's, it's quite over-engineered, but it works.

Jonathan: Oh, that's hilarious. No, I, I like that. I like that a lot. I expect

Peter: it. Still might be in that car 'cause we sold it some time ago.

Jonathan: I just, wow. I pity, I pity the poor tech that opens it up for the first time and tries to figure out what exactly you did.

Yes.

Peter: Yeah, no doubt. It's,

Jonathan: I've not done that to my vehicles, but I've done similar things to my house and I have thought multiple times, like the next person that moves in here, it's, it's going to be, you know, if I ever sell it, it's gonna be quite the archeological dig to figure out what's going on with all of this cabling and raspberry pies everywhere.

What's going on here?

Peter: Yeah. I should leave a menu when you sell it.

Jonathan: Yeah, yeah, I suppose I could. So I was also told that apparently your shirt has a special message for us.

Peter: Oh yeah. My shirt says I hacked the Dutch government and all I got was this lousy t-shirt.

Jonathan: There must be a

Peter: story there. Yes. The dodgy government sent me this T-shirt.

Wow, okay. I was so they, they're, they're quite confident Now. I was helping a friend with a Powers problem and he said, I cannot get to this government website. So we looked at it and his resolve for his recur has gotten into a state where indeed he could not resolve the name of this website. And I looked at it some more and I recognized a problem that's actually widely documented with one brand of load balances.

But then I realized that this state that his, his resolver had gotten into I could cause at will in other people's resolvers, which means I could decide, or some attacker could decide you that resolver over there. For the next hour is not going to be able to visit this government website. So I spoke to a bunch, to a couple of people some of them with connections to the right part of government and they said, well, we've been trying to get this fixed for quite, quite some time, but the pressure is not coming from the right place.

So I reported it not to this department, but to the Dutch Cybersecurity Center, the NCSC. And they emailed back within 10 minutes going, we see what you got. We replicated the finding, we're gonna get it fixed. And then they did. So that, that's a good trick in general, if you want something fixed and you cannot get people to do it, you need to find out who is suffering from this problem or who will feel the pain or the responsibility for this and maybe go there.

Jonathan: Mm-hmm.

Peter: So then they fixed it and they sent me this t-shirt.

Jonathan: That's

Peter: very cool. Which I don't wear out in public because I don't like questions, but I figured it will be fun for, for today.

Jonathan: You, you don't like questions from random people? I get that.

Peter: Yeah.

Jonathan: Yep. Yep.

Peter: And that reminds me of something fun.

I bought an open BSD T-shirt once, which with some artwork and it said open Bs D mm-hmm. And I also bought a kid's version for one of the kits, and that one did not say open Bs d it was just a, just a cute shirt. Mm-hmm. And I don't know if they did that on purpose, but I can imagine a designer going, the kit doesn't even know what open BSD is.

It does not want these questions.

Jonathan: Indeed, indeed. I see in your bio that you've done some work with open WRT, is that correct? Yes,

Peter: yes. Yes. I. So a couple years ago there, there was actually a customer project for power DNS. Somebody said this low benefit you have can you run it on open WRT?

We were like, that's a good question because most people who deploy it have 1 million gigabytes of RAM and not 1 million bytes of ram. Mm-hmm. So we spent quite some time shrinking the software, fixing defaults may allowing you to strip out components you don't need. And in that process I gotten, I've gotten quite used to open WRT.

Mm-hmm. So then I also started doing hobby things with open WRT. I have half ported it to a couple of devices that did not support it before. I've worked on the package manager a bit. Yeah. I, I often, easily let myself get distracted by things I think can be better, and then I go fix them. Or maybe half fix them.

Jonathan: I, as I, as I love to say we are capable of a nearly infinite amount of work, so long as it is not the work we are supposed to be doing at a given time.

Peter: Yes. Mm-hmm. Yes, exactly.

Jonathan: How, how slim did you get the that that load balancer for open WRTI

Peter: think we got it down to five or six megabytes of disc and seven or eight megabytes of memory usage.

Yeah. Which is quite a bit more than the DNS stack called DNS mask that is shipped normally. But then this is with DNS of DLS and HTPS and a bunch of other stuff and the whole scripting engine. Yes.

Jonathan: Yeah. I, I remember using open WRT back when it fully supported four meg discs and, back at that time, it was sort of rumored that you could leave out all of the web interface stuff and actually fit it into a router that just had two megs worth of disc space.

So, you know, a, a single package of seven or eight megs is huge in comparison to that, but I, I think the open WRT guys have since then said that's not a good idea to run on something that small. We recommend like eight or 16 megs of disc space at the minimum now.

Peter: No, no, no. That, no. The way, way past that, there, there's like a 4 32 warning on the website for four Ram 32 flesh and an 8 64 warning on the website for eight megs of RAM is 64 megabytes of flesh.

So I, I believe they recommend 16 ram and 1 28 flash now, or something along those lines.

Jonathan: Isn't that swapped, isn't it the other way around like eight megs of flash and 32 megs of ram. I think, I think yes. Yeah.

Peter: Yes. That, that makes more sense. Yeah. But anyway, yeah, they, they're, they've grown quite a bit since the four or two megabyte days.

Jonathan: Yes. I is Power D itself, power DNS itself on open WRT,

Peter: Over in the packages tree where you can just tell OPKG or the new one, they use A A PK to download the software or have it included when you rebuilt an image. Mm-hmm. And all the work we did to slim down DNS dis is currently an open PR for that packages tree because it's quite a big PR Understood.

And they are clearly understaffed in that area.

Jonathan: Yeah. The nice thing about running stuff on open WRT is you can you can do it on like a virtual machine. So I've got a, I've got a X 64 VM of open WRT, and that's got, you know, all the space that you want. So I could, I could totally put power DNS on that.

I don't think I need to. I'm not sure why I would other than because I can. Although maybe that's a reason in and of itself.

Peter: Yeah. Well if it's safe, you're not a vm maybe. That's fair.

Jonathan: That's fair. Dan, did you have any questions that you wanna get in before we before we wrap?

Dan: That's, yeah, that is a good question.

So I'm just trying to think of something now. So a lot of the, I mean, you talked about some of the smallest distribu smallest distribution, small, like putting it in open WRT, running it with stuff like that is some of the, the kind of edge IOT stuff, does that present extra challenges with when it comes to things like DNS and security?

Because I know a lot of people have problems with iot devices that are unsecure because they haven't been set up properly or whatever reason, but it's, I noticed that. With power. With power DNS, you can kind of mitigate a lot of problems. Could you speak to the, anything to do with that?

Jonathan: In iot? The S stands for security?

Yes. Yes, exactly.

Peter: I, I, I don't know much, but I know a little have you heard of Dane? DENE? Mm-hmm. Not sure. Mm-hmm. Okay. So we have the whole TLS ecosystem with the X 5 0 9 certificates and the CAB run by the browsers where Illa and Google decides who gets to be a certificate authority. Mm-hmm. Then on the, on the other hand, we have DN Ssec, which de facto has one certificate authority, which is ican.

But in the NS sec, you can form a chain of trust from the route to com to example.com or whatever. Mm-hmm. Mm-hmm. And then some people went, what if we then tie our web, or whatever certificates into the DNS SEC chain? The browser vendors decided. We don't want that. Mill operators do use this. And then recently in the ITF, there's a new working group called Dance, which is like Dane plus the C for clients.

Mm-hmm. And I wish I could tell you what Dane stands for, but I forgot right now. So they're doing things with client certificates in DNS. So some people envision that if you have DNS sec, you, you can build a whole structure around your infrastructure and the edge and tie all those things together.

But it's, it's not really my area.

Jonathan: Huh. DNS based authentication of named entities is Dan. Of course.

Peter: Yeah.

Jonathan: How could you forget that? It rolls right off the tongue. Yes. Yes. Oh, that's actually really interesting. That's fun.

Dan: So what's the is there anything coming up on the roadmap or anything that you exciting that you want to tell us about that you're working towards or you've got on the horizon?

Yeah, that's a good question. For the power DNS.

Peter: Yeah. Yeah. So the thing I've been working on recently, or actually been trying to work on for the last two years, but recently finally got into in bind the original name server, basically. Mm-hmm. You have a concept called views where you go, these people on this network, they see this version of a zone, the people on the internet see this other version of a zone.

Mm-hmm. And it's basically like running multiple bind instances in one process.

Jonathan: Yeah.

Peter: And as I understand it, bind has always been designed for that feature. Power DS has not we have geo things you can do per record or whatever. But there's always been a demand from people to say who, who said, I want one power D Ns to pretend to be a hundred power dnss, but I don't want to run a hundred power dnss.

Jonathan: Mm-hmm.

Peter: And our founder, Bert, he always said, we're not doing that because adding that now will be a trail of destruction through the code base. So for the last three months, me and a coworker have been utterly destroying our code base, right. To get that feature in. And with enough time and effort, we've been able to split this out in a bunch of patches.

The first five of which did nothing except restructure. And now we're getting to a point that the actual adding of the feature is not that destructive. Mm-hmm. But he was not wrong. It is it's, it is quite the exercise. Oh yeah, I'm sure. So anyway, that's, that's coming up for the authoritative and for the others.

Mm-hmm. I don't know if I have any major highlights.

Jonathan: Does, does the, does the project power DNS, the rest of the projects do they have like a continuous integration set up and a test suite? So when you go and blaze a trail of destruction through your code base, you, you kind of have some tools to make sure you didn't actually break things.

Peter: I don't know if you can see my whiteboards. There's a matrix there with three zeroes and the three

Jonathan: mm-hmm.

Peter: Oh yeah. That, that three has been counting down failing tests all week in, in one combination of settings. So yes.

Jonathan: That's good. That's, yeah, that's one of the things that actually started to bother me about the, the Lennox kernel itself, is that they don't have any of that set up and I don't know, that's actually a little, it's a little concerning.

So I'm glad to hear that at least one of those basic building blocks of the internet power DNS, which we all use, whether we know it or not, has some things going for it. Yes. Alright. So Dan kind of asked you this question. I'll ask it in a slightly different way. Is there anything that we didn't ask you about that you wanted to cover that that, that we can, we can touch on real quick?

Peter: No, I wouldn't know something immediately. I mean, you have to list the center before. Mm-hmm.

Jonathan: Yes. Our, our panic questions for when we can't think of anything at that particular moment. Yeah, I think we, I think we actually covered those fairly well. Yeah. So I do, I have to ask, and again, I know the answer to at least one of these what, what is your favorite text editor and scripting language?

Peter: Oh, you don't know the answer to the first one? You think you do? I think I do. No, I, I use, I use VI for administration uhhuh but I use Sublime Text for Development Uhhuh. Oh, okay. It's just Sublime is not great with doing things remotely. Sure. Hmm. And when you're in a shell fixing things, it's easier to just pop out fi.

Yeah, makes sense. But otherwise, youli text with the language server integration. So it talks to the c compiler for completion and information and introspection, et cetera. And the other ones the scripting language. I'm quite fond of Lua, which is great to embed in other software. It's in all the power in software too.

The DNS dis configuration is Lua. If you wanna do any, anything fancy in Recurs or it's Lua,

Jonathan: that works out well. I like, works out well for working with Open WRT stuff as well. 'cause so much of that is based on lu.

Peter: They're getting rid of that.

Jonathan: Are they really? Yes. Oh my goodness. I don't even, I won't even recognize the project anymore.

I'd be lost.

Peter: So they replace, they're replacing it with U Code, which at least spiritually is quite Lua like. Mm-hmm. But I guess it integrates better with what they have. They're building it themselves.

Jonathan: Hmm hmm. Very cool.

Peter: Besides that, I, I, I grab Python a lot because it's just so powerful in so little codes.

Yeah. But there's always a risk of that code then going to production and it's so slow. Yeah. It's, it's wasteful. But, but for anything quick that I wanna do for myself, I really like Python.

Jonathan: Yeah. Yeah. It's, it's ubiquitous at this point. Yes. Yeah. Alright, well, hey, we appreciate you being here, talking all about power, DNS and some of the other projects you've going on have going on.

It's been a lot of fun and I don't know, maybe in a few months we'll have to have you back and talk about what's changed and what's new. I, I think that'd be great.

Peter: Why not? Thank you for having me.

Jonathan: Yep. Thank you so much.

Dan: Dan, what do you think? I thought it was really interesting, really great to talk to Peter about so many different things.

Yeah. And I always like to know, I know I always ask that question, but I always like to know what got people into open source. Mm-hmm. And he had another great little story there about that, which I think is very interesting to hear. But I'm not convinced yet whether I'm gonna be hosting my own DNS next time we speak.

I'm I'm on the fence a little bit. Not, not because of anything to do with power. DNS power DNS seems great. I just, I'm not sure I trust myself as much as I need to for that.

Jonathan: I mean, you could, you could set up a a power DNS just resolver on your own network.

Dan: True. Yeah, that's true. Because yeah, maybe that's something I should do.

Jonathan: You're already, you're already hosting that, right? You just mm-hmm. Well, unless you've set all of your computers just to, to look over at Google, DNS, which no is, is a temptation. It's very easy to do because it does just work. But there is definitely something to be said for hosting your own DNS and not, not just giving all of that information away to Google.

Dan: Mm-hmm. Yeah. I, I mean, it's such a, an amazing project as well. Power, power, DNS, it is lit, as you've already said. I, even though before today, before I started really looking into it, I couldn't say I knew much about it. I've probably interacted with it millions of times without even knowing.

Jonathan: Yeah, it sounds like it, it runs some large-ish percentage of the internet.

Mm-hmm. So yeah, definitely. It's definitely one of those projects that. Is is sort of foundational as a building block in the internet. And I, I'm actually, I'm really glad to hear that they've got their, their funding model sorted and they could actually pay some people to work on it full time. That's, that's really encouraging to hear.

I like those stories.

Mm-hmm.

Yeah. Alright. Is there anything that you wanted to plug before we let folks go?

Dan: Same as last time, even though I haven't been on here for a long time, I haven't got any new plugs. So it's, it's, it's the back to the traditional ones, so, yeah. I mentioned Liverpool Make Fest, which is an event that I help to run.

Mm-hmm. It's a, it's a one day maker festival where we have all kinds of stuff from, I mean, we do have all the high tech stuff there. We've got drones, we've got robots and, and all kinds of other stuff, but everything right down to like crafting stuff people make out of wood mm-hmm. Or other things.

There's any kind of making is, is, is good with us. If you go to Liverpool make fest.org, you can find all the information about that, but it's coming upon July 5th, so it's getting closer. Yeah. And it's a one day event, so if anybody is near to. Liverpool and you want to come, it's completely free. Free as in beer, and free as in freedom.

So you can come along and, and come to the library and, and discover four whole floors of, of maker stuff. So yeah, go and check that out. That, that's my plug for this week.

Jonathan: And if somebody wants to follow you, is there a good place to do it?

Dan: Yeah dan lynch.org is the place to go where you can find all my social media stuff, which tends to mostly be mastered on these days.

Mm-hmm. But I, I still have a Twitter account and I occasionally post to it as well. Sorry, X as I supposed to. Yeah, I still have a few things on there and Instagram and other stuff, but yeah, dan lynch.org is the, gr is the best place to find anything about me.

Jonathan: Very good. Alright. Appreciate you being here, man.

Thank you. All right. If you wanna find more of me, there is of course Hackaday. We appreciate Hackaday being the home of Floss Weekly these days. I have the security column goes there every Friday morning. We have a lot of fun with that. And the occasional byline article that'll go live from me, but it's mostly Floss weekly and this week in security these days.

And then there's also the Untitled Linux Show over at twit tv. We just, we just had our 200th episode and Leo actually jumped in for a few minutes in the middle of it to help us celebrate. We told him when we get to episode 255, which will be a more mathematically significant number, at least for us, programmers, that we're gonna bring him in and make him, make him do the whole show with us.

And he said that sounded good. So look forward to that too. But you can check out ULS for Linux News and some how-tos, some tips, all kinds of fun stuff there. We appreciate everybody that watches and listens to Floss Weekly. Those will get us live and on the download, and we will see you next week. Bye.

Episode 831: Let's Have Lunch

30 april 2025 | 64 min

Episode 830 transcript

23 april 2025 | min

FLOSS-830

Jonathan: Hey folks. This week Randall joins me and we talk with Alan Furstenberg about Google's ai Open source, AI in general, vibe coding, and lots more. You don't wanna miss it. This is Floss Weekly, episode 830, recorded Tuesday, April the 22nd vibes. It is Time for Floss Weekly. That's the show about free Libre and open source software.

I'm your host, Jonathan Bennett, and today it's all about the vibes, man. Speaking of vibes, I've got a wonderful and talented co-host, the, the wonderful Mr. Randall Schwartz. That is the wrong. Where, where do you, where do you have that button? Not to, not to bury the lead, the button. Ah, not to bury the lead, but there we go.

Randall, welcome.

Randal: Yeah. Hi. Hi, Jonathan. Thanks for having me back. This is a lot of fun being able to. Not produced Floss Weekly. Yes. As you know, I did that for 13 years. Yes. So I'm very happy to just be the occasional drop in co-host and guest. I think it's a lot more fun this way. It is a blast. I kind of miss the, I miss the show a bit, but I've got my own weekly show that I do for two hours anyways, so I don't miss the, the the fact that there is a routine in my week always as a result of something.

Jonathan: Yes.

Randal: But but I'm sure false weekly. I'm glad, I'm glad I did those 13 years. Happy that you're doing it now. So thank you for taking up the reins.

Jonathan: Definitely one of these days we'll have you and Doc on the same show that that'll, that'll be interesting. Oh yeah. That'll be fun. Yeah, that'll be fun.

We'll do it. So I, I put the call out a couple of days ago and I said, I've been scheduling guests. I've got like three invitations out, but nobody's, nobody's committed for this Tuesday. What can we do? And Randall's like, I. I've been vibe coding. I'm like, oh, I hate it. That's great. Let's talk about it.

And then Randall says, oh, by the way, there's this guy over at Google that is doing some of their AI stuff, Alan Furstenberg. We'll see if I got that right. And so it is not just the two of us talking about vibe coding. We've actually got the three of us now talking about. Stuff going on that Google is doing with ai.

Alan has, I believe it's an AI slash llm di di directional open source project that we're gonna talk about as well. But let's go ahead, let's bring him on and let's the three of us sort of dive into this. And it's an episode today that's, it's all about the vibes. A great to be on. Yeah.

It's good to have you. So where, where do you, where do you fit into all of this? What's your, what is your project? What do you do at Google? What, what, so what's your, what's your feeling about the vibes?

Allen: So, so let's be clear about a couple of things. First. I don't work for Google.

Jonathan: Okay.

Allen: I I work for a, a small consulting company called Objective Consulting.

It's spiders.com. But like Randall, I'm what, what Google refers to as a Google developer expert.

Jonathan: Mm-hmm.

Allen: So we are a bunch of little over a thousand outside developers that work closely with Google. Mm-hmm. So we don't work for them. My paycheck doesn't come from them. But we work closely both with Google's engineers, Google's developer relations team, and with developers to kind of help bridge, make, make that bridge between the, the two organizations, you know, the, the two sets of people.

Jonathan: Sure.

Allen: Help developers understand what it is that Google can offer and help Google understand where they are falling short and need to help developers more.

Jonathan: Got it. And so do you do, are, do you work with with Flutter like Randall does or do you work with other Google stuff? I

Allen: do not. Randall and I, you know, we, we've, it feels like we've known each other for years at this point.

But we work in very, very different domains. A lot of the work that I do tends to be more on the, the server side of things. Historically I've worked with Google Glass, historically I've worked with Google Assistant. Mm-hmm. Neither of which had any flutter components at all really. These days I'm mostly doing work with their AI product with Gemini, with Gemma.

And in their cloud-based resources. Mm-hmm. So that, that's where you see most of my focus these days.

Jonathan: See, I noticed you mentioned Google Glass. You are, you are wearing a a Google glass headset, aren't you?

Allen: Yes. This is, this is one of the classic 12-year-old versions of Google Glass at this point. Wow.

I, I like to, to put it on every so often to, to kind of remind me. Where the focus needs to be sometimes, you know what, what that, that really ultra personal right computing platform can be like. And because I feel like there are some people out there that wouldn't recognize me if I didn't wear it.

Jonathan: It's a reminder to me of the future that might have been,

Allen: still will be.

I I still feel like it, it still will be.

Randal: Yeah. Yeah. Maybe if, if it ever gets to be lightweight enough and, and and, and, and comfortable enough and and easy and easily interacts with me on some level. I'm sure those are the big, you know, how do you handle the UX and something like that. The thing is, I

Allen: think the UX is, is always critical, right?

We could, we could look, we could. Sidetrack and go on that for an hour or so, but that's, that's way outside of of of floss types discussions at this point.

Jonathan: I think there is actually an interesting tie together there, because one of the things that really sort of doomed Google Glass back in the days is that people hated it.

Not the people that used it. The people that used it liked it, but everybody else that had to be around them, I'm sure you remember the term that that was coined Yes. For those that wore Google Glass. Other people hated it. And I, I, I never really quite understood why maybe it was the thought that all were always being recorded, which I don't think is quite true.

But it was just, it was just weird that people had such a, it was almost a visceral reaction to it.

Allen: There was that I, I tend to maintain that the biggest reason why Glass never got the adoption that it did was was because of the Google bus system. That was being set up at the time. Mm. 12 years ago.

So it created this very negative attitude around Google in the Bay Area. Mm. Where all of the tech reporters were. So the tech reporters had this narrative about, well, Google's evil and glass kind of got carried on with that.

Jonathan: Did somebody was wearing Google glass, you, you automatically knew, oh look, it's a, it's a Googler.

Or somebody Googled Google aligned.

Allen: Right. Every person that I ever spoke with about glass itself was, was always thrilled with it. You know, they always, you know mm-hmm. Once I explained to them after a couple of minutes what the UX was like and, and how it was not invading in their privacy and how Google was very careful about doing that.

They got it and they were excited about it. Mm-hmm. And I think we're now seeing that as we're seeing more and more head-mounted wearable platforms actually coming to market now.

Jonathan: So this idea of people being. Annoyed and offended by a product without really understanding what it's for. It seems like maybe there are some parallels between that and the AI slash LLM craze that we're in and what Google is doing with Gemini.

Is that too a stretch? Good segue.

Randal: Nice.

So tell us if I could, if I can jump in here just for a second. Yeah. I want to sort of also bring context in as to why I immediately thought of Alan to come on Today's show is that

Jonathan: mm-hmm.

Randal: Right now, at least in the flutter and angular world, we are going through about a three or four week push to talk more and more about how Gemini can be used in products and Gemma can be used in products and just AI in general can be used in, in products.

The relationship to open source of course, is that many of those products are open source. Mm-hmm. So it's kind of fun to be able to talk about that and how, how you can share your code. But then what are they, what are the, what does the recipient need if he wants to plug in, you know, 'cause he's gonna need a key, he's gonna need a you know, a Firebase account, things like that.

And, and so there's this push, and then there's also the push that's sort of happening in general. And the timing of this is because we're between next and io and we, a few things happened next, and I think Alan can probably explain a little better than me about what happened at Next. And we sort of anticipate some things that are gonna happen in io.

Of course, we won't know. We, well, we kind of know, but we don't know Ally know. So, you know, something Randall, I can't say I'm under Indiana

Allen: as Randall and I

Randal: about is what with Fred Protic? Yeah.

Allen: As Randall and I both like, like to say those, those who know don't say, and those who say don't know.

Randal: Right? Right. So have to be a little careful. So, so what happened, what, what happened next? And what's happening in the Gemini world? It's so rapid, it's so hard for me to keep up even with it all.

Why don't you give us a summary of sort of what's recent, what, what's sort of anticipated and, and where are we going with this and why is it important to people to know about this AI stuff?

Allen: So I think, I think the big the big buzzword of this year is, is agents and agentic. You know, everything is is agentic these days.

We're seeing lots and lots of agents. You know, the big announcements at next were all around this, you know, Google announced what they refer to as their agent development kit, which was a, a software package, an open source library for Python that helped people develop agents. This is similar to a lot of similar Agentic tool sets that you're seeing both from, from other companies like Microsoft and from startups like Lang Chain and it's Lang Chain's products that I tend to work with the most.

I help them a lot with their, their open source adaptations for Gemini. So that's where, where, where I come from mostly in a lot of open source stuff these days. Mm-hmm. But that's kind of, you know, the big, that's the, the big hotness is how do we build agents? How do we maintain agents, how do we get agents so that everyone can connect to everybody else's agents?

And Google released a, an open protocol called the Agent to Agent Protocol. At next, and this, it compliments something that is, is currently the, the big hotness in the AI space called the Model Control Protocol Model, CMCP. Where oh, I'm having

Jonathan: flashbacks to Tron.

Allen: Man, I, I, I tell you, I, I, I'm

Jonathan: trying to tron, I, I've

Allen: been trying to figure out for a while if that was intentional or not, and if it was intentional, why?

But it, it was very much their way of trying to create a standard way that people could build servers. That an individual who is writing an agent or using an AI system can attach to these servers and have it do things on their behalf. Mm-hmm. There's a lot of discussion about it. A lot of discussion about how this is a, a great thing.

There's some gaps in the protocol I, at least in my opinion. Mm-hmm. But one of the great things that it allows people to do is just kind of give these nice open-ended questions to a, to their personal agent and say, go figure out how to get the data that I just asked you about, or do what I just asked you about.

Find the MCP servers that are out there and take advantage of them.

Speaker 5: Hmm.

Jonathan: So this is when, when you, when you use the term agent I've heard that with ai, I couldn't give you a definition for that. What exactly do we mean when we're talking about an AI agent?

Allen: Well, I could give you my definition and my perspective.

Sure. And I'm sure Randall could give you his, and I'm sure if you invited two other people on the show, they'd give you theirs and they would all be slightly different. The way I like to think about agents at this point is that we aren't so much giving our AI systems I, I guess when I think about it, when we think of how we have traditionally developed software mm-hmm.

We are giving it very, very rigid rules, rigid mathematical rules. Mm-hmm. When we're talking about agents, we're giving them more broad rules and sets of tools to use, and it is the, the AI system that is determining how do we take those instructions that a user has given it, and how do we apply these tools to achieve the outcome.

So whereas before we might say, okay, here is the structured input that we need. Here are the algorithms that we will use. Here are the function calls that we will make. Here are the libraries that we will invoke to get a particular set, you know, out to get a particular output. Mm-hmm. Instead with an agent, we're saying, a user may say something, here are the tools that you have at your command.

You figure out how to invoke those tools to execute what, what they want you to do. It's the agent that is kind of the controlling system in there. It's the one that will work with the ai that, that communicates the instructions to the ai. It goes, you know, if the AI has said that it needs to invoke a tool, the agent is the one that actually invokes the tool, possibly with the user's additional permissions, possibly with other context or information that may be necessary, and sends the results back to the AI to determine what the next step is.

So, rather than have the determinism be in the code that we have written, the determinism is now in the AI system and the agent is just a master controller. So

Randal: it was kind of a, is that, and can I fill in, can I fill in the background please? A little bit too? I, I, I just wanna say how we got here was pretty interesting.

Mm-hmm. Because originally the LLMs were just big auto complete machines. They would you would say something and they would say something back and you'd say something to them and they'd say something back again. Until you're building this long context of whatever it's you were covering on. Then somebody got the clever idea of, Hey, if the LLM says something of a particular shape, like Wikipedia and an item tell it that.

If it says that, then I will go look up that Wikipedia item and give it the contents of the top part of the page. Mm-hmm. And all of a sudden now that becomes part of its, I don't want to use the th word thinking, but part of its way of, of auto completing is like, oh, it needs some Wikipedia information.

It now knows if it says Wikipedia, Randall Schwartz. Then I'm going to, as my next chat item say, here's the Wikipedia answer for Randall Schwartz, and it can then continue, its out of completion further and further with that. So this was the beginning of tools, and tools had sort of a really. Fuzzy protocol to start with, then they sort of more formalized that to, okay, here's the best practices we figured out about telling these lms, here's what you say and here's what you're gonna get back.

So that moved it up to the next level. And that was functions and that's all we had for a while. And people built some really cool stuff with that. But then MCP came along where now there's a standard protocol for registering the exchange rather than just, here's a tool and here's the command you use.

And so the CPS are becoming sort of this next level building block. And then you get these orchestration cps, which go even further and no, to call other ones in, in orchestration. And it started to get, and this is where Vibe Gooding comes in. I know I had to, I had to bring it all the way back to the original, original topic.

So, so that was that. Fill in if I've got anything wrong there, Holly. Well, I wanna, I wanna

Jonathan: ask, when we talk about agents then we're, it sounds more like what we're doing is we're giving AI's agency. The ability to go out and do stuff. Is that kind of where that, that term agent comes from? Yes. Is that, is that's, that's what it's getting at?

Allen: Yes. Okay. And, and the notion of agents and internet agents is not a new one. Mm-hmm. You know, I, I have a, a book behind me that's literally called Internet Agents. It was written in 1996, and it was discussing how do we start building what, what we would now call a bot. Mm-hmm. But that was where we started getting these notions of code that was semi-independently traversing the internet in search of resources and what it can do with those resources.

And over the years we have created formalized protocols to exchange information, you know, for, for people who remember what SOAP was like. That was a formalized protocol to. Enable. Yes. I see Randall, I'm sorry Randall.

Randal: You triggered me there. Stop that.

Allen: I know. It's, it's scary. But these were, you know, previous attempts to, to, to formalize and structure how we can locate and bring back structured information.

The, the big deal now is that our structured information doesn't need to be as structured as it once was.

Randal: Mm-hmm. It is sort of like we, we also in, in the middle there too, past past soap, there was rest sort of in general. So that we had basically he, here's a js o payload going this way. Here's a JS O payload coming back that allows you to have some sort of boundaries and delimiters and ability to decode it and encode in both ends.

So if you think of it MCP and these other a to a protocols that, that Google released last month are, are about. It's kind of the next thing of what rest would be replaced with. It's sort of like if we want to talk to something using a much looser sort of set of requests and a set of responses, then what would it look like?

So it's kind of at that level.

Allen: Mm-hmm. The things like the orientation protocol also enable even more independence for our, for our agents. So nowadays people tend to think of agents as as tied to a chat bot. I'm saying something to a chat bot. Mm-hmm. It's spending a few seconds going and getting the answer and it's coming back.

Increasingly what we're we're seeing are people coming up with these much more independent systems, that we will give them broad guidelines and they will kind of live on their own. Doing what they wanted every so often, sending us a message through the other, the other means sending us a text message, sending us an email saying, oh, by the way, I found out this information for you.

Or, oh, by the way, I just booked those concert tickets for you that I figured you would like, or, you know, things like this. So yeah, we're, we haven't quite gotten to mass adaptation of truly independent agents, but we are starting to see movement in that kind of a direction. Mm-hmm.

Jonathan: It seems like one of the things that, that sort of formalizing this protocol could also help you do is put guardrails around what your agent is allowed to do.

Yes, you're allowed to make an order from Amazon, but you're not allowed to order $10,000 from Amazon all at once.

Allen: Well, and I think this is actually where we start getting, again, we start bringing some of it back to, to to floss software.

Jonathan: Mm-hmm.

Allen: Is that it's kind of churning it loose, but it also does mean that people need to, to exercise that control on their own because there's no central, there's no centralized place for doing this.

Jonathan: Mm-hmm.

Allen: What agents and, and MCP servers are letting people do is on their own, in their chat system, say, here's what I want, and kind of letting things turn loose. And if you think of, you know, the, the very earliest days of open source and kind of the dreams that, that a lot of those developers were, were thinking about, they were thinking of open sources and, and an empowering tool.

Mm-hmm. As being able to say, coding is empowering you too. Command all of these resources have access to data. Mm-hmm. Do what you want. Mm-hmm. Now we're kind of saying the same thing, except we're saying you don't, you don't need to learn C anymore. Mm-hmm. You can, you can maybe, and I think this is where we're gonna start getting into our, our vibe, coding debate.

Maybe you could just issue English language commands.

Speaker 5: Yeah.

Allen: And control your agent through. Through natural language natural language should also be triggering to you, Randall.

Randal: Yeah. Yes. But, but, but the question is, will we end up with the Skynet or will we end up with an Starship Enterprise computer world?

I don't know which of the two this human interface is gonna get us to you know, which direction it could go. Well, we'll probably end with the Mad Max World before either of those, so we'll see.

Allen: Thanks for the optimism there. I, I,

Jonathan: yeah. So I've, I've got a, I've got a buddy that just this morning he was telling me his story.

Now he was using co-pilot, not one of the Google products, but he was asking copilot to look at some code and copilot says, Hey, you've got a memory leak here. It's some c plus plus code, I think. And he looks at it and he looks at it and goes, no copilot. You, you have access to this entire code base. This function is a void function.

If this function were a Boolean function, then yes, there could be a code a, a memory leak because of this. But there's not, and there's no way that it could be, and you should be able to see this. You told me this story just this morning. It's like that's sort of where we're at with vibe coding and AI writing code and looking at code.

Like it can be useful sometimes, but it still inherently doesn't actually understand. It doesn't actually have a clue what it's looking at. That's where I'm at with this. It doesn't,

Allen: as, as Randall said earlier, these are really just pattern systems. They're really, really sophisticated pattern systems, but they're just pattern systems.

And fortunately, coding also tends to be a pretty good pattern system. Sure. But it's not a perfect one. No. So it's, it's fairly easy to kind of trick the systems. Still Now that doesn't mean they're not helpful. Mm-hmm. You know, to, to a, to a developer, to a programmer, to somebody who's used to code and who is good at, you know, debugging their own code or somebody else's code, it's invaluable and it really can't help.

Still, you're in the right direction, but I'm still saying what I, what I, oh, go ahead Rick.

Randal: Go ahead. Yeah, I was gonna say what I, what I've found in practice over the last month or so, playing with these, especially my sort of go-to stack right now has been using RU code with VS code and using Gemini two five flash now for all of my little quick edits and Gemini two five Pro.

Occasionally when I wanted to architect or just kind of see the overall picture for things, what I've found is that it's sort of helpful in two directions. The one main direction is typing out the boring stuff, the bo the stuff that is just like everybody else has to build for the boilerplate. You know, how to build the data class and dart all the boilerplate, all that.

It's great at that because there's so many examples of that and there's so many correct examples of that. That's the key thing you gotta have here. Here's the problem. It's, it's, it's taking the average intelligence of everything it reads well. The, I tell you, the average intelligence of most code on, on on GitHub is mediocre.

Okay? So that's the best it can kind of do outta the box, but where it also fills in, and this is what I'm also finding very useful, is if I'm drawing a blank about a possibility for something, it's pretty good at just randomly, I. Generating things into that space that kind of make it help. And so I think as long as you're using it for these two precise things and aren't just straight out vibe cutting, I don't know what I'm doing, but I'm gonna say, make it through a box and do the thing over here on the right side.

You see, if you're doing that, I think you're gonna not get far enough, except accidentally.

Allen: Well, I have two thoughts there. One is I really like how you say it kind of randomly fills it in. 'cause that's very much how I feel sometimes about this is, you know, it will, I will usually I'm trying to do a complicated SQL query or, you know, debug something that already doesn't work and it gives me a suggestion and I'll look at it and I'm like, I wouldn't have thought of that, but you know what, let me, let me give it a try and mm-hmm.

See if that works. And you know, it works about 50% of the time. Yeah. But I need to. I still need to apply a lot of my own thinking to, to really know that that's, you know, where that 50% is gonna be. But we are seeing people who are at least claiming, you know, they can just give in long, detailed instructions in a natural language and they'll get back some code That mostly works.

You get some code, well that gets, that's, you know, it, it mostly works, it works well enough as a lot of people are saying. And I'm like, okay. But what I'm seeing a lot of people saying is, you know, they're taking their well enough code, which worked great for them. And then when someone says, oh, hey, wouldn't it also be cool if, you know, it was also, it could switch between red and blue?

And when they go and try to plug that into their description again, they get a completely different program that they don't understand anything about. They don't understand why the whole thing changed.

Jonathan: Yeah. So. We have a commenter in our discord that says something that's also very similar to what I'm thinking about as you guys are describing this, I'm thinking, oh, so it's like googling, finding something on Stack Overflow and then copying and pasting out of Stack Overflow into your program.

Yes. And XEF six says he's curious about the thoughts of LLM as a search engine versus LLM as a super intelligent consciousness. I think those are two very parallel thoughts.

Allen: So, so my opinion on that is it is neither, and we need to understand why it is neither.

Speaker 5: Mm-hmm.

Allen: The first is, it is, it is not a super intelligent anything.

It is not even a non-intelligent, there is no int intelligence because inside an LLM, right? None whatsoever. Let's be absolutely positively clear about this. It's a little complete, it's just auto complete. Right. It's a really, really, really good pattern system and the people behind it thought that if we could understand, and I use.

Understand in a human sense of the word, if we, we could teach a system language that we also teach a system intelligence. And what we have found, I think, is that if we have taught a system patterns of language, we have not taught it language and we certainly have not taught it. Intelligence.

Jonathan: Mm-hmm.

Allen: Yeah. The other thing to, to keep in mind is that it's also not a search engine. It is what, what I find it's really good at is it's good at understanding and I, I very much put those in air quotes. What a human has said and turning that into a structure. LLMs are really good at transformations like transforming natural language into a data structure.

Really, really good at that.

Jonathan: Mm-hmm.

Allen: But it's not good at actually finding knowledge 'cause it doesn't have knowledge. It has sentence patterns, it has word and letter patterns, and we don't know if those are accurate. We don't know if they're inaccurate, we don't know what they are. And it doesn't know. And it doesn't care.

All it knows is that statistically speaking, certain patterns happen more often than other patterns do.

Jonathan: Mm-hmm.

Allen: And we have kind of associated that with that means that certain patterns are more right, or more truthful or more accurate than other patterns are. And that's not always true. Certainly when we look at what humans have written over time, we write a lot of things that are wrong.

And now those have been incorporated into the large language models as just examples of language.

Randal: And one thing I think that's interesting here is that it's also though that can we accurately demonstrate that this, that what the LLM is doing with auto completion and nearby cosign angles and things like that, if that isn't also what our gray matter is doing.

You know, it's like maybe, maybe this appears to have intelligence because it does mimic in a lot of ways how like I don't have the end of my sentence already computed when I start a sentence. So somehow I'm actually auto completing this sentence. I have no idea how I do that, but it's what I do automatically.

And I'm just kinda wondering. But, but, but then, so what's the limit of what we can compute? Can we compute enough to make it close enough to a human brain? That's what I'm curious about. Right.

Allen: And I, and I think that's what the, the, the data scientists have been assuming. When you talk though to people who have, have studied things like linguistics and have studied cognitive science and who have studied the human side of this over the years.

Jonathan: Mm-hmm.

Allen: Most of them will say no. How LLMs work is nothing like how a human thinks. Yeah. How we approach questions and problem solving. We, you know, humans are wonderful pattern machines as well. Yeah. But that's not how we approach a lot of our problem solving and language skills. Mm-hmm. At least. And, and I am neither a linguist nor a cognitive scientist.

So I, I, I don't wanna really misrepresent I don't wanna misrepresent them.

Jonathan: Mm-hmm.

Allen: But that is my understanding from, from a lot of conversations with with both of those professions that I've, I've been able to, to do over the past several years.

Jonathan: I, I think we can sort of understand that intuitively as well.

Just like a, a really simple thought experiment. And that is there, there must be more to human consciousness than just language, because there are a few instances of people that, you know, were blind and deaf and therefore had no language, but still had consciousness. Well, but you, and much of our consciousness happens in sort of that visual space inside of our head that is not necessarily attached to language until we try to tell somebody about it.

So I think just intuitively we can sort of come to that same conclusion.

Allen: There, there are, there are cognitive scientists, however, that would argue, no. People who do not see or do not hear, do, have, or you know, or do not speak, do have a language. It may not be the same language that we would associate with a language, but they do have it.

And there, you know, there are sentence structures and there are linguistic structures that linguists and cognitive science can recognize and identify.

Jonathan: I, I think at some, I think at some point we're just, that argument just changes the definition of what language is to match whatever it is that we find and is therefore not very useful.

Allen: That could be, but that would also start including what an LLM is doing.

Jonathan: Or could it could, it could. So what what, what, what practically, how does this work If someone says, I want to get started vibe coding or something that I've thought about that I think makes a lot more sense. Can you guys make a good point?

That's okay. Getting boiler plate written is great. AI is a great for that. Something that I think makes a lot of sense is AI guided fuzzing where you say, Hey, I would like for this program to crash ai, look at the code, give me inputs, and then take the output for the program and feed it back into the AI and say, make it crash.

Spin through as many cycles you need to make it crash. And then, you know, you can keep track of your actual code coverage that way. AI guided fuzzing, I think that is a pretty interesting thing, and there's at least one or two companies out there working on that. But how when we, when we take an idea like this, how do we actually put rubber to the road and make it happen?

Allen: I think what we're seeing a lot of, and, and not so much from the automated test generation, 'cause we are seeing companies that are, are kind of looking at that. Mm-hmm. But from the, the code generation point of view, we're seeing companies like Cursor. Cursor is probably the biggest one out there these days that people are just feeding it descriptions and getting code out of it.

Mm-hmm. Even as something as simple as Google now has what they call Canvas as part of their Gemini chat app, which will generate and run code for you. Mm-hmm. So all of these are, you know, it's those kinds of tools that people are, are starting to use. You'll also see see things like what, what chat GPT calls their GPTs, where you're giving it descriptions and possibly some data and turning it loose.

The reason why MCP itself was created was so that you could just say as part of a, a chat system. I've got these mcps that are out there, use these in particular and go solve this problem. Mm-hmm. And in, in a way that is programming. Sure. I mean, I, I, I can't say it's not, you're giving instructions and you are logically trying to put together what you want your inputs and outputs to be, and that's, that's programming fundamentally.

Jonathan: Mm-hmm. Mm-hmm. Interesting.

Randal: Yeah. Related to the, the show here, since this is mostly about open source and about developers working with open source there are, I. If I'm writing open source software, I can certainly talk to Gemini. I might have to use a paid backend of some kind for most of what I wanna do.

But isn't there also models like it can't Gemma also be run completely on-prem and Gemma is a completely, can be

Allen: a completely local model. Yes. Gemma is what they, what Google refers to as an open model. And then, you know, I, if, if you have not had this conversation, I'm, I'm sure you're going to more and more you know, what, what is the difference between an open model and an open source model and, and what is,

Jonathan: we had some guys from OSI talking about what open source AI means and the debate over that.

Allen: Yeah.

Jonathan: Not a subtle question yet.

Allen: It's, it's not, yeah. And it's complicated. It's, it's really taking, it's really taking on a whole new shape. You know, it's, it's requiring other. It's requiring is to create new definitions. Mm-hmm. But Gemma is what they refer to as an open model. So you're able to download the model's weights.

Jonathan: Mm-hmm.

Allen: You're able to modify those weights yourself and, and tune them as you wish. And you can do additional fine tuning. And that's giving, the way I like to think of, of fine tuning is giving these models new words, new language, new patterns to help shape how the existing patterns already work.

So you're, you're building up a new layer of patterning that is tailored to your needs. And when you do these or when you adjust the weights, you are, you are skewing it so that it understands. And again, I use that, that term with air quotes. You better. Yeah.

Randal: Yes. So, so Gemma is basically providing sort of the core.

Here's, here's how English works, or here's how it, whatever language you're speaking to it in works and, and here's some core knowledge. Not a lot. I mean, I can tell you probably how many continents there are on the earth and a few other sort of common data, but it's not really a repository of data. You can use, can you use rags with Gemma?

You can.

Allen: Well, the, the beauty of regs, what is it, is that they are, they're a separate component. They're basically the, the way I like to think of rag or retrieval augmented generation is that you use your LLM to understand again, what it is that the A person has asked for and turn that into a data structure of some sort.

Mm-hmm. Once you have it in a data structure, we as developers know what to do with it. You know, we're, we're used to data structures, so we can feed that into a keyword search engine or feed that into. A nearby, you know, a nearest neighbor algorithm or do something with it, transform it into another data structure, and then take that data structure, feed it into an LLM again to turn it into something more human-like.

So that, and that's really the, the place where I see LLMs is their biggest use, is that transformation of human to machine and then machine back into human.

Randal: So maybe like if I had a mobile app that had Gemma running on it to kind of interact with me about maybe finding a good place to eat and so I'm interacting with it.

So it, it's kind of understanding the search parameters. Then it could create a more formalized search request to Google Maps or Yelp or whatever the back ends are. That one use exactly to provide the real time, accurate data. And then it could repackage that back to me. In some sort of format that makes sense.

Like in like in with my search history and with maybe some notes about the previous time I went there or whatever. Exactly. In terms of a local data store. So Gemma could be that glue in the middle in your device.

Allen: Exactly. And, and the advantage with Gemma is a, it runs locally on your device. But b, we can also teach it new words and new patterns so that we would be able to teach it things, you know, when we're referring to food specific terms, you know, when we're talking about different kinds of cuisine, we can help it associate the fact that, you know, things like linguini and carbonara and, you know, all of these other specific dishes would be associated with Italian cuisine.

Mm-hmm. Or any of, a bunch of other words would be associated with other cuisines and it may not have been trained on those. So we are giving it those additional patterns to think about. Now that's not giving it the answers, we're not training it with where those restaurants are located mm-hmm. Or exactly what the names of those restaurants are.

But we are training it with a better understanding. And I, again, I have to emphasize the air quotes for the audio audience here.

Randal: Yeah, yeah.

Allen: A better understanding of the fact about what. When I'm referring to, you know, when I say, you know, it would be great if I had a really good mozzarella tonight.

What does that mean? What, you know, what, what should it go looking for? 'cause if it just did a search for mozzarella, it's not gonna find it.

Jonathan: Did you mean mozzarella?

Allen: Well, I don't know where you, you're out west, aren't you? Yeah,

Jonathan: I was central MidAmerica.

Allen: Yeah. So, so when I talk about a mozzarella, you know that yes, it's mozzarella, but, but somebody may not say it that way.

Mm-hmm. Or type it that way.

Jonathan: Yes.

Allen: And, and that's the training that we may need to give an LLM.

Jonathan: Yep. Yep. Doing something like Gemma, if someone really, really wanted to run it. OnPrem, what's sort of the system requirements? Is this something you can run literally on your phone? Are we talking about a desktop?

Do we need three top end GPUs? What, where does that sort of come down?

Allen: So I don't know, r have you played much with Gemma on, on device?

Randal: I have, and I've seen other people do it, but I haven't done it myself.

Allen: We are, we are certainly starting to see the most recent generations of Gemma are able to run on fairly low level devices.

Mm-hmm. Meta announced with, with their Llama four that it could run on devices that have a, a single GPU that it would run on just one GPU and the memory in that GPU, which is mm-hmm. Really, really impressive. So we're seeing some of these seeing some of these models being able to be shoehorned into the lowest level devices.

I haven't tested Gemma itself. I, on, on a mobile device. I've tested it on my laptop, which is, you know, a, a Mac from a couple years ago.

Jonathan: Mm-hmm.

Allen: And it works really well. Hmm. You know, the performance on, on the Mac. Is really quite good. And I think as we're seeing more and more devices that are being able to handle it, you know, it is almost guaranteed that the next few generations of both Apple's devices and Google's devices mm-hmm.

We'll be having dedicated hardware that can handle some of the processing. We're gonna start seeing it working better and better on device. We're not gonna see the same capabilities that we've seen from, you know Gemini running on a server somewhere. But we are gonna see some impressive stuff.

And I think what we may start seeing over time is kind of a, a division of labor so that, you know, if it can run it quickly locally, it will if it has a network, maybe it's gonna farm some of the tasks out to the network interface mm-hmm. To, to run it remotely and then process it again locally. So we'll, we'll start seeing that split, you know, maybe some of the more intensive tasks it will try to do remotely if it can.

Jonathan: Mm-hmm.

Allen: And the easier stuff it'll take care of locally. Locally. And there's gonna be a lot of software associated with, with, how do we split that up? When do we split that up? Mm-hmm. What are the questions we can handle locally and what, what do we need to handle, you know, what do we need to form out?

Randal: Like, like say in code editing, you'd probably want auto completions to be done locally and quickly as opposed to say architecting a new rollout of an app or a new feature for an app that can take time and can be taken time on a server.

Jonathan: Mm-hmm.

Randal: So might, you might divide the LA labor that way, where we do auto completions all in local devices.

Jonathan: Yeah. We, I, I talked with Drss on an episode several weeks ago, and he's got this idea that he's been working on and trying, trying to find some people to work with him on it. And that is he calls it personal ai. And it's the idea of, I don't want, I don't care about all this stuff on the internet. I care about my stuff.

And so we were talking about this idea of, well, what if we take all of the pictures that I've ever taken and all of the receipts that I have, like scan all my receipts, scan all my bills, and give all my pictures to this I, this AI that's mine. And we were trying to figure out like, how, how would that even work?

Would this be built into a NAS that you have sitting on your desk? And that, that actually seems to be one of the, one of the ways that someone could go about trying to build this. But then what ideally you could do is you could then say to your personal ai, Hey, I have a charge on my credit card for $85 and 73 cents.

What do you think that's for? It would have all of that stuff already sucked in and be able to go and say, oh, you went and got, you know, guests the other day at this place. And that's probably what that is. One of the things that comes to mind here is, especially with this idea of running them offline and locally, how close are we to being able to build something like that?

What would be involved with building your own personal AI using say something like Gemma, and I'm sure a bunch of other open source libraries that you would sort of duct tape together to make that work.

Allen: Well, I, I don't know. This is your thoughts on this one, Randall. 'cause I'm, I'm curious what you would say on it first.

Randal: I, I would, I'm ju I, I think, I think we're close to that. In fact, I think there's already companies advertising having a, like a, a pendant that you're wearing all the time. No, seriously. I know there are went by that for one. Yeah. Right. Yeah. And, and that way everything you say, everything you say around you, you all gets recorded.

Mm-hmm. And now you've got access to it later in, in some sort of you know, natural language query. I'm not sure I'd want one of those. Right. And in fact, like I said, I, I, I went to the website and now I'm on their mailing list. So I get annoyed by, oh, we came up with a new version of it. It's crazy now or something, I dunno.

But, but longer battery life, whatever. And so I'm finding out about advancements that are happening that I am not taking advantage of in my crazy world yet. So, i, I, I think so. So I think the technology is getting to the point where we could have personal assistance. I'm thinking what's, is it Jarvis?

Jarvis Iron? Yeah. Ironman. Yeah. That, that, that's sort of like thing you interact with mm-hmm. That knows everything that you need to know in the moment and can sort of help you out with that. I, I, I, I, I, yeah. I think, I think we're getting closer and closer to that. And I, I think that's also scarier and scarier.

It's gonna come close to Skynet at some point. Yeah.

Allen: Well, but the advantage is if it does, you know, if it does become Skynet, it's, it's purely local. You could kill it, you know, as long as you don't, maybe as long as you don't give an access to the network, you're fine. I, I think we are, I, I think we're there, to be honest.

I think we're there. I think that the biggest challenge really is that we, we always seem to have this, you know, this ebb and flow. You know, we. We get, you know, we started with everything you know, is, is centralized on the network and we take advantage of the the, the, the, the magnitudes, you know, the, the opportunities of scale that, that, that affords us.

But then we get concerned about everything being centralized. So we need to distribute it all and bring it back local. And then we get concerned about the fact that, you know, we've got everything local, but we've got no reliable backup or safety mechanism. So we move it out again, and then we get concerned about the privacy.

So we move it local. You know, we, we always have this continuous. Back and forth.

Jonathan: Yep.

Allen: And I feel like I've seen this happen Good four or five times now.

Randal: Someone wants to commented that you could set up a tidal tidal wave machine that would actually make energy based on that whoosh you back and forth over and over again.

Yeah.

Allen: So, you know, I don't think we have real technological challenges to running things locally. What we do have is the ability to really make that convenient and easy right now.

Jonathan: Yes. Nobody, nobody is building it as a product. Right. People want a product. Nobody's

Allen: building. Nobody's building it as an appliance.

Jonathan: Yes.

Allen: Because it's not like, I don't wanna run this on my nas, I wanna buy a box

Speaker 5: mm-hmm.

Allen: And put it in my house and I want it to just work. Yeah. You know, I, I don't know about you. I used to run my own mail server. There's a reason I outsource that to Google a long time ago. And I think it's the same sort of thing is I just want this box sitting there that will just work.

Jonathan: Mm-hmm.

Allen: And that was kind of why I liked Glass and kind of why I liked Google Assistant

Jonathan: mm-hmm.

Allen: Is because they were these things that were, were getting me there and they just worked until they didn't. And I feel like,

Randal: I think if you, I think if you want a, a single box in your home that's that smart, you haven't watched enough horror computer films or, I mean, demon seed comes to mind right away.

I mean, things like that, you gotta be careful if you're gonna have something smart in your home.

Allen: But I, I, I think though, this gets back to the other, the other issue that we keep skirting around and that is vibe coating, is that in a sense I want to be able to have this box that I can just say. Oh, I need this software to do this for me right now.

Mm-hmm. Figure out how to do that and do it. And I feel like that's the really, really, really long game when we're talking about vibe coding.

Jonathan: Mm-hmm.

Allen: Right. And kind of what, you know, again, and again, this ties back to the origins of open source, is empowering everybody to just have this thing that does exactly what you want.

And it doesn't matter if nobody else uses it, you know? It, it doesn't matter. Yeah. I, I, if I've written a program that does exactly what I wanted to do, doesn't matter if it meets your need. It doesn't matter if it's this, if it's 80% complete. Mm-hmm. Because that's the only, you know, I only care about that 80%.

If you need something else, you go ask your LLM to do it. I don't care.

Randal: And one, one of the things I, I've been playing with recently is the, is the new streaming stuff. So I think they call it Astra in one format or another. Yeah. But being able, being able to have Gemini in real time, watch out my camera or on my screen and talk to me about what it's seeing and what, how it can help me, I, I think that level of assistant is gonna be a lot more interesting to embed than just something where I'm typing at it occasionally saying something.

Allen: Right. And one of one, you know, we saw this demo at IO last year and I'm, Google's been hinting about it in their, their current crop of glasses. I'm sure we're gonna see this again at this year's io, this notion that, you know, you could just be wearing a pair of glasses and wandering around and suddenly saying, where did I leave my keys?

And your ai, which has been watching you all along, will tell you where your keys are.

Jonathan: So that's very similar. That is very similar to what Microsoft is doing with recall. Yes. And I'm afraid it's going to have the same problem. Yes. The, the, the thing that killed, the thing that killed recall, and I don't, Microsoft's trying to bring it back and I don't know if they're gonna succeed with it or not.

The thing that killed it is that people heard what Microsoft were gonna do and they just, again, they had this visceral reaction of, Ooh, I don't want my computer to be taking screenshots of my desktop every three seconds. I know what shows up on my desktop. I don't want pictures of that taken. Right. It's, it's, it was creepy and I'm afraid it was that, you know, when you, when you get into the AI stuff, in that same, it's kind of that same vein, people are gonna go, that's a little creepy.

Some of us go, that's really cool. Yeah. But a lot of people are gonna find it as creepy as the Google

Allen: Glass. Well, well, but one of the things I'd like to point out about Google Glass is, you know, 12 years ago everyone was saying, well, I don't know, you know, cameras pointing everywhere, or kind of creepy.

And I'm like, you know, 12 years on, nobody cares. The fact that I'm walking around with cameras pointing everywhere.

Jonathan: Hmm. Right. We've, we've gotten

Allen: past that, you know, 12 years ago, it used to be that when you took a picture on your phone, it needed to make an audible noise.

Jonathan: Hmm.

Allen: There was legislation to that effect, and nowadays we just kind of assume people are taking pictures all over the place.

What's the attitude gonna be like in 12 years? I think my, what I think about when I think about that though, is okay, if I've got something that is recording. Consistent, you know, continuously what I'm seeing continuously. And it's not taking the snapshot every minute. It's taking, you know, maybe a snapshot every second.

Mm-hmm. What is the data cap? You know, how much data are we now collecting? Most of it will never be used.

Jonathan: Mm-hmm.

Allen: But how much data am I collecting? How much data do I need to store? And I feel like that's where the notion of storing it all local starts to get a little bit complicated.

Jonathan: Yeah.

Allen: Because now we're talking petabytes of data for an indeterminate amount of time.

Jonathan: You're, you're also talking about a lot of a lot of CPU or GPU cycles. If you don't wanna just store them as JPEGs or video, but you want to, you know, turn them into a format that your AI really can natively understand them well, it's a lot less storage, but man, that takes a lot of, a lot of cycles to, to make that transformation happen.

And that

Allen: turns into a lot of power.

Jonathan: Yes.

Allen: And that's, you know, yes. On one hand we can distribute that, you know, if you're putting it local, okay. That's now distributed power consumption.

Jonathan: Mm-hmm.

Allen: But now that's a lot of locally billable, distributed power that everyone is suddenly doing, and that's, that's dangerous.

You know, that, that's, that's a bad road to head down.

Randal: Yeah, sure. There's, there's also the other meaning of power there. And I actually posted on this on Facebook a couple days ago, which is I wonder if we've hit the knee in the curve with ai that the amount of power consumed by AI now exceeds blockchain.

Yeah, because it, it's gonna be some crossing point at some point, and I don't know if we've hit it there or not yet, but and then somebody was telling me they were running an LLM over blockchain and I went, it's like, let's just burn all the remaining

Jonathan: electrons in the universe. Big brain, big brain move there.

Yeah. That's great. AI over blockchain. That is, that is definitely a it's a sign of the times that we live in that those are the two things that they put together. That's

Randal: great. I, I'm curious Ellen just crystal balling at five years from now, maybe far might be too far, what, what's your, what's your best and worst scenario for five years from now driving v eights

Jonathan: through the desert?

Allen: I, I'm, you know, I'm really not sure I'm the best person to ask about forecasting for the future. 'cause I was the person who, when they, when they saw Glass, said that that's the future. And when I saw Google Assistant, I said that, that's the future, that's the follow one from glass. Oh, what's the term?

Randal: Well,

Allen: so, you know, maybe, maybe I'm not the best person to do prognostication, but what I, I truly do see that.

We're going to see these ultra personal devices. You know, the, what, what we now think of as a phone. I don't think it will look like a phone in five years. The phone's a terrible form factor in a lot of ways, so I think we're going to see, we may call it a phone. It's not gonna be a phone. It's going to be driven primarily.

It may, you know, it probably will have some visual interface, but that won't be the primary interface for it. The user experience for it will be natural conversational mostly audible, although not necessarily all audible and very, very multimodal. So it will be audio when it's appropriate, and video when it's appropriate and tactile where it's appropriate.

We will have multiple ways of, of input and output, but they will all feel far, far more natural. Than anything we have today. And that's gonna be, you know, AI itself will fall away. It will just be an inherent part of what we're doing. And we, you know, we won't be thinking of it as this is an ai, this is an LLM, we are using this model.

Mm-hmm. It will just be a core part of the operating system. And we interact with it as we would interact with anything else. It, it becomes, it, it kind of goes away, which is where it should.

Jonathan: We've seen that, where computer should, we've seen that with so many bubbles throughout the years. You know, personal computers was a huge bubble and then the bubble popped, but everybody's got personal computers, right?

The web, you know, dot com bubble was a huge bubble and it popped. But man, the web has still changed everything. And so, you're right, these, these things, they do kind of, they go away, but they never go away because everything is different afterwards.

Allen: Yep.

Jonathan: I, I think it's, it's inevitably going to be the same with ai.

Like the, we, we had a we had a lawyer on the show a few weeks back, and we were talking about some of the legal questions with ai. Oh my goodness. And, and his, his opinion was, it, it doesn't matter. You cannot put the genie back in the bottle. It does not matter what the court say. You cannot put the genie back into the bottle, and they will eventually have to come to that conclusion.

Yeah, that's probably true. Agree. That's probably true. So you've got a you've got an open source library. I was trying to I was trying to guess what it was about and get you to, to plug it. And what is Lang Chain AI and Lang Chain js What is all this

Allen: about? So, Lang Chain, it's not my library. It's, I am, I am a tiny, tiny contributor to it.

Jonathan: Gotcha.

Allen: But that's, and but that's very much the nature of open source, right? Yes. Is, you know, you've got many hands make light work.

Jonathan: Mm-hmm.

Allen: So Lang Chain was kind of one of the, the early libraries that was out there that was helping developers use some of these AI tools and tap into AI tools.

Mm-hmm. One of the first things they were, they were one of the first companies, it, it's run by a company called Lang Chain the project, but the, the project was one of the first ones to say, look, if you're using Open AI's model, or if you're using perplexities model, or if you're using any of these models, doesn't matter.

They're providing their, their API, we're gonna provide an abstracted API for you to use. And then as part of that abstraction, we're gonna come up with this concept called chains. So that you're essentially, you're, you're taking your input and you're passing it through stages one after the other. And one of those stages is sending it through an LLM.

Whereas other stages may just be things like. Taking the input from a human and combining it with a template. And other ones may be taking the output from the LLM and extracting chunks from it, and other ones may be, you know, combining it with a data store or sending this to a vector database or whatever.

So it, it has these abstractions that you could put together in chains. Mm-hmm. Hence the name Lang Chain. Over time it's really grown and it was one of the first ones, for example, to illustrate the concepts of what we would now call tool calling or function calling. Mm-hmm. Before the LLMs implemented that feature themselves, it implemented structured outputs.

You know, it, it found ways to kind of convince the LLMs to structure things and use them before the LLMs had the built, that built in as a feature. It started, it was used to illustrate. What we would now call thinking models or reasoning models mm-hmm. Before the models did it themselves. And now it was the one that really started introducing the notion of react agents and, and had built in modules to do that before the models started getting agentic and before everyone else started coming up with AG Agentic libraries, Lang Chain was the first ones to do that.

It then spun off another project called Lang Graph, which takes a, a very, very computer science notion of a directed cyclic graph.

Jonathan: Mm-hmm.

Allen: Structures it in a way that makes it useful for agent building. My involvement with it is this teeny tiny part of the whole thing that basically is the compatibility layer for Google's, Gemini and, and parts of Google's Gemma

Jonathan: Uhhuh.

Allen: So if you are, if you're using it to talk to Gemini or using it to talk to Google's Vertex platform, that's my code. That's been my involvement all along. And it's it's significant. It's a significant chunk of code, but what's great about it and what, you know, one of the things that I, I like about in working with Floss is talking to people who have different perspectives than I do.

Jonathan: Mm-hmm.

Allen: And being able to learn how to negotiate well, what's the right approach to this? Here's how one system does it, here's how another system does it. What's the right abstraction? Can we find a right? Is there a right abstraction? What's the right way to do this? And how do we build our, our very different libraries so that they all at least appear to work the same to other developers?

How do we write tests? How do we write documentation? How do we help people understand this? How do we write those articles? Everything that goes into really, really good open libraries and systems. That's been a tremendous learning opportunity for me and, and why I love participating with this project.

Jonathan: Yeah. Oh, that's an interesting question. How, how do you write test cases for ai? Like, what, what does that even look like?

Allen: That's oddly enough. It's really, really tough.

Jonathan: Yeah.

Allen: A lot of what you do, because these are non-deterministic systems. If I give it a problem, you know, if I ask it a question, the answer it gives me tomorrow is different than the one it gave me.

Yesterday.

Jonathan: I was doing some reading about that particular problem, so apparently a lot of ai, ah, what, I forget the term that they use, there's a, there's a term, it's basically, it's a knob inside the AI for temperature maybe, right? So temperature, yeah. Yeah. And so if you turn the temperature like all the way to zero, it's supposed to give you the same answer every time.

And it turns out it's supposed to, like, theoretically it's supposed to, but it turns out that many AI models are just incapable of doing that because the, the, the data, the source essentially comes off the disc at, at non, in non-deterministic speeds. And so those bits of backend code get loaded into different places in the ram and you get non-determinism out of that and all that.

That fascinated me to read about that. So very much.

Allen: What, what's even worse though, is in some cases. When you're doing things like you want to test the temperature knob to see if it works, to see if you've sent it correctly. You know, one of the most interesting things I've seen are people discussing how to test the output is to say, well, what you do then is you take the input, you take the output, feed that into an LLM and say, did this answer the question?

And that's a, to me, a very strange approach. It's kind of a, that's kind of a scary there's, isn't it? But, but

Jonathan: yeah.

Allen: The data scientists are telling me that there's solid math behind doing this. But yeah, it's kind of scary to say

Randal: lms, LMS as I'm vibe coding lms, do act, do, act as if I can do meta conversations with them about what they did.

And I, I, I, I appreciate that and I, I take advantage of that all the time. Of course. I'm also the same guy who says, please and thank you when they get the right answer. I'm a little weird that way too. So I, you know,

Allen: a story about that one, when I first started doing demonstrations with Google Assistant it was not, you know, some of my code was not working well, it wasn't hearing me correctly.

And when it did work, I would say thank you. And I had an audience member ask me, when you say thank you, is that reinforcement training that it, it got it right? And I'm like, no, it is in no way that smart. And they're still not, in fact the, the head of Open AI recently said. That people saying, please and thank you are, it's, it's millions of dollars worth of energy.

Millions of dollars going into handling please, and thank you.

Jonathan: That's hilarious. That's hilarious. Well,

Randal: I've learned it, I've learned it for, for my my my phone device here, which I, I won't mention the name of the lady that would respond to it from my phone here. But and don't, don't, don't say out loud because we can't.

Oh no. It's only going in my ears. It's safe. I gotta release sometimes when I'm on a call and somebody's gonna trigger my phone or not. Usually I turn my phone upside down, but, but I've learned to say thank you at the end. Because it stops at listening. Hmm. And so it's been a positive feedback to me to do that every time.

So because I, because otherwise if, if I don't finish the conversation, it continues to listen to me maybe playing a videotape or something. Videotape. What the hell's those playing? It's,

I'm dating myself. No way. I'm not just dating myself. I'm carbon dating myself. So anyway but when I'm playing, I'm playing a video. Those words will end up in the, in the onboard device if I don't say thank you at the end of my conversation with the device. So I have learned to say thank you. It is crazy.

Jonathan: Yeah. So you were talking about LLMs testing, LLMs. We've got a question in the chat room actually that goes along with this. And it also reminded me of, of something that I've heard about, and I think maybe all these three are kind of tied together, and that is the idea that LLMs, sometimes there's been some instances of this where like they'll get the right answer, but they do it in a way that nobody understands how they did it.

Like they, they didn't do it the right way, but somehow managed to come up with the right answer. This idea of an LLM testing, another LLM, the thing I would be afraid of is the, the test LLM would start saying, yes, that is correct. It gave you the right answer, but a human will look at it and go, no, not at all.

Right. And then the, the question the mashed potato has is what happens when AI starts ingesting other AI slop? And I think all three of those sort of, sort of fit together. Like what, what does that world look like when we run out of we run out of human stuff that we've made and we start doing training with?

We already have,

Randal: we, we already have, they've already been announcing that we kind of have pretty much consumed all of the human produced output out there already. I heard somebody in one form or another,

Jonathan: I heard somebody talking about like simulated training data or something really weird sounding like that, that apparently it's what they do when they, when they run out of human stuff.

Yeah. Yes.

Allen: Yeah. Once again, you know, the, the data scientists tell me that the simulated data is, is actually good. I don't quite understand it. I'm still trying to wrap my head around it, but they tell me it's okay. That the bigger problem is that when we see that there was, that was some research that was done.

I feel like it was about a year, year and a half ago now at this point. That was saying that when you overtrained the AI on generated data, increasingly gen increasing generations got worse and worse. Mm-hmm. We saw clear regressions in their understanding. Again, the air quotes. Of the content and there, especially in their generation mm-hmm.

Of content. And that kind of makes sense because again, you're now reinforcing it. You're, you're overtraining it on certain patterns and those patterns are not necessarily the best patterns.

Jonathan: Yeah. You kinda have to figure that when an AI produces something, it puts patterns in there that maybe we as humans don't pick up on, but other AI does pick up on those patterns.

I guess that's kind of the thread that ties all that together.

Allen: That's some of that. And I think some of what we're also seeing is the, the various companies are, are working on ways on how you can identify AI generated works. Mm-hmm. And the way they're doing that is by skewing the probabilities of what gets generated.

It's is one of the content ID techniques that are out there. And then of course you, you get people saying things like, oh, the M dash, you see too many M dashes. That's clearly a, a chat GPT thing. Whether that's true or not, I

Jonathan: I, I like using dashes in my writing. Kim dashes. Yeah. Well,

Allen: are are we sure that you're not an LLMI mean, you know.

Jonathan: Yeah. So I'm an avatar that works out. I, that it, that is one thing that I've taken quite a strong stand on. None of my, none of my work that you ever read or see anywhere is AI generated at all. And because of the things that I do, that's sort of a stand that I've had to take. So you can, if, if you believe that I am actually here and the one talking to you, then there you go.

Randal: Are we absolutely sure? Are we absolutely sure.

Jonathan: Anyway. Wow. So you've got a you've got a podcast that you, you wanna plug real quick and it sounds, it sounds pretty interesting. What's, what is your podcast and what's it about?

Allen: So it is the two voice devs, DEVS. Mm-hmm. For, for developers. Two voice devs podcast.

Jonathan: Yeah.

Allen: Where my, my co-host, Mark Tucker and I, or one of us and a guest host Will, will come on and we will be talking about pretty much what's going on in the AI world. We started about coming up on five years ago now Wow. Where we were mostly helping developers understand how to create good user experiences with Alexa and Google Assistant.

Mm-hmm. And now we've kind of like everyone else, we've, we've moved into the AI world, but we're kind of seeing it come around full circle, that the new Alexa and the new Google Assistant under the name Gemini

Jonathan: mm-hmm.

Allen: Are these AI systems at heart. So we're helping developers and we are very much a developer show, understand how we can build with ai, how we use these AI tools or how we use AI interfaces to help us.

Really kind of meet the needs of of people to be understood better how do, how we can deliver better products using, using this new UX paradigm.

Jonathan: Mm-hmm.

Allen: You can find us on, on YouTube your favorite podcast location. Or follow me on LinkedIn. Great ways to, to follow me and I post when we have a new episode.

Jonathan: Yeah. And what is what is prisoner.com and spiders.com?

Allen: So, spiders.com is the company that I work for. Mm-hmm. Objective Consulting. We do, we're, we're a small consulting shop. We mostly help our clients understand how to use this new technology.

Jonathan: Mm-hmm.

Allen: And it's, you know, how do you, where when we say this new technology, we mean this new technology for the past 25 years.

So, you know, when a client comes to us and says, what is this voice thing? Or what is this web thing? We've been there to help them kind of understand it and build and deploy in these new environments. Mm-hmm. Spite, prisoner.com is my, my personal domain, and you can find links to wherever I am. The blogs that I'm posting, the, the stuff that I'm posting.

You can find everything from from prisoner.com.

Jonathan: Perfect. We appreciate you being here and putting up with my dumb questions about ai. And there, there

Allen: aren't, you know, one, one of the things I like to point out is we're all learning this still. This, you know, even people who are experts in this field are learning new stuff all the time.

So there really more than ever, there are no dumb questions. You, you know what? There's stuff that we're all trying to figure out.

Jonathan: You know, what I've discovered is a superpower, being willing to ask dumb questions. It is a superpower. How many times I've gotten myself in trouble by thinking, oh, I would love to ask this question, but somebody's gonna think I'm dumb or don't know what I'm talking about.

If I ask, just ask the damn question. You'll learn way more about by asking the questions rather than trying to sit there and try to look smart. Asking dumb questions is a superpower, and I, I, I learned so much on this show particularly, but just in general by being willing to, being willing to look a little dumb and figure out what are going on.

Same,

Allen: same thing on my podcast is, you know, I'll have a guest host on and they're saying something. I'm like, say that again for me slowly, because I didn't get it the first three times.

Jonathan: Yes, yes, absolutely. Thank you so

Allen: much for having me on. Yeah. Really appreciate. Yeah. We appreciate you being here.

Jonathan: Yes, very much so.

Did I, did I say your name right? Is it Furstenberg? Is that close? You got it. Alright, Alan Furstenberg, thank you so much, sir. We appreciate you. Thank you. All right,

Randal: Randall,

Jonathan: what do you think?

Randal: Wow. So we didn't talk much about vibe coding, but I think we got the point across pretty clearly in the middle of that.

I'm, I'm glad for the other stuff that we filled in though. Mm-hmm. I mean, it's, it's, it's good to get kind of like the background and mm-hmm. And of course, Alan spends a lot more time heads down with that stuff. I'm, I'm sort of looking at it only from the perspective of how do I put it on a mobile device and, and how do I drop it into my Flutter app because that's my audience, my GE audience that I'm speaking to.

Mm-hmm. But I, but I've had a personal fascination with this for decades. 'cause I remember the first markoff chains, which mm-hmm. Which became the sort of for all the auto completion stuff going on Yeah. And how a lot of the early markoff stuff was done in Pearl. 'cause it was like two, two stings next to each other.

Mm-hmm. And see what their correlations are. Hey, sounds exactly what a pearl as array is, huh? Mm-hmm. All right. Yeah, exactly. So what, but what, what I've always had a sense of, and I was calling this 30 years ago, was that in order for us to get smarter systems in things that did more than what programmers were able to achieve traditionally mm-hmm.

We would have to ultimately create a system that we couldn't debug by single stepping it. Hmm. And I wanna explain that very carefully. When we write normal code, we can pull up the debugger and go step, step, step, step, step and kind of see where it finally went off rails. Mm-hmm. We can't do that with an LLM.

It's become so complex that there is no equivalent mm-hmm. To single stepping it to actually

Jonathan: doing one thing at a time. You, you can single step it, but the data that's in there is just totally incomprehensible.

Randal: Yeah. It's, it's an intermediate stage. It's almost, if it's been translated across multiple dimensions into some foreign thing, right?

Then that foreign thing gets manipulated and then brought back to reality and, oh, it's a flower. Yes. You know, that kind of thing. But, but it was like, I pr but I called this 30 years ago. I said, in order for us to get smarter, we're gonna have to write code that we don't understand.

Jonathan: Hmm.

Randal: And, and, and can't single step to fix it.

We have to fix it with other means. And, and we've done that. And I kind of wonder. Then what the next generation of that kind of major breakthrough is gonna be. Is it gonna be just more in terms of making neural nets do better things? I mean, neural nets are great neural nets. Help us then do the, the, the, the LLM auto completion stuff, the you cosign calculations, stuff like that to make that all work.

Vector space, all the fun stuff that I've, I've only just barely learned the terms to, but it's great 'cause I know that's all in there somewhere.

Jonathan: Mm-hmm.

Randal: The I, so I wonder, I wonder what the next revolution is and I haven't had any thoughts on that yet. So that's, is, that's kind of a, i, I, I called this 30 years ago, but I'm not calling right now what I think.

So I'm, I'm still waiting to kind of see what, you know, what, when that ship's gonna sail. If ever when, when you figure it out, when you figure

Jonathan: it out, let us know. Yeah.

Randal: Yeah. I'm enjoying all this. I'm enjoying the vibe coding. I've been like updating my website and, you know, and I found a, you know, I, I re I watch probably 15, 20 AI videos a day, at least of people that are doing vibe coding or doing cool things with lms.

And it, it, it, it's just moving so quickly. That's true. It's like if I wasn't, if I had, if I had a 40 hour a week job, I probably couldn't keep up with all this. So it's probably good that I'm between gigs right now, but I, I would rather have a gig, by the way. But I'm think between gigs right now my banker doesn't think so.

My banker wants me to have a gig. Imagine that, but but, but, but, but, but at least I have time to kind of be heads down on this stuff and have time to play with it and figure out where I'm going next with it. And I, I can't wait to see where we are in six months. Yeah. Where we are in a year. I think it's gonna

Jonathan: be pretty exciting.

Yeah. Very cool. You wanna plug that thing you do on Wednesdays and anything else?

Randal: Sure. So I am a flutter Google developer expert. I'm one of the 11 in the US and about 150 in the country or in the world. I mean and one of the things that involves me doing is doing outreach, which means I'm doing I'm, I'm trying to answer questions on discord and we stack overflow and Reddit and anywhere else I kinda see them.

I keep thinking four square belongs on the list, but Foursquare is nothing to do with any of those things. Yeah. I'm answering questions about locations and how they relate to water. No, and I don't think that's gonna work. And we do this Wednesday show every week. Mm-hmm. Called Humpday Q and a a MA.

And so we get experts like Simon Lightfoot, who's a really smart guy in the flatter area. He actually answers most of the questions. I'm mostly color commentary. Mm-hmm. So that's the fun part. I also answer, I manage the, the question flow on the show. 'cause it's a live call in show. Go do that every Wednesday.

That's pretty cool. I also have a YouTube channel. I'll leave you in the show notes for that. That's about Dart and flutter if you're interested in that sort of stuff. Yeah. I'm also just, I just bought a, like, not a steady cam, what do they call these? Gyro cam. Mm-hmm. Mount for my phone. Mm-hmm. So I'm probably gonna be doing more walking selfies for fun.

And that will be in my personal channel though, not my YouTube, not my professional channel.

Jonathan: Yeah.

Cool. Cool. But yeah. Alright. I'm working on Appreciate it. Thank you for being here, man, for jumping in kind of last minute. Thanks. Alright, if you wanna find my stuff, of course, at Hackaday, we've got the security column goes live on Fridays and we sure appreciate Hackaday being the home of Floss Weekly.

There's also the Untitled Linux show that other podcasts I do that's over at twit twi tv slash uls record there on Saturday afternoon. And you can find it. Go the, the, the edited, the, the for consumer for consumption that goes, goes live usually Sunday or Monday. And that we have a lot of fun with that.

So feel free to follow me over there. We appreciate everybody that watches and listens, both those who get us live and on the download, and we will see you next week on Floss Weekly.

Episode 830: Vibes

23 april 2025 | 79 min

Episode 829 transcript

16 april 2025 | min

FLOSS-829

Jonathan: This week I chat with Herbert Verson and Frank Bick about Buffer Bloat, Lipper, QOS, and remembering Dave Tot you don't wanna miss it. So stay tuned. This is Floss Weekly, episode 829, recorded Tuesday, April the 15th. This machine kills Gons.

It is Time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and today we've got something well, really fascinating that we're gonna dive into. But it's not one of the things that we really wanted to talk about today. We wanted to talk about something else.

And so I'm gonna bring our guests on and we've got Frank and Herbert. And these are a couple of the guys from Libre, QOS, the the, the saving the internet from Buffer Bloat organization basically. And today we're gonna talk about Lib, QOS, but we're also, we're, we're gonna remember Dave Todd, and.

If you don't know Dave did pass about, was it about a month ago? It's been, it's been a little while now, right guys?

Herbert: April 1st was the announcement I got, but

Jonathan: April 1st, and there are, so Dave lived in California. Dave was a huge space nerd, like, like I am. And I think like these guys are too and was actually in Brownsville, Texas and passed there.

And from what we can tell of natural causes, the, the weird thing about dying outside of your state is the paperwork is terrible. So like, we don't have all of the answers yet. We can tell you though, even though the announcement was made on April 1st, it's not an April Fool's joke. I wish it were. But it's not Dave, Dave Todd has passed.

And so we we don't wanna make this a downer of a show, but what we wanna do is, is talk about like the, the legacy, the buffer bloat legacy and all some of the other things that Dave has done. Also talk about Liberty QOS, where it's at now and where it's going, the, the neat things coming for the future.

Some of the exciting things that that project has done. And so that's, that's kind of what we're up to today. So like I said, we've got Herbert Verson and I, I guess technically it's, it's France sec Bik and you go by Frank.

Frank: Yeah, Frank is easier for, for Frank. Any Frank? Yes. Frank is easier. Ango Saxon world and even for India, basically, so Yeah.

Yeah, it works. I can, I can imagine. Did I, did I get it at least reasonably close? Absolutely. Okay. You know in the, like in the English Pro pronunciation, I think it's the best you can do.

Jonathan: I'll take that, I'll take that. I get, I get a lot of unusual to me names and trying to get through them sometimes is, is it's a challenge.

It's fun. Alright, so let's I don't know, where do we wanna start? Let's, let's talk about what, let's talk about what Liberty QOS is. Maybe that's a, a good starting point. And let's see, Frank, you told me in your notes that you are the, the gopher for Liberty QOS Let's start there. What, what does that mean and how did you get there and what things do you go for?

Frank: Right. So I go for everything basically. And I, I, I got this nickname when I was living in Iceland back then in 2013, 2014. And like besides anything else, what I was doing there. I was also at the fishing boat, and I was helping the maintenance engineers to do all the fixes and stuff. And, you know, the, the, basically the, the best thing that I was able to do was to fetch like all the things they needed.

And I was going for everything. And they said that You are really good, good, good in, in that. And basically, it, it, it stuck with me. And then I. Tried like everywhere. I was doing some work. I was trying to do the same. I was trying to help mm-hmm. You know, to do things that nobody liked to do and don't want to do and things like that.

So I guess that's a really helpful role to, to do. And you know, I have really good colleagues. They are you know, rock stars. Dave was a rockstar. Herbert is a rockstar. Robert is a rockstar even though like not many people know them yet, right? Mm-hmm. So I'm, I'm the, that one guy that want to help

Jonathan: them shine.

Yeah. I remember, I remember being like a young teenager working with mainly my grandfather at the time. And I didn't, I didn't know anything. And so that was my job. I was the gopher for a while. And, and the, the idea there was that you hang around long enough, you go for the right tools and you kind of pick up, you learn as you go.

And finally you, you get to the point to where. You have a younger cousin that gets to be the gopher and you get to actually work on this stuff. Right.

Frank: So maybe one day I will get my gopher as well,

Jonathan: graduate from Gopher School. Right. And then, and then Herbert, we know you as the rest guy. How do you, how do you fit in with Lu QOS?

Or, or do you, you are part of it, right?

Herbert: I am I'm the Chief product Officer. Yes. We picked entirely because I wanted to be called C3 po. The you know, the background for this is that on top of teaching rust and writing books about rust, I've been helping out with an internet service provider in Central Missouri for, I don't know, well over a decade now.

Mm-hmm. And we're kind of a weird ISP that we don't actually make any money. We, our mission is basically to reach out to the areas that nobody else can get to and get them the best service we can, we can manage 'cause the Potter, Missouri I'm in has some seriously swampy areas mm-hmm. That if you lay fiber, it just washes away areas that flood every year.

Areas that get hit by tornadoes every year. So it's really, so aerial fiber doesn't work because the tornadoes take it away. Mm-hmm. Buried fiber doesn't work because you're in a swamp and so you end up running a whole lot of wireless links. Mm-hmm. And that's what Ison does. And, you know, we fell afoul of serpentine Missouri tax low and we tried to actually be public benefit, but, so we're public benefit in what we do, but not on, not on the actual structure.

Jonathan: Mm-hmm.

Herbert: And one of the terrible things about things like 10 mile long wireless links is that buffer bloat gets absurd.

Jonathan: Mm-hmm.

Herbert: And we'll talk about what buffer bloat is in a bit, but I originally signed up at the com product called Proce that uses FQ cuddle. That's how I first heard about Dave and

Jonathan: mm-hmm.

Herbert: He actually didn't remember it, but I was lucky enough to run into him at a, at a convention about that year. Then we didn't meet again for a while. And anyway, Robert started up Liber, QOS around the time the pandemic was hitting and really wanted to tackle the buffer bloat. Situation to try and get even the people with not so good connections, good usable internet because suddenly everybody's at home.

Every network is suddenly massively over capacity because everybody's at home. Whereas previously you could reliably plan that you know, eight to five people go to work. In the evenings they watch videos do home things. Well now everybody's a business customer at home. Suddenly you have a lot less leeway.

And he started Libra QOS with Dave. They put together a giant script initially that set up FQ cuddle for everybody. And then Dave talked him into turning that into cake, which is basically the next version of FQ Cuddle. It adds a whole lot of bells, whistles, and works unbelievably well. And that's what got my interest was.

I saw them on GitHub. I said, Hey, I could, I could use this. And because I am, I dunno, probably autistic. I sat down and basically wrote a rest version in a week. Shared it with Robert and Robert showed me some of the pain points he had. Like IPV six wasn't working right. Mm-hmm. And he was having troubles scaling it up to multiple CPUs because of some bottlenecks in the Linux kernel.

And so I worked closely with Robert and Dave understanding what they needed. And Dave put me in touch with Taki whose name, last name I just forgot.

Frank: Taki ho Jorgenson.

Herbert: Thank you. Yeah. I'm actually really glad you said that because that will never make it out unscathed. I'm, I apologize. I but we.

Patched a project called XT, PS X-T-P-C-P-U MAP and managed to make the Linux Traffic Shaping System use multiple cars. And all of a sudden now we've got a system that can, you know, scale up well beyond a gig. You know, we, at the time we were celebrating having just hit 10 gigs. We patched in IPV six, which is quite important.

Mm-hmm. The original setup couldn't do that, which was a real problem because Robert has IPV six. So we got that in and then we were like, Hey, this needs a gooey. So I sat down and wrote one and Libra QoS 1.4 came out with bells whistles, gooey, and it. Really took off tar terrifyingly. So to the point that suddenly I was, we were getting messages from parts of the world that I had to look up where they were.

Mm-hmm. I think last time we checked, we're in what, 58 countries now and most US states and the most recent version we're working on has a whole new gooey because the first one was my was also me learning JavaScript. And this time around it's a, this time around it's a nice gooey we're pretty proud of it.

An optional paid stats backend system. We do, we do a lot, a lot of stuff with that, but we've gotta cover our hosting phase. Mm-hmm. So we're working on a self-hosted version, but at least for now that's, that's in the cloud because nobody wants to self-host something I'm updating three times a day.

Yeah. Which, and I, I don't blame them. But the scale has absolutely astounded me. We've gone from celebrating 10 gigs to talking to a guy who's pushing 80 to 90 gigs a second every night through one of these boxes. You know, it was one of Dave's real happy moments was, you know, wait, this code I wrote can do 90 gigs.

And it's like I mean we, we were doing a, basically a happy dance over Zoom on that one. Yeah. And so it's been an incredible ride. And, you know, Frank doesn't do himself justice. If Frank wasn't reminding me in the morning to drink my coffee and reply to support tickets, support tickets would never be replied to.

Frank: Yeah. So, so ba basically, you know you know, Jonathan and like everybody listening just this is the demonstration how how a rock star actually Herbert is and, and, and yeah. It's my pleasure, my, my pleasure to be part of it.

Jonathan: Yeah. You know, it's, i, I've gotten to be both of those roles throughout the years in various projects.

Right now I'm doing sort of an operations, or all, technically they're calling me CEO, but very much just taking care of operations where, where I'm at. And it is, it's, it's interesting. You're, you're, you call 'em a rockstar. You're your, your developers really, they really do need somebody to help handle that paperwork stuff.

It is extremely useful to not have to Yeah. Not have to deal with it. Yeah. I,

Frank: yes, I, I I, I like to take over anything that yeah, that's, you know, so your, your, your, your hourly rate is higher, you know, for, for such nonsense. Mm-hmm. So it's it's really it is really something like that. Yeah. And yeah, you know,

Herbert: that's not good.

Both Dave and I would forget to put socks on in the morning if someone didn't remind us.

Jonathan: So Frank, how did you come onto the, the Liberty Qs project? How did you find it and, and decide it was something you wanted to be involved with?

Frank: Right. So I, I, I knew Dave through the events, IP events in the US mm-hmm. And I was, I was working with the realtor company open, WRT based company from Republic tourists.

Oh, right, right. And yes. And, and, and they, they free, like all laughed, the, the, the product. So we, we started to talk, you know, I was one of the, the people that got to the US in 2021 mm-hmm. To osa. And it wa I, I was probably the only foreigner that managed to sneak in, in the US and we, we, we, we, we, we talked a lot with Dave and we find a lot of common ground.

Mm-hmm. And he, he offered me a help. With tourists and he tested the router. He wrote a, a really great book about how good tourist omnia is for, for the people you know from, especially from the security standpoint. I. Because the tourists or the parent company, CS Nick, they added lot of security features like firewall and things like that, that, that are not typical part of open WRT.

Jonathan: Mm-hmm.

Frank: And then in 2022, they've told me about Libra QoS and I understood that that's the missing piece that Dave was looking for, because most of the router companies, they will never accept or most probably never accept FCU coon cake for many reasons. You know, I want, I want to stay nice, so I will, don't name the reasons why big companies that like to brag.

That they have the biggest buffers in the industry. They, you know, they, they don't want to do that. They want to sell the new, the newest a OA eight oh 8 0 2 11 standard like every few years. They don't want you to have a router at home or, or in the office or in the stadium that will last 10, 10 years.

Jonathan: I remember, this is a bit of a rabbit trail, but I was actually in Ireland at an open WRT mini conference when the tourist Omnia, the first time I heard of it at least, was announced. And we all thought that it was just the coolest thing. Then we were excited about it. I don't know, I don't think I ever got my hands on one of them.

It, the, the price point wasn't quite what I was after for the, the small companies in Oklahoma that I was dealing with. But I remember it was just, it was the, it was the coolest thing that somebody was doing this making. Dedicated hardware, like powerful hardware for open WRT? Yes. It was exciting at the time.

Frank: Yes. So the, so the goal was to, to have it like, at least for 10 years, you know, the typical router won't stand, mm-hmm. Than much longer. Than much longer. So,

Jonathan: yeah. At, at the time I was taking TP link routers, little, little white UFOs from TP Link. We were flashing open WRT on 'em and selling them to companies for, I think.

$85 US installed, which was ridiculously cheap. Looking back on it now, it's like I was not charging enough for that, that I was doing. Yeah.

Frank: So, so, so ba basically, OO open source is really changing the world. And open WRT is one of the, the, the biggest examples because it's, it was just a three people that started open WRT.

Mm-hmm. One of them wa was from Hungary. One of them was from Canada. Canada. And the other was US guy, if I remember correctly. I remember only the name of the Hungarian guy because he's my friend. It's, his name is em. And he's still part of open WRT and he's doing a lot of cool stuff.

He's also a rockstar, but most of, most of the things that he's doing is under NDA and you can, you can say it out out loud, but So these, these are the people that are changing the world, right? Starlink was built on open WRT. Yep.

Jonathan: I've got, I've got the gut somewhere over there on my junk shelf.

I've got the guts of the first, the, the version one starlink router where we were taking it apart and trying to figure out if we could put a, a newer open WRT image on it. I worked a little bit, I worked a little bit with Dave on that, actually. Not, not a whole lot. I ran outta time to really do much with it, but I know we were, we were both poking at it at about the same time.

Yeah, I think, I think the first time I met Dave was when Doc Searles brought him on to a floss weekly to talk about buffer bloat. And that was, that was really interesting. He had his I think it was a funnel and a, you know, a picture, a picture of water and a funnel. Yes, yes. Pour he, he was pouring down and pour.

Yeah. We could see that. Oh yeah, you, you get a lot of water that, that stands up there. It doesn't make it through the funnel fast enough. And he, he shared that clip. We shared that clip all over the place. That was, that was a lot of fun. That was kind of when I, when I first met Dave, I realized that Yeah, he is, he is one of, one of my people, one of one of our people.

He is, he was a cool guy. We, we very much enjoyed having him on the show. Had he, had he back several times throughout the years. Talk about various things. So let's see, with with Liberty QOS it is it's really a, it's like a toolbox for ISPs. That's, that's really what it's made for, right?

Herbert: Yeah, we started out very ISP focused, 'cause Robert and I both have wireless networks.

Mm-hmm. We've been running into more and more people using it for different things. Apartment complexes. Mm-hmm. University campuses cruise ships cruise ships was the one that surprised me, but it does a great job of smoothing out satellite internet and giving the people on board a much better experience.

Yeah. It's, you know, honestly I'm surprised by all the, all the places that it keeps popping up. I've actually had people tell me they've heard of the project now, which is which, which is amazingly cool. You know, I've I'd got at Rust Con somebody actually remembered me as Herbert, the Libert Q os guy, not Herbert, the guy who wrote the book on Rust.

So I was like, Hey, that's cool.

And you know, from an ISP point of view, we were focused, we were tackling one of Dave's major frustrations, which is we can't get every home user out there to run a good router.

Jonathan: Right.

Herbert: And Dave's original plan was to make a nice router and basically give it to everybody on the planet so that we can stop buffer bloat that way.

So the discovery that you could put something in line and have everything in the ISP go through it and do a good job of fixing everybody's buffer bloat all at once.

Jonathan: Yeah.

Herbert: Was kind of, you know, Dave's eyes lit up. 'cause he figured out, Hey, I can do thousands of thousands of places at a time instead of having to go to everybody's house.

Because Dave was a man on O Mission. He Oh yeah. Dave was, Dave was absolutely determined the buffer bloat was going to going to be destroyed. And

Jonathan: let's, let's dig into that. Let's, let's dig into that part of this. First off, what, give us the, the quick, the, the quick definition. What is buffer bloat?

Herbert: Okay.

So when packets come into routers or whatever's forwarding them

Jonathan: mm-hmm.

Herbert: If there's any sort of congestion, so you. You can't immediately just pass it right through. Then packets get queued up in a buffer. And so traditionally the b when, when you've got more traffic than you have capacity, the buffer will eventually fill up and packets bounce off the back of it, just get randomly dropped.

Jonathan: Mm-hmm.

Herbert: And problem with this is that people solve that problem by, by getting ever bigger buffers bigger and bigger and bigger. And now the transit time of your packet all the way through the buffer gets ridiculous. And it gets unfair as well because Johnny downstairs downloading the Call of Duty update is going to be 99% of that buffer.

Jonathan: Mm-hmm.

Herbert: And Mary upstairs trying to make a video conference call for work is gonna be 1% of that buffer, but all her packets have to wait because Call of Duty's in the way. And so she gets that dreaded, you know, your video quality is un is unstable. Things stutter. It looks terrible. And so buffer bloat is, your buffer is much bigger than it needs to be.

And if you look at the round trip time of packets going through the system, there's actually an optimal amount of time for how long something should live in the buffer. And so cuddle, the first algorithm that they've worked on introduced the concept of sojourn time. So measuring the amount of time.

That this packet has been buffered. Putting it through an algorithm relative to approximately one round trip and preemptively dropping packets if they've been in the buffer too long. And that tends to, that as a result slows the call of duty download down a little bit, but makes room for everything else.

So FQ coddle was the follow up to that. In addition to, so on time it added fair queuing. So everybody's data gets basically put in buckets and you make sure that each bucket gets a chance to go first. So this ends up favoring the sparse, the small buckets. So your VoIP call no longer stutters because your wipe call doesn't have a lot in it.

The packets each get a fair chance to get out. The big bulky download that is designed to max out whatever it's got. But nobody's gonna notice that their download was 30 seconds longer when it, you know, when it's a huge multi-gig file.

Jonathan: Yep.

Herbert: That's okay. So slow that one down a little bit. But you don't, you're not trying to do, you know, DPI packet inspection.

You're not trying to say, Hey, this is YouTube, this is Netflix. You're just treating everything. The better behaved the protocol, the better FQ cuddle treats it. And so cake was FQ cuddle on steroids where Dave and a bunch of other people got together and thought of everything they could possibly come up with to make FQ cuddle better.

So the algorithm that handles so time in the queue improved substantially. Tuck's thesis Buff Bullet and beyond is great if you want to try and understand how that works. It's math heavy to the point that I have to read it twice every now and again. The fair queuing got extended out to have both have a better algorithm and also start looking at the SCP tags to tell you when traffic is willing to tell you what it is.

Allowing it to be divided up into tens with a little bit of preferential treatment for the real time and a little bit of slowing down for bulk traffic. Honestly, the tins, the tins are the first thing everybody goes to because it looks, hey, it's classifying my traffic. They're honestly the smallest, they're honestly the smallest component to smoothness.

They help. But the algorithm, the coddle algorithm the cobalt algorithm that replaced it and the fair queuing mm-hmm. Is the real magic. You can go to best. You can dump all the traffic into best effort and still see things a whole lot better. And so buffer bloat is problematic because it's not just your writer.

Your router is talking to the ISPs router mm-hmm. Which is talking to the internet exchange router, talking to another internet exchange writer and so on all the way to the hosting center. And so there's a good chance that somebody somewhere has buffer below whether and kind of the magic of K and FQ cuddle is that because they measure timings on packets as they go through.

Jonathan: Mm-hmm.

Herbert: It's got enough information to start making the streams self-pacing, even though the rest of the network kind of sucks. I mean, you can't make, you can't do something about a router in Chicago that's decided to drop half of its traffic that's outta your control.

Jonathan: Right. But if

Herbert: it's mostly working, then you know, fq coddle will make your experience a whole lot better just by working with what, what it sees going through your router.

Jonathan: Mm-hmm.

Herbert: And assuming that everybody else sucks. Now, if you can get, you know, that guy in Chicago to also install fq Coddle or Cake and maybe the hosting provider to provide. Some buffer improvements. Everything gets better for everybody, but you know, one, one bottle at a time. So buffer bloat. If I had, when I, when I give the elevator summary, you know, the summary to some guy in the, in the elevator who wants to know what I do, it's I make sure that you can still make a phone call.

When your daughter has decided that she wants to download an entire season of papa peg.

Jonathan: Yes. Very important. When, when did this become, when did this first become a problem? And I'm, I'm thinking about, I'm reminded of TCP slow start. It sounds like it's sort of the similar problem that got fixed back in the eighties. When, when did, when did it become that TCP slow start was not enough to, to solve this problem?

Herbert: Sorry, I'm trying to think of dates. TCP slow start was something that Dave liked to grumble about a lot. Yes.

Jonathan: Yes. That's probably why it came to mind. So,

Frank: so, but, so, but if, if I may, yes. Basically, they, Dave realized the problem when he went down to Nicaragua, which was he his favorite country.

And he, he was thinking about retiring after, like, everything, what he, what he was doing in the, in the, in the ISP world in the nineties before the dotcom bubble or telecom bubble rather. And he, he, there was, there was open WRT after 2000 and four like already on the market, right? So people were using flashing all, all the routers we did.

And so he, he, he, he saw a problem down there in Nicaragua. So he decided that he will help people. So he was flashing all the routers around him in hotels, in in the cafes in, in homes of people. But the problem was there. So there was a first like a video conferencing at the time. Video calls on Skype and he saw the issue like so like prior to 2010, so maybe 2008, he started to talk with it with Jim Gatis online.

And Jim was the he was the guy, if I remember correctly, that actually coined the term buffer boat. And so they started to talk and Dave decided to go after it and to, to help Jim to fix it.

Jonathan: Yeah. Interesting. So that was, that was kind of, that's when, that's when Dave got on this. I always think of Dave as the buffer boat guy.

But Jim, Jim was, Jim was working on it even before then, I suppose. Yes.

Frank: So he, he started early, but he, you know, Dave picked, picked it up later on. Mm-hmm. You know, Jim he, he did, did a lot of work and he was kind of burned out. So there was a, a space for more people to plug in and to actually continue, continue his his mission.

Mm-hmm. So, but they if I, if I get the date right, like buffer ball do net was start, started at 2010. Mm-hmm. And, and so that was the, that was the basically the, the starting, like the main starting point for Dave, and this Dave started to be phase of it. He was chasing like everybody right.

To, to change stuff for, for the good of people and, and you know, the or Right, right, right from the beginning of like, after Cordell that was FQ code and the, the uptake was really, really, really fast. Right. Linux opened WRT previous SD quite a lot of stuff. Right. And Microti as a, as a first probably the, the like mainstream closed source company.

Right. So the expectations where that, yeah, it'll be done in no time.

Speaker 5: Mm-hmm.

Jonathan: Yeah, it's interesting. I, I was just linked to RFC nine 70, which is on packet switches with infinite storage from 1985. Right? So people, people were, people were thinking about this even back then that this might not be a good thing.

Yeah, I, I'm not, I'm not immediately seeing when Buffer bloat was first uttered, when, when Jim put that word together. But it's such, it's such an interesting history and it's such an interesting problem too. The, the thing you do to try to fix your your router, fix your network switches, is introduces its own problem.

Interesting stuff. So what's what's, what's next in the buffer bloat story? Where, where are we? Is it a solved problem? If not, what's coming next?

Herbert: Let's see. They've got a bunch of enhancements to cake into production not very long ago at all. There in net next on Lennox added a good 10% to throughput and fixed a couple of things he called bugs that I honestly honestly, the math behind it is stretching my abilities.

But he I understood, I understood the performance improvements. I actually worked with him a little bit on some of that and pushed that up to Toki and Lennox is putting in, ePF based Q discs system in net next right now. Mm-hmm. I know there is some talk about getting cake pushed over to that to make it a little easier to customize because custom kernel modules have become a little problematic with everybody's having signed boot loaders now.

Yeah. So e that's the immediate focus. I am trying to fulfill what turned out to be Dave's last wish and fix a memory issue in cake. It, it's gonna be a while because it's it low level C and I'm more low level rust guy, but I, I mean, I've been using c since the nineties. It's just I'm rust.

I'm, you know, I don't wanna say I'm rusty, but you know what I mean. So that's, that's the immediate part of trying to carry on his legacy. But we're also talking to. Every yeah, everyone will listen to try and find next gen, next generation of researchers, see if we can feed them data. Mm-hmm. So they can look at the problem, maybe come up with something cool.

Like we're working with one PhD researcher right now, giving him vast amounts of TCP RET and round trip time estimates from various parts of the world. He's taking a geographic focus, see if there's anything you can do about long distance lengths that automatically give you some buffering.

Mm-hmm. Because it just takes so long to get there in packet terms. But yeah, it's also, Dave's really big shoes, you know, it's mm-hmm. I don't know if my feet are big enough to fill them, but we're, we're trying.

Frank: Yeah. And, and, and there is also like at least three groups. Of people that are working on, like something that it's called like Cake two Cake mq, MQ cake.

So like all the usual su suspects that that you can think about like Dave was, was scoping something that there was supposed to be like, at least the working title was cake mq. Then there is Stocky and his guys working on, on improvements and I heard from Dave that Jonathan Morton was also working on something to improve cake.

So there are things like that. And also we see like all the time new stuff coming toker, like I, I think it was last week, right, Herbert, that there was that BPF new stuff from, from Amari.

Herbert: Yeah, I'm, I'm actually excited to play with that. I. A BPF is what got me started on Libra QOS.

So not too bad for a fresh project.

Jonathan: Yeah. BBPF is something I find fascinating, I've never gotten a chance to play with yet. Maybe, maybe one of these days I'll have a project where I get to go in there and fiddle with it. You mentioned something Herbert, that, that really got me thinking, this idea of the, the buffering that's sort of built into long distance links and you mentioned geography and I I immediately in my mind, 'cause we talked about Dave being a space nut like I am, is the, the astronomy or, or space travel really implications of that.

And I don't know if this is something Dave was working on. I'm sure somebody is, but like when you start talking about, well, the ISS for one thing, but that's low earth orbit. What about when you start talking about putting people on the moon or on Mars? Mars what does. What does the buffer bloat situation look like?

What does the internet look like when you start talking about Mars and links between planets where just your, your round trip time of the speed of light is measured in seconds to minutes?

Herbert: Well, you're not gonna be doing interactive video unfortunately.

Frank: Right? For, for, for navo.

Herbert: You know, I, I actually read a paper yesterday about how I think it's toia proposing to put a 5G network on the moon.

Jonathan: Mm-hmm.

Herbert: I saw that. Which is clever because they're limit, they're going to be having, by having local nodes, you'll get good latency for at least talking to other people on the moon. I know the deep space network is something they've talked about a lot. It's not something I've ever worked on, but extensions to T-C-P-I-P where were made, especially for it.

So that. Packet latency to Jupiter would work at all. The, it, you know, it's a, it's a tough one. I, because it's the same problem that you have on earth, but on a, on a bigger scale, you, you often need a big buffer. I mean, you know, we laugh about, we've got the biggest buffers in the industry, but sometimes you need it because when there's packet loss going on, fragments being reassembled and so on, those fragments all have to get into a buffer somewhere so that you can reassemble 'em.

At the same time. There's, there's such a thing as too much buffer, hence buffer bloat.

Jonathan: Mm-hmm.

Herbert: So, I remember Dave telling me when we were chatting one night that in theory FQ cuddle would work in space if you changed the RTT parameter to assume. That really hideous latency is okay because you can't avoid it.

Right. I don't know if he I don't know quite how far he got. I know he was talking to some people putting satellites up. I know one of the first, actually starlink was one of the last things he worked on.

Jonathan: Mm-hmm.

Herbert: But starling's, you know, low hanging fruit because it's relatively low, relatively low orbit, it's low hanging the geo.

But we do know that FQ coddle helps with geostationary orbit.

Jonathan: Mm-hmm.

Herbert: So there's no reason it shouldn't help with the moon. It's just gonna be a matter of unfortunately, you know, experimentation requires that we get to the moon, put up a network and see what happens.

Jonathan: Mm-hmm.

Herbert: I will volunteer for that if anybody wants, wants me, by the way.

Yeah,

Jonathan: there you go. SCPS, the Space Communications Pro Protocol Specifications, that is at least one of the projects that people are working on to do this, to, to make extensions on things like FTP and TCP sec even. And, and what do you do when you're trying to talk between planets? I, I find this, I find this utterly fascinating.

I, it, this is one of these things that it really does make me, make me kind of, it makes me sad that Dave's not gonna be around to work on this. 'cause I know this is the kind of thing that he loved too. And we just eat it up.

Herbert: And can you imagine a three-way handshake with minute, one minute delays for each.

Just to establish, just to establish the TCP connection, that's mm-hmm. I mean, that's pretty nightmare-ish.

Frank: Yeah. So maybe, especially if all you want

Herbert: to do is send one packet saying hello,

Frank: right. Yeah. So may maybe UDP for, for such a situation.

Jonathan: Yeah. My, my thought on this was always that you would, you would have to have a, I mean, it would be storing forward of some sort, but you would almost have to have two, two discreet internets that would sink from time to time.

Just because doing anything live between here and there is just a total no-go. The, the, the round trip time to and from Mars is like 24 minutes. Or at least my quick Google suggests 24 minutes. And, you know, nothing, none of the internet that we, that we use works on that kind of a round trip time.

It just all completely breaks down. So, yeah, I find this, I find this super fascinating and I, I too, I look forward to, actually being able to run the experiment and see what happens. See if we can make it work.

Herbert: Do you remember a phyto net from the BBS days? It always, it always makes me think of that because you sent the process of sending email was you wrote it, you send it by modem to your local hop.

Jonathan: Mm-hmm.

Herbert: And it would, at some point during the night, send it off to the next exchange and mm-hmm. Your email might take three days to make its way across the world, but it would get there eventually.

Jonathan: Yep.

Herbert: And so I could see something like that working. Mm-hmm. And the old network news protocol too.

Jonathan: Yeah. Yep.

Yeah. Interesting stuff. But it's al it's not quite that extreme, but it's almost gonna go back to the days of days of sale when you didn't get any updates until the ship came into Harbor that had the latest news dispatches. Right. It's, it's. It's a fascinating thought that as we, you know, kind of expand the human race out it's gonna be a a in some, to some extent, a return to that sort of a, a paradigm.

Herbert: You know, at the same time, I still can't believe I wake up and see videos of a probe crashing into, you know, a, a, a moon off of satin. Mm-hmm. And and that's old tech, you know, had launched years ago.

Jonathan: Right. Yeah. Yeah. Fascinating stuff. What is speaking of all the state space stuff, what is the the, the current status of, of starlink with the open WRT version and Cake or kode built, built into it?

Frank: R Right, right. So, so in, in the beginning of, of March, they finally publicly announced that last year, I'm sorry beginning of March 2024, they publicly announced that they, they implemented FQ call. So they was ecstatic. That's great. And it, it really cut down the latency and, and the goal was 20 milliseconds.

And in, in certain instances, they are able to achieve that, which is incredible.

Speaker 5: Mm-hmm.

Frank: And they, they also have like air airtime, airtime, fairness, and some other, other stuff I was told mm-hmm. That Dave was fighting for. There, there there is one particular amazing guy that got into Starling and he's one of Dave.

Dave, Dave apostles Nathan os. So like, if we can credit someone, someone from, from that is on the inside right now, it's certainly him. Like he was really active for years on, on starlink mailing list and, and various buffer ball mailing lists. And like, I don't know for sure, but he, he might be the, probably the last drop to really make it happen.

Yeah. After like all those days runs and, and, and emails and, and songs and everything.

Jonathan: Yeah. I'm trying to, I'm trying to remember, I'm trying to find it actually. A apparently I, so I wrote a Hackaday article about Buffer bloat several years ago. And I think it was Dave that sent me an email and it was like who was it?

One of the, one of the guys from Doom, oh, I can't remember his name at the moment. Anyway John Carmack, I think it was, yeah, I think it was John Carmack forwarded my Hackaday article onto Elon Musk and he's just like, okay, that's cool. That's a good, that's a neat, deep bit of news to wake up to in the morning.

I'm gonna see if I can find that email. I've got it. I've got it here somewhere. I've just got, I've got get the right search term going. But you know, that's one of those things that. For, for one thing, the this space that I found myself in between doing hack a day and getting to talk with people like you guys and people like Dave, it's, it's really cool the, the connections I've been able to make, but also it's kind of a good feeling that like the fact that starlink is getting better, I would like to think I've had at least a tiny little part in making that happen.

Frank: Yes. Like pretty much everybody that was like tweeting to Eon talking to Starling, sending emails singing songs using Starling, you know, that was the, that was the push from, from all places. Yes. And it was really necessary for something like that to happen, so. Yeah. Yeah.

Herbert: You know, John, John Carmack was actually backing Dave on Patreon for a long time.

Mm-hmm. Big supporter of the Buffer BLO thing. You know, that that was one of the things I couldn't believe was after we posted the announcement I opened my inbox and I see email from Vince Surf. Jim Gdes. John Carmack. And it's like any other occasion I'd been jumping up and down Fanboying.

Jonathan: I've moved that.

A little bit of that. Yes. Yeah, I, I found the email Dave emailed me back in August of 2022. He wanted to know if I could see stats on Hackaday, and then he says, you'll be pleased to know John Carmack. Read your article and forwarded it to Musk. That was, that was one of those, that was one of those happy little dances for me.

I got, I got a lot of pleasure out of that. Well, well deserved. Yeah, it was fun. It was a lot of fun. Alright so Starlink is doing well. One of the other things I remember about Dave is he, would you call it flk? The, the music he was, he was definitely a, a, a music fan, but it was, it was very much his own sort of take on doing things like it.

Gpls me is one of the songs that I remember. He had a he had a sticker on his guitar case that said this, this machine

Frank: kill,

Kills Gans.

Jonathan: Yes, that was it. That was it. He, he said he was inspired by the, this machine kills fascists, but didn't want to quite take that I forget, I forget exactly what he said, but he went in Gons instead.

That was, that was sort of a quirky part of his personality, wasn't it? Do you know where that came from? Would you call that Fi? Is that, would we call that Fi Is it like a, a tech version of Fi?

Frank: Oh, yeah. Yeah. He, he was writing in the style, so it's like folk music but in sci-fi. Mm-hmm. So like everything tech tech wise people like Dave were like writing lyrics down and it was like always always a great success.

Yeah. You could always tell he

Jonathan: he really enjoyed it. He had fun with it. Yeah.

Frank: At the events, at the podcasts or as a, as an outro or as an intro at at, at the webinar even. Yeah.

Jonathan: Yeah. That, that's good. Mashed potato. One of our listeners has a suggestion that, that be the, that'd be the show title for today.

This machine kills Bogans. I like that. Absolutely. I think that would be, let's do that appropriate way to, to honor, to honor Dave.

Herbert: I think we've got pictures of Dave busking at at least 20 major conventions. Mm-hmm. Yeah.

Frank: And a also like close to the the actual SpaceX. Yep. Yep. That makes

Jonathan: sense. It's, that's, that's something that he would definitely do.

Alright, so what, what else Are there some, are there some memories that you guys have of Dave that you, we want to touch on that we haven't,

anything, anything comes to mind? Yeah, go, go ahead Bert.

Herbert: Oh, okay. You know, I just, a hu really a human memory. Mm-hmm. We were having a labor QOS staff meeting and my three-year-old ran upstairs, jumped onto my lap and introduced herself and Dave just lit up. Interacted with, with her, made her feel like the center of the room.

And we, we stopped talking for long enough that she ran over to her little toy piano and she's a really smart kid. She hit demo mode and Mimed playing for her leaves, and Dave is just egging her on, encouraging her. She goes down, tells me what a great kid I have. Meeting continues, but you know, I've, I've talked to a lot of people for whom the 3-year-old invading is cute for 10 seconds, but get rid of her.

Jonathan: Mm-hmm.

Herbert: Dave was not like that. Dave, Dave really just was a warm, kind person who would make time for you. We, you know, we've had, we've had emails from people saying, Hey, you know, I, I had a problem and Dave appeared outta the blue, fixed everything and swept away again. And that's just kind of who he was.

Jonathan: Yep, yep. Frank, did you have do you have something similar about Yeah.

Frank: Probably the time that I cherished the most that I was lucky to spend with Dave. Was was in Mexico after wifi now global Summit in Cancun in 2022. Like the rest of the tourist tourist guys, they already left, but I was still there.

Staying with Dave for a couple of days and we spend it you know playing his guitar, singing, sharing all of songs, stories, and also working on, on, on tourist, tourist routes because they was showing me all of stuff. He was measuring things with Flint and other tools. Mm-hmm. And he was showing me things that should be improved.

And we were just, you know, like enjoying time, walking around the, around the beaches. Talking about things, talk, talking about life, and you, you know, that these are, these were the things that like, he was able to pair, like, like he said, he was able to connect to, to, to to small three years old. Three years old kid.

Mm-hmm. And also like, you know, to the biggest names in the industry and also to like some strange guy from Europe that is working on, on routers and, and, you know, want to, want to learn stuff. So that was, that was the, like a bigger, bigger than life type of character. Like they, they, they was. And it, it's amazing to go to all, all the, all the media and to, to read comments from people.

And, and it's it's overwhelming. And, you know, I, I do believe that he's like up there you know, up there somewhere. Mm-hmm. And he, he, he sees stuff and like he, he likes to like, he like to set things like per aspera ra, right? Mm-hmm. So that's something that we can say to

Jonathan: him, for sure. I'm gonna start introducing myself as some strange guy from Oklahoma that works on tech and likes to learn stuff.

I think that's great, right? It's,

Frank: Ca can, can I, can I read a poem? Sure. Yeah. So Dave, like, you know, back then when, when, when the, when the work on, on Buffer board was going strong mm-hmm. Jim Gettis, John, John Morton, and many, many other people but they were working with Dave and there was a, like poem, poem from, from John God Godfrey Socks.

And the, the new version that John Mor, Jonathan Morton and Jim GTIs put together was Na was named The Blind Man. And and elephant and Dave read the poem at a couple of podcasts. And yeah, like I, you know, I would like to read it for him once again. Sure.

I am sorry. It's alright. It was six men of Stan to learn to learning. Much inclined who? So, so to fix the internet with market share in mind that evil of congestion might soon be left behind. Definitely the piman say why the link V view out for congestion occurs only when the bandwidth exceeds bout if only the earlier reverse were to, we would all live like gold, but count the cost.

If bandwidth had no end monument, say, but count the cost. If bandwidth had no end monument, say perpetual upgrade, cycle through madness. Right? Like that way. From the, this surf land, they, they're tried and true solution. Now, the parts, some packets are more equal, equal than they're the true lesser counterparts.

But choosing which is which presence new problems in all arts. Every pa every packet is sacred Christ. The data center clan demanding bigger buffers for, for their ever faster land. To them, a millisecond is eternity to plan. The end-to-end principles guides a certain band of man detecting. When congestion strikes inferring bandwidth, then aas or metals outcompete their algorithmic maven.

The is is o relation, is prefer explicit signaling, ensuring each and every flow is equally a king. If only all the other man would listen to them think. And so this man of industry disputed wild and wrong. Each in is in, is in each, in his own opinion, exceeding stiff and strong to each was partly in the right, yet mostly in the wrong.

So often in technology war, the disputants I been rail on in utter ignorance of which each other mean. And prayed about an internet. Not one of them has seen,

Jonathan: you know, I, I was thinking about it. My, my experience with Dave, I'm pretty sure I also ended up run it, rubbing shoulders with him. When the FCC was about to tell people they had to lock down routers that that actually happened right about the time the Tour Omnia came out. If I remember correctly. It was right in that same time period.

And I did, I didn't, you know, I didn't really get to know him then, but I'm pretty sure we did end up, you know, we were at least on some of the same mailing lists and cared about some of the same things. So yeah. And that I, I just, I went to my I, I went to my email address that I've had for the longest time.

I've never done this before. This is the first time I've done it. And I searched for Dave's name and now I'm, you know, I'm going back to like the oldest email that I've got that had him in it. And I see some from like 2011 when I was on the open. That's so cool. Open WRT mailing list. And he had something he was trying to add.

So, to it, you know, it just for, for so long, like we've just, we've been kind of in, in involved in the same things. And that's just, that's just neat to me. He's gonna be missed for sure. Right,

Frank: right. And and you, you, you have mentioned FCC, right? So Dave was picking up the fight. Mm-hmm. Like every time when, when it was really needed.

And, and Winder and others, like, we we're always ready to help. Right. Because like this, this kind of stuff like open source, it's really power to the people. And Dave, he was believe a strong believer in this principle in pretty much everything. And like the other, some of the other QOE solutions on the market, do they do DPI, right?

And we, we will never do that. Mm-hmm. And, and, and, and so like, you know, these are the things like guiding principles that Dave had. And like, I, I like to say that he never sold his soul, right? I, I don't have like anything against working for a big company. Right. And it's necessary, right? But there must be some people.

In the industry that are guiding you know, they are the guiding lights, basically lightning houses somewhere. And they are doing these like not, not so, not so well paid jobs, but they are, they are, they are actually changing the industry. And, and we all know that the, the meme of that one guy in Nebraska that's basically holding the, the weight of, of the whole stack, basically Dave was something like that.

For, for way way so many companies. Way too many companies.

Jonathan: Yeah. So that's funny. I use that meme all the time because we the XKCD comic, because we, we interview those people. That's what we're all about here on Floss Weekly. Dave was definitely one of those people. I, I kind of suspect that that was actually written about the NTP guy.

I. Because so many things work on NTP and for the longest time it was literally one guy. I think he may have been in Nebraska, I don't remember for sure. But yeah, it's, it is definitely true about Dave and, and you guys and, and a lot of other people that we talked to.

Herbert: Yeah. You know, it's funny, I went on the Den Labs, that's the company I teach Rust through Slack and me and mentioned that, you know, my friend David passed away.

Nobody had the faintest idea, who Dave taught was. I mentioned FQ Cuddle, they're all a bunch of Kubernetes nerds. And everybody's like, you knew that guy? You knew. And it's like it's like, wait, you know, I, I, I've told you that before, but you didn't pick the best time to be interested. But it's

Jonathan: oh,

Herbert: you know, it's fun.

You know, I've talked to a bunch of people, it's like. Not that many people knew Dave, but everybody used his stuff.

Jonathan: Yep, yep. He was, he was involved with other stuff too. I'm looking through my notes. He was, he worked for one laptop per child. He did obviously a lot of open WRT stuff. He worked on the, the C-R-W-R-T project make wifi fast.

As we said, he did some, some stuff with the FCC, trying to talk them out of their insane ideas all kinds of stuff. He was a, he was working for open access in academia. Lots, lots and lots of things that, you know, I think, I think most of us, sort of hacker types, we, we get that they're good ideas.

He was, he, he was, he was one of us. I don't know how, I don't know how else to put it. He's, he was our people. Right. And we'll be greatly missed. I. Alright. Is there anything that we didn't talk about either with, with memories of Dave or the future of Liberty, QOS or where it's at now that we, we should have, that I should have asked you all about?

And I know that's a hard problem 'cause you gotta think about all the things we did talk about and your list of things you wanted to talk about.

Frank: May, maybe, maybe a's AI stuff Hebert would, would you, would you think? Sure.

Herbert: You know, it's Dave was funny with AI when Che GPT-4 came out. He asked, asked it, the question we all ask, who is Dave Tu?

And it gave a pretty random answer. So we went over and tried Google Gemini and it said, Dave Tot is a fictional character from the Hit Jacket Guide to the Galaxy.

Jonathan: That's right. I remember seeing that.

Frank: Yeah. And he, he still has he still has that I think it's either like hero picture on the LinkedIn or Twitter x.

Yes. I remember when he tweeted about that, that that was, that was Chet

Jonathan: GT's understanding of him.

Frank: And, and, and basically today the former tourist CCEO asked about Dave Chet, GPT. And the answer was a really good, and he, he sent it over to me and I was like, oh, thank you. That's going well.

Jonathan: Yeah. Yeah. That's funny. Did, did Dave use any of the AI tools? I know some of us, so for example, myself, I am, I wouldn't say I'm morally against it, but like in practice, I'm pretty strictly against using AI tools for any of my stuff. I, I would like all of my work to actually be my work and I don't necessarily put that in anybody else.

That's just, that's where I'm at right now. Did, did Dave make use of any of 'em or was he kind of a, a purist as well? Do you guys know?

Herbert: A little bit. Now I. I use them because I love trying new stuff. Sure. I don't really let AI write anything more complicated than HTL form, but and that speak, because I could do it.

I know how to debug it, but it can do it faster.

Jonathan: Right.

Herbert: So Dave, I know Dave fired up Cursor, gave it a Linux kernel module, asked it to make some changes, and we spent the best part of a day laughing at what it was trying to do. That sounds about right. We struggled. We spent an afternoon once struggling to try and get emax to talk to AI code completion service that somebody, I think it may have been copilot, said that he, he could have for free.

So he was like, I wanna try this out. Mm-hmm. And his conclusion was not positive. We spent more time setting it up than we did throwing it away. It's kind, it's kind of funny. It. Autocompletes, HTML, really well, JavaScript and Python pretty well.

Jonathan: Mm-hmm.

Herbert: And it CSC the way I first did when I came from Pascal back in the nineties as a random stream of characters.

And that's what it autocompletes, you know, you just get this random barf of letters that might do something.

Jonathan: No, no, no. Ai. That's Pearl. Come on.

Herbert: All right. Once Pearl, I'm glad I don't have that job anymore.

Jonathan: Yeah. But fun. Alright, well that's actually a good segue. Let me ask each of you, as we wrap the show, what's your, what's your favorite text editor in scripting language with Frank?

Frank: I don't, I don't cult at all, which is really, really weird to thing to say on, on podcasts like this.

No, that's fine. And I, I, like, I, I, I. You know, the, the, the closest to did I ever get to something like that was, was that when I was trying to get my CCNA and there were some like, you know, command, non command line stuff. So I, I never actually did any programming yet. But I, I would like to try to understand stuff like more on, on a, on a deeper, deeper level.

However, like when I'm thinking about it, you know, some of those, the things that Robert Hubbard and Dave were doing, you know, I was not able, and I, I'm still not able to understand on that deep level. Mm-hmm. But I, I, I, I, I, I I consider it as some kind of advantage even in some parts of, of my, my job.

Mm-hmm. Because sometimes it's, it's a rabbit hole and it's easy to get stuck. Yeah. And, and to discuss. Things and you know, I will, it flies over me so I can focus on what, what it, what it delivers to the customer. Mm-hmm. So, you know that that's the story that I am, I, I'm always saying to myself.

And I hope that yeah. So basically I use like notes app on my, on my, on my Mac.

Jonathan: If you ever decide to learn rust, I know a guy that's pretty okay at teaching it.

Frank: Yeah, probably some, her, some Herbert, I guess.

Jonathan: Yeah. One of those Herbert guys. And, and you, sir, what is your favorite text editor in scripting language?

Herbert: Oh boy. I get asked this a lot and I learned the hard way. Never try and teach a major workshop with, with them because you'll spend more time answering questions about your text editor than you will actually getting to teach. So if I'm in public, I am using Rest Rover because JetBrains sponsor my Den Labs talks.

Jonathan: Ah, okay.

Herbert: If I'm teaching a private class, I'm using either Rest Rover or Visual Studio Code again, because I don't want the editor to be in the way.

Jonathan: Sure.

Herbert: When nobody's watching. I'm running Neo Vim with Rest Analyzer. Mm-hmm. And a ridiculously customized color scheme because I've finally found a color scheme that's easy on the eyes for eight hour sessions and mm-hmm.

I'm sticking with it. Yeah.

Jonathan: Yeah. Scripting language. And so here you have to ask yourself, does rust count as a scripting language?

Herbert: Not really. I mean, it, it compiles, it takes a while to compile. There is a crazy program there is a crazy project out there that lets you do shebang bin rust and have it work, but I wouldn't recommend it.

I actually, like, I actually like Python these days. Mm-hmm. I was a pearl guy for a long time. Robert has converted me to the Python gang, so,

Frank: yep. And, and, and, and I, I, I think I remember answer for, for, for Dave, and that was emax. That sounds right. And and, and Python, I guess. Yeah. Was Python, emac

Herbert: EMAX was his operating system.

He once told me he was on a little Chromebook and it took 20, it took two and a half hours to finish loading his customized EMAX setup.

Jonathan: That sounds about right. Oh goodness. All right. Lots of fun. Thank you guys both thank you guys both for being here. We appreciate it and it was good to chat with you both.

Herbert: Oh, thank you for having us.

Jonathan: Thanks for having us. Yeah. Glad to be able to do it. Alright. If you want more of me, oh, I didn't, I tell you what, we're gonna let, we're gonna let each of you plug something if you want to. I don't have a co-host. Normally I would have a co-host plug something and I, I don't have it.

So you guys are the co-host today too. So Frank, is there anything you wanted to plug before we let folks go?

Frank: AB absolutely. Liberty c io and there, there, there are all the links people need. There is a way how to contact contact us. Contact me. Mm-hmm. There is, I, I think there, there are our LinkedIns.

Yeah. So people can can reach me out. Yeah. And at Herbert,

Herbert: My most recent book, advanced Tons on Rest, went into final Beta yesterday. Should be actually in print and ebook print. I get paid a lot more for eBooks so interest Digital print. Digital print and is obviously what I should be pushing.

But I, I love physical books and if anybody wants to chat with me, I'll be at Rusko and this September in Seattle. Oh, cool.

Jonathan: Yep. Very good. Is there, and I didn't warn you guys, I was gonna ask this, but it just comes to mind. I, is there, is there some sort of nonprofit or like a a memorial fund?

If somebody wanted to give something in in memory of, of Dave, is there something you would recommend?

Frank: We don't know of any yet. Okay.

Herbert: We've been talking with Dave's family. Okay. They, they have a lot going on 'cause his mother, his mother also passed away recently.

Jonathan: Yeah.

Herbert: So as soon as we know, we'll, we'll make sure it's up on the memorial page.

Frank: Okay. Ca can, can I plug in Dave's mother? Sure. So Dave's, Dave's mother was a really good programmer at IBM.

She, yeah. And, and they also like to play a lot of music together. So they were like talking about technical stuff, but also about music. So she was, she was probably someone that really influenced his like development. Yeah. I'm sure that that leads, you know, him to, to this buffer boat world in the end.

Speaker 5: Mm-hmm.

Jonathan: Yeah. Interesting. Good stuff. Alright if we do get an answer on that, I will try to try to get the, the eventual article and such updated with that information. But we'll try to get it shared out. Again, thank you guys. Appreciate it.

Frank: Thank you, Jonathan. Yeah. Thank you.

Jonathan: All right. If you wanna find more of me, there is Hackaday.

We appreciate Hackaday being the home of Floss Weekly these days. I've got the security column that goes live every Friday morning and the articles in between things that I find interesting sometimes Buffer bloat, lots of times, other things. You can find that on Hacka Day. I've also got the Untitled Linux Show over at twit TWIT TV slash uls.

Make sure to check that out as well for all of your Linux news and tips and analysis. And we have a lot of fun there. We sure appreciate everybody that's here, both those that watch us live and get us on the download, and we will see you next week on Floss Weekly.

Episode 829: This Machine Kills Vogons

16 april 2025 | 69 min

Episode 828 transcript

9 april 2025 | min

FLOSS-828

Jonathan: Hey folks. This week Rob joins me and we talk with Stefan Grabber about Incus and LXC, a pair of solutions that it seems everyone from home labes to Fortune 500 companies are interested in these days. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 828, recorded Tuesday, April the eighth, Incas inception.

It's time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett. Today we are talking about L-X-C-L-X-D. Incus Linux containers and some of the drama that happened around that in the past few months. First off, I've got a talented and wonderful co-host and it's, it's Rob.

Hey Rob. Hey. That's the best introduction I've ever had. Well, there you go. It's kind of like when universes collide and one of the things that I've, I've enjoyed doing with Floss Weekly is to bring some outside blood in as co-host, and Rob is one of those. So I appreciate you being here.

Rob: Always great to be here.

Jonathan: Yeah. So we we're actually, we're talking with Stefan Grabber, and this is this is sort of, so he is the maintainer for several projects, but one of which you actually brought to the Untitled Linux show as a story. And that's when LXD was forked from Incus because Ubuntu was sort of asserting their dominance over the project, let's say.

Rob: Yeah, they were making, you know, doing the typical, making a lot of big changes to LXD that you know, people didn't like. So yeah. Had to bring that story.

Jonathan: Yeah. Yeah. So as we, as we said in the pre-show, that's one of those easy stories for the cliques. 'cause people love the drama. Alright, well let's go ahead and bring Stefan on our guest for the day.

And so welcome sir. We appreciate you being here. Hello.

Stéphane: Yeah. Thanks for getting me. Thanks for inviting me here now.

Jonathan: Yes, I'm, I'm very much looking forward to this. So you've got, you've got kind of your fingers in a lot of pies as it were with the, the entire Linux containers ecosystem. What, what all, what all with that are you maintaining and what, I mean, I'm not even familiar with the entire ecosystem of Linux containers.

I used, I used a little bit of Docker and I used lib vert and that's about it. And you guys are kind of like. The other side of the pond, it seems.

Stéphane: Yeah. I mean, so some of it has amount of overlap, especially at the, towards the beginning of the Dock Up project, for example. Mm-hmm. But yeah, Linux containers has been around for quite a long time, well over a decade or probably around 15 years or something at this point.

Mm-hmm. And it, it started mostly with the LXC project, which was the first user space implementation of containers on Linux as container support was being pushed into the Linux channel. Mm-hmm. So that came around the time where containers were mostly Linux V server or Open vz, both of which required extensive patches on top of the Linux channel.

Mm-hmm. And so the idea was to implement containers properly in Linux itself. And so Alexi became the, effectively the reference implementation on in user space. So that grew quite a lot of over a number of years. And if we, we go all the way to when Docker started, the first versions of Docker actually used Lxe.

So Docker was effectively a wrapper a layer on top of Lxe to do all of the layering and image management parts. That was really the innovation. Mm-hmm. But when it came to running the ER itself, they were using alexy. And then my, my best understanding, I'm not sure if that's accurate, is that as they were building that into a company, it made a lot more sense from like an IP point of view for them to also to own the entire thing, not just be a rapper on someone else's thing.

Imagine that. Yeah. Especially when you're looking for funding and VC and all that kind of stuff. Sure. That's something that, you know, keeps coming up. So I think that's when they decided to start with like the container, the aspect and all of that stuff. So effectively running. The container themselves instead of, instead of just setting things into place and then asking Alexa to do it.

So that's when that changed. Like as far as the user experience to the user, there wasn't really much in the way of changes there. It was really more like a, I suspect it was actually more about control and it was about about the technical difference at that stage. Mm-hmm. Because as someone who writes a lot of go these days and do written in, go writing a container manager directly in go is a terrible idea.

Go is really not good at doing low level camera level interactions. They had to jump through so many hoops to get something like Docker working properly. Whereas when they were using Alex, Alex is written in C and so has that very tight integration within external that's a lot easier in that way. So yeah, it was mostly kinda the, you know, how Alex kind of started a long time ago.

Mm-hmm. Was initially created by a, a small French company, then was funded by IBM for a while. Then after that it was mostly canonical funding it while I and a bunch of others were working there quite a while later. We effectively felt the need. Well, we felt that LXC as a tool worked fine, but was not user friendly.

Mm-hmm. And also was not using most of the new features in a good way because we always wanted Alexa to be backward compatible so we could never make it use all the shiny stuff out of the box. Yep. And so a bit of a reset was, was a good idea. So that's when we came up with the Legacy LEAP project.

Like kind of, yeah. Creating of legacy product was kind interesting. Like, we wanted to do something on the LXC side, like maybe writing a new tool or something.

Mm-hmm.

And then canonical management, as in Mark came down with like, Hey, I really want a new thing. More like dockers, like, well, that kind of works out.

We, we, we were looking at doing that anyways. Then came a pool, bunch of design and bike shedding on naming and programming language and all that kind of stuff.

Yeah.

After some back and forth and some agreements and disagreements, we found a compromise that we ended up with, which was a project called Lex D.

The name was a compromise, not something that we liked. Mm-hmm. Written in go. That was another compromise that wasn't gonna be our first pick. But still part of the Alex community? No CLA released and and Apache, the Apache to license. Those were the bits that we effectively worn during the negotiation.

Mm-hmm. And we kinda kept going with that for quite a few years, probably. I mean, at this point we are, we are getting when was it? Yeah, well, probably the decade point now. I think I left before the first decade of, of Flex D but now at this point they've crossed a decade. And yeah, that's, you know, been, that's been growing quite, quite a bit.

The user experience and everything was good. We did manage to make it a, as much of a committee project as possible, which Right, while working for Canonical can be a bit tricky. Like the, so some of the views and some of the ways they normally run projects at Canonical are very much gonna.

Company driven and with a, a, a bunch of obstacles in the way of con company contributions. Mm-hmm. We thankfully managed to negotiate most of that away. And so I think as far as public facing open source projects and Canonical LexID was one of the kind of good ones. There were a bunch of other kind of drama internally with Canonical and stuff, which led to me wanting to leave.

And so I announced that I was leaving towards the, well, halfway through 2023. Mm-hmm. And that's when we very quickly noticed how much of, of, of, of those exceptions and things were there, literally because of me. 'cause the moment I announced I was leaving, that's when it's like, okay, let's rebrand the project.

Let's move it away from the electric community. Let's kick out all of the community maintainers out of the project and only keep employees in there. Mm-hmm. All of those changes happened basically within weeks of me announcing I was leaving. I wasn't even out yet. That that stuff was all happening effectively.

Interesting. So that's kind of unfortunate. Like, my, my goal at the time was actually to leave the company, but remain a contributor to Lexi. I would've liked to remain one of the maintainers. I could have still done a bunch of the release management, run a YouTube channel, all that kind of stuff. I was quite happy to do that outside of the company.

But they've, they made it very clear that they really didn't want that. And hmm. That then caused our community to get quite frustrated by this. Which then led to Alexa ra at SUSE to go and fork Lex D into Inca. And well then we had a Lex D sized hole left in the UX containers project at that time.

So we figured, hey, well that Incas thing would be a good fit to fill in that hole. And we just took it over. And effectively all of the lab resources, all of the build machinery, all of the stuff that we had as part of the open source project just all went straight back into Incas. And that's how we've been running this thing ever since.

It's, it's actually kinda interesting 'cause I've been running effectively have, yeah. I've been running in effectively the same way I used to with Flexy, but we've seen a lot more interest, a lot more contributions, a lot more active, other companies getting involved in that kind of stuff with Incur and we have added with Flexy.

Mm-hmm. Despite. Like on the face of it, it's still the same license, it's still on GitHub and the same project. It's everything is identical. The only thing is that there's no more canonical being tied to it. Mm-hmm. And that had a lot more impact than I expected. Like I was actually expecting the opposite, being like, well, you know, companies might.

Like having a bit more of, a bit more backing, full-time paid employees and stuff like that, working on a project before they decide to use it. Right. And no. If anything, we've seen a bunch of, of, of, of folks like large companies and stuff jumping from legacy over to Inca just being like, Hey, that's a way to get rid of that canonical thing and have, and have a project that we can use on whatever district we want that's not gonna cause us issues for support and that kind of stuff that we can feel like we can contribute to and is not at risk of being canned at any point.

Mm-hmm. Like Canonical is not Google, like they don't kill projects every five minutes, but. Still, like it's, there's always that risk. That's happened a few times, like a big company pivot type stuff, and like three, four or five projects just immediately become unmaintained.

Mm-hmm.

I think is not particularly good at killing off projects, but they're very good at moving everyone working on it to do something else which is kind of the same, just less obvious.

Yep.

Jonathan: Yep. So you mentioned the Linux containers project. What, what is that and how is it distinct from say, Incas?

Stéphane: Yeah, so Linux contain project is kind the, the umbrella project. We've put around a bunch of things. Initially. Initially we just add LXC, which actually stands for Linux containers, but that's just kinda confusing.

And we then added more things. We created a project called l Ccf s Alex CFS, which is a file system to expose exact resource usage inside of the containers so that when you go in a container and you run. Top three or whatever, it shows you the actual amount of memory that just that you're located in a container and not the system, the global system value.

Mm-hmm.

So we implemented that a long time ago now. So that was our second project and like Okay, what do we put it in? Like, what's gonna holding all of those projects together? Right. Thankfully we didn't say it is canonical Winsted went with, there's the Linux containers project, which then effectively hosts the project like CFS project.

And it also has incur for a while. We had another project called CG Manager, which was like a legacy cgroup manager until System D kind took over. Mm-hmm. That from us and also host technically probably another 12 or 15 other projects that are just not significant enough for to be listed directly on the website.

But it's like all of the related stuff like incur deploying SOS for like incur deployment. Terraform provider there's like a CRI, so if you want to use LXC wooden Kubernetes, there's, there's a thing for that. There's like a whole bunch of other projects and, and repositories that are all within the LLXC org on GitHub.

And so therefore part of the Linux containers project and the, the main benefit, like we don't, like, it's, it's not formally registered, like it's not a nonprofit or anything like that. Mm-hmm. But it still benefits from having effectively a strong set of org, like organization wide maintainers that can step in at any point in any of the projects to deal with things like security issues or that kind of stuff.

And then we have per project specific maintainers. Mm-hmm. So, so that kinda, that makes it easier to handle cases where like one of those projects might go and maintain because the one guy who was working on it is not really focused on that stuff anymore. Now we can go look for someone else or we can, you know, step in and at least handle critical bug fixes and security on it.

Jonathan: Yeah. So I'm, I'm curious about LXC. Is that you, you said it was kind of the original reference implementation for containers and Lennox. Mm-hmm. Is it, is it like, is it still maintained? Do people still use it? Is it still Yes. Recommended for new installs? Where, how does that like,

Stéphane: so yeah. So it really depends direct, so there's kind of two aspects to LXC.

There is the library itself, so live LXC mm-hmm. Which is very much used by everyone using Lex D or Incus, because that's still how we actually run containers behind the scenes. Like, as mentioned, we're also written in Go, but we didn't do the same thing as Docker, which is try to run containers natively from Go.

'cause that's a bad idea. Instead we, instead we use C Go so that we call into a C library, which is the, and then we run things from there. Mm-hmm. So that's effectively if you use Ingo, if you use Lexi, you are using Label xe just not something you really notice because it's driven by that upper layer for you.

It's also directly used by some other folks. Especially if you think of like the, the guys running open the WRT for example.

Jonathan: Mm-hmm.

Stéphane: Those platforms are like so tiny that running a piece of gold that requires 50 megs of memory just to start the demon is not exactly a good fit. So on those platforms, when you want to do containers, quite often it's using a.

And the c tooling, which is like the, the most like interesting kind of old school way of running containers. Yeah. And, and that's still being used there. So like embedded platforms. Yeah, that's, that's definitely a thing. Like open the RTI know has it, mm-hmm. We've seen it on some phones as well where you know, if you think something like a Samsung deck or those kind of things where like you can dock your phone and get a desktop.

Mm-hmm. I think these days most of them use virtual machines 'cause phones can do that. But for a while, yeah, we had some contributions from Samsung, Huawei, and some others. We have a suggesting that they were using Alexa.

Jonathan: We have a listener who says that his mashed potatoes, this is Chromebook, Linux VM actually uses LXC.

So apparently,

Stéphane: No not quite. So on, on Chromebooks the, the, the Linux development environment on Chromebooks is Lex D so that's, that's using Lux d That was, that's actually the largest, that's actually the largest user of Lday arc Chromebooks. So there's like one point something million active machines that are counted as users of Flex D, but they're effectively older Chromebooks, vast majority of whom are effectively school kids who want to play video games.

So it uses

Jonathan: l it uses the Chrome OS used to use LXD, not LXC.

Stéphane: Exactly. Yeah. So it's and they do a mix of both because Google is quite paranoid with security. So they actually run a virtual machine layer using something called Cross vm, which is the, the Chromes VM layer. And then in that VM they, they have Lex D running on a gen two kind of baso s thing in the vm.

And then when you get to the terminal, you actually get it inside of a Lexi D container running on there. Now this is gonna go away not clear when, but it's gonna go away for a very simple reason. Google has a company wide band on A-G-P-L-V three software. And canonical lic Lex D to a GPL. So Lex D is not, is no longer allowed on within Google Development teams, which means it's still fine on Chromebooks because they're using the older LTS, which is still Apache two licensed, but they can't upgrade and they effectively have two ways out of that.

One is they switch to incur which they're quite open to it. Like I, I know the guys that Google reasonably well. Mm-hmm. But the other approach, which is a lot more likely at this point is that they will just offer a straight up vm from cross VM to the user and not do that container layer because that would actually line up better with what we've they've done on other platforms.

Right. Like they, last month they rolled out on pixel phones that you can get a DN vm and they want to, like, ChromeOS and Android are gonna like, effectively on the collision course at this point. Mm-hmm. It looks like they want to save some costs and start effectively having the, the Chromebooks move slowly move towards Android.

So using the same code base or very similar code base to what's used on the phones makes a lot more sense. Mm-hmm. So I don't expect the Chromebooks to move over to Inca, so I expect them to just move over to Pure VM and, and get rid of Lexi D that way effectively.

Jonathan: Yeah. So back to LXC, keep our acronyms right here.

The, the whole world seems to have gone in the docker direction and you know, I'm, I'm humor that even if you want to use something other than Docker, like I make quite, quite a bit of use of Pod Man. It has all the integrations so that you can use Docker commands with Pod man, and it's pretty much a drop in replacement.

What, what is, what's the situation like to use LXC instead? Is there kind of a similar compatibility layer? You talked about a way to be able to use LXC with Kubernetes, which is very interesting. What, what's that situation like?

Stéphane: Yeah, so mean Alexa can be used as, as a runtime. It doesn't really care what the image is.

Most people, like when they think Alexa containers, they think full distros because Alex was mostly seen as a replacement for Linux V server or open vc, both of which really focused on providing VPSs. So when you wanted to, to have high density VPS hosting, virtual machines were not really viable.

You could use containers and give an environment was a lot cheaper to run. And that's gonna, that's, that was the market for V seven open vvc, and then eventually LLC. Mm-hmm. That's still a big thing. There's still a lot of cheap VPS and stuff that still uses containers that way ing. There's also quite a lot mostly on the web hosting side, which uses Lxe quite extensively because they want to run a lot of like run, you know, a lamp stack or something else, just like web server, MySQL, PHP go.

And they, like, you could run that through Docker. Sure. But then you need to run like three containers and interconnect them and deal with restarting them and all that kind of stuff when you could just install the packages in the container and just move on. So those, like, we've see quite a of those guys still going on.

But I said like, XE fundamentally doesn't care. We have when you create a low level XE container, we've got multiple templates you can choose from, and one of the templates is called OCI and literally pulls an image from the docker hub and just runs. It. So, so we can, we can do that just fine. It's not, again 'cause it's LXC, which is the low level kind sample tool type thing.

At this point, it's not super nice user end, user friendly. But it's a kind of feature that we've then in added into projects like Inca, right? Incas these days let's you do system containers. So the traditional LXE type thing let's you do virtual machines and let's you do OCI containers. We can point it a docker hub, point it at another registry and just run those containers.

Yeah. So then you get kinda all of them covered.

Rob: So with the LXD, Incas and LXC, what is, what is the relationship between those? Like I'm familiar, I'm familiar with L-X-D-L-X-C, I mean not functionally, but it sounds like LXC. My understanding is LXC is like the container and is LXD like. A manager?

Is that more than just that?

Stéphane: Yeah, effectively that's what it is. Like both Flex D and Incur are effectively managers on top of other things. So both of them do LXC and QMU as far as what they, they allow driving. Okay. LXC for containers, QMU for virtual machines. And then the, the main benefit is that they, they give you a nice rest, API, they provide image management.

They provide like multi-tenancy clustering like complex storage and network management, all that kind of stuff. Which if you are using old school LXA, you would have to do everything USSH by hand on a machine which is very appealing to some folks, but also very much not appealing to a lot of other folks.

So yeah, that's, that's really like a, a full management layer on top of it. The idea is to make it feel like a private cloud like it supports object storage, it supports like virtual networking supports, all those kind of things. So that it looks like, and acts like a private cloud, that you can run this on your laptop or you can draw it to a few machines in your home lab.

Or you can be like one of my customers and run it on 2,500 servers. Like it's, you, you, you can do it at different scales and it's remains mostly the same. Like someone can who's working for another. Those organizations running dozens of large clusters, they can still run it on their laptop, they can do their development on their laptop, they can run even the Terraform provider and all of the development deployment tooling on their laptop, make sure everything behaves.

And they basically show this is gonna work the same way in production. Which is not something you, you can do quite as easily with something like Open Stock or even Kubernetes. Like there are some single machine Kubernetes diss, like Micro Case or K three s that gets you most of the way there, but there's still a lot of moving pieces as part of those.

Whereas yeah, something like Lexi Incur is, is. Very tightly integrated, very lightweight and, and simple to run. Like typically you've got one process running on your machine, that's it. Like you don't have, I don't know, 50 plus I guess, for Kubernetes these days.

Rob: Yeah. So, so when you said when you're correcting on the Chromebooks that they run LXD and not LXC, it's because they're using LXD and, and Qmu for, for the virtual machine product.

So they don't actually

Stéphane: use the VM parts, so they, oh, they're the, the vm the, their VM layer is cross vm, which is their own virtualization manager. Okay. So they've written, they written their own equivalent to QMU effectively, and then inside that VM they run next day. And then the Chrome Os os talks to like the other, other address, API, to then create containers.

So if you enable all the features on the Chromebook, you can make it so that in your terminal, you can have one terminal, that's Deb 12, one terminal that's open to something, one terminal that's, that's sent to us. And those are gonna be three different containers running on, on the Lex D instance that they're running in wm.

Rob: Okay. So let's get on to Incas more again. So, is Incas still feature compatible with LXD today? And do you see them, you know, is it gonna stay upstream or do you see it splitting off kind of going its own way?

Stéphane: So it's, it's already split off to an extent 'cause we don't have a choice. For the first six months or so, Lex D was still Apache two, so we could exchange code back and forth.

So we would add new stuff, we would fix things, they would look at our code, they would integrate that. We would do the same. And some features were just on one side or the other. Like if it was a feature we didn't want in Incus, we're just gonna leave it in next day and vice versa. But there was good back and forth for the initial six months.

And then around December, 2023 is when they realized to a GPL. Which now makes thing make things as symmetric because we are still Apache two, which means they can take our code and put it inside of next day and release it as a combined work under A GPL. We can't even look at their code without being tainted.

So we on purpose do not look at the report or issues or anything on the LD side because even just seeing patches and stuff on that side could cause us to be tainted by a GPL codes that could cause issues. So, yeah, that's so now we've, we've diverged quite a bit. Like I mentioned earlier, we support training or CI containers so we can run Docker containers on Lex D doesn't have that they just didn't take that feature.

We support some storage options like Lin store or like using an NVMe TCP or Fiber channel san, they didn't take those changes either. So there's like some fairly big features now that are quite different between the two. It's. And it also limits our ability to handle kind of moving from one to the other since we can't go look at their code or look at the database che, look at any of that.

Basically anyone who's running on next day prior up until 5 21, which is their last LTS to be released on the Apache two, we can migrate from that to Twink gu. We've got a tool that just does it in place. It takes like 30 seconds and you've converted from, let's say twin Gus. That's very nice and easy.

Anything after that we, we can't do anything about. So then you're looking at twin installing,

Rob: So you're likely to diverge quite a bit as as time goes on, if anyone copies anyone though, they'll be copying you. Yeah, exactly.

Stéphane: So my understanding, 'cause I'm still friends with some folks on that team, is that they have a massive spreadsheet where like they look at all of Apple requests and they put them on a spreadsheet for when they've got some time, have someone go look at it and then include and backboard the code.

Switches. Yeah, that's kind of, it is, it's kind of funny 'cause I think at this point we've got a very active community. Like some, some weeks I'm, I mean, like right now I've got I think, 45 issues that are actively being worked on by different people.

Jonathan: Mm-hmm.

Stéphane: So some weeks we, we merge like dozens of port requests.

And I know that on the canonical side, they only have like one, if not two full, maybe two full-time people. So if they just try to keep up with us, they're already spending all their time just keeping up with us. They don't actually get to do anything. Which is, which is kind of funny.

Rob: That is a, that's an interesting place for for a fork to be in where they're kind of really taking over and they kind of did to themselves.

Stéphane: I mean, it's not the first time. I mean, imagine that, you know, open office, lib office, there are a bunch of other similar things that happened. You know, it's, it's, it's not, it's, it's not really the first time this happens. And yeah, we've also kind of changed, like at the beginning we were always introducing ourselves as like, oh yeah, incurs is a for of legacy.

Now we've actually gotten enough people that just started with Inca from the beginning that we're not even mentioning the fork part anymore. It's just like Inca is like a private cloud platform that drives Alexia and QMU and stuff, and we don't even mention the Lexi thing anymore.

Mm-hmm.

Because everyone was gonna move away from Lexi to Inca.

We'll have done so by now, and everyone else upgraded to a version that can't, that can't move seamlessly to Incas anyways. So they're gonna need to, to reinstall and redo things. So, yeah.

Jonathan: I'm, I'm curious, does Incas have options for high availability?

Stéphane: Yep.

Jonathan: How, how, how sophisticated are those? So I, I've, I've done, I've not built it, but I've, I sketched out the plan for, and I actually wrote the estimate up for building a high, a very small, high availability cluster for a customer.

And there's a lot that goes into that.

Stéphane: Oh yeah. It's, it's not easy. But yes, so the clustering piece we did in Legacy and then improved significantly in Incas handle that, handles that pretty well. The main thing is you. For all the kind of hha to work properly, you need to have your storage and network BHA and that's usually the hard part.

So like on the storage front we have three options these days. So we've got cef if you're using CEF or storage, then it's already distributed and you've got h ha on your storage. Mm-hmm. We've got LVM cluster that we can run on a shared block storage that all machines can access. Mm-hmm. So that can be like at home you could use like a, a NA that supports ICE KZ and then have your servers use that, that ice KZ volume as their shared storage and Okay, cool.

Now your storage is accessible from all three or all five or whatever number of machines you have. You're good. And the third option, which we merged in the previous industrial release is Lin Store, which is it's by the company that. Did most of the work on DRBD and it's effectively a managed layer on top of DRBD that feels like cef, but is a lot lower latency and bio throughputs than sef because you effectively only ever rights to your local disc and it gets replicated to one other machine next to you.

Instead of doing like the full distributed rights across all of the machine type stuff that CEF does. So we've got those three options on the storage front. Once you've got your, your instances volumes and everything on, on shared storage, then network is the other one. And for network, then you have two options.

One is you've got a physical network that you just want to use. So you've got VLANs or something like that with a physical router sitting on that vlan. Mm-hmm. All machines can see the same vlan. Cool. Just dump the containers or VMs on that and you're good. Or you need to go with distributed networking.

And for that we support oven. So that's the distributed open V switch layer, which. Yeah. Then lets you do virtual L two networks and that kinda stuff that's also distributed where like if one machine dies, the virtual router just seamlessly moves to the others. And things just, yeah, handles. Ha So once you've got storage and network, then you've do the easy part, which is Incas itself.

And Lex D and Inca both can do it the same way where the database that we use to store most of our state can be natively clustered. Mm-hmm. So once you start adding more servers in the cluster, as soon as you get three, the database switches to clustered mode and it uses raft for distributed computing elections, all that kinda stuff.

Mm-hmm. To distribute that, that state and make sure that machines like effectively vote for to achieve quorum on any changes before it actually gets committed to the database. Mm-hmm. At which point all of your servers effectively see the exact same thing. One of them gets elected, the leader and is the one that handles most of the, most of the right changes within the cluster.

If that machine goes down typically within five seconds a new leader has been elected within the cluster. Mm-hmm. And then you've got the remaining part of ha, which is, what do you do with the instances that just went boom, when the seven went down?

Jonathan: Stone. Stone?

Stéphane: Well, so we don't need to do that. So stone, so you need to do it in some cases, like you don't need to do it from the database standpoint, because since we use raft, raft avoids the split brain issue completely by requiring a quo vote.

So you need Right, right. You need uneven number of machines, a majority of whom agree on a change. For it to go through, which avoids any kind of split brainin type situation. Stone

Jonathan: stone, by the way, for those listening that are not ha nerds, STONEY stands for shoot the other node in the head. And it is one of the ways to solve this problem.

And it it, it really comes up if you have two machines and they each have the ability to just turn the power off on the other machine. Mm-hmm. And so whichever one, you know, if there, if, if a problem happens, whichever one wins the race, just kills the other one. And then you have one kind of canonical VM or whatever service you're hosting.

Mm-hmm.

Stéphane: Yep. And, and so like typically we don't need that at the in cost level. Mm-hmm. But in the storage case you do so with c or in store we don't have that problem because there's like a built in lock lock manager and everything that prevents contract rights and that kind of stuff. So it's fine.

Also c is typically network. So if a machine goes down, it usually means it's not responding on the network. Therefore its storage is probably also. Broken. Mm-hmm. So you don't risk to have rights. It's fine. The issue is really with the LVM cluster option, because if you're using fiber channel, for example, fiber channel is a completely separate network from your normal network.

Yeah. Which means you could get into a case where your network went down, the machine looks like it's offline, but it's storage is still functional Good. At which point, if you start the same VM a second time, it's now running twice. So in that scenario, that's when a documentation is like, yeah, it's, so you need to have a way for, we emit an event that's saying this machine is dead.

You need to ensure it's actually dead. So that you avoid, you avoid that whole issue. Yep. But internally, within have effectively a, a counter you can turn on, which is like self-healing. And you say, you know, after a minute or five minutes or whatever of that machine ha being seen as being dead as in, has not been responding to the database pings and is also not responding on the network.

Then treat everything that's running on it as being dead and start them back up on the remaining machines. So it just goes back through. Our scheduler picks different machines and starts them back up on that side. And when the machine that was detected as dead comes back online, it's gonna automatically come back in maintenance mode.

So it doesn't take back anything that's running.

Mm-hmm.

Which again, avoids another kind of potential issue with some split brain type stuff. Yeah. So it never starts anything back up when it comes back online. And then you can go and check and if it looks like it's safe, as in the hardware is not broken and everything seems to behaved, then you can get it out of maintenance mode, at which point it's gonna go and move back the stuff that was like spread across the rest of the cluster.

Jonathan: Yeah, that's, that's impressive. This is actually being used in production in places I assume.

Stéphane: Oh yeah. So I mean, currently, so I mean, the way I can set up is I've got, after I left Canonical, I created my own company. Mm-hmm. So I'm basically self-employed for a bunch of that. I'm helping out with a bunch of folks that do anything from like game streaming platforms, web hosting government stuff, kind of all over the place.

Public clouds, that kind of thing, that directly use Incus and just need the occasional, very technical, low level help. Mm-hmm. But are also the CTO of another company that we created back in July whose job it is to move Fortune five hundreds from VMware to not VMware, so, and anyone to follow the news.

Broadcom is quite expensive these days.

Jonathan: Yeah.

Stéphane: So a lot, a lot of those guys don't really enjoy getting eight. Figures bills. Yeah, so they, they, they were, they're quite actively looking at moving and for that we are, we're effectively, we've written tooling and are moving VMware VMs in tens of thousands from, from VMware clusters over to Incas.

And that's where like a lot of those kind of ha balancing type stuff was needed because they're running VMware clusters that have self-healing. The DRS features, like the automatic rebalancing and mm-hmm. And load sharing within the cluster, all those kind of things. So we effectively made sure that we have equivalence for, for all of the useful features.

Mm-hmm. We obviously don't want to be identical one-to-one with VMware. 'cause VMware probably has like 70% of its features that nobody uses and others. You know, dead weight, we don't want to replicate any on that. Right. But the, the useful bit, especially on the transition side Yeah. We've, we've been working quite actively to support on that.

Jonathan: Yeah. Interesting. You, you talked about having a, an upgrade path going from LXD to Incas. Is there also an upgrade path going from one version of to the next?

Stéphane: Oh yeah. So in both Flex Day and Incus have always been extremely backward compatible. I think, I mean, for a long time with Flex Day, you could go at any point from next D 0.1 to next D 4.0, so effectively a eight or nine years jump.

Mm-hmm. And you could, that would handle it perfectly fine. Like the database would just detect what version it was and apply all of the changes and do all of the data migration needed. And it was perfectly fine. Yes. It's with 4.0 we switched the database from some database packages from using a raft implementation written in, go to one in C and that goes enough bumpiness.

That supporting, upgrading from like 0.1 all the way to 6.0 would've caused us to keep so much debt code around just in the migration. That didn't make sense. So in Lex D when we did 5.0, we made it that you needed to do a stops through 4.0. Mm-hmm. So if you were not on 4.0 already, you need to upgrade to 4.0 first, then you could go all the way to five or whatever.

And now with Incur, we mostly have the same thing where if you were running Lex D 4.0 or newer we can do that upgrade in place. It's fine. And if you were running Lex D prior to that and indeed to first do an upgrade to like Z 4.0 and then you can jump onto Incur 6.0 6, 11 6, whatever. Mm-hmm.

And within Incur itself yet the upgrades are extremely seamless, mostly because we do monthly releases. So people are kind of used to doing, going through updates and we don't want that to be a bumpy process. So it's basically new package comes in interest restarts, you're done. And it usually, even when you do a big jump, like an LTS to LTS jump, it still takes less than a second for database to have done the update.

So that's, that's very, very quick and, and seamless.

Jonathan: You, you kind of, you kind of jumped on a train that I was already thinking about, and that is the, you mentioned earlier that you were not super happy with having everything written in go, and it sounds like you've started moving back to writing some things, rewriting some things in sea.

Well,

Stéphane: so actually of, so running containers from Go is not a good idea and we've never done that. So thankfully we've avoided that kind of worms. Mm-hmm. When I keep seeing what some of the container D and residency folks have gone through, as far as like the weird bugs and workarounds and stuff they had to contribute to go proper to make things behave in all cases, I'm very happy that we don't need to deal with any of that.

Right. That being said even though we were forced into doing Go when we created the LD project. That was actually a good fit. Like Go is actually something I enjoyed writing these days, especially for those kind of larger code bases that write that have a very large rest, API and it to, to run a lot of background tasks, go is actually a pretty darn good fit for that Go is not a good fit for anything low, super low level.

So that's why yes, for example, the database running the database purely in Go was not a particularly good fit because the on this format is cite, cite is C only. And so go having to constantly kind of back like contact switch between go and see was just extremely slow and, and problematic. So instead the way it works now and, and that's been the case for a while, both in Lex and, and Incur, Lex uses something called dite Incurs uses something called CAR sql.

Same project, but the author of the cite also left Canonical, so he also fucked this project on the but like in effectively how that works for anyone who's deep into go stuff is with Spawn what looks like a go routine that loads all of the, the car SQL stuff, which is 99% cgo. And then we lock that go into a single thread.

And that thread from that point on effectively only runs C stuff. And the way we actually access the database from the goal size to that C thread is we go over the network. So we use a unique socket between the two threads. Yep. Because the go socket API is very, very fast. No block, no blocks, no C context switch, none of that.

Mm-hmm. And the other thread just runs C only so it's, it's nice and fast.

Jonathan: Mm-hmm.

Stéphane: So that's kind the workaround we took on that one. And that's been working fairly well.

Jonathan: So there's no, no movement to rewrite the whole project in Rust?

Stéphane: No. It doesn't, it wouldn't make too much sense on this one. Like like I, I, I mean, in general, I'm someone who, who's or doesn't really get the, why are we rewriting stuff that's been written 10 years ago and has been quite well maintained and fixed.

And, you know, if we look at the Lxe project, it's been in, it's been going through security audits by security firms probably a dozen times by now. Mm-hmm. It's gone through, God knows how many similar of code security research stuff with Google. We're pretty sure that the quality of the, the, the Lxe project is pretty good.

Now rewriting it in rust, you're gonna spend six months to a year or something rewriting it, where your best case scenario is nobody notices.

Jonathan: Yeah.

Stéphane: Like that's literally your best case scenario is like you didn't introduce any new bug, any new trends and nobody notices, but you've also not implemented anything people want during that time.

Jonathan: Yep.

Stéphane: Yep. So it, it's quite hard to justify, re

Jonathan: rewrite it, rewrite it in rust is become quite the meme in, in our in our circles. I think much much the same way that, you know, how are, how are you adding crypto coin to, this was five years ago.

Stéphane: Yeah. So I mean, like we, we are actually looking at adding some new side projects to Alexi for, for some stuff.

And like when we're looking at writing a complete new code base that needs to be low level talking to the camera and stuff, that's when like, okay, we should probably do this in rest because if we're gonna do something new from scratch now that needs to be low level and memory safe. Mm-hmm. That's a good fit.

Wasting resources, rewriting something, not so much. And I know, I mean, we've probably seen the news like, you know, open to switching to more rest. Detailss and stuff. Mm-hmm. And I, I just kind of came back from a kernel developer conference two weeks ago and we just for fun of our lunch, we're looking through that code and we're like, why?

Like why would you switch away from the good util Lennox that yes, the code is absolutely horrific to read, but from a kernel point of view is doing things right to something that is not memory safe, but uses the wrong kernel APIs everywhere. So you've got time of check, time of use, security flows in every single function we looked at.

Mm-hmm.

Like the, that that's utility Linux RS thing or whatever they call it,

the

mount implementation and some of the other common helpers in there. There is like, we found probably a dozen security f flows in the first function we looked at. Oh, interesting. Because yes, there's no memory safety issue, but that doesn't mean that you've not screwed up the Canada APIs and have race conditions everywhere.

So that's, that's a bit of an issue for sure. And it also kind of funny because util Lennox, it's not a demon, it doesn't run with Set your ID bit. Mm-hmm. It doesn't, it's not exploitable. Like even if it's badly written, like even if the C code had a bunch of memory issues all over the place, it's only ever runs as the user was running it.

So it doesn't matter,

Jonathan: huh? I if, if I remember correctly are, is it, is it core utils? Yeah,

Stéphane: I think Core Utils, yeah. Yeah, yeah. Core utils are, yeah.

Jonathan: I, I think we actually I think we actually had the, the, the Russ core utils maintainer on the show here, and I don't think he was calling for any distros to use the rust version.

No. If I remember, and this has been multiple shows back, but if I remember his, his comment on it was essentially. They were sort of doing it for fun.

Stéphane: Yeah, I mean, I, I I can definitely get that like, it, it's got some interesting characteristics like trying to run on multiple operating systems. That's interesting.

Mm-hmm.

It, it, it, it's doing some, they've got some interesting concepts on there, but this shows not just wanting to jump on it because this is rust. This must be better than see No because something is memory safe doesn't, by default make it better than what's there. Like, you need to actually understand what your attack surface is, what you actually want to guard yourself against.

And yeah, when you're looking like I'm, I'm a lot more interested in something like sudu, like yeah, Sudu. That one is a set binary that's used for privilege escalation. Mm-hmm. Yeah, that one, I could see the point of not wanting any potential memory safety issue in that thing. Call tails. Nah, not really.

So it, I, I think it's gonna be interesting, like, you know, like I'm surprised that Ubuntu seems to be jumping the guns scan on this one. I, I think it's the kind of thing that would make a lot more sense for some of the more fringe diss basically to pick up, like, you know, do a build of Alpine that's got all the rust stuff or do void or something like that.

Right. Right. Prove that it makes sense that it works well somewhere else before you pick it into the most used distro. Maybe it's just for the cliques that's quite plausible. I mean, the, the new VP of engineering a burner is a big Rust fan, so I'm not super surprised. John is, def has been pushing for Ghost for Rest stuff for quite a while now that he's got the VP title.

I'm not super surprised to have seen that change show up like a month later. But, you know, at the end of the day I can have to, to wonder like, is that really. A good change for Aun user? Should you have made that more, you know, optional and stuff like, do you really want that in the next L? Yes. I don't know.

Again, I, I would've, I would've picked some, probably some things before. Cote is like, so do 70 pretty high on the layer stuff. Like effectively take your system, look for all the secure ID binaries. Those should be the priority for moving away to the not see look at what demons are running out of the box on the machines.

Those would be good candidates for wanting memory, safety random details, not really.

Jonathan: So we did, we did talk with Sylvester La Drew back in July of 2024. That's episode 7 92. It was, it was a great conversation. He was not calling for this. Yep.

Stéphane: Yes. I mean, that's, that's another thing is like, you know, maybe talk to the upstream project to see.

You know, also on that, on their front, like what are they, what do they think the set of their project is? Are they happy with this being more widely used? Is this production ready? Do you have a story around security updates, security disclosure? Do you have, you know, that kind of stuff in place before A major distrust, which is over to it.

That's, that's kind of thing I would've expected to see. And I've not actually checked half hour long. They got into that way, Ubuntu. 'cause I used to be quite involved in Ubuntu. I used to be on the, the Ubuntu Technical Board. I, I'm still in Ubuntu, core developer. I used to be on the archive team, release team, a whole bunch of other things.

Mm-hmm. So I used to be someone who was in the way of those kind of massive changes. And normally for a change like that to happen, it would have to go through an archive review, which involves not only the legal review, just like. Copyright license combination, all that kind of stuff, but also a full code review by the security team and looking at how does the project handle security disclosure, how does, like, all that kinda stuff before we change a, a, a main component over.

Mm-hmm. So I don't know if that already happened or if it's still yet to happen, and maybe there will be some bumpiness at that point. But yeah, like I think rust is a good thing in general, but there's definitely the the usual kind of early adopter thing of people wanting to use it for absolutely everything that exists.

Before kind of focusing back to where does it make sense? What should, what should actually be useful?

Jonathan: Yeah. So I, I don't wanna put words in anybody's mouth. I did find a comment, it's from Ubuntu, and I think it, I think it must be from, let's see JNS Ruck. Is that?

Stéphane: Yes, that's John sga, John SGA uk John.

Okay. That's the VP of engineering at Canon.

Jonathan: Okay. So he did have a conversation with, with Sylvester okay. And they, they said that he met with Sylvester to discuss the proposal to make U Utils core utils the default in Aun 2 25 10. And he was pleased to see, to hear that he Sylvester feels the project is ready for that level of exposure.

So I don't wanna put words in anybody's mouth. Okay. That is the, that is the conclusion that they came to. All right. Okay. So

Stéphane: it looks, looks like I, well, that is, John had the discussion that, that, that was needed there and that's Sylvester fails confident that this is fine. So I guess we'll see where that goes.

Yeah, I'm, I'm, I'm sure we're gonna see a bunch of bug fixes resulting from that. If anything else, like, you know, that's the kind of thing where sk at some point, those kind of projects, if they, if they seriously want to be an alternative to, to the incumbent they need the exposure at some point so that people flood them with bugs and things get solved.

Like it's. Otherwise, you've got a bit of a chicken and egg thing. Like if you're gonna wait for the project that's just developed by, by one or two people to be perfect before it gets adopted massively. Mm-hmm. It's never gonna get there. So I, I will, I will say, I'll

Jonathan: say specifically and, and we, we can leave this tangent, but I will say specifically with core utils, there is a huge test suite and all of the different core utils implementations actually work together on that test suite.

And so they, they already have a pretty good idea of what works and what doesn't. Although there's nothing quite like being thrown into the fire of being the default on a, on a dis drew like a button too. So I'm sure there will be surprises.

Stéphane: Yeah. We've been seeing to see like a, like if some of the stuff that, that we've noticed, I, I was with five system developers who were looking at mostly five system APIs.

Mm-hmm. And the Linux can support something called O Path for a while now, which lets you effectively open a internal file descriptor to a path that you can then use everywhere else to do. Open something relative to that or reopen this path as something else or pass it to mount or that kind of stuff so that you never have the risk condition of, I've checked something nar called Mount or NAR called something and the file has been changed in between.

And from looking at the call tails, this was not done in there, which means that like all of the functions interacting with files, you could raise them if you were very quick in parallel at like swapping the file. The usual attack would be you swap the file for a same link pointing to another path and it's just kind of triggers to the same link and, and do and access that.

Mm-hmm. So that's one of the things that, that we've briefly noticed, but it was also like on someone's random laptop over lunchtime. So, you know, not gonna go into much more details on that. Like there's also a good chance it was, it could have been an old version, it could have been, who knows? Sure. But yeah, there's that's most, that's mostly to say like there's more to security done, memory safety.

Yes. And. And yeah, just, yep. Because it's rust doesn't mean it's better. Like you still needs to, to tick all of the other boxes and make sure you are, you're doing all your interactions with the US and everything else correctly and safely.

Jonathan: Mm-hmm. So if somebody wants to use ink is back to our main topic.

Stéphane: Yep.

Jonathan: What, how's, what's the way to do it? Can I, you know, can I do A-A-D-N-F install cus from my fedora or my Red Hat box? Is it available on a buntu?

Stéphane: Yeah. So we've got it basically everywhere at this point. That's another big difference with Lexi. Lexi was basically only available as a snap, which okay.

Not everyone enjoys it's, put it that way. With, with incur it's it's a lot more widely available. So, I mean, if I'm looking right now, we've got native packages in Alpine Arch Chime, Linux Debian Federa Gen two Ns Open, Rocky Linux, open Tovo. I think there are a couple more that people have just not contributed in instructions for.

Mm-hmm. So yeah, on federal you can DNF installing 'cause that will work perfectly fine. Same thing for Yeah. Most distros mm-hmm. On Ubuntu you can, because it's actively packaged in, in Debian and Ubuntu is still auto import from Debian, so they got it that way. The main downside is don't count on it being updated in Ubuntu after the initial import from Debian because nobody really seems to be pushing for that and I don't care enough to do it myself.

What I do, however, for both Debian and Ubuntu is that I've got my own package repository that has full builds of Inca on Ubuntu 22 20 0 4, 22 0 4, 24 4 Ian 11, 12, 13. And those packages gonna, they're, they're not your standard package where like everything is nicely split. They're more of the fat package kind, so they come with the right version of QMU with the right version of the firmware for EFI, with the right version of everything.

So that basically it has the optimal combination of everything and we don't have that kind of friction. Your kitchen, they get in these shows where, oh, the QMU version is a bit old, so some of the features are not gonna work because of that. Or like one of the examples would be the the EDK two firmware on a bunch of Distros doesn't support Secure Boot because the distro didn't build the secure boot keys in there.

Or I think another patch I've got for again, for EDK two is. By default, it leaves you a second to hit the boot menu, which basically no human being can do. So my package has that bump to three seconds and some distros have done similar bumps, some haven't. Mm-hmm. So that's why there's a bunch of differences between Distros, but that's what newly happens when you install open source software on, on Distros.

So yeah, that's the main, that's the main way. Someone also, and that's mostly for fun, but actually works surprisingly well. Someone also made a Docker package for Inca. So you can install a dock, you can run that Docker container, mostly turning off all the security features Uhhuh. But you know, if you can run that thing and then you would get Inus running that way, which Incus Yeah.

Surprisingly enough works. And then you can in, in

Jonathan: Docker

Stéphane: basically, and then you can use Doc, then you can use Incus running Docker images on top of it, you know, just to, to make things confusing. Wow. So, so that was mostly a bit of a workaround. I think it's, it's a snap style work around effectively for like any other distro.

That, that will have Docker but doesn't have a native version of Inca. Hey, you can use that, that to, to make it work. We're also working on, on couple of other initiatives around Inca to make it easier to install. So we've got Incas Deploy, which is a set of Terraform and Ansible files to deploy in cost typically at scale.

So if you want to deploy a cluster with, with oven, everything secured properly with a PKI, you know, that kind of stuff that can do it very easily. You basically, it's one YAML file. You just give all of the machines you want, watch what services you want on them, go and it deploys it. Mm-hmm. And that works on open to Deion and CentOS.

And the other thing I've been working on mostly myself for, for the past couple of months now is Incurs os. So that's a lot closer to your VMware ESXI or promo type approach where we build a full OS image. So we're using System D Micros I to make a. Os image that's based on Deb end, that's completely immutable, fully signed TPM measured all of the security features effectively.

Jonathan: Mm-hmm.

Stéphane: So that you can run that on your machine and know that there's like effectively no way for it to have been modified in any, any way whatsoever. And that if only one tries to hack your machine in any way, it'll break the encryption effectively so that you can't access the disc anymore. And it's gonna be very obvious that something bad happened.

So we've got that and we see two targets for this project. One is people wanting to run a home lab at home and don't want to run too much Linux. They just want to acquisition platform. Mm-hmm. That's convenient. That's kind of the mox use case. Deploy that on a few machines. You can access the web ui, you can just click around things and get your containers, VMs, network storage, everything sorted That sorted out that way.

The other thing is, our large customers, when you're running in the thousands of servers. You really don't want them all to run some version of open tool centers or whatever and have to apply updates on thousands of machines. Right. And then as everything hap as it always happens, even if you automate your updates, machines will somehow diverge.

You're gonna end up with like slightly different files and stuff over time. It's, it's a mess. So those environments, very much like the idea of like, this is completely locked down, it's identical bit for bit on all the machines. Mm-hmm. Updates are done with like an ab partition scheme. So that's, you don't know the update in the different slot.

You reboot anatomically into the new slot. If something goes wrong, reverts onto the old, the old slot. That works really well. And it's also like another thing that's annoying with large companies is compliance. They like running, scanning software that in theory tells them what security issues are.

Mm-hmm. Usually those software absolutely terrible, they slow down the machine and they don't know what they're talking about. So imagine that our fix for that is that there's literally no way to get a shell. And the entire system is read only. So we don't have SSH on Inca source. You can't SSH into it.

There is no local shell on the, on the machine itself. So even if you physically go to the machine, you can't log in. There's no shell. Hmm. The only thing it exposes is the rest, API to manage it. That's it. And if you want to actually do a full security audit for compliance, what we tell you to do is download the image onto another machine, unpack the image, run your scanner stuff there.

Jonathan: Mm-hmm.

Stéphane: And then you'll know that because the image is guaranteed bit for bit identical on all your machines. That's what's running on your machines. And don't run scanners on thousands of machines are gonna slow them down and like every time you open a support. Question or something. We're gonna first have you turn off the scanner and wipe all that stuff out to make sure it's not what's causing issues, because

Jonathan: Yep.

Stéphane: Yeah.

Jonathan: Yep. That's a, that's a, that's a hard earned lesson. I can, I can tell.

Stéphane: Yeah. I mean, I, yeah, like one of my pastime is I'm, I'm the VP of Engineering for Cybersecurity conference in Montreal. So I, I, I've unfortunately talked to a lot of those vendors dealing with Yeah. Security scanners. Security tooling and security reports.

And it's, it's all yes. Yes. It's, it's, it's, it's, it is, it is really a mess. And like best case scenario, they're harmless and they just keep reporting security flaws that don't exist. But they have also a bunch of them that are very much not harmless, that are actually like causing either massive CP usage, massive memory usage or street modified stuff on the system and cause a bunch of damage.

So, yeah, the, if I can keep those far away from the production systems, that, that's

Rob: would be nice.

Jonathan: Understood.

Rob: So you brought up a Prox box as a comparison to Incas. Yep. How, how does that compare? Like what kind of person would maybe pick one over the other?

Stéphane: It's surprisingly similar, yet the communities are completely distinct, which, which I'm not really sure why.

So I mean, prox Marks uses Lxe for containers. We've got fourth Gang Boot Metal as one of the droppers at Prox. Marks is one of the maintainers on the Lxe project. So

are quite familiar with what they're doing, how they're doing things. And on the VM side, they also use QMU, same thing as we do.

They have a ui, same thing we do. They have a way to cluster systems, same thing we do. So it's extremely similar. The, the main difference I think at this point is how do you install it? Like Mox is taking the. Linux distro approach of like, there's PX marks va, you don't load it to install it on a machine and it's running PX marks from that point on was up until now, incur is for people who already like to run Linux distros and then want to install something on top of it.

That's where Incas West might make things interesting because we're effectively stepping into that game now of like, well, we also have a pre-installed image thing that you can just dump on a machine and use. I mean, we don't have it yet. We've got built on GitHub that we can play with. I would not recommend putting that on actual machine right now, but in a couple of months we definitely will.

At which point it's gonna get a lot closer and it's gonna be, I think mostly a matter of preference on how it behaves, I guess.

Mm-hmm.

PX marks it's UI and everything to me feels a lot more like VMware in many ways. It's, it's really kind of modeled after it. Whereas Incur is a lot more modeled after the public cloud.

We, we use a lot of the same terminology around projects and that kind of stuff. We've got. Multi-tenancy. We've got like all of the virtual networking pieces are there to, with a CS and that kind of stuff. The same as a public cloud. We do object storage. So Incur is, is a lot closer to the public cloud world.

We also have like a full Terraform provider, so, and Ansible plugin. So for people who are used to to driving cloud deployments through from Ansible, you can do it the exact same way against thinkers. So I think that's currently the main kind of split is that it's the cloud crowd versus the VMware crowd.

The problem is that the VMware crowd is being pushed into the cloud right now 'cause they can't afford VMware anymore. And Hom Marks could have been a good alternative effectively to VMware in that, that situation. But that's not what I've seen large. And at this. So think for the SMB, maybe like the small medium businesses, they might be going PX marks.

That would make sense. It's mostly the same way to operate it. They've got some migration tooling. I think that works well. But for larger scale environments, like, I mean, I'm mostly dealing with Fortune 500 super large companies, but those guys, they, when they look at Prox marks, they see a company that's really focused towards SMBs and home ladders and don't really have clear vision or focus for how to run a full multi-region private cloud effectively.

Mm-hmm. And so they, they get discounted pretty quickly based on that. Whereas Incas is already used by some to run public cloud environments. So we, we know that we can run pretty large individual clusters of like anywhere from like 50 to 100 plus servers, and then usually have, you can treat that as an availability zone.

You can have multiple ones of those inside of a region for like a small region. And for environments that needs to be much, much larger like the public clouds with. Tens of thousands of servers per site, then you just create a lot of a lot of clusters usually based on like your hardware generation or that kind of stuff.

And each of those clusters maybe like have 50 to hundred 50 each servers. So that gets you basically, you can make it so that it's an aisle in data center, you can do that kind of mapping easily enough. Mm-hmm. And yeah, we've, we work pretty well at that kind of scale. And then again, you can use things like Terraform to then easily blast changes at all of your clusters in, in one shot.

So you don't need to keep replicating same changes over another again. So yeah, that's kinda the, the, the thing we've seen is that currently SMB Home Lab tends to go oxs, cloud folks tend to go InCast. That's mostly what what we've seen. It's gonna be interesting to see what happens within Qs os and whether we end up picking up some of the Mox folks at that point.

Jonathan: Yeah. I'm, I'm super curious. What's the experience right now for trying to use. Trying to use Incus in the s and b and the home lab. Like is there a, is there a GUI to, to be able to do this? You know, because I'm, I'm used to using, like I said, lib Vert, so I use Vert Manager. Mm-hmm. And you know, that's handy.

You can, you can just click, set up a new vm. Is there something like that with Incus or is it all command line driven?

Stéphane: Yeah, so that's incus. So officially, as far as the Inca project is concerned, we don't have an official web ui. We just offer hosting any web UI that you want. And there's like probably a dozen or so on GitHub.

Jonathan: Oh, okay.

Stéphane: Now, in my own, in the packages I produce, as far as my company, I package what's called Incas ui, which is actually a rebranded and now pretty heavily patched Lex D ui. And that gets you like a full a, a full UI that you can play with. You can create instances, you can access like your VGA console on Windows and stuff works just fine through it.

You can do all that kind of stuff through the UI and. Y Yeah, like we, that's pretty easy to install. If it's installed on your system, you can literally run Inca web UI as the command, and then it just opens the web UI in your web browser and you can play with it from there. Mm-hmm. We also have a online demo service for Incas on our website.

So you can go on our website and just try Incas online, which gets you access to effectively a VM running incur that you can play with for 30 minutes. And that's lets you, so there's a walkthrough terminal walkthrough to discover KU Works, but there's also the link to access the web UI for that that test environment.

So you can play with the UI instead if you want to.

Jonathan: Mm-hmm.

Stéphane: I think the, the, the, the main kinda missing piece maybe compared to prom as the UI at this stage is that assembling the cluster itself, that is not something you can easily do from the ui. Yeah, yeah. Once the cluster is assembled, you can, but you currently, you are better off using something like InCast deploy even just like three machines in your home lab.

To get the cluster correctly deployed with oven, with staff. Mm-hmm. All of the security bits and making sure that everything is nice and safe. That's gonna change within s UI because within cus os because Inca OS is fully read only. Mm-hmm. So we can't have Ansible deploy random stuff on it. Right.

Which means we we're moving to a different model for how to run those kind of complex services, likes and oven to instead of running them bare metal on the machine, at which point, if you're running a cluster of like 10 machines, you've got like three special snowflakes that run the controllers, fors and oven.

Mm-hmm. That's never ideal because what happens if like those three go down or you're evacuating them and you forgot that those three are the, the critical ones, which happened to me before. Yes. Well then you've got a bit of a problem. You're like, oh crap, I couldn't, I couldn't take down those two machines 'cause now I don't have a chrome on safe or another anymore, and now my storage on network is down.

Yeah.

So we're actually changing that with the work on InCast Os to run those kind of API like storage and, and network APIs as containers on top of Incus cluster itself so that they, they can then move around if we evacuate the machine or if it goes down or something, it'll just be spawned back on any of the others.

Keeping thing a HA obviously we need to be careful of the chicken and egg problem. Mm-hmm. Which is we can't have the SEF control plane be stored in an instance that's stored on sef, obviously. And same thing for, for the oven control plane, you can't do that on a virtual network. You need to have, so we're effectively making it so that you, we will have what we call a project internally.

So that's kind of how we do our multi-tenancy. Mm-hmm. There will be like an internal project that's used to, to run those kind of workloads. And that one is gonna be on local storage local network so that it. It doesn't have any of that kind of complexity and, and problems with like the chicken and egg issue and that that will fix that.

It'll also let us make it easy to deploy additional services because Incur can do open ID connect authentication can use open FGA for fine grain authorization. It can do all of the metrics stuff with Grafana, Prometheus, and Loki. And currently it's gonna up to the user to have deployed that somewhere if they want to use it.

And then configure Incas. What we wanna do is that with ware, it should be as easy as just like enabling some features and then we will spawn those containers for yeah, for Grafana, Prometheus, low key open, FGA key lock, all that kind of stuff for you, and integrate with it so that it just works. So the idea is that like you within os, once we're basically at the end of our vision you should be able to put it on a machine.

It'll give you IP address, you go to the web UI to the IP address. You can then either just configure in, configure it for standalone use or as part of a cluster. Mm-hmm. And then you can enable the bits you want. So if you want oven and surf shock, enable that, you get the control plane for that being deployed.

If you want to have all of the metrics gathering and everything, you can turn on the observability stack and that's gonna get deployed on top of that Inca cluster two. And then same thing for authentication authorization. If you want to do, have like fine grade control over the cluster, you can do it that way too.

Yeah. So that's basically the goal, which should at that point be very appealing, especially for a crowd that's more used to Yeah. VMware or pox or something. Mm-hmm. Because that will be like literally you'll be able to, to run in custom from your cell phone if you want to. Right. Like you, you start the machine, you just need the IP address.

And then from there on in, you can do everything through, through the UI if you want to. While we also make sure that everything is still easy to do 3D API and the CLI. 'cause every time we do more things in the ui, we always have a crowd that's like, please don't I want my CLI. It's like, yes, we know we always do the CLI first.

Yes. Because that's how we do testing. And then we UI afterwards. Anything the UI can do, you can already do it to the CLI. Maybe you just haven't found it yet, but you believe me, you can do it to the CI.

Jonathan: Yeah. Yeah. Is, is Incus os going to be fully open source and freely available or Yeah. You guys thinking about a freemium model?

Stéphane: Nope. Completely open source. So Incus OS is on, is on GitHub today. I've been mostly developing it through YouTube live streams. Actually. I, I tend to do live streams on, on Thursday. I've been, I've been doing a bunch of development on that that way. Mm-hmm. With the other company I've got that focuses to Fortune 500.

We will effectively have a rebranded version of Inca Os that's called Hypervisor os. Mm-hmm. And the main reason for the rebranding is simply we want a build with a different secure boot key that the customer can trust, which is then just that build. And they can't be tricked into loading any of the random build from GitHub.

But other than the key and the name that shows up on the board screen being different, that's basically gonna be it. The idea simply for that company being, if you're using hypervisor os and the other components coming from that company, we'll know that you will have paid the relevant licenses so that you get access to support.

And that's, that's effectively the model on that front. But Sure. All of the soft, like even for people who do pay for the future version cloud paid offering thing, they get the exact same open, the exact same bits as if you just consuming it from the open source side. The main difference being that you're using the build.

We built, we tested, and we are supporting. Yeah. Awesome. And like, and the reality is that, like, looking at the market right now and looking at how things work, there's really no need to go through the open core model or any of that kind of stuff. It's, it's, the open core model is extremely good at pissing off your community because they never know whether, whether a feature is gonna remain open source or it's gonna go on the, on the page one.

Contributors are also never really tempted to contribute because they don't know if the echo isn't gonna get just moved into the, the other version, you know. It, it, I just don't see it. And also kind of selfishly from like an open source, maintaining now point of view, it's painful enough to maintain one open source project.

I don't want to have to deal with the, the open source and the paid version at the same time. Yeah. Like, that's, that's just, that just seems like a lot of work.

Rob: So Anca os won't have a subscription nag that holds back updates like proxim Max? Nope.

Jonathan: Oh, that's,

Stéphane: yeah. And, and the, and, and the, and the way it's all built currently, it's literally built through GitHub action so we can, I mean, it's using like a, a local runner because we need something much larger. So I've, I've effectively have integration to use large Inca VMs on some of my clusters as cub runners.

And that's what we use to, to, to run to build and test all that kind of stuff. Mm-hmm. But yeah, like it, this will literally be able to put a schedule so that every day a build just happens. And then you put it or not, depending on if you want it or not, like the, the daily bills, you might want that.

The stable update channel will be slightly well updated, slightly less often. That being said, we're gonna have to follow at least the channel, release it as far as security updates and stuff. So you can probably expect a weekly update, basically. Mm-hmm. Now whether you want to install it or not, that's kind up to you.

I think a lot of folks update more like once a month, but these days just like a new point release of the channel coming out every week and it always has a bunch of CVS listed, so it basically needs to, to spin up a new image every week.

Jonathan: If we had more time, I would talk to you about the kernel CVEs because that is a fascinating subject in and of itself, but we are, we are over time already.

So I wanna get in a couple of last, last questions for you. And the first is, is there anything that we didn't talk about that you wanted to make sure and let folks know about? And I know we've covered a lot of things.

Stéphane: I mean, may, I might just go over briefly what Inca actually does because we didn't touch too, too much on that, actually.

That's true, that's true. So yeah. Inca is effectively a private cloud platform. Hs u run CONT system containers. So that's full Linux, TROs, HSU Run virtual machines industry run OCI containers, application containers. We've got an image server with prebuilt images for just about every single Linux distro.

You can think of most of who, most of which we have available as both VMs and container images. Some just as container images, like if you want to run open the value rt. Currently it's container only. Mm-hmm. There are few other that like that, but that means that unlike if you are used to ver you don't need to attach an ISO and go through the install process.

Us. Like you can literally say, I want 2 24 for vm, and we'll just pull a print image and start it for you, and you're gonna have it up and running in less than 10 seconds. So that's, that's a lot more cloudy in the feeling again, where like, it's a lot more what you get on the public cloud. Mm-hmm. As mentioned earlier, like we do clustering we've got a lot of storage features anywhere from like attaching extra discs, file systems, sharing file systems object storage, all that kind of stuff.

On the network front, we do anything from. Physical networking with VLANs, BGP. We've got a builtin D server we've got, and then we go onto the software defined stuff with oven. And we support all kind of pass throughs. So like PCI pass through GPUs GPU slicing, all of that stuff is all all supported.

So even if you want to do like AI stuff you can do AI stuff.

Jonathan: There you go. Fill that box in on your on your Bingo card. We talked about it.

Stéphane: Yep. There you go. Don't really care much about that, but a lot of people do. So lot people. Yeah. You can do it right now.

Jonathan: Yes. Alright. You mentioned your YouTube channel.

Are you the Zli on YouTube?

Stéphane: Yeah. Yes, that's me. So my, the name of my company private. My, my self-employed company is Zli. So yeah, that's the, that's the account on YouTube. Well, I usually do at least a video for every in release. Then I usually live stream the, the other weeks on mostly Inca West lately.

A lot of them are in case West related at this point. Mm-hmm. And do some occasional other things. So one thing maybe worth mentioning is that starting with the next Trona release that's coming up later this month. Mm-hmm. Inca will be part of true now and will be have virtual machines are being run on.

True now. Very cool. So I've, I've published a video just kind of talking at that, seeing how that works. So that's gonna bring us a bunch more new Incas users too, because a NAS is often the only server that, that people have Yep. At home or in their business. So having the ability to run virtual machines directly on that using InCast would be pretty neat.

Jonathan: Yeah. Very cool. Alright, so I'm required to ask you two final questions before we let you go. And that is, what's your favorite text editor in scripting language?

Stéphane: But text editor is vi and scripting language, I don't know. It, it, it moves a lot, I guess these days. It's probably good old fashioned shell and not bash like actual Shell.

Actual shell, yeah. So like yeah, deposit type. But I also used to do a lot of Python before, so, but I've not done much Python lately. I, image Server Logic and whatnot is written in Python, but it's mostly maintenance for the past year or so. So I think I've been doing more. More shared scripting than I've done anything else.

Jonathan: Awesome. Awesome. Alright, Stefan, thank you so much for being here. There was absolutely a blast to hear about Incas and LXC and the history with LXD. We'll, we'll have you back here after a while and talk about how the, the Incas Os release is going. But for today, man, thank you so much.

Stéphane: Yeah, thanks.

Looking forward to, to coming back and thanks for this.

Jonathan: Yes. All right, Rob, what do you think? Have we convinced you to get rid of that old prox Max install and go over to Incas?

Rob: I, I'm definitely interested to try it out. You know, obviously I've been falling Lex d and Incas a little bit. For quite some time now, and I've, I've known what they were and what they did, but hearing this talk definitely gave me a lot more insight and desire to really want to at least throw it on one machine and check it out and give it a good comparison.

Rundown.

Jonathan: You could probably install Incus inside of Prox Mox as a VM and play with it that way. I really

Rob: cross crossed my mind, but I, I, I would need to do nesting too, I guess if I really wanted to get the. The full test.

Jonathan: Yes, yes. But that's the thing most hardware can do these days. Goodness. Yeah. Which is kind of nuts.

But yes, they made it so that actually works. Alright. We do not yet have a guest for next week, so if anyone listening has an open source project and wants to be on, let us know. We've got a couple of openings coming up. And then we are eventually talking with Kuai, who is one of those lytic security projects.

Hopefully they're one of the good guys and not one of the ones that Stefan was talking about earlier. But that'll be super interesting to hear what they are all about. Rob, do you have anything you wanna plug before we let folks go?

Rob: For those who want to know more about me, you can come to my website, Robert p camp be.com, and on there you can find links to my LinkedIn blue Sky.

Yeah, all those social media sites and you can come connect with me there if you want.

Jonathan: Yep. Awesome. Alright, I appreciate it. Thank you man for being here. If you wanna follow my stuff, of course over on Hackaday, it's where the Home of Floss Weekly is and we appreciate that. It's also where you can find my security column talking about all those Linux CVEs and CVEs all over the place that goes live every Friday morning.

And then Rob and I do have the Untitled Linux Show with a couple of other guys over at twit tv. And you can get to that, it's twit tv slash uls and we'd love to have you there as well. We appreciate everybody that's here, those that get us both live and on the download. We will see you next week on Floss Weekly.

Episode 828: Incus Inception

9 april 2025 | 80 min

Episode 827 transcript

2 april 2025 | min

FLOSS-827

Jonathan: Hey folks. This week I talk with Bash only about Y-T-D-L-P. That is the cutting edge, YouTube and every other video service imaginable downloader. It's a lot of fun. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 827 recorded April 1st, Y-T-D-L-P. Sometimes you can't see the tail.

Hey folks, it's time for Floss Weekly Lead. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something a little different. Today we're talking with Bash only. Yes. First name Bash, last name only about Y-T-D-L-P. That is. One of the YouTube downloaders.

And as far as I could tell, it's sort of the premier YouTube downloader these days since that old project YouTube DL sort of went away or went on hiatus. There's a lot of history here but we've got the right person to talk with about it. So I'm gonna go ahead and bring on Mr. Bash. Mr. Only Bash only.

Welcome to the show.

Bashonly: Hello. Thanks for having me.

Jonathan: Hey, it is, it is great to have you here. I am. I'm excited. I know we've got at least one listener in the chat room that's excited to talk about Y-T-D-L-P because it's a tool that so many of us find ourselves using a lot for. You know, the things I've been using it for recently is to download the HDR version of certain videos to be able to play it in MPV to do HDR testing.

And you know, there's a whole bunch of those different like oddball use cases that you wouldn't ever think of that people end up using. Tools like this, this one in particular for that's gotta be kind of surprising, isn't it? Do you, do you have kind of a, a line to the, the way people use it? You get told stories?

Bashonly: You know, I came into the project in 2022 mm-hmm. And it was forked from YouTube dl, which we can get into later. Mm-hmm. But you know, it's, it's been around for over a decade, you know, in one form or another. And it just, it supports so many sites. That every week there's, you know, some new issue reported of a site that I didn't even know we supported.

And then I find out like thousands of people used Y-T-D-L-P just for that site.

Jonathan: Mm-hmm.

Bashonly: And, you know, I had never even heard of it before, so it really is sort of like the, the Swiss Army knife of downloading.

Jonathan: Yes. Which is interesting because I think it's, it's primarily based on ffm peg, which is the, the Swiss Army knife of anything related with media.

Right.

Bashonly: It uses FFM, peg it, it's, it's one of the libraries.

Jonathan: It depends. Right, right, right. That's, that's fair. Right. Okay, so let's, let's start with the elephant in the room then. And I got, I got permission beforehand to ask this. What is the, what is kind of the, the primary reason that you are Mr.

Bash only today and only a a logo and not going by, you know, your real name on the internet and putting your, your picture out there like some of us have chosen to What's what, what's behind that decision? I.

Bashonly: I mean, who says Bash only isn't my legal name?

Jonathan: It's on the, here's your, here's my social security card.

It's on the driver's license. Miss, that's Mr. Only to you.

Bashonly: Yeah, exactly. I was thinking about, you know, PowerShell only, but it didn't really have the same ring to it. Oh. But it's, and I also, I don't think I would agree with that name, but yeah, I just don't really want my name out there for reasons.

Mm-hmm. Like if you look at what happened to YouTube DL about five years ago there are people that want to take this project down. Mm-hmm. And I just don't know what angle they're gonna. Come at that from, right. So just to protect myself, I just, you know, rather stay semi-anonymous.

Jonathan: Mm-hmm. Yeah, that makes sense.

It, it's hard, it's hard to be served with paperwork if your real name and your address isn't out there anywhere. Yeah,

Bashonly: exactly.

Jonathan: So let's, let's talk about kind of the, I guess the origin story of Y-T-D-L-P. Honestly, I, I did not know for sure that it was a fork of YouTube dl, but like, what's the, what's the history there?

What was the, kind of the original problem that somebody wanted to solve and, and the, the original project got started.

Bashonly: I think the original intent with it was.

To sort of create a digital VCR, and by that I, I don't mean, you know, like DVR mm-hmm. You know, A DVR is owned by, you know, your service provider. You don't actually own the content. You know, it's encrypted.

Jonathan: Mm-hmm.

Bashonly: A digital VCR is, you know, you see a video on the internet, you can record it, you can watch it offline, you can, you know, use it for.

For personal use and I think with video streaming becoming more and more complex, that had be, become impossible without a tool like YouTube dl.

Jonathan: Mm-hmm. That's a, that's an interesting analogy too. This is something I was thinking about in, in preparing for the show. You know, there's gotta be legal questions around doing this.

You know, can you download videos? Let's just take YouTube. Like the, the question has to be, can you legally download videos from YouTube? And I think the, the VHS player is a really good analogy because here in the United States, we had a Supreme Court case about this. I mean, Mr. Rogers went and testified about this, right?

That it makes sense to be able to, and the term they use is time shifting, right? That's, that's the official term for what we do here. We time shift things. I. You, you, you take it and you make a local copy of it so you can watch it at another time. You can watch it repeatedly if you want to. And that's that's pretty apt for what, at least, you know, as far as I is what I do with it with, with Y-T-D-L-P and, and previously with YouTube dl.

You just, you get a, you get a local copy of it so you can watch it whenever you want to. But that's sort of been disputed over the years, hasn't it? By, by maybe Google to some extent, but other people like the RIAA, not, not everybody shares that same legal theory. Do they?

Bashonly: Well, the RIAA, they've, they have their own view on this issue and.

You know, GitHub apparently disagrees with the RIA. Right. 'cause they rein, they reinstated YouTube dl. I think I'm, I'm trying to be careful with my words here. Sure, sure.

Jonathan: So we, we are not lawyers, right? Neither of us have legal, as far as I know. Besh only may have a legal degree and just isn't telling anybody. I certainly do not. So this is not legal advice. This is us being internet lawyers based on what we can read, right?

Bashonly: So essentially the way I view it is. You know, your web browser visits a website.

Mm-hmm. It streams the video. You are stream the content. Mm-hmm. You receive the data of the video. It's just not viewable outside of your web browser. Right. What tools like YouTube, DL and Y-T-D-L-P do is they act essentially like a web browser. Mm-hmm. The way that, that they. Interact with the websites is they basically recreate everything the browser does, but then it saves it to your hard disk instead of, you know, just an ural stream.

Mm-hmm. So I don't think, you know what the RIA said in the the DMCA take down notice for YouTube dl. I don't think that applies here. Because what YouTube DL was doing really wasn't anything different than what your web browser was doing.

Jonathan: Right, right. And, and so there's a, and I think this was one of the things that, like the RIAA got totally wrong.

They talked about DRM and when I looked into it at the time, it was the case that YouTube dl and I'm, I'm assuming this is the case with Y-T-T-L-P as well, explicitly and intentionally does not break. DRM isn't, isn't that, isn't that accurate? Correct. Yeah. Yep. Yeah. So

Bashonly: yeah, if a, if a website only serves DRM content, then Y-T-D-L-P doesn't support it.

Yeah. And you know, if the, if there are DRM protected formats offered by a website, Y-T-D-L-P does not download those.

Jonathan: Mm-hmm.

Bashonly: So we deliberately avoid anything DRM protected. Because, because that would be unquestionably illegal. Yeah.

Jonathan: Yeah. Does YouTube still recommend that like Y-T-T-L-P for archiving?

That was, that was the funny thing at the time when it got taken down, there was still like active YouTube documentation saying, you know, if you need to download a copy of your video, here's this great command line tool that you can use.

Bashonly: I actually wasn't aware of that.

Jonathan: Yeah, that was, that was part of the document, part of the, part of the argument, the kind, like the counter argument people made about YouTube dl and it's like, look, Google, here's your own documentation where you tell people that this is the tool to use to do downloads.

It was, it was wild. It was a wild time. I'll have to,

Bashonly: I'll have to look into that. I wasn't aware of that.

Jonathan: Yeah. I'll see if I can, I'll see if I can go back and find some of those sources. Okay, so lots of, lots of directions I could potentially go here, but I'm real curious about, about the fork. So how did we get from YouTube DL and it got the take down notice and then eventually got reinstated.

To now it got forked to Y-T-D-L-P and as far as I can tell, like if you go to install things in say Fedora, Y-T-D-L-P is the one that's available. Not, not YouTube, dl. And in doing the prep for the show, I was kind of surprised when I went back and saw that YouTube DL does actually have a little bit of activity back in, its in its distro or in its GitHub again.

But what, what led to the fork and what led to where we're at now?

Bashonly: Sure. So, well, YouTube DL started I think sometime in the early 2010s. I think it might have been 2010 even. And it started out as just YouTube as a simple downloading tool. Mm-hmm. And then, you know, the scope started expanding. It supported more sites and more and more features were added.

Mm-hmm. More maintainers were added on, and I think the original maintainers sort of. Bit off more than they could, they could chew maybe the, you know, the, there was a lot of churn through the maintainers, right. And and. The maintainers before the take down. I think they were understandably overwhelmed by the sheer volume of issues and pull requests mm-hmm.

That they had on their tracker. You know, even I'm overwhelmed with what Y-T-D-L-P has and that's a, a fraction of what YouTube DL had at the time. Mm-hmm. And, they just, they weren't merging pull requests, you know, they, they had standards, which was good. Yeah. They had high standards for code quality and coding conventions and whatever.

Mm-hmm. But stuff wasn't getting merged. So then the first significant fork from YouTube B was called YouTube, DLC. And that happened, that happened I think a few months before the take down. And basically the idea was it'll be YouTube dl, except we'll merge the poll requests that should be merged mm-hmm.

That aren't getting merged. And then the takedown happened a few months after that. I think it was October, November, 2020. Sounds about right. And so when YouTube DL was taken down, all of its forks were also taken down. Anything that was marked as forked from YouTube dl mm-hmm. Was also got taken down.

So then YouTube, DLC, they started new repository called Y-T-D-L-C, which was. To get around the mm-hmm. The, the forking problem. Yes. And then I think the maintainer of Y-T-D-L-C eventually got burned out, or there was, you know, other life priorities there. Sure. And so then, this is actually the first time I've had to say his name out loud, so I'm not, I'm not sure I'm gonna pronounce it right here, but Pandan.

Mm-hmm. P Pandan he, you know, the torture of pass to him and he started Y-T-D-L-P, which the P in Y-T-D-L-P is his name. His name, yeah. So it's you, you, yeah. That's basically how we got here, how, where we are today.

Jonathan: Mm-hmm. Yeah. It's a, so YouTube go. No, go ahead. Go ahead.

Bashonly: YouTube, DL still does exist.

So after, after the reinstatement development had still pretty much frozen, the maintainers weren't really doing anything and it's, it's understandable. They were already overwhelmed and then they got hit with this, you know, legal nonsense. Yeah. And sort of, sort of sours

Jonathan: you wanting to work on the project anymore.

Bashonly: Yeah. Who wouldn't be burned out at that point, you know? So they basically removed themselves from the project and they, the reins were handed over to a new maintainer. And he's pretty much just been going it alone ever since. Mm. So development is slow over at YouTube dl. But YouTube downloading is still getting fixed, updated.

You know it if you use the. The master branch of U2 BL, it still works. Mm-hmm. There just hasn't been a proper release since I think 2021. Ah, so that's why it's no longer in package repositories. And, you know, the, the version you download with PIP doesn't work. But if you run from source or you, you know, you build it yourself.

Mm-hmm. It, it still works.

Jonathan: Yeah. Interesting stuff. So what, what, what is it like, are you the, are you the sole maintainer of Y-T-D-L-P or do you have a a, a group of people that work with you on it?

Bashonly: Oh, no, I'm not the sole maintainer. I would lose my mind if I was the sole maintainer.

Jonathan: That's, that's good. Yeah.

Bashonly: Yeah. Yeah, there's a team of us, there's about five or six of us, depending on who's active at any given moment. Mm-hmm. And it's sort of a, a democracy. There's not really a leader right now. Okay. We all just sort of, you know, go by consensus and mm-hmm. You know, work together.

Jonathan: Yeah. What's it, what's it like being part of that team?

Like how, how crazy is it? How many, how many sites do you support and how crazy is it to try to stay on top of it?

Bashonly: To be honest with you, I don't even know how many sites we support. I know we have over a thousand modules for extractors. Oh my. But then each of those modules can support more than one site.

And then Y-T-D-L-P also has what's called a generic extractor, which, can be used on any site we don't support. And it looks for generic embedding techniques in the webpage to see if there's any video or audio that can be extracted. Mm-hmm. So it's thousands of sites and I, oh my goodness, I couldn't tell you the exact number.

Jonathan: It's just, it's just drinking from the fire hose all the time then, isn't it?

Bashonly: Yeah. Yeah. I don't know if you've looked at our issue tracker, but I think we're up to 1.5 K.

Jonathan: Mm-hmm.

Bashonly: And most of those are, you know, the site's broken. Can you support this site? And yeah. It's, it's fun.

Jonathan: Yeah. Yeah. Do, do you get any support from anywhere?

Do you, like, does anybody have a Patreon account or does. Does YouTube give you any money to make this work?

Bashonly: Oh yeah. YouTube funds those all the way.

Jonathan: Got those big YouTube bucks. Google Bucks.

Bashonly: Yeah. I don't even work. YouTube just, you know, funds my lifestyle.

Oh no. We have a. We have a donations page on our GitHub that you can check out. Mm-hmm. I don't accept donations just 'cause I don't, you know, again, I, I don't wanna de anonymize myself or Right. Yeah. I wanna avoid that as much as possible.

Jonathan: I was just thinking about that there might be some legal ramifications for taking payment for the work.

Bashonly: Exactly. Yeah.

Jonathan: Mm-hmm. Yeah. So why. To get into a little bit more technical question, why is it, why is this so hard? Like, okay, so I remember years and years ago where back when everything was Flash video somebody came up with this, I don't remember if it was a browser plugin or it ran on Windows. I'm like, this was, this was like 20 years ago at this point.

Somebody had this little tool that would just, it would watch your network connection for incoming flash shock wave video files, and when it saw one, it would just download it right off the wire. We've come, we've come a very long way since then. I know. But why, why is this such a hard problem? Why isn't there just, why isn't there just one extension that works on all of the sites?

Bashonly: Well, that'd be nice, wouldn't it? It would be yeah. You know, back then, you know, I think that was probably around like. 2007, 2010, somewhere around that. Yeah. Time range. Streaming video was mostly just like an FLE file or an MP four file mm-hmm. Or something that you could just, you know, find the URL and download it.

And so for a browser extension to do that, it was simple. And then, you know, then there was HLS dash and. You know, YouTube is experimenting with its own new protocol. What fun. And now you, you know, you can't just download a file anymore. You need FFM peg, you need, you know, something that can stitch together the segments.

Something that can, you know, even find the URLs of the segments now, you know, parse the dash manifest and whatnot.

Jonathan: Mm-hmm. The is, is there quite a bit of churn? Like, does YouTube constantly change stuff to where you guys have to go in and, and ref fiddle with whatever? I mean, I don't, I don't even know, honestly, like, what is the, what is the process for downloading a YouTube video?

You, you've mentioned that you kind of have to act like a browser. So do you go and grab some huge obfuscated JavaScript file and try to de obfuscate it and, and run through it and like what, what does that process even look like?

Bashonly: It's. Similar to what you described. So to answer the first question about breakage and churn it depends I guess, on how upset Google is about.

People downloading and bypassing their ads at any given moment? Like I would say about two years ago, YouTube downing was, was pretty stable. Mm-hmm. It was only every few months something broke and we'd have to fix it. I think for about the past year now, it's been a weekly struggle. Something has been breaking.

At least every two weeks, if not every few days.

Jonathan: Hmm.

Bashonly: We have done about, I think, five releases just in March, just to fix YouTube support. So then to get into this second part of your question about how YouTube extraction works there's a lot of steps to it. And it does involve JavaScript, which has been the problem this month.

Jonathan: One, one might argue that JavaScript is always the problem, but that aside.

Bashonly: Yeah. Yeah. It's sort of me and the maintainers, some of the maintainers have a, a phrase called JavaScript hell, which is what we're in when this happens. Yeah. So there's a. JavaScript player file that, that basically, you know, that runs the whole operation when you're watching a video on YouTube.

Mm-hmm. And within that file, there's there's some functions used to descramble, a query parameter that you need in order to. Actually access the video stream without it, you'll get a 4 0 3 error. Or, you know, in the past you would get throttled so that it was basically unwatchable or downloadable.

Jonathan: I'm familiar, I'm familiar with both of those problems.

Bashonly: I, yeah, yeah. But now then I think about. Six to eight months ago, they changed it. So now you just get a 4 0 3. You can't watch or download anything unless you scramble this query parameter properly. Mm-hmm. And for a while, the, the player JavaScript was stable and then about a year ago they started changing it.

Very frequently and they would try new obfuscation techniques. They would, you know, add, recently they added an array that has all the strings in it. So you can't find the function by searching for the strings. 'cause YT DLP is written in Python. We don't use JavaScript. So, you know, we don't have access to an A ST or anything like that.

We just, I mean we, we essentially first parse it with. RegX. Okay. So we find the function names like that, and then we have our own little JavaScript interpreter module you've built, which is

Jonathan: you built a JavaScript parser in Python.

Bashonly: Yeah. Well, it, it's very primitive, right? It just in, it just interprets what it needs to interpret.

Sure. And it returns a python function. Mm-hmm. So that's been breaking all the time, but

yeah, well there's, there's two parameters like that, that need to be decoded. Mm-hmm. Anyways I mean, does that answer your question? Did you want me to get into more technical details?

Jonathan: No, I, I think so. I think that pretty much gets us into the into the ballpark. Yes. I, I, I remember when everything was throttle to be very slow and so, you know, if you had a, if you had a video you wanted to test something with, you would start it and then go to bed for the night and hope that it didn't time out and you got it by the morning time.

It was very, it was very reminiscent of using the internet 25, 30 years ago. Right, right. Yeah. Oh, fun times. There are some other projects one in particular that I know of that's kind of in this same space is a smart tube. I. And that's a, a replacement YouTube player for like Android tv. Is that, is that something you're aware of and is there any cross pollination?

You guys are obviously solving some of the same problems.

Bashonly: I don't know much about Smart Tube off the top of my head. Okay. I don't keep up with many of the other projects. There's a few of them that I do keep up with because they help us out. Mm-hmm. Quite a bit. There's YouTube js, which is sort of like the reference implementation for Yeah.

All things having to do with the YouTube API. Right. And I'm very aware of that and, you know, I, I watch their changes and everything. But I'm not, I'm not that familiar with Smart Tube.

Jonathan: One of our live listeners mashed potato points out that you, you may have similar problems to New Pipe as well.

And that's another, that's another app. That's an Android app particularly that I'm, I'm aware of. I never got it to work for me, but that's fine.

Bashonly: Yeah. Yeah. New, new Pipe. I'm aware of. New Pipe. And they help us out too. That's a good project. I, I believe they, it's an Android. Project? Yes,

Jonathan: I think so. I shall have to, I shall have to try this one again.

Last time I tried it, it just did not work for me at all. But

Bashonly: yeah, I mean, they're having the similar kinds of problems that we're having, right. They don't really have to deal with the JavaScript part of it that much because it's the Android side of things. Mm-hmm. But the Android side of things has its own set of problems, so

Jonathan: Of course.

Bashonly: Yeah.

Jonathan: In notably. Okay. So surely, surely, your, your career path, your education path, you, you studied software development or computer science, right? And that's what your day job is. That's how you're so good at this.

Bashonly: No. Nope. My, I did not go to school for computer science, software engineering, any of that.

My my real job has nothing to do with this. I'm just a pure hobby developer and pretty much self-taught.

Jonathan: It amazes it. I am, I am basically in that same boat. I didn't even finish college. It amazes me how many people, particularly in the open source world, although. In corporate world in some cases too, sort of took that same path.

It's like we didn't go to college for it. We found it, you know, maybe sometimes in high school, some of us earlier than that, some of us later than that, but just, it's like, oh, this is, this is the thing that I wanna do. Maybe not for a job, but I, I won't, I won't ever really be able to get away from this like, writing, writing code for some project somewhere.

It's just, it's just part of who we are. I find that real fascinating that that's such a, a common story.

Bashonly: Yeah. You know, when I was a kid, I was a huge nerd. I would I would write my own webpages in Notepad, you know, write h tm l by hand. Mm-hmm. JavaScript, and I, you know, I, I, I started out writing batch files.

Mm-hmm. And yep. And I didn't really keep up with that that much. You know, I, I wasn't like an early adopter or Linux or anything like that. And then, you know, I had, you know, sort of a, a busy time in my twenties and then life slowed down after that. Mm-hmm. And I had more time on my hands and I thought.

You know, why don't I, I get back into that. 'cause I, I thought I was pretty good at, you know, programming and Yeah. Doing that sort of thing. And then Covid happened and then I had a whole bunch, I had a whole bunch of time on my hands.

Jonathan: Yes.

Bashonly: So I started teaching myself. You know, I, I started with Bash 'cause I had started using Linux a few years before that.

And I thought. You know, there's so many things that I, I just wish I could just press a button and it would happen.

Jonathan: Mm-hmm.

Bashonly: So I started writing BA scripts for everything that I need to do. And then eventually I taught myself Python. 'cause I thought, you know, that's the logical next step here. Yeah. And I, before I got into Python, I was.

I sort of wrote my own YouTube DL in Bash. Hmm. 'cause I, I wanted to, I wanted to just, I, I wrote like a, a live stream, not downloader, but something that would pass the live stream to MPV. Mm-hmm. So this was back before YouTube became. As difficult as it was today. I don't think I could write this in Bash today, but Right.

And then I thought, you know, this is something I'm good at. And, you know, I was aware of YouTube DL and then Y-T-D-L-P wrote rolled along and I, I figured, why don't I just contribute to this instead of wasting my time on my silly little strips? So I opened a poll request and. I just kept going with it and that's how I ended up where I am today.

Jonathan: That's how they get you that first pull request. Yeah, the first one's free and then you're hooked. Yep. Oh, I'm involved in multiple projects with the, with exactly the same thing. Oh, there's this one thing that annoys me about this. I'm gonna go fix it. And the next thing you know, you've written a quarter of their code base and.

Yeah. That's just how it works. Yep. I must admit, I do still maintain a little bit of HTML by hand. I use Nano, not notepad, but that, that part of your story really spoke to me. I still have one website that I go in and do by hand.

Bashonly: I mean, there's something to be said for that. Sure. So, yeah. Not Notepad.

I don't use Notepad anymore.

Jonathan: Not notepad. No, no. HTML by hand. Yeah, for sure. For sure. Okay, so what what does the, what's the future look like for Y-T-D-L-P? Do you guys have any long-term plans? Are you going to get rid of some of the plugin, some of the modules to be able to make it more maintainable?

Or is it, is it just gas to the floor all the way out?

Bashonly: So probably the biggest undertaking that we're planning right now is to separate the extractor modules from the core project. Mm-hmm. We eventually wanna move all of the extractors, you know, that, that deal with the web, the specific websites. We wanna move those into their own repository or repositories, and then have the the core in its own repo, and it would basically treat every extractor as if it is a plugin.

Mm-hmm. You know, you can plug them in, unplug them. And that would sort of, you know, first it would, it would shield the project. You know, if, if any one site causes trouble for us, then that extractor repository would get, you know, taken down or, yeah. Had to deal with the repercussions of that or the core project would be fine.

That's still a ways out I think, because

Jonathan: that kind of a code reorganization is not true real.

Bashonly: Yeah, I mean, we're dealing with a code base that goes back like about 15 years. Yes. And compatibility has sort of been our enemy lately. Backwards compatibility.

Jonathan: Yeah.

Bashonly: It, it, it's sort of the, the enemy of new ideas and it's sort of been holding us back.

So it's one of the struggles we face is sort of balancing backwards compatibility with. Implementing these new ideas. Yeah. You know, we have to find a way.

Jonathan: What, what isn't, sorry, go ahead. What, what wouldn't be backwards compatible? So like, are you talking like internal API changes that break some of your extractors?

Bashonly: Well it on the, the user side of things, so. We tend to, you know, never remove command line options. Ah, okay. Okay. They're just deprecated and, you know,

Jonathan: forever deprecated. Yes.

Bashonly: Yeah. Yeah. And just I think the way that you would have to operate Y-T-D-L-P would be slightly different because it wouldn't necessarily come with all sites supported outta the box like it does now.

Jonathan: Mm-hmm.

Bashonly: You would have to, you know, pick and choose which extractor repositories you wanna add. I don't want to get into too much, too many specifics here 'cause That's fine. We haven't really hammered down the details Yeah. Of how it's gonna work ourselves. So I don't, I don't wanna give anyone the wrong idea, but

Jonathan: Sure, sure.

That makes sense. So I have, I have noticed something when you go to Google and you type in YouTube downloader or, well, yeah, let's, we'll just go with that. That's what most people are gonna search for. You have to go quite a few pages before you actually get to something like Y-T-T-L-P and you first find a whole bunch of websites.

That will download YouTube videos for you and very likely run the Y-T-D-L-P source in the background on their server. I assume you've noticed that.

Bashonly: Yeah. That's interesting, isn't it?

Jonathan: That's interesting. That's, that's gotta kind of annoy you guys.

Bashonly: It, it makes me wonder why, I mean, I think. Y-T-D-L-P being deprioritized in the search results has gotta be intentional. Unless it's just some, some sort of, you know, there these other sites are somehow gaming the, the search engine optimization, what have you, but you just think you search YouTube downloader, at least YouTube DL would be the top result.

If not Y-T-D-L-P. Yeah.

So I think we've been de boosted most likely, but it just makes you wonder why would they want these other sites,

Jonathan: some of which are certain malware, some of which are certain malware and adware and all kinds of garbage on them.

Bashonly: And I mean, yeah, it just makes you wonder. Just makes you wonder. And yeah, most of those sites, they are either.

You know, powered by YouTube dl or, you know, they're not,

Jonathan: I don't know, third page. You're on the third page. Which third page?

Bashonly: Wow.

Jonathan: I mean, it could be worse, but Yeah. On this site, on, on this particular search for YouTube downloader, the, the GitHub eight is on the third page. And above you is a whole bunch of websites that I would prefer not to mess with.

I mean, I'm sure some of 'em are legitimate and, and aren't going to try to eat your babies, but I'm sure some of them are serving malware if they can.

Bashonly: I know a, a big one right now is Cobalt. And I think they're, they're fairly legit. I don't know if they show up in the search results or if they're.

Yeah. Also down on the third page of this also

Jonathan: I, I think some of it is just that the vast majority of people don't wanna run a Python script, and so YouTube knows that, or excuse me, Google sort of knows that and is going to direct them to the services.

Bashonly: I just think that they wouldn't want people using those sites at all, though. Well, that's why it's confusing to me is you'd think, you know, yeah, let's send 'em to a GitHub page where they have to figure out how to, how to install this thing and run it from the command line instead of, you know, go to a webpage where they type in the URL and click a button.

Jonathan: Yeah. Yeah. That's fair. I guess I have to wonder. Are, are people at Google and people at YouTube sort of in the back of their mind thinking about like antitrust. And so if they do, if they manipulate the results too much, then you know, at some point they might get hit with some kind of antitrust lawsuit.

I don't know,

Bashonly: maybe, but I, I don't think they're worried about that from Y-T-D-L-P, though.

Jonathan: That's probably true. That's probably way, way down. That's on the third page of their list of concerns.

Bashonly: Yeah. Yeah, though, I mean, I think, I think we are on the radar though.

Jonathan: Oh, I'm sure. I tell you what else you are, you know, I, I what I can guarantee you, you are, you guys are used internally at Google.

I can guarantee you, you guys are used internally at Google. Absolutely.

Bashonly: I wouldn't, I wouldn't doubt that

Jonathan: By, by somebody. Probably multiple people. 'cause it's such a useful tool. Yeah. Which is, which is nuts to think about.

Bashonly: Yeah. I think we're, we're definitely used internally when they write their new versions of their player JavaScript to see if they can, they can break it.

Jonathan: Yeah. That's likely true. So if somebody goes to use Y-T-D-L-P. Whatever they're trying to get to doesn't work. Where should they go? Complain slash send patches? What's the process?

Bashonly: They can always add another issue onto the issue tracker. We're 1500 going strong. We can always use some more. As long as they're willing to submit complete, verbose output.

If they can give us a log with all the details we need, yeah, then feel free to open an issue on the issue tracker. That's, that's all we ask. 'cause just follow our issue template checklist and give us the details we need. If they have more general questions or they don't feel like. You know, if they, if they feel like they can't even get to the point where they have a verbose log to give us, then they can join our Discord and ask us on there.

Jonathan: Hmm. Yeah. We'll have to make sure and get the link to the discord and put it in the show notes. 'cause that'll, that'll be very interesting for folks to follow. How difficult is it to submit a patch and get it picked up?

Bashonly: It's, we accept pull requests from anybody. We do ask that you follow recording conventions and we ask that you understand that, that we have a high volume of pull requests and it might take a while for us to get to them mm-hmm. To review them. But yeah, anyone, if they wanna add support for a site and it's not, you know, it's not a sketchy site, nothing, you know, DRM protected, nothing piracy related, then, you know, we'll, we'll review it and, you know, merge it if it's acceptable.

Jonathan: Yeah. That's interesting. So you, you guys do have kind of a list of sites that you're just not willing to support. Not piracy stuff. Ma mashed potato, mashed potato points out. People might not want to admit that they're downloading furry videos.

Bashonly: Yeah. Oh, well, I mean, you can make a burner account on GitHub and open the, open the poll request.

As you know, anonymous, furry four 20.

Jonathan: Oh, that's you. You probably have that actually. People doing burner accounts just to complain about a site not working. That's probably a real thing.

Bashonly: Oh yeah,

Jonathan: yeah.

Bashonly: Oh yeah. And then they turn into ghost two days later. Mm-hmm. So, you know, the, the account's deleted, never be seen again.

Jonathan: Yes,

Bashonly: exactly. Yep.

Jonathan: Some, someone somewhere is living in shame that they had to do that,

but thus is the internet. This is what we live with. So we talked about what's coming. What do you, what do you see? Like what, what's, what worries you about this project? Right? I know, I know some of the plans you have, but what's your what do you, what do you see coming that's gonna be hard to deal with?

Bashonly: YouTube? Just

Jonathan: YouTube and mostly YouTube. Yeah. Yeah. Are they gonna, there's a, there's a lot of, are they gonna ever roll something out like DRM for all videos that we just sort of put it into it?

Bashonly: Well, they partially did already. It's not something that's been applied to all forms of all videos, but, so there are different clients that can access the YouTube, API, there's, you know, the Android client, the web client, and then there's the TV client or TV HTML five as they call it. Mm-hmm. And that's what Y-T-D-L-P is primarily using right now because it's not subject to some of the restrictions and what have you that the other clients are.

Okay. And if someone is downloading too much, if they're making too many requests with the TVH TM L client, then they will get DRM only formats right now. Or at least,

Jonathan: Hmm.

Bashonly: It's, it's, it's like an AB test. It's not that widespread, but you know, if they. Cool it on the downloading for a few days, everything will go back to normal.

So it's something they're testing out right now. We don't think that it's gonna be applied to any other client other than the TV HTML five client. Mm-hmm. But we don't know, you know, it's not like YouTube lets us know anything.

Jonathan: You don't have the inside line.

Bashonly: Yeah, exactly. But you know. That's sort of like their nuclear option.

If they do that, then YouTube support is gone. Mm-hmm. In Y-T-D-L-P and YouTube dl. I don't think they're gonna do that though, so I'm not that worried about that. I am worried though, that it'll make the TV client useless for us and we'll have to find another way of doing things.

Jonathan: Hmm.

Bashonly: There's something else happening right now called.

Proof of origin tokens, and this is what your web browser generates to prove that it's a real web browser. Hmm. And this is, it's it generated with JavaScript and it goes beyond what we're capable of, capable of doing, and Python. And so if we have to move to the web client, then we would need to add.

An actual,

Jonathan: more like a full blown added, a full-blown JavaScript parser.

Bashonly: Yeah. We, and you know, we're not gonna write that in Python, so we'd need to add a dependency on like Dino. Mm-hmm. So that could be something that happens where if you want to download from YouTube, you also need to have Dino installed.

Mm-hmm. Yeah. Yeah, I can see that. Those are the biggest concerns right now.

Jonathan: Yeah. We've, we've talked a lot about YouTube and rightfully so, right? It's, it's the, as far as I know, it's the most popular video site in the world, but you guys have a bunch of other sort of backends that work. Do, do you have any favorites?

Like is there anything out there that you, you, you really think people should know about? Like, Hey, we can download from crunchy roll so that you can archive anime, or, you know, is there, is there anything like that that, that should be highlighted? I

Bashonly: well, crunch roll, we can't download from anymore. Oh, it went full DRM So that, well, that got removed. That got removed a couple months ago. It

Jonathan: used to work. Interesting, interesting. What I would, interesting example, I pulled outta the hat then.

Bashonly: Well, what I would say is, you know, we get these questions on discord and on the issue tracker all the time of, you know, does it work with this site?

I would say just. Get the URL and plug it in and see if it works. Mm-hmm. And I think more often than not, it will work or it'll start to work and then it'll break and then you can report the bug to us and, you know, it'll get fixed eventually. Right. But yeah, not, not any one particular site. But you know, like I said at the beginning of the show, I'm surprised, you know, at least every month.

It that there's this site that dozens, hundreds, thousands of people use Y-T-D-L-P for that I'd never even heard of before. Mm-hmm. And then it breaks and all these people come outta the woodwork and it's like, wow, you know, I've never even looked at this code before and now I gotta fix it. So yeah, just try, just try it with anything.

Mm-hmm. Is what I'd say.

Jonathan: Yeah. Yeah. Was, did the code base start in like Python two? Was there a, was there a big Python two to Python three transition for this?

Bashonly: Yeah. And that is one of the big differences between Y-T-D-L-P and YouTube. DL is YouTube. DL started in, I don't know if it started before Python three, but they have maintained compatibility going back to Python 2.6 this entire time. So they have all these compatibility modules that. With, you know, compact functions that you know will, will work the same in Python two and Python three and Y-T-D-L-P.

After it was forked it dropped Python two support pretty soon after that. So we don't have to worry about that anymore. Mm-hmm. Thank God. Yeah. But again, that's, that's actually one of the reasons where I decided I'm gonna contribute to Y-T-D-L-P instead of YouTube dl. 'cause I didn't want to have to deal with, you know, maintaining compatibility, going back to 2.6,

Jonathan: the, the mess that is trying to work in both versions of Python.

Yeah. Right, right. That's kind of ugly. Are there, are there any skeletons like that still hidden in the code base?

Bashonly: Probably we've gotten rid a lot, gotten rid of a lot of 'em and we've been sort of keeping on the same timeline as the PSF itself with, you know, if they drop support for. You know, the, the end of life of, you know, Python 3.7, 3.8.

We've dropped support for 3.7, 3.8. Mm-hmm. Right now, I think we only support 3.9 and above. So every time that happens, then we go and we try to, you know. Purge a bunch of old code, right. That we don't need anymore, but I'm sure there's still some lingering around that we haven't found.

Jonathan: Yeah. So I, I'm kind of remiss I should have asked you earlier about this.

We, this is open source. What, what is the license that Y-T-D-L-P is under?

Bashonly: So it's under the unlicensed, which is basically a fancy way of saying public domain. So when we. You know, push code. It's just out there for anyone to use however they want. Which sort of explains the proliferation of all those YouTube download sites that just use Yes, YouTube, DL, my DLP.

'cause I mean, they're free to do that if they want, but you know, I wouldn't use those.

Jonathan: Oh. I'm looking forward to the inevitable fork of this into a source available licensed business clause. Something, something, something. Yeah. Probably, probably not on the roadmap.

Bashonly: Probably not. No. We've, you know, we've thought about relicensing, but there's really not, there's just really not a great solution to that problem, I think.

Yeah. And I, I, I actually kind of like that it's public domain. Sure. I feel like, you know, anyone can use the tower they want. It's sort of, you know, takes responsibility off of me, I feel like, is, you know. Mm-hmm. This code just exists is, is what I say. Yeah. You know, it's not really my code, you know, it's not our code, it's just, it's out there so anyone can use it.

Yeah.

Jonathan: So I, I was speaking of course, very tongue in cheek about the idea of the, the relicensing and the, the point with that is like the worst of these sites that do this, that use your code, the ones that you would really like to stop, those are the ones that don't care what the license is, right? So it would, it would be a, a pointless gesture.

Bashonly: Yeah, exactly. You know, and it's not like they're, they're saying like, oh, we're powered by Y-T-D-L-P, you know, it'd be kind of nice, but no, all these sites are like, look at what we can do, you know? So.

Jonathan: Our proprietary YouTube support plugin.

Yeah, yeah,

yeah. I I've seen it. I know how that goes. Alright.

Hey, so is there anything I didn't ask you about that I should have? I know this is a hard question, just gotta do, set math in your head. Think about all the things we talked about and what your, your list of what you want to is there anything we didn't cover if we didn't get to?

Bashonly: No, I think we, we touched on a lot of stuff.

Jonathan: Okay. You got to plug your discord. You told people where to go to open issues and where to go to submit code. Those are, those are the big things, I think.

Bashonly: Yeah. github.com/yt. DLP slash yt dlp, as long as you can remember those letters, it's pretty easy. Yep. We got the link to the discord on our GitHub.

Pretty much, you know, that that's where you can download from. That's where you can get the source code from. That's our, that's our hub.

Jonathan: Yeah. Is it fair to say, let me ask you this before I get into the final two questions. If, if Y-T-D-L-P like permanently breaks with YouTube, I, I kind of have this feeling that there are gonna be a lot of repercussions from that, that people don't immediately think of.

Is, is that kind of your feeling too? Like there's a lot of, obviously these, these fly by night website services, they're gonna go down, but I, I feel like this probably gets used in a lot of other tools and, and there's a lot of places that are suddenly going to stop working if, if this goes down for a long period of time.

Bashonly: Yeah. Well, I think probably the most noticeable thing is I think a lot of creators on YouTube use it. To, you know, clip, clip other videos and include them in their videos. Yeah. So I think if YouTube support goes away, I think there will be a lot of annoyed creators that they have to find another way, or that, you know, they, they can't make these videos that clip other videos and stuff like that.

I think it's, I know it's used by journalists. Yeah. And it's, you know, to,

I think it's, it's used, yeah. Like you said, it's used for a lot more applications than, than you'd think. It's, you know, it's not just. It's not just, you know, yeah. Someone at home, you know, downloading a, downloading a funny video. They like, you know. Yeah. Right. So it's,

Jonathan: it is that too, but there's some, right, right.

I mean, yeah. Yeah. It is something we find with a lot of these open source projects that we talk about, there's a tail to them. Right. And, and not everybody can see the tail sometimes, you know, we're all, it touches things as hidden. You, you know, you guys are not in quite as many places as like lib curl.

Where there's probably a billion installs, but there's a lot, there's a lot of things that would, that would stop work. You would eventually notice that things would go away if, if Y-T-T-L-P went down or even if just YouTube shut it down. Completely interesting stuff. Oh,

Bashonly: well hopefully that doesn't happen.

Jonathan: Hopefully indeed. So I've gotta ask you two final questions before we let you go, and that is, what is your favorite scripting language and text editor?

I mean you Well, can I say

Bashonly: anything o other than Bash,

Jonathan: based on the introduction, I thought you were gonna say Notepad and HTML.

Bashonly: I, I think, well, I think I've gotta say Python. That's, that's what I write every day. So Python an editor was the other question. Yeah, a text editor. I like Genie. Or Genie, I pronounce it Genie.

G-E-A-N-Y.

Jonathan: I am not familiar with this one actually.

Bashonly: It's, it's really lightweight, which is why I like it. I don't like, like the IDs and stuff like that. Mm-hmm. And it has, it has syntax highlighting. That's pretty good. That's all I really ask for. So.

Jonathan: Cool. That is, that is not one that was on my on my radar.

I will definitely have to look into that. Very cool. Alright Mr. Bash, only, thank you so much for being here with us today. I sure appreciate you taking the time to do it and we learned a lot about Y-T-T-L-P and kind of the history and some of the things that go on, on the background that we didn't know about.

So thank you. Appreciate it very much.

Bashonly: Yeah. Thanks for having me on.

Jonathan: Yeah. Alright. That was bash only of Y-T-D-L-P and definitely a lot of fun. And like we said, there's a, there's a tail to some of these projects that you don't really think about, but they get used a lot, may or may not realize it. We've got something really fun coming up next week.

We're talking with Han Graver about LXC, the Linux containers project. Definitely looking forward to that. And then we have a couple of weeks open and then we'll be talking about. Kuna, which is a Linux security product. And excited about that one too. Appreciate everybody that's here, both those that watch us live and get us on the download, and we will see you next week on Floss Weekly.

Episode 827: Yt-dlp, Sometimes You Can't See the Tail

2 april 2025 | 59 min

Episode 826 transcript

26 mars 2025 | min

FLOSS-826

Jonathan: Hey folks. This week Neil Gompa joins me and we talk about Fedora the new 42 beta, the new KDE flagship news, fedora on arm and Risk five Asaki Linux and more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 826, recorded Tuesday, March 25th, fedora 40 twos new flag. It is Time for Floss Weekly.

That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett. And today we have, well, it's kind of a, a, a hark back to I think the very first pair of episodes that we did here on Under the New Hackaday Flag. We interviewed, I think it was the first, the first two we interviewed Neil Gompa.

And I touched base with him this week and I said, Hey, lots of stuff that hap has happened at Fedora and at KDE and at Sahi, all these different things that he and other, other places too. All these different places that he sort of has his fingers into that he works with. And he said, sure, we'll come, we'll talk about it.

It's, it's been a, a long time, almost two years now. So let's do a recap, a revisit. And so I've got Mr. Neil Gmpa here with us and welcome to the show. Hi. It's been, it's great.

Neal: Like, hey, at least it wasn't another 10 years. Yes. Yes.

Jonathan: I think that was our joke when we, when we did the, a couple years ago when we did the episode.

We'll have to wait 10 years to have you back. Well, we, we did not follow through on that when we, we get you back sooner. Ah, I'm fine with that. Yes. That's okay. Yes, me too. So stuff's, stuff's been happening. I'm, I'm trying to think about where to start and maybe, maybe the most interesting immediate news is Fedora 42, that beta is now out.

And that has sort of kept you busy, hasn't it?

Neal: Oh my gosh, yeah. It's been it's been a big release like. Lots of things have been happening in there from a technical as well as non-technical perspective. And I've just been running myself ragged dealing with all the different things along with everybody else who's working in, in fedora kd.

Jonathan: Now, remind us what, how, how do you, how do you fit into this? Like, what is your role at, at Fedora, at Red Hat? Question mark. KDD? No, no, no. I I do

Neal: not. I do not. Red Hat things. I do not. I am not, I'm not a Red Hatter. You know, there was the brief stint where I was in Red Hat as a salesperson. We talked about this in that first Hack a day episode.

And that was it. I, and I wasn't even in engineering, I was in sales, right? Like, it, it, it doesn't, I I am not a Red Hat hat.

Jonathan: It doesn't

Neal: count. It doesn't count. I don't do Red Hat things. No one, no one ever construe anything I say for something that Red Hat does or wants, or cares or anything of the sort.

Yep. Yep, that's it. But with within a fedora context, right? Mm-hmm. So. I am a member of the Fedora Engineering Steering Committee, which is elected every in my election term, basically every spring because I started with Fedora 32 and have been reelected every year, every year since. Mm-hmm. And Fedora 32, by the way, was spring of 2020, so that's, it's five years now.

Jonathan: That doesn't seem like five years ago, but, okay. No, it

Neal: doesn't, but time hurts a little. It's starting

Jonathan: to, yes.

Neal: I am also the lead of Fedra, KDE. I was previously co-chair rather than the sole chair, and before that I was a member of it. And I've been doing Feda KD stuff for going on 10, 10 years now.

Mm-hmm. I am also involved in Fedora workstation, fedora Cloud, fedora Server, Python, go Rust. Lots of different things in fedora space from the development and engineering perspective. Right. And the, but the main, I think the main thing we're here to talk about is kind of like my role within Fedor, Katie, and doing all this stuff around, around that.

But I also, like most recently, I helped mentor new people into Fedora to create their own special interest groups.

Jonathan: Mm-hmm.

Neal: Matthew Kosar, who makes the Miracle Window Manager, a tiling window manager based on Mirror. I helped him create the Miracle Sig and set up the Miracle Window Manager spin in Fedora 41.

Nice. And then for Fedora 42, I mentored and assisted Ryan Brew to build out Fedora Cosmic, which is also part of this release. Yes, so. There is that, and I also, you know, helped people with like getting Fedora Katie mobile and all this other stuff that happens. Like, so like I think you could, it, it's probably, oh, and also Fedora LX Q type kind of revived and now, you know, that's for Fedora 42.

That's Wayland using MiWay based on mirror as well. Mm-hmm. So I think like I have my fingers in a lot of pies in Fedora is I think, the way we want to put this. Yes,

Jonathan: yes.

Neal: So with, and we didn't even talk about as Sahi here because where I created the Fedora Sahi Sig and we'll get there made Fedora, sorry,

Jonathan: remix and all that other stuff.

We'll get there. That's, that's an interesting story in and of itself. So there's, there's something big with Fedora 42. Like we said, we just had the beta release. There's a big change with KDE and the workstation, like flagship. Right. What's going on there? So, historically speaking, since

Neal: Fedora 21

Jonathan: mm-hmm.

Neal: There we, we've had a rejiggering of how things have worked. So Fedora 21 was the first release where we had the change to have this concept of working groups with additions. And these additions are essentially top level curated variants of fedora that are promoted front and center. They are considered the flagships mm-hmm.

Of, of the, of the community. And we started with Fedora Workstation being the OME product or OME Edition. Mm-hmm. And we had Fedora Server and Fedora Cloud. Later on they added Atomic with CoreOS. Mm-hmm. Replacing Atomic later. And then there was Fedora iot added on top of that. And now we are adding fedora Katy Plasma Desktop as an addition.

Jonathan: Mm-hmm.

Neal: And so the additions are. Essentially variants that are in addition to being flagships, they're actually, they're backed by what is known as a working group, which is a special team that has to report both to the engineering steering committee, fesco mm-hmm. As well as essentially being required to be part of the other organs of fedora like marketing and council for the top level leadership and things like that.

Mm-hmm. So these teams have extra responsibilities that they have to fulfill as part of being a top level you know, a top level team. So then below additions are Spence, which is actually an ancient. Ancient variant term that we've used since the beginning of introducing spins in Fedora seven.

Jonathan: Mm-hmm.

Neal: These are essentially deliverables like media curated collections that are done by teams within Fedora, usually special interest groups that go in and take a collection of things that they care about.

Jonathan: Mm-hmm.

Neal: And produce a, a you know, something to, to show off. So, so you can download,

Jonathan: you can download an ISO that has your preferred desktop environment in it.

Neal: Right. So Fedora, KDE was the first live CD in Fedora. Mm-hmm. It was where the live CD stuff started. And you know, there's, there's many others now. Like, as I briefly mentioned earlier, there's Miracle Window Manager, there's cosmic, but there's also lx cute, there's a sway and much, much more. Right.

So I think there's like, I don't know. 15 of them or something like that. Goodness. I, I, yeah, there's a lot now. I, I have not counted. I guess if I go to the website and go look at the spins. Let's see. So three, three columns and 1, 2, 3, 4. So three by four. So 12. 12.

Jonathan: Yeah.

Neal: Yeah. I'm subtracting out automatically the fact that for some reason the website still shows the KD desktop as a spin as well as in addition.

But like, that's fine. Takes time. It'll get the it, everything eventually consistent. Yes. Is the way that this works. Yes. Eventually. So like there's, there's 12 spins and then then labs are essentially, you take a spin or an addition and you provide a workload opinion. So like for example, if you are an artist, you probably want the design suite.

Or if you are an audio enthusiast or an or a podcast producer, you probably want jam.

Jonathan: Yeah.

Neal: These take an existing desktop edition or spin. For example jam takes KDE Design Suite takes ome, and then layers changes the application set and some of the default settings to optimize it for a particular workload.

Mm-hmm. So for example, for Jam, they change some of the kernel settings to make it better for real time audio.

Jonathan: Right.

Neal: And things like that. And of course the applications are changed and things like that. So for the labs, there's three by three by three. So it's, it's 9, 10, 11 of them. So there's not that many, but mm-hmm.

The idea is that there is a way for a fedora community contributor if a group of people come together who want to do a particular type of solution, I think is the official word, to create a workload that, to, to express a workload type. There's a path for them.

Jonathan: Can you, can you mix and match the desktop that you want with the lab?

Like is there a way to say, I want an ISO that is jam, but running on cosmic?

Neal: Well, so not right now. Technically you could install cosmic and then install the, what we call composition group. Mm-hmm. Or comms groups for short, which actually describe the workload. Mm-hmm. And then you could just install it on top of an existing cosmic one.

Mm-hmm. So if you wanna do audio production on Cosmic, by All means you can, yeah. You could install the software that is used on those, on that spin, on that lab, and then put it on a cosmic environment. Like, there's nothing particularly nuts about being able to do that. But these are just like opinionated combinations that people have decided that they like and want to put together.

And sometimes it makes natural sense, for example. A lot of multimedia production applications are cube based. Mm-hmm. And they have integration with KD or they're from kd. Yeah. And so it makes a lot of sense to pair that with a KD desktop. It perplexes me that design suite ison based as opposed to KD based, given like half the software for kd.

But hey, I'm not in charge of design suite. If they want to switch to kd, I'm happy to have that conversation. Mm-hmm. But like that is the, the idea is that the people that are working on this basically come up with how they want to express a, the workflow and for the workload.

Jonathan: Yeah.

Neal: But you can always mix and match yourself if you want.

Jonathan: Sure. With, with KDE in 42, what, like, what's the exact terminology? Is it Fedora workstation? KDE plasma? Is that the, is that the title? No, no, no, no, no, no, no. So

Neal: the workstation working group owns the workstation name within Fedora. Okay. And they wanted to represent the GNOME desktop. Okay. For a variety of reasons.

This is how, that is Uhhuh and we've accepted it and that's fine. So for Fedora 42, the edition is the Fedora Katy Plasma Desktop Edition. It is a mouthful. I, I, I know. Yeah. But we get to have the word desktop in it, which is always important. Yeah. We have the word edition in it, which is even more important.

Yeah. And we have the word KDE in it, which is awesome. Yeah, KD is the best. So like, we want to be able to, and, and this has actually been beneficial to us because one of the things that I, we've taken a slightly different approach compared to some of the other additions, is that we have done integrated cross marketing between fedora and kd.

So if you go to the KD edition Katie plasma edition website on Fedora. You'll notice that the learn more links across the page actually take you to brochure sites on the KDE website. Nice. To explain more about this stuff. I'm taking the approach of let's not do it all by myself and let's, let's, let's kind of sharing this, and this was actually, this website was actually helped put together by a member of the KD Fedora community Carl Schwan, who's one of the KD artists that like puts together the websites and stuff for kd.

Jonathan: Cool.

Neal: Actually helped make this, like, basically wrote this website.

Jonathan: Mm-hmm.

Neal: Like, I gave him some headlines and some pointers and did some review of it, but he basically wrote it for us and I very much appreciate that he put in the time to make this work. Yeah. And it looks fantastic, I think.

Jonathan: Yeah. And so, you know, I, I assume this kind of means that you, whenever Fedora has an official presence somewhere, like we were talking before the show about Foz dim or, you know, wherever it is, this means that the, the.

The KDE Plasma Desktop Edition is sort of at the headline just as much as Workspace

Neal: Workstation. Yes. See works. Yes, for sure. That is the, the goal, the plan, the hope. It may take a few reminders to make sure that we get all this straightened out. Like, I mean, of this is brand new. This is, this is the first time we have something like an addition that people can easily demonstrate

Jonathan: mm-hmm.

Neal: Coming that, that we need to start prioritizing in terms of marketing. But yes, like the idea is that in the breadth where we talk about fedora and workstation, we should also be talking about KD plasma. The goal is to make them equal.

Jonathan: Mm-hmm.

Neal: Because that is what they are at this point. Yeah.

Jonathan: Yeah. I, I imagine that you to, to sort of talk about them with a shorthand, do you just refer to them as workstation and desktop? I.

Not officially.

Neal: I call it fedora Katie and Fedora workstation. Okay. That works Because, because while yours is, is also accurate. I, I try not to upset people. I try to keep, I try to keep the peace. I know some people don't believe it, but I do in fact try to keep the peace. Oh,

Jonathan: yes, I understand. Okay.

Something else I've wondered about with this is there is another distro out there that uses fedora as sort of, its, testing and source. So Red Hat, red Hat Enterprise Linux. Yeah. Is is in some ways a downstream of Fedora. Is the change that KDE is now a first class citizen, is that going to get reflected, do you think?

Not speaking as a, you're not a Red Hatter but just sort of, I guess, reading the tea leaves and what conversations you are aware of, is there gonna be any change to Red Hat as a result of that? Or is Red Hat always gonna be a known first distribution?

Neal: Red Hat doesn't have the staff right now to add a second desktop back into the portfolio.

Jonathan: Hmm.

Neal: The desktop team has, you know, this is the fir, I don't know if you've noticed, but like, for the first time in many, many years, red Hat is hiring in the desktop team. Hmm. They're hiring for known people to work on G stuff. I, I don't know. Whether this will eventually result in some kind of change on the Red Hat level to reintroduce KD plasma at in Red Hat Enterprise Linux.

Honestly, that's more of a conversation with Red Hat customers. Sure. If Red Hat customers start really pushing back and wanting to have KD plasma as their preferred desktop in on Red Hat Enterprise Linux, then I'm sure that there will be a change. But as of right now I don't, I don't see that happening.

That being said the up the immediate upstream for Red Hat Enterprise Linux is sent off stream,

Jonathan: right.

Neal: And pretty much all the cool people in CentOS Stream use Katy Plasma. So if you look at, for example, CentOS Hyperscale and the alternative images like KA plasma is a big part of, of those as well.

You know, like Cintas Hyperscale has not yet put not yet put out. It's Cintas Stream 10 based deliverable, but as the one making those deliverables, I could tell you that Katie plasma is very, very important to me. Right. And Troy Dawson, who works on Fedor, Katie e and supports Fedora k supports KD plasma on enterprise Linux distributions in Fedora kde.

Mm-hmm. He makes the alternative images and KD plasma is, you know, a very important part of it for him.

Jonathan: Sure.

Neal: And so I think if you look at the community of people who actually use Enterprise Linux in a way where they're part of. Developing this stuff or putting together things, I think you'll see a higher degree of KD plasma usage than you, than you would expect.

Jonathan: Mm-hmm.

Neal: There are a lot of people, even within Red Hat, who are KD fans, even if they're not in charge of the product.

Jonathan: Right.

Neal: KD plasma is very important to a lot of people, and it is a high quality experience that deserves to be shown out more and, and commercially supported more than it is right now.

So, so I'm just doing my part to make it better.

Jonathan: So in, in the realm of RL there's nothing official, but one, one might say that there are some undercurrents that are at play.

Neal: Maybe I, I, I can't say one way or the other 'cause I don't know. Yeah. But if Red Hat customers start asking for KD plasma and keep seeing or, or keep deploying it and, and, and ignoring Danone.

Maybe that will change. I don't know.

Jonathan: Yeah.

Neal: As it is, you know, like it was unthinkable five years ago that Red Hat would even provide any kind of engineering support for extra packages for Enterprise Linux. And here we are now, they have two people on staff that help work on supporting Apple and community people to work with re people mm-hmm.

On making Apple workout. So like, you know, that's a Red Hat is culturally very slow. We, we've talked about this before Yes. That they're, they're like this mishmash of the eighties and nineties cultures.

Jonathan: Right. And

Neal: they don't, they haven't really taken in a whole lot of the newer newer community cultures.

Yeah. So it, it takes time.

Jonathan: Sure. Sure. That's fair. Let's get, let's get back to Fedora and looking at the Fedora 42 beta. Is there, is there anything really exciting in there that you're just, so we, we talked about KDE becoming a, a. You know, flagship level. But other than that, is there anything in 42 that you're just, you're really excited about?

Neal: Oh, I mean, there's a bunch of good stuff in, in Florida 42. Like, I mean, honestly like the, one of the cool things is that we now have Fedora, Katie e specifically out of the box, we have support for running X 86 applications on Arm.

Jonathan: Mm-hmm.

Neal: Because we integrated the f stuff from the Fedora Sahi people.

You know, I say that as if they're separate people. It's all the same people, but like, you know, the stuff that we did for Fedora Sahi remix in 41. Mm-hmm. We are bringing forward into 42 in Fedora kde. Right. So if you're using Fedora KDE on Arm, you know, we have that stuff integrated so that it, it becomes a much easier experience.

Another thing that I'm really excited about, I mean, from a user facing point of view, is. The, the stuff that we've done to introduce the cosmic spin, right? Mm-hmm. The cosmic spin is very exciting to me because the system 76 folks have done a lot of great work mm-hmm. To build out a brand new ecosystem.

Yes. Nobody, people are not ballsy enough anymore to do stuff like that. And I'm very, very humbled and honored to have been part of help putting that together. Mm-hmm. And the co the folks that at System 76 working on Cosmic, I've met them in person multiple times now and we've talked a lot about Fedora Cosmic and they've been extremely helpful

Jonathan: Yeah.

Neal: Every step of the way. And they're very, very pleased with the progress we've made as well. Mm-hmm. Supporting Cosmic and Fedora. And they really do wanna see how it grows on, on its own. And they're very, they're very much more in kind the kind of spirit I have about open source where I feel like. Where you're building all this stuff and doing all these things to create these you know, these things that are different, the different options, opinions.

Mm-hmm. Like, you know, people decry, fragmentation, whatever. But like open source is all about you being able to do you? Yes. And, and if you can't do you then, like, what's the point? Right. Right. Like the and, and cosmic is a desktop that is made by people that embrace that philosophy. Mm-hmm. They are very, very happy to see Linux distributions adopt cosmic.

They're very happy to see Linux distributions adapt. Cosmic. That's a very big deal. So like, you know, this is in contrast with ome, which would generally prefer that you don't touch the ome experience. You keep it very close to what they're shipping. Yeah. Upstream and, and deviations are not welcome.

Jonathan: Right. Nome has always been very opinionated.

Neal: Right. And cosmic is opinion aid too, but it, but in many respects, they're more flexible. Mm-hmm. And their idea is we want you to show us how you can make cosmic awesome for you. Yeah. And I think that that is a wonderful philosophy. It's basically the reason why I use KD plasmas is because this philosophy's built into it.

Like you look at what we have for the marketing material for Fedora Katie Plasma Desktop, and we call it the Next Generation Personal Desktop, and I mean it. Mm-hmm. The idea is this is a desktop that you get to own. It is, if there's something you don't like about it, you can change it and make it more fit your needs.

Jonathan: I, I'm reminded of a conversation I had just the other day about h the way HDR works, but we'll, we'll set that to the side for the moment.

Neal: I mean, okay. Look man, sometimes technology. Sometimes sand doesn't work the way we want it to. Just give me the slider. But you know, they're working on it, right? Oh, yeah.

Like it's, yeah, yeah, yeah. And I, this is all, so here's, this is all new stuff.

Jonathan: Yes. Yeah. And we could, we could talk about H-D-R-I-I, I'll give everybody just a little bit of a, of a background as to what I complaining about privately and what I was told. Right? So like, I've got the big, I've got the OLED TV behind me and my normal use case for that.

I, it's, it's a 4K. And I discovered very early on after getting that, that it splits up into 4, 10 80 p windows very nicely. Right? And so I, I end up using it like, as if it were a two by two monitor grid. And there is a, there's a little plugin for KDE that lets you use keyboard shortcuts. Moodier is what it's called, lets you use keyboard shortcuts to like full screen things and then put things full screen into these tiles.

And so I use that a lot. And one of the things that I like to do, because it's HDR and there's now support for HDR and KDE is I'll have like three places I'm actually working on stuff. And then I'll go grab an HDR video. Usually it's like a nighttime walk through some big city and I'll throw it on the, in the fourth quadrant.

And. I do not like the white windows being blindingly bright. And in KDE 6.2, there's a nice little slider where you can tone down the brightness of your SDR content and yet have your HDR content at full brightness. And for my particular use case, that is amazing. And they took it away in 6.3. And so I've gone and I've complained and I was told, no, no, no, we're not gonna add that back.

But at the same time the, the, the, I think it's Xavier, it's aver, right? It's Xavier Hug Z

Neal: Huckle. Yeah, Z

Jonathan: Okay. Yeah. At the same time though, like he, he listened, he understood what I was talking about. And, and I feel that there is, you know, if enough of us go in and say, man, we really like this feature, like there's a possibility of getting it back.

I've also. Contemplated a little bit like, well, what would it take for me to patch my own KDE install, you know, my own plasma install so that I can get the, the cool stuff in 6.3 and yet still have my slider back. Yeah. I mean, that's the kind, that's the kind of thing I love about Fedora though, and KDE, like, I can go in there and find the patch.

It, honestly, it wouldn't be that difficult to go and undo whatever they did in 6.3 and make it work the way I want it to, and then recompile my own plasma packages like that is not sure the hardest thing in the world to do.

Neal: Also, you could make an applet or an extension that does it, because I'm pretty sure the underlying, the underlying setting mm-hmm.

Still exists in Quinn. Because you can manipulate all these settings using the tool case screen. Doctor,

Jonathan: I went and looked in case screen doctor and couldn't quite find it. But that's, that's an interesting point that it might, well, case screen doctor's terrible about documenting itself. So like there is that too,

Neal: like I, I wind up struggling figuring out how to use it.

'cause it doesn't have a man page and the help is dynamic and it's like, and it will crash if you try to use it outside of the desktop running. So like it is, it's a

Jonathan: very special

Neal: application.

Jonathan: Yes, yes. It is not, it is not intended necessarily for regular people to work with. I get that.

Neal: Yeah. But like essentially it, it just talks to, it talks to Quinn News against private, the private interfaces.

Yeah. But it is possible, you know, if the underlying infrastructure still exists. For the controls.

Jonathan: Mm-hmm.

Neal: There's no reason you couldn't make a pla, a plasma applet or extension that does it.

Jonathan: Yeah. That's an interesting idea. I may have to look into that in all of my copious spare time with all of the, all the other things on the to-do list.

Neal: All of us with our copious spare time can get so much done, right? Yes,

Jonathan: yes. I've, I've discovered though that I'm capable of a nearly infinite amount of work, so long as it's not the work that I'm supposed to be doing at a given time. Oh, geez. I know. It's the same thing. It's so, so very hard. Yes. Yes. Okay, so let's, you, you guys did something kind of crazy.

You, you brought a sahi in to KDE et su, these two fedora fedora, I guess the kde too, probably. What, what's the, what's the story with Fedora and Ashi? Fedora is kind of the flagship for Asaki now, isn't it? Yep. What, what happened there?

Neal: Well, I mean, so originally Hector Martin was working on this with ar, with Arch yeah, I had actually approached like near the beginning, just after the announcement on Christmas 2020

Jonathan: mm-hmm.

Neal: About working with him to produce a fedora ASA distribution. Sure. There wasn't enough stuff ready to be able to do it. So like early days, in very early days, yes. But he made his original prototypes on arm on Arch arch Arm, which is an unofficial derivative that rebuilds the arch packages for arm architectures.

Mm-hmm. Arch as a project does not support arm. So this was just one guy rebuilding everything for Arm because he wanted to very wildcat sort of approach. Sure. And that's fine and all. Sure. The problem was that a Sahi alarm, as it was called. Was not very stable and kind of broken a lot. Mm-hmm. I think the thing that, that popped his cork at the end was the fact that Firefox web r TC was turned off because it was too hard to figure out how to make it work on arm at build time. Yeah. And so that's like, well, okay, now I can't video conference, I can't do any of my work job things.

Jonathan: Yeah.

Neal: This is not good. And I was still plugging away at Fedora Sahi, and I'd offered him that, Hey, you want to, I'm, I'm happy to like, kind of essentially take over the distro part of this for you so you can focus on all the rest of it.

And I gave him an early prototype where I had. Things like kind of stroll together.

Jonathan: Mm-hmm.

Neal: And he was like, this is already better than what I have with, with arch alarm. So the Sahi alarm. So then we made the plans to switch to it. I started formalizing it. My partner in crime on this, David Alka helped with getting all the infrastructure in place and putting the SIG together.

And essentially we poured it over the, a little bit of the things that came from a Saki alarm to Fedora. And then the rest of it was All right, we, we just packaged the stuff and then put it together and we created a new opinionated base integrating Fedora features into it.

Jonathan: Mm-hmm.

Neal: And I think we've had a really good relationship since then because one of the things that has kind of happened since is that desktop Linux packages that are in arm on Fedora are actually really getting used.

And like there were fun surprises in the beginning where it turned out. No one had ever actually used these on real computers and they didn't work and they would crash in bad ways because Apple, Silicon CPUs, particularly the M two and newer, had features that fedora packages were built to, to, to use such as pointer authentication and branch protection and all this other stuff.

Jonathan: Yeah. Yeah.

Neal: That was enforced by the CPU. Well, it turned out no one had ever actually verified that those would work. And so these packages would just straight up crash when they were used. And it took, and these kinds of things, these kinds of things are the inhibitors, the real inhibitors, to being able to use another architecture as a daily driver for a desktop.

Like if you don't have people actively using it and you don't have a platform that's performing enough for people to use, you're just not going to see these things get fixed.

Jonathan: Yeah.

Neal: All the things are theoretical until someone uses it. Yeah. So, you know, in, in many ways this has allowed us to leapfrog, you know, fedora Katie has seriously benefited from it being the flagship variant for Fedora Sahi remix.

Jonathan: Yeah.

Neal: Because now ARM support in K is pretty much the best of all the open source desktops.

Jonathan: Mm-hmm.

Neal: And we have support for, you know, some of the weird things that you do with arm graphics and things like that, that pretty much none of the other desktops have.

Jonathan: Mm-hmm.

Neal: And because none of the other desktops have it, the experience is not as good.

And it until, and some of those desktops just don't want to care about it. And that's fine. That's their prerogative. Sure. But Katie is, was perfectly happy to have us contribute and fix things and get things working. And so that's, and like things like supporting HDR, making it a priority to handle high color depth

Jonathan: Yeah.

Neal: And stuff like that, those are important features that you need because the Apple platform. Pretty much mandates all of it.

Jonathan: Yeah. If somebody wanted to have an arm, laptop or desktop for that matter to run K to e fedora on what, what's the go-to right now? I wish

Neal: there was a go-to right now. Right now this is this, it's, it's a rough world outside of Apple silicon because to put it bluntly, not many people care.

Like you look at, you have Qualcomm with the X one extreme. Yeah. Right. The Windows Onar devices that they released last year. Right. Those devices have the beginnings of Linux support, but they're not in a way that is usable. Yeah. And this is in part because I don't know of qual, I am not assured Qualcomm really cares about making this work.

Jonathan: Right. So for some, for some devices, you get the what do they call it, like server ready, where it's where Arm has UAFI and boy that helps. Yes. That helps a lot to get boot working without having to go in and and mess with Device Tree. Oh my goodness. I've lost so many hours of my life fighting with Device Tree.

I still lose hours fighting with Device Tree. Yeah. U-U-E-F-I makes it a little bit better.

Neal: So it's UEFI with a CPI because what Qualcomm has done for their, for lytic support for their devices is they've taken the worst road in the middle, which is UFI with Device Train.

Jonathan: Okay.

Neal: Which you can do now.

Jonathan: Yes.

Neal: Because Yes, because ARM has made it a optional feature to have A-C-P-I-I

Jonathan: see

Neal: uhhuh. Yep.

Jonathan: Yep. That's, and

Neal: so system Ready now lets you validate a platform without a CPI support and all the ARM embedded people are like, oh, but A CPI is terrible anyway. We should device tree's. Awesome. It's like as a person that has to ship something for generic hardware, as a software integrator, as an operating system vendor, I.

Device Tree is the worst possible choice that you could have made for everyone because Device Tree is not standardized or specified. It's actually controlled by the Linux kernel project.

Jonathan: Yeah. And

Neal: changes based on what Linux kernel developers want. And that makes a ton of sense in the embedded place where no one cares about anything.

Jonathan: Yeah.

Neal: But like once you get past that and you start talking about scale out long lived devices that need to be updated continually, this breaks down fast. Yes.

Jonathan: Yes.

Neal: And and like

Jonathan: it is, it is not designed for consumers. It is, it is built for developers or kernel developers in particular, not for consumers.

Yes. That is exactly what I found with device three.

Neal: Yeah. That is, it's a big problem. And like, so right now, if any, any, all I should say, all current Windows on arm devices have incomplete UEFI implementations and thus cannot boot, S-U-E-F-I on Linux. Oh fine. Yes. It is something that I have complained about and is something that no one else seems to care about.

So it is the current state of affairs. As far as a potential go-to device, Lenovo has actually been fairly good about their proof of concept, as they like to call it. Mm-hmm. Work for supporting Linux on their windows on arm devices.

Jonathan: Hmm.

Neal: So there is a couple of them that you could buy where probably by the end of this year you will be able to reasonably load Fedra k arm on yeah.

The Lenovo ThinkPad T 14 s Snapdragon version. Mm-hmm. Gen Six is one candidate.

Jonathan: Yeah.

Neal: And I think there's like a idea pad slim that's being worked on as well. I think the Slim seven X or something like that has a Snapdragon version. Mm-hmm. And that one's also being worked on those two. And of course the ThinkPad X 13 s Snapdragon version.

All three of those devices have varying degrees of you can do stuff.

Jonathan: Yeah.

Neal: With it. Because the firmware that is required for those things has been made available. So you have full hardware enablement and combined with someone wrote device trees. It, it turns on and, and actually kind of works.

Jonathan: Yeah.

Neal: Performance, your mileage may vary, but between the difficulty of getting them set up the middling performance of the, of the socks, in reality I, I, there just isn't, I mean, the best of the go-to is still not great. Mm-hmm. But hey, it's an option. And we now have, we, we have UEFI enabled. Live ISOs for arm for fedora kd, as well as the raw disc images that you typically flash and you work with in the first for, for things like raspberry pies and whatever.

So like they're available you can try. If someone is actually interested in, in driving this more. I am, you know, within Fedra KD we are very interested in making this work. Mm-hmm. But getting access to hardware is the difficult part. Sure. Because they're not cheap and they are not fun to work with.

Jonathan: Don't hate me for this observation, but we're kind of in this weird place where we need windows on arm to do well enough for enough people to buy these, for the price to come down, for us to really be able to support Linux on arm on them. You're

Neal: right. But I'll also point out that even Windows is now making concessions for Qualcomm.

Craziness.

Jonathan: Right?

Neal: Like, so the way that it works for Windows to boot is they have just enough a CPI for the NT boot environment to work.

Jonathan: Mm-hmm.

Neal: And then they use drivers to pun fill in the holes, essentially Device tree style on Windows to fill out the rest of the platform enablement. That means that these devices are effectively going to be orphaned as soon as Qualcomm stops caring about them.

Right. Which is probably within two to three years. Yeah. That's the real, real dangerous part about the way that things work right now with ARM, is that unless someone cares and has the ability to make things work, like this is go that that stuff's gonna turn into E-Waste way faster than any X 86 machine will.

Jonathan: Yeah.

Neal: Or even Power PC machine. You know, and open and, and Device tree came from freaking open firmware like, and, and Open Firmware device. Trees are not this bad because they're expected to be on the board. They're expected to. They have, they have like restrictions on what, how they can be filled out. It's only on Arm and Risk five where device tree is a mess because it's controlled by the kernel rather than by a specification.

Jonathan: Yeah. Although, to be, to be fair, this is in some cases a pro. So this sort of flavor of thing is a problem on X 86 sometimes too. Sure. I've, I've got a desktop right over there that I'm working on for a, a local customer and it needs, it needs a Windows 11 reinstall, right? Oh boy. It's got, it's got some kind of proprietary NVME thing in there that need like, so Windows will not see the drive.

I know it's there. Linux can see the drive, no problem. Mm-hmm. Windows will not see the drive. So it needs, you know, it needs the driver installed during install time. Yep. That machine will not boot into recovery. It blew screens because it does not have the driver.

Neal: It's like it's, it's broken. So back there, there's an HP desktop that I have

Jonathan: mm-hmm.

Neal: Which has the opposite problem. The disc controller cannot be seen by Linux because it's sufficiently weird and not initialized properly by the bios. But Windows can see it because, because for some reason there, the windows, the windows, early boot stages seize the hardware differently than, than Linux does.

Jonathan: Yeah, yeah.

Neal: So like, yes, this is not exclusive to those platforms, but I would argue the problem is far Oh yes. It, it's far worse on arm and, and, and risk five than it is on X 86 on power. Like, and you know, speaking of power, right? Like Fed Door 42, Katie has now power PC support and it's Power nine and, and newer, basically it's little Indian power.

And like, I'm sure most people don't care. But like, hey, if you're one of those people that have a Talu workstation, hey,

Jonathan: you can now run KDE on it. Yeah, that's true. I guess there is Telus I was gonna ask whether there is any, any existing power devices out there, but there is Talu I've found about that.

Neal: I mean, in theory you could also install it on IBMP series servers, but like, I don't know why you want to put a, you know, door K on a P series server.

Right, right, right. But hey, you do you, it's, yeah, like the, the whole reason SUSE added like remote desktop support to gno for GNO Wayland was because people run no on mainframes. Yeah. So like Yeah. People do some

Jonathan: strange stuff. Yep. Speaking of the KDE. Arm fedora experience. What about like the raspberry pie?

I know it's not, well, you can make it into a laptop. In fact, I've got the the Ella Crow crow view, I think they call it, which is like a laptop form factor with ports to be able to slap a pie into the side of it. That's neat. Yeah, it, it is. I've actually, I've actually talked to them and I said that when you make the version two of this, just give us the give us the slot in the bottom to be able to slot the, the compute module into it.

None of this popping into the slide. Just let me plug a compute module into it in the bottom, and then well, if the,

Neal: if the, if the thing has like portage for like, you know, the N VMEs and, and all the other things like in the, in the enclosure. Yeah. Then the compute module makes sense. Otherwise, you might actually still want to use the standard F form factor board.

Jonathan: Yeah.

Neal: I, I would think that the, I, again, I know nothing about this product. I, I you, you're, I'm literally hearing about it just now from you. Let's see. Here

Jonathan: do. It's, it's neat. It's the crow view note, that's what they call it. And so, oh, it looks cool. It is, it's neat. And it's, it's literally just got USBC for power.

It's got an hd, like an HDI mini connection and A-U-S-B-A, and then they have this little carrier board. I, I literally just happened to have it here, this little carrier board that snaps in beside the pie and then the whole thing. Oh, and ports over the interfaces. Yes. And so the whole thing just kind of goes together like this, which it's, it's super cool that it works, but you have a laptop, but it's super ja.

You have a laptop with a sidecar, right? Yeah. It's a super Jake setup, but it's so cool. Exactly. Yeah,

Neal: I, I, I think I agree with you that you'd probably want to use the compute module there, but I would also probably say that if they go the compute module route, they need to make sure that the main board actually exposes all the interfaces that the CM five has.

Like for example, you probably want to be able to add MVME storage, definitely want NVME to make that work. Yes. Yes, absolutely. You also probably want to plumb through the interface to plug in the wifi card because Right. If you're using the compute module, you no longer have built in wifi.

Jonathan: I think some of them do.

Neal: Pretty sure some of them do. Wow. I am so out of touch with the raspberry pie people. Like this is, let's see, let's go look at their products. I'm just gonna quickly, the compute module is, oh yeah. Wow. I'm impressed and horrified. But point is right, the, you know, when you do it with the compute module, you gotta do all the interfaces that are normally on the board itself.

Right, right, right. But yeah, like the Raspberry PI four is supported. The Raspberry PI five is still being up. The work is still being upstreamed. My understanding is the Raspberry Pi Foundation has paid Sosa to do the work. Okay. So eventually it will show up and then we can do something with it.

But right now we can't. But you can do with a PI four, you could use the Raspberry PI 400 computer

Jonathan: mm-hmm.

Neal: And, and put it on there.

Jonathan: Yeah.

Neal: That's actually how I test on the rasp for, for Fedor Katie on arm is 'cause the Raspberry PI 400 is easy to get. It has all the things. I don't have to MIT fuss with it and it's fine.

Yep. Yeah. So that is probably the closest to a go-to arm device I could give is the Raspberry PI 400 right now. Which is, which is depressing,

Jonathan: but I mean, it's a, it's a really cool, awesome device. But yes, that's, that's less than ideal. Is it any better or any worse on risk? Risk? Five.

Neal: So risk five, I think there's been pluses or minuses with risk five.

Mm-hmm. One of the pluses for risk five is that I think people have made it a point to not make the same mistake that ARM did and say that. You know, arm arm's mistake was they said that they're not the stewards of the ecosystem. It's not their job.

Jonathan: Right. To make

Neal: sure people that are integrating arm IP do the good job of doing it.

I think they've seen the folly of their ways, but it's too late for arm to fix it.

Jonathan: Yeah.

Neal: For risk five, I think people are trying to be better about this. There are far more products that use standard UEFI for risk five now than there were for, than there are for ARM today after 10 years of it. So that is, that is a blow to the heart.

If I ever had any, if I ever said any, if I ever saw any, yeah. From the perspective of, you know, device trees and stuff like that. I think for a lot of the single board computers, they're still doing it this way because. They're still doing the open SBI and they're doing the, the crazy jump route. But like for example, the Milk five, the Pioneer Machine is full on U-E-F-I-A-C-P-I.

So it works like a normal computer.

Jonathan: Yeah.

Neal: I think the framework 13 risk five board while slow as balls is actually a standard UEFI setup. Mm-hmm. So like, I think we are seeing this mistake not being repeated as badly. Yes. Now, what is a problem right now is that risk five chips themselves, their feature sets are fragmented.

Jonathan: Yes.

Neal: Because the instruction set architecture is a puzzle, and, and people have not, I think my understanding is that the, they have just agreed on how to baseline for Linux, like just like a couple months ago.

Jonathan: Yeah.

Neal: So it's so new that there's no hardware with that pro, with that configuration.

Jonathan: So yeah, they, they need, they needed Intel to say, you know, this is the, the pen two feature set and then mm-hmm.

And then everybody else, but nobody's there. Right. There is no intel in the, in the, well, I mean, Intel works with Risk five, but they're not, they're not doing this. They're not,

Neal: they're not pushing anything.

Jonathan: Right. They, they, Intel uses risk five internally. They don't necessarily tell anybody about it. They, they put it, they put it like, I mean everybody does.

Yeah. Yeah. Well, I mean, that's kind of the point of it, right? Like all your Intel chips have little risk five cores in them. That's what runs the, the intel management stuff.

Neal: Right. And so does the the Nvidia GPUs, they're all risk five cores. That makes sense. Makes sense. Yeah. So, and same for Western digital hard drives.

The microcontrollers are all risk, five cores, stuff like that. It. Yep. And I, I don't care about those things, right. Because they're not, but, but they are endemic to the problem because, because there is no one who says, I care about this from a gen pop perspective, and I want PC experiences to work well, and I need Linux to actually not be crap on these platforms.

Jonathan: Yeah. How do

Neal: we do this? Yeah.

Jonathan: What's, what's the workflow look like for a regular person to do this install and use it as a daily driver?

Neal: Flash it to an SD card?

Jonathan: Yeah.

Neal: Right now, that's pretty much it. I mean, to be fair, right? Risk five in most distributions is brand spanking new. Nobody has any workflows.

Nobody's built any tooling. No one's done any serious testing. Fedora is one of the early builders of risk five. Fedora risk five has been around for a while, just quietly in the background. And then of course there's a buntu risk five and there's open source risk five, and there these are all efforts that exist.

Mm-hmm. All Mo Linux is working on a risk five build, and they'll be the first, you know, my, my, my bet is they'll be the first enterprise Linux based platform to have a risk five offering. Mm-hmm. And that will, that will change things because essentially hardware makers have to figure out how to work with the software vendors.

Because what happened with ARM was most of the ARM products for a long time were vertically integrated, and that's not what's happening with Risk five because the ecosystem is not vertically integrated, you are seeing much more collaboration on these fronts.

Jonathan: Mm-hmm. And so it, when, when we say vertically integrated, like let's just talk, let's dive into for just a second about what we're talking about.

Like, so this, this is Google makes Android and makes a pixel handset. And is paying, you know, Qualcomm to make the specific CPU that goes in it. Right. And so, like, they have control over the entire thing. And so they don't necessarily care about what gets upstream to the Linux kernel and what the experience is because they, they control the whole thing so they can just make it work for their, their platform.

So, well, they

Neal: don't have to ca they don't have to care about a generic case. Yeah. Is essentially what it

Jonathan: boils

Neal: down to. Yes. Yeah. That's although to be, to be clear, Google's actually one of the better Android handset manufacturers. That's, that's fair. Actually do upstream their platforms. And you can use them with standard Linux after you beat them to death a little bit.

But mm-hmm. You know, like a better example probably would be like Motorola, Sony Sure. Like the other Android manufacturers just don't do these things.

Jonathan: Yeah, yeah. Yeah. Interesting. Alright, man. We have we have been going for almost an hour here. Is there, is there anything that you wanted to talk about that we didn't get to?

Neal: Well, I don't know. I mean, like, this was mostly like, this was kind of spur of the moment, so I didn't exactly have a plan of things to talk about. But like, I always enjoy coming on Oh, yeah. And talking, even though we've only done it twice and now thrice, so

Jonathan: yes.

Neal: Let's not, let's not make it take too long the next time.

Jonathan: Indeed. Indeed. I did want to ask you about, I'm trying to go through my mental list too. There's been sort of a shakeup recently with Ishi Linux.

Neal: Oh yeah, sure.

Jonathan: And, and even some drama with Rust in the Colonel. And I'm, I'm curious, one, what your take is on all that, and two, is that gonna have an impact on the Fedora Sahi remix?

Neal: I'll answer the questions in reverse order because it's faster. It has no impact on Fedora Sahi remix. Mm-hmm. Because Fedora Sahi remix just chugs along, just doing the thing. I work on it. I take the patch sets, I build the kernel and produce the images and push them all out. That all continues as it does now.

So for the upstream components, Hector Martin essentially

Jonathan: burned out, right? Like that's, that's he got tired of fighting the fight and like that's understandable. That happens. Yep. When it comes

Neal: to rust in the Linux kernel it has always been contentious because mm-hmm. As that community doesn't churn a lot.

Right. So, like, one of the things I like to point out to people is it is very rare for someone to leave the Linux kernel project

Jonathan: mm-hmm. External project.

Neal: And that is actually a bad thing because there is no churn, which means there is no fresh blood, which means there is no new leadership, which means there is no new ideas coming in very often.

Jonathan: Hmm.

Neal: So this is actually the first like really new idea that has come into the Linux kernel in a long time. Right,

Jonathan: right.

Neal: And I mean, we're also talking about a kernel project that doesn't really use ci. They don't have they don't have good workflows for continuous testing and integration and delivery.

Their code review style is painful to use because the modern internet does not really handle having tons of patches being sent as emails. Mm-hmm. You have, you know, these people that have been doing the same thing for a very long time and things have, and, and they're all paid by their companies to just basically do the thing.

Jonathan: Yeah.

Neal: And they are in a position of which if they disagree with their company they can kind of do whatever they want. Sure. Because. They have their pickings of places to go work.

Jonathan: Right.

Neal: And so things are relatively stale and stable up in terms of personalities, in terms of opinions and things like that.

Jonathan: Mm-hmm. So

Neal: that's the stage here.

Jonathan: Mm-hmm.

Neal: It's also like overwhelmingly paid people. Like there are very, very few hobbyists that work in the Linx Turtle project these days. I believe that Greg Crow Hartman once said that, like even back in 2003, it was like 89% you know, these, and, and the churn rate from then to now is almost zero.

Right. So, you know, if, if in a 20 year time span you have all the same people doing all the same things in all the same ways and nothing has changed.

Jonathan: Yeah. And so, and then

Neal: you try to introduce change.

Jonathan: It's not surprising when you try to introduce a change, it's gonna be sort of cataclysmic.

Neal: Yeah.

Jonathan: And cataclysmic and cataclysmic is overselling it, but just to make the point.

Neal: Right. It, it is going to just be difficult. Yeah. Right. It's gonna be a difficult process. And I, the, the problem is that when you have a really old project who has really old opinions, who really doesn't get a lot of new people

Jonathan: mm-hmm.

Neal: And then you get a bunch of new people who are working in their free time to do something interesting

Jonathan: mm-hmm.

Neal: That is kind of hearkening back to the early days of how Linux was being built out. But they have different opinions of how to do things than you do. Yep. You're, you're just a culture clashes everywhere. Yeah, exactly. And, and to be clear, not everybody who works on a Sahi Linux has. Is a complete outsider in the, in, in the, in those things.

Like for example, Alyssa has been working in graphics drivers for decades at this point. Yeah. And she has very good experience working with both the kernel and the user space people on graphics.

Jonathan: Mm-hmm.

Neal: You know, Jean Grau has worked in multimedia for a long time and is a fff m peg contributor and fff m peg is a very similar cultural mindset.

Yeah. Warts and all. And you know, it kind of goes on like, it's not like these, and you have me who exists in a lot of places. Right. Like, so you, you, it's basically a story of burnout and the fact that like. You know, there's only so many fronts you can fight. And he tried to take them all on.

Jonathan: Yeah.

Neal: And it's just, you know, you can read the story.

He has written a blog post about it. Mm-hmm. And, and that's pretty much like the, I mean, I'm just happy it, it went well, the transition because mm-hmm. Like, it could have, it could have been different and yes, thankfully everything went fairly well as it has. Yeah. So

Jonathan: there you go. HEC Hector stepped down.

There are other people still at the Ishi project and Yeah. Someone else has already stepped up and said, okay, I'm gonna help maintain what's in the kernel. Yep. So it, it's gonna, it's gonna continue on maybe a little bit slower with the, with the loss of one or two of the people at the head, but,

Neal: well, we don't know yet because the current focus is now around Upstreaming stuff rather than writing, you know, new, I mean, some new features are being made.

Like we just released microphone support for like, just this past week, I think.

Jonathan: Yeah.

Neal: So. But, you know, we'll, we'll see how it goes. Right? Like the, it, it's still early days in the transition. We've only made the change like a month ago. Mm-hmm. So we'll see. Yeah.

Jonathan: Yep. Alright. Very cool. Let's see. I think about the only thing we have left is to is to ask you what your favorite text editor and scripting language is.

Neal: Well, it's still Python. Still Python, but I now use Kate as my text editor for things now. Ah, I fell in love with being like, so I told you before that for simple stuff, I tend to use Nano. And then for big programmatic stuff I use emax. Mm-hmm. But what I didn't tell you before was that in my, my original heritage of doing software development, I always used IDs.

Because big, big projects, I can't keep all the context in my head. I need a way to manage it. And k is actually a nice middle ground here. It has a language, language, LSP support. Mm-hmm. It has you know, many of the features that you would want in id, it's very much like vs code except not written an electron and actually fast.

Ouch. And like, if I wanted a full blown IDE, I could move up to k develop, right. Like so. Mm-hmm. Kate has actually been really nice and I've been using it for the past year or so for, for bigger projects and I've been very happy with using it. So Kate is now my preferred editor of Choice graphically and I still use Nano for small things.

Mm-hmm. Because, eh, I don't need the whole id, I don't need an Id like experience when I'm just editing a file. Sure,

Jonathan: sure. Alright. If somebody wants to try out Fedora KDE specifically, say the 42 beta, where's the, where's the best place to go? Look for it. KD dot fedora project.org Super easy. And if somebody wants to get involved with sort of the backend tinkering, is that the same place or where, where should

Neal: all that, all that, all that information is present on the, on the [email protected].

Jonathan: Yep. Yeah. Very cool. And then there's, there's various places like the Matrix chat room and, and all of that where people can come and hang out, ask questions Yeah. And bug you about features that are not there yet and all of that good stuff. Oh my

gosh, yes.

Yes, there are. I try to, I try to behave myself, but I've been lurking in Yeah, you're fine.

Chat for a very long time. You're fine. I've been on both sides of that equation. Right. I have been the guy saying, man, it doesn't work. Just fix it. And I've also been the guy saying, yes, we know it doesn't work. We're trying to fix it, or this is the reason why we're not fixing that. I've been on both sides.

Right. Yes. Empathy. Yeah. It helps. It helps. All right, Neil, thank you so much for being here, man. It has been, once again, a, a joy and a pleasure to have you and we appreciate the time. Yeah, sure. And I'm happy. I'm happy to be on

Neal: the, on the show and I'm happy to come back again in the future if you so desire.

Jonathan: Yep. We will definitely have to do it. I appreciate it. All right, so that was Neil Gmpa talking about KDE in Fedora. And definitely I can recommend it. It's what's running on the machine behind me. It's not what's running on this one for reasons, mainly because this is this laptop ship with Papa West.

And so inertia kind of got me there, but it's what I run on the main desktop behind me. But we've got some fun stuff coming up in the future on Floss Weekly. So next week we're talking the plan is to talk to one of the developers behind YT Dash dlp. That is one of the u the new YouTube downloader.

And then the week after that we're talking with Stefan about LXC, the Linux Containers project. And then a couple weeks after that we're working with someone from the Kuna Project, which is a Linux sort of, I don't think it's exactly an antivirus, but it's a Linux security project and platform. And definitely looking forward to that as well.

We appreciate everybody that is here, that watches and listens and whether you get us live or on the download, we will be back next week for another episode of Floss Weekly.

Episode 826: Fedora 42 and KDE

26 mars 2025 | 60 min

Episode 825 transcript

19 mars 2025 | min

FLOSS-825

Jonathan: Hey folks, this week Ben Metters joins me and we talk with Darko Fabian about Semaphore, a recently open sourced continuous integration continuous development platform that's got some neat tricks up its sleeve. You might want to check it out, so stay tuned. This is Floss Weekly, episode 825, recorded Tuesday, March 18th.

Open source CI with Semaphore.

It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host. Jonathan Bennett, and today we're talking about continuous integration and continuous development, and I've got a co host that we have some shared battle scars over this exact topic. Ben Meadors, welcome!

Ben: How's it going, Jonathan?

Jonathan: Hey, it is great, it is good to have you here. This is the first time that Ben has been on as a co host. He's been a guest twice before, I think? And, yeah, and so when we got the, when we got the guests lined up for today to do the CICD thing, my mind immediately went to Ben because, as I said, we have, we have wrestled the, mainly the GitHub CICD system.

And, you know, we were, we were joking before the show. It's like, when you have 27 force pushes in a row to your Git, that's probably what it is that you're fighting with. So.

Ben: Yeah. And it's always the, the, the step at the end of the work. Flow that you're trying to fight.

Jonathan: You got to wait 45 minutes, 30 minutes.

Yeah, exactly. Well, our guest today has something to say about this. Maybe he has some solutions for us. It's Darko Fabian, and let's go ahead and bring him on. Let me push the right buttons to make that happen. Darko, welcome. Welcome to the show. Hey, guys. Hey.

Darko: Hey, guys.

Jonathan: Good to be here. We are glad to have you.

So, you've got a you've got a company, Semaphore, that is sort of in this space. And let's sort of start with that. What's the, what's the background of Semaphore, and what does it have to do with CI and CD?

Darko: Yeah. Well it I think it started in 2010 or so. Mm-hmm. Maybe 2009. We are essentially like small six or seven people, Rubian rail consulting shop work for, you know, various kinds of startups.

Those were the, I would say in some ways golden area of, of SaaS. So whatever you had on your desktop or you know, somewhere, you kind of ported it into the cloud and we. At the time, we, we were, you know, embracing TDD, BDD, all, all that good stuff. We just needed, needed a CI and such a small team, we all wanted to, you know, just write code and deploy features and all of that.

That, that was valued, you know, for our size and the nature of our, the nature of our business. So we installed, you know Jenkins and experience was initially amazing. It's super easy to install. But the problems kind of start later when we need to scale it and, and manage it. And the other service that appeared in, in those days were, you know GitHub and Heroku.

And GitHub and Heroku, as you guys know, had great, great UX. And Jenkins didn't, was not on that, on that level. So, yeah, what we did is just, okay, let's, let's take that, you know, tiny bit of what we needed and bring it to the cloud. And I would say the main innovation was to say You connect to the GitHub, you click next, next, next, and you, you can use as many as those agents, you know, that will build your coders, and you don't have to, you don't have to worry about them.

And it just turned out that if you call that bit innovation, I don't know what's the, what's the threshold for showing something innovation or not. It changes over time. But that was enough. That was enough for Ruby on Rails community to embrace us. And that was our beginning. There are other interesting technical challenges and stories along the way, but yeah, we can touch upon them later.

Jonathan: Sure, and absolutely we will. I know we've got questions about the technical thing. I want to jump right to the new news, and that is that you guys are releasing this to the world. Or at least parts of it as an open source project.

Darko: That's correct. So I would say about a month ago, we released Sanford Community Edition under Apache, Apache 2 license.

It has vast majority of the features of our cloud, cloud offering. More, more things will, you know, will, will go in into the, the Community Edition. We just have to, you know, prepare the code and all of that. And yeah, there, there is also like an enterprise edition, which exists for a couple of years now which has all those like.

RBAC and various compliance security stuff, which is not in the, in the community edition. We are trying not to be too creative, at least in the beginning with this and do what, what others have done and some achieved success, some not, but, you know, open core open core model. Yeah, in a nutshell, that's, that's the story so far.

Jonathan: And I know it's only a month in, but how how is it going? Have you had a mass exodus of existing customers saying, Oh, we're going to host it ourselves. I, I don't imagine that's going to be the case. Have you started getting pull requests back where people fix things that bug them?

Darko: Well, to start with the first maybe observation no exodus, you know, so far.

I mean When it comes to our cloud customers, we actually, we don't charge, you know. There's no per seat cost, so what people are paying is for resources they use and kind of software comes for free. And our customer base in that realm is, you know, are just people who want something that works and they don't want to manage it.

They're in that mentality that I described as we were. So it's a bunch of developers that want to do as little ops as possible. So, I mean, there might be some people that, you know, for various reasons, might end up going that routes. Our risk estimate in that area is that that's not going to, you know, hurt us in any, any, any, any major way.

And the same thing goes for the. Customers that are, that they're using our enterprise edition where we provide, you know, 24 seven, you know, support, we set SLAs and the contracts and all the, you know, security and all the other stuff that they need to. And those big companies just want to, you know, manage risk in the best way that they can.

And one of the way that they manage, just manage risk is that they have our, you know, 24 seven support and, you know, there's also a certain license clause. But yeah, those are the two segments. But moving to the next next piece. So I mean, in terms of the reactions as you guys know, the main Our audience are developers, DevOps folks, so when they heard that something is going to be open source and they can, you know, fix something that they hated for years and we did not fix it, and it's maybe just like a CSS, I mean, the first contribution that we got was something in the realm of like, What is a selectable element.

So they have a need sometimes to create, to select part of the ui, and they don't want that bullet point, you know, they just want the name or, or something. So it's I dunno, it's something Nons. Selectable. So that was the, the first thing. And yeah, the direction from, from whole customer base was, was very positive.

Mm-hmm .

Jonathan: Yeah. Well, I mean, you, you've got, i i, I have the I assume it's github.com/sema four io slash sema four. That's the, the main repo. You're up to, you're up to 855 stars. You've got 27 forks and it looks like 10 contributors. I'm sure not all of those are from from Semaphore. You've got a couple of releases out, like for being a month into this project, this looks really healthy.

Like this looks really good.

Darko: Yeah, thanks. To be honest, I didn't have time to really benchmark us against, you know, other people, you know, going open source. I mean, there, there, there is a difference. Obviously Sanford is a mature product, you know, which is, which is going open source, which is unique.

I was searching a lot, you know, to find who other crazy folks, you know, took that route and they're like. Very, very few that I could, you know identify.

Jonathan: Yeah, there's, there's a few of them out there, but it's not, it's not what usually happens. Yeah. So let's, let's dive in a little bit to like some technical questions about the CI itself.

Ben, do you want to kick this off? I know you've got some.

Ben: Yeah. Yeah. And actually one that just came up, you know, you, you mentioned you know, kind of getting started. I think you said Ruby on Rails was kind of where, you know, where you cut your teeth. Can you talk a little bit about the, the stack of, of that Semaphore runs on?

Is that a Ruby on Rails application or is it kind of a mix?

Darko: Yeah. Well, it started as a Ruby on Rails application. And it was a Ruby on Rails application maybe until 2017 or something like that. Essentially just a bit of history after, you know, starting the, the, you know, product and getting the traction, we literally spent maybe four years or so just, you know, scaling and making it more liable because you make it reliable for a hundred customers, you know, that can run, I don't know, whatever number of jobs in a day.

And then a couple of months later, you are maybe order of magnitude, you know, higher, you know, volume of, of just work that you are, that you are processing. And it turns out that your state machines and the patterns that you use and all of that is just not, you know, not made for that scale. It literally took, took quite a while.

And reliability is number one attribute of good CI CD system. If, you know, it doesn't run, you know, it doesn't matter how pretty it is or all the features that it has. And as, as we kind of just wrapped that up, you know, and it was supporting customers at a decent, decent scale that they needed. It turned out that it, it lacked some, some, you know, major features that, you know, bigger customers wanted, you know, that, that I want to make dependencies, fan in, fan out, you know, all the, all those kinds of things.

And at that point in time we, we had a bunch of struggles with, you know, running everything as a monolithic application. The main reason was that you have certain components that need to be very, very stable, you know, and you don't want to shake them a lot because of the UI change or dependency change or whatever.

That would be the scheduling algorithm, that would be the the component which is receiving the hooks from GitHub, and the component which is just talking to the agents which are running the jobs. So then we initially started running. That monolithic application has like multiple instances of that.

One would just boot a router for accepting GitHub hooks. And, you know, maybe GitHub, you know, during last decade goes in a somewhat major way down, a couple of times, and then they DDoS that component. And then to scale that component, you have to scale the whole Ruby on Rails app, which has a significant memory footprint.

So going back to the, to the original question, we. In 2017 made a, made a, you know, decision to do like a rewrite to support all those like fancy things that we did not have, like a highly customizable CI and There was there was on one hand, the, the hype of, of microservices was alive. On the other hand, we had that experience already that for us, that being chopped up a bit in a couple of services at that point in time, maybe we had, you know, seven or eight.

And we liked the pattern and then we, we chose Elixir. So Elixir is now something like 80 percent of our code. And when we launched that 2. 0 version, we were Ruby on a list monolith, which has maybe 30 percent of the original code base plus roughly 15 services that, that, that we're running. So it was. I would still classify it as a highly risky move of, of rewrite.

That's yeah, I would not recommend that to, to many people. I think it worked for us mainly because we were couple of years in, we know the patterns, we know the, that would work for us and it ended up working for us. But

Ben: yeah, that's a, that's a all too common story. I've, I've heard of, you know, kind of outgrowing the ruby on rail stack and, and pivoting to the elixir where you have a little bit more. And here it's scalability, so that's encouraging to hear that you guys were able to kind of move over to that and, and solve a lot of those problems.

Darko: Yeah, I haven't spoke about the global interpreter lock and the concurrency and parallelism and, you know, those kind of fights that you get with Ruby that comes for free. But yes, it is there. I haven't. Yeah, haven't had the chance to, to, to connect yet with Jose Willem, the creator of Elixir language.

And I want to hear is, is, is for maybe the biggest, you know, Elixir Phoenix application open source. So, yeah, maybe it is.

Jonathan: It might be. One of the things that I know GitHub has struggled with, with its CI CD, or maybe it's more accurate to say users of GitHub have struggled with, is keeping those things secure.

Particularly when you've got instances like, someone has sent us a pull request, we're going to automatically kick off a CI run. that uses that pull request, and so essentially you're going to run some untrusted code on your, your CICD service. How does, how does Sima4 handle that? How do you avoid getting owned every time somebody sends a PR and it kicks off a run?

Darko: Yeah, I mean, there are two features that I'm familiar with, there may be something else that I don't know, so I think that the first one that we implemented is that there is a radio button or a checkbox disable Runs from four, four tripos. That was the, the first, after that, there was a feature of adding a whitelist.

So I could take maybe GitHub handles from you guys and I can add it to the whitelist. So other people can. The CI will not kick off unless, you know you are in the whitelist. And I think the last iteration of that in terms of the, the UX is that you can, if someone opens a pull request from a fork, you can go and put a comment slash Sam dash approve.

Which would be kind of a manual approval for a random person from the internet To run to run to run the ci job and it seems that it it covered the Most of the corner cases for for a lot of people.

Jonathan: Yeah. Yeah, it's it's something that For the longest time, people didn't realize that this was going to be a security problem, right?

And, and then suddenly it was discovered that RCI has all of these, they've got tokens, they've got GitHub tokens. Sometimes they have AWS tokens, they have all this stuff. And it, it kind of was thrust on the, on the scene that those tokens are sometimes accessible in ways that you don't want them to be.

And so I know places like GitHub have done a lot of work to try to make that secure. We, we got an email not too terribly long ago that GitHub, Was made aware of a problem where one of the action scripts because on github you get to like this There's this whole community of actions that individuals have written and one of those actions got Maliciously modified to have some code added to it.

And so We then had this issue where it's like you didn't You didn't accept code intentionally that was malicious, but because, you know, you may have used this action, you may have run it. And so like, that's a, that's a security problem that has a lot of hair and. We're really still trying to figure out how to, how to, how to deal with all of that.

Darko: Yeah, I would classify it as I mean, somewhat typical open source security problem. As there are so many dependencies that, that are, that are out there maintained by You know various kind of people and we have these, you know takeovers of how it's called xz Yep, okay, which is on a completely another scale.

Maybe not the best example to to use but They're just thousands and thousands of dependencies and tracking What's in all of them is a major

Jonathan: so we've talked about to go a different direction We've talked a lot about github because that's obviously where a lot of people are at And I do want to ask about the github integration But I first have to ask what about get lab and some of the other get labs not the only other one out there But is there any semaphore support for some of those other?

Services that are not get hub.

Darko: We don't have a bit bucket support and and a get live support. Okay.

Jonathan: And so what, like, what does the, what is the integration look like? How let's just take for get hub, for instance, because I feel like that's what most people are going to be most familiar with. Does it just sort of drop in, replace all of get hubs C.

I. Stuff so that you, you don't have to mess with it. Or does it, does it use some of the same interfaces? Like, what's the, what's the user experience like?

Darko: Well, once you like have account, let's say Sam for cloud, and you you want to add a repo, you want to connect the repo, there is like a GitHub app that, that needs to be installed.

Mm-hmm. You click, you, you kind of initiate the process. Sam, for side it, you know, kicks you to, to GitHub. You have to select a repo to which you want to give access to s for on oral field repos. And then you authorize that, and then you go back to, to Xamphore and Xamphore through the API can list your repositories, you pick the repo in which you want to configure Xamphore then it's a single click action.

And in the next step, you need to essentially choose one of our templates for, you know, your CI process, you know, some YAML, maybe play around with it and click next, and you have a first fail built.

Ben: One feature I was kind of interested in that when I was looking through the docs on Semaphore is, I saw mention of an interactive debug with SSH, and that sounded particularly intriguing from my experience that Jonathan mentioned earlier of, of constantly force pushing to try to, to try to kick off A failed feedback loop and in a CICD pipeline and trying to figure out what's going on in the in the constant print of like a LS command or something to try to figure out why some file isn't there.

Can you talk a little bit about that feature?

Darko: Absolutely. Yeah, it, it comes in, in, in two flavors. There is a CLI tool that we have. And essentially if you would go to a failed job, there is like an icon that you click or, you know, debug and you, you copy a line, which does SAM debug and that job ID. You paste it in your terminal.

And what SAM4 is going to do is going to spin up the. The VM or a container, however the job is configured, up for you, and it's going to export all the environment variables, credentials, and all of that into that environment. And it's going to SH you into that. So what would end up happening after five seconds or so, you would be SHed into that, into that box.

And then There is like a script like in commands. sh where we export into that script all the commands that were configured for that job and you can just kick off that script and it would just run all the commands that were meant to be running that or you can, you know, start manually and do Call the checkout command to check out the code and then crunch slowly through your commands.

And it is extremely useful for those, for those situations that you are describing. And it comes also in another flavor, which is sematach. So let's say you have a job, which should finish in five minutes, but let's say 30 minutes in, and it's still hanging, it's saying hanging on some dot from your, your, your test report formatter, and you have no clue what it is, then you can do a SAM attach and give that job ID, and it's going to give you the SH access.

Into that job, you would have those processes running or being stuck or whatever. And this is the moment where your sysadmin will go hardcore sysadmin skills, you know, kicking. The first thing that I suggest to people is that they do S trace on that, on the ID of that process, which is running those, I mean, check out the logs first, but if it's messier than that, then you'll do the S trace there.

And you can maybe see that. If it's some browser heavy test, you will see in the logs that it's maybe polling on some external network resource in a loop, or there is a lock, you know, you are waiting on some SQL query which is now never going to end, or those kinds of things. And Yeah, I have to say that we kind of build those features more for ourselves than for the customers.

Well, that's, that's the way you imagine support. That's the way, that's the way the

Jonathan: best features happen though. That's right. That's the way the best features happen. Goodness, I, I tell people all the time, I'm good at X, Y, or Z, because I've been bad at it for so long and had to figure out how to make do, right?

It's the same thing. The features that you dog food are absolutely the ones that people get the most use out of.

Darko: Yeah, it's in the realm of like, if it hurts, do it more often. Yes!

Jonathan: Oh, that's so true. What about architectures? What architectures do we support? So one of the things that we've run into is trying to do ARM64 builds on an ARM64 runs on CICD.

And that's just the tip of the iceberg. You know, there might be people that want to do old school power PC MIPS RISC, all kinds of stuff out there. What does support for that look like?

Darko: I mean, what is support is, you know, Mac, any flavor of Linux and and, and windows out, out of the box. And then there is a support for also one, which falls into that category, which you mentioned of mm-hmm.

Like, you know, arm 64 and so on. There was, the agent is open source actually for many years now. And, you know, people can compile that. I mean I remember that some people were doing some, battling with some how it's called A IX. Is that the IBM Linux Unix, right? Yes.

Jonathan: Yes.

Darko: That, that's the most exotic thing that I remember.

Plus people were, I think, adding FIPs support for, you know, encryption for the agent.

Speaker 5: Mm-hmm .

Darko: And, that's in the realm of exotic, you know, platforms that I'm aware of. But there are many others.

Jonathan: Well, I guess, I guess that is one of the huge advantages that you have over GitHub is if that agent is open source, then somebody can just come along and run it on just about whatever platform they want to.

Obviously, obviously there might be some technical issues you have to work through to actually make that work. But that's actually, that

Darko: component is, you know, that component is written in Go. Just to be fair to also all the Go folks, there is, there is some Go, I mean Yeah, forcing people to run Erlang VM on their on their agents would not be

Jonathan: We interviewed some of the people behind Erlang, and I think specifically the NERVS project.

And that thing is, that thing is interesting. That's a, that's a cool project. I hear nothing

Darko: against Erlang VM. I think Erlang VM is great. It's just that it's you cannot beat that single binary, which is, you know, 5 megs or so.

Jonathan: Yes. Yes, it is. It is a different way. It's a different way to look at the world.

Speaker 4: No.

Jonathan: All right. So do you, do you know of, or do you have people doing crazy things like running this on a Raspberry Pi? Is that, is that sort of in scope or do you know of anybody doing that?

Darko: I mean us, you know, when we started the open source was owned by Travis CI. You'll probably remember those guys.

They're kind of out of the scene now. And then it comes, when it comes to the open source community, you know, GitHub Actions, you know, just by the nature of where the code is, you know, took over the ownership of that and those kind of like super cool, geeky fun projects unfortunately never belonged to us in,

Speaker 5: in,

Darko: in, in, in that phase.

So Rosemary Pi is not, but on the other hand it was great for us that, you know, various startups in. You know, various domains did use, did use for, for their commercial work. And I would say there are like a couple of interesting things here. So let's say Tegera, a company that makes Calico, like a networking layer for Kubernetes.

They, they need some very custom, you know, networking stuff to be accessible for them to be able to run all that. So then that's one of the projects. Where such, such capabilities are needed. There are now a lot of people that have different kinds of GPUs and those kind of things that also have a lot of data that's sitting somewhere on some very expensive NVMEs and then run those kinds of workloads there.

So I would say it's more kind of that, you know, there is a lot of exciting hardware that people use, but it is, um, more in the commercial setting. And, you know, so for Rosemary Pi. No one reported it. I don't know.

Jonathan: I could, I could just tell you from knowing the open source community been around for a while now.

It's coming, right? As this, as this gets popular outside of the enterprise, somebody is going to be like, I've got an old Raspberry Pi. Now it's probably going to work best on like a Raspberry Pi five with an NVMe, but somebody is going to be, I can guarantee you because I've In the communities I've been in, I've had this happen.

Somebody will come along and say, I got it running on a Raspberry Pi one or the, the zero, the original zero or the zero two. And they'll come along you, they tell you that and just kinda scratch your head. Like, I didn't even think that was possible. .

Darko: Yeah. We are looking forward to that. I mean, honestly speaking, we were missing that, you know, kind of fun element.

Yeah. That, that was initially, initially very present, but you have to be. You have to be glad that you're getting, getting these bigger and bigger customers, you know, it makes your whole story more viable. Yes. But on the other hand, those guys have a very clear KPIs, you know, and OKRs and all of that.

And it's more like CTO comes and says, we want our build to finish under three minutes. And it needs to be parallelized across 60 jobs, you know, in parallel to achieve that. And then. Okay, let's scratch our heads. Let's, let's make, let's help those people make that happen. So there is that element of like, fun is sometimes replaced by a serious stuff.

Jonathan: Yes. Yes. That's, that's the danger of being enterprise first and trying to bring the open source along, but it'll, it'll get there. You can't, you catch the attention of the right people and they bring the fun element with them.

Darko: Yeah. I mean, originally we were not enterprise first, you know, originally we were like, you know, Small folks just trying to make their SaaS thing, you know, go to the next level.

And then it was like, literally we had a lot of those customers who were like maybe five, 10, 15 people for first couple of years, and then something takes off. And then in 18 months, they're like 150 or 200 engineers. And yeah, yeah. And that's how it translated into Datrum. And then over time it just kept going in that direction.

And now the biggest team that is using CERN for is 1, 800 people. You probably know the folks, um, confluent behind Kafka.

Speaker 5: Mm hmm.

Darko: And we, we just entered that, that, that space over time. And I have to say that it's, it, it was a good thing for us, although it, you know, it, it was a struggle to, to, to achieve it because as GitHub Actions, you know, came along, they did cut off that, you know, layer of those five, 10 people, you know, small folks for who's, you know, for whom GitHub Actions, you know, it's enough, it's enough up to a certain scale.

Sure.

Ben: To that point what is kind of the big value add of, of Semaphore when, you know, if, if I'm an open source project on GitHub, at what point can you kind of point to the key features set that Semaphore has that, that it can do above and beyond what GitHub Actions does?

Darko: In the open source world

I don't, I don't, I don't think that there is like a A single huge thing for open source people, you know, this debug, you know, features, these features, you know, being able to manage who can run the full pull request and so on. I mean, builds of most or vast majority of open source project tends to be pretty simple.

You know, if it's like most of the libraries and it's relatively, relatively simple for biggest projects. I agree. You know, someone has to compile Linux and, you know, run all the little stuff and so on in the commercial realm. It's it's maybe a bit different or around that you are iterating on a, on a bigger project with a larger team.

There is this debugging features that we spoke about. Then there is what you see is what you get, like a visual builder. You don't have to write the ML code. You can draw your pipeline. And it generates the code for you in the background. And I don't know if you have like a 50 people team, there are, you know, a couple of CI geeks that, you know, love to mess with that stuff.

But the vast majority of engineering doesn't want to learn your DSL and, you know, how do you run, you know, conditional expressions, if something's going to run or not, and so on. And I would say that the biggest other component is that, you know, continuous delivery component, where for us it has only been the kind of the first class citizens.

So promotions, which can, you know, trigger manual automatically, you can model your deployment process and to end tests and so on. So those are. Those are the big differences comparing to GitHub Actions. It's not that impossible to do that in GitHub Actions, but it's definitely not a first class citizen.

You cannot see your deployment environment with, you know, in such a nice way with, you know, everything was listed, who deployed where, under which conditions, and so on.

Jonathan: If somebody is running Semaphore, making use of all this, and they run into a problem with their CI not necessarily a Semaphore bug, but just CI is hard and therefore you run into problems.

Do you guys have sort of a, a, a forum a way to, to go to y'all's engineers and say, this is the crazy thing my code is doing. Can you please help me? Is that in scope?

Darko: Again, I mean for this, for the open source, you know, piece, we, we have discord that we had for a while mm-hmm . So some people, you know, will probably able to give, you know, a piece of advice there.

For the customers, it's, you know, depending on kind of the level of the, you know, support plan, there is a possibility of, you know, hitting P1 and, you know, we jump on a call and we help you but that, that's that, you know, commercial enterprise ish setting and for other folks, you know, there, there's a, you know, a typical, you know, support forum where, you know, we share advice and tips and, you know, some debug, some attach.

Jonathan: Yeah. No, that's, that's great. I don't think any, no one reasonable using this expects to get that kind of that kind of support for free, right? I think, I think that's fully understood. It's just, it's interesting to know what the options are that are out there because man, I've, I've lost hours and hours of my life fighting with CI and CD stuff.

Darko: Yeah. I mean, as, as every project, every company is, you know, fighting for their space in the world. I mean What our customers think is unique about Sanford, it's not that kind of support that they're not going to get anywhere else and certainly not for, you know, huge vendors such as, you know GitHub, you know, you hit the P1 there because your whole team of, you know, let's just say a couple of thousand engineers are stuck and you have a support plan with us.

We are going to jump on a call with you. You know, even if it's not our fault, you know, to, to, to help you out. We have a luxury of doing that. They, they don't, I don't.

Jonathan: Yeah. That's, that's one of the things I've noticed about GitHub is that when something goes wrong, even if you can point to like, here's a reproducible, reproducible test case.

Where something is wrong inside the, the GitHub code, like getting someone's attention to actually get that fixed is sometimes very difficult. And so I think that in and of itself is, you know, an advantage to working with maybe a little bit smaller team than the behemoth that is GitHub and trying to get support there.

Darko: Yeah, yeah. I'm sure that they're going to get to help you. If you're paying, like, millions or tens of millions a year. Yes. You don't have to

Jonathan: worry about that. That's true. That's true. Us normal folks don't have They're a free customer. Yeah.

Ben: Doesn't get priority support with the millions of other fires they're dealing with, I'm sure, on a day to day basis.

Jonathan: And honestly, though, that's where the fact that all of this is open source for Simba 4 comes into play. Because there, there are bugs in GitHub or, or maybe not even bugs, but just quirks in GitHub CI that I would absolutely go and fix if I could get to it, but it's code that I can't get to, so I can't fix it.

And I think I think, you know, as time goes by and you get more people kind of aware of this as an open source project, you're going to get a lot of those. It's a lot of just little pain points, like your example about this bullet point shouldn't be selectable because I don't want to copy and paste it like that's a trivial thing to fix.

But a very small people group of people would realize that that's an issue. But you then also have a large, like you've got this real small group of people that go, this is an issue. This is annoying in a certain way, and I know how to fix it. And then you have a larger group of people that are like Oh, it doesn't copy the bullet point now.

Cool. I didn't even realize that annoyed me, but I'm glad that they fixed it right. And so you get sort of those knock on follow on effects, and I'm excited. I'm excited for you guys to see some of that start happening, because it's gonna be cool.

Darko: Absolutely. Absolutely. I mean you know, just a few variables, few numbers can change things drastically as like a response time to a debugging CI request.

The other component is that, you know, we are by the By measures of, you know, super successful startups, we are a small, small team. We are 40 people, you know, roughly half of that, you know, engineering. And as you can imagine, our backlog is huge, you know, for various small improvements that people want to make.

And one of the things that we are kind of counting on, we don't expect anyone to at least initially implement anything bigger, but you know, these small paper cuts. You know, are the things that you're hoping that, you know, people will jump, jump in and, you know, work on.

Jonathan: Is there anything that's been a real challenge in this process of going from closed source to open source?

Like, have there been any particular pain points or we, we talk about Some companies, when they try it, when they even think about doing this, they've got like NDAs. And so some of their code, some of the code are either licensed from another company or there's some NDA keeping them from being able to open source.

It was, you know, it's something of that of that nature. Were there any pain points?

Darko: I would say that this happened in two big jumps. The first one was when I spoke about that, you know, growing up from serving small. Small shops to serving the bigger companies. And then at some point people came and said, well, the Sanford is great.

It always with features and so on, but you guys aren't in the cloud and our security compliance, legal teams are not going to take it. They might take it from, I don't know, GitHub or, you know. AWS, but you guys being like a smaller vendor and all of that, you know, does not satisfy. But, on the other hand, if you would let us run it, we would be happy to pay.

And then it was a process of, roughly speaking, like 18 months of packaging the app so other people can install it and run it. And that was, that was the first jump. And once, once we did that, kind of relatively soon after, we had that experience of, you know, some customers want to contribute to some area, have the influence in some area to a larger extent that, you know, maybe we would not be able to, you know, in a short period of time address, you know, if it, even if it's like a relatively small thing, it really depends.

And, That was one of the inspirations for, for, for going open source. The other obviously being, you know, more people being able to use it. But to get back to your question then there was an element of open sourcing it. Obviously one of the first thing on the list was, you know, security and the code that was written kind of over a decade in a closed environment, under, you know, How many eyes, you know, 20 people or so looking at it.

And now the whole world will be able to, to, you know, take a look at that. So that period of like figuring, figuring that stuff out and polishing stuff, also defining what is, what is dedicated to SAS. So you don't want to open source your billing. And let's say billing is part of your main app, you need to drop that out.

You need to drop out the, the scheduler for, which is somewhat specific for the SaaS, you know, workloads and so on. Not, not, not to go into all the details. So microservices architecture helped here. If it was a monolith, there would have been a lot of cutting. Now we use gRPC in between, and then you can rewrite certain service for this particular, you know, use case and, you know, chop it up.

But I would say that roughly speaking, it was a period of roughly a year where I would say larger part of the team was dedicated to preparing all it you know, to, to go, to go open source. You also want to make it as simple as possible to install. And that was not your priority previously in such a, such a, such a huge way.

Jonathan: Indeed. Yeah, that makes sense.

Darko: If anyone, if anyone wants to take this route, I will be happy to track the chat and, you know, Offer maybe some specific piece of advice? It is one, it is a it is a significant investment.

Jonathan: Yeah. Yeah, for sure. And then kind of looking forwards in the, the future of the project like the, the maintenance of it going forwards what are your, what are your thoughts on that?

Like it kind of helps. One of the things we think about with open source is maintainer burnout, and if. You're paying some of the maintainers. That helps a lot avoiding burnout. But like other than that, what are some of your, your future going thoughts about the maintenance of the project?

Darko: Yeah, there are like two major changes that are happening.

One is, you know, making just the, the source code open source, which we are obviously excited about, but I'm personally more excited about our transition to, you know, building in public and moving the engineering product, all of those, you know, organizations to, you know, like you guys are streaming right now, we are, you know, streaming our meetings and making all the.

You know designs and architecture discussions and all that, all that in, in, in public. I mean, on one hand, that obviously has has a huge benefit to, you know, just showing the dedication to, you know, doing stuff in open and, you know, other people can learn and join and it's not it's not just in that code.

There is there are people around it that you can talk with and so on. So, yeah, I mean, as you mentioned, there are people being paid to contribute to Sanford, so Sanford It definitely has that component of being, you know, commercial open source. Our goal is, it's now a hundred percent commercial open source.

We want to make it less commercial open source and, you know, move more into the community realm. It will take time, definitely. But we hope to eventually, you know, get more and more people who are just involved as a community and it's not just us who are contributing to it.

Jonathan: Yeah, what what is the funding model?

Like, so obviously you've got your enterprise customers and they're, they're paying for they're paying for the, you know, whatever it's services they're getting. What's the ramifications of going open source on the funding model? And do you have some interesting ideas for how to make how to make money particularly with the open source project?

Darko: I mean, there is I would say almost nothing and definitely nothing ingenious, you know in, in this realm. So we started as a SAS. So you paid for the, for the minutes that you use to, to build your projects no per seat cost. You just come in, you and you pay for, for, for what you use. That was initially the case.

On top of that, we added that enterprise model there, where there is per seat cost plus the, you know, support elements to that. And we are we are a fully bootstrapped company, you know, maybe not. Directly related to your question, but just to mention that, so when it comes to open source, I mean, those two revenue, those two revenue streams would fund the development going forward as they did so far.

And yeah, we will see. Basing on who are the people that get involved, and who are the people that end up running, you know, CI for themselves. We'll see that that might change might change over time.

Jonathan: Yeah, there's, there's a couple of, there's a couple of revenue streams, ways to make money, that sort of come to mind with this, that you might see some, some traction in.

And the first one is simply, you know, some big company wants to run this internally. They don't want to necessarily pay you a support contract for it, but they say, this is the feature we want to see. Let us contract with you guys to add that to the open source project. And now that happens a lot in various cases.

One of the other really interesting ones that we're sort of as the open source community, we're only now really figuring out is. The, the new, the new security laws in Europe and the United States, you know, they're, they're beginning to say that there has to be some attestations that come with this, this product, you know, if you're going to run it in certain places or if you're going to run it as a business, you know, things like that.

And it seems to me that there's sort of a, there, there's a space for. Open source projects to, and I don't know if this exactly applies to you guys, because you sort of come at this from the other side. But for an open source project to say, Okay, fine, you run it, and we will sell you, like, the attestation that this is, as far as we know, is CVE free.

And, you know, we will sell to you the, um, the, the, the list of other programs that are in it, right? The the BOM, the Bill of Materials, the Software Bill of Materials. You know, and, the, the, one of the places that comes from is you, we've had projects that they have this complaint that, oh, this big corporation came to us and started demanding an S BOM from us, and we don't have time to work on that.

And every time my response is, Sure you do, you just have to tell them this is your hourly rate to do it, right? And, and I, I really think that when it comes to open source, Particularly with the interaction with other corporations, That there, there really are some potential there to, For the an outside corporation to really support the project.

And I'm not sure how much of that makes sense with you guys, Because like I said, you came at this from the other direction. You've got the, the corporation first, and then you open source something. I'm curious your thoughts on all that. Yeah.

Darko: Yeah, I mean, for the second piece, that's something that I mean, SBOM is a big thing, I would say it is somewhat solved in the industry, but the whole problem around like making open source secure is not really solved because you have no clue who contributed to certain libraries and, you know, what's the, what's the, what could be the motivation behind the first thing is something that we talked about.

We don't have many data points. Um, maybe it will be scalable, maybe not. It's, it's, it's hard, hard to say. I mean, I do have a background in consulting. Consulting is a tricky business in its, on its own. Yes. The second piece is, is, you know, is possible to in some realm. But yeah, honestly, I don't have any deep thoughts around that and if that can work out or not.

Jonathan: That is fine. Is there anything like a CLA that contributors have to sign to be able to add code to the project?

Darko: I mean for the Apache two piece no, but for, there will be a much smaller portion of code that we will keep under that commercial license piece.

For that one yes, but we are just in the process of kind of moving that code in the, in, in, in the repo. I would say more than half of it is already there, but we are still, you know, polishing some off and just transferring it there.

Jonathan: Makes sense. There is a question that I know Ben, he told me ahead of time he wanted to ask about, and that's a particular language that he is a fan of that I've never ever done anything with, but he probably wants to know whether you have support for that.

Ben: Yeah, yeah, I was looking at your, your language and database support on, on the docs, and I'm, I do a lot of C sharp. net development and have for a number of years, so I was curious if there was any roadmap for that or interest.

Darko: There are some people that are running it, so it's I mean, with this Sanford 2.

0 thing that we launched, like, roughly like five, six years ago, we tried to make Sanford as kind of extensible as possible there so people don't have to depend on us, you know, to run run that so it's definitely possible. Definitely possible to run. It's just that our, our roots are in that, you know, mainly that SAS ecosystem, which makes things with open source technologies for the web, like kind of web apps and I don't know, yeah, honestly speaking, there are.

There are a few C sharp projects in that realm. I know that you can run open source stuff with C sharp also these days, but

Jonathan: You can install C sharp. So this, this is my, my guideline for how, how open source something is or not. If you can go to a Fedora Linux install and just install it with yum, then it is, it's got its ducks in a line for being open source.

Obviously, some projects are more complicated than that, and the newer thing, like Semaphore, is not going to be there because it's so recently open sourced. But the point I get to is, you can install NET with YUM on Fedora, so like, it's pretty dang open source friendly these days. Yeah, it just has a stigma.

It has a

Ben: stigma associated with it, where people kind of see it in the same way as old You know, java enterprise apps, except it's under microsoft's thumb and

Jonathan: well, so many of us grew up working on windows XP machines and you have to sit there for an hour and a half and pray the whole time that the dot net three dot x install update would work.

Right? And our days of the net framework. Yes, they were. They were dark indeed.

Darko: I'm a little bit older. For me the, the Visual Studio experience was main, mainly tied to Visual Studio four and Visual Studio Visual Basic four, and Visual Basic six. .

Jonathan: Oh,

Darko: tho those are nineties. Yeah.

Jonathan: Well, I, I have some of those memories, but actually doing it doing it professionally, I, I maintained a lot of Windows XP machines and.

Boy, we would just sit there and wait for the dot net updates to finish. Oh, it took so long. So I have a little PTSD over dot net. All right, so you've got a note here. One of the things that you wanted us to ask you about is one of our most slash least favorite topics, and that's a I can't get away from it these days.

Is there a place inside Semaphore or inside sort of the development that you guys make happen for AI assistance? How does, how does that fit? How does that fit without just being AI slop were added to be able to make our backers, you know, our, our, our venture capital backers happy? What's, what's the story there?

Darko: I hear, I first hear a story, a short story to tell. So we are kind of in the process of booting this project of, you know Revamping part of the CD, how it works in Sanford. So it's, um, much more flexible and then you can model how you are deploying across different regions, continents, and so on. And that's automated.

And so not to get into the weeds, weeds of it. And I'm kind of working on the architecture, you know, this feature design of that, and I'm talking with, I'm talking with AI and then I have, you know, a lot of ideas that I. Paste as like markdown to it. And I say, okay, do you see any problems with it? And so on and so on.

And tell me how you would how you would maybe implement that, you know, what are the other options? And then it's literally starts writing SAM for YAML and saying how it would implement it within the SAM for features as it today, you know, and it turns to be like a hacky workaround, but the effect is the same.

And that's not what I asked at all. I asked the guy, you know, dude. The backend architecture and the system architecture. So for, you know, writing YAMLs and those kinds of things, I mean, yes, we're probably going to bring in some input box, you know, to, so you can do it in the app, but there are a lot of these, you know, smart AI.

LLMs around there that you can use, use to interact with software. Right. An area where, where, you know, people are asking us to do, to do more, more in, and yeah, it's not easy finding the time for that, although it would be hot. It's in the realm of that, you know, agentic AI, if something would.

Observe your pipeline, duration of the pipeline, the frequency of failures, the flaky tests, and all of that. And maybe not do something really hands on initially, but just based on the observations, give some feedback to people, you know, what, what, what they could, what they could do better. It is, it is a kind of thing, which is very hard.

You have to really have a process around it. To be good at that, if, if you are doing it in a traditional way, you have to have some, you know, charts and some people and some weeklies, and you know, to review all the flaky tests, to review why is our build getting longer, who contributed this, you know, tests that take two minutes, are you crazy?

You know, all, all, all those kinds of things. So I'm just

Ben: imagining an AI Clippy saying, I noticed that your build is 40 percent longer after this commit, right?

Darko: Yes.

Ben: Would you like help with this?

Darko: And then there is a picture of the person who made it. Yeah. Yeah.

Ben: Automatically get blamed and sends an email to their supervisor.

Jonathan: Oh, that's, that's terrible. This is the dystopian future that we do not want. So there is, and I've, I've done a lot of thinking about this, but also talking to people sort of in this broad industry. And there's one, there's one use case for AI in particular that I think really makes sense. And that is AI guided fuzzing.

And I'm curious, Just to start with, is there sort of an infrastructure for doing fuzzing inside of Sima4? And then is, is that something that someone could, you know, shoelace and bubblegum together to get some AI guided fuzzing happening inside of Sima4?

Darko: Yeah. I mean, there is no, no first class citizen on implementation for fuzzing within, within Cepher.

That's kind of the, the short answer to that in terms of AI, AI and fuzzing. I also think that that would be super interesting. I mean, not, not related to fuzzing in particular. There is another realm, which is very interesting to me. And, spoken to a number of people that, that share the same views.

So you have a successful 10 year old Ruby on Redis application, let's say, to keep in that, in that world. And the build, when it's spread across, you know, maybe 50 parallel jobs, it takes five minutes. So that's okay. So you can imagine you have a crazy amount of, of tests there. You know, can be unique, but can be also those expensive end to end, whatever.

And we were, you know, graphing through the database of tests, you know, and there is roughly speaking 25 percent of tests that have not failed in the last three years. Are those tests useful? Are those tests even valid? Maybe they, maybe they have been always green. Maybe they have never been red.

Maybe it's

Jonathan: not possible for them to be read.

Darko: Yes. And there is a certain, I would bet, you know, a few bucks that, you know, 5 percent of the tests can just never be read. But the point being that they are adding, they are this taking away from your, Velocity and you know, how fast can you run things and so on?

I mean if I would retire And have nothing else to do but write academic papers. I would work on those, you know tests that that That are just wasteful, you know and how to how to figure out what's the percentage of those? Yeah How how to eliminate them?

Jonathan: And so the the thought here being that that might be something that you could turn an ai loose on and say hey Look at my Everything.

You have access to the CI runs. You have access to the source code. You have access to the glue code that holds it together. Give me some. Yeah, they hit the history of runs, all of that. Pull out, you know, 5 percent of the tests that you think might be entirely superfluous and see what it comes up with.

Yeah, that's that's pretty interesting. Yeah, so, okay, we are, we are basically back down at the bottom of the hour. I'm gonna give Ben a chance to get in any final questions he has, if he has any. And then we're gonna ask Darko if there's anything that we didn't ask him about that he wanted us to.

And that's sort of a that's sort of an interesting question to get, because you have to do set theory in your head to figure out all the things we talked about, and all the things I wanted to talk about, and how much do they overlap. So Ben first, is there anything else you wanted to get in before we let him go?

Ben: Yeah, I was gonna ask like how a semaphore kind of fits into, Like containerized deployments now, how does that and orchestration? I know a lot of a lot of Businesses and, and SaaS products are using technology like Kubernetes now, how does, how does Semaphore kind of fit into that paradigm?

Darko: Yeah, I mean, when it comes to I guess that maybe indirectly you are, you are referring to GitHub, GitOps as a pattern of like having the Spec and description of what what the environment should look like. We don't, we don't deal directly with, with Kubernetes. We are not touching those APIs directly.

I mean, over the years we kind of burned ourselves. We had integration with dozens of, you know. Deployment targets over the years, and it was quite expensive, you know, to maintain all those integrations over time. And luckily Kubernetes came along and there is now somewhat of a standard there but it was also the case that we were busy doing other stuff.

And I would say Flux, CD and Argo did that, you know, okay, here's the spec in my repo. I'm going to apply it. I'm going to do sync. So we never ventured into, let's say re implementing or implementing that, you know, sync functionality when it comes to the, to the Kubernetes. We kind of sit on a more abstract way.

So what you can very successfully do with Sanford with promotions being the first class citizen, you can maintain that those YAML definitions for your Kubernetes within Sanford and Sanford can, you know, update the. Particular YAML file with a new tag of an image. And from that point on you know, Flux or Argo or something else can take over and do that synchronization process.

And SAMH4 can do the check post that process. You know, okay, has this been applied, you know, in a certain way.

Jonathan: Very cool. Very cool. And then is there anything that we didn't ask you about? Anything that, burning that you want to let folks know about?

Darko: Not in particular. What I'm I, as a student, wanted to contribute to Blender.

I don't know if you know that piece of software for, you know, 3D modeling and so on. I was very unsuccessful in doing that. First, I wanted to contribute to that C or C code base. I think it's C. And then I wanted to contribute to the Python plugins. And it was, you know, quite hard. I did some work in the end.

I didn't do anything there. I ended up writing in C my own, you know, clone of space invaders and lived in my own little cave, having fun. So I'm, I'm just very curious, you know, to which extent AI can help people onboard them to sign up for, to contribute something. You know, with those new, I don't know, Claude and, and so AI takes on, does clone on the semaphore and says to the, to the CLI tool or wherever, Hey, I would like to make a contribution in this area to fix this and this in this way, you know, onward me to the projects.

Yeah. Tell me five things that I need to know before I hop in. So, yeah. That's kind of answer question that I'm really curious about, because we do have to work a lot on that, you know, onboarding of people and make it easy to onboard an open source project. Right. There is a big to do there, to do more, we just didn't have time.

So that, that's the area. Yeah. I don't know if you have guys seen something there or what are your thoughts on that?

Jonathan: That's, that's an actually a really interesting point I have. I've made the statement before that when someone goes to work on an open source project that's new to them, or I guess a code coding project of any sort, one of the, one of the first challenges you run into is sort of like.

mentally mapping where the code is, right? And so to put that in more concrete terms, which source file do I even need to go look at to try to figure out how to make this happen? And what you're kind of describing is using AI. As a search engine to find what file to start with inside the code. And that, that's actually really interesting.

I would, I would love to hear people's reports on trying to do that with various bits of source code. That is actually a really interesting thought. I kind of like that.

Ben: Yeah, that seems like a better approach than just kind of using the AI to make a drive by pull request and not understanding what it does.

Which I've seen as well. Yeah. The difference I have no idea what this works, but YOLO. Yes, the difference between write

Jonathan: this code for me and tell me where to look to write this code.

Ben: Yeah, help me understand.

Jonathan: Yes. One of those things AI is very good at. One of those things AI is Not,

Darko: yeah, we will see, I mean, for the, let me just mention the last thing for the open source code.

I, I interacted with, um, open AI API and wanted, wanted it to review each of the source code files. For me, I mean, on one hand, it didn't really give the very, it was very verbose when giving the output, you know, is there something that, you know, shouldn't be there potentially and so on. It also ended up costing a lot.

If I would run the most expensive model on each of the files, it would be in kind of thousands of dollars. So yeah, I don't know how it's, how achievable is this one, or is it my. But yeah, interesting,

Jonathan: interesting. All right. A couple of questions we are required to ask you before we let you go. And that is, what is your favorite scripting language and text editor?

Darko: Text editor, Vim, scripting language, Ruby.

Jonathan: Ruby and Vim. There we go. All right. Darko, Fabian, thank you so much for being here, talking about Semaphore, a really neat project and something that if I ever get a chance to, I'll go check it out, maybe I'll be the one that runs it on a Raspberry Pi and to see how that works.

All right, awesome. Thank you. Yeah. All right. What, what do you think? What are your thoughts?

Ben: Looks interesting. Are you ready to to hook all of our GitHub actions over to simf4 and start the exodus?

Jonathan: I, maybe let's start on a small scale. Start small. Take one of the small projects and just do a test run on it.

Yeah, I don't, I don't know. For some projects, that might make sense, particularly when you get into the realm of self hosting. You know, a real small project, self hosted a runner on a raspberry pi, maybe that is the way to go. If there's something that GitHub Actions can't do. And then I can also see, obviously, a really big project or a really big company.

They want to be able to move all that stuff in house and not fiddle with. The, the GitHub runners at all. So there are various places where that makes sense. And I really have the feeling that the more, the more people from the community sort of bang on this and add stuff to it, the more it's going to make sense for all of those different use cases.

Absolutely. All right, then. Thank you for being here. Is there anything that you wanted to plug, let folks know about?

Ben: Sure. If you want to check out any of my open source work github. com slash thebenturn. That's my handle. And most of my work is on the Meshtastic project. So if you're interested in communications and off grid technology using LoRa meshtastic.

org. It's a really neat project. It's really fun.

Jonathan: Yeah, absolutely. Appreciate it. If you want to find my stuff, most of it is at Hackaday these days. We appreciate Hackaday being the home of FlossWeekly. That's also where my computer security, not just computer security, but my security column goes live every Friday morning.

We occasionally dive into physical security and lockpicking and all of that fun stuff. So you can check that out. And there's also the Untitled Linux Show, which is still over at Twit, twit. tv. I think it's twit. tv slash ULS. And you can find that to stay up to date with what's going on on Linux and hardware and open source and all kinds of stuff we talked about there.

Have a lot of fun. We appreciate it. Thank you everyone that's here that caught us both live and on the download and we will see you next week on Floss Weekly.

Episode 825: Open Source CI With Semaphore

19 mars 2025 | 68 min

Episode 824 transcript

12 mars 2025 | min

FLOSS-824

Jonathan: Hey folks, this week Doc joins me and we talk about personal AI. Sort of a wish list for what we want that to be and then some thoughts about how to get there. We look up a couple of projects that may get us there if you're willing to put the pieces together and more. It's a lot of fun, you don't want to miss it, so stay tuned.

This is Floss Weekly, episode 824, recorded Tuesday, March 11th.

Hey folks, it is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host Jonathan Bennett. And today we've got a co host and a guest who are the same person. It's because we've got Mr. Doc Searles with us. today. And we were, we were scrambling for a guest today. And I said, well, I thought I had one, but then it turned out he couldn't do it today.

And Doc said, well, you know, I was supposed to be at scale and do a presentation and I didn't get to do it because I had some medical thing come up. I believe you said you're fine by the way. It was just one of those things.

Doc: Yeah.

Jonathan: Yep. And I said, come, let's do the, let's do the presentation you were going to do at scale.

Let's do it on the show. And he said, Let's do it. So what are we,

Doc: what are we talking about, Doc? So here we are. Well, the, at scale, there's a, there was also the QI summit. QI is an open source, um, project toward personal AI And I'm involved with it it has like seven or eight hundred volunteers at this point.

It's got some code mostly working on an OS. There, but I don't want to go into that because I don't know enough about it. And I think it's beside the kind of points that I wanted to make in the talk I was going to give there.

Sure.

But I couldn't do it, you know, so and and I thought well, I'll just give the talk here But I I ended up having this really interesting Conversation with chat GPT about what personally I is and should be There was a leading conversation in the sense that I I kind of urged in the directions.

I wanted to go You know it now Yeah,

Jonathan: the skeptic in me wants to say that having a conversation with one AI about another AI that's sort of gratuitous Navel gazing.

Doc: Well, it's in a way except the the The baby that produces the navel hasn't been born yet. So it's there's, I mean, personally, I, and I've said this a bunch of times, I think personal AI in 2025 is pretty much exactly where personal computing was.

50 years earlier in 1975. A really good idea? We don't have it yet. I mean, there's some examples of it out there. I think it's 75 by that time the Apple 1 was out and the Apple 2 was on its way, and in the next several years we had the TRS 80 and the Sinclair and the Osborne and a bunch of others.

But, It was, it was, it was hardly thinkable. And, and so we tended to think of personal computing that as well. How do we put a mainframe on your desktop? Right. And, and it's the same way now I'd say, well, how do we take corporate AI and make it personal for you? And then we're still thinking inside the corporate world, we're still thinking.

And we're getting a difference now, of course, is that there was no access to computing in 90, 95, 1975 rather nobody had any access to it unless they worked for a big company or something like that. And they were doing batch processing with cards and stuff like that and, oh, the Altair was out and you could.

Program it with switches.

And, you know, snap a tape, a paper tape or something. And that was personal computing. But that's what got Bill Gates going on it. Like he saw Altair and said, I have to do this. And he dropped out of Harvard to do it, and the rest is history. He did rather

Jonathan: well for himself. He did rather well for himself

Doc: with that.

And but the I think there's a sort of a paucity of imagination about, it's not even imagination, I think we're just not considering. we need from a personal AI that we're not going to get from the corporate kind. Starting with the fact that it's yours and it's private. The, the main thing for me is the data.

You, all of corporate AI has been trained on, on the entire web. Plus lots and lots of books lots of other things. It's a massive amount of stuff that is trained on the models are great They're not what they're going to be but they're you know, they've used that data and a lot of programming and logic to Do a really good job of speaking in English or whatever language you want to chose and making a lot of inferences and for one thing or another and Emulating humanity and and human conversation and stuff like that that we're all very familiar with and to the point where it's very You know, very useful.

And in fact, I think they've done a terrific job of leapfrogging Google. I mean, Google should have been here five years ago. They're not, they weren't, but we're, we've replaced this again. We're in the corporate world right now. We've replaced searching as if you're looking through a library with Q and A, with query, with queries, you, you don't need to look in the books in the library, you just ask the library.

You know, a question and, and it's because Sears goes and looks through the library and gives you an answer or a collection of answers that are quite credible and useful. And you go past that and say, great library, now, you know, do my term paper for me and it'll do that. Right. So, it's, it's very handy that way, but we have these sort of two modalities there, which is one is search and index.

And the other is ask it a question and and chat GPD kind of modally ask it a question, but now Meta and Microsoft and Anthropic and a bunch of others have come up with alternatives that are all roughly the same some are better than others. I like perplexity because it gives you sources.

But again, it's looking at a giant index and. And give, and answering questions. The key piece here for personal AI is, how do we do the same thing for our lives that corporate AI is doing for the world? And and our lives are. Our contacts, our calendars, all our health records, all our financial records, our obligations, our recurring payments when we bring the garbage and the recycling out to the curb what do we pay for that?

Who do we pay for that? Our travels. Where were we when we had lunch with so and so? Our, our work, you know, I mean, keeping records of our work, especially if, and you and I are both kind of solo operators in some ways, we have consulting businesses. When do we do such and such for so and so? What do I owe them?

What did they pay? When, how long did it take them? I mean, all this kind of stuff. And We don't have that yet, and we don't have it in coherent form, and an awful lot of the data that we would need is in the hands of people who are not us, okay? So for example you know, so, we're watching White Lotus and Reacher right now, and and we're watching them on our new Samsung TV, which has, and we have an Apple Gizmo that hangs off the side.

And how does, you know, and I don't remember where we were with either of these, you know, we haven't watched them in a couple months, how many episodes are we into this? I don't want to just do a Q& A on that. I want to look at something. I want to look at something that's like a spreadsheet that I can refine, say, okay, give me a list of all of the subscriptions that I have to streaming services, and that includes, in my case, SiriusXM on two cars and portable devices.

Plus Sonos, I've got that. I've got but all of these streaming services and, and when, when, when does it come up for renewal on this? When, I mean, all of these things depend to some degree and you're not knowing something, but the TV knows. All your new TVs come with spyware in them, you know, I mean, you need to find where the camera is on it and put a black tape over it because it's there, you know, I mean, and I was even looking up on ChatGPT or whatever, how do I turn off the surveillance on our new Samsung OLED TV and what they told me was current like three weeks ago but Samsung has changed it and now it's something else and they want to make it as hard as possible for you to turn off the surveillance.

Of course. And they all do this but that's that's another gripe but I'd rather have is show me what I've watched and when and there are countless shows and we don't even watch that much TV we all watch some that I don't remember. We'll watch two episodes, and wait a minute, we've seen this. You know, it's, you know, you go into that stuff, so it'd be nice to have that.

That's a fairly trivial example. But what's happening in the world right now is that old fashioned over the air TV that got replaced by cable TV and satellite TV, both, all of that is being replaced by subscription TV. Where For example, if you're a sports fan, you're, oh my gosh, you know, the NFL or the NBA could be on any one of five different services.

There are all kinds of conditionalities in there. We could use personal AI to solve this. Now we could go to ChatGBT and ask it questions about that and probably get adequate answers. We're not going to get personalized answers. The personalized, or if we get a personalized answer. It's good. I hate the term hallucinate.

I'm sorry, BS is the right term here because that's exactly what it is. It's making something up it hopes you believe, right? That's plausible sounding, but it is BS and And it does that, and it does it very innocently, like a butter wouldn't melt in its mouth when it does that, and But, but rather than critique what we have with corporate AI, let's start building personal AI starting with the data we need to be in control of our lives.

Not just know stuff about it, but control it. You know how, you know, now I'm old now, but I mean, even back in 2008 when I went, when I started in the Harvard medical system cause I was hanging out there at the time and I could get into it, and that's such, by the way, exclusive to Harvard. It's like I'm in the Indiana University system here and it's called IU Health and it has nothing to do with the university anymore.

It's a separate thing. I don't know if Harvard's like that now, but my point is. I could send a request to 19 different prior healthcare providers and get back data from them. What I got were scans of faxes and copies of scans of faxes. They're upside down, right side up. Sometimes it looked like they were done by Da Vinci because they were written backwards because they were like the mirror.

I mean, it was beyond useless. I mean, beyond useless. Anyway, my, my point about this, well, AI might be able to make sense out of this, right? Okay. Give me all that crap, there's copies of faxes of scans. Do OCR have a model that understands this kind of stuff? Apply to it. I would like to be able, I don't know if you've noticed this with Amazon if you buy stuff on Amazon, it kind of breaks out, if you look at your account records, by shipment.

But Your visa bill doesn't line up with that. The dollar amount you pay may be the same, but they're not the same because the visa bill says 45. 23 and you have, you know, a 13. 45 on one and 17. 65 on another. And you'd have to start doing algebra or something or make your own spreadsheet to figure this out.

Whereas NAI, smart enough to do this. Okay. I recognize that as an Amazon bill. I know these conditionalities that are embedded in this. I, I see this over here. These match up. Okay, we'll, we'll work that out. You know, our property. I have, I have books here in Indiana. In my new bookshelf, my new old bookshelf.

We have nothing new. We get it used. It's like this in Oklahoma too. I mean, I'm sure. If you go to a thrift shop in California, you're paying 10x what you'll pay in Oklahoma or Indiana. Yeah, that's true. It's just true. I mean, when you, it isn't the sticker shop, when you go to the coast, you get horrible sticker shop.

It's like, wait a minute, the dollar doesn't work here because this is much higher. And when you go from the coast to like we did in Indiana, it's like, Wait a minute. This, this state is on sale. Let's, let's live here. So that's what we're doing. But my point is that we, you know, I want to be able to take the phone and I want to point it at the bookshelf.

It's got very high resolution on the phones now. And I, and take a picture of the spines of those books, have pattern recognition happen and have. A list of all those books, right, and I do that in California where I have books, in New York where I have books, and Los Angeles where I have books, and shoot those, and here's a list of all the books and where they are, okay, I would like to know that, and, I mean, the list goes on, I mean, our travels, okay, Google Maps and Apple Maps and, and you know, All of those know where you've been.

They're, they're tracking you. Your car is tracking you. If you have a late model car, it has a cell phone in it. You don't even know the number of that cell phone. It's known by Toyota or Chevy or whoever it is, and they're busy selling information about you to insurance companies because that's an extra way they make money.

We need that data. We can use that data. We can make much better use of that data than the insurance companies. that they're selling, I'm sorry, that they're selling it to me, anyway, so I've got

Jonathan: a question for you then.

Doc: Go for

Jonathan: it. So some of these things that you're talking about like already exist, let's just say photos for instance.

So when I take a photo on my Android cell phone, it all goes up to Google Photos and then I can go to Google Photos and say, I want to see pictures of. Weather or I want to see pictures of fireworks or I want to see I want to see tech pictures computer pictures You know and it's got an AI it's got pattern recognition in there and it'll do that search and we'll get it back to you The problem well multiple problems But one of the problem that I have with that is the reason that Google does that is because they also have a clause in their Terms of Service that you give us the right to use your pictures in such and such way and they also use that data to train their AI, right?

So like there's strings attached. And then the other problem is, you know, that's one service. And then if I want to say, well, where have I been over the past two years, that's a completely different service. Those two don't talk to each other correctly. And I'm curious And they're not working for you. Well, that's the other thing.

They're not working for me. My data is the product. I am not the customer, right? And so the question is, how do we get there? Is there a, is there a path where we get from Google is the AI and I'm the product, to I own the AI and it's the product?

Doc: I think we start with the low hanging data fruit that is actually ours and we can, not already sitting out in clouds of various kinds.

So So I mean like the book example is one of them. Our own personal property is another. Our, our purchase records are another. Our contacts and calendars, even though most of our calendars and our email, I hate to, even that's the case are, you know, I mean, your email is yours, even if it's a Gmail, okay?

It's still yours. You could, you could take that. Thank you, SOTP and IMAP. I can copy that out.

Yep.

I can. I can, I can move it to some other service. I don't have to have it there. There's let's take the data that is. or enough hours and where the gating factor for using it does not require that we go into Google or that we shake down Chevy and Nissan for the data that they've collected about us.

That's more of a longterm thing. And then prove and then, and then prove the utility of that data to us individually. That makes us, you know, better human beings, citizens, customers, workers, Parents, whatever else is involved in that, you know, the, because there are many different, we all wear many different hats and play many different roles.

But I mean, you know, I mean, even like on the parental side, you know, you know, what sports teams are the kids on? What are their grades? What are they doing in school? Where, where, you know, where are they? When did they do this? Right. There's a list of their friends. When are their birthdays? I mean, there's that kind of stuff that's of interest to us, but not necessarily to the whole world.

But anyway, once we prove some of that out, then we can go after the higher fruit. Sure.

Jonathan: I'm trying to, I'm trying to figure out, I guess, the form factor? Like, what, what makes sense? So what, what, what would this look like? Are we talking about, you know, Western digital or Synology building a, a NAS? A little, you know, a home use desktop NAS, network to stack storage, that also has a GPU in it?

And then you put all your data in that, and it's going to also index it, and you've got your own little AI that lives inside that desktop hard drive that you can ask questions of? Is that sort of the, the form factor? That is my current fantasy.

Doc: That's my current fantasy, actually. It's I wouldn't call it a fantasy.

That kind of exaggerates you know, what, what it might be. I think, I think we need our own, You know, it's not, I mean, let's say that what, I'm not sure everything I'm talking about, I've been talking about, requires all that much compute. I'm not really sure it does. But let's, let's say it does, it requires a lot of compute.

And the, and the you know, the, the hardware that you have in your laptop isn't going to cut it. You know, I'd, I'd want a separate device. When you, I mean, you could. Certainly get from Synology or any one of those companies, you know, some kind of box that'll sit out there. I imagine somebody will make an appliance at some point that'll do the same thing, that'll have GPUs in it that are optimized for, for AI kind of work.

But in a way though, we're before the beginning of this, right? All of this stuff is on the table. That's why I'm just sort of focused on Actually, before we even get to how the processing is done, let's look at what it's working with. And, and by working with it, I just mean our lives. I mean, some people have almost no property, or they rent it all, and they don't care about the property.

And other people, that's just an enormous part of their lives. And with other people Travel is almost everything or it's kids or whatever else or work. I mean you work all the time, but We go from but you know, most of us go from one job to another And what do we want to take with us? Is it only in LinkedIn where you're basically?

You know, exaggerating or you know, whatever to those people. I mean, it's, I mean, LinkedIn, I mean, if you're looking at a way to share employment history and, or even know what you've done in the world, what a horrible interface to do that. It's certainly, I mean, we're all social. To me, my, you know, the contact, the contacts I have on my phone and my computer are far more a platform for.

Being social than anything I have in what's called social media. I mean, I mean take Twitter. I mean, I, you know, I followed, you know, a few thousand people on Twitter back when I was active on it. I don't know who most of those people were. I don't remember at all. You know, I sorted them out. Oh, these are journalists.

These are all my open source contacts for Linux journal when I was doing that. It was an enormous number of people. And, but that was, that was on Twitter and it was a list on Twitter. I don't even know those lists are still there. I've, I've no idea, I've just haven't been much on Twitter in, in recent years.

And then, you know, Blue Sky and the rest of it is kind of trying to replicate this really inadequate, horrible writing platform that we had with Twitter, and, and do the whole thing over again. It's like, wait a minute, oh my God, it's like, we could do better than that. But my point is that I, I actually think that if we hit, if we're on top of our lives at home.

We are going to be better at doing the things that we do as people, you know, whether it's this as simple as knowing who else to go to church with. I mean, it's all, all this stuff is, is, is personal for us and it matters to us, but it's not, it not only isn't, but shouldn't be of that much interest to.

The Googles of the world, except they're just trying to give you better advertising, which I also think is a flawed model that's going to die in the long run, too. We just haven't, it hasn't died as fast as I wanted it to, but it's just, it's a pretty lousy way to run an economy.

Jonathan: Yeah, oh I've thought about that.

One of the, one of the depressing conclusions that I've come to is pretty much as long as we've had technology, it's been, it's been paid for by advertising money. I mean, that's how they made the soap operas back on the radio back in the 30s. This, this show is sponsored by Clabber Baking Soda.

Doc: Yeah, what, what I love about podcasting right now is, A, I mean, my favorite line about podcasting is wherever you get your

podcasts.

It turns out RSS is something everybody could use. It costs nothing. And and the other thing is I've got a little thing on every different podcast app I could use that says, Go forward 30 seconds, you know, and I can skip over that stuff. It still helps pay for it. But we had the same thing with, you know, VCRs and DVRs and we could skip over the ads on those too.

We just spool stuff and skip over it. But I, this is a side thing, but I, it, it always bothers me when somebody says, you're not saying it, I'm not saying you said it, but I hear this a lot, you know, the internet is paid for by advertising. No, it's not. Google and Facebook are paid for by advertising. They are not the internet.

The internet runs a TCP IP. If you look at the entities that pull the wiring through, we're now on a two two gigabit fiber connection here, by the way, this is very cool. It's brand new. But I'm not quite on it. It's sphinctered right now through a Wi Fi hotspot that will do more than 600, 600 megs.

Never mind that. I, I don't, I don't have the router hooked up yet that I'm going to get, but I'm debating which one it's going to be. But but the The, you know, I'm paying gigabit now, Indiana for that. Okay. They're, I'm not, and they're not paid for by advertising. They're paid for by rate payers. Right.

Before that I had Xfinity. I was a rate payer for Xfinity. Xfinity pulled the cable. The coaxial cable that I used and they had mostly cable TV on that. They changed their business model to internet, you know, primarily, it used to be cable TV with internet gravy. Now it's internet with cable TV gravy, but it's, but it's still rate paying either way and, and, and all the biggest operators.

There's no longer a phone system in there that is all about billing every, across every border crossing. That's, that's the great thing that TCPIP did. It's like, I mean, suddenly all of a sudden all the phone and cable companies are looking at each other saying, holy shit, we don't have to pay each other for passage.

Because it's not, because the TCPIP doesn't do that. And that, I mean, that to me is one of the greatest open source successes in, in, in world history. And same, same thing with HTTP, you know, we did, Adobe wasn't telling us how to, how to, how to write, you know, there's this open standard where you can just write in HTML and that's, that is fine.

Anyway, um, I think that there's, I mean, I think something analogous is going to happen. At the personal level, when, once we, whether, whether we have those boxes, does the box come first? Does what we do in the box come first? I don't know, I know, I mean, Apple is building into their, you know, M2, Chipsets that some GPU ish kind of stuff, but it's all for their purposes And and what they have right now, and I'm using Apple stuff here.

I hate to say but there it is Their Apple intelligence is all candy. There's nothing nutritive in it at all It's I mean I not not as far as I could tell and and what there may be this Nutritive is that they've got a kind of chat GPT back in there's no imagination about what you can do As an individual when they're probably among the bigs the best position to do something with that Yeah, but they're not going to because they're I don't know why they're just be there who they are But I think I really I mean, here's an interesting thought experiment what would have happened if AT& T had opened Unix In 1983 or in 1982 or one, you know when, when it literally, it was, it was Ms.

Doss, there's a CPM, I mean, an Ms. Doss beat CPM because Doss is cheaper. You know, Gary Kal wanted to charge you a thousand dollars a box or some crazy thing like that, and somebody will come in and say, no, you're wrong. SRLs, . I probably have wrong. So anybody who's listening and correct me after the fact, but , uh, there's.

You know, but they wanted to charge for operating systems, but what if, what if AT& T had a brain back then and said, you know what? We should make Unix free and open. What would have happened? Oh yeah. Right? Holy, holy Christmas. I, well, I think there's a similar moment right now for open source personal AI.

Yeah. We don't have open source in corporate AI. That's not there. Open AI had that sort of mission in the first place and then they. Screw that, and

Jonathan: Yeah, well, so there Boy, a lot of different directions, so I'm going to go with that. There are some open source AIs. Although, for various definitions of what it means for an AI to be open source, right?

But like, for example, DeepSeek, all of the models on that is available under the MIT license. You can download, now you need a little bit of a Horsepower y video card to do it. Yeah. Or an accelerator, but you can download the DeepSeq model and run it locally, which, you know, in and of itself is really interesting.

I have been playing around, not with not with those sort of LLM models, but I've been playing around a lot with image generation. Because I find that really fascinating. Yeah. And I run that. The first thing I found was a website where you could do image generation there. And you know, one of the first things that I was looking at doing is, like, creating characters for tabletop roleplaying games.

That was something I was into and I thought, well, you know, if you could make a portrait that was, cause I, I, I had this whole digital system for doing it for, for playing like a tabletop game virtually. And it came to me like, well, if you could make a portrait, that would be really cool. And so I got into that idea.

And then, you know, here recently I got to look and it's like, well, I've got a reasonably decent video card in my desktop. How much of this is available in, in, you know, open source, where you can actually use it locally? And I found that there are, they're like, Hugging Face is one of the big names.

Hugging Face is there you go. Yes, yes, it's, it's a whole bunch of open source models, and you can run them locally. And then they've got plugins too, and so one of the plugins they've got is a a, a plugin, I forget what it's called, but it'll, you can give it Pictures and it will learn to recognize a face and then you can say okay now Give and people use this for nefarious purposes, right?

Don't do that. That's not what i'm talking about But you can say okay. I want to see a you know, show me a picture of A barbarian standing on the field of battle.

Doc: Yeah

Jonathan: and put my face on it, right? You can do that sort of thing. You can do it locally on your own video card and so what i'm thinking about with that is You talk about the other revolutions like we're we've we're all here Enjoying the fruit of the personal computer revolution.

Well, how did that start? Well that started with just a few people that thought hey What what happens if we take this this nifty new chip from intel and we put it into a little tiny box with with? Switches on the front and they built some of those and you had a little niche of people that said that's the coolest thing I've ever seen here take my money and the next thing you know, you know, we've got Dell and IBM, IBM, the IBM PC and eventually, you know, Dell and HP and everybody wants to make them.

I think maybe we're at the same point with personal AI where right now you have just a small group of enthusiasts and they're in this niche and in some ways, maybe we're waiting for. the next IBM to come and say, well, what if we turn this into a product? You know, what if we make this personal AI thing into a box or, you know, do something that makes sense and put it on people's phones?

Doc: Well, an interesting thing that chat GPT does in these dialogues is it answers every question with a question at the end. It says, well, have you thought about this? Like, you know, what, what is it going to take to make demand happen for this? And that was one of the questions it had, I think. And. And I said, we need the invention that mothers the necessity, right?

That's, you know, we need the thing, you know, I remember it was actually with the, I mean, I had a VIC 20 briefly, but which Lin has also had, and it went somewhere with him and never did with me, but the, but I remember when I saw the Osborne, the Osborne looked affordable and I thought, Oh my God, I have that, you know, it's like, it's a.

You take one look at it and you have to have it. I mean, one reason that we're using iPhones today in this household is my wife took one look at the original iPhone and said, I can work with that. Yeah. You know, I mean a funny thing about her is, is it. She, when she was in the third grade, I think she was second or third grade, she's at this Catholic school and she said, she remember looking at the nun and pointing at the nun with her, with her pencil and her head saying to herself, kids like seven years old, I can work with this.

That's her mind, right? That's great. I, I, I came to that at about age 30, you know. But Yeah. Yeah. Yeah. But that's, that's, that's what we need. I mean, I think that's, that's one reason I think we may need the box. We may need the box that is a Linux box that's independent of, of Apple and Microsoft and everybody else that's full of GPUs or whatever else it needs that you could suddenly throw a bunch of data in and have fun with it and do cool stuff that's AI ish.

But probably, I mean, with, I mean, just put another card in your desktop, I suppose, if you have, have one of those where you still have back planes in some worlds, you know, if you have that, you can do that. But it's, we, we, we need some things, I think, that will spark imagination and, you know, toward what we can do here.

You know, everything

Jonathan: you've, you've talked about, it, it kind of, it strikes me as really the problem we're talking about is search indexing. But for our stuff, not for the internet. And in some cases, this is indexing physical things. In some cases, it's indexing digital things. It almost seems like the big problem that we're trying to solve here is just a personalized index.

And I know, you know, like, even in Even Linux desktops like KDE. We've had three or four different file indexing tools, and they've all been terrible They've all caused problems, right? But like that's sort of what this boils down to is a really really good Indexing tool and then plug that into an AI somewhere which again is not much more than a good search engine and say hey I want to see my receipts that match this criteria.

You put it in natural language, very much, very much like how Google has worked for 20 years. Right, yeah,

Doc: no, the natural language, well understood at this point, having a sense of that. So, a couple thoughts about that. First, what Google did and if we go back to 1998 when you were two years old, I was 11.

Thank you very much. But I mean, back then the, the understanding of the web before Google came up with its index, and this is the Yahoo model,

but

the others came at it the same way

was

that this is, It's a Linux, this is a Unix directory, right? And it has directory paths and it's, it's all files and paths.

That's what the web is. It's all files on paths. That's what a universal resource locator is that I'm going to locate that. It's in a place. It's in a, it's a file on a computer somewhere. And I'm going to catalog it. That was the original idea. We're going to catalog this, and that's where Yahoo came from.

We're going to catalog it. They hired people to like basically look at the web as a, as a library, and what Google did instead was saying, we are going to look at all of this as unstructured. We're going to, we're just going to, we're going to, we're going to, we're going to make our own catalog, but it's not, A card catalog like the library, it is going to be this index of, of, of data that comes from these links, these locations, but we'll get you back to those locations by looking through this index.

So where I'm going with that is, I'm not sure if I take all of, I may be wrong about this, I'm just thinking out loud. If I take all of my, you know, health travel work. contacts calendar, schedule, property and look at it like, or have a, an AI or whatever feeds the ai, whatever the base pile of data that le, that the AI is gonna look at.

Mm-hmm .

And look at it as, this is all unstructured actually. And I'll leave it up to the AI to figure out what's a photo and, and what's a bill. And know the difference or at least have enough association so it can recognize the difference. And work that out for me, in which case that's kind of like a search index.

But it might, but I'm thinking there might be some other breed here that's not either the catalog model. Or, meaning the, you know, the Unix path model, nor the pile of data index model that Google pioneered. It's something else, and I don't, I'm not technical enough to go to where that might be, but I suspect that unconstraining ourselves from those two models might be better than saying, let's just do it like, A search index, but there's probably ways of indexing and ways of cataloging and ways of making sense of data that I don't know when I'm making shit up right now, so.

Jonathan: You know, so I'm struck by this question. And of course. Google knows the answer or more more accurately a bunch of nerds on reddit know the answer and that is Can you can we even do this now with pictures? Could I? Upload all of my picture. No not upload Sorry, could I dump all of my pictures on a folder on my computer and then start something that runs?

Probably on my GPU locally that'll go through And tag, and give me a description of each of these. And ideally try to do some, some OCR and actually get data off of it. And there is at least one project that I'm immediately coming up with, it's LLMII. LLMII. It's the LLM index or no image indexer. And I will, oh, cool.

I will have to try this out because like, obviously there's more, there's more data here than just images. That's a, that in and of itself is a really good starting point. And yeah, and I'm curious, I, I really wonder whether we have. To, to, to do, let's, let's, again, let's just say just images. Do we have the pieces out there to be able to do this?

Do we already have the pieces out there to say index? All of these photos that I give you that may include scans of receipts that may include, you know, scans of bills, pictures I've taken with my phone, the pictures of my bookshelf here, my, my local L-L-M-I-I, my, my local LLM powered in, in image indexer.

Wow. That's hard to say. Mm-hmm . Take all this data, ingest it, and then is there another piece that may even be DeepSeq, you know, you could, because you can run the, like, like we said, you can run the DeepSeq, DeepSeq model locally, can you then connect the two, and be able to query the one, and let it use the pictures as part of its data set to look at, and it may be that those pieces are already out there, and there's just not yet a guide that somebody has written on how to make it happen.

Doc: I think so you're probably familiar with LM studio. Are you familiar with that when you had LM studios? Basically lets you run lots of different models just and I don't think it it runs on Linux, but it's not open source I don't think because the service but it's a way to say I want to run D I'm gonna run deep seek on this one.

I want to run this llama on that one. I want to you know, I got that kind of thing I'm sure what you're talking about exists in some ways. What we need is the listeners to the show, you know, to, to come up with the, the form of this that is user friendly, that's, that's muggle friendly. It's not just for wizards, it's for muggles.

Well, that's

Jonathan: always, that's always the challenge, right?

Doc: Yeah. Yeah.

Jonathan: I know. I mean, I, how many, I mean,

Doc: I've been, I was around this for, since 1996, okay. So. What is it, like 30, 29 years, okay, that you know, well, just, just open a shell and do these things, right? So. And 99. x percent of people aren't going to do that.

And just, so

Jonathan: just buy a Raspberry Pi and install this on it and then log into your router and forward ports and then go buy a DNS address and boom, you're running your own website and everybody else. You want me to do what's a raspberry? You want me to bake a Raspberry Pi?

Doc: Yeah, exactly. Exactly. Want me to close my windows?

How will I have to walk around the house? Yeah. But yeah, I mean, we, you know, we. We need, we need the invention that mothers that necessity and I'm sure most of the you know, I mean as I was saying earlier the the the reason you know that the resemblance between now and 1975 ends with the simple fact that What we didn't have in 1975 was the power of mainframes at all, you know, it wasn't available.

So we had to like go from the bottom up and build everything from stolen parts, as Wozniak did from his work at HP. And we're not there now. We can, you know, all the mainframe stuff is being, you know, going bleh out to us. Thank you, China, for DeepSea, you know, it's fantastic, you know, so, so we have so much more to work with there, but I think they're, you know, I think the, the main thing to realize is that we're going to have so much more market activity when we're done.

All the individuals are far more equipped than they are now to do things that they can't do now. And I mean, that's what happened with PCs. It's like, I mean, if, if you go back to like 1982 and asks an MIS director, because they called IT directors that then you know, ask your MIS director, could you mind if we bring some PCs into this company and say, no, no way, we have mainframes.

We have guys in white coats on raised floors. We can, we will do it our way. And then within a year or two, They're all running PCs all over the company because it turned out that the workers could do more with PCs than they could with the mainframe and it's There's we we need a similar moment here where and it's not just inside enterprises because in a way that's a bad example Is it I mean?

it's because Bob Frankston and and Dan Bricklin came up with spreadsheets so they could do math work, you know that business changed utterly because we had spreadsheets now, right and You know, there's there's You know, we need people Scratching personal itches to pull and basically to pull their lives together how do I how do I make a better sense out of my life than I'm making right now and And and we have to and once but we have to prove To The, the big makers of the world that we can do more with our data and data about us than they can.

We don't have that right now. I mean, the example you gave earlier about, well, geez, you know, I could dump all my photos into, into Google photos and I can query against that. I mean, it works the same way with Apple, you know, give me clouds or give me, you know, whatever. And they do facial recognition. You'd say, this is Joe and, and it'll say, here are all the photos with Joe.

You know, but again, it's in their system. It's using their, you know, their off world intelligence to, to make sense of that, where, but there's some kind of breakthrough we need on our side that It isn't so much that it proves it to the world, it proves it to ourselves, and then we generalize from there.

You know, I just think there's so much that happens starting with individuals that can be generalized rather than starting with corporations that could be individualized.

Jonathan: Most of the, most of the big breakthroughs and the world changing things have kind of worked both ways. Like they start, usually the very, very beginning is with individuals, where somebody has an idea, they make a, they make a proof of concept.

And then you, you get some company that sees it and turns it into a product. And so it kind of starts on one extreme and then it zips all the way over to the other extreme. Yeah, I know. And then the candle sort of burns from both ends. And you know, the next thing you know, everybody's got a computer on their desk.

Doc: Right, exactly, exactly. And they have to have it. I mean, it's, it's I mean, it's amazing to me that. Almost everybody is walking around with a 500 or more phone in their pockets, right? And but they're, they're extensions of ourselves, you know, we need them to extend ourselves. But then again, they're also tentacles of Google and Apple.

So that's another problem that's not as well recognized as it needs to be. I mean, you know, and Apple glosses it a bit by saying, we give you privacy except with us, you know, but that's in certain, certain ways that we're not fully explaining. And

Jonathan: Yeah. Fast. Fascinating stuff. I assume you, speaking of Apple, I assume you, you have watched the the, the UK snooping charter and its repercussions for Apple and all of that?

Yeah,

Doc: it, they did the wrong thing. I think that they should have, I think they should have told the uk No, we're not gonna do that. We're just not. And and seen what happened, but they didn't. I think

Jonathan: that's, I think that's kind of what they did, though.

Doc: Did they? I don't know. I haven't, I haven't followed it that closely.

I mean, I know that, I mean, didn't the UK, the UK wanted backdoors or something like that?

Jonathan: So, so the, the issue is iCloud backups and Apple rolled this feature out where you could have end to end, like actually truly 100 percent encrypted iCloud backups. So when, you know, you took a picture on your phone, it got encrypted on your phone.

It got sent to the iCloud server, still encrypted, and Apple doesn't have the encryption key to decrypt it. It's only on your phone. Right. And so this was a new feature that they rolled out. They had a fancy name for it, like a extended encryption or extended security. I forget exactly what they called it.

And the UK said, that's a really nice feature you have there, but we're going to make you put a backdoor in it. We want to be able to decrypt these iCloud backups. And that was a couple of months ago, apparently that they, they notified them of that and Apple pushed back on it. And then the order got leaked, right?

Like this was a secret order. It got leaked. By Apple, obviously. And then Apple said, okay, in order to comply, because they are required under the law to comply with the order, we're just going to disable this feature. That's right. For UK users. And so, on one hand, yes, they did, like, decrease the security of UK users, but on the other hand, they didn't actually put the back door in, which You know?

Right. One of the, one of the important things to remember here is if you put a backdoor in your software, it's there for everybody. Right? Not just for your UK users. And the way the, the way the Snooper chart, snoopers charter, and the order was written you know, it, it did not even tell them. Make sure this is only for UK users.

No, no, no, the order said for everyone. And so apple is basically as far as I read the situation apple is taking the first step of If you're going to make us do that, we're just not going to do any business in the uk

Doc: Yeah. Yeah. And

Jonathan: that, you know, that's a, that's a pretty hard stance to take. And they're not there all the way, like they're taking just the smallest step in that direction that they can.

But that's, that's where this is inevitably leads. If the UK government does not back down Apple, apple has this option of saying, okay, fine. We don't have any brick and mortar stores. We don't sell any devices there. We don't have a, a Nexus in the uk and therefore we're not liable to follow any of your laws.

Doc: Yeah, except they have a lot of stores in the UK. Well, they

Jonathan: do at the moment, yeah.

Doc: Yeah, I don't think they'll I mean, it would be interesting if they pulled out I mean, um, Google pulled out of China at one point. Yeah. And it was a gigantic deal, because they didn't want to do what China wanted them to do.

This was a while back. I doubt, I doubt they would do that, but I don't know. That's an interesting question. I've I've, I've always been more fascinated and knowledgeable about the weird things that Apple does. You know, I mean, like, I don't know why they actually have this thing on their apps where it says you can ask the app not to track.

Why not tell it you're not going to track? Just prevent it. You know, just prevent it. Tell them. Tell, don't ask. Tell them. But they don't. They ask and they say, well, what does that mean? I mean, it means they don't have to Obey your request. It's a request. It's not a demand, you know, but they advertise it like, hey, look, I blew up these spies.

They all turned into vapor. No, you didn't. No, you asked them not to spy. They're still intact. You know, all those molecules go here, you know. It didn't make sense to me. You

Jonathan: think you, you think you dropped the Moab on them? No, no, you just put a sticky note on your door. That's all you just did. Exactly.

Exactly. Yeah.

Doc: Yeah. Yeah, it was, I dunno. So

Jonathan: the with the, with the backdoor thing one of the. He's actually a cryptographer, but I followed his thoughts on it. He was one of the guys that broke the story, I think, and wrote, you would ask me that now, wouldn't you? I'm sure I can. His name is, his name is he.

I don't remember. He's a, he's a professor and a cryptographer, but he, he had this idea that I thought, I don't know that it's going to happen, particularly not in our current, like polarized. Political climate, but it was a brilliant idea and that is We should just pass a law in the u. s That says that u. s companies are not allowed to add but add back doors at the requests of other governments

Doc: Yeah,

Jonathan: and that would that would pretty much take care of it.

And you know, whenever there was a Conflict in a case like this you would suddenly have the u. s government as a party to the conflict That would sort of change everything interesting thought

Doc: Having hung out in a lot of universities, I'm, I'm always struck how It's basically because they have law schools, you know that The answer is policy.

What is the question? Right? That's almost where they always come from. We we need laws here and regulations and I would rather have standards and practices and and And let the world take care of that, you know, not that we don't need laws. Obviously we need to be able to drive on the right and stop on red and stuff like that.

I'm not kind of full libertarian on all that stuff, but I, there's a, but I, I just see so much more opportunity where we're not starting with regulation. If we, I always like regulation to be the cart that follows the horse rather than the cart in front of the horse. Even I mean like right now with AI, I just think all AI regulation is just not even going to happen.

It's. You know, and now at least in the U. S. we're, you know, we have a, an administration that, that, I don't know, it's not worth, not worth going into, but but I mean, every, everybody I know who wants policy to happen is basically looking at their hands right now, it's like, I guess not for the next four years, you know, it's sort of, so, yes. Matthew Green. And I'm not saying I favor that, by the way. I don't have a political position. Yes.

Jonathan: Yes, I understand. Matthew, Matthew Green was the is the cryptographer. He's the, okay. Yeah. Yeah. And he's got a. Cryptographyengineering. com is his website and he's got thoughts on this and a bunch of other things.

I've, he's one of the, one of the voices that I follow. But yeah, he has the suggestion that we just need to pass a law in the United States that and here's the thing. Here's the thing that I really liked about this. Like if you could get that idea before the right people that's something that in theory should make sense to everybody, right?

We will not allow U. S. companies to add encryption backdoors at the behest of other governments. And that's specifically the way that he put it. And it's like, that seems like a, a, a win no matter which side of the aisle you're Yeah, I guess the company

Doc: can now say, well, we're a U. S. company. We can't do that.

Yeah. You know, and the U. S. has, has this law that we're operating, you know, we're registered in Delaware, Texas, wherever it is now. And yeah, that would, I mean, that makes sense. And I would hope that it would appeal to You know to both sides in Washington, but who knows? I mean, that's I've given up betting on any of that.

Jonathan: Indeed. It's kind of like horse racing, right? What's, what's the quick, to be able to win a small fortune doing it, what do you have to do? Well, you have to invest in a large, yes. Yes. That's how betting on all that works. Goodness. All right, Doc. I feel like we've I feel like we've covered this fairly well, sort of our,

Doc: well, it's, I should add, I mean, it's so early.

It's, I mean, that's part of my point with this is that we're early enough with personal AI that at least of the kind that I envision that that's on the PC model, it's personal computing model. The important fact about which is, or aspect of which is that it's yours. That's basically it. When it's personal, it's yours.

We're just completely early with that. We'll just, we'll see where it goes. Mm-hmm . Or we'll make it go where we'll see it goes that place a better thing.

Jonathan: That's, that's the part that intrigues me the most. That's why I wanted to have you talk to you about this, because we are, we're in the position now where those of us that are sort of the, well, the programmers, those that are researchers, that's those that are working on open source projects we're still kind of in the driver's seat on this thing.

In a way, in the direction that we want it to go for personal, because nobody, none of the big companies right now seem to be working on that aspect. I mean, sure, you've got, you know, OpenAI with ChatGPT. They've got this idea of, oh, well, as you talk to ChatGPT, it remembers your conversations, and therefore the AI becomes personal.

But that's not really what we're talking about.

Doc: Yeah, I mean, we outnumber them, whatever them are. That's also true. I mean, I'm reminded of what, Bill Joyce said at Sun Microsystems back when they were a thing because it was so fond of calling itself smart about everything, he said, most of the smart people in the world don't work here.

They work somewhere else or on their own. And there are a lot of them. There are only some of us. And that's, and that's sort of, I mean, to me, that was, it was always the most interesting thing about, about Linux and all the open source development is that there's just a load of people out there doing the whole thing.

Right. So, so. I mean, I mean, it's, I mean, even what you were saying earlier, well, isn't this already being done here or there or there? Go to Hugging Face. Look at all this stuff happening there. What a brilliant site that is just to have that list. It's massive. It's massive. And you know, so it's just. I mean, I think there's just a, there's a lot to play with right now.

You know, and as the models get better, you know, we can use them personally, we can use them locally. Thank you for your model. We'll just work with that. I, I just think it's really early in the whole thing, whatever that thing is.

Jonathan: Well, I am definitely going to look at the LLMII. I will actually reach out to these guys and see if they want to do an interview as well.

LLMII. Yeah, it's, it's to take your images and go through and index them, add tags to them, and you know, try to figure out what is in there. Do it on your own local machine, and that's I get

Doc: Luthier's Mercantile International, so but then if I had images no, I get stock images, and so, well, okay, well, I'll look it up.

Jonathan: I will add the I'll add the link to the show notes. It's on, it's on GitHub.

Doc: And yeah. But take a look at LM Studio too, because it's Yeah, I pulled that one up. They've

Jonathan: got they say that their, their models and their CLI are all open source. And then the, the GUI, the fancy looking thing that you actually run locally is not.

Right. Yeah. But yeah, that is, that is pretty

Doc: interesting as well. Yeah, I mean, I, I, I, I, somebody demonstrated that to me when I was in New York a couple months ago. And it was pretty remarkable. I said, well, I'll yank this one out, I'll put this one in, I'll yank that one out, I'll put this other one in, see what it says, you know, and have it, I'll have to use this one to ingest all this stuff and Pretty interesting.

Yeah, interesting.

Jonathan: Yeah, for sure. All right, is there anything, we kind of, we kind of talked about your plug for the whole, for the whole show, but is there anything else that you wanted to plug? Oh my gosh.

Doc: Well, I'll, I'll plug KWAI, K W A A I dot A I. It's an all volunteer thing. Yeah, A A I dot A I. They now have a new user interface where my face rotates among some others of people that are also unpaid volunteers, you know.

But you know, a fun thing is that it, it is, you know, that there are there are three significant organizations that have base their work on, at least to some degree, on stuff I've written. And they're one of them. Consumer Reports is to some degree with their future plans. And and Tim Berners Lee's Solid Project, you know, so, which Tim himself told me was inspired.

This book I wrote that was a worst seller except for the fact that he bought it, which is like, okay, I don't know if he bought it, but he had it. And You know, good, good for that. Yeah. So you know, I'm, and, and, and it doesn't mean I'm right about anything, by the way. It doesn't mean that I, I said some stuff.

Anybody can write a book. Yeah, exactly. Anybody can write a book, you know, just get an AI to write your book for you. So it sounds so generic, by the way, I've done all kinds of stuff with AI images. It's so interesting to me, just a quickie, quick party thought that. You know, six, eight months ago, being able to come up with an AI image for something that was so brilliant and fantastic.

And now it's so cliched. It's like, you see them and you know, that was done by an AI. Yes. And so that didn't take very long. No, no, it

Jonathan: didn't. So I'll, I'll get your final prediction then. What's, what's the next thing after AI? What's going to be the next flash in the pan? Are we going back to

Doc: cryptocurrency or?

No, I mean it's, I mean, I, I do think I'm I'm really scared of the strategic Bitcoin reserve that our current administration came up with, with help from Elon Musk. Not only because I worry about lose, the dollar losing. It's it's a reserve currency status. I think that would be a bad thing for the world And but I think it raises the possibility of that So and I say that based on nothing other than my own but I it's just like how is this not a good thing?

It might be not a good thing that way, but I don't know. As for what's coming. I mean

Well, I'll tell you one thing. We, we need, we need words other than AI, because it's not artificial. It is not intelligence. It's a collection of other capacities, and we anthropomorphize this stuff. We say, oh, it's trained and it learned. No, it didn't. We use those terms because they make sense to us, but that's not what actually happened.

And I, I sort of see come, in coming years, multiple Two, three, and four letter acronyms that, that mean these other things that these machines do, you know, and or non machines and software, whatever, you know but, but I don't know. I mean, I, I, I, I think actually one big thing I think will happen, and I just say it only because of motorized bicycles, is that there could be a lot more people traveling around on those, you know, I like playing with those.

I think it's a better way to get around in a city or even a town. They're pretty handy and they're, I think we just see a lot of change around that. It's not self driving cars that are going to change everything. I think it's going to be motorized, you know, motorized bicycles, but. Self driving cars, that's, if, I mean, go to San Francisco, holy crap, I mean, they're everywhere.

And it changes the way traffic works, I mean, it literally does, like, oh, these things obey the lights, okay, I mean, it's sort of like that.

Jonathan: Oh, that's funny. That's great. So I'm starting to see them here in, you know, basically the middle of nowhere, Oklahoma. We've got several. It used to, I remember taking a trip and going, wow, it's a Tesla.

And the guy I was with was like, I saw like five of those on my way here. That's the first one I've ever seen in person. And the other day I went to the doctor and there's a Cybertruck sitting there. I'm like, oh, cool. It's the That's one of

Doc: the 2, 000 that got sold? Yeah.

Jonathan: Not a whole lot of those out there yet.

Doc: But yeah, we'll see. Aaron Lawton. Wow, wow. Yeah, well, why not? You know, go play with it. Sure. I mean, I, I'm a, I mean, Tesla is going from new hat to old hat rather quickly. And and it hasn't I mean, I think in a certain way it could end up being like the Volkswagen. Every Volkswagen bug sold between like 1952 and 1970 was the same.

I mean, it looked the same. It changed a lot on the inside. But but they were basically one design. They just kind of kept going like that. Yeah. Okay. Tesla has a good It is in a good position, I think, to normalize the way electric cars work. But I don't know. I don't, I have a hard time figuring it out.

Figuring, figuring out, I just think that there's a weird a weird a weird moment in time with that company.

Jonathan: Yes. If we, if we could really actually read those tea leaves successfully, we would, we would be millionaires from stock market trading and all of that stuff. So I know it's like,

Doc: I mean, if, if buying Bitcoin had been easier for me way back when I thought of doing it, it's like.

Yes. How many of us think the same thing? Right? Oh my

Jonathan: god. I, I, I distinctly remember thinking, oh man, I wish I had a couple hundred extra bucks so I could buy some of these bitcoins. I

Doc: know. Right. Yeah. And I, and there are people we know. I mean, john, our producer on Twitter in the past, he had a whole bunch of Bitcoin, doesn't know where it is.

Oh. You know, and I know other people who say the same thing, well, I, I have it. I think I did a Xerox copy of the

key or something like that,

but I don't know where that is. Okay. Yep. Yep. Yep.

Jonathan: All right. Last two questions I'm going to ask you before I let you go. And that is, what is your favorite text editor and scripting language?

Doc: Oh boy. All Oh, I guess, I mean, it is, this is, it's funny because I was thinking today, I really need a text editor for this one and I'm not sure. I mean, I, you know, back in the latest journal, I use VI. So there was that, you know, and But for scripting, nothing. I mean, I don't do any of it, so it's like, it doesn't, it, it, it's, it, it's sort of like, you know, what's, what are your favorite skis?

Well, I don't ski anymore, so I don't know. It's like, been a long time. Yep, yep, I understand that. My favorite is whatever somebody else uses for me. I'm at that stage. Yep, there you go. All right. Hey,

Jonathan: thank you for being here. Appreciate you coming in at the last minute. Thank you too.

Doc: I miss this thing. It's always a fun thing to do.

Oh yeah, yeah. I'm glad you've kept it going.

Jonathan: I am too, and I'm, you know, I'm super glad that I've gotten you, and even Randall, in the rotation of co hosts. That's great you brought Randall back. Yeah, we've had a lot of fun with that. We've had a lot of fun with both of you.

Doc: The old hand at this. Both of you guys

Jonathan: have been a lot of fun to have back in, so we will definitely do it again.

Maybe we'll have both of you on the same show at some point. That would be interesting.

Doc: Oh, that'd be interesting. That's never happened. We've, we've hung out, we hung out on boats together a lot, a thousand years ago, when he was on every cruise that ever was. Yeah. I'd love to talk to him about that actually, like, what was it about cruises for you?

Jonathan: Maybe, maybe for the thousandth show or the 1024th show we'll have both you guys on. That'll be fun. Oh,

Doc: that'd be great.

Jonathan: Yeah, that'd be great. Let's do it. All right. Thank you, man. We'll see you next time. Thanks. Yeah. All right. If you want to find more of my stuff, there is, of course, Hackaday. It's where the security column goes live every Friday morning.

And we appreciate Hackaday being the new home of, not new, not so new anymore, but the home of Floss Weekly. I've also got that other show, the Untitled Linux Show, that's still over at Twit. Have a lot of fun there if you want to find that one. And we appreciate everybody that's here. to catch us live and those that get us on the download and we will see you next week on Floss Weekly.

Episode 824: Gratuitous Navel Gazing

12 mars 2025 | 67 min

Episode 823 transcript

5 mars 2025 | min

FLOSS-823

Jonathan: Hey folks, this week Aaron and I talk with Joao Correa about TuxCare, LivePatching, and NET 6. It's a lot of fun, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 823, recorded Tuesday, March 4th. TuxCare, 10 years without rebooting.

It's time for Floss Weekly, that's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. And we've got something a little out of the box today. We are leaning heavily into our Linux roots today. Going to talk about something Microsoft did, something Microsoft stopped doing that someone else's it's going to be fun.

We're talking about TuxCare and specifically NET 6, which a lot of you might not be big into the NET ecosystem. I am certainly not. I've got a co host and I'm curious Aaron. Are you a NET user? Is this something that's in your wheelhouse?

Aaron: Not really. Although from a, not as a user, I would say, but I do remember I don't know when, but probably in the 2000s, the aughts as they're also known, you know, there was.

net and then there was mono. So that was more of a, of a open source Linuxy project that people, you know, it was kind of like the Java wars as well, when you open JDK and some of those things started coming out as well, trying to give open source alternatives to closed source Platforms like that.

But yeah, I don't really, the only thing I know about NET these days is when I'm installing a game or something. And as part of the install process, I get, Oh, you have to install a NET version of blah, blah, blah to make your game work. And I'm like, okay, this is the longest part of the install. Just get it over with and let me play my game or, or my, use my application or whatever.

So yeah, these days I've never been a NET. Programmer. So I don't know a ton about it. It just seems like it gets in the way a lot. So it'll be interesting to see maybe how how we can fix that. Yeah, I

Jonathan: remember back in the Windows XP era waiting on the dot net framework. Updates when doing windows updates and some of the old machines that I was trying to keep working man I would sit there for like literally an hour just trying to wait for dotnet to finish whatever it was doing Exactly.

I I know that the story really changed in the 20 teens when Microsoft released I think they called it originally NET Core. And that's when they went and they open sourced a whole bunch of the stuff around NET. And I know that today because I was checking this out for another project, today, even in Fedora, you can just, you can go do a DNF install NET, and it will just happily pull those packages down.

Because Microsoft actually open sourced them. I think that's fairly important to what we're talking about today. But I'm also very interested in the, the tux care side of this conversation. So, let's go ahead and bring our guest on. He gave us a quick primer on how to pronounce his name, and I'm sure I'm still gonna get it wrong.

It's essentially Joao Correa. Close? Yeah, it's close enough. Close enough, okay. It's close enough.

Joao: Hi, thank you for having me. Hey, it's good to have you, yeah. Yeah, so, yeah, we at Tuxeer, we do lots of different things. We essentially support open source when the original maintainers don't want anything else to do with it.

But specifically on NET, and we'll cover this more in detail. It's actually interesting that, and I was looking at some JetBrains data, the guys behind ReSharper and a lot of other tools. About 20 percent of their users actually come from the Linux world, are using their tools on Linux to develop C sharp.

When you're using C sharp, it's very, it's a very good guess that you're going to rely on some stuff from dot net. You don't have to, but you're essentially going to find your basic libraries there. So the presence of dot net on Linux, it's. Let's rewind this a bit. We actually have to go through another phase that Microsoft went through in the early 2000s, late 90s, where Linux and open source was a cancer and attached to everything and it broke their licenses and they really hated it at Redmond.

They actually turned around and they changed their minds and they started to embrace it and they started to embrace open source and they started contributing to lots of processes. And It tells something that whenever you're talking about Microsoft in the Linux space, you always have to make this distinction between that period.

That's way back there. And the way that they're doing change that they're doing their things right now. So the, the thing with. net, like you were saying with on the XP days, it was. net framework. It went through lots of revisions. There are. To version 4. 8, something like that, which I think is still supported on the Windows side of things, but it was very Windows centric.

It was closed source. It was very Windows centric. The idea behind it was to provide a standardized look and feel for the components that you use on your, or UI and your user experience components. So the. net framework gave you that. With subsequent releases with NET Core and now just NET. What they're doing is that they're giving you components, not just focused on the, on the interface.

So you have components for messaging between processes, for talking to other systems, for distributing load, for lots of different tasks, not just related to them, to the UI. And historically, one of the holdovers at the enterprise level. The prevented companies from moving fully to open source and fully to Linux was that they would always have this in house developed tool, this line of business application that had been built in the days of NET Framework or NET Core, which they couldn't cleanly migrate To Linux and the cost of doing the full refactoring of that code of getting things to run on top of Linux was just too expensive.

So even if the majority of the systems were moved to Linux, they would always have a cluster or two or three just running those line of business applications because they couldn't get rid of them. So dotnet comes and helps out in that dotnet is cross platform. You can run the same software. With some caveats on Windows and Linux and Mac OS if you're very adventurous.

But yeah, it aims to fix that. It's not just that technology that works on Windows and it's not just Windows centric anymore.

Jonathan: Yeah, I've got a, I've got a programming buddy that is a dot net developer. And he has kind of shown me the light on, you know, this is actually worth using these days. Like you said, particularly if you want to do anything with C Sharp.

It's, it's kind of the, the standard lib for C Sharp. And it, it works. And like I said during the, the intro, you can actually, Fedora is kind of my litmus test because they are very strongly open source only. And so, you know, if I can just go into a vanilla Fedora install and do a DNF install of the NET packages, then they must be set up fairly well, they must actually be open source, like everything must be kosher there.

And so I've, you know, it's one of those things where, like you said, Microsoft used to be just absolutely the bad guys in the Linux world. And so much has changed in the last 20 years.

Joao: It absolutely has, it absolutely has, their stance has turned around. I mean, they probably just started looking at their usage data from Azure that says that Linux is the most used system there.

So, I mean, they're not blind, they know their customers, they know what they're running, they know the workloads. If the majority of the customers are running Linux, then they have to be able to provide service to those Linux users. And the way that they do this. Essentially, so far has been open sourcing stuff.

I mean, you have NET, you can run PowerShell on your Linux systems for whatever reason you might want to. want to, but you can, yes. You can deploy it, it can run. It actually does a fairly decent job at letting you run exactly the same scripts on Windows and Linux for a variety of tasks, and that helps with automation if you have mixed environments.

But yeah, being on Linux and natively a Linux user, I do prefer my bash scripts every day. But again, the option is there if you want it.

Jonathan: You know, I can't help but think that Microsoft is aware of what happened with like IIS when it went head to head with Apache and how Apache just literally decimated, like reduced it by, by 90%.

You know, the actual definition of decimated, probably more than that. Apache just absolutely owned the internet there for a while. And that was one of the big things that really made Linux popular, particularly for servers back in the day. Because Apache let you do some things that IIS just could not touch.

And that really killed that section of Microsoft's business. And I can't, I can't help but think that they were aware of that and going We don't want to lose, you know, our compiler stuff and our, our library stuff too. And so they were sort of forced into this of, well, I guess we're going to play nice, right?

Joao: Yeah, so they essentially had to go with the flow. It's interesting what you said there that Apache killed the IIS. It did. The usage statistics don't lie. It really did. The majority of the top million websites are running on Linux and that's just factual and nobody can actually dispute that. But the fact that the Windows ecosystem is so large, even 10 percent of that ecosystem still represents a huge amount of systems they still pull a lot of weight.

It's still useful to, to develop applications that run on their platform. Having those same applications now be able to run on Linux and on other, on other platforms. That's just icing on the cake at this point.

Jonathan: Yeah, agreed. So let's, let's go into, for just a bit, the history of TuxCare, because I don't know much about TuxCare.

Somebody mentioned this to us in an interview we did a couple of weeks ago, was talking about some of the, the big Linux what, what was it? It was the companies that were doing support for Linux machines, and just kind of mentioned in passing, like, there's this one, and this one, and the TuxCare.

And that was the first I'd heard of it for a while. So what, like, what, Generally, does TuxCare do, but also, what's its history? Where did it come from?

Joao: Okay, so Tech Care started tech Care is a brand, so to say of Cloud Linux. Cloud Linux is a company in the, in the open source space. It has a cloud Linux os, which is very popular with with hosts and that side of things.

It has tools for production of Word Press and all of that. And Taxpayer Tech Care specifically focuses on the enterprise side. We focus specifically on providing support for. Out of service distributions into past the end of life date we offer support if you're running say 8, we continue to offer you security updates for that.

We also provide you support with many different distributions like that. Many different things that you're using and you're using just fine and you want to continue using because they're working just perfectly. So we give you the security updates to be able to do that securely. We also have life patching, and this is where the conversation usually gets interesting.

We have life patching in a way that's different from all of the other life patching vendors. For starters, we have life patching for many different distributions, not just vendor specific stuff. Like, say, Red Hat will offer you life patching for RHEL, Canonical will offer you life patching for Ubuntu. Each vendor has Some type of live patching for their own distribution, Oracle as well.

We actually support live patching on all different distributions and we'll provide you with live patch for all of those. And we actually treat live patching seriously. All of those vendors, they will add that just as an extra line item on your support agreement, and they will. Pretend to be very important for them, but then they will only patch like a handful of CVs every year.

We cover hundreds of them. In fact, if you have a CV for the kernel, we will live patch it. We've been doing that for many, many, many years now. We've deployed thousands and thousands of life patches. We have customers running systems. I believe the top one is a heating 10 years now without a single reboot.

Yeah. And they're not doing that. If you had told me 10 years ago that you were running a system for 10 years without any support, that's no longer your system. That has been hacked all over again many times. But the way that we let you do this is that you continue to receive your updates. Your kernel will be running at the equivalent of the kernel that is out right now.

And you didn't have to reboot that system to apply those updates. We'll let you live patch stuff like glibc and OpenSSL without having to restart the services that are using it. We'll go ahead, we'll find all the instances where it's running in memory, we'll apply the patches and all the services to just continue to run.

And we've been doing that consistently for many, many years. Now, the company was started, I believe, in 2009. We've been offering live patching since about 2015. We've been doing that. We have lots and lots of customers using our solutions. And a critical component of that work is that we have to work with the open source projects.

We have to backport a lot of stuff. We have to pull stuff from upstream projects and contribute back to them. And we do that. We, we have. A deep understanding of how that works and how we operate in that space. And that's how NET gets tied back into the picture of what we're discussing. As an open source project, we are fully able to integrate it into the workflow that we already have and continue to provide support for it after the original vendor, in this case, Microsoft, it's, it's an open source project, but everybody knows that it's Microsoft behind it.

We can continue to look at the vulnerabilities that are out. We can continue to develop fixes and distribute those. We have the pipelines just working perfectly by this point.

Aaron: Yeah, that's pretty amazing. I guess, I guess I didn't realize it, but I should have just given my background trying to, trying to support things, or working with companies that support things, because, you know, these days, for modern systems, you'd probably just have a container running, right, and you I Create a patch for something and then just redeploy that container and Everybody's happy because of the way that you've architected your your systems and your software to handle that sort of things but for legacy environments where You know, I'm assuming correct me if I'm wrong but I mean my ancient history came out of a regular regulated environment for pharmaceuticals and Where there was requirements to keep things around and keep things up and available with a certain period of uptime.

I mean, I'm assuming that at this point that those live patching Like, like when a, when a vulnerability comes out, for example, I mean, how do you even keep up with? The request, because I'm assuming that when a, when a really bad vulnerability comes out for something that's old, but still has to be available, you guys must have to really like jump into emergency mode, right?

Joao: I mean, we don't jump into emergency mode because that's essentially what we do. So it's not different from any other day. Essentially, right now, since the Linux kernel became a CNA, and I'm sure you guys have talked about this. I mean, the number of CVs just went through the roof, everything gets a CV. And now everybody's in trouble for not complying with patching that within X amount of days.

Yeah. That meant a lot more work for us as well. When the CV comes out, when people are struggling to find the fix and rebooting and getting their maintenance windows scheduled and all of that to reboot the systems, to apply the patches. That doesn't need to happen with live patching. The update is made available.

We have it on our repos. It gets picked up by the, the agent that deploys them. Sorry, not an agent. It's a scheduled task that every now and then checks to see if there's a, there are patches available if there are, and you're configured to do it completely automatically. As you can, it gets applied immediately.

It's the fastest way that you can respond to new CVs right now. There's no other technology that lets you do this faster. As soon as the patch is available, you can apply it. That's minutes. That's not even hours or days or weeks like you see in many places. Oh, that's awesome. The patch is applied.

Aaron: Yeah, I'm thinking like, you know, there's code monkeys out there, like actually manually going in, SSH ing into these servers and like trying to figure out something, a manual, deploy a manual patch for something on the fly, but the way that you're set up with your agents and everything, it just kind of happens.

Automatically if, if the user chooses to, to make it happen, assuming

Joao: you can still have your different tiers, you can still have your lab environment where you receive the patches first. That's all you can all you can accommodate that with life patching. But if you so choose, you can receive them completely automatically.

Jonathan: I, I can't help but think of some of the other vendors out there that do not live patching. None, none, it's exactly what you do, but do like hardening for the Linux kernel. And there's, there's one that comes to mind that is rather infamous. And that is, I believe it's GR security is the name of them that have the, the hardened Linux patches and the, the The very strained relationship with the upstream kernel.

I'm just curious, has TuxCare managed to do live patching and all of this? What is that relationship with the upstream kernel, but then also like all these distros that you support? Is that sort of a friendly relationship still, or is there some strife that happens?

Joao: We try to be friendly with everybody.

Pretty sure that's the general stance. Here's the thing. We try to get our patches to have a proper live patch that applies cleanly and doesn't introduce any side effects. You have to compile it with exactly the same configuration that the kernel that you're running has. So we do that by distribution and inside it's.

distribution. We do that by version, by kernel version, because they might change the flags that get compiled in on different versions. So we have this build matrix that's very huge, as you can imagine, it has lots of different kernel versions for every single distribution out there. And we need to backport the patches and make that compile the same way for everybody.

So we get patches from upstream developers and in the distribution side so that it aligns cleanly with what they're releasing officially on their distribution packages. But when that's not manageable, either because they're taking too long to deploy the patches or they just don't patch a given vulnerability and that happens.

Or they decide that they're not going to patch it until a subsequent version. We go upstream, or if not, we go directly to the upstream projects in the case of the kernel or the GLibc or OpenSSL, which is not the kernel, but we also livepatch it varies a lot. We don't have to my knowledge. We don't have any feud with anybody over this.

We don't Do any underhanded stuff to getting the source code. We don't essentially steal the code from anywhere. This is open source projects. We contribute back when we create the fixes ourselves. I do believe we have a healthy relationship with everybody in the environment.

Jonathan: Yeah. Yeah. Very good. So let's talk about T net then.

So T net six, I was looking and I was actually really surprised by this. T Net six was released yeah. Released state for T net six was 2021. This thing is only three years old. And Microsoft already pulled the plug on it back in November of 2024, like it did not, it did not last long. I'm, I'm real, I, I'm very surprised by this.

What's the what's the story with the NET lifecycle?

Joao: And that was a long term release. Yes. Yeah. The long term releases get three years of support. The regular ones get 18 months. That's how they roll. We're already seeing previews for NET 10 coming out right now. The thing with that. is that while Microsoft does a pretty decent job, and you have to acknowledge this, you can run stuff on Windows 11 that was built for Windows 3.

11 or something like that. Yes. And that's like 30 or 35 years ago. Yes. And you still have a way to run that. And I mean, without using emulation. So they have a pretty decent track record of maintaining backwards compatibility whenever they release something. And the thing with T net is that T net comprises a lot of namespace.

Namespace is the name that they give to the individual stuff inside of T net. You have stuff for files, you have stuff for communications, for signaling, for different subsystems in there. There will be breaking changes on some of those subsystems. So if you happen to have an application that's written and uses that.

You can't cleanly move to something more recent because you will have to refactor a lot of the code to accommodate those changes, or you'll have to find alternative libraries to fill in the gap of what you no longer have. So having this short life cycle is tricky in that regard. When you develop code against it, you should already be prepared that in at least three years you're going to have to rewrite some of it, but most companies don't.

Jonathan: That's that's interesting. I'm It it reminds me of I mean so like other open source projects do this sometimes But it's because it's it's all open to look at this kind of life cycle works when you're talking about open source code Right because it's constantly being worked on and it's being worked on in the clear And so, like, a distro maintainer, it's easy enough to just pull the patch down to be able to update it.

I'm just, I'm just really surprised that Microsoft decided to go with this route when people are writing, I assume, proprietary code against the NET framework, against NET 6. It's just, I don't know, it's just such an odd choice. Do we have, do we have, like, some insight on why they break things every three years or year and a half?

Joao: Sure. Their licensing their licensing agreement, and they need you to get new updates. Other than that it's just the way that the, the Windows ecosystem works. You have to remember that Microsoft is a Windows company. They're still focused on that, even if they do contribute to open source the way things happen on that side of the fence is not exactly the way that you're used to on the open source world.

It's just how everybody does business on the Microsoft space. It's how things are done and you either go with it or you just shout at the wind, but it will get you nowhere. So you either have to adapt or you have to to be ready to face the consequences of that. But there's no, there's no particular insight on this.

It's just the way that things are set up. It's the way that it's actually the way that companies expect this to work. In some sense, if you're forced to upgrade for some reason, then it might force you to actually look at the code. And if you have the ability to do that, that's great. The problem is that at most places you don't have that ability, especially when we're seeing the environment right now where developers are being fired left and right and development.

Teams are being are struggling just to keep up. It's really difficult to be to do that when you have internal internal applications that your team developed five, 10 years ago, they might be running fine. They might be running your accounting system and that's just fine. They might be running at say university and they're running the scores or the students enrollment and all of that and it's working fine.

And now you're forced to go back and work on it. Most teams can't accommodate the

Jonathan: extra work. So What are the what are the, what are the alternatives? Like, what, what does a company do if they have a 10 year old program that was written for NET 6? Obviously it's not going to be 10 years old. The math doesn't work there, but

Joao: yeah,

Jonathan: can they just keep running NET 6?

Is there, is there any downside to doing that?

Joao: Okay, so since NET 6 launched in November, something like that, 2021 there have been an average of about two new CVEs per month since that time. So it's pretty easy to run the numbers. If you continue to run that application without any support, you're going to be seeing CVEs that will eventually affect the code that you're using.

It will eventually affect whatever functionality you're pulling from NET and your application, no matter how well written it is, how secure it is, how your code is great. If the language fails, if you have issues in the language, then your application will suffer. You need to find an alternative source for the updates, in this case, the service we provide, or you need to refactor the code.

If you go down the route of refactoring the code, which is perfectly valid, you, you'll need to look at whatever new functionality was brought in. You'll need to rewrite code around the changes. You'll need to make sure that everything still works and that your code continues to be great. Sometimes that's doable.

Other times that's not doable because of the amount of extra work that it takes and it's extra work that it's very difficult to actually justify. Extra work means extra costs and it's all just to be at the exact same position that you were when you started this endeavor with the same application running at the exact same level, doing the exact same thing that it was doing.

You're not taking advantage of any new functionality. Taking advantage of any new tooling that comes with dotnet. You just have your application running as it was running before. And for some companies that's not a very healthy proposition.

Jonathan: And so what about just staying with dotnet six? Like if it's installed and it's working?

Is there, is there any problem? Does it really matter that Microsoft's not pushing out updates?

Joao: Well, that's the same reason why you're not running CentOS 5 these days, because ever since it was launched, it has accrued like 10, 000 new vulnerabilities. So you can continue to run it. I mean, you can bite the bullet and just do it.

It won't be your application for much longer, but hey, if that suits you, then. Go ahead. But honestly, you shouldn't be facing that proposition. That's not really something that you should be contemplating running without support, especially at the enterprise environment. That's not something that you can do.

For starters, you're going to lose all compliance. There's no compliance regulation out there that says you're going to be running an application with no support at all, knowing that there are vulnerabilities that are going to be coming out. That just doesn't fly. Your auditors, they're going to flag that you're going to get in trouble and you're.

I'm going to get in very trouble. If you say you're in healthcare or you're in finance or something like that, where you have very strict compliance regulations and you need a way around that. So again, your option is rewrite the codes or get support from somewhere else.

Jonathan: So there, there are actually vulnerability.

There are CVEs that get found in the. net frame, the, in, in. net itself. I keep calling it framework How, how how severe and how often do we get those CVEs? Like, there, there, there are, I cover, I cover security, right? On a weekly basis, and so there's like two broad categories of CVEs. And, you know, you've got the one where it's like, Okay, yes, I guess technically that could be a problem.

And then you've got the other category, it's like, Oh my goodness, my hair is on fire.

Joao: You get both, obviously you get the ones that affect say the installer, but that's something that you run one time only and you're not going to run again. So who really cares if you have the installer at the extreme case, you just unplug the network cable, you install it and you plug it back in again and then the wiser, but then there's remote code execution CVS and those have happened.

Those have been released out there for dot net. And those are more dangerous if you happen to use that specific part of the code, if you happen to rely on that specific function that was found to be abusable or exploitable, then you're in trouble again, it covers the whole, the whole spectrum here.

Jonathan: One of our listeners that's listening to us live mashed potato says it wouldn't be a Microsoft product without a good healthy bunch of CVS to go along with it.

Which is hilarious, but not actually fair because we have our, we have our own share of CVEs and in other open source projects. So it's not just Microsoft, as fun as it is to say.

Joao: Saying that when the Linux kernel had 3, 400 just last year or 4, 000 or something like that, that's not true. Probably a good road to, to go down.

Jonathan: I am convinced that the Linux kernel is publishing CVEs out of malicious compliance. I am, I am convinced that they are, they are sort of playing a game because nobody wanted to use their long term releases. And so they're sort of playing a game saying, you don't want, you don't want to use the LTSs that we provided.

Okay, that's fine. I guess because of the way the laws are written, you're going to use the most up to date kernel then, huh?

Joao: You know, I've written about this in many different outlets. I don't want to believe that's the case, but they're making it really hard to continue to believe it's not the case.

Jonathan: I, I am, I am convinced that maybe malicious is, is a little strong, but yes, I am, I am convinced that there is a bit of strategy to that.

It's, it's not an accident.

Joao: And what's up with those year identifiers? I mean, the, the kernel CNA has been issuing about half of the CVS that they put out have you have your identifiers prior to the start of the CNA activity. They've been issuing CVS for. 2016, 2017, in the extreme, and I don't know if this is true or not, I haven't checked.

If there's any code still remaining in the kernel from the nineties, you're going to run into this great situation where you're going to have a CVE issued for a year prior to having the CVE system in place, which is amazing. That's great. And we're running that risk right now.

Jonathan: It, it's fun. I'm sure it's a headache for you guys, but it's.

At the same time, it's very fun to watch as an outside observer. It's

Joao: also fun for auditors when they look at your security compliance reports and you say, Oh, you're missing patches for this CVE from 2010. What are you doing? Yeah, it was released last week.

Jonathan: Yes. Yes. That's hilarious.

Aaron: I'm kind of curious if I could jump in for a second.

Yeah, go for it. What, what other, besides NET. I mean, what, what other languages basically or programming environments have this type of issue? I'm thinking Python, maybe like people have like a Python 2 script, but for some reason they can't update to 3.

Joao: Python is one of those that has incredibly Incredibly descriptive instructions on how to take your code from one version to the next.

It's like 50 or 60 pages that you need to go through with weird examples all the way. Yeah. But yeah, Python, PHP, Java, Spring itself, the Java framework. Those are all also services that we provide support exactly so that you have, you can avoid going through all of the hassle of rewriting the code just to accommodate all of the changes.

So it would be insane to be running an out of date PHP version on your website, for example. We all love PHP like that, but yeah, if you have the updates that fix the security problems, then you can continue to do that. It's not typical, but you can continue.

Aaron: I mean, what about if dependencies change? I mean, I know that's probably a little bit more difficult problem to solve, but I know I've, I've had old Python scripts that I thought would still work, but for some reason, something happened with one of the dependencies or modules I was using and, and that didn't work.

Is there anything you can do for, for that user or you just have to say, no, we've got to set up this entire environment. We

Joao: support many, many such modules. We support many packages, popular ones, and we actually are open to supporting new ones. If you, if the user is interested in just reach out to us and talk to us, we're very open about that for all the languages Python, PHP, Java.

If you have something that you want us to support and it's open source, just talk to us. We're very likely able to help you.

Jonathan: How many, do you have an idea of how many discrete things between like languages and kernels and packages that TuxCare offers this sort of support for?

Joao: If you want to go to that level, then it's going to be hundreds or maybe thousands across all the different distributions.

Yeah. It's a very large support matrix. Like I said, the build environment is massive.

Jonathan: Do you know off the top of your head what the oldest thing is that you guys still do support for?

Joao: Not really. We cover CentOS 6 has very old stuff in there. But I can, from the top of my mind, just give you that one example that we're looking

Jonathan: for. That's, that's fine. That's fine. Can

Aaron: I, can I ask another question? This is now starting, my interest has peaked hearing all this conversation. So, I noticed on the website that there's that you also support QEMU.

Joao: Yeah.

Aaron: Which is curious to me I mean, I've been a fan of QEMU since the old days, I think Fabrice Bellard, I don't know if he's still involved, I think he was one of the original contributors or started the project, I started using it a long, long time ago when I was still at Sun Microsystems, I think and it seems to be coming, be more and more popular these days, so I'm kind of curious, what do you, what needs to be fixed with QEMU that, A customer would come to you and say, I will pay you money.

If you can take care of this problem for me.

Joao: It's getting popular probably because VMware helps that. So you know how the maintenance operations go around when you have to patch the hypervisor, when you have to patch the host system, you need to migrate every single way you need to patch the system, you need to migrate everything back, that's incredibly.

Easy to do when you have two systems. That's very, very hard when you go into the hundreds or thousands of systems. I may have done in the past migrated every single way from the node, then to apply updates and reboot the node where I had migrated to instead of from. That was a fun day for all of the 20 or 30 VMs that went down.

It's really easy to mess up the system. Sure, you can talk about all the orchestration tools that you want and all your automation scripts and all of that, but again, it takes a long time. It's very bandwidth intensive. It's really error, very easy to make mistakes there. So what we do with QMU Is that we can apply life patches to it.

If there are flaws found in QAMU and there are security issues, instead of compromising your VMs, because the hypervisor was, was breached. We can apply life patches to it. You don't have to move any VMs away. In fact, the VM workloads don't even know that the QAMU was patched. They're not even aware of that.

It doesn't stop the process.

Jonathan: I've got to ask, and you guys may not want to say this publicly, and that's fine, I suppose, but what, and Aaron touched on it just now, if somebody wants to pay you money, you can do this. And like, that's how business works. Lots of companies are willing to do lots of things if people are paying enough money.

What does that look like? And I'm also curious, like, is there a free tier that individuals can play around with, if they just want to try out some of this live patching stuff from Tuxcare?

Joao: Yeah, there is. Just go to the website, you can sign up for for a trial for that. But if you actually want to look at source code and see all the steps involved with that, we have a GitHub repo with how to create patches, live patches for libcare, which is a component of KernelCare, the live patching solution.

And we go through the motions of explaining how you take the code from a regular patch. Do all the conversions that need to happen and you turn it into a live patch and how to apply it. We explain all of that. Oh, very good. It's not about the actual code. That's our expertise or expertise in, in creating the patches and preparing and testing them and being, yeah, being able to deploy them securely and confidently that it won't bot your systems when you apply them.

Jonathan: Is, is the, is the pricing model sort of per system or is it like per library or?

Joao: It depends on the product. The pricing system is typically per system, but again, depends on the product. We have all of that information on the website. It's easy to find or just get in touch with us. We won't bite. We're very friendly.

We love talking to people, but no, but really sometimes you see that, Oh, we don't want to talk to somebody because it's going to be a hassle and they're going to try to push stuff. Get in touch with us. We have ways to help you. We can help you get in touch with us. We're friendly. Yeah, we really

Jonathan: are. Very good.

It's it's a it's a neat thing. And I'm not, I guess I'm not aware of anybody really else sort of in this niche. So that that's, that's a good thing. You're in a very interesting niche, and you're sort of there by yourselves. Yeah, what, what, how I asked you how many products you support? Do you have an idea of like, how many servers are using the different Tuxcare package or the service?

You got it.

Joao: Hundreds of thousands probably. Yeah. At this point, upwards of a million.

Jonathan: Yeah. Yeah. That's a, that's a, that's a bunch, that's a bunch of machines that is is one intern sneeze away from having a problem .

Joao: The, the median NEP time for the, for the systems running kernel care, which is the life patching solution.

Mm-hmm . Is over a year and a half right now.

Jonathan: I mentioned it in the back chat. I got real excited when one of my systems hit a thousand days of uptime. I didn't have live patching on that. I didn't feel too bad about it because it was on a segregated network. It wasn't, you know, exposed to the internet and nobody ever used it for anything other than file storage.

But yeah, I, I got real proud of a thousand days of uptime. You said you've got, you guys know of one that's over 10 years. That's just kind of ridiculous.

Joao: Again, it's the type of thing that if you're not using life patching, it's absurd to run a system for a year without updates, because especially if it's network connected and internet facing, that's just the best sentence at that point,

Aaron: that's what I was going to ask.

Like, what are, what are the, if you don't go with something like this. As a business, how, how would you even solve this? I'm thinking the stuff that I used to do, like I said, working for, for those regular regulated environments, we had high availability. And so we could take one side down, patch everything, get it up to spec and then cut over and then do the other side.

That's basically how we would have to do it. But that's super expensive because we were maintaining very expensive Unix servers. on those systems with high redundancy, live swapping hardware parts, you know, all of that kind of stuff. We were maintaining all of that twice just for the off chance that we would have to use it because we had to comply with the regulations that, that we were under and then the other, the other thing I'm thinking, maybe I'm answering my own question here.

I hope not. I'm going to give you a chance to talk. I promise. The other thing I'm thinking is that you could possibly use a container scenario. To do kind of the same thing where you were controlling things putting putting the things that you need to do control for a particular cv inside a container and then I guess it all depends on what what the problem is, but you could potentially like put a wall around it and say okay, I know that there's a vulnerability here because It's accessing something, it's not supposed to, we're just gonna like, put a firewall around that and make sure it doesn't happen.

What else would you do? I don't know, is that even valid anymore?

Joao: If you have very deep pockets, then you can go down the high availability route and you have higher upfront cost because you have to buy twice the amount that you need, or at least enough to support the amount of high availability that you went, the amount of nodes that you want to take down at one given moment and.

Yeah, if you're able to do that, if you're able to foot the bill for doing that, because it gets very expensive, as you were saying, it's very expensive hardware that you need to accommodate for, or even cloud capacity, if that's the route that you're going if you're able to do that, then that's one alternative.

But the thing with high availability is that it was created not to support that high availability using high availability as your shield for your maintenance operations. That's not really the intended usage. High availability was used. To accommodate hardware failures. Those are unexpected and predictable.

And when they hit, you need to have some way of keeping your systems up. If you're using it to do stuff that you can anticipate, then again, that's not really the intended scenario. It works. It's a solution. It's something possible, but you're essentially wasting resources because you have stuff lying around consuming electricity that's not doing anything.

And again, over provisioning when you probably could save the cost there. The other thing about containers is that The Linux kernel that's running on the host is the exact same kernel that's running inside and visible inside of the containers. The containers aren't magical, it's a kernel functionality that it gives out, it isolates the processes, it doesn't let one process talk to another, and that's what you get, what you call a container.

The kernel still needs updates. We can still live patch the kernel on the host and that way you'll get the updates to be visible and reflect inside of the containers as well. So yeah, you can go that route. You can fully isolate the container depending on the CVS. If you have a container breakout, that might be not.

That possible to do if the breakout allows you to, to go around that. But again, it all depends on the workload that you're trying to save. If it's stuff that really needs hardware access, direct hardware access, for some reason, it's controlling a tape controller or a CT scanner or something like that, then it might not work inside the container due to the restrictions.

Depends very much on the workload.

Aaron: Yeah. And the ugly truth there too, is that we know that developers and system maintainers don't always go back. They'll architect something in development, which. Maybe has ports open that it shouldn't, or maybe has root privileges that it shouldn't. And then when that goes into production, I mean, you want to call it lazy, but I think it's also a complexity problem as well, where you just can't keep track of this stuff, but it happens and you get stuff in production that has root privileges or elevated privileges available or ports open and, and then it becomes very difficult to go back and fix that after the fact, so, yeah, stuff like that is just, and then of course when a CVE comes out that can take advantage of that, then that's really when, when the, you know, what hits the fan, so yeah, yeah, interesting, it's an interesting problem to think about, and seems like a really interesting solution.

That you offer both for. net, as we've been talking today, but for lots of other things too.

Joao: Yeah. And what you were describing there, you don't just do one thing. You don't just prepare one workload and then you just sit around waiting for any changes to come that need to be applied to that. You have lots of other things to do.

You have users to deal with. You have other problems to handle. You have lots of tickets popping up. I mean. That's just a day in the life of a sysadmin, that's just what you do. So focusing on just one thing like that, like you said, with that level of attention that, Oh, I need to remember to fix that part.

And I'm going to do it tomorrow. And I'll be able to go back to that as that never happens.

Jonathan: So I, I've, I've done some looking into some of the other live patch options and there's, you know, for the kernel in particular, there's a couple of others. And one thing that I remember seeing on some of those is that.

It's like, not every part of the kernel code can be live patched. And so then, at least in theory, you could have some CVEs, some vulnerabilities, that you just couldn't ever effectively live patch, because they're, you know, they're so, for lack of a better term, they're so deep inside of the kernel. Is that, is that a problem that you guys have hit, or do you have some secret sauce that makes it possible to live patch anything inside of this?

Joao: We don't have, we don't have that problem. Okay, we need to delve a bit more into life patching. There's the, there's a consistency problem, and this is true of all of the life patching solutions. You need to maintain consistency when you're applying a life patch. Right. What I mean by this is that when you apply a change to the code, it either has to go all in and be completely applied or none at all.

So that you don't catch code that's executing in the middle of a function, then you switch the end of a function, and for example, you remove a release, and now you have a memory, a memory loss there. You need to make sure that the kernel when you apply the live patch is in a consistent state. That means that when you're applying a patch to a function, that function must not be in use.

Okay? Kernel care is smart enough to wait for a moment where the functions that it wants to patch are not being in use. One of the very few CVs, and I mean, you can count them on the fingers in one hand, that we haven't been able to live patch, has been when one of the changes a few years ago changed all of the, all of the way that functions returned, and it was for one of those high profile functions one of those high profile C vulnerabilities out there.

This one's trampolines. It's

Jonathan: just trampolines, right? The retpolines, I think they call them. Exactly.

Joao: When all the functions in the code, the, I mean, the diff for that was massive. When all of the functions in the code had the, the return poll added there. So. You had to switch all of them at the same time. We could create a live patch, and we did create a live patch.

The problem was that there was never a good time where all of those functions were not in use simultaneously, so you would never be able to apply the patch. The, the problem was not so much with the live patch creation, it was that it was impossible to maintain consistency.

Jonathan: You would have to convince the colonel to park in Maine, and good luck.

I don't even know if colonel has a Maine function.

Oh, that's fun. That is an excellent answer for that. Okay, so one of the things that I've seen is, from you guys, and in fact in your

Joao: Sorry, sorry to interrupt. Let me just backtrack there. The, those, those issues that you were seeing with other solutions, let's say that they can patch certain functions, tie into the way that they apply the life patches and into the way that they treat life patching.

Canonical, for example, the, they will tell you that you have to reboot after 90 days. If you apply the life patch, we don't tell you that you can run the systems for as long as you want and you can apply the subsequent life patches as Often as you want, you will always be running an equivalent version to whatever is out there at the moment.

So you will never be in a state where your kernel is patched up to this level plus one or two more CVEs. You will always be equivalent to whatever is out there released. So the thing with with Canonical's approach, the way that they do live patching, for example, is that they will patch specific and individual CVEs.

And they will be able, and they will often leave the kernel in a state where they can't cleanly apply another one because there will be lots of different changes that they haven't accounted for. We're never in that situation with our solution. So that's why some of the life patching solutions out there and some of the vendors out there, they don't tell you that you can patch every single, every single CVE that comes out because they will often find themselves in dead ends where they can no longer move forward with life patching.

Jonathan: Interesting. I, I am very, I'm now very intrigued and going to your GitHub repo and learning more about how exactly this works because it is, it is pretty fascinating stuff. I, I want to ask one of the things that you mentioned in your notes that you sent to us was this idea of endless. And I guess that kind of goes along, but with NET 6, you're going to support it forever?

Question mark.

Joao: So we're going to support it for as long as there are people interested in us keeping it supported. The, the mantra here being that will probably outlast the hardware where you're running your workloads on. We can continue to support when that hardware dies, you'll be able to do that because there's no technical reason why we can't.

And that's the fundamental truth of it. There will always be a way forward and there will always be a way to change the code and fix the bugs that come up. There's no reason why we can't do it. So as long as there's interest, rather than saying, Oh, no, we will only provide you with the service for three years or five additional years.

What we would be doing in that situation was just kicking the can down the road. You would eventually find yourself in the same situation when our support ended. We don't want to do that for as long as you need the service for as long as there's interest, we have the technical ability to continue to support it.

Jonathan: Are there, are there other vendors out there that are doing something similar? Well, Provide you have keeping a, a fork of dotnet six up to date fixing CVEs, and then I'm, I'm, I'm really curious if there are other vendors doing this. Are you doing, are you working together at all? Is there a kind of a shared, semi-official do net six code base that all the CVE fixes are landing in?

Joao: No. To my knowledge, there is no such air code base. We're not working together doing that. For one particular reason, the distribution vendors that are doing that, Canonical and Red Hat, they are doing that only for their own private, their own particular distributions. Canonical is supporting it for particular Ubuntu versions, RHEL is doing that for particular RHEL versions.

Right. We support all of them. We're not distribution specific. We're agnostic in that we can support all the distributions that have the package. So we're not essentially focused on one over the other or on just doing the fixes for one. We cover, we create the packages for all of those distributions, which also helps.

In most infrastructure scenarios where you're not just having one single distribution running all the systems, all the distributions try to come up with some marketing strategy or with some advantage over the others. They're all essentially just running the kernel and some tooling, but some will tell you, Oh, we're better at doing raw calculations.

We're better for AI. We're better for file systems. We're better for whatever. So we try to get past that and we support all of them.

Jonathan: A lot of that just boils down to whether users are more comfortable using dnf and yum or app app to get

Joao: essentially It's more of a philosophical choice right now than the technical one, because you can do the same thing with all of them, essentially.

Yep,

Jonathan: yep, basically. Okay, so what about NET 7? It's out of support as well. Is that on the radar?

Joao: Not necessarily 7, but we're looking at the next one because 7 is one of the regular supported ones not a long term supported one. We could eventually do it, but right now I don't know if there are plans for it.

I honestly don't.

Jonathan: You would have to have someone come along and say, We really need this service for NET 7. Here, take our money.

Joao: That would be a reason, obviously. I mean, who wouldn't now, but again, going back to distributions, because this is something that's more relatable. There are distributions and there are distribution versions that get more usage than others.

For example, CentOS 8 had much less usage than CentOS 7. That's why it ridiculously went out of support earlier than CentOS 7. So it's expectable that more people are running CentOS 7 today and using extended life cycle support from us, the endless life cycle support from us than the ones that are using CentOS 8 because it has less usage overall.

The same is true with NET 6. It's just the more common Library out there. It's just more has more usage right now than seven because seven has a shorter lifespan. So people might just have jumped over it and went for the next one the next long term after that. Again, it goes back to the life cycle and the expectation of the life cycle that you get around dot net.

When you're developing applications, you don't just take a week to develop an application. It's usually a months long process or even years long process. So you're not gonna target something that's gonna go out to support a year after a month after you release. You're gonna go with the one that gives you the most amount of time.

So developers will opt for long term support.

Jonathan: Yeah. Yeah. Very cool. Alright, Aaron, did you wanna get anything else in before I move to our wrap questions?

Aaron: No, not at all. I, I mean, this has been very, I I've learned a lot actually. Yeah. Through the discussion. It's not something I typically touch on in my day to day for work because my work focuses mostly around Kubernetes and containers and monitoring, but it's tangentially related, right?

Because we're always running into customers that have. Oh, I've got mainframe still, you know that I've got to support or I've got cobalt applications that I've got to support And this is just one more of those where I also have legacy Applications written in non supported versions of programming languages like Python or NET that I also have to, it always happens, always, without fail.

And of course there's the issue with all the CVEs around Linux and everything else. So all that to say, I think it's a really interesting discussion and certainly a service that is pretty much required these days to have around, especially for big companies, you know, with very broad deployments.

They're going to run into something that's like, it's easier just to have someone support this ongoing than it is to hire Somebody to come in and write it over from scratch. It's just the way it goes. Yeah,

Jonathan: you know that that has a lead on question Aaron and that is is is the the tux care offerings are they x86 64 specific?

Or is there support in there for, you know, arm 64? Something like some of the old IBM's mainframe, like the Z series and all of that? Or is it just, is it strictly X 86 64? At this point,

Joao: when we're looking at do net, right now, it's X 86 64 right now. Mm-hmm . We have arm, we've tossed around the idea. If there's enough interest, we, we'll probably log into that as well.

Right now it's for X 86 64. Sure. Windows and Linux. We offer this for both Windows and Linux.

Jonathan: I did not realize that. That is good. I'm glad I asked. Alright, so we've got two final questions that we are absolutely required to ask everybody, and that is, what is your favorite text editor and scripting language?

Joao: Joe, which I believe is not a very common response there, but it was the one that I learned first, and I fell in love with it, and I I still remember the keystrokes from WordStar back in the day, and it's very similar in Joe, so yeah, that's one for Joe. The scripting language, again, I do everything in Bash.

Jonathan: Ah, yes. We had the creator of Bash on one time, and I asked him this question, and I asked him, Does Bash even count as a scripting language? And he, he sort of He sort of got offended that I asked the question. He's like, well, I think so, of course it does. I was like, okay, just, just checking. So, I have, I have it on a good authority that Bash, Bash counts and is an acceptable answer.

Joao: Could be years of learning.

Jonathan: Indeed, indeed. Alright, thank you so much for being here. It has been a blast talking with you about TuxCare and NET, picking your brain about all this stuff. We appreciate you being here.

Joao: Thank you very much for the invitation. It was a pleasure.

Jonathan: Yes. All right. Aaron, what

Aaron: do you think?

Yeah, I mean, it's something that I think a lot of people don't realize or maybe don't think about it unless you're working in one of those environments where you have to provide legacy support, not just because You want to, to give your customers a good experience, but because you have to because if you don't, you're either going to leave yourself open to some sort of security problem, or because you've got to comply with some sort of either government or industry imposed regulation, and there could be severe penalties if you don't I forget how many millions of dollars it was for.

I think it was something like, Okay. It was 500, 000 an hour or something crazy like that for the systems that I worked on that if, if they weren't available, that's how much we would get fined. Oh my goodness. And so we had like an eight hour outage one time and it was like, yeah, that's 4 million to the company.

Not nothing that I did. I was going

Jonathan: to say, I'd hate to be that intern that tripped over that power cable.

Aaron: No, no, no. I mean, that's, that's what does happen. You know, there has to be people out there that are willing to take care, or there doesn't have to be, but it's nice that there are people out there that, that are, are, are in this field.

To help companies with this because sometimes like I said, it is the cheapest and easiest alternative to Standing up a new dot net 10 whatever they're on now Application.

Jonathan: Yeah. Yeah, you know, I was I was basically not aware of tux care being I've always found live patching to be super interesting, but I was not aware that tux care was Apparently the name in live patching.

I'm honestly, I'm thinking now about my, my one lone server. I now have down in Dallas. We were talking before the show, do we have to do some server transfers? And I'm down to one. And it's like, I don't like rebooting that thing. So I may be contacting TuxCare. How much would you charge me to keep one server up to date so I don't have to reboot it again?

And that's a, that's fascinating.

Aaron: Yeah, for sure. For sure. I knew about TuxCare. But it never really looked into it specifically to see, you know, I just figured, oh, okay, they offer support and live patching, okay, whatever. But it's, like I said, the QEMU stuff, the virtualization things, the fact at the end there where they mentioned they support Windows, I didn't realize that.

So it's much more involved than I assumed it was. And so really interesting to learn about.

Jonathan: Yeah, absolutely. All right, man, do you have anything you want to plug?

Aaron: Well of course you can go to either of my YouTube channels, right? So, RetroHackShack or RetroHackShack After Hours. The my, my frequency of posting is a little less these days with my eye problems.

It just makes it more difficult to do things. And so, that should get better once that once I get that fixed, hopefully. In the future, but still posting a lot of videos. You can go out there and see, I'm working on one right now. Give people a preview of, I found a whole stack of old IBM PCs. And when I say old, I mean the original IBM PC, the 5160, which is the XT and the 5170, which was the AT.

So, original hardware from IBM, I found a whole stack of those that were missing their top cover and cards had been mangled and, and, you know, state, state of the hardware was unknown. So, I'm working my way through that as kind of a repair a thon or a salvage a thon right now, and the first of those videos should be coming out this weekend on the main channel, so go to RetroHackShack on YouTube and you can see a 5150 motherboard that I had a little trouble.

Diagnosing the problems with and fixing, but yeah, spoiler alert, I got there in the end.

Jonathan: Very cool. Yes, I, I very much enjoy. Watching the, the Retro Hatchback and the After Hours channel. I'm a, I'm a sucker for retro computing as well. And I, I like your, your combination of doing it on original hardware.

But also you'll take these, these gadgets that people are still making. A lot of them are open source and that's cool about it too. And you know, you plug it into the old hardware to be able to give it, teach the old dog new tricks as they say. I, I think it's a, it's a fun combination of things.

Aaron: Yeah, yeah, I love modding things.

I mean, that's my background. You know, I started a makerspace. I love 3d printing and all that kind of stuff. So I do try to work in both open source software and hardware and modding and making things into the channel, you know, as, as often as I can, because it's just what I naturally gravitate towards.

Yeah. Very cool.

Jonathan: All right. Well, appreciate you being here, man.

Aaron: Absolutely.

Jonathan: So we don't yet have a guest scheduled for next week. We do have Darko Fabian of Semaphore on the 18th. I have, I've had some real life stuff come up, and so I've been a bit distracted from the scheduling, so we are on the hunt for guests.

If you or someone you know runs an open source project and you want to be on the show, give us an email. It's floss at hackaday dot com, and that will come to me. We'll get you scheduled up. We sure appreciate that. You can also follow, of course, us. Also on Hackaday, the home of the show, you can follow my security column that goes live every Friday morning.

And that is my take on the interesting news on the, in the security world from the week. Lots of fun and interesting and crazy stories there. And then we've also got the Untitled Linux Show that's over at twit. tv. And we have a lot of fun there covering the Linux news of the week each week. And make sure to check those out.

We appreciate everybody that is here, both that get us live and on the download, and we will see you next week with another great guest on Floss Weekly.

Episode 823: TuxCare, 10 Years Without Rebooting!

5 mars 2025 | 64 min

Episode 822: Nand2Tetris

26 februari 2025 | 59 min

Episode 822 transcript

26 februari 2025 | min

FLOSS-822

Jonathan: Hey folks, this week, Rob joins me and we talk with Shimon Shachin about NAND to Tetris. That is the open course for college and high school students, all about starting with the NAND block as the basic building block of computer architecture, designing an ALU, a CPU, a full computer your own programming language, a compiler, all sorts of great stuff, and eventually ending up with the playable game of Tetris.

It's a lot of fun, and you don't want to miss it, so stay tuned. This is Floss Weekly, episode 822, recorded Tuesday, February the 25th. Nand to Tetris.

Hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host This is Jonathan Bennett, and today we've got a lot of fun planned. We've been fighting with technical difficulties, but that's okay. We're used to that. We're Linux users. And speaking of Linux users, I've got a co host today.

The one, the only Mr. Robert P. Campbell.

Rob: Excited to be here as always. Yeah, it'll be

Jonathan: fun. So you've got, you've got a little, a little bit of a teaching experience, right? Sort of on the, on the college level.

Rob: I have a little bit of college teaching experience, taught for a year mostly computer hardware software kind of like A plus stuff and then a little basically a security plus class.

So I've done a little bit nothing really in the programming side of of teaching.

Jonathan: Still, you've done more teaching at that level than I have. So I think probably than most of our other co hosts. So that's why it came to mind to invite you on because today we have Shimon Shacham of the Nand to Tetris.

Project and book and course and I have been recommending this for a long time since I first found out about it I just immediately went. Oh my goodness. That's one of the coolest things i've ever seen And so I reached out several months ago. I reached out to shimon and said Hey, would you be on the show and he was he was kind enough to to agree to it And this morning we went to get everything set up and ran into some technical difficulties So i'm gonna bring shimon on the show and he could say hi But we won't for those of you watching we won't actually be able to see him You have to be stuck with looking at Rob and I but welcome, welcome, sir, to the show.

I'm glad you're able to be here, even though it's just as a sort of disembodied voice today.

Shimon: Yeah, sure. Very glad to be here with you folks. And sorry that we have some camera problems, but I'm not, I'm not a great looker, so the voice will

Jonathan: do better. Alright, so I'm gonna start with the assumption that most people listening have no idea what we're talking about.

Give us kind of the intro, like what is Nand to Tetris, where would someone run into it, and why should geeks like us find it cool?

Shimon: Well, Nand to Techbase is a project, or a course, or a book, as you said, that guides you through the process of doing actually 12 standalone projects, 6 hardware projects and 6 software projects, in which You start with the humble, low level NAND gate, which is possibly the lowest logic gate that you can think of, and work your way through building a complete, modern, general purpose computer system.

Consisting of a typical von Neumann hardware platform and a typical software stack consisting of an assembler, a virtual machine, a compiler. And a little operating system so that at the end of this journey, you have a computer on which you can run or write and execute any program that comes to your mind, including, of course, Tetris.

So. That's the story.

Jonathan: Yeah. Where did, where did the idea for this come from? What was the spark of inspiration where you said, let's, let's put all of this together in a course?

Shimon: Well, I developed this course together with my colleague Noam Nisan, who is like me, a computer science professor. And both Norm and I served as, um, directors of various programs computer science programs, undergraduate, graduate, and so on.

And we were always frustrated from the fact that we are teaching Many different fragmented courses and students are getting all sorts of I, I, I should say rather deep views about various corners of computer science and, you know, mostly applied computer science, but they have no idea how things work together.

So, you know, they may understand the hardware and they understand the compilation. They may know to write programs in Java and Python, but there's no one course that puts all these pieces together and lets them sort of see the forest for the trees. So, the idea was to To sort of plug all these holes, all these interfaces between the different courses that one typically gets in a computer science program and glue things together.

So this was sort of the underlying pedagogical motivation. And the other motivation was just the, you know, the fun of building something big. And And, and taking the time to do it because both of us are, you know, busy people like, you know, all of us. And so this became kind of on the side work of love that that we did for several years, you know, before we actually.

Jonathan: I would, I would pull my copy of the book out and show it to everybody. It is on loan to a teenager here in town that I heard he was, you know, interested in computers and languages, and he'd done a little bit of programming. It's like, I want to take the next step. Well, if you're adventurous, I've got something for you to try.

So that's where my copy of the book is at. And I've, I've got to admit, I've not made it all the way through the process. I've, I've built the, the ALU. I may have finished the CPU design. I was somewhere, somewhere along those steps when. Real, real life busyness took over and I had to step away from it. But I was actually, and this is what I want to ask you about.

I was actually going through one of the courses that are offered. And so this, this is a course that just anyone that's out there that wants to take it. There are, there are some opportunities for just someone to jump on the internet and take it, right?

Shimon: Yeah, absolutely. The, the course is available presently in Coursera.

And pretty soon it'll be available in, in other platforms as well and anyone can take it. And I think you have to pay something to Coursera in order to get a grade in the course.

Yeah. But you don't

have to get, but you don't have to get a grade. You can just take the course. As an auditor and do everything, do all the projects.

And, you know, when you do a project, you submit it to a server that we control. So even, even if you submit it through Coursera, it ends up in our server. So anyone, you don't have to be a registered to Coursera to submit any, to submit stuff. And so you submit it to our server. We grade it, you know, we, we, we test everything that you do.

We, we, we send you a grade. And and then, you know, the grade is just a nice feedback that you did what was expected for you. And so, you know, because we work with a version of HDL, how to description language that we Designed for our course. And we also wrote a language that, you know, we called it TDL, Testing Description Language.

And we wrote many scripts for, for every chip that you develop. And so when you submit your chip to our server, we run a test script on it. And if the chip produces the expected outputs, you know, we compare it to a compare file, then, you know, you get a pass grade or, you know, whatever you want to call it.

And then the student knows that. that he or she did what was expected. And so anyone can take this course. And surprisingly, you know, we didn't expect it, but it turns out that the hardware part of the course does not require any prerequisite knowledge. Now, you don't have to be a computer science student because everything is based on the basic logic.

And you know, logic is something that anyone can understand or pick up easily. Yep. So You can build, you know, you mentioned the ALU. So the ALU is built, you know, is designed on, you know, it requires to put together some, you know, endgates and muxes and so on. Nothing much more beyond that. And then you have a working ALU, which is really exciting.

You know, moving on to the CPU is, you know, a few more steps. It gets a little bit more complex because now you need what is known as sequential logic. You know, you need to print this clock and so on. But once again, we have very detailed lectures, video lectures. The book is very detailed. And so.

Anyone, practically speaking, including high school students, can, can build a computer. The hardware part of the computer.

Jonathan: So I wanna, I wanna dive more into that. I, I have a question first though, because you mentioned it. That you guys have your grading server. And I'm sure Rob has questions he wants to get to as well.

But you have your grading server where people send in their programs and you kind of simulate them. Has, has anyone ever tried to be particularly clever and do something like really surprising? Have you, have you had anyone, you know, escape your sandbox and, and cause havoc on the grading server?

Shimon: Not that I remember, but no, it's an interesting thought.

Yeah, I should have expected questions like that from you guys. But no, I think so far, you know, people behaved like happy campers. And I think all they want to do is submit a cheap. Get a grade, you know, stay alive and move on to the next gyp, so.

Rob: Yeah. Go ahead, Rob. So, you know, this being all online, is the book required to take this course or is that supplemental material?

Shimon: The book is not required. It's everything that written in the book is given in the course lectures. It's nice to have the book, you know, you know, in your hand. It's either as a souvenir or as some additional source of documentation, but but it's not required. And and so I think most people who take the course don't buy the book.

Rob: Yeah, but it is reasonably priced, especially for any kind of course textbook kind of kind of book. Cause I, I took, looked it up before the show and US dollars, it's on Amazon, it's less than 30. So that's definitely reasonable for most people.

Shimon: Yeah, we insisted when we When we made the deal with the publisher, we insisted, we insisted that the price would be relatively low because, you know, many textbooks in computer science and maybe in other subjects as well are very expensive.

So in our case, it's I think the price is okay.

Jonathan: Oh yeah, definitely. And also, I

Shimon: must say that we put half of the book online, so the first half that I mentioned, the hardware part, is available in PDF form online, and so that's another reason why, you know, you don't need to buy the book, although I'm happy to say that many people buy it, so.

That's always

Jonathan: nice.

Shimon: Yeah. You're welcome. Yeah, I think it's a nice souvenir.

Jonathan: Yeah, oh, I would encourage anybody that wants to go through the class, or even that just wants to learn this stuff, buy the book. It's a great book as well. And to your point of it being simple enough for anyone to go through, that was what I was struck by, is that like, the way, not only the way you approach it, but even the way The whole, the Von Neumann architecture works.

When you start with the NAND gates, it's all very incremental. Like, there's, this is a weird thing to say. There's nothing about modern computing that's actually black magic, right? It's all very incremental, one layer on top of another. And once you understand what this layer does, the next one really makes sense.

And it's, I would, I would make the argument that it really is accessible for a lot of people if they're just, if they have the time to do it and are willing to put the work in. And that's one of the things that I think is just the coolest about this is that it gives someone that, that top to bottom understanding of a, admittedly, a very simple computer in comparison to the, you know, the behemoths we have on our desks, but it's still the, the exact same principles at play.

And that's, that's, it's, it's a service. honestly to the community. And so for that, thank you.

Shimon: Yeah. Well, thanks. Thanks for making this observation, which is very, very true. You know, both of us are computer scientists Norm and myself, and we are trained and and, and, and we enjoy. You know, this, this trend on and enjoy logic and and first principles.

And, you know, whenever we write the paper or do some work, we have to start with the first principles and prove, you know, every step in the way until we can make a solid point about anything. And we, we decided to build the course exactly in the, in that, in that spirit. So as you mentioned, you know, the computers that you use today are, are monsters and or monstrous and, and they are so complicated that that it's very difficult to.

To explain, you know, how a real computer works because there's so many layers of hardware and software and proprietary stuff and so many interfaces that it's impossible to explain how a real modern computer works in a typical academic course or to some, you know, layman or high school student. So one of the first decisions that we made.

Was that we're going to design an architecture, which first of all, will be relatively easy to build. And second of all, we'll have will be, you know, I guess I should use this word that it will be beautiful, something which is very elegant and very simple and the same time, you know, very powerful. So and I'm glad that You know, we, we managed to do it.

Jonathan: Yeah. Yeah. It's good stuff. With that idea of, of beauty said, I'm curious about the, the license behind all this, right? So we're open source enthusiasts here, and obviously you guys have made this sort of an open course. It's on open course where it's, it's open, open source friendly. What I guess the book itself, you said about half of it is completely available online.

And then I know a lot of the example code there's an IDE, I think. that is also open source. What what's the, what's the license and what's the story with all of that?

Shimon: Well, everything is wide open. I don't even remember the exact license that we are using, but it's a common what's the term?

I forgot.

Jonathan: I think, I think some of it's creative commons at least.

Shimon: Yeah, exactly. It's creative commons. Anyone can, uh, you know, fork our ID and, you know, do whatever he or she wants with it. And Same goes with the lectures. You know, it's, it's I think one reason why this course became very popular is that because not only anyone can take it, it's also anyone can teach it.

So, if you want to teach this course, all you have to do is send me an email, and I'll give you access to all the lecture materials and all the projects you know, past exams and so on. So, very easy to Adopt the course and teach it anywhere you want and as a result the course is now being taught in something like 400 universities and that's awesome numerous numerous high schools and hacker clubs and and whatnot so so yeah, it's widely available

Jonathan: I think it really appeals to that sort of original hacker mentality of, I want to understand this thing on my desk as much as I can from top to bottom.

And as far as I know, Nanda Tetris is about the only course that really offers that. I'm not aware of anything else.

Shimon: I think it's the only course that offers a complete ascent, if that's the right word, from basic logic gates to a high level programming language. There are quite a few courses that teach you how to build a computer, but typically they get stuck with well, getting stuck is not the right word.

They, they go all the way up to a symbolic machine language and an assembler. And that's it. We, we go the extra, you know, distance and we, we also cover a very nice gap between machine language and high level language. It looks pretty much like Java. You know, we, we decided to design our own language.

We call it Jack. Now the machine is called Hack. The language is called Jack. And it's a Java lookalike with object orientation and no inheritance. That's one major concession that we made. And, you know, we make all sorts of concessions so that students will be able to build significant tools like compilers and little operating systems in maybe two or three weeks, you know, each.

So, so you have to, you know, you have to. make some compromises. And also, you know, when you, when you stick, when you, when you, let me give you another example. Take a compilation. If you look at a typical compilation course, college level, a great deal of the course is devoted to handling errors.

Compilation errors. Now, this is, you know, terribly important, you know, error diagnostics, error handling, and so on. But, when you deal with errors, you often sort of lose sight of the most fundamental ideas, which are program translation. You know, taking one language, parsing it, analyzing it, and generate, generating code from it to another language, which I think is, is the essence of, of translation.

And so we decided that when we write a compiler, we will make an assumption that the source code is error free. There are no errors, and once you don't have to deal with errors, you can focus on, on the most important ideas in compilation, in my view. And so we made several such concessions in, in, in many different steps along the way.

And the nice thing about all these so called missing things, in Antutu is that every one of them is a great extension project. You know, you want to handle errors? Great. You know, you want to add some features to the high level language? It's a very nice extension of the compiler, you know. So, for example, we don't have a for loop.

You know, we have while, we have if. We don't have for. You want for? Great, edit. You know, we give you everything that you need in order to edit yourself. And so, indeed, over the years people made all sorts of incredible extensions and additions to the BASIC platform. And which, you know, gives us great joy too.

Every once in a while, I get an email from someone who says, look, look what I did. And and I try to publish some of in in the non detectors website. So, You know, we have a section called cool stuff in which you can see some of the extension that people built. That's

Rob: great. So, so in this project, you build up a virtual computer, virtualized, you know, real chips and all up to the computer to, I think you said, like, even like an assembly language layer to your own Java like programming.

And you said, you know, the language is the Java language is your own, but is the assembly like at the step where you're at assembly, is that like. Real assembly or is that assembly like that works with the chips that are made in this? Yeah,

Shimon: it's the assembly language is also our own quote unquote, proprietary language.

I mean, we made it up and, you know, when you build a computer, you have to design also the instruction set. And so, we designed the instruction set, and it comes in both binary and symbolic versions, and it's a very elegant language. And it's, it's really a pleasure to write programs in our assembly language.

And it's, it's much more user friendly than a typical assembly language. And you know, a little bit difficult to talk about it. I mean, you have to put your hands and play with it and write some programs and see for yourself. But it's I think students who write were introduced to assembly language using Our language, you know, gets very smooth and user friendly intro.

And indeed, this was, you know, we made many different sort of major decisions along the way, and one of them was should we develop sort of a stripped down version of an x86 or a RISC machine? And, or, or maybe design our own. And, and we decided that if you want to do this elegantly we should, we should do, we have to build our own our own design.

So that's what we did. So the assembly, once again, is our own language.

Jonathan: Yeah. It's real assembly, but it's custom.

Rob: Yeah.

Shimon: Exactly.

Rob: So, so having that assembly layer theoretically one could create their own. language above that instead of using your, the jack, I think you called it. Yeah, or change their jack, however they want.

Shimon: Well, yeah, you can. Well, you know, first of all, you can easily, and this is something you can do in high school also, you can build a macro assembler and design all sorts of macro commands and bring it to a level where you have a C like language or a basic like language. And And this, you know, is a significant step up in writing high level code.

But if you want to implement a powerful object based language, you need a compiler. I mean, you have to, you have to go through parsing, code generation, and we decided also to introduce a virtual machine. And the reason we did that was because You know, most computer science students today work with either Java or Python, and both Java and Python have a two tier translation model.

So the high level code is translated into some virtual machine, some intermediate language, and then the intermediate language is translated further into assembly. So that's what we do also in our course. You know, we thought it's important to, to train students to understand this elusive VM that sits in the middle of everything they do.

Jonathan: I, I, I'm looking at the, the cool stuff and one of the ones here that basically for as long as I've known about Nand to Tetris, it's what I've wanted to do with it. And that is running, running the, the virtual CPU as a real CPU in an FPGA. And I assume I'm not the only one that just immediately thought of that and thought it would be the coolest thing to do.

Shimon: Yes, absolutely. It's a very nice project in which you take the HDL programs that we wrote that you wrote in Nandu Tetris, and you Quite easily migrate them into either Verilog or VHDL. It's a very simple process. And then using the incredible FPGA world that we have today, you can actually sort of commit this whole thing to silicon.

And have a little board that, you know, with which you can play Tetris and any other program that you can think of and, you know, show off this board in job interviews or to your parents or whatever and, you know, tell them, you know, that that's a computer that, that I built from scratch and built it all the way from NAND.

It's a very empowering feeling.

Jonathan: Has anybody ever laid it out on, on actual NAND chips without the FPGA?

Shimon: Yes. There was, there was one project in which, uh, someone built the whole thing from using what is known as RTL or sort of real elementary logic gates. And, I don't remember if you did, if you went all the way up or just built the the CPU, but yes, people did that as well.

Jonathan: Yeah. I, so I'm sure you're familiar with Tiny Tape Out.

Yes. Yeah. Has any, has anybody managed to convince the, the powers that be that, The NAND to Tetris CPU, that it should be on one of the Tiny Tape Out builds? I don't know,

Shimon: I have to check, I don't know.

Jonathan: I see here where it looks like someone tried to, I just, I can't, I can't find out for sure if they managed to You know, because lots of people send in their designs for Tiny Tape Out.

For those that don't know, Tiny Tape Out is, I think Google is one of the companies behind it, but several companies. What they do is they say, okay, we're going to do a run of this. It's an older stepping, but a run of a lithographic process to build, essentially like you would build CPUs. But instead of doing one big CPU design, they'll say, people from the community, give us your designs, and they'll pick.

Oh, I don't know, 40 or so designs because they're all much smaller than a modern CPU. And so they'll do chips with all 40 of those designs on them and then just pin them out differently. And so that you can get, just like students and people researching, they can get real hardware designs done in lithography on these chips.

And being able to get a hacked CPU on a tiny tape out chip would just be It would be the, the, the ultimate. Well, I mean, it would be a real feather in y'all's cap too, but it would, it would be neat.

Shimon: Yeah. Actually, I have a friend by the name of URI, who who is very active in this in this area.

And, uh, we've been talking about it. So I'm sure that we'll do it. Pretty soon. And so if we get it done, I'll let you know. And I'm sure you know, maybe you can put something about it in a heck of a definitely. Yes, definitely.

Jonathan: That is our that sort of thing is our bread and butter. So I okay.

Every student that goes through this, do they sort of end up with the same design? Or like, is there, is there room for creative differences? Right? So like, am I gonna put all of my instructions are the same, you know, machine codes gonna be the same instructions for everybody that goes through it?

Just for one example.

Shimon: Yeah, well

you know, this course is It's very demanding because every week you have to come up with a significant module, like like, for instance, in project one, you have to design 1515 basic chips, and then in project two, you have to design an ALU, in project three, you have to design memory unit, RAM, and so on.

So, there isn't much time to come up with your own designs. And yet, toward the end of the course, if time permits, We have a special session, sort of one or two weeks, in which we encourage students to come with all sorts of extensions. So, it's not that they come up with different computer models, but they come up with different extensions.

So, one student may develop a literal disk. Another student can develop a nice extension to the high level language. There are many different possible extensions. You know, you can develop an, you know, HTTP server using our system, you know, make two head computers begin talking to each other and so on.

By the way, I mentioned before I forget, I mentioned Uri Shaked his website is called walkway. com. It's W O K W I. com. And it's a wonderful website for TinyTap and TinyTapOut and related projects. So you may want to take a look at this and maybe interview him at some point.

Jonathan: Yeah, yeah, that'd be great.

So to that, to that question about the differences between different students, you got in, in the course, the, the sort of the framework is provided then, like you, you guys have already assigned, you know, this machine code is going to be this instruction. You know, this one is gonna be this one. This is the way that the CPU is gonna be pinned out.

Shimon: Yeah. Well, is that, you know, we, we provide only the OB obstructions, you know, we mm-hmm. We describe, you know, if you take, for example, the alu, we describe the language or we describe the the, the arithmetic and logical operations that the air should perform. Mm-hmm . And, but, but we say nothing about implementation.

You know, we expect students to come up with their own implementations. Mm-hmm . So you have to decide how you put together the lower level chips in order to deliver the the required functionality. So the functionality is the obstruction that's on us. The implementation is of new. That, that's the deal in, in every one of our projects.

You know, we and you know, one reason why. Students can build, for example, a compiler in two weeks is because we, we are very, you know, very careful in giving, giving them very specific obstructions, very specific specifications, and We also give them the design in the form of APIs. So we might give them an API for a passer, an API for, for a code generator.

And the APIs are proposed. They're not you know, they're not mandatory. Mm-hmm . But if you follow our APIs, you will realize that. You know, every method that we specify in the API comes with a, with a test method that we wrote. So you can immediately put your work to test and you can, you know, you can unit test everything.

So, once again, you know, you get the overall design. Which is what I call an obstruction. And then you have to implement it. And you can implement it, you know, in any language that you want. And that's another nice feature of the course. You know, if you build a compiler, you can build it in Java or Python or Scala or Ah, right, right, right.

Yeah, or Perl is probably the best language for these things. It's up to you.

Jonathan: How about in Jack? Is it possible to build the, is it possible to eat, for Jack to eat its own dog food?

Shimon: It's, it's difficult. It's difficult. It's, it's possible. But, but it's, it's challenging. It's, it's a nice project. Yep,

Jonathan: understood.

So it's, it's one of those good add ons where someone can do it if they want to. But you're not going to require it from everybody. So that answers the question that I was thinking of, so like, if one person writes A Jack program on their CPU, they can share it with their classmate or someone around the world for that matter, and pretty much know that it's going to run on the CPU and the, the, the system that someone else designed because you guys kind of lay out the, the, the, the bones.

Maybe we could say that that assures that all of this, for the most part will work together the way it's supposed to. Yeah.

Shimon: Executed on any machine that was built in any of the Tetris cores.

Rob: Are there, are there any Jack repositories? Well, we don't have a

Shimon: repository. We should, yeah, we should build one. There are numerous computer games that people built with our computer. If you go to YouTube and type non tetris games, you'll find, you know, thousands of games.

And these are typical, typically they are kind of classical retro games like Space Invaders, Snake, Tetris, Sokoban, Maze Games, and so on. And You know, Jack is surprisingly good for interactive graphics applications, you know, as long as you don't do 3D modeling and things like that. Although, I must say that, you know, some people even did ray tracing and very fancy image processing without computer, which was quite astonishing to see.

And yet, once again, most programs are kind of classical computer games and there are thousands of them.

Jonathan: I, I see a YouTube video here. It's linked to from your cool stuff page. It's Hackenstein 3D, and it's a it's black and white, but it's sort of a re implementation of the old Wolfenstein 3D game. Yeah, and

Shimon: it's impressive.

It's impressive. It's like, you know, real time graphics. It's a very nice game.

Jonathan: Yes. Yes, very much so. It's very cool.

Rob: Do you know if anyone's re implemented your jack on just a regular computer at all?

Shimon: Yeah, that's a great question. And I don't know, but I think it's a very nice project, you know, to implement Jack on, you know, maybe on, I don't know, on your cell phone or whatever.

There's no reason why it cannot cannot be done because all you have to do is, you know, you have to generate VM code. For a virtual machine, and then you have to write a VM translator that translates the VM code into the assembly of any computer that that is out there, you know, so what is missing is is that trust is sort of what is Sometimes referred to as the back end of the compiler That's the the translator that translates VM commands You know, push, pop, and so on to assembly.

And the target assembly can be on any computer's assembly language.

Jonathan: Yeah, that's cool. So obviously there's, there's a lot of these add on projects where people have done extremely impressive things. Have you guys thought about a follow on course, like a next year? Because I think Nanda Tetris is a full year long, right?

Shimon: Well, it depends. I think the recommended pace is, yes, it's a, well, I'm kind of, I'm, I'm, I'm selecting my words carefully because I'm, you know, I guess some people are hearing this and they're not sure, you know, what's the right format to teach this course. So I should say that in many universities, Nantu Tetris is taught in one semester only.

Jonathan: Okay,

Shimon: but it's very intensive and you know, it's, it's, it's also tends to cannibalize, you know, many other courses because the students are very engaged and they end up spending, I don't know, 15, 20 hours a week just on this course. And so that's, that's what many people do. That's what I do. But in other universities, it's It's being taught in two semesters.

One semester to build the hardware, another semester to build the software, which I think is a more sort of normal pace. So it depends. It's either a two semester or one semester course.

Jonathan: And then the follow on question to that is, have you guys thought about a second cut to it, you know, once you have the simple software

Shimon: done?

Yeah, so, you know, we've been approached by many people who wanted to do something like that. And we always Encourage other people to do it, but I'm not sure that we have enough energy to develop a sequel. But you know, we've been thinking about some, some wild things like, believe it or not, even something I called Named to Life.

And Named to Life is a course in which you, you know, in the same style of doing everything from the ground up, you begin by building You know, very basic sort of DNA sequences.

And then

from the DNA sequences, you build proteins. And from the proteins, you build ribosomes. And before you know what, you have a little functioning cell.

Yeah.

You know, so, so that's, you know, one thing that we've been thinking about. And in the same spirit. We thought about building logic itself, you know or a Turing machine or something like this from basic logical constructs in the process explaining, you know, what is an axiom, what is the Theorem, what is the proposition and building everything from the ground up in the form of little objects, you know, that you can build in a language like Python.

And so I think, you know, the basic spirit of things, you know, which I think is very characteristic in what, what Norm and I are doing is that I should say, you know, what innovation of non contemporaries. is this notion of bottom up construction. And I'm saying this because computer science is very much a top down field.

And when you start a computer science program, the first thing that you do is you learn a language like Java or Python, you know, which is, if you think about it, it's it's, it's something which is so farther away. From the real metal, you know, from the real computer. And so, you begin with the most high level things in computer science, and then, through the program, throughout the program, you work your way, sort of, downward.

And at some point, you begin to talk to learn about hardware, operating systems, typically TEPPEN. Your third year of studies or something like that. And we, I think Nantou Tetris has shown that there's also a great virtue in doing things the other way around and starting from scratch, you know, and building everything from very simple building blocks.

And, you know, one reason why it works is because it's very exciting. You know, it's very exciting to do something simple and see it becoming more and more powerful, and more and more potent, and to understand every step along the way, which is terribly important. You know, there's nothing hidden. You know, everything is out there, and you can play with it and extend it and do whatever you want with it.

So So unlike a typical computer science student who who thinks that the underlying computer, you know, maybe, I don't know, a coffee machine or something, you know, you don't know what's really working there under the hood. In Nandu Tetris, you build the whole damn thing, you know, from, from A to Z. And I remember when we, when we, when we wrote The abstract of the book, and we sent it to the publisher.

The publisher sent it for a review, and two of the reviewers were two well known professors who wrote a wonderful book called Structure and Interpretation of Computer Programs. It's a very famous book that came out from, from MIT. And it's the ultimate top down computer book, because it starts with a language called LISP.

And, you know.

Which is a very high level,

yeah, very, very high level abstract language. And so when I heard that they're going to review our book, you know, I was kind of very worried. But the review that they wrote was very warm. It was, you know, exactly the opposite of what I thought. They said that you know, what a refreshing thing it is to see a bottom up approach to teaching computer science.

Because we, we used to do everything the other way around. So, so I think that this, this notion of bottom up Construction has you know, many people find it engaging, many educators, and so I'm sure that we'll see more non tutorialist courses non tutorialist like courses in, in, in the future.

Different areas. I'll give you one more example communications and networks, you know would be very nice to build, you know, all the, all the protocols, you know, from, you know, starting with having two machines that send bits to each other. And then build the whole, you know, OSI layers and everything and end up with a web browser and an HTTP server somewhere.

So that's another very nice project, you know, sort of, you know, building the internet.

I'm also, I'm

Jonathan: also struck by the idea that we should show the electrical engineers some love and that there should be an electrons to NAND course that actually talks about how you get to the point of a NAND gate.

Shimon: Yeah, that's a huge

Jonathan: missing,

Shimon: you know, missing gap in our journey. I've seen

Jonathan: somewhere where someone has written a chapter zero, there may be more than one, but at least one person has written a chapter zero talking about, okay, this black box that we're calling a NAND gate in this course, what's actually inside of it?

Rob: That'd be a great prequel.

Jonathan: Yeah. Yeah,

Shimon: absolutely.

Jonathan: Yeah, it, it really, it really does seem like there's a, there's a, there's a great niche there for writing a couple more courses, maybe more than just two that sort of slot into this same idea. And I think, I think that one would be great. The Electron to NAND as sort of an intro to electronics, getting up to the NAND and then.

You know, maybe on the other side of it, there needs to be a build your own GPU, you know, a very, a very simple GPU, like the old retro computers used to have that can do sprites and such. But then that idea of building network protocols and, you know, maybe you re implement token ring and then finally Ethernet that.

That would be very cool. And then, of course, on top of all of that, you have to, you have to talk about computer security. And so then, you know, hacking the hack chip just naturally falls in there.

Shimon: Yeah, absolutely. You know, another, another very important building block in, in, in all such courses is to build.

A suite of simulators that kick in, kick in, in every major step along the way.

for example, in Antutu, we have, you mentioned, you know, we have our IDE and the IDE includes a hardware simulator, a VM emulator. a CPU emulator, an assembler, and so whenever you build a significant piece of software, you can actually test it and play with it before you actually set out to build it.

And we have executable versions of everything, of the compiler, the VM translator, the operating system. So And you have to do something like that in every non tetris like course, you know. So, you know, you play with DNA. You need some, you know, DNA simulator. You, you, you get it up to the level of proteins.

You need some, you know, some way to, to simulate how protein Things are, you know, being constructed, and so, and when you do that, you also get some very nice decoupling of the various topics, you know, you can sort of teach, teach the course in different orders, if you like so yeah, you know, I was thinking about you know, preparing a little lecture that sort of focuses on On the various tricks that, that Norm and I used in order to make this course, uh, feasible.

And, uh, you know, all the concessions that we made, all the tools that we developed, and, uh, coming up with a whole litany of, of, of advice, of tips to, to people who want to develop similar courses. You know to find time to do that.

Jonathan: Yeah, that would be, that would be really interesting. I, I, I like I like this style of teaching and learning a lot.

So in, in my high school education, one of the educators it's actually a quote from scripture from, yeah, and it's to him that understandeth knowledge is easy. And I am, that was, that was very much their viewpoint on things. And I'm really seeing echoes of that in the Nanda Tetris course, because it's all about, if you understand the layer below, then when you go to the layer above that and go to work on it, you understand it.

So actually doing the work is fairly easy. And I think, I think that's, I think that's great. I just. I'm just in love with the course. It, I, it makes me want to go back and start it all over again, get my book back or go buy another copy and actually go through it again. I don't think I have actually. I have to

Shimon: tell you that, you know, the best, the best way to learn it and to build a computer is to actually teach it.

So, you know, find a few people who are interested in this stuff and just teaching them to test this. Tetris course, it will be a great pleasure. Oh yeah. Hmm.

Rob: Community class.

Jonathan: Do a community class.

Shimon: Yeah, exactly. Exactly. That's the way to do it.

Jonathan: That would be a lot of fun. I don't know if I have time to do that.

I've got, I've got too many businesses I'm already running and juggling, but it would be a lot of fun. Maybe one of these days. Speaking of which, if someone wants to either take the class or run a group through the class, what are some where would you tell them to start?

Shimon: Well you have to go to nanddetectives.

org, our website, and Everything's there. It's, it's NAND, N A N D, the digit two, tetris. org, and you will find there are links to the Coursera courses. You will find an email that if you send us email and just say that you want to teach a course, I will give you links to to all the materials.

You will find links to our IDE, to all our programs. Projects, everything is, is, is in the website.

Jonathan: Very cool.

Shimon: So that's the go to, that's the go to place.

Jonathan: Yeah, that's, that's neat, that's neat. And then people that are taking the course, is there sort of a a hangout place? Is there like a Discord server or anything for, for students going through it?

Shimon: Not really. I mean, we we didn't want to Sort of, you know, we didn't want to put together something that we cannot support. Right. So so you want to teach the course? I'd say you can build your own space and do whatever you want with it. Up to you.

Jonathan: Yeah, I would, I would love to see maybe a little bit more official, easy to use repository one of these days for, for people doing these add on projects.

Like, look, I added inheritance to Jack. Here's my code. You know, I, I wrote this video game engine. Here's, here's my code, you know, put a GitHub or GitLab server up somewhere. I think that would be fine.

Shimon: I agree. I guess. I guess we have to do it. It's too, too lazy, but we'll do it at some point. I understand

Jonathan: that part.

Yeah, so do I. I mean, you guys surely don't have anything else keeping you busy, right? Exactly. Oh, okay. That's cool. Is there anything coming down the pike and beyond our source code server that we just established that we're going to create? Is there anything, is there anything you guys are excited that's coming up that you want to plug?

Shimon: Yeah. Well you know, I'm so glad that we. We managed to talk for about almost an hour without mentioning the two letters A I. But one thing that I'm working on now is pouring all this stuff, you know, all the course materials, the book and everything, into a larger sort of language model. And and providing a chatbot in which you can actually, in which you can actually interact.

with so called me, you know, with an avatar that represents me, and you can ask any questions that you want about Nand to Tetris, projects, teaching, lectures, whatever, and, and hopefully get some sensible answers, so Hopefully, yeah, so I'm working, you know, I'm not going to release it until until it will be good looking and sensible, but that's something that I'm working on.

And other than that plug in, I don't know, you know, I don't know. I think I'm quite happy with what we did so far, and I don't want, I don't want to sort of describe all sorts of half baked, half baked ideas. So, I'll let you know when I'm ready for another interview. Okay,

Jonathan: we can, we can do that. Yeah, for sure.

Alright, good. Okay, so I've got to ask, we are, we are required by our audience to ask you two final questions before we let you go, and that is, what is your favorite text editor and scripting language?

Shimon: Favorite text editor is Tico. If you, you know, sort of an Emacs like editor, and scripting language?

Yes. Perl. Ah.

Jonathan: Yeah, pretty cool. Okay. We don't get very many Perl answers, but somewhere Randall Schwartz is, is applauding and giving you, you know, a virtual high five. I started

Rob: on Perl, so.

Jonathan: Pearl, yeah, Pearl's one of my first languages too. I moved away from it pretty quick, but I enjoyed it back in the day.

Every once in a while I'll pull it out and do just a little bit in it. Okay. Alright, Shimon, thank you, thank you so much for being here. It was an absolute delight to get to chat with you and talk about the project. And like I said, I'm, I'm gonna have to carve out some time to get back to it and go through it again.

I just enjoy it so much.

Shimon: All right. Thank you for having me. Bye

Jonathan: bye. Bye. All right, Rob, what do you think? You gonna, you gonna teach the course?

Rob: Well,

Jonathan: I don't know if I'm going to set

Rob: up, I don't know if I'll set up a community class or anything like that, but definitely heard about it. It gets me excited to try it out.

I don't know if with all my commitments, if I'll be able to, but I may at least start with buying the book and, you know, baby steps.

Jonathan: Yeah, yep.

Rob: And then get into it, you know, take a, take a little bit into it. But that seems really awesome.

Jonathan: I, I really want to get the the recommended FPGA and screen and set up and do the do the actual hardware version of it.

To be able to have the thing running now in an FPGA, but still be able to run the thing for real. That's just, that, that tickles every geek bone in my body. I love it so much.

Rob: Maybe we know someone who can give us some chips.

Jonathan: Yeah, there you go. I don't think that company makes FPGAs, but.

Rob: Well, I don't know.

Guessing they're all NAN chips or something. Yeah,

Jonathan: yeah, I'm sure he can give us lots of NAN. All right, anything you want to plug, Rob?

Rob: You know, if you want to check out more of me, come find me. My website is robertpcampbell. com. There's links to various social medias there. Otherwise you can come join me and Jonathan over at the Untitled Lennox Show on the Twit Network every Saturday at 5 30 central time.

Jonathan: Because central time is all that matters, right?

Rob: Centralized for a reason.

Jonathan: Alright, thank you for being here man, I appreciate it. Alright, if you want to find more of me hiding behind that command line window where I've been moving virtual machines around, back there is Hackaday. You can find my stuff on Hackaday, that's where Floss Weekly is.

We appreciate Hackaday being the home of Floss Weekly these days. It's also where my security column goes live every Friday morning. Friday morning, central time. The only time that matters. And then if you want more of Rob and I, there is the Untitled Linux Show over on the Twit Network. And other than that, we just appreciate everybody being here.

We always have a lot of fun on Floss Weekly. And we will be back next week for more. And we don't have a guest scheduled yet. So if you know of somebody Let us know, either come into the discord and tell us, tell us you want to be on the show or I believe it's floss at hackaday. com. You can send an email there.

We will get it. It'll come to me and we'll make it happen. Appreciate it. Everyone that watches that listens both live and on a download, and we will see you next time.

Episode 821 transcript

19 februari 2025 | min

FLOSS-821

Jonathan: Hey folks, this week we chat with Gregory Kertzer and Krista Burdeen about Rocky Linux, it's history, based in CentOS, where it's been, where it's going, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 821, recorded Tuesday, February the 11th. Rocky Linux.

It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something, we've got a treat today. We've got something really fun. We're going to be talking about CIQ. That is a technology company that you may be familiar with. Better know as the home of Rocky Linux.

And we've got Gregory Kurtzer and Miss Krista, whose last name I've forgotten. Shame on me. But we've got both of them for the show today and I am once again, flying solo, no co host today, definitely need to work on that scheduling issue. We will get that straightened up for our next show. I am sure. But let's not let's not talk about that.

We're not here to talk about the Floss Weekly Scheduling. Talk about the software and the people and the projects. So welcome both to Christy, Krista and Gregory. Thank you both for being here. Thanks. It's great to have you. It's great to be here.

Gregory: Yeah. And Chris actually. I have a quick question for you.

Okay. The way you framed it in the intro is going to mislead people right off the bat. I'm happy to go with it and then correct it, but I wanted to give you the opportunity to not have it misleading right from the beginning that we have to go back and correct, but it's up to you, however you want to do it.

And sorry. Go ahead. Immediately as you go to us, we're immediately already in edit mode. I'm sorry,

Jonathan: edit mode. Whatcha talking about? This is content, so go . So go ahead and correct it. So, and I, I say things like that and I, I thought of this when I was saying it, you know, I don't know that that's exactly how their structure is set up, but I know that that's the way people are gonna think about it.

And so this is a question, this is an instance of me saying something dumb on purpose so that you guys get the chance to set the record straight.

Gregory: I love that. And and, and, and so I'm going to jump in on this and Krista, I'm sure you have thoughts on this as well. There is a. There is a lot of confusion about what C.

I. Q. Has done and what Rocky has done. And what is the relation between the two? And I'm actually, I love the fact that we're starting this literally talking about this because there's been so much confusion on this. So when, when, when Rocky first came to be, and this may go a little out of order because maybe we'll come back and talk about this.

But when Rocky first came to be I was, you know, I had a startup called C. I. Q. And we needed to We needed CentOS, like what we were doing, we required an operating system to be part of the stack that we're creating. And so when CentOS went away, it kind of left us in, in kind of this, this, this weird spot where we needed a, we needed a solution.

I raised my hand to the world and said, you know, I'm going to, I'm one of the co founders, original co founders of CentOS, and I'd love to recreate the same, but he wants to join me. I'm hanging out over in the Slack over here and, you know, come up, come and hang out. And immediately, like so many people joined.

I think I even put in that original posts that or the comment to the, to the post where I first announced this, that I'm even willing to hire some people to go, to go help me and do this. So again, if anybody wants to come and join. You know, coming to this community now, it's, it's really important, I think, to mention that one of the first decisions that C.

I. Q. made here was that we don't want to own this. We don't want to control this. This is something that needs to be a community project. It belongs in the community. And so C. I. Q. did not try it. To like own it or, or, or control it. As a matter of fact, right from the beginning, this was a hundred percent community and CIQ did something that I'm actually unaware of any, but any other companies doing most companies that, that operate in open source have a commercial open source or core open source business model where they have a an open source project.

Which they control, like it's in their GitHub, they control it. Decisions in many cases happen behind closed doors. I don't like running open source projects like that. This is not my first company and none of my companies have ever done that. So I always try to keep everything as a community project as much as absolutely possible.

So, Rocky always stayed outside of CIQ. From where it was hosted, to how it was hosted, to the leadership. And the people who were in charge. It was never, never CIQ and, and directly controlled or anything like that by CIQ. Now, CIQ was a huge supporter of Rocky Linux. And so there's a lot of confusion there because CIQ has done so much to facilitate Rocky Linux.

But I really want to stress this point. CIQ does not own or control Rocky Linux in the slightest. As a matter of fact, there's a huge community behind Rocky Linux that is so much bigger. Then, then what? Then, then any single company that is bigger than C. I. Q. It's bigger than any other company in my opinion, because I love this community and I love what the community is doing.

And again, C. I. Q. Is here to help to foster and to provide assistance as necessary. But we even have like charter bylaws in the charter that says, That no single company can ever control the board and that includes CIQ. So I just wanted to jump in, Jonathan. I apologize for totally like side channeling the, the, the

Jonathan: podcast.

We had a plan, man. We talked about this.

All right. So let's jump back. I appreciate that because if, if people only hear one thing, I suppose that's sort of the most important but let's jump back in history and talk about Gregg's. where he comes from and why you had a passion for community enterprise Linux and why you were in this place to even be able to respond at all.

So what's, what's the backstory? Where, where do you come from? And why, why is why is CentOS something that you, that you have cared about for so many years?

Gregory: So I've been, I've been part of the open source community. Really since I got super interested in, in, in computers I graduated with a degree in biochemistry.

And from there went to work at a bioinformatics company and this bioinformatics company was using big Unix systems to do a lot of our genomic searches and whatnot. And then, and then the person I was working for said. You know, I think we can do this cheaper and more effectively with this thing called Linux.

And this was in the mid 90s, so this was still fairly early. I didn't hear of Linux before that. I thought he was just mispronouncing Unix or was being funny. And but it turns out, like, there's this really cool thing, Linux. And we actually went to Fry's Electronics. If you remember Fry's way back in the day.

And we got a bunch of motherboards, a bunch of processors, memory. built up a few kind of, you know, well equipped computers and we downloaded. I don't even remember what the first version of Linux it was that we downloaded, but we ended up settling going between Debian and early, early, early Red Hat.

And and, and we installed it, you know, downloaded it off of, I think it was a 28. 8 K modem and installed it via floppy disks onto this, this hardware. And we created a scientific tool. We created something that I was able to go and do research on, and my colleagues were able to do research on. And. It completely boggled my mind because I've never heard of anything that anything like this was possible, that you can just go and download an operating system or or or software that is truly valuable without having some sort of tie in back for, you know, making money and having it held over people's head.

And you know, you get like, you know, a freemium model, like you get certain amount of things like, you know, I remember like McAfee, you know, antivirus at this point was just starting to get big and but they had, then it was free, but there was a commercial model behind it. There was no commercial model behind this and it totally intrigued me.

So I got super enthralled and interested in Linux and open source and it became a very, very strong passion of mine to the point where I gave away, I kind of gave up my. Where I focused on my degree and I kind of transformed it into, you know, I want to do, I want to do Linux, I want to do open source and, you know, I ended up doing a few, a few cool jobs.

My first job in Linux was working at a a company called Linux care. And LinuxScare was one of the three main kind of initial companies in the late nineties doing Linux. There was Red Hat, there was VA Linux, and there was LinuxScare. Red Hat still around, obviously. VA made a huge amount of of you know, noise out there because they blew away all the records for an IPO.

And LinuxScare, well, we were on a great trajectory, but we didn't quite make it past that, that hump. And as a result, a lot of people have kind of forgotten about LinuxScare. There's a lot of really cool stuff. Jonathan, at some point, it'd be a lot of fun to talk about kind of the early days back then and what things were like, but I got even more enthralled with the culture of open source through Linux care.

I worked with a ton of amazing people in the open source community. You know, everybody from kernel developers to user space developers to device driver writers. We were part of early LSB and I worked with some of the people doing LSB. I worked with Angel, Andrew Trigil Rusty. A whole bunch of like amazing people, Chris Yeo.

And what I learned was that there was this so much passion behind open source and it just completely transformed how I saw the world. I actually, at some point, maybe I'm somewhat embarrassed to say, but I actually thought open source was the way to solve All world problems because with I worked with developers and other people that were in other countries.

And in some cases, our government called the hostile nations. But those were people just like everybody else. And we all got to work together and it was super cool. And, so I actually saw it is this is like this is even above politics. This is this is like we're all just people. We're all working together for the same goals.

So I got super interested in it. When you have a background of open source and Linux and air and interest in that in a background in science it's pretty easy to kind of land squarely and high performance computing. And I spent a long time working in high performance computing and in the In 2000 I ended up leaving Linux care and going to Lawrence Berkeley National Laboratory.

And in that role, I was immediately kind of put into high performance computing areas. And I created my first open source projects there. My third or fourth open source project was a little operating system called chaos Linux. And the idea of chaos Linux was it's Debian, but kind of for RPM based.

And the idea was I loved RPM. I liked RPM better than Deb, although Apt was super cool. And I wish. That, that RPM had a version, something like apt, which did come later. And which was Seth Vidal's work initially with yum. And but, but I felt like there was this whole part of the ecosystem that was missing, which is this community around the Linux distribution.

And so chaos Linux was, was designed to, to, to be that. And we were, we were. We were, we had, we started growing in this community, kept growing. It started off meeting and if anybody knows Berkeley started off meeting in Berkeley at a cool little pizza beer joint called Jupiter. And we always used to hang out at Jupiter and that's where, that's where all this started.

And from there again, community kept growing, moved to IRC for most things, and it kept going from there. And over time you know, the community was, was quite substantial. We had a number of people that were contributing in and being part of this. And then, then Red Hat changed their business model and they got rid of something called Red Hat Linux.

Now Red Hat Linux up until this point was the free version. It was, it was their we'll call it community version of, of Red Hat. And and Red Hat Enterprise Linux was just kind of getting spun up. So Red Hat Enterprise Linux was, Oh, here's what you go pay for. If you want commercial, you know, enterprise grade solution, support, et cetera, but the free solution Red Hat Linux is just got end of life and one of the things about chaos Linux that we were trying to do was trying to maintain core operating system compatibility.

And so we're like, well, we want the core APIs to be compatible with Red Hat. So a lot, there was a, it was a shared. Common base of, of, of, with, with Red Hat. So we kind of needed that, that Red Hat base. And there were a few people on the team that were like, you know, this is, this is not good, but we can still get access to those source RPMs.

So we can still get our, our, our base and then extend, extend the operating system from there. At about this point, white box enterprise Linux came, came about. And the, the maintainer of it, I believe his name is John, was John Morris. Don't quote me on that if I got that wrong, John, sorry. , but he put it out there and he, he told me in an email thread direct to him that, you know, he didn't really want to maintain, you know, a whole operating system himself.

So I asked if we should work together and he said, no, he says, if this is something you're interested in, go, go ahead and do it. And CentOS was kind of born from that. The first developer on it was a guy named Rocky McGaugh and Rocky was super passionate about open source. He loved open source and he also came from an HBC background.

So him and I, you know, we got along really, really well, really well. And it was a fantastic, you know, kind of initial drive. He worked for a company, an HPC company that had their own rebuild of Red Hat already. And so he was, he already knew how to do it. We had a couple other people on the team that were just tremendous, fantastic engineers.

Michael Jennings is another one of these these engineers. And he also, he ran VA's Linux distribution, which again was a rebuild of, of, of Red Hat. So we had a number of people on the team who knew exactly how to do this. We had build resources already. So. When, you know, white box kind of, you know, hit the market and then john mentioned, you didn't want to do this.

We're like, well. Let's just throw all the source RPMs through our build system and we'll just, you know, we'll, we'll end up with something that's more community base than white box, but, you know, and maybe something can go bigger and CentOS was born Lance Davis came up with the name. And brought it to me and I thought the name was great.

A little bit of humor and irony. I said, I actually, I don't like the OS being separate than sent. I said, cause sent is cheap, but it's infinite increase from zero. So I said, is great sent to OS and I don't like that, but sent us is great. And of course, you know, we put it towards the community and the community ended up liking the logos and liking it as CentOS.

So whatever.

Jonathan: Oh, that's, that's fun. That's a, that's a, that's a lot of neat history about how we got to CentOS and You know, I, I was around for a little bit of that. My first Linux install was like one of the real early Fedoras, Fedora 3 or 4. So I came onto the scene a little bit after all of this happened but I, I found CentOS and, you know, started installing it on some of my machines and on some customer machines too, and I, so I was, I was around for that blog post we talked about before the show that was in 2020. So it's not been that terribly long ago, but you know, Red Hat had put out, CentOS 7 was out, CentOS 8 was out, and I thought CentOS 8 was just the coolest thing. So I started installing it on machines, on servers.

I had it, I had it deployed. And then Red Hat puts this blog post out and they say, we're not going to continue. Well, okay, so there's, there's a, there's a missing step here, right? How did we go from So Red Hat put out this statement that they're not going to continue doing CentOS. How did we get from the point where you and some other open source developers were doing rebuilds of Red Hat's Enterprise Linux completely unofficially to Red Hat now has the ability to put the brakes on the project.

I think that's, that's an important step that we almost missed there. What happened?

Gregory: There was, there was, there was ChaosLinux which, which had a similar base as, as CentOS and that was the overlap. And then there was CentOS, which kind of had a. Slightly different trajectory. To be clear, my passion was always on, you know, making something new initially.

Like I wanted to do chaos, like chaos Linux to me was the coolest thing. And while even though people out there wanted CentOS more, I thought. You know, I thought it was more fun. I thought it was more interesting to be developing something new from that perspective. So I was really more focused on the chaos Linux project.

I did lead both projects as part of the chaos foundation for a few years. And there was a, there was some history that I'm not going to go into. But it was decided at some point, you know, we came to a mutual agreement that the project should. Should go into separate directions. And so Sentos ended up going in one direction.

Chaos Linux kept going in, in another direction. Chaos Linux stayed alive for a number of years after that. But a lot of the interest kind of died out because when the, the split occurred excuse me, Sentos kind of took all of the fame and glory, even for good reason, like that was what everybody wanted.

And so Chaos Linux kind of slowly started dwindling out. And the way that, and, and this. This is going to come across as a criticism and I don't mean it to just in terms of matter of fact, the way that CentOS was initially kind of instantiated by no fault of anybody's just because the technology wasn't there we had individuals who, who had the signing keys.

We had build hosts, which in some cases were individual people's homes in their garages, in, in hosting facilities, and we did anything we could. We built the packages anywhere we could, but as a result of that, it's really hard to bring in external people. And, and make them part of this community because there's, there's, you can't just like hand over the signing keys, which are the keys to the kingdom for the operating system to somebody who hasn't really been there for a while.

And, and it's hard to kind of open up that, that envelope of trust. And so. We, we, we, early on, it was a kind of small team and that team persisted. Now there was a little bit of growth in the team and a little, you know, some people, some people joined but for the most part, it was always a very small team that maintained it.

As you can probably imagine with, especially with a small team, after 10 years of doing this, a number of those engineers, well. You know, I have, I have my day jobs I've been spending a lot of time on this and weekends and evenings and off hours and. This is kind of hard to, you know, have a life and whatnot.

So things started to slow down and a lot of people probably do remember that pre 2014, CentOS was actually starting to, to, to slow down. Like it took in some cases, three, four months. For new major releases to come out. And then sometimes even a couple months for point releases to

Jonathan: come out.

Gregory: And as a result of that, like things were just kind of slowing down and slowing down.

So at some point Red Hat made offers, job offers. To a lot of the Santos maintainers that we're still doing this. I was not privy to these discussions. I don't know exactly what happened, but what you can see from the outside is, you know, number of these engineers and whatnot got employment from red hat and the project moved from the community to red hat.

I I was very optimistic, honestly, when this first happened, I was very hopeful because I actually felt as though Red Hat is the right home for, for this initially, for a long time, actually, I thought that I even proposed to the Red Hat people that I knew, and I think it went up the executive chain at least a little bit.

At the time that I actually wanted to hand over CentOS to Red Hat back when I was still running the project and I wanted to offer it to them. They said no, they didn't want it. Now granted that was 12 years, 10 or 12 years prior to them kind of acquiring the project. But they ended up acquiring it.

And that was in 2014, if I remember correctly. And in 2019. IBM acquired Red Hat, and then in 2020, CentOS, we got this announcement. And, and that's where we merged back up to what you were mentioning just a moment ago.

Jonathan: Yes. Yes. Okay. So, the, we, we get the announcement. Let's, let's continue on from there.

We get the announcement, and those of us that had trusted in CentOS, we, we sort of got burned. Now, if we were on CentOS 7, everything was fine. Because that was going to end of life when the end of life was supposed to be. But for CentOS 8, we got basically a year, not even a full year. I think it was several months, like nine months.

It's like, Oh, we're going to end, we're going to end of life this in nine months. And it was. In fact, my wife watches the show and she made a comment. She says, I remember the reaction in the house to all of that. It was a significant event for a lot of us. And on that blog post from Red Hat where they made this announcement, you came along and you made, you made a comment.

And I don't remember where it got picked up. It may have, it may have gotten picked up on Slashdot, or maybe it didn't. Maybe we were just all going to the blog post, and you, it may be that we were all going to the blog post and going down to the comments and going, Surely somebody down here has an idea for what we can do.

And there was. There was somebody down in the comments that had an idea for what we could do. What, what Let's see. What possessed you to make the an offer like that? And what was the offer? How did, how did that turn out?

Gregory: Well so the company that I have is named CIQ. And the goal of CIQ is to build a computing platform.

And we were heads down building this computing platform. And we, we recognize that the computing platform is super complicated. It's actually, especially at the beginning. There's a lot of moving pieces to, to, to this computing platform. It's built off of Kubernetes. It's a whole HPC type and AI and ML computing platform that is different than anything else that has existed.

So it's just super complicated. That's the point. I'm not trying to plug what it is that we were doing, but just super complicated. So when we thought about going to market. With this, we recognize that we're going to have to create an entire turnkey stack and support that entire terms, key stack. And obviously we're like, Centos is the right solution.

And we showed it to some vendors and some partners. And we actually even had a few vendors and partners ask us if we would do this on Ubuntu. And our response was, well. You know, the, we felt as though that the best and safest solution was, was CentOS. And so we were moving forward with basing this whole computing stack on CentOS.

Well, when we got this announcement, it kind of made us put our foot in our mouth a little bit, as you can probably imagine. So we, we, we, we had to figure out. In kind of short order, what are we going to do? Cause this affects us. It affects our go to market. It affects our customers and partners that we've already been talking to.

And to be honest, it affects the whole community, like everybody. And so I had some quick discussions with, with some of my stakeholders at the company. And I said, look, I may need to be part of a project or help to, to, to bring this back because this is a really big piece of the ecosystem and.

Everybody needs this. And so I'm like, I'm going to go raise my hand to the community and say, if anybody wants to do this, you know, I'm going to be hanging over in the Slack. And this Slack just blew up. Like, I don't even know who ended up reposting this. I couldn't keep up with anything at the time.

As a matter of fact, I got some people that, that even got upset at me because I didn't get back to them because in this Slack, the free tier of Slack. Supports 10, 000 messages and it's rolling buffer. If you've got 10, 000 plus people in a slack that all just joined, everybody's talking. I had pages of DMS.

I would just scroll and scroll and scroll. When I wake up in the morning, go to the bottom one and just see, Oh, you can't see this message because it's past 10, 000. Go to the next one. You can't see this message past 10, 000. And it's not until I get halfway up to the list where I'm basically just kind of catching up to what I can even see.

And then I start responding to people now. This was a really interesting situation because going from zero to 10, 000 people, and I'm not joking, it really was over 10, 000 people all talking at the same time, all excited about going and recreating and rebuilding Santos and managing anything with 10, 000 people who just all showed up like without any management hierarchy, no like structure, Oh, my gosh, I, that was, that was crazy.

So first thing that I did was I created a bunch of different channels and I said, everybody go to, go to the channels in which interest you the most. And so we had people kind of break themselves out into different channels and kind of trickle trickle into different channels. And then each one of these channels kind of had different groups or different initiatives, excuse me.

And these initiatives ranged from everything from. Okay, let's start mapping out what packages we need. Where are we going to get these packages? How are we going to build these packages? What are the build orders for these packages? Are all of the dependencies present? If not, what dependencies do we have to go?

So it was that from what infrastructure do we need? To build this. How are we going to manage that infrastructure? How are we going to make sure we don't succumb to the same issues that sent early sent us did where there just wasn't the capability to even bring in new people from the community. So how are we going to manage that?

How are we going to manage the project as a whole? Like, is there a, you know, like, like a leadership umbrella of this? And then we had people that came along that says we know nothing about how to build a Linux operating system, but We want to be part. And so we had documentation teams, all of a sudden joining up.

We had a design team who was like, you know, we're artists. Like we want to make stuff. We want to make stuff pretty web team. That's like, yeah, we want to make a website. It's actually funny, we had swag and merchandise before having a product before having an out, like an output, because the design team is like, we can do that.

Like, we're not going to build an operating system, but we can go build, we can go build shirts, right? We can go make this, we can go do something over here. And and they did that. And so. Next thing you know is we have these, these cool rocky swags and shirts and stickers. And the first ones actually said Rocky Lennox on the shirt and underneath and bracketed text, early supporters.

And they're there. I don't even know if they're possible to get anymore, but I always have people asking me like, can we get an early supporter shirt? It's like, it's not really an early supporter anymore, but I'll see if I can pick some

Jonathan: up.

Gregory: It's been, it's been remarkable and, and the team just continued to grow.

Now I want to stress something really important here, which is my knowledge in terms of how to build an open source community is, is, is relevant, my knowledge of how to build an operating system is 20 years dated. Like I haven't built an operating system for 20 years at this point, when these teams started coming together.

They immediately outstrip my knowledge of how to actually do this using today's technology, today's capabilities, today's security practices, et cetera. So I'm watching these teams grow and I'm like, Hey, how do I help be? How do I not get in the way? And how do I just keep everybody in online? And, and that was really my first kind of goal was just to help build this, this community and, and help build partners and, and, and other people who want to be part of this.

And, and. C. I. Q. Again, kind of, as I mentioned at the beginning was here to help. C. I. Q. Obviously funded me. We ended up just like I said in the comments of the blog post that we will hire some people to go do this and to go focus on this. And we did. And we do have people that are hired by C. I. Q. In order to kind of help this community grow and thrive.

And C. I. Q. Does not own or control or, or hold hostage this community. Like this community makes its own decisions of what's best for the community. And as a result, this community kept growing and thriving and others started wanting, wanting to join and be part of this all of the hyperscalers jumped on various vendors, ISVs, IHVs joined up, joined on and, and all these people wanted to be part of this and it was just beautiful how the whole thing kind of, kind of came up to be and I take credit.

Really for just bringing people together and, and really just kind of having division of here's how we need to do this at a high level. But I want to stress like the people that came together, the teams that came together are some of the best people in the industry. And Rocky Linux has been amazing.

We built everything from scratch. Literally we wouldn't release versions of the operating system unless we knew for a fact that others could also build

Jonathan: it.

Gregory: So we've always adhered 100 percent to open source principles. And, and everything we've done has been by the community. And I really want to stress that.

And that's why I wanted to go back when you first mentioned the cross pollination between CIQ and Rocky. Because I don't want Like, I want to make like this, this, this confusion actually happens all the time. And we are trying our best both on the CIQ side as well as on the Rocky side to clarify this.

But and, and, and I would love advice if anybody has, like, we're trying to do this now. CIQ definitely is very proud of, of what CIQ has helped to create. And so sometimes we message, you know, and say things like, you know, the Rocky community is doing fantastic and whatnot. And so people may assume that there's a, there's a more of a direct correlation than, than exists.

But they are two separate entities and CIQ is just here to make sure that that open source community continues to grow, thrive and has whatever it needs to be successful. And, and there are. Commercial needs that that people have. So a lot of people say we need support. We need optimizations.

We need modifications. We need X, Y, or Z and C. I. Q. does help in a lot of those areas. So when a corporate entity is needed, C. I. Q. raises our hand. We say we're here. We're here to help in any way we can. But you don't need us like go use the operating system without us. Like we can never, we're never want to hold it back.

So, but if you do need us, we want to win business based on us just trying to be good, good, good contributors and good partners with the open source community. And so we, we try to do very good ethical business on, on that front. But again, the community has grown and maybe I've talked enough. I've talked plenty.

So probably where Krista starts to come into the,

Jonathan: into the. I was thinking about that. And so the organization that is not CIQ, that sort of is the host for Rocky Linux then, is RESF. And I think this is a perfect time to kick it over to Krista and ask what, what is RESF and what is it all about?

Krista: Thank you. Hi. The RESF is the Rocky Enterprise Software Foundation. And the whole point of its existence is to probably to inspire both individuals and corporations to be able to come together in support of a common goal, which is that of preserving the the longevity and the stewardship and also of enterprise type software and also of helping to.

Drive innovation in that, in that piece of open source.

Jonathan: And so what, what's your, what's your backstory, Krista? How did you get involved in all of this? Were, were you one of the early adopters that saw the blog posts and just couldn't, couldn't get away from it? Or, you know, where, where does, where do those threads tie in?

Krista: You know, I, I traveled my own path. I, I kind of stumbled into the Rocky Linux project. In retrospect, I feel like it was the beginning of chapter two of, of Rocky's birth and getting going because they had worked so hard to just get the product up and get stable with that. And I think they were starting to realize, Oh, we need to pay attention to the community part of this open source community, technical community.

And in doing that, I. I didn't even know what was coming. I had just in my own world had been trying to develop my skills in technical writing, and I knew that open source documentation was a way to do that. So I joined this new project. I knew somebody who was involved with it and they pointed out that with open source documentation, anybody can be contributing to it, but also.

So it's a new project. So this is a great time to be involved because we're in the process of creating a whole lot of documentation. Unlike some projects that may be 20 years old by now are a little more less, I don't know, volatile. That's the wrong word. But they're not changing as much.

Jonathan: Sure.

Krista: So I just came in to.

Develop my technical writing chops. And after a few months of that, somebody reached out and said, Hey, did you have any interest, would you have any interest in participating in the community? And I said, well, what's involved? And they said, well, we have a couple of social media accounts and sometimes we go to events.

So I said, okay, yeah, I could do that. I don't have a job right now.

Jonathan: It works out nicely.

Krista: Yeah, time to do that. And it very quickly turned into something much bigger than that. Because I, it was right as the original board was forming, as far as the oversight committees, they had just really gotten the RESF formalized and ready to be inhabited.

So I got to be a part of voting in the first board, which was the very first thing I did as an official Rocky Enterprise Software Foundation member. So, I, I was, I, Think of myself as the first one who joined after the beginning, which is what makes me part of the beginning of chapter two.

Jonathan: Yeah, that makes sense.

And so what what does the community itself look like these days? If you were to if you were to give someone a quick primer on tell me about the community involved with rocky linux What does what does it look like?

Krista: Well I have attended a lot of events, and in so doing, I've had to learn how to articulate the project, and it helped us all to acknowledge and realize we have really two separate sides of the project that we like to think everybody all works together as a whole, and we're all pretty unique.

I feel like we regard ourselves pretty laterally. It's not a lot of hierarchy, but it's a team of leaders that each has their eyes on a different part of the project. And we realized that there's a side that's more technical, such as the release engineering and the testing team and yeah, the infrastructure side.

And then there's the side that is more non code based, but also important and valuable, which is the design team, the web team, the documentation in the community. Itself. And so I've, my eyes are pretty much on the community side and on those non technical or it's not non technical, but it's non code on that non code side that.

I try to include everybody to communicate with everybody. It just, I came in and started talking to people and tried to create points of connection because I knew I may not know open source that well, and I may not know all of this technical code stuff, but I know people and I know that when people feel connected to a thing, they come back, they hang around.

And they, and when you give them ownership of a thing, they get involved. And then they also, when they have ownership, then they enter and pass on that enthusiasm. So I, I would say that our community now is something that's more engaged and is also there and feeling like this is a thing that's fun.

And this is our, our social. Our social group.

Jonathan: So one of the things that we often ask about it kind of at this point in the show is like, who sets the roadmap? And the idea there is we kind of, we're kind of asking like, is the project community driven versus top down driven? And that's sort of an, maybe a category error question here.

Because in some ways, Red Hat sets the roadmap, right? Because since the Rocky Linux is a repackaging of Red Hat Enterprise Linux. And so I'm just curious, like how much roadmap is there? How much wiggle room do you guys take for yourselves?

Krista: Well, I think I'll answer from my perspective. And then maybe Greg has more to say, but I know that our group looks ahead.

As much as they can as far as they can and tries to be prepared for the changes that come the new releases of red hat of rel Don't come in a vacuum. They come out of incubation and out of testing from other projects the fedora and what's now centos stream Which is what the centos project became which we haven't really addressed yet It didn't just go away.

It became a new thing and it is now upstream And It is still generating what is today that we know as Enterprise Linux, so got lost on the question, but they're looking at the people in our group in Rocky Linux are looking upstream to see what changes are coming and already putting together test versions of future releases just to make sure that they're prepared.

I know that the primary function and the primary purpose of Rocky Linux is to exist in the space that it exists in, and to meet the needs of the demographic that has flocked to it for those reasons.

Jonathan: Greg, some questions?

Krista: Oh, sorry. Go ahead. Well, as far as the community goes our Purpose of existing now is to just continue as far as our roadmap is to look down the road at what can help us be successful long term at supporting the community.

So, you know, we try to make sure that we have support from a diversity of sources. We try to make sure that we have everything documented so that to insulate ourselves in case somebody has to go away, then there's somebody else can come in and take over their role or. Solve the problems that have been solved.

And so we set a roadmap in that way of trying to just articulate what we are and where we want to go.

Jonathan: Very good. Okay. So same question, but this is good to go. I'm sure a very different direction because we're going to go to Greg and talk about roadmap. And I'm again, it seems that. There, there's maybe not a lot of wiggle room.

And, and at some point we're gonna have to make comparisons between Rocky Linux and Alma Linux. Right? They're, they're the, they're the other big player sort of in this space outside of, of red Hat. And so, you know what, we'll, we'll dive more into this, and I know, again, I know part of the answer here, but like, what, what does your roadmap look like and how much is that set by upstream?

Red Hat?

Gregory: A lot of the ups of the roadmap, of course, is set by upstream Red Hat.

Jonathan: Yeah.

Gregory: When, when we started this, the whole point was to continue where CentOS has left off and and, and I want to actually go back to Chris's point CentOS Linux has left off because CentOS stream does exist and that's what CentOS has become, which is now in front of Red Hat.

So I'll differentiate that more pedantically and usually I'm pretty lax about that, but I will be more pedantic. So CentOS Linux left off. And Rocky was, was there to take its place in a matter of speaking and to become the successor of CentOS Linux. Now, to do that means that we have a a set of goals that we need to adhere to, which originally CentOS adhered, CentOS Linux adhered to.

And, and we are, our mission, our vision for the product is how do we Continue with with exactly where that left off. So when we think of road map, a lot of it is how do we ensure that we can always do this? And how do we ensure that we can always, you know, fulfill the promise that we made to the community that we set out to do at least to the best of our.

And we've seen some some difficulties. There's been some, some changes with regards to access to source code just as an example. And, uh, Rocky and R. E. S. F. said, you know, we're going to stick to our guns. We're going to stick to the promise that we made because we don't want to change, make any changes to all of the people that have, that are, that are running Rocky Linux.

They entrusted us with their infrastructure. So we're going to continue with exactly with what we said we're going to do and what we set out to do until the point where if it ever comes that we cannot. And so that's our goal. That's our mission. That's our strategy. And to some extent, that's our road map.

But then the road map becomes A little bit more strategic. How do we ensure that we can continue to do this? So when source code kind of dried up a little bit and it became more difficult to get access to the source code Rocky came up with a number of different ways to do this and to continue building the operating system.

But at the same token. You know, I'm I'm all I've been burnt with with failed missions many times in the past. So I'm always thinking about contingencies. And what do we what do we do for contingency planning? And how do we ensure that everything that we're doing is going to be able to continue moving forward?

So I reached out to Oracle and I reached out to Sousa and I said, look, we're kind of all in the same sort of, sort of boat. Like, how do we all ensure that the source code for Enterprise Linux always remains open and, and we all can get it, we all can leverage it. And Oracle said, yeah, this is something we would love to be part of.

And Sousa said, we'd love to be part of this. And an open ELA was created. Now, to be clear. Rocky already had a solution for moving forward. So Rocky is continuing to use that solution. So nothing has changed on the Rocky side, but the fact that now open ELA exists provides a contingency plan for Rocky and, and ensures that Rocky will always exist and we'll always be able to continue moving forward.

And, and that is backed by not only CIQ, but Oracle and SUSE. We actually. Are opening up membership that's actually becoming a 5 0 1 C six nonprofit. And we're opening up membership right now and we actually have I believe it's two other organizations who are proposing membership right now and, and looking to do that.

So this, the open ELA is, is also growing and that's going to provide greater stability. And when I'm jokingly jokingly going to call the land of enterprise Linux, which is this pseudo standard of what does it mean to create an operating system that is compatible with other operating systems, which makes life better for system administrators for end users for IHVs, ISVs, other partners as well.

Everybody really wants compatibility across the spectrum. So we have this pseudo standard. Which is today to follow red hats lead. But again, this whole thing is about making sure we have contingencies and making sure that we are properly set up that no matter what fluctuates within corporate agendas, that Rocky Linux can continue.

And that we can continue doing exactly what we set out to do. Now you brought up Alma. I'll address this initially. Krista may have some thoughts on this as well. When, when the source code kind of became more difficult to, to obtain Alma Linux took a different, a different trajectory, slightly different trajectory.

Their trajectory was, well, we don't have to be exactly one to one compatible. We can be close, but not exact. And they de risked Alma Linux. In the sense that doesn't have to be exact. It just needs to be close. And they've done a very good job at keeping it close. And they've done a really good job also on adding additional features and capabilities that, that people want that red hat doesn't do

Jonathan: backporting, backporting some pretty important security fixes earlier than red hat does.

That's one of them. Yeah, yes, exactly.

Gregory: So. There is some, some, there's a great opportunity now for Alma. There's a great opportunity for Rocky Linux. And the most important thing that now we have in this ecosystem of Enterprise Linux is choice. And now we have, so Oracle Linux also is not identical. It's really dang close, but it's not identical.

And so, but we now have choice and people can choose what operating system fits their needs and requirements best. And this is, but, but we still have compatibility, which is exactly kind of where we need to be again to solve problems for users to solve problems for our partners and, and vendors like.

We need that compatibility. Otherwise, we end up with Linux distributions that aren't compatible with each other. And while there's there could be some benefit to that in the sense that, you know, people could do different things with their Linux distributions when you're talking about professional I. T.

Staff, whether it's their jobs to maintain. A fairly significantly, you know, large infrastructure and they don't want surprises. They want to sleep at night. They want to know everything's exactly the same, right? And so they want that like boringness, super boringness. And some Linux distributions are going to be bleeding edge.

They're going to be moving faster. They're going to be focused on different areas. In enterprise Linux, it's really, it's hate to put it this way, but it's kind of about the boring and, and how do we. Help system administrators and I. T. Professionals and partners and whatnot bring value and maintain their infrastructure in a way that, you know, helps them sleep at night.

Jonathan: So I, I commented at the time when both Rocky and all the Linux both were announced to became things. There were some people that went, well, why do we need two of these? And I, at the time said, well, this is, these are going to be run by different people. At some point, there's going to be different decisions made.

And I thought from the very beginning that it was a great thing to have both projects in the space. It, from one perspective, sort of as an insurance policy, but from the other perspective also just because they are eventually going to go in slightly different directions and they're going to meet different needs.

So I, I am, and I've said this publicly, I am thrilled that both Rocky Linux and Alma Linux exist. And I think they're both super important. I was

Gregory: on a, I was on a podcast with, with Igor who was the original founder of, of Alma Linux with, you know, with cloud Linux. And I was on a podcast with him and somebody asked us at some point you know, should we, shouldn't we just join forces?

And there's definitely, there's some there's some good reasons to join forces, but both Igor and I agreed at the time that, you know, really what the community needs, especially after one whole operating system was just end of life. Unilaterally for the most part, right? Really what everybody needs is they, they, they need to know that there's multiple solutions out there.

They need to know that they have safety in numbers speaking. And it was, it's very good that there are multiple solutions out there. And it's even better that we're all compatible. Yeah. That's the key in my mind. The moment that somebody. has to deal with an incompatibility between operating systems, between infrastructure tooling or whatever that actually makes life harder on people.

Having standards actually makes life better, makes life easier. So even though right now it's a pseudo standard this is something that we, we do have kind of like a goal. We, we have this, this golden image of what we're trying to, to emulate. And I think I think Rocky, I think Oracle, I think Sousa Liberty and, and others out there are doing a fantastic job maintaining that compatibility.

And this is this is, by the way, Open ELA does more than just distribute the source code and ensure that it's always available. Open ELA is also now talking about, well, how do we help guarantee compatibility across all of these different variants? And how do we demonstrate compatibility across all these different variants, such that we encourage variants to some extent.

And at the same token, we're also encouraging stability and compatibility between those variants. Because again, at the, for the end of the, at the end of the day. For for end users, for the people that are using this, the people that are relying on this, their safety in numbers. So let's make sure that we've got backup plans, we've got contingencies and and everybody knows that the enterprise Linux ecosystem is, as a result, the most stable ecosystem in Linux, that you can go and run any of these operating systems and end up in a great

Jonathan: place.

Have there been conversations with AlmaLinux about joining OpenELA?

Gregory: I have not I can't say if there have been other conversations about it. I know that there have been questions that just, again, not with the open, it was not, excuse me, not with the Alma people and team, but just questions on, you know, where are they going to end up getting source code once, once CentOS stream end of life, because they're only, you know, CentOS stream is only alive for five years.

So, where are they going to end up getting updates from and whatnot? And I, I make the the, the, the, the invitation that OpenELA, OpenELA is there. Like, if they want to use OpenELA, absolutely. We would love it. And we, we think that that would be terrific.

Jonathan: Yeah. It, it, it, it, it occurs to people, it occurs to me, like you guys are doing similar things.

It just, there's an obvious, there's obvious room for collaboration, even if, you know, as we, as we said, it, it doesn't make sense to try to merge the projects, right? We don't want that, but there's all kinds of room for collaboration between the two and with, and with the others in the space. So that's that's a question you'll probably get as long as there are both projects out there.

Alright, so let's, one other thing I wanted to touch on, and maybe this is just going to be commentary for me, although I'm sure you'll have something, you'll have something to add to it Red Hat, Red Hat, do not, for those of you listening do not think that we are bashing on Red Hat do not think that Red Hat are the bad guys, Particularly the Red Hat engineers that are actually doing the work.

There are some excellent engineers. There are some really good people. We, well, I, I guess I shouldn't, well, no, I'll say, I'll say we, because I do as well. We have a difference of opinion on what we think of their business model and particularly the fact that they ended of life. The CentOS, when there were some commitments already out in the wild to not do so.

But, don't, don't don't go harass your local Red Hat employee. Have you seen, have you seen some of that? I've seen, I've seen a few threads. From people at Red Hat like please don't it's it's not our fault. There's more to the story. You know, what have you there were different threads about it I'm sure that's been on your radar, too

Gregory: so You bring up something that actually I think is super important for us to address and and again I think Krista may have some thoughts on this but You know, a lot of people have their sports teams and they're super excited about their sports teams.

Like I've seen fights emerge. I used to love to go to hockey games and Oh my goodness. The fights in the stands and whatnot. Like, yes, like people have their religions associated with things that they love and the tech industry especially the Linux industry and, and, and community. We've got a lot of our own religions, no matter of speaking.

Sure. And. Whether it's what editor you're using, what programming language you're using, what IDE or or Linux distribution or window manager. Like we all have like, like passion on what it is that we, we are excited about. And, and we kind of defend those like, like other people would defend their sports teams in many cases.

Yes. My point is. There's a lot of passion. There's a lot of excitement, and there's a lot of emotion that has occurred over the last few years. There's been a lot of emotion both you know, around Red Hat, around Rocky, around Alma. Around all of this. A lot of times, you know, this, this, these, this unfolds in different ways.

But I, I'm glad to say that at this point, I think most of that drama and that, that, that. Rockiness, not to use that pun in, in purposely, but has actually kind of gone by at this point and I don't see very much of it anymore, which I'm really thankful for I've made mistakes. Other people, I think I'm not going to call anyone out, but I think there's been a lot of mistakes made.

In terms of how we allowed Our emotion and our personal feelings and whatnot to get involved with this. And the good news is I think we are in a fundamentally better place from an enterprise Linux perspective than we ever have been before, meaning even when CentOS was separate from Red Hat.

It was still, except for the very beginning, we actually had multiple different types of versions of CentOS, there was Tau Linux, and there was Whitebox, and there was others, but it all kind of merged into CentOS, which now meant there was one kind of operating system, and we saw how that kind of ground down, just slowed down for a while and whatnot, um, but there was just, just really one.

I believe we're actually in a fundamentally better place, and I know we had to go through some drama, and we had to go through some emotions to get to this place, but I think we're in a great place right now, and I think it's good for, to be honest, I think it's good for Red Hat I think it's good for the users, I think it's good for vendors.

I think it's good for everybody. So I fundamentally believe we're in a better place. Krista, did you have any thoughts? I wanted to kind of poke and yes, I wanted, I wanted to ask

Jonathan: Krista too, because she, she works as the community manager and a lot of times the community manager is the frontline on dealing with.

Some of those passions in the community, is that something that you see?

Krista: I think it has diminished a lot since I first started working with the project. What we've tried to frame it as, and to ourselves as much as externally sometimes is to remind ourselves and to remind anybody who wants to say, why not, why don't you guys.

Fighting like you're in a hockey match is the enterprise Linux is really a neighborhood. It is its own little branch of Linux of the greater world of Linux, and I think it helps the entire enterprise Linux ecosystem. If we work together and consider ourselves co laborers instead of enemies within, because nobody's going to say, I'm going to go to Rocky versus Alma.

They're going to say, Oh, there's too much drama here. I'm going to Debian, which, you know, Debian is its own thing, but for people for whom Enterprise Linux is a good solution. It's best for us to all work together for the good of Enterprise Linux in general in order to continue to allow that whole platform to move forward to that end.

What we've done is we support the. The Fedora and CentOS stream is we consider that our upstream and so we actually contribute financially to their gathering conferences and we go and attend and participate together and all the Linux people are there and we all sort of just mingle together and we are at least friendly to each other.

Jonathan: Well, that's, that's good. That's better than any of the alternatives,

Krista: right? I mean, you might as well be able to get along with the people that you're at events with.

Jonathan: Yes. Yes. Okay. So what, we talked a little bit about roadmap and sort of a theoretical idea, but is there, is there anything sort of on the horizon that you guys know is coming that you want to plug and any, any any big plans to announce?

Gregory: 10 is coming. Risk five is coming. So those are, those are the big pieces of, of what we're excited about. I can also share that for our users that are leveraging you know, government compliance needs and whatnot. FIPS is finally on its way. Now we've been pushing on FIPS. So CIQ has funded the FIPS development.

For Rocky for quite some time we promised to the user base that we would start with eight version eight. And the problem is, is that, and not to get too far in the compliance world, but FIPS is is really hard. Like there's a lot. Of difficulty with managing and navigating the whole FIPS ecosystem and process and couple that with 140 2 just end of life as, as this whole thing was spinning up and 140 3 is the new standard for FIPS operating system module compliance.

And. Version 8 was never built for 140 3, it was built for 140 2. And so CIQ, it took us a long time and a ton of development to actually get version 8, 140 3 compliant. And as of right now, all of the modules are in MIP status. And they are available and CIQ is making all of that available back to the community as well.

For,

Jonathan: so for someone that heard, for someone that heard FIPS and is thinking of Simon FIPS, what, what is FIPS compliance? So FIPS is

Gregory: a compliance that guarantees. The cryptography is running properly within the operating system and it is meeting the government standards defined by one 40 dash three. So there's a whole bunch of cryptography that occurs within the operating system and occurs within applications.

What this is stating is that there are various components of the operating system that are providing cryptography. And each one of those is, is functioning, functioning and certified. That it is, it is working properly and it is validated approved by the government. So it's taken some time to get to that point.

Anybody who has to deal with federal compliance is, is dealing with FIPS and it is a big deal. And, and this is why you know, it, it has been a. Many multi million dollar investment from C. I. Q. to do this. And so that's gonna be released here shortly. As well. So we're going to see FIPS. And again, just to reiterate, version 10 is coming.

I believe the, the guesses of, of when that's going to come out is in summer. And we are, we have been working on RISC V for quite some time. So RISC V is also on its way.

Jonathan: Yeah, interesting. I am, I am personally, I'm fairly excited about RISC V as a thing as it moves into the Linux ecosystem. I'm, I'm looking forward to some even better hardware.

being available for some less ridiculous prices. There's, there's some good out there, but I feel like we're, we're ready for the next really good one to come out. All right, let's see. So is there, is there any talk within, within either CIQ or Rocky about doing something like a repackage of CentOS stream or repackage of Fedora or anything crazy like that?

Or does that just not make any sense to you guys?

Gregory: From my perspective, I haven't heard much rumblings at all through the community on that. I know that there's a lot of our community members, even our developers, that run and leverage Fedora, you know, quite a bit. Fedora is a fantastic collaborative operating system, and it has definitely been extraordinarily valuable.

Uh, area of the enterprise Linux ecosystem, but it is kind of the bleeding edge area of the enterprise

Jonathan: Linux

Gregory: ecosystem. On the Fentos stream side, this is a little bit more where you know, I don't want to put words in Alma's you know, you know, from their perspective, but it's a little bit more.

We're almost kind of focusing right

Jonathan: now.

Gregory: So almost a little bit more on the, on the central stream side and then guaranteeing, going back, guaranteeing that compatibility and then adding some additional value, which again, totally fantastic. I love that they're doing that from the Rocky perspective.

We're really just focused on being that. The successor of CentOS and to continue moving forward just to make sure that everybody has a very clear understanding of what CentOS was, CentOS Linux, excuse me, what CentOS Linux was, and now what Rocky is.

Jonathan: Do you guys still have a line of communication with?

Like the redhead engineers. So, and the reason I asked this is because I know that you like all of the rest of us, when you spend much time with an operating system, you find, you find problems, you find bugs, and you guys are in an interesting position to find those. Figure out the fixes and, you know, maybe get them pushed upstream.

You're also, because of the bug for bug compatibility, you're in an interesting place where you may know what the fix is, but you don't want to roll that fix out until Red Hat does. So is there, and I'm sure there must be some work where you're trying to push things back upstream to Red Hat. Please fix this.

Please pull this patch into your sources. What does that process look like?

Gregory: So our release engineering team has actually had a number of tickets in with, with Red Hat to, to either fix certain things or to provide feedback on certain things. I can tell you it is a, it is a difficult situation that and I'll say this kind of bluntly.

And again, I don't mean it in a negative way. But all of the enterprise Linux ecosystem, it's following the lead of Red Hat. And so as a result of that, it is not really community and collaborative. Yes. There's some opportunity to provide some collaboration back into Red Hat, but it is not really an open collaborative community.

So it makes it difficult for us. I would love to invite Red Hat to, be more open with that. But I'm not complaining. I think what we have right now is, I think, fantastic. And I think we're working through it in a very good way. With that being said this is, this is also provided difficulty and again, this is, this is not about plugging C.

I. Q. but to kind of talk about a difficulty is we have customer contracts where we are providing additional value. On top of what currently exists. In some cases, that's optimization. In some cases, that's hardening. In some cases, that's performance tuning et cetera. And some, there are some times in which we, CIQ decides, well, we need to go fix a, a security bug that Red Hat has decided that they, they don't want to fix.

Or something just happened recently, and I can get you the details. I don't have them off the top of my head. Where CIQ actually just pushed back on the CVE and we just got a CVE unvalidated, meaning it's not truly a CVE. So that's quite the process to do too. Yeah. Yeah. So we did that and that affected Red Hat in a positive way as well.

Cause Red Hat wasn't fixing it. So we actually got that CVE Nixed. So, oh, Nix is an operating system. Linux variant, so I shouldn't use that. A very cool one, by the way, I may add. So we're talking about kind of value add operating systems. Nix is super cool.

Jonathan: Oh yeah, there's some clever ones out there.

Gregory: So it has been, it has been a little bit of a challenge for, for CIQ. We have additional repositories for our customers that when we fix things that are either not fixed upstream yet and whatnot we fix it there. And in cases where we can, we do, we do push it back up to the, to the upstream communities.

In some cases that could be Red Hat Fedora. In some cases it was Stream. Some cases it's the, it's the upstream project itself. We have a lot of customers, for example. that are leveraging the upstream stable kernel. So we tend to work very closely also with the upstream stable kernel team because we're seeing performance and security enhancements on that front as well.

And again, it's actually, it's kind of an interesting way to consider the enterprise Linux ecosystem because you still get that completely compatible user space. So all your applications work, everything works as expected, but you're also now getting, you know, a better hardware support. In some cases, again, better performance.

And in some cases, you know, many times better security coverage by using the upstream stable kernel. So we've we've kind of played and dabbled with that for some of our big customers.

Jonathan: Yeah. All right. Well, we are at the end of our time. But I did want to ask each of you give the opportunity. Is there anything that I didn't ask about that we didn't cover that you want to make sure and get in?

And let's let let's let Christa go first, actually. Is there anything we didn't talk about that you wanted to let folks know about?

Krista: Well, I just want to say, I think one of the things that makes our community stand out is that we try to really engage with the people in the community. We think, we like to say, with the different Enterprise Linux distributions, to choose the one that fits you the best.

For us, I think some of the things that help us stand out are the community and the documentation. And our group of special interest groups that are SIGs that they are the groups that provide the things that are in addition to what Rocky Linux is, such as when we, when we do security fixes that need to be done before they're fixed upstream, they, they can be found there.

So people can add those for additional hardening. We love our documentation tree and then with our community, our community. I've been thinking about was the fact that there are several things that motivate people to engage in open source community and that can be anything from financial to psychological, like, as far as well being is concerned, it can be social, it can be educational, which is what brought me in, and it can be just for the chance to contribute freely to something on your own terms.

So as a community, the way I'm trying to grow it is by acknowledging that those are all ways that people can get something from our community and to address them in the ways that we can. So that as many people as possible, when they come to our community, they give themselves what they have to give, and then that they give something in return.

So that is my great goal for our community as the way to make it stand out.

Jonathan: Yeah, very good. And anything, anything else that we didn't talk about that you guys wanted to cover?

Gregory: I'll throw something out there. Sure. So before Rocky was created and it was literally like a month or two, I think, before December of

Jonathan: 2020,

Gregory: I was having conversation with some of my, my friends that are, you know, in open source and whatnot, and we were kind of talking about.

You know, comparing the shades of gray in our beards and, and, and talking about kind of what things were like, you know, in early open source and, and how amazing it was that, that an engineer contributor could make such a profound impact to the world. And and it, it, it kind of, we reflected that it was, it seemed a little sad that.

It kind of felt like that wasn't going to happen ever again. And the, the, the people that had that opportunity again, you know generations of, of, you know, graveyards in a matter of speaking. And, and we, we, you know, people that are newer to the field. May not ever see something on and then almost like, like serendipity magic, rocky happened and rocky has been so influential to so many people in terms of what it, what it does to Chris's point.

It is a, it is a. Social gathering place. It is a collaboration. It is a place where people who have never actually even maybe even just joining the field of technology or don't have kind of this long pedigree contributing an open source is a tremendous thing to put on your resume. It's a tremendous amount of experience that you can put under your belt.

And again, I mean, Kristen nailed this like everyone can go do this and you can do it on your time. So I want to stress whether you, whether, you know, somebody decides to join Rocky or they decide to join another open source project, get involved with open source. It really is truly special and choose, choose what community you want to be part of.

And I also want to say that. You don't have to be a software engineer, a software developer, to bring value to an open source project. Now, younger projects, sometimes, yeah, it's probably a little bit more on the technical side. But especially as the projects are a project starts to grow. They need every single role that you can imagine from documentation to marketing to even sales.

Like in some cases you are many, many projects need donations. Like it is a very different personality from an engineer to a salesperson, to somebody who's going to go and be successful in helping to gather funds. For, for an open source project there is governance, there is leadership, there is management there is program management, project management is even like, like everything you can imagine, like, like, come and join, you have an interest in being part of some of this I just want to encourage people that you may not be a traditional kind of engineer.

Software developer. There's value you can bring to an open source community. And I can tell you, if you decide that you want to check out Rocky, please do. Our community is wonderful. It's, it's very welcoming. Krista runs this cause she's, she's much friendlier than I am. And nobody likes Krista more than me.

So so Krista runs this and reach out, like come join our matter most at chat. rockylinux. org and come join in you'll see Krista there. You'll see me there. You'll see everybody there. And it's a fun place to hang out. It's a fun place to talk. And there's a lot of really cool things that are happening.

So come

Jonathan: join. Very cool. All right. Final questions for each of you and Well, the final question is, and we'll start with Krista, favorite text editor and scripting language. And I know sometimes for the community people that that question doesn't mean quite the same as it does to, to the, the engineer types, but all the same.

Krista: My favorite scripting language is in English.

Jonathan: That's fair. And text editor?

Krista: I use Windows.

Jonathan: That's fine. Like I said, we get I would love

Krista: to be more technical than I am. And I would love to sound more technical than I just did. Yeah. That's perfect. I use commonly available tools.

Jonathan: That's perfectly fine. Like I said, we get It's obviously a set of questions that are very much geared towards developers.

And when we get people that are from either either the business side of things or the community side of things It's just it's sort of a mismatch of the questions. But at the same time, it's fun. So Greg same two questions.

Gregory: So they're related for me. For almost all of my programming career I used vim and Leverage Vim, I still think in Vim, even when I'm in a Google Doc, I think in Vim.

So you'll see lots of random I's, A's, and colon W's all over my docs. With that being said going to the next question, again, which is related I've coded a lot historically, and you can make fun of me, in Perl, and in the past. And I'm really, I prefer C. As my primary programming language but within the last four years within with through a container system that I, that I created called a singularity and I moved it to Linux foundation.

It's now called obtainer. That's now in go I recoded a, where cluster provisioning system called werewolf that I've been running since 2001. I could recode that and go. And when I did that, I decided to use an IDE because picking up a new language with an IDE is much easier.

Jonathan: And

Gregory: so I chose VS Code for that.

So my text editor of choice is still Vim. But when I'm coding nowadays, I end up in VS Code. More often than not and Go is both a programming language as well as a scripting language.

Jonathan: Someone needs to come up with a browser extension to give us evil mode for Google Docs. Doesn't that sound like fun?

Somebody out there listening, let's get on that. That'd be awesome. Yeah. Alright Gregory and Krista, thank you both so much for being here. It's been a blast getting to talk about Rocky Linux and CIQ and the Rocky Software Foundation and the community there. Thank you both so much for your time.

Krista: Thanks for having us.

Jonathan: Thank you. Yeah. All right. And boy, that was a great conversation. Really, really enjoyed that. And, and love the the, the comparison and contrast with all Millennics and just that, that agreement that we're all happy that both of them are there. And I think that's great. That I think that's a neat story out of this.

So next week we've actually got something really cool and it's gonna be two weeks from when we record today, but if you watch this on the download after it's edited, it'll just be a week away. We're talking with Shimon Shacham about NAND to Tetris. That is an open source. But of courseware where you start with the most basic building block, the NAND gate, and you work through building a an A LU and A CPU, and then writing assembly code for that and building a very simple compiler all the way up to running Tetris on this computer that as a student, you.

And the whole thing is open source, and I think it's one of the most fun courses out there, and I asked Simon about it, and he's willing to come on and talk about it on the show. So that is the plan. We record that on the 25th, and it'll go live on the 26th, and so we hope to see everybody there. We appreciate everybody that watches and listens, both live and on the download, and we will see you all next week on Floss Weekly.

Episode 820 transcript

12 februari 2025 | min

FLOSS-820

Jonathan: Hey folks, this week I talk with Ryan Sipes about Thunderbird, it's history, it's future, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 820, recorded Tuesday, February the 4th. Please don't add A. I. Clippy to Thunderbird. Hey folks, it's time for Floss Weekly, that's the show about free, libre, and open source software.

I'm your host, This is Jonathan Bennett. We've got something really fun today. We're talking with Ryan Sipes about Thunderbird. That's the email client that you, you should be using. And in fact, a lot of us are using. And it's, it's, it's pretty good. It is part of the Mozilla suite and you know, one of the things I've talked about for a while, I was told by a little birdie that it almost became part of the LibreOffice suite.

And so I'm going to ask about that. Cause I'm really curious about how that story went down. But I am going to go ahead and bring Ryan on. Mr. Ryan Sipes. Well, welcome to the show. It's so good to have you.

Ryan: It's good to be here. Thank you for having me.

Jonathan: Yeah, Ryan and I've been chatting before the before the recording started and we talked about a couple of different things and it's like, oh, put a pin in that.

We need to make sure and get back to that. Oh, put a pin in that. When they asked me about that during the show. So let's, let's start with the overview. What, what is Thunderbird? Why does it exist? What's the problem it's trying to solve?

Ryan: Sure. Thunderbird is an email client, first and foremost. It's an email, calendaring, and contact management client.

Those aren't in vogue right now. They, I mean, maybe they're coming back into vogue but for the longest time, folks were like, well, I'm just going to use webmail, which would be like Gmail or the Outlook online experience. Right. What Thunderbird's main difference between everyone else and us is, is that we are open source software, free and open source software, and we really value our users freedom, whether that's freedom of choice, freedom to change Thunderbird to meet their needs and just freedom to own their email we have free your inbox, that's our kind of tagline right now.

And we sincerely believe that that's a worthy cause. And, ultimately, we want people to control their communication experience. We want people to control their own personal communication information. And that's what Thunderbird strives to do in the world.

Jonathan: This Thunderbird, I've, I've, I've used Thunderbird for a long time.

And I've played with multiple features over the years. Does Thunderbird also do chat to support some of the chat protocols? It does have

Ryan: IRC built in and it does have Matrix, although the Matrix one is currently under development. It doesn't have every feature that you, all the other Matrix clients might have.

But yes, it does do chat. It also does tasks. And it kind of follows the CalDAV standard to do that. Tasks is a part of the CalDAV standard. And so if you have a CalDAV calendar that's, that can play nicely it'll do, it'll connect to that to do tasks. Or you can just do it locally on the client.

The tasks and, and chat are going to be seeing some love over the next couple years to try to make them a, More competitive experience versus what else is out there.

Jonathan: Yeah. So let's see, there's, oh, there's so many directions I want to go. Let's, let's talk about, let's talk about the LibreOffice thing, right?

I tease that. Let's just go ahead and dive into that. So I, I've, I've said this for the longest time that, and it's, I say it less now, cause things have changed, but there for a while, Thunderbird. But, well, I mean, it's just, it sort of struggled as part of, as part of Mozilla, right? And I know there was a conversation had when OpenOffice forked and became LibreOffice under the Document Foundation.

There was a conversation that was had of, well, why don't we just pull Thunderbird in as part of the LibreOffice suite? You know, Microsoft and their suite has all text editor and, and spreadsheet and an email client, so why doesn't Library office have the same thing and that didn't happen and for a while.

I said well, maybe that should have happened It might have been a better place for it And then obviously Thunderbird has sort of had a resurgence of sorts at at Mozilla, which we'll talk about that, too But but let's let's go back and cover some of that history maybe even why Thunderbird struggled at Mozilla Why it was not moved into LibreOffice.

Like, there's a whole, there's a whole thread of a story through here, I think.

Ryan: Yeah, so, we, to, to talk about this, we have to go back. We have to go way back. Way back. We can go back as far as 2012 to really set the stage. There's this TechCrunch article which I put in My presentations that I've given at FOSDEM and other places that it's an article that says that's it for Thunderbird And it's our logo, but it's got X's over the eyes.

Oh, I remember that. I remember that. And it declares the death of Thunderbird because Mozilla Corporation in it shares that they're not planning to actively develop it. They're gonna find some other way forward. for the project. And that began Thunderbird's time in the wilderness. Where a bunch of different things were tried in order to make it sustainable.

And if you, if folks have heard me before they'll know that when you're talking about an open source project about starting one, about continuing one, I think a conversation that developers don't have often enough, right from the outset. is how is this going to be sustainable? Let's say we get users, or maybe you already have users, who rely on the the software.

Okay, well, that's great. How are we going to ensure that we continue to develop the software into the future? And the problem with Thunderbird was that no one had found A good way for it to sustain itself. It didn't have a Google contract, search contract. It didn't have anything like that. Not for lack of trying, just folks didn't find a suitable way to sustain the software.

And So, by the time, I think it would have been, oh gosh, I don't know the exact year, 2014, maybe 2015, conversations I think were happening, this might be a year off in any which direction, conversations were happening with other homes, other potential homes for Thunderbird because Mozilla actually was and still is a good steward for the project, so much so that they were saying, Listen, you know, yes, this, this is a piece of software that we've spent a ton of resources building, developing, but we need to find where it's gonna be able to thrive.

And so to their credit, they were, there were discussions with a variety of organizations who thought that they could carry it forward. One of them was the Libre Office folks. And ultimately there was an elected Thunderbird Council that participated in those conversations of the contributors.

It, they elected seven representatives who participated in those conversations. And ultimately that group was the one who made the decision that they wanted to stay with the Mozilla Foundation. And so it became a Mozilla Foundation project. And. I know that there were a variety of reasons for that. The biggest one is probably they just felt like they were still a part of the Mozilla community.

And still, that the Mozilla manifesto and its values were perfectly aligned with the project. Which made sense, because that was the organization that developed it. And so they, they stuck around there. And Even I had some questions, I came on in 2017, and it was still, this conversation was still fresh in everyone's mind, and I came on as community manager, and so I spent some time getting up to date on what had happened, why we were where we were, and what we were planning to do, and what I found was that folks thought there was still a story to tell, Within Mozilla.

And at the time that was I was like, I don't know. Like, let's, let's see if that's actually true. And and it turned out to be right. No one knew how it was gonna look, but there was a lot of faith that the community would figure it out.

Jonathan: And so, and so what, what happened? Where did, where did the sort of the corner get turned where you went from?

We want to be part of Mozilla Foundation, but we don't know, we don't know how to pay rent for our developers, essentially. Yeah. To Well, there

Ryan: was

Jonathan: Yeah.

Ryan: Yeah, yeah. There was a little bit of money coming in, through donations. Mm

Jonathan: hmm.

Ryan: And it was just enough to hire me, and I think there was one other person on when I first came on.

Jonathan: Okay.

Ryan: But They were part time contractors, including myself. And my funny contribution to the story is, I saw that there was a listing for this part time, and I said, you know what, I could spend evenings, you know, helping out as a community manager on this project. And so I applied, and I, I came on, and as I worked on it, I started to realize How much untapped potential was within Thunderbird.

We had 20, we had over 20 million users. We, and those were users who were dedicated. They opened Thunderbird almost every day. And when I would interview users, I would ask, how, how long do you have the application open? And they said, well, the entire workday. It's like six to eight hours. Yeah, of course. And so you know, my background was Mycroft.

I started Mycroft. I worked at System76. I did some other stuff in there. And I remember thinking at, at, in my previous experience to have users use your thing every day and to use it for most of your day, that would And to have 20 million of those people would be, it's shocking. You know, that's a to use a word, I'm not sure I like exactly, but it's a sticky application.

That means it's, it's sticky in people's lives. They love it. And they, they need, they rely on it. And so the thought I had, and what I started talking to the Thunderbird council about was how. How are we, how is this thing that's so important to people just going to like, slowly die? Because that's kind of the glide path

Jonathan: it

Ryan: was on.

It, it was not necessarily growing all that much. And there were a lot of issues. Like we couldn't actually build the product most days. It was in an unbuildable state. And the technical debt was drowning everyone. Yeah. But. What we were able to do and my kind of contribution was, I told folks, we've got to get out in front of our users and we have to tell them what's happening and how they can help.

Because when I interviewed our users, I asked, you know, I just said, tell me what you know about Thunderbird. Tell me what you know about Mozilla. And most of our users who are just polled like that randomly said they thought that Thunderbird was supported by the Firefox money. By, you know, search contract money.

Or they didn't know, but they thought it was a very A very lucrative

Jonathan: project. Yeah, Google's involved there somewhere. You guys are, yeah. Like that story I told before. You guys are getting the big Google money, right? Yeah, exactly.

Ryan: But the truth was that it wasn't. It wasn't. It was just like pennies from the couch.

You know, the project was coming up with. And and once, it took a while to convince folks. Once we, once we managed to convince the folks in the community, the folks in the council, the folks in who, the other folks who were working there, Hey, this is actually a problem that we need to address.

Jonathan: Yeah.

Ryan: Well, we started really making some, some headway. Mm hmm. The, the turnaround, I, I hate to say it because a lot of open, open source, Folks, which I include myself in, cringe about the money thing, but even the infrastructure to build and distribute Thunderbird is

Jonathan: expensive.

Ryan: If you have, in any given year, I don't remember the exact number, but we have I think 10, we can have 10 million downloads in any given year or more.

Jonathan: Yeah. It's huge

Ryan: Distributing that is just expensive. Mm-hmm . And the, and so not having the money to even, to like, just barely sustain that, which is what we had, just barely the money to sustain. That meant we weren't addressing. Problems, because there were, there was work that we couldn't pay people to do.

There was technical debt work that people were like, I don't have the time for that. It doesn't sound interesting to me. It just sounds like a grind. And so, we knew we needed to be able to pay people to get us out of the hole that Thunderbird was in.

Jonathan: Yeah.

Ryan: And ultimately the turning point was getting in front of our users, telling them that we needed money and having them give money so we could hire.

Developers to help us improve the product and pay down our technical debt.

Jonathan: Yeah Okay, that's that's interesting. I'm reminded of I don't know if you've followed like linux desktops, but kde Did something that sort of sounds familiar? It's similar to this. And it's the dreaded request for donations, right?

And what kde did is they I think it's like once or twice a year. They have it built into their code I think it's twice a year now. So like christmas and then july It'll pop up and say hey We need money to be able to keep this thing going. Click here to donate. And they, they brought in like a hundred thousand dollars in a month by doing that.

It was ridiculous.

Ryan: I know about that because I was, because somebody came and talked to me before they did it. To ask me questions about what we did. And I told them, get right in their face. Because that's what we did. We ultimately popped up a big notice that just said, Thunderbird needs you to survive.

Jonathan: Yeah.

Ryan: And we explained. All the things that I just laid out. We pay for expensive infrastructure to distribute Thunderbird. We have to pay people to triage bugs. We have to pay people to, you know, develop the code and, and ultimately it worked. Was, was there any,

Jonathan: how much pushback did you get from that?

Like, was there, did you get angry emails? Did people picket at Mozilla headquarters?

Ryan: Yeah, so what I tell people is If you imagine I had to bring people along at the beginning. People didn't like this idea when I first raised it. They said it's just, it just seems gross. To ask people for money. And And, and they also said, well what if they just decide to uninstall Thunderbird because we asked for money?

There's a few different threads to pull on here.

Jonathan: Fewer downloads you have to pay for, in that case.

Ryan: Yeah, well, one of which is, One of which is, and some people are really going to hate this, but, You need your users to, to want to support the software. However they can. If you have users who are just taking and giving nothing back in return, From a sustainability perspective, those users are actually a, a cost, only a cost.

And, I mean, there are ways you could say, like, okay, well, by using it, they're potentially marketing it to others. That's true. There are a number of ways in which they help the project. But, from a, just at the end of the year, like you said, like, they downloaded. You know, that's, that's just a cost and we're not getting back anything in return.

Now, that's not to say it's not good to have folks who just use the software and do nothing in return. But, it doesn't make for a sustainable project. What we weren't doing was giving users who didn't, who couldn't dedicate time, an avenue, a clear avenue, to dedicate something to help the project. We weren't letting folks know that we needed support.

And so they weren't even having the opportunity to say, Okay, I get value from this. I'm gonna give some of that value back to the project. And the truth is, is that a lot of the people I talked to said, Oh yeah, well, if people want to support us, they can come develop the software. But I was like, most people don't have either the time nor the expertise to actually come and do that, so we have to give them another avenue to participate, even if that's just, send us five dollars.

And then, once we did that, the pushback we got was super minimal because people don't even see the full screen appeals that we make. And what I mean by that is that they're so used to seeing advertisements and appeals from software that the other software that they use that says pay for our monthly thing, pay for our, you know, I think of Evernote, every time I opened Evernote, I saw like, you need, you should pay.

And it was, and that they actually just like, they're either going to give or a lot of them. They don't even see it. It's just like a

Jonathan: reflex. Yeah.

Ryan: Yeah, and so don't

Jonathan: ask me again. Just look for that button and press it Okay. Yeah, exactly. Yeah,

Ryan: and so We do get some pushback, but it was very minimal, you know, we hit 20 million users We may be you know over the past we've been doing this since what was the first year we did it really big 2021 maybe we have gotten A random email every once in a while, just saying like, I don't want to see the appeal anymore.

But it's, to your point about KDE, we've only ever, at first we just did it once a year, and we recently moved to twice a year. You know, so twice a year you see, a tab pop up in Thunderbird that asks if you want to give, and you can hit X, and then you won't see it for six months. It's not

Jonathan: particularly onerous.

Ryan: Yeah. I mean, for what, for all the value, if you're one of our users who are using it every day, 8 hours a day, we're at, we take about maybe 8 seconds of your time with this appeal. If you give, you know, it's probably like a minute. And, and then you're, you're done. You don't, you know, you'll see, you'll spend another 8 seconds on it in 6 months.

Jonathan: And so what, what has the what is the positive response been like? How, how much how many, how many donations does that result in?

Ryan: So this last year, which we, we published a financial report, usually sometime in the middle of the next year on the previous year, once we've really had time to make sure that we have settled everything and we know exactly what the, everything looked like.

But. The, this year we in donations, we raised 10. 3 million.

Jonathan: Wow. That is impressive. Out of, out of a user base of how many? 20 million. Roughly.

Ryan: Okay.

Jonathan: So that's, yeah, that's impressive. That's really good. That lets you, that lets you hire a few more developers.

Ryan: Yeah. We're up to about 40 people.

Jonathan: Yeah. You know, you talk about sustainability, and I imagine that most of our audience realizes this.

And we touched on it briefly, but, especially at these scales, when you talk about that many downloads and that many users, like, bandwidth costs money. The ability to have a server somewhere costs money. Having a, having a continuous integration running costs money. And for a, for a tiny project, yes, you can use GitHub for those things and, and get away from most of those costs.

But the bigger a project gets, the, the less sort of tenable that becomes and you have to have some sort of infrastructure. And so, you know, I, I, and I say this because the, the term sustainability means different things in different contexts. And to some people it's kind of a It's a, it's a throwaway word that, you know, you tune out everything that comes afterwards because sustainability is sometimes what people are talking about.

But we're just, really what we're talking about here is the ability to pay your bills as, as a as an organization. And I'm, I'm, I'm thrilled that Thunderbird is in a better position now to be able to do that going forwards. Have you, where, have you peaked? Let me, let me ask that. Have you hit peak donation yet?

You know, like, was there a point where you You had more and then it's slowly tapering off, or is it still growing? What's the trajectory look

Ryan: like? We haven't. We, we, every year we've watched to see, because every year it's grown since I came on. When I came on, it was well, we, the, the track we were on was anywhere between 300, 000 for that year and 500,

Jonathan: 000.

Ryan: And we, And we, we already did some donation things in the first year that I was on. So, to improve that. So we actually ended, I think, at 700 and some. And then, but that was still, like you said, there, there were still So many infrastructure costs and things like that, that it's not as big as I've had smaller projects be like, that's everything you'd ever need.

And I'm like, yeah, well, distributing this on all platforms and all languages in multiple build types, it's, it's, it's not as much as you, as you think. But the, the yeah, the Kind of trajectory each year has so far gone up, and we, like, for instance, and you can see this on our blog, We went from, if I'm remembering right, we went from, what was it?

From three, so, from essentially just below three million, to six million, to eight point two, two three? To 10. 3. So those are the last few years. Yeah. So we, we were curious, we've been very curious about is there a peak, what is, I mean, there is a peak eventually, somewhere in there. What is that, when do we hit fatigue from people seeing these messages?

And, or, or the people who are going to give have already given and, you know. They don't want to give anymore. But what we've found is that so far our users respond to us just engaging with them and telling them what we're doing why we're doing it, and then what we need in order to do that. And so I think we've done a really good job of just Just bringing folks into the conversation who otherwise were just using the software and not thinking about any of this, to where now they realize, oh, these features that I want, these improvements that I want, I can get them by, and this is how I can participate.

Jonathan: Yeah. I am, I'm looking at the the, the Thunderbird. net slash donate page here. And a couple of things that, that, that stick out to me is you do have the recurring gift option. So like sort of, sort of the, the Patreon model where I'll give you 5 every month. And then it's not tax deductible, which is, which is kind of surprising.

Why is that? Why does Thunderbird not count as a charity?

Ryan: Well, there's a variety of reasons. At one point we were sitting in the foundation and we were getting big. We were starting to get big. Donations were going up and we were hiring developers in order to work on the software.

Jonathan: The

Ryan: IRS doesn't count development of software as a charitable.

activity.

Jonathan: Oh.

Ryan: And you can only have such a certain amount of threshold. You, a majority of what you do has to be this type of charitable activity. Charitable

Jonathan: activity, yes, yes.

Ryan: And there are a couple ways we could do that. We, we could find some way to spend a majority of the money on charitable activities, which could be Messaging about open source software and Thunderbird, putting on conferences, putting on other types of, you know, events, but, but that seemed kind of wrong.

Yeah. At some point

Jonathan: that would be a betrayal of your users. If you say, Oh, you've got to spend more than 50 percent of our money on stuff. That's not what people are paying us to do.

Ryan: Yeah, exactly. They were paying us to develop the software. And so, and additionally, and we'll talk about this here in a moment, there was more that we wanted to do.

Because how we look at developing Thunderbird is not about necessarily just developing an email client. We start from where are our values. What are the values behind the software? If we just wanted to develop an email client, we could be some proprietary, you know, nice email client that charges everybody ten bucks, you know, to use it or whatever.

What the, what we want to be, what we want to do is promote our values. And our values are open standard communication, open source software, whatever. And the freedom to choose how you communicate on the internet. And and so, in being a non profit entity, with the restrictions that we had, there were also things like, what if, what if we wanted to offer an email service that had those values at some point?

You know, how would we do that within the non profit model? It wouldn't be a It wouldn't probably be sustainable because you're going to want to Charge people for the server time and every, you know, for that service. And so there was also this conversation about, well, what else do we want to do that would enhance our users lives that we might be hamstrung by a 501c3.

And so for all those reasons, we chose to go the for profit route and Mozilla created an entity. That Thunderbird lives in and that's healthy, I think, because it also means that Thunderbird lives or dies on its own merits.

Jonathan: So I was gonna, I was gonna ask, I was gonna go here next, actually, and you're, you're, you're poking around the same question and that is, Is Thunderbird part of Mozilla Corporation?

Ryan: No. Thunderbird is not a part of Mozilla Corporation. That is another entity. That is owned by the Mozilla Foundation. We are owned by the non profit Mozilla Foundation.

Jonathan: So, when we see in the news that the Mozilla Corporation has re branded themselves and changed their logo and all of that fun stuff, that doesn't, that's not necessarily having anything to do with Thunderbird.

Ryan: Yeah, I mean there are, we are siblings, and we frankly develop some of the same software, and there are a number of things that we collaborate on, for instance our build system. We share a build system with Firefox and we pay for our part of it. And doing that alone might be even harder than, you know, right now where we all collectively work together towards creating this thing that,

Jonathan: that

Ryan: makes Firefox and, builds Firefox and Thunderbird and, and all that's, those different languages and build types, whether it's ARM or, you know, whatever it is.

That is shared work, and I can't imagine I mean, I can't imagine doing it alone, but I think it's much better that, that we are these siblings that work together on the things that have shared value for both

Jonathan: entities. Yeah so, The question about, and again, I'm looking at the FAQ here on the donate page, and one of the last ones here is, well, doesn't Thunderbird earn any income?

And I am, I'm super curious about this. Like, is there, is there, it's mostly donations, I get that. Are there any other commercial things where Thunderbird brings income in? Obviously, you guys have explored that over the years you know, have, let's see, who, who would it, who would it be you know, has somebody thought about, well, let's take the, the Thunderbird code base and rebrand it and sell it, you know, you've got some open source projects make money that way has that just, has nobody stepped forward and wanted to do that?

Is there just not a

Ryan: Other companies did that. Postbox. That got sold to just, just this year, well, this past year. Got sold to EM Client, I think is what it's called. They were at Thunderbird, they were building off our code base and selling it. Which was, which was kind of a sad thing for a bit because when they were making more money than we were selling our software, they were adding things, they were adding some value, they were adding some features and things, but you know, the, the underlying work was, you know, enabled by us and we weren't getting a cent from it, you know, that was a rough period where you really have to look at the project and say, you know, wow, people are, you know, Actually making money off of this and we're, we're really not, not, we're not.

Jonathan: Yeah. And It's always, it's always a challenge as an open source project. That was tough.

Ryan: That was tough. But the, but yeah, we, we considered for a while you could get an email address from a provider. When you start up Thunderbird, you could hit, get an email address. And if you signed up with one of those providers, we would get a cut of your subscription.

But it was, It was miniscule. Yeah, yeah. And and ultimately we, we always partnered with providers who we liked and we thought were, shared our values, but it just, for the amount of just even maintaining the future, it didn't make sense for the amount of money we were getting because it, it was like, not, not much at all.

Right,

Jonathan: right.

Ryan: And that was because when that was That was because what the aim of that was, was not to make Thunderbird rich. I mean, maybe it was. It predated me. But it was it was to provide users with, if you're coming here and you need an email account, you know, here's what we recommend. And ultimately where we're at these days is, What are the value adds that we can provide to users?

And this is what we're looking at now. That will make the experience of using Thunderbird better, and will make our users lives better. That will also help support and sustain Thunderbird. And so that's probably, if we're going to make another jump, that's probably where the next jump is, which is, are there any services that we can offer that can help?

You know, require, we're not going to charge for anything that doesn't require running server infrastructure to provide it. If it can be shipped with the client, it will. But the, for free. But if there are things that enhance people's lives, then we're going to Look at what of those we can provide as a service to users.

Jonathan: Yeah. This goes right along. Is there anything that is in Thunderbird that is limited to people that donate or pay for it? And I kind of have in the back of my mind what Ardor does with this. So their code is totally open source, but to get access to their builds, particularly their Windows builds, is where it's tricky.

You, if you donate, you get access to the builds. And that, that actually works out fairly well for them. Is that something you guys thought about?

Ryan: We're not doing that We, we are starting to come up with things that are What's the, what's the word? Cosmetic? That we might offer folks? You know, it doesn't, it doesn't necessarily affect the experience at all.

It's just like, are there things in the UI that we can recognize that you're a donor? Are there, we're providing, donors are getting emailed a a wallpaper. A Thunderbird, a cool wallpaper that our design folks did.

Jonathan: Skin unlocks.

Ryan: Yeah, exactly. I think about it like Yes, it's like Fortnite. You can play Fortnite.

You know, you don't have to buy the skins and everything. But, and, so we're looking at things like that. But we don't really want to, I want the internet to hear this. We do not want to gatekeep people from a full Thunderbird experience based on whether or not they paid. We're still, we're, it's a part of our values.

That we, you know, we want everyone to have access to this communication experience we're providing. And we don't want it to be gate kept by subscription. Unless we absolutely have to.

Jonathan: Yeah.

Ryan: When we talk about the services, what we're talking about is if there's something that costs us every month.

Jonathan: Yeah, you've got to, you've got to pass that on.

Ryan: We need to, we need to at least cover our costs.

Jonathan: Yeah. Okay. So we do have some live listeners and mashed potato, one of our, one of our longterm listeners says, I hope dark mode won't go behind a paywall tier.

Ryan: No, it's not going behind a paywall tier. There you go. And dark mode, dark mode just got a lot better.

By the way, if you're using the daily builds now, when you're in dark mode, the emails. are made dark. And there's a little toggle now. So if you're looking at an email and something about the mechanism by which we use to make the email content dark just isn't working, something's hard to read, there's just a nice little sun toggle there where you can turn it on and see it, see the original, the content of the email in its original form.

Jonathan: Yeah, nice. Nice. I do not run Thunderbird. I do run Thunderbird. I do not run it in dark mode. I will have to look at that because that sounds intriguing. Yeah. Okay, so with some other open source projects, I don't know that this, this works here, but particularly when we talk about libraries and things that are dependencies there's beginning to be a bit of a commercialization change because Particularly the European Union has passed some laws about, you know, there have to be S bombs and, you know, you have to have, if you're selling this thing as part of a commercial product, you have to be able to, you know, give these guarantees.

And so, you know, every once in a while, one of these open source projects, they'll get reached, they'll get touched, talked to by a company that says, Hey, we need you to provide us an S bomb. And I am convinced that the correct answer for that is we will be glad to do that. Here is the amount that that will cost.

And that is a way that the company can then support the project financially because there's a give and a take here does that make is that something that's going to be? Part of thunderbird going forwards or because thunderbird is sort of that End product. Is there no one else at the end of the chain that's paying for?

You know, getting an S bomb and getting all of those things.

Ryan: Well, I think there's, there's so much wrapped up in that, first off. This is some place, this is an area where our relationship to the Mozilla Foundation is really helpful. Because some of this is when you have regulations. that regulate open source software without a real understanding that open, with, with a convenient forgetting that open source software exists and that some of these things don't make sense, regulations don't make sense when they're applied to open source software.

It happens really often across all countries. And what's kind of interesting is folks will reach out to us with all sorts of requests from, from, from governments or to participate. That, mostly like, we want to use this software in this context, but my region requires x, y, and z for that to happen. And, there's some level of education that goes into that, where we say, listen, you know, what, a lot of times what they're asking for is some kind of security

Jonathan: Some kind of security guarantee, right?

Yeah, exactly. I can send you an email with a problem and you'll get me a security fix within 72 hours.

Ryan: Yeah, well what usually they want is like, have you gone through X, Y, or Z?

Jonathan: Oh,

Ryan: okay. And what they're asking makes sense when it's a SAS product. Or when it's a product where you can't see what data they're taking off of your machine, out of your use.

And sending it up to themselves. And they're looking for like, hey, is my data safe essentially? Like, and what we've been doing is pointing them at our audit, providing, you know, figuring out what the regulation is, and providing information on this is why it doesn't apply here. Because ultimately, there's a lot of cases where The regulation doesn't necessarily apply to our software.

Or, if it does, and we don't think that's right, we then raise that issue to the Mozilla Foundation and to our peers to say, do we need to contact, you know, regulators in this, in this jurisdiction and tell them, hey, you didn't actually account for Open source software here, and that is actually, that is actually created, that has resulted in real changes to laws and regulations as a result, and I think that's important, like a small software, open source software project can't do that, but Mozilla scale can do that sort of thing, and so when you see people online, kind of denigrate Mozilla, You know, and some of the stuff that they're doing, whether it's around policy or other things like that, you have to remember that that's some of the work that Mozilla does that's vitally important for open source software to be able to exist and be competitive in some of these areas.

Jonathan: Yeah, we've talked about this several times on the show. People in, the, the legislatures in Europe and the United States and other places, they don't necessarily understand how software works. They don't understand how open source works. And so there is, there's a desperate need for, like, people that get it to go and try to help educate, explain why, you know, we understand what you're trying to do to make things more secure, and that's great and all, but here's where there's a disconnect from reality.

And so yeah, that sort of thing is, is super important.

Ryan: And at least in, in, My experience talking to folks in government, generally, they want to do the right thing. They're ultimately trying to make it so that, I mean, this isn't always true, of course. And not true of all countries. But, but let's say, like, in Europe.

In Europe, I've gotten a distinct, and in the US, but in Europe especially, I've gotten a distinct from talking to the folks who I have, who are trying to do like the DMA, the and other stuff like that, that they do want to ultimately help make their, the users in their countries. Privacy be respected.

Mm-hmm . They, they, the, the outcomes they're shooting for are ones that I think most folks in the open source community want, but unless there's someone in the room from our community sharing how it affects the software that we make, open source software broadly, then you can have misses where they don't account for, you know, what they, they're thinking of it oftentimes in a.

This is provided by a business that sells it, you know, what is, what are we going to do around that? And so it's good to have someone in the room saying, Hey, you, but think about how this affects this software developed by a community of people. Yeah. Yeah.

Jonathan: Is, is there a future that you can imagine where, you know, someone can pay for some ISO certified copy of the Thunderbird software?

Ryan: The questions. Keep coming up, and I do think we're going to find a way to do that, probably in the context of an enterprise support contract of some sort that will come with additional things, which would be important, such as help deploying at scale help with specific problems that folks face when they're running Thunderbird in an organization.

This is something we've discussed quite a lot. And I think it's really important because who, in Thunderbird's context, who, what else do you use instead? Well, you use Outlook. And Outlook and Microsoft have a very, you know, complete offering around deploying in large organizations and trying to solve their problems.

And so We have to, if we want to be truly competitive and, and also a good alternative, we need to develop some of that capability as well to be able to support, you know, those environments. And I think that's another thing where, when folks think about Thunderbird, they think about it as, oftentimes, one guy choosing to use it.

On his personal computer. But actually, we have entire governments that use Thunderbird at scale. And so, when we think about supporting our users, we don't just think about the individuals, but also, how do we support these large organizations that use Thunderbird, and what their requirements are. And it can be complicated.

Jonathan: Yeah, so this is probably a good question a good time to get this question in One of the guys that co hosts on the show from time to time David Ruggles He says he wants to know about support for office 365 email Especially multiple accounts from different tenants. So this is one of the backends for email backend.

So obviously Thunderbird is great with pop and IMAP. If you want to use one of those two, we're, we're good to go all day. What about, what about the, that, that other backend from the company that shall not be named?

Ryan: This was a enormous, your, your audience is going to love this. This was an enormous argument in our community because I.

Tried to give Thunderbird to a few people in my family to use and they were going oh, they were like looks great I like the customizability. I like the ability to you know The Thunderbird has a lot of competitive great features and the interface, you know Especially folks who are coming from the ribbon interface of Outlook.

We're like, this is much better and Then they went to add their Outlook or office 365 account and they had issues What a bummer, right? And to them, to us, it's like, Oh, your administrator doesn't have the IMAP setting turned on. To them, it's, this is email. It's an email client. Why can't, why doesn't it just work?

The community, we had this discussion about whether or not we should support closed standards. And I can understand why someone would argue that we shouldn't. I was on the other side of that, in that if we can't, if we can't let them in the door, we can't then bring them to our way of thinking. If they can't even use the product to begin with, then they can't begin to understand why open standards and open source software is important.

They're just gonna go use Outlook, which is what they have done, which is what my family members had to do. Eh. And so we've actually been on a couple year long quest to support EWS. And that's the protocol that will allow us to connect Office 365 accounts.

Jonathan: And that's, that's Exchange, right? That's like the Exchange backend?

Yeah.

Ryan: Yeah, yeah, yeah, that's exactly right, and it's complicated because it's not an open standard,

Jonathan: So there's You mean, you mean Microsoft wrote a, a, a server protocol that's not straightforward and clean and isn't just great to work with?

Ryan: It's not intended for, it's not intended for Thunderbird to use in order to connect to their service.

In fact, they would prefer that No third party connected to their service.

Jonathan: Yeah.

Ryan: But we, we made a lot of progress. And I really truly believe that by the end of 2025 majority of users using Office 365 or Outlook. com or any of the Microsoft accounts will be able to connect to their accounts and use it alongside their other accounts.

Jonathan: Very good. Very good. Has there been any even an attempt? at convincing Microsoft to make this an open standard? Has that been a conversation that you guys have tried to have?

Ryan: Earlier on, we've had some informal conversations. I do think, I do think we need to go back to them now that we've done work in this area and we understand the problem space more as far as they're concerned, and talk with them.

I don't think, I don't think it makes sense to Not try and convince them to either turn these into open standards or to have them at least in the future think about how other software projects will interact with their services.

Jonathan: Yeah.

Ryan: So, and, and, so my answer is yes, we will talk to them. We're now getting to a point where we actually can talk to them.

Jonathan: Yeah.

Ryan: Before, I feel like we were, we just didn't have the bandwidth to carry on a ongoing conversation with them about collaboration. But I think, I think now we're at a size and, and I know some folks are going to, hair is going to go on fire when I say this, but Microsoft does seem like They're more and more interested in engaging with communities.

That's

Jonathan: exactly the direction I was going to go. You will have a better chance of making this happen with Microsoft of today than the Microsoft of even four or five years ago.

Ryan: And I think that, I don't know if you've seen this, but there are people from our community, from our wider open source community, who now work at Microsoft.

And so it's becoming easier too because it's like, Oh, I know that guy. Yeah, exactly. And you say, like, okay, can you, can you go talk to this person for me and tell them why this is important? And that's, that's a good thing. That's, that's a really good thing that that we have kind of infiltrated the, the, you know.

No, no,

Jonathan: cross pollinated. That's a better, that's a better term. We don't infiltrate.

Ryan: No, no, no, cross pollinated. That's right, that's right. And and haven't you heard, Microsoft loves open source now, so. I am

Jonathan: sort of starting to believe that when they first started saying that I was very skeptical and cynical about that It's like yeah, tell me that when you're no longer making millions of dollars from Android manufacturers from your patent wars But well, here's a long way.

Here's

Ryan: the thing and I think this is this goes back to something I said at the beginning when you develop a product what they teach product managers is Is before you Before you even build a product, when you're just in the ideation stage, Part of product management training is, How are you going to, what's a business model?

How is a product going to make money? And continue to exist? And what we have to do with folks like Microsoft is tell them, in part, This doesn't serve you or your customers to gatekeep this, to make it so that, you know, folks who run these accounts can't use third party email clients. And, that is not, we, you can already in your head come up with reasons why they wouldn't want to allow that.

So, you have to really think about it from their perspective. What is the, what is the, What do they gain out of this? And But, you know, corporations have people in them too. And sometimes those people are open source enthusiasts. So sometimes they can be brought around just by the argument of this is just better for the world, it's better for everyone.

Jonathan: I, I, I think there really is an argument to be made there, probably, probably privately, but An argument to be made that an open standard for exchange gives Microsoft one less frontier of antitrust liability.

Ryan: Yeah.

Jonathan: And that's, that is something that tech companies are starting to have to think about again.

Ryan: Well, with the Digital Markets Act and is that the only one? In Europe, this message interoperability thing has mostly been approached from a chat perspective. But there's an email piece there too. And and ultimately what Thunderbird doesn't want to see, what Mozilla doesn't want to see, is this Email shouldn't is like the best decentralized.

messaging platform we have. It has the highest amount of adoption. It, I email you, you email me, we have no idea who our providers are necessarily.

Jonathan: You don't have to know. You don't have to care.

Ryan: Exactly. And so we should preserve that. And, you know, the, we've seen as folks have moved to webmail that more and more of these providers have closed down.

Their connections with their ability to easily connect with clients or to essentially move your email and use it in the way that you want and So that's an area where We kind of have to push back so that at least the one really great decentralized communication platform that we have, communication technology that we have, doesn't ultimately end up a fiefdom, you know.

Jonathan: Haven't you heard? Email is dead. Spam killed it.

Yeah, yeah, exactly.

That actually does lead right into a question about how Thunderbird handles spam. Surely this is one of your constant headaches, right?

Ryan: Yeah, we have a, a built in spam filter, and it learns, you know, from what you mark as spam. It does have some basic idea, and I think it does a pretty good job.

But, if you tell it, additionally, by just marking the little spam thing, when you see spam make it It learns from that. Providers also are doing a much better job of this across the board. And so they kind of automatically filter your spam to your junk folder. Right. And I've seen over time that's getting better and better.

For instance, phishing emails seven years ago, eight years ago, were much more likely to make it into your inbox than they are today.

Jonathan: Yeah.

Ryan: And so I think it's like A two way thing. On the client, we're going to continue to do more, and I think that now that we have these LLMs that can run on device, we can roll that technology in for spam detection, which I think will go a long way, but also the providers aren't sitting on their hands, and we see that they're getting better and better at detecting When a message is potentially harmful, and, or, or it's something that you don't want to see.

And putting it in the right place.

Jonathan: Yeah, so you said LLM, so we can all wipe that off on our finger guards. Yeah. No, so okay, this is, this is, this is an interesting kind of direction to go. There is a little, it sounds like there's a tiny bit of AI built in already. This is old,

Ryan: this is old stuff though.

Right, right. Well, okay,

Jonathan: so that's where I'm going with this, right? So, like, the current hotness, like, the bubble is, well, let's put AI in everything. And everybody is coming out with announcements of this version is AI enhanced! And, I can, I can legitimately, though, think of a couple of places in Thunderbird where better AI support sort of makes sense.

And I, I'm sure you can think of them, too, right? So, like, spam filtering is one of them. And then the other one that really comes to mind is automatic translation of incoming and outgoing emails. Are those things that are on the radar?

Ryan: Absolutely. Yeah. I get, I've, I still get pushback from, Folks around LLMs and small language models and AI in general.

There's a, there's, there's some corners of the internet where, including in the open source community, where they just don't want to hear anything about it. But the truth is, is if you, if you take a step back, and you just look at the technology for its own sake, don't think about the hype, don't think about the just what is, what is, Is the current state of it and what does it enable that I know I'm already getting comments

Jonathan: Please don't add a I clippy to Thunderbird.

Ryan: Yeah But in our testing for instance, we you know, we've been playing around since before the chat GPT we were playing around with with these AI models

Jonathan: and

Ryan: And seeing what they could do and they they fall flat on their face Yeah, it's it's true, but they also Can be very helpful. Some of the, some of the, even the smaller language models now do great translation.

And you sit, you sit there with a native speaker and you say, was that, is that right? And they're like, yeah, that's, that's perfect. And, and additionally, you know, the one I have spent some time on is trying to take your inbox, that day's contents, and trying to pull out the useful nuggets right at a glance.

And we, we see it just getting better and better so that when you're starting your day you can just potentially see a digest and decide from there what, okay, what is it, what is the thing that, you know, instead of, if you're me, you get a hundred emails before you even sit down at the computer. Yep. And so, that's a, a lot to Even if you are good and you can spend three to six seconds per email reviewing it, it's still 600 seconds.

That's a long time. It's ten minutes. And that's if you're fast and you don't get caught on one. That's

Jonathan: really fast. Yeah, that's not, that's not a practical, you cannot go through your entire inbox that fast. Yeah, so it

Ryan: is useful to have LLMs kind of comb through that and make sure that you're putting your, attention at the, at the things that have the highest priority right back.

They can't do it alone. It's a user experience challenge.

Jonathan: Yeah.

Ryan: You can't just say, throw an LLM in and it will fix everything. You have to build around it to provide an experience that does the right thing and does it consistently good. But I think that for us as an open source community, this is an opportunity for us to say, We don't want everyone who uses AI to have to use.

Google and open AI and whatever. Let's find ways to roll it into our, our technology, but in, but using models, you doing it either on device or whatever it is that makes it so that there's some ownership and some privacy around its use.

Jonathan: I will, I will echo though, what mashed potato says, please don't ever prompt me.

I see you're writing an email. Would you like some help? No, I don't want help writing my email. I would like spell checking. Maybe a little bit of grammar checking, but other than that, no, I don't want AISlop getting sent instead of the words that I actually wrote. I think that is the sort of thing that I think people, when they hear, we're adding AI to Thunderbird, which you didn't say those exact words, I know, but, you know, when they hear something like that, that's the sort of thing that a lot of us go, no, I don't want that, I don't want the LLM to rewrite the email that I wrote.

Ryan: Yeah, I, what I think is, what I think we should be doing, everyone, The folks in the chat is, is come help us figure out the best way to roll this out. Because, you know, I do think that there are ways that using these models can improve people's lives. Can help them churn through their email faster, you know.

Ultimately, I think it's a bad situation. When I talk to folks who use, who are heavy email users who say, I only really get about an hour of work done a day because I spend the rest of the time in my email client. Yeah. And I think, gosh, that's not good. What, what I see from this technology is an opportunity to help sift.

Continually, like we've already done this before with the models that I'm talking about the machine learning stuff for spam detection and things like that. This just allows us to go one level deeper and say, can we further sift the inbox so that you see the most important things first and you can deal with them first.

And then, you know, the other stuff kind of, you can get to that after, after you're done with the most important things and after you've.

Jonathan: Does Thunderbird have like an extension system? Some of these things sound like, you know, they are great tools for some users and then other users aren't going to want to have anything to do with them.

And so it seems like it would be a great candidate for here's the extension store, go look at our AI stuff, or don't if you don't want to.

Ryan: Yeah, so even with our current experimentation, it's been, it's not We haven't published a repo yet. We will, but it's been done as an extension because we know a lot of people won't want anything to do with it.

And so we're not going to shove anything down people's throats. And frankly, right now, the only way I would roll it out to everyone is if it, if I felt like it could be done privately on their device and most devices aren't capable of. Handling this yet.

Jonathan: Yeah.

Ryan: And so there are, there are devices that have these NPUs shipping, you know, with in the, in the machine, they can't handle some of the, some of the impressive models, but they, but it's not the time yet for, for that to be done and rolled out across all users.

And I'm not even sure we ever will. But the, but yeah, this is a perfect area for, for, for that An extension to fill the gaps, whether it's an extension from us or an extension from community members and developers who, you know, want to develop for Thunderbird.

Jonathan: Yeah, so something I really want to make sure and ask you about before I let you go is Thunderbird on Android.

I, I have recently gotten that and it's, it's kind of funny because I've used K9 mail. For the longest time, and then, you know, I got the notification that it was essentially getting rebranded. Man, we've been, we've been at this for, you know, quite a while already, so we don't have a whole lot of time. But I do want to, I do want to touch on this.

So what, like, what's the story there from, from your perspective? What what led to this, where we end up with, with K 9 male becoming Thunderbird on Android?

Ryan: Yeah, so it started at FOSDEM. I sat down with Like many good things do. Yes, I sat down with K9 developer, Kedi, and we talked through what the future of K9 looked like.

Because for me, that was, I was using Thunderbird on desktop, I was using K9 on my Android phone. And as he described the state, Of canine, it sounded a lot like old Thunderbird, you know, there wasn't very, there weren't many resources to develop it, you know, and, and it was kind of moving slower as a result.

And there were a lot of things that he wanted to see done for the qual to improve the quality of the application and, uh, but it was going to take forever to do. And so I invite, I just said, Hey, you should, this should just become a part of the Thunderbird project and we can accelerate its development.

And ultimately he agreed and the Thunderbird council agreed and we, we brought the project over. We adopted the project. Some people call it an acquisition. It was not an acquisition. There was, there was no payout. Sorry for Keddy and other folks who contributed to the project. But, ultimately, we we got it and we, so rebranding is a bit of a misnomer.

There will be a K9 for the foreseeable future.

Jonathan: Okay.

Ryan: But there will also be a Thunderbird and they share the same code base. But there are some features and some, some things that we're just not gonna turn on in K9. Not because We don't need the users to have it, but because we don't think it matches kind of the K9 experience.

And but if users tell us otherwise, we'll turn it on. But, yeah, the, in Thunderbird for Android, we really want to create kind of this seamless desktop mobile, you know, experience where you can jump from one to the other and back. And it feels like one cohesive story. And so that's where we're going with that.

And and I love K9. And I love Thunderbird on Android. It's a great experience and it's improving rapidly every day.

Jonathan: Yeah, I'm, I'm looking forward to, you know, some of those, some of those advantages, fixes and new things getting baked into it and rolled out. I'm sort of eagerly looking forward to that.

I think it's a neat potential crossover. Is anybody talking about Thunderbird for our iOS brethren?

Ryan: Yes. I think that we'll see at least an alpha this year, if not a beta release. I won't go as far as to commit to a full, fully fledged release. Only because K9, we had a, the great thing about K9 was we had a project.

Jonathan: Yeah. We had code. That was in, that was in reasonably good shape.

Ryan: Yes, exactly. The story on iOS. is pretty grim as far as open source email clients go. Yeah. And so we're going to have to develop a lot of that from scratch. And that's going to take time. And for, you know, a couple years ago I said, I think we're going to start on iOS.

And we didn't. Because the amount of just upfront investment to do that, you know, would have took away from some of the things we were doing on, on desktop and on Android where we really did. And we still do need to spend time and money and effort to do it. But, I think that it's been long enough, and we do need to give folks with iOS devices a way to have a open source email client.

And so, it's time, and I think we will get in front of people this year.

Jonathan: Yeah. Now, iOS is interesting in that, I think this is true of the email client as well, you don't get to write, you don't get to use Gecko, essentially. Everything web based on iOS is actually Safari, just with a different skin on.

And so that's, that's sort of a an interesting challenge there, isn't it? That you, you sort of have to, you're almost packaging Safari for looking at least HTML based email.

Ryan: Yeah, that's, it's really,

it's, it sucks. But the but what we've been trying to do with Thunderbird, and this is true of every platform we're on, we're trying to make it the best experience for that platform. And when we have conversations about, like, for instance, where we've rolled in. It's, this hasn't hit the ESR release yet.

But we've begun using native notifications on each platform. And that's a lot of work, but ultimately it was this idea that we should, when you use Thunderbird on Linux, it should feel like a Linux application. And when you use it on Windows, it should feel like a Windows application. And it shouldn't feel like some weird creature from another planet.

Right. Like, it should feel at home alongside your other applications. And so, As much as stuff like that frustrates me with iOS, our goal is to provide the best open source email experience on the platform. And so we're gonna, we're gonna Use each part of that platform, integrate as best we can, and make sure that users have a good, good time with it.

So, I can be frustrated, but we're gonna make it work, and people are gonna be happy.

Jonathan: Yeah. I am just, I'm looking at my Thunderbird install on my phone, and I realize I'm running the Thunderbird beta, which is not updated since December, which is not that long ago. Should I be installing the Well, you should check

Ryan: for updates.

You should check for updates. It might have an update now. Yeah,

Jonathan: it may. Let's see. Oh, okay. So I, I asked the, a few weeks ago when we had the other Mozilla guys here talking about Firefox, I asked them this on iOS, we are seeing, at least in Europe, they are required to support third party stores.

And I was asking is, is there a future where we see Firefox with the Gecko engine on the third party iOS stores? And so I'll ask you the same thing about Thunderbird. Do you, do you eventually see, and this probably is dependent upon what Do you see a Thunderbird internal fork, I guess it would probably be for iOS just for those third party stores?

Ryan: It's very possible. We've been, I've been, I've at least been following that really closely. And I would like to see on, on both the platforms, on Android and on iOS, Better competitors, I guess, to the existing app distribution, you know, Play Store or iOS or the App Store. And we're gonna pay really close attention to that, and if there, as, if that usage really grows, I think we're gonna continue to put more and more effort into building what we want to build for those platforms versus what we may be required to in order to, you know.

Exist on the, you know, App Store to use iOS as the example. And so, yeah, there's a lot of, I see that as a wonderful change and something that we're going to lean more and more into as, as it evolves. Hopefully it continues to evolve and, and folks actually use these other stores.

Jonathan: I would like to see it come to devices in the United States instead of just in Europe.

Ryan: Yep, yep, me too,

Jonathan: me too. We'll see, we'll see, hopefully. All right, man, we have covered a lot and we are basically out of time. I could definitely, we could definitely keep going. I do want to know though, is there anything that we didn't get to, anything just burning that you want to let folks know about?

Ryan: Sure, I just want to share kind of a high level view of Thunderbird here to close this out. If you're not using Thunderbird, friends,

Jonathan: you

Ryan: should, you should really consider it. It's a really powerful tool for managing your email, your calendar, and your contacts. And not only that, but by using it, you support open source software in this space, in a space that really needs.

Changes. We don't want everyone using proprietary Gmail webmail for email. That's just going to result in more and more stuff sitting behind the Gmail gate. It is a gate, because right now there are features that Gmail does that we can't tap into. Because it's, it's not in the API and it's, it's not something that we can even interact with.

If we don't If there are not more and more people using these clients, more, more is going to move that direction. And so I just encourage folks, you know I guess vote with your, with your software use. And use this and, and, and Google and other providers will know. Additionally, keep your eyes open on what we do in the future.

Because we're really focused on impacting this area. And that's all I'll say today, but I think we're going to offer some great stuff, products to folks that, well, at least you have choice.

Jonathan: Yeah. There, there is quite a there's quite a useful use case in, you know, even if somebody is using Gmail and likes the Gmail interface.

Getting a pop feed of it and throwing it into a local Thunderbird and opening it from time to time, just to have an offline copy of your emails. Like it's useful for that by

Ryan: itself. Yeah, I remember the, and then we can close. One story I, I saw was folks, there was a folk, a per, a guy, there were a few of these like back to back in the news, who erroneously his account got.

Suspended, and he shared that he couldn't access any of his email. And he, he had like a lot of important stuff in there that he was trying to reference. And he didn't realize how much of an inconvenience that would be to just suddenly have Google block your access.

Jonathan: Yes.

Ryan: And I, when I was reading that story I thought, if he was a Thunderbird user, you know, he would have had a, even if he was using IMAP, like, he would have a local copy.

On his machine that he could refer back to and you know, do you own your data? Do you own your communication on the internet and The answer is I just use Webmail, then the answer is no, you don't.

Jonathan: Indeed. Indeed. Alright, this has been great. We'll genuinely have to have you back in, say, six months or so to talk about how these things have shaken out.

Before I let you go, I am just about contractually required to ask you two final questions. And that is, what's your favorite text editor and scripting language?

Ryan: My favorite scripting language is probably Okay. Hold up. There's two different, there's two questions in there. What do I use? And what do, what do I wish I could use?

Yeah. Or feel more empowered to use? I use Python as my scripting language. I love Ruby. I have a deep love for Ruby and, and used to use it a lot more but, you know, really don't have the same opportunities as when Rails was taking over the world and so it's Python, I guess, but my first love is Ruby.

Jonathan: Makes sense. Makes sense. And then my text

Ryan: editor is sadly, I have moved over to Visual Studio. Code I, I have to say I was a big Atom user, and ultimately moved over. If anybody has any recommendations You know, I'm open to it, but I'm pretty happy with Visual Studio Code, so.

Jonathan: Yeah, the guys behind Atom are working on another editor.

I think it's going to be, yeah, I think it's going to be open source, too. I think so. I was talking with them about doing an interview. I need to go back. I don't think we actually did it. I need to go back and touch base with them again, see if we can actually make that happen. Yeah, I know some people are really excited about that one, so.

But, you know, VS Code, it's open source. Now, you might want to run VS Code EM instead of VS Code if you, if you. Come from a certain, you know, open source purist sort of mindset. But lots of, lots of people like it. Lots of people like VS Code and it's actually really, it's

Ryan: pretty neat. It's the, for me, it's the extensions.

Yeah. All the different extensions, you know, there's just moments when I need something and I go out there and I get what I need. I'm, I'm always surprised at just how much is out there.

Jonathan: Yep. All right. Thank you so much, sir. I appreciate you being here. It was a delightful conversation.

Ryan: Yeah, thank you so much.

I appreciate it.

Jonathan: Alright. So that was, that was Thunderbird. And that was Ryan Sipes. And had just a great time talking about it. And I, man, I'm excited even more now for the future of Thunderbird. Get, I'm, I've already installed the newer version on my phone. I think I was stuck on an old beta.

So maybe some of my little, little quirks and annoyances will go away. I'll get to get to the new stuff. But yeah, I am super excited for the future of Thunderbird as well. Also excited for the future of the show because next week it looks like we're gonna have the guys from CIQ, and that's the, that's the company behind Rocky, the Linux distro.

Looking forward to talking to them. And then after that, recorded on, February the 25th, and should go live on the 26th, we're talking with Simon Schocken about NAND to Tetris. And that is a really neat project. It's a, it's a university course, but it's also an open source project about taking the building blocks of literally transistors with the NAND circuit and putting them together and, you know, building, building precept upon precept, as you might say, to get all the way to a computer that can play Tetris.

And it's a really, really cool course. I'm, I'm very excited about talking with him about that. If you want to find my work, of course, over at Hackaday, we've got the security column that goes live every Friday, you can check that out. We've got the Untitled Linux Show over at Twit, and you are welcome there as well.

Appreciate everybody being here and we will see you next week on Floss Weekly.

Episode 819 transcript

5 februari 2025 | min

FLOSS-819

Jonathan: Hey folks, this week we have a conversation with Key Jeffries, one of the minds behind Session, an open source encrypted messaging app that has to do with the blockchain and Tor and all kinds of other interesting stuff. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 819.

It's time for Floss Weekly, that's the show about free, libre, and open source software. I'm your host, of course, Jonathan Bennett, and I am sort of flying solo today. And those of you watching live or getting this on the immediate rerun will realize that FlossWeekly. com. Today is Thursday. It's not Tuesday.

What's up with that? Well, we're doing something a little bit different. Our guest today, Key Jeffries, by the way, is from the land down under, and it is actually Friday morning for him, and it would have been in the middle of the night at our normal recording time. So we we made an allowance. And we're doing an off, off the regular rotation show.

That's why it's just me. And I've got a conference coming up mid February. So this is gonna work out very well. We're gonna get an extra recording in here. And then things will be off a little bit. Then when, when I'm out of town, it'll all work. And so the, the, the universe will be right again. But Key works on a, a really interesting little project.

I don't think it's a little project anymore. But the project Session, which is, this sounds like buzzword bingo, but I think there's some actual cool stuff in here. It is encrypted messaging on the blockchain, which I know some of you out there want blockchain. Ah this, this might be interesting.

I'm hoping this is going to be really interesting. I'm not going to waste any more time telling you what I think it's about. I've got the man here and I'm going to ask him, but, but first off key. Welcome. Welcome to the show. And it is it is lovely to have you here.

Kee: Yeah, lovely to be here and thanks for accommodating for the australian time zone So it's always tricky when we're trying to set up meetings with the u.

s. So It's appreciated.

Jonathan: Yeah, we've had one other guest from australia and when I told him what time it was he's like, oh Well, it's not the first time that i've done it

Kee: Yeah, yeah, there's definitely been some early mornings and late nights here, especially when the time zones Both drift out and you're even further apart.

That's probably the worst time for like, I think there's three or four months there where it's really hard to make meetings with the US people.

Jonathan: Yep, I understand. So, let's talk Session. Like, right, that's your big claim to fame. What is we say the 30, 000 foot view. What's the problem that Session is trying to solve?

What do I do with it? Why would I install the Session app on my phone?

Kee: Yeah, so Session is really focused on Being different from other private and encrypted messengers, mainly from a metadata perspective. So if you use Signal or WhatsApp, you're typically signing up with your phone number, for example, to use those services.

And that creates a lot of metadata linkage between your accounts and all of the messages that you send. So session doesn't require a phone number to sign up. So that's one really big thing that we do there. It also uses onion routing to hide people's IP addresses when they're connecting to the decentralized set of nodes that session uses.

And it doesn't have a single central server. Like a lot of the models out there. So if you're using signal or WhatsApp, they have a centralized server, which stores your messages for a period of time in session, it's actually a decentralized network of nodes, which stores those messages. And I will just like clarify on the, like what, like when you came out in the show notes, you said like session is messaging on the blockchain.

It's like, it does use a blockchain for. The rewards layer, like kind of the incentivization layer, but the messages themselves don't actually get stored on the blockchain itself. Cause that would kind of create some like inefficiencies storing every message forever. And it would cost transaction fees to send messages.

So yeah, we did think about that approach there and it is an approach that some other applications use, but we think it has like some kind of downsides to, to, to, to doing it that way.

Jonathan: I have been contemplating a very lightweight messaging protocol myself for a project that I'm in for the longest time.

And it makes sense to me to store things on the blockchain. But when I say that, I mean the blockchain in the same way that Git is a blockchain, not a cryptocurrency blockchain. And so I, you know, there's, there's a lot of different, different directions that a project could go with that. So I find it real fascinating.

That, that you guys, you guys went that way. Now let's talk about some history. It's pretty, yeah, go ahead.

Kee: So just to, just to jump off that, like it's, it's pretty interesting to use something like Git for version control of messages, especially like if you want to maintain sequencing, like you want to be always sure that, you know, once one message is sent, then the next message is sent, it's like chained in hashes so that you can always verify and go back.

I think, like, with Session, one of the things that, like, has been difficult for us because we're building a decentralized network is maintaining, like, synchronicity between the nodes that are storing your messages can be really tricky. So if you were to have really, yeah, if you had to have really strong guarantees about like, what message comes first and the sequences there, there's potential that, like, You know, what would happen if one message came in first and then the next message came in like they were sent in sequence, but then they arrived out of sequence, like, you know, how, how long do you wait to kind of sort that problem out?

So we don't, we don't do like strong sequencing guarantees yet on session because of that decentralized structure, but it is something that we think about often. And I like the idea of kind of using git to kind of version control your. Your message is in the messaging system. Well,

Jonathan: I've, I've said for the longest time, this is not original with me.

I think the person that originally said it was trolling and I'm a little less trolly about it, but you know, there is already the killer app for blockchain and that is Git. Git is the killer app for blockchain. And with all of these other things that we do with it, we're trying to find other things that are interesting to use it for, but Git was the first blockchain.

It just wasn't monetized.

Kee: Yeah, that's true. Like, I think you know, there's a lot of hype, obviously, about blockchain and every time the Bitcoin price goes up, like, people come back and I think a lot of people have a very initial, like, negative reaction to anything that's like blockchain think, oh, it's a scam or everything that's not Bitcoin is a scam.

And I understand that sentiment because like, you know, probably 80 percent of the projects, 90 percent of the projects out there that have cryptocurrency or blockchain or AI in their title, you know, are just kind of using the title for, for hype. But then I think there are like a few kinds of projects which are legitimately using some of the very interesting things about blockchain.

Things that I think is the most interesting about blockchain is its ability to incentivize large, like, public infrastructure networks. So, Bitcoin is the biggest like, decentralized computer in the world. Like, it doesn't do very interesting compute. It's just running SHA 256 over and over and over again, right?

Right. But it is the biggest. Like physical compute network in the entire world. And the only way that that's been able to be created is through that decentralized rewards mechanism that they've built, which is based on blockchain. And I think like, kind of what we're doing at Session is taking some of that decentralized reward and consensus model and applying it to a problem, like building out a decentralized network of nodes, which can store messages for users.

So very similar to. to how Tor works. You have this network of community run nodes which participate in routing and storing data on the network. I mean, Tor doesn't really do storage. They more just do routing, but they don't have an incentive model behind that. So, you know, there's no rewards provided for Tor nodes.

You just provide them. Because you're an altruistic individual. There's also no barrier to running Tor nodes as well. Like you can go and spin up a thousand Tor nodes and join the network and you'll start seeing a portion of users traffic. So by having crypto currency. There's

Jonathan: some three letter agencies that are very fond of doing that, I think.

Kee: Yeah, exactly. Exactly. So by like introducing a kind of barrier to entry. There, like you cut down on some of the Sybil attack problems and you can also provide a reward for people who are acting honestly on the network. So like, that's where we come at cryptocurrency like from, it's more from an incentives and you know, building out physical infrastructure network side of things rather than, you know, meme coins and, you know, let's hype it up type thing.

Jonathan: Sure. I want to, I want to dive into the history of the project. Now, look, looking at it, I think some of the source code originally came from, was it from Signal that some of the, some of the source code came from? And I'm curious about that. Like how, how did, how did Session get started?

Kee: Yeah. So the project itself started in 2018.

I think I was going to university in 2016 for computer science and I had kind of gotten a year and a half through my degree and I think in one of our lectures they mentioned like asymmetric encryption or we were kind of doing our cryptography units and they were talking about asymmetric encryption and public private key, key pairs and stuff and my lecturer interestingly like just mentioned offhandedly.

like, Oh, Bitcoin is an interesting, like practical application of asymmetric encryption. I was like, Oh, okay. Let's like, go and look into this. It's like right into the white paper. And I started, like, I really dived down that rabbit hole. I think it has most people too. And they first find about out about, like, especially if you're kind of libertarian minded and like very interested in like uncensorable technology.

Like when you first hear about Bitcoin, it's like, Whoa, like there's this thing that I don't need to ask. Permission for so like I kind of dove down that rabbit hole.

Jonathan: I have distinct memories of thinking to myself, man, I wish I had a couple of hundred dollars so I could buy a couple of these Bitcoins.

Kee: Yeah. I think I remember the only way to, for me, cause I think when I found out about Bitcoin for the first time, I was under 18. So I didn't have a credit card or anything. I think the way that you, you were able to buy it or the way that I tried to buy it was I don't know if you remember Mount Gox.

Yeah, so like Mt. Cross is one of the earlier like Bitcoin exchanges that went bust and you know Took a took down a bunch of people's bitcoins with it. Yes They're still in proceedings to get those back. By the way, I think they may be paying them back some of them back now So the way that you could do it was that you could use the second life currency So you would buy the second life currency, which I think was Sylvan was it?

I don't

Jonathan: remember. I know what you're talking about. Yeah. Yeah,

Kee: you would buy You could buy the second life currency with paypal. So I did that I bought the second life currency then you could transfer the second life currency to Mt. Gox And then you could trade the second life currency for bitcoin on Mt.

Gox and then you would withdraw it From Mt. Gox and I tried to do this whole process And they like banned my account as I was trying to withdraw all the bitcoins So I ended up with a bunch of this second life currency, which It wasn't that useful to me because I didn't play second life

Jonathan: And unfortunately unlike bitcoin that did not that your second life currency didn't make you a millionaire

Kee: No, no, it didn't appreciate quite in quite the same way as bitcoin did anyway, like coming back to the story.

So I was like you know, kind of dove down the Bitcoin rabbit hole and started going to meetups in Melbourne. And that's where I met all of the other co founders of the project. And these were all mostly like people, like guys that were very interested in crypto. We were all super interested in the privacy space as well.

We had all kind of been privacy maximalists and we saw this kind of interesting way that we could combine elements of say the Tor network and elements of. Like private cryptocurrencies like Monero and like join them together so you could have kind of these incentivized kind of backbones to networks and applications that you could build on top of those networks.

So if you could incentivize a set of nodes to operate and you could have an underlying private currency underneath those, then you could have very interesting properties or you could build very interesting applications on top of that network. And in the original white paper that we wrote in 2018 we, we called it Loki messenger at the time, but that's what session is now was one of those applications that we had proposed to build on top of this private network.

So we kind of spent 2018 to 2020 building out the network and the backbone. of session. And then we were kind of like, okay, you know, let's build a test application here in the, in the white paper. We said we were going to build a session, which, you know, it's, it's a messaging application. So as soon as you can kind of send messages, messages back and forth, that's generalizable to a bunch of different other like protocols that you might want to build on the network.

So we set out with the idea of. Building this is like a proof of concept and that's when we forked Signal because we thought, okay, Signal's got a really good code base, like they've done a lot of the work for us, we're basically like just replacing the back end and that's all we need to do. So we did that and I think like people immediately were like, this is like, this is amazing, like there's no other network.

out there like this, which is kind of like Signal, but it's on a decentralized back end. He's on his onion routing. So like, that was very interesting to people. And there was a lot of demand to like push more development effort into like that application and focus less on like just building out infrastructure.

Cause I think a lot of like these blockchain projects that you see, like they've built these massive networks, but no one actually uses them. Like. They built a lot of supply, but not a lot of demand for the actual network. So we decided like, okay, let's actually just work on an application and let's like build that up.

And that's what we've kind of done from 2021 to, to about now. And just, I think last year surpassed 1 million monthly active users on session. So yeah, so it's big milestone for like a decentralized application, privacy preserving application.

Jonathan: So, so your, your, your demo program became the killer app.

Yeah,

Kee: essentially, yeah, we didn't, I guess there's a, it's like a happy coincidence, you know, like we didn't plan for it to be this way. Like we kind of thought originally that we were going to focus more on building the backbone of the network and other people would build applications on top of it. But I think it ended up being like, sometimes it's just best to, to work on the apps yourself.

Jonathan: Session is the messaging application. What's, what's the term for that underlying network?

Kee: Yeah. So it's called the service node network right now. It'll be renamed to the session node network soon because kind of had this, like the naming as well, like with a lot of these things comes along with like how the strategies changed over time.

So the service node network was. To describe this like layer of nodes that would provide service to other applications But at this point like it's kind of developed into like session is really the main thing that it it serves right now So changing the name

Jonathan: of that that leads naturally on to the next question Are there any other applications other than session that run on it at this point?

Kee: There is one other application which is called LukiNet And that's an open source decentralized onion router. So kind of competitive with like Tor or ITP. It's more similar to ITP but it's kind of still in development and we have kind of put it in a backseat a little bit to focus more on session.

Although like Lokenet is still going to be used quite heavily. To replace the onion routing layer that we use in session right now. But yeah, that also runs on that decentralized network and uses the nodes in that network. So there's around 2200 nodes in the network right now. So it's similar to TOR in that you, you know, kind of choose three nodes in that network and then you create a route through those nodes to hide your IP address.

Jonathan: Interesting. And how does, how does crypto, there's so many directions I want to go with this, I want to ask you about, how does the cryptocurrency part tie into this? So it's not, it sounds like it's, it's in some ways not a base part of the of the network or the application, but yet it's in there some, somewhere.

What's the tie in?

Kee: Yeah, so, like, I think that was important for us, like like when we were building Session was that, you know, if you downloaded an installed Session that you wouldn't need to like provide some sort of cryptocurrency or like make a payment or pay fees in a cryptocurrency because the experience for most users when they use a messaging application is that they just download it and they start talking to their friends if you like.

create this big barrier of entry of like you need to buy a cryptocurrency now like to get involved in the network then it's like you're never going to be able to be competitive with like the whatsapps or the signals of the world which are free to use so that was never our kind of goal the way that the network works right now is that that backend, those 2200 nodes that you use to store and route like onion route messages on the network.

Those are incentivized by a cryptocurrency and they also have to stake a cryptocurrency to become a node in the network. So it's really for the staking aspect of things and then for the rewards that are provided to nodes as well. So if you run a node in the network and you route users data and you store users encrypted messages, then you receive like some amount of tokens per month.

For doing so, and you also have to provide an amount of tokens per month locked up so that if you perform poorly on the network, we can kind of remove you and, and punish you. All of that system is decentralized. It's like a self policing system that they use to like check on each other. But yeah, it gives us some interesting properties which make us a little bit different from some of the other decentralized networks that exist.

Jonathan: That that is interesting Are there any are there any scenarios now? I'm i'm I have my open source hat off and i've put my you know, Amateur economist hat on are there any scenarios where users get to pay? So like are is there? This is going to sound mean, I don't mean it to be mean. Is there any real money that's being put into the system?

Or is it all the cryptocurrency money?

Kee: Yeah, yeah, yeah. I mean, that's a natural, that's a very natural question. It's like, you're kind of, you're paying out rewards to nodes as they operate. But you can't just keep doing that forever, otherwise it wouldn't be sustainable. It's like There has to be some sort of income, like income for the network, right?

Or like money coming in. So the idea there, and it's not implemented yet is that there'll be a premium version of session. So we're calling that session pro. So most users will use the free version of the application. So it's like 97, 98 percent of the users will use the free version. And then session pro will be a subscription fee.

That's like, you know, 4. 99 a month, and that subscription fee essentially is paid in a fixed, like, US dollar price, for example. That is then converted into session tokens, and those session tokens go back to the network to provide the rewards for the network. So, right now, that isn't implemented right now, but that is the future plan, that there's essentially Income from the users that are using the protocol.

Like, you know, using the messaging application That flows back to reward the the the service nodes in the end

Jonathan: I can I can imagine. Almost like a what discord has done with nitro Sort of so with nitro you get to unlock the ability to send bigger files and bigger messages And you could imagine your pro version would have something like that, which makes sense because you're talking about sending more data, and I think it would be pretty reasonable to ask people to pay to be able to send the bigger data types.

Kee: Yeah, that's exactly the thought as well. Telegram also has Telegram premium and that's, you know, larger files, larger groups more like profile customization and cosmetic like features for the user. That's the kind of stuff that we're thinking about. Like it's, it's not you get more privacy or less privacy by paying.

That's not what we're thinking. It's more like,

you know, if

you are a power user of the application, then you may want to send larger files or have larger groups. And this is a way to be able to. To do that and allow people who who want to support like the continued sustainability of the network to do so

Jonathan: Do do the session tokens just sort of exist on their own or is it its own network?

Or do they exist on like on top of the monero blockchain? I know a lot of a lot of different cryptocurrencies sort of tie themselves to that.

Kee: Yeah, so originally like the coin, the network in its current operation uses a coin called Oxen, which is a fork of Monero. So that's like the private cryptocurrency underlying the network.

But We're in, in the process of transitioning that coin to be on top of Arbitrum. So that's been a big transition for us. And it's going to change from Oxen to Session Token when that transition happens. So there'll be a swap for people who have Oxen already, and then there'll be new Session Token holders on, on Arbitrum.

So that's been a big process for us to move blockchains. Cause it's kind of like swapping the engine from a car while it's still going down the road at a hundred k's an hour.

Jonathan: Yeah, no, that's, that's challenging. And, and so, you know, I can, I can imagine a scenario where someone runs one of these nodes that generates the tokens for them, and then they can just turn right back around and buy themselves pro with it, right?

Like you can, you can you can sort of dog food it that way, which is also interesting.

Kee: Yeah, definitely. I think a lot of the people who do run like service nodes or session nodes and the network are session users as well. So there's a bit of natural feedback. Yeah.

Jonathan: How many, how many session nodes does the session company run?

Kee: So with the foundation has a mandate that's written into its constitution that it can never run more than 10 percent of the network. So, I think it runs around 170 nodes right now, and there's around 2200 nodes. So that's less than 10 percent of the network. I think it's like 7 or 8 percent or something like that.

Jonathan: Is there baked into it an idea of knowing where these nodes are geographically? It seems like On one hand, it would be useful to be able to route to the closest one. But on the other hand, you're giving away a bit, well, that metadata is what you're talking about, to be able to even do that sort of routing.

That must be a challenge.

Kee: Yeah, so we don't use any type of routing algorithm. I mean, the algorithm is basically you choose random nodes from the set of nodes you just, You know, download this set of 200, 2, 200 nodes, and then you just randomly choose nodes from that list to, to form your route through the network.

There is a lot of thinking about how to most efficiently use the network. So Tor does like what is called bandwidth based routing. So if you, if your node is able to serve more bandwidth to users, then it gets used more frequently and uses paths. That has some advantages in terms of that. You're kind of fully exploiting the entire power of the network because you're routing based on bandwidth, but you're also giving attackers that have more bandwidth in the network, more of the chance to be in the user's path.

So. And if they're kind of the start point and the end point of a user's path, then the anonymity is removed. So we never, we didn't really want to go down that approach. Latency based routing is also a thing as well. Like, you know, if you're in Australia, like you might only want to choose nodes that are in Australia.

So you only have to do 20 milliseconds for the first one. And then the next hop is 20 milliseconds, you know, so on and so on. But you do also at the same time want to have. Nodes from diverse geographic locations in your path, so you're kind of doing a bit of, like, jurisdictional arbitrage, right? Like, you're kind of crossing different barriers, like, and hopefully you end up in some, like, non Five Eyes country at some point.

So that, you know, someone can't fully collect all of the information there as well. So it does always end up being a bit of a balance between kind of performance and privacy. I think what we have right now is a pretty good approach because everyone in the system is fairly equal and the distribution of nodes is, is not too bad either.

Like I think it, it does tend to be like centralized around Europe and the U S these countries that have kind of, The best in internet infrastructure. But that's more kind of, cause operators tend to optimize on price to a certain extent. So yeah.

Jonathan: Is there, is there built into the, the network or the session app the ability to send session tokens?

So does it, does it sort of accidentally work as a payment application too?

Kee: Not, not yet but that probably is something that we will look into in the future, so. Yeah, I think like the idea of a, of a, of a super app, like has been quite popular and I think we're trying to like, we're seeing people kind of go in that direction as well.

It's like you see it a lot in China, for example, with WeChat, like people use WeChat for everything. Like, it's not just a messaging application. It's like, The main, it's their everything. Yeah. People use for payments, like they order their food through it. There is a bit of danger there as well, but like from a convenience perspective, like, and if you are providing privacy, it does make sense I think, to, to push more functionality into messaging applications just because those are the things that we use kind of a lot day to day.

Jonathan: Yeah. So mining in, in the. In the Session world, is mining routing data, or is it being on the network with a stake?

Kee: Yeah, so there's, there's no kind of mining in a traditional sense, I suppose you'd say it's more akin to a proof of stake network, so similar to Ethereum you know, because Ethereum has transitioned from being a proof of work currency into a proof of stake currency.

So yeah, you earn rewards on the network by meeting a minimum level of like basically requirements. So you need to have a certain level of bandwidth. You need to store users messages in this warm and you need to participate in the onion routing network as well. And then basically like the nodes and network will check each other periodically.

And then they have a system of, of voting on each other as well. So yeah, every couple of rounds, they'll assess a certain number of nodes. And then they'll vote on whether those nodes were, you know, acting honestly for that period of time. And if they are, then they, they're able to, you know, earn a reward which is paid in session tokens.

Jonathan: Yeah, I have, I have been trying to crack this nut myself for the longest time, because obviously mining, if it's just shaw hashing, is ridiculous. You know, we're, we're, we're just burning coal, right? It's not actually doing anything. Proof of stake is neat and it's better, but I've always thought that the ideal solution would be if the thing you were mining was also, in and of itself, something useful.

And if I can ever crack this nut and figure out the solution to that, I will start my own cryptocurrency and I'll become the next, you know, next Tech Bro billionaire. I've not found it yet. It's not a, there's not an obvious solution to that problem.

Kee: Yeah, I think there was some kind of projects working on protein folding, like, for, you know, if you were able to generate a solution for some sort of protein folding problem, then you would be able to earn some cryptocurrency or awards for doing so.

But, I think there's it's hard to scale kind of real world problems to the crypto space, which I think is why it's been difficult. It is pretty easy to take a hashing algorithm and you know, you just have to produce a certain number of zeros in it and then like there's your block reward. So

Jonathan: I was, I was honestly, I was excited for helium for the longest time.

I thought maybe they had cracked the nut, but obviously that is. Not taken off and gone where they were hoping it would so I don't know maybe yeah, maybe i'll be the one to do it

Kee: I haven't looked into Helium for a while. I know that they were building the the LoRa, LoRaWAN network for a while, and I think they've kind of transitioned into more of the 4G or the 5G world, but I haven't looked into how that's kind of going.

Jonathan: It is it is not as big as it once was, I'll put it that way. It's not as hot as it once was. Turns out not enough people were willing to pay money to be to track things.

Kee: Yeah, that, that, that always seems to be the problem. You build these big especially like these, these class projects are usually called like D Pin projects.

You could, they, they have this kind of problem of The incentives to build the supply side of the network are really good at the start and they build these massive networks of like hundreds of thousands of nodes, but then eventually you need the demand side for the network. You need people actually using it to make the economics of paying out these nodes sustainable, and if you aren't able to develop that, then the project is kind of doomed long term.

Jonathan: You sort of just described what a bubble is. Right. And I've, I was told just on this show, in fact, when we had one of the older co hosts and we were talking about bubbles in regards to AI, of course, and I made my analogy that I've made multiple times that, you know, what. com was a bubble, the bubble finally burst, but we still have the internet and it still changed the world.

And I think that is probably true of, I mean, it's obviously, it's true of AI, the world will never be the same, but at the same time, the bubble will burst. It's obviously true of cryptocurrency as well. Right? Like there are, there are things about cryptocurrency that. In some way we'll change the way the world works.

But we're not in the middle of the same crazy bubble that we're, where he pointed out to me, the personal computers were the bubble once upon a time. And the mainframe guys, IBM was going, Oh no, personal computers will never take off. And it was a bubble. It did eventually pop and there was a crash in the market, but here, you know, here we are with computers on all of our desks, not to mention it on our inner pockets and everywhere else.

So, and I, I kind of sell that because One of the things I've been trying to figure out, particularly with cryptocurrency, is what parts of it are going to survive, now that we're kind of on to the next big thing. What are the actual use cases that are going to continue to survive? What parts of cryptocurrency are going to change the world?

Session is interesting in that, I don't know if it is going to be the big one or not, but You know, it sounds like you guys are getting closer to a sort of a economics model that makes sense

Kee: Yeah, yeah, I think Like if if I think long term about what what projects are gonna kind of survive and thrive In the next like, you know, five years or ten years because like I do think like long term It's not you know, we've been running since 2018, which is a lot longer than A lot of the projects, right?

How to fit right in. So we do think on those longer timescales, but I think the models which are going to be successful are models where there is some sort of level of income for the protocol or some sort of level of, you know, natural demand for. the token outside of just speculation, like, you know, if it's all speculation, then, you know, when, when the bubble pops, it really does pop very hard.

And there is no, there is no kind of long term future for the project. So I think like those are going to be the protocols that succeed, like everything that's kind of in the top 10 coins right now is probably, you know, Ethereum and Bitcoin are probably going to be fine. But, you know, like the, the more, the, the projects lower down on the list, I think the ones that are going to thrive are the ones that actually have real users that are willing to financially, like you know, buy, buy something in the protocol and continue that income for the nodes that run the protocol itself.

Jonathan: Yeah. So talking about session. How does somebody get on it? What are the options for joining the session network?

Kee: Yeah, so we have applications like published on all platforms. So Ios, android, mac, linux, windows In terms of like, probably have five different Linux, like packages, we've got an app image, we've got a deb, we've got a snap package, I think we have a, you know, you basically just go on the GitHub on our releases tab, you can see all the releases that we have there.

It's cross platform as well, so if you have an account on, you have a session account on Android, you can link that to your desktop account and all of the messages will show up and you can Message back and forth that way. So yeah, it's available basically everywhere. Probably won't find something that's not supported.

Jonathan: Interesting. I am just out of curiosity. I'm looking at F Droid, which is one of the open source. It's for those that don't know, it is an open source sort of app store for Android. And it's, it's very useful for people that don't want to run the Google services on their phone. But F Droid has this policy that it only will run fully open source software.

And other, other policies, like that's not the only thing that'll keep you off of F Droid, but that's one of the things that can keep you off of F Droid. And there is a it says it's an unofficial rebrand of Session. Without the firebase push service probably, probably the official stuff will violate their, their, some of their network access rules, something like that.

Kee: Yeah, I, I give you, I give you a little bit of background on that. Sure. So like we have our own F Droid repository, which you can add. to FDROID because you can add, you know, other repositories to FDROID. So that's just fdroid. getsession. org if you want to add that repository and then you'll get the official session.

The reason that we can't be in the FDROID store is because of the like Google push notifications. So that's closed source, not on our side, on Google's side, but we do want to be able to provide push notifications via Google because Different Android phones have very interesting like battery management limitations.

So if you're trying to do push through non Google services, you could, your app can basically like be shut down in the background and then you're not providing You know, reliable notifications, and that's very important for a messaging application to provide reliable notifications. So yeah, it's not the perfect situation to be in right now.

I think we have some longer term plans to get around those things. But yeah, it's it's where we are right now. You can also download the APK as well. That's on the GitHub as well. If you don't want to use FDroid or build it yourself, if you want as well. What,

Jonathan: what is the what does a developer base look like?

How many, how many folks from the you know, your community jump in and I'm sure you have some that just are, I call them drive by they'll just, they'll drive by and they'll give you a single patch, like this is the one thing that drives me nuts, here's the patch, please pull it and then, you know, every once in a while you get somebody that gets real excited about it and jumps in and does a lot of work long term.

What, what does that look like for you guys? How many, how many contributors do you have?

Kee: We probably have a few more kind of drive by contributors right now than like actual people that work on, on session day to day. So like, you know, I'd say we probably get, the other thing is like across like different repositories, like you don't see that much open source activity on our iOS repository, for example, because that tends to attract, it has a bit of a higher barrier to entry and I think iOS people, people who work on iOS don't tend to be Like super open source, like people, there's a bit of a kind of disconnect there, whereas like say Android and desktop, like desktop is written in like JavaScript and TypeScript.

So the barrier is like way lower to enter there. So most people understand how, how to write JavaScript or TypeScript, and then there does tend to be like a bit of a. More crossover between the open source world and people who are using desktop applications. So, we probably get a bit more open source contribution through there.

The other interesting thing is, like, we have this on Session, we have, like, larger communities. So if you're familiar with the idea of like a discord server, it's like very similar to that. So you can run a server in session. This is like separate from the staking component and stuff. You can run a server in session, which a session clients can connect to, and then you can have these really large communities of.

You know, thousands of members that co that that code is also open source, like that's called the session open group server or SOGS. And there's a lot of community contribution to that because a lot of people like to, you know, run their own communities and want to develop their own little tweaks and, you know, how, how long things are stored for.

And like we're kind of on the, we're working on a plugin system for that as well. So people can kind of develop their own bots and stuff around that.

Jonathan: So I want to ask this before I forget it, because this has entered my mind and left it a couple of times now. Session is sort of built on optional anonymity?

Right? So like, you don't have to, you don't have to give your phone number. You can just create it, you create a new you know, cryptographic identity, and you can connect. But I assume that's optional. Like, so are there any ways built into the system to be able to verify your identity? Right? Can I, can I get a blue check?

Going back to the old days of Twitter. Can I get a blue check inside of Session to prove that I am who I say I am?

Kee: There is no blue checks right now. Like, so the system is optional in the, like the identity system is anonymity optional in the sense that when you create a session account, you get this long, you know, alpha numeric.

Identity on the network, which is called your account ID or your session ID. And that is what you give to other people for them to message you. So you can go and post that like on your Twitter or like on a forum or like, you know, you can send it to your friends. And in that sense, like then you're connecting that.

you know, public key or your account ID to your person through another means, or you can just say like, I'm only going to share this session ID, this account ID when I'm in person with someone and they can like scan a QR code or like I can write it down on a piece of paper, although it's very long and that they can then use that.

So it's kind of like a choose your own adventure in that sense, but. Like native to the platform, we don't have any like blue check marks or anything because it's, it's not really a social media network in the same way, like the, the way that, you know, we see people using session is like more that they would use it with their friends and family and they can then like give their identity to the people that they want to use it with rather than, you know, it's a public network where you, if someone is claiming something, you want to be sure that they are who they say they are.

Jonathan: Did, were you Did you, did you pay attention to, were you a part of Keybase at all? I don't remember who got me on that, but Keybase was this, it used to be, before it got acquired, of course it was this service where you could generate a cryptographic identity, and then you would post proofs to, you know, your Reddit, your Twitter, your GitHub.

You know, all these different places and it was a way to cryptographically prove that I am me and that is my account there and that is my account there. It seems like that, something like that could be, could be really interesting inside Session for someone that, you know, wanted to be public, was not looking for anonymity.

Kee: Yeah, and I think that's, like, I am aware of Keybase but yeah, after it got ported, I think, like, not too many people use it anymore. But yeah, like, the same thing is The same thing is possible like for session as well. Like if you're publishing your, I mean, your, your account ID or your session ID is your public key.

There is no mapping or anything that occurs between those two things. It's not like a, a username where I need to look up someone's username and then the server tells me what their public key is. The account ID or the session ID is the public key. So if you're posting that. In different places, like it's the same as posting your public key and saying, you know, this is mine.

You know, your service could be hacked, you know, if, if, if you like, you know, your Twitter could be hacked and you put out a fake session ID, that's possible. Like if you required a signature. On when you're posted your public key that would be maybe a little bit stronger, but then makes the message longer so you kind of have to You know do you kind of have to make it easy for people?

So this is kind of what we've we've come up with so far.

Jonathan: So is there I had it and then I lost it Goodness, I usually have a co host that I can I can bounce to when this happens And I don't have one at the moment oh I remember so We've danced around this a little bit, talking about Five Eyes and, and has Session gotten official notice from law enforcement or other government agencies at this point?

Kee: Yeah. Like I, when you're running a messaging operation, like there's a lot of things that. You have to consider and the good thing about session is that it's built in a very decentralized way from the start. So as I was saying before, there isn't a central server that, you know, the session foundation has access to, which has everyone's messages.

Even if those messages are enter and encrypted. For example, in Signal's case, Signal still has a central server where everyone's encrypted messages are stored and users sign up for the service with their phone number. So, and they connect to that, you know, central server with their IP address, usually if they're not using like a VPN or something.

So even though Signal doesn't get the contents of your messages, they end up seeing your IP address, potentially your phone number when you sign up to the service. And like the frequency of when you're sending messages as well. For Session, it's like completely different because we don't run like a central server in the network.

It's a decentralized network of 2200 nodes. And when you use Session, you essentially like are, you're assigned to a swarm, which is a collection of 7 of those nodes. That's where your messages are stored, and that's where, like, senders will send messages to you. They send it to that swarm of notes. And that's a random selection of notes.

So, if someone comes to us and asks us, you know, give us all of the data for a particular session ID or account ID, we can honestly say to them, we do not have that data. Not, not only do we not have the encrypted data, we don't even have access to the data at all. So that puts us in a very different position from other messaging applications from that perspective.

And I think that has helped us a lot. Obviously, like, you know, there's been requests from law enforcement agencies and stuff like that for information on particular users. But our standard answer is always, you know, here's our limitations and what we can do. The protocol has been built, you know, such that we don't have access.

to the network in that way. So, you know, we're being fully compliant, like with you, we just don't actually have access to the data.

Jonathan: Does Session have a warrant canary posted somewhere?

Kee: Yeah, I think we do on the, I think on the I think on the STF website, I think we do.

Jonathan: For those that, for those that don't know, warrant canary is sort of a legal hack.

And the theory is that a government can force you to do a lot of things, but it can't force you to put out a statement. And there's also some cryptographic keys that get added to that to make it difficult to fake it. And essentially it's a statement that says we have not been served by any national security letters.

We have not, we've not answered any warrants. And a company will put it or an open source project will put it out usually once a year. And so, you know, if they get on that once a year timetable and then A year goes by and no warrant canary gets posted. Well, everybody sort of knows that it's time to move on to a different service because something went wrong there.

Kee: We, we, we also have transparency reports as well. So if you go up, like I think cause we've just been doing this transition right now from the OPTF to the SDF. So it's basically like moving the operation of the company out of Australia to Switzerland. So that's been a. that we've been undergoing.

But if you go to the OPTF website, they publish a report every quarter about the like law enforcement requests that we've gotten and how we've responded to those reports as well. So it's yeah, very transparent on, on that front.

Jonathan: Yeah. And I guess it helps that, well, so all of the code is open source, right?

There's no, there's no closed source bits that live anywhere.

Kee: No, apart from like what I was talking about before, like the Google push, like stuff. You know, to handle like interaction with the essentialized things, but yeah, like Android, iOS Desktop, all of those applications are fully open source and then the full routing server.

Yeah, right So so like the server code is open source, too

Jonathan: so even if you got a in the united states with we have a Incredibly terrible thing called national security letter where essentially the government can say because of terrorism We're going to force you to do something. And One of the things that apparently that they can force you to do is to add malicious code to an application, right?

The fact that you guys are fully open source and nothing runs well, I guess 10%, 10 percent of the network can run through your servers. Like that's not even particularly a danger there because if you were to add something malicious to the code and push it out, somebody would find it. Enough people, I hope, are looking through your code base that somebody would find it right away.

Kee: Yeah, that, that, that's a big like thing to talk about with like session as well and like a big differentiator. It's like a lot of these projects are open source, so like Signal for example is fully open source, like including the server. But there has been times where the signal code has, like, that's been running on the server has deviated from the signal code that's been published on GitHub.

That happened for a period of time, like six to eight months there, and then they push, you know, updated code up. So I'm not, like, claiming that there's, like, some sort of vulnerability or something. Maybe they just, like, you know, didn't want to update their code, like, for that period of time. But with session, like the, the way that like code is deployed to the network, because there are these, you know, 2200 nodes, we can't push an update on those 2200 nodes, they actually have to pull it down.

So, you know, when there's an up update, people check that code that's running or the code that's published on GitHub, and then they pull in the release. So it's not something that we can do to them. They actually have to agree and there's a consensus protocol. And that's, it's kind of like very similar to the way that upgrades in Bitcoin work, right?

Like there can be these huge, like social. Like problems in the network like if things are being pushed on operators that don't want to run them So, you know in bitcoin you had this really big hard fork between like bitcoin cash and bitcoin people over the block size And that was eventually like it's a code.

It's something that has to be put into the code, like the block size limit. So some percentage of the nodes in Bitcoin decided that they're going to run one software. And then some decided they're going to run another software and that caused the fork in the network. So it's a similar thing with session.

If we tried to push some sort of backdoor on people the nodes in the network, like, cause we only control 10%, you need a consensus of the network. The nodes will just tell us to like. Basically and create their own like little network and then we would be segmented off on the like backdoored network if you would call it that.

So it's a, it's a completely different model from how a lot of the centralized companies operate when they're pushing software updates.

Jonathan: Yeah that, that's. Yeah, that is, that is super interesting and definitely the way to go about doing that. So what, what language is all of this written in? I, and I'm sure there's a smattering of different languages, but like, if someone wants to get involved, what, what languages are useful to know?

Kee: So on the desktop side, it's a React application. It's like a Electron application written with React in JavaScript and TypeScript. On Android, it's mainly Kotlin. You know, we've kind of moved away from java quite a bit on that side. I think everybody's Yeah, yeah, I mean like we still got a lot of skeletons in the closet though Because like we are a fork of signal and at that point like signal had a lot of java code So like we're in the process of kind of rewriting everything like with the ui layer being in Jetpack Compose.

So like we're kind of moving a lot of our XML screens right now to Jetpack and doing that process. And then on iOS it's Swift. Nearly all of the Objective C is gone on iOS and we're kind of transitioning a lot of our screens to be in Swift UI now as well. Yeah, that's a kind of breakdown. And then on the like server, like side of things that's pretty much all C so like the storage server the coin side of things is all C And then like some of the elements, like where we really want community interaction, like the community server that's written in Python, so.

There's a bit of a spattering of all of the

Jonathan: languages. When are you planning to rewrite that C code in Rust? I,

Kee: not, not soon, I don't think. Yeah, I mean, I think we have a lot of fairly opinionated C developers in the in the codebase right now. So, yeah.

Jonathan: I could definitely something that we've thought about.

Yeah. Oh, I'm sure I am a fan. I'm a fan of rust. I like it but boy, it's it's both annoying and hilarious the constant you should have written that rust

Kee: Yeah, yeah. Yeah. Yeah,

I think like right right now. We've kind of developed a team That's pretty good at writing c as well. Like obviously it's a dangerous language but if you have like people who are Like veterans like it writing it like it's a bit easier.

You don't tend to run into some of the same issues, so It yeah, it's but it's an interesting discussion as well

Jonathan: It helps to have something like a decent ci suite where you're doing some fuzzing and some Static analysis and I hope hopefully there's a few of those things that get run over at least the c code

Kee: Yeah, yeah, we have a we have a CI, suite which runs like aux and core every time that there's a A new commit to a PR or a new PR that comes in.

Jonathan: That that that helps that helps weed out some of those mental oopsies that we all make from time to time so is signal still upstream at all Have there been any, any fixes since the fork that Signal has made that you guys have pulled?

Kee: We have probably deviated so significantly from Signal code at this point that, like, it's almost two different projects.

So I mean, like, I think we forked Signal in 2020 and we haven't really pulled much from upstream for four or five years, so. At this point, we would be so significantly deviated that like, if you tried to merge them back together, it would be complete

Jonathan: hellishness. Right, right. I just, the only thing that really comes to mind with that is, you know, if Signal, because your, so much of your stuff was based on Signal originally if Signal found a big vulnerability, is it possible that that would be in your code as well?

Kee: Well possibly. It would depend where the vulnerability was. Like, if it was A cryptographic vulnerability or it was a vulnerability and like the version of electron or like, you know, like the, the way that messages are stored, like a lot of those things have kind of changed over time. So there's significant difference now.

But we do like these keep track of like, what is going on in signal and like, If there's like major things, but like, I think the last thing that happened to them was like the Twilio SMS verification attack that they had, where essentially like some attacker hacked into their SMS provider and was able to like send messages to signal accounts to kind of.

Take over those accounts via their phone numbers. That's not something that exists in session because we don't have phone numbers So it wasn't a concern

Jonathan: for us, right? And so what What was the what was really the the the thing that sparked? Wanting to be privacy first. Like, I'm curious where, where your mind, where you guys mindset was that privacy first was really the, the thing that you, you were so interested in this. And then I guess I'm also curious, because I'll get questions about this.

Do you, do you regret that? In fact, I, I can guarantee you that at least somebody, at least one, well, there'll be at least one person out there that will ask themselves or maybe ask me, How do they sleep at night? Knowing that they, you know, enable whatever.

Kee: Yeah, I think, like, we were all kind of very interested in the privacy space from, like, using Tor and, like, what that kind of enabled human rights activists and journalists to be able to do all around the world.

Like, we kind of started the project around the same time that WikiLeaks was kind of coming up and putting out this information about how the US government was, you know, spying on citizens and Edward Snowden and those revelations that came out. So I think for us, like It was kind of in that same vein that that session took a lot of those kind of political ideas and, and put those into software and, and put those into a tool that people could actually use.

Mm-hmm . So I mean, like the, the question about like, you know, do I sleep well at night and like, you know, am I worried about people using the, the service in malicious ways? My approach to this is like fairly utilitarian, so like. I, I, I talk to session users every day and like, I have a pretty good, you know, we don't, we don't see in depth what people are messaging about, obviously, otherwise that would compromise the way the system works.

But I talk to enough session users that I know that, you know, the vast majority, 99. 99 percent of session users are using session just because they want more privacy in their life, or they want to talk to their friends and their family without feeling like there's someone like watching over their shoulder.

And that they're limited in what they can say. So, you know, if you're taking a utilitarian approach to this, like you're kind of trying to weigh off the, or weigh out the privacy benefit that all of those users get against like whatever the malicious use is. And I think that equation is so heavily imbalanced in, you know, providing privacy for those users that, you know, if there is some malicious.

malicious use, that's totally outweighed by, you know, the, the, the service that you're providing to most users. So yeah, that's, that's kind of my perspective on it.

Jonathan: Yeah. Yeah. I, I asked that just because, like I said, I know, I know there will be people out there that have that same question. That's why I'm acting as an audience proxy in that.

Can, can people access session from places? I don't know what the big three are anymore, but say, let's just say Russia, China and Saudi Arabia, those sorts of places. What does it look, as I know Tor has had to do a lot of work over the years to try to be accessible in China particularly. What does that, what does that equation look like for Session?

Kee: Yeah, this is a really tricky one. So like we're available. So we're banned in the Chinese App Store so you can't download Session in the Chinese App Store you can get Session, like, if you get the APK, or, like, you've changed your region to, like, outside of China, you can download the application.

And I believe, as of my last reports, like, It session is still working well in China sessions fully banned in Russia like blocked in Russia from a perspective of they have actually blocked the decentralized network. So that's also what happened to Tor as well. So essentially what they do is like, cause there's this network of 2200 nodes, which you need to connect to, to be able to send a message.

They basically just like get the IP addresses of all of those nodes, put them into like, Either their national firewall, they go and tell their ISPs like you need to block these IP addresses and then no one can connect to the decentralized network anymore. There are ways around that and that's kind of like, you know, Tor has this like bridging system and they use these like domain fronting as well to be able to distribute like these lists of bridges to people who are in censored regions.

I'd say like we're not quite there yet because that's like quite a resource like intensive operation to run. Like those censorship evasion activities, but we do have some really good ideas there, and I think it's important for us to be available in censored regions, like, one of the biggest growth events for Session was in Iran during the Masa Amani protests, and The session was growing at like 50, 000 users a day, like when those protests were happening because the Iranian government had blocked signups to Signal and WhatsApp and these other messaging applications actually via the phone number services that those applications require to sign up.

So when you sign up Signal, you need to receive a text message with your verification code. And If they, if you can't receive the verification message, then you can't sign up for Signal. So they were actually blocking them through those methods, and because Session doesn't use a phone number, people are able to sign up to Session really easily.

That went on for like two, two, two weeks or thereabouts, and then the Iranian government blocked like, access to the decentralized Session network. So. Those are the kind of, like, attacks that we're kind of facing from nation states. But it does, like, prove that, like, Session is being used in this really positive way as well.

Jonathan: Yeah. What's, what's your, what does the funding of Session look like now? I know, I know you've got ideas for what to do in the future. But like, have you guys taken VC funding to be able to get to where you're at now?

Kee: Yeah. So like most of the funding has come from 2018. So that's like, we sold tokens in 2018 to be able to fund the operation.

Yeah. In 2018. And we've raised some funding recently as well. Like with the transition to the session token that's going on as well. So it's mainly been like funding from people who would traditionally invest in coins basically.

Jonathan: Yeah, I, I have, I have for the longest time thought that, and it's unfortunate because there have been so many scams that have happened, so many pump and dumps.

But on the other hand, the idea of an ICO where you don't have a venture capital fund, you just have a whole bunch of, of the little people that think you have a good enough idea to invest in it, like, I, I think that is, like, genuinely one of the coolest potentials of of the entire cryptocurrency ecosystem.

System that we've got and at the same time, unfortunately, it's been one of the biggest downfalls because so many of those have been Little more than scams over the years and so much money has been lost in them I think I think that's neat though that you guys made that work and are Continuing to do the thing, you know, we're what seven years later six or seven years later that you started out doing

Kee: Yeah, yeah, I think like there aren't too many projects that started in, in 2018 in that kind of ICO bubble that are still going right.

I think like the, the tricky thing has always been like the balance between like, I don't want to say protecting retail investors, cause I think that's the wrong way to look at it, but you don't want to have like these ICO booms where retail investors are getting very misled, like into what they're actually investing in.

And that I think is what we actually saw in 2018. It's like, there was a lot of projects who you know, basically like maybe not even writing white paper, but just like. You're writing something on Twitter and then well, this is a repeat of is happening of this as well with the meme coin craze that's going on right now.

And then like retail is just like jumping into that understanding. Yeah, yeah

Jonathan: Bitcoin stocks are up. Bye. Bye. Bye.

Kee: Yeah, essentially essentially like people don't really understand what they're Investing in and that's a dangerous like thing. So we kind of, we kind of balanced that off. So like, you know, it's, oxygen is like tradable on all of the public cryptocurrency exchanges and same session token.

So people can come in and buy, but like, at least they know that, you know, what they're buying has a liquid market behind it and that they can resell it, you know, certain prices if they want. And then like, we did take, like, I'll be clear with you as well. Like we did take some VC money as well, like in, in, in the ICO too.

So it's like, You know, a combination of all sources, I think it's probably the best balance.

Jonathan: Yeah, what I've seen with open source projects trying to turn into a business, the thing that's just poison is when you take enough VC money that a venture capital group. Has control, they have access to the steering wheel, right?

Because then, you know, a year goes by and you haven't made a billion dollars yet. And all, well, it's time to change the license. Let's get, you know, this and that. And it never ends well.

Kee: Yeah, it's I'd heard so many horror stories about that. Like, you know, when, when we were starting the company about like, you know, VCs that were overly controlling.

And who would kind of try and steer the ship in like the most profitable direction. But. not necessarily what is best for the users and that like the misalignment of those things would cause both no profits and no users, right? Because you were kind of compromising on both fronts. I think like actually the, the crypto VCs are quite different from traditional VCs in that most of them are very hands.

Like hands off, like in terms of, I, I don't know if that's, it's different because they have their liquidity straight away. Cause usually like you're going to sell tokens and then those tokens will be given to the VC and then they can sell on a liquid market straight away if they don't like the project.

Whereas like with a more traditional startup, you're giving them equity, but there has to be a. Some sort of exit event, either the company gets sold or at IPOs, like there isn't as much liquidity. So if the investor doesn't like the, you know, how things are running at the company, then they don't really have any exit opportunity.

Like they basically need to like try and get the company to IPO as soon as possible, or. Find some other private investor that's willing to buy their that equity. And so it's much different from kind of a crypto market where from day one of the launch, like the investors become liquid or maybe there's a lockup or something, but yeah, it functions quite a bit differently.

And I think that. creates a bit, a bit of a different dynamic between the project and the, and the investor.

Jonathan: Do you, do you foresee governments getting more involved with ICOs and trying to regulate them more?

Kee: Well, I mean, I, if you asked me six months ago, I would have said yes. But since like Trump's election, I think that's obviously changed a lot in the crypto space.

And the U S still is. Although it's like been diminished from a crypto perspective just because of the way that the SEC has operated over the last, like say 10 years, like in a very punitive way against crypto projects. It has diminished. It's like the U S is kind of standing in the space, but I think with, with Trump, like launching his own, literally his own coin

Jonathan: Two of them, two, two Trump meme coins now.

Kee: Yes. Yeah. Yeah. Like, you know, with the, with those coins launching, I think I, I can't see how Trump would be, you know, appoint someone who was going to be super aggressive against crypto. And I think also crypto has shown like the crypto crypto industry has shown how. Powerful they are in terms of lobbying efforts.

I think they spent crypto industry. I think we're the largest donors to Political campaigns in the last cycle. So it's you know that money talks obviously like in in politics, you know, you never really want it to be that way, but I think like It's it's going to be potentially a regulatory shift in the U.

S. And I think we may see that follow on to other countries as well.

Jonathan: I, you talked about having a libertarian bent. I, I too have something of a libertarian bent in a lot of ways. And so I am very sympathetic to the idea of let's not regulate this thing to death. The only thing I would say with that is I would like to see in cases of outright fraud.

I would like to see the the departments of justices from around the world to go after people that are intentionally being fraudulent And that seems like maybe that's one of the pieces that was missing from some of the icos over the years

Kee: it it doesn't really make sense because You know when when governments were kind of talking about passing all of these new regulations to regulate Crypto or kind of putting them under the idea of like being a security it's like just enforce the existing laws that you have like if a project has claimed that it's going to You know pump to a million dollars and it doesn't pump to a million dollars That's literally fraud like can't you just prosecute them for Like being fraudulent because that's literally the definition of like claiming something and then it doesn't Regular like you've said it's guaranteed regular

Jonathan: old fashioned fraud.

You don't need a you don't need a law for cryptocurrency fraud. It's it's just fraud

Kee: Yeah, it's like it's it seems like I think that's always been a problem, like governments, like they're built to pass laws, so like they want to pass as many laws as they can. They don't want to, you know, if they can create some weird variant of a law that sounds good on paper and they can sign a bill, you know, they're incentivized to do that rather than just enforce.

Existing laws.

Jonathan: Yes. Yes. So true. All right So let's say somebody wants to get involved and wants to get a server up and running on the Well, it's not really quite on the blockchain on the network I believe would be the way to say that what does it take to be able to run a server? Do I have to have an unfettered ipv4 address?

Can I run it behind a NAT? How much server hardware does somebody need? What does that part look like?

Kee: Yeah, so we recommend not running behind NAT You do need to have a publicly routable IP address. That's just so people can actually like send messages to you because you're routing like public messages.

In terms of like the hardware requirements, the requirements are fairly low actually. Like a lot of these blockchain networks you need like hundreds of gigabytes of RAM and like very powerful CPUs. Because we're dealing with messaging like users messages, they tend to be pretty small, like packets of data.

And Another thing that Session does as well is that it deletes messages off the nodes every 14 days. So you're not storing every message forever. You're only storing the last 14 days of messages for users. So that really radically constrains the amount of storage that nodes need to have. So we recommend around 30 to 40 gigabytes of storage.

Although we're not actually using that much yet. Three to four gigabytes of storage that I think are being used right now. Actually, I think it's actually smaller than that. The, the, the other component there is not every node stores every message on the network, right? There's these swarms. So you're only ever storing in the current network one, 330th of the network messaging capacity, and you're deleting that over 14 days.

So it really constrains the amount of messages that you have to deal with. You have to be online. So like you, you can't go like online and offline a lot. So most people run nodes like as a VPS, like in a data center. It's the most common thing. And I think we recommend four gigabytes of RAM and two virtual cores.

So it's something that you could realistically run on a Raspberry Pi. As long as you have a publicly routable address and you're not going online and offline. You know, every time your router gets reset or something.

Jonathan: Is it IPv4 only? Is your IPv6 support yet? Just IPv4 right now. Okay. That, that is. And so I've, I've had conversations with both my ISP.

And my data center, both saying, Hey, I would love to be able to get some IPv6 addresses, because there's some cool stuff you could do with IPv6. And I've been told multiple times over the years, No, we don't have IPv6 yet. I finally have an ISP now that's giving me IPv6 addresses. And I am about to move my servers to a different co location host that comes with IPv6 by default.

So I'm, I'm about to get on the IPv6 train everywhere. And I'm kind of excited for that. IPv6 is interesting. IPv6 is interesting for the the, the firewall circumvention stuff too, just because it's, the, the space is so huge and you can so trivially get so many IPv6 addresses. So that's and I understand, I've been a part of a project that, that got, in fact, we got sent an IPv6 patch and it's like, this is so big, we don't have the resources to even.

Look at it at the moment. It was not great. But let me put that as a bug in your ear IPv6 is interesting for what you're doing

Kee: Yeah, yeah, I think we've looked into it in the past, but i'll I haven't gotten updated in a while So I should look into it again. Yeah,

Jonathan: it's it's definitely interesting definitely the future, right?

I've got i've got a buddy that tells me i'm not doing ipv6. I'm gonna wait for ipv7 or Maybe he's, he may be savvy enough, he tells me he's waiting for IPv8, because that's the next one. How many,

Kee: how many addresses does he need? Like, once the human population has expanded to the entire galaxy or something, you know, ran out of addresses, IPv6 addresses then, but it's like some massive space, right?

Like, yeah, more than the atoms in the universe or something crazy. It's a

Jonathan: ridiculously high number. Yes. Yeah. All right. So, Boy, I think we've been going for about an hour. Let me ask you this. Is there anything that you wanted to let folks know about that you wanted to cover that I didn't get to, that I didn't ask about?

Kee: No, I think actually we, we covered things pretty widely on this. Like, obviously, like, the, the audience here is, like Definitely from an open source perspective. So I'd encourage you to like go and have a look at our GitHub repositories and download the app and just test it out. Like everything's open source and viewable.

Jonathan: yeah. What, what license is it primarily under?

Kee: I think most of the code is GPL three. I think there's a few packages, which maybe MIT think the core code base, although we're moving that kind of now is B is it BDS three? I forget what that license is. BSD probably. BSC, that's right, BSC 3,

Jonathan: yeah.

Alright and so I've got to ask, this is sort of a troll question are you adding AI elements to the network or to the application at this point?

Kee: No, no, no AI plans. I think the only way that it would really make sense is maybe concession as a messaging application, like maybe we'd put some sort of chat bot in there or something, like maybe running off some sort of open source.

Like a framework, but there's no plans right now, like, it's not really something that I focus on at all.

Jonathan: That's, that's good. That's good. That's the right answer. Yeah. Alright so I've got two questions that I've got to end with, I'm required to, it's part of the tradition of the show, and that is, you personally, what is your favorite Text editor and scripting language.

Kee: I'm definitely a nano, I'm definitely a nano user. Which is very simple and I'm, I'm usually not doing anything too complex which requires like Vim or something like that. Sorry, what was the second question? Scripting language. Oh, scripting language. Yeah. I don't know, it's like I end up, I end up writing a lot of scripts in Python, to be honest.

I'm not super familiar with Bash. But yeah, probably Python for scripting is what I end up using a lot.

Jonathan: That's a

Kee: very popular answer. We get that a

Jonathan: lot. Lots of people like Python.

Kee: Nano

Jonathan: and Python, or? Not as much nano, although we do get that combination. But Python quite a bit. Well, Python's very popular.

A lot of people like it. What about you? I tend to use, if I'm on the command line, I use nano. Not on the command line for actually doing real coding. It tends to be VS Code these days. And for, and for scripting language I've actually gotten, gotten hooked on Amber. Which is, it compiles down to bash code.

It's like, it's a dude that said, there's these three things that just kill me about bash. And everything else is great. So he wrote his own little, you know, add on to bash. That compiles down to bash code that fixes those little annoyances that he had with it. And I kind of like it. So I'd say that might be my favorite scripting language at this time.

Kee: I would say like if I'm doing something beyond editing a text file, I would also use VS Code as well. Like if I'm writing code, I would use VS Code. But like if I'm just seeing what's in a text file or editing one line, then I'm probably just going to use Nano. Yep.

Jonathan: Yeah, makes sense. All right. Very cool.

Hey key. I appreciate you being here. Thanks for coming on a weird time and and making it do with the time zones and putting up with just me and not a co host. I could kick things off to but thank you man for being here very much.

Kee: Yeah. And thanks for having me. I hope to be back at some point.

Jonathan: Yeah, we'll have to do that.

Come back in six months or a year and talk about what's changed to be fun. All right. That was key. Jeffrey is talking about session which You know, you may have thought when we first started out. Oh, no, it's cryptocurrency. We're back It sounds like they've got some interesting things that they're doing and whether it makes sense for you I will of course leave that up to you You can find me if you want to on Hackaday, of course That's where my security column goes live every Friday.

And as soon as they get done with this recording I'm going right back to working on that I've also got the Untitled Linux Show over at Twit, twit. tv. You can find that as well. And I appreciate everybody that's been here, both live and on the download. And we will see you next time on Floss Weekly.

Episode 818 transcript

29 januari 2025 | min

FLOSS-818

Jonathan: Hey folks, this week, Doc and Jeff joined me and we talk about DeepSeek. That is the new AI out of China. That's really making waves and a couple of other open source things. It's a lot of fun. You don't want to miss it. This is Floss Weekly episode 818 recorded Tuesday, January 28th. I don't care about the Roman empire.

It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today, well, we had a little bit of a scramble. We had a guest. scheduled who had to reschedule. And then we had another guest that started to schedule and then realized that it's about 4 a.

m. For him right now in Australia. And so that's not going to work either. We may have a special recording session later in the week to get that one. I'm pretty excited about that. So I sent an email out, a hurried email out last night that said, ah, help guys, there's news we can talk about news. And so we had a couple of brave souls that answered the call.

We've got Mr. Jeff Massey, and then. Very good friend of the show, Mr. Doc Searles. Welcome to both of you. Hey guys. Hey, really glad to be here.

Doc: Yeah, this is great. It's great to be back.

Jonathan: Yeah. I'm, I'm particularly excited to have Doc back. It seems like it's been too long since we've had you back. Every time I would, I would text you or send you an email and ask like, Hey, do you think you could make it for a show?

And Doc would constantly tell me, no, it's, it's traveling season. It's convention time. I'm out. It's always something. It's always something. Yeah. So. We stay busy. This, this is kind of an interesting opportunity. What, what conventions, what, what, what has been keeping you busy, doc? Is there anything particularly interesting for our show that they would love to hear about?

Doc: There's, there's a lot. They're probably the, the biggest thing that just takes a lot of prep is the twice yearly. Internet, Internet Identity Workshop. And we're coming up on our, I don't know, our 30, our 40th. We've had them two a year since 2005. There's always about 300 people. It's a lot of stuff's come out of there.

Um, OAuth and just all kinds of things having to do with identity, but lots of other topics. And that's the kind of a self sustaining thing that's at the Computer History Museum in Mountain View, California. But we also organize some events here at Indiana University. So, where Joyce and I are both visiting scholars, which is Half right, in my case, I am visiting but we like Bloomington enough that we moved here and so we've been very busy building a house in which I'm now in and and this is if the room sounds reverberant, it's because there's no furniture in it other than being a computer and a microphone and not much else.

So yeah, I mean, and, and, and I'm writing a lot. We've actually I've been following, um, the fires in California because Joyce is from Los Angeles and we know, we either know personally or are no more than one or two handshakes away from many dozens of people who've lost their homes. So that's been And I've been writing about that on my blog and especially covering the coverage because I'm still a journalist and I don't get paid, but journalists don't get paid anymore for the most part.

So unless they sell out and I haven't, I haven't been smart enough to do that in a while. So that's been fun.

Jonathan: So are you guys no longer doing the two houses in two different states sort of deal?

Doc: Oh no. It's like we have two houses in Indiana, one in California, and we still have. We partly pay for our old apartment in New York, which my son, our son occupies.

And he's getting ready to go to law school next year, so somewhere. It'll probably be in New York, but if not, somewhere else. We'll see how that goes, and we may lose that.

Jonathan: So you simplified things. Now you only have four houses in three states.

Doc: We only have four houses in three states. That's right. Which sounds so much fancier than it really is.

It would be great if one of them was like an estate or something, but Nope, they're just four houses.

Jonathan: So out of the out of the identity workshops, the internet identity workshops, is there anything particularly interesting that, that came out of this last one that was, what was the hot topic?

Doc: Actually, I write, so there are two things.

One, one is that I wrote this, I wrote this book called the, an earlier book called the Cloutier Manifesto was very popular and hot. It's actually 25 years since that came out. That was a bestseller. Right. It's been wrong now for 25 years. It's very optimistic about the future. That, that future is one in which we're all trapped now.

It hasn't quite happened, but people love it. The other one was called The Intention Economy, which turned out to be The inspiration, it was a worst seller. It's like number two million on Amazon, but Tim Bergers Lee based a solid project on it. Kawhi, which is this open source personal AI project is based on it.

And And more recently, Consumer Reports, the, the, the organization that rates products and all that kind of stuff has decided that they, or at least their Skunk Works inside of it, but I think it's an important Skunk Works that they want to base the future of the magazine, I think, to some degree on what.

Not just the services they'll do on what I wrote about in that book. So that, that's, and they were there at IW this last time. Taking input and engaging in dialogue and planning things out. And very optimistic about where that's going to go. And another thing, which is this workshop that I have to jump to on the hour, is a new thing called IRA, A Y R A.

And I think it's at ira. forum, A Y R A. Yeah, Aira. Forum, A Y R A, and a trusted relationship network, but basically it's, if it works out and there's open source stuff in it, I don't, I won't know until this workshop exactly what's going on with it, but but it's, it's, it's the new, it's, it, it will do for identity kind of what credit cards did for, money, another, a better way to do it.

So, and I, and as I'm looking for things, I lost the zoom. So there you go. So that, that,

Jonathan: that might not be the best analogy doc I hear. No, it might not do for money. I think, Oh no, it's going to enslave all of us and make everybody poor.

Doc: I mean. If you look at, if you look at what, what D Hock, the guy who invented Visa did, basically got a lot of competing banks to get along and gave us this very convenient thing we can carry around in our wallet.

Now, all the bad things that happened since then are beside the point that it made that a lot easier. So. IRA is a, is a cabal of a lot of companies that none of them are like big household names, but there are some pretty good sized ones and some banks is based in Switzerland. But included in that is going to be IEEE P 7012, which is.

Machine readable personal privacy terms. And this is something that we started in 2013 or 14. The standard is finally being finished. It's, it's finished on an asymptote. I mean, it's never going to quite arrive, I think, but it, but we're, we're really, really, really close. But it will completely, if this works out, it'll completely reverse the way we do we do privacy online.

They agree to our terms rather than we always agree to theirs. We blow up the entire cookie bullshit. We, you go to a website, you go to a service, you have your terms. It's just like creative comments. Where I have this, I have a term, I've chosen it. It's that one there. And you can accept that or not, but if you accept it, we both have a copy of it and we agree and we can adjudicate it later if we have a dispute, but we probably won't and because we're not trying to screw each other and this is friendly and and that's what we've been working on with customer comments for some time.

But I think it's quite simply the most important standard in development today. And I, and I Pretty sure we'll have it done real soon.

Jonathan: So

that's, that's where

we are with that. Yeah. So this is an interesting thought. The idea is that I, I have, I've picked out, so I'm assuming that there's going to be like an onboarding wizard for this.

At some point I pick out, these are the things that I think are appropriate for a site to do. These are things that are not appropriate for a site to do with my. data, our business relationship, whatever. And then I go to a new website and they have done essentially the same thing. And so then there's going to be this algorithm that compares the two and we'll say, well, you're trying to access this site, but they're asking it.

So it's going to be like, it's on a web area, a phone application, right? This site is asking for these additional permissions. Like that's inevitably what's going to happen.

Doc: It's actually not that complicated. It's there's no. There's no centrality to it. You have in your browser or your app, whatever it's going to be, your agent, all these things have been called agents now, by the way but you have an agent and it says, it's, in a way it's probably just going to be just like.

Do Not Track, only it has teeth. And Do Not Track was way the hell ahead of its time. It just was, it was the wrong time to build that sandcastle. You know, before the tsunami. You know. Anyway but I think privacy matters a whole lot more now. Everybody wants it. And the. The GDPR is that it's thing, but it's kind of failing.

And so now it's, it's a good time to have something where, Hey, the user is not just the user. This is an independent human being and it has this terms and, and it's a simple term. Like one of them is, is called no stock music. Just don't track me out of here. Okay. And don't sell my data to anybody. Give my data away to anybody else.

But you and I, you could track me while I'm here. That's one of them. And and the, and, and it's, and they either accept it or they don't. And, and their row, their, their agent on their side says yes or no. And they'll notice that this is you know, term number one on the list of 10 that are on customer commons.

And that's a known thing. And it'll have a, you know, a unique ID for that one. It's just like, just like creative commons has, you know CC by, or whatever, whatever that they're like, like there's six. at Creative Commons. It'll be that simple. You know, it, you, it's a contract. It's, it's not more complicated than that.

Contracts are like the oldest legal ceremony in the world. You know, it's just, you know, I rent a wheelbarrow from you, I give you forty bucks in deposit, I sign something that says I'm gonna bring it back in good shape, you keep, keep the deposit and I bring it back and we've got a deal. It's over. It is contracts are laws that two parties make.

for themselves with each other. And contract law really only applies when there's a dispute. And that'll apply here too. But it doesn't have to be complicated. And we're optimistic that, for example, that WordPress will pick up on this. WordPress runs like 40 or 50 percent of the world's websites and most of them are not based on, on tracking you.

Don't make money on advertising that way. I think this will free up sites and services to make money a better way. You know, than just advertising or get better advertising because you're asking for exactly what you want. I mean, there's just lots of possibilities here that don't start with surveillance.

That's basically it. Like, hey, hey, world, stop starting with surveillance and then let's work this out. That's basically what it is. But we don't even have room. We have room for negotiation, but the assumption is There's not going to be negotiation involved. This is simple. I don't want to be tracked off your place.

Good, you got that? Great. I

Jonathan: can't help but think that the devil in this case, the devil in the details is going to be exceptions. Right? Like there's going to be some valid use case someone has for, I, we need to be able to do tracking. And so then there's going to have to be an extension added to this where it says, this website is asking permission to track you.

And it'll be, you know, an addendum to the contract. Yeah, and then, yeah,

Doc: and that, that will happen. And I think, I mean, in the beginning with this, I mean, All kinds of that stuff is possible. If we start there though, we're going to be exactly where we are now. You know, um, if, if we start with the assumption that privacy is the default, okay, I have privacy here.

It's, it's basically, that's what we have in the regular world, in the everyday world. It's true. You know, we don't walk around wearing name tags and we don't walk around with. with the fuzzy side of Velcro all over ourselves so that, so that, so that the prickly side of Velcro can be stuck to us everywhere we go.

Right. You know, that's, that's not what we do in the everyday world and, and we want to replicate online something that resembles the everyday world, but we're, there are tacit understandings. There's, if you look at two kinds of knowledge, there's explicit knowledge, there's tacit knowledge. Tacit knowledge is You know, I know how to drive a hammer with a, a nail with a hammer, but I can't explain it.

That's tacit knowledge. We have, the world is full of tacit agreements. We have tacit agreements, we're not going to track everybody everywhere they go in the everyday world. We can't have them in the digital world because everything needs to be explicit. So this is a way of making some things explicit.

Like, we're going to start with privacy. We have privacy. What a layer of stuff on to that, we'll work that out, but we'll start here. We'll start with We're all private here and then we can work out other stuff. So and, and our, and our like 10 terms that maybe they're 12 now to worked out by some people who are in our little cabal that are in, in Scotland and Ireland right now.

And and I think they have 10, but they have graduated ones where, okay, some of my data could be used, for example, to train AI. Some of it doesn't have to, you know, you can add that on like, I was going to ask. And we can transition from that to, you know, something else. So,

Jonathan: yeah, that, that that sounds good because there is an AI story that has just exploded upon the news scene in these past couple of days.

Oh, it's huge. And that is it's Deep Seek. Right? It's the new AI out of China from a Chinese company. And I think the reason that it's so huge right now is not necessarily because it's the best AI that's ever been made. I think the reason it's so huge is because it's an AI that was made for like 6.

5 million dollars. As opposed to the billions of dollars that places like OpenAI and all of those are throwing at this particular problem.

Jeff: Well, do we know for sure it really cost that? I mean, they just said that, right? Right. Yeah.

Jonathan: So that's, that is one of the things, right? Like, so we don't know for sure exactly how much was put into this.

This is a claim that is coming from a, you know, a relatively, Unknown corporation out of China. They are, they are claiming that is an open source AI and I've been chatting back and forth with Simon who, you know, had a hand in creating the definition of what it means for an AI to be open source.

And his take on it was it is more open than less, less closed is what he said. It is less closed than Llama from Meta. But, yeah. So far, they have not been able to find the training data. So it does fail that test of what OSI considers an open source AI. Right, so it's not a completely open source AI by that definition, because the training data is not published.

But you can. Git clone it, and run it locally. Lots of people have been doing that, and have had good success with it. I have been playing with that a little bit, last night and today. So far I've not been able to make it work on my AMD video card. Although, there was, there was part of the story was that it didn't necessarily rely on CUDA.

Which is interesting, so maybe it will work on AMD cards, I don't know. But, yeah, the, the big thing is that it came out of nowhere, and apparently Was so cheap to make. And so NVIDIA stock is cratered as a result. I think Meta stock is cratered.

Doc: It bounced down, but I, I'm certain it'll go back. Oh, I'm sure.

Jeff: I mean, yeah, it's just, you know, a lot of semiconductor stocks went down, but the, one of the official statements from. NVIDIA was well, it doesn't matter what the engine is. You still have to have the hardware to run it on cheap cheap or not It's still gonna be a lot of purchase of hardware. That's what they're banking to get the confidence back into the market, right?

Jonathan: So that's it. That's actually an interesting thought though because NVIDIA right now They're making their money by selling their hardware to meta and open AI and Twitter X Because it takes so much horsepower to do the back end work of actually training these things. Well, if we're going to see a swing back where it becomes easier to train them on the front end, And then you, you do your, your, your inference work at the user facing side.

That kind of suggests that NVIDIA is going to start selling more cards to consumers, more consumer level cards, perhaps. Does that, does that check out?

Jeff: Yeah, I mean, it, it really does, I think, which is might even tie into their. Recent 5090 launch where it's kind of geared less to the gamer and more towards the prosumer or home AI Enthusiast where they comes with 32 gigs of RAM in it to help train and you know, there's a lot of AI Specialized chips in there to just help do those computations that the specialized Neuronet computations and so I could totally see that You know, and less, less of the monster racks, just full of GPUs that take, you know, ungodly amounts of power and cooling and,

Doc: And, and have to know the whole world, you know, where I think, I think when, when this stuff actually does get personal, which is you have a box.

You know, it's either your laptop that you have already, or it's a second box that's a specialty box that, that's going to do your AI for you, but, but helps you run your life. You know, what, what's my data that I need to know? What, what are, what are my travel, my calendar, my contacts, my finances, my all the stuff that's out of control in our lives right now.

You know, especially when you start gathering it up, like, I mean, I have no idea what TV series I've watched. I don't remember them. But you know what? Our Roku TV's been narcing on me to, for, to God knows who for some time. They have that data. If you have a late model car, it is busy reporting on you to insurance companies at all times.

Your car comes with a cell phone for that matter, the cell phone you carry in your pocket has all kinds of travel data that you don't have access to, but Google does and Apple does. What about when we have that data and, and, you know, who, you know, who have I had lunch with the most? When, who, where was I last Tuesday?

What, you know, what is all my health data? How can I pull all that together? That's, to me, that's the most interesting stuff in the longer run. It's not the, you know mean, it was wonderful that we can have Claude give us a better recipe and, and I can do all kinds of research, right? Which I do. I mean, I spend.

you know, probably two hours a day on an AI of some kind or another doing research. But but I think the interesting stuff for us is going to be what we do on our desks, you know, in our phones.

Jeff: Well, and I could totally see that because I, I think, you know, a good case in point is Leo Laporte, where he, he ran his own AI and he fed his programming books into it.

Was it Go? I, I forget what the exact language was, but it, he said, okay, I wanna program. And I'm going to use this language. He had open source books that he fed in and said, okay, only use the. The source material from these books. So it was a very specialized AI. And if you start thinking, okay, I'm going to have an AI at home.

I'm going to do some programming. I want to know what's going to be on the next TV series I want to watch. You can feed in what you like and don't like. You wouldn't, wouldn't have to You wouldn't need so much data when you, when you start getting these very specialized niches to you know, have it, have it work on, you don't, you don't need, you know, if I go, Oh, I don't care about the Roman empire and I don't care about, you know, the rainfall in Uruguay or something, Oh, well, there's a lot of data that can.

You can start just filtering out to make the data sets more manageable. And thank you, mashed potato. Lisp was the language I was looking for and Emacs.

Jonathan: I can't help but think of what George Hotz, AKA GeoHotz is doing with tiny corp and the tiny box, which tiny is maybe not the right term for this, but he is putting together AI boxes and he's got a couple of them that are sort of.

Aimed at the consumer. Now, the cheapest one is 15, 000 and it's got six, you know, 7, 900 XTX cards in it, right? So these are not cheap and they're not tiny, but compared to a full data center, they are, they're, they're affordable.

Doc: Wow. The more expensive ones. I'm looking at this now that the red one, which is the cheaper one, 15 grand, The green one is 25 and the pro one is 40 and the latter two are sold out.

Jonathan: Yeah. Yeah. Yeah. Wow. But his, you know, his Focus with that is the same thing. Let, let people run their own little AI engines. Do their own machine learning project. You don't have to have something that big. So I've, I've, I mentioned I was spent yesterday and a little bit this morning trying to get The deep seek engine running on my machine.

I did not get that going, at least not yet. I will continue to try to do that, but what I have gotten on my, my poor little single AMD card, which is multiple generations old now I've got a text to image working fairly well. Now they're not creating great images yet, but it's, it's doing it. It's working.

In fact, this is the first time that I've, I've gotten that working on this machine. Which is, it's cool. Like, it's really neat to be able to do that on a local machine. And, you know, not using somebody else's service that does who knows what with the images you create and the text that you create and all of that.

It, it is interesting to see this. Sort of step further and further towards the consumer, towards the user. It's becoming a little bit more accessible for us, for everyday people.

Doc: And I think what happens is when this reaches critical mass, when this thing is 15, 000 and, or the box that you've put together, you know, your AMD box You know, becomes the price of an ATM withdrawal and somebody makes a cheap way to get a zillion of them out there.

And we're going to do more, so much more with this stuff than, than the bigs can even imagine at this point. I think there's going to be so much invention that happens at the edge. It's like, I mean. I, I watched this happen with PCs, you know, it's like when PCs came along in the late 70s it was like, Oh, you could put your recipes on there and you could do some art and you know, but then VisiCalc happened and then, you know, and all of a sudden it's like, I mean, if you told a, an MIS director, which is what an IT director was back then, an MIS director.

You know, you're you're gonna let PCs into your company and they'll say, no, no, no, we have a mainframe here. We don't need that. Right? We have guys in lab coats. And, and two years later, the, the buildings are full of PCs because people could do more with that, with these things than the, than the bigs could.

I think we're, now the bigs here could do so much more than mainframes ever could way back then. But the personal computing revolution was mass, was massive. And I think something like that can happen with AI as well. Once we have independence and power and it's ours personally, and not just corporately.

Was there, it was personal, not personalized.

Jonathan: Was there a feeling doc back in the day that PCs were a bubble? You know, we talk about the crypto bubble. Oh yeah.

Doc: No, not only that, it was a bubble in a sense there. The the the IBM pc, which really came out of a, a skunk works in Boca Raton, Florida. Was, you know, was a big hit in business, but it wasn't a big hit personally because it was too expensive for ordinary people.

It's still like three grand. You know, you, oh, you buy that. Then you get the peripherals. You put it in the back plane, right? And that wasn't, oh, you need an ethernet card. You need a storage card. You need a bunch of other things. Right. And. You need the micro domain frame card. I worked for the company that sold those.

So you could pretend to be a VT 100 or a 32 70 display terminal. But, and then the Macintosh came out Apple had a failure with the Lisa, which was 10, 000 and. But the Macintosh was a kind of a hit, but still kind of expensive for people. And there was a huge crash in like 84 in Silicon Valley.

And Atari was in the business. Commodore, a bunch of other companies kind of Came and went but, but they pulled out of it because the damn things proved too useful. I mean, that's really what happened. Yeah. They became too useful.

Jonathan: So I have, I have pointed out several times you hear on this show and on others that we are, we are obviously in an AI bubble, but the AI bubble reminds me of the.

com bubble. And I suppose it reminds me now and hearing this, it reminds me of the, the, the PC bubble back in the seventies. And it reminds me of that in this way, things are, Hotter than they should be, right? And there will eventually be a crash. Maybe, maybe this is the crash that we've been waiting for. I don't know, with DeepSea coming out, maybe this is the crash.

And the, the dumb businesses are going to go away. You know, the whole, let's add AI to your blender. Now with AI, you're AI powered coffee pots. Like hopefully at some point that ridiculousness is going to go away. Very similar to the. com bubble and the PC bubble. It's still going to change everything.

It's just not going to change everything in necessarily the way that we thought it would.

Doc: Right,

Jeff: right. I think

Jonathan: that's

Jeff: absolutely true. Maybe I'm making a bad analogy here, but I wonder if we could almost. Have this, the, be analogous to, remember when PC started getting cheaper and then laptops and note little notepads came out and people are like, Oh, this is the death of the PC.

It's going to go, nobody will have a PC in a few years. Well, I'm wondering if AI is going to be somewhat similar, only the PC is going to be a really large language models, you know, your chat GPTs and so on, but then you're going to have these smaller, more personalized units that took over You know, it would be the same as like your little notepads first, because just like when they said the PC is going to die, well, it didn't because it was just people, not everybody needed a PC, you know, grandma's just writing some email to the kids or somebody who's just kind of surfing the web.

They're not creating, they're not really doing anything with a heavy lift. They don't need the whole big. PC, the expensive PC when it's like, Oh, this little tiny, you know, now it's phones can do a lot of this, but you still need, when you're content creating, doing a lot of bigger stuff, you have to have that horsepower.

Well, maybe, you know, the chat GPTs and the other large language models, there's going to be a space for them where you really need the power, but a lot of stuff, you know, you're going to be able to specialize and it's like, well, I don't. I don't need all that power. I just can have my specialized topics on my home PC or my device and that will, that smaller model with limited data will, will do what I need most of the time for my specialized things I care about.

Jonathan: You know, I, I could, I could even see a future where you have, like, NVMe, I know there are some GPUs that have slots for NVMe's on them. Right? I could see a future where you've got a 1TB NVMe that's on your compute module, like it's connected to your GPU, and that's where your bit and most of the power for your LLM comes from.

And that just gets maybe updated every once in a while, because once these things are trained it can, it can take new information in and sort of work with the new information. I, I think, I think part of this becoming useful for everyday people. Not having to deal with those huge data sets is going to be a big part of it.

But I mean, let's, let's talk, let's think about like all of the other things, like why, for example, why did the cloud take off? Why is that a thing that everybody wants to do? Well, it's because nobody wants to run their own servers. Fundamentally. And I, I kind of wonder whether there's a similar problem, nobody wants to run their own AI.

Because it's still fiddly right now to get it set up.

Doc: Oh, we're not even close to that yet, I think, in terms of the convenience. But, you know, it's kind of, you put the resources where you need them. Obviously, I mean, like we're getting fiber here shortly. And so I'm going to have two gigabytes, symmetrical.

Back when I had You know, a few kilobits, I could get 16 IP addresses and I could set up a server and I could, I ran my own email and I ran my own website and it was all in my house. You know, I mean, I had 16 IP address, here's 16 fixed, there's not even thinkable now, right? You're going to have to use the service.

And it makes total sense. I mean. These guys have set up giant server farms and they're up 100 percent uptime and you're, you know, you pay rent for that and, and they could assume that privacy has worked out at least to some degree. And but there are going to be things that you need locally. I mean, you need the phone in your hand, you need the laptop in your, in your, in your, in your briefcase, you know, it's or on your desk or on your lap.

There's, there's, there. We're going to need compute resources on site and in our pockets and things like that and and we'll work it out But we need the invention that mothers the necessity. We don't have those yet You know, that's that's the missing piece and I have to get off. I hate to say this. Yep understood, but you weren't it's fine I'll leave you with one thing, for the next time you talk to Leo, have you noticed that um, Google's Notebook LM, which will take anything and turn it into a podcast, that the male voice in the podcast has to be trained on, on Leo.

It sounds like a young Leo of some sort. It's sort of interesting. Listen to one of those, one of these times where they're, they're really, they're really weird that they put this podcast together, but it's, you know, it's good, but anyway. They knocked off Leo for that, for the sound of that. Huh?

Jonathan: That's funny.

All right. Thank you, Doc, for being here. Appreciate it, man. Thank you. Talk to you. Good to see you both. Okay.

Doc: Bye bye.

Jonathan: All right. So, you know, talking about that idea of, of nobody wants to run their own servers, I wonder if you could sort of make an observation that we all run our own servers. They're just kind of become invisible, you know, because there, there are daemons that are built into the phones and there are daemons that are built into our, our desktops and all of that.

It would be, it would be interesting to kind of do, try to step through all of the different things running on a device and figure out, like, how many of these are invisible servers?

Jeff: Well, I was even thinking of another kind of analogy when you look at, so I run my own server, like a Plex server, and I have all my videos, you know, my own movies and music on it, so it takes care of what I want.

But it doesn't fully replace Netflix, Amazon Prime, Hulu, anything like that. So, I guess that's another way to look at it of, it's, it's gonna be smaller targets, I think, where the next real innovation in AI is gonna be. Which, if you look at the hardware, there's already, almost every CPU now is coming out with, they have a little neural processor in there for AI, I mean, even the pretty small, low power ones have that specialized chip in there, and we're not really utilizing it right now, but it's there, you know,

is the data going to become the real commodity? You know, okay, I've got AI everywhere, I've got all these little AIs. Oh, you want access to the data? What, you know, what are you going to train it on? Are you going to be limited?

Jonathan: Yeah, there's definitely some interesting questions in there about that. And it kind of, we're kind of at the point now where all of that, and it's a quirk of the way copyright law works, that like once you take all that data and put it through an artificial intelligence training it, the legal theory is that it strips all of the copyright away from that data.

And we, we had a lawyer on the show back a few weeks ago, and he sort of looked at that and said the genie is out of the bottle. There's no putting it back where that's just kind of because that's what we've done. That's where we're just going to have to live from now on. I thought that was really interesting.

So, you know, it depends, it depends on which side of the data you're talking about. One of the other things that I find really interesting with AI right now is They, well, you know, they, they use terms like trustworthy AI. But one of the things that you really, you have to dive, kind of dive into is, is this AI going to give, well, one, everybody, is it going to give me the right answers?

Is it going to give everybody the same answer? Is it going to give me the same answer every time? And I, I did some looking into this for a security article. And, you know, on AIs, on LLMs, there's a setting called temperature. And if you set temperature to zero, it is supposed to give you the same, you know, a given input gives you the same output every time.

If you turn the temperature up, it randomizes the output so that it gives you something different, because that can be useful to get a little bit different output every time. And there is at least one of the AI models that even in, I think it's I think it's OpenAI, one of their, one of their leading ones if you turn the temperature all the way to zero so that it should be you know, a completely deterministic model.

It will still give you different answers every time. Because, as it's loading those tokens from the disk and putting them into the GPU, it does it in batches. And which tokens end up in which batch, from what I could tell, depend upon the load speed off the disk. And sometimes the load speed off the disc is slightly randomized because physics.

And I found that really, really fascinating that some of these AIs are not deterministic just by not necessarily intended to be that way, but like a quirk of the way they're designed. They're non deterministic.

Jeff: Huh. That's really interesting. I, now, you know, I, I will put this caveat. I've only used a couple different AI is basically Google's Gemini and Microsoft's Copilot.

So I haven't played with ChatGPT, but I've put in documents that I've written into each of them and just say, okay, clean this up, you know, make sure there's no grammatical errors and spelling and, you know, sometimes maybe I use a little too much slang or whatever, you know, clean it up and make it, make it more.

You know, generally appropriate so that a larger audience can understand it. And I found it interesting that in my experience, co pilot does a pretty good job of just, you know, Oh, we've changed the phrasing here and you had mixed some tenses and, you know, maybe changed the vernacular slightly, but, you know, and improved the document.

Whenever I'd use, try that with Gemini, I would come up with a document that was different meaning. You know, I'd, I'd read through it and go, wait a minute, this, this changed the meaning or, you know, it, it, this phrase was credited to somebody that didn't actually say it. And it's like, You know, it's temperature was really high and, you know, kind of really, really my document out was not my document in at all.

Jonathan: And that kind of gets to like the danger that we've seen with people trying to overuse LLMs for different business things. So some of the some of the most hilarious cases were like lawyers. And they would go to the LLM and say, you know, here's the thing I want to prove, find me case law, and the LLM would hallucinate case law, and then it was discovered, and I think at least one lawyer, probably several, got disbarred over that because that's a definite no no.

So, you know, don't, don't trust your LLM. Who, who is it? Is it Simon? It may be Simon that makes the statement that LLMs are not telling you the truth. They're just trying to convince you. Yeah, I

Jeff: like that. Yeah, that's, that's I know in business, whenever you use them, they tell you, check the output because it's not always right.

And I always double check, you know, whenever I say, hey, clean this up, or, you know, it's, it's nice for some business documents to, Okay, but Here's what I'm trying to say. Put this in a nice business format and it'll format it nicely for you, but you have to read it because sometimes it's like, where did that come from?

You know, it goes off into left field and

Jonathan: yeah, well, there were a couple of other open source stories from this week that I thought were pretty interesting that we might touch on briefly. One of the big ones is that Microsoft has open sourced a new database format. Did you see this? What are they calling it?

DocumentDB or something? FerretDB? I did.

Jeff: Yeah. I'm kind of wondering what Microsoft is doing. They're open sourcing a lot of stuff lately. And I don't know What's the long game? That's, that's kind of what I want to understand because there has to be a business reason for it. Whatever that is. And

right now I'm just not sure what it is. It's working for them. Yeah. Other than I have said on the other podcast is I think Microsoft is trying to basically Make its operating system the similar to what Apple did. It's going to be open source underneath. Eventually be the Linux kernel, and they'll have their own custom interface to it.

And that way they can take a lot of the resources that are working on the operating system and have them working on cloud and AI and other other things. And the under under the hood just runs. through the open source.

Jonathan: I, ironically, I think you're absolutely right. I don't know about windows, but what you, what you just described pretty much encapsulates what Azure is these days, their big cloud system.

And that's where a lot of this stuff gets run. So, you know, Microsoft is sort of becoming a cloud company where you just you, you rent the resources from them, and you use Microsoft because they're the one that wrote all of these tools and therefore it's going to work well on their cloud. They're, they're moving into An open source friendly cloud company, which is very interesting to see.

So they're sort of trying to become Amazon.

Jeff: Oh yeah, and that's, that's a lot of money, you know, because Amazon doesn't make very much on their web store. When you buy a, you know, bag of cookies from them, it's all the web services and the cloud. And that's where, you know, they're making hand over fist. And I think, you know, Windows is not making them.

Well, I should say it makes them a lot of money, but it's kind of a stagnant area. I mean, they're not going to double their market share. They have basically, what, 95 percent of the market or, you know, somewhere thereabouts. And it's kind of where it's going to sit for a while.

Jonathan: Well, and you

Jeff: mentioned

Jonathan: it earlier, Microsoft saw the writing on the wall that maybe the desktop, the PC is going to go away in some ways.

Right? I think that was part of their their, the push to get them to look into new markets. And they tried, they tried the Windows phone, and that obviously did not work out very well for them. And so, you know, Microsoft sort of looked in the future and saw, well, what do we do when nobody is running desktops anymore, when the desktop market is tiny?

What is our company going to look like then?

Jeff: Well, and they're back to innovating again. I mean, they're trying a lot of things. I mean, you know, and I've said this many times, I worry when a non technology minded person runs a technology company. You know, that we The bomber years, or you can look at Intel when they had I can't, can't think of the person before yeah, against look good before guns.

I don't know who it was either. Yeah, but they weren't technology minded. And, you know, there's several other examples where can cause problems. Bob Swan was directly before Gelsinger. Yeah. Yes. So, I mean, I think now that Microsoft is back into technology and into innovation, that's, you know, there's a lot of stories where they're, they're opening things up and they're becoming more compatible with Linux.

They're open sourcing, they're, and I, and I think it probably eases the transition to The future because people say, Oh, Linux. Yeah. Ha ha. You know, is it? But people don't realize the world runs on Linux. You know, the desktop is like the one place that Linux isn't dominant. You know, you can argue that the handheld devices, the big servers, the big infrastructure stuff.

Pretty much all runs on Linux and desktop is the only thing that isn't. And well, if they're going to go cloud, they're going to make money. They better be compatible with the operating system. That's running it all.

Jonathan: Yeah. The most fun thing I ever, ever heard somebody do with that was if I said a Linux conference, the guy's like, how many of you run Linux on the desktop and a few hands went up.

It's like, how many of you used Google? In the past 24 hours, everybody's hand went up. It's like, well, you know, you were running then a service based on Linux on your desktop. It was just sort of this, this sort of a, you know, interestingly different way to look at it. Yeah, you know, when, when Microsoft bought GitHub, that, that was kind of the one that really, yeah, that was the one that really scared me the most.

Microsoft bought GitHub and a lot of people were already at that point using GitHub. That was kind of scary. That worried a lot of us. There was a bunch of GitLab installs that that popped up right after that. But I must say, you know, so far they have shepherded GitHub very well. And it's, it's grown and become more useful as a service, not less.

Jeff: Well, and it wasn't that, but, you know, when they bought it, it wasn't that far off of the, was it Embrace, Enhance, Extinguish era? Mm hmm. So, I mean, that was pretty close in the rear view mirror. So there's a lot of people that I, I totally get the. You know, when I heard about it, I thought, oh, you know, this can't be good.

But so far, you know, knock on wood, it's going great. And it doesn't seem like it's the Microsoft of old anymore. It's, they've, they've kind of evolved to change with the markets, change with, you know, what's, what's important and where their next revenue stream is going to come from. And, and, and realistically, you have to, you know, otherwise you're going to wind up The next Kodak or Xerox or, you know, you're

Jonathan: just not.

So there is a there's another story that sort of is the inverse of that. And I'm not sure if you've seen this one SimGrep and the OpenGrep fork. Did you get to look at this one at all? No, I haven't. Okay, so Semgrep is it's an open source security tool. It does, I believe it's offline code scanning, right?

Like it's it looks at your source code and it tries to find common errors in it. It's like a static analysis tool, I believe, is what you would call it. And they Made changes to their license back a few months ago You know, they they essentially added to their open source offering. They added a non compete clause is essentially what they did which is the same this is the same direction that we've seen with other source available licenses like the the Open business license.

I think it's one of the other ones and you know, we, we've seen multiple, multiple companies go in that direction. And what's interesting is with Semgrep, the exact same thing happened. The, the outside of the company group of people that were using it went to the last release that was under that. that truly open source license and said, okay, we're going to start at this point.

We're going to fork it and you know, going, going forwards, it's, it's no longer to be SimGrep and we're not going to use your new silly license. And you know, OpenGrep is the is the name that they chose. I'm trying to find the list of who all is involved in it. Oh, interesting. So in 2020, the the, the company behind Simgrip took a, a round of VC funding.

And then in 2023, they took a 53 million series C round of venture capital funding. Yeah, it's probably not a good sign when an open source company takes. Venture capital.

Jeff: No, I, I, in general, I'm not a fan of that at all. I I've seen a lot of companies kind of turn because they, they want their money back.

It's an investment.

Jonathan: Well, yeah, you've been, and so what, what ends up happening is you've got these outside investors that in most cases don't understand the company, right? They don't, they don't understand open source. They don't understand why the company was successful to start with. And it's just like, well, okay, we've given you a year now.

And you're not making the millions of dollars that you promised us. We now have this big lever in the form of stock ownership. It's time to make changes. Here's the changes we want you to make. And that sort of sometimes gets rammed through and does not always work well. In fact, I would say it usually does not work

Jeff: well.

Well, you know, I, I would argue though, if you, if you have an open source company, okay, I'm, I'm running the next, you know, super grip we'll say whatever. And it's becoming really popular. And someone said, Oh, we want to invest all this money. I think one of the things I would have to say is, look, this is not going to probably make, you know, we want to give you 50 million.

Well, it's probably not ever going to pay that out or not in any kind of reasonable time frame that you would ever want to see. You know, there would be better investments to make. Right. Because, you know, I, but I mean, maybe people get lured in by the dollar signs and want to, oh, we want a big investment.

But that's not what. Open source, you know, I mean, it can make money, but it's, it's, I don't see open source ever being, you know, a, you know, a Microsoft, a, anything like that, you know, you have a, you have a few, you have a

Jonathan: few unicorns that can do that. Right? Like, I mean, Red Hat. Red Hat was is still a completely open source company.

Well, almost, almost completely. I think they have a few closed source products. But they got, they got huge, and that acquisition by IBM was quite large. There are a few unicorns that really become big like that, but in general, you're right. I don't know if that is a failure with the open source funding model.

Like the, the ways that people make money with it, or if that's just sort of designed in, maybe it's, it's that way by design and that's a good thing. But you, you don't tend to see open source companies become huge like some of the others do.

Jeff: Well, and I guess, I mean, Red Hat's big, but it's still not private sector big.

I mean, there's, it's, it's one of the biggest examples. But you know, it's near the, near the top of the list, if not the top of the list, but it's not going to be, I actually, I'd have to, I don't know what their revenue was, you know, say even last year.

Jonathan: I mean, IBM, IBM acquired red hat for 34 billion with a B that's, that's chump change compared to somebody like Exxon mobile or, you know, where a company like say Tesla was at the top of their valuation, but that's, that's still quite a bit.

Yeah. Well, I mean,

I'm, I'm curious now. I guess Apple is also one of the ones at the top.

Jeff: Oh my goodness. It says, well, this is, this is old 2019. They said 3. 4 billion. So I guess, I mean, it was, it was bigger than I thought it was. I didn't realize it was in the billions.

Now I don't know what that is after. I don't know what the net income is.

Jonathan: You would imagine quite a bit because they don't have like manufacturing expenses and they have a lot of people expenses and paying a lot of salaries for engineers to work on stuff. You know, it's interesting with, with some of the legal changes happening, like in the European Union, there, there is sort of this new funding model that's, is beginning to suggest itself for open source.

And I, I don't know that this is going to create more unicorns, but I think it'll be a, a good source of funding. And that is essentially, you know, companies use open source products, open source license, libraries, and it's like, oh, we need your help. To get a software bill of materials because the law requires it.

We need your help to make sure that we close vulnerabilities because the law requires it. And you know, the, the appropriate answer for an open source library and open source developer to give is sure. I will be glad to help you. Here's my hourly rate, or let's negotiate a contract where you support the project and we support you as a business.

And you know, there is this opportunity that, that we're starting to see because of things like the Cybersecurity, the Resilience Act in Europe and some similar things in the United States that is kind of moving us, I think, I hope to a little bit more, I don't love the word sustainable, but it's, it's fairly accurate in this case, a more sustainable place for open source.

Jeff: Well, I mean, it, I, I, I would like to see it because, I mean, sustainable, I'm okay with it, I, I'm okay. You know, even, even if I had to pay for the, the Linux kernel or Linux operating system, you know, if it was a reasonable amount say, and it's going to vary based on, you know, your economic place in life, you know, but if it was pretty cheap.

You know, based on what you, whatever you make, you go, you know, it's pretty cheap. I'd be okay. Cause I, I at least understand there's people that are putting a lot of time in this and they've got to eat too. They've got to, you know, as long as we stay away from some of the outrageous licensing, you know, if they just go, Hey.

How about, you know, and, and you kind of have that with the donations you can do through various open source organizations, but, you know, and I try to give a little money to here and there to various, you know, I've donated to KDE, I've donated to Code Weavers. Well, I've kind of, to wine, I've, I've purchased a license through Code Weavers who was in the major backers of wine.

I've, you know, things like that where I. I try to support open source and, you know, give, give a few, few dollars, you know, I'm not breaking the bank, but you know, I'll, I'll throw them, I'll throw them a few bucks, you know, just to help out.

Jonathan: There is one project that immediately comes to mind that has made that work and that is Ardour, the the, the audio editor, multi track audio recording and editing suite.

And they've got a system where you can become a subscriber and they offer they offer different tiers you can either do you can chip in one dollar a month and I believe they they label that tier as the sort of the You know the low income from a from a country where this is all you can do I forget exactly how they put it, but they have that tier They have a four dollar tier, which is the one that i'm on a ten dollar tier or a fifty dollar a month tier and when you pay, that unlocks the ability to download the binaries that they compile for you, as opposed to, you know, if you're running on Linux, it's reasonably easy because most most of the links just shows also package it.

But particularly for people running on Windows, I think Mac as well. It's sort of a challenge to get our door installed if you're not a subscriber, and they make that work. They pay a couple of developers on that. And it's, it's really interesting to see that they have managed to turn this open source project into, into a business and able to pay a couple of people to work on it.

Jeff: Well, and I think even on Linux, the, the packaged versions are older versions. I thought, I thought you were always behind.

Jonathan: Well, they're going to be a little bit behind just because it takes time for Ardor to push. Ardor will push a release out, and then you have your packager will come along and go, Oh, hey, there's a new version of Ardor.

Go through the steps to package it, and then most distros, once it gets packaged, it gets pushed to some testing repository. And then it tests there for a couple of weeks. And if nobody finds any bugs, then it gets pushed to stable. So you do, you end up running, you know, probably two weeks behind the full release.

That's. Honestly, that's one of the reasons why I went ahead and paid for it. When there is a new release of Ardour, I can just go grab the installer script and run it and back up and up and running with the new version. I also, I, I had some feature requests and it felt kind of weird to go and ask for features and not as a paying subscriber.

So I figure I'd give them 4 a month. That gives me the right to complain about things that are broken.

Jeff: There you go. Yeah, I could see that. I, I guess I was mistaken a little bit. I thought, I thought part of their licensing or whatever was the, the free version was. Or maybe it's if you download it from their site or something.

I, it's, I, I want to say it was. You

Jonathan: can, there, so there is, there is a there's something like that. You can download. I'm trying to remember the details you may be able to download the old version but if you get the most up to date version, it'll do something like only let you render out 30 seconds at a time, or after every 5 minutes it does 30 seconds of silence, like it's got something like that built into it, I can't remember the details.

Oh, okay. But yeah, there is, there is something in there kind of a, I guess you would call it annoy aware, but yeah, that makes it kind of, kind of encourages people to pay the money.

Jeff: Yeah. And you know, and I'm okay with that too, because for me, I don't pay anything to them if I'm running an old version, you know, I use it so infrequently.

You know, it's like the, the newest feature bug fix probably doesn't even affect me cause I'm not a power user. I'm not, you know, Oh, I'm just recording 30 seconds of audio for something. And then I'm done for six months or whatever.

Jonathan: Yeah. I'm, I'm looking for their their, so they call it their demo version, the demo copy.

I guess it just has a timer. It, it it'll, it'll work for, and they don't, they don't say on this page how long, but you know, a certain number of minutes that it'll work. And then you just get a message, the timer's expired. And, I'm sure people don't like that sometimes, but they're, they're they have an FAQ on the Ardor site.

I thought this was free software. Ardor is free in the following ways. You're free to do anything that you want with it, including You know, make changes, put it on as many machines as you want. You can get the source code without charge and build it yourself. That's funny. I think it's worth it for, for

Jeff: Ardor.

Well, I was gonna say, and they have a single payment. Okay. So you can just say, okay. If you choose to pay less than 45, you'll get the current version and updates to version, all the point versions, but not eight. If you go over that, then you also get the next major version and a bunch more. So, okay.

Yeah, that's,

Jonathan: yeah, it works for them. All right. Well, we have hit the bottom of the hour again. We've managed to fill a full hour talking about deep seek and some other open source news things going on these days. Let's go ahead and see about. Wrapping this thing up and letting the folks go. We didn't ask Doc.

I should have asked Doc before he had to run. Because we are required to ask people about their favorite their favorite scripting language and text editor. I'm not sure what, I'm not sure what his would be these days. I doubt that he does a whole lot of scripting anymore. But what about, what about you, Jeff?

If you have to, if you have to smack together a script to make something work, what, what language and editor do you use?

Jeff: Language would be Python. Mm hmm. So I That's the main language I'm used to be fluent in, I guess. I don't do a lot of programming and scripting these days. But if I do, Python is my main tool in the toolbox.

Editor? I use Kate. I'm not a, you know, unless you need command line, you know, then it would be Nano. I'm not a VI or EMAX person. I never learned it.

Jonathan: I tried to use Emacs once. I'm like, Oh, this will be cool. I I'll, I'll get Emacs installed. For whatever reason, when I went to install Emacs, it gave me the GUI version of it and it was just terrible.

And I've never gone back and really tried to work with it ever since then. I can, I can get by in VI, but man, I, I prefer Nano myself.

Jeff: Yeah, I, I, I can like change something and then save it and that's about it. I don't have all the keyboard shortcuts and, but, but you know, I've never really had to learn because I've either been on the system directly or I redirect the display.

Or I'm running in a remote desktop, so I don't, I'm usually not limited to command line only. I'm a hardware person, so I, you know, I, I help build semiconductors. I don't do a lot of coding, I'm not running big banks of servers.

Jonathan: Makes sense, makes sense. Alright man, is there anything you want to plug before we let the folks

Jeff: go?

If you enjoyed us talking, you can catch more of us over on the Untitled Linux Show on the Twit. tv network and thank you for having me as a guest. I always enjoy being here. A lot of times I can't make it just because of scheduling, but. Always a pleasure when I can, and thank you for listening.

Jonathan: I don't suppose you have any poetry for us?

Jeff does the poetry corner at the end of the Untitled Linux show, and it's always a lot of fun.

Jeff: I don't, I wasn't prepared. Ha ha ha! Nobody's ever asked me on this show for poetry, so.

Jonathan: I know! If I remember next time I have you on, I'll make sure and bug you to see if I can get you to have something, because it's fun.

Jeff: Technological poetry. Yes,

Jonathan: yes. They're all, they're all fun. Our, our audience would appreciate them. I'm sure. All right. Thank you for being here, man. I appreciate it very much. All right. If you want to follow me, of course, there is Hackaday. We appreciate Hackaday being the home of Floss Weekly these days.

My security column goes there live on. And yeah, we've got the Untitled Linux show over on Twit that we've talked about. I am going to be taking a break from the security column for one week. It looks like that is in February. Because I am going to be traveling. So February the 21st, we will probably not have a column.

We are going to try to go ahead and have a Floss Weekly that week. And we're going to get a little bit ahead, I believe, on doing the recordings. But anyway, yeah, and if anyone happens to be at the Zero Trust World 2025, make sure to stop by and say hi. You can find me there. But yeah, we appreciate everybody that watches and listens, those that get us both live and on the download, and we will see you next week on Floss Weekly.

Episode 817 transcript

22 januari 2025 | min

FLOSS-817

Jonathan: Hey folks, this week Dan joins me and we talk with Stefano Zaccaroli about Debian, but also Software Heritage. That's the source code archiving project that you probably haven't heard of, but really should know about. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 817, recorded Tuesday, January the 21st.

Incompatible with reality. It's time for Floss Weekly. That's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something very fun today. First off, we've got Dan the man, Method Dan, the original Linux outlaw. I like that intro, Dan. It's, it's fun to introduce you that way.

Dan: Thank you very much. I like that. It's fun to be introduced that way. I have to say. Yeah, it's great to be back. Good to see you, Jonathan. Hello everyone.

Jonathan: Yeah, you know one of the things I enjoy we talked briefly in the in the pre show But one of the things I enjoy about doing this show is like the group of people that we get to hang out with You know, we've got we've got dan.

We've got simon aaron I've brought some guys on in the form of you know, rob and jeff and those guys But we also will have randall and even we're having the The plan is for next week doc to come back as co host, right? And so just the fact that we get to hang out with all of, all of these guys as part of the hosting team, I enjoy.

And then, you know, there's the fact that we have all of these guests that are just, you know, every one of them are great, have, have super interesting things to talk about. We've got another one of those today. And I understand that you've got you've got some connection with Stefano Zach Zaccaroli.

What's, what's your history with, with Zach, with Stefano? I interviewed I interviewed Stefano a few times for Ledex Outlaws back in those days. Once we were just discussing earlier before we got started once when we were both pressed against the window of a bus, quite literally, physically in Brussels on, at Fosdem because Fosdem is very busy.

Dan: Busy and those buses are not built to take that many people. So it was quite funny. We were just next to each other. And then I re I didn't actually realize who he was at first. And then we got chatting and it was like, Oh, can I interview you? So we, we did it right there. And then

Jonathan: Foster must be great for that because like almost every, basically everybody that is there is potential interview material.

Dan: Yeah, yeah, you need to take a lot of spare SD cards or whatever it is You use to record on when you go to somewhere like Fosdome.

Jonathan: Yeah All right. Well, let's go ahead and bring him on So Stefano, I know that he has been involved in Debian over the years Probably still involved to some extent and the other real no Not the only other.

Been involved in OSI but the thing that really interests me is software archive, because I, I am a, I am a fan of the idea of archiving software, and I'm also aware of some of the challenges, and so I want to, I want to pick Stefano's brain about all of that but first, just Stefano's Welcome. We are absolutely delighted to have you here.

Stefano: Hello. Welcome. Well, thanks a lot. Thanks Jonathan. Thanks Dan. And thanks also for the memory, Dan. Well, that was a long time ago. It was a really fun interview to actually do.

Jonathan: So what, give us your background. Like, so if somebody asks you, what, what, what do you do? What open source stuff do you do? What's, what's your answer?

Stefano: Yeah, so I've been around for a while, so I'm myself in let's say in my real life, I'm a computer science professor at the Polytechnic Institute of Paris, but I've been a geek for my, a computer geek for my entire life. So I started getting into free software when I was a computer science undergraduate at the university.

And we were, we had the lab at the university that was managed by the faculty itself, and it was some proprietary Unix at the time. And we have a second lab, which was managed by volunteer students. And of course it was Linux. And so that. Things intrigued me from the very beginning. And I had one of my first objective was becoming one of those, you know, volunteers is that mean that was helping out students and that's, that's how I started.

Yeah. It was that yet, of course. And then the rest of history, as you say,

Jonathan: there's an interesting crossover. And I think there always has been between that idea of political science and open source. And, you know, we, On the show, one of our rules is that we don't dive into partisan politics because nobody, nobody's happy about that.

But with that set aside, there is an interesting crossover and it is a thing that it is important to have people that sort of are aware of both of those realms, particularly when you start talking about legislation that's going to deal with cybersecurity and copyright law and things like that. Is that something that you, you focused on where those two overlap?

Stefano: Oh, no, I was actually doing computer science. Oh, computer science! But, since then, especially in my days at Debian, I've been involved mostly in policy work, and so, yeah, what you say, I completely agree. So software and technology is really political, and free software, if possible, is even more political.

So, yeah.

Jonathan: Well, yes, but actually no, but actually yes. Yeah.

Stefano: It's complicated.

Jonathan: Yes. Yes. So how, how did the how did the Debian piece come along? I've, I've always thought of Debian as difficult to get started with. Like you, from what I understand, to even be able to publish packages on Debian, you've got to go into like physically beat one of the other packagers to get your keys signed.

And it's just, you know, there's some, there, there's some challenges.

Stefano: Yeah, that was part of the thrill and of the challenge at the time. So the technically, the reason why I started looking into that was that, so I was passionate about for now, nowadays, somewhat obscure programming language, which is OCaml, Objective, Objective Camel, which is a functional programming language.

And I was really passionate about functional programming. And we were using Debian, and we were using GoCamo. And as you start coding, you, you discover that, well, some library is missing, in the sense that you cannot just, you know, apt get install that library and say, Hmm, let's look at this. How could I make this package, this library, which exists in this free software available to myself, my, you know, students, colleagues, and everyone else in the world?

And that's why, that's why I started looking into this Debian thing. thing, which was indeed quite intimidating at the time. But then, you know, I find friends that were already collaborating. A friend of mine, a student as well, was already a Debian developer. So it basically got me started into the process.

That must have been 98 something. So yeah, at some point you need to meet people and then the chance to meet a lot of Debian developers that were, you know. Traveling around and, you know, getting to my city at a time that was Bologna and finally get my keys signed.

Jonathan: It was pretty

Stefano: amazing.

Jonathan: Now you, you did, you were not just content as being a Debian developer.

You actually became a, you became the head of Debian for a bit, right? The Debian project leader for a while?

Stefano: Yeah, I, I, I've been dev project leaders for three terms in between 2010 and 2013, I think. Mm-hmm . And that was, so at the time I was a postdoc researcher. I already moved to, to Paris at the time, and I was really, really involved in the project and I, I was realizing that there is a technical part, but the, I think the most interesting innovation, innovation of Dion is actually the development model.

It was the first distribution of being actually a community driven distribution. And it's also a community that essentially self determined itself. So it's a project with a constitution that developers themselves created that was before my time, but it was not really common at the time to think of self determination of online communities.

Later, it was a notion that started to become more discussed at least, but at the time it was really, really an innovation. And so I was really passionate about that. Thing. And I say, okay, what can I do to help? And beca I did a bunch of things before , you know nominating myself for for the DBL election.

But I was really passionate about improving the processes and, you know, try to see how we can make the community give. The best of themselves.

Jonathan: And you're still involved with Debbie and I assume

Stefano: so. I'm still formally a, a developer, technically it's called a non uploader developer because I, I'm not doing technical work anymore in Debbie and this day, so I, I announced the, you know, the, the, the right, the upload right associated to my GPG key to avoid, you know, reducing the tax surface.

But i'm still a voting member of the project and i'm still following The yearly election the policy decision the political discussion on the project and some legal work work That's the kind of stuff i'm still involved with

Jonathan: well while we're on the topic of debian If you if you have your sort of finger on the pulse of the project still something i'm curious about is What's what does the health of debian look like these days and i'm thinking, you know, primarily in terms of Are there still developers coming on board?

Are there enough developers that are there to handle the workload? What direction is that trending? And like, is there a, is there a funding model there? Does there need to be? Are people getting paid to work on Debian? I honestly don't know how all that works inside Debian land.

Stefano: Okay. So many questions there?

Yes. So Debian as up. So starting from the, the funding project, Uhhuh Debian as a project is actually a nonprofit project in the sense that the Deion project doesn't pay developers or maintainers or other technical roles or even non-technical roles to, to, to actually do stuff. But there are a number of companies that are.

present in the broader let's say Debian ecosystem. A big example is of course Canonical that the, that the company behind Ubuntu, which is a commercial distribution based on Debian. And so all these sorts of realities, not only Canonical pays people for making something happen either downstream distribution or in Debian itself.

So that is part of the, of the Debian, let's say sustainability model. But what's beautiful about the Debian model is that a lot of people are just. You know, volunteers to the project there might be paid by research labs or or other institutions, but they might also be just working in their spare time.

And that's correct. And it poses some challenges because when you start mixing the work of volunteers with the work of people pay to do something, you might have some sort of tensions. But it's I think it's really working well for that. And regarding the turnover question, I think it's still pretty healthy.

So probably I don't have the exact number numbers in mind, but it might be a bit lower than it used to be in the in the early days. But it's still quite healthy. People come and go and and actually. You know, making packages is becoming more efficient. So with team maintenance and with the number of automation tools, the workload is actually cheaper in terms of the number of packages you can maintain and you can act on than what it was like 10 years or 15 years ago.

Dan: Yeah. Pretty cool. Wow. So did they still have to Debbie in kilts by any chance? Cause that's something that you used to be able to identify people at conferences. I certainly do.

Stefano: I still have my Debbie in kilts and I even wore it at my wedding. Not for the entire time of the ceremony. My wife wasn't really okay with that, but at some point at my wedding party, I did wear the Debian kilt.

And that was created for the DEBCOV in Edinburgh and by an actual Scottish Debian developer who registered a Tartan. So we actually have a Tartan registered in the name of Debian in. Whatever official Tartan registry there is out there.

Dan: It's probably a bit like software heritage. There's a Tartan heritage somewhere where they've got it all archived.

And, and they're very serious about that kind of stuff. Don't get

Stefano: me started about the trademarks of Tartans because I have no idea. I'm sure there is.

Dan: Yeah. Yeah. And so sticking to the Debian kind of subject, a lot of people listening to this I know we have a lot of very technical listeners, but we have some who may be less technical might be thinking I've never used Debian.

Why does that matter to me? But you mentioned it just there about the fact that the upstream thing. So Debian to me feels like the upstream of so many distributions that I suppose I'm interested in like the relationships that Debian has with say Ubuntu, Canonical is a big one that we all know, but also things like Linux Mint, and I'm not going to go into listing how many Debian derivatives there are so many.

So how does that relationship work? Do they give back to the Debian project?

Stefano: Yeah, so that's a very interesting question, because one of the things that I I. I sit down and thought about a lot when I, when I first become DPL, Debian Project Leader, was actually understanding the role of Debian in the ecosystem of distribution at the time.

And I realized that a lot of the impact that Debian had was not directly in the sense that people were directly or primarily using Debian. There were a lot of people that were benefiting from Debian. Indirectly, possibly not even knowing that Debian exists. And so I started measuring at the time, essentially the tree of derivative distribution.

And it was hundreds of downstream distributions, meaning that people have installed those distribution. They know the name of that distribution. They don't necessarily know it comes from Debian, but they're still benefiting from all the work that Debian people were doing and are still doing today. So, It's free software, right?

So there is no strict obligation to give back changes. The license says that you can just use the code for whatever you want, as long as you respect the license. But there is no obligation to contribute back. So you have some distribution that do contribute back and that benefits Debian a lot. And then from there, it also goes down again to other distribution.

And there are others that Contribute less, maybe because they don't have the means, maybe because they are not interested, and that's fine. So it starts to get, you know, tense and complicated when people in Debian expect contributions that are not coming. And so that was part of the time of the debates between Debian, Ubuntu, and Canonical.

Dan: Yeah, it seems to me that there's a lot of scope for discussion. I hesitate to quote argument, but discussion about things like system D, for example, when Debian brought system D in Jonathan's a Fedora guy. So he's looking at us a little both like everybody knows about that. Oh, I'm aware. I am

Jonathan: aware.

Dan: So there was arguments about system D or no system D. Is it difficult to wrangle? Is it difficult to get like a clear purpose? Is it it's a lot of debate going on all the time?

Stefano: So it's I mean, I think it's the the reverse of the coin of you know, having a large community of maintainers in the sense that in theory the Debian model and the Debian Constitution actually it's it's designed so that everyone can work on their own packages and essentially they can work independently from others but you can only work independently from others As long as there are no strict technical interactions between packages.

And when you start touching like low level packages that are pre installed and they deal with the operating system and they manage services like systemd, you can have very complicated technical decisions and you can have maintainers that are on opposite side of the decision you need to take. So this is the example was a big one and the way that Debian has to govern all that is that it has a body which is called the technical committee, which is some sort of judge it's not a single person, it's multiple people, but it's a committee that can actually make decision when they are not in the scope of a single package and they are like they have broad implication on the on the technical setup of the distribution and the system, the one at the time was, was a big one.

Dan: Yeah, that was a big fallout at the time. So while I'm kind of on this tack, I need to ask a bit, but as well about the relationship with FSF and all of that kind of stuff, because am I right in thinking that Debian isn't on the approved list of, of FSF distributions?

Stefano: That, that is correct. So still to this day.

So the, I guess we, the, the relationship within Debian and FSF went through many different periods. So in the very beginning, the FSF actually funded. The work that then led to the creation of Debian. So Ian Murdoch, the founder of Debian, for a time was funded by the FSF to essentially create a distribution on top of the free software that the FSF was providing.

Either developing themselves or funding to achieve the GNU system and the GNU operating system. So that was something at the time. Then I would say that culturally and even before my time, Debian went through a big, to put it simple, open source moment. So let's say we are here for the tech part. We are here for the collaboration.

We are not here for the political part of the software and for specifically for user freedoms. Then, and that was Around my time as a Debian project leader, the two projects got closed together again. So I, I insisted a lot on the discourse that Debian is primarily about free software and one of the main goal of Debian is creating a completely free operating system, but Debian also has some non free parts that needs to be explicitly enabled by user to be used.

And the FSF didn't like that part. So this was one of the reasons. And then there were another big reason. For the FSF for not listing Debian as a free distribution is firmware. So Debian historically took a decision that essentially the core firmware that you need to make some of your devices run was okay in some part of the archive and the FSF was not okay with that.

We got closer at the time, and at some point, FSF sent out a statement saying, well, Debian is not a community free distribution yet, but they made a lot of progress toward that goal, and that was a big achievement at the time.

Jonathan: I am personally not a fan of the line in the sand in the place that FSF has drawn it there.

Years ago, we had somebody from FSF on the show, and I tried to make the point It's actually rather ironic because all of these devices have firmware on them And it's actually a little bit more user friendly if you give the user the ability to replace that firmware as opposed to just baking It in forever and the FSF has the exact opposite take on that that no No, if the If the user has to touch the firmware, that's when it's a problem.

But if the firmware is baked in forever, that's, that respects your freedom. And I just think that's the weirdest take.

Stefano: Yeah, I think I personally agree, but I, so one point that I made to the FSF specifically, so I attended LibrePlanet, the main FSF conference as Debian the time. And one argument I use is that knowing that there is a problem, Part of software, which is completely free software, and then, you know, a red line separating that from a part of non free software is actually an opportunity to teach users about the importance of their user freedoms.

So saying, okay, there is software that is not enabled by default, and it's firmware that you need. To have your specific device run. You can enable it, but we take that moment in which you need to enable it as a teaching moment. We teach users that, well, you can enable it, but then here you don't know what the software is doing or what the firmware is doing.

You cannot change it or that kind of stuff. It can be something useful for the free software movement to, you know, to use rather than just pretending that non free software does not exist. Because essentially the results of some of the takes of the FSF is just pretending that. Yes,

Jonathan: sad but true. Have things changed very recently in Debian?

I, I, I don't remember the details, but I've, I've had at least one person tell me that there are some things that are now a little bit easier in Debian, like getting, I think it was getting NVIDIA. Drivers working that that recently changed in it.

Stefano: So what they changed recently is that essentially before there was a Essentially a big part of the archive which contains either known free software or software depending on non free software And essentially to get for instance just a non free firmware you had to enable all that software So that means that you enable that to get your firmware, but then suddenly you are exposed to all the rest of the software Non free software that's there like I don't know, Acrobat Reader or you name it.

And what has changed recently is that there is a specific section of the archive to just enable the non free firmware part.

Jonathan: Which I

Stefano: think is a good improvement.

Jonathan: Yeah, yeah.

Stefano: I mean, I also have here, unfortunately, a laptop with an Nvidia chipset. So I could say that technically installing that kind of proprietary model, it's technically easy.

But I don't know if it has changed recently or not.

Jonathan: Yeah, does that change and some of the other things that Debian has done recently to make it easier to use? Does that steal some of the thunder you think from? Distros like Ubuntu and Pop! OS?

Stefano: I think it's different targets. So I, so I, I haven't been using Ubuntu for, for many, many years.

So I don't know what's the state of, you know, usability and and of those, let's say desktop distributions, but I think Debian really is a different target. So it's, it's a perfectly fine distribution for the desktop. I'm using it myself here and it works just fine. But I think the main target is being like a foundational distribution for, well, servers, of course, everything that is in the cloud.

A while ago, I was checking and in the major public, so called public clouds, Debian was the most installed VM.

Dan: It

Stefano: was a few years back, but, and that's really, you know, that's a key thing. Because to get your. Autonomy, for instance, if you want to self post stuff, being very easy to use there is really, really important.

Jonathan: Makes sense. All right. I know I am itching to, and Dan is as well, that is to get into the idea of software heritage. And so what, give us the, give us the overview to start with. What, what is this project? What is it looking to accomplish?

Stefano: Okay, so it's a project that started 10 years ago now. I co founded it with my colleague Roberto Di Cosmo and it was initially supported by INRIA, which is a national research center here in France for computer science and UNESCO for, you know, preserving important stuff for humanity.

And the key idea for this software is to retrieve Preserve in the very long term and share with everyone the entire body of software that is available in source code form that of course is a super set. So it's a larger set of all of free software. And the starting idea was that essentially. asking ourselves, well, is anyone archiving free software so that it does not get lost for future generations?

And when we first asked ourselves that question, we, we thought, well, of course, someone should be doing it. It would be crazy if nobody is right. And actually nobody was doing that. And that was a huge surprise for us. So we, we've, there were a number of other initiatives, including by UNESCO for preserving the executable form of software, like for doing retro gaming.

Or also for making sure that images stored in some proprietary formats, which are only implemented by some software 20 years ago, will still be viewable 20 years from now. So that kind of initiative existed already, but the preservation of source code was not a thing, at least not at the scale that we were imagining.

And so we set to actually create this this project. And as it happens, while we were thinking about that, Turned out that it can be useful for many different use cases. You have preservation for just, you know, not losing all the important knowledge that is embedded in source code. Do you have a bunch of research use cases?

Like I'm a researcher myself, and I've always been envious of physicians that create these? Yeah. amazing infrastructures like, you know the CERN or the very large telescope in the, in the Atacama desert. And I wanted to create something similar for computer scientists tuning software. A place where scientists with an hypothesis about how software is developed can go, run their experiment.

And then go away and do something else. And then finally, you have also a bunch of important industrial applications like create, helping with creating software bill of materials, of indexing all the software components that are in your smartphone or in your IoT devices. It's something that was not easy before and that's something that software editors can help with.

Jonathan: So, I'm going to ask a bit of a troll question, but it might be one that people are actually wondering. And that is, why do we need this? We have GitHub.

Stefano: That's such a serious question. You couldn't imagine. So another big field of use for software is open science. So, open science is the idea that all artifacts related to science should be open.

The papers themselves. Software created for scientific experiments, data, should all be open. And it's amazing how many scientific papers out there, when they want to tell you where the software used for that experiment is, they just add a GitHub link. Which is, which is a great development platform, aside from the fact that it's not free software itself, of course.

But The repositories on GitHub disappear

Dan: with

Stefano: very, very quickly. There are a number of scientific papers that that studied that that try to retrieve software associated to papers in any scientific fields. And you literally get the answer like, I'm sorry, the dog ate. The the homework that kind of answer and say, well, that was a good time owned by a student that is no longer it's been longer with us for the past five years.

We don't know where the code is. So that's why you need that sort of archive.

Jonathan: So when, when software, when you guys archive a piece of software, do you get like the entire get history? Or is it just a snapshot of where something is at?

Stefano: Yeah, so the short pitch I mentioned before is that we archive the source code, but in reality, the form of code that we archive is indeed development history.

And that is really, really important. So it means that essentially, if a Git repository disappears, we can recreate it from the archive, obtaining a hash that matches the one of the repositories that disappeared. And it's actually not only Git, it's multiple version control systems, and it's not only GitHub, for sure, All the different software platforms out there, and there are a number of very interesting use cases.

We are seeing these days maintainers of our repository hosted on public get rewriting the history to retroactively change the license and to make sure that the code that today is free software will no longer be advertised as free software in the future. So when this kind of stuff starts to happen, the importance of an archive is even greater.

Because you need to, you know, to empower users to show that that code was under the GPL one month ago or two years ago. So it means it's still under the GPL today.

Dan: So

Stefano: that are a number of use cases we are seeing. So people are really using the archive for defending themselves. Against, against these strategies of, you know, stripping rights that were there in the past.

Jonathan: So there is a, this is something that, that really concerns me with all of these data archiving things. And that is that there are laws out there, especially in the European Union, that give a right to be, I, I'll, I'll, I'll use the term right to be forgotten. That's not technically the right term, but that's one that's easy to be understood.

How does that work? When you have, well, for one thing, when you're trying to archive all of these things, but also, like, just mechanically speaking, when we're talking about Git, how does it work if someone says, I want to be removed from your database, but it's like, but your name is on half of these Git commits on this particular project.

Like, that's gotta be a nightmare.

Stefano: I mean, that is a tricky issue for all archives out there. And of course, all the applicable laws in this case in Europe, because the data, the data owner is in the world, the data, the DPO, the Data Protection Office is the one from India. And of course, if there are takedown requests, that request to remove code from the archive, that should not remain public anymore.

Oh,

Jonathan: so you, you could go in and you know, this person is now John Doe as opposed to whatever, you know, a real name. If someone wants their real name to be forgotten.

Stefano: people for reason for removing code from the archive that are legitimate under the law, they are implemented and people can remove their code.

And then ideally, if it is free software, the ideal way would be people republishing the code under some other format. And we are kind of like, yeah,

Jonathan: it's, it's such a, it's such a tricky thing. I, I guess the, the place that I come down on this is if something was made intentionally made public, I don't think there should be a right to be forgotten.

I think that is incompatible with reality.

Stefano: I mean, I don't have a specific opinion on that topic. What I say is that in some cases there is, there is to consider also the right of, you know, free software users to not lose access to some piece of free software. Yes. So, and how to balance those two rights in some cases can be very complicated.

Jonathan: Yeah. Dan, did you want to jump in here? I was about to send you a message on the back chat and he beat me to it.

Dan: That's okay. We're, we're revealing the inner workings of the show. Much like maybe we can archive them in future. Maybe Stefano can archive the inner workings of the show. Yeah. So I'm curious about how some of this works technically, really.

I suppose I've been reading through your documentation, which is excellent, by the way. And I've been reading about some really cool acronyms like sword got me. I was like, Ooh, they've got a tool called sword. That sounds really cool. So can you tell us a little bit about the kind of actual physical process of doing this?

So how does it work?

Stefano: So essentially what Git does, and I'm taking the example of Git because essentially models of most other version control system out there can be mapped to Git, is essentially creating a big graph. So your Git repository is itself a graph and each object that you store in Git as an identifier can be a file, can be a directory, can be a commit, can be a release.

And essentially what we're doing is that We are either crawling, if they're available from some public development platform, or receiving requests for archiving specific repositories, as you would do for a web page on the Internet Archive, for instance. And essentially, we are storing all the objects we retrieve from the version control system into a unified model.

This way, if a file that is in your repository is also present in a gazillion different repositories out there, we store it only once. Same thing with a single commit, and same thing if you have one million forks. Of the exact same version of the Linux kernel git repository while you store it on the watch.

You don't need to store it 1, 000, 000, 000 times. And so essentially it's a fully duplicated draft data model keeping information about where stuff you have archived. Come from in the in the public Internet and something else. So it's a protocol that we did not create ourselves. It's a protocol used by open access and platforms for store papers.

It's an interchange format so that if you deposit a paper on a specific open access archive, it can be pushed. To a different archive, and we support that for integration with open access platforms, for instance, scientists, depositing paper on those platforms can say, hey, by the way, I have an associated software in source code form for this paper.

And when it is recognized as being. source code, it's also pushed to the Software Heritage Archive via this work protocol. That's super useful because you, you can know that some specific software you have archived is associated to some scientific paper and then keep the link between the two.

Dan: Oh wow, that's excellent.

Is there a mass import of stuff from GitHub? How do projects get, do they have to request to be included in the archive? Or do you proactively kind of go out and say, these are all public? In some platforms, we archive all publicly available Packages from pipe high or indeed repository from the public guitar.

Stefano: In other cases, users request that we explicitly archive some specific forges like a GitLab instance, also operated by a public administration or operated by a research institution. People from those institutions or third party users of those repositories can request that we archive all those GitLab instances, that GitLab instance in full.

Dan: Wow, excellent. And where, I imagine physically there's quite a lot of data here, so do you have masses of cloud servers and stuff out there that you put the stuff on? Are you duplicating it all over the place so it can never be lost?

Stefano: So in terms of data size, it's about two petabytes of data. So it's big, but it's not, you know, a video of video archive big.

And we have, of course, multiple copies because the only way you can keep something safe is to have multiple copies of it. So we, we make a distinction between copies that we operate ourselves, like to protect against the virus. Discs failing or that kind of stuff and independent mirrors that even if we wanted to, we couldn't destroy ourselves.

So we have three copies of ourselves that we operate. One is on bare metal, also here in France. The Inria data center is a couple of racks full of, completely full, I think, to this day. And Two other copies on public cloud that offer in kind storage space. One is Azure from Microsoft and one is Amazon S3.

And then we have independent mirrors. So one full mirror is already operating, operating from Italy. And we're in the process of setting up another mirror in Germany and another one in Spain.

Dan: Oh, wow. Excellent. And talking about the legal side of this as well, you've mentioned a bit about licensing and stuff.

That really interests me. What what set of licenses do you accept? Is it the stuff that relates to, say, the OSI approved licenses and things like that?

Stefano: So right now is really is much larger than that. So everything that is public is archived. And in some cases you can archive stuff for Public interest, even if the license is not speaking for software, but in addition to that, we do some, let's say, mining of the stuff.

We have archives that we recognize the license of the software. We have archives. So you can actually query that guy. Not in batch mode, but for a specific project, you can see what the license is of that software. And if you only care about, you know, OSI compliant licenses or FSF compliant licenses. We have the metadata to actually support that kind of searches.

Dan: Oh, wow. That sounds, that sounds brilliant. So how many people are involved in this? Because obviously, I know you started off with just two people, I believe. So is there, is there now a growing team?

Stefano: Yeah, that is correct. So in, in terms of the different sort of roles, but so we are about 20 people operating on this, and I should mention also that is a completely nonprofit initiative.

So there is no company associated with that. It's really for the public good. And you have all sorts of different roles. So you have software developers. So we have operators, system administrators. We have a bunch of researchers that are also doing research on this kind of stuff. And that is the part I'm mostly focusing on these days.

Okay.

Dan: And how do you keep it going financially then? Is it through donations and so on?

Stefano: So the the source of funding is diversified because it's the best strategy for financing any non profit initiative So we do accept donations, but the bulk of the funding comes from sponsors that are in part public sponsors like I've already mentioned inria, but there are also a bunch of other institutions that can be specific universities or public bodies around europe And we also have a few of the tech sponsors that finance a part of the project, but it's quite diversified.

So we are not dominated by any single sponsor and that's a very healthy position to be.

Dan: And on the subject of diversity, sorry, I'm jumping in again, but on the subject of diversity you've got some stuff to do with DEI. Dave, do you want Diversity, equality, and inclusion and trends and so on that you've been looking at through the research.

Can you tell us any more about that?

Stefano: Yeah. So as part of my, so here changing hat a little bit. So I'm a researcher myself and I'm a lot of the stuff I'm researching is related to free software. And of course I'm using the a lot of the public data that software heritage is producing for, for my kind of research, which tend to be very large scale.

So some of the studies I've done in the past are related to diversity in at least a couple of sense, gender diversity, and. Geographic origin, diversity, and essentially you can mine the software heritage archive to see who is committing code and seeing what is the ratio of male contributors, female contributors, and doubt that is evolving over time.

And you have a very long window of observation because we have not, well, we started archiving in 2015, but of course, version control system preserve history. So, and you have the timestamps. And so you can observe essentially. Something like 50 years of development history as recorded by version control systems.

So in one of the study I did a few years ago, we observed the ratio of yearly contribution coming from women. And as you can imagine, in the entire body of public code out there, it's pretty abysmal. So it essentially, overall, overall, 5%. But the trend. was actually growing yearly, and it reached for the first time, just before the pandemic, to 10 percent of the yearly contribution coming from women.

And unfortunately, the COVID pandemic changed the trend. And we have actually proved in a recent paper that it was not just a coincidence. And actually, the COVID pandemic caused the inability of women to contribute to free software, or at least to public code in general. This is not surprising, because essentially, The pandemic has heightened some of the interesting discrimination that exists in households and whatnot.

But we've actually proven that it has been the case also for public local contributions.

Jonathan: So in, in that particular realm, one of the things that's always interested me is, and I kind of see this as a strength of open source and the way things are done, but like a lot of commits just come with an email address and that is sort of inherently non gendered, right?

I, I'm just. Sort of curious, how do you go from all I have is an email address or even all I have is a name? and And it's it's sort of dangerous these days to even make an assumption about gender from a name how do you how do you go from that to determining whether a commit comes from a man or a woman?

Stefano: Yeah, absolutely. So there is essentially a spectrum of the kind of methods you can apply to this kind of studies Which depends on the size of your sample. So if your sample is small enough The right way to do is just go and interview people and you ask themselves which gender, what gender you identify with.

But here we're talking about tens of millions of authors, so you cannot possibly go to ask tens of millions of authors and you shouldn't do that either because they've not opted in to be interviewed or that kind of stuff. And so the approach that you use, you actually recognize not from the email, because it's usually not very informative, but from the full name, you recognize their there's a number of characteristics can be the gender, or it can be geographic origin, and you validate that with external data sets.

For instance, people can self declare. The gender on the top metadata or their own pages, but you also have some ground truth data set. For instance, we obtain a data set from the Olympic committee, essentially, with all the statistic of all the athletes that have ever participated into the Olympics with their name, their origin, their gender, the country that we're coming from, and you can use that to essentially.

Test if your automatic tools for detecting gender or geographic origin is correct or not. Of course, it's not as good as interviewing people, but for analyzing such huge data set in aggregate form is, is good.

Jonathan: Yeah. Interesting. What, what are some of the other fascinating sort of scientific queries that people are doing?

What, what, so we kind of dug into gender and you've mentioned a geographic area, which is super interesting to think about. So if you want to speak on that briefly, you can, but then I'm curious, like, what, what else is there that people are looking for? In, in the status set.

Stefano: A quick thing about geographic origin, because it was really fascinating that we were being able to observe essentially the history of technology through the lens of geographic origin, like in the early sixties, you see a lot of contributions coming from North America.

That was the early year, next day. And then you see later in Europe coming up and in more recent years China and other countries coming up again. So this is really, really fascinating, but so more than that, there are a bunch of other things. So one thing is software evolution. Like, you would be sort of surprising, or maybe not, to know that the ratio at which new code is published has been exponentially growing for more than 20 years now.

And is stable in both the number of new commits that you see coming up, or of new files that get published into public code. So there is exponential growth of code that is, at the very least, Publicly available and a big part of that is also called is actually free software. So that's great. So the humanity is producing more and more Digital commons and releasing it to the public for others to see that's another thing and you also have a bunch of Studies related to security for example You can observe how vulnerabilities propagate from one repository to another.

You can observe vulnerabilities that are fixed in an upstream repository, but not integrated into downstream forks. That's another big example. And then you have a bunch of technical advancements that you need to create, essentially, to be able to analyze data at this scale. And so we've been working a lot on compression techniques.

So, can we make the entire body of software that we have archived fit on your laptop one day using specific compression techniques? Maybe yes, maybe no. Can we make the entire development graph fit in the memory of your single server that you have at home? The answer to this is yes, for instance. We have developed techniques to do that.

Based on compression techniques. So graph compression techniques. Sorry. So you see, it's really a huge ramification of thing that you can do. You can study using this data and you have to develop yourself to make studying these data actually possible.

Jonathan: Yeah. Super interesting. So what, what about.

Documentations like I'm curious about the things that are that are included in this kind of repository and is documentation in scope. So obviously, in some cases, documentation is just part of the source code, you know, it's part of the get repository. And I would imagine that that sort of gets hoovered up automatically, but brought more broadly than that is is documentation.

And I'll get to a more specific question about this in a second. But like website content, is that is that in scope? Is that something you're thinking about?

Stefano: Yeah, so the essentially the choice in software heritage was to at least initially archive what is stored in version control system repositories that are publicly available and we select development platforms that are usually used to develop software, but you're absolutely right that incidentally archive a lot more than that per capita documentation, archive essentially taking the example of guitar, Everything that people could put on GitHub.

And there is a lot of trash in there. Okay, you have, of course, all your random website. You have people using GitHub just to create their portfolio of contributions to look good on, you know, on the market. Yeah, that sort of thing. So the solution there that we well, our approach is being we archive everything.

And then we develop techniques to recognize what's what. For instance, we develop classifiers to decide whether automatically whether a repository is a website, is a data set, is a is some source code. And you get from there. So downstream users of this data can search what they care about. And only consider that in your analysis, for instance.

Jonathan: So you could, someone could put together a query and say, show me all of the websites that are you know, hosted on GitHub pages, you know, or show me all of the places where people use GitHub to manage their website as a, you know, as you just do a Git pull to be able to do updates to their website.

Stefano: So that is correct.

That said, we are not operating ourselves a cloud service that can, you know, at the CPU capacity or memory capacity or storage capacity to execute queries. So we're essentially the storage and computing resources of the project are usually mostly for their cutting part, but we create data sets, for instance, that.

Researchers can actually use and do these kind of queries on their own hardware. Sure.

Jonathan: Sure makes sense So this this leads into sort of an obvious next question and that is Are you do you anticipate are you planning on changes for what exactly? The the archive holds, you know, are you going to expand into we want to archive more web pages?

Stefano: Web page is not really but we are getting into the territory of for instance Issues, bugs discussions in pull request, because essentially if you take like an archivist or a museum curator, curator approach to rebuild the history of a specific software project, especially a community one, all that kind of things are really, really relevant.

We decided not to archive that kind of stuff in the beginning because a common data model for that stuff looked way more complicated than the common data model just for the code. And but we are starting to look into that right now. So archiving issues are kind of a pull request. Discussions is something we're looking into right now.

Jonathan: So something something along the lines of archiving the Linux kernel mailing list.

Stefano: Well, yes, I guess so. Not really. We're not really specifically looking into mailing list. We're first focusing on what is Part of the common development platforms out there, and if you look at most of the Gitforges out there the GitHub one, but also all the GitLab or GTI instances usually support at least issues, discussion in issues and discussion associated to pull requests.

So that's what we're starting to look into.

Jonathan: Yeah, I imagine at some point though that's going to have to include emails because there are still Quite a few projects actually that their discussion and their issues and their pull requests all happen over email and their mailing list

Stefano: Yeah, they are correct.

Jonathan: Yeah,

Stefano: but to be honest, that's easier to handle than you know All the different pull request discussion and whatnot, but they are more difficult to link To be interesting objects in the archive. More difficult to link to the commits and what not.

Jonathan: Right, we already have tools to be able to to collect emails all in one place.

Stefano: Yeah, and so actually Jemaine, that was a, I don't know, has it been restarted? Because Jemaine was a precious resource, but at some point it was discontinued. I don't know if it has been restarted since. Okay, anyway

Jonathan: so in the kind of in the pre show, maybe before Stefano joined even I, I was talking to Dan about the internet archive and, you know, that is a super useful resource, but they are also sort of seeing some trouble and, and it, it kind of occurs to me that you guys are in, in a very small slice of it are a backup to what the internet archive does.

Stefano: Yeah, so the archival model is pretty different in the sense that the Internet Archive takes a traditional archivist model in which you archive stuff and you put the stuff you archive in a box, and you can retrieve that box to obtain this website, this webpage, or this zip file that you have archived, while we take a more unified model in which everything is intertwined.

So that's the big difference in terms of archival model. But other than that, yes, it's a precious resource. I've seen some of their str technical struggle due to attacks and whatnot in recent years, and that was really scary

Jonathan: even for us. Well, yeah, so I, I genuinely, I, I hope so. Like, I hope you guys are looking at that and taking notes of there are trolls out there, there are these problems out there, and it's probably going to come to your door at some point.

Stefano: Yeah, I mean, it's, it's, it's only an uphill battle, right? Because we're, we're, we're fairly size. Team, like 20 people is not nothing, but it's not the fire capacity of the huge companies out there that have, you know, if they have an attack, they can just deploy 50 people to defend against that attack. So, but, so we do what we can and we take, of course, security very seriously.

Jonathan: Yeah. What do you think the history, excuse me, the, the future of archiving looks like? Like, so this is obviously something that's very important, I would say to humanity. Both the internet archive and software heritage, both of what you're doing. And there are, there are other archives out there. I'm, I'm not sure of the details of all of them, but I know there are some other ones out there.

What is, what does this look like in the future? You know, are we, are we, are we getting to the point to where the things that are being archived are just going to become so overwhelmingly huge that it's not, it's not practical for a small organization like yours to be able to do it are we looking towards a future where maybe governments need to step in and.

Budget towards this.

Stefano: The digital preservation community is something really fascinating. So there is a huge community of both practitioners, companies working on that researchers doing the groundwork for how to do that properly. And it's fascinating. Everyone focuses on different, let's say, kind of artifacts.

And the question of size is really interesting. And I was surprised that one of the first. questions we got from the digital preservation community when we present a software heritage was indeed about sustainability. And sustainability is not only funding that we have already discussed, but there's also sustainability in terms of sizing.

So is it doable to archive what you're archiving? I think with software source code, we are in a kind of a sweet spot in the sense that very precious human knowledge, Takes a lot of time to produce, usually brain time, and then is, let's say, serialized into a fairly compact representation. Like, every developer knows that they can spend a day finding a bug, then result in a single line change in a piece of code.

So in comparison to, like, archiving videos, public videos, where you can do a 10 second HD video of a cat and get a gigabyte of data, I don't know, maybe not a gigabyte, but a lot of data. We are in a safer spot than, well, let's say archiving the other kind of format. But still there is a question of how you will find something interesting into the trove of the 300 million projects that we have already archived today.

And we're working with archivists, curators, and to actually enable this kind of this kind of research. But we see ourselves as a necessary building block for doing that kind of work in the future, rather than being the ones that should, you know, answer that right now before, instead of letting the expert of that part work on it later.

Dan: So I'm interested in security as well. You talked about. The security of the archive itself, but I'm thinking of the useful, there'd be really good use case for the archive would be tracing security vulnerabilities through different pieces of software and how they've affected it over time. Is anyone doing that kind of thing with it?

Stefano: Yeah, as a matter of fact, we're working on that right now. And so it's something that is already done via other techniques. So for instance, you already have a bunch of. even, you know industry products that do that based on package metadata. The way we work on that is looking at the commit graph. And we're, we're trying to see if the increased granularity, which you can do can help finding vulnerabilities in projects that might be unaware of that.

But there are a bunch of other secure, relevant security use cases for the archive. Like there are people that are trying to reproduce vulnerabilities. So there was a vulnerability 10 years ago in a specific piece of open source software. Can we reproduce it today to see if the resultant time were sound?

Or even just for historical reasons, like the previous talk, right? And that are kind of activities in which a software archive, a comprehensive software archive like Software Heritage can help. Because you need the vulnerability, you need all its dependencies. And, and, and only with a comprehensive archive, you can know finally all you need for that.

Dan: And I heard you say 300 million projects there. I mean, I'm sure that's a ballpark figure, but I was going to ask, do you know how many how many projects are actually being archived right now?

Stefano: So we do know. And so project is kind of a blurry term that we use in, in for software a lot without Defining it properly because it's very hard to define.

And so the figure I've mentioned is if you want the number of URLs that we archive, where a URL can be the URL of a specific Git repository, or the identifier of a specific package in some repository. Package manager repository like pipe or NPM or or or whatnot. And we have 300, I think 30 millions such software origins that, the way we call them archived right now.

Dan: Wow, that's amazing. Is there a a target that you'd like to get to at some point? Like is, you know, we want to get to a billion or we, no,

Stefano: not really. There is no target question. We, we get a lot, and it's really hard to answer is that, can you estimate how much you're missing? And the answer is that noise.

That's very hard to do.

Jonathan: So I am, I am listening to this and I'm, I'm, there's a thought that's going through my head. And that is, you guys saw this need to archive source code. And you jumped in and it's there. And so it's, it's taken care of now. What is not taken care of, right? And so I'm thinking of things like you know, the old GeoCities.

Lots of people had little websites put on GeoCities, and there were some efforts to archive those. And I think Perhaps all of them were archived, or at least the majority of them. You've got the internet archive archiving things that they can get to from the internet, but then you have places like YouTube where people are archiving things to YouTube.

You can imagine a future where. Google pulls the plug on YouTube because they, they do that sort of thing. And then I'm thinking of even things like, you know, there are, there are TV shows and movies from years and years and years ago that never got digitized and there were fires in the you know, in the place where the studio was storing their master copy of it.

And there are some of those things that, you know, as far as anybody knows, they're just gone forever. And it brings to mind this question of like, what's the next frontier or, or what, what are we missing? Is there some place out there that. There needs to be an archiving effort that's, that's not there. You know, do, do you have any, any feeling of something that, boy, it would be nice if the, if this was archived in a, in a safer way than it is now.

Stefano: Yeah. So let me stick to my comfort zone or focusing on computer stuff. And what we discussed thus far is essentially. Public code that we archive, but there is a lot of important code that is not public, not because it's proprietary, but just because maybe it's not even digitized, it can be on a punch card.

Okay. And we, in computer science, we're in computing in general, we are still in a pretty lucky let's say period in which a lot of the funding fathers and funding mothers of computer science are still around and it's the time to go see them and see if they have that floppy disk or that punch card or that.

magnetic tape still in a drawer and work with them to digitize that stuff and archive it. We have done this for a number, for a very small number of historical software, but it's really time we do that for essentially all the important historic software that have made the history of the world before it gets lost because People just are no longer with us anymore, and nobody else knows that there is something important in that doorway.

Jonathan: Yeah, so you talk about not proprietary software. I think there's a, an argument to be made that proprietary software, really, it would be ideal to have archives of that source code as well. And just, you mentioned earlier on retro computing and retro gaming, like, I just, I kind of shudder to think, but I wonder, like, how many of the 8 bit and 16 bit era games is the source code just irrevocably lost?

Stefano: Yeah, and I've been talking with a lot of people developing retro gaming emulators, so free software retro gaming emulators, but they have to deal with, okay, how do we support this specific game that is, you know, From this specific realm, and they get a lot of issues in which in some cases they cannot do that.

And it's even hard in most cases to identify who are the right owners of that piece of code. So maybe you have the bytes, but you don't know legally who has the rights on that piece of software. And I mean, don't get me started on proprietary software. So for me, we could even as well expropriate all the proprietary software, make it free software and archive it all.

But it doesn't look like we're in a world where this is going to happen anytime soon. But the day it happens I want to archive it for sure.

Jonathan: Yeah. I don't know.

Stefano: On the other hand, it's more complicated for companies, but for code written by individuals, if it is publicly available at some point, the copyright on it will expire.

It will take decades, but at some point it will expire. So it is indeed important to archive that as well. You usually cannot do that with a public archive like Software Heritage, but you can imagine an escrow schemes in which it is deposited somewhere and the day it becomes the day copyright expires is automatically published and automatically archived.

Jonathan: Yeah, you know, I think, I think a lot of people are with the resurgence of retrocomputing and retro gaming. I think a lot of people are sort of starting to think about that. And every time there's another story about, you know, how something is just lost to time. I think more and more people are sort of becoming aware of the problem.

And I don't know that we're, I don't know that we're ever going to completely solve it, but just having more people paying attention to it is, is a win at least.

Stefano: Absolutely. I mean, people think about that. I mean, people think that, you know, GitHub is not an archive or YouTube is not an archive, you know, sending out that message, at least thinking about where I should put this stuff to maximize my chances that it will be available in the long run.

This is a very important message to send out.

Jonathan: And even, even the fact that you know, in the United States, when things started entering public domain again, that the, the, the companies, the, the, the usual suspects of companies that don't like that, they did not try to pass another, you know, get another law passed that would extend copyright, you know, another 20 years.

And I, I did some reading on, I've done some reading on that several times, and the statement that they made, that those companies made was, There is enough awareness of the issue now that they didn't feel like that they could make it happen. That there would be so much pushback from the, so much outcry from the public that it wasn't worth even attempting it.

And that's, it's, it's a huge win.

Stefano: Okay, that's interesting. So I, I think we all have in mind that there's a diagram on, on Wikipedia on the evolution, the periodic evolution of copyright terms over time. And so you're saying the, pulling this out this day would be more difficult than in the past? Well, I hope you're right.

I'm skeptical, but I hope you're right.

Jonathan: Well, so in the United States, at least things are entering public domain. And so we have public domain and it was not that very many years ago that nothing was entering public domain. Right. And, and so I, I saw an interview where one of the CEOs from one of the big companies that was behind a lot of those laws, in fact, The, the law was, was named after them, not, you know, not on the, on the actual law, but people were referred to it as the, the nickname that it was named after this company.

And there was an interview made where they, they made the statement that there would be enough public outcry against trying to do this again, that they opted not to. They just didn't think it could happen. Very interesting. So, yeah. All right. We are. At the bottom of the hour. And there's a couple of questions that I am required to ask you or else I get in trouble.

And you, as a computer science major, I'm sure you've got answers for these and this, what's your favorite text editor and scripting language. So I made the news when at the time I was the Debian Vim maintainer and they switched to Emacs. So I'm still on Emacs these days, but with evil mode. So with the VI key bindings into Emacs.

Stefano: And

Jonathan: scripting language

Stefano: scripting language I would say Python.

Jonathan: Yes. Yeah. That's, that is very, very typical of these days. Do you do anything with OCaml anymore?

Stefano: No, but so that place in my heart has been replaced by Rust. So I'm a big Rust fan. So for everything that needs to really scale, I ditched Python and switched to Rust.

Both of them with bindings and whatnot.

Jonathan: Yeah, yeah, I I finally sat down with the this year's Advent of last year's Advent of code. I finally sat down and started doing some rust work. I made it about 3 days in before everything else started happening and I had to shelve that again, but I will eventually get back to it.

Stefano: I got started with Rust. I also decided to do Advent of Code. It was a few years back.

Jonathan: Yeah. It's a, it's a good way to get started on it. It kind of gives you an excuse. Yeah, exactly. Exactly. All right. Stefano, thank you so much for being here. It was an absolute pleasure. And we will definitely have to have you back in a year or two maybe six months.

I don't know when talk about when things have changed because things are obviously always changing in this particular realm, but we sure appreciate you being here.

Stefano: Thank

Jonathan: you both for the invitation. Yes. Bye. All right. What do you think, Dan?

Dan: A great discussion and such an important project as well, because as, as Stefano was saying there, you know, so much of this stuff could be lost otherwise.

And it's really important that we have, you know, a good archive and they're doing a great job of it, so it seems.

Jonathan: Yeah, you know, and we, we mentioned this a couple of times, but there are, there are things that you read about that have been lost to time. And it's just, it's, it kills me every time I come across one of those, you know, like you, you have a memory from childhood.

Oh, I remember this, this TV show we used to watch when we were little kids. And sometimes you, you go and you search for it and here's all the, here's all the episodes. They've done a re release on DVD, or here's a bunch of the episodes on YouTube. But sometimes you go and you look one of those up and it's, yeah.

Those were never digitized and they were lost in a fire in 1987 and it's like, oh man, just forever gone.

Dan: In the case of the BBC and in the UK, where I am, they actually ran out of tape and started to record over old episodes of things. So they recorded over lots of episodes of Dr. Who. So if you're a Whovian, you know, you'd be upset by that.

They, yeah, they recorded. new episodes of soap operas and things like that over these classic episodes of sci fi and all kinds of stuff just because they were like, we're not going to buy more tape. We've got lots of tape here. We're just going to reuse it, which is such a shame. Yeah, it is. It is.

Jonathan: Yeah. We could, we could dive into that more.

I, I have questions. I've heard that before, but there are, there are just questions that come to mind, but we don't have time for that. So there, is there anything that you want to plug before we let the folks go?

Dan: Yeah, so I'm, I'm heavily involved with Liverpool Makefest, which we've talked about in the past.

It's the biggest maker kind of celebration in the UK. We have about three, 4, 000 people come along. Sometimes we've had up to 6, 000 in the past. Not all at once because they tend to pass through during the day, but I'm just in the early process of sorting stuff out for that this year with my colleagues who run that.

So if you go to liverpoolmakefest. org, you can find all the information on there. And I'd encourage people to to give that a look. And we're coming up on the, it's going to be in July. It's the first Saturday in July. So you've got quite a bit of time. So if anybody fancies coming over from other countries, we have people coming from Canada, we have people coming from all over.

So a distance is not an excuse, Jonathan.

Jonathan: I am, I am making it out to one convention this year and assuming that goes well, maybe next year we'll expand a little bit more, but I don't, I don't know what the future holds for me. So we will just have to see. But yeah, I, I, I've got several friends now that are going to be at FOSDEM.

So that'll, that'll be fun. I'll get reports from everybody on how it went. Alright, I got a couple things that I would like to plug as well. Of course, we have Hackaday, my work there. You got the security column goes live every Friday morning. Make sure and check that out. We appreciate Hackaday as the home of Floss Weekly.

Also have the Untitled Linux Show, which is over at Twit. Dot TV, make sure and check that out as well. If you want more of my ramblings, as we said today. But yeah, we sure appreciate everybody that watches the show and listens, both those that get us live and on the download next week, as we said, we're going to have Doc Searls back as the co host and we're going to talk with someone from CIQ and that's actually the company behind Rocky Linux.

I am very much looking forward to that. We've had Alma lakes on the show. It's only fair that we have rocky links as well. So make sure and be back for that next week. And again, thank you to everybody. And we will see you next week on Floss Weekly.

Episode 816 transcript

15 januari 2025 | min

FLOSS-816

Jonathan: Hey folks, this week Aaron joins me and we talk with Simon and Stefano about the open source AI definition. That's something that was just minted and did not come a moment too soon. It's a super interesting conversation and you don't want to miss it, so stay tuned. This is Floss Weekly, episode 816, recorded Tuesday, January the 14th.

Open source AI. Hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something that we've been teasing, it seems like, for months now. We're actually going to do a deep dive on the open source AI definition, and I've got a co host that is not also part of the team that wrote the definition, and you'll get this joke here in just a second.

But I've got Aaron with me. Welcome, Aaron. Hey, thanks. Thanks for having me and Aaron is going to be sort of our AI optimist. Is that how you described yourself?

Aaron: Yeah, a little bit of an optimist. I don't think it's taking over the world as quickly as people think, although it's doing a lot of crazy things these days.

It seems like But, but I, I don't, especially since I do use it on a daily basis for work and for other stuff I'm working on. I, I, I tend to be, if there's one person in the room, that's a little bit more optimistic, I tend to be that person. So,

Jonathan: yeah. And I've, I've explained this on the show before my, my theory, working theory as of now, is it artificial intelligence falls into.

Sort of the same place that the crypto bubble did, but also the same place that the dot com bubble did. And that is that it's a bubble, people are doing dumb things with it, trying to figure out where it makes sense, but once the bubble bursts, it's going to stick around, and it's going to be something that sort of changes the way the world works, indelibly.

And that's obvious with dot com, and you know, with cryptocurrency, we're still sort of in the process of figuring out what that looks like. I think it's a good analogy. Yeah.

Aaron: 10, 10 years from now, we'll, we'll kind of have to think back like what life was like before we had AI just auto, you know, just assuming that it was there and that we could ask it to do things.

Yeah.

Jonathan: It's, it's the same, it's the same difference of, of worldview that I have with my children. Because I remember before the internet was just always there, and my kids don't. It is always, they've always been connected, you know, in some way, the household has been at least. So it's just totally different.

All right, well, let's let's bring the guys on. So we have Both Simon Phipps, which that is, that is the, the joke that I was making that one of our, one of our co-hosts is also a guest. But we've also got Stefano, Mali, and welcome guys. And you two are the absolute experts, from what I can tell in , what it means for an AI or for an LLM to be open source.

And let's, I guess, go to Stefano first and sort of taking that as the prompt to use an LLM term. Tell us what, what is all of this about?

Stefano: What's oh, thanks for being thanks for

having me. It's it's a pleasure to be here. Yes, sir. It's so, so what is open source AI? Yes, let's start there.

It's. Yeah, it's really not that different from other open and, and open materials or or artifacts that we've been thinking of. We need to have, we want to have freely, free access, freely available access. to all the components and all the pieces that have made that artifact that you have received.

So it sounds, sounded really simple and almost trivial at the very beginning of the conversation that we had almost three years ago, but it was quickly, we quickly realized that All the paradigms that we were used to apply, like the term source just to get started with, didn't really match the technology.

And so we had to study a little bit the issue and we had a long process and the end result that I can today, I can simply say that an open source AI is an AI is a system. It's, it's it's a system that makes you. gives you a free availability to all the pieces that made it. That includes the, the parameters, so the, the results of the training, the, the code that does all the, all the training, the code that produces the training data set and the data itself, when it's possible to distribute that.

And that's, that's it. Sounds simple.

Jonathan: I'm, I'm kind of thinking of the, and I know this is not exactly the same thing, the free software and the OSI, those are two like overlapping but at the same time separate things. But I can't help but think of the, the five freedoms that the Free Software Foundation talks about, and that's the freedom to, when it comes to software, to run, to copy, to distribute, to study, and then to change and improve a piece of software.

Are, are we kind of talking about the same thing when it comes to open source ai or is it, is it a similar sort of overlapping definition? The way that the OSI and the free software definition is l

Stefano: right. Let, lemme start from, from scratch because maybe, maybe the whole concept of open source AI definition throws, throws people off balance because what we have done.

The principles are really the same. And in fact, if you look the, at the document that we have published the OSI published at the end of last year, the, the document really lists all those freedoms that you are talking about, like an AI, and we talk about systems, I can talk about it later, but the, the, an AI pieces, in other words, something that something that produces an output infers an output based on an input is, needs to be made available.

Needs to be able, you as a recipient of that system, need to be able to study, to share it, to understand to to run it you know, to execute, to have those to have those outputs generated for you. And you have to have also the freedom to modify it, change it, change how it works so that others can enjoy the same freedoms that you have received.

So those principles are really the same. What is really missing when you look at the definition of free software? It is that sentence that says the precondition to exercise the freedom to study and the freedom to modify the software is access to the source code. And that's the, the words, those are the words that we were missing when we started looking into AI specifically.

We didn't have a very good way, we didn't have any way of understanding how an AI system can be really studied deeply. Modified to change its behavior, right? That was the, that that's what we have researched for the past almost three years and what we came out with is a, is a way to describe. The equivalent of source code for software and, and which in open the, in the open source definition, it's a definition point number two, it's called the preferred form of making modifications to the, to the code, right?

It's not just the source code, but also instructions on how to build dependencies. Of libraries and versions and, and of course, language compilers and things like that. All those things, knowledge about those pieces need to be shared in order to make, to have access to the source code.

Jonathan: Yeah, that's interesting that you mentioned that.

I'm trying to find the exact my notes on this, but there was a, there was a court case in I believe it was in Germany here recently, where a router manufacturer used some code that was under the GNU, the lesser public license of the LGPL, and they got sued, not because the code itself was missing, but that extra stuff, like the scripts needed for compilation and installation.

were missing from, you know, their, their source code repository. And someone sued them and said, no, you've got to, you've got to provide this under the LGPL as well. And I, I've commented before that every time, like, the GPL, the LGPL, or any of the other, but particularly those two, because they have, like, the strongest copy left protections in them, every time they go to court, I'm kind of, like, I hope this turns out okay Because the there's sort of this nightmare scenario that a court says no no No, this this is a this is a contract not a license and it's not a valid contractor You know, there's various ways that that could go poorly but the german court found that yes, not you know, not only is it a valid is it a valid agreement, but they they It looks like, you know, reading the, between the lines here, it looks like a conclusion was come to before the court case happened, but then the court case kind of rubber stamped it.

But the extra stuff was shared. And so it was it was confirmed that yes, you can force someone to share your compilation steps as a part of the, the source code license. So that's kind of a, it's, it's good to see that confirmed in

Stefano: court, right? It, it's really, it's a really a, a fundamental component.

A fundamental piece of, of the whole movement is to have access to the source code as defined as the, the preferred form in which a programmer would modify the program.

Simon: Mm-hmm .

Stefano: You, you, you know, obfuscated source code is not source code in this, in the context of the open source definition. Missing information about.

Like, build scripts is really not open source code.

Jonathan: Yeah, so, in the, what OSI done, has done, is they've put together a definition for what open source AI, what it has to be, like the minimum requirements. I guess the next step then, or maybe this has already been done, is to begin to produce OSI compliant licenses for AI models?

Is that, is that what's sort of next on the horizon? I'm jumping way to the end with this question, but it just comes to mind and I'm very curious.

Stefano: We, we are, as OSI, we are ready to start evaluating licenses that do not cover squarely or exclusively software. Like historically, the OSI has never Taken into consideration licenses that cover, for example, content music or.

Databases or other things that, that are not necessarily software. But with we are ready to evaluate other sorts of licenses against the, the open source definition, the original one, the 10 points, the 10 points one, are there, there are efforts that we are aware of. Of groups that are writing new documents.

They're not licenses, technically, or they're not necessarily going to be licenses, but they're, they're terms of use and distributions. There are other legal terms that are squarely new and, and cover only specifically parameters data and data sets and, and code also. And, and they're all comprehensive.

That's it. put together. Yes, we're ready to do it. Yeah. To review that

Jonathan: has in, and this is a, this is another sort of strange tangential question, but when we talk about open source beyond just software, what immediately comes to mind is open source hardware. Has OSI been involved with any of the open source hardware efforts, like defining what that looks like, or has that been left to you?

You want to jump in Simon? Yeah. So you have, you have thoughts about open source hardware,

Simon: don't you? Yeah, well, it's, so this came up before Steph's time. There, there is another organization called Open Source Hardware Association, or Oshawa.

Jonathan: Mm-hmm .

Simon: And Oshawa you know, did what frequently happens in open source communities.

They gave us the great. Using our logo as the basis for their logo and in trademark law, that's a big problem because that means you have to ask them very politely not to do that. And and so at that time, which was about 10 years ago, 12 years ago OSI and Oshawa had to reach a legal agreement agreeing that they would deal with open source hardware and we would deal with open source software.

And as a consequence, OSI has never done that. Actually got into defining what open source means in the world of hardware in the same way that the Open Data Institute, ODI, talked about what open data was and OSI again has never got into defining what open data would be. So the open source hardware definition, sorry, the open source AI definition is something of a departure because it's really OSI's first move into something which is not open source.

But I think it was a necessary thing to do because the, the boundary is such a a series of dotted lines that it's very, whereas with hardware, you can tell, well, you know, that's, that's fairly clear, even actually open source hardware is fairly unclear. One of the things I did when I was at Sun in 2006 was release all of the Spark designs for the Spark silicon chips as open source under the GPLV2.

We called it OpenSpark and we released all of the Verilog designs because it turns out that silicon chips are actually software as well. They're just software that's compiled to silicon. So the dividing lines are kind of hazy there as well, but OSI, as an organization doesn't doesn't harbor an opinion about open source hardware and does not produce a definition in that region.

Stefano: Can I jump in because I, I want to augment a little bit this, this part, like, why did we jump in? Why did we feel the urge when software is not we haven't done the same for hardware? I think what Simon just said hardware is programmed by a human. In, in, in some sense, like the chip design is done by a human.

And then it is compiled into Silicon. So you can see the mapping is pretty easily translated into source code being written by human compiled by a compiler into executable code for, for AI. What we noticed was that these systems look a lot like software, but they're not programmed by humans. They.

They apprehend, they learned by themselves, like they, they, they they have capabilities that emerge semi randomly. I don't want to get, I mean, I'm not a technician, I don't understand exactly why, but what I've been told is that these things just start to execute and become they have new capabilities.

They're not programmed. And, and the question. is therefore, how do you fix it? If it's consistently creating issues or spitting out the wrong answers, or, you know, or whatever, you, you know, you want to fix it, you want to change it, you want to give it to others, what is it that you actually want to ask?

It was easy for the hardware piece, it's not that simple, it wasn't easy, it wasn't immediate for the AI piece, that's what triggered it.

Jonathan: Yeah, that makes sense. Before we get any further into this, we really ought to stop and define who we are talking to. And I know that both Stefano and Simon are involved with OSI, and I, I could, honestly could not tell you what each of your exact roles are.

So let's let's go there next. Stefano, I guess. Where, where are you in the OSI org chart?

Stefano: I'm the executive director of the open source initiative. I started three years ago and I'm in Italy now. You're at the top

Jonathan: of the

Stefano: chart. No, no, the board is actually at the top. Well,

Jonathan: okay, that's fair. Simon, where

Simon: do you fall in this?

Well, I was the president of OSI for about a decade, give or take the odd year here or there, and when I quit the board of directors, I had foolishly started some work on open standards and public policy. And there was nobody to carry that work on if I just ran away and disappeared. So OSI hired me to be their director of policy.

And so for the last three years, I've been OSI's director of policy and standards, and then we hired somebody else to do, to look after us policy. So I've been the director. Director of EU policy and standards for the last year or so. And I, I, I actually don't do anything at all to do with AI. I, I, I look after making sure that things like the cyber resilience act doesn't break open source.

I'm making sure that the standards organizations create standards as if open source was real. Those are the two things that I actually do as the day job. But one of the things I'm going to have to do now, Steph made a A very clear statement when he took over as executive director that OSI needed to do something about defining open source in the context of AI.

And I'm now going to be taking that definition forward into the the, the, into Brussels and performing the necessary education to help people understand that, for example, Meta's LLAMA AI system is not open source. because the licensing includes field of use restrictions and other insights drawn from understanding what AI is and working out how we should legislate.

And that's necessary because people have already started writing legislation about AI. I mean, it, it. It may be really young, but there is already on the statute books in Europe, the the artificial intelligence act. And it contains within it an exception for open source AI. So somebody had to define what open source AI means.

And that was, that was Steph.

Stefano: Yeah. No, no, no, no. Wait a second. Wait a second. It wasn't me who defined it. I mean, that's.

Well, that's crucial because, because different from the free software definition and the open source definition itself, this is not the work of a lone person you know, smart or otherwise the coming out of the, of their, of their garage with, with with the sacred text. This was the. This process had to come from the community of AI developers, researchers, lawyers, copyright holders subjects of, of of AI systems, all of these different, different stakeholders had to be consulted and we needed to find a definition that matched what was actually happening.

In the space, providing some, some guidance, of course, and bringing our expertise and experience from 30 plus years of free software. But it was definitely not the work of my work.

Jonathan: We, we absolutely need now to go ask one of the one of the AI image generators to generate the image of Stefano carrying the tablets of stone out of his garage.

Simon: You know, I think that the point Steph's making there is actually really important. Yes. Because some, there has been the question asked, you know, what writers OSI got to define what open source AI is. Sure. And, and this then reads back to the open source definition itself, you know, what writers OSI got to define what open source means.

And the answer is that it's actually, The OSI's role is to collect together the consensus on what it means. That's what we did with the open source definition. There was a definition that came out of Debian that became the open source definition. And over a fairly short space of time it became obvious that That the consensus of the global community was that the open source definition was the the, the, the canonical explanation of how you can tell that a license is an open source license.

And now what Steph has done for the last few years is he has run this exhausting global program where he's held public meetings. He's hired facilitators. He's hired authors and writers. And what's happened is he's asked AI experts and open source experts and people who, who work in, in social development.

He said to them, what, what is open source AI? And over that time, he's gradually evolved what the answer is not by being clever and working it out for himself, but by. Identifying the consensus of this huge crowd of people. And that means that the, the, the, the definition is really a consensus definition in many ways.

Now because the, the field is younger, there are some dissenting voices. You go out and ask anybody who actually works in AI on open source nine out of 10 of them, and I can tell you the name of them. The 10th, but nine out of 10 of them will tell you that this definition is, it's pretty much right.

They might want to nuance some of the words or some of the concepts. And indeed OSI is putting in place a plan to evolve the definition to a, you know, a V 1. 1 or a V2 in the future. But this really is a consensus definition rather than an imposed definition from Steph. There you go. He's

Jonathan: got it.

There's Steph with the open source definition coming out of his garage, that's great. I love just that hair. So, I think, I think we can take a moment and just say, like, on some level, that, that is impressive, right? Like, that's pretty amazing that we could do that. That we could go, hey, here's a cool idea.

Let's, let's have a machine draw a picture for us. And it comes out that well. Like that, it is cool that we live in these times where we could do that. Yeah, can

Stefano: you imagine, can, do you

Jonathan: remember when you had to

Stefano: go to a bookstore to order a book?

Jonathan: I still enjoy going to bookstores, but yes. Me too. Take it. We take it.

Aaron: Yeah. Hey, I wanted to dig in if I could a little bit more into the nuances of what we're dealing with, because one of the things that really interested me two things you know, one is that there could be a dependency on something that lives outside the source code for the thing to work right as a problem statement.

And then the other, er, That it may have self generation capabilities and how do you license them? So I'm just kind of curious if what made me think initially was about yesterday, I discovered and started playing around with whisper from open AI. I'm not sure if you're familiar with it, but it's a.

text to speech translation tool. And you can, you can, it's, it's under the MIT license. So you can go, you know, do whatever you want with it under that license. But I'm just kind of curious, like if there's some examples of things that are already out there where those two things would be problematic or are problematic already.

Stefano: Two things that are problematic. Sorry, I, I wasn't, I'm not sure what, What do you mean? An

Aaron: example of some, of, of some AI tool or, or AI LLM or something that's out there that is. Either self generating, do you have an example of, of something like that, that could be problematic or the other one is where it's totally dependent on things that are outside of the, what could be covered under a license or what could be considered open source and thereby kind of invalidates the idea.

Stefano: Self generate, do you mean self generating as in Skynet? I don't know if

Aaron: I'm going that far, but something where I think you were talking about it, Simon, where it would be difficult to categorize the what was covered under there because it wasn't necessarily programmed by a human.

Stefano: All right, okay, now I understand.

So the case of whisper is interesting because the license. is, is extremely permissive. We're very familiar with it and it covers the parameters, like the weights, the train, the trained weights of that engine. What we don't know about Whisper is how it's been built, how, what kind of training, how, how the training happened, what kind of data sources they use to, to, to, you know, to, to have those results, those pieces are missing.

And that is why we don't consider, we wouldn't consider. Whisper and Open Source AI, despite the fact that the weights are openly and freely available. Because what we have defined in the document, the Open Source AI definition, what's defined in there that is brand new is the preferred form, the definition of preferred form of making modifications to an AI system.

If you want to change the behavior of Whisper, you can, you can, sure, you can fine tune it. You can, you can add layers to it. Or you can modify manually some of the, or randomly poke at at the weights themselves inside the, inside the matrix, you will not have, you will have a much harder, easier time if you knew what kind of data went in there, have a full list of it, the, if you had the training, the training code in order to understand how that training was done, you had all the code that was used to Change the, to generate the training data set because training data, I mean, the original data needs to be massaged, needs to be filtered, needs to be duplicated, tokenized, etc.

Before it can be fed into the training machine. So all of these pieces are required to understand to, to have a fully access to preferred form of making modifications to the, to the, to the code. Sorry, to the system, to the AI. That's the new thing that's in the definition.

Jonathan: So would it be fair to say then that that particular model, some of it is still a black box?

You don't know where it came from, it's just you have this black box kind of artifact that's part of it. And, and to make something in open source AI, we want to get rid of all of those black boxes and be able to see inside of all of them.

Stefano: Yep, that's exactly. Or

Simon: at least know where they came from. You know, so, so you can't, because some of the things that you do to make an AI are ephemeral.

They are transient. What matters more is that you've got a, an adequate description of how it was, how the AI acquired its knowledge that somebody else who is sufficiently experienced could take your recipe and do the same thing. They may not need exactly the same data. But they do need the same recipe that you had for how to train it on, you know, the health data of everyone in an emergency room for a month with these, with, you know, the, these inputs from the equipment and from with these sign offs from the patients and so on.

An expert can take that description and. produce the same transparent box. They don't necessarily need all the exact same data to do that, but that that's a corner case as the general case. Yes, indeed. You know, we don't want any black boxes. We want to know what was, what they were shown in order to be populated.

Jonathan: And I suppose that's sort of a hinge point here, right? Like there are, there are certain to medical is a really good example, right? There are certain times where you would want an LLM. That has been trained on medical data. And the idea of releasing all of that medical source data is just, it's a complete non starter.

Like, legally, you just cannot do it. But you would want that LLM to still be as open as possible and ideally be able to fall under the open source definition of AI. And so that's, that's kind of the corner case that's been the, the sticking point maybe through all of this?

Stefano: Honestly, a lot of the corner cases that have been, that have been, that are circulating We must wait and see, like, I, a lot of the conversations that we're having about these corner cases require new science or they may, they may, they may become obsolete tomorrow. Like a lot of the, a lot of the ideas that we had two or three years ago about the technology, specifically LLMs are starting to go away, like, or, or are becoming less relevant.

So, before we talk about corner cases, I would really love to see more work and more analysis of the actual good examples that we have today of, of groups, research institutions, nonprofit. Hackers that are really releasing datasets with with full instructions on how to build them full code releases.

They're making attempts at, at creating platforms, including hardware descriptions of clusters, training clusters. To, to build AI, AI themselves, like all of these examples, the, the virtuous examples are the ones that we are losing track of. Everyone talks about Lama and then on the other side there is a lot of other groups that are releasing much, very, very compelling technology.

With full access to all of the underlying components and pieces. So respecting that, the concept of preferred form of making modifications to an AI system.

Jonathan: Have you, have you gotten a decent bit of a contact from like the industry? People saying we, we acknowledge the work that you've done and we would like to make changes to make our license or model or, you know, whatever.

Open source compliant. Has there been some reach out?

Stefano: There has rather than the industry, the most the, the most productive and conversations we had in collaboration, we had So we had collaboration with industry, large corporations, small corporations, startups, and research institutions, and non profit groups.

The non profit groups are the ones who have endorsed more happily the definition as it came out, because it really supports that idea of creating a framework for collaboration, a shared understanding of what are the principles of furthering the science. furthering the knowledge on how systems have been built, how you train and therefore how to improve without having to reinvent the wheel.

But from the industry perspective, because of the technology, the way it is built, the way it's of its complexity, the fact that it has To take into account multiple layers of the companies themselves, like they go from the data scientists, but even on the legal departments, just to give you an example, in the legal department for software, copyright experts and maybe patent experts are sufficient, all of a sudden for AI components and pieces, you end up having to involve the whole Expertise of the firm plus consulting consultants from outside, because you have to have export regulation, the, the, the privacy regulation across multiple, multiple countries.

And world like it becoming, it becomes a lot more complex. Companies generally haven't been very happy with the way the definition came out. It's too restrictive for their point of view. Yeah.

Jonathan: And then you've got people on the other side that don't think it's restrictive enough, I'm sure.

Stefano: Right. Yes, there, there are some, some groups who say, yes, we should be asking for more.

Jonathan: The thing that comes to mind with that is that there's nothing in this that would prevent someone from writing a more restrictive license. Right? And I would imagine that you could write a more restrictive license and it would still be considered an open source AI license under these guidelines. So someone could come along and try to, you know, bring the idea of copyleft.

Into into this, it's the, the AI, the open source AI definition. Someone could come along and try to, you know, bring in an, a Fero, Genie, Genie public license. To where, you know, if anyone touches it, even on a website, they need to have a way to be able to get the source. Like, I would imagine that it would be possible to write these now, they're not going to get a whole lot of, of uptake.

But it would be possible to write these license and to to release something, to release an LLM under, you know, a less restrictive or a copy left license, right? Like that's an option, isn't it?

Stefano: It is an option and it's a conscious one. I don't think it's a negative one, to be honest. Like if you. And I don't classify the GNU GPL or the GPL as restrictive licenses.

They're permissive. They just add requirements that you may want or not. They're not really restrictive. So in the same vein, I do think that we It's a good idea to have the possibility to have legal frameworks, legal documents that would let someone like a user downstream to, to say, Hey, you're coming up with this system is spitting out this, this this output like I'm asking for a mortgage and it's consistently telling me that I'm not qualified.

To, to, to have it like, but why can I get access to all the instructions of how it's been built, can I have my experts review it, you know, that kind of stuff is not, is, is good for society and in general let me put it also on the other way. Like, think about the fact that you. We, collectively, have created a lot of content that has been crawled and, and and spidered and archived into, into repositories like Common Crawl, or the Internet Archive, or our code is into Software Heritage and GitHub repositories, right?

That content is now, can be used and, and is being used to train wonderful machines that create images and, and and spit out code. Right? Do we want, as a society, to have the possibility to say, Hey, you're using my code, I want the parameters to go back to me, and under the free, same conditions I gave the code to you, or my pictures.

I don't think it's wrong to think about it that way. It's a choice that we need to allow.

Jonathan: Yeah. And that, that actually kind of touches on a, a much bigger question with, with the, the, the way AI works right now, and there's this sort of legal theory that AI is put, putting data into a large language model, it is so transformative that it pretty much removes the original copyright, right?

So you, you put it, you put it in. And the, the language model trains on it, but does not inherit the copyright of the training data. That's essentially the way that it's being used. I mean, you look at something like Copilot on GitHub. It is trained on a whole bunch of GPL code. But then you can say to Copilot, write me code, and there's no expectation that the code that Copilot writes carries the, the GPL license.

So just as an example, that's the sort of thing that I mean. And I know that there are some there are some legal theories out there that that's not going to survive now We we talked last week with a lawyer and his comment was the genie is out of the bottle And I don't think we can ever put the genie back in the bottle but it is interesting to think that there is there is sort of this push for We should inherit some of the copyright of the original training data.

And I'm, what, I guess, what do you, what is your thought on that? Do you think that's a reasonable thing to think about? Or is the genie just entirely out of the bottle and there's no way to go back?

Stefano: It's, it's not, it's not a, I don't have an easy answer. It's a complicated, complicated and nuanced conversation because, and I have a dual, dual approach to this.

Do that on one hand. First of all, these theories, they are being proven in in American courts. And I'm, I'm saying American courts specifically because there are different in law. The copyright law is fairly uniformly applied around the world. But some things are different in the United States, in Europe, in Great Britain, China, et cetera, et cetera.

So, yes, the, the genie, I, I guess the lawyer was may, may have been referring to the the theory that has been proven by Google Scholar Google Books, actually the when, when if you remember, was it 10 years plus 10 plus years ago, Google started scanning books and making them available. Search available through the content of the books.

They were buying the books and paper, scanning them, doing character recognition and offering those as search engine products. And and they got sued. Google got sued by the American Publisher Association and the publisher, the Google won that case because the judge said basically that what Google was doing was not damaging the copyright holders.

And also being was transformative work. And, and so I believe that Copilot and GitHub and Microsoft are building their cases, their defenses against the lawsuits pretty much on the similar theories. We'll see if that survives the courts. But there is a, there is a different, there is another angle that I keep on thinking.

That the fact that. Collectively, we have created content. Collectively, the society has created content. Every blogger, every podcaster, we are creating content. We're making it available on their permissive licenses, like Creative Commons licenses and others. And we're telling the world, use it, do whatever you want with it.

And now, if we start saying individually, like, you can use it, but not for training, or you can use it not for training under these conditions, Then all of a sudden, the larger copyright hold, the larger corporations who can license content from content providers, large aggregators, et cetera, or they will, they will have an edge.

They will have an advantage because they have the money and the resources. To get access to large quantities of data and groups like that are building their, their, their systems openly in 3d and they want to have access to freely available content, they will have to jump through hoops and license individually from millions of people and copyright holders.

So we, it's a, it's a more complicated. Question in my mind than, than what it looks like at surface.

Jonathan: With some of those being more complicated questions and the way that all of this is still being developed and changing so much, do you foresee the possibility of changing the open source AI definition?

Do you think there's going to be some updates to it where you go back and you say, this needed to be stronger or this doesn't need to be in there?

Stefano: Absolutely. I think we need to, we need to really pay attention to what's happening in, in the space, like in the field, what, what are not only the technologies, how they're evolving and changing, but also how the developers and developers of AI, builders of AI, builders of datasets, how they're behaving, how they're adjusting their.

Their, their tooling, their expectations, how are they, they're, they're releasing. What, where is the collaboration happening and how or what's the potential for collaboration? Where is the safety coming from? All of these habits, we need to watch them. Like, consider the fact that the open source definition appeared more than a decade, almost two decades after the free software definition appeared.

The, and the free software. Definition was basically a statement of principles and in a manifesto that the open source definition is more like a checklist what we have today, a checklist that was based on 20 years of experience, right? So we're basically watching the space as it evolves. More systems are released.

More data sets are released, more, more experience we gain, and we can generalize from there. What we have right now in the open source AI definition version one is a stake in the ground. It's basically like, like a conversation starter, if you want. The place that we can use, like someone was saying, to have conversations with policymakers, with researchers and developers, with corporations.

To say, this is, these are the principles that came out of the, of a large conversation. Where do you disagree? What do you think is coming, is, is wrong? You know, we'll, we'll keep collecting that information. And we're also going to keep watching the data space because that's where I think that we, We need to pay more attention to like we have right now Really been talking about open data as if it's the only and most important thing that that we when we have But the open data alone is not sufficient to describe the complexity of training data sets and training data.

There's more nuance to that.

Aaron: There's so I've got a kind of a reverse question. We talked about this a little bit when Jonathan was talking about protecting creators rights or protecting the rights of the people who's Data, whether it was images or whatever, ever kind of data that was used for training.

Is there, I'm reading through the definition now. Is, is there, because one of the nice things about open source licenses, and we've got probably also distinguished between the definition and the license, but you know, the license does provide protection. At various levels, depending on which license you choose for the creator of the software, in that case to make sure that they get credit, for example, as the code is, is copied by others or used in other projects, et cetera.

Is there also that same protection in the definition, or is that left up to the various licenses that will come that meet the, The, the definition to provide that type of protection.

Stefano: I think it will have to be coming from the legal terms as they get developed.

Simon: So keep in mind that that, that the the credit to the author is a license term. It's one that's permitted by the open source definition, but it's not actually required. Actually a little while ago, somebody, I think it was Jonathan talked about restrictive licenses.

There are no open, there are no restrictive open source licenses. I

Jonathan: knew that was coming. Simon's had his B in his bonnet ever since I said that. And look at him go. There

Simon: are no restrictive open source licenses because a restriction is something that you have to negotiate removal of. And open source licenses do include conditions.

And so I, I say you can use this license if, blah. But that's not a restriction, that's a condition. A restriction is you can't, no, you, you, you guys with, with, who have, don't have white hair, you can't use this software. That's a restriction. And the only way you can get rid of that restriction is to go to the person who owns the copyright and ask them to waive the restriction, probably in exchange for some money.

So open, no all open source licenses are permissive in the, because they contain no restrictions. They only contain conditions and credit to the authors is a condition. And it's an optional condition. There are open source licenses that don't include that condition. And so if there are going to be those sorts of conditions placed on open source AIs, they will have to be licensed terms because they're not predicated by the open source AI definition itself.

Aaron: That's a good point. I the other question I had as you guys were talking, was there ever any thought put into this or maybe it's just outside the bounds and it doesn't apply because it's too high level of of intelligence itself, right? So what happens when we have an AI that, I don't want to go all the way to sentience, but I mean, as, as, as these AI tools and things become more and more intelligent and able to do things on their own, was there any thought in terms of the definition into we need to somehow Accommodate that, or think about that, or do the rights change at that point?

When, when AIs become or, or approach sentience?

Simon: That's got to be your question, because I don't believe that can ever happen. I'm, I'm a weak AI guy. I read Marvin Minsky in the 80s. And I, I believe that the society of mind does not lead to emergent intelligence. I believe that that evil, the evolution of, of machine learning only goes to just below that point.

But, so it's gotta be his question,

Stefano: right? My question, your question. I'll take it. I, I, I like we have this, this this sticker that says Skynet won't be open source . So I love, I love that. But it's just a working theory right now.

Jonathan: Oh, Aaron, I think that is a great question. But I think that's probably also a great answer. Yeah, that's pretty funny. Oh, all right. Let's see. Where, where do we, where do we want to go from here? So, the Do it. Simon told me where one of the bodies was buried. Do I want to do? I want to dig around in that.

Somebody's like, I don't care. I know, Simon. So there's this, there's this thought that in the, so there, there are exceptions for where, you know, sometimes all of the data for whatever reason cannot be provided. And we talked about the medical exception. And there are some others. And is it sort of a challenge to just to explain the data.

Open source definition, because those exceptions are so prominent and are we going to maybe see a version in the future that puts the exceptions sort of in the, in the back half of that section?

Stefano: We might. So, okay, let me put it that way. If I started today, rewriting everything from scratch. With with the help of a few people without having to, you know, go through the whole process and,

and interviewing hundreds of people, et cetera and traveling the world that blah, blah, blah probably the, the wording might.

I think that could change, but honestly, I, I think that we are panicking a little bit and, and we need to, we need to take a little bit of a step back, like really, we're going to release next week, a paper on about data specifically about the issue of data. And because if you put things away, like I said, the open source, an open source AI is one that gives you all the means to understand exactly how it's been built and be able to build something that is.

working exactly like the one that you have received. Knowing that software is, we have decades of experience with, we have understood it well enough that we have now reproducible builds, or at least a theory of it. Like, we can really take source code and rebuild a binary bit by bit exactly the same as exactly equivalent to the one that we have received.

It took us decades to get there with, with AI, with LLMs, with, with with this sort of neural networks technology, we still don't have the science to to do the same thing, to take I've been told by a notable AI developer that releases everything in open source, that they tried to Take the same data set.

Okay. Take one data set, feed it through. I

had an ambulance. I live near a hospital. So ambulances every now and then, I forgot about this issue. So they, they had this one data set and they were feeding the same data set to through two different pipelines into the same cluster split into halves. And they would put these two streams. would produce two different models.

Okay. So it's not, it's not easy. If not, it's not easy. It's not a solved problem on how to replicate a training so that you get an exact identical model model weights. So knowing that this is different than software, and this is a very young discipline, we need to take a little bit of a step back and, and we need to really understand the issue of data.

is that the issue of data is not going to be solved quickly. It requires much deeper understanding that what I was talking about, like we're now thinking of open data versus not. And I give you a very quick example. I was talking to a developer who's been thinking about building a data set that is purely made of.

Copyright free or unencumbered material content. And they were adamant on using public domain movies only. They step into the public domain issue, which is in the United States, public domain for a movie is 70 years after the death of the director in France. A movie goes into public domain 70 years after.

The last person who worked on the movie dies. It's impossible to calculate. So a lot of the, so where is a data set that looks like it's, it's built on public domain material in the United States may not be in public domain in, in in Europe or, or in France, right, or in German. Completely different story.

We need to get a little bit, take a step back and say, okay, well, this is different than source code. We need, we need more. We need to talk about changing policies maybe. Have new laws or build new habits, or change the technology, right? All of these questions are all open, all of these possibilities are open.

Jonathan: One of the, one of the things as we, as we get towards wrapping up, this is something that's kind of been in the back of my mind the whole time. When we talk about open source software, generally all it takes is a computer to be able to make changes and compile. When it comes to open source AI, like the, just the barrier to entry to really play with these things is much higher, you know, in some, in some cases you can do it on a CPU, in some cases a GPU, but like these, these rather on the edge AI models, you, multiple GPUs to be able to do anything interesting with it and I guess that doesn't really change anything as far as what the open source definition says for it, but surely that changes things for how accessible, like, On a practical level, how accessible it is to be able to actually get into and mess with some of these things.

Stefano: Yeah, there is definitely something that changed, like the scale of some of these training is, is prohibitive. But again, in a couple of years, I've seen there's been some difference. And small language models and other technologies. They seem to be making things more accessible. Like if before, if open AI, the company trains on clusters that are worth billions of dollars there are smaller groups that are doing something on tens of thousands of dollars, like, and getting similar results.

And I'm not saying that, you know, they can build something as big as Powerful of OpenAI's GPT and being able to respond that quickly. But, you know, these models, like Mozilla is doing something extremely fascinating with with, with smaller models running inside the browser, even a mobile. So, you know, training versus execution, that's another big conversation.

It's early. It's really early. We need, we need to give it time.

Jonathan: Oh, yeah. We are, we are definitely in the well, even, even faster than Moore's Law, right? So, like, there's this idea with early computing that every, what was it 16 months, it, it, the computing power would double. Some, something like that.

And we're, we're moving even faster than that with AI at this point. So, you know, give it a year or two and just who knows where we'll be as far as the accessibility part of it goes.

Aaron: And it definitely speaks to the reason why something like this is needed in my opinion, even though even though I'm an optimist, I'm not a skeptic.

I'm not, I'm not saying we don't need any, any of these protections and definitions for things. Speaking of definitions, I'm kind of curious, like, you know, you've been saying it's early days, this is, you know, You know, 1. 0, the definition, right, is, is what's out today. By the way, if people are looking for it, it's on open, I think it's opensource.

org slash AI. If they want to check it out so definitely go there and have a read as I've been doing, as we've been talking here. But it's still early days. I'm kind of curious, like, for example, you know, there's, I only see really two definitions in version 1. 0. One is of an AI system and the other one These are definitions in the definition, , but one is of a, of an AI system.

And then the other one is for machine learning. Mm-hmm . And of course AI is is lots of different things, right? Like LLMs and, and different types of things. So I'm kind of curious where, where do you go from here? Will there, where will there be other definitions that you know, you already have to add?

Are there things that, you know, what's the roadmap here of, of the definition? Where do we go?

Stefano: Yeah. Very good point. So. So why did we include the definition of an AI system in there is because we needed to have an anchor to understand what we were talking about, what we were defining three years ago, it wasn't really clear what we were talking about, like, what is it, why, what's different in here.

And so we use the definition of the. OECD, the Organization for Economic Cooperation and Development, which is what's been used also in the Artificial Intelligence Act, very similar definition. And why we're targeting machine learning specifically to talk about the preferred form to make modifications is because the new LLMs, the ones that require training, that have the dependency of data, etc.,

the ones that learn how to magically, those are machine learning systems. So we We said, okay, we don't want to define. The preferred form for everything all at forever at at every time, let's focus on what we know today that needs clarifications. And that's what we, we got it from what we take it from, I mean, what, where do we go from here?

We need in the next year, year and a half, we will need to understand better how groups like LLM 360, LL3i, The Allen Institute for AI, Falcon Foundation, TII, and other groups like this are LLM France. These groups that are releasing software parameters and data sets. As much as possible to the, to the comments into the group, into the, into the, the open source communities to understand how they operate and how they work and what they need in terms of legal frameworks, legal documents, opinions and, and generalize from there.

Jonathan: How soon do you think we'll see the first license that is OSI approved specifically for AI?

Stefano: I know that the Linux Foundation is working on a new license specifically for that. I would love to see even the, the current ones that are, would not pass definition, but I would love to see the debate, like the ones, like the responsible AI license.

I would love to see one of those being submitted, but also would love to see submitted the data licenses, for example, the, like the CD, CDLA, I think it's called. From the Linux Foundation, they have, they have developed a couple of licenses that are suitable for, for, for data sets. It would be a nice exercise to, to start getting, you know, start getting the, the habits but also understanding where the open source definition terms like require clarifications.

Jonathan: All right. Very interesting. Good stuff. We have basically reached the bottom of the hour. Aaron, was there anything that you. Desperately wanted to ask before we let him or we wrap.

Aaron: Yeah, I've got two and they're kind of, they, they, they weren't immediately applicable to the discussion. But I'm curious about their thoughts on this.

So I'm kind of curious if you have any thoughts on what other open source projects can do besides getting more. Developers involved to incorporate some AI tools into other open source programs or existing open source programs. And the ones that come to mind for me, cause I use them all the time, of course, are like Inkscape and GIMP.

I find myself using Photoshop more than I want to these days because I just need a tool that can quickly remove the background and generate a forest behind my image or something. And they do that really well. And I'm just concerned that. People will stop using these great tools that have been in the community for so long because they're now they're missing these AI function, this AI functionality because they don't have the development bench to go build it.

Any, any thoughts on that?

Jonathan: That's interesting.

Stefano: Yeah, it's I, I love to see Colabora. And, and LibreOffice get some summarization stuff. Like Mozilla is doing some very interesting work on that front. I, I think that it's just a matter of time to get the, to get on one hand, developers with skills and understanding these, these tools and how to do things like reducing the size of a model, like something that looks really gigantic that works only on a hardcore platform.

And high level GPUs to make it run on CPUs, for example. Yeah, so that it can be distributed freely by Debian. But also we need a little bit,

speaking of Debian, right? We need to also I think that we need to understand a little bit better what the legal frameworks are. Because I, I don't think that Debian is going to be very happily distributing Whisper. For example, or something based on a whisper, right? So we will have to come to, we will have to have more conversations about, about that, this reluctance, or maybe let me put it that way.

Maybe we need to think about how we're going to be solving that, that challenge between. Oh my God, this thing is stealing my content, it's stealing my stuff. And, and, oh my God, this, this is useful. I would use it or my friends are using it. So there's that tension in there that we need to resolve.

Jonathan: And you had one more Aaron.

Aaron: Yeah. One, one more quick one. And maybe this can be part of the wrap up instead of our usual questions, but I'm kind of curious what your favorite AI tool is that you're using at the moment.

Jonathan: Oh, that's a really good one.

Stefano: Yeah. I really don't. Don't use them that much, but we do use Google workspace at, at USI and Gemini is included in it.

So every now and then the temptation is to, to go and check what Gemini does.

Simon: Simon? You know, I'm not knowingly using any AIs at the moment. I, I do go and kick the tires on them every now and then. So like Steph, I, I kick the tires on Gemini. Because I can, you do that without needing to buy tokens to put in the slot machine.

Right. I, I'm going to be quite interested, so to go back to Aaron's earlier question, you know, the biggest challenges with doing that are indeed the, the the belief that the copyleft is, is carried into the statistical model. And I, I, I, personally, I am of the view that the courts are going to find that ridiculous.

They have done so far in each attempt in the U. S. We've, we've really not seen very much happening elsewhere. But until we get over the idea that the copyleft has been carried into the model, we're not going to see anybody then using the model in some software. And I think that's a big obstacle to seeing AI tools, AI ending up in open source tools.

I, I also think that we're going to see another generational change in AI coming about, where the technology is going to be made differently and deployed differently to the way that it is now. And I think that's, that's a big obstacle. Quite likely to wait for those, those cool things to make their way in to open source tools is going to need to wait for that.

Having said that, there are already AI hooks in an awful lot of places. I don't know if you've realized this, you know, you look in home assistant, for example, so I do use home assistant. There's a great big AI hook in the middle of home assistant so that it can do voice recognition. There are great big hooks in Mastodon for going and doing AI translation.

So actually, the, the, the question you asked, you know, when are we going to see open source supporting AI, it's already happening, but the way it's happening at the moment is by providing hooks to go use external systems rather than by building the capability into the product itself.

Jonathan: And

Simon: there's a lot of that already happening.

But as a community or a community of communities, we've got some very hard conversations to have sometime soon, if we're going to see freedom respecting software, do AI rather than just call out to freedom, non respecting software that does AI.

Yeah,

Aaron: right. To your, the, the agentic model, I guess, as it's known, it seems like that's a big word.

That's thrown around a lot these days.

Jonathan: To your point, Simon, there, about whether, and it's the same question that I asked about the kind of inherent, the inherence of the copyleft when it's part of the training data and how that's gone in the courts. I imagine what we're eventually going to see is essentially a test.

Right? There's going to be a legal test that says, you know, if, and I don't know for sure what it's going to look like, but like if you can create a prompt that gives you this many words in a row that matches the the input, then you have inherited the copyright from the input, you know, it's, and it may not be exactly that, but like, I have to assume that there's going to be some court case that's going to give us a test that sort of everybody can agree on.

Okay. Because, obviously, like, you could just copy a file through a black box AI that doesn't do anything, call it AI, and

Simon: I mean, there was a cartoon to that effect, wasn't there? Well, someone

Jonathan: tried to do that with Beatles music, years ago. And they, they ran the Beatles records through their sonic maximizer.

And I don't know if they even use the AI buzzword or not, but they're, they tried to make the point that, Oh, this transforms it so much. It's a completely new work. And of course the courts completely shot that down and, but there's got to be a happy medium in there somewhere. So I, I expect that at some point there's going to be an agreed upon test of some sort that here's the, here's the guideline for where your AI is transformative enough.

that you're not inheriting that copyright.

Simon: Yeah. Well, I'm not going to second guess it. You know, but I, as an accident of what I do for a living, I know a lot of lawyers. And I have yet to meet one that thinks that that the license that the, the the statistical model is a derivative work of the source data.

Now that maybe that's going to change sooner or later, but at the moment If you, if you want to go make a court case that the some AI has copied your work, it's going to be a really tough case to even get legal counsel to defend you on, and I draw your attention to the fact there's a lot of no win, no fee work going on in this area.

But Steph disagrees. I can see him waving. No, I, I don't disagree.

Stefano: I, I think that it's the, it's a less interesting question for me because the more interesting question is, should it consider a derivative or not? And I, I think that we should be running that exercise a little bit more consciously and think about the consequences of either way.

Like, either way, what happens and what's the worst outcome possible in, in either parts? And and maybe, maybe we can even influence courts. But we should do it with consciousness, right? It's not just the gut reaction. Say, Hey, that code is mine. I should be getting something remuneration for, for copilots capabilities, for example,

Simon: or,

Stefano: or Dolly, the fact that it, you know, reproduces works that look like mine, like I need to get compensated.

All right. Should you, what happens if we do?

Jonathan: Yeah, there's a hole and we do not have time to go down this rabbit hole, but there is an entire rabbit hole about whether it's a good thing that we can have a machine produce an image and we're no longer paying an artist to do it, right? Like, that's its own entire conversation that, yeah, I think is important to think about, but as I said, we are not going down that rabbit hole today.

Maybe someday in the future. I do, I am required to get a couple of final questions in myself. And that is, what are each of your favorite text editor and scripting language? Stefano! Tsk.

Stefano: Yeah, it's passed on the scripting language. I really don't code anymore. You know, I played with Bash and Python.

But I, right now, I'd probably ask a co pilot at JGPT or something to do it for me. Is there a text editor in there? Text

editors VI, VIM is one that I usually fire up when I do some quick stuff, but I'm not really a text editor guy. Yeah.

Simon: And I'm very disloyal to my text editors. I've been using different ones throughout my career.

Yep. The one I fire up most often at the moment is Nano. Sure.

Yep.

But hey, you know when, when I was IBM, we were using e and I, I used to work on, I used to work on, on a word processor. I used, I worked with WordStar and Word Perfect back in the day.

Jonathan: Yeah.

Simon: And I've got everything loaded on a, on various computers around here.

Scripting language is more interesting though. I'm doing it all in YAML at the moment because I'm doing all this home assistant stuff. Yeah. You

Stefano: consider that a

Simon: scripting

Stefano: language?

Jonathan: I'm not sure that, that yet another markup language really counts as a scripting language, but

Simon: And yet I'm programming the whole of my home assistant deployment using YAML, using YAML pages.

Jonathan: All right. That's fair. I suppose. All right. Thank you guys both so much for being here. We appreciate it. And it's a fascinating dive into the, some of the questions and some of the answers about open source AI. Appreciate it very much. Thank you. All right. Man. What do you think?

Aaron: Yeah, I mean, it is, I'm, I'm glad that it's being done, I feel like mm-hmm

With other areas in tech, we kind of missed the boat. Open source hardware and some of those other things. Yeah. Social media, for example. Yeah. You know, it, it, it feels like now there's all this concern around social media. It's like, well, where were you five, 10 years ago? Right.

Simon: Yeah.

Aaron: When people could see some of the problems that it was gonna have, and at least with AI moving so fast, I'm glad that.

Somebody at least is thinking about these things and coming up with these definitions because otherwise, you know you who knows what what could happen? Right? And like Stefano said, we need to have the discussion, right? Even if you don't like 1. 0 of the definition that's out today. Talk about it. Have the discussion.

It's what it's what we've always done going. Back to early days of networking, for example, think of all the discussions and all of the groups that were around to try to define how we're going to make this thing work, right? Without discussing it, you're, you're, you're just opening the door for not necessarily bad things to happen, but unwanted things to happen.

And so yeah, I'm glad it's taking place.

Jonathan: So I think it was Simon that told us something that absolutely terrifies me. And that is that in Europe, there are laws that refer to open source AI, and they referred to that with there being no definition for what open source AI is. And it's like, that encapsulates the central problem that many of us have with like over regulation and, and just government's, Stepping into things that they really don't understand.

That's, that's, that's humorous. And all of that to say, I'm glad that there are people working on this and, you know, people that sort of have an idea of what they're doing.

Aaron: Exactly. People that understand these things instead of. Politicians who don't just coming up with, you know, random things that they throw darts on the wall and come up with sometimes.

Jonathan: Yes. Yes. All right. You have anything?

Aaron: What we, you know, we should, we should talk about our favorite AI tools that were you, what's your favorite, Jonathan, what are you using?

Jonathan: About the only one that I use these days is when you Google something. Sometimes you'll get like the, the, and I guess it's Gemini, right?

So you'll get like the AI answer to your question. And that is getting to the point to where it's useful. I, I do not consider it trustworthy, but it is useful. And then I've done a little bit similar to what you did with like here's a prompt, give me an image that looks like this. I've, I've done some playing around with that.

With varying degrees of success. Some, sometimes the prompts can be, or if you're good at writing the prompts, sometimes you can get really good results. And then sometimes you ask for something a little esoteric or off the beaten path, and you just get weird results, which I guess is kind of fun too, but not really what you were looking for.

What, what tool do you make you most use of?

Aaron: I've probably got to pick JAT GPT. I mean, I do pay for the plus license and I use it multiple times a day. Okay. For personal and business use. And. It's, it's, it, it impresses me more than it disappoints me, that's for sure. So I use it to summarize notes from, from meetings.

So, you know, I'll have like hour and a half long meetings. We get the transcription, feed it into chat GPT, say, can you summarize this? And it does a great job. And there are other tools, of course, that do that. I use it as part of my YouTube channel for sometimes creating thumbnails. Images, if I just can't find anything that I'm allowed to use sometimes.

I'll use it for that. And of course I use it for you know, like I said, removing background on things, just things, shortcuts that, that, that I can get it to do that would take me. You know, 10 times as long, I can say, look, just like generating that image today, for example of the guy coming out of the garage with the open source tablets, right?

I mean, yeah, that was

Jonathan: pretty good.

Aaron: Yeah. So things like that, that you can do just as time savers really is the biggest way that I use it today.

Jonathan: I think, I think I'm a little gun shy on using AI because I write for Hackaday. And like, I do not want there to ever be the even the appearance of crossing those streams.

Right. So every word that I write for hackaday always comes directly from me. The, the most in AI is ever involved with is Google will tell me that I misspelled something and that's it. And we've had some conversations internally at hackaday and that's pretty much the conclusion we've come to as well.

You know, there, I think there's, was. Once one of our writers used an AI image as like the headline image and we kind of got shot down internally it's like let's not do this unless we decide that we're gonna be okay with this and So I'm I'm pretty careful to not use much AI just kind of because of that even as a

Aaron: research tool though for for the background for the For the piece that you're writing

Jonathan: about the only thing is if I google If I Google something, you know, it'll, it'll come up with the the, the, the Google summary of what it thinks the answer is.

And again, I've kind of found out that that's not necessarily always trustworthy. Sometimes it'll tell you the exact, now it's getting better, obviously, but sometimes it'll tell you the exact opposite, or it'll pull something from an article that's not really about the thing that you asked about. So even, even then I've, I'm pretty careful to go in and try to find actual hard sources.

I use

Aaron: it. I use it for research and I tell it to give me sources and then I can go take a look at the sources and validate.

Jonathan: Yeah. Yeah. And that's, that's apparently a really useful hack. Like you make your, you make your AI assistant. Way more accurate. If you tell it, it's, it's kind of like with people, you ask them, Hey, what's the answer to this?

And give me your source. Well, we're going to work a little bit harder to make sure we give you the right answer. If we also have to give you a source, apparently that works for AI too.

Aaron: Exactly. Exactly. Yeah.

Jonathan: All right. What do you want to plug if we let folks go?

Aaron: Well, I mentioned the YouTube channel.

Check it out. I've got two channels. That's important because I've actually been publishing more on the second channel than on the first. So there's a RetroHackShack, of course, which is the main channel. I do more history videos there, more big projects there, and they come out a little bit less frequency, less frequently.

And then there's also RetroHackShack After Hours. And the last two videos I did there, one was a repair on a 1990s motherboard where the keyboard didn't work. I couldn't get any keyboards to work, so that was a short one, and you can go watch that one and figure out what happened. And then the one before that was oh, I do a lot of e waste stuff on the second channel.

I find stuff at e waste and just, just bring it home and start the camera, and I go through and figure out what happened. Like on the fly, like what it is, what's working, what's not. And so that's, that's a lot of fun too. So check out both channels and you know, hopefully there's, there's still people out there that like vintage computers.

Jonathan: Oh, definitely. I know. I watched that, that last one about the the keyboard not working and I won't give it away, but I will say the component that you replaced, I would have looked at and probably just thought it was a weird resistor. So that's actually, that's actually pretty, a pretty useful bit of knowledge for me to add to my own.

Fix it toolkit.

Aaron: Right. Right. And I would say probably it wasn't long ago, you know, 10 years ago, I would have been in the same boat. I would have been like, Oh, what's that thing? So, yeah, it is, it's, it's fun. And that's, what's fun about it is you learn about what things are and how they work and how they used to work.

So yes,

Jonathan: absolutely. Absolutely. All right. So next week we've actually got something really interesting. I believe, yeah, we're talking with according to the calendar. Hopefully it's right. We have another Stefano. We're talking with Stefano Zaccaroli about the Software Heritage group. And that's going to be a lot of fun.

And then the week after that, hopefully we're going to have someone from CIQ to talk about Rocky, which we've talked to all Malinux. We figure we might as well talk Rocky. They were there at the very beginning as well. And then the week after that, we're talking Thunderbolt. Thunderbird, so all kinds of fun stuff coming up.

You don't want to miss it. If you want to follow me and my work, of course, there is Hackaday. We appreciate Hackaday being the home of Floss Weekly. I've got the security column that goes live pretty much every Friday morning there, and you can check that out. Got a YouTube channel that you can find.

There's also the Untitled Linux Show still over at. Twit, that's twit. tv. I think it's twit. tv slash ULS. But we have a lot of fun there talking about what's going on with Linux and lots of open source stuff, but more news of the week over there as opposed to the long interview form here. Yeah, we appreciate everyone that watches that get us both live and on the download, and we will see you next week on Floss Weekly.

Episode 815 transcript

8 januari 2025 | min

FLOSS 815

Jonathan: Hey folks, this week, Randall joins me. We talked with Matthias Schakla about open source and the law. We can talk about, uh, contributor license agreements, software licenses themselves, and lots more. You don't want to miss it. So stay tuned. This is Floss Weekly episode 815 recorded Tuesday, January 7th.

You win some.

It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something a lot of fun today, but also really, really important. And, uh, if you are, but let's see, let's count. If you're part of an open source project, If you're doing any security research at all, uh, if you're an employee anywhere in the IT sector, you probably need to pay attention to this because we're talking about the, the intersection between the law and open source software and probably other technology questions.

And, uh, it's not just me, of course, I've got a lovely and talented co host, as we like to say, Mr. Randall Schwartz. Welcome.

Randal: Hey, Jonathan, thanks for having me back on this show. This is great.

Jonathan: Yeah. Thanks for jumping in kind of at the last minute. My scheduling got further behind that I wanted it to. And so last night I was just messaging a few people like, I know it's like 10 PM where you're at.

Could you please come on the show tomorrow? And I asked Randall, he's like, yeah, yeah, yeah, sure. I'll do it.

Randal: Yeah, uh, well, I was looking at, uh, at this week's schedule and I said, Oh, Tuesday, I'm blocking out FOSS Weekly. I wonder if I'm needed for that. And that's when I contacted you at 10 o'clock last night saying, I still have an open slot tomorrow morning.

What do you think? Yeah, it worked out.

Jonathan: It worked out. We are. It worked out real well. Um, so our guest is Mattia, Mattia Cioche. And I, I'm sure I killed that last name. I did not pronounce that correctly. It sounded right in my brain and then my lips started moving and it was terrible. Um, but he is a legal expert and sort of a geek.

And, uh, he said it's been a while since he has compiled the kernel, but used to do it all the time back when he was running Gentoo. And, uh, I think that makes him one of our people, you know, at least, uh, one of my people, I think tangential to being one of Randall's people, right? Have you ever compiled the Linux kernel, Randall?

Randal: I don't believe I ever have. It's true.

Jonathan: That's all right. We still love you.

Randal: I've installed it a few times, but it's not my primary install when I pick up a computer. So yeah, sure.

Jonathan: So let's bring it. Mattia on welcome, sir, to the show. Hello. Uh, it's an honor to be here. We are absolutely thrilled to have you.

Um, so what, let's start, let's start with your background. Because you, you come at this from sort of an interesting, uh, direction. Your, your background is mainly the law and the, the IT computer stuff was sort of a hobby that got picked up. And then from what I understand, it's sort of become part of your career now.

Matija: Um, yeah, that's the long, long short version of it. Um, so. The longer, slightly longer version is, I've been interested in computers since forever. Um, me and my brother Noah, we had, we got some really old books on computers, um, which, that included like, uh, you know, the whole page is full of basic, um, that you could, you know, type in to your computer to get, uh, something that looked like Tetris out of it.

It never worked. Because that was some other basic and we tried it on DOS. So the QBasic didn't work, but we spent like days doing that. So I then got infected by the Linux virus, mid nineties at, uh, uh, high school. Um, because there was like all these computers in the hallway, um, that you could use to go online and all had, you know, nine, Windows 95 or 98 or something like that.

But in the middle there was this one with black screen, white text.

Jonathan: But what I

Matija: saw is like, they changed, like on every other computer, it took like, because we had like the whole high school had a dual ISDN line for the whole school. So somebody knows what that meant, right? Uh, I see nodding, which meant that when you logged into your Yahoo mail, which was usually what you had back then.

And you know, the Netscape times, um, it took like the whole break before you managed to log in. And so you, you saw you had new mail. So you, then you had to go through into the next break and try to be the first one at the computer to manage, to click on the first mail, to see what it is. But then on the.

That black screen computer I saw like people rotating and it turned out that was a Linux machine and people had mails on the machine itself. And I think we used the pine or something like that. I think it was already pine. And so we just learned from each other how to use this weird contraption. Um, so, you know, we learned how to use, I apologize, the program was actually called like that, uh, Bitch X, which is an awesome IRC client back then, um, especially with the ASCII art.

Yeah. Um, and we learned together how to use that machine. And then through that I got into the, into the early essays, uh, by Stallman and, uh, uh, from the esr Cathed, yeah. Sr. Yeah. Um, and that's kind how I got into the whole thing. Um, but I always want, I always knew law was what interested me in general. Um, and that's another.

You know, interesting story why, but that was kind of how it went. It's like always while I was studying law, um, I was also part of the local hackerspace, um, so, and the Linux user group. Um, so, you know, I don't have any IT, um, training official, but, um, I've dabbled with it for a while. Many years. Far from being a good coder.

Far from being a coder.

Jonathan: I do some tech support for some lawyers, and I just have to say, The fact that you know what Linux is, puts you so far ahead of some of that crowd. Yeah.

Matija: I mean, I probably run around a different crowd of lawyers than the average crowd of lawyers. Yeah. But yes, in general, I do agree.

Like, when I remember when, when it was, when, um, at university, yeah, it was, turning on the PC was a challenge for some people.

Jonathan: And so did that, did that kind of affect the, uh, the trajectory of your career, your legal career that you knew sort of this, this whole. computer and open source side of things existed.

Um, is, is that sort of where you, where you went with your legal career?

Matija: Um, kind of maybe, but it was also a bit of a chance. So, while I was, um, working, uh, with the hackerspace and, uh, and the Linux user group, um, we did, you know, we clearly, we did some advocating like in the late 90s and, you know, against software patents in Europe, et cetera, et cetera.

Um, I did. You know, through that, through advocating for more FOSS and, uh, and all that, um, I found out that there's actually a pan European, um, organization that does that, uh, the Free Software Foundation Europe, which is like the sister, uh, organization of the USFSF. Um, and I messaged them, just said, well, you know, you know, do you have anything in Slovenia, which, which is where I'm from?

Um, and if not, can we do anything? And they said, yeah, you just, you know, there you are, you know, you should start a chapter. And so we did. Um, and then at some point, um, when I was volunteering for the FSV as a, you know, running the Slovenian chapter, um, at some point, In my studies, um, there was a, I had an option to go study abroad, um, and, uh, in the U.

S. actually. And, uh, I, um, asked the president back then, of the FSFE if that was okay, um, because there was some, you know, then I wouldn't be in Europe anymore, et cetera, et cetera. And uh, he messaged me back like saying, I'll, um, I'll give you a call and I'm like, oh crap. What did I do now? Um, and so I run out of the law, uh, faculty's library and I, you know, pick up the phone and, you know, that's still when, you know, even in Europe, you know, the.

And he was in Germany, I was in Slovenia, that was still like kind of a costly call. It wasn't super expensive, but it wasn't local yet, as it is nowadays. So, um, I pick it up and he's like, well, we actually wanted to, we wanted to employ you. Like, what? So, that's how I became, uh, FSFE's, um, What was it called back then?

Legal coordinator. So I was responsible for open source or free software, uh, legal matters within FSFE. And that kind of, I'll be honest, I mean, if that didn't happen, I don't know what I would have been nowadays, because that, you know, not just, you know, open doors, et cetera, but introduced me, part of the, part of the job was, um, there is this, uh, legal network.

Um, that, uh, FSFE facilitates, which is the largest, uh, network of, uh, FOSS specialists, lawyers, and technical people who have a lot of knowledge when it comes to, like, license compliance, etc. Um, and, uh, for many years, I was in charge of that network. So that's clearly, you know, you learn a lot on that by that, doing that by.

Organizing conferences for, and workshops and all that great stuff. Um, so that's, that's how it happened. And I'm very grateful for that because otherwise. I don't know, I would probably as a, just, you know, a box standard attorney, I probably wouldn't be great, pretty good at what I do, but, you know, when, when I was looking for a job and I was looking at my grades, um, in school, because I know I was, you know, Helping with the hackerspace, you know, it took me forever to finish law school.

It was, you know, on paper, this was not looking good, man.

Jonathan: Yes, I very much get that. My, uh, my college career went very similarly to that. Uh, so one of the things that Randall and I were wondering before the show, Is, are you a, and I don't, honestly I don't even know how this works for sure, are you an American lawyer, a Slovenian lawyer, are you licensed in both places?

How, how does that, that bit of international law work?

Matija: Um, I'm a lawyer, I'm not an attorney. That's the first thing. So I am, I am, I finished Master of Laws. Okay.

Jonathan: So

Matija: I have, um, it's because it was the old system. It's, um, we're not allowed to call it LLM, but it translates to LLM. Um, not the new LLM. Not the new, yeah, yeah, yeah.

We're not talking about artificial intelligence here. We're talking like the title. Right, right. And, um, but then if you want to be an attorney, so attorney at law, that means you can represent people in court. I mean, there's to, to some extent, to some level of court, anybody can represent. Right. But from a certain point onward, you have to have an attorney.

Um, and that is, and different places have different, uh, exams, how they do that. Um, and, um, Um, when you do an exam, you're only allowed to do it in that jurisdiction. So even in, within the EU, if I did mine in Slovenia, I couldn't be practicing in Austria, which is across the border. I would need to take, I would need to take at least the partial exam to get it re evaluated from just for Austrian law.

Because the, the thing is like, Counter to how, you know, tech and, and natural sciences work is like jurisdictions, like every country has their own laws and their own history of laws. So you need to be trained in that specific history. Um, so I'm not admitted to any bar. So attorney at law means also like.

In English, it would be a barrister, um, so that is, you know, when you take the bar exam and you're admitted to the bar. Got it. I

Jonathan: appreciate the, uh, the clarification there. Um, Randall, this seems like this would be a good, a good time for you to, uh, fill us in on your, your history with legal issues, and maybe we can kind of jump off from there.

Randal: Yeah, so I also wanted to jump in with like maybe what you're closer to and say the U. S. would be like a paralegal, which is somebody who isn't a lawyer because lawyer here actually means probably the same thing that attorney means to you. If I say somebody is a lawyer in the U. S., they, they are somebody who can represent me in a court, in all court actions, things like that.

So, um, so I think, I think our terminology is a little different here in the U. S. So I would say you're probably closer to someone who can do filings on behalf of people and, and, uh, and, and, and understand the law in an area and have been at least certified to a certain degree. That, that would be paralegal here.

I don't think so,

Matija: honestly. I think paralegal is somebody who just has, uh, who just has an exam. It doesn't have like the whole schooling.

Randal: Oh, okay. Okay. Maybe, maybe not, but here, here, a lawyer and attorney mean exactly the same thing. There's not a distinction for those.

Jonathan: Although that may be, that may be one of those cases where if you're a lawyer, there is a distinction, but just for the rest of us, we don't know what the distinction is.

Maybe that might be what's going on there too.

Randal: A software engineer, not a programmer. Okay. Okay, here we go. It's probably the same sort of thing. I'm not a coder. I'm actually a software engineer, right? Like that means anything really come on guys Um, so just uh, just the brief version many people in our audience are familiar with my legal case But just to bring you up to speed a little bit on that Uh, I, uh, I am, uh, I am a three times over felon as a result of doing my job for one of my clients with a bit too much enthusiasm, as in I was doing, uh, unsolicited pen testing.

Jonathan: Uh,

Randal: against a machine that, uh, was no longer in my jurisdiction, but had been earlier and I was still concerned about the new people that had taken my job over there. So, uh, so I just got crazy and started doing some pen testing without asking my boss. And, uh, that eventually led to cops showing up at my door and, uh, 270, 000 later, um, I'm still ultimately restricted from going to Canada or Australia.

Although I've been in 68 countries, I can't go back to Canada or Australia ever again, uh, which is really just weird. But, um, uh, I think from that, I actually want to start asking questions about liability, uh, in an open source project. So here I am contributing to an open source project. How careful do I need to be as an individual contributor, contributing to a business project that the project's not going to get in trouble somehow, but also like, I'm not going to be individually liable somehow as I'm contributing.

That was kind of my first volley of a question.

Matija: Oh, that's an ungrateful question because that like literally I just started reading there's there's you guys in the US have the executive order number, whatever, whatever, uh, national security thing, uh, that was a big thing like five years ago or a few years ago.

Um, And I did read through that one, uh, because of work, um, and I'm just right now reading the European kind of variation of that, which is the Cyber Resilience Act, um, and I haven't gone through it yet, um, so I know just, you know, Secondhand information, um, but from what I gather so far, at least it, it depends on jurisdiction to jurisdiction, but in this case, it would be like EU will be pretty stable.

Standardized between itself, uh, within itself and, uh, U. S. will have its own thing. And then I think Brazil has something already cooking if it's not done yet. Um, so there is legislation happening on that case, uh, on that side. Um, I don't know about the rest, but from what I seen. And, uh, in the CRA, so the Cyber Resilience Act in the EU, um, I noticed that the, uh, the, the likes of, uh, the FSFE and, uh, OSI, so the Open Source Initiative, and I think the Eclipse Foundation was involved as well.

Um, they were very involved with the European Union, uh, when it came to, you know, Honestly, to lobbying to make sure that FOSS gets excluded or, you know, gets preferential treatment. Um, and to some extent that's what happened. Um, so, in your case, I mean, you were pen testing, that's different. That's not contributing to an open source project, let's be honest.

Um, And I don't know enough about that case to, you know, make it sure, make a statement, what I think about it. Um, but, um, yeah, it sucks to be you.

Randal: I have said that frequently. Yes. Yeah. Uh, but, but, but we, there, there are issues, I think, both from the level of civilian penalties, as well as criminal penalties, if you cross certain lines, even being an individual contributor to an open source project. Right.

Matija: Um, the question of an individual contributor is a good one.

Um, because from, again, I haven't finished reading it yet. Um, but from what I've read in the CRA, um, they do distinguish between Manufacturer which may also include open source project or components or, you know, maybe the whole thing is an open source thing and then they just make money, um, shipping it on, you know, Computers or, or offering as a SaaS or whatever, and, um, open source stewards, that is actually like a defined term in the, in that law and, um, open source stewards is defined to encapsulate things like, um, like the Eclipse foundation or, you know, you're Linux Foundation is already kind of huge.

Um, they do a lot of stuff, but you know, kind of like those, like GNOME Foundation, KDEV and stuff like that. So you have like, it's a preferably non profit or not even necessarily an organization that is a steward of an open source project. Um, they may have, they may get some money from donations or otherwise to work on it, but they're not.

You know, they're making the software, you know, publicly available and they're not necessarily primarily monetizing it. Um, and there they got like preferential treatment according to the CRA. So this is what is happening, at least in Europe is that we've seen now that the trend is that. for years now, they, they, they had preferential treatment for small and medium enterprises and, you know, even micro enterprises, of course, um, that they're now also are starting, like starting to, um, make exceptions also for FOSS, uh, how that works in the U S I don't know.

I've followed the, that executive order and, um, That's as far as I got, um, because that was, uh, relevant for work. Um, but, the, and, and here's a, as I said, this is what is interesting when you point it out from an individual contributor is,

Here, in this case, it could be like a company is using, you know, some foundations, open source software, the company would need to make sure that it's secure. And it has, like, depending on how serious the use case is, um, whether it's critical or, or just severe or just You know, everyday stuff, um, depending on the severity of the use case, they have different levels of what they need to, how much they need to take into account that the use of that software that they, you know, manufacture, you know, this is actually secure to use, uh, safe to use.

Um, and then the foundation, the open source foundation. The project has a much, much, much, much, much lower, um, barrier they need to meet, meet is, um, I, I mean, there's one of the obligations is, for example, if you see as a manufacturer, if you see a, if you find, uh, security hole, you have to report it to the EU agency.

That's, you know, for security of software. I forgot what the name was. It's a new one. Um, and also if there's an upstream also reported to the upstream, and if you have a fix, ideally also ship the fix. So in this use case, it would be if the, if the company finds a bug in the open source component and they patched it, they have to report it.

And then they also have to submit the patch. The patch with the report to the upstream, uh, open source, uh, project as well. Um, and there's, you know, the open source project doesn't have as many obligations. I don't think there's, I think they need to, again, I haven't gone that far. Uh, so, so far it seems like they might need to report an issue if they found a serious issue, but they don't have to do it in 24 hours because there's, you If you're critical infrastructure, you need to report it within 24 or 48 hours.

Um, and I don't think I've seen individual contributors mentioned yet in the text. So that's an interesting question. It is a good question who they would be, who they would be liable to. I mean, most Pretty much all open source licenses have, you know, the, as honestly, pretty much all software licenses. We don't, there's no warranty to the extent, to the extent permitted by law, et cetera, et cetera.

So it's going to be, I would say, following on that, I would say that, building on that, I would say that the contributor wouldn't be liable Unless they would be liable under just general law, period. For example, like, if they would be, if they were malicious interacting. So, and a malicious actor would be liable regardless.

Um, uh, but a, you know, if a contributor just, Made a boo boo. I don't think that would be a big problem. Um, but again, haven't finished reading it yet. So just got feelings so far.

Randal: As a follow up to that, I'm looking at the angle for individual contributor to an open source project. I think one of the other obligations as an individual contributor I have is that.

Uh, the, the chain of, of creation is proper. In other words, I created this software, therefore I have the rights to contribute it, but we can get into trouble by, uh, taking things that actually have proprietary licenses or incompatible licenses and contributing them to an open source project. And I just, that's one point that I want to look at, but I also, I'm thinking about.

Like licenses, like say the Mac, Mac OS user licenses, you have to be running on Apple hardware. Does that mean that if I'm using a hackintosh to make software that I'm contributing to an open source project that relates to a Mac M M where, where did that cross the line and is the project liable for accepting my stuff?

Or am I the only one liable here? Cause I'm the one that violated Apple's contract. I mean, where, where do we go there? Wow.

Matija: I mean, you'd be the one who violated the Apple's contract. So that, that would be on you. I mean, how, how would, and, and, I mean, there's also what you, what the output of your work here.

And I'm slightly oversimplifying here, of course, but it's also like. If I understand the situation here is, it's kind of like, if I use a, if I use a cracked pirated version of, for example, Microsoft Office to write a document, who's, who's liable for using that document? They're like, I don't care. I mean, the document is a separate piece of work.

It just happened to be, you know, I managed, I, I did something illegal to get it. done, but, um, the document itself should be fine. There's no, you know, if, if that was a problem, we'd be in all sorts of hell.

Jonathan: I I'm, I'm curious with this idea of, uh, the individual contributors and liability for open source, uh, in the United States, we've seen, I don't think it's a Supreme court precedent, but we've seen a court precedent that. Source code is First Amendment protected speech. And I'm wondering if that helps. Like, does that help protect the individual contributors?

When we're talking about these projects.

Matija: I'm not U. S. trained, so I do know a bit of U. S. legislation. I do have a Harvard online course in copyright, in U. S. copyright, apart from my own master's, but, um, I think like the First Amendment gets thrown around quite a bit. Um, and, uh, From the history I've heard about the First Amendment, it's, um, it's been stretched in different ways and different forms through the decades, so I don't know how that would work.

Um, but

I'm not sure. I'm not sure. I'd be, I'd be, literally, I'd be just pulling something. Well, ding,

Jonathan: ding, ding, ding. We've

Matija: stumped the expert. No, I mean, I can theorize on it, just that this is not a jurisdiction I'm familiar with. Familiar with enough, you know, specifically the first amendment. I'm not as familiar with the first amendment to make it I can make an educated guess but right well I don't know if an educated guess would be a fair answer at this point

Jonathan: So I mean we we see things the United States like there are there are open source projects that are essentially Malicious software, right?

And so, you know, someone will publish it and throw on there, you know, this is for uh Entertainment not even entertainment education education purposes only gets published under an open source project license And then the next thing you know that piece of software is used in an attack somewhere and I don't know that i've ever seen someone get Prosecuted just for writing the software and my understanding has been that that's always basically been because they are protected because that is protected speech, even though it's code.

Matija: Eh, I think it's more like if, if it's, like it's similar to the, um, to the CRA here, right? When I said, I mean, if, if it's an open source project, they're not liable, but as soon as you, you know, use it, the, the one who's using it is liable. Um, and I mean, from the first amendment, I mean, here's the way I would approach the first amendment question is, um, if I.

Violated, if I copied like half of, you know, let's, let's, let's use a blatant, um, examples. If I copied like half of a Disney movie, um, and remixed it a bit, maybe, and then release that as my own. Would the First Amendment protect me? I mean, it would probably protect me that I can say that, but I'm still gonna get into trouble.

I mean, nobody's preventing me. Nobody's telling me I'm not, I can't do that. It's just that, you know, I will face the consequences. So I think it would be similar in this case. But again, I'm not an expert on the U. S. constitutional law and its amendments, but Sure. So, um, but you're, you're the other part of, uh, Randall's question was, um, if I understood you correctly, was the liability for introducing not a security issue or a bug, but, uh, introducing a license issue or copyright infringement.

Um, and that's something I'm usually working with. So that's more of a home territory for me. But if you, but, um, if you had a. Further question, just please continue and we can get back to that one later.

Jonathan: No, I think that's an interesting, an interesting direction to go. Um, and one of the, one of the things I saw that you were involved in is some, some, uh, some CLAs.

The, uh, the, what is it, the CLA 2. 0? What, um, Fiduciary License Agreement. The FLA 2. 0. Yeah. Okay. So that's, that's kind of a, it's, it's one of the solutions, I think, to this question about, you know, licenses and whether they're compatible, at least it's something that I've seen businesses try to use as a solution is, well, we'll just have everybody sign a CLA and then if we find a license incompatibility in the future, well, then we can just relicense it.

And, uh, I, I'm, yeah. I'm very curious, like, kind of your perspective, maybe even your background on the idea of CLAs and what the FLA 2. 0 is and what the purpose is.

Matija: Ooh, oh yeah, that's a, that's a pet project of mine. I actually, my master's thesis was specifically about the FLA, so the versions 1. x and the 2.

0 was a result of my thesis. Um, but you know, if we just go back to the basics, as if, you know, as Randall said, like, what you're, what you're As a project, you're getting in code, you have to understand there's like this concept, you know, you, you understand the concept of inbound and outbound code or, you know, upstream, downstream, right?

Um, so the same is with rights. So, you know, rights have to flow with the code. Otherwise, you're not allowed to do anything with it. Because of copyright and it being automatic and all that stuff. So you also have like inbound licenses and outbound licenses. So the outbound license of a project would be, you know, what you would typically find in the readme or the license file.

So this is everybody out there to the public. This is the license the project is published under. It, you know, in practice, it gets more complicated. There's usually like a dozen of them in there. But, you know, let's keep it simple. Um, and, you know, typically a. Open source project pretty,

I'm not going to say easily, maybe sometimes too easily decides what their outbound license is and then they try to backpedal later on. But, um, at least, you know, that's a relatively, you know, at least you have like one main license that you choose as a project. But then, you know, if every piece of code is protected by copyright, By default, um, then what do you do with all the contributions?

So the contributors have to agree to give it to you under a license. So for a long time, and, you know, still, um, the assumption. Was, and it is an assumption, you know, this has to be said. If it's not explicit, it's an assumption that contributors are contributing the code under the same license that the project is under.

Um, but that's an assumption, right? Um, and that is usually called like the inbound equals outbound, um, licensing situation, which is, uh, I think a term that Richard Fontana coined. Um, he's a really smart lawyer, uh, with red hats. And you can formulate, the best way would be that you formalize that. You put somewhere in the README or some other documentation that's easy to find, a contributing file or something like that, where you explicitly say that, you know, every contribution you make is going to be under the same license as our outbound license.

So if the project is under GPL2, then whatever you contribute is under the GPL2. Like, for example, for the Linux kernel. Then, as you guys alluded to, there's, there's, there's The CLAs and CAAs. So, and there's a distinction, a small, but big distinction between the two that people mix up because, um, a CLA is a contributor licensing agreement, um, which means that it is an agreement on what the licensing will be for contributions or between contributors, and that can be whatever.

That is a very, very broad spectrum. What you can write in there. Um, It's just an umbrella term. It's like, you know, just basically, you know, this is our inbound licensing policy. Um, But then you also have, um, the CAAs, so which are copyright assignment agreements, which, you know, means, you know, if you sign that one, you're assigning all your copyrights to, you know, this project or this company, typically.

And this is usually, you know, people conflate the two, uh, you know, some companies explain especially make sure that they, they're vague about that. They call it the CLA, but in practice, it's a copyright assignment. So you actually need to read the text to figure out the difference. Um, and that's, you know, that's as Randall said, I was like, that's, you know, the basis of it is if, you know, the company has, if, if, They only allow contributions that are, that have signed a CAA or CLA, all the contributions are covered by those.

And then, you know, the, the upside is that the company or open source project then has all the rights it wants. or needs to change the license if they need to do it later on. For companies, there may be different reasons for, you know, open source foundations or projects. There could be different reasons to do that.

Um, CIA is the, the copyright assignment is like the Most extreme way, but then you have like with the CLA, as I said, there's a big spectrum and the fiduciary license agreement is one of those. Um, and the fiduciary license agreement specifically What it does differently than most CLAs is that it ensures that, you know, first of all, you know, if I signed the FLA, um, you know, for example, like even with a company, like, let's say I signed the FLA with a company.

And that means I give all my exclusive rights. Um, and, you know, we can discuss this, there's, you know, jurisdiction reasons why you cannot assign full copyright in most jurisdictions, um, outside of US, um, mainly most of Europe, you cannot assign copyright, period. You can just assign exclusive rights.

Effectively, same crap, but different and for practical purposes, this is like 99 percent the same. So with the FLA, I would assign all my rights to the code that I contributed to the, you know, foundation, um, company, whatever. Um, but the company foundation would at the same time contribute, uh, give me back a non exclusive rights of the exact same kind, like everything that I could you know, gave them the rights to, I get all the rights back.

They keep the exclusivity, but I get non exclusive rights and I can do with my code, whatever I want. So I can still, I can run my own code as a closed source. If I want to, I can contribute to a different project under a different condition if I want to. The only difference is that they keep the exclusivity, which is a bit not really because I get a full copy of the rights as well.

Um, so that's. One part of the balance and the other part of the balance that the FLA brings is crucial and that is that as soon as, um, that it includes the clause that the fiduciary, so the company or foundation that received my rights, they are under the obligation by the FLA. They're under the obligation to keep the code that I contributed to them also under an open source license.

So they can, they can run a dual license situation where they have a proprietary version that they monetize or whatever. But my code, if there's code in the, proprietary version that I wrote, that, that code also has to be present in an open source version. And if they don't do that, I can just say, I can automatically say, well, I returned, I, you know, you broke the FLA.

Which means you don't have my rights anymore. So they lose, automatically lose the rights to my contribution. So that's the, that's the interesting part of the FA.

Jonathan: Yeah. And so I, I'm, I'm, I'm super curious that that bit right there, that they lose their rights if they break it. Like that's, that's in a lot of other open source licenses, I believe, like that's part of the GPL and, and all of that.

I, I think, uh, something, something sort of to that effect. Yeah. Um, it, it in that it has, it's, it, it is. Ideally, it has some teeth, right? Of some sort. Um, has, has that ever actually been tested in court?

Matija: Yes. How did that go? Um, I don't think I'm allowed to say because I was. Part of the, I was part of the, it was not part of the court case, but, um, there was a situation where a company and a different company, um, both held, uh, used code, a lot of code, um, that was contributed, that was, uh, under the FLA to a different entity, to a foundation.

Um, and they, interestingly enough, they sued each other. So the company and the company, the other company sued each other for copyright infringement, which was interesting because most copyright, like most exclusive rights were held by the foundation. Right. So the foundation had to go, you know, had to, it wasn't directly involved in the court case, but, You know, they had to bring in statements, et cetera, et cetera.

Um, and because that's when I was kind of involved with the foundation. I don't think I'm allowed to say too much, but it did it, I can tell you, it does have teeth. Both companies are still alive and there's a, there had to be a three way agreement on how to settle that. And it was settled in a way where, um, both companies and both projects are alive and they both have to also be open source.

Jonathan: And so that was the, the FLA. And so it, it survived because I know this is a,

Matija: It was in court, but it was, it was settled. So it wasn't,

Jonathan: I know this is, I know this is one of the things that it's kind of like a nightmare scenario to some open source people. It was pretty bad. Well, so the, the nightmare scenario is that the, say the GPL, Gets pulled up in a court case.

You've got a, you've got a court case and the GPL is the center star. And in that court case, the judge looks at it and goes, for reason XYZ, this is not binding, right? This is, this is not a valid license agreement. It doesn't have the teeth that you thought it did. Um, and that's, that's kind of the for, for some of us that are all in on open source, like that is the scenario that keeps us up at night.

Matija: But I mean, here's the thing is like on the other hand, um, and there's the distinction, this is going to be similar distinction as with the CIA and the CLA is what's the distinction between a license, like an open source license, um, and, uh, and a EULA. And if, you know, by default, copyright law allows you to run a piece of software as long as you legally obtained it.

So if, you know, I get some code from Randall, um, and he will willingly gives it to me, but there's no license attached, you know, nothing. Um, and I run that code. Um, he can't sue me for running the code if he didn't explicitly provide you know, prevent me from running the code. He gave me the code, I illegally obtained it.

And if I downloaded from, you know, if I found it on the internet, um, I knew that I don't have the rights in it, then I'm, you know, I'm probably not allowed to run it, but, you know, like pirated games and stuff, you know, you didn't legally obtain it. But if you did, Did obtain a legal copy, um, then, then it's fine.

So the thing is like a license is by definition something that gives you more rights than what you get by law. So an open source license is a license by legal definition because it gives you, you know, the right to use, study, share, and approve. Mm, the code. Whereas, you know, by, by copyright law, you only get the right to use it.

You're not allowed to copy it. I mean, it's copy left or right, right? Mm-hmm . So you're not allowed to copy it. So also the other things don't fall off. I mean, there's some exceptions for interoperability reasons, et cetera, et cetera. But so whatever gives you more rights as you know, the receiver, then you don't need to agree with it because.

You know, you did notice you don't have to click on I agree when you download a piece of when you install a piece of software with GPL or BSD or whatever, because you're not agreeing to less, which is opposed to the EULA, which is, you know, it does say end user license agreement, but the trick is it's not a license, it's an agreement, it just happens to be called the EULA, but it's technically it's an agreement because typically in a EULA, Um, and they differ wildly, but typically in a EULA, uh, when you read it, you notice that they tell you you're not allowed, you're only allowed to use it on X amount of CPUs, you're only allowed to use it for X amount of time.

You're only allowed to use it for certain purposes. Um, your. You're not allowed to, to, to share it. You're not allowed to do copies. You're not allowed to do basically anything. So you agree because you get less rights, uh, through the EULA, you actively have to agree with it as you know, the other party. Um, because it's not in your benefit.

It's, it's, you know, compared to just copyright law, it is, you know, potentially to your detriment.

Jonathan: Interesting. Are there, are there any cases where like, and this is a question from David Ruggles, one of our live audience members, uh, are there any cases or what would happen if different nations? Treat the same open source license differently.

Are there is that happens? Yeah, so what what what do you what do we do? Like what's the that just has to make everything more complex, right?

Matija: I mean it does but so coming back to to finish that thought on the GPL where you said what happens if If if a court says well the GPL is you know, it's not it's worth this piece of paper piece of text It's what happens then so that means that the GPL You Didn't give you all the four rights or the four freedoms.

So, you know, the cop, the copyright owners, uh, are the ones who still have all the rights. And, um, so depends on what you're looking at. If you're, you know, if you're. The court cases rarely go into the point of are you allowed to use this as an end user. They typically go into the, into, oh, somebody violated the GPL because they didn't provide the source code or they didn't provide the, You know, provide the full source code or they didn't mention it's under the GPL to the, to their users, et cetera.

And in that case, if the GPL went away, you are still completely violating the copyright. So, so, eh.

When it comes to treating licenses differently, interpreting licenses differently in different jurisdictions, that actually does happen. But that's also, you know, it's, it's a question of, What is, again, what is the issue here is like, I know that I think there was a Chinese or a Korean court case that was a bit different than what you would expect.

Um, but ultimately it didn't, you know, it's usually the court cases, um, in most jurisdictions out there, of course. You know, U. S. and the U. K., et cetera, do have this common law system where you have to take into account legal proceedings, et cetera. In most other jurisdictions, the law is the law, and the court cases are court cases.

They're between the parties. Um, yes, you know, in If it comes to a, you know, very high instance, you know, it would be odd for a lower court to, to say otherwise than the higher court in an exact same or a very similar case. Um, but they would just need to, you know, argument why this is not the same as it, you know, the, you know, 10 years ago, um, uh, Supreme Court, uh, ruled.

Um, but yeah, potentially. You know, potentially it could introduce some ambiguity and the ecosystem, but in practice, it doesn't affect that much. I mean, there's very, for how long we've known free and open source software and for how, in how many places it's used https: otter. ai

And, uh, in most of those court cases, it was more like a between the parties thing. And a lot of them just settle out of court anyway. So there's what is,

Jonathan: ah, now I lost the thought. Are you, are you watching, have you been watching the, uh, the software Freedom Conservancy versus Vizio out in California?

Matija: I, I'm following on the sidelines.

I'm not deeply following. Um, personally, I mean, from, if, if I did that in Europe, I would Take a different approach, but this is also probably a bit of a, you know, jurisdiction difference. Um, but, um, yeah, it's an interesting one. It's an interesting one. It's let's see happens. Um, it's is it going to like, what, whether it goes one way or the other.

Will it defect the whole world? I don't think so. I mean, it's, you know, we we've been People do, in other jurisdictions, do sometimes quote, you know, courts from other jurisdictions. But that's, I mean, if, if quoting your local court isn't binding, this is very much not binding. This is just like, somebody looked at this in their own jurisdiction, and this is what they found out.

And they're like, okay, that's an interesting thought. But I just had that thought again and then I lost it. Oh, yeah, so an example of You know different jurisdictions and different courts interpreting it differently on the mailing list of the Open source legal experts that I mentioned there's been like an evergreen discussion Or argument was whether the GPL is a license or a contract

Jonathan: mm-hmm

Matija: And, uh, in the US that's actually, you know, it, it depends on which court it goes to, et cetera. Right. Um, and then also, you know, which, what kind of damages you can expect or what kind of outcome is possible at all. Uh, and eu, that's a nonsense question because it's both. It's a license in a contract, and the U.

S. situation here is, it cannot be a contract, or at least used to be, that it's, it's, it could be weird to be a contract because a contract needs to have, um, consideration. So it needs to have, like, both parties need to have something to exchange. And that was the main argument for a lot of U. S. lawyers to say that, you know, it's not a contract.

Um, I think it was like a few years ago that it came to a, it was another court case. I don't remember. I remember which one it was, where in the U. S. where it actually did say, well, it's also a contract. But the, you know, the consideration is like all the worth that The software brings, right? Like all the rights are actually a consideration.

Jonathan: Yeah. I'm, I'm looking for this. It was a, uh, a California district court. Um, she ruled that the GNU General public License is an enforceable legal contract and I'm not seeing exactly why. Yeah. Yeah. Uh, south Korean company hand com. Yeah. Hand com. And what was the

Matija: artifacts thing?

Jonathan: Yes. Yeah. Yeah. Yeah. The artifacts thing.

Yeah. It's hand combed artifacts. Yeah. Yeah. So that, uh, that's, that's one of those, that's one of those examples of, uh, you know, it, it actually got tested and the results of that sort of changed things for, well, I guess didn't change in this case, didn't change anything, just clarified. Um, but yeah, super interesting.

Matija: Yeah. They did change, you know, the things in a lot of U S lawyers mindsets when it comes to, yeah. This situation, but yeah, it just clarified it.

Jonathan: Yeah, yeah. Okay, so I would be, I would be remiss if I didn't ask you about this, and that is, If an open source project or an open source contributor has a concern, and I can even give you, I can demonstrate one for you that, that is not very real for us, Um, I've, I'm part of an open source project and we run an MQTT server and we let people upload data to it.

And one of the things that gets uploaded is location information, totally opt in. You know, we're not collecting that, but still it gets uploaded there sometimes. And the question that we have then is, well, what about the PII laws? You know, the, the California consumer protection act and all of that. Uh, What sort of, um, you know, like what, what's our liability under that?

And I know that you haven't researched like this particular question, so you don't have an answer for me at the moment. But the bigger question is, though, when an open source project has a concern like this, where is there a place that they can turn to? Is there a, you know, a panel of experts out there?

Is there a legal group that we can say, Hey, we don't, we don't really have a whole lot of money, but we need, You know, we need opinion, but we also need like the equivalent of a lawyer on retainer just in case this blows up in our face. Is, is there any kind of solution like that out there?

Matija: Oh, um, back when I was, when I was the legal coordinator for FSFP, I would have had the answer.

Um, um, so we did use to have the, the, FSFE used to have a legal team, uh, which did comprise several, um, mostly European though. It was actually only European, but very, very experienced lawyers in these things. Um, and we did answer, like, if it was something that you can just send an email and it's not a thing that you need an attorney for, so you don't need to actually go to court or have some, you know, attorney privilege, et cetera, uh, somebody would just answer.

Um, So that was a mailing list for that. I don't know if, I think they started something similar with FSV again. Um, so that's one way I would, I would look at, uh, FSV. org and then search on their, um, legal team or whatever they call it nowadays, I think they have a. Place where you can ask questions. Um, and if it's something that's like a deep problem, like a very specialized problem, they have access to the, to the legal network.

So there's like, I think like 400 or 500, uh, legal experts on that mailing list, um, that they can just, you know, ask, um, if there's any questions. There's, I know OSI, uh, the open source initiative has, um, uh, license mailing list, which is public, um, and there's also a license review, but it's, uh, for a different use.

Um, so that one is a, that is one where you can ask potentially questions about licensing, not maybe personal information, uh, liability laws, but, um, there's a lot of lawyers there as well. Um, and, um, probably FSF has something as well set up, I don't know. Um, I was never actively involved with the FSF, so I don't know what they have, but they, I, probably they have something.

Um, so when it comes to GPL, that they might have something. Um, so that would be options. I mean, you're, it's not 100 percent sure you're going to get an answer. Um, because you have like the attorney privilege is. is a very important thing when it comes to lawyers. Um, and there's also liability. You know, if you're acting as a lawyer for somebody or acting as an attorney for somebody and you're not, especially in the U.

S., there's serious liability for that, for the lawyer in question. Um, so So, but if any, it, at least it's going to be, it's, I would say that would be the best foot in the door. Um, so it could peak, you could pick, pick some, uh, somebody's interest and they'll just, you know, answer your off list and say, well, you know, you didn't hear it from me, but maybe you should read this and this for more information.

Um, oh yeah. And the, yeah, yeah, of course. And this SFLC as well.

Jonathan: So that's the, those would be. Yeah. The Electronic Frontier Foundation and the Software Freedom Law Center.

Matija: I don't know how active they are. Right now anymore, but yes, FSFLC was, uh, had a thing going for a while. And there's also the Software Freedom Conservancy, um, that's also an option.

And clearly if you're near or if you're a member or near one of the big, Open source foundation. So like, you know, Apache, Eclipse, um, Linux foundation, et cetera. All of those have, um, legal people on board or have access to teams. So if you have a problem, especially if you're a member, you know, you ask it there and you might, you know, you probably will either get an answer or get, you know, put Told where to go.

Jonathan: Yeah. Oh, that's, that's interesting. I hadn't, I had not thought of the Linux foundation as one of the places to turn to, that's actually really useful. Um, all right, Randall, did you have any, any final questions that you wanted to get in?

Randal: It was, uh, I was just going to point out that, uh, when my case first started, the people of the EFF were very helpful in contacting me with, uh, Mike Godwin.

As sort of initial contact, and he was their point guy for years on, uh, where to go to about legal issues relating to look to your foundation's sort of charter. Of course, he couldn't represent me, so he got me directions to get to the local guys, and he also supported my local guys. And, uh, and helping me defend against that, uh, the pen testing shouldn't make you a triple felon.

But, you know, um, it was, uh, it was really interesting, um, uh, being able to work with them as an outsider. Uh, it was kind of fun.

Matija: Yeah, but maybe an important thing here also to mention is if we're not talking about the big Foundations that have like a very wide net You know EFF is is it's relatively big and pretty important, but it's also very u.

s. Centric. Yeah And the same would be for a lot of others. So As, as, as I said, like, you know, the jurisdiction by jurisdiction, you're going to have different issues. And as Randall said, you're, you know, if you get, if you get into a court case, it's going to be a local thing. So, you know, if you're going to come, if you're going to go have a, have something in Canada, you know, a U.

S. lawyer is not going to do you much. Um, they could be really smart and everything, but, you know, they're, they're different. There's, you know, they're not allowed to, they're not admitted to the bar. And if they're not admitted to the bar there, then they're not admitted to the bar. Um, so maybe, you know, depending on, you know, who listens to this podcast as also, you know, make sure that you search for something in your jurisdictions.

Like, you know, for the EFF equivalent in Europe would be something in France. It would be something like, like what the, or April. For open source or, uh, Edry, so European digital rights initiative is maybe something that if you don't have an open source issue or a free software issue, but you have like something else, like EFF ish, um, Edry would be one of those that I would go to if I had a question that was, you know, maybe PII related or something like that.

And because the, the really good thing about Edry is also their network. Um, Um, they're not just an organization, so they know who to send you to, so that would be

Randal: the, this might be a 15 minute question, but let me at least get it started and see how far we can get in a short amount of time, um, over the years, there have been.

Sort of misunderstandings and gaps in interpretations of what legislators think software is about and what we all know software is about leading to some really weird laws like the DMCA. Um, is that getting better? You think over time, or is that getting worse? Is it getting worse because it's international?

I mean, what, what's, what's your take on how well. Legislation is keeping up with the technologies that we have.

Matija: I mean, legislation will always, almost always. So that, that's come, that brings me back to my, you know, something I mentioned, like at the very start is why I started studying law is, um, I found. I find law interesting because on one hand, it has two functions.

On one hand, it has the function of, it has to be, it has to represent the situation, um, that we have. So it has to be applicable to the real world. Um, And if it doesn't, there's a legal theory, I think it was by Jelinek, an important figure in legal theory, uh, who said that if it doesn't, if law doesn't correspond with, you know, real life anymore, it becomes unlaw, and therefore ignored.

Um, and on the other hand, you have the other side is law Not always, um, but sometimes also has the function of pushing society in a specific direction. So it predicts the future in a way. Um, so this is something that you would have, like, when you know, you know, where you have things like, um, when, uh, women were given voting rights, where they didn't have them before, that was definitely something that, it didn't just happen, you know, you had to enact a law for that.

Um, and this is, you know, the interesting, this bipolarity of, um, the functions is what I find interesting. And I think, I mean, the cardinal sin of, of. Intellectual property in general is that it's not something that's very new. It's like, you know, 200, depends on how you look at it, 200, 400 years old. Um, so like most of the stuff that was written before or drawn before or composed before doesn't fall under copyright at all.

It never did. It didn't exist.

Jonathan: Right.

Matija: Um, And now then we suddenly have this thing, which started depending on, you know, where you look at it, if you look at it from the UK, or which then, you know, developed into the US as well, copyright law, or you'd look at it from the, from the French authors rights, which developed to most of continental Europe, you have different spins on how, what was important.

Um, and why, uh, copyright happened, uh, roughly ish at the same time. Um, so that's already a cardinal sin in IP is that it exists. Um, and then it's how do you shoehorn something that is software into this? Um, and early there was a debate whether software is, you know, before, before I, I'm gonna, it, it, interestingly, I like, there was a time where software wasn't copyrightable.

It was public domain, period. And it was actually, I think IBM, actually, software licenses predate software copyright because IBM was slapping on licenses on their software, on their source code, because back then it was just source code. Um, on source code. With the intent of, you know, this is either going to be a contract, um, uh, or ignored or might happen to be something.

Um, so

It was always, I think copyright was always a bad option, um, for software because software, like part of, there's a, you have like the technical distinction is like, there's a test of originality, um, in software, uh, and in, in, in copyright in general, and the thing is like the more formal something is, Okay, now, coming back, um, copyright doesn't, copyright protects the expression, it doesn't express an idea.

So, If the expression and the idea, you know, if there's only one way how you can express an idea, then it becomes a fact, then it's not protected by copyright. Because that is, for example, would be, you know, arguably is the case with, uh, config files. Um, because there's your, you know, if you want to set up a config configuration in a file to do this, you know, the idea, very, very little you can do, you can be completely wrong.

interesting things around there. Um, so it's, it's a mismatch already. And then, you know, the length of the term for copyright protection is, you know, for in general, is, you know, A bit silly, you know, who gives a, I mean, if I die, I mean, now I need to, now, now, now, now people need to wait another 70 years or 90 years after I croak that they can use my, that my stuff gets into public domain.

I mean, what, how does that benefit me? It doesn't mean theory benefits my heirs, et cetera, et cetera. Um, but it's also, um,

It's also a mismatch in the sense of, I lost it. Um, it's, it's late here. So just so you, just so you know, there's a, there's a, it's, it's not dark because I'm in a basement.

Jonathan: It, it, it reminds me of the, uh, the recipe book, right? Like the actual recipes in a recipe book. Yeah. You can't copyright because they're just, they're mechanical.

Matija: Yeah. So, and, and this is why there's a mismatch already in the start. Um, you know, you could argue that. You know, patents would be a better option, but then, you know, we get into a completely different problem. Patents have their own problems. Yes, everything has its own problems. So the question originally was, is it getting better or worse?

Both.

Randal: Wonderful answer.

Matija: That's fair. So it's, it's like, it's similar is like with the CRAs as an example or the security legislation that we have on even in also in the U. S. Um, There's on one hand, you know, legislators seem to understand software better from the point of view. Oh, there's Security issues there.

Somebody needs to make sure it's it's it's done properly and then you know, how do you do that? So, I don't know. I'm

I have mixed feelings on this. I wouldn't say I would I wouldn't say I'm cautiously optimistic. I wouldn't say I'm pessimistic. I'm just, um, it's, it's the way it is. And, um, you win some, you lose some. Yeah. I think, I think they understand more than they used to. Um, but whether they understand enough to make everything better is a good question.

Yeah. Yeah. And the problem with law is always, I mean, it's not gonna, the law is typically Um, written by politicians and you have a lot of lobbying and a lot of, uh, stakeholders involved in pushing the agenda here and there. And this is maybe an important call through for all the open source, uh, you know, software advocates out there that you need to make yourself heard because, you know, the, those who have an interest to have, you know, to keep copyright as difficult to use in software Those guys typically also have a lot of money, and they do lobby.

It's not like they don't.

Jonathan: All right. Well, we have hit the bottom of the hour again, and I've, there's a couple of questions that I've absolutely got to ask you, and that is, what's your favorite, and it's totally different, what's your favorite text editor and scripting language?

Matija: Ooh, um, I mean, the geeky answer is Vim and Python, but in practice, I mean, uh, I use most of the time I use Kate.

So the Katie's, uh, advanced, uh, text editor. Um, I did try Helix and I liked it a lot. Um, there's one or two things that I don't like about it. I, if you like the idea of VIM or VI, but you're not in, you know, you don't have the muscle memory yet. Helix makes a lot of sense. Um, it does things in a more logical way than VIM does.

Um, But I just, when I, when I don't use it for a while, I just start doing Vim things and then they don't work. It's like, ah, um, but, and same with, I'm, I'm not really good at programming. I did, you know, ages ago I did in school. I did try to learn, uh, I did go to a course on, uh, Pascal. Uh, I was never good at it.

I just know, I just still remember a lot of semicolons and read lines and write lines. Um, I was, at some point I knew, I learned how to program in Python, I, but, um, in practice, I use fish a lot. I have a lot of scripts in fish. I love fish. Cool. Fish is great.

Jonathan: Yeah.

All right. Uh, we appreciate you being here.

Uh, very, very much. That is, uh, Mattia Mattia Shukla. Yes. Uh, that's, that's what it sounded like in my brain. I just couldn't get it to come out of my lips. We appreciate you being here very much talking about some legal issues and, uh, it was a lot of fun. I appreciate it.

Matija: Likewise. Thanks for having me.

Jonathan: All right.

Uh, so yes, thanks for staying up. We appreciate it. Yeah, no problem. All right. What do you think, Randall?

Randal: Well, uh, covered a lot of areas. Uh, it's, uh, it's, it's never uninteresting to talk about computers in the law. Uh, particularly for those of us who have been at the, Short end of the pointy stick. So it's like, okay, what's going on now with how these are all working out?

A couple of subjects that we didn't get into that would have been interesting, but you know, for a matter of time, I didn't want to bring up anything else. But, um, the idea of, uh, my personal liability is an individual contributor contributor using LLM based solutions to create my code that are later found Themselves liable for reproducing copyrighted code without disclosure, and I'm wondering at what point Who's we haven't tested this yet in court like we were talking about earlier It says it hasn't been tested yet It's going to be interesting to see what the first test case actually comes out to be where Uh, uh, I'm using something like co pilot and it inserts GPL code into my code.

I then submit it to a BSD style project, uh, which would not have the same GPL protections. Uh, what, where do the fingers point to in that chain?

Jonathan: Yeah.

Randal: And

Jonathan: that is interesting. It is, it is such a big question that several projects have simply said, we will not accept any code that's written by AI or LLM,

Randal: right?

Jonathan: So

Randal: that's, uh, it's interesting. And that's also why I stopped using Copilot once I sort of discovered that was happening. Uh, Codium, uh, C O D E I U M promises that they're training only on licensed appropriate material. So that does help a bit. Uh, and I'm hoping that Gemini also trains only that, because I'm also using Gemini quite a lot these days for that sort of stuff, so.

Yeah, Gemini is getting pretty smart. Uh, for those of you who haven't tried it yet, check it out. They're doing some really cool stuff.

Jonathan: That is an interesting segue, because next week, the plan is to have the guys on and talk about the open source AI definition. And that has been, yeah, that's a big deal.

That has been kind of a, uh, a brouhaha because everybody has a little bit different idea of what open source should mean when you're talking about artificial intelligence. Yeah.

Randal: So

Jonathan: OSI has put their oar in and they've got a definition and we're going to talk to them about it. Hopefully. That's the plan.

Awesome.

Randal: Awesome. That sounds good. I may watch that show.

Jonathan: Yeah. Did you have anything you wanted to plug, Randall, before we let folks go? Ooh.

Randal: Just in general. I am doing, uh, these days, I'm not doing anything with Pearl. I am, uh, for those of you that know me as the Pearl guy, uh, I'm doing everything with, as you can see, by reading the hat.

Flutter, flutter and dart are my, uh, primary things. I'm one of the 10. Uh, Google developer experts in the United States about the Dart and Flutter arena. So I spent a lot of my time teaching online, uh, making presentations, uh, answering questions on things like Stack Overflow and Discord and stuff. And, uh, I'm also always looking for commercial contracts for that and have to keep me off the street and out of jails.

I like to say that was more literal than before.

I would say it's a joke before all that happened and then I started to say, oh, that's a little too real now, but, um, but yeah, uh, and, uh, other than that, uh, I'm, uh, I'm, I'm happy. I'm getting a lot of stuff done in my life. Uh, I'm, uh, I've taken up karaoke, uh, running a karaoke show on Friday nights now, so I'm having a great time helping other people sing and choose songs, stuff like that.

It's a lot of fun. So yeah, it's all good. And, and thanks for having me back on the show as always. Cause I. I do kind of miss the hosting role for this,

Jonathan: but

Randal: not enough, not enough to actually do it, except when asked on temporarily. I, I, I am very happy to have the five hours a week back that I was putting into this show for 13 years.

So I'm very happy that you're doing that now. And all y'all do is tell me. Minutes before we go on air that here's the button. I gotta press so i'm much happier with that now

Jonathan: Yeah, well, we we appreciate having you it's always fun to have you back and we'll have you again in a few weeks All right If uh, if folks want to find me, of course Pack a day is one of the places is where the security column glows live on friday morning That is the home of floss weekly now.

We appreciate them for that And then I still have the untitled linux show over at twitch. tv Twitter. tv and we have a lot of fun with that covering the news and the tips and all the things with Linux. So you can check that out as well. We appreciate everybody that caught us that watch us live and those that get us on the download and we will see you next week on Floss Weekly.

Episode 814 transcript

1 januari 2025 | min

FLOSS-814

Jonathan: Hey folks, it's time for Floss Weekly and this week, Rob joins me. We talk with Alistair Woodman about FR routing, but also the future of open source, open source business, software liability, and all kinds of other stuff. It's a lot of fun. You don't want to miss it. So stay tuned. This is Floss Weekly, episode 814, recorded December 31st.

The Banksy situation.

It's time for Floss Weekly. That's it. It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we're back after a couple of weeks break for the holidays, which is kind of ironic considering that today is December 31st, so we're not quite through all of the holidays yet, but that's okay.

Uh, we've got a guest, we've got a co host speaking of which, uh, let's go ahead and bring Rob on our other resident, uh, networking engineer, networking experts. And, uh, Rob, I think we're going to have to take our networking expert hats off for today because we've got, we've got a, a real expert, or, uh, he was telling us before the show that he doesn't do a whole lot of coding these days.

So, so maybe he's the business manager for the real experts. We'll, uh, we'll ask him

Rob: about that. Yeah. I'm interested to see what he has to say about, uh, about the business side of things.

Jonathan: Yeah, yeah, that's something we're very interested in around here. Um, so our our guest today is Alistair Woodman and he he has an interesting background Uh, he says by education.

He's a physicist And by inclination he is into Product management and business development. He's done apparently some coding in the past, but he said he retired from that. He hung his head up on that one. Um, but he's involved with some really neat things here, like, uh, FR routing, which I've been telling people this week is going to be about like sort of the, the deep internet protocols because they do, uh, they, they do some of the, the routing things that you've never heard of, or maybe you've heard of once or twice, but you don't have like, BGP, the Border Gateway Protocol.

I know, I've never messed with BGP. Uh, I've written about it because every once in a while you'll find a story where, uh, somebody advertised a bad BGP route and we accidentally routed the entire internet through Brazil for a few hours. It's like, I'm sure nothing malicious happened as associated with that, right?

It'll be fun. All right, let's go ahead and bring him on. Mr. Alistair Woodstock. Welcome to the show, sir.

Alistair: Hello, good morning, good evening, whatever good next year for those of you, uh further afield in the east So,

Jonathan: oh,

Alistair: yeah.

Jonathan: Yes. I've i've already gotten some well wishes for 2025 in my various discord channels and that's always fun Yeah All right.

So What, what, what all do you have your fingers in? What all are you involved in? Um, are you a wizard of the deep internet magic? What, uh, what are you up to these days, sir?

Alistair: Well, so, well, I, I left, uh, formal, you know, working business stuff in 2011, uh, after putting in a solid 25 years in high tech. Um, for companies being well paid.

Um, so I took early retirement and then I pretty much figured out that I didn't enjoy golfing. Um, so I think the last time I did play golf was as an 11 year old. So I pretty much figured that I wasn't going to enjoy it. So I decided what else am I going to do? And I actually figured out that I really like hanging out with nerds anyway so, um, I still hang out with nerds and I work on open source projects and And hanging out with nerds is either a terrible thing for many people and uh, but I actually enjoy dealing with people who are sort of uh, uh, very diverse very focused on what they're interested in and um, I mean It's sort of a popular meme here in silicon valley that we have a lot of you know on spectrum Uh type individuals and I I like that Working with them and I like getting, you know, good results out of them.

And I like helping them, um, interface with, uh, business issues because people in the tech community tend to be very focused on what they're doing. And of course the thing that they've done. Is super important and super useful. And the question is how are you, you know, keeping a roof over your head and paying your bills?

So a lot of the time, um, I spend my time running around, making sure that that part of the process is working for open source projects, unfortunately, I can only do a couple. Um, but I'm heavily engaged in the two that I'm active in at the moment.

Jonathan: Okay, so there is, there's a lot, there's a lot in just that opening statement and uh, I didn't realize we were gonna go down this route but it is super interesting to talk about.

Um, and that is that it's quite a, it's quite a skill set to be able to talk to nerds. And, and I, I very much appreciate that thought of, Some of these people are on the spectrum and I think that's true because I've known some of them over the years and It sort of is a Useful, but also important skill set to be able to I I call it translating.

I refer to it I translate for nerds and I think that is that is super useful. Um, I'm i'm thinking of a there's a there's a youtuber, uh, david plumber, I think um that I follow he worked at microsoft for years and he's like at some point people told me enough times that I ought to go get tested to see whether it's on the spectrum and I'm like You know after laughing them off enough times I finally went and did it and then I started reading about being on the autism spectrum And he goes so many things about my life started making sense.

I think it's that's super fascinating I think it is a real thing. I think it definitely is.

Alistair: Okay. Well what I think it attracts people with um, special Gifts and skills, right? So some of it can be just pure IQ related stuff, but a lot of the requirements for being a very good coder are sort of very good memory, understanding of structural things across space and time, being able to sort of figure out what that really is.

means in terms of where to find something in code. Um, and these are very interesting skill sets. Um, you know, people who, you know, study the Hebrew Bible, um, also have those types of skill sets, right? It's, it's very focused a lot on the written word and, and what that means, right. And it attracts a very gifted set of people.

People and sometimes they also have, you know, um, other proclivities, right? And as we all do, and Uh, they sometimes want things to happen in a particular way and get frustrated when you know, it doesn't happen Um, or the rest of the universe doesn't appreciate that. They aren't really the center of the universe So

Rob: I feel like I feel like a key strength may be the ability to hyper focus on something Yeah, whereas me when I got into it and technology I feel like I went the opposite way and I just so many things are flying at me from so many directions that I'm all over the place and and if I had maybe more of a hyper focus, I could really focus and go down one of those paths, but I'm more of a jack of a lot of things

Alistair: and that is an interesting point because when you building larger scale.

Communities and interrelationship you can't afford to be hyper focused on You need to be able to Frequency shift from being very focused on a particular thing at the time being and then try to build connectivity or causal relationships to other things and and that's um, That's an interesting challenge that that one faces when you're doing business development, right trying to find the Yeah The way of, of connecting something that a project's doing with a need that somebody else has somewhere else.

Jonathan: Yeah, so one of the, one of the interesting things that I've been faced with a few times, honestly, trying to make money writing code. Um, when, when you, when we're writing code, things are very, um, binary, you could say. You know, either this code is written correctly and it works, or this code is not written correctly, and either the compiler tells me, or I get, you know, runtime errors and things break.

Uh, the business world is not quite that, uh, cut and dried. And oftentimes when we are convinced, you know, our internal compiler is telling us that we've got just the awesome, the best business idea. This is going to make us millions. And then you go to launch it and nobody cares. Uh, it's, it's quite a different world.

Um, I have found in trying to take my coding acumen and move it into the business world.

Alistair: Uh, that's very true. Um, and both in the business world and even just people using hobby projects from one another, right, amongst open source stuff, right, the new mousetrap or somebody's done something that already exists somewhere else and everybody goes, I don't care, right, the marginal improvement isn't good enough.

So. You know, have a nice day, right?

Jonathan: Or, or the idea is so niche, there's, you know, all of five people around the world that are ever going to care about it. Yes. Yes. Been there. Um, All right. So let's, let's get into, um, a little bit of maybe, maybe background. So you're involved from what I understand heavily with, uh, the FR routing.

Project is that is that the only one you said maybe there was two projects that you you kind of have a hand in

Alistair: Well, so the other thing that I'm involved in is the Erlang ecosystem foundation, which is a group Of projects so, um, the the two are different in scope and scale um, but if we talk about the free range routing project or frr as most people refer to it as um, you probably Um, well certainly people listening to this will be getting their traffic over, um, routers that are running FRR.

Um, we're usually, typically in OpenWRT, um, releases. So anybody who's running on a lot of the embedded hardware for home routing stuff, it's got OpenWRT in some flavor underneath and with a routing stack on that. Most people don't turn on all the different daemons that we have and support. But, uh, some people do run, um, decent routing topologies in their own home.

Um, so some people turn OSPF and ISIS and other type of stuff on and have relatively large, um, home networks already, especially if they've got a lot of IoT devices. So, um, that can be. Um, n not the typical home. I've just got a wifi and I need need an uplink type of thing. Um, which is still what open WRT will serve, but many people turn naturally a lot more knobs on.

Um, the FR R is also used at the other end of the scale on Sonic, um, which is a white box nos, and there were a couple of white box noss out there. So via, um, BISD analytics. Um, that use FRR for general routing purposes. So the forwarding plane and control plane stuff is not part of our project, but as many things in the open source software go, we're, they're downstream of us in terms of integrating FRR into their projects.

And they, Then talk to the appropriate hardware, silicon board support packages and whatever it is to build the embedded system, um, installations. So we've got a lot of code, um, on different code bases, uh, usually. As far as the users are concerned, there's a binary, although many home hobbyists will compile their own and put it on their hardware, but most people will be just downloading something and running it there.

Um, so that's what FRR does. It's also used internally, um, By telcos and some of the hyperscalers for um, sometimes purely just operational backup But other people are using it for more interesting projects so it's um, it's been around for over 20 odd years now because it used to start out. It was quagga and then it got forked Well zebra quagga and then free range routing.

So we stuck with the animals, but we went from Uh, we went from the individual animal to the the paddock basically when we changed the name to free range routing Yeah,

Jonathan: yeah, that's that's funny Um, one of the other protocols that I saw that is part of frr is uh, BGP, the board of gateway protocol. Um, I'm curious.

Are there, are there any of the, like the big level one ISPs that use any of the FR routing stack in their sort of big iron routers?

Alistair: The answer to that is yes, and then you'll have to go figure out the details yourself. So, so yes.

Jonathan: That's fair. That's fair. So it is, it is very, very likely, if not certain, that our, our bits that are being broadcast right now are going through the FRR stack. At least

Alistair: in one of the nodes, yes. At one of the nodes, yeah.

So somewhere along the path, yeah.

Rob: Is that because it's built into hardware they're buying, or are they building stuff upon this, are you suggesting?

Alistair: Um, I would suggest that from purely a percentage likelihood, it's mostly people have got a binary that they've got that came bundled with the hardware, and this will typically be the home router of most cases.

But if there are more. FRR on that link and possibly further up towards the data center or hyperscaler or some service that you're accessing, um, then they probably built it themselves on, on white boxes and possibly be tinkering with that. Um, but it is possible to just download the NOS, uh, you know, from, from folks and then just turn a binary on as well.

Jonathan: What's that people

Rob: have.

Jonathan: What's the, what's the license? What's the license FRR is under? Then we'll give it back to Rob.

Alistair: GPL. Okay.

Rob: So that sort

Alistair: of makes a, okay. Sorry, Rob. Go ahead. Go ahead.

Rob: Go ahead on the license.

Alistair: Right. So, so the interesting thing about licenses, I mean, I think it makes a hell of a lot of sense that routing is under a GPL licensing model because if people make modifications to it, they should be making modifications to a standard routing protocol that you really don't want routing protocols that can't interoperate with one another.

So, um, you sort of really, from a community perspective, it's a better solution from a licensing perspective than to allow um, People to just fork their own and then do what they want. Um, there are some hyperscalers who actually do use it internally and have built their own flavors of routing and because it's, uh, they're running it as an internal, uh, software, they do not have to give those, uh, modifications back, but many of them do.

Um, and it, but it makes sense if you want to interoperate with a router on the outside that you better be, you know, capable of doing that. So, forking and, and not reconverging is a bad thing in networking.

Rob: So, if people are using these, uh, higher end routing protocols on, on their Linux system, something they're building or something they bought, Um, Is, is this, you know, this being around for 20 years, is this what they are using or are there other, um, alternatives out there that people use?

Alistair: Oh, there are several other open source alternatives. Uh, BIRD's a BGP, um, code base that, that is quite popular amongst, um, service providers and inter exchange carriers. Um, the, um, The, if you want to be doing multi protocol routing stuff, uh, you're basically in the FRR domain, or you're going to be using, you know, Juniper or Cisco because you, you know, you, nobody got fired for buying Cisco or Juniper products, right?

That problem, right? But if you are willing to, you know, be fired and want to run multi protocol routing, then you'll typically be running FRR.

Jonathan: Yeah, so if, uh, if, if I do actually finally pony up the money and, uh, buy, buy into and get an ASN, an autonomous system number to be able to truly own my own IP addresses, uh, it sounds like FR routing is the, uh, is the project that I would want to go with because I don't have Cisco or Juniper money.

Alistair: Well, so you're certainly being a bit of a piker if you haven't bought your own AS, right? I mean, that's relatively cheap. It's like 500 a year. Exactly, right? That's chump change, right? So you should at least go get the AS. Now, whether you do anything with it is a separate topic.

Jonathan: I look forward to being successful enough that I can say the 500 a year is chump change.

Okay.

Rob: Question from Discord. Does FRR support MPLS?

Alistair: Um, yeah, that's an interesting question. Um, we do. We're on boxes that support MPLS. Um, but the, the routing it, so it, so the answer is partially we don't get in the way, but, uh, we don't typically, um, It usually requires people to have done something else or want to do something else.

The Linux stack does actually support MPLS and you can turn it on. And it will work under those, uh, circumstances. If you're running on dedicated hardware, you'll need the drivers and other types of things to do MPLS on the hardware. So, the answer is maybe, uh, probably, if you just want to run it on Linux by yourself.

It doesn't work on FreeBSD.

Rob: I imagine MPLS being, being that path, um, FRR would support the routing over the MPLS.

Jonathan: Correct. Makes sense. Um, so it sounds like this is a, a fairly complicated project, FRR, and it supports some fairly complicated protocols. Uh, How did, how did we get, how did we get to this point?

Like how, first off, how did, how did FRR get started? And then at what point did it kind of turn into, cause it sounds like it's been commercialized now. It's been used in a lot of places. Hopefully there is some money being paid to help keep it supported. And so how did we get to the point we're at now?

Alistair: Um, so as I said, it was started over, uh, 20 years ago as, um, By in Japan to actually try building routing and as a self study, uh, type of project it as, as it was. Under the Quagga flagship thing, it was already being used and was a part of, so Quagga still was and is used in some of the embedded systems.

So it's probably still out there in, in old routers that somebody hasn't updated for a long time. Um, which is one of the other topics that we might get into is, you know, how to end of life and, and maintain software over time. Um, it's. So when we, when we forked it to FRR, there were, um, business interests.

So there was, um, couple of startups that at the time, if you remember software defined networking and a whole white boxes were a thing. Um, so it became, there was enough startup money being put into the space to actually make it from a very good. Um, code base, but we're still not really competitive when you looked at, you know, Cisco and Juniper and, you know, people would use it, but many people would scoff at it, um, as not being sort of industry ready.

And certainly if you wanted to run networks, the old adage about, you know, You're sleeping at night and your page are not going off. Most people will buy Cisco and Juniper gear. But with the advent of white box, uh, switching, um, there was enough money flowing into that that startups, um, worked on making the code a lot better.

So Cumulus Networks was, um, around at that particular period of time. They no longer, Uh, exist as a separate entity. They were acquired by Mellanox and Mellanox was acquired by NVIDIA at about the same time. So there are folks over at NVIDIA now working on the project and they're a, um, major contributor to FRR.

Jonathan: And just for, for anyone listening that might not know what, what exactly do we mean when we say something is a white box, like a white box switch?

Alistair: Um, it basically a, uh, uh, Vendors have built something to spec, um, so this happened as well with, uh, what happened in the hyperscalers, where they, where they basically said, we don't need to be buying all these pieces of equipment, we can have servers that we will specify, please don't put these extra chips in there, so there's a lot of, you know, specifications for white box servers, and then there were also specifications associated with um, switching infrastructure, we can just have layer two on layer three switching, which doesn't need all the bells and whistles, um, or rather, they were so focused on chip count and hardware costs that they would specify this is all you need to put in this, you know, don't, don't over Egg the hardware.

This is all we want. Please put some open source software on this and we're off to the races. So um white box white box anything is a Statement about the fact that it wants to be relatively generic um and not particularly complex, although Now they are actually sometimes quite complex, right? Uh, but, but you know, they're, they're trying to create a lot of the interest in the white box switching thing was to remove, um, to, to restructure the business in the sense of having a vendor specific, um, my things better than your thing to actually, this is just a utility thing.

It needs to do these basic things. Please just build them at scale cheaply.

Jonathan: Yeah, definitely. Um, okay, so back to FR routing, and I think, I think back into the kind of the business side of things. Maybe the first thing I want to ask is how did, how, how did you go about, so I don't know if you were there at the time, how did they, whoever was, was doing FR routing, um, How did they get in touch with and start getting that startup money to flow into the open source project?

Uh, we see sometimes that open source projects get used for startups, and sometimes it's difficult to get that money to flow back to the actual people working on the code. So

Alistair: that is actually a very interesting question, and I do think we're a relatively rare bird, and if we segue later in the conversation to other examples, I think we are, we are Accidentally rare and that it ended up with this good symbiosis and so, I mean you you Yeah, you were talking about the whole concept of curl a couple of weeks ago, right and just how well supported curl is correct Uh, how, how well used it is and now that it is almost has a business model, uh, associated with it.

Right. And, um, that's just astounding to me, right? I mean that, that we've got something that's, that well used and that's still struggling to get, uh, a viable business model. Yeah. And so when I look at that and some of the other. Uh, projects which obviously do struggle, um, the, the reason that I think we got lucky is that routing is a complicated problem and back to this other issue, it needs a lot of interworking support.

So, when, uh, Where the F are an active project, there are probably 30 very active committers to the project and there are probably another hundred that commit, um, more than 10% of their work time to adding features. And then there's a long tail of. People who just do some feature or bug fix, you know, that show, show up, but because this all has to interoperate with other people's equipment, specifically Juniper, Cisco, and the installed base out there, um, you have to be able to prove that you've not in, you've not just added code, but you haven't put any entropy into the system.

Uh, And so there's a lot of testing necessary with those things to, to ensure that. That you've not broken the code and to your observation earlier, that you think it's a bit of a miracle that networking sort of works and you've got all these things that can connect to things, but it's not quite clear and some of them are in the process of going down or being rebooted and the network, you know, continues to work.

That's because there's all this complicated state machine machinery in there, which can easily get broken by neophytes. And even the experienced folks can break it too. So we have a very, very large CICD. Pipeline and we throw a huge amount of resources at this And so the our budget for doing testing on this type of thing is, you know over You know half a million dollars Uh, a year in terms of testing, both in terms of people and CPU utilization to keep the project around.

And I think it's obvious to anybody who works in the networking space that that's the cost of doing that business. So, The the pool of companies that are involved using the software understand that it's that hard because Everybody has worked in those companies Basically used to work at cisco or juniper in the you know sometime in their career Um, or they might have worked at you know, some of the other guys like alcatel lucent or whatever it is, but people know What the cost is?

And and how much work you need to do to avoid regressions creeping into the system So It it's not too much of us of a hard sell. I still need to go talk to people about this um, but it's it's it's a communicable problem amongst a very small number of Consumers that allow the business development process to occur Um, and we sort of all know one another that helps.

Um, so this is not very we're not an anonymous project So if you if you thought of stacking um some relationship relationship diagram or sort of two by two matrix of how how well people so Daniel, for instance as he Set on the program, right? Doesn't know most of knows nobody who uses his code.

Everybody uses it, but he doesn't know what they're using it for and how they're using it. Right. And we're the opposite end of that spectrum. We have a reasonably small number of. People who use the code seriously, and they also know why they're using it and they know who we are. So we can, we can forge business relationships in a much tighter manner.

So that's how we manage to run an, an, a system which is, um, which is working from a business perspective.

Jonathan: Yeah. What, what is the, what is the actual, um, like funding model? There's a couple of different ways I can imagine going about this. So do you, do you go to all of those white boxers that include your code and say, Hey, why don't you pay us 2 percent of your MSRP on everything you sell?

Because we're in there and you know, we need support. Or do you go to, you know, one of the companies and say the, the amount of, of support we're giving you, we think is worth a million dollars a year. Why don't you write us that check each year? How does that part of it work? I'm very curious about the nuts and bolts.

Alistair: it's it's certainly not on a per copy basis um So it's much more along the lines of the this is our support load this is how much it costs to run this infrastructure and um, and then just saying what a fair fair cost payment would be To those people and and some of it scales by the size of the entity in the organization um, and It's it's just a usual sales negotiation thing, uh, because the the quantum size the number of Number of parties that i'm having conversations with is as I said reasonably small.

It could be larger Uh, there is a group of people who still fall into the free rider You know user category that I can't get to or I don't know how to get to them in organizations um, but it's very much You You know, I basically tell them how much it's costing to do the job and then discussing, you know Whether they're a bigger or a smaller user than somebody else and and what does the pecking order look like?

Yeah, and again, this is a tractable problem because it's small right So those

Rob: aren't those, uh, fees or whatever you discuss with them, that's not backed by say a support contract. Is it just complete voluntary?

Alistair: Well, we do also provide support to, to companies who want support. We do that too, but we do get a, a set of donations, which are pure donations to run the CICD.

So we do both.

Jonathan: Yeah, that's interesting. Um, How, I guess the first thing that comes to mind is not every, not, that doesn't make sense for every project,

Alistair: right? Well, that's, that's my observation, right? We are, I think we are quite peculiar or special in that particular regard. The, the other class, similar class that looks like this is, is projects which were started by some very large, Entity that then get put into the public domain, uh, and still have large support from those entities, right?

So if you think about something like Kafka or any of those types of projects, um, they typically, um, Have a similar model where people understand how that's going to work and people continue to contribute. Um, as I said, I think we're relatively is that we organically grew around a type of thing. The code itself did not come out of a large entity as a, an open sourced project.

It, it organically grew but has been used because of networking's focus by larger entities. Um, so we certainly, Are at an opposite end of other some spectrum than say daniel right as he's definitely at the other end um, and and I think we're not typical so if I go and look at if I put my Erlang ecosystem hat on for a while, right?

That, that, that area encompasses multiple different, uh, projects. And so the, because there's a virtual machine underneath that, that is the Erlang virtual machine, which is run by People who want to program in Erlang, or Elixir, or Gleam, or any of the other languages that run on, um, the virtual machine.

There's a much larger tail of people doing stuff, from the very, from hobbyists that are just doing stuff, uh, in their own area, to the other end of the scale. Um, Ericsson and, uh, Facebook with WhatsApp. that use this at, at industrial scale, um, in networks, but we have a much longer tail of people and it's much more difficult to, there are a multiple projects that run on top of the VM.

So for instance, Phoenix, uh, you entered, you had a conversation with Lars recently about the nerves project. Um, so there's a whole bunch of different projects and they all have their own scaling. Systems and they're much broader from a scaling system. So it's much harder for us as a community to reach out to everybody to encourage them to support because it's diverse, right?

So we we have in that space, I think over 1000 commercial users of the technology, probably more. And most of them are dark Uh, and many of them are

Jonathan: free riders There's there's something that's sort of changing that I I am hopeful is going to help with the open source funding problem Um, and and that is there's been some laws passed and there's laws being passed Um, I don't remember the name.

I don't remember what project it was But I saw this statement where someone from a project was was saying We had a business reach out to us and say we need this documentation And for us to be, you know, to, to comply with such and such law. I think it was, was something related to a, you know, software bill of materials.

And it's like, so this, this company reached out to us and said, we need you to do this work for us. And the person running the project was like, I couldn't believe that they had the nerve to do that. And my take on that is no, no, no, no, no, no, no, no. You don't get it. That's a good thing. And your response needs to be.

I will do that for you. Here's my fee.

Alistair: Right. So, so you're entirely articulating the level of excitement that I had when I first saw the Cyber Resiliency Act over two years ago. So, um, most of the people When I first started engaging with the European Union Cyber Resiliency Act wanted to put their libertarian hats on and run away screaming and say, you know, keep, keep these bureaucrats out of my, you know, code base foo, leave me alone.

And I have,

Jonathan: I have a lot of sympathy for that point of view. I sometimes fall into that myself. I have one of those hats and I occasionally wear it.

Alistair: And, and I did it for like all of about three years. Three seconds before the penny dropped that's like, oh Oh This law so the interesting thing is not the law What it says you must do that's not the interesting bit.

There's a whole there's Hundreds of pages telling you what you must do and there will be much more Specifications written about you follow this standard and do these types of things from an implementation standpoint the interesting thing is what happens if you don't do these things and the The law sees that companies will be accountable for Not following business best Practices so you can be held, you know culpable as a company and there's even um One of the other european directives the product liability directive will see criminal Um law associated with some of these types of things if companies and their senior executives Do not follow the law So the interesting question is, oh, that allows me to talk to people about what they need to do to follow the law and the Um, I had a very useful conversation with the Apache Foundation, uh, again about 18, well actually almost 2 years ago on this very topic, is that their observation was that if you, if you just look at the law the way it's written, As you're a company, right?

You'd say to yourself, okay, fine. I've got 10 software devs in my company and I'm shipping something at the moment. And if I look at all the requirements, I'm now going to have to put in place to manage this software, then, um, I probably need 30 percent more resources to do the testing at the backend and the documentation and all this other type of stuff.

So how hard can that be? Right. Um, I just need to hire three people. But. Then your VP of engineering turns around and says, uh, actually 90 percent of our code is open sourced. Yes. And. The, the executive team sits around and goes, okay, so we have to hang on. We have 10 people, but they're only writing 10 percent of the code.

So that really means we've got a virtual engineering team of a hundred. So we now need to hire 30 people. To do the testing and verification That's where we don't want to do that, right? So that's why Everybody thinks that the government's interfering with their you know, their business plans But the obvious other solution to this problem is well Turn around and talk to the folks who are actually maintaining this stuff, right?

Um,

Jonathan: yeah, it it seems like a a real golden opportunity for a lot of open source projects, and I'm, I'm, I'm very thankful that when the law was written and then amended and they worked on it, um, they, they did go in and acknowledge that not everyone publishing source code. Is making a product right because from what I understand in one of the early, uh drafts of the law Um individual open source developers were going to be liable and that was terrifying and a terrible idea Um, so thankfully that got that got yeah,

Alistair: I don't think it was ever I think they clarified their position.

I don't think it was ever their intent to do that, right? I I do think that um that they So i've had the I I discussed I describe this as the Banksy, um, situation, right? So, if somebody just goes and posts something on a GitHub repo of the, you know, I've done the best bubble sort algorithm, you know, look at this and, you know, weep, right?

In terms of, you know, how efficient this is, right? And that's it, right? Everybody looks at it. It's a work of art, right? This is not a product. Product. This is a a political statement. This certainly should fall under the category of freedom of speech and um Fundamentally somebody does pure work like that.

It's just matter of algorithmic genius, so people will be able to do that and you know, I tend to I did talk to many people and they were they were throwing their toys out of the pram to start with and I said look They're not going after the banksy Situation, right? The interesting question is are you actually then maintaining it and do people use it?

Jonathan: And

Alistair: this gets to uh, toma de de pierre's Conversation about well, i'm not your supplier, right? And at some point Teams need to figure out whether they are a supplier And they just don't know their consumers or that they are not a supplier the law will create a bifurcation in thinking and um, you know in the In the conversations that you had recently, right?

I mean it would look like curl is already You know, in the situation of, of accepting and looking for maintenance, um, money, right? So, that has become, that's moved that side of the boundary line in terms of that. And I think a lot of projects are going to have to ask themselves, which side of the boundary line do we want to be on?

Yes. Yes. And the other problem is, is if you want to be on the You know the support In some way shape or form side of the maintenance line, right? How do people this gets to your other question is like, how do you how do you figure out how to pay? That right. So I think the ethos of the communities are most people Don't want to have a per unit License fee for maintenance that I mean that would go back to the you know, the the times before we had You know open source stuff, but people need to cover the costs of the maintenance and That's We need mechanisms of expressing what the costs of the project will be and what a reasonable apportionment is.

Uh, and I, I don't like the idea of just one benefactor turning up and saying, Oh, we'll take care of this cost, right? I think that's also bad from a relationship perspective. You actually want Um, major users who are using it commercially to actually be able to put their hand up and say, yes, I need to have a, I need to know something about these people, especially if they're going to have to fill in S bombs, figure out when CVS are going to get fixed and all these things.

And this is, this is all new territory at the moment. So, um, to Thomas point. The regulators want to see a tighter supply chain, um, by definition, open source and Open sources doesn't have that connection built into it. And we sort of, we need to introduce that reverse path into the system, which is an interesting problem.

Jonathan: Yeah. And you have other people that are working on this, right? So like, um, uh, post open source, Bruce Perens is working on the, the idea of post open source. And it's, it's intended to be sort of an addendum to all of the open source licenses that says you get to pay for your software. And his, his idea is let's make a central clearing house that handles this.

And I think it's really fascinating. I'm hoping to have him on the show to talk about it. Um, but I, I think there's been just sort of a awakening in a lot of different places to this idea that there's gotta be some money that flow back into these projects because developers need to eat. And we expect open source projects to have, um, quality standards.

Of commercial offerings, but and honestly a lot of times the quality standards open source projects are higher than commercial offerings But you find out, you know after a while, that's not necessarily sustainable unless somebody's paying the bills

Alistair: Well, well, it's it's that and more right? Um, the So so let's take the I I you know, I I'm not in it.

So I I spoke To daniel recently in a couple of months ago in stockholm on this topic, right? Um, it's not it's good that he's now in a situation where he has Uh, he has a sustainable business model and he can actually be paid to work on the thing that he enjoys working on Oh, yeah, right, but what happens if he gets hit by a bus, right?

We need In principle some a project like that should have You Couple of three apprentices working on, you know, a sustaining plan for when he decides he wants to, you know, hang up his clogs and do something else. Yeah, right. And there is, if you think about this, this is sort of in the Middle Ages, they had better plans for these types of things.

Yeah. You know, the, the, the famous artists would have people in their studios who were helping them, you know, and learning to do stuff. And when they died, right, the, the guys took over and it took us a long time to even be able to figure out that some of the works of art were done by the apprentice, right?

And, and we need a system like that for, uh, You know, artisanal software, which is of the highest quality, right? I mean, it really is, right? Daniel's Daniel stuff is the highest quality. Yes. And so there are systemically. At least a thousand projects, possibly 10, 000 that should have that same level of coverage, right?

I mean, the NTP keeps coming up as the topic, right? Um, I'm pretty sure that that's the one that's the, you know, the one maintainer in, you know, um, that's exactly right. So, so you've got all these things that if they were not managed and, and what's the succession plan? Nobody's thinking about this stuff and we need to figure out how to make this happen Because otherwise something's terrible is going to happen and then we end up with these Oh, let's just throw a large amount of money at it from the linux foundation.

It's like We could Plan better than this, people, right? We could have, we don't need to have, you know, Y2K bug problems just randomly turn up on a Friday because somebody got hit by a bug. Bus, right? I mean, we can work better than this.

Rob: Yeah, and that's a problem. That's a problem just all over the place in commercial endeavors where you have one person who knows or does everything and you don't know what tomorrow's gonna bring with that one person.

Yep. Yep. Right. Well, which

Alistair: is one of the reasons that industry has said, well, this is why we don't want to use open source. But then, unbeknownst to them, from the executive perspective, their engineering teams have been, you know, borrowing the code because it got them away, got them to their finish line faster and better by doing that, right?

So they've inherited this problem, but they don't want to face the fact that they've had all this benefit and what do you do about it? Yeah.

Rob: And that really could be quite the contrary, where in a small if you have one or two people doing the development and something happens to them.

Jonathan: Yeah.

Rob: You know, you've got to start from scratch.

Whereas open source, you have people at least looking at someone who can probably step in.

Alistair: Well, yeah probably step in but let's be let's be real about this right what we would really like Is somebody people who were actually wing wing people? On a regular basis and that they were actually capable and knew how to step in right?

And and then you can avoid The sort of xz exploit problem, right? Because the the we're talking about The same thing that whoever came up with the cunning plan Of doing xz looked at the way open source works and go there's a poor maintainer over there who needs help So guess what? We'll provide him with the help and this time it was with malicious hats on right?

But why don't we look at this and go well? That's a systemic problem. We should be providing the help so that there's, there's a general, I mean, doctors, for instance, when they go on holiday, they have backup. Other doctors that you can call. My dentist says, you know, call this other person, and they have access to my records, right?

Other industries have figured these types of things out, and we haven't.

Rob: You know, in your notes, you, uh, you mentioned abandon, abandon wear and, you know, even myself, I, I, I don't do a lot of programming these days, but I used to do quite a bit, a little bit of, uh, open source and, and what I found personally, I mean, I never grew to the heights of really anybody, but I'd start to build something.

I'd put it out there in the hopes that, okay, this is going to get some attraction. Some people will come along and help me. Um, maybe I'll get some, some other contributors to it, and then I just Get tired and burned out because I'm doing it myself and it's like well Nobody's interested and I am banning it myself So I mean

Jonathan: so it's interesting to me you have some of these projects where people just that Sounds like fr routing was one of these to go back to that as an example It starts as a single person sort of a labor of love And you started throwing code together.

And, you know, Rob says he's tried this a couple of times and never, never gotten anywhere with it. But what happens when you do get to like critical mass and there are some people out there to depend upon your code? I guess the real question that so many of us have is how do you go from that single person project where nobody really cares and it's okay if you get hit by a bus to there are people out there depending upon it and we suddenly have a budget, how does a How does a project get from, from that point to the point of having a budget and having sponsors and is there, is there a blueprint out there for this?

Alistair: Well, at the moment, it's entirely luck, right? And a lot of that luck is based on, as I mentioned earlier, the circumstance of the connectivity between the producers and the consumers of the code, right? And whether they know one another. If you end up in these situations where there is Coupling between things like people are using.

So in the, the Erlang space, right? We have a web web server on, on Elixir called Phoenix, which if you just want to do live view, Phoenix stuff. It's it's excessively popular with people who want to use web services stuff, right? But if you're just doing web content, you don't really need to talk to the guys who are keeping the html Um machinery working underneath you use a bunch of features you do turn up right?

but in principle you're You're writing, you know, your web service stuff on top. So there are a lot of web servers that have been run by, uh, very large companies that are on top of the Phoenix framework stuff, but their connectivity, I would guess less than 10 percent of those users are actually reaching back into the community other than, you know, to hang around a mailing lists and figure out how to get stuff done and, you know, Bitch about stuff, but they're not actually there is no formal way of setting up that relationship um, you know, so it's it's a difficult problem and and I think This gets back to the the topic of if you do now need to verify your supply chain It's it creates a bit of a magnetic field on the businesses that they want to start to have that conversation

Jonathan: So the gpl specifically I think most licenses have this but I know the gpl does i'm familiar with its text It has a big section in there.

This code includes no warranty, you know express or written Or is express or implied it's the term it uses And so it's, it's very expressly part of the GPL that you do not get a warranty. The, I am not your supplier is basically built into the GPL. I do not take any liability for this code. Um, do we just need an addendum that some open source projects can add that sits sort of on top of the GPL that says, if you need a warranty, if you need me to be your supplier, there is going to be a cost involved and here's how we handle that.

Alistair: So that's a way of solving the problem right sort of the essentially the dual license Model and that might be the way that folks decide to go. I don't think it has to be the only way That that you could do that, right? I mean you can think about things in a very different manner of of Um, showing support and sponsorship or any of those other types of things.

So, it doesn't have to be that way, right? It could be that way, and I'm relatively neutral on that topic. Um, so, What what is obvious is that we need a better way than the one we have at the moment, right? That's clearly not working, right? And I think you know getting people to sit down and talk about What the appropriate ways of making things work would be better, right?

Um You know, I, I, the, it, it amazes me that, that we are at this particular stage that, you know, there's so much dependency on this technology from a societal perspective, and that we as communities haven't quite figured out what the appropriate, uh, ways of, of, Running this are now. I'm not saying that the conversations are not happening, right?

Because I've been on many of these calls You know with the folks in European Union spaces as well as folks in the u. s Domain, and I think things are move that the the pressure is there To reach solutions. What's not clear is that I think we as Community activists are not actually sitting down and saying okay Well, there are four or five different ways of doing this.

Well, how do we think this could work? Right. Sometimes we need to actually talk to industry itself and ask them how they want to play. Yeah

Jonathan: Oh, yeah. So I've heard, I've heard stories of like a big business wants to sponsor an open source project and they're like, but we need you to send us an invoice and the open source project goes, we don't have any kind of business things set up.

We can't even send you an invoice and the business walks away and it's just like, well, we wanted to support them, but I guess we can't. Yes.

Alistair: Yeah. So, so there is, um, there's certainly that, and this leads to these perverse situations that either a project gets over. Supported by some company that thinks it's being like beneficial and in fact, we could talk about the you know So it's it's great when those things happen, right?

but the We need something that's going to work generically across scales and works for small and medium business customer companies that are consuming software and works for larger business and and and I think tomodipierre was very free He made a very cogent point about one of the reasons that open source has become So successful is that it's a one way graph of People downstream of you do not know What's upstream from a commercial perspective?

Yeah, right So the business, you know a the vp of engineering, you know doesn't even tell The CFO that they're bringing in a, you know, a project and nobody even knows all the branches and twigs that are associated with that project when they bring it in, right? Which, uh, led to the log4j surprise for everybody, which is like, oh, but we don't have that in our, oh shit we do.

Right? So, so, so that. We need and and this is what I think is certainly happening about the supply chain software bill of materials Technology. So all the the work that's going in there about generating s bombs will allow people to identify what that back path Looks like right and then the interesting question is like, okay fine.

Now, we know what you've got in there How what's an appropriate way? You Of providing funding down that path, right? So for instance, if i'm a consumer of a project should I also be somehow trying to make a micropayment to the log4j guys or Should the project that's using the log4j thing be making a larger donation and said Where is it turtles all the way down?

Yeah, right or and and Those are the types of questions Questions that I don't think we're we're sensibly asking ourselves because the solution to many of these things is oh Let's just throw a lot of money at the log 4j guys and that problem went away and it's like well No, you just you avoided Making the connectivity work and you've just fixed.

That's a point. It's a diving catch Solution which is not very good

Jonathan: Yeah, and I know there are groups out trying to trying to work on these problems like we years ago We interviewed the guys from Tidelift And so they were kind of, they were kind of way ahead of this. Um, we need to have them back on, because I'm very curious to hear how that has gone for them, whether they've had continued success or not.

Um, so I've got to ask, I've got to ask, this is kind of a, maybe a weird question, but I think you'll understand where I'm coming from. As, as we try, open source, historically, obviously, has been lightning in a bottle. Right? It has just been wildly successful. It has made things possible that were impossible before.

It has literally changed the world. As we try to sort of solve these problems and come up with blueprints for open source projects to solve them, do you think we have a danger that we, uh, we let the lightning out of the bottle? That we change open source in such a way that it becomes less impactful than it was?

Alistair: No, I th I

Jonathan: th So,

Alistair: I think if you

There are ways that you could Over engineer it and break the system, right? But on balance, it's working. It's working incredibly well in the downstream mechanism. The thing that isn't working. Is we're not getting the nutrients back up in the upstream mechanism and we don't have, you know, these longer to, you know, who's the, who's the apprentice for the master, right?

Who, what's the replacement, the backup strategy, those types of things are not happening and they, there needs to be a structural mechanism and making that happening. And to your point, this requires. organization on behalf of the, the open source projects. So good example is that in the EF, um, we've, we're in the process at the moment of applying for a CNA.

Um, and you, I think talk to Daniel at length about the CVE problem associated with this. So, um, I had the same conversation with him as well. And, um, it was pretty, it'd been pretty obvious to me for the last 12 months that we, as a, uh, an ecosystem wanted to have. Uh, you know, uh, autonomy over CVE, um, registration stuff just to avoid some of this frivolous stuff and be able to understand what was going on.

Right? So, um, I've articulated this as running your, you know, local fire station. Right. I mean, you need to have a volunteer fire station and you, and we don't want to avoid, we want to be able to manage that process and you need to set up your resources. That means we're actually hiring somebody to, to do this full time.

It means that we also need people to support this and we need to be able to talk to our community about why a fire station is necessary. So it's an interesting problem about how. Open source projects are going to be able to manage that back office infrastructure, right? And to your point, if somebody doesn't know how to write an invoice to somebody, right?

They probably need to bulk up. So, maybe we need group collectives. Maybe we need things that look like the mini Apache Foundations. Because, I mean, the Apache Foundation doesn't want to take on, um, lots of small stuff, right? But, you know, Um, that they take on big substantive projects, and I think they're doing a great job, but we need something that manages and looks like that for smaller projects.

Fortunately, we can do that for our language ecosystem, and I think you're seeing that happening in the other language ecosystems, like the Ruby project, the Rust project, they're all focused on doing things in their particular realm. Um, you know, there's, there's a whole bunch of stuff that, that needs to understand that it needs to be even thinking like this, right?

Jonathan: Yeah. Uh, is there, is there a resource somewhere that, uh, if someone is part of an open source project and they're kind of stuck in the middle of this, they're, they feel like the project has gotten too big for its own good and they've got to make a change. Is, is there someplace that you would recommend people go to, to kind of start learning about the options or to get help?

Alistair: Ooh, that's a very good question. Um. So actually that, uh, so coming up in my future is I'm off to FOSDEM, um, at the beginning of, um, the first weekend in February. Um, there are a lot of folks out there. I mean, one of the reasons talking to you guys, I think from this perspective is, uh, Is um, a lot of my knowledge about what people think and feel at the moment comes around about listening to podcasts So we are communicating over radio um Related to you know, what what people think and care about so I don't know I mean, maybe you want to put your hand up and and say i'm certainly happy to take redirections, uh Anytime you want to send them my way, but um, yeah Not clear who's trying to work on this at a total global scale.

I think it's probably too big for one person to work on. Yes.

Jonathan: Yes. Um, I, I do know of a couple of places that are, that are sort of thinking about this and trying to help. Uh, the Open Source Collective is one of them. I believe they're based in the United States. Uh, our friend Simon Phipps helps with a couple of companies, or a couple of projects, excuse me, and trying to, to Solve some of these same problems.

Um, and there, there are people out there, but I, I, I can't help but think that it would be useful. And, uh, I'm sure people are working on this, but it would be useful to have a blueprint where. You know, maybe, maybe we publish it as its own open source project. And it just, here, here's the documentation.

Here's what we suggest you put on your GitHub repository. Um, here's the way that you, you form a company to be able to take donations or, or what have you. It just seems like there, it would be useful to have a, a blueprint document out there for people to try to bootstrap from. I'm a single person. Open source project, but people depend upon me too.

Okay. Now I'm a healthy open source project with a succession plan, all of that.

Alistair: Uh, I couldn't agree with you more. Um, I've been on a couple of calls with Simon, um, possibly earlier on this year related to the EU cyber resiliency act and, um, um, and that there are obviously folks in some of the larger foundations that are, that are worried about.

This class of problem everywhere from the Linux Foundation to the Apache Foundation to the Eclipse Foundation. Um, what's, what is missing at the moment is something that scales. Across all the domains and, and, and to a great extent, even thinks about the really hard problems of the apprentice ship and survival.

So that's my usual test question. When somebody said, Oh yes, we're thinking about these things, right. Related to the, you know, cyber resiliency act. Then I say, okay, so if you've thought about them, what's the plan for that? And everybody goes, Oh, we hadn't thought about that bit. So, I mean, the, the, the, We sort of need to have a sort of discussion about how ambitious We want this to be and how you would think about stuff, right?

I mean at one level um You know other fields of endeavor have fixed this right? So they have things like the fields medal and they have whatever it is, right? I could imagine Somewhere where you actually decided and said, okay, we view these things as systemically interesting. We're not going to give um You Say a boatload of cash to um the curl project because They might just all then goof off and you know buy a boat and you know, hang out in the stockholm harbor Um, but we will we will pay You know the apprenticeship salary for three or four years or something like that, right?

There is there needs to be something that's going to go work on that level of sustaining. Yeah um So Yeah, don't know yeah interesting stuff

Jonathan: All right, we could go on for I think another hour chatting about all of this it is super fascinating Rob Is there anything you want to ask before we before we let Alistair go?

Rob: Um, no, I don't have any final questions. I

Jonathan: want to make sure and let you get, let you get the last word in if you want to do. Uh, Alistair, thank you so much for being here. The hour is just absolutely flown by and we will have to have you back here before too long because again, there's so much more to cover here and more thoughts to be had.

Um, but for now, I just want to say thanks. Thank you for coming and talking about FR routing. And then also this, obviously your passion is about having healthy open source projects. Yep, my pleasure. Before I let you go, I do have to ask two final questions. Uh, and then I guess I'll get a third one in too.

Um, so the, the, the third one that we usually ask, is there anything that you wanted to cover that we didn't ask you about?

Alistair: I think that, that, well actually, that, that you, you, that the one shout out that I would put out there is for the, you know, the first robotics, uh, programs. So, um, uh, I'm, Whole of january is going to be lined up doing uh, both Judging as well as inspector management for the local competitions in the bay area And for those people who haven't looked into first robotics and what they're doing.

It is absolutely wonderful um you You hear so many people especially my generation, right? But maybe even your generation like to complain about the kids are today And whatever it is, and i'm sure that there are problems With some of the kids are today, but the self selecting subgroup that actually turn up to these robotic competitions are absolutely wonderful.

And so it's a pleasure to be engaged in that activity.

Jonathan: Yeah, this is things like battle bots and line following robots and just all of that?

Alistair: Yeah, they don't. So the interesting thing about first is that they don't battle it's um, There is a competition. There is a scoring. It's a bit like doubles tennis that you play with other team members and it really Uh, it's not a zero sum game there.

There there is a Point system and you're trying to beat the other team, but it's not at their At the disadvantage of the others and there's you know, highly refereed and curated type of stuff and it it's a very interesting gaming theory uh process that whoever thought that up i'd like to go and you know have dinner with them to figure out whether it Came out as whole cloth or whether it took them several years to figure out how to make this process work.

So well, but um, But yeah, it they they're always striving to do some particular thing right as opposed to beat the other robot so so it's uh That

Rob: sounds similar to uh, we have a vex robotics in in our area

Alistair: Yes, it, it, yes, same thing, yes. Very cool. Yep.

Jonathan: Alright, and then Alistair, I've got to ask, what is, and this, this may be a historical question for you, but your favorite text editor and scripting language?

Alistair: Um, so certainly, uh, text editor and stuff, I would use nano, um, and I think the argument's been made before that it's just so ubiquitous. And I think that's, that's a good answer. Certainly if I want to get something done and I'll use bash if I have to do scripting stuff. Sure. Sure.

Jonathan: All right. Well, again, thank you so much for being here and, uh, we, we sure appreciate it.

Rob: Okay. Talk to you.

Jonathan: All right. Rob, what do you think?

Rob: Well, I definitely approve of his, uh, last two, uh, answers, nano and batch. But, uh, as for, you know, the, the open source projects he's involved in and FRR, um, routing, I was really surprised how ubiquitous that was. And definitely, uh, you know, in agreement with all the, the The needs we have to fund and, and have apprenticeships for open source and all that.

Jonathan: Yeah. So he, he threw out the numbers, uh, a thousand mission critical open source projects, maybe as many as 10, 000. I think 10, 000 is a low ball estimate. Of the number of open source projects that are mission critical to various things Um, they're probably there may be 10 times that many that if any one of them were to go down it would be It would well it'd be the left pad situation right when when the guy behind the left pad Uh did javascript like library, you know all one line of it Deleted it in protest and half of the internet broke.

I mean there are so many of those even In

Rob: one large piece of software they're Yeah, maybe I'm stretching it, but I was going to say there could be a thousand different little open source projects. It's possible, depending upon

Jonathan: what language it is. If you, if you expand that and say in a single server, there's going to be at least a thousand open source projects.

You probably have at least that many installed binaries. Um, yeah, there's a, there's a lot of them out there and, uh, it is, uh, it's quite a, Problem, but also an opportunity to really think through some of these things about how do you What does it look like for a project to be healthy? And then how do you get to that point?

Definitely interesting. We're going to be talking about more of this sort of thing next week as we have uh, Matija Silke Of FOSS, well, it's about FOSS legal. We're going to be talking about free and open source software, legal issues. Um, and he is involved with a couple of different organizations where I believe open source projects can reach out to and have a legal representation, get opinions and such as needed.

And I am super interested in that because one of the projects that I'm involved with, we have legal concerns because we occasionally deal with people's personal, uh, inter. Personal, identifiable information. And, uh, that is, that is sort of a hairy issue these days. So I'm gonna try to pick his brain next week.

Not sure who's gonna be with me as a co host. I need to figure that out. Um, but we will be back next week on the 7th to talk about that. Legal issues and open source. Be a lot of fun. Uh, Rob, is there anything that you want to plug before we let everybody go?

Rob: Well, you guys can catch me on the Untitled Lenox Show with Jonathan Bennett.

Yeah, and if you want to connect with me directly, you could find my website, Robert p camp bell.com, and there'll be links there to, uh, various social medias and things and information about me.

Jonathan: Yep. Great. I appreciate you being here. All right. As Rob said, you've got my work over at twit tv, the Untitled Linux Show, and then there's also Hack a Day.

Where you can find the security column goes live every Friday morning when it's not a holiday or I'm not sick Which was a very interesting past week or two for me Um, but we're back. We're gonna be back at the grind for a while at least Um, and yeah, make sure to check that out I've also got a youtube channel where you find a smorgasbord of things talk about meshtastic talk about modular synthesis That video did well, so we're gonna have to do more of that.

Uh, that's uh, that's you can just You search for me, Jonathan Bennett, on YouTube and find that. Um, but anyway, we will, uh, we'll be back next week with more Foss and Floss goodness. We'll see you then. We appreciate everybody that listens, that watches both live and on the download, and we'll see you next week on Floss Weekly.

Episode 813 transcript

11 december 2024 | min

FLOSS-813

Jonathan: Hey folks, this week, Simon Phipps and Aaron Newcomb joined me and we talk about an interesting smattering of topics. We start with talking about legacy computing and retro hardware and how open source still makes that work. And then end up with politics. It's actually a lot of fun and you don't want to miss it.

So stay tuned. This is Floss Weekly episode 813 recorded Tuesday, December the 10th. Turn off the internet. It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and sometimes we talk about a little bit of hardware here. And today is interesting because I was absolutely certain that we had a guest schedule to talk about the open source AI definition, and I sent an email yesterday, just checking up to make sure that everything was good.

And I got sent back, you never confirmed that we were going to be here. So we don't have it on our calendars. And I went, Oh no, yeah, that's right. That's one. So. Thankfully we have a couple of willing volunteers. Simon and Aaron jumped in sort of at the last moment and they're here and we're going to talk about a couple of things.

First off. Welcome guys. Thank you so much for being here.

Aaron: Absolutely. It's all your fault, Jonathan. It is.

Jonathan: It's all my fault. So this, this was basically the crew that was supposed to be here. We were just supposed to have one more, sort of another expert in the field of open source AI to talk about, for one thing, what that even means.

And we may get into that a little bit here towards the end of the conversation. But I've got, I've got here two experts in retrocomputing. For two very different reasons. ,

Aaron: well, I don't know, is, I don't know that they're that different. I mean experts because we lived through it. Is that what you're getting at?

Because we're so old 1, 1,

Jonathan: 1 being an expert because he lived through it and the other being an expert because you've got the YouTube channel on it. Yes. That was sort of what I was getting at. Okay. Yeah. Okay.

Simon: You know, I have to say that I, I, the, one of the things that makes you feel old is going to the Computer History Museum and walking around and going.

Hey, that's my computer.

Aaron: Yeah, I worked

Simon: on that. Yes. And I, I, my, my SWH 68, 000 is indeed in the computer history museum. So you know, that's retro computing. Yeah. We, we call that a live systems production. That's cool. Yeah,

Aaron: I get asked, I get asked a lot by the younger crowd, like, you know, the question always starts with back in your day.

I was like, all right, were Atari joysticks really like this when you, yeah. Okay. Yeah, I get it now.

Jonathan: I'm old. Yeah. Yeah. I'm starting to feel that too. I had somebody the other day. We've got some really cool retro computers. You should come and take them, take a look at it. And you know, I'm expecting a certain era of machines and I get in there and I'm like, yeah, here, it's this Windows 95 machine.

I'm like, okay,

Aaron: That's not right. I try not to be too. I try not to be too. Yeah, because there is starting to be That nostalgia for things even out of the 2000s Now, you know and it's like well, yeah, it was 20 years ago. So I I get it. I it's not my thing, but I understand, you know So that's what I usually tell people.

It's about a 20 year window, right, where people start to feel nostalgic. Until then, it's just old crap, you know, and then all of a sudden it's like, oh, I remember how long that took to do.

Simon: Yeah. Yeah. So one of the curious consequences of always assuming that Microsoft was going to audit me I've, I've kept all of the original media for, I have it, I have it for Windows 3.

I have Windows for workgroups, I have Word version 1, I have all the original media and original packaging all down in the office, so I'm waiting for a museum to call, or I need an auction house to give me a call.

Jonathan: So one of the, one of the things that I have seen both on Aaron's channel on RetroHackShack and some other people's channels is that we're starting to see like open source, both software and hardware get applied to some of these old retro computing things.

And like just a couple of things that come to mind. I've seen, I've seen people take like Commodore 64s. I am the age that I consider a Commodore 64 to be retro. Simon, that's probably. That's probably just old to you, right? ?

Simon: Yeah. I, so I never, I never had a Commodore 64. I, I, I worked on Sinclair qls.

Mm-hmm. They were, they were my, my thing. 'cause they had built in networking and everything. But yeah. That's good. You know, I've got friends who've, who've got those on a shelf somewhere.

Aaron: Yeah. And the, the QL was the one with that little tiny tape drive, right? Yeah. With

Simon: the micro drive. Oh yeah, yeah.

Aaron: The micro drive.

Yeah. Yeah.

Simon: But I, so I actually used to run a network of them for an office. And because the QL had had built in networking in the hardware. So you just ran a cable from QL to QL, and they could all share each other's drives and resources. And so I added a hard drive on, and so that we effectively had a hard drive in the cluster.

And you add a printer on, and everyone can access that. It was actually really great computing. It was much easier than anything I ever encountered subsequently on PCs.

Aaron: And I think that, I think that one thing that lends itself to open source in retrocomputing is there was always an element of, even though they didn't call it open source back then, but certainly hacking right on these things.

And those became projects that were published in a magazine that anybody could pick up and do. So there, there is some, for me anyway, there is some root, some kernel that was going on back in the seventies and early eighties. That was akin to open source to an open source project where someone would come up with something It would get published in bite or it would get published in popular electronics or something like that and maybe they got paid for their article, but the point was they were sharing that information Yeah with the community and anybody could pick that up and do whatever they want with it if you go back all the way to The TV typewriter, for example, and different people will say, Oh, the TV typewriter, I think like Waz, for example, said the TV typewriter wasn't an influence in the Apple one, but it was, there was some, some sort of DNA there because what was used in the Apple one, which led to the Apple two, which led to.

Apple, as we know it today used all the same elements that were in Don Lancaster's TV typewriter. There was some memory there. There was a character generator. There was a keyboard. There was the way that it interacted with the rest of the circuitry was there. And that was published in a, in a, you know, one of those early magazines.

So I really do think that there's some open source DNA in there somewhere to the whole. Computing back when it was actually happening in the seventies and eighties.

Jonathan: Well, so something, something important on that, on that note, something important to keep in mind is that when, when, when RMS came along and came up with free software, he wasn't inventing the idea of let's make software freely available.

Like he was, he was in fact saying, this is the way it was. And this is the way it should still be and then they came up with the licensing hack that is the gpl And it's the same thing with open source when when the open source definition first was was kind of developed They were not they were not inventing open source.

They were looking at this is this is what people have been doing This is what's worked and let's try to codify that. So yeah It's, it is a much longer tradition than, you know, just the, even the free software foundation or the, the, even just the OSI, it has existed longer than that. It's just, those are, those are the groups that we sort of have to thank for, for taking all of this nebulous idea of let's share the code, let's share the information and sort of distilling it down into, let's use this as a definition for that.

Simon: Yeah, I mean, it all comes from a real long time ago. I've been working with somebody on a short book recently, and you'll likely love it, you know, because he's written a book about why open source is capitalist, and he started out by talking about how Stallman invented free software, and then he looked back at history and discovered that Stallman was really quite late to the game.

Yeah.

the folks over it to Berkeley University were there ahead of him. Were it's just, they, they had a different way of expressing their expectation. Their expectation was that this was before the IBM consent decree. And so it was before. The productization of software. And so there was no need for a GPL because there was no value to software.

People just gave it away. And but the bill joy was there creating BSD and then going on and creating the world's first open source software company, which was some microsystems and people look, they, they, that's so far back that people assume that it all happened in the nineties, you know,

Jonathan: It's interesting. You mentioned that book, you know, Steve Levy's book, Hacker's Heroes of the Computer Revolution. That was sort of the introduction for, at least for me. And I think for a lot of us to some of this history and he makes the exact same point, but with Stallman and I think it was the, the, the hacker club at MIT where they were playing with the PDP machines and Stallman came again.

He was late to that game. He was towards the end of it and he saw things. Changing away from the, let's just all share this code so that we can all make it better to again, this idea of productizing it and the free software foundation and the GPL itself was sort of a response to that. And so, yeah, it's super interesting to see the, you can sort of trace that line down through history to where we are today.

Aaron: Yeah, so the, so the, so the nuts and bolts have been there for a long time. But now it's kind of come full circle where, you know, people are creating these, these cool hardware and software projects that can then bring these retro computers and give them more functionality or make it easier to do things for people that, that maybe aren't you know, that don't want to do it on their own.

And there's some great examples of that. Of course you know, the one that comes first to my mind is RGB. Yeah. To HDMI which is an open source project. And I sell some of those boards on my site, shameless plug. But you can also, it's an open source project. So you can go get, I had a guy this morning that emailed me saying, are there DIY versions of this thing that you sell that I can just go do on my own?

And of course the answer is yes, you can. But yeah, this project allows you to take an old. Sinclair QL, for example, or TRS 80 or whatever, you know, computer you, you want to use that had some of these older protocols and allows you to hook them up to your modern HDMI. Monitor. So you take the output, convert it to HTMI and, you know, cause a lot of these old monitors aren't available or are very expensive.

I mean, try finding a IBM 5154 EGA monitor these days, it's going to cost you three or 400 bucks and hopefully it's working. It probably isn't, or probably won't be for long.

Jonathan: Or it's dim, or, yeah.

Aaron: Yeah, and not everybody can fix those. They're awfully hard to work on. So yeah. So that's a great example that, that first comes to mind is, is that project because it works with so many retro computers and is so accessible for people.

But there are also a lot of projects that are coming up right now around the Raspberry Pi Pico. So, and of course there's tons of things going on, but, but especially in the retro community, people are taking the Raspberry Pi Pico and making it do things inside retro computers as well. So there's two projects there that come to mind.

One is the PicoMem. Which adds memory for older IBM computers with a ISA slot. You can put this in there and the Raspberry Pi Pico will emulate the memory and give you, you know, two, four megs even. Can you believe it? Of memory for your old IBM computer, right? And what people found, I think, partly based on that product and based on other products, once they had the ISA bus figured out, is that they could do all sorts of other things.

So there's another project out there called the PicoGus and that project has gotten a lot of momentum lately, which actually takes the Raspberry Pi Pico and emulates a Gravis ultrasound. That's what it is. I was trying to think, what does G U S stand for? Gravis ultrasound, again, incredibly hard to find these days.

Not very many people had them cause they were expensive when they came out. But you know, this is an a sound card that was that worked with a lot of games. You added it onto your Sound Blaster and it did incredible, incredible things, but they're super hard to find these days, but it'll emulate a Gravis ultrasound.

Or a sound blaster or a a Roland system or a Tandy three voice system. It'll do all of that just based on this little, I don't know what they cost these days, 4, 5, a little open source or open hardware microcontroller. That's, that's cheap and easy to use. So there really is a lot of, a lot of great open source projects out there.

And I, I, I think it speaks to the legacy of open source. The fact that open source is established so much now that when people create these projects, the software that they create. Of course, it's going to be open. Of course, I'm going to give it a GPL license or whatever. It's not like a question where you have to convince people, Oh, can you please, like, just make that open source?

No, it's just like by default, right? They're choosing to make this open source and it's for the betterment of the community.

Jonathan: Yeah, every once in a while I come across a project where it's like somebody obviously, like they'll put it on GitHub and everything. They obviously intend for people to just grab it and use it, but they won't put a license on it.

Aaron: Yeah,

Jonathan: like you you realize that you're all rights reserved by default, right? Please pick one of the open source licenses I don't care which one you pick just one of the OSI approved licenses and just to add the file like here I'll even make the pull request for you. Just accept the pull request put a file on there.

That way we can use it Yeah,

Aaron: yeah. Yeah for people that don't know I think github does a pretty good job though of pushing people into choosing a license when you start a project Which is good. They try to at least Yeah Yeah. Yeah. Now , so there's a lot of there's a, there's a lot of great projects. One last project I'll, I'll mention is one that I just found and I'm super excited about it.

It's called The HID Man. Hid Man. Mm-hmm . And you might know HID from USB HID devices. You know that that's what makes USB so ubiquitous and easy to use. You can kind of like plug. A USB mouse into anything and it just works. And so there's a guy, he goes by the name of Rastiri on GitHub, but he he also has a YouTube channel and he's been working on this device that takes.

USB devices and converts them into either PS2 compatible or AT or even XT compatible protocols so that you can take, if you don't have one of these old keyboards or whatever, or you just want to use, like in my case, I have a fancy new mechanical keyboard that sounds wonderful. I love to type on it. But I want to use that on my retro devices.

But it's a USB keyboard. You can now take that plug it into this device and then plug from that device into your computer and use, you know, a mouse or, or a keyboard on your, your retro computer. And it's just a little tiny device that does a wonderful job. He made it open source. And I talked to him about it.

I'm going to be carrying the hardware portion of that on my, the hardware is all open source as well, but I'm going to be going ahead and making some of those and putting them on my shop as well. Because. I think it's just a, it's just a great, it does such a great job. One of the things I love about it is the, in order to do the configuration, because he has in, in the software, he has control over the PS2 keyboard.

Let's say you're using, when you want to change the configuration, you open up any text editor. Maybe it's just edit in DOS, right? And you push a button on the device and it prints out in the editor, what the configuration is. So instead of having to open a file or go in and change something, you know, it just prints out the configuration and it says, what do you want to change?

And you hit one. And of course he can interpret that he knows you hit one. So then he says, okay, I'll change that configuration to this. You know, this is like a brilliant in my mind is something that would have been done back in the early days. And. He's using this to actually allow people to change the configuration on this device.

And I was like, that's, that's cool. That's cool. And it's brilliant. And I love it. It's

Jonathan: kind of like using a teletype machine almost. Yeah, exactly.

Aaron: Exactly. That's fine. So

Jonathan: one of the things that fascinates me about this, and I've got the same bug and it's this idea of, of so someone, someone might ask.

Why don't you just emulate? Right? Like, so if you, if you want to play with a Commodore 64, well, there's Commodore 64 emulators. Why bother with the real hardware? Why bother with any of these adapters? You know, why would you use a, a machine that actually has an AT keyboard or an XT keyboard plug? Why not just emulate it all?

And, I don't know that I know the exact answer, but there's just something special about running the real hardware. And, and, Why, why is that?

Aaron: Yeah, there is something about, about running on actual hardware. It's, it's a great test bed. I use emulators to test stuff all the time. Testing code. If you're writing code, for example, for some of these old things, it's, it's a lot easier to pull out an emulator and just do it that way.

But even what I find interesting is even the younger crowd, right? Kids to me, anybody younger than 20 still has that same response to the actual hardware, you know, and you would think they would be like, I'm just going to run this on my steam deck or on my phone or something, but no, when you. When you give them the actual hardware, it like something clicks in their brain.

It's like, Oh, it's tactile. I can use it. I see how that worked. I understand like how the cartridges work and how the joysticks works and how that all came together. In, in a special way, I think because all of those systems at the time were, you know, there wasn't a lot of standards back then, you mentioned the Commodore 64, you know, that was as different from an IBM PC as it could be.

Right. But. you know, learning how the systems work and how the hard drives work. And you had to kind of be your own expert on some of those things. I think there is just an appeal to people that like to learn about old systems of having the physical cartridge, the physical hardware to play with, seeing how the keyboard felt in those days, how it was either bushy or clicky, or, you know, there's just something, something about that tactile response to those systems.

I mean, Simon, you mentioned those, the micro drive, right? Yep. For the Sinclair QL or the keyboard on the Sinclair QL, I know was quite a bit different. Of course, it wasn't the the, the old plasticky one of the original Sinclair, you know, ZX Spectrum, but

Simon: the membrane keyboard, only one keyboard. And so with the key, with the, the space on the top of the case with the keys attached to the, the the, the printed circuit board that was underneath, I mean, the whole thing was a single, was a single board computer.

And that's obviously a desirable thing because I, you know, Raspberry Pi are coming out with the Pi 500 at the moment, which is just the same. And it looks just the same. Just like what I remember my Sinclair QL looking. I mean, it's white, but it's very similar kind of experience and they expose all the ports on the back.

The GPIO pins are all out there. I think that one of the things that makes it compelling. Knowing that you can

yeah,

I think that the the difficulty with with lots of stuff, you know I've got it. I've got a device here that I picked up from a well known Scandinavian furniture shop that's for controlling my Sonos speakers And and it's it's all sealed And I can't do anything to it.

And if I take it apart, what's inside is sealed and the box that it's controlling over there is sealed and I can't mess with it. And all this stuff is stuff you can, you can get your hands on and not because anyone who made it intended you to do that, but because the people who wanted to stop you doing it are all dead now.

And there is that sense of, I've got this. This, you know, ancestral artifact here and I, and I can, I can touch it and I can, and nobody is compelling me to do things in a particular way and there's problems to solve, but I'm not going to find myself getting a cease and desist letter from Sony for doing it.

And, and I think that part of that is what's compelling, and I think it's, that's the same thing that's compelling about Raspberry Pi. It, it is that you can, you know, it's got all the GPI pins are, are on the back. You, you know, you can take it apart. They sell the whole thing as a chip you can put on your own circuit board, and people do.

And I think that's, that's always been, you know, the things that have always driven me in computing are the fact that I can and also the idea that I can do things here that make something happen over there. Those have been the two things that have always driven me in computing is action at a distance and being in control of my own destiny with my hardware.

This when when people try and take those things away from me, that's when I feel diminished

Jonathan: This is why simon and I get along so well We we have those two very core core to our being things in common All right. So simon, do you find yourself or maybe you have this? You mentioned the the Sinclair. Do you, do you have some of those still kicking around that you, you boot?

Is that something that even appeals to you?

Simon: Yeah, so I do have a box up in the attic that's got a couple of Sinclair QLs and I've got, I do have a, a, I do have some other Sinclair hardware up there and we have a dinner guest who comes every weekend for, for Sunday lunch who keeps a compute library.

He's, so he's got his old systems and we have our eight inch floppy drives and things, but I found with the QLs that the QLs are very

mechanical

and their mechanicalness is also very what we would call in this country, Heath Robinson. I forget what your term is for it over there. It, it, it, you know, it, it does involve rubber bands and, and it does involve little thin pieces of cassette tape in a, in a little plastic box.

And they've all stopped working and getting them back to being, working again is gonna be really quiet. A challenge. Yeah. And and I know I could, I've got a degree in electronic engineering. I could get them working again.

Yeah.

But there's other things that are more interesting to me at the moment.

Sure. So. They stay up in the loft, really.

Jonathan: And that's something, you know, kind of coming at things from the, like, even the Hackaday perspective. That's something I find fascinating about some of these retro machines, you know, in the right era. The parts inside are just the right size that you can actually get in and work on.

And so they're, they're super useful for young people. Teenagers, people in their 20s that are like working, maybe working to get their electronics degree or just got it Or they're in high school and they find all this stuff really fascinating You can actually get inside of them and work on them you can actually take these old machines and fix them and We're still thankfully in a kind of at a time period in history where if someone has a you know a not working Again, I keep coming back to the Commodore 64.

It's the best selling computer in history, so I guess that's why. But like, you can still find not working Commodore 64s on eBay and places like that without having to spend a fortune on them. And so it is, it's really fascinating for, for all people of all ages, but specifically young people. You can grab them and really get into the hardware, and it's a good, it's a good learning platform, I think, among, among other things.

Aaron: Oh, it's a tremendous learning platform, especially since in the olden days people would actually have schematics for these things as well, right? So if you get a bug in your, under your whatever, and you want to actually go in and fix that thing, or you want to learn why it's not working, you can pull out a schematic usually.

Or a service manual, you know, I mean, to Simon's point about things being sealed up and not people don't give you any information about what it is or how it works anymore, but they're used to and give you a full service manual with a theory of operations and a schematic and you could go in and you could learn to your point, Jonathan, about how it works.

You could learn about electronics or any of those things just by, you know, breaking open your home computer and getting in there with a screwdriver and a soldering iron. So it is a tremendous learning experience just because those things were available at one point in history.

Jonathan: I've got a couple of devices, old devices that You can tell like you pull you pull it apart and you look at the pcb And you can tell someone wrote this pcb out by hand Right, like there was a time where that was that was the way things were done and then you know, you would get a you You would replicate it, but it would the first thing it would be someone would literally draw it out by hand And on the other extreme of that today, you know you talk about You get five layer PCBs, you get eight layer PCBs made to be able to fit all the things into a tiny little space.

And so, you know, on on one hand, we get enormously more powerful devices. And and so, you know, we're we're not luddites. We appreciate the the enormously more powerful devices. But on the other hand, that means that it's so much more difficult to work on, to understand, and The, the old stuff, the old stuff is pretty neat, because, again, you can, you can see inside of it.

You can understand how it works on the inside. That, maybe maybe not by accident, is an interesting segue into something else that is enormously powerful that is very difficult to see inside of.

So we were going to talk about open source AI and the open source AI definition, and we've got half of our expert crew here. And so

Simon: You got the skeptic department here.

Jonathan: Well, that's interesting because I find myself being sort of skeptical about a lot of these things too. So maybe, maybe Aaron will hold down the, the AI enthusiast side of the conversation for us.

Sure.

Simon: So we, we, we, we've worked on the open source AI definition at OSI. And I was the staff skeptic. I stayed out of the process completely because somebody had to do the other stuff that wasn't AI, so I got on with it. With you know, looking at the legislative schedule of the European Union and influencing that for two years and, and I kept on throwing rocks at the AI stuff, you know, I, I would make rude comments like, isn't it strange how all those people who were into crypto last year into AI this year?

No. And but nonetheless, you know, I've, I've got a reasonable amount of respect for where we've got to. We do have open source AI definition. It exists for excellent reasons and it's the subject of a program that you'll be doing in January, I think.

Jonathan: Looking forward. I am very much looking forward to that.

You guys, you guys have gotten some pushback on that though. People don't like your definition.

Simon: Well, actually, when you, when you look at what people are saying about it, you find that Generally speaking, free and open source software people analyze the problem the same way. You look at what the, the, the way that the Software Freedom Conservancy has analyzed the problem.

You look at the way the Free Software Foundation has analyzed the problem. And they've, they, you know, we look at it the same way. The, the, we, we recognize that it's a, A system that involves software and data that it involves life, life cycle phases. We, we, we assume that for something to be open or free, it has to include all the source code to the software under an approved license, that it must include all the code, all the, the, the parameters that you need to configure the system, and that it must include all the data that you need to populate the model.

And, and like, like all good high detail movements what we're arguing about is one of the tiny details, which is, well, how much of the data do you actually need to trade and train the model? Yeah. And the answer to that question is quite complicated. The answer is that you don't actually need all the data to train the model.

The problem you have is that you don't know which of the data you do need to train the model. And so you give the model all the data, and it It then trains itself with the bits you needed, and if you could magically know which bits it needed, or if you could artificially synthesize those bits so that the model could be primed, that would be great, but we keep on being told by AI experts, and it's important to talk to AI experts as well as free software experts about this subject.

We keep on being told the data doesn't really matter. You can use any old data because the model will end up roughly the same. And also that they, the other thing that they say, which is really head scratching, is they say that if they run the training process twice, they get a different, a different model at the end with even using the same data.

And so the things that free software people want is they want the, The full corresponding source, you know, they want the, the, the form normally used for making modifications to use the software language about it. And you talk to the AI experts and they say, well, there isn't really one of those. And so all the arguments we're having have been up have been from free software and open source experts trying to force fit the free software concepts.

Yeah. into the world of A. I. And the answer is you can't do it. And we then disagree with each other about about the compromises that have to be made to try and do it. Now, I think the A. I. Definition we've got is is perfectly adequate for the next step, which is stopping meta taking the market over by claiming that their llama is open source, even though every any fool can see that it's not.

That's the big presenting problem. The big presenting problem isn't, you know, is it okay to train a model on transient data that you can't give to people? The real presenting problem is we've got these completely closed AIs that you can't see anything inside that people are saying are open AI.

I mean, it's even in the name of the company.

Or they're saying is open source AI. They say about LLAMA and neither of them is in any way. Well, LLAMA is actually, you know, less closed actually. But the real reason we need the definition now is to, to make sure that we've got time to have this conversation, because if we hadn't made this definition now, the conversation would be as moot as the conversation about you know, about cloud is the free software and open source community completely blue cloud.

Yeah, we made no ethical directive statements that helped. And consequently, the cloud industry has gone whoosh off in that direction. It's completely out of our reach. And the same thing was going to happen with AI. We were going to have an argument for five years about, you know, whether we should call it AI or machine learning.

And meanwhile, Meta was going to completely control the market and tax everybody who innovated. So you know, i'm i'm glad we've got the definition. I don't necessarily agree with all of it but I'm glad

Jonathan: we've got it. And it's, I guess, it's important to point out that like, you guys are not writing the licenses.

You're writing the things that must be in those licenses, like the minimum baseline requirements for them to be considered an open source license, right?

Simon: Yeah, we don't, so ISI has never written an open source license. What, what we've, what we have in the open source definition is a you've heard people say, you know, I don't know, I don't know what the, I don't know what one of those is, but I know it when I see it.

Well, the open source definition is the list of things to check to make sure that it's one of those. And that's AI definition is as well. It's a, it's a list of characteristics that let you know, Whether your freedoms are going to be respect respected that whether you're going to be able to take that AI and do the thing that you wanted to do rather than the thing that somebody else wants you to do.

And I, I think the definition we've got is pretty good for that. I think it's a little bit over generous with what it allows in terms of availability of data. But I think it's got it absolutely right on the source code. I think it's got it absolutely right on the model weights. I think it's got it, Absolutely right.

All the way down to this continuum of how much code do we need. And no one has been able to say how much code you need. And so there's some people say, well, so to be on the safe side, we want all the code. That means we'll never have an open source medical AI. Because, you know, medical AI is never going to ship with all the original data.

Aaron: Right.

Simon: And that's, that's the AI I want most is an open source medical AI because I can never get an appointment with a doctor. So, you know, I want to be able to ask my computer what that rash is down there. Why I can't bend this arm. And yeah, here's a,

Aaron: here's a picture. What, what, what is this? Yeah, yeah, yeah, exactly.

Simon: Do you recognize this rash?

Aaron: Yeah. Yeah. Here's my symptoms. Here's a picture, diagnose it for me. And there's already been examples of that happening, right. Where people have used AI to get a diagnosis that, that many doctors couldn't figure out. And then when they see it, the answer that they got back, like, Oh, of course you put these things together and it means you have this, this thing.

But, you know, AI did it in. A couple of seconds.

Simon: Thank God it got it right. I mean, the big problem with all these AI models. So, you know, you're getting into my skeptic self now.

Aaron: You

Simon: know, the generative AI models, you have to remember that their purpose is to persuade you that they have responded to your prompt.

Their purpose is not to tell you the truth.

Yeah.

And sometimes in persuading you, They accidentally tell the truth, and that's really handy. But the people who should be interpreting the output are the people who don't need the AI. The AI is there to help them get to an answer sooner, not to help them get to an answer without an expert.

Aaron: And,

Simon: I hear so much of this discussion being about, oh, you know, the AI hallucinated. The AI made an error. No, it didn't make an error. It was trying to persuade you that it was right. It has no idea what right actually is. Yeah. What it does for a living is persuade you. And, and it did a great job because you were persuaded.

And and the fact that you're persuaded about something that was completely false is beside the point because it's not here to tell you the truth.

Jonathan: Yeah. Yeah. You've got even, you've got even another, like, angle, another point to dig into with this, and people call it trusted AI, which sometimes boils down to we always want the AI to give you the answer that we've deemed appropriate, and there's a whole, like, Boy, that's a whole ball of wax in and of itself, right?

But then on the other side, you have things like one of the car manufacturers here in the United States, you know, Ford or Chevy, one of those guys, they put put an AI chat bot on their website and someone convinced it to sell them a car for a dollar. And apparently the courts upheld it and said you put the AI out there as your official spokesman So you you sold him a car for a dollar have fun with that.

Yeah

Aaron: That's funny.

Jonathan: I I've I've made the point repeatedly that I am I am just I am waiting for the next big thing So simon mentioned this with the idea of all the cryptocurrency guys are now into AI I'm waiting for that crowd to move off to the next big thing. Whatever it may be Maybe it'll be back to cryptocurrency or whatever like the the AI bubble will kind of pop And then we'll actually be able to see what it'll be useful for as a tool, once people stop trying to fit it into everything.

I'm looking forward to that.

Aaron: Yeah. That'd be nice. But again, that speaks to why this type of definition is so important right now, right? And even regulation. I'm not a big fan of regulation. But if you're going to be looking at putting regulation in, now's the time to do it. Because as fast as cloud moved, AI is moving a hundred, a thousand times faster, maybe, maybe even faster.

So if you don't do it now. It's, it's over, right? It's gonna be very hard to put the genie back in the bottle. Yeah,

Simon: I'm, I'm curiously a fan of that. I, you know, regulate early, regulate often is the, is the, the watchword. I think that what happened in Europe is we saw the, the absolute need to have privacy regulation on the internet.

Which was which became that became necessary like 15 years ago and GDPR was way late to the To the to the scene and as a result the scene that it was way late to was already extremely important economically, and so they compromised and they, they got it, they got it wrong. It's, it's an awful piece of legislation with that, that, you know, if anything, strengthens the arm of the people that it was trying to control.

And that's why I quite like the AI act in Europe, you know, it's not, it isn't perfect, but it's come out early enough that you can change it. And it can actually address the real problems rather than coming out in 10 years time when it, when it, when I is already a trillion dollar industry and you can't possibly regulate it because everybody will simply remove your political funding, regulate, regulate early and then you Change it every time you find it's wrong and that way you end up with a regulation that fits.

So I'm, I'm a big fan of regulate early, regulate often. I think that's the way to go.

Jonathan: You know, when you first said that I bristled and I'm like, ah, regulation, it's terrible. We shouldn't be doing it. And then you actually explain what you mean. It's like, okay, that's a reasonable approach.

Aaron: As long as there's a mechanism to fix it, right?

As long as it's not.

Jonathan: Yes.

Aaron: You know, a U. S. Amendment to the Constitution, which never happens anymore, right? But as long as there's a mechanism in there where we can say, go get, you know, go, go fix this thing and make it better, then it's fine. What is that? I'm not familiar with the European. What is it called?

The act or the act?

Simon: Yes. Yeah. So it's it's it's what it's it's It's been quite widely criticized by, by tech utopians. But it, it puts in place some regulations that will stop harms happening and they may be over conservative, but I think that it's being held reasonably lightly in the hands of the regulators.

And when you look at. In, in Europe regulations, you find they get changed, they get amended quite often. And so you look at a regulation like GDPR, you find it's, it's getting modified many times a year. By other political instruments. So by, by delegated powers from other pieces of legislation or from new pieces of legislation that modify the early legislation.

And the problem with GDPR is there's, there's all that awful money associated with spying on us and regulating it harms the people who are getting rich out of all of your and my private information. And so I, you know, what was needed was to stop them from creating the advertising surveillance industry in the first place, rather than to try and regulate it after it existed, because you know, when you try and regulate something evil, you have something evil and regulated.

Jonathan: And the other, the other side of that that we see here in the States from time to time is when you add regulation onto an industry, it is a minor burden. For the established players, but is a nearly insurmountable burden for anyone coming along trying to do anything new And so, you know when when you hear people in in my country decrying regulation, that's sort of the thing that they're getting at and again, it's one of those where when We try not to get political on here on the show, but I think we're going to, we're going to get into it just a little bit today by the very nature of the same things we're talking about.

So when you hear like a conservative talking about, we got to get rid of regulation, that's what they're getting at. We've got some of these regulations that are just because there's so much of it and because they're written to the benefit of the established players, it, it can be a problem. And again, it's one of those deals where, If you, if you just, oh, let's see, how should I put this?

If you just come at it from the kind of the partisan viewpoint, well, you have an an A versus B and, and, you know, the other side is wrong. But if, if you have something to where you can kind of cut across that and I think open source is an interesting tool for doing this and, and listen to some of the points that are being made, there, there are some things that we have in common here that people on the very far left and the very far right, if you actually have a, like a A well intentioned, good faith conversation.

There are some points that you can come and actually agree about. And that's sort of a useful thing.

Simon: Yeah. It's a, it's a, it's a turning fine line to walk. So, you know, before, before we, before we, you pressed record, we were talking a little bit about that. Yes. And What we found at OSI over the last 25 years is it's really important that we focus on the, on our swim lane,

you

know, and our swim lane is open source licensing.

And the reason that swim lane is so important is because every side. And there is, there is at least three sides to this conversation. Every, every side recognizes the need for a copyright anchored license that gives you the rights to exercise your freedoms. And. So as long as we focus on that, everyone's happy with that because, you know, there are some people who are, you know, they're very keen to have that happen so that their freedom to make sure everybody can use the software under copyright on the copyleft terms is protected, there's other people who are very keen that there's The license makes it available to everybody without restriction.

There's other people who are very keen that the license protects their, their rights in a way that is, is, is, is lightly touching the software, like Mozilla license does. And So that means that within the community of people who are wanting this to happen, we can have people who are, you know, over at the libertarian front or over the, the, the social good front or at the, Over at the intensely personal.

Leave me alone front. And and they can all be served and we don't have to argue about the things that we disagree about. And that's where we're going to run on the rocks with AI is because already in the conversation there is all of this is being woven together with ethical purpose and acceptable use policies.

And if there's one thing I've learned over the last decade, it's that what is an acceptable use to one person is an invasion of freedom to another person.

Jonathan: Yep. Or, or, One of the,

Aaron: one of the,

Jonathan: oh, go ahead, Jonathan. Or, so, or, it falls into their definition of, of evil. Like, that's not a thing that everybody agrees on.

And there've been some, there've been some brouhaha's in various open source projects, like the X, Y, or Z military uses some of this source code. And so, and to, to one group, that's the U. S. military uses it and therefore it's evil. And to another group, well, the Russian military uses it and therefore it's evil.

And I don't know that I could say that either of those. are like illegitimate points of view. They don't work well together when you try to add the, except the idea of acceptable use into the conversation, which I think, I think that's why it was, it was brilliant that when the open source definition was first founded, it was explicitly stated that we are not going to include.

Definitions of even no, no acceptable use policy because even back then they realized, and they could see it even in culture and, and, and all of that, that nobody's, we are not going to get on the same page on this.

Simon: That doesn't mean there's no place for them. So so I think it's absolutely fine for a project community to say that we are here to achieve these objectives.

Jonathan: Sure.

Simon: And the, and we're going to make sure that there's no one in our community that doesn't want to achieve these objectives. And when you find that excludes you, you fork, because you're free to do that, because the software license makes you free to fork. And then if there's a critical mass of people who share the other view, you get two pieces of software that are serving.

But you, that, that is a, the, that is a role for the governance of the community, not for the license of the software. very much. And so that's, that's the line that I, I think I'm still trying to work out how we draw on AI. Yeah. Because we're being encouraged by legislators to fold acceptable use policies into the core licensing.

And that isn't gonna work. Right. That's That's going to result in the fragmentation of the community down to individual company sizes. And that's going to result in there not being open source AI and thus not the, we won't get the social good of the network effect of massive collaboration.

Aaron: Yeah, I think there is a, I'm hoping this is true and maybe you guys can corroborate this or not, but I think that what you were alluding to there is this idea, I always worry, right, about open source, that it's going to be somehow influenced by, The conspiracy theorist, right?

It's going to be given a bad name. Just the reputation of open source could be affected or something. But I remember to your point earlier, one of the most interesting times I ever had at Sun was when we were down at Fizzle in Puerto Alegre. And I was giving a talk, so I was in the speaker room, and I'm sitting there, and there's two guys talking, right, in the corner.

And one of them is Richard Stallman. The other one is Peter Sund. I don't know if you guys know who Peter Sund is. I have a

Simon: photograph of that conversation. Do you? That's awesome. I do, yes.

Aaron: Wasn't that crazy? That was like That was reasonably

Simon: wild. You know, Michael Tiemann was also involved in that conversation.

Aaron: Yes! Like, how do these people get on the same page, right? Like, you would think they would be diametrically opposed or in some way, you know, how does the guy that created, or one of the guys that created Pirate Bay, sit down and have this philosophical conversation with Richard Stallman? But they were having that conversation, and I think it was, you know, my point is, can open source be more of a unifying factor?

And of course, this is again, why we need to get there with AI. But can it be the leverage that we need between all of these different political groups with different opinions to say, look, That's the nature of it. If you don't like it, fork it and do your own thing and see if other people like it. You don't have to have this conversation like this is, we need to change this or it's the end of the world because you can just go change it.

Jonathan: Yeah. So I'm, I'm, I'm involved in a, to, to kind of tie into this, I'm involved in a project, it's mischastic. And we, we have kind of a, because of what it is, it's off grid radio. We kind of have this like higher than usual population that are I do not, I do not want to insult them. So how shall I put this?

Well, they're, I guess, prepper community, right? We could, we could just say that. And we were talking again before the show, like I, I have some level of very much empathy towards that. If nor the reason, then I live in Tornado Alley. And so I definitely get the idea of let's try to be prepared. Let's try to prep.

And Then, of course, you have some people that just go way off into the deep end. And I, I've made the, I've made the point here before also that you, you have sometimes these people that, like, they will have reasonable points to make, but it's so wrapped up in the, sort of, the tinfoil hat language. So, there was a There was a deal where we, we founded a corporation to, to try to, you know, do some things around this, this particular open source project.

And we had a guy that goes, Oh, well, that's it. They're getting the Rothschild money now. And, you know, it was just sort of, it was off, it was off the rails. But when I actually read what he was saying, the, the point that he was trying to make was essentially I'm worried that now that there's a corporation, there is a leverage point for, you know, the, the government or whoever to be able to come in and say, we would like back doors in this encryption as well.

And like, I don't know if you've been watching the news, but that is not a that is not a tinfoil hat theory. That is a thing that government officials are actively trying to do. That's not a conspiracy theory, that's being done out in the open. And so like, yes, he was off the rails and he was wearing the tinfoil hat in the way that he presented it.

But there was a legitimate kernel of concern that you may not agree with but is a legitimate point to make And so this is something I spent a lot of time thinking about like for one thing How have we come to the point to where people just cannot make their legitimate? Concerns known without getting into this ridiculous language.

And then how do we how do we help? How do we help somebody get back from that point? It's all the internet's fault, you know yeah Yeah, that's probably true social media

Simon: you know, seriously, I think it is because I think that the The ability to compromise involves having a small number of opinions that you can think through rather than having a constant echo chamber reinforcing your first, your, your, your first surmise.

And I think that we've, the, the internet has produced this this dynamic for human beings that results in an inability to, to to, to empathize with. People from, people that you only know virtually. And I, I, I have no idea how we fix it. You know, the most obvious fix is to turn off the internet.

Bring

Aaron: back modems. Modems were underrated.

Jonathan: Media types.

Aaron: Not from a hardware perspective, but from a social fabric perspective. Yeah.

Simon: But I can't see that working out very well in the current environment. So I, I've no idea what we do, but and then the social media has then put that that effect on steroids and allowed people to create divisively boundaried communities for profit.

Yeah. And and you know, so what do you do about that? Well the best, very best thing you can do about that, if you're, if I'm honest with you is regulate it. Right.

Aaron: Right.

Simon: And and I don't see a great deal of that effectively happening just at the moment. So it's

Aaron: especially hard here. Cause we already have regulation that says you can't sue the social media companies, right?

So, you know, that's the thing that's like, I go back and forth on that one. I was like, well, I like that regulate. Cause I like technology and I don't want them to be sued. And I don't want to lose my. Whatever the latest social media thing is. Right. But at the same time, it's like, well, yeah, but the outcome of that could be this.

Other bad thing which we see happening right now, which is they can't be held responsible for what people say On their on their platforms.

Jonathan: Yeah, that is very much a two sided sword So I I fall into the camp of and I I acknowledge this is not working all that Well at the moment, but I very much fall into the camp of rather than solving it with regulation Let's solve it with open source technology This is why i've been trying to push people to mastodon because it it it cuts out all of these problems You can run it yourself.

You can, you can do, you know, whatever you want to with it because you can host it yourself. The problem really with it is two problems. One, there's not quite the critical mass that enough people are on it. Everybody is on, everybody's still on Twitter and then everybody's moving to blue sky and then you have a few people on threads.

But the other problem with that is that people just don't want to host their own mastodon server. Like the majority of people don't want to. And that. That sort of dries up a lot of that Advantage to it. So it's a it is it's a it's a tricky problem So simon, I see that I see the wheels turn and the smoke pouring out.

Yeah

Simon: so, you know that does concern me because I last I checked the the little islands of of isolationism Were running on mastodon Yeah, you know truth social is

mastodon

for example And I actually don't think that it with that that solves the problem because the problem is not a software problem.

The problem is a social problem and that social problem is being exacerbated by the technology and applying more technology to the social problem just gives you a bigger social problem. And I, so I. I don't really think that's the answer. I tell you what does happen though, when you get lots of open source software forming the basis of the economy like it does now, like whatever it is, I forget what the most recent research was, it's between 80 and 95 percent of all software systems are open source.

Yeah. And the what does happen is it's driving sufficient economic value to need regulating. And so that's what's happened over here in Europe. The European Union has decided to, to the Cyber Resilience Act is applying product liability to the whole product, including the software portion of it.

And so manufacturers are now liable for what their software does as well as what their hardware does. And that's going to change everything in Europe. I can't see you being able to get away without doing the same thing in the U. S., honestly. But Europe is a big enough block that it can, it can freely do that and not need everyone else to come along too.

And then we're going to see more of this. We're going to see more of people saying, hang on a minute. What is it that's driving this trillion dollar economy, economic force here? Shouldn't we be taxing it? Shouldn't we be regulating it? Shouldn't, shouldn't we, or, or, or alternatively, shouldn't, shouldn't we give, be giving it pork out of our barrel?

And those things are going to happen because it's big enough now to be visible from space. And as a consequence, the regulators are coming. And there may be regulators that, that pork barrel things that they took to, to, to help their electors. There may be regulators who want to stop people having fun.

There may be all sorts of different regulators, but now that open source software can be seen from space, the regulators are coming for it and you can't stop it.

Jonathan: And so there there is an interesting little quirk here and it's it's a difference between the united states and sort of the rest of the world And this is this is one of the things that was one of the latest kernel dust ups really had me livid in the united states we have the first amendment that that like enshrines freedom of speech as As one of the core principles of our entire existence And not to say that other countries don't have freedom of speech, but it's just it's so central To to the u.

s. Government to what the u. s. Government is supposed to be about and then you have some court rulings over here that basically say that code is a form of speech And so, there is a, there is a very interesting blocker to regulating, particularly open source code written by individuals, because that code is an expression of speech, and the freedom to be able to do, you know, to say what you believe is so bedrock to the United States.

It, it, it's just, it's interesting. I, I, I don't know that I have anything more to say at the moment other than that, but it's, it's really interesting what that's going to do when you try to get into regulation and all the other things.

Simon: I don't think that's going to cause too many problems for regulating it, because, you know, you regulate banks and people talk in banks.

And so you're regulating a modem and the software in the modem is what makes the modem do its things. I think you can regulate the behavior of the system. And when you recognize that the, that you may well have freedom of speech to create whatever system you want, you have freedom of speech to say whatever you want in the U.

S. And it's only when you start acting on it that you get, that you get restrained by the state. And that's, that's all this regulation is doing. It's, it isn't going to infringe anybody's freedom to write the software that they want. What it is going to do is remove their ability to be able to avoid the consequences of having written that software.

Jonathan: So isn't, isn't this though one of the things that, that you particularly had to fight for in the European Union and the laws there, the idea that the liability for the systems would be reflected back to the individual open source developers?

Simon: Well, well, in actual fact, the, the European Commissioner had done a pretty good job of understanding that, that, that there should not be mm-hmm

Liability imposed on people who were not benefiting from the availability on the market. What they hadn't understood was that the language that they used accidentally affected the people who were in the supply chain.

Mm-hmm .

Because they've, they, they'd had a view of the supply chain that, that there was always somebody placing the software on the market and that there was always a company and they were always an entity that was able to accept liability and they hadn't really considered some of the way that they had phrased things meant that You know, a guy in a barn in Omaha maintaining an open source library might well discover that he is a party to a liability claim.

Yeah, and that was what we had to fix. We had to to make sure that that their original intent of the legislation, which was to make parties placing software on the market in Europe liable for the consequences of doing so, which is a very fair proposition in my view, and we need to make sure that that intent Was the, was both real and bounded to the people who had actually placed it on the market and not the people whose work they had exploited to do so.

And, and we succeeded at that. We've, we largely succeeded at that. There's a couple of corner cases still. And so it's the, I, I believe the CRA is in pretty good shape in that regard.

Jonathan: And I would, I would, I would actually agree with you that I think similar legislation is coming to the U. S. You know, eventually, even, even if it's not like a federal legislation, probably what we'll see here is California will introduce it first.

Maybe. Because that's, that's the way things tend to go in this country, but

Simon: It can be you know, I, I also, I don't think you should underestimate the will of certain political actors to to cut the tech sector down to size. This, this is true? And I think you might well see some regulation that that has effects that are not unlike the CRA, but are there to make sure that the Googles and Facebooks and Microsofts of this world are not the the political forces that they have been for the last 15 years.

Jonathan: Isn't it, isn't it interesting how different groups and different political forces act in surprising ways sometimes? Yes. Well, I, so, I think something that we would all agree on with that is we hope they don't mess it up.

Simon: Well, yes. And, you know, that was, that was where, you know, I'm allowed to make political statements about the UK, aren't I?

That was where we, we really screwed up with, with Brexit, was we did something that was not Not realistically reversible still at the moment in the U. S. everything that's happening is still you know, conceivably reversible and it's not obvious that there's anything that's going to be different about that in this political, this, this upcoming political presidency.

And I think what you've got to watch out for is people erecting no return barriers. And you know, the place that that happens is actually in, in, in much more subtle places like the appointment of judges than the adjustment of electoral laws. I think that's where you want to be watching very carefully to make sure that your freedom isn't being abridged by anybody of whatever color.

Jonathan: Yeah, yeah, that's a, I think that's a reasonable statement to make. Let's see, we are, we are just about at an hour. Is there anything else that you guys want to touch on that we haven't gotten to? We, we talked about retrocomputing and politics.

Simon: I mean, you know, we, we, we could talk about sex or religion. We've never had shows on those. No, I think, I think we have avoided that on

Jonathan: purpose. I think I just try

Aaron: to imagine how that conversation would go. I can't even like.

Jonathan: So I've got, I've got somebody that follows me on, I think on Mastodon that will occasionally like and make comments and it's like an app to be able to get to some religious text.

So yes, there are, there are ways that that, that circle can be squared and that we could pull it in, but I, nope, not interested. We

Simon: can all, we've all seen the Saint Ignatius photographs.

I will say that so over here in Europe if you're going to be in Europe at the end of January. FOSDEM is happening. FOSDEM is coming, folks. You need to get yourselves over to Brussels for the first weekend in February, where more open source and free software developers than you've ever seen in a single place will all be trying to squeeze into too small a gap.

And we'll all be geeking out and probably drinking rather too much beer and pizza. And that is a completely free event. So FOSDEM you can attend completely free of charge. All you have to do is get yourself there. I think it's probably the highlight of the event year for me. The event that I, that I enjoy most from a health and welfare position is the SFS con that happened in Italy in November, but the one for simply seeing everybody and making the most progress ready for the new year that's got to be forced them.

So come and come and come, come on over. It's really easy to fly over to Europe these days.

Aaron: And when is that event again? I'm sorry I missed it.

Simon: That's the first weekend in February. So that's February the 1st and

Aaron: 2nd.

Simon: And there are actually quite a lot of adjacent events now. So they, it's now being called EU Tech Week.

And there are events as early as 29th of January. And there's events carrying on as late as 4th of February. So you can quite easily fill your calendar up there. And the best way to get to Brussels if you're flying from America is to fly to Schiphol, to Amsterdam airport and take the train.

There's a high speed train from Schiphol to Brussels. So you don't have to work out how to fly yourself to Brussels. And you get to try a really good high speed train.

Jonathan: Yeah, it's a lot of fun. Insider tips. Yeah. Aaron, anything you want to cover?

Aaron: Well, just the usual, you know, go check out my YouTube channels on Retro computing, vintage computers, all that kind of good stuff.

I just released an episode this morning covering the controllers that Atari put out the CX 40 plus and the CX 78 plus, which are reproduction Atari controllers but they are wireless. So, and, and they have dongles nine pin dongles, so you can actually connect them to your original hardware and get a wireless experience with a, with a controller that looks ju identical to the original, but it's wireless.

So, but then you miss out, they're pretty cool.

Jonathan: Then you miss out on the experience of playing your game and trying to dodge something and ripping your entire console off the shelf, ripping the controller out.

Aaron: Yeah, exactly. I did find a bug inadvertently actually, because, you know, me being me, the first thing I did was like, okay, well, let's really put this to the test.

So I grabbed my, my original heavy sixer, right? The first system that Atari made, you can't get any older than that and plugged it into there. And when I turned on the game, all of a sudden I was hearing all of these, all this new music that wasn't in the original game. And there's something actually in the wireless Controller that doesn't interact correctly with the heavy sixer and the light sixer It's only when you get to the one the four switch model that that problem goes away So there's something that was connected differently in those original ataris that causes the wireless.

Noise essentially that's going on in the circuit to bleed through Into the system and so it was really weird. I was playing with my son. We're like, I don't remember there being a soundtrack to joust on the atari 2600 And then we disconnected the dongle and it went away. I was like, oh So I wrote atari and said yeah, I think you need to be testing this stuff with the even older equipment than you're using but anyway, fun episode, check it out.

And there's a bunch of other Atari 10 things you didn't know about the Atari 2600. If you like Atari stuff, go check out my channel, RetroHackShack on YouTube.

Jonathan: Aaron, that's, that's not a bug. That's a feature. It, it introduces generative music to games. They tried to tell me that. Yeah, they were joking.

Aaron: They were joking.

It was all tongue in cheek, but yeah.

Jonathan: They

Aaron: said, Oh, I think I'd love to tell you that we planned it that way. But I think I'll forward this to our engineers.

Jonathan: All right. Anything else you wanted to plug Simon, you didn't get a, is there, is there a radio

Simon: I

Jonathan: just,

Simon: Put a new post up on, if people go to blog.

opensource. org, they can see my my article today all about the role that open standards are playing in the new regulatory environment and some unexpected consequences of it. I would love Focom Mastodon to, To to follow the OSI blog and anyone that finds any of the outrageous things that I'm saying interesting can follow me on Mastodon.

I'm webmink at mesh dot cloud. To follow along, say hi. I even reply sometimes.

Jonathan: All right. Very good. Thank you guys both for being here. I sure appreciate it.

Aaron: Absolutely. Anytime.

Jonathan: Yep. All right. So you can find my work of course, at Hackaday. We appreciate Hackaday being the home for Floss Weekly these days.

We've also got the Untitled Linux Show over at Twit still, and you can you can find my security column talking about sort of the other side of the coin there at Hackaday on every Friday morning and have a lot of fun with that. We appreciate everybody that watch both live and on the download, and we will see you next week on Floss Weekly.

Episode 812 transcript

4 december 2024 | min

FLOSS-812

Jonathan: Hey folks, this week David Ruggles joins me and we talk with Sylvester and Brian about Firefox, the original open source browser. Something of a perpetual underdog, but maybe it's time to take another look at. You don't want to miss it, so stay tuned. This is Floss Weekly Episode 812, recorded Tuesday, December the 3rd.

Firefox and the future. It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we're going to have a lot of fun today. First off, we've got the one, the only David Ruggles, the David Factor, the, the original David Ruggles, as opposed to all of those clones running around out there.

Hey, David, welcome.

David: You're just trying to bring in every single one of my handles from every possible source. Oh, I'm excited to be here today. In proper form, I am calling in from a Firefox based browser, so this is, this is awesome. I'm excited. In fact, I plan to do a lot of listening and not a whole lot of talking.

Jonathan: Oh, well, but I mean, you're kind of the, you're the, you're the expert co host this time. You're, you're the Firefox user. So you gotta, you gotta be on the ball to be able to jump in and ask questions. All the things that I wouldn't think to ask. I can do that. Yeah. So for those that don't know, that are not watching the schedule the way that we are, our, our Guests today are Sylvester and Brian of the Firefox project.

So from Mozilla and boy, there's a lot, there's a lot going on with browsers these days, and I am just thrilled that these guys are willing to be here and chat about it. There's. There's some things to talk about. So let's, let's go ahead and we will just go ahead and bring them on. So Sylvester and Brian, both welcome.

Glad to have you both. Thank you.

Brian: Glad to be here. Thanks. Thanks for having us.

Jonathan: Yeah. Let me start with Sylvester and just kind of give us the rundown of like where, where are you in Firefox? Where are you in Mozilla? Well, how do you, how do you fit into this, this thing that we all know and love?

Sylvestre: So I am, I am.

I started at Mozilla 11 years ago, and now I'm a director of engineering. I'm managing an organization of about 50 people. We do release management, release engineers, OS integration, engineering workflow, and some security work also. So we do a lot of things, and we touch a lot of different parts of the project.

All

Jonathan: right. And then, Brian, same question for you. Kind of help us get the mental map. Where do you fit into the organization?

Brian: Yeah, I'm a senior principal engineer working on Firefox and Gecko, the web engine that powers it. I joined Mozilla in 2013 and I've worked on a whole bunch of stuff from kind of dev tools to the desktop front end to platform and recently performance and a whole bunch of other fundamental stuff.

Jonathan: Yeah. Okay, perfect. Perfect. So both both very senior engineering types, which we love. We'd love to be able to ask the engineering questions. But before we get to engineering, I think there's some, there's some interesting history stuff to talk about. I don't think we've ever had like Mozilla or Firefox on the show.

So let's, let's talk about some ancient history and Firefox and Mozilla used to be Netscape. Is that, is that fair to say there's at least a connection to Netscape?

Sylvestre: Oh yeah. In a while, right? So we just celebrated the 20th birthday of Firefox a few weeks ago. Basically Firefox started on the ashes of Netscape.

So, you know, Microsoft decided to ship IE with Windows for free, and back in the day, we, Netscape was sold. Two users. So obviously it Netscape took a hit and some people decided to refactor on Netscape into what we call back then the Matia project. And then the project was renamed to Firefox and here 20 years after.

So, but it's also a, Firefox was really a revolution. We think about it like most of us in that call are pretty old. And remember what it in the day when had popups everywhere. And you didn't have tabs. So if you wanted to look three different websites, you have to open three windows. Now we have tabs and a proper blogger.

Firefox is a browser that created that.

Jonathan: Yeah. Boy, there's some, there's some cool history there. I remember Firefox 4. It's kind of a big memory I have because I was just getting active like in things and on the internet. And I remember when Firefox 4 was sort of the big push and I don't remember when that was.

Early, early 2000s I guess. And that was, that was a big deal because you know, there were cool things happening in Firefox and it's been a long time ago now. And I was, I was talking with David before the show and, you know, you kind of look at the, the, the current browser wars and the, the how shall we put this, the market share of Firefox is maybe not quite as big as what everybody would like.

And it, it occurred to me, Firefox has been in this position before. This is, this is not the first time that Firefox is, is approaching the the market as the underdog. And this in some ways is a comfortable position for you guys. And one of the things I think that we can dig into maybe today is like what, what cool things are coming because you know, when you're, when you're a bit of an underdog, that kind of gives you a bit of freedom, I think.

to to experiment and try things out. And so I'm, I'm hoping that you guys will have, have some neat things to talk about that you're kind of looking at with that, that you know, things coming, things down the pike where Firefox is going to once again, be sort of shaking up the entire ecosystem.

Brian: Yeah. Yeah. It's, it's a, it's a, tough position to be sort of the independent choice and not be the default on on most, most devices. But I think at the same time, in terms of, I think that's important for the platform in terms of having an open interoperable platform and then being able to compete on, you know, basic browsing fundamentals, performance, security, stability you know, managing web compatibility, and then you know, adding a bunch of, a bunch of new features to just make getting through that.

Getting through the day better, you know, people use their browsers for all sorts of things today. And so there's a bunch of stuff coming down the pike in the next year that we'd love to get into.

Jonathan: Yeah, but before we, before we dive into sort of what's coming, maybe let's, let's talk about some more of the like fundamentals.

So Firefox has been around for 20 years and If, if I forget this, David, you particularly, I'm gonna put a pin in this and you help me remember, if I forget, I want to talk about Rust too, because that's something big that Mozilla has been involved in. But I guess first, what, how, how big of a project is Firefox?

Like what, what does the line of code look like? How about, do you know, just off the top of your head, like in the, in the ballpark, how many commits there are? Let's, let's chat about the size of the thing.

Sylvestre: Sure. So it's one of the biggest project that we can find on the planet. Like we we love the expression.

It's not rocket science, but here it might be more complex at rocket science at time Because when you think about a brother you have to run untrusted code all day long and you have to keep the user safe And secure so for firefox to give you a scale of what we have i'm looking at the number We have more than 10 000 Contributor over the history.

We have about 1000 contributor per year. We have about 31 million of code. Of course, some of them are platform specific. So you don't always get all those code in your binary. Some of the code is Android specific. Some of them are Mac specific, et cetera. And we are close to 1 million commits in the code base.

So when you think about it software that has been alive for 20 years, how many of those are still alive? The Linux kernel.

Jonathan: Yeah.

Sylvestre: So windows. And Photoshop, there's only a few. So yeah, it's crazy the scale of what we do.

Jonathan: Yeah. You know, we, we had, we've interviewed some of the people that have been around for 20 years and there's, there's projects that you wouldn't necessarily think of, you know, like the core utils, you know, some of those have been around for a long time, Emacs and VI, but.

Most of those have gone through, like, big shakeups, and you know, we're not still using the same Unix core util source code from the Unix days, right? People have re implemented that now a couple of times. And there's now, you know, this interesting move to re implement it in Rust, which is really fascinating.

I have put my foot in my mouth a few times and said that Mozilla created Rust, and I've been corrected each time. No, no, no, no, no, that was this professor guy. And Mozilla just came along and used it. But, is it fair to say that Mozilla is responsible for a big part of where Rust is today? I think that's probably fair to say.

Sylvestre: I think Graydon was a Mozilla employee when he started that project. Oh, so I think it's a, I think it is a Mozilla product. Like it is, maybe I'm showing off, but it is something that Mozilla created and we had like 10 or 10 employees at some point working full time on Rust. And we are still contributing to Rust in many different places.

So to me, it's a Mozilla product. Correct me if I'm wrong, Brian.

Brian: Yeah. And to, to add on to that, I think nowadays it's a separate sort of spun off into an independent foundation and has huge traction in the ecosystem from tons of, you know, companies and open source supporters and, and and, and whatnot.

But I think in terms of what it was sort of designed to do was to be a systems programming language that could build a browser and certainly to, to my knowledge it was sort of incubated and started at Mozilla.

Jonathan: Yeah, does that put Mozilla in an interesting place as kind of still well? So is Mozilla still sort of the main stakeholder?

Are they are they sort of the The the place where rust still lives and does that put Mozilla kind of in an interesting place as like the engineering experts?

Brian: So now it lives in in an independent foundation separate from Mozilla. It was sort of spun out To, you know, to, to broaden the ownership and sort of governance of the project some years ago, but we still have a user still involved in in the, in the core language and we are landing new Rust, you know, every day.

So we're certainly a big fan to continue to be a stakeholder.

Jonathan: Yeah. Are, are you just adding new features in rust or is there kind of an, an active development process to port things over from the old C code? Or is it c plus plus? Honestly, not sure. Are you actively porting old bits of the code base to rust?

Brian: Yeah. So it's mostly c plus plus. Mm-Hmm. , we are not, we don't have a concerted effort to kind of port you know, the millions of lines of existing code. But what we do is when we have a chance to redo a component. Or some sort of nice encapsulated portion of code. We tend to reach for that just for the security and sort of ability to parallelize code and so on.

So we had a really big project where we took the Stylo CSS engine out of Servo, the engine that we had sort of incubated. Within Mozilla and put it into, into a Gecko. And so that's a huge sort of rust component that handles all of the CSS, you know, parsing and matching and interaction with the layout engine.

That's entirely in rust. And we've seen amazing performance out of that over the years. And we've done the same with. Parts of the rendering stack, I think, crash reporter and, and other components.

Jonathan: Yeah, very cool. I, I, I have been saying for months, if not years now that, oh, well, I really just need to sit down and learn rust at some point.

And I have finally with the the 2024 advent of code, it's a, it's a fun little project where you get a tiny little code challenge. For every day for December up until the 25th. And, and I got to looking at that and they go, Oh, we support every single language. I was like, okay, it's time to learn Rust then.

So I've done the first couple of days of the advent of code and Rust. And I now understand why people, when they first start learning Rust say that it drives them nuts, but then it makes them better programs. I'm beginning to understand what they mean by that.

Brian: You'll have to ask Sylvester you know, about that one as well.

I'm sure he's got some stories from that.

Jonathan: Oh, I'm sure. All

Sylvestre: right. It's not always easy to start with Rust for sure. Yeah.

Jonathan: Well, it's just different. It's a different, it's a different paradigm, really. Because, you know, with C, you return an object, like you return a string, or a value, or whatever. And with Rust, you have this idea of, we're going to return either a value, or We're not going to return anything.

And Russ doesn't have an idea of a null. So you've, you've got this, like this week. So coming at it as a C plus plus developer, you've got this weird thing that you've got to handle. And then, you know, the first time you start using it, it's like, do I need an ampersand? Do I need to do a dot unwrap? How do I actually get it?

The thing,

Brian: I'll be interested to hear how the rest of the advent goes for you and sort of, as you get through, I assume it gets more challenging as it goes. So. Maybe it'll be a good ramp up.

Jonathan: That, that was, that was my hope. That was my hope that it would be a, a good sort of soft introduction. And there, the, the nice thing about it's they're bite-sized problems, right?

Like the first one was you're give, or let's see, the second one, I don't remember exactly what the first one was, but the second one was like, you're given. five numbers, and you have, and it gives you a very simple pattern, like each of the numbers have to be within two, and they all have to be either incrementing or decrementing, and so you've got to do some things like, all right, so we're going to take this, this text file, we're going to split it by new lines, and then we're going to Split it up by whitespace, and then we're going to convert each of those into an integer, and then we're going to do some simple math on the integers to make sure.

And then, you know, return a number and then do a, you know, a running calculation. It's all very, very simple stuff, but the fact that you're doing it in a new language, for me at least, I'm doing it in a whole new language to me. So it's like, over half the time I'm spending there on Google trying to figure out, alright, how does, How does one convert from a string to an integer in Rust?

And what kind of thing is that actually going to give me? But it's, it's, it's been fun. It's been a lot of fun. But it is very, it is different for this, this old C programmer.

Sylvestre: I can, I can share an anecdote about Rust. Given the scale of Firefox, we have, we need to develop plenty of tools around the product itself.

So for example the crash management is one of them. So we used to use bpa, and BPA was not very maintained and upstream was not really accepting our patches. Mm-Hmm. And it was quite old and hard to maintain and we decided to be right. It in rust. And and I, one, one of the anecdotes is that we, we wew wrote a pt B, so the Microsoft par Microsoft debugging format.

We wrote that, we rewrote the parser in Rust and it significantly decreased the load on the server, which were doing the parsing of the crashes, to the point that I think that some Microsoft team are now using our crate that is doing PDB parsing for their own format they created. And we have those kind of anecdotes with Rust everywhere.

Rewriting in Rust sometimes can be expensive, so it's not something that we always want to do, but the return on the investment can be huge.

Jonathan: Yeah, yeah, definitely makes sense. Okay. So what about contributions is, are there still quite a few contributions that come from outside of Mozilla? And what does that process look like?

Like how difficult is it to land something in Firefox?

Brian: Yeah, I think Sylvester may have the numbers pulled up on sort of the, the outside contributions. We, we definitely get a lot of outside contributions. I think in terms of difficulty. It's, you know, it's a browser. So I think finding, we try to identify sort of good bite sized places to start. I think we put a lot into the tool chain to make it easy to, you know, pull down, bootstrap all the dependencies and do the build.

I was just looking, you know, at some of the sort of. Depending on how fast your machine is, you could get built as fast as sort of four or five minutes or, you know, 20 plus minutes or 30. It just depends on the much. It's actually improved since I started at Mozilla a lot. We've invested a lot in that and then it's, it's open source.

So it's MPL licensed. Anybody can, can kind of pull it down and make contributions. And so I think finding the right. You know, we have, we have pages and stuff and a chat server if you want to hop in and get involved with the project, but we've gotten a lot of great contributions from people outside of Mozilla and continue to.

Jonathan: Yeah. So you mentioned the MPL license. Let me cover this real quick and then I'll toss it to David. What, what kind of license is MPL? Is that based on the GPL? Is it a BSD style? You know, how permissive is it?

Sylvestre: It's very close from the Apache. License. It was written by Mozilla. I think it's one of the first licenses of that style.

So it's a, it's, I wouldn't say that, but maybe to take a shortcut about license, not to base it about license it's between GPL and MIT. So we get some protection. It's very flexible. So you can fork Firefox and do what you want with it.

David: And David, feel free. I know that this is the Firefox show, not the Rust show, but one question with the public contributions that you're seeing, are you seeing a uptick in Rust contributions over C It's

Jonathan: a good question.

Sylvestre: I'm not sure. I, I'm not. You know, when I'm bored, I'm going between two meetings. Most of my life is spent in meetings. And sometimes between two meetings, you want to do something concrete and writing docs or talking with colleagues. So sometimes I'm running good first bug. And I know that I'm opening Python and C and Rust.

And usually, so, So those good first bug are taken very quickly by a contributor. Most of them are clippy fixes, that kind of stuff. With clippy, it's a static analyzer in Rust.

Brian: Yeah, and I don't know within the tree. That would be an interesting query to run, I think. We also have a lot of projects that are outside of Mozilla Central.

So the main sort of Mercurial source tree. That get vendored in that we still maintain. So, for example, we're working on a WebGPU backend which is a new API in JavaScript that gives you sort of platform independent graphics programming. And that is a separate GitHub repository called WGPU. That's, that's Rust based and is used by other projects as well.

And so we have that there's a quick stack. There's a lot of and then there's a lot of code sort of that's, that's very independent from the browser in terms of the repository itself that I think have also picked up some good outside contributions.

Jonathan: Yeah. So kind of in this vein of talking about the the, the various contributions and what gets worked on how, how does Firefox, how do you guys make the call?

Like what, what do we spend our, our development time on and kind of related to this? One of the I guess I could call it a criticism, suggestion, scuttlebutt. Something people say not just about Firefox, but about a lot of different projects, is I wish they would spend more time on the fundamentals and not work on and then X, Y, or Z, whatever that person is frustrated about at the time.

And so sometimes that's, I wish they wouldn't waste their time doing AI integration or, you know. And what's ironic about this is people have said that about different things over the years and Sometimes those criticisms turned out to be well founded and sometimes those criticisms were very not well founded.

And the thing that people were, you know, wasting their time on turned out to be the next killer feature. And so that's something I found is that it's very hard to know. But inside of Mozilla, like what, what does that process look like? How do you guys come to that conclusion of, okay, this is the thing we're going to work on.

And I leave that to the two of you to figure out which one is, is more appropriate to answer the question. Both of us, we have different

Sylvestre: perspectives. Brian, you want to start? Well, I think,

Brian: yeah, in terms of, in terms of priority, an enormous amount of effort goes into fundamentals. I think that's been true over the years.

I think. We've had a huge focus in the last couple of years on performance in particular that we could we could talk about sort of speedometer three benchmark and a lot of wins we've seen on that. But, you know, performance security stability. We put Mozilla puts an enormous amount of the investment into Firefox into those things.

They're not always super visible which is 11 difficulty with that. But I think they're they're super important. Like, in terms of performance. Advancing the web platform, both for the, you know, millions of Firefox users. And then also just for the competition in the, in the ecosystem, it's important that you're sort of never slow down on that stuff.

And that can be basically an infinite amount of resources you can take onto those things. And so, of course, you have to make decisions. On priorities and prioritization, especially for a new feature development. So I think in the, in the coming year, you know, there's a lot of investment planned in just improvements for getting through the day on the browser.

So things like tab groups vertical tabs for people who want that with the sidebar that has kind of more rich. Capabilities to interact with the web page, better you. I for managing user profiles. So for completely independent profiles, you could always kind of switch between those in Firefox, but making that really pleasant to use and then, yeah, building features that are using you know, newer technologies, AI inference, you know, such as local translations for language translation alternative text captioning and so on.

And so I think that would be an area interesting to, you know, to dig more into. But I'd also, you know, love to talk more about the performance end of things. And also maybe I'll give Sylvester a chance to jump in there before we go deep on that. Yeah, sure. Yeah.

Sylvestre: One of, one of the things that we, we have to keep in mind when we talk about browser roadmap and prioritization is that a significant part of our work is defined by the startup bodies.

So when there is a new, when there is a new startup coming, we, If it makes sense and if our competitors and friends are implementing it, we have to implement it. Otherwise, you are going to get some window on some website saying not supported on Firefox of things that we used to have back in the day. So we, we have some of our roadmap is driven by, by those standards.

So it's always a trade off. And to be honest, when, when you, you mentioned some comments that we can read online, we read those comments and we have also the tools to Those discussion internally, like, is it the best use of our time? Is it the best way to advance Mod Mission? So, as you know maybe not always the business know, but Modia is driven their mission with, and our goal is to keep internet healthy available to everyone and the public resource.

And all decisions that we make on Firefox and at Madia in general are, are based on that mission statement. So Brian mentioned performances. We have web compact, we have offline translation, privacy processing translation. So those kinds of things are clearly what what is advancing the web.

Jonathan: So Brian obviously wants to tell us about performance.

And so I'm going to, I'm going to let him, I'm going to give him a little space. Brian, what, what is, what's a recent performance win that you are proud of? What's something that's coming down the pike when it comes to performance?

Brian: Yeah. Yeah. Thanks. Thanks for that. Yeah, I think. So we had a, a project actually with, with with all of the browser vendors.

It's, it's not, not the standards based, but kind of standards adjacent project to ship a cross browser benchmark. So obviously people have benchmark browsers for a really long time. We got together and worked on a new benchmark called Speedometer 3 in the last couple of years and have shipped that shipped that this year.

That really focused on kind of end to end user journeys. And so one hazard of benchmarking as, as people probably are aware is you can create these little tests that are not indicative of the real world in any way. And then you optimize those tests to no end and it actually doesn't really help anybody.

And so we wanted to build a benchmark that was really end to end. Like it, it opens a webpage, it renders a chart, you measure kind of how long it takes to do. These different things. So I identified a bunch of use cases, vetted all the code work together with the other vendors to put that together and then sort of set off individually to go compete on that you know, a solid kind of.

framework. And so we, we have an amazing set of performance tools internally. I think that, that lets you sort of take profiles, share them, a great sort of web UI and user experience to share and pinpoint exactly what's slow here and there. And it's really satisfying work, I think, to kind of go and you take a thing, you see exactly where it's slow and how you make it faster and you go you know, work on the code to make it faster.

So a bunch of examples, I think. One that stands out is there's a JavaScript feature called proxies, which is kind of a weird metaprogramming feature. Where you can wrap an object and then sort of intercept any call into that object and like customize it before it happens. And we, when it shipped, it was not used really.

And so all we did was just focus on correctness for the feature and we had never noticed any usage in the wild. We had never heard complaints about our performance. As part of the benchmark, we updated to the latest view JavaScript framework, and it turned out we were like very slow at this test and it was, you know, very concerning.

We're landing these tests that were really bad at, and we started to look at it. Well, they started to use the proxy feature because it's kind of convenient for this, like, reactive programming model to, to intercept calls and know when some state changed. And so that was like evidence that this thing is used in the wild.

Then we went and we, you know, made sure everything happened in the JIT and made it way faster and. You know, land of the change and move made the benchmark faster, and we start to see improvements on real pages, you know, real user metrics and so on from stuff like that. And so there's just. You know, hundreds and hundreds of bugs like that, that, that we've done in the past couple of years.

And I think you may not notice any individual one as a user, but all in all, you know, when you press a key and the text appears quickly, or you click a link and the page is still responsive, you can feel it. And it makes a difference. I think in a, in a in a, in an incremental way, when not, not just Firefox, but everybody, all the browsers are really pushing against that.

Jonathan: Yeah. So something comes to mind with that is. What about on the browser? Like how much of the, how much of the code base is and I know there's some interesting questions with the browser. We'll get into that in a second or excuse me, with, with mobile how much of the browser code base is shared on the mobile platforms, like on Android and then even on iOS.

Brian: Yeah. So we split the the kind of, when we talk about the browser, we split it into two parts. One is called content and one is called Chrome. Kind of confusingly, the browser Chrome is like the tabs and the URL bar. And so. The content is on on desktop and Android is rendered by Gecko. The Chrome, so like the tabs and URL bar on desktop is actually also rendered by Gecko.

So it's, it's a web app. This actually kind of blew my mind when I started and joined Mozilla. Is I was like, okay, so how are we at a feature here? And it's like, oh, well, it's HTML and CSS, basically. And so, so when you're like typing in the URL bar, there's pop ups. That's all just like JavaScript. It's very nice to write as a front end web developer.

On Android, it used to be done that way. And then we switched to use the native UI controls for the browser Chrome on Android. And then on iOS, we use the native UI controls for the Chrome and we use WebKit for the content. Due to, you know, app store policy.

Jonathan: Right. Now, is that, is that changing? I know at least in, and so here, here's the, the background for this.

I, I don't know how many people know this. Am I going to get myself in trouble? Yes. Apple is terrible in my opinion. And so one of the things that in, in fact, I'm, I'm not sure why it is company A that we have an antitrust suit going on with and not Apple but that's just me. I think David agrees.

Anyway so what Apple has done, as I said, you know, if your browser is going to be in the app store, you're actually going to use our browser code. You don't get to bring your own code. for the actual rendering of the webpages. So for the longest time, when you, when someone runs Firefox on an iPad or an iO, any, any iOS device, you're actually using the Safari code to render the webpage.

And in Europe, there has recently been a, a new law passed that basically says, no, no, no, Apple, you don't actually get to do this. You need to support it. Third party app stores. And so the obvious question that I have about this is, does that change anything for Firefox and are we going to see another version of Firefox show up in one of those third party app stores?

It's

Sylvestre: a good question. So we, we, we are investigating. How much work would it be to have a part of Gecko? So what Brian described, so the core part of Firefox. How much work would it be to port Gecko and to maintain it on iOS? But as you can imagine, it's a completely different platform. So the constraint in terms of performances and integration CI are completely different.

So we are spending time on it. Maybe we will ship it, maybe we won't. It's still TBD, it depends on the ecosystem, the work that we have to do and the next step in the process.

Jonathan: Yeah,

Brian: I'd say we're definitely fans of the direction here. We've been working, you know, with with speaking with regulators, with Apple on the proposal and really digging into the technical fundamentals to make sure that it's sort of suitable for being able to run, run on a device, being able capable for the architectural differences between the engines.

And so there's a lot of work there that we're very interested in and continuing to pursue.

Jonathan: I, as an aside, I am extremely hopeful that the third party app stores will come to other countries outside of Europe. I, I do have a single iOS device and it is I am not using it to its full potential because of that, because there are apps that I would like to be able to use.

There are just. Not available inside of the official app store. So it is what it is. David, you want to jump in for a while and get some of these questions off your chest.

David: Oh, yeah, I've got lots of burning questions. So to back up a little bit when we were talking about your focus and what you're able to focus on, one of the things about Firefox that I love and as I mentioned at the beginning, I am using Firefox right now to have this conversation is your extension ecosystem.

And so, when you talk about, you know, the Chrome of Firefox and some of the, some of the other features, even ad blocking and stuff and you, we can see, like, the moves that the Chromium side of the house has moved away from ad blocking and, and that sort of stuff. So, that has actually driven some interest in Firefox.

But does that ecosystem, of extensions, allow you to focus more on that core speed and that core rendering engine and everything, and kind of watch the extensions to say, Hey. This is where people, there's interest in stuff and maybe it makes sense to like do the tabs on the side in the future, but you can kind of maintain your focus while letting the extension community experiment with Chrome and the other attributes around that.

Brian: Yeah, I think it's a great point. I think just to reiterate, like we, we support the web extension APIs, including the you know, previous or the newest version, the previous version, we continue to support that, you know, ad blockers, all of the existing add ons in the store on both desktop and android. Now that's a somewhat newer development that took quite a bit of engineering work.

And so we big fans of that. And people use that for all sorts of things. Things I think in terms of using it as a way to sort of identify features that people want the direction things are going. I think it's a good point. And there's times where that extension APIs are a bit too limited. I think like we've seen interest in, you know, vertical tabs, tree tabs and so on.

And there's some really good extensions to help manage that. But to get something that's like really tightly integrated into the UI that supports all of the different features, you know, we talked about grouping tab groups, making sure that's a first class thing that works in kind of both modes. It's just really hard to do that when you're sort of here's a surface, render what you want and through the just like anything with an extension API, it's always a bit harder to kind of coordinate changes.

So I think it's kind of a case by case thing, but to the extent that we can. Let the extension ecosystem sort of pave a cow path or show a direction forward on, Hey, this is a useful feature. It's getting a lot of traction. I think that's a great way to go.

Sylvestre: I was looking at the numbers and we have 600 extension on Android. Now you can look at how many Chrome extension you will find on Android. Probably zero. Yeah,

David: definitely. Let's see,

Jonathan: I didn't know, I didn't know that there were extensions on Android. Actually. I'm, I, I definitely need to go look into that because that's, that's a thing that has been missing on, I think all of the browsers for quite a while now is able to do extensions on mobile and there are some times that that would be extremely handy.

So that's, that's a win.

Brian: Yeah. Yeah. Check it out. I think especially, you know, you're concerned about, about bandwidth performance, maybe there's productivity things you want to add into the browser to customize that, you know, download Firefox on Android and check it out and see, see if it see if it works for you,

David: David.

Yeah, not to rehash it, but that's where I feel like if you are able to be successful in bringing Gecko to iOS, that extension ecosystem will be a huge win over there as well. Well, it's

Sylvestre: not only an extension, right? It's limiting our capability to innovate on iOS because we are constrained by WebKit and WebKit is not shipping as Often as as chrome, for example rs and it is limiting the innovation and we have so everybody Like if you as jonathan said earlier if you use firefox or chrome or brave on ios You always get the same bugs as the same bug of the platform.

So Shipping on this platform. We can do some nice, cool stuff with web assembly with we were talking about offline translation using was some advanced security things that you don't have with WebKit. So have plenty of things that you can do when you control the full stack.

David: I actually wanted to ask about WebAssembly and the effort that it took to integrate that and kind of where you see that going because that's becoming very hot in the development ecosystem.

Not

Sylvestre: an easy question. I think it's one of the things where Mozilla was at the beginning of that technology like 10 years ago, we had different technologies. So Google was pushing PNaCl Microsoft at some point was pushing ActiveX. We, we were pushing a a subset of JavaScript called ASM ts, which was high performance JavaScript and then with, with our partners, we coordinated to ship web assembly to a product origin.

So we have been, part of the people who created that technology. Brian, Brian can probably say more than myself about the next step in that space.

Brian: Well, yeah, it's really there's a lot of momentum in that space. And the interesting thing is, it's not all in the browser. You know, you think about it, it's web assembly.

It feels like a browser feature. But I think there's a lot of interest in sort of server side you know performance secure kind of language independent runtime. And so one of the, one of the challenges to my understanding is sort of making sure that the features are roughly aligned and make sense sort of in both of those context.

And then also just honestly, there's a lot of low hanging fruit still to adopt it, even with the capabilities that exist today. Like the, the translations infrastructure I didn't know if that would work when we started the, the project, but the way that works is we, we you start Firefox, we don't ship any code related to translations other than like a small little WASM thing to check and quickly guess what the language of the page is.

And then if it appears that the page is a different language than your Firefox is, then we prompt the user. Hey, do you want to translate this page? It will actually go and download the WASM program off the, off the internet. And so you're not shipping, you know, a couple megabytes or whatever size of the inference itself with the binary until the user chooses to.

And then it will go down with sort of language pair weights, maybe 10, 20 megabytes, something like this, and then you can integrate it really tightly with the browser, because, as I was mentioning, the browser has, you know, a javascript and wasm runtime, both for the front end and sort of under the hood.

And so then we can integrate it, translate the page, you know english to french, whatever it is very performantly, like, it took quite a bit of work to do this, but we've even been able to start shipping that on android devices. And yeah, when I started, I was thinking, okay, doing this sort of an abstraction layer above C Are you really going to be able to run this?

It kind of. In a way that makes sense for a user and they're looking at a page and it's sort of you can, you can see it very quickly kind of work through the content of the page and translate it. So it's really exciting. And I think there's lots of other use cases. You can just build on from there. And it's very secure, portable.

You know, it's not it's not. Included in the binary and so on. So just a lot of benefits to it.

Jonathan: I just, I just went and looked at Firefox installed on mobile on my Android. And I just went and looked at the extensions available. And one of the ones that's there is tamper monkey. And I am now entirely sidetracked because the ability to write little snippets of JavaScript and change websites on mobile is just been forever out of, out of reach.

And I'm, I'm super excited about that. Oh, that'll be fun. Let's see. We, we, we touched, we touched briefly on this and I think it might be worth kind of digging into that, that other browser. I don't know. Is it the browser that should not be named? Over and over in Chromium land, they are, they are changing the way extensions work.

And you mentioned briefly that you guys are supporting both the, the new version of extensions and the previous version of extensions. And I know there are some people that are really up in arms about that change over in the Chromium code base. Is this, is this something you guys foresee happening, like long term is supporting both of those versions of extensions and like, what does, what does that look like going forwards?

Brian: I, I'm not super familiar with this area of the, of the code itself, but I think in terms of extension support in Firefox, yeah, I don't think there's any. Intent to, to change. We sort of support the current set of APIs. A lot of add ons are programmed to those. I'm not quite sure the sort of deprecation plan at what point would it be if you're going to.

Ship it. It's only supported at Firefox. And if so, how does that change people's mind on kind of shipping it? And, you know, I think on some, you know, on things like ad blockers, I think there's some cases where the new model is maybe a little bit less resource hungry. And maybe some people would prefer that.

Whereas the older model is more feature rich and people prefer that. I think in terms of, you know, I think just the users could choose. Which is their preference. So long as the extension is supported.

Jonathan: In the, again, this may be outside of y'all's expertise, but in the kind of extension store for Firefox I assume there's, there's gotta be like some, some checking done for malicious code.

Has that ever been a problem? And, and how, like, how does that get sorted? How often do you see malicious extensions get uploaded and do they get caught pretty much right away before they get offered to users? I mean, I would assume that that's the goal.

Sylvestre: So we have a team who is in charge of doing moderation.

So we, we have a lot of tooling, we have a lot of static analyzer and just kind of tool to verify that nothing bad is happening to the user. But yeah, we, we have precedent of some, some extension being used against the user at the end. Yeah, fortunately, but yeah, we are, we are very careful. So that, that's why sometimes it takes, a few days for some new extension to be approved. It's because we need to make sure that every new extension is safe and is not going to do some crypto mining on the system of the user, those kind of things. Yeah,

Jonathan: and crypto mining is really the least of our worries, right? That's, that's the starting point.

And it just gets worse from there. What about, what about detecting those sorts of things in, in websites? Because like a website gets to run basically arbitrary JavaScript. Does, does Firefox do things like trying to detect crypto miners and JS in a website? Or I know obviously there are some things that are malicious that you guys have to, to stay on top of.

What, what does that process look like?

Brian: Yeah, we there's a built in features called tracking protection that has sort of support for each of those classes of sort of nasty code, you know, crypto miners and cross site trackers and that sort of thing. That's you know, there's a sort of curation process. It's tough because you're sort of on defense with that.

So you have to identify. Which, you know which scripts are doing the tracking, add them to a list, deploy that list and notify the clients. And so it's really a hard problem. A lot of challenges with the browser is trying to sort of understand and navigate, you know, unknown web content. And so it would be great to come up with some kind of heuristic.

Where we could say, okay, we have some confidence that this script is doing something like crypto mining as far as I know, that's sort of in the domain of research at this point. And we're starting to, you know, I would be interested to I know we're doing research on fingerprinting and other things, but it's just so hard because.

The APIs that you might want to use for doing something malicious or the same APIs that you would want to do for something legitimate. And so it just gets into really messy heuristics of, well, if you call it this many times and this long, is there some kind of budgeting thing? It's really messy. It's very hard to do in a reliable way.

That's part of what makes browsers challenging.

Jonathan: Yeah, and even a step beyond that, like, so what would you do? Let's just, you know, kind of game theory this up for a second. You know, say you had a heuristic that could detect crypto mining. Okay, so we're going to block that in the browser. Well, what do you do when someone wants to run a website that that is the point of it?

You have an account, pull this page up whenever you're not using a computer and make a few cents an hour doing crypto mining with us, right? And so then you're in this, you know, you're in this weird position of, okay, well, we have to go and approve these URLs to be able to do it. And oh, that, that gets very, very hairy and complicated very quickly.

And. I mean, I guess that's what that's what it is, though, running a browser. It is hairy and complicated because you have all that stuff all the time.

Brian: Yeah. And then every prompt that you're kind of showing the user at some overhead of sort of explaining and making sure they understand what they're consenting to or not.

And so, you know, obviously, ideally, you would, you would set defaults that are like, right for what you feel is right for users and give them an easy way to change it. But it's super, super hard to do that. In a way when the, when the content is not sort of deterministic or controlled by us, it's sort of decentralized content.

So yes, you're right. Yes.

Jonathan: And then, and then, you know, someone passes a law to try to help it. And the next thing you know, every website you visit has a cookie banner. Click here to accept cookies. Yeah. I, I would love, I would love a, a feature in Firefox could do it. Maybe maybe we make this a web API that says I actually know how cookies work and websites are allowed to do local cookies and please don't show me any more banners.

Could we make that feature work? Well, I

David: think that's going to wind up being a legislative question more than a technical question.

Sylvestre: Well, we are back to the web extension point, right? You have web extension doing that for you if you want. Yeah, there you go. There you go. It's a tricky. It's a tricky thing to do that one.

David: Yeah Continuing on the security questions, though. How How does mozilla which I mean, I know you've got the security bug bounty program Maybe you want to talk a little bit about that But there was actually a news article in the last week or so about rom com out of russia that took advantage Of a zero day in firefox that had actually been patched two months ago So obviously Some of the issue is end users not updating.

So I guess I have two questions that you can answer however you feel. Number one, how, how are you guys staying ahead of those evolving threats and issues in code base? And then two, we just talked about, you know, prompts to the user and stuff. Other than the automated updating, which is built in, unless you build it.

Disable it. Are there any other prompts or things that you're pushing the user saying? Hey, you really need to update

Sylvestre: Well as brian said earlier You have to be careful what you are going to prompt to the user because it can be an annoyance very quickly So you have to be very careful. So we do everything that we can to to help the user to update So for example, you will find firefox on the windows store we have a Debian repo an official one where our binary is going to update the user on every platform on Google Play.

The platform is going to update you. So we are trying to update the user as fast as we can, but there is so much we can do. So what we are doing in terms of security, we are, we are doing a lot of things. So we, the bounty program, as you mentioned, is one of the best way. One of the silver bullet that we have been using for a long time that if one day you want to talk about that into details it's an amazing subjects of fuzzing part of what it takes to fuzz a web browser.

So fuzzing is a technology where you send some correct or incorrectly formatted data into the various endpoints. So sometimes it can be just. It's by spider monkey. So JavaScript, but sometimes it's going to be the API or the IPC or Firefox. So you send that and you see what happens and you manage your crashes.

It is one of the silver bullets that web browser have been using for more than 10 years now And we are using static analysis. We are also experimenting with LLM. Like if an LLM can help us identifying some security issue, but we are investing a lot of money to keep our users safe.

Jonathan: Yeah. I think I'm going to jump in right there for a second.

I think one of the most interesting kind of frontiers and security research right now is essentially LLM guided fuzzing. I think that is extremely fascinating, because, and I'm curious about this, whether you've had this problem as well, we've heard from like the Curl project, where they've had people that have asked the LLM, you know, ChatGPT, find me a vulnerability in Curl, because Curl has a bug bounty and some people like I make money by getting chat GPT to find me bug bounties Well, and of course the person doing it doesn't actually understand what they're doing And so they say, you know, find me a vulnerability and chat GPT will happily hallucinate a vulnerability for them And so I am curious whether the bug bounty project at Firefox has seen some of that But there's been an uptick in you know, sort of fake fake vulnerabilities.

And then on the other side, is that something that you're, you're looking into is this idea of LLM guided fuzzing, because I think that is absolutely fascinating.

Sylvestre: So we do monitor those bugs. I don't know the percentage of of those kind of bug report, but we have a few people that are working on, on those kind of bug report full time. So, but given the complexity and the investment that we have done in security, there is, there are no longer, no hindering foods anymore.

The bugs are usually quite complex when you want to find one. Then for fuzzing, we are investigating because we know that it is the next big thing, and we know that the attackers are listening to your podcast and they know now that it is one of the, one of the tricks to find exploits. No, more seriously, it's obvious that it is a big deal.

I guess currently with Firefox, given the scale, it would be hard. But if you know how it works very well, you can probably find a few. And to be honest, we found a few security issues playing with LLM in Firefox. Sure.

Brian: Yeah, and I think there's a lot of interesting stuff you could do by hooking this technology up to a browser, an automated browser.

You know, when you think about looking for compatibility issues, you know, a site that says, oops, we don't support Firefox, you know. Is there some kind of vision analysis? Could you even for more complex workflows? Like you get a bug that says, here's the steps to reproduce. You know, if somebody does a really nice job filing the bug and they give you a very detailed bullet list of exactly what to do, can you sort of wire up a tool to help you know, validate, capture a performance profile, these sorts of things.

I think there's a lot of things to still explore in that, in that space, in addition to security.

Jonathan: Yeah, yeah, very much. So so I, I have said, I have said before, and I will continue to say that I am eagerly looking forward to the flash in the pan of AI going away because like with every other sort of bubble kind of technology, it's only when the bubble bursts that you really start to see the usefulness of the thing, right?

So like you have the debt, the. com bubble. Which when it burst the internet did not go away It's just people started using it more as a real tool rather than doing dumb things like renaming their city city. com and I I see something very very very similar happening with ai it is it is going to stick around We're going to use it for things but you know, hopefully at some point every company will will get the It will get the pat point past the point where every company feels like they need to make AI do everything for them.

And at that point, we get to see where it's, where it's going to be really useful.

Brian: Yeah, I felt I think the local translation thing changed my mentality a little bit on it. Like, I think I was pretty default skeptical. And when I saw that we with an example, I, I remember having a conversation with the accessibility team at Mozilla, maybe five years ago about the idea of doing auto image captioning.

So if you're a screen reader user, you have a vision impairment. If you're building a website, you're supposed to put alternative text on an image. So this is in the HTML tag, you write alt equals and then you describe what it is. In case the image doesn't load or if the user can't see it. But the problem is most authors don't do that.

And so most images don't have it. And so if you're using a screen reader, You're reading the paragraph, you're reading it, and then you get to the image, and it's like, oh, there's an image there, and then you keep going, which sucks.

Jonathan: And

Brian: so, we had looked into, some browsers had started to explore doing, using a cloud service to sort of send the image up, describe it, and send the text back down, which, it was not something that we were, you know, considering or wanting to do, we wanted to do it locally.

And I spoke with the team at the time, and it was like, no, the stuff is, It's just too bad. Like it's just not helpful. It's worse. It's kind of harmful for users because the descriptions are bad. It requires the super advanced hardware to do effectively. Maybe that was maybe five years ago. And then when we looked at it a year ago, it was like, Oh, my God, you can actually do this now.

And, you know, we want sort of controls around it. Like the user should be informed that this is machine generated, and they should opt into it. But if they do, it's actually a huge service to users, people who are using screen readers and like we've, we've sort of taken the steps to generalize the translations platform, basically.

Into more of a general inference platform within Firefox and started to dip our toes into that image captioning task a bit starting with the PDF editor where it's more like an authoring tool right now, but as we sort of validate the performance and quality of those captions, we're hoping to bring that into the main content for first accessibility tooling.

And so those types of things where it's a very clear user value to the feature, we can do it sort of locally performantly. It's to me, it's just like a no brainer from a product standpoint because it's a good feature. You know, regardless of what tech is sort of underneath it.

Jonathan: Yeah.

Sylvestre: And I would add to that we talked about fancy brand new technology with Rust and WebAssembly, and now I'm going to talk about all the boring tech with PDF.

We in Firefox are not showing off. We have the best PDF reader in Firefox currently. You can do editing of pages and it is fully written in JavaScript. We had only a couple security issue over the last 15 years with it. And if you look at our competitor, It sounds right. And in PDF, one of the cool things that we have been working on is you know, when you, when you, when you have a form and you need to put your signature, you can do it with your touchpad or your mouse, but it's not very good.

And usually what you want to do is you take a picture of your written signature with a pen and But it's hard to do, you know doing image recognition and to, to do the background removal and with with a simple model with LLM with a simple model with AI inside the browser, you can easily.

Identify the signature and convert that into an SVG locally without sending your signature to third party and then you will have a very nice signature That you can insert into your tax form for example, and it is that kind of of of boring use case I would say that that is going to stay after the AI bubble, I think.

And you have plenty of those cases. Summarizing a PDF, for example.

Jonathan: That's actually a very good way to put that. It's the boring use cases that are probably going to stick around. That is very apt. One of the other things that I find so fascinating, this goes back to the security angle, that I find so fascinating about AI is that in some cases it just approaches problems differently than a human would.

And so, you know, if you were to tell it you know, This is maybe a little bit, a little bit more advanced of a, of an example than what we can do right now with AI, but it's, it's close. It's not too far off. So you could say, look at the Firefox source code or, you know, maybe one particular, a function even.

Okay. So if you've got a AI that can pull something from the internet, which some of them can now, you could say, well, look at this function, find me a security vulnerability here, and then write, you know, a, a text input that would trigger it. All right, so this gets back to the idea of doing fuzzing from LLM.

Sometimes, the sorts of vulnerabilities, and even if it's a hallucination, right, even if it doesn't exist, sometimes the sorts of things that an LLM will find is very different from what a human would find, which in and of itself is interesting. And so that may not be quite boring, but I think one of the things we're also going to see is the times where that's useful.

And again, security research is one of those times. It's very, very useful that an LLM thinks, heavy air quotes around that, thinks, it thinks about things differently than the human would. And in some cases, it's an advantage.

Sylvestre: Yeah. And sometimes it is connecting dots that a human would not connect. So for example, it's not exactly fuzzing, but One of the, so we have a partnership with Ubisoft, you know, the video game company, Assassin's Creed and so on.

And one of the experiments that we have been doing with them is using an LLM at video phase. So you send a patch, you say, I'm a Firefox developer. And and I, here is a patch and please tell me how you can improve it. And you have the the LLM, who which is going to tell you, okay, you, you should refactor that one or that code may be duplicated by this one and so on.

So you can do some very cool stuff. And that as a human, you wouldn't think, or a very good reviewer would find it, but it's not always easy to identify what kind of improvement you could do in a patch when you are a reviewer. So I'll ping the reviewer. Not replacing it.

Jonathan: Yeah. Yeah. And so along that same line, something else that comes to mind is you were talking about the alt text for images.

I could see it being useful. And, and I say this sometimes people use the alt text to hide jokes, like XKCD really comes to mind. The alt text for XKCD is. Hilarious, but not actually usually very helpful to understand what the picture is. And so there's this, there's this use case that comes to mind that it's like, even if there is an alt text for an image, you might also want the AI generated one because it might, well, for one thing, it's going to be more useful than the joke that's hidden in the alt text.

But it also might turn out to be helpful in giving additional context as to what an image is. And so there, again, it kind of, it kind of taps into that idea of, well, it, an AI might approach this a little bit differently than a human would, and that can be useful. It's fascinating stuff there.

David: Yeah.

Because as a web developer, I've even seen cases where, you know, you're trying to pass your W3C checker. So you get alt equals image.

Brian: Yeah, exactly. Yeah, yeah, the analysis, we should make sure on some of the analysis, because the analysis I've seen is like empty alt text. We should, we should also make sure to have alt equals image in that list.

Yeah,

Jonathan: yeah.

Brian: The tricky part, you know what would be, that's such an interesting idea. I'm trying to think the user experience of that is like, Maybe, maybe there's a keyboard shortcut when you're hovering or something like this. I've always found picking a keyboard shortcut in a browser is one of the hardest hardest problems.

Jonathan: Yeah, yeah. All right. So let's, let's talk for a bit about about maybe privacy. I don't think we've, we've dug into this one yet. What are, what are some of the big privacy wins that somebody has using Firefox? And I'm trying to remember there was a there was a project that was being pushed.

I think it was at Firefox to do cookie isolation. Was that a Firefox project? What, where, where's that at as well?

Brian: Yeah. So there's a, there's a bunch of privacy features you get when you use Firefox. We had, you do get cookie isolation. There's a feature called total cookie protection that does the sort of you know, domain specific isolation of cookies and other storage. There's tracking protection features, which you can turn on either to different levels and settings, or you get by default in the private browsing mode that goes further and actually blocks or shims, even tracking scripts.

This again gets into the area of like reaching into content gets complicated area because, you know, some sites might rely on a tracking scripts call back to, to operate the page. And so we have all sorts of kind of interventions to work around that. And then we have fingerprinting protections. So fingerprinting is when.

A shared script pulls sort of identifying information off of ordinary web APIs to try to build a profile of you without relying on storage. And so that's also a, you know, feature set that, that we provide as well as, you know, the extension the extension ecosystem. Sylvester, I'm probably missing some some other things, but that's some of the highlights.

Sylvestre: One of the things that I love, which is technology, very technologically, very interesting. It's DNS over HTTPS. It's one of the cool features. So, you know, DNS is a protocol that it is very easy to spy on and your ISP is often doing it. And with DNS over HTTPS, you, everything is secure. So it's one of the cool things that your ISP cannot see where you are going.

Yeah,

Jonathan: I remember that. I remember that being a strangely controversial feature when it was, when it was first rolled out. And yeah, there was, there was some fights over that about the way, the right way to do it and who do you want to use as the end point and you know, I, I imagine some of that was that ISPs were gonna, gonna lose their way to look into what people were doing.

Sylvestre: Also to block you, right? In some country like mine you can do some blocking at the DNS level,

Jonathan: for example. Yeah, yeah. True. Alright, so I know we are, we're actually getting a little bit towards the end of our time, and I wanted to make sure and get this question, and it's one of the ones that you guys you guys included in the rundown, and that is what's, what's some of the fun stuff?

What's, what's something you're, you know, surprising or that you're particularly proud of? Something unique, a story from the history of Firefox, maybe? Let's start with Sylvester. We'll go from left to right on my radio dial.

Sylvestre: Well what I, what I love at Mozilla is I, I know that it can sound cheesy, but it's the, the expertise of the colleagues and the impact that we have on the industry.

We we We are always the underdog, as we discussed earlier. We are tiny compared to our competitors, but we still have a huge impact on the web. And what I see often and some we cannot communicate is how behind closed doors, sometimes we influence the web standards. So seeing that every day, how Mozilla is impacting the future of the web.

It's it's part of the fun of working on that project. And I see that almost every day. And as an anecdote myself, I as a kid, I the fun stuff is working on a project that I, I have seen Netscape and I think it was Mosaic when I was 13 and working on a project like 30 years later on a project that I've seen when I was a kid and you know, it was back in the day when you had one, one frame for me.

per minute seeing a guy in California surfing. And now you have a HD quality on, on the browser. It's to me, one of the fun stuff working on that project and seeing the web waiting for a while.

Jonathan: Yeah. You know, that's to, to, to take a brief detour. That's actually one of the interesting things I'm sure about this is Chromium is.

a competitor, but they're also very much, I'm sure an ally in some ways in making some of these things work. So they're, they're a, a frenemy, I suppose. It's got to be an interesting relationship.

Sylvestre: Well, we collaborate with our competitors in the standard buddies and and many times we, we collaborate with them to tell them, so look, this one is a privacy concern, but they are not going to sign up on the new feature because we have privacy concern.

With that feature, and we collaborate to make the standard better in those cases with them. So we give feedback, we make improvements, suggestions, and so on.

Jonathan: There, there was a Oh, this has been a long time ago. One of the, one of the browser teams always would like, send a cake. To the other browsers, whenever there were big releases.

Was that, was that Chrome was sending that or was that Firefox? Oh, we all do that together. We all do that. Okay. It's still happening. Yeah. One of the, one of the really fun stories I remember from that. So the browser versioning numbers have changed, right? It used to be the semantic versioning where, you know, when, when we, so for example, in Firefox, when we rolled over from three to four, that was a major change.

And then, you know, eventually got from four to five. So there would be like. 3. 0. 1. buildnumber and 3. 0. 2. buildnumber. And then, you know, it eventually has gotten to the point, I think most of the browsers are on board with this now where the, the, the major number is not nearly as important. And so now, you know, we're up to a hundred and something or 200 and something on the version numbers.

And one of the, one of the things that I found so funny there, there, there was, I think it was the Chrome team, the Chromium team, they would send a cake for The major versions and then when it was it was either edge or or I don't remember Firefox. I'll have to go I'll have to go and google this story instead of sending a cake when the versioning numbers switched It was a cupcake because it's like it's not nearly as big a deal as it was but we still want to be nice about it

Lots of lots of fun little stuff like that in there All right. So let's let's ask that same question. To To Brian, like, is there, is there a fun story or something you're particularly proud of? What's the fun stuff that you want to talk about?

Brian: Yeah, I was just thinking earlier in the conversation, I was, I was joking about the keyboard shortcut thing.

This is sort of a funny, but small thing is I was, when I started Mozilla, I was working on the developer tool. So this is the little toolbox that you use if you're a web developer debug your. Your styles and, and, and stuff on your page. Well, well, on desktop, I mentioned it, it is the a web app, basically the browser.

Jonathan: And

Brian: so we have a thing called the browser toolbox, which is the dev tools, but it's sort of popped out in a second window and it's pointing at the browser itself over sort of a remote protocol. And I had joined the team when we were kind of standing all this infrastructure up and I had to you know, come up with a keyboard shortcut for this thing.

And it took me, I was like, okay, well, you know, I think control I that's taken cause that's page info controls shift. I that's taken cause that's dev tools. I ended up landing on control alt shift. I to open the best, the best that I could do. And so but, but I guess, you know, it's been a privilege, I think, to work on the, on the web platform and on a, you know, mainstream production browser that has so many people using it, I think, especially doing it and sort of You know, I came in as a, as a web developer and getting to sort of build that platform and use it also within the browser and sort of co develop features.

You know, migrating stuff in place, some of the really old stuff, migrating it to modern standards and just sort of. I don't know. I just say the, the sort of grind, like every day, make a little bit of improvement, make it a bit faster, make it a bit better, a bit, a bit cleaner. It just sort of watching that compound, even over the time that I've been at Mozilla into the, the, the.

Quality of the product it is today is just really it's just really satisfying and to work with, with great people who are smarter than me in different areas and know there's, there's stuff so deeply, it's just been really a great a great. Yeah.

David: I have one final question as we get to wrapping this up and I promise it's not a gotcha question, but it is a personal one, and I have this platform. So several years ago, Firefox removed PWA support, or specifically the single site browser, where you can make a, Web page act like an app and that is something as a full stack developer I actually use and that's the one thing that I can't implement in firefox Is there any hope of getting that back?

Brian: Let me so the single site browser, yeah, there was like a prototype of doing that to be honest, I don't remember there was just some issues basically with it from like a like it's quite hard to You know to manage kind of the context of this is, do you carry the cookies over? Do you not? So on and so forth.

However, I will say we have started to look at at something like this again. I'll have to find you the details sort of afterwards, but I think we found, we think we found an affordance that we're maybe more happy with. And so I'll have to pull it up. I don't have it handy, but I think it's an area that that, that would be, Worth looking into I'd say it's probably hard in different platforms to do in different ways, but appreciate the The the feature request on that I think sylvester.

I don't know if you have any context on that one

Sylvestre: but we I recommend that you look into connect. media. org. It's a platform where we are receiving feedback from the user and maybe that one is already there and We'd be happy to discuss internally about those kind of things and we use that a lot like We we are not in our in an ivory tower We look at what people are writing on ICONews.

Sometimes we are crying and some, we look at Reddit and we look at connect. media. org and discourse. We, we read a lot what people have been telling us and writing about us.

Jonathan: That's gotta, it's gotta be a, it's gotta be a trip. Right on being on such a huge project that like everybody in the world knows about Like i'm i'm involved with a big project and we have you know We measure our users in the thousands Maybe a million users if we're feeling real, you know, if we're feeling real bold We'll say we might have a million users you guys are just you At least a probably a billion people recognize the name of the project right like it's just insane That's that's got to be a trip to be that big of a project.

Sylvestre: Oh, yeah. It's fascinating. It's also funny to see how people can perceive the company. And and sometimes reading some of the comments and, you know, we don't, sometimes we take the time to answer some of the comments, except when they are very snarky. And knowing how it works behind a closed door, or even behind open doors, because most of the stuff that we do is public, is is quite interesting.

Yeah. Yeah. It's fascinating at times. Yeah. So that Brian has some anecdotes also. Maybe we shortcut.

Brian: You know, I, I'm trying to, you know, I think trying to change and remove features can be really challenging in software that people this won't be a surprise to the audience, I think, but in software that's sort of widely used and people learn to rely on quirks or bugs, or just ways things work removing or changing features just gets really, you Complicated, I think, and so that's something that we, I don't know that there's like a perfect solution for that.

We try our best with sort of experimental infrastructure and communicating and things like that. But you're definitely end up sort of building on functionality. I remember removing. Some really kind of obscure air console UI. Once we built the same functionality into DevTools and just getting like roasted on on Bugzilla or some other, some other forum.

And I was just thinking these are the first time I was pretty new. And, and I just thought, Oh, I just like made a huge mistake on this. You know, I have to go and undo this. And there, and somebody was like, No, no, it's okay. Like this, this does you know, we have the functionality, we can take bug reports and we can improve any, any gaps that are missing, but it, you know, it can be a bit of a challenge.

Jonathan: Yeah. I kind of wanted to ask a follow up on that. And it's like, how do you keep your sanity? If you, if you pay attention to places like, like Reddit and you know, all of the other, all of the other places where people give feedback or talk about your project, how, how do you handle that? It's got to be a challenge sometimes, right?

Sylvestre: I can start. I am a Debian developer, so I have a good training with that. I have been contributing to Debian for like 20 years, so I have a tough skin now. So I learned, I, I'm, I'm I'm always taking feedback with a grain of salt. I know that some people wouldn't talk to me face to face the same way or in the cold.

So I'm always I have a tough skin, but, but what Brian said is we are very careful with newcomers. I had many conversations with engineers who have been reading Reddit or Hacker News, and the feedback that we receive sometimes can be very harsh, so we always work with with with the younger engineers who are not exposed to that, or who don't have a very good experience, and we are telling them, don't worry, that's fine.

If if you make a mistake, What is cool with code is that it's very easy to weave out. We can make changes. We'll ship every a major release every two, every four weeks, a dot release every two weeks. So it's not a big deal, like, at the end. Yeah. And we have plenty of CI. Yeah. So CI is very good to cast those errors.

Jonathan: Yeah, I guess, I guess with that, with dealing with feedback, maybe the, maybe the secret is to realize that the, the people that are talking to you on the internet, they're, they're not talking to you as people, they see you as just, you know, Mozilla, the, the company. And so you can't take any of that feedback personally.

Brian: Yeah. And I mean, I think it's often, you know, people are doing that because they care about the project and they care about their, their app. And so there's, there's some sort of kernel of you know, As I said, it's, it's sort of, it's a good thing to work on something that people care about. I think you do have to learn certain tactics to have the humor and thick skin and stuff to sort of navigate some of that.

But yeah, for the most part, if you look at like connected stuff, it's really funny. People are, people are you know, contributing, like it's for people, especially if somebody is not going to show up and sort of write a patch, but they have, they really care if they want this feature to work or that add on to work and they're sort of contributing.

It's a way to contribute to the project.

Jonathan: Yeah. Very good. All right. Normally at this time, I would ask you guys what we didn't cover that you wanted to, but I, I sort of feel like we have just barely scratched the surface. In fact, pretty much as soon as this call is done, I'm going to start negotiations, see if I can have you guys back because I feel like there's legitimately a lot that we did not get to, and I would like to, because it's, it's a dare and it's super interesting.

So instead I will ask you both this, and that is what what's your favorite text editor and scripting language? And again, we'll start with Sylvester.

Sylvestre: Well I am, I'm biased, I'm bash because of the coroutines, you know why? So that's why I have been contributing to the Rust implementation. And as a deleter, it really depends.

I think VS code is getting better. Better and better. I on my 20 years ago, I wouldn't have said that I was going to use a Microsoft editor, but here I am. Thank you,

Brian: Brian. Yeah, for me, I think for scripting language, it's gotta be JavaScript. That's what I've, that's my sort of go to language. I've been really.

I'm pretty excited lately with some of the new runtimes and technologies. I use the Dino runtime has sort of a really good CLI and system utilities and stuff. So if I need to turn through some files or glue some stuff together, that's my. That's my go to. And then editor. Yeah, I'll use Vim if I'm in the, if I'm in the shell, but VSCode usually for for editing code.

Jonathan: Yeah, understandable. And it's amazing how many people have basically come to that point where it's like, VSCode is just so good. It's what we're using.

Brian: I've got to, I've got to ask what about you guys on your on your answers to those questions? Maybe your audience already knows these, but

Jonathan: They might.

So, my scripting language of choice it depends very much on what I'm working on. The last few times I've had to do shell scripting, I've actually used something called Amber, which compiles down to bash code. We actually had the author on a few weeks ago in Floss Weekly. And then for text editor, my default in the command line is actually nano.

And I think that's because I actually got started on Microsoft products way back in the day and so, like, my first programming was in QBasic. And Nano is sort of familiar. It looks very much like the old Microsoft editor in the command line. So it's just where I feel at home. So those are my two.

David, what would you say?

David: I've actually never answered this question. Python is my scripting language. If I'm on the command line, remote is a server or something. It's a Vim. And my IDE of choice is JetBrains.

Jonathan: That's kind of unusual, though, for a Python developer to be a JetBrains user, isn't it?

High charm! I mean, I know it exists, I know it does, but it's just an odd combination. Alright, well, Sylvester and Brian, both, thank you so much for being here, it was an absolute pleasure. And, like I said, I feel like we very much just scratched the surface of all the things we talk about with Firefox and Mozilla.

All right. Thanks for having us. Yeah. Yeah. Glad to have you both. Okay. What what do you think,

David: David? Oh, awesome. I completely agree. And if I can swing it amongst all the wonderful co host at Floss Weekly has, I would love to come back as well. And I promise no more gotcha questions. That wasn't too bad of a gotcha question.

We've done worse on this show, but yeah, I mean, As a full stack developer having variety in the ecosystem is important. Gecko's always been a little bit of an underdog, maybe a large underdog at times. But just their support for open source for the web ecosystem, everything is just. Second to none in my personal humble opinion and just getting to talk about it is awesome.

Jonathan: Yeah, yeah, absolutely. I am, I'm actually super excited for Firefox on the, on mobile now on Android. I did not know that they had extension support there and that, that is legitimately really cool. I am really excited about that. I think, I think Firefox just became my mobile browser of choice. Which that's, that's saying something, but I think they have a convert.

So I will definitely be giving that a try. So next week on the show, we've actually got something super interesting that I am excited about, and that is the open source AI definition. We're going to have Simon and someone else from from the From OSI here. We're going to talk about that and get into all of that, which that is that is very much the cutting edge of what's going on right now with open source and AI and very much looking forward to that conversation.

Is there anything, David, that you want to plug before we let folks go?

David: Not specifically I would encourage everyone to check out the Twit Network if you don't already. We, I get to hang out with the Untitled Linux Show, the ULS guys from time to time. And there's lots of great news and tech stuff over there.

So I would say that.

Jonathan: Yeah, thank you so much for being here. I appreciate it. All right, and as far as plugs that I have, well of course there's Hackaday. We appreciate them being the home of Floss Weekly. Some fun, some fun stuff going on in the Floss Weekly world as well. Looks like we're gonna be able to get the, the old Twitter slash X account back.

And so you can follow there. That is X. com that feels weird to say x. com slash Floss Weekly. And hopefully by the time you hear this, that will be back back in our hands. Appreciate Twit working to get that back to us as well. And then, yeah, you can follow my work at Hackaday. You can follow me over at still at Twit, the Untitled Linux show.

I appreciate all those that do. And just want to say thank you to our listeners and our watchers. We appreciate it to those that get us live and on the download, and we will see you next week on Floss Weekly.

Episode 811 transcript

27 november 2024 | min

FLOSS-811

Jonathan: Hey folks, this week we talked with Lars Wickman about Elixir, a modern take on Erlang, and NURVs, a really clever way to run it with Linux on embedded devices. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 811, recorded Tuesday, November 26th. NURVs, real embedded Linux. It's time for Floss Weekly.

It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something real fun this week. I've got Lars Wickman with me, and we're going to talk about Elixir and nerves and running things on embedded devices and the Raspberry Pi. It's going to be a lot of fun.

It is just me alone in the hot seat this week, and so I'm, I'm just going to go ahead and bring Lars right on. And first off, welcome to the show, sir. Thank you very much. Yes, I appreciate you kind of stepping in, not exactly last minute, but a little late and we didn't have anybody. And so we, we avoided doing yet another round table discussion or, Oh, God forbid, even worse me doing a monologue for an entire hour.

That probably wouldn't go well. So thank you for being here. Oh, my pleasure. So let's see, where do we, where do we start? You, you are, you're responsible for something called NERVS, which I don't know what that is, I know a little bit, but I don't know what that is. And I

Lars: refuse to take responsibility, but I can't tell you who is responsible.

Okay, well, you're,

Jonathan: you're involved with something called NERVS, which is based on something called Elixir, which is maybe based on something called Erlang. And I, I sort of know what Erlang is. And so maybe let's, let's start with what, what is this elixir thing and why should people care about it?

Lars: Yeah.

So starting with Erlang is not a bad idea. So 40 years ago, the Swedish forests of Stockholm, no, I'm not going to do full story time, but Erlang was created. in Sweden by the telecom company Ericsson, who people might know from like Sony Ericsson and a few Ericsson phones. But they're a massive company that does a lot of telecom equipment, 3G, 4G, 5G, all the Gs.

No longer 2G, I think, I don't think anyone does that anymore. But back in the 80s, they were developing telecom switches. And they wanted to experiment with whether they could develop a language for Developing resilient, reliable, performant telecom equipment because it's, it was a really hard problem at the time the computers were not that fast and they needed to have sort of massive concurrency.

So many phone calls at a time. Yeah. One phone call having an oopsie should not bring down a thousand other phone calls. So a sort of error error handling and resilience. They also ship these sort of resilient redundant boxes that were physically connected. It's not like, Oh, they were network.

No, no. They, they shared a backplane essentially. So they needed to actually run it as clustered devices.

Jonathan: Yeah, so back in the 80s, telecom equipment was basically big iron, right? That's kind of the era still that we're talking about now.

Lars: It's the era of, okay, we're going to put this under, we're going to put this in the ground before we put a parking garage on it.

Jonathan: Yes, I've been in those rooms a time or two, yes. Yeah.

Lars: They also did not want to have to remove the parking garage or even go to the parking garage to update the software. Yeah. So Erlang has supported concurrency distributed computing like fault isolation or reliability, consistently low latency is kind of important for, for telecom and hot code updates since somewhere around the 80s.

It's almost 40 years old now. Wow. I just. recently interviewed on Beam Radio one of the developers that have been maintaining the virtual machine for Erlang. And he's just been doing it for like 27 years now. He didn't make it. He's just the guy who maintained

Jonathan: it. He's a real noob in the field at that point.

Lars: Yeah. So, so Erlang has been around for a long time. Yeah. It's what Like RabbitMQ was built on Erlang. WhatsApp was built on Erlang famously. So they had like 10 years of uptime before they actually had to reboot all their servers for the first time. They did hot code updates. They did all the fancy things and they had very few developers, but a very performant, massive system.

And then they sold for a lot of money apparently, but Erlang has always been sort of a bit of an open secret. It's like only deep nerd wizards. known to use it or, and sort of fought through the documentation to use it. It has not been, it has not been focused on adoption. Not in, in the way that modern languages are.

And at some point Ruby showed up like Ruby, Ruby on Rails. And in that community, Was a guy called Joseph Lim. Well called it's his name. And he, he built up a decent following in Ruby. He did a lot of cool libraries, a lot of useful stuff for the community, very active. And then he got a little bit sick of Ruby being very slow and sort of not great operationally and started exploring other things.

options. And he settled on Erlang, and then he built Elixir on top of the Erlang virtual machine. So it is a separate language, compiles to the same thing.

Jonathan: Ah, okay. And Elixir

Lars: is a modern ecosystem with modern tooling, good build tools, good dependency management, a nice web framework, database layers, all the niceties built out and built up over the last 10 years.

So it's new in the sense that Just 10 years old.

Jonathan: Yeah. And so the original Erlang is, it sort of reminds me of Java and in that it's got, it's based on a virtual machine and it's got some of these, you know, really impressive features under the hood. And so would that make a Elixir kind of like the equivalent to I know there's, I guess Kotlin is one of them, right?

Languages that are compiled to the same kind of bytecode that runs on that existing VM.

Lars: I would say it's kind of the a closure or similar? Well, Erlang and Elixir are very equivalent languages. Okay. With Java and the JVM, if you look at like, I guess Kotlin is a fair comparison in that Kotlin is not a very different language from language from Java, aside from trying to modernize and make things nicer.

Erlang is a functional programming language, so it's functional programming, but it's the functional programming because they wanted the massive concurrency and like, so they, they do, they spawn lightweight processes and communicate via message passing, a bunch of that stuff. And then it really helps if your data is immutable.

So. They chose, they ended up doing functional programming as a consequence of what they were trying to solve for. So Elixir is also a functional programming language. Where I would say the JVM is a fairly general purpose virtual machine as far as I know. I don't spend a lot of time in, with it. I mean, object oriented programming is a very general, Right.

Paradigm. Right. It's not specific in any real sense, but the Erlang virtual machine has, has these primitives of like spawning processes and sending messages. Those are things inside of the machine code, well, the virtual machine's machine code. So it's like, it's a fairly high level of abstraction for, for a system language like that.

And Erlang was built. To build systems. It's like, yeah, you have an OS, but you want to find out if you have disk space. Well, you can just ask the airline system. You, you can stop and start parts of the system. You can inspect the running system at runtime. You can of course, update the system at runtime.

That's the whole hot code updates part. And it had preemptive multitasking from very early on. So it's, it's very much a language and a tool for building systems.

Jonathan: Yeah. And so is Elixir kind of focused on that same thing? And then I guess along with that Elixir Erlang, do they have object oriented stuff added onto them or are they strictly functional?

Lars: They, they are. The least strict, strictly functional language you can run into. So they are high level dynamic languages. So it's not static typing and it's not like Haskell or Elm. And some people are very upset about that. Some people are very happy about that. I come from like PHP and on into Python and then I went to Elixir.

So high level and dynamic has been my tool for the longest time. Sure. So didn't bother me. The FP part was kind of weird to get used to, but it also clarifies a lot of stuff. You don't get this very messy thing where you have classes and objects that interact in weird ways. It's like, no, no, the data you have in front of you is all the data you can deal with and have to deal with.

They're, they're not pure in the sense of the purest functional programming languages, because you have, you're allowed to do a bunch of side effects. You can do logging, you can do IO, you can. Do a bunch of things and you don't have to involve any monads at all. So,

Jonathan: okay. So what, what are people primarily using Elixir for these days?

What's the, what's kind of the problem set somebody today would have that you would say, Hey, Elixir would be great for that.

Lars: A lot of people pick it up to do something like messaging related, like chat or real timey stuff, because it is very good at. sort of soft real time. But it is also just generally good.

So Erlang was built to run services. Services turned out to be like the entire world. Yeah. It's like, okay. Yeah. Define services. But Elixir comparing Elixir to Erlang. Elixir has much more clearly built out like the web toolkit. So the Phoenix web framework, very nice to work with for building APIs and WebSockety stuff and all of that.

And then they built Phoenix live view, which is like, honestly, the best pitch for it is you probably don't have to write very much JavaScript.

Jonathan: I can see that being a winning pitch for, for some people at least.

Lars: Yeah. So it is a server side driven Interactive you, it's a interactive UI framework, but you drive your components and your templates and all that, and keep the state on the server. And you just ship optimized diffs over to, to update the browser side.

So there's, there's a whole thing for like. Very small teams can build fairly ambitious web applications using, using that tool set, but it also plugs together with all the other cool stuff like Erlang comes with clustering out of the box. So if you shove like Elixir onto fly. io that has this. Good private networking across geographic regions.

Like all your machines can just talk to each other and you can do RPC calls across the, across the cluster transparently, you can pass messages around the cluster transparently. It really brings a whole paradigm with it and often cuts out a lot of tools that you end up needing. So if you've done Python web development, for example, that there's like a counter That is just like, what's your time to Redis?

Like, how soon will you introduce Redis to cache something or do a janky queue of some sort, or will you bring in Celery for the queue? Celery has always scared me. It's so strange. But. But I don't need it in Elixir generally. And there are good like queuing libraries and all that tooling if you need it.

But for some very simple queuing needs, it's like, it's almost like you have a small microservice architecture inside of your application. Because you can do multiple things at once and they generally don't impact each other very negatively.

Jonathan: How, how big is the sort of the community, the user base for Elixir these days?

Is it still fairly niche?

Lars: It's very hard to say because, well, ElixirConf EU polled like, I think 800 people, total 650 or 700 in person this year in the EU. So it's not tiny but it's not massive. It's not KubeCon, like, but like, I've worked with Elixir professionally full time for six years now, I think self employed and I keep finding work.

I'm a fairly loud person in the Elixir ecosystem, which helps. Yeah, but yes that

Jonathan: does help a lot. What what is the this was this was not on the the list of topics that We were going to talk about ahead of time, but i'm very curious what's like the security status of it? Like is there are there bug bounties?

Are there people that are? Sort of looking specifically at the, the security of the system, finding vulnerabilities and fixing them. What, what does that look like?

Lars: So for Erlang I don't think there are any bounty programs, not that I've heard, but Erlang is maintained by Ericsson. It still is. Okay. Yes, and it is heavily used by WhatsApp.

Okay. And it is heavily used by Cisco. And also backs a lot of big game company tools. So you'll find Blizzard uses Erlang. League of Legends has used it. It's a bunch of Elixir discord runs on Elixir. So there are large companies that are using it and they take their security reasonably seriously.

Then there's the Erlang ecosystem foundation, which is kind of the umbrella foundation for, for the whole ecosystem. And they do have a security working group. So when things come up, like right now, there's a bunch of conversation around supply chain. Because EU legislation is going to force, force a lot of people's hands in that regard.

So there's a bit of that. And I think the EF recently turned into its own. So if you file a CVE for Elixir for, for Erlang, it's Ericsson but for, for Elixir and a few other things, it should now end up in, in the EF. So. The, the sort of underpinnings for the security work is there. I think it's still fairly early if you compare to the big ones, but Somebody is thinking about it,

Jonathan: at least.

Yeah, I wouldn't say

Lars: I know what the security story of Node. js is either frankly.

Jonathan: Oh yeah, they're, they're, they're working on it.

Lars: That doesn't sound good. Not reassuring at all. So, okay.

Jonathan: That's actually an interesting question because let's see. So does, does Elixir kind of have its own library repository?

So like when you, when you get it, when you start talking about the, and this is what I mean by that. When you start talking about the things, the, the the dependency tree for something like node js, you get into this, this. problem space where just anybody can throw a library up. And so one of the security problems that they have is people put malicious libraries on the you know, on that repository.

Is that a problem that exists in the Elixir space?

Lars: I have not seen or heard people note that there are like malicious libraries coming on to hex but hex. pm very nice repository has a lot of good things in it. Of course, like we can have the same problem there. There's nothing stopping people from publishing.

We don't have the culture of. Sort of proliferating packages in the way the node ecosystem is i've i've run into a package and gone. Huh. This is seven years without updates. I wonder if it still works Cause it seems like it might solve my problem and it just works. Like the, the Elixir language was announced to be generally done.

What was that? Like Elixir 1. 7 or something? It's a ton of versions back. It was probably like three or four years ago and it was, it was, Just met with a standing ovations because it's not a cultural or ecosystem that wants a ton of churn. There are libraries that are sort of leading edge, for example, live you changes fairly frequently because that's still heavily under development, but the language itself.

Changes very little and they have no plans for a 2. 0 because they don't see the need interesting.

Jonathan: So there's no You guys don't have the problem of left pad where you have a whole bunch of you know Libraries, that's just one line of javascript No, that's

Lars: fairly unusual. I think the the general cultural consensus is you can vendor that It's like come on.

Yeah, you just you just add that that's a snippet not a library But of course like there are small libraries I'm sure, but I don't know that they get a lot of use.

Jonathan: Yeah, yeah, that's fair. Okay, so that seems like a reasonable overview of Elixir. We could spend more time on it, I'm sure. But we have other things to talk about.

What, what is what's Nerves?

Lars: Yeah. So when I got into Elixir, I got in, like, I poked around with the web framework and thought, this is cool. But then I saw a talk by Frank Hunleth, who created NERVS. Okay. And he showed putting together some Elixir application and. Compiling it and then burning the firmware to an SD card in record time, because apparently it was like 16 megs instead of two gigs.

And then starting up the device and poking away at like the IAX prompt and running web apps or whatever he was doing, I don't recall the details. And then when he wanted to change something, he just recompiled it and then did. Mix upload and mixes the general tool runner for Elixir mix upload. And it would just shove the new firmware onto the device over that network and do an AB partition switch.

And that compare that to like working with a Raspberry Pi on Raspberry Pi OS, and you'll. either be installing a bunch of stuff inside of Raspberry Pi OS that you will never ever reproduce or you keep needing to re flash Raspberry Pi OS and that takes time. Yes. Yeah, I think the, the base image for nerves has been bloated to 30 megs now, but it's still seconds to burn to an SD card.

Jonathan: And so that's, that's what, the Linux kernel, enough of a minimal system to get you booted, and then the Elixir virtual machine?

Lars: Yeah, so that's fundamentally it. The Erlang runtime system and virtual machine gets started by a tool called Erlinit. Early net runs as process one, and then there, it churns through a few pits before it hits before it hits the Erlang virtual machine, but you don't have anything like system D under nerves because the Erlang the Erlang ecosystem was built to run services and orchestrate workloads.

Interesting. And. This, this is a hearsay, or I'm taking this as second hand information, but I hear that there are challenges running systemd on a read only root file system.

Jonathan: There, I know it's possible but I can imagine that it might not have been designed for that specifically. Sure. Yeah,

Lars: so NERVs by default ships a read only root file system and then a data partition, which means like you can have the really crappy SD cards and they still won't bail on you, or and you won't churn through your eMMC.

So a lot of people take nerves for being like a DIY hobbyist thing. And it's like, Oh, it's a cool thing. You can put nerves on your, and do elixir on your raspberry pie. It was never designed for that. It was designed for industrial and professional use right out the gate. Frank has worked. in embedded in various ways for a long, long time.

And he just found that like the Erlang philosophy of like, Oh, something is failing. Let's find a known good state to re to recover from.

Jonathan: Right.

Lars: Resonated very well with his experiences of trying to make embedded devices behave themselves.

Jonathan: Can, can you run Erlang and Elixir under systemd and, and, you know, more traditional Linux stack?

Yeah, yeah, yeah, yeah.

Lars: I mean, you can, you can put it on anything, essentially. People run it on FreeBSD, people run it on, MacOS, Windows, Linux, all the things. Windows too. That's impressive.

Jonathan: Okay, so it's just when you're using it for sort of this, this embedded or the system is only doing Erlang. That's when you have the, the Er and it start as a, as process one.

Lars: Yeah, yeah, yeah. So that's an ERVs thing. And there's a lot of niceties to that as well. I'm pretty sure, yeah. You can have the, There's a mechanism called the Erlang heart that can monitor your Erlang runtime and restart it if it fails. So it's essentially a watchdog process. And then you can integrate it.

Nervs integrates that with the hardware watchdog. So if either the hard watchdog fails or the Erlang runtime fails for any reason, it knows to restart the device. And this is like stand, kind of standard embedded stuff. If you've done embedded long enough to find out that it's standard. Cause a lot of people just Like, pick up Debian and ship it.

And go, Ha! Embedded device! And that will work until it doesn't.

Jonathan: Right. So, I have to take a moment and say, Isn't it just a crazy wild world where you can put a full Linux Debian install on something and call it an embedded device? Like, we're at the point where that's a thing. That's just, that's nuts.

That is the way it works, but that's just nuts at the same time. Yeah.

Lars: Yeah. Yeah. I mean, two gigs of embedded goodness. Honestly there, there are definitely embedded devices that end up going with like two or four gigs of payload, but that's because they pulled in CUDA. If you, if you buy one of the small Nvidia devices, I would still consider them embedded devices, but they need so much supporting code that it's ludicrous.

Jonathan: Yeah,

Lars: no, but nerves. is the entire concept. Like it's the, it is the system that helps you build your underlying Linux actually I had an interesting conversation with Frank at one point that clarified something for me. He doesn't particularly call NERVs a Linux. It's not like a Linux distribution.

It's not a tool for embedded Linux. He could, he considers. Linux and implementation detail in that regard. And I think if he had his had his way, he would have probably preferred FreeBSD, but Linux gives a lot more sort of support and tooling and like, you get a lot of stuff if you pick Linux. So I think he picked Linux for that reason.

But essentially, NURBS tries to paper over a lot of that. We use the network stack because the network stack is kind of proven. We use a lot of the kernel. facilities because they're useful and good, but You can run a lot of your device from Elixir land, which means you get this message passing. You get like various parts of your system like, Oh, we have this sensor and that needs to inform the backlight.

If you have an ambient light sensor, for example, and it needs to inform some part of the system to tell the backlight to actually bump it up a little bit, because we're in broad daylight, that type of internal communication needs to happen over something.

Jonathan: Yeah.

Lars: And Erlang already has message passing and pub sub and like broadcasting and all that stuff.

So it's very easy to build out this stuff with within an Erlang system. So everything that you can do within The Erlang system is very nice to deal with, but you can pop off like, Oh, we need to start a Python process here. We need to pull in all of like Podman and run a Docker container. Like it's doable.

But the more you can run in Elixir, the more niceties you get, essentially.

Jonathan: Yeah. So, well, this is interesting. Is this, what is, what is nerves primarily targeted at then? Is it for the raspberry pi or did that just kind of come about by accident?

Lars: If you look at the supported systems, it's going to look like it only supports the Raspberry Pi.

Okay. But that's because there are so many Raspberry Pis and nerves, like the base systems we use in nerves are very specific. That's why they can be small. And they, once you put one into production, you can get to know it and know what's in it and make it yours. So it's prefers explicit systems. So that's why we end up having the Raspberry Pi 0, 1, 2 0, 2, W.

Jonathan: I'm very familiar with this. 3a, 4

Lars: and 5. Yeah. And also I think we have kiosk systems that pack in like a whole browser shower thingy. Essentially small compositor and cog, which is a nice embeddable browser. And those are also for the Raspberry Pi, mostly as sort of getting started systems.

But then there's also the BeagleBone Black. We support a bunch of different systems. OSD32, MP1. I, there's some IMX board in there that I don't know off the top of my head. There are also non sort of non officially supported ones. I have a couple of devices from TI's like Sitara processors. AM62 series boards.

And there are contributors that have developed those for, for their clients and made them publicly available. So that's kind of base systems and the fun part is of course If you have an application and you want to put it on a bunch of various Raspberry Pis, if your application doesn't rely on anything too weird, you can just pop it on a variety of them.

Like the, the system part is pre built and separate from your, from your application. You just plop the application down on top.

Jonathan: It's probably the application doesn't necessarily care about even what, what architecture. So, like, one of the things I've run into with the Pi is that most of them are Arm V.

Eight, I think it is. The recent ones are, and some of the old, there's a V

Lars: seven in there somewhere. I think it's the zero in the old ones.

Jonathan: The, no, the zero is actually a V six. So like the, the, oh, yeah, yeah. So that, that tripped me up for a while. I was, I was shipping binaries that weren't working for anybody because we were compiling for armed V seven and those are actually armed V six and that is not the same thing.

Yeah. Yeah. I, I guess with, with the elixir and nerves, you don't, you don't have to care when you're writing an application because all of that gets handled on the, on the OS side.

Lars: Yeah, so like, I know in the embedded space, a lot of people are like, yeah, I'll, I'll write this and see why would I write it in anything else?

But actually I would beg people to consider the fact that there's a very good high level framework for writing C it's called Erlang. So Erlang interrupts very well with C and as a consequence Elixir interrupts very well with C and with SIG there's also good interrupt with Rust and so on as well.

Jonathan: By nature of Rust working so hard to have good interoperability with C I'm sure.

Lars: Yeah. You're catching me at a weird time when I'm very upset at a Rust based library that will not precomp cross compile cleanly. But aside from from the ONNX runtime that I'm having trouble with, generally the Rust libraries have been well behaved. Right, right.

Jonathan: So I, I, I want to know a couple of different things and it's related.

And so I'm curious what, what, like what real work people are doing with nerves. And then I'm also curious, is there a, is there kind of a place where it makes sense to run nerves on x86? And it, it kind of seems like there that you might even do like. VM based microservices, right? Where do you have a virtual machine?

That's just nerves with, you know, whatever elixir application on top of it. I'm curious if people are using that out in the real world.

Lars: I know some people have taken the so we have a generic x86 system in the nurse project that you usually Like you can put it on almost any computer and it will probably boot and work.

But you, if you want a bunch of hardware support, you'll have to adapt it. Sure. But someone took that and hacked on it to get it to run on Vultr, which is a virtual machine cloud provider. So they definitely run nerves based cloud cloud instances. I've been. Curious to experiment with it, but I haven't.

Also like there's a bunch of industrial PC type stuff where you could definitely want to put nerves as a single. So the point of nerves is essentially like it should not really be running your desktop or anything, right? It might, it might work out, but it's not what it's for. It's Single purpose devices is kind of the best explanation, but essentially it's like IOT, IOT gateways, kiosks, that type of thing, where it's like this device has one job.

It might may need to do a lot of different things to do that one job, but it has roughly speaking one job.

Jonathan: Yeah. So, okay. On the other end of that spectrum, then how, how embedded can it get? You know, does this thing run on ESP32s? Does it run on any of the STM devices?

Lars: So, hypothetically, you could get it to run on one or two of those devices just because people have made Linux run on them.

Jonathan: Okay.

Lars: But you shouldn't. Fundamentally. Generally, I would say they have they're too resource constrained. Like the Erlang VM is not terribly, heavy, but like Linux prefers an MMU and a microprocessor, not a microcontroller. And we, NURFS is a thing that runs on top of Linux as it stands.

Jonathan: Has anybody, has anybody tried, and this is nuts, but has anybody tried something crazy like running, running the Elixir, the, so the Erlang VM on like free RTOS?

Lars: So there is a project called Grisp that runs that runs the Erlang VM on Artemis. And then they coupled in the FreeBSD networking stack. It's a cool project, but I haven't done anything with it. It's, it's a wild wild German. I think it's a professor. Professor. But yeah wild German engineering.

Interesting stuff. Per Stritzinger. But then there is also actually a project called AtomVM, which is a re implementation of the The Erlang VM for tiny, tiny embedded devices. So they, they managed to fit it. If the device has 400 K of Ram and they need like two megs of space, I think, for, for a small image, and then they can do a lot more if you have more than that, it's a very impressive project.

They claim it's not production ready. I think they should should experiment more in that regard, but it's very cool. I saw a talk about it in Berlin this year and it's a very cool project. Yeah. Yeah. That's neat. Not. As battle tested and mature as the virtual machine, which has been tuned for like 30 years.

So that's the trade off. I, but I would say like nerves is not particularly heavy, right? So smart is a company I, I work fair bit with him, Frank Oleth, who made nerves. Mm-Hmm. works at Smart Rent and they make like smart thermostats and ho smart home stuff for rental. rental properties. And they are probably the big, as far as we know, they're the biggest user of Nervs.

So it's around a hundred thousand devices is what I've heard. And their most used device is a smart thermostat, LEDs and stuff. It doesn't have a touchscreen and it's a 32 bit single core. All the winter processor clocked down to 325 megahertz. I think it shipped 256 megs of Ram and the U they have like a hundred makes despair.

And I think the overall image is like 30 megs on disk. Yeah.

Jonathan: I am reminded of some work that I've done with OpenWrt, which is very similarly, you know, it's an embedded Linux system. They can get down to very, very small targets. You know, back at one time, they fully supported running on boards with 4 megs of flash.

And, you know, ideally, was it megs of RAM, something like that, but just insanely small targets. And so this, this. Brings to mind an obvious question. Is anybody making routers with NURBS? Is that is that even possible? I know

Lars: some people have definitely explored it. I know a guy who does wifi somewhere somewhere in Europe, Austria, I would hazard a guess, but I don't think he has switched over.

He's still do doing open WRT, I think. And, but we heard. Like, randomly in the Elixir forum, this is, you never find out who's using an embedded framework because, like, it's just secretive industries and not quite as open generally as, as like the web world or whatever. But we found out Southwest Airlines uses NERVs for their in the flight Wi Fi.

Jonathan: Oh, cool.

Lars: On a bunch of their flights. It's like okay had no idea but of course, it's perfectly suited for networking like cisco does a bunch of networking using airline Yeah, that makes sense. It makes sense.

Jonathan: I could I could even imagine honestly I could imagine a mashup project that takes like the open wrt kernel and puts the the nerves runtime So the the whole the whole user space would be nerves and then you'd use all the kernel work The open wrt guys have done and you could probably make a really impressive little embedded device with something like that You

Lars: Yeah, and the way we build our Linux is just Buildroot.

Okay, very similar then. So, very straightforward. I don't,

Jonathan: does OpenWrt use Buildroot? I, it's, it's similar. I think it's sort of their own take on Buildroot, but I'm sure there's some cross pollination that goes on there. So what what, what license is all of this under? So if obviously if people are doing secretive IOT stuff, it's not going to be a viral GPL sort of license, at least.

No, hopefully not. That would be problematic.

Lars: I think most of NURVs is under Apache, but it's like Apache or MIT or some BSD class it's all fairly permissive licenses. So no, no real concerns for doing commercial work. Right,

Jonathan: right, right. Yeah although boy, it's interesting, you know, I'm sure you know this and like a lot of our listeners do too, but the fact that that, that Linksys used Linux and a bunch of GPL stuff is the reason that we have OpenWrt and a lot of this, right, because Linksys built the, built a router and they didn't.

Put Linux on it. And some, some smart user figured it out and said, Hey, by the way, how about you release all the sources that you're supposed to under the GPL. And to their credit, Linksys did it. And then a group of hackers hacked on it and said, well, hey, why don't we just compile this ourselves? And that's where open wrt came from and a lot of this stuff came from that

Lars: and that was how you fixed your blue and black little bug like Linksys router.

I used dd wrt at the time. Yep Yeah, it definitely made it work. Yes. Yes compared to the links firmware

Jonathan: and all people did some crazy stuff with that too Like what what if we desoldered the leds and put a usb port there instead and just bit bang the usb? And oh, it was crazy people did some crazy stuff.

Very impressive. Okay. So now what about what about nerves hub? What is is that sort of a repository for for packages or what's up with nerves hub? You

Lars: Yeah. So nerves hub and here's like disclosure. I'm trying to start a business called nerves cloud hosting nerves hub. Okay. But but I'm also one of the maintainers of nerves hub and I'm in the nerves core team since a little while back.

But yeah, nerves hub is an over the air update service, completely open source project for firmware updates. So you essentially set up your device. We have a, like the lightweight developer offering of like, yeah, you can just set a shared key on your device and can connect, but the good way to do it is of course, with some, some little device certificate chip.

So. There's a good library for the attack 6 0 8, which is a nice little cheap chip that everyone has seen at this point. But you can also do it with a TP except it

Jonathan: doesn't support the 2 5, 5 1 9 curves. Sorry, I, I have been, I have been fighting. There are so many upsetting

Lars: things about that. I have fighting many upsetting exact

Jonathan: problem for days now, but anyway, continue.

I'm sorry.

Lars: Yeah, . But some, something it does do is that it will, do a TLS handshakes with a reasonably private device certificate, which means we can use that for authentication and letting, and then letting the device connect and we can know which device gets which firmware that type of thing.

So fundamentally it's like Nervs Hub is a whole suite of things. It's like, there's some CLI tools for when you built your firmware, if you want to sign it and upload it. You can only use signed firmware because like this, this is not hobbyist grade. It's production grade. It's being used with real devices.

And you actually want to be sure that someone that the right, someone is sending that firmware. Yes. And. All of this is built on top of a small tool that Frank built not in Elixir, and it can be used for, for many other things, but it's called fwup. And one of the cool things it can do is that it can stream writes from like one of these firmware files that it packages.

It can stream the writes essentially like DD would. But straight to the partition you want to update. So NERVs by default ships A and B partitions for switching back and forth and being able to revert a bad firmware. Yeah. Resiliency. It keeps coming

Jonathan: back to resiliency IoT, right? You, when you're doing IoT devices, you never want to have to tell somebody, All right, take the device apart.

Connect to the serial port and then, yeah, just, it's just like, yeah,

Lars: we're going to have to send a technician out to 10, 000 devices across the U S like, no, thanks. I also know a company that does nerves that have power meters across Africa. Getting to one of these devices might take a week. Yeah. Yeah.

Yeah. So important to be able to revert and also important to be able to operate under. Non ideal network conditions. And it's also kind of important to not use too much memory or too much disk space when you're dealing with IoT devices. So what FW up will do by default is it will stream the update.

and only use a fixed amount of memory and no disk while it is fetching, fetching the update. And it will just write them at the pace it can get from the network and like whatever throughput bottlenecks it might have. It will write that to the new partition. When it's done, it will do the switch over, but that means it can sip through a straw if that's what it has, or it can do.

Do it fairly quickly in many cases.

Jonathan: And I assume with, with checksumming and all that built in to make sure that you don't Yeah, yeah, yeah. Flash bad code.

Lars: Yeah. There's a bunch of Blake checksums in there and all that stuff. Yeah, so FW up is kind of the underpinning there. But then of course, Nerf's hub is built with Elixir because Elixir is very good at building scalable, fault tolerant web services.

So You want to run a couple of hundred thousand web sockets. That's not really a problem. And actually there was a blog post very early in the Lexer's history. Excuse me. Where Joseph Lim and Chris McCord, so Elixir's creator and Phoenix creator, dear, my throat is just seizing up. That's fine.

They wanted to see how far they could push the web socket implementation, Phoenix channels. They ran out of big beefy boxes to try it on. But essentially they, I think they scaled it on one machine to 2 million. Oh wow. Active chattering web socket connections. Wow. Without too much issue.

Jonathan: That's impressive.

Lars: And that's kind of the, the scale, like the scalability we're building on top of, and so there's web ui, there's ACL I, there's APIs, and then there's a lot of shipping firmware.

Jonathan: So Nerves hub, is it, is it available to hobbyists if I've only got a couple of devices. Yeah. Does it make sense for me to, to, to get on board too?

Lars: So you could set it up. It's a Docker container. It's very straightforward to set up. Okay. Or you could sign up for Nerves Cloud and use like our hobbyist plan, which is essentially 10 devices. Do whatever you. Because like the real customers in this case is not a hobbyist, it's, it's someone who has a couple of thousand devices.

At least, yeah.

Jonathan: I, I imagine that hobbyist here is also nice to be able to give those potential customers a way to do a test run. Yeah, for sure. Works out pretty well. Cool.

Lars: Yeah, so it's a, it's a clear differentiator. Like, There are many over the air update tools. There are also a lot of assorted, sort of, Embedded frameworks, but most of them don't come with any opinions.

It's like, if you lock, look at Yocto Linux, like what is the opinions of Yocto? It doesn't have any, it's supposed to be general. Buildroot is slightly more opinionated about how to design the whole thing, but it's fundamentally also very open ended. It's just like, here, we will help you get a Linux and some stuff on it.

But nerves is very much like a whole package. It's, yeah, it's actually a good starting point for embedded. I didn't come from embedded, but I'm learning a lot about how to do embedded devices based off of the fact that there's a lot of experience written into the framework and written into the tooling.

Jonathan: Right. So what does it look like if somebody wants to bring something from outside of the Elixir world onto a NERVs device? So the example that you give in our notes here is Python, but you know, there's a, there's even a world outside of Python that, you know, people have stuff that they want to do. So like, can I run a Java virtual machine alongside the Erlang virtual machine on a NERVs device?

You know, can I do Python? Can I do Perl, Flutter, you know, whatever, is there is it possible to run that stuff alongside NURBS?

Lars: Yeah, so you have a bunch of different And some things are easier to run than others. Like, I don't know how much work it would be to get a decent JVM into a Buildroot project.

I've never tried. I don't particularly want to try.

Jonathan: If somebody pays you enough money, I'm sure you'd be glad to, right? I would do it. Yeah. That's the way that works.

Lars: But there are also some blessed paths. Like if you want to do. If you want to integrate Rust, there's a few nice ways to do it. Sure. As I mentioned, C and C interop, pretty straightforward.

Then there's cursed things, like there's a community member called named Coco, and she has implemented an embedded Python, like you run Python as a NIF under Erlang. So essentially you have Python in your Erlang virtual machine. And that's cursed, but it also seems to work pretty well. It's very early days for that particular experiment, but you have a lot of options.

You can also just I mean, if you have Go, Go code, Go makes very nice little binaries. You plop that in just as a file and you can start it easy. But then of course, if you actually want a proper Python. In your system, Buildroot has that, like you can just, you'll have to modify your system usually, is the answer to running more complex things.

Jonathan: It sounds kind of like anything that you can do in Buildroot, you can actually make nerves do as well. Yeah, pretty much. And honestly, again, I have experience with OpenWrt's flavor of Buildroot, but I'm pretty sure it is similar. It's not that difficult to add stuff to Buildroot. You just, you know, you write, you go find somebody else that did something similar, you take their makefile, you modify it a bit, and hey, Bob's your uncle, you got it working.

Yeah, defconfig,

Lars: defconfig, defconfig. Yeah. It's been a wild ride for me since I, like, I've run Linux since I was a teenager, but I've Not really. So for one thing, like actually engaging actively and contributing to open source was something that I started like seven years ago when I got into Elixir because the community was very good and since I've started poking around with nerves, I've had to pick up like kernel configs and def configs that I Just like, I've probably poked some of that when I was doing Slackware and my audio wasn't working, like, but not, not really, I didn't know what I was doing at all.

And now I, I do have a patch for a build root thing where like the WPA supplicant couldn't be configured the way I needed it to. Like that's, that's an interesting journey. It's like get email patches and all of that.

Jonathan: Yeah. I, it's been, it's been years ago, but I when I was messing around with OpenWrt a lot I, I found this edge case where if you were trying to do a compile from scratch, but a multi core compile, the OpenWrt builder would just fall on its face during compile.

And so I've one day, I cleared off some time and I, you know, I troubleshoot it and really dove into it turned out that because, you know, it, it, it builds GCC to be able to build GCC to be able to build the final GCC. Right? It's, it's like cross compiling. That's the way that tends to work. And it would start building GCC and then download the next copy of it and then unzip it over the copy.

It was already downloading. And that was, that was the problem. That seems rude. Well, it, it was broken. It was causing problems. Yeah, the compiler thought it was very rude. But it was one of those deals where it's like, well, once you understood the problem, oh, that was trivial to fix. You could just go in and say, well, don't actually compile GCC with with multiple threads.

And so I've got, I've got that fixed. It's been an open WRT for like a decade now. And it's, it's really, it's really fascinating to me. Like this world of open source that we're in. How easy it is for someone to, you know, if they find a problem like that, to jump in, figure it out, and then, you know, you, you email the patch, and here, you know, here's the fix.

Or, you know, these days, you can just make a PR on GitHub for a lot of these projects. And something that's always really fascinating to me is that it's so it's, it is so accessible for people to jump in and, and make a fix like that, and, and Make the world just a little bit better place, right? I love it.

Yeah, it's beautiful. It is.

Lars: It is. Now you talked about multiple cores. That's kind of one of the sweet parts of running on top of Erlang. So a lot of Embedded devices these days have a few cores to play with. And because Erlang was built to do concurrency and to be able to, like, share nothing, immutability, and also being able to run workloads in a distributed fashion, when multi core came, they were like, if we start more schedulers.

Maybe we can just use all the cores. Of course, it was more complicated than that in truth, but essentially that so by default, when you start Erlang, it will start one scheduler on a thread per core. So like your Erlang program will run your number of cores, number of threads, and then it will distribute the, these like, well, green thread type of, they're called Erlang processes.

across these and like in our most Erlang systems can manage like millions of these if you ask it to but the max parallelism is of course the number of the course you have right but But yeah, so adding more cores is kind of straightforward. So scaling up your, your cloud like, Oh, let's do 32 cores.

And it will actually just use them all. This was not my experience with Python.

Yeah.

Jonathan: Most languages, you've got to work fairly hard to be able to do that. Yeah.

Lars: Yeah, no, it's it's kind of ridiculous in many ways, but the, the probably nicest thing for, for example, in IoT for this is like, Oh, we are downloading a firmware update, but at the same time we're serving the user what they need.

And because it's like preempted multi, multitasking doesn't even show up to the user. Yeah. Because. Nothing can block the the scheduler for very long before it gets kicked to the back of the

Jonathan: line. Cool. All right. So let's say somebody listens and they go, man, Elixir and Nerve sound really good. Where's a good place to go and learn, let's just say Elixir first, so like if I want to dive into the world of Elixir, where's a good starting point?

Lars: So I think the Elixirlang website has a good starting documentation. There's also Elixir School, there's also oh, well, I don't know if exorcism. io is still around. I think they might have folded. They had a very good Elixir track. I have a, I have a good overview, I would say, of Elixir. It's not like, this is how you learn the language.

This is more, it's more, this is how the language works. And this is how the runtime works. But I have a blog series called Unpacking Elixir. And that might give a good idea. I've also given a talk that can, that's kind of a good, slightly longer overview of Elixir, LiveView and all the assorted cool things you can do with it.

But there's a lot of good resources. So Elixir, the language itself, I would start from their website. Then there's the Phoenix web framework. Of course, if you're in the web space that's probably what you want to pick up next. Also good documentation like Elixir and Phoenix. came out the gate with just really good documentation and the documentation tooling for Elixir is very good.

So the whole culture is essentially documented well because we have very nice docs and you like it when the docs are nice. So when you write a library, you better make the docs nice or you will be ashamed. You will feel shame. Yes. Yeah. So I would Just essentially dive in there. And if you want to try nerves, there's a few nice ways to do it.

There's nerves live book, which is a cool sort of interactive thing. You can put on a recipe pie and then open in the web browser and tell it to do things like you can. flip the GPIOs from, from essentially a code notebook or tell it to find your Wi Fi from a code notebook.

Jonathan: So you can, you can write, you can write Elixir code right in the browser then and get it to run kind of live.

Yep. Very cool. Yeah.

Lars: Livebook is also good while you're learning Elixir and you can run it as a desktop app. That's what it's usually Livebook is a bit of a special one. Then there's the Nervs Quickstart project, which is also like just Nervs packages. Bundled up with most of what you would need. And there's some, some decent documentation around for, for all the different parts of Nerve.

So it's just again, and like. There's a good forum, the Elixir forum. We're not typically on Stack Overflow as a community. So there's the Elixir forum and the Erlang forum and they've, they've served us very well. And then there's a Slack and a Discord of course, so. If you start poking around with nerves, dive into any of those and you'll find me right there.

Jonathan: There you go. Goodness, I don't know if I can handle programming in a language that doesn't have all the answers on Stack Overflow.

Lars: But you have all the answers on a real forum.

Jonathan: There you go. So I would be, I would be remiss if I didn't ask on the, on the Raspberry Pi does, do, does nerves have support?

You've mentioned GPIO, but there are a bunch of other sort of interfaces on the Pi. Things like SPI and I2C. You mentioned the ATEC device, right? The little crypto chip, you know, that's over I2C.

Lars: Yeah, so there are libraries for I2C, GPIO, UART, SPI. My first library that I The way I got into nerves, I think we sidetracked from that was that I was trying to get an E Ink display, the Inky from Adafruit, I think running.

from Elixir instead of from Python. And since I'd done Python, I could read the Python code and try to bash my way there. I got pretty far. Then I got a lot of help. The tricky part was like the, the spy stuff and the LUTs and like, oh, a display is not the best part to start, especially not E Ink.

They're kind of complex. Yes, they are. But it worked out. The library works. Cool. And that's it. Like that was GPIO and I2C and SPI, I think, at the same time. Yes. And that was six years ago. And like I have a couple of these like more serious boards. Someone recently ported. Well made a system for this cute little thing.

So this is the M five stack. Mm-Hmm. , which is an adorable little yeah, little device. M five. I don't have an SD card in it right now. Otherwise it would show the lo nerves logo when it booted. But there's a German company that has been working with the Seed Studio re terminal dm. Mm-Hmm. , which is a raspberry pie fundamentally, but it has a lot of nice things on top of it.

So. I've been playing around with that. So this is like Phoenix and live view for listeners. It's an industrial kiosk that you can buy. It's kind of pricey, but it's also IP certified. If you do a proper enclosure, it's like meant for industrial. I also got it working on the clockwork pie you console, which is this adorable little data pad looking thing.

And yeah, people get it running on all sorts of things. So. If you have some hardware that could run Linux some people show up with a, like an ESP 32 and get very disappointed. Yeah, if it can run Linux, generally, you could put nerves on it. It might be a challenge if you're not sort of used to the Linux bits, but generally generally, it's very doable.

Jonathan: Yeah. Yeah. Interesting. It's, it's fun too. You're working with some of the same brands and companies that we are. One of my, one of my side projects is mesh tastic, which is Oh, putting Laura radios on. Some of those things. And I've

Lars: heard of mesh tastic. You're involved in developing it?

Jonathan: Yes. I, I'm, I'm the Linux guy over there, so we now, we now run the firm.

We can run the firmware as a Linux binary, and I, I made, I made that work with real hardware, so that's my claim to fame over in that system. You now have me thinking that, hmm, we ought to see if we can get our stuff put into Buildroot, and I think we may already have a user trying to do that. So, you know, at some point in the future, maybe there'll be an Elixir front end to be able to talk to the Meshtastic binary running on your, you know, little embedded device.

I mean,

Lars: is it just a binary that does most of the radio?

Jonathan: Yes. There's, there's a, there's a couple of there's a couple of libraries that have to be there. Like lib, lib yaml C is one of them. And then there's some kernel stuff that's got to be there, but I can't imagine that would be too difficult.

And it's, it's pretty much all contained in a binary after that.

Lars: Yeah, and like, yeah, the way you do things when they're not in Buildroot is you just add your patches and you add, you make your own Buildroot packages like there's endless ways to get code into your nerf system, because that's necessary if you're doing professional work, it's not like, Oh, no, I'll have to wait.

Eight months for this to be accepted by Buildroot. I refuse to run patches.

Jonathan: Oh yeah, indeed. Indeed. Okay. So is there, is there anything that we did not get to? We, we've been going for, I think over an hour now, which is fine. Is there anything we didn't get to that you wanted to make sure and let folks know about?

Lars: Oh, there's, there's a lot there. I feel like we covered

Jonathan: a lot too, though.

So,

Lars: yeah, I mean, I need to do an honorary mention of like the membrane media framework. This is something I don't see in a lot of ecosystems. There's a company in Poland, like a consultancy that specialized in media streaming, and this, this means that. Instead of trying to make GStreamer do what you want it to do, you actually build your pipeline with Elixir data structures and like the the actor model of Erlang and do a bunch of message passing.

They're of course calling ffmpeg and libmad and whatever you need because That stuff's horribly complex, but, but you can operate all of that from Elixir. And if you've ever done like media streaming and also wanting to tell people what's going on live in a web UI, that's kind of complicated. Yes, it is.

But I very recently, well, a year ago or so, I gave a talk where I was controlling the entire slide. Show with my voice. So that's the elixir machine learning effort is also very interesting, but we, I don't think we have time for it, but membrane was my media pipeline for that. Getting my voice from the browser in processing it, sending out like.

Audio measurements that I could make a waveform from, and then shoving it into a machine learning model and getting everything live into a web view. And this is like, you just stitch these things together and everything's just message passing. Like everything follows the same paradigm. This is. This is kind of the wild thing, whether you're doing a web UI, whether you're doing a sort of a, the internals of an embedded device, whether you're talking to your database, if a thing can tell you that something is happening, like interrupts are lovely in Elixir because.

events that can trigger messages can do such nice things across your system. It's not like you have to poll anything. And that's, that's kind of the ideal. Sometimes the hardware won't let you get away from polling. Thankfully it's not that hard to do polling yet, but yeah, it's a, it's a big ecosystem with very, very particular things going on in it.

I think it's always worth a look. There's a lot of good talks about it as well. There is one particular talk that I think has flipped a lot of people to trying it, which is the soul of Erlang and Elixir by Sasha Yurich. His book Elixir in Action is also a very good starting point for people that are already experienced developers that just want to pick up the language and the concepts.

Sure.

Jonathan: Where can, where should people go to find your work in particular? And the Nervous Hub as well. Plug the URL for that.

Lars: Yeah, so go to underjord. io. So I, hopefully this will be in the show notes because Cause I have no idea what you just said. Underjord you don't, you don't speak Swedish? No.

See,

Jonathan: I read that as underjord.

Lars: Yeah, yeah, you would. The Americans do. It's fine. But yeah, I would search for, if you want to find Nerves, I would search for Nerves and Nerves Elixir, not just Nerves. Yes. It's a common word. You get a lot of neurological stuff. But you can find it. NervesHub, Nerves, it's all out

Jonathan: there.

Yeah, we'll make sure to get it in the show notes. Alright, before I let you go, I've got two final questions that I am required to ask. And that is, What is your favorite scripting language? And I suppose you can, you can talk for a moment about whether Elixir counts as a scripting language, scripting language and text editor.

Lars: Yes, Elixir is my favorite scripting language. It used to be Python, but there was actually a very important, small, small, but important facility introduced in Elixir where you can just write mix install at the top of your script and state which dependencies you wanted to pull.

Jonathan: Mm

Lars: hmm. And having a script that can fetch its own dependencies.

It's pretty cool. It's like, okay, if I have Elixir installed, the script can do the rest. And there's a good example repo of like full of sort of one, one file, do something weird with Elixir. Like start a web server or whatever that, that you can look up, but yeah, I, I generally do my scripting in either bash or elixir depending on sort of how evolved it gets.

Can you do

Jonathan: a, can you do a splat bang elixir to actually run a script?

Lars: Yeah, yeah, yeah. Cool. Userbin and elixir is the usual thing. I was like splat bang. I was like shebang is what I've heard. But I thought you were going to, into Python. And my mind was going to Python because that used to be my scripting language.

It was like splat and double splat and all that. And Python is still a mighty fine scripting language. But actually the dependency management is hell.

Jonathan: Yeah, it has problems. And then text editor.

Lars: So I like Vim or Neo Vim the most but I use Visual Studio Code the most.

Jonathan: That is a very common answer, that exact combination of things.

And yeah, I find myself in a very similar, very similar case. So,

Lars: yeah, so usually Vim mode in VS code is, is how I get the bulk of the work done.

Jonathan: Yep, that makes sense. Alright Lars Wickman. thank you so much for being here. It was absolutely a blast and I got to learn about Elixir and about Nerves, two things I knew very, very little about, and now I feel like I at least have an idea of what they're about.

So, thank you.

Lars: Oh, thank you very much for having me. It's it was a fun time and happy to come back and clarify anything. If you get the weird questions about it.

Jonathan: Yeah. Yeah. All right. And so, yeah, a lot of fun. I do want to plug a couple of things before we let everybody go. Of course, first off, we've got Hackaday.

You can find my security column. Goes live every Friday morning there. And of course, we appreciate Hackaday being the home of Floss Weekly. Over at twit. tv, there's still the untitled Linux show. We record those on Saturday afternoon. And they go live Sunday or Monday, just depends upon when things get finished up.

But you could follow my stuff there and we sure appreciate it. Thank you everyone that watched us both live and on the download and we will see you next week on Floss Weekly.

Episode 810 transcript

20 november 2024 | min

Jonathan: Hey folks, this week, Randall Schwartz and Aaron Newcomb join me and we talk about Linux, and some geopolitics, and some open source challenges. It's a great show, you don't want to miss it. This is Floss Weekly, episode 810, recorded November 19th. A rising wallet pays for all boats.

It's time for Floss Weekly, that's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today is a little bit different. We do, of course, have a co host. We sort of have two co hosts. We've got Rundle Schwartz. We've got Aaron Newcomb and I'm not sure which one is the guest and which one is the co host.

But it is, it is great to have both of you here today. Yeah. Great to be here.

Randal: It's like the, what is it? The three sort of Sopranos, not three amigos,

Jonathan: three Sopranos,

Randal: three amigos too. But I was thinking, isn't there like. Three opera singers.

Jonathan: Oh, three

Aaron: tenors. Yeah.

Randal: Three tenors. There we go. Not sopranos on the wrong band.

I'm more of a baritone actually.

Yeah, yeah. Me too, actually. So, and as I get older, my karaoke goes lower. So

Jonathan: instead of doing Kermit, the frog, you're going to eventually be You, one of the one of the big animal boy, I don't, I don't even know the Muppets well enough. Animal! Animal! Yeah, that's

Aaron: right. Practice your drums. Yeah, there you go.

Randal: Something like that, yes. I'm going to regret this video already. This is great. Your background

Jonathan: removal tool, I hated that, by the way.

Randal: Oh, oh, I'm sure, I'm sure. I wasn't even looking. I had my eyes half closed. Clothes like animal. That's great. That's great.

Aaron: It's either that, or we're just going to be the, the, the two guys that sit in the back, the two old guys, the two grumpy old guys that sit in the back.

Randal: yeah. Waldorf and Statler. Oh yeah, absolutely. That's what I feel like more and

Aaron: more these days. I'm

Randal: already doing that. I'm already, I'm already channeling them. Sometimes when people ask me questions, I act like one of those guys. I don't know which one brought both of them.

Jonathan: Yeah, that's a little too real, man.

All right. So, hey, there is, there is some news that I have came across this week and I think it's something that's worth talking about. We'll see where all this leads us. The, the main, the main bit of news though that I thought was interesting is that there is a, there is an Ubuntu, I guess it's a derivative, it's a repackaging of Ubuntu.

It's made specifically for Rockchips, which those are the things in, Well in a lot of here. I've got one here actually you see them in little devices like this And so this is a well, it's a it's it's kind of the equivalent of a Raspberry Pi CM4 Yeah But the the SOC so the system on a chip that runs this is a rock chip and in this case It's the RK3 3566 and we're sort of in this weird world where Rockchip does a little bit of work on like pushing things upstream to the Linux kernel.

In fact, I found out just the other day that there are a whopping, I think it's three kernel maintainers that are at rockchip. com. Right? So, like, it's a little bit better than I thought it was. You have some companies that they'll push hardware out. And it's like, eh, if you want to put it in the kernel, that's fine.

But, if you want to run anything official, here's our kernel from 10 years ago that has all of our patches on top of it. Oh, I hate that. Rockchip is doing a little bit better. But when it comes to actual distros to put on it, one of the real leading ones was the Ubuntu Rockchip distro. And that was run by a guy named Josh.

And a couple of weeks ago he put out an announcement that says, I'm not going to be able to do this anymore. So he's frozen the GitHub, he's frozen the Discussions on it. There's no more releases and yeah, and it's it's it's not super great and one of the things that people have pointed out about this is It's, it's a real problem for all of the people that use this, because RockChip doesn't put out a distro, and a lot of the, you know, like, so, so this is a, what is this one, a Banana, something by BananaPie maybe?

I forget. Or no, this is the Pine64 board. Oh yeah, I've got a

Aaron: Pine64 sitting up here too.

Jonathan: Yeah, so this is one of Pine's boards, and it's, it's, again, it's in that little CM4 format. And a lot of these companies don't put out a distro either, they don't put out images. It's just, oh, go grab the the, the Ubuntu image from from Josh Reich.

And that is now frozen and it's not, it's not great. I'm not sure what direction we go from here.

Randal: So, again, I don't know a lot about Linux, which I've been saying over the decades, I really don't know that much about Linux. Is it that the special Rockchip Linux distribution has like extra drivers to support unique hardware?

I mean, what's missing in the standard Ubuntu distribution? You can't just plop it on there. Yeah.

Jonathan: And that's a, that, that is actually a really good question. I've spent some time over the last few weeks figuring that out. Good question. Because it's not, it's not immediately apparent what the answer to that is.

Maybe. Ah, and I think there's two main things that's, that's different about this. One is it's got U boot built right into it because a lot of these devices you actually flash U Boot. And U Boot then boots the kernel. Well, in some cases, you've got to flash U Boot with something called a device tree that's specific to the device.

And so, you know, you've got that where you've got to build this kind of special image that's got U Boot at the very beginning of it, the right device tree, U Boot, and then your kernel after that. And you've got to have the right things turned on in the kernel, right? Because when you build Linux, there's a whole bunch of modules, there's a whole bunch of compile options, like, do we support this piece of hardware?

Do we support that piece of hardware? Because you have to imagine if you just turn every switch on, you get really, really big kernel binaries. And so one of the other things that this project was doing was, you know, these are the boards we support. Let's make sure all of the different switches are turned on for the images.

And then one more thing I know that it does is there are some like out of tree patches. Where, you know, let's say HDMI for one of these particular boards. Well, that hasn't landed upstream in the kernel, but there is, there are some open source patches floating around. That's like, well, you can patch the kernel with this and this and this, and it makes HDMI work.

So he would, he would integrate some of those things as well. I've got another bore, the, the, this one's by TuringPi, it's the RK1, which has another of these RockChip chips on it. And while, while resear I'm trying to do a review on it for Hackaday, actually, and I kept running into the problem that, like, Trying to boot Linux on it was fundamentally broken in one way or another, unless you were using, you know, this specific Ubuntu image.

And I finally got cobbled together all the patches that you boot needs to be able to then boot like Fedora. So I have one of these now that I can boot Fedora 41 on just the, the regular image and it finally works. But getting to that point, it's just been a pain.

Aaron: I mean, isn't there just, I mean, so I fell away from, from.

Ubuntu, especially on the desktop a long time ago when they stopped supporting Gnome and they came up with their own thing. And I just don't like the, the whole, the whole paradigm of Ubuntu right now, but it is pretty stable. So I use derivatives that are based on Ubuntu or I just run stock Debian, right?

Which Ubuntu is based on. So you've got Debian, then you've got Ubuntu, which is built on top of that. And then other like Linux mint and other popular distributions built on top of Ubuntu. So isn't there, can't you just run Debian? Is the problem just that nobody's, nobody's working on all the special things to get the rock chip and all the peripherals to work correctly?

I mean, it seems like, I mean, this is one of the beauties of Linux actually, right? So if this was like Windows or, or Apple saying, we're not going to support this version of Linux. This Mac book anymore, right? Like that would be bad, right? Cause then you have no recourse, but it's Linux. So one of the things about why should I run Linux when we talk about that a little bit is that, you know, someone else can come along and build it.

Like you have started to test things on. Right. And because it's open source, you can actually come up with something. So I see this as a setback. But I don't see it as the end of the world for Rockchip or, you know, it's a pain in the butt basically for people that want to run the latest and greatest because now they have to run whatever the last version was, go get Debian and build the rest yourself.

Also, the other thing I would say is on these boards, I know, at least for myself, having worked on these boards for the past 20 years or whatever. I very rarely run those boards with a GUI with the desktop anyway, because I want those boards to go do something. I want those boards to go run my lights. I want those boards to go run my 3D printer.

And so very rarely am I running a GUI anyway. So as long as I can get to the command line, as someone actually, as long as someone supports all of the peripherals on that device, I don't really care because I just need to get in there, get to the command line and get something done. That's, that's my take on it.

That's, that's my quick take. Yeah. Not so quick.

Randal: So, so this, that sort of begs the question for me is there not a community around this fork of Linux? Does he not have people that are talking to each other about the distribution that he's no longer maintaining?

Jonathan: I, I know there are people talking because like I am one of the people talking it's just okay.

It's it's early enough that it's unclear what's going to happen here. So one of the, one of the frustrations that Josh had is that he said, I've been trying to get in contact with rock chip for a very long time now. All of the vendors I've talked to have not been able to assist me. I need access to the latest SDKs to be able to fix things.

And then also there was another, there was another Oh he's using GitHub and so each of his artifacts, which is essentially your release binaries have to be under two gigabytes on the free version of GitHub. So like there's, there was beginning to be some need for financial backing for this.

To really make it make sense for people. And so those were a couple of the problems that yes, other people could step forward. And I'm sure, I'm sure somebody will, right. There are some sort of endemic problems that just more volunteers are not necessarily going to fix.

Randal: You would, you would think that Rockchip having a financial interest in pushing these boards out would have an interest in making sure that.

Linux runs on them.

Jonathan: That, that is my take on this. I, I really feel like Rockchip needs to step up. That seems really obvious. It does. It does seem obvious. Yeah. But obviously not all of these companies think about things the exact same way as we do. You know, you, you could imagine that, boy, it would be nice if, like, if we were just describing what we would like the future to look like, it would be great if there was a working group put together that was Rockchip and, you know, Pine, the Pine Store Turing Pi, and every one of these other guys that make one of these boards based on RockChip.

Like, put a working group together, and your thing is, you push things upstream to the kernel, and then build distro images for people to use. Like that would, that would solve the problem. But, you know, there's corporate interests are not necessarily always aligned with the interests of the open source community.

Aaron: And sometimes it's up to the Oh, sorry, go ahead.

Randal: I was just going to say, so explain the hardware relationship again. So Rockchip is a company making some part of this. They make the CPU, the SOC. So I guess I should define And that gets used in people's, people's small devices somehow.

Jonathan: Right, so we should, we should define What an SoC is, it's a system on a chip, which that's actually, that's what CPUs are now.

Like AMD kind of went in this direction and Intel has followed. So everything is an SoC, right? It's a system on a chip. The idea is that that is your, your CPU and what used to be called like the North Bridge and the South Bridge. It's now just all on one piece, one piece of silicon. Or one, one physical chip, I guess, anymore.

It's not necessarily one piece of silicon because they have chiplet designs, which is fun too, but that's not the point. Anyway, so they make if somebody's watching the video, like the, the, the black blob right in the middle of this is the CPU. Rockchip makes that thing. And then you've got the rest of the board around it.

And in this case, you know you've got probably a Samsung memory module on there and maybe a you know, another chip by rock chip. So I can't quite read all of these, but you know, there's one, two, three, four, there's six. I can't read any of

Aaron: them right now.

Jonathan: Yeah. Well, yeah. Yeah. Close one eye and tell me how much you can read.

Hopefully not this one. Right. Oh, I

Aaron: can't

Jonathan: see anything. So, you know, you've got, you've got the people that make the silicon and then an integrator will come along like again, in this case, it's pine, the pine store, pine 64. And they made the, the whole board and then this will slot on a carrier board somewhere.

Well, so to be able to get Linux to boot on this thing and know where all of the ports are for the kernel, like to even know how much, what hardware is there. You've got to have some it's called a device tree. And that's basically just a list of here's the peripherals that are connected. So it tells the kernel what drivers to load and where to find them.

And so somebody has got to maintain that. And that's sort of the piece that's missing is getting that maintainership. So

Randal: it sounds like there should be a cooperative amongst all the people who are Rockchip integrators.

Jonathan: Yes, and I think there is a little bit, like ad hoc, but definitely nothing formal.

Yeah,

Aaron: usually it comes down to the individual manufacturer of the board. Like OrangePi does a pretty good job with their boards. They always have, now it's not as nice as RaspberryPi, I'll get to that in a minute. But OrangePi, at least, you can go to their site for just about any board and you can download their, they will build as part of their process, they will build a distribution that works for, you know, or a bunch of different distributions at the time they release the board that work.

So they'll have their own orange, OrangePi. org. I forget what it's called. Orange Ubuntu based thing. They'll have Debian. They'll have Armbian usually. And, and usually Android as well. So they'll have like those four or five. They might not get updated, but they'll be there. And at least you can pick one to run and maybe you can update it yourself.

So they do a decent job, but the best one out there, of course, that does this is, is Raspberry Pi and Raspberry Pi owns the whole chain. Right. They, they own everything from the Silicon up through the operating system, including all the boards. And, and so they, that's why when someone asks me, Hey, there's this new orange pipe board that has never, you know, they never used Raspberry Pi or anything, but they're like, I'm trying to figure out if I should use orange pie or Raspberry.

It's like, no, just go get the Raspberry Pi because. There's the community there, you know, you're going to get a, a, a semi modern operating system and may not be, you know, last night's release, but it'll probably be last month's release and, you know, you're going to be okay. You're going to get support. You can go to the forums, ask people how to do it.

I, I don't, I haven't even bought the latest Raspberry Pi, but I don't need to because I understand how to do this on other things. But for someone that's just starting out, use the Raspberry Pi. And the reason is exactly this problem. It's exactly this problem.

Jonathan: And you run into, you run into edge cases on the other boards.

Like I've, I have more than once been running one of these not Raspberry Pi development boards, and it's like you install updates and it pulls a new kernel, and guess what? That kernel is from the wrong repository and your board no longer boots. Like that is, that is distressingly common.

Randal: Yeah. Oh geez.

Yeah. Yeah. I don't have these problems with my Mac.

Jonathan: Yeah, well, so, I mean, it's, it, I guess it's, it might be worth kind of talking about, like, why we, Aaron and I, have gone down this rabbit hole, like, what are the things that we want to be able to do with these, with these boards, and the, the two big ones for me, and I've talked about this before, the Raspberry Pi blew my mind, because suddenly I had a little computer running Linux that could talk to the outside world via hardware, So And not just like video on a video camera, it had, it had GPIO.

And so then you can know, you can switch lights on and off. You can wire that into a garage door and open your garage door from the command line terminal. And, and then, you know, the next step on top of that, of course, is you can write a webpage, which calls a command line on the back end. And you can hit a button on a webpage and open your garage door.

The, the thermostat Not while you're driving. Right. You can do it while you're driving. Never. I would, I would never

Aaron: do that while driving. I do it all the time because then the next step from that, the next step after the webpage is you hook it into Google Assistant or Siri or any of those. And then while you're driving, as you're almost home, you say, Open the garage door and you pull in and the garage door's open.

Randal: Yeah. And you just triggered 72 people at home by saying those words. Yeah. Well I didn't say key. He didn't actually say the word. I just said the Oh yeah, I was listening for that. Gotta be careful. Oh wait, I turn my phone over right now. There we go. Alright, now it won't respond. Thank God, .

Jonathan: So like the, the thermostat at my house.

is a Raspberry Pi with a board on the back of it to be able to do it's not actually relays because relays have problems because so it's a it's a solid state relay, but it switches the air conditioner and the heating on at my house. And so I've got a little script that pulls the temperature and switches on whichever one is needed.

And That's been, that's been running now for me for like five or six years, pretty successfully. And so those are the sorts of things, like those are the things that people might want to do with one of these. HTPC, that's another one. Home theater PC. Right? And, and I know people use Raspberry Pi for that a lot because you can do things like you can, you know, emulate your old video games.

You can watch movies, you can have YouTube on it, all kinds of stuff people use it for that you don't run Plex, right? Do you run Plex or Kodi on it? Yeah, absolutely. Cool. And so, you know, there, there's, there's a lot of interesting things there. One of the things I've been doing with them recently is GitHub runners.

So I've got, again, the Turing Pi, I've got it right over there, that's why it's on my mind. It's got, so it's got four slots on it, and I've got four RK1s in it. And I've got two of those set up as remote GitHub runners. And so when we need to be able to build, you know, ARM32 or ARM64 builds on GitHub, it can just, it can farm out to those and do it.

Because, you know, GitHub doesn't yet offer. on their free tier in any ARM builders. So, there there are some places there where it really makes sense.

Aaron: Yeah I always tell people if you're a, I used to be a strong advocate for Linux desktop. Like I used to tell people, oh yeah you gotta ditch Windows and run Linux. Well, not so much anymore, right? I mean, Linux really comes into play, as you've been describing Jonathan on those small boards of course. Linux also comes into play if you're a developer.

Pretty much anything you're going to do in the cloud, right, is going to be Linux based. It doesn't necessarily have to be, but you're almost maybe 90, 95 percent of the time you're going to be distributing your applications out to a Linux based virtual machine, which is running You know, Kubernetes and containers, but the base of that is going to be Linux.

So for developers, I've always said, even when my son was going to school learning software development, I was like, you need to at least have a good understanding of Linux. You may not have to support it, but you have to understand the concept so that when you talk to your DevOps person or your support person and try to figure out why your application isn't working, and they tell you that there's a kernel bug or there's a kernel panic.

Or something like that. You need to at least know what that is. Like you don't have to support it yourself, but some developers do support themselves. So I think like, it's not for grandma. It's, it's not for the person that's never used it before. It's not, I wouldn't even say you could use it for, it's good enough to use for, for your daily driver.

I D I used to at work. I've got two of them here

Jonathan: that are Linux boxes that are my daily drivers.

Aaron: But you know, sometimes, sometimes you're not, and I know plenty of developers that obviously I think probably the number one OS for developers is, is they run on Macintosh, right? So. So, yeah, I think, I think that's one really big area is just to understand, you know, if you have an Android phone, you're, you're running Linux already.

You just don't know it. You know, if you're, if you're using an application on the web, it's running on Linux, probably somewhere in the cloud and you just don't see it. And there's reasons for why, why people choose to run their applications there. Customizable, stable, upgradable supportable, all those kinds of things.

But yeah, for your daily driver, I'm not going to try to convince anyone to try to try to drop their MacBook and go pick up a PC and run, run Linux desktop on it anymore.

Randal: Right, right. Like I'm probably never going to get rid of like a Mac interface unless something really outstandingly better comes along for cheaper.

And that's not going to happen on either axis. The, but the thing is like, I'm running right now sort of because I'm, I'm just a shoemaker's children problem. I want to put time into my cloud servers, but I never seem to get around to it. You know, it's like, eh, that'll, that'll still run tomorrow. It would be fine.

So I've got three free BSD servers in the cloud. And now that I'm doing a lot of stuff with Dart and Flutter. I regret this because FreeBSD is not one of the target platforms either to compile on or to target to build for. And so I can't do anything that I really want to do. Like I want to move my website so that it's a, a Dart application spitting out HTML, which is perfectly reasonable if the, the host box is a Linux box or can at least run Docker.

And even running Docker underneath FreeBSD is. A bit dicey from what I've heard. So so I'm thinking strongly about, let's see, first farming it out. So I'm actually buying separate machines for my web and my mail and all that, rather than having one big monolithic box that I'm doing all of the maintenance on as a free BSD box.

But that would be a break in multiple years of, of history for me. So I've, I have to then learn enough about Linux to be able to have a hosted service of my web and my mail and all that stuff correctly. I could go to. Platform as a service stuff and start doing, you know, like all my mail through Google or something.

I don't quite trust that yet. And I have too many special things I've wired in. So like the way you guys wire up hardware, you should see my proc mail RC and my, and my Pearl script that sits in front of that. And, and it's like all these things. Are very, very heavily customized. And so I have to track the upstreams of all these things I'm using to make sure that my stuff still keeps working at the software level.

So I, I can relate to the hardware problems that you're facing because I've got incompatibilities of this hardware or this software doesn't work with that software.

Aaron: Yeah.

Randal: So it's, it's, it's a similar problem. Luckily I've got things like unfortunately now I have both original stuff on my Mac and brew and Mac ports because they each had their own things.

And my path is about 85 elements long, trying to suck all that in. And what JQ am I invoking today? I don't always know, and they have different features. So it's like, Oh, boy, the path has to be exactly right. And I don't dare move it now that everything's sort of working on my Mac. It's, it's definitely scary times for me.

And I, I also, I just don't want to deal with it anymore. I just want it to work too, which is sort of the anti tinkerer theory.

Aaron: Yeah, but,

Randal: But it means you have to be a bit of a tinkerer to get it sometimes to play well with the stuff that you've also tinkered with. Do

Jonathan: you hope? So, so Randall, a couple of things here.

Yeah. First off,

Randal: sure.

Jonathan: From the little bit of BSD that I've worked with, I don't think you're going to have much of a problem going from an A net B, S, D, open BS, D to, to Linux. So much of it is almost exactly the same. And the , there's, there's better compatibility with Linux. I think in a lot of cases there's more packages available too.

So I, I just, yeah, I don't think it's gonna be a problem for you. I think you can dive into that and not, not really have much issue. Talking about, you know, I,

Randal: I have to, I have to learn the GPL version of all the argument lists, the arguments, because that's, that for some reason, GNU just went off sideways and a lot of things that I use every day.

And it's kind of annoying because I grew up with, I, you know, I used Unix V6, Unix V7 2. 8 So I grew up through real Unix. And I still don't consider Linux to be real Unix that because it doesn't have the same history.

Jonathan: That's fair.

Randal: You know what? Yeah. Is that fair? I think that's fair. Yeah. I

Jonathan: think that's

Randal: fair.

Okay. Yeah.

Jonathan: The same problem. So I have to learn the

Randal: Linux. Yeah. I

Aaron: have the same problem going to b sd.

Jonathan: Yeah. I have the opposite or the opposite. Yeah.

Aaron: The mirror. The mirror problem. Right,

Jonathan: right, right. A a lot of the i's like, why doesn't

Aaron: this work?

Jonathan: A lot of the GNU Corps utilities though, or at least some of them.

Actually support the BSD flags, don't they? I'm pretty sure some of them do.

Randal: They, they've been sort of forced to because POSIX was sort of a compromise in some ways between both worlds.

Aaron: Yeah.

Randal: And so as everybody started moving more towards POSIX compatibility, they ended up having to, the BSD made some concessions and FreeBSD, and Linux made some concessions to be able to add things in to say that they're POSIX compatible.

POSIX compatible if you only use these switches. And so, and so that, that helped quite a bit.

Jonathan: David Ruggles in the chat says, Linux is not real Unix is a feature, not a bug.

You know,

Randal: the thing is though, I remember when Unix fit in 32k. And, and that was considered huge back then. So it's, you know, it, it, it, it's come quite a long way since then. I'm sure there's some parts of the Mac OS I'm running that are under 30 TK, maybe one of the files or something. I don't know, maybe, but, but.

But now they've become these giant monoliths, but there's, but they're also so feature full now. I mean, there's, there's so many features, then they keep adding more features and telling me to upgrade again, and then they're adding more features and I have to learn those new features and then those features aren't compatible with my phone.

So I got to upgrade my phone now. Yes. So

Jonathan: did the BSDs have a package manager? Cause that's kind of a Linux. Oh, lots of them. That's kind of, well, yeah. So that's sort of a thing that was originally, I think originally developed in Linux, like the, the, the yellow dog. So Red Hat's yellow dog package manager was one of, I don't think it was the very first one, but it was one of the first ones and you know,

Aaron: RPM, right.

I think RPM was before the yellow dog. That's the first one I ever used back in 1990. Eight when I was messing around with a red hat, I was like, Oh, I can just use this RPM command to like update my software, get a new program on you. That's pretty cool. Previous D

Randal: from out nearly its very beginning.

It's always had some sort of quartz managing system. That can record dependencies between ports and install dependent ports. That was mostly a source based system. And then they moved to the PKG system later on. And that's kind of where all my systems are right now. They're all. PKG related, which is kind of laughing now that I've got brew, which does some source compiles and some binary downloads and a Mac port still does some source compiles and some binary downloads.

And it's like you know, upgrade time. I never know whether it's going to be four minutes or three hours. I don't know what it's going to compile something or just download a few things. Yeah, so

Jonathan: I, I'm, you know, I've thought about over the years, like, what is it about Linux that really has drawn me to it?

And what is it that I don't like about, you know, those other operating systems? I've got to say the package manager is one of the things that just really, really makes a difference. And in thinking about this, like on Windows boxes, I've had times where, because I, I, I do tech support, I find people's Windows machines that are just in terrible shape.

And one of the things you can do with a package manager, particularly with something like RPM obviously that's what I'm used to, so that's what I'll reference, is every file that is part of your operating system, it lives inside of one of these RPMs, in one of these packages. And so if you have a file that gets corrupted, you can just say, Look at all, basically what you can do is say, look at all the hashes of all the system files and if one of them doesn't match, just reinstall it.

And I, I, I still don't understand why they have not added that to Windows. Now, I know they've got a package manager now that is apparently fairly decent. I don't think it applies to all of the system files though. And that's just a, it's a killer feature for me for Linux.

Randal: And let me, so, so Mac OS is slowly getting better in that third party apps can now create sandboxes and containers for their machines, for their, for their installations, but it didn't always used to be that way.

It used to be that what Apple told you was just make sure it installs in user local somewhere. And it's like, well, that's not a package manager. That's well, there's, it's installed. You want to remove it later? You want to see what's corrupt? You can't. It's

Aaron: yeah.

Randal: And so that's why the third party Mac ports came along and then brew, brew does it a whole lot better, by the way.

I would say Mac ports now seems. So chaotic at this point. I've been slowly looking at my list of things installed with ports and one by one installing, reinstalling them with brew. And then I hope that that's a similar version to the one I was already running so I don't break something. The one I've been dealing with lately that's crazy is YQ.

I don't know if you've ever seen the YAML version of JQ. But it's really, really cool for a lot of different transformations YAML in, transform it the same way JQ does and spit it back out. Except there's two things in the world that are called YQ and they do entirely different things, but they're both related to processing YAML using a JQ like syntax.

One of them actually uses JQ behind the scenes. So it turns everything into JSON and then back, which is weird. And the other. Has a subset of JQ's instructions built in. And so, if I accidentally get the wrong YQ in my path ahead of the other one, because I have them both installed, because they both are handy in different ways, then, ah, it's such a mess.

It's such a mess. One of them is written in Python, by the way, which I think Aaron will probably like. Namespacing,

Jonathan: it's hard sometimes.

Randal: Yeah, yeah, yeah. Hard to think of good names for stuff. All the good names are taken, that's definitely true. All the two letter names are definitely taken. Yeah, that's probably true.

Jonathan: Oh, I did not know about YQ. This is actually really interesting. Oh no, now I distracted you. There you go. Now

Randal: there's two of them. One of them does, like I said, one of them is based on Python and only, and has a built in sort of YAML interpreter. So it has, but it's also then cloned a lot of the JQ operations.

The other one that has YQ. TomlQ and XMLQ, which basically means you can manipulate from one to the other of any of those four things, JSON, YAML, Toml, and XML, and manipulate all of them. That's the cooler one. That's the one that uses JQ behind the scenes and then does in and out mappings for those other things.

Languages, that one is really, really cool as well. So I, again, that's why I have both of them installed. I have the one that does the Python version. I have it as PYQ just to make sure that I can get to it whenever I want to. But yeah as an example, I was using probably going deep off of, I don't know what your, our actual agenda was going to be today, but let me just finish this and we can go back to Linux again, but in in the, in the Flutter world.

You have this tool called pubspec. yaml, which describes all of the things that belong in your release. So it has a list of all the version numbers and everything like that. But what actually gets installed is derived from that, because it also has to have all the sub dependencies and all that.

That gets locked up in a file called pubspec. loc, which is a JSON file. Okay, so now I've got JSON representations of YAML stuff, and I wanted to take the actual versions that got installed and put that in my dependency range in my source pubspec YAML file. So I wrote this About this long single command line that unpacked the, the, the JSON file got to where it picked out the actual dependencies and sub dependencies that were used and spit that out as a nicely formatted YAML file.

And then I dropped that into my PubSpec YAML. That's great. And really cool that I can get two different languages on both ends of that pipeline. So. Yeah, really, look into YQ. Again, there's two versions of it. There's the one that has the multiple languages thing with TomoQ and, and XMLQ, and I've used that too.

I've actually got, oh, XQ I think they call it. It actually goes from XML to and from XML with JQ like operations on it. It's pretty cool.

Jonathan: Yeah, I've actually become quite fond of YAML for config files. When I had to write a schema for doing config files for Project M1, I was like, let's use YAML. It's great.

I like YAML. Now, occasionally The great

Randal: thing about YAML, the great thing about YAML is that JSON is a perfect subset of it.

Aaron: Mm hmm.

Randal: So at any point you're freaking out about indent, you could just go open curly brace, blah, blah, blah, blah, close

Aaron: curly brace.

Randal: And it's, and it, and that can all be indented whatever way you want.

Yeah. You can run it over multiple lines, all that stuff. But what's also cool about it is it's like relaxed JSON. You don't need to quote your keys. You don't need to do it. It's, it's, it's, and you, and you have comments. Yep. So it's like the best JSON parser you can get is a YAML parser.

Jonathan: Yes. Yes. I, I have, I very much enjoyed YAML.

The only thing is every once in a while, somebody will come into it and work on it. It's like, nothing's working. It's like, okay, copy and paste. Okay. YAML is whitespace dependent. You have to actually get, it's like Python, man. You have to actually get your whitespace lined up. That's,

Randal: that's tripped up a few users, but, but again, we'll drop in a JSON mode when that happens and it's fine.

Jonathan: Yeah, I have to look at whether our parser could reason us in this project reason a C library to parse it. I got to look and make sure that our parser can actually handle the JSON curly brackets. That's

Randal: in the spec for YAML. So if it's not able

Jonathan: to parse it, it's not spec compliant. If it's not able to parse it, we need to find a better library.

Is that is that what you're telling me? Yeah, and I know that I

Randal: know that C one that comes out of library does it. Pretty accurately. It's pretty, pretty on spec. So,

Jonathan: yeah, cool. Yeah, cool. All right. Let's see what, what direction, what direction were we going? What direction did we want to go north? Let me, let me, let me throw out,

Aaron: let me throw out one more Linux use case that I think people should be at least cognizant of.

Yeah.

And that is for anybody building systems for other people. Which I used to do. Or if you're just techie enough and you want to have a good backup in case something goes wrong. I always recommend that people have some, some easy form of, you know, Linux mint or something like that on a live USB stick.

Yeah.

And if you're building systems, as I mentioned, I would always ask people, family members or whatever. Hey, building this system for me. Great. Do you want me to use me? I'm going through puberty there. Do you want me to throw Linux on this system as a backup? And you'll never have to deal with it.

It'll automatically boot into Windows. But just in case you ever have a problem, you can at least select the Linux option and get to your files or get to the internet in case windows decides to throw up on you, which mine is doing. If I disconnect right now, it's because I got a blue screen. And that, that has saved my bacon more times than I can count really, because I've had, I've had cases where, for example, my raid.

Running on the motherboard failed and Windows doesn't know anything about it. Right? I mean, it doesn't even know that there's RAID underneath, but I could go into Linux and run. I think it's test disc test discs, analyzes the, the, the bare bones of your hard drive and it can tell where the files are.

And you can even copy those files out onto a different drive. And I've saved like picture, family pictures and all kinds of stuff just by having. Either a thumb, you know, USB thumb drive with Linux on it, or you know, Linux as a, as a separate partition on the drive that I can recover from. So the one more, one more use case where I think people should at least be aware of Linux and also if you're having trouble understanding Linux, you can always go get my book, Linux for Makers, because the whole reason.

The whole reason I wrote that book was because people were coming to the makerspace from either macOS or Windows backgrounds. They wanted to run Raspberry Pi. And they're like, what are all these directories? I don't understand what they mean. You know, And so that's a big part of the book is just explaining what Linux is.

It doesn't go into great detail because it's not meant to, but it's like, here's what you need to know about Linux to make your experience with Raspberry Pi better. And here's what those directories, here's why they came to be called. USR and ETC and all that, you know, and so people just get a basic understanding of it.

And I think once people get past the, the translation between how it works on MacOS, how it works on windows and how it works on Linux, then they can start to at least function a little bit better.

Jonathan: Yeah, you know, the ironic thing is that Mac OS under the hood is so similar to Linux when you actually get on the command line, because they're both, they're both Unix compatible.

And so you get a program like brew installed and you start installing the binaries, you can, you can get your Mac machine set up to the point to where it almost behaves like a, like a Linux box does. Windows, Windows is like a

Randal: Unix box,

Aaron: like a Unix box, technically BSD.

Jonathan: Yeah,

Aaron: yeah.

Jonathan: Well, so no, that's that's the point.

I mean, they're both open step. They're both Unix boxes. So you can make your your your Mac OS install act very much like your Linux install does because they're both Unix.

Randal: Yeah.

Jonathan: Yeah. That's, that's the

Randal: point. No. Stop making me say that. Linux is not Unix. No, it's not. You can make, you can make Unix a lot like Linux.

It can be a lot like it. Yes. They both have LS. Both of the LSs have 65 parameters, but there are different 65 parameters.

Jonathan: Yeah. I suppose that is always going to be a thing.

Aaron: Yeah. But like, even in my book, I, I showed people how to write the Raspberry Pi operating. system to to an SD card, right? And both Linux and Mac OS use DD to write that image out to the SD card.

And it works the same way basically on either one. So there's enough similarities there. Yes. It's not Linux. It's Unix. But There's enough similarities there. I think people, people could get a, that were, were at an introductory level would look at it and say, Oh, this is just like that other thing I was using.

Sure. But

Randal: there's enough similarities. I mean, I I've, I've in practical terms, I've had Linux in the cloud from time to time for some of my clients. And I've, I've, I've sort of. Poked and prodded my way through those. It's not like I can't do anything at all. It's just that every once in a while I'll type a command and it goes, I don't know what this switch is.

And I go, oh man this, oh, oh, this doesn't have GPL or GNU code does not have that, that switch. Okay, I get it now. Yep. I'm used to the BSD switches, not the GNU switches.

Jonathan: Randall, are you familiar with TLDR? You long didn't read the the command line application tldr. Oh, no. I didn't know there was one. There is one I I assume that you can get it with brew but it's definitely a thing on pretty much all the linux distros So you install tldr and then you just it's tldr and then the name of the command that you want to run And what it'll do is it gives you like the minimal version of the man page And so it'll, these are the four most commonly used switches.

And here's three examples that use them. And it's for, for, yeah, it's really nice. Leo actually is the one that got me hooked on this. Back during one of our first ULS episodes, he, he plugged it and I'm like, Oh, that's really great. So

Randal: brew does have TLDR. I'll have to install it and

Jonathan: play with it later.

TLDR. It's really cool. I won't be like,

Randal: I won't be like like Leo used to do where he would install something the moment I say it on the show, that was really weird. Wait, wait, we're still doing the show. They're like, yeah,

Jonathan: that was. That was his thing. I think that's still his thing in some cases. I think it's just because he's been doing it for so long.

He doesn't have to think about doing the show. It just comes naturally. So he could, Oh, okay. I'm going to install this while we're talking. He probably

Randal: has a completely spare PC sitting next to him that he can just immediately swap to if he screws the one up that he's in front of him. Yeah, I think so.

Aaron: Especially these days, right? Without the studio.

Randal: Yeah.

Jonathan: Yeah. Well, I don't know. Gotta be

Aaron: self sufficient.

Jonathan: Maybe, maybe that means that he's running the he's running the show off of his computer. I don't know. That's what I do. So I can't, I can't mess around with this machine in front of me. I can't mess around with it too much.

Cause it's doing the streaming.

Randal: Yeah, me too. Cool.

Jonathan: All right. Let's see what what, what direction do we want to go next? Hey, we got about 15 minutes left. There's another interesting thing that happened with the Linux kernel. And I guess I'm curious on y'all's take on it. I've covered this in other venues, but since we're a very Linux centric show today, it makes sense.

Do you guys follow where the Russian maintainers got kicked out of the kernel?

Randal: Yeah. Weird. That. But then, then you start wondering how many other, Oh, but you should go ahead and tell the story first for our audience. And then I'll, I'll fill in my comment.

Jonathan: Sure. I'll, I'll give you the, and maybe I'll answer some of the questions.

Cause I had, I had questions and I've looked into it too. I've actually, I've actually got questions.

Randal: We got,

Jonathan: I got a hold of a couple of people and, and anyway, so there was a rather cryptic message, you know the, the, The commit history of the, the part of the email, you know, when, when Greg Carl Hartman wrote it, I think he said something like for compliance issues, we're having to change the maintainers file a bit.

You know, it was, it was real cryptic like that. And people got to looking at what it was is just about all of the maintainers. So that's the, the people that are responsible for different parts of the Linux kernel. The maintainers that had like email addresses that ended in dart. ru just got removed from the maintainers list, no longer maintainers.

Geopolitics and politics and all that being what they are, people got a little frustrated by that and let people know, and there was a lot of trolling that went on some of it was hilarious, some of it was not very helpful, and then of course Torvalds was Torvalds. And, and stuck his oar in, in, in a very Torvald sort of way, which is fine.

I have no problem with that. But so what, what it came out to be is that apparently, obviously, again, obviously geopolitics there are Russian companies that are under My mind has completely gone blank. I've forgotten the word. Sanctions. They're under sanctions. They're under sanctions from the U S and other countries around the world.

And so one of the things that those sanctions mean is that like, there are limits to how you can cooperate with them. And when it comes to source code, even, and apparently the Linux foundation got a visit or a phone call from the U S treasury department, and then said, Hey, I see your open source project.

I see you have people that are partially responsible for it, that are on our sanction list, and this cannot be. Yeah. And I think that's, like, fairly reasonable, because they're, they are the U. S. Treasury is responsible for enforcing the laws and the executive orders that we have in the United States, and the Linux Foundation is based in the United States.

And so, it's just sort of an inevitable thing for that to have happened. I think the colonel could have done better with the way that they like announced it and messaged around it. It came out later that apparently the the internal lawyers of the Linux foundation Gave them guidance, but did not tell them how much of that guidance they could make public And so there was that it's like our lawyers have not yet told us that we can make a statement on this Right, and that's that's just one of the terrible things about the way the legal system works, but it is part of how it works There has been, I think at least one of those developers was able to document that.

No, no, I am not actually employed by, you know, by call or whoever it is. And so is back as a maintainer. The thing that the thing that worried me the most about all of this, and this is mainly, I guess, mainly what I'd like to get y'all's opinion on is that there was a statement made by somebody at the Linux Foundation that said essentially We hope that removing these people as maintainers is going to be sufficient to make the treasury department happy and we hope we will not be required to remove their contributions.

And that, yes, I, I have a real problem with that I have a real problem with that because in the United States we have the First Amendment, which says that, you know, you can say whatever you want to, and we have some, some legal opinions that have pretty well established that code is protected speech, and so I, I really am concerned that the U.

S. Treasury Department even made that threat. But I'm real curious what you guys have to think about that. I

Randal: don't think that so much as a First Amendment protection issue as I do recognizing the open source development process, which is that those lines of code clearly had to get reviewed by non Russian infiltrators. No, I won't say that, that's just evil. Non Russian residents. Right, right. I don't want anybody to misinterpret.

I actually, you know, I'm a world traveler. I've been in 68 countries. I do have a much greater sense of recognizing my kindred humanity in every part of the world. So let's just start with that. Okay. But but I think Every piece of code that went into the kernel ultimately was viewed by many people.

So to say you won't have to unwind particular contributions because of their source is a bit confusing to me. It's not understanding the process.

Aaron: Yeah.

Randal: I also am curious about how it was only Perhaps anybody who had a email that ended in RU, how does that prove anything? So it,

Jonathan: it was, there's a little bit more detail on that.

I think, I think all of the dot RU email addresses were taken out. And then there were some that were also things like Gmail. So essentially what it was is it was, it was people that were known to be Russian nationals. I think

Randal: it

Jonathan: was, it was a little bit more intelligent. It was not, you know, it was not just a said script, right?

Let's just set and remove all of the lines that have dot R U, just greedy. For Hawk or

Randal: Pearl. Yeah. Well, okay. Hawk and Pearl, whatever that is.

Jonathan: Yeah. Yeah. Well, that's the fun thing about the world we live in. There's 15 different tools that can do that for you. Anyway it was a little bit more intelligent than that.

One of my buddies that I talk with And we discussed this idea of what would you have to unwind the patches? And he's like, his, his opinion was you would just have to reset the entire code base to whenever that executive order went out. There's just no other way to do it because you have so many patches built on top of each other.

He's like, that's what you would have to do to comply if they actually wanted you to do this.

Aaron: That's BS. That's, that's, well, first of all, that's a hypothetical, right?

Jonathan: Yes, correct. Correct. So that's not,

Aaron: that's not what they're saying, but the way

Jonathan: I will say this, the way that the mailing list message reads the way I read it.

Is there was at least the threat made that, let me rephrase. There is at least the consideration that this could theoretically be possible. Like, so this is not just me suggesting this could be possible. This is a Linux foundation member saying, we hope that we do not have to do this.

Randal: There's a dangerous component to this that I also want to bring up.

And I, I'm sure everybody's thought of this anyway, but let me say it for the few that haven't. Is this also going to apply to every other us based company?

Aaron: And

Randal: does that mean that the treasury department will be coming after the software freedom conservancy and especially these umbrella organizations like, like the SFC to say, your members all have to comply with this ITAR regulation.

And sanctions that we've had on, on particular governments and individuals, and that would be horrendous for the SFC because they've got like 50 projects, a hundred projects, something like that. Right. And, but they're us based and, and so where, where do we draw the line here? I recognize the intent.

But I don't recognize the practicality of it, given the world of open source.

Jonathan: Yeah. So let me, let me tell you what I think the, the, the real intent was, what I think annoyed people in the U. S. government. So obviously we have a, a, we have a war zone where it's a, it's a somewhat hot war right now and Russia's on one side and U.

S. allies are on the other. Like this is just the reality of the situation. And you have Some of these Russian weapons that are literally running the Linux kernel. Like this is, this is a thing. This is a thing that is known to happen. And I think what happened is the, the treasury department, the United States looked at that and went in essence, our developers that are working on the Linux kernel are assisting.

These Russian developers, essentially because the kernel is running on these weapon systems, we essentially have U. S. developers that are helping to maintain the Russian weapon systems. Right? Like, I think that is probably the thought process that went into this.

Randal: And didn't we also have a recent example of where some apparently cleverly hidden code caused a backdoor in some SSL transport mechanism?

Jonathan: Yes, and so that could be this, it could be related to that. That's the XZ I'd be triggered by it. The XZ libraries, in a very clever way, were attempting to backdoor SSH, and that did actually make it out on a few machines. There were some machines in the wild that had it backdoored for a very short amount of time.

Although ironically, The the time zone work on that happens to line up with us east coast. So I have theories on who is behind that Could be cuba right Yes Anyway, so I I guess I guess this this whole story the thing that really comes to mind for me about it is like Open source has sort of avoided having to play the geopolitical game.

And I'm not sure that that's going to be able to continue forever. And that's sort of unfortunate, but maybe also just the reality of the world.

Randal: Along that line some of the new EU requirements for software in general might apply in very chaotic ways to open source.

Jonathan: So we've, we've talked to

Randal: a lot of issues.

Jonathan: We've talked briefly about that with people like Simon Phipps. Simon actually was a big part of lobbying in how those laws do and don't apply. And so there is actually quite a big carve out for open source in the, in the EU software laws. Which, you know, it, it really, it really makes sense. If someone is not making money for a, for a project, trying to force them to, to follow all of these requirements is just, it's, it's a non starter.

Randal: Right. Yeah. Open source starts at a negative 5, 000 a month deficit. Yeah. You can't quite do that. Right. I don't mind doing it for free, but I don't want to pay 5, 000 a month out of my pocket to hire these six guys they need for compliance with the EU regulations. Yep.

Jonathan: Yep. Yep. And so one of the interesting things with that is, so now you can get it, what it, what those laws do from what I understand is they, they put the onus on the companies that are commercializing.

And so I, we've heard some stories now where these companies are going to open source projects and saying, okay, here's the documentation that we need to be able to use your project. And they sort of expect it for free. And the thing I've been telling people is when you get one of those emails, you just respond and say, we will be glad to help you with that.

Here's our hourly rate.

Randal: Yes, absolutely. Yes. You making money. We making money. Yeah. Yes.

Jonathan: That sounds

Randal: great. Yeah.

Jonathan: So hopefully that'll become more and more of a thing and

Randal: maybe it'll be, maybe it'll be a good thing. A rising wallet pays for all boats. I like it. Oh, I like it.

Jonathan: Just

Randal: made that up. That could be the title for today.

Jonathan: It could be, we've had two or three. Yeah, we've had two or three suggested. Okay. Let's see, we are getting close to the end of the hour. Shall I ask each of you what your favorite scripting language and text editor is? We haven't, we haven't done this for a while. I think I, I think I remember Randall's.

But have they, have they changed since the last time I asked you?

Randal: Not in the last three or four weeks, but just to, just to remind everybody, you know, what's funny is I don't think I ever answered that question when I was asking it all those years. Oh yeah, that might be. So although I would, I would say, yeah, I'm also an Emacs user, but now I do most of my work in VS code, particularly because.

For one it's embedded in GitHub, so I can just type dot when I'm looking at a repo on GitHub and I'm in VS code on that project. So cool. Yeah. Right. I didn't realize that until like, like three or four months ago. And I went,

oh, this is, why didn't I know about this for all these years? I've been downloading this stuff, putting in an opening in my own.

Right. Anyway, so I'm learning, I'm using VS code parse because of that. And also because VS code is being used in the IDX project, which is now open to everybody with the Gmail headers, by the way. IDX. google. com is essentially you get your own little Linux backend, you get VS code, you can suck down Git projects, you can run them in any of a number of architectures, you can control how the Linux is configured with somebody called mix or something that's S S M S I S there's like a configuration language for Linux layouts and stuff.

So, but it's great. And it's free idx. google. com. Go check it out. And it also has a templates for starting up, like say a typical angular project or a typical, in our case, dart and flutter project. And it even will put a web view and a Android view. As emulators right in your window. So you can do incredible development with nothing.

Yeah. So VS code is running in that. That's kind of a long way of saying that. And my favorite programming language right now is now Dart because it's part of Flutter. And I'm getting much better at Dart. Dart's really cool. Dart's like Smalltalk that won finally. So that's great.

Aaron: And of course they've integrated or they're trying to integrate Gemini into this as well.

Oh, of

Randal: course. Oh yeah. Of course. Of course.

Aaron: Yeah. Now it all makes sense. Yes.

Jonathan: Aaron, I, I, boy, I don't, it's been a while since I've asked you this. What what's your favorite text editor and scripting language these days?

Aaron: Yeah. Nothing's changed for me, actually. I mean, text editing. Is still, if I have to drop to the command line and edit a file, it's still nano because all of my systems pretty much are Debian based systems and nano is almost always there on, on all the operating systems, all the distributions that I run.

So, so nano is just easy. It's just easy. I used to be VI when I started, cause that was what was there and Emacs was too hard for me to learn. So, you know. So now it's nano and then languages are still the same Python when I need to do something more higher level and C when I have to do something a little lower level, I go back and forth.

I spend a lot of time in, in you know, the Arduino IDE working on ESP 32s and. And that kind of stuff. But I will say I have been trying to learn more of the Pico platform, which has brought me to VS code more and C and learning the back end stack that I need to get that stuff compiled and all that because because Arduino obvious gates, all that stuff.

Right? So you just like, yeah, there's Build it and upload it to my board, you know? And then it's like, Oh, wait for the Pico. I've got to know like what's behind here. So I've been learning that a little bit too. So yeah, VS code is going to become more important in my life than it has in the past, cause now I have a reason to use it.

Jonathan: have you worked with the platform? I owe any. Has that been on your radar?

Aaron: Yeah. Yeah, I did. I tried it. It was kind of, it was kind of a similar experience to me. It's like there's so many dependencies. And once you get it all set up and running, it seems to work. But for me, it was between Arduino, which I had been using and understood and could figure out and then going to platform, which was seemed like it was supposed to make it easier.

But for me, it ended up making it more complicated because things didn't work. When I would try to build stuff. And so I was like, I don't, I just, there was no reason to use it. And like, now there's just a lot of stuff in VS code that I'm just like, yeah, I can do my Python work here. I can do my C plus plus work here and I can, there's so many integrations and you know, all the stuff Randall was talking about.

So yeah.

Jonathan: I have, I have a recently. So I've been told that I have to answer this question too, because it's around table and that's fine. I've recently been doing a lot of work because I've been working on the mesh testing project. So I'm doing a lot of platform IO in VS code. And that's pretty interesting.

What are the, one of the cool things about platform IO is that it lets you define a whole bunch of different targets. And so, you know, we can, out of the same code base, we can support ESP 32 devices. We've also got it rigged up to where we can compile for native Linux, which is what I've been working on mainly.

NRF, the NRF 15. I think that 42, whatever the, the, the thing is STM 32 boards, all kinds of stuff you can do out of platform IO. So that's cool.

Aaron: Is it any easier to set up now than that was before? Because I swear I went through it and I spent a couple of days on it and I was like, if I, if I, if I have to spend a couple of days to get this thing set up, it's not worth it.

Jonathan: I think it's one of those deals where it's like, you want to copy a working config from somebody else's project that I find that's true though. So many of those different things that have boilerplate, like goodness, the first time I went to do an Android application, like the, there was just so much boilerplate you had to get in, get in play now, the, the, the.

Android studio made that a lot easier. You can just say generate new. But still, I think that's, I think for any framework, that's just going to be the thing.

Randal: Yes. Right. On LLMs, LLMs made it also simpler because pretty much that's the thing they trained on is so you'd say, give me a configuration for, and it goes, ah, I've seen that somewhere.

Right.

Aaron: Yep.

Randal: Yep.

Aaron: Speaking of LLMs Randall, I ha I hate to say it, but I'm gonna be going through just a test. It's more of an LLLM test than it's anything else, but I'm gonna be taking my last Running Pearl program and using chat GPT to convert it over to Python just to see how well it does,

Randal: does PT Gemini Advanced is smarter

It's been smarter in every test bed.

Aaron: Yep. But that's my last running Perl script. It's a recipe database that I built 25 years ago. Nice. Wow. For my wife. And now you don't cook,

Randal: and now you don't cook. So it's great.

Aaron: Oh no, I cook a lot. And we use the, we still use the recipe database every, I mean, multiple times a week,

it's,

it's crazy.

But at the same time, it's like, I don't remember what I did 30 years ago now. And so a couple of times I've had to go back in and fix it. And I'm like, crap, I don't remember how this works in Perl. It's Perl code.

Jonathan: So, you know, it's pretty much obfuscated already.

Aaron: Yeah. And I didn't want to call you, Randall.

I didn't want to take advantage of our relationship.

Randal: I've done very little Pearl. I have to now go look things up now. Again, I'm at that stage in your own words. Luckily that's online, but yeah, no, but I read the man pages once again. Although I heard that from Larry as well. So Larry told me about I don't know, about five or 10 years into you know, after the camera book had been published, that That he said yeah, I got to look things in the man page occasionally, because I forgot how I did this.

And it's like, you know, it's, it makes sense. I mean, it's, it's stuff. It's, we only have so much room in our heads. And now we got to worry about new things like LLMs and stuff. I,

Jonathan: I have been told that it's when you know that you're sort of an expert in your field when you go to Google a question and the answer, you find it in something you wrote.

Randal: Yeah, oh, that's that's happened plenty of times for me because I have 255 magazine articles online Yeah, and just the other day not about two years ago. I was trying to solve something for somebody For a client and I went I know there's a way to do this I googled and up came one of my columns. These are all online.

So it's like, Oh yeah, of course, that's how I remember now. I remember writing this. I've started, I've

Jonathan: started finding answers to my questions in my own Hackaday articles. That is a, that is a very fun experience. Yeah, definitely.

Randal: Good with being published, yes. Yeah.

Jonathan: Anyway, I don't know if they actually fully answered.

So if I'm in the command line, I like to use nano. And otherwise I use a lot of VS code. And then on the language. Very much like what Aaron had to say. I very much enjoy C these days. But for scripting, here recently it's just been straight bash code for pretty much everything I have to do. The fact that you can just glue commands together with pipes and all that is really powerful.

I have used a little bit of Amber. We interviewed the guys behind this and Amber is a really cool program. It is a compiler for bash script. And it lets you, the things that you wish Bash script would do that it doesn't do, the Amber language has it, and then it compiles down to Bash code, which is really pretty cool.

So if you find, if you find yourself really annoyed with Bash, but you want to use it still, Amber is something to take a look at.

Aaron: I'll have to go back and watch that show.

Randal: Yeah, it was fun. That's, that's similar to the reason that I converted from using Perl to using Dart for my command line stuff is that Dart actually compiles to local binaries, which Perl never knew how to do really well.

So that's really nice. All my apps are running much faster now.

Jonathan: Yeah. Yeah. Let's see. Do I, I assume that you guys have stuff you want to plug. We'll let

Randal: Randall go first. Sure. So I am again, pretty much heads down in the in, in, at Transparent apparently. If they OBS is being really mad at me.

Yes. So is in the, in the dart and flutter world. So I'm spending a lot of time attending virtual meetings, giving like Q and a sessions there. Sometimes they do presentations. My big presentation coming up is if you're a Pearl user or have been in the past there is the annual Pearl advent calendar and on the 3rd of December 13th, I'm doing a live presentation of a talk I gave seven years ago at OSCON, back when I was still going to OSCONs about half my life with Pearl, which is, at that point I was turning 50 and Pearl was turning, So it was like for exactly about half of my life, Pearl had been around.

And I talk about the early, early history of Pearl and how Pearl impacted me personally, impacted my company, how my company and the trainings we did impacted the Pearl world. So there's this incestuous, you know, loop in, in there that I'm able to kind of highlight some behind the scenes, sort of how did that all happen?

I'm doing it live. There's some sort of registration link that I don't have in front of me, but I can probably give it to you for the show notes. You don't have to register to actually be there. It's just going to be like a a zoom call. But I think they want to make sure they know how many people are going to be there to know what size Zoom package to buy for the one day or whatever.

So yeah I can't imagine seeing a hundred faces in the zoom, but apparently they're close to 100 already registered for this. So I it's gonna it'll be fun. It'll i'm not changing the talk at all, even though it's been I gave it seven years ago, mostly because I, I haven't really done anything else with Pearl.

So the talk really doesn't matter, but I'll, I'll speak of it as if it was the historical moment in time seven years ago when I was half my life, I had spent exactly the Pearl. So that's coming out. That's probably the only thing I really want to pitch in the Pearl world. And again, all dart and flutter stuff.

I have a, I have a YouTube channel, just Randall Schwartz, dart and flutter. You'll find it.

Aaron: Aaron. Well, as usual, you can find me at on YouTube, RetroHackShack and RetroHackShackAfterHours. I'm getting there's been a little bit of a delay because of my eye and I actually did a video on the main channel that talking about what's going on there as a little bit of a warning for people as well, so that they can avoid having to wear an eye patch like this all the time.

So I'm not going to go over it here, but if you want to know like what's going on health wise, you can go watch that video. But I finally got back into actually producing some content after a bit of a break. So I just uploaded a video to RetroHackShack After Hours, which will come out tomorrow morning, 6 a.

m. Eastern Time. And it's all about this really cool Vintage piece of electronics that helps me diagnose vintage computer monitors and TVs.

Ooh, wow.

And it's made, made by Extron and it came out in 2002. So it was right at that sweet spot where you had to, you know, there was no HDMI, you know, you had to support everything.

So this box supports Mac monitors. It supports IBM monitors. It supports Sun monitors. It supports SGI monitors. It supports Apple too. Monochrome composite monitors. And I test all that on that video. And so, yeah, if you're the kind of person like me who likes to pick up this old stuff and test it, but you may not have the machine you need to actually.

Hook up to it. Or the machine you have might not be working enough to produce a video signal. So you can test that monitor. It might be something interesting for people to watch.

Jonathan: It'll help you figure out whether the problem is in the machine or in the monitor.

Aaron: A hundred percent.

Jonathan: Yeah,

Aaron: but, but actually what I found out was that the box that I bought is old enough that the problem could be in the box because I had to buy two of them.

The first one didn't work. So I had to buy another one, which is going to help me fix the first one, but when it works, when it works, it's all pretty much in hardware. So when it works, it, it works really well.

Randal: Oh, that's great. All right. Hey, thank you. Thank you. You mean you mean Aaron? You're not still dressed up for September 19th and

Aaron: just didn't stop No, or halloween either.

Okay Halloween for halloween. I had a I had the eye patch on just because it helps me focus more on this eye and I had a a Shirt like this and I had a ghostbusters t shirt because I wear geeky stuff, you know I just that's just what I had on that day and some little six year old girl Oh When they came to the door, she's like, what are you dressed up as for Halloween?

And I said, I'm a ghostbuster pirate, of course, you know, and she had no idea, but the parents were back in the back and they were laughing. It's like, no, this is just how I look

Randal: these

Jonathan: days.

Randal: Know your audience. Yes.

Jonathan: All right. Thank you guys both for being here. Appreciate it. Jumping in at the last minute. And thanks so much.

Thanks, Jonathan. Yeah. Thank you guys. All right. Thanks. As far as my plugs do want to mention, we appreciate Hackaday being the home of Floss Weekly and you can find my security column goes live every Friday morning. There's of course also the Untitled Linux Show. If today was not enough Linux for you and you want to keep up to date with the, the latest news of what's going on there.

That's over at the Twit Network, twit. tv and be glad to have everybody there. We appreciate everyone that joined us, both live and those of you that get it on the download, and we will see you next week on Floss Weekly.

Episode 809 transcript

13 november 2024 | min

FLOSS-809

Jonathan: Hey folks, this week, David Ruggles joins me and we talk with Frank Delport about Py4j, a Java library for exposing GPIO and SPI, all kinds of stuff on the Raspberry Pi in Java. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 809, recorded Tuesday, November the 12th. Pi4j, stable and boring on the Raspberry Pi.

It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we've got something a little different, out of the box. Maybe outside of my comfort zone. But first we have a co host. We have Mr. David Ruggles, the David Factor. Welcome, sir.

Good to be here. I appreciate you kind of popping in at the last moment here saying, Oh, you don't have a co host yet. I can, I can do that. I could be that guy. I very much appreciate it. You're very welcome. Because I think, I, I, it's on me. It was my fault. It was a scheduling snafu that I made. But I think I was going to do the show solo if you hadn't popped in.

So, very much appreciate that.

David: It's going to be interesting. I don't remember which one it was. It was one of the recent ones I was on. It was about Mac and I ended up, oh, Brew! Brew I said I'm not a big Mac guy so many times it became a joke and, you know, now it's, and we ended up calling it the Whopper episode.

So, for this one, I'm not a big Java guy, but I don't know of any funny saying about that, so.

Jonathan: I am here to learn. The question may be, are you a Raspberry Pi guy?

David: Not, I own a couple. And I've run some code on one, but I just, It doesn't fit into my day job, and I don't have enough time for that hobby.

But I love the idea of the Raspberry Pis and that whole ecosystem, and I've done a little bit of Arduino stuff, it's kind of that same you know, software meets hardware. Bye

Jonathan: bye. Yeah, that's, that's the thing about the Raspberry Pi, and so, like, before the Raspberry Pi, there was the, the Beagleboard and Beaglebone, and before that even, there was the Arduino, of course.

If you go even further back, you have things like the BASIC stamp, so this idea has been around for a long time. But the idea of, well, let's take our, our computers that we're used to, that we interact with, and let's give them more ways to reach out and interact with the real world. And that's just the, that's the addictive part for me.

Like that you can, you can hang real hardware. You can, you could, you could do things. And so a lot of times that's turning lights on and off. That's actually where I really got started with some of this was, you know, okay, let's put a relay inside a lamp and then switch the relay. With the way that I first started doing it was the the relay was connected to the arduino The arduino could switch the light back and forth and then the arduino just had a super simple script on it That talked to the local computer That you know so that it could tell the arduino to tell the light to go off and on and then of course You know you you make a web page with a button on it so that you can you know Pull it up from anywhere and turn your light on and off and that's just you know That's geek nirvana.

Like that's that you made it when you could do that So today's guest is Frank Delport, and we're talking about Java on the Raspberry Pi, which, you know, is kind of an interesting thing. So I could, I could see this. With Java, you could do very, very quick sort of prototyping of programs. And there's so many libraries out there.

And then, you know, you bring it to the Raspberry Pi, and it, it is kind of this, like, really easy to get started, really quick prototyping platform, perhaps. So I guess I can see the point of it. Are you familiar with this project at all?

David: I looked at it a little bit enough to have a question about the change in versions.

Okay. That's about as far as I got, so I'll wait for our guests to ask that question.

Jonathan: We will definitely get to that. Let's go ahead and bring bring Frank on. Welcome to the show, sir.

Frank: Hello good evening from Belgium.

Jonathan: Huh. Well, good morning from Oklahoma.

Frank: And thanks for inviting me. You Java lovers.

Jonathan: I've, I've explained it before. My experience with Java happened. Well, one with a much earlier version of Java and two, a much earlier place in my programming career. So I am sure that some of the frustrations that I have would That I had would no longer be things because hopefully I know a little bit better what I'm doing now.

And from what I understand, the Java ecosystem has sort of come along and things are a bit nicer to work with as well.

Frank: Yeah, so people who are listening to this podcast and have a bad feeling about Java, if you're feeling dates from Java 8 or around that time, Please ignore those feelings and, and take a look at, at nowadays Java.

The fun thing is the Java from then still runs on the new run times, but the language has evolved a lot. The run times has evolved a lot of you can write much cleaner codes than you could do. A long time ago and yeah, debugging and the frustrations you have possibly are all gone now with better tooling and, but yeah, everything has evolved as we have so much better IDEs for all languages.

We have a visual studio code for people who want to start coding for free with a very good IDE. What's that? Plugins for all languages. So yeah Java has evolved a lot and, and I hope I can inspire some people who are, who have some frustrations to retry some of these and see what, what has changed.

David: In all fairness, my experience actually is even further back.

It was Java 6, I think the last Java that I touched. Yeah. As soon as I get copious amounts of free time, I will revisit Java. I've committed to that.

Frank: Okay. If you, if you're looking for a starting point, I also write blog posts for fuji. io, which is the website for the friends of OpenJDK, which is a community platform.

And on that Side, I have a small tutorial getting started with Java 30 minutes of video with all the basic functionality of Java which could be a good starting point if, if it's really new to you or you just want to see what the current state is.

Jonathan: Yeah, so let's, let's start with this question that I, I threatened that I was going to ask, and that is, why, why, who, who thought it was a good idea to put Java on the Raspberry Pi?

And of course I asked this very tongue in cheek but I'm sure you, I'm sure you get it a lot. Like, why Java on the Pi?

Frank: Well we go back, I think, about five years, In history, and I was doing Java development at that time for a company in Belgium, Televic Rail, who builds display systems for, for trains and communication systems for trains.

And we were also using Javavix. So those are, Javavix is a graphical user tool for Java. So those were my two tools that I was used to use to, to work with, and I wanted to create a touch based touch screen based application for my son for his drum boot. So it was then nine years old. So a little application on a touch screen where he could start some lights in his drum boot and, and some flushing let strips.

And I wanted to create that with Java and Javavix because those are the tools I know. I wanted to use Raspberry Pis because I had a few of them because I was I'm also doing a Coder Dojo, the coding club for kids. So we have some, some of the stuff laying around and, and, and they say, if you want to learn something new, you should focus on one thing.

And for me, it was the electronics, communicating with. electronics hardware from software. So I needed to use the tools I already knew, Java, Java, Vix, and, and, and Raspberry Pis. I've used that a bit. I used Linux before. So the new thing for me was communicating with, with those devices. And that's how I started with some basic experiments, blinking a LED.

The hello world in, in electronics and I found at that time that there was not that much documentation about running Java on a small PC like the Raspberry Pi. We go five years back. So there were not the Raspberry Pi five we have now, which is a full power device. So we had the Raspberry Pi three, I think at that time.

So I, I needed to find out a lot of stuff and experiment, and that's when I started writing blog posts and, and people who are watching this podcast as a video, they can see my book behind me here on the side. So I ended up writing a book which landed me a new job. So it's, for me, it was a personal journey of, of combining software and hardware.

And Java is my language, the language I use the most and love the most. So that's why I thought that Java on Raspberry Pi is actually a good, good idea. And during that journey, I, I bumped on Pi4j, which is a project which is already I think it started in 2012 a Java library to make it easy for Java developers to communicate with the GPIOs of the Raspberry Pi.

So interact with electronic devices and, and, and, and in those, those last years, I got involved in the, into the Py4j project itself. Thank you. And that's why I'm here to share this message and this love for electronics on the Raspberry Pi.

Jonathan: Yeah. And, and so, you know, with the, with the Pi it's kind of been interesting because the, the, the Linux kernel in general, the Raspberry Pi, it seems has sort of dragged some of these things into the kernel, like not necessarily GPIO access from the kernel, but exposing that to user space.

Like from, from what I can tell, that, that subsystem Was created because of the Raspberry Pi and other devices like it. So how much, how much of this worked when you got involved with Py4j a few years ago? How much of this was already there and working and how much, how much stuff did you bring to it?

Frank: Everything worked actually. So, so on the Raspberry Pi, you had SPI, I2C PBM, all those protocols. You could use them on the Raspberry Pi from Java. And how that was done was by combining the Java code with WiringPy, which is a library At that time to bring this GPIO to different languages also to Python.

So that was actually using wiring pi. And and that's the same thing that still have is happening. So from pi for J some C code is used. to interact with this GPIOs through JNI, which is a way to include C headers in Java code and being able to call this. And what happens is, is when I found this project about five years ago, It, the original creator of it, Robert Savage, an American guy, he was working on a new version to bump it to newer Java versions and new architecture and modern Java, let's say.

But he got involved into other projects, his personal, his work projects. So he had something which was nearly finished. But not completely finished. And that's when I joined. I mainly focus on documentation, writing about this, telling about this. And then some other people like Robert a Swiss guy, Robert, we call him in the project.

And then Tom arts, who is also an American guy who, who creates a lot of example projects. They also joined the project. And now Robert, the original creator moved out a bit. And that's what happens in, in a lot of open source projects.

Yeah.

People start it, then get in, yeah, get a new job, get, get other priorities.

And we are lucky with Pi4j that we have this community joining in. And like you said, the kernel support or, or what Raspberry Pi is doing inside the operating system to be able to interact with this, with this GPIOs shifted a lot now with the new Raspberry Pi 5. They have this new chip the RP1 and that chip is responsible for the communication with these GPIOs.

And what happened with Pi4j is we assumed that it wouldn't work as is when the Raspberry Pi 5 launched. So it took some time before I had the first device here. I tested it and didn't work. So we created the ticket. We need to find out why. And, and the problem is pretty clear. You have these, these C class files we include to interact with the GPOs and they were not compatible with this new chip and lucky, luckily this is an open source project.

So at some point someone said, Hey, I have a pull request for you, which brings RPI5 support to Py4j and it needed some fine tuning. Then, then Robert Eich joined again into this, this pull request. He fixed it. Then Tom did some improves. So we had this whole thing. Suddenly we have support for RPI5 without planning it, which was really great.

This is a problem with, with this, with this new boards. Yeah. This suddenly appear I guess the compute five is already. Available for some people. We don't have it. So we don't know if it will be compatible. We hope so. The

Jonathan: response that I've gotten to that is we can neither confirm nor deny.

Frank: Yeah, everyone is waiting for it.

So yeah, it will be there. But but that's yeah, that's how this project evolves. And it's nice to see that that people love it. Because they are Java developers. I want to experiment with this. I talked about this at several conferences and then people are building very fancy stuff with it. We know that there are some companies who use it, but don't talk about it because they are not publicly used.

Please. Telling we actually just buy Raspberry Pi fives and put them in a fancy box. But yeah, those things are happening and that's also the disadvantage of open source. It's there. It's on GitHub. It's a Maven library. It's a Java library. You can get from the Maven repository, so anyone can use it without telling us.

It seems pretty stable because we don't get a lot of issues. Issues we get on GitHub also get solved very easily by the community again, so, which is really great. And this is, yeah. How this project is, is evolving and, yeah, for me, so many years ago when I started doing this, the blinking let's.

Was after five days of trying out to get Java on the Raspberry Pi and then having something doing something Yeah, that was my aha moment. It works on my machine. I can blink let's which is no rocket science at all, but Once you get beyond that point and you know, yeah, I can use this library and I can communicate to the GPOs.

The next thing is, yeah, a button and a relay and, and, and an I square C display and we can control everything. And another fun thing is, is a Swiss university, the FHNW university FACS school and something in, in Switzerland, they use. Pi4j, Raspberry Pi, Java, in their education system. So, so hardware and software engineers work together on projects where they build some kind of, like, for instance, the crazy, most crazy example is, is you know, the, the shortest route solution.

So you have a driver and he has to go to point B and he has to pass through all these points, calculate the shortest route, which is a typical software problem to solve. Now they combine it with the hardware. Student who has to build this on a 64 by 64 let matrix where they put some sheets of paper on top of it with different routes and they have to match and then they turn it into a board game.

You have to compete against the computer and you get a few seconds to indicate which is the shortest route. And then the computer tells you if you're right. So they have this whole combination of software and hardware, and they use Py4j and they contribute back with examples and with documentation changes.

And they built I don't know if you know the CrowPy, which is a little suitcase. With the Raspberry Pi and all electronic components as an experimentation kit. And again, they contributed all these example code back to the project. So it's, it's, it's fun to see if you have a, have a, have an interesting library project on, on, on open source on GitHub and you get some traction.

That it starts living by itself.

Jonathan: Yeah, some interesting stuff there. So, was Pi4J sort of originally made like for education? Was that the scope originally for it? As far as I

Frank: know, Robert just made it because he needed it himself. Also as a Java developer, he wanted to use, and at some point Oracle who, who took over the Java brand.

So they are the owner of the Java brand and, and, and also steering this, the evolutions within Java. At some point they also started something like that. So at some point there was an idea to bring this officially as a part of the, of the core of Java. Oh, interesting. And Rob, and Robert was working on his thing and he thought, yeah, at some point I can stop my project because it will be in Java.

But Oracle has taken the part of, of, of Java. Backend server stuff, databases Java fix, which is another love of me to build user interfaces. Also has bits. Yeah. Went, went on the sidetrack because of that, because they focus on the server side. But Java was originally born on embedded devices. The first Java.

Was something which ran on, on set of boxes on, on refrigerators on, on, on car systems. So that's where Java was born, but it's, yeah, it, it has this promise of, of right ones run everywhere. So yeah, it runs on the Raspberry Pi. Of course it runs everywhere.

Jonathan: Is, is Pi for J limited to running on the official?

I don't know if it's the official Raspberry Pi OS or if you throw a different distro on your Raspberry Pi, like say Fedora, does it run there as well?

Frank: I have tested Fedora, Ubuntu, that's no issue. I should try it again with the Raspberry Pi 5. I'll put Ubuntu there. I'm not sure. It depends on a few of It depends on what you want to use of the GPIOs because if it's just an input or an output or a PBM or, or you want to use I square C, those are different.

Mm-Hmm. providers we have within PI four J. And if you start it, if you start the Java application with the PI four J library, it first starts detecting which version. Of the Raspberry Pi board, am I on? Because Raspberry Pi 5 has different, needs different native code than, than, than other boards.

So it needs some way to define on which board it's running. So we, we should try it on Fido. I, I have no idea.

Jonathan: It, it might not be a problem now. I just, I have memories of back several years ago when I, it was actually Fedora. I just first started working on the Raspberry Pi and actually having access to the GPIO, which was great, except if you tried to use, in this case it was Python, if you tried to use the same Python libraries that worked on the official Raspberry Pi OS, distro, it just didn't work because the way the upstream kernel implemented GPIO support was different than the way that Raspberry Pi was doing it at that time.

Frank: yeah, so yeah, probably the same native C code, which is missing somewhere or, or yeah, not compatible with, with how it works. And

Jonathan: I think in that case, it was actually the, the kernel drivers themselves. I think that was that was right as the slash dev slash GPIO chip. It was coming online and making it into the kernel.

Yeah, an interesting time. David, you want to, you said you had a question that you wanted to get in. Let's let, let's let David have some fun.

David: Okay so I was looking at your website a little bit. Well, not yours specifically, but the project's website. And I actually went down into the GitHub repo.

And we hadn't really talked about the community around this yet, but I noticed that you are number four in contributors, so that is always a good sign. That means you've got other people committing code and supporting the project. So, and it looks like you've got about 20 committers. But. The actual question I had was around the versions.

Version 2. And 1, yeah. Yep, Version 2 though, you're starting to, or you did develop a new plugin model. To kind of, I guess, allow, More third party integration and stuff. So what is really your catalyst in moving? Yeah,

Frank: so that's dates back to the history of Robert Savage and the creator of the project.

So version 1 is what he initially started in 2012. It's based on Java 8. And it is a massive, big project. It included implementations for specific devices like that kind of LCD display, that kind of IO expander, like this chip, this number of chip, which made it very difficult to expand, change something in the, in the project.

And also difficult, yeah, to maintain it. So, and, and, and have people contribute some, something back to the project. So that's why at some point Robert together with some other people from, from other projects, he looked back at how did I design this project? How do I maintain it? And is there a better way to do that?

And I was around the time that also Java 11 became available. And for people who are not really aware of the history of Java, there is a big change that happened after Java 9 with how Java is architectured. So there is quite some change between Java 8 and 11 and then what happened since 11. So once you're on Java 11 on modern Java, as you can call it And what he started doing is he started working on a new repository on version two, which was Java 11, based on Java 11, and it all, the implementations for specific devices for specific electronic components got removed.

So the idea of this version two is the basic IO functionality that is needed to interact with components. So that's the core of PI four J really provide methods. to interact with the GPIOs, with all the different protocols, I2C, PBM, all those SPI, all those protocols are there. And by providing and building in a modular approach, it is easier now to extend it.

And that's exactly what happens with GPIOs. With the RPI5, with the Raspberry Pi 5, thanks to this modular approach, someone was able to say, for the Raspberry Pi 5, we need a complete different provider, which is compatible with the Raspberry Pi 5. And I can do that because I can just create a new module in this project, which is compatible with Raspberry Pi 5, and implement this specific functionality.

See? And it was very easy to test that because it didn't break any other code. But because the project changed so much and it was in a different repository, we decided to create a new repository, which is, it was a big question at the time, how should we do this? Should we clean that existing repository and put a new code in place?

But I would also break the whole arc history. So it was a bit, yeah, we had a question there, and at some point, yeah, you have to decide. And we decided to keep them as separate repositories. So version one is still a repository on GitHub, but it's archived, so you can still look at it, but it won't change anymore.

And now we have this version two, and now we are again at the point where we have to decide to move forward from Java 11 to a newer one. So, but we will stay in the, in the current repository. That's for sure. But there is we have an open issue about that. The idea is for instance, to, to bump the version to version three, for instance and then have some, some forks where we can.

If needed, go back to add security fixes, but do some breaking change that we're not compatible anymore with, with all the Java versions. But that's something which is now ongoing. It's a discussion which is now open on the Py4j version two discussion list on GitHub. So people who have some opinions there, they can definitely join and share how they think about this.

And, and that's also what we do with this GitHub discussions. They ask people what they think about. What should be the next step? Like support for cellular communication. There are, there is a very good Java library, which does only this serial communication. And we have some basic implementation in Py4j.

Should we keep it or just tell people use this library for this specific use case? And that's also how we use, use the discussion list a bit. So makes, make, does it make a bit clear what we did with the versioning there?

David: Yeah. Yeah, I think it makes a lot of sense to because I've, I've run into that situation with repositories myself where you don't want to lose all the history, you don't want to lose those commits and the logs and everything else, but it's a big enough change that it doesn't make sense to continue that.

So, yeah, that's really cool. like

Frank: if you now would go from Java 11 to Java 21, which is now the current long term support version, it's also already. One year old. So it makes sense to bump to this newest version. But actually that kind of change is just changing some version numbers. In the config files, it doesn't break or change any code.

What will happen if we bump to Java 21 is that we could dive into the code again and write some cleaner code in some places. Java, the language got some new methods which makes it possible to write cleaner code in some cases, like where you have a lot of switch statements and stuff like that, and text blocks.

I don't know if there are any A lot of text blocks in Py4j, but that kind of code can be optimized and can be written in a cleaner way now. So if we bump to this Java 21, which is just a config change and then in the next version, we could go into the sources and decide, yeah, this code now can be come cleaner and we should clean this up.

And in Java 21, I think we also got a new way to talk with C code. So, and, and there are more changes coming in Java about foreign memory, how you can write into the memory, which is shared with some C code, for instance, which will become a lot easier. Those are the kind of changes which are coming to Java inspired by what's happening in machine learning in the eye.

You want to be able to interact with C classes with C code, which is very good in doing this kind of stuff. And, and, and these foreign memory APIs, which are coming to Java, they are inspired by that goal, but it's something we can use to interact with the GPIOs also. And, and so. If we evolve to newer Java versions, that's also how Py4j can keep evolving.

Jonathan: That's actually interesting. Something I have on my list here I wanted to ask you about is the way that the question was going to go is like this, so it, does Py4j have support for the Raspberry Pi camera inputs? And then I know something that people are doing with those, it's interesting, is like AI acceleration to be able to make sense of what the camera is seeing.

Is that, are those two things that are in Pi4j yet? Are they on the roadmap? They

Frank: are not yet. We have got a question very recently, indeed. Can I access the camera? No, it's not unless someone stands up again and says, yeah, I know how to do this. As a matter of fact, I do. I have this, actually for the people again watching at this podcast, you're looking at me through a Raspberry Pi because I use Raspberry Pis as my camera.

Oh, interesting. I have an ATEM Mini to switch between different cameras. So I have other stuff lying around. But I use Raspberry Pi cameras and I want to create a little Java application that I can run on my desktop that I can zoom in with this camera and stuff like that, but yeah, it doesn't exist in Java.

So I'm a bit stuck. Yeah, of course. I could look into the Python version, but I'm a bit that's

Jonathan: heresy.

Frank: I want to do this with Java. So yeah, no, no, we don't have camera support, but it's probably possible. And there are a lot of fun stuff, things you could do at that time. I'm wondering if we should add it to Pi4j itself, because we're really focusing on those 40 pins.

And, and the camera is something different. I guess there are also already enough of Java libraries who are doing stuff with, with images. So, yeah, I don't know. It's, it's something to think about. Yeah. I'm

Jonathan: trying, I'm trying to remember. If the Raspberry Pi cameras, if they show up as like a V4L2 device instead of Linux.

Frank: I don't know. I only played with this, what is it called? Pi camera? What is this? The, the, the, the, the lip camera I think that you can, because that's what I do with this thing. And so I, I just have a little service that starts the, the camera as a full screen. thing on top of everything else. And I just use the HDMI output.

So that's what I'm using, but yeah, that's, that's very basic coding. It's somewhere hidden on my block somewhere there. You can find how I done this because yeah, that's also what I do if, if I experiment something I'm the kind of person that forgets what he's done one hour ago. So then it's fun.

Yeah. Then I have to write it down and it's fun to find back your own that can also be frustrating. I, when I started experimenting with Java on the Raspberry Pi, I always came up on my own questions or on, on Stack Overflow. So I was, yeah, going back to the same problems over and over again.

Jonathan: So one of the most fun experiences I've had is googling for a question and finding the answer in my own Hackaday article that I had written and forgotten about.

That's, yes, that's always fun.

Frank: That turns, that proves that you're an expert in that topic because you had the problem, you fix it, and then you had it again.

Jonathan: Yeah, I guess. Okay, so you have support for the Raspberry Pi devices. What about, what about some of the other. Single board computers out there, do any of the, do any of the Pi4j libraries and does the, does the code run on, say, the Pine64 or the BananaPi devices or the OrangePi devices?

Frank: That was the original idea. It got a bit abandoned in version one and it got totally abandoned in version two, but again, it's a modular approach architecture. So there is a provider in there, which is called the Raspberry Pi provider. So that means that, that when the application, the library starts, it detects I'm running on the Raspberry Pi.

So I'm loading that one. If you execute your code on a Windows machine, for instance, or a Mac, it will load the mock provider. So you can toggle an IO. But actually nothing is happening because the provider just returns. Okay. So we got the question recently. It is an issue again on, on, I have to look it up for another board.

And again, we have the same message for everyone. If you really need it, it's an open source project. We accept pull requests. We cannot support it with the people we have now in the team. We cannot do it because we are, we are Raspberry Pi people. That's the device we are using. If you have another device and you are, you know how to do this, then please, yes, join us.

And, and, and I'm looking in the issues, but I don't find it back. Immediately, which port it was. But that's exactly what, what we want to achieve. Yeah. Little issue we have is, yeah. How do they call these contributions to an open source project? Drive by contributions. Yes. Someone really needs this as this was a project and then it's gone.

Yes. And if you then need an update, yeah, you're stuck. So. I don't find which port it was, but indeed yet it's, it is a question which we got already a few times. And at this moment we are focusing on the Raspberry Pi. If someone has a good idea on how to implement this, the problem with that board was that for instance, the GPIOs were in different headers or different sections or different addresses.

I don't know. So it was not How it is now implemented in the Py4j library. It would become a bit complicated to edit, but it's codes. Anything can be done at the end, but someone has to do it.

Jonathan: You have to have somebody that cares enough about it to want to go in and make it all you know, generic to use the GPIO chip interfaces.

And then you have to have that person care enough to stick around in the project to answer questions and maintain it.

Frank: Yeah, and maintain. And I think with the modular approach we now have, It's pretty easy to add it and to keep it stable. We don't need to, the way how, how electronics are, are, are programmed or how you interact with them will not change much.

SPI is there, I2C is there, serial communication is there just simple input outputs is there. There's not much which will change in the field of electronics. So it's not that we will need to invent something completely new within the library that will need to be implemented for Raspberry Pi for all the other boards, but still the library keeps evolving.

We have some improvements. There were quite some I2C improvements and SPI lately. So yeah, those changes can go back to those specific implementations for. Raspberry Pi and other Pi types. So yeah, we need people to stick around indeed. And that's a bit, yeah. I think a lot of open source projects have that issue.

If, if people join for a short time and then yeah, that specific topic, and that's for people who say that Java doesn't change fast, that's actually Also the reason, and, and what we tend to say within Java is Java is boring and boring is good.

You, you have

fun in your private time, have fun with, with things which break whenever they want to break.

But within a company, you want to have stable stuff.

If you

start your application tomorrow, it should be fine. They behave exactly like it did yesterday and, and, and that's exactly what they want to achieve and what they achieve with Java. Java evolves and at, at this since many years now, we have a new Java version every six months.

There are a lot of evolutions within those new Java versions, but you can always run your old Java code on the new runtime. They're like, like, if you went from Python two to three, no way, but that's exactly what Java wants to have. I still have

Jonathan: PTSD from that transition.

Frank: Yeah, me, me too. And I didn't do Python development teams.

I only have some, some small tools on, on Python, so and that's what Java wants to be. It wants to be stable and boring.

Jonathan: Yeah.

Frank: But on the other hand, it is evolving and it's evolving pretty fast, even Stability in mind, and if you want to contribute something to OpenJDK, actually Java is OpenJDK, that's the open source project it's not that easy to get something in.

And that's the reason they don't want this drive by contributions from someone who has a very good ID. But then, yeah, who is going to maintain it? And, and Yeah, I think that's something we can learn from the Java, the OpenJDK project, definitely.

Jonathan: So speaking of OpenJDK, and I was thinking about this in terms of the kernel as well as Raspberry Pi, do you guys have lines of communication upstream?

Do you have a guy at Raspberry Pi Foundation or a guy at Java or somebody at the kernel that, you know, you could shoot an email to when something breaks and And, and get some opinions or,

Frank: no, since, since I started writing about this, I'm trying to get into contact with, with the Raspberry Pi Company, foundation, whatever.

I contributed a few articles to Magpie Magazine. Mm-Hmm. . But they're not that big fan of Java. Hmm. I know . But for, for reasons but the same thing like they had in the past, some, some, yeah, problems with migrating to new versions. And so, yeah, maybe that history is also from there. Do I have a line with, with Java?

Yes, I'm a Java champion. That means that I I'm one of those 400 people who are community. Selected by the other Java champions to, to, to, to talk about this. I'm definitely no OpenJDK specialist but I know some people. The fun thing is I work now at Azul, which is one of the OpenJDK distributors.

We built OpenJDK versions and, and runtimes. And, and just one example we have a project crack, which is aiming to have very fast startup of Java applications. And when we first announced this, I think one or two years ago, I immediately got the question as being the Java and Raspberry Pi guy, does this work on, does this work on Raspberry Pi?

And I tried it and it failed. For two reasons, the library itself was not compatible, the Java version. But there was something missing in the kernel of Raspberry Pi. And then I found out that, that if you just stand up and make a, make a, make a ticket on GitHub and clearly explain what's happening and what's missing and what's wrong, you don't have to fix it yourself.

If you just are able to tell, this is what I'm missing and you can do it there. It was just something, some, some something which needed to be enabled in the kernel settings, whatever. It wasn't the next version of the Raspberry Pi operating system. It was fixed together with 6, 000 and something other fixes, because I didn't know that the kernel was that big of a project.

And so that was also the first, very first time in my life, I compiled the kernel myself. Which is apparently something you have to do if you want to be a real developer. Yes,

Jonathan: yes, of course, of course. I'm glad to hear that you as well had that experience interacting with the Raspberry Pi Foundation, the technical support there, because I've had a similar instance where, and in my case it was something that the documentation said you can do with the device tree overlay to do to do fun things with the SPI chip select pins.

But and I raised an issue on their GitHub and like, Hey, your documentation says this should work. And from what I could tell, it doesn't work. And it did not take long. And they got back to me like, You're right. That doesn't work. Here's the problem. We'll have it fixed in the next version of the kernel.

I was like,

Frank: Wow. And that's, that's, you know, Jeff Gehling. I think the Raspberry Pi video creator.

Jonathan: One of, one of many, but yes.

Frank: One of many, but he's really great. Yes, he is. And, and that's exactly his message. They're all. Many boards like the Raspberry Pi, there are way better boards and Raspberry Pi for the same price, but the effort you need to take to get them running compared to the Raspberry Pi, yes, that's miles apart.

I have, I have the same experience. I have a few boards, only a few that I tried out and just finding An operating system and how you put it on the cart and how you put it on the device is already a problem and that's, there are many things you can say about Raspberry Pi company foundation and their ideas and how they are evolving, but the tooling they create and how they, how easy they make it.

To use these devices and what you can do with it. Yeah.

Jonathan: I ironically, I was, I was on discord last night talking with Jeff about this exact topic, because there's, there is another, there is another board that we both have copies of. We. But we both got sent copies of them to review and in fact, my review of it is hopefully coming out soon.

The review, my review of this thing is like six months past due, because putting Linux on it was fundamentally broken. And as far as I could tell, nobody cared about it. So I just, it's like I, I can't install the, in this case it was Fedora. It's like I can't install Fedora on this thing because the device trees were just goobered and nobody seemed to care enough to try to get it fixed or, you know, maybe, maybe these people knew how to fix it, but they didn't care enough to put out a You know, an actual guide where it'll tell you now to their credit.

I did this, the discord I was in just explain the program, the problem that people are willing to help. So like, it's not like it was complete radio silence. Yeah. But

Frank: it's strange. Why do they make this hardware? I guess they want to sell it. So they want people to use it. So why, why don't they? Try a little bit harder.

Jonathan: I mean, so in this particular case, you've got, you've got Rockchip at the top that actually makes the CPU and Rockchip, I learned last night actually that there are four kernel maintainers that are at rockchip. com email addresses. So it's not like they're entirely disconnected, but. The amount of effort that Rockchip puts into making things work is not as great as it could be.

And then you've got these small companies that repackage these chips for, you know, for various boards. And they're small, and they do not have an unlimited budget to be able to make things work in the kernel and in uBoot. And so, and then, you know, they sell their chips and they invest the money into trying to make it work, and the money runs out.

And they get to this point where it's like, we can't do anything more for you. We're sorry. Good, good luck. And which I mean, on one hand, like financially, I understand where you come to that point, but on the other hand, that's, that's not a great experience for your users. And Raspberry Pi is just, I don't know, I guess that's why they're the unicorn.

They've, they've managed to make that work. And they are slowly, but they are pushing things up to the kernel. It's just. The, the raspberry pies are such a pleasure to use .

Frank: And, and definitely when you combine it with Java, . And, and I wanna come back to, to one of your earlier podcasts. I, I've written down the number 8 0 1 mm-Hmm.

You talked with the creator of J Bank. Mm-Hmm. . And, and yes, if you want to build a Java application, a full size Java application, you need tools, you need Maven, Gradle, something to build it. To compile it, and then you have someone. Some, some tool like JBang, which takes all of this out of your hands and, and, and within the Py4j website, I have a getting started examples based on JBang.

So what is JBang? You write a Java file and a set of Java. And then the name of your file, you do jbang, and the name of your file, and it will fetch dependencies so you can use libraries it will compile it, it will make sure that Java is installed on the machine, on the Raspberry Pi in this case, so it will do all this for you, and jbang installation is just one line of code, you have one script you have to execute in the terminal, so once you have jbang, you can execute this Java code.

And, and so I have created a few examples where you don't need Maven or whatever. You just need this one file, this one Java file, and it will blink a let, or it will listen for the, for the button, or it will, what did I create with, with a little LCD display, I think, and, and then not even using Py4j but the serial the serial library that JFost serial, I think to communicate with a little microcontroller to control LED strips.

So I have all these examples are on the py4j. com website because they make it so easy to get started with something even create a Javavix user interface application with just one file. One file, all your code in that one file, which is a bad coding practice for Java developers, having everything in one file, but to get started, it's really great.

So you can have all these, this, yeah, pretty complicated functionality in one file. And that's what I love about, yeah, for instance, JPEG, which is a great tool.

Jonathan: Yeah, I'm going to let David get back to, I'm sure he's got some questions queued up. But I've got to ask you about this. And this is, so I think, I think this may be Jeff actually that first suggested this.

And it, I've just loved the idea ever since the new RP1 chip. So that is all of that GPIO and SPI and I squared C. off of PCI Express. That is how the RP1 talks to. And so, the idea is, well, why don't we just take the RP1 and put it on a PCI Express card, and put it in desktops, and maybe laptops. And I don't know if, I don't know if the Raspberry Pi Foundation, I don't know if, If I don't know if you're willing to do that, if Evan, I don't know if Evan is willing to do that, but that would be really cool.

And I have to imagine that if that happens, you guys are going to be right there saying, Hey, look, you could, you could use Java with this thing. Yes, because

Frank: we already have the implementation of, of what's happening on the RPA one. Exactly. It's just sending the data to it. That's also why I love this Raspberry Pi.

I'm, I'm, I'm from the generation from the Commodore 64. So yes, I'm that old 40, 40 years ago. But at that time I found the book in the library as was 14 years old, I think about how you can interact with electronic components with the Commodore 64. So And with the book came a print board for eight relays and I had to go to a shop to buy all these relays and all the connectors and I had to solder it and I've never done this before and I fried my my, my Commodore 64.

It was making this electronic buzz when I started it and it didn't do anything. And I unplugged it and tried it again and it worked. And from that time on, I was controlling my Lego trains

Jonathan: with

Frank: basic codes. And some of the less on my, on my Commodore 64. And then we had all these fancy PCs. And the only thing you had was a keyboard and a mouse and, and now you have this Raspberry Pi four, which is way more powerful than this Commodore 64 was for a price.

I once calculated the price of a Commodore 64. I think it's, it would now be 1500. Euros or dollars calculated to current price and you have a RedBull pie for 50. So you have so much power in this, this little device and you can control whatever device you want to it being less or less or displays or, or, or yeah, let's trips.

Which is a difficult topic, but even those kind of stuff you can control with, with, with, with Java or whatever code you want. So the idea of, you know, that the GPIO header of the Raspberry Pi was actually an accident.

Jonathan: I did not know that.

Frank: I know, maybe it's not true, but when they were designing the first Raspberry Pis, they had this chip and they connected it.

And the idea was let's build a computer for everyone. Everyone has already a television. So they put an analog TV connection in it, so nobody had to buy a new one. And then they find out, oh, but this chip has some IO pins and we are not using them. Let's just put them on a header and see what we can do with it.

And those first headers only had the IO pins. 20 connections, I think.

Jonathan: Yeah, it was, it was different. It was different. So

Frank: the header evolved also, so now we have these 40 pins, but to me, these pins, those are for me, the real differentiator between any other computer and the Raspberry Pi. Of course, you can also do a lot of other amazing stuff when you just use software, but those pins are really.

Yeah, those are great.

Jonathan: Yeah. I don't remember if it was during the show or before we started, but the, the Raspberry Pi, it's, it lets you reach out and work with the real world and it's, you know, it's, it's similar to your story with the the controlling the model trains with the Commodore 64. For me, it was the first, the very first thing was being able to switch a light on and off, and I still think that's great.

So I now have smart switches that have custom firmware on them so that I can, you know, I can I can push buttons here on my desk and turn my lights down and on, you know, and it's just, I don't know, it's the, it's the best thing. It's just the best thing to be able to have a computer to reach out and actually do things with the real world.

Frank: Yeah, and that's, yeah.

Jonathan: David, you want to jump in? I'm sure you've got you've got some stuff queued.

David: Well, I was actually going to ask about J Bang, but

Jonathan: he beat you to it.

David: I exactly. But so the other Hackaday thing that you tied to was J Releaser. Yeah. So how does Py4J and J Releaser work together?

Frank: So J Releaser is a tool you use within your Java build chain. At some point it jumps in and it can create a release and, and, and create release notes and that kind of stuff. And you also had them in the, in the, in the podcast, indeed. We don't use it for Py4j, the library itself, because we already had a build flow there.

But we use it from, for some other libraries and tools. So it's part of our GitHub actions. I, I have to be honest, I really hate pipelines and, and building stuff. And, and this, for me, this is the boring stuff. I want to build new codes but packaging and distributing it. And like live Java libraries, when you have a new life Java library, you have to to be, to, to make it available to others.

You put it in the, in the Maven repository, which is a public thing. This whole flow. is boring. And once you have it set up, it will run forever. And that's exactly what we do with JReleaser. So we have it in a few of the other repositories in the Py4j website Py4j GitHub projects where we use it to create releases, but it's really amazing tool.

It's, it's again, something which those are the best tools, tools that people need for themselves to automate the boring stuff. And then suddenly they find I other people are also interested in this, and then it becomes better and better and better. Mm-Hmm. . And, and, and that's exactly what happened with Bo Jang and, and j Releaser.

They are such an amazing tools within the Java ecosystem. And yes, I using where they are meant for, to, to automate the boring stuff and make it easy to, to publish a new version of a library in the, in the Maven repository.

Jonathan: So I've gotta ask, what, what license is PI four J under?

Frank: It's yeah, that's a very good question.

One of the free ones what is it again? It's Apache 2.

Jonathan: Okay, that's a, that's a permissive license, I think, right? Like BSD celled?

Frank: Yeah, I'm, I'm not familiar with all these licenses. There's a whole discussion about licenses I once had when I started a new Java project. What is the best license? It depends if you ever want to make money out of it.

It depends. That's, that's the big question from the start, but it was already there. So this project is not meant to make money out of it. Not the library itself. You can hire me. No, that's not the idea. Yeah, the license is there. We're never going to change that because I heard from, I think, one of your earlier podcasts is, is you have to go back to all your contributors and ask for their approval.

Yeah. Ask for their approval. So we're never going to change that, I think. And I think this, this, this license is free enough for the project that we have.

Jonathan: Yeah. Okay. So the Apache two is a, is a permissive license. There are few restrictions on the use of the code. It is not a copy left license. And there's like this whole flow chart of like what you could do with licenses and what kind of code you can include in other projects.

And not everybody gets it right, which is unfortunate. But yeah, for, for a library like this, honestly one of these permissive licenses is really what seems to make the most sense.

Frank: I think so. Yeah. If you just look around, what others are using. That's what I do within certain new projects for myself is how do others do this?

Yeah.

Jonathan: Okay. I think I know the answer to this already, but is there any support for using the Py4j on say the Raspberry Pi Pico or any of the other embedded targets?

Frank: No, because the Pico is not a Linux system. So yes, it works on the Raspberry Pi zero. The small ones, which are also a full Linux system.

The problem you have there is there is not a lot of memory. It's 512 megabytes, but you can I have Java Spring applications running on the Raspberry Pi 0, 2, the 0. 2, which has, I think, the chip of the Raspberry Pi 4 or something similar. So the chip itself I know it's

Jonathan: 64 bit. I know Yeah, yeah. I've learned that one the hard way, that it's a 64 bit and the early ones are not.

Yeah.

Frank: So the, the, the last recipe by zero is actually, again, a very powerful device, although it doesn't have a lot of memory, but it is very powerful. So yes, you can run Java. And then the Pico is, it is a Raspberry Pi, but this is microcontroller. So it's more on Arduino. Then it is a Raspberry Pi, if I may say so yeah there are projects I know that compile Java to microcontroller codes.

Jonathan: I was going to say we have MicroPython, surely there's a MicroJava.

Frank: There are things like that. But I didn't try them. I should write this down again. That's another one for my, my, for my to do list. It would be really funny if we could just. Make one video as an example of yes. We are running Java on the Raspberry Pi Pico and then see what happens at the Raspberry Pi company.

But yeah, I have no idea what the status is of it.

Jonathan: Alright, so what's coming, what are we looking forward to next with Pi4j? What are some big things on the horizon?

Frank: The big thing will be the move to a newer Java version. So that will make it easy for us to implement new support for yeah, maybe Raspberry Pi 6, although I think that will be the same RPI.

So that we don't, that's the idea I think of, of the, of the company to have this chip now as, as the, the central point of communicating with USB network GPOs, all these things. So that will probably not break. And then just. Seeing what appears as issues for the moment, the main changes are bug fixes like better support for SPI, I2C, some things that were discovered by users, and, and we are lucky again that some of these users fixed the problem themselves.

With a pull request. So that's really, really great. And thanks to all those people. And for the rest, what we are trying to do with the website is we want this website, pubvj. com, to be the starting point for Java on Raspberry Pi. Even if you don't use the library, how to install Java, how to install Java on a Raspberry Pi 0 of the first generation.

Which has a 32 bit on V6 processor, which is an issue. I have to try that out again with the latest operating system because I think there's an issue again there, but I got, I got it working for, for many years. So keeping that website up to date having all the information available there for people to get started, we get.

quite some issues being reported, which are actually returning issues of, I didn't start the right way. So that means that something's missing in the documentation. And that's what I try to focus on is, is keeping these websites as good as possible with all the information. Like if you want to use a Java VIX user interface application on the Raspberry Pi, what do I need to install?

How can I run this? So that's what we want. I'm now working on a J bank script. To update the, the, the wallpaper, the desktop background of the Raspberry Pi that you, that you see the IP number and, and, and which version of Java is installed. Just a little helper thing, but yeah, this exists of course in Python, but.

With Py4j, we want to have this in Java, of course, so let's do this with JBang, and I have it running, I just need to clean it up, commit it, and document it, but those kind of stuff is making developer life easier for Java people who want to experiment with this.

Jonathan: Yeah, so I've got to ask, what was the, what was the craziest or most surprising thing that you're aware of?

that somebody has done with Pi4j. What has somebody done that you've heard about? That's really surprised you.

Frank: We have a few of these on the website. If you go to my English featured projects, I didn't say it well. Uh, Robert von Berg. So he has a company which uses Pi4j and Raspberry Pis. So he has this, this cabinets for the pharmacy of a hospital, and you have drawers with all the medicines.

So if you start picking medicines for a patient, the drawer you have to pick open turns green. You pull it open the box with the pills turns green. And if you put your hand in the wrong box, it turns red. So that's a business use case. But then on the website, we have a cocktail pie. So that's from the same guy who contributed the code for the Raspberry Pi 5.

He has a cocktail maker, which is based on the Raspberry Pi. We have a street artist robot. So a robot that some street artists is taken with him on his shows. And there are other things. I've built a few gaming examples where you have really have a joystick. And then a game created with Javavix and NavigGL, there are different things but I do invite people who have created stuff with Py4j, let us know and if you have some videos or, or, or.

Pictures. We are more than happy to add them to this, to this section of the website.

Jonathan: Yeah. All right. David, is there anything you want to get in before we actually wrap?

David: I am reading through these featured projects. This is pretty cool.

The jukebox,

Frank: the jukebox for instance, it's just, and that's, that's what's happening in the, in this maker space. Hey, you, you. Upgrade an existing old radio or a television and make it something new. And Raspberry Pi is a really great device for all those. And in some cases people use Java and Py4j to make something nice of it.

And that's like the Pi jukebox.

David: So I You mentioned your connection with trains and I also have been a model Railroader in the past and a real fan. So it seems like a project for the future is going to be Running a model railroad on a Raspberry Pi using Java.

Frank: That would be really, I think I saw a tweet of someone using a Raspberry Pi Zero A train, a model train with a camera to have a front facing camera and live stream.

So yeah, why not put Java on it and have some control over the lets and then the button on the website to turn them on and off. Yeah,

Jonathan: it's, yeah,

Frank: definitely.

Jonathan: So we actually have a question from, from the YouTube. from the YouTubes. We don't usually get questions from there, but I see it. Ozcan asks for open source projects, are there any rules that you really care about, like to keep the community positive and productive?

Is that, is that something that you guys have run into? Do you have any, any of those community rules?

Frank: We don't have a rule based, we don't have a rule written down somewhere. I, I find the community pretty friendly Java community. Definitely. I don't know other ones, but I only once stopped a discussion on GitHub on an issue where someone says.

I really need this. You have to fix this. And that's not how it works. Sorry. And that's what I told him. It's, this is not how open source works. If you really have an issue, then please hire someone. If you cannot fix it yourself,

Jonathan: I will be glad to fix that for you. Here's my hourly rate.

Frank: Yeah, you could also interpret it like that, but yeah, sorry, this whole Py4j project is a pet project for everyone involved.

We don't make money out of this. I, I get money for writing articles about it for some magazines at one euro cent per letter. So it, it, it, no, I don't earn money out of it. Sad, but true. So we have, we luckily have a bit of support from this, from this university. Because they really need updated docs and that's the only support we have in this project.

And that's with all projects, with all open source projects. Maven. I talked a few times about it. Maven is a built tool in Java to build your application. It's, it's how libraries are distributed. It's how the whole Java system runs on Maven. And there are four main contributors doing this. next to the job.

And that's, that, that's what's a returning problem with open source. You know, this, this, this this, this cartoon with a lot of blocks stacked on top of each other, and then one little block at the bottom. is holding up all the rest. And there's one contributor and it's an illustration of the open sources.

And I'm happy that you looked into the contributors of PI4J and said you have 20 contributors. I don't even know. And those are the people keeping a project alive. And I can only ask to people, if you have an issue, yes, we definitely want to help you. We cannot promise And as long as you stay friendly and, and help us find, if you have a problem and, and that those are the issues which are fixed very rapidly.

Someone says, I see this problem. I tracked it down. I think this part of the code could be wrong because this and this and this, and then someone says, yeah, you're right. And we can fix this. And you have the next version tomorrow. That's, that's how things get fixed. But if you just throw something like, yeah, it doesn't work.

On my machine. Yeah. Sorry. I cannot help you. Not always the most

Jonathan: helpful. Yeah.

Frank: I cannot help you. It's keep in mind that people doing this. Want to help you, but they're really doing this in spare time. Tom arts, who is creating all this example implementation, half of the website is creating has examples of him.

He just has had the surgery and he messaged this afternoon. I'm no. Dragging myself back to my desk to find Raspberry Pi that I can take with me that I can do something for the project. Those are the people working on a project like this. These are the people who really want to get things evolving, but it's in their spare time.

It's yeah. Don't, don't yeah, keep, be polite.

Jonathan: So to, to condense all that down into a single statement, your, your most useful rule is users don't get to abuse the developers.

Frank: Yes. Stay friendly. Stay friendly.

Jonathan: Alright, so I am required to ask two final questions before we let you go. And I know the answer to at least one of them, maybe, maybe it depends upon how you understand the question.

What's your favorite text editor and scripting language? I know what

Frank: you think about the language.

Jonathan: Well, see the, the obvious answer there is going to be Java, but that's a, it's a question of, do you consider Java to be a scripting language, which is kind of a loophole.

Frank: Yeah. And since Java, I don't know which one you had to do two things.

So you write Java code in a Java file, dot Java. Then you had to go through the compiler. Then you get a class file and then you could execute the class file. So you had to do two things. But since Java, I think 11, you can just, just do Java, my file, dot Java, and it will actually do the compilation for you in the back.

And then run it. So as long as you don't have external dependencies, you can just execute a file like that. So all the things I would do in bash or whatever, like going through a list of files and find something. I can do that in Java because that's, that's the code I know. So yes, it's my scripting language.

Sure. And would you bang that? And then, yeah which text editor? I am the guy. I actually did film school, so I'm actually a video editor. So everything I use is graphical. So no, I don't know how to exit film or VI. If I have to, I use nano. And yeah. Intelligent ID is the Java, itd but also Visual Studio Code.

My website is, is in, is in Cohero. So that's something I do with visual studio codes. Mm-Hmm. And those are the tools I use.

Jonathan: Yeah. Yeah. Good stuff. Thank you so much, sir, for being here today was a lot of fun and for me too. Got to learn some about Java on the pie. I've, I've gotta admit, I, I do not have time to do it and so I am very much.

restricting myself to not go in and try to hack together a quick backend to talk to GPIO on all of the different devices. Cause I just, it would, it would very much be a drive by contribution. It would not help you guys very much. So.

Frank: And, but if you make something, send us some pictures for the future projects.

Yeah.

Jonathan: Yeah. There you go. All right. Frank Delport, thank you so much for being here.

Frank: It was a pleasure. Thank you.

Jonathan: All right. So you've already told us that you feel the need to go and make your own little Java based Raspberry Pi powered train set. Whoa,

David: whoa, whoa. You might be reading a little too much into that, but okay.

Jonathan: I mean, I'll have to roll back the tape. I have to feel like that's almost exactly what you said. No, I said

David: it needed to happen, not that I was going to do it.

Jonathan: Ah, I see. I see. So, what do you think? Have we talked you into picking up Java as your next programming language?

David: Maybe? Again, you know, it's that whole copious amounts of free time, you know, that most people experience.

But, I will say that I will not be as excited Automatically negative of job going forward, as I have been to this point.

Jonathan: Oh yeah, indeed. Indeed. So Frank has his evangelism has made a little bit of progress there. Yes, absolutely. Oh, alright. Yeah, no, it's cool. And I love to see people sort of thinking outside of the box.

You know, the box that I would put them in. And You know, if you had asked me a couple of weeks ago, what about Java on the Raspberry Pi, I would have thought it was a bananas idea. And now maybe it makes a little bit more sense. And obviously there are people doing it! Doing cool stuff with it. Love to see it.

Love to see it. Alright anything you want to plug?

David: Ah, the only thing that I would plug is something and I actually plugged it on ULS, so I'll plug Twit, the Twit Network and the Untitled Linux Show, but check out the Zen Browser if you've never heard of it. Ah, yes. It, I've seen several people talk about how Arc, the browser company, they really love its simplicity and features and everything.

But the com, the Arc browser. Company went a different direction and Zen is out there trying to fill that niche and they are based on Firefox and they're open source. So I would say go check out the Zen browser.

Jonathan: All right. Appreciate you being here, man. Thank you for stepping in at the last minute.

David: Glad to do it.

Jonathan: Yeah. All right. So as far as my stuff, of course, I'm going to plug Hackaday. We appreciate Hackaday being the home of Floss Weekly now. We've also got my security column goes live Friday mornings. And then there's some other Hackaday stuff that happens for me from time to time. Just keep an eye out for that.

Do check out the Untitled Linux show over on Twit. And as far as next week, we don't have a guest yet. So if you want to be on the show or know somebody that should, you can let us know. Either tag us on the socials or you can send an email to floss at hackaday dot com and that'll get to me and we can get people scheduled.

So sure, appreciate it. Thank you everyone that was here, caught us live and those on the download. And we will see you next week on Floss Weekly.

Episode 808 transcript

6 november 2024 | min

FLOSS-808

Jonathan: Hey folks, this week, Randall joins me and we talk with Daniel Stenberg about Curl. It's an open source success story that lets you download anything and everything from the internet. It's a great show, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 808, Curl. Gotta download em all.

It's time for Floss Weekly, that's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something really fun today. We're talking with Daniel Stenberg about Curl. That is gonna be, ooh, I've written, I've written code a couple of times now to use libcurl, which is part of the curl project.

I know somebody else that I'm sure has used curl, maybe written some code around it. And that is the the amazing Randall Schwartz. Welcome back as co host, sir!

Randal: Hey, hi David. Now Jonathan, sorry. And I Dave, what, where the heck did that come from? Oh my God. It's not enough caffeine and I'm just drinking out of my giant Stanley cup that you can't see anyway.

But no, yes. Jonathan, thank you, thank you for having me back on this show, despite my guffaw, this may be my last show then, we'll see. We'll fix it in post, it'll be fine. Okay.

Jonathan: Nobody will ever see this. Yeah,

Randal: right. Except the live stream and the actual show, I know it'll be there somewhere. It'll be in the transcript too, people can search for it.

When did Randall call Jonathan David for some reason?

Jonathan: Ah, goodness. So you've, I am sure you've done stuff with curl. Have you ever used libcurl? Have you written, have you written code with libcurl? I think at one

Randal: point I wrote some Perl stuff that used curl, the libcurl binding there to get some stuff downloaded.

Of course it was because Perl had direct bindings to sockets and stuff. I actually wrote a FTP mirroring software using just direct sockets. It was pretty cool. It's what actually became the basis for the CPAN. So, the CPAN had been mirrored for many, many years by just having pure Perl code do all that.

I guess I could be ensured that live girl was everywhere and everybody had it installed. I imagine we probably could have shifted to that. And I don't know what they're currently doing these days. I am, I'm sort of out of the Pearl world. I'm sort of, there is, you know something emeritus, something, something, you know, but but I actually went full time dart and flutter starting about four years ago.

So in that respect, I've been using curl from the command line a lot because there are tools that seem like they always want to install themselves on your machine using the extremely dangerous. We can talk about this and the show really dangerous, something, something piped into your root shell. Oh, that always bugs me when I see that.

How does, how does this keep becoming the meme for easy installation? But it's, I have used curl multiple times based on that. So so yeah I'm a heavy, I'm a heavy curl command line user. Yes.

Jonathan: Yeah. So let's bring, let's bring Daniel on and let's ask him about it. First off, welcome to the show, Daniel.

Glad to have you here. Hello, hello. Thank you. Good to be here. Yeah, so Hi. Hi, Daniel. Get it right, Randall. Almost David. That's probably where it came from. Almost David. Ah, it was a

Randal: blur.

Jonathan: It was a blur. So we've had Daniel on the show to talk about Curl and some other things, but it's been like, 15 years ago, back before I had anything to do with Floss Weekly, but Randall was the, the host or maybe the co host, even at that point.

I don't, I'm not sure. I think it was with Leo, like episode 53 or something a long time ago. So welcome, welcome back.

Daniel: Yeah, it's been a while. Yeah. Leo and Randall back in 2009. Yeah. So.

Randal: I think that may be the record for the longest gap on this show. Counting all the shows together. Yeah. The radio show, the TV show.

And now this show I'm sure this is like yeah, this is, this has got to be a record.

Jonathan: Yeah. So we've started out with already we have a question from the chat room and I'm going to corrupt this question just a little bit because I'm a troll and I can. And David Ruggles asks, Why is W get better than curl?

Daniel: Yeah, why is it? So I never wanted to do W curl replacement so I never did. So I have really never. Intended it to be a W get replacement. So sort of, for me, it's always been a case of both are open source and you use the right tool for the right job. So if you want to use W get, you use W get, and if you want to do something else, if you want to fiddle with the protocol in a, in a more sort of more flexible way, rather than just doing a download, then curl is probably your better friend.

Jonathan: Is it, is it fair to say then that W get is kind of like. A pocket knife to curl's swiss army knife plus plus maybe

Daniel: Yes, I would say so. I mean wget is a more limited single more narrow use case basically just download this url to file on disk while curl has a lot more use cases and Really a swiss army knife in comparison.

Yeah.

Jonathan: Yeah, but You know, I talked about using libcurl there for a while, and I think we actually, we still do, we, in ZoneMinder, we use libcurl to get video streams, video and audio streams. And that's, that's part of the process there because it, it talks the right protocol. And so it'll just reach out there and it'll grab it and shuffle bits around.

Curl gets used for sort of everything these days. Do you have a feel of like how many curl installs there are in the world? Are there more curls than there are people?

Daniel: There are way many more curls. So, and usually when I talk to people like this people always think about the. command line tool that hence the comparison with wget, right?

But the, the much more used component is libcurl and libcurl is really, really everywhere. And since that is a little component that everyone is using these days. So we, yeah, we estimate that it's used in somewhere around 20 billion installations. So it's way more than humans on the globe and, and really an average human in, in the Western world, like, like us here, we have a lot of libcurl installations everywhere in our households, you know, phones, cars, fridges, printers, TVs, video games.

So basically all of us, we have 10, 15, 25 installations of curl per person.

Jonathan: That's gotta be kind of surreal, right? To just like look around and go, my camera probably has libcurl in it. My monitor probably has libcurl in it. My led ring light might have libcurl in it Like that's just going to be a surreal feeling like my code is in all of these places sprinkled around the entire world

Randal: Well, so here would be the challenge on that.

Is there more sqlite installations than libcurl installations? Because that's also embedded. That's everywhere, too

Daniel: Yeah, and of course, it's impossible to know, and it's impossible to tell. So it's just a matter of religion, I guess, or guessing. How do you know? I mean, they have the same situation as we do, that it's completely open source, someone downloads it and installs it in a billion devices, but they never tell us, right?

So how do we know this? We can just guess and see some traces somewhere, and I don't know. I mean, I say 20 billion installations. It could be 25. It could be 17. I don't know. So it's more of a. How do you actually know? So comparing, so I often say that it's one of the most installed software components in the world, but I cannot say that it's the most, because as you say, SQ light, maybe Lib Z is one of those also.

Mm-Hmm. And, and the TLS libraries. There are other libraries that are tiny and used in a lot of places. Could be those. I don't know. It's, and it really doesn't matter. It's just in a humongous amount of places. And of course it's, it's, pretty mind boggling.

Jonathan: You've got it. You've got it. You've got enough installs.

You have enough installs now to be like a lot of installs by any metric. And beyond that, it's like, Oh, who's counting?

Daniel: Yes, exactly. And I have this graph I showed once when they, when they told me about that, they actually use curl with the, in the Mars helicopter landing. Right. And then suddenly, you know, two planets as well.

So not only those three, It's 20 billion installations. Now it can also save on more than one planet. That's pretty awesome. I wonder if

Randal: anybody's done an SQLite extension that calls libcurl.

Daniel: I would not rule that out because it's been done on a lot of other databases at least. So yeah. Cool.

Jonathan: I'm sure there's plenty of instances of using libcurl to fetch SQLite databases.

Probably. What license is curl under? Is it, is it GPL? Is it one of the more permissive?

Daniel: It's an MIT, but, but actually, but I filled with it a long time ago. So it's actually not exactly an MIT. I was stupid enough. I should say, because it's not recommended. I wouldn't say that anyone should do it, but it's an MIT.

But so it's actually in the SPDX registry, it's actually its own license. It's actually listed as a curl license, but it's basically MIT with just a few words twisted.

Jonathan: Don't do that. Don't do this at home for no good

Daniel: reason at all. Exactly. Don't do that.

Jonathan: Ah, licensing, licensing will get you every time. So it's been, like we said, it's been like 16 years.

What, what's, what's changed? What is and. The highlights only, please.

Daniel: Yeah, what hasn't changed. Right, right. It seems like 2009 is, I mean, we lost our innocence and open source has exploded in every aspect possible since then. So I think pretty much everything has changed. The internet as an as an So the internet changed, open source has changed, software has changed.

And so there's not a lot of things that that's still the same as it was back then. And suddenly, I mean, curl as a project has changed and my own. approach to it and my own situation has changed. So it's everything is different. And I mean, back in that 2009, that was also before we didn't even do a lot of TLS and SSL back then.

So a lot of clear text protocols that was only the days with HTTP one only. And so it was a lot of, a lot of things were much simpler internet transfers and protocols in general have really, they have become much more complicated, much more, they have evolved a lot in both in security and in complexity.

Jonathan: Yeah. So security is something that's real interesting for curl. I've, I've been following your blog and we've gotten some fun. So I, I, I handle the security beat. I'm not the only one, but I handle the security beat at a hack a day. And I've, I've written up your blog posts a few times here in the last couple of months because Curl is doing some interesting things.

You guys are now, you're a, you're a CNA, so a CVE numbering authority for the Curl project. And that sort of came about, from what I understand, because you had some people that were submitting, maybe you could say, low quality CVEs and trying to make more of them than they should have. And then there's also been a couple of interesting stories about We su suspect you, you suspect where AI has been involved with people trying to find CVEs.

I'd love to love to dive into this.

Daniel: Yeah. So, so first, first, the cv C the CVE side of things. So CVEs, I mean, that's, CVS are made like this's, a database of. Basically issue or backtrack numbers, right? So anyone can pretty much request a CVE ID as a number from MITRE, who's the sort of the head organization of CVEs.

And there's really no requirements from the, I mean, they don't require anything pretty much from the reporter. They just, so I want to have a CVE ID. And if you ask for that for any product, you can just get it. It's, I mean, you can do it today for, for free. And they will just give you that CVE with no questions asked, no requirements, nothing.

And then you as a, you imagine that you have found an issue. And then when you get your CVE, you can publish that at some point later on. And, and there's really nothing in the system that puts a break to that. So that, that just happens one day and that's how the system works. And it's completely. stupid and unhinged, I would say.

So there's really no filter in this. Well, there are filters, but, but, but for the majority of people, there's no filter involved, so that just happens. And, and so then suddenly one day someone can just publish a CV about your product and they can claim whatever because there's really no filter. Nothing involved that prevents that.

I mean, MITRE could theoretically have some technical person who would read it and complain, but they don't. And I'm. There's nothing. So the only way is that this numbering, the numbering authorities within the CVE, they have a veto right so they can deny your CVE, but only for the products that they are responsible for.

So there are a lot of those organizations that there are over 400 now. Well, and now curl is one of those. So now we can actually deny your CVE being created. If someone says we found a problem in curl and if they don't have enough details or enough proof of that, we can just say, no, it's not go away. Try again another time.

So basically we did that to put that filter on. So now stop that. So now we cannot get those. Ridiculously stupid CVs and in the particular one that the last one that I blogged about was one One user found a commit message from a few years ago when I mentioned integer overflow in the commit message The person extracted that and and whoa the integer overflow that must be bad So it's sort of extrapolate make a cve Publish score 9.

8 because, you know, the sky is falling. Everything is going to burn clearly because it's an integral flowing curl, but it was super stupid. It was just a delay. That was. sort of miscalculated. So sure it could do completely wrong delay, but it was really completely harmless. And it was a bug, stupid one that I fixed also years ago.

So it was sort of, yeah, it's, it should never have been a CV in the first place. It's impossible to reject because MITRE refuses to reject it because. They argue that it could be a security problem and NVD that sets the score. They also won't refuse, refuse it. So it's, it's in there. So that's, that's the CV part of things.

And this is potentially a really a danger for any open source project or any project at all, because there's, as I said, there's nothing that stops this from happening again and again, and again, really, unless you have a CNA that says this product is our responsibility. And now suddenly we are the judges of if we should deem this or allow this to be a CV or not.

So that, so now we can stop that. But then the, the, but the, the part about AI is that's a sort of a slightly different angle to everything. So, because at the same time we run a bug bounty program in curl. It's actually sponsored by the sort of the upper project called the Internet Bug Bounty, IBB. And they, they have a.

idea of sort of running bug bounties for internet infrastructure, open source projects. So they they've sponsored bug bounties for, I don't know, 20 different projects or something. Anyway, so we're part of that. And that means that anyone who reports a security problem to curl may then get a potentially big reward cash, big bunch of cash if they find a security problem, which I think it's a really great thing because it really makes people.

Put in the extra effort and spend a lot of time and to research and actually find problems. But I mean, that's a good part of it. But of course, when we say, hey, we can give you a lot of money. It also attracts the people who are sure I want the money, but I might not actually understand what I'm doing. So I'm just going to run some tools.

And then, of course, people ask some questions. friendly AI to find problems in curl and use that and report those findings to us. And, and that's one of the, some of those more hilarious reports when, when, well, yeah, hallucinations is a friendly way to say it, but it's just blatant lies, right? Sort of And sometimes that's hard sometimes to, well, it's hard to say immediately that it is wrong because the AIs are also very good at English and they're very good at, you know, producing a lot of texts and, you know, you get a report that is, I don't know, very long, very detailed and how it takes time to assess, I mean, what are we talking about?

Is it true? Is it not? And in most cases when someone has said, actually found a serious problem. It can actually take a lot of time, right? Because it's the details and you have to understand things. What's the possible outcome, blah, blah, blah. So a very good crap report takes a lot of time to dismiss.

Usually, I mean, the really bad ones, they're just, you know, someone ran a scanner, found some completely bogus things. It was a mistake. I didn't understand it. They, those are the ones you can dismiss like in seconds, right? Those are easy. They're not a burden to us. We get those all the time, but the better the crap, the harder it is.

Threw away.

Jonathan: Do you see a, do you see a scenario or a future where AI actually becomes useful for finding real security problems?

Daniel: I'm sure that they can be useful. It's probably already useful in some senses, but I think right now, I think the ones who are using these tools are not the sharpest knives in the drawer.

So, I mean, you can, you should use it as a tool and then sort of understand the output, maybe try some further dig a little deeper, and then maybe you will find something right. Not just throw some code at a tool, see what it says. Thanks. Copy and paste that report back into us without understanding at all what it said or what it meant, or what it po possibly was really, then, then that's not gonna fly for a long time.

Jonathan: Yeah. I, I think there are some companies working on this, but the, the idea there that really makes sense to me. I think there could actually be some, some use for it is an LLM that is, that is running a fuzzing scanner. And so you're actually taking the output from the AI and actually running the code.

And then you've got a test suite that will find something. And that seems like there's some potential there for actually being useful.

Daniel: Right, exactly. That that's, I mean, taking what it finds, actually throwing that back into the code to actually verify that what you found actually finds the problem for real.

That's, that's a really good way to do it. But, and of course I can imagine that you could, you know, Do something with fuzzers in general and maybe use the AI to improve the fuzzer. So sure. I'm sure AI can be used in a lot of good ways here, but I think as most things with AI right now is that you need humans involved to sort of filter a little bit and guide it and extract and, and, you know, back and forth a little bit before you actually just throw it out on someone.

Jonathan: Yeah, absolutely.

Randal: It's, it's, it's obvious that if the if the fuzz is trying to find things that are similar to what has been bro appear to have broken other things, LMS gonna be pretty good at locating those kinds of things. Mm-Hmm. . So, because you know, really what l m's doing is it's saying.

Here's a whole body of existing text. And here's a little bit of a new piece of text. Is there anything kind of in this whole body of text and sequences of things that would sort of extend this a little bit? And so, so it is the kind of thing that, that works that way. It's, it's You know, it's we could do a whole show on AI.

I'm sure actually dozens of shows have been done entirely on AI. I think that's pretty much every

Daniel: show since the last couple of years.

Randal: I was going to wonder how soon before AI comes into this show, actually, I was sort of taking mental bets about, is it going to be right after this question or right after that question?

And really,

Daniel: I mean, You can also get a sense that that's exactly the way you mentioned is that when when we have got a single security report reported and published and everything, and that's, you can already tell that that's exactly what the humans already do. When there's been one security problem, you, you can be sure that people will investigate sort of nearby situations.

Almost that or the same thing in a different part of the code or something. So it's, it's very common that when we have one issue reported, people suddenly start to report similar things in nearby code or nearby situations. And so on. So yeah, I agree that that's exactly, and I think it's actually a pretty good way to do it.

Because then you have proven that you had one of these flaws. Maybe you have another one.

Randal: Yeah, that that definitely once you crack that egg open, you know, it's like, it's you gotta make lots of, lots of eggs from there. I get it. And I, that apparently I don't cook cause it doesn't even make sense.

Oh man. Oh man. Well, I, I'm not sure what it was before the show or whether in the show, we were talking about the use of curl pipe directly into the shell. Do you have any comments about all these ways of installing things that seem to show up that are curl piped into a root shell?

Daniel: Well, I don't do that Well, I don't think I don't take that I don't take responsibility for that I I I would say perhaps that sometimes I think I mean People have downloaded things from the internet for a very long time without verifying anything.

And people tend to sort of, yeah, and if you just, if you install a code from somewhere and you don't check it out, that's roughly the same thing. So as long as you actually trust the site that you do download this from, I would say it's not that hard. that bad. I shouldn't say that. I wouldn't encourage it still.

Randal: How is curl giving enough information to the host computer to tell whether it's being piped into a shell or just going into a file? Because it doesn't, it doesn't

Daniel: at all, but you can. I've seen people do tricks that you can actually guess if it does that or not, depending on how the shell works. So there's some timing differences, potentially.

So I've seen those that have actually done it differently so that you can do some different display in the shell than the download. So you can do really nasty tricks. That's what works most of the time.

Jonathan: By default, when you URL, it'll just spit the outputs to standard out, right?

Daniel: Yes.

Jonathan: Yeah.

Daniel: As long as it doesn't detect binary in the first hundred bytes.

Randal: So what protocols have been added in 19 years? All of them. I mean, right. Well, right.

Daniel: How many protocols

Randal: are you still, I still primarily only do HTTP and FTP, or is there some other protocols that curl now is including.

Daniel: So in 2009. We were still innocent and young, and then and then all, everything broke loose.

So we implemented pretty much every email protocol after that. imap Pop three SMTP, and then we went into RTSP, RTMP and our tm, and of course all the TL s versions of those.

Randal: Mm-hmm, .

Daniel: And then we went further and went into SMB Microsoft to. File transfer things. And then after that took a little longer and then we did MQTT.

We did, and then we did Gopher over TLS. And then we added WebSockets a few years ago.

Randal: Gopher, TLS. I love it. Gopher over TLS. It's the two ends of the internet spectrum of time. It's kind of,

Daniel: yeah. And you know, this, those enthusiasts are still Gopher users. And apparently some of them are even Gopher over TLS users.

No. And those three people, they contacted us about it.

Randal: And they're just shipping files to each other.

Jonathan: That's incredible. So like HTTP 3 and QUIC, have those been added? Stuff like that? Yeah. Yeah. Yeah. The real, the real new shiny stuff.

Daniel: Yes. So an HTTP is of course, one of the primary protocols that for curl and we added HTTP two in.

So we shipped that already when the standard came, the standard came in 2015 and then we added HTTP three support early on. So we had HTTP three already in 2019. I think we started having that. And we've been shipping it to be three cents. And we Nowadays we have, as we do with, with all the, we support a lot of different TLS libraries for TLS, and we support a lot of different QUIC and HTTP 3 libraries for HTTP 3.

The situation is really complicated for HTTP 3 because of the weird Situation for APIs and the components involved to actually do quick HTTP 3. So it's, it's a little bit of a messy state there. So when I say, yeah, we support it, it's still the fact that in most places where you install curl, it won't be enabled because of situations in the surrounding support libraries.

Randal: So you just mentioned IMAP. I didn't realize that I could actually go fetch some email with my curl. Now you can fetch email and you can

Daniel: even upload email with IMAP.

Randal: Wow. Okay. Cause I'm actually trying to diagnose some problems with my current mail server. And I've been trying to like. Type they're all the right you know, Dart commands and stuff to get that to happen.

But it sounds like I might pop curl out. And so that must mean the curl dash dash help command is now what? How long? Super long.

Daniel: We have actually the other day we added the 266. Command line options. Wow. I'm not sure that's a good thing.

Randal: With all those protocols, how do you test all that? Do you have like a, a extensive test suite that's able to simulate both ends of the conversation and curl complaint? Yeah, well. In the middle

Daniel: or whatever?

We, yeah, we have custom servers written for every protocol pretty much. So yes, and so we test against our own test servers and we have unit tests and we have tests. So yeah, we do a lot of tests and of course, Fuzzing and scanning and everything. Wow. So when you, you know, when you have your code in 20 billion installations, you want it to be at least decent.

Randal: Yeah, you definitely want it. You definitely don't want to show up on this top CV list at all. And I mean, now that you can control that, you can say, Oh no, that's not a bug. But, but I mean, you don't have any bugs anymore. Yeah. Right. Gone. It's all, it's all gone. So so a lot of times I'm typing curl and I get like halfway through the command.

Then I go, I don't know what the switch is. And so I've, I think if you just type like dash dash help in the middle of the command line, will it sort of take what's already there to kind of give a hint about what the rest of the band page to show or not?

Daniel: Well, there are a lot of these command line completion scripts.

Randal: yeah,

Daniel: if, so for, for a lot of shells, you can just have those completion scripts and then it can complete all those command line options. And since since a while back, I also offer a, so you can do curl dash H and then do the so you can get a help for that particular switch and get a huge chunk of The man page output in the terminal.

Randal: That's what I was looking for. So there is a way to not have to read the 10, 000 line version every time. Exactly. So you can

Daniel: get the help for just a particular option you want to do.

Randal: Oh, cool. Cool. So I would ask for curl help for IMAP or whatever. So to try to understand those protocols. Cool. That way it makes sense.

Jonathan: So David in the chat room he has a metaphor here that he thinks maybe helps us understand. He says, is curl to data transfer what OpenSSH is to encryption?

Daniel: Yeah, a little bit like that. So we've tried to, or I tried to narrow what, what, So, so the question then of course comes what, what is, what is included in cURL and what isn't included in cURL, right?

What, what, what's ever is subject to be supported and what's not supported. But as long as we have a URL format for protocol and it's related to upload download, then I think it's, it's fine. Fair game for Curl to support it. But now I think there aren't that many more protocols we can add support for it.

Jonathan: Somebody will come along. It's an, it's inevitable. Like how often, how often does that happen that you get a support request that somebody goes, this obscure protocol that I care about, it would be great if it was added to Curl, like that's going to happen. A lot? Weekly? Monthly? Yearly?

Daniel: Yeah, well most of the requests are not as a pull request and then most of them will be someone just asking for hey Why don't we support some weird protocol?

Or something strange because in many cases people They found find themselves in a situation when they already have curl right and then they just want this little extra thing as well. So why doesn't curl do that as well? Because that would be really easy for them because then they could just use curl for everything.

But that's not how we work, right? So we cannot just add support for everything just because of that. So that happens regularly. And, and then of course, I mean, it's not always that easy to just add support for another protocol either. So yes, we also get the occasional pull request that just, you know, here's a new protocol someone made and, and It just never actually materializes because the one who actually wants that to happen doesn't really have the energy and effort or energy or time and whatever to actually make it happen all the way.

Jonathan: Yeah. So speaking about these requests, one of the, one of the things you mentioned in the in the show notes when we were prepping beforehand was you occasionally get crazy emails. And I went and I looked at a couple of these and I I too, not emails usually, but I get support requests for various things like that.

And it's, it's, Okay, set the stage for us. What are the crazy emails that you get?

Daniel: Well, I could set the stage by going back to the, so, curl, runs into a lot of places, right? And it is an MIT license and in the curl license, it actually says copyright Daniel Stenberg, blah, blah. And my email address. And that is, I think that is the key here.

And because Not a lot of other licenses have email addresses in them. So, so anyway, so fast forward. So when the person sits there in his car and you know, I want to enter my GPS position in my map and I can't figure out how it works. I need to contact someone and you

Randal: know, and then

Daniel: how does that person find someone to contact?

I, I guess in some cases they find some kind of open source license screen, you know, scroll through this needs to be an email address somewhere. Oh, there's a guy. I'm going to email that guy and send him some questions. I mean, that's my guess how this is happening because I get so many car questions that it's ridiculous because why, why would I get car questions?

Jonathan: Oh, jeez. That's, that's hilarious. Okay, I, I, I had not, I did not realize it was going to take that direction because I, I am used to getting, I am used to, bless them, I'm used to getting dumb questions, or, or really poorly thought out questions, or sometimes even questions where someone's native language is not English.

I'm trying to have a little bit more grace with those. I get, I get dumb questions all the time. But that's, that's a special kind of special.

Daniel: Yeah, but my, my questions, they're beyond, they're not really dumb. They're just completely out of my league, right? They're just, you know, someone asking about their account on some weird service because they found my name in a game they're playing, right?

And I had no idea. I didn't even know that it was a game or that my code existed in the game in the first place. You know, so I'm so far away from. Whatever they are talking about. So that's quite impossible for me to do anything except, you know, giggling and putting it up and say, here's another funny email.

Jonathan: Do you respond to these, these email them back and say, I'm sorry, I can't help you.

Daniel: I used to do that, but, but I realized over time that it was just, it was just. It actually turned, I mean, it was, I gave the opposite effect many times that they just think that I'm ducking the question, avoiding it and being rude by not answering it really.

So, so they, they just get upset with me and sort of, no, no, no, shut up. Help me instead of, you know, avoiding the question when I think it's really fun as this, I have this, Among those emails, one of the, that woman from emailed me about her Instagram account getting hacked, right? Why do you email me about your Instagram account?

I have no idea what you're talking about. And then she sent me a screenshot from Instagram. Look, your name is in there. Cool. So she did not think that was cool, you know,

but I of course had no idea that my name was in that app. And so, and that's kind of, it's just impossible for me to. help them with their question, actually, because I'm not involved in that. I don't know what they're talking about. I don't have any contacts. It's not my thing to answer.

Jonathan: You mean you didn't put a backdoor in Libcurl so that you can just go in and solve all these problems for people?

Randal: You can't tell us that. You can't put them on the spot like that there, Jonathan.

Jonathan: So that actually, that actually raises an interesting, an interesting thought. With, with Libcurl being as big as it is. Have you, have you, have you had any contributors that you think are trying to, to Jian, Jian Tei, John, John T, whatever, the, the, the, the, the random Chinese, yeah, the random Chinese name that tried to get the back, that managed to get the backdoor into XZ.

Have you seen anything that matches that sort of a, that sort of a pattern?

Daniel: No, never. I've never even seen an attempt like that. And that's also the, that also made me more impressed by that attack because it was so brilliantly performed in so many ways. But no, I've never seen that attempt, or maybe I've just been naive enough to not understand or dismissed it early enough so that it never materialized.

Or maybe it hasn't happened yet. And, you know, one of the main trainers I already have. relationship with is actually JR 10, right? So,

Randal: Or maybe it has happened and, Or maybe it has happened. Yeah. There's

Daniel: so many

Randal: dimensions. And we just haven't found it yet.

Jonathan: That's, that's the terrifying one, Randall.

Randal: Yeah.

Sorry. That's what I'm good at. Yes. Yeah.

Daniel: But no, I think, I, I mean, it's so, so, really, really difficult to actually perform that kind of attack. I mean, not even that guy managed that attack, right? So even though it's so excellently well performed, it failed anyway. So I think there are much better ways to invest all that money and time to actually exploit real issues that we have landed anyway.

Yeah. Those are my probably The better way to attack a current situation

Randal: in the last recent times who's contributed most of the The patches is that are you mostly still the primary author or are you have a lot of it delegated out now and you're just Taking poll requests. How does this? How does it keep getting moved forward?

Daniel: think this there's a lot of contributors the other day, I think we are over 1300 Authors and commit authors over time But of course, it's a narrow as a much smaller subset of people that are actually contributing a lot I still do a lot, but there are a bunch of other maintainers who do a significant portions of the commits these days So maybe 10, 20 people that are actually contributing frequently.

Randal: How well has the design held up over all these years? Do you think if you started over, would you redesign it in a different architecture?

Daniel: It has held up surprisingly well, which I think is Part of the explanation why we have succeeded so well, because we haven't had to, you know, rip it apart and do it all over and change the APIs and everything.

So I blogged about it the other day when we celebrated our 18th year of not breaking the

Randal: API.

Daniel: So I think that is an explanation why everyone can still keep using libcurl, right? Because they haven't had to change their applications for 18 years. They can just upgrade to the latest libcurl and everything keeps working.

So in that sense, we have actually succeeded. Excellently well in, in sort of separating the app from the internals of the library. But then of course, we have a few things that I sort of have regretted over the years. We should not have done that, but now we're stuck with it because we won't break the ABI.

So we have to support this until the end of time, right?

Jonathan: I mean, I've written C and C or C one of the other code against libcurl. And it was decidedly not terrible. Which is much more than I can say about some libraries that I've worked with.

Daniel: No, I think it actually is pretty good in many aspects. Then it becomes a little bit hard to use sometimes because you have so many options and so many ways to use it.

Could sort of, you can get lost in among all this.

Randal: It's good to hear that you haven't done much architecture change. Cause I know that I get a lot of questions from people who want like the perfect architecture for their Dart application. And it's like, you really don't know until you get a ways into it.

Right. But I think, I think we,

Daniel: I think we have done architectural changes within, I mean, limited, but we have still supported the same external API, but we have refactored the internals several times, but we've, we've sort of with a little bit of skill and a little bit of luck, we managed to do a fairly good API that I abstracted a lot of internals good enough so that we can, could actually remodel things.

So we could introduce things, for example you know, when, when HTTP2 was introduced back in 2015, right? So suddenly, that was really one of the first protocols that suddenly it was not a single request for a single connection, right? Then suddenly you could do multiple requests over the same connection.

That's a completely different paradigm. So, and by, but by pure luck, really, we had an API that really was agnostic to that. So we could just Transition into that world without breaking the API, we could just add an option and suddenly we could do that. And that was, of course, I mean, we couldn't have foreseen that like 15 years before it was just.

Yeah, we just decided on that API and it happened to work that way when we transitioned how we do protocols. I mean, it's not unlikely or I mean in the future, maybe we can come up with something else that will not be possible with existing existing API because some new thing is not possible to do with this old API.

Jonathan: When, when did the dual API come about because in lib curl you've got lib curl easy and lib curl multi, and so easy. The lib curl easy. You can, you could be grabbing files in just literally like four or five lines of C code. It is like, it says on the 10, it's easy, and then if you wanna do something more complicated, you've got the multi what the, what you guys call the multi interface.

What did that come about? Was that from the beginning?

Daniel: No, well, from the beginning I had the idea of making layers of APIs sort of provide something that would be easy to use to just get the file or get the transfer done when you don't need to do anything fancy, sort of quote unquote fancy. And, but but then I wanted to also have a way to do more than one transfer in parallel.

I struggled a bit to, to do that. figure out how to do that in the API. And then I came up with this solution so that we sort of build the same API into the other API in, so to speak, so that you can just build a lot of transfers and make sure that all of those transfers happen in parallel instead of serially.

So it wasn't really, Sort of from the beginning, but we had this sort of the foundation for the beginning and then we sort of worked it out over the years.

Jonathan: Yeah So we've talked about the the maintainer burden and like how you're still very involved with it does does libcurl pay your rent? Is this how you feed your family?

Have you, have you managed to turn libcurl into a career?

Daniel: Yes. So in, since 2019, I do curl full time now. Yeah, that's great. And yes, so that's, this is the only thing I do. So now I have paying customers, paying to get the code for free.

Jonathan: Yay. How do you, how do you make that transition? Like there are, there are a multitude of programs, open source programs in the world that just, they would love to make, to turn that corner.

What, what does that process look like? How did, how did you, how did you get to that point?

Daniel: It's a really tough Well, yeah, that helps. Yeah, but still, I still struggle with sort of that exact step, right? How do you go from not doing it as a work at all until sort of do that only? Or even doing gradually, that might be even harder, right?

So how do we actually do that? Turn that from not having it as a job and having it as a job. And for me, it was I was in a fortunate position because I had friends who worked then at Wolf SSL and they believed in this as a business concept. So I sort of, I just moved my product into their library pretty much and say, now we sell support under this umbrella.

You pay me my salary from day one and we start selling this and they believed in it and we could do it this way. So I think that was a great way for me to. Kickstart the business and also have a get an infrastructure for the support and for contracts and stuff like that already from day one.

Jonathan: Yeah, that's great.

So is that still the case? If somebody needs some sort of support for for curl, do they go through wolf SSL?

Daniel: Yes, they still handle the business. They signed the contracts and I do. I do pretty much all the support. Well, I have a few others that could do. You know, the basic ask the follow up questions, which version have you tried to blah, blah, blah.

But then otherwise it's pretty much me

Jonathan: Does that does that keep you busy? All right Are you just running around like a chicken with your head cut off trying to keep up with that or is it manageable?

Daniel: It is it's quite manageable. It's actually it's actually pretty good Pretty neat situation, I think. For me, all of this has just turned into sort of landing, landing the dream job, really, because now I work full time on my spare time projects.

And most of the time I'm not occupied by projects. Customer support cases. Most of the time I can pretty much decide myself what I want to spend my time on. And that means, you know, working on much code, reviewing, writing new stuff, whatever you want. Whatever you think is the project needs right now.

Randal: Is this do you have a big community?

Do you have like, like live curl meetups and stuff and annual conferences?

Daniel: We have an annual conference. I wouldn't call it big. But we can we can gather maybe 25 people in a room. All right. All right. But it's an awesome conference. Just curl stuff for two days. Yes.

Randal: Wow. That's great. That's great. Do you have like a, you have like mailing lists and like maybe a discord or something?

Daniel: We have a mailing list. We have IRC channel on of course a lot of things on GitHub.

Randal: Yeah.

Daniel: So yeah, we have a pretty active community of people. It's happening a lot of things. So, I mean we're doing a release tomorrow, actually November six. And in this, we have, we have a eight week release cadence.

So we do new releases every eight weeks sort of on the clock pretty much. And tomorrow we ship 260 bug fixes and five changes. So there's a lot of things happening. Even though we've been around for 28 years and everything seems to be the same.

Jonathan: Do you have problems with distros and distros in particular, but I guess this will be true for a lot of things.

Shipping super, super old versions of curl and lip curl.

Daniel: Yes. Period. Of course. I, I, I don't know. I, I, I sometimes I feel like I'm, I'm even more of, I don't know, hurt or affected by this than others, because for some reason people like to get stuck on really, really old versions of curl. And I think sometimes the distros are not the worst offenders here or people using old outdated end of life versions of distros.

But even a lot of these device manufacturers, they, you know, they got that libcurl install 12 years ago and they installed it in a device and it runs there, you know, and now suddenly they want to upgrade So sometimes we see that oh i'm upgrading from blah blah blah and now this doesn't work anymore And you know, and then you can see that.

Okay They have not touched this in a decade or so and things have changed. Yeah

Jonathan: well, I mean, I think I think a big part of that is SDKs honestly is what I blame SDKs for pieces for for hardware are terrible Pretty much across the board. You have ancient kernels that'll get shipped as part of the SDK.

All of the support infrastructure around it is terrible. It'll, a lot of times it'll have weird, you know, custom code bolted onto it that's terrible. And so you have people that, you know, they get, it's like, oh, here's this new chip to build a router out of. And it's got Ethernet built into it. It's got Wi Fi built into it.

Oh, and here's the SDK from 2006. I guess we're shipping the 2. 6 kernel with it then, huh?

Daniel: I agree, but I think it's also sometimes we have had a culture when people have allowed this to happen. So the device manufacturer, they don't want to upgrade because upgrading is a sort of, yeah, it might break, it might be work, you know, it could just stick to the old thing and it seems to work and we can be fine with that and earning sort of just, you know, Earned some more money on these devices and just avoid it.

So I think it's a little bit of that too, because in some cases it's clearly, they could have upgraded a long time ago, but somehow they decided not to.

Jonathan: Yeah. Have, have you guys felt any impact yet? Or do you anticipate any impact from some of the legislation around this that's happening, both the United States and Europe where, you know, they're, they're trying to put some of the onus back onto.

The people writing the software to, to make sure bugs are fixed and insecure. Yeah.

Daniel: There's a lot of talk about that. Yeah. The CRA, for example, in, in, in the EU. And so, yeah, I'm, I'm sort of looking forward to see where that goes and how that could possibly be converted into an opportunity rather than a problem.

Punishment for me or for us, right? So, so for example, I want to be the one who guarantees my functionality. I don't want to have any middleman on top of me saying that I'm going to guarantee functionality on these things. Just because it could actually be a, you know, a business opportunity, right? If a lot of companies say they ship products with curling and they need to guarantee that there's a functionality.

Sure. I want to be the one who guarantees that functionality, because I think I can do that, but I don't know how that will play out.

Jonathan: Yeah. You know, I've, I've, I've heard people tell stories about, you know, Open source developers, and they'll get a request from some business that, you know, they have no relationship with and like, Oh, we've got to have this and this and this to be able to put our S bomb together.

And, you know, the open source guys are complaining about this. And my take has always been, tell them your hourly rate. Yes, I will be glad to help you with this. This is my hourly rate. Let's make it happen. And sometimes open source devs are terrible about that.

Daniel: Yeah. But That's has actually been my response for the last few years, but the, the typical action from the other end tends to be silence

So it's, it's easier said than done well, but, but it, it might change then with legislation. Oh, sure. So it might. Actually go in the right direction going

Jonathan: forward. The golden part of that silence is that's one less support request that you have to deal with. It at least doesn't waste your time having to fiddle with

Daniel: it.

Exactly. So at least I don't bother to actually respond in a very long and elaborate way. I can just say, sure, I can respond if we just get a support contract first.

Jonathan: Yeah, absolutely. In some ways, I'm gonna, I'm gonna wax philosophical here for a moment. I

Daniel: just wanted to insert a little bit about that, that exact thing, because a few years ago I got several questions from NASA about exactly that.

You know, I got three emails from different departments at NASA asking me questions about, you know, who's writing this and where are you from? Is any contribution from one of these five, seven band Chinese companies or something?

Randal: I don't

Daniel: know. You know, I have no ideas from on, from which companies people are working on it, you know, blah, blah, blah.

But so I got a little bit annoyed by all those questions from NASA and I blogged about it, right? Look at all these silly questions from NASA. Ha ha. And then I moved on, of course, because they didn't answer my, I asked them, well, fun. You're using my code. You can tell me about what are you using curve for?

They didn't, they didn't want to answer that. They say we're using it to further blah, blah, blah. It was just some nonsense. Anyway, I forgot about it, but I think it was last year or the year before that a NASA guy appeared on did a presentation at FOSDEM in Brussels. I talked about their use of open source and what do you know they showed my blog post in his presentation.

At least, you know, it was red. You had an impact. Exactly. At least someone got it. I don't know if it actually had an effect, but it actually appeared on the slide anyway.

Jonathan: Yeah. So I was, I was going to point out that I think open source kind of needs to have this growing up moment where we become more willing to Ask businesses to financially support what we're doing.

And I'm, I'm just, I've got to say I am tickled pink. I am delighted that you have made this work and you've, you've actually turned it into a career and it's taken care of you. I think that is, I think that is awesome. Yeah. One of the, one of the things that recently has been in the news, I've done some writing about it and talking about it is and, and, You mentioned this question from NASA, this is what brings it to mind the Colonel has had to do some reshuffling of maintainers because of U.

S. laws and executive orders and, and geopolitics coming in and messing up everybody's open source fun. I don't, I don't want to dive into like the geopolitics of that. That's not what the show is about, but I do want to ask, is that, is that something that's had an impact on on your work as well?

Have you felt that?

Daniel: I have not I, I guess I don't have any prominent contributor from any of those countries. I think, I mean, how would I know, but not that I have detected anyway. So no one has said anything and I have no one has remarked it. So it hasn't happened, but I actually, I have actually had the reverse because I have right now a support customer who actually dropped, he had to, they actually used a Russian tool as a replacement as, I mean, as current alternative.

Hmm, which one, which they can't pay for anymore because the author is actually Russian. So they can't pay him. So actually, in that case, you

Jonathan: picked up a company.

Daniel: So it's weird. I mean, yeah, I didn't select that. It just happened.

Randal: To Jonathan's point, I think also the you may not be subject to the U. S.

restrictions. And so you know, exactly not the U. S.

Daniel: restriction. I could be subject to EU restrictions because we have a lot of Russian similar restrictions here. So I couldn't do business with a lot of Russian businesses, for example. Yeah.

Jonathan: Yeah.

Daniel: Wow.

Jonathan: So I managed to not ask about it and I meant to ask about it.

You've got something new coming that's kind of related to this support contract thing. A like a long term release, an LTS version of curl.

Daniel: Yeah. So, so I've had this actually debated that question back and forth for a long time, because we're in curl, as I mentioned, we all, We don't break the API, right?

So we keep supporting the old API all the time. So it should just be a question of upgrading to the latest. And that's what we always support in the open source project. We support the latest version. But over time, I've just come to realize that companies are still scared of that jump, even though we say that everything is safe.

And that's one of the reasons why right back again to why people get stuck on old versions. It's safe to just get stuck on an old version because then you don't have to risk getting something bad when you upgrade. So now we're finally going to bite that bullet and start offering a proper long term support version so we will get stuck in time.

Randal: Yeah.

Daniel: Or offer a version stuck a little bit in time at least but still patch it with security fixes and Important stability fixes really, and try to support that in a little bit style, the style similar to how distros do it already. And a little bit try to get some of their business back to us.

Jonathan: Yeah.

This is something that, that you've had requests for, I assume, like, you know, there's a market there for doing this.

Daniel: Yes, I, I assume there is one because I've had questions and I think there is so it's a little bit about testing the waters too. And it's not, at least in the beginning, it's, it's not that hard.

I mean, the support job grows over time, right? Because it's going to be harder and harder to backport stuff and maintain. Code the older it gets so it's a little bit of a Experiment here and testing to see how How interested companies and customers actually are in this but I have faith.

Jonathan: Yeah, i'm sure there will be some interest because you would hope at least that some of these people Places that are using these old versions of curl are doing some of this internally Watching for very severe CVEs and doing fixes.

Daniel: Exactly. So, so, so that's what I'm hoping for here. I want to aim this at these companies that are already doing stuff like this, or they want stuff like this. And you know, they're concerned about their product services and they're already having some support businesses involved for, for, I mean, obviously commercial companies are this directed to.

Commercial businesses.

Jonathan: How, how long, how long are you planning to do support? Like how old of a, how old of a kernel of a curl lib curl install. Do you think you'll backport patches to

Daniel: I've said five years now. And it's, you know, right now it's just putting the finger in there and say, how long do I actually need it to be?

Maybe five. I haven't actually no idea really how long it actually needs to be. So I say five now, maybe I need to. Adjust that as we go and I think As also as it gets more work as older as it gets I also think it gets more valuable to the customers the older it gets so I guess it's also that's also maybe an opportunity to Get more customers for the older ones the longer the support term is but I I don't know Yes, that's the easy answer.

Jonathan: Yeah, I used to think that five years was a Almost eternity for technology. And then I started a business and ran on that for a while. And the next thing, you know, I've got servers that have been installed for five or six years and have almost five years of uptime in a couple of instances. And

Daniel: yeah, I have a customer right now.

They offer their customers 13 years of API stability, 13. So they want that stability from me. So I already have that support burden sort of implied. So. I have already, you know, this set up that we don't break the API anyway. So in theory, it should be possible.

Jonathan: Yeah. We'll see. Yeah. It's

Daniel: a, it's an adventure.

We'll see. We'll

Jonathan: have to, we'll have to bring you back in five years and ask how that went.

Daniel: So see if there's any tears or joy.

Jonathan: Yeah, for sure. Randall, is there anything you wanted to get in before we, we start the wrap?

Randal: No, I was trying to think of anything else that we really haven't covered, but we've kind of covered the whole thing.

I, I think I guess the only last question is sort of a historical question. When you started all this, did you figure it would get this far?

Daniel: Oh, absolutely not.

And it is actually kind of ridiculous, right? Because I started with something that was just such a tiny little piece of toy thing, and it now has, I mean, I actually, you know, I've gone back and looked at the first code that I've actually started working with. That was 160 lines. And now it's actually more than, A thousand times more code, right?

So that's actually, it's now approaching 180 K from 160 lines. So it's the, the growth is, it's, it's amazing. No, I had no idea when we started, where we were going, what I wanted, what would happen.

Randal: I can imagine it must be similar to what, how Linus felt about. Minix when he first released it, like who knew that it would grow into like a universal world industry, you know,

Daniel: exactly.

It's just a simple little thing. Sure. We can have some fun with it.

Randal: A little kernel running on this processor. Who cares? It'll

Jonathan: never be big and important like a GNU's kernel. Right. Exactly. Yeah. Alright. There are a couple things you mentioned in your notes here that we did not get to. And I wanna give you just a quick chance to to plug what these are.

You, you mentioned earlier, and I honestly don't know what it is, W Curl. And then I also see, twirl? T R U R L. I'm not sure how one pronounces that. No. It's quite

Daniel: impossible to pronounce. But I, I, so I like to pronounce it True rel, with an extra E there at the end. So True rel is actually a new tool that I, that we started last year.

It's actually a tool for The tr is actually for the tr command as in transpose, translate as for, and for URL. So transpose, translate URLs, basically parse and manipulate URLs on the command line. Pretty much when you write a script somehow and you'll need to, you know, extract the host name from the URL or add something to the path or add a query part to the URL.

That's really tricky when you do that in the shell script because manipulating URLs is really tricky. It's next to impossible to do in a shell script. And well, you can reg exit somehow, or you can mess it up.

Jonathan: I was just thinking it's a replacement for using Perl with reg X.

Daniel: It is. And, and, and that also goes back to this.

There's been several papers over the years, how there's a common security. Problem when you mix and match URL parsers pretty much use one URL parser to do one thing and you pass on the result or you And they get some results

Jonathan: slightly differently Exactly

Daniel: because urls are a messy. Sorry thing that we should talk too much about because i'm ah, it's it's It's terrible in every aspect.

So anyway, so there's just impossible for two parsers to parse a URL the same way. So you, so, so basically, this is a way to, if you're going to use curl anyway, and you want to manipulate and work with URLs, it's a pretty convenient way to just do that.

Jonathan: So true rel uses the exact same parser code that curl uses.

Daniel: Exactly. So it's based on the exact same URL parser. So it's just a small command line tool for manipulating and fiddling with URLs.

Jonathan: And then what is wcurl?

Daniel: So wcurl, back to the, what's the difference with wget? So one of the most questions, so now people are going to giggle and tell me that the only reason, or the reason I use wget is because then you can type wget.

And apparently you can do that with your left hand only on a QWERTY keyboard. That's, that's apparently a benefit. But anyway, and then you can type a URL. So you can type wget and a URL and it will download it. And you know, you don't have to remember any options at all. It'll just do that magically. And with curl, you actually have to remember some option, capital O.

But people don't do that. So, so someone then introduced the wcurl. Program is actually the replacement for W get because you can type W curl and the URL or many URLs actually, and it'll download that URL to file on disk. So basically the W get version of curl.

Jonathan: Yeah, if you wanted to really have fun with it, you could, you could just add a little bit of code in curl that looks for the binary name that it was called with.

And if it's called as w curl just imply the dash capital o and the people could just make their own sim link and get It free.

Daniel: Yes, that's one way to do it That has also been debated Of course

Jonathan: Of course, i'm not the only one that thinks about but

Daniel: w curl is then only just now It's just a shell script anyway, so it's just a shell script that invokes curl with the right the right way Yeah makes sense Is

Jonathan: is there anything that we didn't ask you about that we should have that we that we neglected to cover?

You

Daniel: Not that I can think of. We covered a lot of area here. We did. We did. I think we're good.

Jonathan: Yeah. Okay. So I've got to ask. I don't know if we were doing this. Randall, do you think we were asking the two questions way back then? I

Randal: invented

Jonathan: the two

Randal: questions.

Jonathan: So,

Randal: yes.

Daniel: And we can, if it's the same questions, we can compare with 2009.

Jonathan: Yes.

Daniel: Yes. Yeah.

Jonathan: So, favorite, favorite text editor and shell script?

Daniel: Well, text editor is still Emacs as it was in 2009. I'm pretty sure that's not going to change.

Jonathan: It's amazing how many people are like, yeah, it used to be VI or Emacs back in the day, but I use VS code a lot now. Yeah, we don't like that. No, I used to be, I used to be as good a lot now too.

Daniel: well, I can see Randall is on my side here

Jonathan: in shell script,

Daniel: shell script. Yeah. When it

Jonathan: comes

Daniel: to scripting, I tend to default to Perl actually. I'm one of those old, old people who still do that.

Jonathan: You're two for two, Randall.

Daniel: Yeah, so yeah, exactly. And I'm pretty sure I said the same things in 2009. Pretty much you could imagine that not a lot of things, you know, I'm writing code in C still too, so my life is pretty the same.

Anyway, you know, I type make in terminals. I use Emacs. Yeah. 16 years later here I am doing the same things.

Jonathan: And you just have more cores and more memory and more hard drive space to do it with now.

Daniel: Exactly. So, yeah, that's true. That's true.

Jonathan: That's and a lot

Daniel: more code to handle too.

Jonathan: Yeah. Yeah. Yeah.

Alright, Daniel Sandberg, thank you so much for being here. This has been a blast and we, we sure appreciate it. It's been really good to catch back.

Daniel: It was fun.

Jonathan: Yeah. Alright Randall, what

Randal: what do you think? He may have more disk space, but I bet he has the same percentage free. Oh, it's sad, but true.

I have seen this as a constant over the decades now. It's sad, but true. It's, it's somehow, for some reason, it always ends up being about 80, 85 percent free. Full no matter how many outside discs you bought and how many things you've wired up. It's always 85 percent full They should maybe they just ship them that way.

I don't know. Maybe that's Maybe i'm not actually getting empty discs. I'm only getting discs that have some a bunch of stuff already on them. But yeah But no, no, that's, that's, that's guaranteed. And just to, just to update. Yes, I do use VS code primarily. Now I do hierarchy max occasionally. I'm still happy that there's a little bit of me in every copy of Guru Emacs that goes out.

Cause I wrote the project for Guru Emacs from many, many decades ago. It now seems, and I do have Perl scripts doing backend work for me, but I don't actually, I don't think I've written a line of Perl and probably I don't know, a couple of months, six months, something like that. So it's all been Dart and Flutter because it's my new.

Claim to fame on you operation stuff. So about our guest, let's get back to that. Not just me. So, so it's, it's again, this is and I, and I sort of mentioned it in the show, it's like, it's, it's this fascinating thing where you invent something small and useful, and then you share it with people and they also find it useful and it grows sort of organically on its own.

It wasn't like either Linus or. Or that guy, the other guy Stallman. No, no, no. Our guests, Daniel, Daniel, see it's the D words. I kept thinking David in my head now. So that's really messed up. It's cause I, just before I was on this show, I watched something with Larry David in it and it just, it just messed up my brain.

So, so so these guys, they start out, you know, not knowing That what they're going to do is eventually going to have such a huge impact. And, and I'm, I'm happy that we live in a world where that can happen and that And, and, and yes, to give to give some credit to Stallman, the idea of fighting for free software, which became the fight for open source software, which is less restrictive,

which

became an opportunity for us to do what we're doing and, and for the world to be expanding and building upon each other's work, at least at the, the software.

Engineering level. So

Jonathan: yeah,

Randal: yeah, it's, it's, it's really good. And it's, it sounds like he's had a great time with the project too. He's still with it. So that's that means he's, it's probably going to stick around for a while. And he's probably gonna stick around for a while around it, which is good.

Have being a BFDL, you kind of have to do that, but, but you know, he could have abandoned it a long time ago, or at least having some fun. Making changes to it and fixing things and adding new features. I didn't realize they could do IMAP. Cause like I said, I needed to F I need to diagnose some problems with my IMAP servers.

So I was trying to do it the hard way, watching stuff on, you know, typing the right things with SO cat and things like that, trying to get that working. So cat can kind of get you there, but not the same way as having the whole protocol. Yeah.

Jonathan: I'm, I'm just, I'm so delighted that he's able to make a career out of it and he's, he's not, it's not killing him.

And he's actually able to make some money from it. And I just, I think that's, I think that's great. I wish, I wish more of our foundational tools, we had that sort of success story. Because we don't for all of them. Right? There's, there's some of them where somebody has been, you know, just maintaining it thanklessly and not getting paid for it for 20 years.

And I like these, I like these a lot better.

Randal: Well, like, like when we talked to the NTP guy who had basically been managing single handedly the project for, for years, you know and getting an occasional grant here and there to take care of that, but

Jonathan: yeah,

Randal: yeah. So there's definitely that.

Jonathan: Yeah. And then there's, there's even things that are super important that like nobody's even heard of, like the term info files.

Right? Like how many people across the world do you think know how to write a terminfo file? You could probably count them on one hand. Those things are obscure and ancient and super important.

Randal: And then we have entire projects dedicated to things like TZData, you know, which are critical to everybody. Yep.

And nobody knows it's a handful of guys on a mailing list that That you have to listen to crazy requests from governments to say, can you change the DST time next to next week? We can, and nobody will have it working for six months. It's just the way it is. Crazy stuff. Yep. Yep. All right.

Anything you want to plug Randall? Just that I have been doing a lot of online presence to deal with Dart and Flutter. If you're interested in all that stuff, you'll probably already know how to find me, but I'm on all the major places. Flutter. dev slash community will lead you to the places that I hang out at and I patrol on a regular basis.

I'm also appearing at conferences and doing virtual talks. I am doing a talk coming up in March. The middle of December, if you're a Pearl person, you already know about the Pearl Advent calendar. Well, the Pearl Advent calendar has been going for like 15 years now, I think. So as a special celebration, I've got like December 19th or 18th.

One of those days right in there is going to be me doing a live presentation of My half life with Pearl or half my life with Pearl. I was going to switch the two around to kind of make it funny. Half my life with Pearl where, and I gave this talk about 12 years ago at OSCON, but at that point. I was 50 Pearl was 25 and I was able to talk about the relationship between me and Pearl and my company and the people around me and what is it, what does it meant to me, what does it meant to the world?

Because Pearl at that point had already had a significant impact on creating the. com boom and everything like that. So I have this talk that's about a, about a one hour talk that talks a lot about, about to some people behind the scenes history that you've never heard before. To kind of talk about how that all kept interrelating for those 25 years.

So I'm giving a version of that talk not updated for current because I haven't done much apparel since then, but at least it will be accurate as of the, the talk I gave 12 years ago. So, yeah, yeah,

Jonathan: cool. All right. I appreciate you being here, man.

Randal: Yeah. Thanks for inviting me again.

Jonathan: Yeah. All right.

Let me hit the right button here. That button. Okay. So the things I want to plug, of course, is you can get my security column at Hackaday. It goes live every Friday. And then of course we appreciate Hackaday being the home of Floss Weekly. And the show tapes on Tuesdays and goes live on Wednesdays.

And then the other thing that I'll plug is the Untitled Linux show over at Twit. And that is live every Saturday afternoon, we have a whole lot of fun with that as well, and you should check it out if you can. So to everyone that caught us live, and those that get us on the download, we appreciate it, and we will see you next week on Floss Weekly.

Episode 807 transcript

30 oktober 2024 | min

FLOSS-807

Jonathan: Hey folks, this week Dan joins me and we talk with Josh Bressers about ANCOR and open source security. We talk about the kernel and its many CVEs, the state of the CVE system in general, what an SBOM is and why you should care, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 807, recorded Wednesday, October 29th, bitten by the penguin.

It's time for Floss Weekly. Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and it's going to be a lot of fun today. We're talking about Angkor and security and open source. Surely that's not a thing anybody has to worry about. Uh, that said a little tongue in cheek for those of you that don't realize.

Uh, first off, we've got. A great co host today. The wonderful, the original Linux outlaw, Mr. Uh, Dan Lynch, Dan, welcome to the show. Hey Jonathan, how are you doing? I am, I'm doing great. We were talking just before we started that, uh, this was kind of an early morning for me. So one of two things are going to happen.

Either I'm going to get tired and kind of get a little loopy and punch up happy during the end of the show, or my energy level is going to drop and it's going to be real draggy. So we'll, we'll see. I'm feeling pretty good at the moment. I've. Got the go juice, the coffee to keep me going. So

Dan: that's good stuff.

Yeah. And, uh, I managed to make it on time, even though the hours different here, our clocks have changed in the UK. It's an hour earlier. I once on, uh, on plus weekly with Randall years ago, a long time ago, I turned up an hour late because I didn't realize that the U S clocks changed differently to the UK ones.

Jonathan: I remember that. That was fun. So when we were setting up, when we were setting up for the show, um, In the past week, I made sure and sent both of you guys an email saying, Don't forget, time change is a different weekend in the US and the UK. Yeah, it's, it's no fun to, it's no fun to show up to do the show and not have everybody here.

No, no. Uh, Dan, this is actually a guest that you sort of brought to us, isn't it?

Dan: Yeah, so I was at odd camp recently, which a guest of the show only a few weeks ago now you had Gary on to talk about that with Simon as well and I was at camp and I met my old friend Alan Pope, a poppy as he's known online and he he kind of hooked things up and suggested, you know, the guest for today and I'm really pleased that he did because it's going to be a great show.

Thank you

Jonathan: Yeah, so we've got Josh Bressers, who is the VP of security at ANCOR. And from what I understand, ANCOR does a lot with open source tooling, security tooling around open source, which, I mean, this is sort of a match made in heaven for me. There's a couple of things I really care about, and so I am super interested to hearing about it.

Uh, let's just, let's go ahead and bring him on. Um, let me push the right buttons to make that happen. There we go. Josh, welcome to the show. Awesome. Thank you so much. I'm super excited. Yeah, it's gonna, it's gonna be fun. Now, you are the, you have a VP of security at Anchor. What, let's start with this. What is, what is Anchor?

Like, what is the, the, the, the problem space that you guys as a company are in?

Josh: Yeah, for sure, for sure. So, we have a product, an enterprise product, built on a foundation of open source. We call ourselves, like, Next Generation Software Composition Analysis. Everyone has Tons of open source today in their environment.

And so we have tools that help you scan into your stuff, basically, figure out what the packages are, and then you can do vulnerability scanning. You can do, like, policy management and enforcement, things like that, plug it into your CI system. You know, there's, we, we, the, the, the US federal space runs us a lot.

There's tons of compliance in that universe, and they're running open source absolutely everywhere. It's everywhere. Mm hmm. Mm hmm. And so you've probably heard of things like, you know, like SSDF and STIG like, we have a FedRAMP webinar in a couple of days to talk about all this stuff. And it's just, it's a, it's a strange universe.

And I've been in open source just literally forever at this point. I mean, more than 20 years. And so it's super fun. We've got these two open source projects, SIFT and GRIPE, that get quite, quite a lot of attention. SIFT is an SBOM scanner. You can point it at containers and directories and whatever you have.

It will generate a software bill of materials for. Kind of whatever and then you can take that software bill of materials, and that's the thing that you can then, you know, look at for policy enforcement you can scan it for vulnerabilities. You can just ferreted away for later like that. The famous example is log for J right where we all we are.

Everyone has battle scars from that. And if you had. A catalog of all your open source, you know, do I have log for J means you go to the search box and type log for J and you're like, oh, there it is over here. Let's go look and see what's going on versus saying, do we have log for J? Like, I don't know if we have log for J.

Someone should go find out. Yeah, it, it, so it's, it's been wild. And in fact, it's funny because I started at anchor, I think like a month before the log for J incident happened. And so like it was trial

Jonathan: by fire.

Josh: It was amazing.

Jonathan: Yeah. Yeah. Yeah. So, so SIFT is the S Bomb finder. What is, what is GRIPE then?

Josh: Uh, vulnerabilities.

So you can either point it at the same artifacts, be it directories, containers, whatever, or you can just hand it an S Bomb and the, the, the magic there. Is so you scan something for an S bomb, right? And your software bill of materials theoretically doesn't change because you've got your container or your release or whatever, but the vulnerabilities do because there's new vulnerabilities found every day, every week, whatever.

And so you need to keep re scanning things for vulnerabilities over and over again. And if you have to scan, let's say like, you know, a multi hundred megabyte container that can take, you know, 30 seconds, a minute, whatever, depending upon your hardware. Versus if you've already done the scan and you have like your software bill of materials.

You can usually scan those in milliseconds. And so there's like a huge advantage just hanging onto those documents and their JSON, right? Like maybe a couple of megabytes max.

Jonathan: And so gripe is, is mainly just going out and looking at known CVEs and comparing it to the software and the versions in your, in your S bomb.

Exactly, exactly that, yes. Yeah, it seems like the world kind of woke up to the idea of, boy, we really need to know what software we're running. I think Log4j was really, yeah, like you say, it was the point where everybody said, oh hey, SBOMs might be a good idea.

Josh: Yeah, yeah, for sure. And well, so I think what happened in Log4j is, so I'm, You can imagine I've spoken with many people in corporations and open source, like all over the place.

And I think what happened is a lot of organizations didn't realize how much open source they actually had in the log for J incident happens. And they, you know, send out the minions to go figure out where, what our open source is. And then they're like, holy crap, where did all this stuff come from?

Because, you know, any, any open source nerd. We knew open source had been taking over the world for literally decades, and we knew it was running everything. But I don't think a lot of the, we'll say, business folk had quite figured out what was going on until log4j. And so I feel like that is truly the moment all of the businesses, all of the people were like, holy crap, this is just literally everywhere.

And that's the moment. I knew some people that were like, can we stop using it? And they were just laughed at because like at this point, no, it's completely out of the question. Right.

Jonathan: And unless you want to turn all of your logging off for your product,

Josh: all of your everything. I mean, it's wild because there's all these surveys that keep coming out.

I mean, I think I read one from red hat not too long ago, where they said 80 to 90 percent of all enterprise applications. Our open source underneath and honestly, I have no doubt, no doubt at all that that's the case.

Jonathan: Yeah, yeah, it seems like all the, all the compliance of, you know, you, you mentioned this sort of almost like an alphabet soup of the compliance things.

Some of those I have heard of and some of them I haven't because that's, I'm not in the enterprise or federal space really. Um, but it almost seems like that's just. That's the price that we pay for having made it. We're here and open source is not going away. So it's time to play the game the big boys play.

Josh: Well, okay, so that's a really good point. So there's a famous blog post. It's called, I am not a supplier. A fellow named Tomas de Pierre wrote it back in the log for JDAs. It's been almost, I think it's been two years now since he wrote it, which is like blowing my mind because I feel like he wrote it a week ago.

But. In that he talks about the fact that he was getting these like questionnaires from companies saying, you know, oh, you're my supplier Do you do you have log4j and your stuff and he's like, I'm not your supplier Like I'm some dude who put some software on the internet Like that's not the relationship and I think that's another really wild aspect of this is the open source developers Like they technically don't owe anyone anything, right?

But if I'm an enterprise using open source I'm the one responsible for that because, I mean, if you think about open source, I often say it's more like a natural resource, right? We're like, you're going to harvest lumber from the forest. You don't ask the forest for things, right? Like you're responsible for the things you'd get.

That's a,

Jonathan: that's an interesting analogy. This is something that I know the Europeans are having to sort of wrestle with as they're, they're looking at making some legislation about security and software and wrestling with trying to, trying to fix that in law. That's right.

Josh: And they have two things in Europe that we're keeping our eye on.

And like the compliance universe is there's the CRA, which is, I do not remember what it stands for anymore. I apologize. But that's the one that's talking about, like, you need to track yourself or you need S bombs. You kind of need all that stuff. And then there's another directive called NIST 2, which is more of a, you know, you're an organization, you're a company and you have to follow certain cybersecurity requirements.

And like, Knowing what you're shipping and providing security updates and things like that are part of that. And one of the challenges in the open source world has been the EU directives initially just said like all software and the open source people were like, Whoa, hold on there. Like you can't have this work.

Yeah, right, right. Exactly. And so, I mean, places like the Apache foundation and the eclipse foundation and like the Linux foundation, they really did a nice job of fixing a lot of those problems. So now there are. You know, there, there's sections carved out that say like everyone except open source type things, which is good.

Cause that's what we want.

Jonathan: Yeah. I mean, our, our own Simon Phipps was actually part of that. He would keep, he's in UK, but obviously he would. Several times went to Brussels and was talking to the people making the laws and trying to educate them on you know Here's how this thing called open source works And here's why what you have written this this piece of what you have written in this law just does not reflect reality at all That's right.

That's right. It's some of the you know, I I I Did I read the entire CR? It's the Cyber Resilience Act.

Ah, that's it. Thank

you. At one point, I read either parts of it or the whole thing. I can't remember which for some coverage of it. And like, there's some really good stuff in there.

Yeah. There's actually

some really good stuff in there.

Like, you know, if you are making software, like if you are a company that is making software, if you're putting your name on it, you are required to have a security contact. It's, it's mind blowing how many companies just Don't. It's like, Oh, I found a security vulnerability. How do I tell them? They don't know.

You don't know. Nobody knows. Might as well just drop it on Twitter because there's no better place to do it. You can, you can, you can email their contact at email address, but you'll know, you'll probably never hear back. If you do hear back, what you hear back will be written by Chad GPT in today's world.

I'm experiencing that. Or a lawyer. Well, or a lawyer. Cease and desist. Trying to hack our product. Exactly. Yeah, it's, it's a little, um, so what, uh, You guys, the, the Anchor Company, um, let's turn back to that for just a second. What's sort of the, the space that you're primarily focused in? Because it seems like everybody that is in this space has their own niche.

Um, we, we talked with, uh, we talked with the guys from Workbrew, which that's actually the commercial offering now attached to Homebrew, and they were cool. Like that was, that was a great project to talk about, but they've got a, they've got a very narrow niche, and that is. People running open source stuff from Homebrew on Macs at their workplace.

And they want to be the guys that own that. We've talked with some other people that are all about, you know, uh, JavaScript, you know, the, the, the, the full stack JavaScript, and they want to be able to catch those kinds of patch or watch for CVEs in those packages. Um, and you know, there's some places that kind of seem to get left behind, like the actual.

Linux binaries and packages. There's not as many people working on that, which might be a problem. Uh, but what, what is sort of the niche where Anchor fits in here?

Josh: Yeah. So I would say we're a little to the right. You don't hear the word shift, right? Very often, right? It's always shift left everybody. But the thing we like to do is we work with, we'll say the operation side of DevOps.

Cause we also, it's like, Oh, it's the DevSecOps everybody, but there's still people who are, we'll say a little more operations than dev in every organization. And so what we do is imagine you acquire software from somewhere. It could be internally developed. It could be an external thing, who knows? And you need to deploy it.

And so you scan it when you bring it in and then you know what you have, right? You, you've got your S bomb, you know what it is. And then. There is the vulnerability aspect, which obviously is important these days, because, you know, vulnerabilities are in everything, and then also there's the policy. So the intent being that you have all this software kind of flowing into your organization, and then using the tool Anchor has, Anchor Enterprise, it gets scanned, and then there's policy gates, for example.

So you can say, like, everything looks good, we can just keep flowing it through the system. Or you can say, oh, something is weird. Like, We're going to stop the train here and someone's got to go figure out what's going on and so it's kind of that. And then the other aspect is once you have things like deployed in kubernetes will say again that whole security vulnerability piece.

You can say like I see all of these containers running in my kubernetes cluster. Now, let's reapply the policy continuously, right? That's like you always hear about, you know, continuous compliance and things like that. And this is kind of one of those examples where you can, you can get a report every day, every hour, every whatever.

And you can look at it and be like, okay, all of my stuff is passing our policy. Or you can be like, Oh, holy crap, a bunch of stuff just failed. What's going on? And then the intent is, you know, humans get involved and, um, Kind of figure out what's, what's

Jonathan: up. So is, is Anchor mainly aimed at, uh, the, the container world then?

So can it, could I say, here's my, here's my Linux server that runs my website. I'm going to, you know, point either, uh, either the open source tool or the Anchor enterprise at it and say, I want to catalog every package and every binary that's installed on that Linux server. And I want to know when a CVE hits, is that in scope?

It is not

Josh: for the enterprise product, but with SIFT you could do that, yeah, SIFT can scan just, you can point it at slash and it'll find all the binaries, it'll find the packages, it'll find, I mean, so this is actually one of the really interesting points in this whole universe that doesn't get talked about a lot is there's no such thing as we'll say like just a node application, right?

You've got like the node. js binary that runs it all. You have supporting libraries usually that help with that. To work. You've got your packages on the system. You've got maybe some random binary crap. Someone installed in their direct home directory somewhere. You know, there's like all this random stuff.

And so we like SIFT does a pretty good job of finding a lot of there's still some things that can't identify because like, for example, if you build your own binary, like Maybe we can't catalog that, it just depends how we're detecting it. And so we also have this concept that we're working on. We call it known unknowns, where it's like, I found a thing, but I don't know what it is.

So, maybe someone should go look at that thing and see, like, is it something we care about, or is it something we can be like, Eh, whatever, it's just some random trash someone threw in a

Jonathan: directory. You're almost, that's really interesting, you're almost into the realm of threat detection then, with that sort of a capability.

Interesting.

Josh: A little bit. I mean, I would say there's, there's some strange overlaps in that regard, you know, everything from like observability and you've got threat detection and kind of same. It's one of those. I have all this data. Now, what do I do with it? Right? And in fact, there's a lot of people that take the output of these kind of tools and they just throw them into their Splunk server, their elastic server, whatever.

And then they use it as part of their whole, like, detection process, where they can say, like, Oh, I've got this evidence, then I can follow it around and, and see, like, it started out over here and, you know, that file ended up over there, or whatever.

Jonathan: I could, I could see somebody taking these, these unknown files and just, let's do a mass upload of all of them to, like, Total Virus and see if we get any hits.

Exactly. For sure. Super interesting. So, uh, you're detecting CVEs. We touched on this briefly before the show started, and I think it will be real interesting to get your take on this. So you wear a couple of different hats. You're not just the anchor security guy. You're also the open source security guy.

You've got a couple of podcasts, I think, that cover this. That's right. That's right. I have a podcast called the

Josh: open source security podcast actually today. And we're, I mean, we're like seven years into that one. I mean, we're 400 episodes and change at this point. So yeah, I've been doing it for a while. I love it.

It's a lot of fun.

Jonathan: Yeah. So. You have thoughts, I'm sure, on the, um, the absolute flood of CVEs coming out of the Linux kernel then.

Josh: Yes, I do. In fact, I, so Greg KH was a guest on my show shortly after he started doing this. And I am a massive fan of the work they're doing. I think the Linux kernel is a great example of kind of doing this right.

And it is creating a ton of work and it's a lot of effort on their part and of the people who receive it. So like, if you're on the receiving end of this, I'm I, I, I totally get it. You're overworked. It's ridiculously hard to sort through all this stuff. But at the same time, the reality is these are potential problems.

And that's fundamentally what we're dealing with when we're talking about like vulnerabilities in software. It's like, this is a, a risk that we have to understand. And as Greg, you know, so eloquently puts it, when he talks about this is a Linux kernel is run on everything from like milking machines to literal rocket ships.

And so they can't say, we know exactly how the software is being used. So we can say this bug doesn't matter. They don't know. And so that's okay. But this is where. As the security people on the receiving end, we need better tools that help us categorize and understand what we're getting. And now there's like 7, 000 layers in this.

There's everything from the data in the CVE program is absolute crap and anyone who's ever worked with it will that's not No, one's going to argue with me over that point. Like we need better data. We need better tools. I mean, this is part of what I'm doing is I'm trying to help build better tools to let us deal with all this data and information in a more coherent way.

Like we can't humans have to get as. Far out of this process is possible. We're talking about now this year, we're going to see a little over 40, 000 CVEs across the industry, and that's probably way too low. I have another thing. I think I sent you guys a link to my blog post where I talk about the size of open source.

Like there are 4 million NPM packages, 40, 000 CVEs in 4 million packages. And that's just NPM. Like that feels a little low, you know what I mean? And so it's quite likely that if we. Had good data and we had more research and we had the ability, imagine a world where we had 500, 000, a million CVEs per year, and like all the security people just like, you know, had an accident, but like, that's what we need to think about is like, that's the scale of what we're dealing with.

Like open source is absolutely gigantic. The Linux kernel is absolutely gigantic. And there, there are a lot of vulnerabilities in this stuff, but we have to learn how to deal with it better. Better. That is our new challenge. Now it's not the whole, Oh, I'm going to read every CVE and figure out how it affects me.

Like, no, no, we need tools to be able to say like, this is the CVE. You need to look at today, these other, you know, 7, 000 that came out. Don't matter. Don't worry about those. And that, it's a really hard problem.

Jonathan: Yeah, so I've, I've mused just a little bit that with the kernel, and so you, you touched on it, so what, for those that don't know, what's happened is in the Linux kernel, there's been a push to better report CVEs.

And they have essentially come to the point now to where they consider more or less every bug to be a CVE. And they've got a valid point to this, like, because any bug running in the kernel space, because it's so tightly ingrained with the system, there's a really good chance that it can be used maliciously.

And so, like, that's a fair point. Um, I, I have mused that this might be just a little bit of malicious compliance. I, I, do you think, do you think maybe just a little bit there, there's a little bit of, of, of, uh, would, would that be schoenfreude? Um, So I,

Josh: I, I thought so at first actually, but after talking to Greg, I decided no.

I think like this is like actually legitimately, The right thing to do, which makes me a little sad. Cause the malicious compliance angle like abused me. Exactly. Yes. Uh,

Dan: so with this kind of increased, um, I don't even have a good word verbosity. I'm going to say, which makes me sound clever, but increased verbosity, I suppose, of, of reporting of CVS and stuff.

Is there a danger that people become less. Um, sensitized to them, if you know what I mean, so like the, the, the, the boy that cried wolf or something, do people then start to say, well these guys are always saying there's something going on, and, you know, is, do you think that ever happens, that, that the sheer volume of them now is, is kind of making people think, oh, you know, this is probably nothing?

Josh: I don't think it will because of compliance. I think when you're in any sort of regulated environment, you have auditors that care, and they don't have to do the work, so it's no skin off their back, right? If they, if they have to say, oh, you have to deal with 7, 000 of these things a day, like, whatever. So, I, I think your point could be valid if it wasn't for regulated industry, because yes, you could easily say, like, the Linux kernel is releasing, you know, hundreds of CVEs per week, and actually, I think it's more than hundreds, it's, it's two or three hundred.

Um, but yeah, all these CVEs per week and none of them matter. So I'm just going to completely ignore it. But at the same time, I guess this is actually part of Greg's talking points is if you just follow current, you don't have to care. Right. And I know that's easy to say of, Oh, just upgrade your kernel.

It's not that simple, but if you're trying to follow. More closely than we maybe did in the past, that does make a lot of these problems go away. And I think that's part of his intent, is by just having so many and saying like, just, just follow upstream. Don't try to figure this crap out, because it really doesn't matter as long as you upgrade.

Dan: That kind of goes back to Jonathan's point about malicious compliance a little bit. Is it a good way of forcing people to follow the latest current kernel and say, or more people to kind of go? And, and isn't to get up with.

Jonathan: And is it related to the dropping of all of the long term support kernels?

Because nobody used them anyways.

Josh: Right, right. Well, I mean, that was, that's the business model of the distros, right? Is we're going to support the software for decades. And everything will be fine, except, oh, we can't do it with the kernel now, because it's too big and it's too fast and there's nothing we can do.

So, yeah, I don't, I mean, so dropping the LTS, That's something I watched quite a bit, and I mean, it fundamentally just came down to no one wanted to do the work, right? And I think that's just your classic supply and demand. Like, if there's a demand, then find some money or find some people and make it happen, you know?

And it's also open source, so you can't be like, oh, those darn Linux developers. Like, no, just go do it, right? That's how it works. Yeah.

Dan: Yeah, that's true. That's very true. Okay. So, uh, so josh, I want to back up a little bit and talk cause I, I'm not, I know a little bit about security, but I'm not really up on a lot of this stuff.

So I, for the benefit of some of our listeners, I know we have a very technical crowd who listened to this show, but some of the layman, I suppose. Um, can you tell us exactly what an S bomb is and how it works? So it's a, it's a, it's a, am I right in thinking it's a bill of, uh, what is it now? Well, in fact, why am I trying to answer my own question?

I don't know what I'm doing there. Can you tell us exactly what an S bomb is and why

Josh: it's important? Yeah, I mean, so, it's a software bill of materials, right? The idea is, all of your software has stuff in it. And some of the stuff you wrote, most of the stuff you didn't. And so you have Like you might have NPM packages, you might have Rust crates in there, you might have Go, whatever.

And so basically what a software bill of materials is meant to be is on any system, any application, it catalogs all the stuff in it. So like if you do an NPM install and it installs 100 things, those 100 packages would show up in the software bill of materials. And you can also imagine that you can have like different Kind of pieces of software where you have a, a container image and then you have your software and those, those are two different software bill of materials.

Now you put them together. Now you have a third software bill of materials that is functionally the other two pasted together. And so the idea is just like, it's imagined when you, when you buy something from Ikea. Right. They always have that list of all of the parts that should be in the box. And in fact, I don't think he has ever screwed me over on missing parts, but you know, that's the intent here is if I get a piece of software, what are all the pieces in the box essentially?

And then we can start asking questions about those pieces. And like, one of the really interesting things I like to do is, so if I get an S bomb from somebody, I'll, I can, Look at it and understand it. No, I shouldn't, I should use the word look at loosely because there's usually thousands and thousands of packages of this stuff.

Like I'm, I'm kind of an armchair data nerd. So I just put everything in elastic search and then I do stuff from there, which is not, I know that's not how normal people work and that's okay. That's why we need tooling to help us do this. But anyway, someone gives me an S bomb and says, here's the S bomb for the thing I built.

And then I scan it. And then I say, what's, what do I see? Are they the same? Are there things different? Like, did stuff get added somehow no one knows about? Did stuff get removed? How did that happen? And this is like one of the really cool things you can do, is you can watch your software as it kind of traverses from the left to the right.

And you can be like, why did this file get put here? How did that happen? Or why did we remove these packages over here? I don't understand why, why we did that. And so there's, there's just all this weird stuff you can start doing and asking questions about your software, because, I mean, let's face it for the last, what, 30, 40, 50 years, the vast majority of software has been kind of YOLO.

We're like, we type make and we get something out the other end and we're like, I guess that's it. Like go run that in production.

Jonathan: Yeah.

Josh: It's, it's funny when you think about it, but at the same time, it's like this stuff is like running the world. I mean, I feel like we should know a little more about

Dan: it.

Yeah. Is there a standard format for S bombs? Could I take an S bomb that was generated by SIFT for example, and ingest it into, uh, into another. You know, program another, another, another company's tool or another project's tool? In theory,

Josh: yes. So, Okay. So we are kind of early days still. There are two primary S bomb formats.

There's something called SPDX and Cycle and DX. And They're they basically do the same thing. I mean, I know you can nitpick over Oh, this one does, you know, it records dates a little different or whatever, but what it doesn't matter fundamentally they're the same thing and the intention being that you have interoperability between tools and and whatever and Some of the tools can do this.

Okay, not all of them can because like any good standard nothing makes sense So there's tons of wiggle room And they're working on fixing it, like they're on, I mean, SPDX is I think 2. 6 now, SPD, no, SPDX I think is version 2. 6, and they keep adding changes to this stuff, and they're constantly trying to, I guess, bring the, the formats a little closer together so they work better, like one of the things SIFT does, so, SIFT, if you run SIFT, the default output format is something we call SIFT JSON, And it's funny because the S bomb people like, Oh my goodness, did you make a third format?

I'm like, no, no, we did not make a third format. We are basically just, we just take whatever SIFT has in memory and we dump it out to JSON with the intent. There is more data in that than either CycleNDX or SPDX collect with the intention being you can convert between CycleNDX or SPDX because like if you have an SPDX document you can't today convert it to a CycleNDX document You will lose information because there are those little niggles back and forth So you kind of it's kind of imagine Google Translate where you know, you you can You keep translating something back and forth and you end up with a hilarious output at the end, same sort of problem today where it's being worked on, but I mean, you have to remember too, like these formats are not like, it's not old, right?

We're, we're still very early days here and it's getting better. It's getting better. Thank goodness.

Dan: So you mentioned that the sheer amount of data and so on, is there, I'm going to, I'm sorry to have to invoke this, but I'm going to, um, is there an application for AI in this where you could use something like an LLM or something and train it with some of this data?

I mean, there's people

Josh: looking into that for sure. I, I think it is an excellent question. What can we do with them? Like, I, I haven't seen anything I would say is particularly compelling yet. Just because I feel like this data is kind of boring and There's not a lot of wiggle room. Like if an LLM gives you a, a, a weird version or a weird package name, that's potentially bad, but I know there's, there's interest today in doing things like saying, okay, I've got this S bomb, I've got all these vulnerabilities in it.

You know, what, what should I start working on first? How can we look at this data and maybe unwind it a little bit using something like an LLM to maybe understand the vulnerabilities And your product is like a really good example is so let's say I have a container image and it has a website open to the Internet, right?

That is going to have a very different threat model than if I have an application that is. Downloading a CSV file from a website and then parsing it and uploading it somewhere else, right? And so this is, that's one of the places I've seen some of the LLM efforts look interesting because you can say, like, this is what my application does.

Now, you know, based on this information, look at these vulnerability details and tell me which ones sound like they are going to affect me in a very bad way.

Jonathan: You know, I'm, I'm aware because I cover this stuff for Hackaday that, um, people are using AI in security. Research and it's not necessarily going well.

It's actually, it's actually a real problem for open source projects. They are getting particularly those that have bug bounties. They are getting vulnerability reports that have been generated by AI. And the problem with that, and this is sort of just in general, this is the problem with AI today, Um, they sound very plausible.

And these vulnerability reports, they are like, very well done, well written, very plausible sounding reports, that are totally hallucinated. Yes. Is that something that you're seeing more broadly? Is that something that, uh, I guess you guys are hooked into? To even observe it?

Josh: Not I I'm not affected by that in any way I know some people that have dealt with it and they've they put the hammer down pretty hard because it really really annoyed him a lot However, there is some really cool security research I've seen in the LLM universe where you have Researchers training up models, you know the code models and they're saying like here is a description of a vulnerability I want you to write an exploit for this vulnerability Against this application right here, and that's been reasonably successful, which is a little terrifying sometimes because most vulnerable, like what's the stat?

I think like 3 percent of vulnerabilities are actually exploited, but with some of this technology that could even if it doubled. The workload probably quadruples, right? I don't think we're talking about linear scaling here between the vulnerability rate and the work that the security analysts have to do.

And so that's kind of a weird, scary part. And then there's also people doing research where they're basically saying, I want you to just take this application and find a cross site scripting flaw in it. And that works surprisingly well also, which they are there. We start getting into, you know, the whole like 500, 000 million CVEs per year.

If we have. automated discovery, obviously, you're going to have, you know, a huge influx of vulnerabilities, which we are not at all prepared for in any reasonable way today.

Jonathan: Yeah, boy, you could imagine, uh, uh, you can imagine a scheme where you take a, uh, you take an LLM and you write the prompt that says, you know, find this kind of vulnerability and write an exploit for it.

And then you hook that into on the backend, a fuzzer. That actually tests what it could, what it spit out. And then if you were to feed that output back into the LLM, and I imagine there are people trying to do this right now, but you actually get a positive reinforcement loop that actually works, because it'll tell the, it'll tell the LLM, that one worked, that one didn't, and particularly as you start scaling up the speed in which it can churn through that loop.

Yeah, that's going to find real vulnerabilities. Of course it will. So

Josh: that happened. DARPA had, uh, like basically a security capture the flag event. And there was a, an AI, it was originally done by Carnegie Mellon. It was called mayhem. Um, David Brumley's behind it. He has a company now and he just changed his name.

I can't, it used to be for all secure. I don't remember what they changed it to, but anyway, mayhem was designed to basically look at like a binary. Find problems in the binary. So we're talking about like just operating on straight up binaries. Like we're not even looking at code here. Find problems in binaries, fuzz it, find out how to exploit it, write the exploit, then also write a patch in the binary.

So the other team can't exploit it against you and then going and attacking them with the thing you just found, which is like all automated. It's like, holy crap. This is ridiculous.

Jonathan: Yeah. Well, so that's, I guess that's the next stage of this. When we, when we finally reached security nirvana, you've also got, so you would have to write it, you'd have to write a really good test suite to make sure your program is working correctly.

You run through this process I just described to find the vulnerabilities, and then you also tell the LLM to fix the vulnerabilities, and you feed that back into it as well. And as soon as you fix the vulnerability and still pass your test suite, Might have good code on the other end of it. Hopefully. We can dream.

Yeah, right, right. Okay, so, the many, many, many vulnerabilities found, lots of CVEs. Um, how are the various organizations and agencies that are responsible for Tracking and catalog, cataloging these, you know, you've got NIST, you've got MITRE, you've got the NVD. How, how are they doing with it? Oh my goodness.

So it's, it's

Josh: been a rough year. So, so anyone not in this space might not know this, but in February or March of this year, I forget when it was, NIST runs a database called NVD, which stands for National Vulnerability Database. And what they used to do is. MITRE, which is a government contractor, runs the CVE program, right?

And so people who, like, get CVEs, you get them through MITRE. You can have a company that can assign their own CVEs, but they go through MITRE. Or, like, security researchers can just go to MITRE and ask for a CVE. So those all go into MITRE's data set. And that data is very bad. As anyone who's ever looked at it knows.

So NIST had a program called NVD, and they would take the MITRE data, And they would add, they would enrich it. They would add some version information, some like package information. They would add like severity. They would guess at severity. Their severity scores were often very bad. And they would kind of add this information.

And then they just stopped one day, like no word or warning, no nothing. And, and I, in fact, uh, uh, there's a handful of us security nerds who like, look at this data. And I remember we were like, Do you guys like didn't stop and we're like exchanging notes and being like, holy crap, like they stopped. There's, there's no new data.

What's going on. And then of course we'd email them and crickets. No one knew. And so they just like literally fell off the face of the planet for probably six months almost goodness. And then at the same time you have Sisa. starts a new program, they're calling Vuln enrichment. So now CISA started doing some enrichment.

NVD has sort of come back and they're doing some enrichment, not everything, but some, there's still a giant black hole in the middle where they didn't do anything. And then you also have MITRE trying to ask the CNAs to do a better job of providing nicer data. So now instead of having one, And a half, we'll say organizations providing not good data.

We have three organizations all providing, we'll say incomplete and difficult to use data. And so for, for vulnerability nerds right now, it's like, it's a mess. It's really hard to figure out what to do. It's really hard to understand. I mean, this is a huge challenge we have, because obviously we have gripe at anchor and gripe means vulnerability data, and so we've got, you know, a team of people that are doing their best to try to unwind all this and make sense of it.

And it's. It's not, it's not the most fun, not the most fun at all. We've ever had, but you got to do what you got to do.

Jonathan: So I don't know.

Josh: It just kind of is what it is today.

Jonathan: There's, there's several different directions. I want to go with that first. You mentioned CNA is what's a CNA. They call that a

Josh: CVE numbering authority.

That is where, and this, this is like such a fricking mess too sometimes. So, um, yes, curl, which we've all heard of. So, so the guy behind it, Daniel, um, baggery goes by online. There were all of these CVEs being filed against curl. And he was upset with them because they were kind of garbage and they shouldn't have been filed and he tried to get rid of them, but working with NIST and MITRE is like an impossibility for the most part.

They just, they don't have to listen like their, their, their customer is the U. S. government. So, haha, no. So anyway, CURL has become a CNA. A CVE numbering authority with the sole intent of being able to get rid of these crap CVEs and just not have to file them at all, which is a super heavy handed way to just like match reality.

And, and the thing is like, Daniel's way brighter than a lot of people and like he works on curl full time. So he can pull this off. But like, if you're an open source project that you work on an hour a week.

Jonathan: Yeah,

Josh: there's no way you can do this kind of thing. And so that's one of the complaints I have is just the fact that all of this, all of these organizations, all this data, it's just like such a one way black hole that no one knows how it works.

No one knows what's going on. I mean, I've been complaining forever that this is exactly the sort of thing. That should, like, exist in some sort of open source foundation

Jonathan: and

Josh: have, like, proper bylaws and, you know, proper governance and we understand how everything works and all of that, but I've been screaming into that void for more than 10 years now.

And put it

Jonathan: on

Josh: the

Jonathan: blockchain, right?

Josh: I'm sorry,

Jonathan: I couldn't resist. That will solve everything. Um, you've got a few companies out there that are trying to pick up some of the, like, GitHub really comes to mind. I've been, I've been involved with a couple of CVEs now that, uh, GIF, GitHub has been the CNA for, and they've, they've got a lot of automated tools with that.

I think they do have some human review in there as well. And that's actually been a really good experience. Um, they do.

Josh: So GitHub's doing some very clever things here, actually. Mm-Hmm. So they, they are a CNA, they will assign CVEs. Yeah. But they also have their own data set. They have the GitHub advisory database.

Mm-Hmm. and, no, I'm sorry, . They, they kept yelling at me about this. It's the GitHub vulnerability database holds GitHub advisories. . That's how that works. Okay. It's very important to get this, but they have a team of people that what they do. So actually, I said there were three organizations, there's kind of four, if you consider GitHub, where they have a team of people that look at all of the CVEs that affect the, the ecosystems GitHub cares about.

They focus on things like, like Maven and NPM and Python and Go. And there's a couple of others. I can't, Ruby is one of them. And what they do then is they. Do a really nice job of kind of teasing out what is the ecosystem, what is the package and what version fixes it. And what I love though is if, if you find an error, you just submit a pull request and then they either say, yep, that's it.

Perfect. It'll be like, I don't think this is right. Like. I think this is the thing, and then you can have like an actual discussion in the issue and you can come to a conclusion and everyone can see it because it's, you know, just GitHub and all in the open and their data, quite frankly, is the best vulnerability data that exists today.

But unfortunately, it's just a subset because obviously they do not have infinite resources to do this. So they purposely constrain themselves. But because of that, it is just so good. It's amazing.

Jonathan: Yeah. So I have tried, by the way. I cover vulnerabilities and I've found typos. When in particular comes to mind, I found a typo in a CVE on, I forget whether this is before or after the switchover.

So, you know, maybe it was at MITRE or maybe it was at NIST, you know, things have moved around a little bit over the last couple of years. I found a, a, a, a typo in what version fixed a vulnerability. Like you go to the vendor's website and they say one thing, you go to the CVE and it says something else.

So like, Oh, Hey, I'll get this fixed here. Shoot an email off real quick. Surely they'll get it fixed. I still haven't heard back. I bet it's still wrong.

Josh: Yes. So I've, I used to do this all the time where I would, I would try to get things fixed and I have quite frankly, just given up because what it usually is, is you say, Hey, MITRE.

And yes, you have, like, you can look at the website, you can see the version, right? Like, I have definitive proof this version is wrong.

Jonathan: Yeah.

Josh: And I remember the, the one, like, blazing into my mind when I just threw my hands in the air and gave up is I emailed MITRE, and I'm like, hey, look at this, like, here it is, and they said, oh, no, you have to go to the CNA that filed that, but who's that?

Like, and I, you couldn't tell you now they're putting the CNA details in the CVE data. Thank goodness. But at that time it wasn't there. So I'm like, who's the fricking CNA? And they're like, oh, it's so and so. It's like, so I have to go call, like email this person. And I sent them an email and they never replied.

And I'm like, you know what? This is ridiculous. I'm just, I'm out. So now when I see problems, I just go to GitHub. I make sure it's correct there. And like, I'm happy. And that works.

Jonathan: Yeah. So one of the, one of the other things that has been a big problem for the industry is the, um, the CVSS system, the numbering system, like how, how severe a problem is.

And there's this, what, what, how shall we say it? Um, it's almost a conflict of interest because like the vulnerability researcher is looking for the highest score they can get. Because sometimes they get more payout, right? Like sometimes you get a higher payout because you're working on a bug bounty. So if you find a CVSS 10, you're going to make thousands of dollars.

Whereas if you find a CVSS 5. 5, you'll get a couple of hundred dollars. So like they are, they are, um, there's an incentive. They are incentivized to get a higher. CVSS. And then on the other side of it, you've got, in some cases, the vendors that it's like, they do not want to make the news for having a CVSS of a 10.

0. They want that to be a 5. something. And, oh, getting those fixed when they're wrong has been such a problem over the years, because sometimes they are wildly wrong. Yes.

Josh: Yes, you're exactly correct. And you have researchers will, they'll, they'll mis describe the, the vulnerability. And, and the thing is like NIST has a very limited window into what things should be rated.

And additionally, we're like using CVSS completely wrong. It's not meant to be a, a just like raw way to score this stuff and then rank it. There's, you're supposed to add in other information. Like, how am I using it in my environment? Like, you know what I mean? Yeah. What is, what does it look like? Is it, is it being exploited?

There's actually a system that's newer ish called EPSS, Exploit Prediction Security System. EPSS is really cool. What they do is they, uh, it's part of, um, first. org is where they, they kind of, that's where their home is. And they're collecting like all of this data. And it's not all public data, which, It's like all of this threat data is super expensive and the fact that they're getting threat data for free is amazing.

So I'm not going to harp on that, but they're kind of getting all this threat data. And what the EPSS score is meant to be is it's meant to be like, what are the odds this vulnerability will be exploited in the next 30 days? And so the intent being things with higher odds, those are things you should look at.

And now that's an example where you can like, if it has a high CVSS and a high EPSS, now we're, now we're talking like that's something to look at. But if it has a high CVSS and a low EPSS, it's like, eh, we should probably double check what's going on here. Cause this is weird. And so EPSS, I think has a lot of.

I have a lot of hope for it, and they're on, I think, version 2 of the model, version 3 is coming out very soon, I believe, I might be wrong, version 3 or 4, I can't remember, but they have graphs that show how accurate it's been, and when it started out, it kind of sucked, as all things do. Do when they begin, but they're actually doing a really good job now where their, their prediction models are quite good.

And they update it once a day. This is the same idea of like, you can scan your S bomb for vulnerabilities every day, and it could be different. The EPSS score is going to change every day just because like the threat landscape is constantly changing out there in the universe. And so this is one of those things, like you can look at your EPSS score functionally every day, and you can get a better idea of like, what do I need to work on?

What do I need to worry about? Whereas like, again, CVSS. Once you have a CVSS score, it's kind of set in stone, right? Cause that's just how it works. And that doesn't always make sense. Things change. The universe changes around us and we have to try to adapt to that.

Jonathan: I was, I was pretty excited to see the new CVSS 4.

0, the new revision of it. That's supposed to be a little smarter and, you know, supposed to not give you 10. 0 scores for things that are really, really trivial and not actually that serious. And so every time I go to cover a CVE, I go on, Oh, what's the, what's the CVSS 4. 0 score? Surely, by now, it's been out long enough that somebody's using it.

I have not seen, I have not seen anybody use. The CVS has 4. 0 scoring for anything. So like it, it theoretically exists.

Josh: So the vendors hate it, actually. The secret there is, so for, for we'll say low and moderate things, it definitely bump, it kind of, it brings everything to the middle is what I would say.

The high things are coming down, but the low things are coming up. So the vendors hate it because now things that might've been low or moderate are being bumped into moderate or importance. And like you said, The vendors have an incentive to lower the severity of things. So from their perspective, CVSS4 is an anti feature.

Because it's going to actually create more work for them and their customers.

Dan: Makes sense.

Josh: Yeah, yeah.

Dan: Yeah, so, so Josh, how many people are working on, on SIFT and, and GRIPE? What I'm interested in is the community engagement. Are you getting code in from outside of, of the company? Uh, are you getting pull requests, are you taking pull requests, that sort of thing?

Yeah,

Josh: Yeah, oh for sure. We definitely are like we love pull requests. We love the community. We've got a discourse We've got it's all a github. We've got community meetup every other week I think it is the the open source team with with popy They do like a live stream once a week where they kind of do like some of their bug You know, wrangling to figure out what to, what to do.

I mean, we've got a team of, so I think we have a dedicated team of, I don't know, is it five people, six people? I can't remember exactly what it is, but then obviously you've got more anchor involved as well. And then, yeah, I think we have like hundreds of contributors. I don't remember the number off the top of my head.

I'm, I'm ill prepared for that question, but it's, It's, it, it's pretty impressive, honestly. And there's, we're, we're seeing more and more community involvement, which is a double edged sword though, right? Because you get more bugs, you get more pull requests and it's not like, I always love this. Oh, open source.

We'll give, do our free work for us, but like, it's free, like a puppy. Cause you have to, you know, you have to read the bugs. You have to triage of the bugs. You have to. look at the pull requests and be like, Oh, we'd like to do it this way instead, whatever. And, you know, it's, it's a lot of work, but it's really cool because yeah, we're seeing, we're seeing some pretty cool contributions.

In fact, where there's, um, uh, there's a thing called Bitnami, which is like a container runtime universe. Like there's, they just submitted a, a giant patch to SIFT to properly catalog and index their containers, which is amazing, you know, like that's exactly what we want. We've seen a couple other organizations, you know, Give us, um, we call them catalogers in SIFT, which is where we have the ability to be like, oh, this is Like, um, I think it was just a NET that got added where it's like, this is, you know, a NuGet repository.

So figure out what to do, NuGet cataloger. And, and yeah, like we're, we're pretty pleased with, with how it's going. And, and like, that's the dream, right? Is you start an open source project and ideally you want a community to build up around it, which makes us very excited and very happy.

Dan: Hmm. And I, I know I don't have, I shouldn't have, I shouldn't really have to ask this question on an open source podcast, but I want to anyway.

'cause I think it's always great to get people's opinions. What's the benefits to, uh, Ancor in releasing these things as open source? Because we, we preach it all the time here, but we've, we'd like to. So let you preach it, I suppose. All

Josh: right, all right. So I've got, I spent a decade at Red Hat

Jonathan: and

Josh: I, I spent time at a little Linux startup called Progeny Linux Systems before that, which the, uh, Ian Murdoch, the founder of Debian created a startup and we'll say it wasn't real successful, but, uh, But anyway, so I spent a lot of time at Red Hat.

So like, my career is steeped in open source. And I remember my first job out of college was at a company that wrote life insurance software in COBOL. And this was like right after the dot com crash, so it was dire. I'm like, I will literally take any job. And And I did, but I remember they, like, I'll never forget it.

You know, there were people there that were like, this open source stuff is stupid and it's crap and it's never going to catch on and you're wasting your life on it. And of course being young and, and filled with, you know, optimism, I was like, Oh no, no, it's, it's the future. It's going to win. And like, thank goodness it did.

Cause I'd be, you know, probably broken homeless otherwise, but, but, so the thing is like, Angkor has a bunch of red hat DNA in its blood. Like there's a bunch of red hat folks that are at the company. And so it's. Thank goodness, not something we had to like educate or understand, but fundamentally. So the idea is we've got the, like the, the bottom of the pyramid is open source.

We have sift and gripe and we have a bunch of like this tool called grant. We've got a whole bunch of other like open source libraries that kind of drive all of this stuff. And then, The intent being by making a tool like sift and a tool like gripe open source, you, you get community contributions are awesome and we'd love that, but you also get a certain amount of just kind of engagement in the wider universe, right?

Where you have people running these tools, you have open source projects running these tools. You've got, um, we just had a Allen actually webinar with Google where Google is scanning. Like all of their stuff with SIFT, they're literally scanning, you know, millions and millions of things every day with SIFT.

Like, that's really cool. They have good feedback, obviously, as well. And that's like the power of open source. Right. And then what we did is we took kind of those. Core functions and we built product on top of that because it's easy to say, Oh, you're going to run SIFT across your entire infrastructure.

And if you're Google, you can do that because you have an army of engineers. But if you're a small company working on something or even a medium or even large company working on something, you have to ask, like, is it worth just buying a solution that will do all of the plumbing and all of the monitoring and all of the policy and all that stuff?

Or. Should I hire an army of people that are going to do this as well that will probably cost somewhere between three and ten times as much as just buying a product. And so that's kind of like that very red hat model of there's like that core base functionality and then there's things, there's value added to the top and we've got, you know, we've got support, you've got services, there's, we can help you figure this stuff out because it's weird and hard and, and if there's a problem with your data, like, Open and support issue.

And it's going to come to my team and we're going to tell you what's going on. And often we just fix it because it's like, oh yeah, that data is bad. We'll go fix it. And so that's kind of the beauty of that. And yeah, that's what. It's been a treat being there because I don't have to fight for whether they should or shouldn't be open source or whether this is or isn't a good idea, like everyone totally gets it.

It's awesome.

Dan: That is very cool. Um, so I always like to, to, to ask people how they got involved in, in all of this. How did you get into what you do now? How did you get into open source initially? And how did you get into computing even? How far can we go back with this? You can go pretty far back,

Josh: man. I mean, I've been, I mean, I'm, I'm old.

I've been doing this for a long time. You know, I, I grew up without a computer in the house. Punch cards, was it? No, not quite, not quite that far. Not the punch cards. No, no, I don't. I didn't have access to punch cards. I would have gladly used them if I could have. But no, I mean, the first computer that came into my house was like a, it was a 386 running MS DOS.

Right. So, I mean, we're talking like early nineties, somewhere in there. And I just, it was, I loved computers. I've always loved computers and I'll never forget like my. You'd go to school and you know us old geezers We had like a couple Apple twos at the school and we do things with them and stuff And I remember my my parents came home from a teacher conference one time and they were started yelling at me And I'm like what and they were like you have to stop helping them with the computers because you don't know what you're doing I'm like, what do you mean?

I know what I'm doing. It's like I knew more about the Apple two than the teachers did. Cause like I would read anything I could get my hands on. It talked about computers. Right. And I mean, I remember people laughing when I tell this story, but like, I used to write down basic programs on a notebook paper and I would like, I would run the program in my head.

I didn't have a computer. So I'd be like, what was this program going to do if we run it? And you know, I'd write it out and figure out what the program would do. Which nothing long, of course. course, but like, I, I just always loved it. And so, you know, computing will say for a while, and then this Linux thing you start hearing about, and it's like, wait, I can get free stuff.

Like, well, I mean, we'll say a lot of software was free back then, if you knew the right people, but this is like, you know, free compilers, I can get, you know, a nice. Uh, a Unix terminal that doesn't cost any money and it works and it's awesome. And then I think the real thing that tipped me over the edge was I go to college and they've got sun workstations everywhere, right?

Cause I'm an engineer. It's just sun gear every like the sun, a sun workstation, like 45, 000, right? They were crazy expensive. It was 45, 000 in nineties money. So we're talking a lot of money today and I'll never forget. It was like, there was this. This stuff I wanted to do where there were programs I wanted to run, you had the ability to, you know, do like X windows between systems where you could like run something on the server and you get the X windows.

Window on your system. And I was like, Linux, let's me do all this for nothing. Like this is a no brainer. And so it was really, I mean, I mean, honestly, it was just, I was too poor to buy a sun workstation. That's kind of what really got me into it. If I, if I had a ton of money, I'd be running sun gear at the time, but it was Linux, but then the, you know, the thing is it just, it, it, it gets you, right.

You get that bug of like, oh, I can talk to the people who do this. I can send patches. I can work with them. I can build my own kernel and. You know, completely trash my machine and learn more than I can imagine in the process. And, and it just, I don't know, I caught that bug and it's just, I love the community.

I love how it works. I love everyone helping each other. It's just everything about this universe has been a treat. And like, I, I've built my life and my career around it. And I've, it's been lovely. I, I have absolutely no complaints or regrets and I love open source. It's, it's absolutely amazing. And. Every day I just think like what's the next crazy thing that's going to happen and I'm never disappointed.

We'll say that but yeah, it's fabulous.

Dan: Awesome. So, uh, I just came up when you were saying that you got bitten by the bug. I was thinking bitten by the penguin by a penguin. The penguins don't have teeth though, do they? So I don't know how quite that would work, but

Josh: they have like sharp. Barbs, I saw a picture one time inside of a penguin's mouth.

It's like a Lovecraftian horror.

Jonathan: Oh, that's great. All right. We are, boy, we're starting to run out of time here. Um, is there anything that we didn't ask you about that you want to want to cover and make sure folks know about? Many

Josh: things.

Jonathan: Yeah, I'm sure.

Josh: No, it's, I mean, I think the one thing I would say is, so I, I adore, this universe, like the open source security, all that stuff.

And if like anything I've talked about catches your interest and it's a topic you'd want to discuss or, or learn more about or anything, like reach out. I love talking about this stuff. I will talk your ear off if I can. I mean, I have a podcast just cause I love it. I've actually have two podcasts. We talked about the open source security podcast.

My other one, I call Hacker History, where I just take like old people and I say, tell me your hacker story and that's it. And then that's it. We learn about crazy things from the days of yore. And it's, that was super fun. It's, it's, it's, it's tough to find guests, but I'm always looking for guests. So if anyone wants to tell their story, like hit me up there too.

And yeah, it's just, that's kind of the biggest thing is just, if you want to have a chat ever reach out, I would absolutely love to, cause there's so much going on, there's so much interesting stuff and it's just so much fun to talk about, I love it.

Jonathan: I was going to have you make sure and plug your podcast, but I think you just snuck that in there, didn't you?

I did.

Josh: I did. I'm pretty shameless like that, so.

Jonathan: A skilled, a skilled commentator.

Dan: That's

Jonathan: right.

Dan: In security speak, it was the payload in the, uh. That's right. Yes, exactly. Quickly run off the end of my security knowledge there. I went payload. Yeah. Okay. Show code has been executed. Yeah, that's right.

Jonathan: All right.

So, uh, what's your, what's your favorite then, uh, scripting language and, uh, editor?

Josh: Okay. So I, I was alive for the VI versus Emacs wars and I still run VI all the time. It is. Probably my favorite editor. I think I use VS code, we'll say for anything remotely difficult, but VI still is like, that's where it's at.

It's amazing.

Jonathan: It's amazing. How many people answer VS code to that question either directly or indirectly. It's so good,

Josh: but it works. It does what I need.

Dan: Yeah. It's a good tool.

Josh: And then scripting language. So I, I know many of your guests, because I, I listened to the show. There's always a, is Python a scripting language?

And I, I, I've been thinking about this like since we, you booked me, I'm like, should I say Python? I'm like, I gotta say Python. Cause the thing is, every problem I have at this point. Ends up with a Python script to do it, you know, like I'd love to say shell or like bash or whatever, but it's Python. Like it's funny.

It's it's become like the, you know, Swiss army sledgehammer of programming. I think it little things are big things that it all works. It's it's kind of a replacing Pearl

Dan: for that position.

Josh: Oh, yeah,

Dan: for sure. Somewhere. Randall just exploded. I thought I had a faint explosion in the background. Yeah. Yeah,

Jonathan: that was

Dan: Randall.

Jonathan: We have discovered recently that any language could be a scripting language if you really want it too badly enough. We had somebody doing the Java for scripting, for system scripting. There

Josh: you

Jonathan: go. I remember C

Josh: Shell, right?

Jonathan: Yeah. Yeah. Yeah. It exists. It does. I, man, I am, I am in a group right now that, uh, one of the guys is a huge C sharp fan and I'm sure he, he does.

No, he actually, he does. I'm sure he uses that C

Josh: sharp C shell. There was a shell from long ago that had a C syntax, but it was a shell. I know. Right? Like I remember that too. Yeah. We'll say corn shell and flood us all with horrible memories of each like some pain with your pain.

Jonathan: Oh, that's right. All right, this has been this has been great. I don't want to let you go. It's been so much fun We'll have to head back. I actually wrote a note in our back channel Like next time we need a uh, we need to we need to do a roundtable because a guest flakes on us I'll have to reach out to to this guy and I think our guest from last week.

We also made that note about so Maybe we'll do that, maybe we'll have a running list of guests and co hosts and when we have to scramble we'll just shoot a mass email out. So this is what we're doing, this is where we're doing it. Awesome. Uh, it'll be fun. Alright, thank you man, I appreciate it so much, it's been glad to, it's been really good to talk to you, uh, Josh Bressers everybody, thanks for being here.

Thank you much. Alright. Uh, what do you think, uh,

Dan: what do you think Dan? I thought it was great. What a great discussion. As you said, uh, we, we could, we, we seem to say this on every show, but we could have kept going for a long time now talking about all kinds. I really thought we're going to get into penguins at the end there and whether, you know, with the whole penguins and teeth and all that, I was, I could have gone in that direction.

Um, I was, yeah, I was interested. Something I'll, I'll ask, I'll have to ask Josh when he's talking, I'll check his podcast out, the hacker history one, because I'm really interested in the whole phone freak thing and from the seventies and the sixties and all of that original. Audio hacking of, of phone switches and so on.

So, so I'll ask him if he's got any, had any of those kind of people on Yeah. Uh, yeah. Fascinating guest. Fascinating guest.

Jonathan: Yeah. That, that is, that is really interesting history. And if he hasn't, we'll have to make sure and request some. Um, there are, there are still some people out there that are active. Um, right, right.

That, uh, one guy that I've, I follow on Twitter and he, he. Evan Dorbel is what he goes by and just some great stuff. The cool thing about Evan is that he, he was also like an audio, he was a phone nerd, but also an audio nerd. And so he has recordings of the phone system from way back in the day. And so like, he's got these narrated clips of.

He, this is, this is what you hear when you do this and this is what the equipment on the other end is actually doing. And it's, it's really cool. So we'll have clear all clips from like all the different exchanges and like, here's how this one sounds different from this one. That's why. So Josh, I know he's still listening to us.

Josh, if you haven't had him on yet, see about Evan Doorbell and some of those guys. That'd be a lot of fun. Uh, alright, uh, do you wanna plug anything Dan?

Dan: Um, I want to, yeah, I mean, uh, go to my website, danlynch. org, if you want to find out what I'm up to, and, um, I'm going to do a community style plug, because, um, I run our, I'm one of the people who run our Linux user group in this area, and I think everyone should, you know, uh, the whole Linux user group movement kind of has lost, you know, Momentum in the U.

K. And I'd like to say, you know, if you're interested in Linux, have a look, look dot org dot U. K. Find out where your local log is. Or if you're near the northwest, come to Liverpool. We run Liverpool log. We have stuff on every month. So check that out.

Jonathan: The Linux user group scene has just moved online. It's still, it's still alive and well, it's just It happens in Slack and Discord and a little bit in the IRC and Telegram and

Dan: Signal.

I have been to Linux user group meetings with people all like 10 people sat in a room all hunched over computers like this talking to each other online and I think, why are you here? You could have done this from anywhere, you know, you're not even talking to each other, you know, anyway, that's just my own personal gripe.

Jonathan: Yeah, we, we nerds are, we're a weird breed sometimes. All right. Well, thank you, man, for being here. I have a couple of things that I want to plug, and, uh, of course, the first off is Hackaday. We've got my security column. If, if this is not enough security for you, you can get even more of it, week to week coverage there.

It goes live every Friday, uh, and we also sure appreciate Hackaday being the home for Floss Weekly. Um, there's also the Untitled Linux Show over at Twit. If Linux is more your thing, you can find our weekly musings about that over there. Uh, we do not yet have a guest for this upcoming week. on the 5th, but on November the 12th, we're talking with Frank Delaporte about Pi4j, which is another Java thing, but it's, it's doing, uh, it's doing Java on the Raspberry Pi, and specifically, it's about being able to access all of those peripherals, you know, GPIO and SPI and I2C and UART with, with Java bindings.

And so that, that sort of, that sort of tickles my, my interest, interested bone I've been working with. Some of that stuff myself here recently. Um, and then one other thing that I will let folks know about is, uh, later today in my time, I don't know when all of this will go live, but, uh, I'm going to be interviewing with Brody over.

He's got a, the YouTube channel. Um, is it just Brody Brody talks Linux? I don't, I forget what his channel is actually called. You can search for Brody. You'll find him. One of the other things he does though, is he does tech over T where he interviews folks. And I. commented on something on, on Twitter slash X and he sent me a PM.

He's like, Oh, by the way, I've been meaning to invite you to come to this. So that is happening later today. So watch for that. If that's something that you will find interesting. Uh, we sure appreciate everybody that's here. Those that caught us live, those in the discord, making notes and commenting back and forth and all of y'all that get us on the download.

Sure. Appreciate it. We will see you next week on Floss Weekly.

Episode 806 transcript

23 oktober 2024 | min

FLOSS-806

Jonathan: Hey folks, this week David joins me and we talk with James Smith about Manifold, which is the host your own repository for 3D printing. It's just recently been wired up to talk with the rest of the Fediverse and it's got a lot of really neat features going on you don't want to miss it, so stay tuned.

This is Floss Weekly episode 806, recorded October 22nd. Manifold, the dopamine of open source.

It's time for Floss Weekly. That's the show about free Libre and open source hardware, hardware, software, software today, software about hardware today. I'm your host, Jonathan Bennett, the oftentimes tongue tied, and I've got a co host with me, of course. I've got Mr. David Ruggles. Welcome, sir. And we're, we're talking today about, well, about Activity Pub once again, but not Mastodon, not directly Mastodon this week.

Talking about Activity Pub and 3D printing. David, I, I know, let's see, you're on Mastodon already? Yes. But not, you don't have a 3D printer yet?

David: No, I do not.

Jonathan: Okay. But

David: it's something I'm interested in, it's just one of those things that it's down on the priority list and there's never enough time.

Jonathan: We are going to do our best, I think, to talk you into getting on the 3D print bandwagon today.

For this, I'm sorry, and you're welcome. Um, let's, let's go ahead and bring our guest on as well. It's Mr. James Smith, who is part of the Minifold project, which is all about curating and storing 3D printer models, and then sharing some of that metadata on, on Mastodon, on ActivityPub. It's it maps somewhere in there.

It's something like that. James, welcome to the show.

James: Hello. Thank you very much for having me.

Jonathan: It's great to have you here. Yeah. I was, I was saying before the show got started that I was, I was excited about this because I like Macedon. I like activity pub just like on, on its own. The, the idea of it, I think is really great.

And then I am also something of a 3d printing nerd. And you put the two together and it's, it's just, it's just great. I recently picked up, and we're going to try not to make this all about 3D printing, we're going to talk about the technology too, but I recently picked up a a Bamboo Labs Mini, and it lets me do some really fun stuff like this.

You know, I got it with the multi material, and so, you know, you can do a single print that's got a couple of colors in it, and so I've got a, it's actually a Meshtastic radio I printed out a case for, and I've just been going nuts with stuff like that. Because it's such a huge upgrade over the old printer that I had.

And I was, I was telling, I was telling James before the show I, one of the other things I do is I print out some miniatures for tabletop gaming. And I purchased a I purchased a miniature from HeroForge back several years ago. And I, I printed it on my, as has been called the, my first printer. The, the very cheap model that I started with, and then I forgot about it.

And then I got the new bamboo and it's, Oh, I need to go back and try to print that model out again. Oh, where is it? That was like, Two desktops ago. And so I have on the other side of my desk, right over there on the floor, my old, huge desktop is out and running to be able to pull model files and other files off of it now.

And I was telling James before the show, it's like. Why hasn't somebody come up with a repository you host your own yourself that you can put all your models into And that's what many fold is isn't it?

James: That's it. That's it. I mean you've you've described the journey that That made it happen. I think really which is that I've got too many models lying around.

I don't know where any of them are or what they are so I need some way of organizing them And that that became A little side project and that grew into Manifold.

Jonathan: No, so it is, it is Manifold, not Minifold.

James: It's, it's, it's an interesting one. It's kind of between the two because it's, the name comes from like, Manifold being, you know a property of a 3D If it's got no holes and things like that, it's what's known as manifold.

And it's many because there's lots of instances and things like that. So I, I kind of sort of. Drive straight down the middle between the two, but I think, as I said earlier, it's I, I use it written down a lot more than I say it. So I, I don't, whatever. It's fine. There, there,

Jonathan: there is a long tradition of open source projects having names that are very easy to write out and sort of difficult to pronounce with various takes on how to pronounce them.

James: I think of it as a, it's a wide, it's a wide cluster of how to pronounce it. It's fine.

Jonathan: And, and, and manifold has a tie in with activity pub. Is it activity pub or Macedon in particular?

James: No, it's activity pub. So this, this is really, really new. So manifold's been building for three years or so. I've been building it.

Like I say, initially as a side project and and then managed to get some some open source funding for it. But one of the things that I got a funding for was to add in the activity pub integration so that not only can you have your own sort of collection and and organize it and things like that, but also you can connect it up to others.

So yeah, it's pretty new. We properly, properly joined the Fediverse, I suppose, couple of weeks ago. Which is, which is really cool.

Jonathan: Oh, wow. It's that is very new then. So this was not, this was not an original feature. Okay. Okay. So

James: no, no, no, it was always something I thought like, Oh, it'd be cool if I could make it do that, but yeah, it's, it's taken a while to get here.

But but yeah, it does it does do it now. Yeah. So what is the, what does the

Jonathan: integration look like? It's like, what, what what gets posted out to the Fediverse? Is it like, I printed this, I printed this and here's a picture of it. You know, what, what does that look like?

James: So at the minute. So we're taking a very sort of agile approach to to getting features out at the moment.

We've got the sort of the first things that will that will happen in terms of ActivityPub is that if a creator adds a new model, That will get posted as a as a note across on things like Mastodon and so on. So you can subscribe to creators on public manifold instances. You can subscribe to them from Mastodon or other, any other Fediverse software, and you'll get the notes when they release things.

There's a lot more to be added in there. One of the things that the the instances will be able to do is actually directly sort of follow both creators, but individual models and updates collections, things like that to to follow those on other services more directly. So it's not just like, oh yes in my Macedon feed I saw a nice little post.

It's actually like a link. You know, it's like a single archive almost, so that's what we're building towards.

Jonathan: Yeah. Is all of this opt in or are we sort of putting models out there? It's all okay.

James: Oh, no, no. Very opt in. Okay. The everything above Running it as a single user on a private network is, is opt in and controlled by, by various feature flags and things.

It was originally built as a private archive. So I built it as a, a thing to literally just post on a Raspberry Pi in the corner of the room there and be able to catalog a directory full of files in in my browser. But then we've added in multi user capability Lots of other, lots of lots of other bits and pieces.

The, the federation and, and all these things are all optional features. So no, it's, and it will only happen for public content as well. So we've got very fine grained Google doc style permissions on things. So you can say actually, yes, I want to make this item, this creator, this item public. So if you were to look at the, uh, my own.

Manifold instance, which is publicly available on the internet. You can see my stuff that I've published as if it was thingiverse or one of the other model hosting sites, but you can't see my private archive, which is also on there.

Jonathan: Yeah, I was that that was a question that I had. I was going to ask because not every license for STLs.

Lend itself to sharing publicly like that.

James: No, absolutely. I've, like I say, I'm using it to manage my my models that I've published as Creative Commons and things like that. But also it's, it's stuff that I've bought from creators on Patreon and things like that which are, yeah, which are not for sharing, so.

Yeah, handles both of those situations.

Jonathan: Yeah, now is it Is this simply for STLs or are there other file types? Can we, can we load G code in there? Can you do object files?

James: Yeah, it will. Yeah. So a whole load of of model files are supported STL OBJ, lots of sort of older formats 3MF which is finally a decent format for these things.

And a lot of others, but also images last couple of weeks ago, we added a PDF and text files and video, even because some models come with assembly instructions and you want to have those there alongside it. So but yeah, this yeah, all sorts of things. G code, sCAD, FreeCAD. All sorts of stuff.

Jonathan: Now, I've, I've got to ask, this is something that you see some of the other sites will do does, does Minifold render? Like, if you give it an STL, will it give you a preview of what it looks like?

James: Yeah, so the primary, the primary, the thing I did first, where it started was, I want to point, and this is why I did it in the browser, really, it sort of drove the whole thing, I want to point, A server at that folder.

And I want to see the models that are in it and render them. So it uses three JS. If you, if you go to look at a model it, you get a list of all of the. individual files that are there, it renders them all, you can play with them, zoom around, find exactly which one you want, then grab that file to go and take off to printing, which for, for things with lots and lots of parts in is, is very useful.

Jonathan: In just a second here I'm going to give it to David and I'm going to let him ask the questions that I'm not thinking to ask because I am already a 3D printer nerd. First though, I've gotta, I've gotta This is, this is crazy. And you're going to really, really impress me if you answer yes. Is it also a slicer?

Does it slice as well as dice?

James: No. No, sadly not. No, I mean, the slicer is a massive bit of software. And there's some really good ones out there. So that would be reinventing the wheel. People have asked. There is work that people have done in browser based slicers. I look at that as the sort of thing where it'd be really nice to integrate with some of those things.

So it'd be really nice to be able to have things automatically open up in your slicer of choice. Or to be able, if you do have G code in it, to be able to automatically send that over to OctoPrint or something like that. People have asked about things like the The thingiverse customizer where you could actually do some of the editing of models The scope I think for sort of maybe plugins and things that do that.

Yeah, I I tend to be not a fan of things that grow to encompass every possible feature. It's But if I can make plugins work for stuff then Then that would be that would be interesting

Jonathan: I could, I could even see something like editing of OpenSCAD scripts.

James: Yeah, that's exactly, yeah. Exactly what people were suggesting with the customizer approach.

Jonathan: Yeah, no, that would, that would really be cool. You could also imagine a flow where you've got your, your models saved, and then you open one of them. It moves it to your computer, opens it in, Your slicer, your preferred slicer, Cura or whatever, you render it, you slice it, and then it uploads it back to Minifold.

And then from there, you've got a button to press that'll kick it over to your printer. Like, you can imagine this, this workflow that would be really, really cool. But,

James: it's, it's, I, I've recently switched over to OrcaSlicer, and one of the immediate things that, I loved about it was that when you hit the print button and it sends the file to your printer, it's got a, it's got a tab in the UI for the web interface.

If you're running OctoPrint or if I am, I'm running Mainsail,

The Clipper UI online. So right there in the slicer, it's taken me to The web view of my printer, which is amazing. It's, it's integrated that workflow. So now actually what I want is like, what about the first tab there that actually is pulling in a manifold instance or something like that?

So. Yeah, there's lots of possibilities, I think.

Jonathan: Yeah, exciting, exciting ideas for the future.

James: Too many ideas. That is usually the problem. Yes, that is

Jonathan: usually the problem with open source. David, you want to jump in and ask the, ask some questions that I'm just glossing over?

David: Sure, I can ask the dumb questions, or the any man questions.

So, I've got CAD experience, it's mostly 2D stuff. And, I don't have a 3D printer. It's something that I've looked at many times, and it's just down on my priority list. But, so, Some of the terminology and stuff, I'm familiar with some of it, I'm not. First question I have, and maybe this is for both of you, but you said slicer.

I'm assuming that is something that takes a 3D object and creates like, parts of it so that you can print it out where you might not be able to print the entire 3D object because of its complicatedness, so, so you can basically print parts of it and assemble it after the fact. Is that what a slicer is?

James: Nearly. Nearly. The slicer is actually, so all, all 3D printers, I think all of them at the moment anyway work in layers. So if you've got a a filament printer, you've got a, an extruder and it's, it's, it's drawing a path around like, like there's a lot of 2d CAD stuff like laser cutter or anything like that.

In fact, the the language I think is the same as is used for for things like laser cutters. So it's generating this tool path. And if you've if you've got a resin printer, which is often doing like a it works. whereby you have a screen which is curing some UV resin, and you do that layer by layer and it slowly sort of builds up the model.

And the slicer is what turns that 3D model into a series of Basically,

David: okay. So does your slicer maybe resolution or thickness of each slice? Is that dependent on your printer functionality?

James: Yeah. Yeah, that's right. And different printers will go to different different resolutions. And to be honest, the amount of time that you want to spend as well.

It's I can go down to 0. 04 millimeter. That's a very slow print though. It's a very slow print, yeah. I don't do circle

David: back to what you were talking about before about integrating that slicer functionality are slicers something that is printer dependent or is pretty much any slicer able to work with any printer?

James: The good question. The two, I think, I think on a consumer level, there's two big types of printer. You've got the filament type known as FDM fused deposition model. Sorry. My daughter's knocking on the window. That's

Jonathan: fine. Bye.

James: It's better than a barging through the door, which is what I thought she was going to do. Yeah, you've got the the FDM printers and then you've got the resin and they work in two different ways. One is extruding this big wiggly line of plastic and one is building it up almost like a, you know, like a flip book.

Right. So I think in, in, Until fairly recently, you'd probably have a different bit of software for both. There are now some that are combining those two into one bit of software. But there's lots of different Oh, and all the printers will, will have like, Oh yeah, here's the slicer that we recommend and, and things like that.

There's lots of them out there. There are about Five different forks and minor variants of one, one in particular, which is just confusing to me, but it's but yeah, choosing one that, that works well that you like the UI of things like that. They, but they're all The resin printers are a bit different, but all the filament printers tend to work on on G code, which is this language for, like I said, originally for CNC stuff.

So it's that's a, that's a standard. So they're all creating that kind of kind of thing. So they're pretty cross compatible.

David: Okay, so it does seem like there may be the possibility of somebody developing a plugin at some point that would have that slicer functionality. It's possible, yeah.

James: I think Octoprint had one.

I don't know, I've never actually tried it, but Octoprint did have one, I think at one point. Whether it's still there, I don't know. So, it's been done.

David: So we could go a lot of different directions with this. I read through your bio and the website and you've got just an interesting history. So next time we have a round table and we're missing a guest, I recommend that we see if he's available and we just go into some of these other things.

Come

James: and talk about anything, yeah. Yeah, I've done, I've done, you know what we said about too many ideas, right? That's, yeah. That's, that's me. So yeah, I've done too many different things over the years.

David: You combine the creativeness with the open source and the possibilities are endless. And then you get somebody with the mind for it and you want to try every one of those endless possibilities.

James: Yeah, the difficulty is not starting projects, it's getting them, yeah, remotely finished. I mean, that's been really nice about this one is that it's I have, I'm not going to say I've built a community around it because I don't really think I did anything, but there's a community has appeared. There was one day back in May, I think that someone popped up in the support channel that we have on matrix and somebody else answered the question.

And I was like, Oh my God, that's when you know,

Jonathan: you've made it.

James: Oh, how did that happen? This, this has never happened on any project I've ever done before. So Having, having community to, to keep you engaged and actually get a bit of the dopamine hit from delivering things is really nice. Most of what I do has failed over the years.

David: it's, it's not failure. It's it's learning what doesn't necessarily work in the community. Absolutely. Oh, so how large is your community now? Do you have any feel for that?

James: Do you know, I, that's, that's a really good question. I'm in a way, I'm really proud to say, I don't know because knowing too much about them is, is kind of, you know, a bit of an antithesis of the like self hosting kind of control your own information world.

So I'm taking that as a strength. No, but I mean, there's, there's lots of a few, a few sort of proxy measures, I suppose. I mean, that's about. I don't know, 60 people in the support chat. Normally we've got a few hundred followers now on in, in the Fediverse. One of the things I don't know is how many there are, how many instances there are.

No idea. I I've made a I built an anonymous tracking thing so that people could report. How that their thing is running and how many of them there are first of all, it's really difficult to get people to click a button that is completely optional and, and not shoved in their face. And secondly, I made it so privacy enhancing or privacy preserving that actually, whenever I change the code, it forgets everything.

And that just starts from zero again. So it's very much a proxy. The, the. One interesting question, which I don't know if anybody listening knows, if there's any way I can work out from the Docker download stats, the Docker pull numbers, what that actually means. Is it, you know, if I've got like 50 polls, is that going to be one machine pulling it every, you know, 10 minutes?

Cause somebody has got an auto update on or is it 50? I don't know, but I do know that we get Docker pulls within moments of each release. And, you know, we'll have a couple of thousand over a week. So, but I don't know what that means, but it's, it's going up and that's a good thing. So

Jonathan: yeah. I'm pretty sure it's accurate.

Docker pulls are actual downloads, not just checks for version.

James: Yeah,

Jonathan: that seems right. David.

David: Yeah, I've done some, I kind of don't want to assume that and it only pulls it in when it's built. So, I mean, you might have some script that's checking automatically for updates and then pulling it down, but it's still going to be a new download.

James: Yeah, I, it's kind of, I don't, I'm not sure whether I trust GitHub's counting because it's on the GitHub container registry. Do I trust that their number is actually pulls or that it is? People have just asked what the latest version is. I don't know. So if anyone knows that, let me know. I'd love to know.

But no, we've got enough community going on that that it's keeping me going and it's, it's building momentum and we get contributions, code contributions, lots of feature requests. People report bugs really quickly, which is great. That's actually a good

David: sign of it really is right there.

James: Yeah.

Absolutely. Yeah. Yeah, there was one I, I accidentally pushed in the the previous version. And I think within about two minutes, somebody had said, Oh, on this, you know, on the settings page over here, when you go and change something, it crashes. Well, I haven't changed anything in the settings page. So you're just using this.

You're not trying stuff out. Wow. So yeah, it's great. And that, that meant that I could fix it straight away and, and get a release out in the half an hour. Yeah. That's awesome. I'm a,

David: I'm a web developer, so I definitely interested in that side of things. And I'm sure I'll have more questions as we go along.

But you mentioned that You've done audits against the web sustainability guidelines. Yeah. So what does that look like for you? What, what,

Jonathan: what is the web sustainability guidelines first? Yeah. I only dabble at doing web development and I'm not sure what that is.

James: I, I think most people are in the same boat as you.

So You've heard of things like the Web Content Accessibility Guidelines,

WCAG,

and things like that, which is all these good things that we should be doing to make sure sites are accessible and so on. The idea of the Web Sustainability Guidelines, it's now a there is a W3C community group that's put these together.

I think some, some people I used to work with are involved in it. I, I used to be more involved in the sort of sustainability climate side of things. So it's always been like an interest. I used to run a meetup in London around, uh, using the web for sustainability stuff. So but what they've come up with is actually a, a kind of a draft spec of, okay, here's, here's how you.

You know, these are the, these are the guidelines for, for building sustainable science. It's yeah, just, yeah, if you look it up, you'll you'll find the thing, but it's. It's still in draft. It's very early days. It's not that a lot of people are using it, but it's something that I was interested in.

I've been interested in for a long time, like say, so I was like, well, when I had the opportunity to spend some time on it as part of this open source funding, I, I said, right, we need to do an accessibility audit. We need to do a security audit and I want to do a sustainability audit as well. And so I did, there isn't a.

process as such. So I actually sort of rolled my own DIY assessment type process, very much just like a red, amber, green type thing, going through each item on the list and going, yeah, that's, that's okay. And this one, Oh no, I hadn't thought about that, that kind of thing. And then hopefully do it again, maybe in the new year, see how things have moved on, but it's been really useful.

Just in terms of even, even that sort of very rough approach was really useful in into pointing at pointing out where, where the inefficiencies were, where the, you know, what was going to be more impactful about this application. So yeah, it's, it's really good. And obviously one huge thing with this is that It's 3d models and they can be really big and that's by far the probably the biggest impact of the thing.

So you know, that's that's on my sustainability, Roadmap now but it's nice to treat that as a as a sort of primary thing I think alongside accessibility and and security and and things like that to to have it in mind as As a goal so actually in the release nodes now we have like a special section which is Sustainability and performance because those two go hand in hand.

So when those get Improved it it points it out to people. So

Jonathan: So this basically boils down to not wasting cpu cycles I mean put it very very broad strokes at its

James: most at its broadest stroke level. Yeah, network transmission all that kind of thing But it goes into a lot of other stuff as well.

There's a huge overlap with accessibility. Build things that people are able to use and they'll be able to use them more efficiently. There's a big section on product design and things like that, you know, build what people actually want. And then you're not building things that are pointlessly using using power where they don't need to.

No no major current trends mentioned.

That

David: was a very suspicious cough.

James: Yeah. Yeah. I don't know. You'd need something, I don't know, something very clever to analyze what that was.

David: So I

James: used to work in AI and this is, this whole thing is just hilarious to me.

Jonathan: Don't get me started.

David: We'll put that, we'll include that in the red table. That's not for today.

Yeah. So, this, not to go too far down into sustainability, but I assume you can also see it even as a UI UX issue where Absolutely. Think about a web page. If they're having to click through five screens to do one thing, that's five different hits. That's additional. Okay.

James: Yep. And it goes, it's, it goes to that sort of level of pointing out all those all those guidelines of, you know, Around those things and those things that I wouldn't have, you know, I'm a Very much a developer technologist.

So, you know my thinking was like, oh, yeah, where are my data? Where is my server running? Is it, you know powered by renewable energy, etc And that's like one one point is that you know, there's so much else And it goes into all these other things. That really take it and you know a whole team Developing, developing a product can, everyone can do something.

It's really good. Really interesting.

Jonathan: I love the fact that when they made these guidelines, they included a lot of just good hygiene sort of stuff like that in there. So it's useful, you know, it's useful for, for everybody, developers. That, you know, may not have a whole lot of concern about the actual sustainability bit, but it's still useful guidelines.

I think that's really neat.

James: Yeah, absolutely. It's a bit like the universal design thing for accessibility, right? You know, good accessible websites help everybody. Not just, you know, people who need it all the time. It's just good design. So,

Jonathan: yeah. All right, I want to ask about funding because you've, you've talked a little bit about getting some money for things.

And, you know, for some open source projects, that's like the holy grail to be able to get paid for stuff. And it's, it's not quite the panacea that people might think it is, but it is kind of nice to be able to get paid for doing open source work. I believe I also saw that you are a fellow open collective member, which that is cool.

That's where, that's where mish tastics. Open source budget is at and the cool thing about open collective is it's like it's all right there. It's completely open So we can go. Yeah, I love

James: it. We yeah, absolutely. We use that for I also admin, the mastodon. me. uk server which we've got a couple of thousand people on and that's all supported by Donations and through open collective and everybody can see exactly what we're doing where the money is going on the server costs.

So it's great. But yeah, the, the funding is, yeah, it's, it's an interesting thing. I've been very lucky to get some funding from it's, it's basically, it's European union funding through the European union have a research program called next generation internet. NGI and they have various funds within that, like NGI zero I'm never quite sure how it breaks down.

Anyway, that's all administered then by a foundation in the Netherlands called NL net. Who put out calls every, every quarter for it's all. Specifically for open source you know, what you're doing has to be open source in order to to benefit from this, but it can be you know, a theme around bringing control of data back into back into users hands.

It could be a theme around security or, or whatever, but yeah, the funneling, funneling that money into open source Developers is is amazing. So I was lucky enough to get allocated, allocated, get to be awarded some money for that, which has let me spend the last few months full time on this, which is incredible.

I've got another application in, so hopefully it will carry on. But in, in the long run, it ideally needs to build up a, a sort of sustainable base from. From users on, on Open Collective, I'm hoping that's where it goes. We've got now quite a few projects that are sustaining themselves that way.

And it's, it's interesting. People, especially in the, in FedEva seem to be, You know, there, there was always this idea that, Oh, people won't pay for stuff on the internet. It's like, in the early days, I wanted to pay for Twitter. I liked it. I was like, I'll, I'll give you some money, but there was no way of doing so.

So they had some other business model that didn't care about what I wanted. And that, that kind of, no, I am willing to pay for something that I think is good. It's it's, yeah, it's, it's changing, I think. And there's a lot of things doing it.

Jonathan: Have you, have you had reach out or worked with any any of the printer manufacturers?

You can tell us about it. I have no secrets mainly

James: because I have no success on this. I did try. I did try once. I was like, Oh, is anybody interested in sponsoring this thing? And obviously I just went straight in the bin. The one thing that I cannot do is marketing. That's just not, it's, yeah, I don't know how or or whatever.

I mean, that, that would be cool. That would be a really good thing. I think it's, you know I think a natural fit for many of them. It's also a bit of a reaction to what many of the printer manufacturers have done. So. The first sort of big model sharing model publishing site was Thingiverse, which was which is run by Ultimaker.

I think so. I think. And it was, you know, dominant for a long time in sort of the early early days of these things. That's now, well, well, it went through a real. Bad patch where like just search didn't work for like a year and things like that. So people were going to other, other platforms things like that.

It was, it was building up sort of, you know, more more other platforms, but it seems that every printer manufacturer is now bringing out their, their own hosting site. We now have printables, which is. Somebody I can't remember which one is it Prusa. Is it? Yes, it is. It's Prusa. Thank you. And there's there's loads of other I came across another one today, which I think had I don't know if it was Bamboo or whatever, but it was like very much Download the bamboo thing here into bamboo slicer.

Okay, this is this is theirs and None of these things work together, right? So You have creators who will be Publishing on Five, six different platforms now, which is just wild to me. And then you have things like, I can't choose how I use Thingiverse, right? I can't say, Oh, Thingiverse is useful. I can pay for it.

No, I can't do that. Instead I have to watch a whenever I download, there's a 30 second ad that pops up. I mean, it's a static image ad I've seen it in side two seconds and decided that I'm not interested, but then I have to wait. So this, this, they're all trying to find ways to monetize these things, I suppose.

And Corey Corey Doctorow's term which I don't know, family friendly show. I, I won't I won't say I'll let you say it. I is, is definitely a thing, right. And, and. I think we've learned over what we've learned of the sort of the consolidation of the big tech and big social media networks is that these walled gardens aren't great.

So actually publish your own stuff, host it yourself. We make it as easy as possible. And then, you know, syndicate it out from there. So that's very much a kind of a a driving force behind Behind manifold as well. So I think that's something pretty manufacturers should be excited about, but it's also like, I want to stop you all building your own hosting sites.

Although feel free to run one and put your name on it and then join the Fediverse, right? It's that's something they can do by all means. Absolutely. That, that actually brings up, I'd have to see that happen.

Jonathan: Yeah, I would too. It brings up an interesting question though. What, what license is manyfold under?

It's MIT.

James: Okay. So very, very permissive. Very permissive. Yeah. I've long. Got over the the idea of trying to, like I say, nothing I do has ever worked. So, you know, I might as well give it away. No, it's, it's MIT, everything that I've, that I've done with it is, you know, there's other things that we've published as part of that part of the development is MIT as well.

I'm not, you know, looking to make a walled garden and get everyone onto one platform and then, you know. close the gates and and start making money off them. It's just not, it's not a thing. So if I can get people running their own stuff and you know, pay the bills alongside somehow, then that's fine.

Jonathan: Yeah. Yeah. That's the ideal. Do you have any really big instances of Manyfold? Is there anybody out there that's trying to run one as sort of a community instance?

James: I know there are some, I know there are some starting up because a couple of people have asked me in the last couple of weeks about getting things up for community groups.

One of the things that I very quickly put on the roadmap and launched last week was OpenID login, so people can integrate it with their existing authentication systems. And Fortunately, the, the way I'd built it made that very easy, which was nice. So we could get that out. It was a complete surprise feature.

I did not have that in the plan at all. I mean, I don't really have a plan, but you know but yeah, it was really nice to be able to say that actually, yeah, I can do that. There we go. That's the, the dopamine of open source, right. But I know of. I, I thought I had a fair few, I thought I had quite a, quite an archive of models.

I know, One instance that's pointing at 17 terabytes, which is insane.

I've had a few sort of bug reports come in from, from the the person running it. It was like, it gets a bit slow.

I'm surprised it hasn't set your house on fire or whatever. You know, it's, it's a bit slow. Yeah, I, I, I've been impressed actually how well it has coped with, with big instances. I, I'd like to see some, I'm just starting to collect public instances now. I've just put a list on the website of known public instances that people are happy to put there.

Currently, it's the two that I run which is the which is my own one and the demo.

There's a demo instance. You can go and actually play around with all of the features without any authentication or anything. But yeah, it'd be, yeah, I'm looking forward to seeing some, some instances come up, but at some point I'd like to run like a flagship instance so that anyone can just sign up for an account and start publishing stuff.

Yeah. That would be really nice. Yeah, budget is probably the thing at the moment. I've, I've, I've mentioned that as a possible thing in, in the sort of next round of funding as I actually like, it would be great to get a flagship instance up.

Mm hmm.

So,

Jonathan: yeah, is, is there a, is there a way to search across the different public instances of this?

So say I'm looking for a specific model of something and I just want to say, go out and, and, you know, look at all of the many fold instances, all of the things that are public. Is there a way to do that yet?

James: No, not yet. That's a general. So, so there's a few things. One, there's, there's one. main, there's a few main sort of 3D model search engines.

So I think Fangs does a has a search across other sites. And there's one called Yegi as well, which, which does that. I'd like to make it so that manifold instances can be indexed by those. If necessary, so that somebody who is running a search indexer, the stuff will pop up. And then there's a big thing generally about federated search and search of the Fediverse because every instance has this problem, right?

Whether you're on if I search on Mastodon, I'm only seeing what my instance knows about. So how does that work? And I know that there's work happening there, which hopefully I can. I can hook into and build upon in some standards compliant way, that would be ideal. So yeah, at the moment, no, but it's definitely.

Yeah.

Jonathan: And it's something you want to do. I think it would be really cool. I'm looking forward to that. So the the thing that we had to talk about with, with Mastodon, I think it comes up here as well. Maybe, hopefully it does. Hopefully it hasn't come up here, but do you, do you, do you guys ever have any problems with people like uploading inappropriate things or even just straight up trying to spam on Miniverse?

James: at the moment. No, because I don't know of any other instances. But it will be a thing, right? That will be a thing.

Jonathan: That's when you know, you've made it

James: Yeah, definitely. So I mean one thing I did last actually the the sort of current One of the current themes of development that I'm doing now that we are in the ActivityBub world is building in the required moderation and admin tools.

We've got things like moderator permissions in there already. I added in a sensitive content flag last week. But building those things so that it will respond to Reports from other instances give you a, you know, report log, things like that. So yeah, that's been in the, in the roadmap from when we started thinking about federation stuff.

It's definitely something we need to support. We need to be able to control federation as well. So which instances we do and don't federate with or, but that's all individual choice, right? That's not, I say we, it's not. It's everybody separately. But yeah, definitely want to build in those controls and learn from what's worked elsewhere in the Fediverse and what's but also what people have missed as well.

David: It does seem like Moderation around 3D objects is a little bit different than moderation around text. Yes.

James: Yeah, but you know, we've got comments and descriptions. So, you know, Comments and comments. There's text as well. And images.

Jonathan: Images is actually one of the ones that's a little scary. Because if you, if you tick somebody off, they can, there are, there are literally illegal images that you have to potentially deal with.

Absolutely.

James: Yeah. Yeah. Yeah. So, so knowing, I mean, it's been really handy having been a, an admin of a, of a Mastodon instance and bringing that experience to building this and actually going, yeah, okay, I need to be able to say for this server, don't, just don't talk to it and never ever accept any images.

from anywhere over there. That's definitely something we need. So I mean, I say that aren't mastered on moderation journey has been very smooth over on our server. It's been, it's been a nice, a nice ride, but I know, I know it's not that way for everybody. So, yeah. Yes.

David: To get So I know that this could be a potentially deep topic because you do have a PhD in it, but the explain it like I, like I'm five overview.

And going back to that massive survey you were talking about the size of those 3d models and everything you've mentioned in your blog post that you're actually looking at developing ways of 3d compression.

James: Hmm.

David: What does that look like?

James: So, yeah, this is this is one of the fun bits of of the project that I've I've got funding for.

I haven't done it yet. It's going to be coming on hopefully sometime in a month or so. I've really got to get this done. 3d models are big. They're, and some of them aren't. really big. I mean, some, some creators will create quite efficient measures. Some will just have millions of points in there that you really don't need.

So Yeah, things like, and this speaks to the sustainability thing as well, right? These things are expensive to send over the internet. They're they take a lot of space. So one of the things Manifold will do is it'll detect old inefficient formats. So you mentioned OBJ Wavefront. That's really inefficient.

It's all ASCII. It's just, it's massive and it doesn't need to be. There's ASCII encodings for STL, things like that. And we can convert those to 3MF. Which is a nice it's a, it's, it's a sort of more modern format, but then it's also zipped up. It's a zip file basically. So there's that, but then that still doesn't help you sending it over the network because you've still got to send the whole thing.

One of the you know And the internet might've got fast enough that we don't really worry too much about this these days, but you know that PNGs can be, and GIFs actually back in the day, could be progressive. So you could, they've got all the information you need to display them at a low resolution and then it refines and refines and refines.

So there are ways of doing that with 3D models. And like you say, I did a PhD in this years and years and years ago. 25 years ago, probably approaching I started. And It was when it was, it was all around animation of very very dense meshes. And the fun thing was while I was doing my PhD they invented graphics processors and suddenly what was a very dense mesh wasn't by the time we could just do it all with a, with a nice graphics card.

So. What I did for my PhD wasn't massively relevant, but some of the things that I, there are algorithms that I read about while I was while I was preparing to do sort of my research that are, there's a thing called progressive meshes written created by a guy called Hugh Hoppe, who I probably pronounced his name terribly wrong.

I'm sorry. Which basically you can imagine you have like, imagine you've got a cube, right? Or a very simple shape, right? You're going to get that really quickly. And then instead of loading the whole thing, you just say, all right, that, that corner there, split that into like that, split that on that, on that, on that, on that, on that, on that.

And then the thing refines. And. This is a really nice, nice thing, you get this sort of stream of progressive enhancement of this model, which means you can see what it is very quickly, but you can wait for the full detail if you, if you want to. So that's a It's a really, it's a really nice algorithm for that.

And, and I was looking around to see if anybody had done anything with this particularly, around transmitting them over the internet. But, you know, I mean, I was playing around in VRML back in the 2000s, but 3D over the internet hasn't really been a, you know, a huge thing. It's, it's all, you know, in games and things like that.

And you're just downloading gigabytes in one go and then you've got it. That sort of continuous transmission hasn't really been an issue. So nobody had done anything with this particularly in terms of that as a sort of, as sort of as a, as a transmission format. So I got in touch with Guy and said do you know if anyone's done this?

And, you know, we should put together a, a spec to do it. And so I've done a draft spec for it actually as a GLTF extension, which is a sort of format designed for, uh, for sending over the network. People have done mesh compression in that, but it's not this sort of progressive transmission thing.

So I've written a spec, I've got to write the code, and, and actually get it working. But the idea is that if you're looking at a, a very complex model on some other server, you'll actually get a low resolution version of it instantly, and then it'll refine over time. One fun thing about that is that it provides an interesting, interesting thing.

possible approach for commercial models in that I could show you a low resolution version of the commercial model and you can see that and you can look at it and flip it around and all that kind of thing. But actually if you, you know, when you've paid for it, you get the whole stream. And you get all that fine detail.

So, it's, yeah, it's a really interesting thing. It's, it's, it's nice to be able to flex the the 3D graphics brain for the first time in a good many years. Yeah. Doing this. Yeah, that progressive,

David: Concept that you just mentioned, and not just looking at it, but I could envision where it could be something where you could, like, print it in low resolution to test if it's what you want before you spend the money and get the full resolution product.

James: Yeah, that's true. Yeah. Yeah. Yeah. Yeah. Test size, you know, yeah. Yeah, yeah, that's a good point. Super interesting. That's a good point. I haven't really worked out what the sort of commercial obviously there's lots of commercial creators out there, right? So I need a solution if anybody's going to use this thing for that.

I don't quite know what that is yet. But I'd like it to be possible in some way. I'm not, not interested in like payment processing and things like that. Somebody else can handle that. But like, how do I, how do I make sure I've got a permission model that works for. for sort of sharing of a preview and then and then full versions of things.

It's kind of what I'm thinking about. But there's a lot of a lot of commercial modelers out there who who I think would benefit from being in control of their, you know, seizing their means of distribution, right?

Jonathan: All right, we have a, we have just a minute before it's time to wrap and I want to say something we didn't get to. Yeah, it's gone fast. Something we didn't get to. It has gone

James: fast.

Jonathan: I wanted to ask you about is what was the experience like adding ActivityPub? How, how big of a pain was that? Or is that something that is actually fairly simple?

James: It's got some sharp edges. It's no, it's all right. It's all right. There's, there's some good, you've definitely got to learn it and understand it and, you know, read the specs and things like that in order to to be able to, to do it. I think it's not, the basics aren't super difficult. A friend of mine did a a great blog post on how to build an activity pub server in 20 minutes in PHP like in a single page of PHP, which is, you know, you can do that.

There are interesting bits like signed requests, which is a little bit tricky and difficult to to work out exactly exactly why it's not working. Some of this stuff can be hard to debug because one of the things is actually it's all needs to be on the internet and talking to other systems. So I mean, while I was doing a bunch of the development, the deploy process was just the debug process was so slow because I had to build a version, deploy it up to my machine, then talk to other things on the internet, et cetera.

So it was it can be a bit like that. But one of the One of the things that I've done as part of this is I wanted to, so I built the the site's built in Rails which has been a, something I've worked in for many, many years. It's very nice for programmer speed, which for me is a good thing to optimize for.

It's just me on my own. And one of the things you can do with that is it's, you can plug in engines that do you know, that bring a bunch of features in one go. Like we have one for authentication and we have things like that. And I wanted to make a reusable component for activity pub. Mastodon is also built in Rails.

Their activity pub is kind of, well, it was the, the site was built before they adopted activity pub. So it's a little bit It's not sort of easily extractable. I don't think I was kind of going in thinking, Oh, what if you could pull out the activity pub call from masters? I've definitely read some of the code.

It's been useful to see what I've been doing wrong. Mainly actually, why is this thing not responding when I send it stuff? It's where, where is it sending it through debugging through other people's code to, to work out what you've done wrong is, is interesting. But . Yeah. The, I wanted to produce this reusable engine and I found that somebody had already had this idea.

And so I've been contributing to that project. There is a rails engine now called Fed Rails. Hmm. Which, when I started looking at it, it supported things like following back and forth and things like that. It's that was it had some of the basics down. I've over the last little while.

Implemented the the signed activities, different activity types multiple actor types. So in our system in, in Macedon, everything that you can follow is a person, right? That's simple in manifold. It's a creator, which is a person, but also it's a model and it's a collection. And there's all these different things.

So getting multi actor support in there Lots of things. So I've done a bunch of code for that which is, so actually all the ActivityPub core of Manifold is in that reusable engine. So, and, and I'll be carrying on with that and building more and more into it as we as we go. So anybody can pick that up and reuse it.

Hopefully fairly easily. Yeah.

Jonathan: Manifold is built in, in Rails, Ruby on Rails? Yeah. Yeah, that's right. That's one of those languages and frameworks people have very strong opinions about one way or the other. They do. They do.

James: I, I won't start a flight war.

Jonathan: I like

James: it. And that's

Jonathan: what matters to me. That's what matters.

Yes. Mash was data from the chat. Opinions are available. Yes. Yes. I can help supply some of those if you'd like. Master tailor from the chat wants to know what about that other federated protocol at, which that's the one blue sky is working on. Any plans to support that?

James: Not at present. No. I don't know anything.

I, I haven't looked at it that much, to be honest. I know that BlueSky does it. Is there more than one BlueSky server yet? Is there, is that even a thing? I don't really know. I'm not there. That's what I think. We, we've we, I did bridge the manifold. Fediverse account over to Blue Sky the other day, and that's about as far as I've gone.

I don't know. Should I? He asks the

Jonathan: audience. I will say this. I went looking for, oh, have any other open source projects picked up AT and started using it? And as far as I could tell with my quick search, the answer is no. And if that ever changes, I would think maybe you should think about it. But until then, I don't see the point of it.

James: Yeah, I think others. Other bigger, more important applications will pick it up first. And that's when I'll notice and think, Oh, maybe I should do something like that. But I'm always willing to be corrected or to learn more. Yeah. There you go.

Jonathan: All right. So I want to make sure and ask what, where do you guys need help?

So if somebody said, Oh, I know, I know how to program in Ruby on Rails, or maybe even if somebody doesn't, but they want to say, I want to roll up my sleeves and help with many fold. Where's a good place to jump in and, and start at? So

James: it's, so I've tried to make it accessible. I don't know whether I've managed it.

We have a bunch of issues on GitHub that are labeled good first issues that are fairly self contained things that don't involve you understanding the entire application. But it's not just Rails. There is, there is Ruby code. There is plenty of JavaScript code. If you want to do 3d JavaScript, I've got some jobs for you.

There's lots of stuff I want to, to add into three JS for instance. So there's, there's things there, there is design. I am not a designer. The whole thing is built with bootstrap. Because I'm incapable of doing anything better. So, you know, design UX one things we do have people we do have a little team of translators who've stepped up.

We're actually translated fully into four, five, four languages five in total. English doesn't count. French, German. Spanish and Polish Polish, because somebody popped up and said, I want to translate it into Polish. So I went, all right, sir. And, and he did, and that was the first one that actually got to a hundred percent translated.

So people want it in their language. Please do. I would love to have Russian. I, then I think after that it's Japanese. I don't know, I think, but things that I can't understand when they're wrong is, it's definitely, yeah, on the horizon. Although I mean, I can, I can read Russian. I just don't know what the words say.

Understood. But I did it at school so I can read the alphabet, but that's about it. And so but yeah, there's that but also things like documentation, I mean, open source really suffers from. From documentation issues, I've tried to do some. I've discovered that I would rather do almost anything else than write some bits of documentation.

But also i'm a really bad person to write it. I know how it works and how not to break it so those kinds of things like What haven't I explained? Is is not something I can see very well. So There's loads of yeah loads of opportunities for people to get involved. We're on we have a Chat space on matrix.

And there's all the codes up on GitHub and there's GitHub discussions there and things as well. So yeah, if it seems like something that, you know, I, I'm very much of the opinion that there's so much more to open source than there is just code. So, you know, there's, there's a lot more that that people could could do.

Jonathan: If somebody wants to run it, is the Docker image the easiest way to go?

James: Yeah. Docker image is pretty much the only way to go. Somebody did ask the other day how to run it on bare metal and a few of us just sort of all chorused in going, just don't,

Jonathan: you don't want it. You don't actually

James: want to do that.

I mean, nobody wants to set up a Ruby runtime and, and all these different bits and pieces. No, the whole point is all of the self hosting stuff I've done has the, the by far the easiest way it's been through, through Docker containers. Yeah. Normally the Linux server containers. I don't know if you guys have come across those.

It's a group that put out a, a set of applications as containers that all work in a standardized way. And I was very pleased when manifold got into that so you can run our own. Container. We have a version that has a separate database is the standard one. We also have a solo container, which is just run one container.

Everything's in there. And there's the Linux server one as well. There's instructions on the website for running it on things like Synology, Unraid All these various Various methods, but yes, it's all Docker. I imagine it's not

Jonathan: doing anything too obscure so you can run it rootless with Podman and all of that.

And it just works.

James: Yeah, we did a I don't know about Podman. I assume it will. We did yeah, one of the big changes to, to get the the Linux, everything was stopping it running as root and letting it run on read only file systems and things like that. So yeah, it can do that.

Jonathan: I will I'll have to give this a try with Podman then I'm my, my Linux usage is mostly on sort of the red hat fedora side of the fence.

And so I tend to go with Podman when possible. So I'll give it a try. I'll give it a try. And we'll let you know

James: what happens. That's, that's all the that's all I can ask. Let me know when it breaks. I mean, yeah, it's still under very active development. This is, you know, last week I put out three feature releases and a patch release for a broken feature.

So yeah, things break, but then I fixed them. So I've never lost anyone's data yet in 86 releases. Yeah, there's, there's not been any data loss. I'm hoping to get to a 1. 0 release maybe early next year, something like that. There's a few a few of these sort of the, the, the various features that I've got funding for, but a few other little bits to get in there that I can then say, okay, yeah, that's, that's a baseline.

And Then, then we go somewhere else, then we go from there, I don't know. But I do want to get to that one point, I hate projects that just perpetually hang on. A zero dot something or other. Yeah, yeah, yeah. Yes,

Jonathan: yes. Okay, so, I've got a, is there anything that we didn't ask you about that you really wanted to let folks know about?

James: Oh, I have absolutely no idea. I completely lost track of the conversation. No, I think just, no, come and come and check it out and come and have a look. If it sounds interesting, come and come and see. Well, I'll, I'll help people get it up and running. Cause like I say, it's early days. You still get very personal support.

Actually that's, it's not just me. There are now a lot of people in the support channel who will, who will help out getting it up and running. Yeah. None, none of whom I actually know. It's amazing. So

Jonathan: what what's the best place? So I see, I see many fold app. Yeah. If somebody wants, that

James: is the best place to

Jonathan: start.

Yeah. If somebody wants to donate, where, what's the link for that?

James: There is a donate link on that website. I think if you go to manifold.app/donate, I think that will probably work. Okay. He says nervously. But yeah, if you, you can click through to the main site. It's, there's a donate link downside.

There's also community links which will take you to the various channels. That are available there's a source code link for GitHub up in the, up in the top and yeah, all the documentation and getting started guides and all that sort of thing. Very cool.

Jonathan: Very cool. All right. So I've got to ask before we let you go I'm required to just about contractually.

What's your favorite text editor and scripting language?

James: Ooh. Ooh. See, how do you, how do you define text editor? So the thing I use day to day at the minute is VS code. I've gone through so many over the years. I was definitely an Emacs user for a long time. At the moment I'm using VS code. It's reasonably nice.

It's, it doesn't. break. And that's about all I need it to do. Actually it does break sometimes. It does break sometimes, but you know, yeah, yeah, yeah. Everything breaks sometimes though. So no, that's what I'm using at the minute. And I mean, favorite scripting language has got to be Ruby because it's just it's fun.

I've been using it for so long now that my favorite scripting language is one that I can get stuff done quickly in and not think about the language. And solve the problem. Yep. Yep. That's the way it goes. I'm not averse to any of the others. They're all great. I know many languages. Most of them are lovely.

Jonathan: Most

James: of them.

Jonathan: Not naming any names. Not naming any names.

James: I have mixed feelings about some that I have to work with. And that's where, that's where we shall leave it. Yes, yes, that is fair.

Jonathan: All right, James, we are, boy, we are beyond out of time. But thank you so much. It's been a blast. Talking with you and about Manifold.

And I literally have a terminal pulled up. I'm trying to get it installed on the machine behind me to let you know where the pod man works or not. So as soon as, as soon as we get close, yeah, I'll finish up with that. Thanks for being here. Thank you very much.

James: Yeah.

Jonathan: All right, Mr. David Ruggles. What do you think?

Have we convinced you?

David: Oh, well it's still a question of time. I never have enough time, but now I want to definitely get a 3d printer and get into the ecosystem. But yeah, that was fun. And and I just, it's always fun to talk to somebody that's just got such a wealth of diversified experience.

So I definitely, he, he, Hey somebody, if he's available, we keep him in for the roundtables. Yeah, that'd be fun. It'd just be fun just to pick his brain about so many different things.

Jonathan: Yeah, yeah, that was a lot of fun. And I'm, I'm real fascinated by this idea of what he's doing with 3D models, trying to compress them in different ways.

I think there's, there's definitely some room for that. And I didn't, I didn't get to ask him, but I actually wonder whether some of that is going to be cleaning them up as well. Because it's one of the things you get when someone's used like Blender or what have you. To produce a model, there'll be a lot of internal geometry that just either never gets printed or maybe never should get printed.

And there are some algorithms out there to try to clean those up, and not all of them are amazing. So, definitely some room for someone to work on that. But it's cool. I am literally working. I have a tab pulled up right here. I am working to try to get this installed and running using Podman. It's pulling right now, so if my internet connection craps out, that's what's going on.

Alright, a lot of fun though. David, do you have anything you want to plug?

David: Not specifically. I would encourage everyone to go check out Twit. That's the other place I like to frequent. So everybody else should like to frequent it too, of course.

Jonathan: Of course. Of course. So next week we are talking with Josh Bressers from Anchor.

Which that is about automated software compliance. And apparently they have quite a few open source projects that are connected to that. And so looking forward to looking forward to that, it's going to be next week. And then we've got some other things in the pipeline coming down the way. The, we've got the The J4Pi project coming up soon, which is about and we talked about scripting with Java.

Well, this is about scripting with Java on the Raspberry Pi. And that's, that's actually going to be pretty interesting because they're doing some fun stuff, exposing all of the different peripherals, SPI and GPIO and all that. So actually quite, quite a bit looking forward to that one as well. As far as plugs, we do want to make sure and thank Hackaday for being the home of FlossWeekly now.

And there is of course my security column, goes live Friday mornings there. And then there is, as we mentioned, the Untitled Linux Show over on TwitNetwork. And make sure and check that out as well for even more Linux y goodness from your host to co host today. We appreciate it. Thank you everyone for watching, those that get us live and on the download.

And we will see you next week on FlossWeekly.

Episode 805 transcript

16 oktober 2024 | min

FLOSS-805

Jonathan: Hey folks, this week we talk with Andy Piper about Mastodon. There's a new release, 4. 3, with all kinds of new tools. We get into the philosophy of Mastodon, the cool things you can do with it, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 805, recorded Wednesday, October the 15th.

Mastodon, bring your own algorithm.

It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something fun today. We're talking with Andy Piper about Mastodon. And of course, it's not just me here. I've got a co host today, Jeff Massey. How are you doing, sir? Oh, I'm doing good.

Doing good. Happy to be here. I appreciate you sort of sort of stepping in at the last moment I've I've had a bit of a health issue that I have been battling through for Well in retrospect over a month now, but past week or so was real interesting So things got a little behind but we're here. We made it and I appreciate Jeff being here

Jeff: Well, I'm just honored to be on the short list to be able to fill in and you know, I, it's, it's sometimes hit and miss based on, you know, my day job, but the Stars Aligned had an, had an opening and said, sure,

Jonathan: be glad to help.

Very cool. Now today we're talking with, like I said, Andy Piper about Mastodon and Jeff, do you have a Mastodon account? I do not. Okay. So our goal today, our goal today is to talk Jeff in to going and getting a Mastodon account somewhere or host it yourself. Like that's even an option. That's one of the beautiful things about Mastodon.

If you really wanted to, you could prop your own account up somewhere.

Jeff: Well, and you know, the interesting thing is I've heard about it quite a bit, but I haven't done a lot of research. I'm not an expert in it, so I'm, I'm really curious to find out, you know, all the. I know the high level stuff, but I, you know, a little under the hood and kind of what's going on and, you know, where's it been, where's it going?

The whole, the whole nine yards.

Jonathan: Yeah. And that's definitely part of something we're going to talk about because Macedon just had a new release. And I, you know, I don't remember what the version number is. It's a 4. 3 Macedon 4. 3 and we've got the man that knows about it and let's go ahead and just bring him on.

That's Mr. Andy Piper, and first off, we sure appreciate you being here. Welcome to the show, sir.

Andy: Hey, Jonathan. Hey, Jeff. I'm, I'm really, really excited and honored to be on a show, which is not only all about free and open source software, which I'm a huge advocate of Floss Weekly but also on Hackaday, which is another one of my favorite sites.

And I'm one of the things I do is I have a small maker studio here with my wife in the UK and you know, I'm on Hackaday. Literally all the time, all the time, every day checking stuff out. So yeah, so that's great to be here. Thanks for inviting me.

Jonathan: Yeah, we, we appreciate you being here. And now I've got to, I've got to ask, maybe I'm jumping a little too far ahead, but what's, what's the stuffy?

What, what is the, is that an elephant?

Andy: This is our Brand new just launched although we've been talking about it for, gosh, months, I think, because we've been through the whole process, but this is our Mastodon plushie. It's available now in the Mastodon store. Depending on your geography, we are working hard to make sure that as many folks can have their own.

Look at this, look, see, see how squishy he is? But I was away this weekend for reasons I was actually at Ogg Camp, which is something you talked about here on Floss Weekly a couple of weeks ago. And I got back to the studio Monday morning and my new my new work supervisor, let's say, Was here waiting to keep an eye on my contributions to the mastodon project Yeah, really excited.

It's it's a lovely plushie and also got to give a shout out to dopper 2 who is our designer so They've done all of the artwork for mastodon. So you can see the load the mastodon here. Not not this This is this is something different the mastodon logo on my shirt here the the mastodon mascot And all of the beautiful artwork we have You Around our website is, is by somebody who, who I know, only by username doper2.

And that, that's obviously the mascot the plushie. So, yeah, super exciting.

Jonathan: Is there also a rock band that goes by the name Mastodon?

Andy: There is indeed a rock band called Mastodon and so there is sometimes a little bit of search confusion depending on what you're looking for. But if you're looking for the Mastodon social network, then or the social networking software, then that is us.

And Mastodon is also a rock band. That's

Jonathan: fun. So you want to search for Mastodon social to get you guys. That's fun. That's the one exactly right. Yeah. If you're on a website that looks like heavy metal and has whales, you're on the wrong place.

Andy: Reaching down over here because I also have got a little 3d printed Mastodon mascot key ring that I've I made myself over here, but these are not available to buy.

But yeah, I have a lot of fun with it. The mascot is really, really cute. Is

Jeff: the the 3d printer file

Andy: out there so

Jeff: people

Andy: could I I So not at the moment because again, you know, it's done by our designer. I don't have permission to share that. So We'll see whether we can do that in the future. We'll we'll figure that out

Jonathan: There'd be there'd be a lot of fun actually to have that available.

Yeah, absolutely

Jeff: So you'd say with the Oh, I was gonna say with the rock band. Is there any naming conflicts? I mean not counting search but any legal You

Andy: Look, I, I, I am not aware of anything like that. I am not a lawyer, so I I can't comment. But I, I'm certainly not aware of any problems around that.

Jonathan: You mean trademarks? Surely open source projects don't have problems with trademarks. Nobody would fight over stuff like that. Let's not go there right now. Andy, I'm so glad you're here because if you weren't here, that's what we were going to talk about. And I was not looking forward to that. Oh, right.

Okay. Okay. Okay. So let's let's talk about Mastodon. You guys just had a new, a new release and what version of 4. 3? What's, what's the big, what's it, what's new in 4. 3? What's the, what's the roadmap? What did it look like? What new stuff did we get? What new fun things can people do?

Andy: So first of all, I want to say that it's taken about a year to get from 4.

2 to 4. 3. There's a ton of moving parts. We've got kind of the core engine of Mastodon. We've got the web front end. We've got our own Android and iOS apps as well. And then there's an API, which means that anybody can build their own third party client apps if they want to for either for you know, Android and iOS too, cause they, they prefer to, to build something for themselves or for other platforms that we don't currently have apps for, which is pretty exciting.

And. When we think about what's new, there's, there's kind of the new stuff that you get as a user, which is really cool. There's also a bunch of new stuff that if you want to run your own Mastodon instance, because you don't have to come to an existing Mastodon instance, you can run your own for you and your friends or your community.

Hackaday. social. Very excited to see that Hackaday have their own Mastodon instance.

Jonathan: Probably needs to be updated still.

Andy: Need to persuade Elliot. I think he may be the one that runs it to just refresh that up to 4. 3. And then also there's some new stuff for developers. So Mastodon 4. 3 for users gets a nice new look and feel.

It's, it's an evolution of the look and feel, but it's really nice and consistent. We've done a lot of cleanup improvements around, around the user experience. Especially around the media support. So you can now do things like drag if you want to. Attach a bunch of images to your post. You can drag and drop those in the composer to reorder them.

We've got also group notifications. So if you're, if your post gets really popular and you suddenly start getting loads and loads of boosts or likes, then those things will be grouped. The other thing that's really useful I find in notifications in 4. 3 is the ability to filter notifications. So you can do things like filtering out, if you want to, brand new accounts or from domains and things like that.

So there's some really nice stuff there. Another cool thing is you can now add a little tag to your blog posts. Fediverse creator, it's an open graph tag that you add to the top of your page. And then that will add, when you share a link to your post, then that will add a little a credit, an author credit under your blog post.

Link card, so that will enable people if people are sharing your blog posts or for writers and journalists, if people are posting their stuff on, on Mastodon, then it gives a little link to their Mastodon profile so you can discover them and go and find out what they've been talking about, which is really cool.

The embedded posts have been updated as well, so they look a lot nicer, so that's kind of a lot of the stuff for users. That's in the, in the default web interface. Incredible work really by by the very small but dedicated engineering team that do most of the work on the core team and then there's a load of stuff for admins and developers as well.

Jonathan: Yeah,

Andy: pretty cool

Jeff: So that sounds really really cool but i'm pretty new to mastodon. Yeah, never never used it and just just a little background I'm more of a hardware person I'm an open source enthusiast but not Fully informed so I handle some of the basic stuff for Jonathan like asking like at least the 10, 000 foot version the very very simple.

How does it work if everybody runs their own servers? How does how do they connect and talk and

Andy: yeah? Yeah, so I mean the easiest way to think about this and the way that a lot of people talk about the Fediverse or this decentralized system we have is email, right? So easiest way of thinking about it, you might be at outlook.

com and another person might be at Gmail, but you can still send messages to one another, right? So It's it's very similar to that. We have a protocol called activity pub. We have some internet standards. There's one called web finger, which lets me say let's say Jeff at hackaday dot social if I know your address is Jeff at hackaday dot social, I can go and find out where my server needs to send messages or, or, or where your, your server exists.

And then you've got a system of inboxes and outboxes which enable. The service to exchange those those messages. So that's kind of the super high level way of describing it. I can go into tons of detail and we can sort of do this whole seminar on activity pub in the Fediverse, which which is not what we're here to do today.

But that's that's pretty much how it works right now. I've just been at. An event this weekend called Ogcamp, which as I say, I know that you talked about on Floss Weekly a couple of weeks ago, and I was running, running the crew for that. And what we did there was we had a a wall of posts from people talking about the event.

And by the way, tons of hardware people on, on Mastodon Jeff. I, I've got lists of folks that I follow. Who are kind of I've got a group called gadgets and gadgets and makers. So I follow a lot of the kind of hackaday folks. I follow a lot of the people who you'll quite commonly see sharing their, their builds and things on there.

But anyway, back to our camp. So we have this, this, this. This social wall with all the posts and what I did for that was I said, okay, let's get hashtag or camp and hashtag or camp 2024 and anything posted by the odd camp account as well directly. So it was aggregating together all of the stuff that was coming through on mastodon from each of those hashtags and aggregating them onto this nice wall that was in our main social room.

So it was really

Jonathan: fun. Every, every time we have a Mastodon guest, this is at least the third time we've had somebody from Mastodon on the show, which is great, by the way. We're huge fans. The the analogy gets made that it's like email and every time that is said I kind of I have this bit of a shuddery Feeling because I run my own email server.

Oh, and it's a terrible experience one of the reasons is because of It is anybody that runs their own email server will tell you it is terrible It's spam is one of the big problems. Yep, and with mass saddam. We also have to talk about abuse and And let's, let's talk about the spam bit first, and then we're actually going to talk about Matrix for just a little bit, because they've had some of these problems too.

But let me, let me just, let me just ask you this has Mastodon had a spam problem yet? And is there anything new in the new release that helps people deal with spam?

Andy: So, Mastodon does sometimes have. Some, some spam issues has had, it's got way better with 4. 3 and I'll tell you why in a second. But, but let me just describe how that sometimes occurs.

Sometimes if you, if folks do run their own server and then they leave registrations, they leave it on the internet with registrations open with no kind of capture or anything, gating folks signing up. And then they forget about it and then they wander off and they forget that they've set up this container spinning somewhere in the cloud, right, and they're not paying attention to it.

That is sometimes how that can happen. What is, has got a lot better in the last 12 months in particular, A couple of things. First of all, there's an organization called if tasks, which is independent federated trust and safety. And that is an organization, another nonprofit which seeks to provide best practices tools for folks that run either their own instances or small instances, small communities, provide them with some guidance, some education about some of the things they might need to think about.

Around those kinds of topics. They also have a service, I believe, which enables you to kind of get a almost a default block list or a default list of known very bad domains that you don't want to receive information from on your instance. So they're a super, super great organization to keep an eye out for.

Mastodon, the, the, the organization knows, you know, we're in connect, we're in contact with them. But that's a separate organization for, for Mastodon as a whole. And it works to try to bring some of those good practices across the Fediverse, all of the things using ActivityPub. So that's one element here.

The other element is in Mastodon 4. 3, we've got these filtered notifications so you can go in and I did this literally did this in the last couple of days because my instance was the instance I'm on mccall. social got upgraded over this weekend. So again, I came into the studio on Monday morning. I thought, Oh, my friend, Ron, who runs us, our instances upgraded us to 4.

3. I went in and updated my, my, my notification settings and I can do things like filter out posts from accounts that were created just. You know, in the last couple of days don't get rid of it. You can either choose to get rid of them completely and never see them. But then somebody like Jeff might sign up right now or tomorrow or, but, but in the next, I'm going to think Jeff's going to be going to create an account in the next few hours, let's say, right?

Because it's going to make sense. A

lot of sense. And he may, he may post it post. Post it on Mastodon and say, Hey, Andy Piper at McAuliffe social. Great chatting with you. It's been great chatting with you too, Jeff. I'll say that right now. But anyway, let's let's let's carry on. And I don't want to miss that, right?

Just because it's a new account. So what we do, what we do is we have the option to filter those into a separate like inbox, a filtered inbox. So I can go and look at those later and say, Oh, Cool, Jeff's here now. I'm going to follow him, right? And I'm going to allow that, and that's okay. But if I get one from some strange domain I've never come across that may be one of these kind of abandoned instances or that contains something that I don't want to see, then, you know, I can, I know then I can either block it or I can block the domain.

Another feature in Mastodon 4. 3, which is really useful, is because we're talking about these instances that are Separate you can block individual users, mutual block individual users. You can also choose to block an entire domain. Now, if that happens between instances, what that does is it blocks every account on the other domain.

Now, if that other domain has, you know a bunch of people actually you wanted to follow. Maybe something, something has changed around the ownership of the instance or the moderation of the instance. Then if your instance owner, your instance administrator blocks that domain, you as a user now get a notification to say, hey, by the way, your instance owner just blocked baddomain.

com. You were following four users. Bad domain dot com, here's a list, you might want to keep a record in case they move somewhere else in the future. So, we're doing a lot of work to try to improve these kinds of situations. You raised some really good points I beg your pardon, Jonathan, about running a mail server.

And and yeah Unfortunately, we've learned in the history of the internet that some people do some silly things which aren't great fun for the rest of us who just want to have a better experience. But what we want to do with Mastodon is really build a better social web for everyone and do that using open source technologies, do that using internet standards and protocols.

Jonathan: I mean, you sort of, you sort of know that your system Has arrived. You've made it. You're in the big leagues now when you start getting spam through it.

Andy: I, I mean, that's the other argument, right? And, and, and that, that, that could well be the case, but there are controls in place or there are got controls available to, to, to manage some of that.

And we are working on that. There's always more we can do. There will be more we can do. And we, and we listen to the community and we pay attention to, to what's happening.

Jonathan: Yeah, that, that idea of, of an old instance that allows open signups without any sort of protection against it. That very much reminds me of the the open relay problem with email.

Andy: Yeah.

Jonathan: And so I can imagine, I can imagine a future where you, you start having, well you, it may already be here for that matter. You have some sort of a a blocking list service where this server is a known open relay. Therefore, we suggest that you block it.

Andy: That's what if tasks can provide to some folks.

And that's something that we're looking at adding. On our roadmap for the next version of mastodon as well that we'll we'll have some better defaults there But right now you can go to iftas. org And sign up for their forums and they can help you with those kinds of things as well if you're a new administrator

Jonathan: Yeah, so again making the email analogy.

It's the equivalent of spam house Spam host, however they say it. I mean,

Andy: I'm not, I'm sure that Jazz Jazz Michael King who runs that would see things slightly differently but it is similar, there's some similarity for sure. There's enough similarity to make the analogy I think, yeah. Yeah, yeah, yeah, absolutely.

And you're right that it's a sign that the Fediverse is taking off that we need to spend more time thinking about things like Things on this.

Jonathan: Yeah, so let's let's talk. I'll let I'll let Jeff go just a second. There's something else. I want to make sure I get to Let's talk about abuse like that's the other the other big part of this when we talk about blocking people and blocking messages those are the two big things you got spam and then you got abuse and the the tricky thing with abuse is it is not a Nearly as binary as spam is.

Right, so an incoming message is pretty much, it's either spam or it isn't, and everyone sort of agrees, looks at a message and agrees that's spam. Abuse, it's different, because some people, their tolerance for abuse is, if someone, you know, has anything political to say, well, it's abuse. If someone is supporting, A candidate that I don't like it's abuse and like that's sort of like on the very very light gray area and then you've got the extreme opposite is And this is something that matrix has been dealing with matrix is sort of the other federated multi user universe out there I was I was in one of the main matrix rooms and I had to leave it because people were Coming in and dropping images that I did not want to see like I don't I don't I think they were getting to the point of being like CSAM that's child sexual abuse material but they were, they were close, like that was the direction people were trying to go, and the Matrix guys were just, they were struggling trying to get a hold of this, and so like, that's the very dark, completely opposite edge of, of the abuse spectrum, and not everybody agrees on how much of this is actually abuse, and one of the things that I, I love about Mastodon is, You can run your own server and you get to make that decision yourself.

Whereas on the other social networks, you've got a company that makes that decision for you. And I just, I just love the fact that I'm asked it on. It's not the way it works. And so I'm assuming that with with the new version with 4. 3, you've also got more tools out there to help people deal with that sort of thing.

So let's talk about that a little bit. What's what's new? What's new in the world of abuse on Mastodon?

Andy: Well, if I can pivot that, I'm going to include that in my answer. Let's let's pivot it and talk about the sort of the admin side. I gave you kind of an overview of what's new for users. And for people using Mastodon in 4.

3 particularly on the sort of the website but also on, on the, on the. So let me start with the basics. So 4. 3, there's some upgrades to some of the underlying dependencies. So we bump up, I think, from Ruby 3. 1 as the minimum to Ruby 3. 3, and a newer version of Ruby. Newer versions of Postgres and, and those kinds of things.

And what that brings as well is better performance. We've done some work to migrate away from and, and, and deprecate image magic for media handling. So we're now using live vs. And, and that provides better performance. So. But there's some sort of underlying stuff that's new for for admins there.

There's also some better stuff around metrics and management. So we've dropped deprecated stats. D. We've moved over to open telemetry. We've got Redis Sentinel support in there. We've got we've got some other nice things and some little nice. Touches like you can now customize your instance icon.

So if you are a hardware community, you can literally, without having to go and change templates, you can just choose an option to choose what icon you want. But to address your point directly, there are some features in 4. 3 that help related to spam and abuse. For example, moderators can now Search for problematic content using hashtags, which wasn't possible to do previously again.

We really I think the, the administrator community, the instance administrators benefit from things like talking to if task, getting some, some advice from them, getting some help from them as well. They've got some services that they, they help to provide in those instances, but but from the administration side again, we know we can continue to improve and iterate here.

It's been a, been a year long process of building the new release. And that seems like a really long time. It is a long time. The team's worked really hard. There's a lot of moving parts. And every time we think about wanting to add a new feature, it means having to, to, to make a choice about what we, what we focus on and things we can't do straight away.

So, you know, I think we've added some really nice things. We've got a decent balance of. Keeping a nice fresh modern look, adding some user experience features, making it easier for folks to get started, making things a bit easier to use, and then also helping administrators to manage their instances.

Finally, there's some, some Stuff features for developers, which will hopefully help them to plug in and improve the situation as well. So we've done things like really improving the authentication mechanisms for the API and enabling developers to also, if you're using a third party client, take advantage of those same notification filtering mechanisms that we provide so, so that they can provide those to their users as well.

Jonathan: Yeah, it seems like there's a, we probably need to, here in a few months, we probably need to have the folks from IFTTTAS on and talk about this because it's, it's fascinating to, to, and one of the reasons it's so interesting to me is because every other social network does this very opaquely. You don't get to look into it at all.

And with Mastodon, just by its very nature, it has to be very transparent. And that's amazing. That's it.

Andy: Okay. There's challenges with that though you know, we can't, I think most instance administrators We can't necessarily you can say that these are the kinds of content And kinds of things that are okay on our server, and you can be transparent about that.

I think that you, the way you get into difficulties, particularly with spam, it's back to the spam assassin, spam house, you know Bayesian learning kind of stuff, you know, as soon as you start to say, this is what we are filtering out, then the bad guys can figure out how to, to get around that. So You know, I think that's absolutely true, and I agree with you that one of the benefits of building an open platform, building in the open and working with others is that we have to be and can be much more transparent about those, those decision making processes.

But I think the technical level from certainly from my past experience. At one of those large platforms that we've referred to you know, there are challenges with, with how much you can share in public.

Jonathan: That's, that's actually a very, that's a really good point and not something I was thinking about, although it definitely makes sense.

It definitely does. I think that's, that is something that Oh, I forget the name. That's Well, is it, is it Spam Assassin That has some, some open source tooling around that. I

Andy: certainly, you, I mean, I'm, I'm going back 20 years when I'm, when I refer to Spam Assassin, I mean, well, that was when I ran my own mail server at home and, and that was a long, long time ago.

But I think that's

Jonathan: still, I think that is still one of the names in the business that do it. And they, they have that, that same problem is that they publish some of their filtering stuff. And so you can just grab it and write spam to avoid it There's another organization. Yeah, find the loopholes.

Andy: Yeah, exactly.

Well, that's that's that's the the loopholes is the challenge there's another organization that i'll mention because I think it's really important and it's another demonstration of the growth, momentum of the social web and and the fediverse is the social web foundation which was founded by Evan Prodromou, who's one of the authors of the activity pub specification, and he's been really active in the Fediverse for a very long time.

Mallory Nodal and Tom Coates. So they founded this thing called the Social Web Foundation, announced it about two or three weeks ago. And it really exists and is supported by a number of players in, in, in this space ourselves. As well as a ghost flip board, a number of the platforms that are integrating with activity pub, and it really exists to tell the story, help people to understand a bit more because, you know, Jeff, you kind of said, look, Hey, I'm aware of the basics, but I'm not quite sure I get it all right.

And so the social web foundation is there to help to tell the story more about, we've got this opportunity as users to. Make our choices, own our data, not be beholden to Jonathan's point to the decisions made by those big corporates that are, you know, very opaque. We get to the, the, the boring thing as users is that we have to do a little bit more brain work.

And, and, you know, my, my background, my degree was in history and I, I have, I have, humans are incredibly lazy. Okay. We are, we are the most lazy creatures we will, we will, we will let, you know, that's why it's sometimes it's very difficult to, to, to make those hard choices to say, I'm going to go and use something different.

It's going to involve a little bit more work to learn. Relearn some, some, some processes and the way things work because I'm so used to this easy thing that I was doing for the longest time to, to, to get out and change the way you do things that can be difficult, but there's so much benefit to it. And the Social Web Foundation is there to help to tell those stories and to say, look, you know, these are the benefits of using ActivityPub.

These are the, these are the benefits to you as a user of owning your network. I've switched Mastodon servers only once so far. I know, I know folks who've switched more often than that. And when I did that, I was able to go from one active mastodon instance to another and it moved my network with me.

I didn't lose followers. I just, they were all, all of their accounts at a protocol level, the protocol said, hey this guy's changed his address. He's over there now. And all of those accounts re followed me and I re followed them and that was super powerful. It's a really big difference to, again, the way that some of those other networks work.

Jeff: I don't think that makes. Good. Sorry. I was gonna say, I think that makes a lot of sense. I mean, as you know, Steve Gibson says, you know, having good security is inconvenient. Well, having control of the social media is maybe a little inconvenient, but it, you know, you, you're better off because you have that power.

And that's one of the things I do like about Mastodon, is I know that you're in much more control of what you, what you see, and, you know, you, you're not at the mercy of some conglomerate And, you know, there's many stories about how they've made decisions one way or the other that have upset people because wait a minute, you're filtering things.

I don't want to see. And as a Mastodon user, I could say, well, I'm really into hardware, I'm into the software, I want to, I don't want to see cat videos, you know, I just, I want to see any of that, you know, I can just block it myself and not have to worry about it. But one of the things I did want to say is like, okay, I'm going to get a Mastodon account.

Yes. Where, where's step one? Where, where, you know, as, as

Andy: a new user. So you. So go to, you can go to joinmastodon. org and there's a page for you to, to find servers by default. And this is, this was done primarily to reduce that mental. Cognitive load of making that choice because before and something we've heard before is people would go to joinmastodon.

org and it would say, here are all the servers in the world and you'd have to choose one, right? And that's the first thing that people tended to, well, exactly. Right. So, so, so first of all, doesn't matter where you start. You can move if you want to. Right. In the future, you can take your network and you can, you can move somewhere else.

So by default, it will, it will suggest mastodon. social, which is the one that the mastodon nonprofit organization that builds the software runs similar to the matrix project in a sense, right? They've got matrix. org, which is where I've got a matrix account. But if you, if you want to you can go somewhere else.

So for hardware folks, If you're into 3D printing, there's 3dp. chat, there's, there's there's bitbang. social, there's, there's hackaday. social, just saying, just saying, maybe have a nice chat with, with Elliot and get that, get that sorted. But If you go to joinmastodon. org, there's there's a service page where you can do things like filtering by by your geographic region that might be important to you for data privacy reasons servers based in the EU may have different.

Data laws than those based in the U. S. For example so it may be important to you from those perspectives. It may be a maybe a topic based community. You're looking for what I did was I moved. I was on Mastodon at Social. I started there back in 2017 or so with my account had a number of years there and then I ended up moving to New York City.

A server run by a former co worker of mine which is called mccaw. social, which has got a couple of hundred of folks that I, you know, used to work with on that platform. And they're all in their own places now. But I know the person that runs that server. I actually donate to him. I give Give him some give him a small amount of money every month so that, you know, it covers the running costs, help to cover the running costs of his, of his server.

And and, and I know, you know, if it goes away, he would let me know because I'm giving him some money. The, and, and, you know, he would explain what was going on. And that does happen sometimes with, with. Mastodon instances or Fediverse instances. But that would be my suggestion. Another tip I wanted to give you, though Jeff is one of the power moves on mastodon you can do is follow.

You don't not only follow. Users and then needing to know who all those users are. Although Macedon 4. 3, if you join a server that's running Macedon 4. 3, you get a slightly refreshed onboarding experience where we will help you to hopefully find some folks you might be interested in a bit more easily.

We do a little bit of additional recommendation to help folks fill up their. The following list a bit more, but one of the power things you can do is you can follow hashtags and that is super useful. You, you know, I, I follow a ton of hashtags for 3d printing, for ESP 32, for micropython, for Lego, for events I'm involved with and and then if I see those things coming into my timeline because I'll find out how there'll be a note saying why that's shown up, why I'm seeing this then I can actually go and choose to follow that user.

Here's another useful tip in 4. 3 that I didn't mention earlier. In 4. 3, we've got these little hover cards on the web user interface. So if you see a post from someone that you don't recognize, you can hover over their name and it's going to pop up a little floating card so that you can get a bit more information about them, where they are, for example, or what their, their profile links are.

And that's in 4. 3 as well. So again, we've really tried to focus on making it easier to find people that you might be interested in following or topics you might be interested in following. Thank you for asking that question because, because it is, I think it is one of those barriers to folks sometimes is either they've previously had an experience coming to Mastodon and having that, Oh my God, where do I start?

Or they just haven't kind of kept at it. And that goes back to my comment about. Yeah, there's a bit of relearning to do here, but it's really powerful and valuable when you, when you get into the groove and start engaging with it.

Jeff: Awesome. Well, now just say I decide, you know, and I'm going to run a server as well.

Yeah. Is, is there any, like, legal risk or burden with that or what, you know, could I get in trouble running it or? Sure.

Jonathan: Is California or Europe going to come after you for having people's personally identifiable information?

Andy: Right, so so look the the answer to that is it depends i'm afraid It's the honest answer.

I I work in developer relations. I I do my best to give those honest answers. If you're running something for yourself, then, you know, it's about the same as running your own mail server in terms of like What's coming onto your server and those kind of things, if you're, if you're running it for a group of people, your friends or whatever, or then you definitely need to be aware of who you're supporting, what data is coming in and what the rules are, what the laws are, and I can't tell you those, it's going to, it's going to depend on where you, where you're located.

Again, I believe the IFTTT knowledge base has some help on those kinds of topics. They've been such a great supporter for folks running, building and running the social web. So I would definitely encourage people to take a look there. And there are also people in their forums who can also probably point you in the right direction, depending on your, you know, your geographic location or so on.

But, you know, broadly speaking. You can spin up a container. You can go to what there's a couple of two or three hosting organizations that I can think of that will let you sort of buy a small bit of cloud container to run an instance if you wanted to, either for yourself or for a group, and they typically will then Do what a lot of these organizations will do though.

They'll charge you based on usage and size and and How many virtual CPUs and all that sort of stuff that you want and how much you might need to support the size of your community

Jeff: Okay, well that and that answers quite a bit now you said A little bit of cloud. So, and I know it's going to be based on users or whatnot, but say I had 20 users on my server.

Do I need pretty powerful or will it run on a potato or somewhere in the middle? Oh, Jeff,

Andy: you're actually, honestly, genuinely, you're, you're testing my, my ability to answer that directly. I'm going to go and quickly go see if I can find, find that out for you whilst we're talking, but I don't have that in my head.

Okay. Off the top of my off the top of my head, I think sort of for 20 users, you're gonna, you know, you're gonna want a fair, a decent amount of media storage, probably, you know, 50 gig or something. And and something like that. I'm looking at one of the pricing pages of for one of the hosting companies right now.

I'm not going to name them because people should go and check those sort of things out. But they are suggesting that sort of thought for a 20 active users But that would be their kind of mid tier package with, as I say, 50 gig of media storage and, and something like that. But, but please don't take my, my, my, Off the top of my head word for it because I don't have it in my head I've made notes on other things we might talk about but not those topics.

I apologize

Jonathan: I'm gonna i'm gonna ask you about something similar and you you may you may have to hit the eject button on this one, too

Andy: Well, no, listen, i'm not going to eject. I will definitely help out and answer these questions In a different

Jonathan: format later, but but yeah go for it so, this is something we covered on Hackaday, actually, you may have seen it.

The website was Itsfoss, which, great website, we cover stuff from there from time to time. And they made a request, please stop sharing our stuff on Mastodon, because Mastodon is unintentionally DDOSing our website when one of these big places do it. And what was happening, it's a real thing. There's There's more to the story apparently they didn't have any caching turned on for their website But what was happening was when someone with a lot of followers would would you know re do we still say retoot repost?

We would repost something from the site all of the federated so In, in Mastodon, you have this idea of federated followers. So, like, someone from your server follows someone from that server, therefore the server follows, gets the whole feed. Well, when that happens, when that server, when person A posts a link, server B, where someone follows person A, will follow the link and grab it to get, essentially, a summary, if I remember correctly.

But they'll do, they'll do a download of the website. And so what was happening is when someone with a lot of followers, or a server with a lot of followers, posted that link, you had, you know, dozens, if not hundreds of Mastodon servers around the world, sort of instantly doing a lookup for the website.

And it was, it was taking at least one website off the air. It was, it was taking them down. It was, it was a DDoS effect. They were getting slashed on it by Mastodon. Yeah. Yeah. It's so late. Is that, is that still a thing? As the message on devs, I, obviously you're aware of that. Have you all, have you all worked on that?

Yeah, I am aware of that.

Andy: No, I am aware of that. And because also I have my own small podcast I, I do weekly and, and, and that one sometimes has a similar kind of blip when I share our weekly episode URL. So look it depends on the size as you just kind of indicated, and we are aware of it.

We're working on it. We've got ideas. Something I want to mention since you bring this up is there's a new project that we've spun up which is funded by and Elna NGI and It's called Fediverse Discovery Providers, and the reason I mention this is because there's a, there's a, there's an additional element to that, which is called Fediverse Auxiliary Service Providers.

It all sounds very fancy

and

so on, and to be honest, it's not very user facing, but the idea is that you might have these services, which are not necessarily just Mastodon. The, the, Fediverse is all using ActivityPub, we've got other platforms, Lemmy, which is like a Reddit sort of similar clone. We've got PixelFed, which is an Instagram similar style thing.

All of them using ActivityPub and you can follow users across them and so on. And we want to contribute to the broader Fediverse. So there's this project that we that we are working on that is funded by NGI, NLNet NGI search, which is Fediverse Discovery Providers, which will enable in potentially for folks to discover content across different types of Fediverse instance more easily.

It's going to all be opt in. We introduced it at an event called Fediforum last month, and we're working on it in the open on GitHub at the moment. Now, the reason I mention it is because what. The idea of this is a Fediverse Discovery Provider could be an instance of what we're calling a Fediverse Auxiliary Service Provider and other things, the other things that Fediverse, future Fediverse Auxiliary Service Providers might do is provide better caching for situations like that, that would, that would, would prevent.

Or mitigate those kind of situations, there was also something we actually added in the in the previous release. And one of the dot point releases to 4. 2, which is that there is, there's a bit more randomization of a delay between when all of those instances are going to go start. Like trying to fetch the URL that's just being shared, so there should be less of a tsunami of requests that suddenly all come in, in a, in a very short period of time.

And, as you rightly say I think the particular instance you were talking about with, with its FOSS or whichever site it was you know, they came out and said, hey, you know what, okay, we, we weren't caching a few hundred requests in 60 seconds, you know, Most website servers are gonna, are gonna be equipped to deal with that.

As I say, I know that the podcast I have, we are running and, and it's terrible and I really need to fix this. We are running in a shared in a shared web hosting thing. That, that I haven't managed to spend enough time to get us migrated off of yet. So we only have a tiny slice of a of a, of a, of a VM somewhere.

That sometimes that sometimes does that does happen. And yeah, it's on my list of things to to resolve because I can't even do things like upgrading PHP and my SQL on that particular instance because of the deal we have with that host. That's my problem, not anybody else's. So again, yes, the answer is yes, we're aware of it.

Yes, we're working on it. We've got ways of mitigating it. Definitely want to improve it in the future.

Jeff: Speaking of the future, we do have a question from Discord and from MashedPotato. He says, is it still the case that quote posts won't be implemented on Mastodon?

Andy: No,

it's definitely not the case. And MashedPotato, if you go read the launch 4.

3 blog post, which I think I sent to Jonathan to include in the in the post later from Hackaday's perspective. If you go look at the blog post, blog. joinmastodon. org and look at the 4. 3 announcement, you will see at the end of that there is a reference to what we're doing next. And this is a really nice segue into that, actually, because 4.

4 4. 4 or whatever the next version is quote post is on the list. We are, we're doing it. At some stage there was a I think some folks felt that this was not going to happen. We recognize the community wants, wants Qt Quote posts. So that's something we want to do. We're trying to talk to other, again, activity pub implementers.

It's not just Mastodon here. The Fediverse is a, is a collaboration of different. Platforms and we want to do this in a way that that works and that is interoperable that extends ActivityPub if needed in the correct ways to enable this to be done well, there's a ton of things to think about, but we are talking about it.

Jonathan: That's a really interesting point. So the quote post is going to be an extension to, to ActivityPub.

Andy: I may have misspoken. We're looking at what I'm saying is we're, we're looking at what we need to do to make sure we do it well. And in a way that, that, that interoperates with other ActivityPub Services.

Right, right. So I'm not 100 percent clear yet on the exact data

Jonathan: format that it's going to look like. Sure, sure, sure. It might be an extension to ActivityPub, but at the very least, it needs, you guys need to think about the ActivityPub backend. So that brings to mind something actually really interesting about quote posts.

They probably don't have to be just Mastodon posts, right? Exactly. So there are a bunch of other services on ActivityPub. But there's there's picture sharing there's video sharing there's you know, yes think think of a Existing big social media service and someone probably has an open source equivalent that works with activity pub Yeah, and so this brings to mind this this This is actually really exciting.

This amazing thought of someone just posted a video on this other activity pub thing. I would like to share it, and it is a repost. And if you click the repost, it takes you over to the video. Like, that's super cool. That's really cool. It is

Andy: really

Jonathan: cool.

Andy: I love, I love, so I've got, I think there's another element to all of this, which is that we as an activity pub, Fed social web community can do more around accounts because I've got accounts on pixel fed, peer tube, lemi, mastodon my WordPress blog is, has got an activity pub plugin and so on, right?

So you can see content from all of those places under, under my username at different domains.

Mm-Hmm, ,

I, I can reshare. One of the things I love is I can, I can post my images on Pixel Fed, I post them on Mastodon as well. But if it's kind of more of a photographic thing that I want to exist in a picture fair sharing site, I'll post it on Pixel Fed and then I'll repost my Pixel Fed account, boost my pixel Fed account into my Mastodon feed so people can see that.

So I kind of love that interoperability. Mm-Hmm. . It takes some getting used to for people that are used to other platforms and the ways that things have worked elsewhere in the past but for them in the past, but I, I really appreciate it. I think it speaks a lot to the power of having this, this social web protocol and layer.

But yeah, look, one of the things I try to do is I try to attend when I can the W3C social web community group, which is being reformed into a working group meetings and take part in their discussions. I go along to something called Feddy Forum, which happens every six months, which is independently run.

By some folks and that brings together a bunch of the different projects that are interoperating in the, in the social web to talk about what's new or what they're building or how we can make things better on an ongoing basis. So, yeah, there's a, there's a, there's a lot to think about here.

You know, it's not a quite just a question of can you just implement quote posts?

Mm hmm.

Tomorrow, you know, there's we're a tiny team. Our team are the core development team is really small. We've got to keep the platform as a whole stable for lots of people fix bugs and, you know, solve problems. If if things get get raised, we need to be.

We need to respond to them quickly because, you Especially in the age of misinformation and spam and all this other stuff, we want to make sure that there aren't too many vectors for bad actors to misbehave. We want to make sure that things are running well. So we're doing, we're fixing bugs. And releasing patches to existing versions.

By the way, now that we've released version 4. 3, we'll be, we'll be retiring version 4. 1 in in the spring so six months after the 4. 1 series will be will no longer receive updates, but we will, will continue to support 4. 2 and 4. 3. We want to install to it to apply, you know, add new features.

We want to build for as broader community of people as possible. And we want to interoperate with all of those other platforms that are part of the Fediverse as best we can. So it's a, it's a, it's a big, big effort for a

Jonathan: small team. So I've got to ask about something and it may sound like a troll question and it only a tiny bit is that's actually fairly serious.

Are we doing anything with AI in Mastodon? Is LLM coming to it? No, no, no. Is there a future that you see where it could? And I think, I think I can see a place where it might fit, but I'm curious. What do you think?

Andy: Well, I personally. Could see a small number of places where something like an LLM could be useful.

However, it is not AI and LLMs are not at all on the roadmap or plan for the Mastodon core team, we are not in any way interested. And if you follow Eugene, who's the founder, the CEO on Mastodon, he's very, very clear that it's not something he finds acceptable. He's that he's interested in I don't want to speak for him too much on that regard.

He speaks for himself. But no, I it's not something that we are going to be going to be spending any any time on we've got much more interesting problems to solve I I'm curious what your thought is jonathan my thought and this is I want to make it 100 percent clear. This is not me speaking on behalf of the Mastodon team or project.

The place where I could see it potentially, two places I could see it potentially useful. And one of them is already being done by a couple of third party clients, which is doing things like image descriptions. Okay. Right. So, so alternative text for images where, where, where the, the, the, the, the, the app would suggest that for you.

But the, the other one potentially would be some element of recommendation or ranking, but a lot of people come to Mastodon and the Fediverse to get away from algorithms. Again, I've got a very complicated relationship with AI and machine learning. I work in a maker studio. I do some stuff with generative art plots, but generative art in the sense of running algorithms to generate interesting swirly circles, not, not taking stuff that other people have created and mashing up another image, you know, so there's, there's, I've written a blog post about that as well.

So again, that's me speaking personally. I don't think that that is something. That any of those things, are Well, I know for a fact that We have no interest in adding those sorts of things to to mastodon.

Jonathan: Sure. Sure, and that's that's fair You know the the the current, AI craze, we could call it, maybe it might be fair to call it a bubble.

People have very strong opinions on it. And I, I find that really fascinating. Personally, I am waiting for the bubble to pop so that AI and LLMs can become basically just another tool that people use as opposed to the big thing that everybody wants to throw money at right now. .

Andy: Yeah. I, I mean, I think that it, it, it's, yeah, you're right. I mean i've been in the tech industry for 20 years and I think that You know, we've been through a bunch of these waves but yeah, I I the thing that bothers me with it is the the clear intellectual property infringement aspect, in terms of like how all of this stuff has been built and trained Yeah and I and again speaking personally I am Very much struggling to come to terms with the way that You The big tech folks have railroaded their way to access to all of this data and used it to train their models.

I have problems with it. Yeah, I don't, but I, but I do find the outputs when you used well with an intelligent human brain. You know interjecting and saying well actually that's nonsense, but these three bits of information you've given me seem to be useful You know, I think I think that that it can be useful in that respect.

Jonathan: We had a we had an awesome discussion I was actually a little surprised at how good it was But just last week we talked with one of the guys from ibm and he's also working with the ai alliance And just a really good discussion about some of these issues And with with With Mastodon in particular, it kinda gets back to, to one of my thoughts about LLMs and one of the things that they're really good for, particularly the conversational LLMs, they work really well as a replacement for a search engine.

Particularly if you do things like in your prompt, you tell it to cite your sources, then it will, it works very well as a as a search engine and. That is interesting. And so you, you hit the nail on the head. The thing that I really, I was thinking about is in addition to, you know, just show me the posts from the people I follow, just show me the posts from the hashtags I follow.

And then you've also got this option to show me the posts from all of the followed federated servers, which is kind of like the fire hose. There might be an algorithm that says, show me the posts that are similar to posts that I've enjoyed seeing in the past. And I think that could be useful for people.

Andy: I think you're right. But let me give you, let me give a shout out to some of the third party developers because that's what I'm here for. Right? I worked in developer relations. My role. On the team. I'm not one of the core developers. I work with the core developers and I help to communicate what we're building to the third party developer community.

So let me give a shout out to some of our awesome third party developer community. There's, there's a, a web app called Fan, P-P-H-A-N-P-Y Social which is a third party web app progressive web app that lets you you know, browse your master on timeline, but it has a catch up feature. So you know.

If you, if you're not on mastodon whilst all of these non algorithmically sorted posts are coming in mm-Hmm, , then you might miss them. Right. But if, but, but if, but, but I might want a summary of the last eight hours of interesting posts. And FPE can do things like saying, well, you know, if that's had, if, if posts, posts that have had more than five likes or 20 boosts or whatever, that might make them interesting to you.

It can show you them in a timescale. There's another. Service I use called Mermel, M U R M E L, which does something similar. It sends me a daily email digest. There's another one to highlight things I might have missed or links that have gone viral that I might have missed because I wasn't watching my Mastodon timeline at the time.

There's another one called Fediview, which does more of the. Machine learning style, LLM style ranking rather than just doing what is you know, what metrics to reply. So these are places where I think having a, an open API, a publicly accessible, easy to use API and having third parties plug those things in is fine.

If that's what they want to do, and the data is being used appropriately and not just being, you know, churned into a Another LLM to generate more stuff. Right. And we know that those kinds of things can happen. But I think that, you know, we as a platform as a core team are focusing on making a, a, an easy to use, welcoming, better open social experience for people, for a large number of people for a broad range of people.

So, When there are features that are really good for power users, we may not spend as much time on those. We may delegate them to, or suggest that third parties can go build those things that have much more of a niche use case. That, that, that, that Feddy wall that I used at OBCAP this weekend, that was built by somebody else, you know, they built that tool.

We didn't need to build it into Mastodon directly. Not everybody needs to be able to run a wall of Mastodon post at an event. So why would we spend a load of time building something like that ourselves?

Jonathan: yeah, I've I've thought for the longest time and I was at first quite hopeful that twitter was going to implement this with the changes and They've only a tiny bit done So and and I guess I still have a little bit of hope for the future, but not a whole lot there.

Anyway, I would love to see a social network where you could bring your own algorithm Yeah. You know, exact, you know what the algorithm does, you have control of it over it, and it does exactly what you want it to. Well, and that's sort of, so Fed, fed view

Andy: kind of,

Jonathan: that's sort of what you're, what you're describing.

Yeah.

Andy: Yeah. Fed, fed View kind of layers that on, you know, they, they've got four, I think four built in algorithms that they, that you can choose from. Mm-Hmm. Yeah. Look, I, I, I'm not. I'm not opposed to that as a concept. A lot of people who use Mastodon would be completely, I believe, would be very opposed to any kind of algorithm.

Because all algorithms are opinionated in the end and have some kind of agenda. Oh, absolutely. If you are going to spend the time thinking about your agenda for applying your own algorithm. All good. Right. As soon as that gets taken out of your hands and is run opaquely, going back to our conversation earlier, that's when things can get dangerous or things can, you know, be misused.

And I think those are dangerous. But yes, I think you're right. I think let's see where the Fediverse goes next. I mean, there's a ton of interesting stuff happening.

Jeff: Absolutely. Brings up, you know, how do you decide what's going to be next on the roadmap? I mean, cause you talked about earlier community, you know, wanted something.

I mean, is there a place that. To give feed feedback or do you survey or, you know, what's up? How loud does

Jonathan: the community have to be before something is implemented?

Andy: Yeah, yeah, yeah, yeah, yeah. Well, I mean, we, we hear the community a lot and we appreciate them. And, and honestly we do. It's we've got, obviously we've got the atmastodon, atmastodon.

social, and we've also got atmastodonengineering atmastodon. social. That, that second one is one that, that, that, that, that, that. That we've spun up in the last 12 months, and we're also now posting every month. We, we share a monthly engineering update. It's our, our trunk and tidbits blog series which I, I co author.

So we try to help folks keep up with the things we're, we're doing. The last six months have all been about what the changes are that fed into the 4. 3 release. But the other thing I'm very keen to do again in my role is to showcase what the developer community beyond us is doing as well. So I always try to include one or two or three fun things that other people have built on top of Mastodon.

We're on GitHub. Obviously, all of our development is on GitHub. Being FlossWeekly I'm aware that other source control mechanisms exist. We really appreciate all the tools that are available to us on GitHub, and we've used GitHub for a long time, and we don't have plans yet, or any time soon to move off of GitHub.

But again, we hear the community sometimes who suggests that we do that. We, we, we hear, so we hear people talking to us on the platform. We hear people talking to us. We have a discord for people that donate to mastered on on Patreon. They get access to a discord. And, and the, The Patreon donations also, they get a newsletter from us.

I think we just posted one this week to folks that have donated to us via Patreon. We we have people raising issues against the project and raising pull requests. We've got a small team of amazing volunteers who help us to do some issue triage around some of those. But yeah, we, we need to manage a lot of sometimes competing.

Requests our launch blog post of 4. 3 mentioned a few of the key things we want to improve in the next major release, which would be those quote posts and also potentially subscriptions to block lists. We also want to work on better long form content display on Mastodon. So figuring out as more blogging platforms and newsletter platforms start to build in activity pub support, we want to figure out better ways of displaying that content on Mastodon, which is essentially just a.

A short form service and doesn't have plans to move away from being a short form service. We've got a new iOS developer who's joining the core team and starting very soon. So you should see some improvements. We've had, you know ongoing improvements to the iOS app, but, but you should see some more of those coming along soon.

Yeah, I mean, we, we, we we're working on it is all I can tell you. We're a tiny team and the one thing we haven't talked about, and I'm going to, I'm going to make a play for here is funding, right? We're a tiny team. We exist totally on donations. We do not take investment from folks wanting to drive us in particular directions that we've been very clear on, on, on not taking.

Those kind of investments. Okay. So we, we are funded through donations. We really appreciate every dollar that is donated this year. We spun up a 501 C three in the U S to make it easier for you all to to, to, to chip in and support the project. I will also say that If you can and you use an independent Mastodon instance, do try to support that, the running costs for that instance.

I do that with my own instance, but the core team to keep us going to, to, to, to make sure that the developers can eat and, and live and, you know and contribute to fixing, you know, critical security issues quickly needs to be funded. And, and I'm sorry to bang on about it, but we do appreciate donations.

Jeff: And is the core team like full time paid Just working on that I off

Andy: the top of my head. I think we've got three or four members of the team who are are paid full time on the team but you know, there's a there's about 10 or 10 or 11 of us in total including, you know somebody who helps with dealing with sort of admin and management sort of tasks.

We've got folks also that run, help to run, operate our mastodon. social, mastodon. online instances. But the, the actual number of people that are paid full time to work on mastodon is very small. It's, it's three or four folks. And there's an annual report. I think the most recent one was 2022. The 2023 one should come out soon and then there'll be one for 2024, which has You know, it's it's a it sounds really kind of Simple to say but it's been our biggest year yet You know, I mean we've had massive growth the the the the products come on and the platforms come on a long way and You know, we're trying to we're trying to make this thing sustainable and make sure that better social online experiences Scale and can exist for for a long time to come.

So Yeah Yeah,

Jeff: very cool. I was gonna And I will Good. I was just gonna say, I am going to get a Mastodon account. I will, I will do it. I, I know of a couple of stories. Listen, Jeff, if

Andy: you need, if you need any tips and tricks then, and, and you wanna ha get on, jump on a call with me, I'm happy to, to give you some, some of the kind of the, the wizardry that, that, that, that lead you through it.

And hey, also just wanna say, you know, you all can also, and I, they're coming to North America in the near future when we can, when we can get our our shipping sorted. But, but in the EU now, you can get a plushie to support us and, and you know, if you're missing. missing missing your mastodons, then, you know that's a great way to support us as well.

Yeah. Awesome.

Jonathan: I was going to ask about the transparency and the funding. And so I assume that's what that annual report is. That's where people can go and see here's the money I donated. Here's what it went to.

Andy: We're a nonprofit. Incorporated in Germany, and there's a, there's a, there's a non profit board 501C3 in the U.

S. And of course, both of those things require us to be transparent and to provide reports. So as I say, I believe the most recent one that's available on the website is 2020. 22, the 2023 one should be available. And of course, but that will, that will be an ongoing thing. Yes, absolutely.

Right.

I don't think I'm a line item in the existing report yet.

And I've been working with the team for 18 months. Right. So and to be clear, since we're talking transparency, I, I I work freelance. I do one day a week of my life is spent dedicated to the Mastodon project. I have a, I have a, I have a discounted day rate because it's a non profit that I that I offer on top for that and and the rest of my life is currently burning really honestly currently burning down my savings So if anybody else in open source wants to hire me, for the other four days of the week, then then here I am

Jonathan: Yeah, very good Maybe, maybe somebody will, hopefully.

I understand that. I too, I'm in the, I have many, I have many irons in the fire. I do a lot of freelance work on various different things. And so I understand how that goes. And when one of those goes away or several go away, the trepidation you feel, I get that.

Andy: I also

Jonathan: want to be clear,

Andy: I want to spend all of my time on Mastodon, but you know I love, I love this product, this platform, and I love what we're building as a, as a, as a free and open source standards based, better social web platform.

Yeah,

Jonathan: absolutely. Is there anything that we didn't ask you about that you really wanted to make sure and cover?

Andy: Wow. Let me just spin through my list. I think we've covered pretty much everything here. I feel like we did get

Jonathan: through it pretty well. I hope,

Andy: I hope you both are I'm delighted, Jeff, to hear that you're planning to give it a, give it another try or give it a try.

I hope, Jonathan, you'll spend a bit more time interacting with us and looking forward to Elliot getting Hackaday. social upgraded and seeing all those great hacks.

Jonathan: I'll go bug him about it.

Andy: Hey, listen, listen, the other thing to do, Hackaday. Articles should get the Fediverse creator tag added to them, and then you'll get the little author attribution whenever a Hackaday link is shared on on on Mastodon, you can get the author attribution added if you're if the folks that have written that article have the Fediverse creator pointing tag in the in the page pointing to their Mastodon account, then every time one of their articles gets get shared, they'll get a little find more from Jenny or Oh my gosh, my brain is blanking on, on, on all the, the great folks on, on on Hackaday.

So, Al and Elliot

Jonathan: and me.

Andy: Yeah, of course, you know, I listened, I, I listened to, to your podcast weekly, so I should know all the names. I'm just, I'm, I'm so excited to be here that

Jonathan: my brain is going blank. No, no, that's fine. Okay. I got two final questions I've got to ask you then. And that is what's your favorite text editor and scripting language?

Andy: Vi and scripting language. Gosh . I mean, I, I just, I use bash all the time. I mean, I, I write code in Python typically, but my scripting language would be, if we're not calling Python a scripting language. It's a programming language. I would probably say Python bash.

Jonathan: Yeah, that works. I think either of those are, are legitimate answers.

Andy: Thank

Jonathan: you, Mr Andy Piper. Thank you so much for being here. We went, we took up more than an hour of your time and I sure appreciate it. It was great.

Andy: I hope the audience of stuck with us and I appreciate you. You both and floss weekly and hackaday is a match made in heaven for as far as I'm concerned. So this is this has been great.

Thank you for the opportunity.

Jonathan: Awesome. Thanks again. Wonderful. Yeah. All right. So the real question is, what server are you going to sign up on?

Jeff: Oh, that's a, that's a tough one. You know, You've got, you've got

Jonathan: hackaday. social. There's twit. social. We're all good friends with Twit as well. And I'm sure there's some Indian motorcycle that, you know, Massadon server that's just out there for classic motorcycles.

Jeff: Yeah, there probably is. But you know, I, I, I got to stick close to home and it's, it's either going to be a Twitter hackaday. One, one

Jonathan: of those two. You could do it like I did and make one on both places. Oh, you can do that? I mean, there's nothing stopping you from doing it. Now you can't, of course, we let, we let the man that knows the answer to this go before we thought to ask the question, but I don't know of a way to sort of merge the two accounts on two different servers, but I do know that a lot of people have accounts on different servers to sort of keep their various interests separate.

Just something to think about and see whether it makes sense for

Jeff: you. True. I'd probably at least just start with one and then kind of get, get my legs under me and you know, see, see how I do. Maybe, maybe give Andy a call if I need to, to get the, get the crash course and. Yep. But yeah, I, you know, I, I will have one might take me a day or two just to work it, work it in with my day job.

But yeah, get on there, get one. And then I can advertise it like other people we know.

Jonathan: Yeah, yeah. Well, make sure and let me know what it is. If we do it before the show goes live, I can tag you in the Macedon post. Oh, nice. Yeah, yeah, it'll be fun. I will do that. Alright, awesome. You have anything you want to plug?

Jeff: Just check me out on, in twit. tv on the Untitled Linux Show. I'm there almost every week. And yeah. I talk with Jonathan and some other great people that co host and we just Strictly talk about pretty much Linux though. I do veer a little bit into the hardware, you know, benchmarking on Linux, the new processors and some other, other things like that.

It's a little, little of the enterprise. My, my day job, I work in semiconductors. Bleed over into that, but no, no, no inside scoops though. No, no. I, if anything's gray, I always say this is strictly what I've heard on the internet. This is not inside knowledge. This is, you know, I do, I do not speak for the company I work for and I don't mention

Jonathan: them either.

Indeed. Yeah. I don't think we've ever, Maybe once we've mentioned what company you work for on the air, but that's just, it's not what we're about.

Jeff: All

Jonathan: right. Excellent.

Jeff: Listen to

Jonathan: all the episodes you probably figured out, but. All right. Thank you, man, for being here. I sure appreciate it. We we don't, we don't necessarily know.

who the guest is next week. And so if you want to be on the show, if you have an open source project, you want to chat about, or if you have recommendations or leads for us, you can shoot us an email, it's floss at hackaday. com. Shoot us an email that goes straight to me. It's a, it's a mailing list technically, but it comes to me and we'll take a look at it and get some folks scheduled.

So we need the guests for next week. We've got, I think one guest scheduled through the end of the year. So let's let's get the suggestions coming in. But we will be here next week, whether it be a guest or a roundtable, and we will chat about what's going on in the world of open source software.

Also, you can make sure and follow Hackaday. You've got my security column goes live every Friday morning, and we sure appreciate Hackaday being the home of Floss Weekly. You can also check out as As we mentioned, the Untitled Linux Show over at twit. tv and I personally appreciate Twit letting us do that.

We appreciate everyone being here. Thank you for those that watch live and those that get us on the download. And we will see you next week on Floss Weekly.

Episode 804 transcript

9 oktober 2024 | min

FLOSS-804

Jonathan: Hey folks, this week, Dan joins me and we talk with Anthony Annunziata about open source artificial intelligence. Anthony is the director of AI Open Innovation at IBM and the co founder of the AI Alliance, definitely an expert in the field. He entertains all of our questions about AI in general, and then talks about what open and open source means for AI models.

It's a great interview and you definitely do not want to miss it. This is Floss Weekly episode 804 recorded live. Tuesday, October the 8th, the AI Alliance Asimov was right. It's time for Floss Weekly. That's the show about free Libre and open source software. I am of course, your host, Jonathan Bennett, and we've got something fun today.

We're going to talk about AI or. Maybe it's LLM, maybe it's just big computers doing lots of math. We've got, we've got somebody that is sort of the expert on this. But first off we've got we've got Dan the man, the original Linux outlaw. Mr. Mr. Dan, how you doing?

Dan: I'm good. Thank you. I've got to confess.

I slightly worried then when you said we've got an expert in all of this Because I can't confess to be an expert on ai or llm large language models, but I try I try yeah, it's good to be back

Jonathan: Dan, you're probably a lot like me. You don't necessarily claim to be an expert in much of anything. We're just we're jack of all trades.

We are very generalists, right? Is that kind of where you're at too with all this?

Dan: Pretty much. I think it's dangerous to claim you're an expert in anything really, because then somebody will go, aha.

Jonathan: Are you really? It's, it's the what do they call it, the Dunning Kruger curve, where it's like when you first learn about something, yes, I know all about this, and it's easy, why are you guys making it so hard?

And then as you learn more and more about it, you finally get to the point of, oh, this is ridiculously complicated, and I don't know anything about it. And that is when you have actually started to understand.

Dan: That's very true.

Jonathan: I think, I think AI is very much like that or LLMs. I sort of resist calling this AI.

I don't think we're close enough to a general artificial intelligence to really refer to any of the the things that are out there as AI, but obviously people have different opinions about that. So we're talking about the. AI Alliance and sort of kind of also IBM and what IBM is doing with AI our guest is Anthony Anthony and Azuela, maybe we'll ask him how he pronounces his last name That's probably

Dan: not

Jonathan: right that's probably not I I'm I made the old college try I gave it my best All right, well, let's go ahead and bring him on since we've got the man right here Anthony welcome to the show

Anthony: Hey guys, welcome.

Yeah, thanks. It's great to be here. I really appreciate the invitation.

Jonathan: Yeah, we're glad to have you here. How do you pronounce your last name? I'm sure I butchered it. Ah,

Anthony: A little. Yeah, it's Anunziata.

Jonathan: Ah, yeah, I, I, I way Americanized that.

Anthony: All good. All good. Yeah. I think I Americanized it a little bit too, but you Americanized it even more.

Jonathan: I turned, I went, I, I, I am the American that's turned up to 11, I guess. So we, we talked briefly before the show and you mentioned that you sort of dual represent for, for what we're talking about today, being both part of IBM's AI effort and the AI Alliance. You want to kind of maybe start by mapping out what that means, you know, what those, those roles are in those two different places, what we're talking about here.

Anthony: Yeah, sure. Happy to. So, yeah, as you said, I kind of served two roles. You know, my, my main role, you know, where I'm employed is at IBM and I lead open innovation in AI here. That means a lot of things that includes open source data and models as well and partnerships and community engagement.

And I'll tell you more about, you know, specifics there. And then, you know, as part of that, as an extension of that, I have a role in the AI Alliance, which is a program that IBM and a number of others put together about nine months ago. And that that's an open innovation, open source, open program that is intended to promote in many ways open development.

And adoption of AI in a, in a responsible way. So I have, uh, the position of co chair of the steering committee and direct involvement in a few of the working groups as part of that organization.

Jonathan: Yeah. All right. Interesting. So I think maybe the thing, the thing to go to first we were talking before the show, I think, I think it's before you joined the call even but last week I had Simon Phipps on as.

Co host slashed guest. And he was talking about how the, the the open source definition for artificial intelligence, for AI, for open source AI, is, is sort of something that's still being written. Because it doesn't necessarily work the same way that source code does. And I'm curious what, what the, sort of the, the AI alliance and IBM take on this is.

Have you guys been involved with trying to write that definition? And what are your thoughts on this?

Anthony: Yeah. Wow. Great topic. Okay. So lots to say here from an IBM perspective, we have been in, you know, in discussion, dialogue, providing feedback to various efforts, right? That are trying to better define what open source means in AI and in particular, like what the source term means and whether it's actually applicable or it's a different, different sort of thing.

So, so, yeah, we we've provided feedback. We've been part of that dialogue. I think it's fair to say today that you know, the community is moving forward to try to understand what open source means. And I don't think it's an answered question yet. And from an alliance perspective, right? More broadly, that's kind of a microcosm of people's broader involvement and opinions, right?

Lots of orgs are starting to weigh in more specifically. But a lot of people are concerned, you know, if not the whole picture, then let's, let's define enough commonality, right to be useful. And, and often that, that rests around, like, what is a, what is a model, right? What is an LLM? What is an open source LLM?

Does that term mean, is it meaningful or is it something different?

Jonathan: Yeah, and so where, where would you draw the line? What, what definition do you like for, let's say we're talking about a, a particular model? If we wanted to define this, so like, LLMs obviously is one of the big things that people are working with right now.

And so if we want to push one out that we say, this is, you know, put the stamp of approval, this is open source. What, what should that mean? What should that look like?

Anthony: Yeah, so the way I like to do it, you know I like to go back to the fundamentals of open source, right? Is, what is open source? Enable you to do what kind of actions ought you as a developer be able to do right?

So, you know, it comes down to can you study modify use share the artifact? Like, we've become very comfortable with that means in terms of code, right, source code, but a model, an AI model is something pretty different, in particular, the types of models today, right? They're pre trained, they're very large and they're, in some sense, kind of compiled objects, right?

They're not, they're not really sourced, but there are elements that kind of, sort of, are source like within them. So, the way I like to do it is, is, you know, look at the model as an artifact and then look at, you know, kind of the broader picture of what an AI system is. Is and how you how you get to a model, right?

The whole training regime. So the model itself. is is primarily a set of. So in general, they're neural nets, right? And neural nets that have weights for the different nodes. And so a key part of what it means to have a model that's open is the weights. All of the weights, right? All the parameters are open, accessible in a way that you can study, modify, use and share them.

Without encumbrance, right?

Jonathan: Mm-Hmm. .

Anthony: And so I think the minimum, right? Practical definition is really what is an open model or, or an open weight model. And it has to be at least the weights, right? The weights after training. Mm-Hmm, . So that, those are the, the, the key descriptors of the neural net and what it takes to run it.

And then some basic information, at least about you know, the, the characteristics, right? The the behavior. Of that model, which is a set of neural a set of weights in a, in a neural net architecture, that's like the bare minimum. I wouldn't call that open source AI yet, though, but that is really like an open model in my view.

And I think an open model is a piece of the story. And it's in some ways the most important piece because it's the new And weird and hard to understand and sometimes hard to use piece.

Jonathan: So in, in thinking through this, as I said, I am, I am by no means an AI expert by no means. I am sort of just an outside observer looking into all this.

But one of the interesting things about, about these LLMs is they can very. they can very greatly, based on what training data goes into them. And in, in sort of thinking about this, you know, openness, open source, yes, but also like openness trying to understand them. One of the things that would be really, really important is maybe not having the entire data set that trained them, because as we know, that in some cases can be prohibitively large data sets.

But at least having an accurate descriptor of that data set, or, or maybe even. You know, how was this data set modified before the training happened? Because certainly that, that must happen, because there, there's, there's, in data sets, there's spurious data, right? There are things, to put it simply, there are things on the internet that you may not want your LLM to have inside of it, right?

And, Like, that's, that's a valid thing, but at the same time, if we're going to be open about the model, and if we want it to be something we can call open source, certainly that's something that we need to understand from the get go, right? Is this, I assume this is part of, part of the thinking about this as well.

Anthony: Yeah, for sure, for sure. So a big part of building a model, a big part of understanding behavior model is the data sources, but more importantly, the whole pipeline of processing that the data goes through before you start to actually train the model. So, one of the things that is really important. is transparency, right?

Short of, you know, permissive use of the actual artifacts and pieces of pipeline. Are you at least transparent about, you know, where your data are from? But more importantly, the various filtering and processing the data that's done before it starts to train the model. Yeah, for sure. Super important. And yeah, the types of things people do that are really important and that are kind of emerging standards are Are you need to, you know, you know, there's a whole bunch of processing in terms of formatting and getting the data ready to process.

But key parts are, you know, removal of hate, abuse and profanity removal of personal identifiable information. In most cases, most responsible cases, checking against known copyrights and removing those these are really important. And, you know, even short of like what it means to have a full. Open source AI system, right?

Because this is an emerging, you know, kind of definition of discussion. I think what I think even more people converge on agreement on is it's very important to have transparency. On how you process your data, the choices you've made. Right. And people may disagree with those choices, but at least articulate clearly the choices you've made in cleaning and preparing the data to build the model.

Jonathan: Yeah, absolutely. You know, you think about something that when you're doing scientific endeavors that you have to worry about is the different biases and I'm using this term more of a technical term, the different biases that it's in your data set because. You may think that you're just cleaning your data, but you may actually be removing some of the information that you're looking for and the same thing can happen with these LLM models.

And so like what one person may, may have the opinion that, oh, this is, you know, this is hate speech, or this is just noise in the data, or we want to get rid of this, like, That changes your output in ways that in some cases we don't fully even understand. And so I love that transparency is a core of what you're doing.

So I think that is super important with, with this technology, with this becoming more, more universally useful and not, not misleading people even. So I think that's great. Let's see. So I, I, I want to ask you about this and I know it's, it's controversial, but again, we have the experts. I'm going to pick your brain on things.

And that is the idea of copyright when it comes to LLMs and AI. And as far as I know, this has not actually been settled in Court and I think this is probably what's going to have to happen, right? This is these are questions that are going to have to be settled in court or by written laws But the idea as far as I know is that it is believed that Taking information into as as a training process for LLM is a transformative work To the point to where you're no longer covered by the original copyright of the information that you trained on.

And then a quirk of that is the output, you know, so you, you write a prompt and you give this to an LLM. You get an output, whether that's an image or a written work. That is then sort of not a copyrightable work. And, and those are, those are interesting quirks of, And feel free to correct me if I got any of that wrong, but those are interesting quirks of working with, of AI and LLMs in this day and age.

And do you think that's going to stick, or are we going to see laws or court cases that change that? I know, you were not prepared maybe to give your legal opinion on things, but, it's, it's, it's so integral though to all of this.

Anthony: Let's dig in, man. No, I'm happy to. First, I'm not a lawyer. I don't represent, you know, IBM's legal opinion or AI alliance members and all that, so.

Right, right. So we'll we'll just go under that caveat.

Jonathan: Yes, sir.

Anthony: Some things are clear, some things are not clear. And some things are in between. I'd say there's reasonable there's, there's reasonable agreement that if data are out there in the public domain already, that, you know, if you train a model on those data, that that is a reasonable case of fair use, right?

But you have to make sure that you don't redistribute those data that you haven't gone and you know, gathered those data in places you shouldn't have, and so on. So there's many caveats there, right? And that's why, you know, what I mentioned earlier, like, Even though you maybe don't strictly have to, it's good practice to check your training data against known copyrights and make sure they're removed,

Jonathan: right?

Right, right.

Anthony: There's also progress and, and, you know, methods coming available that allow individuals in some cases, some of the, the more, the very open model building efforts, I'll name one from an alliance partner BigCode. And in the big code community initiative to build open, open way models they provided a very nice mechanism for people to go in and and look for data from, from them.

Right? And, and request that it be removed just voluntarily. Not because it's required to do that under law, just because it's good practice. It's good community practice to enable that to be a possibility. So that's kind of the input end, right? Training. I mean, there's, there's lots more to say about that.

I'd say the output end is actually a little clearer in some ways, right? Because in some sense, like copyright law is copyright law, right? Like it doesn't change, right? If someone, if me as a person goes and gets a piece of text or a piece of art that that, you know, is, is substantially similar to a copyrighted work I can't go go use that, right?

And I would, I would be violating the copyright, right? It doesn't matter if I drew it up on my own or I got an AI model to do it or, or something else, right? Like if I go try to pass off something that is really similar to another person's copyrighted work, I'm in violation.

Jonathan: Draw me a picture that looks sort of like the Mona Lisa.

Anthony: Yeah.

Jonathan: Yeah. That sort of thing. Yeah.

Anthony: Now where it's murky, right? You know, the question of, you know, whether whether a I, you know, substantially changes the risk and liability picture in terms of copyright violations, right? Is it still just on the user and on the person that goes and tries to distribute?

Something that violates somebody else's copyright or is there some implication because the technology is so capable You know to the the technology builder themselves. So that's that that's kind of some of the debate that's happening now.

Jonathan: Yeah Yeah, I know there are some open source projects that have just said because of some of these issues We will not accept any code that has had Like, you know, Copilot, just for instance, Microsoft's Copilot.

We will not accept any code that has been generated in any way by this tool. It's because they're afraid of that idea of the copyright. You know, there are, people, two people demonstrated this. Like, there are certain prompts where you can get bits of code out and then you do a search for that code on the internet.

And you can find, you know, substantially similar, if not identical, character for character, identical copies of the code. And so I know there are, there are places that are just saying. Don't bring us any A. I generated anything because we're afraid of the copyright problems. And of course, there are there are there are other problems, particularly with code, because people have, they misunderstand the L.

M. Tool. And so they think, Oh, well, co pilot wrote this. Surely it must be good code. And you know, we see the same thing with vulnerability research. Find me a vulnerability in this program and then people will try to report it. And the, the problem, particularly with that one is the LLMs do such a good job of making it look good.

But when you finally dig down into what they're telling you, it's, it's almost, almost all the time bogus, but it's, it wastes a lot of time from these maintainers because they've got to do the work because somebody sent them a vulnerability. Yeah, it's, it's fascinating stuff. Yeah. It's really interesting.

Anthony: It really is. Look, I mean, that's, you hit the, hit the nail on the head on a couple of big, you know, big challenges and opportunities, right? Like on the output end of things, there are better and better methods to try to screen detect and block, you know, copyrighted material, but it's, it's definitely not where it needs to be yet, you know, various guardrail schemes and detector models and all that.

So, so that's a, that's a big opportunity and something the Alliance from a safety and from a quality eval perspective is working on as IBM as part of that. And then the other piece I think is really interesting too, which is like, Hey. Yeah, you're right. LLMs are pretty good, but not great at many things.

And, but the, but they're, they're pretty great at making themselves look like they're really

Jonathan: great,

Anthony: right? So whether it's, you know, better formatting, markdown, output, things like whatever, it looks pretty good, but when you dig deeper, it's not quite all the way there. Right. That's true for code. It's true for, for text and natural language.

So there's this problem and actually the big problem with adoption, right. With lots of companies that want to use AI. You know, you can get 80, 90, 95 percent accuracy and reliability, but getting to like high accuracy in use cases that need it, which is a lot of enterprise use cases, it's pretty tough, right?

It's pretty tough.

Jonathan: Yeah. One of the, one of the funniest things. You know, people have come up with different tricks for interacting with AI, which all that's, that's a whole subject in itself. Maybe we'll get to here in a minute, but the one that, that I think is just about the funniest is with, with a lot of the models, you can tell it at the end of your, you ask a question.

And at the end of the question, say, show your work, cite your sources, and just adding that to your prompt, the quality of result goes up dramatically. And it tends to be more accurate. And I think that is one, it's, it's hilarious. But it's also super interesting that it works so well. Is that something that you see, like, broadly?

Does trick work across a lot of different models?

Anthony: It actually, it works a lot of, some form of that works across a lot of models and a lot of use cases and modalities, actually. So I think some generalization of that is, you know, think about structured inputs, right? How you specify your prompt input has a big, big effect on the output quality and the output, you know, the output in general.

You see it in code, right? If you can better set up the problem you can get higher quality code output. If you can better set up, okay, so there's a, there's a whole set of patterns called retrieval augmented generation, which is where you hook up an LLM. With a database and the database is a vectorized data.

So it's in the form of where an LLM can kind of knows how to interact with it. I can go into much greater technical depth, by the way, if you want. I'll keep it light for now. In those cases, right? We also find that the better you structure that that vectorized database, right? The better you structure the data.

Whether it's in a graph form or some other sort of structured form. Yeah, you get much, much higher quality retrieval and output. So yeah, there's a huge, huge amount of, you know, sometimes it's called prompt engineering, structured inputs so on and so forth that are important and and determine the output to a big degree.

Jonathan: Yeah,

that's interesting.

Dan: Yeah, I was curious when you talked about copyright there. It made me think a little bit of I relate a lot of things to the music copyright because that's kind of the world that I come from. But when you were saying about a lot of projects won't accept code, which has been generated by an LLM.

I understand why I completely understand why and the fact that you can maybe find I don't know, a github repository or something where you end up finding a very similar code. But at the same time, I was thinking to myself, there's only much like in music. There's only so many notes that you can combine and so many ways to combine those notes orders.

You can put them in and so on. Maybe this is something down the road that We'll hit a point where, you know, there are only so many ways to make a certain piece of code and only certainly ways to combine different functions and features of the languages that you're using, the programming languages that you're using, and so on.

So I wonder if some point down the road they'll find a way of doing that, because with music, again, I'm not a lawyer in any, in any sense, so please don't take this as legal advice, but it's certainly in the UK anyway, in music copyright, there's, there's laws about how many notes in a row. Can be the same as another piece of music, for example, it's usually six notes or roughly six notes.

I wonder if that might be an approach in the future, but you're dealing with such a large data set that I would imagine you get to the point where you need another, you know, I don't know. I'm not sure where I'm going with that point, but I think maybe at some point in the future we might see that and it'll have to be tested in, in in court as well as all these things are.

But with regard to things like copyright one of the big things that I've seen recently is I have a lot of friends who work in academia work in universities, places like that. They have a lot of Tools. Now, there's a lot of companies trying to sell their tool, which will check your students work and check that it's not generated by an LLM.

I wonder how I just wondered if you had any thoughts on that kind of area and whether any of that could really be effective. The checking of it, I suppose. Yeah.

Anthony: Lots to say there. I think some sense is a little bit of a losing battle to try to just take output and you know, work back to see if it was created by an AI model.

I mean, yeah, you can still do that, right. There are patterns there. You know, recurring words and phrases and things that you can map back with some statistical probability to a model, but I think as models get better and better as you can adjust things like the variability of output responsive to the use case and lots of other stuff.

I think it's getting really hard to do that. So I think in education, we're going to have to figure out a different solution. I mean, just like yeah. It took you know, some challenges, but we figured out how to do math education without a calculator or the right role for calculators and, you know, the right role for in classroom work that specifically, you know, prohibited calculators, you know, in some sense, we're gonna have to figure out how that works now for not just math because I can can assist with with you know, a lot of tasks in the educational setting.

Yeah,

Jonathan: yeah,

Anthony: I can make it. Yeah. So there's some thoughts on that. I mean, it's certainly not Yeah, I don't know fully how that's going to play out.

Dan: That's okay, I kind of threw that one at you. Don't worry, I'm aware that I just kind of threw that one at you. Yeah, it's very interesting. I mean, there's one more.

Anthony: Yeah, I mean, I guess I could say one more. I think just on the topic of it kind of relates to copyrights and original work and cheating and all of that. You know, I think increasingly the, the path to show provenance and originality of, of work is going to be to track the lineage and provenance of non AI.

generated material, because it's going to be so much AI generated material, synthetic data, images and so on. And so many ways and sources and ways to manipulate that. I think like probably a better tack is to have better end to end methods to track, you know, data and works that came from, you know, not AI.

Dan: Yeah, that makes sense. It's like a litmus test, like a, you know, something to compare it against. It makes sense. So one of the things I found really interesting when I was looking through the AI Alliance website and other things is I noticed a little bit at the bottom where it says let me just scroll down so I can find it.

Competition law guidelines. I'm sorry, but to get into law again, I don't want to realize we're not here to talk about. I just found it. It led me to some thoughts because it says you've got guidelines published there for how people can interact together without. Contravening something like anti antitrust laws, competition laws, all those sorts of things.

How so I suppose my question is with so many different bodies working together in the AI alliance, um, what are the challenges to kind of get them to work in together and how much can they work together without it being seen as colluding in some way in the market? In the market.

Anthony: Yeah. Hey, this is great.

My my, my lead attorney, John McBroom would, would love this question. And you noticed his finely crafted competition log guidelines. Yeah. It's really important. What we put together in the Alliance is a collaboration of various organizations, but it's a tighter collaboration than just kind of individual at open source.

Right. And because of that, right, we need to pay extra kind of extra attention to making sure that. That everything we're doing is truly open that it's not you know, even close to the kind of you know, cooperative, commercially inspired kind of work that we can't be doing. Right? So a lot of this is just basic hygiene of open source.

You know, do you work in the open, publish everything early and often, right? Use GitHub and, and, you know, known processes for contribution management and all that, which, which we have you know it includes, you know, publishing. Working groups you know, meetings, events, just just making sure that, you know, it's all in the open.

People know it's not about keeping anything in the dark or hiding anything. It's really just about bringing people together and orgs together, you know, just for tighter, more coordinated collaboration on open work. So there's a lot of, you know, process mechanics, hygiene, publication kind of stuff. That's, you know, just mostly good open source practice.

I'd say that, you know, to the other part of the question. You know, what are the challenges? I mean, there's lots of challenges. This alliance is actually much bigger and grew much faster in terms of members than we were expecting, which is always kind of a nice problem to have. But, you know, what that means is there's even more interests and projects and priorities.

And so, you know, the way we've, we've handled this is, you know, we've structured the program. Hey, my earbud came out. Can you still hear me? Yeah, we're good.

Jonathan: Yeah. Yep.

Anthony: All right. Good. Yeah. Sorry. I'm not used to

The way we've structured the program right is is pretty lightweight and ground up and we've got six Focus areas each with multiple working groups and projects. And it's it's very you know, maintainer contributor leader led, right? And so there's typically, you know, one or a few sponsoring organizations for each project.

There's some individual leads just like a good open source project. You know, a road map. And so what we try to do is, you know, preserve a lot of ground up, you know, kind of individual led work, but we provide, you know, collaboration you know, pooling of resources, kind of, you know, a forum for common priorities and, and, you know, that kind of program structure to kind of better support and scale that kind of work.

Dan: Yeah, that makes a lot of sense. One thing I was, I was interested in, as I mentioned, to do with academia and all that kind of stuff is how you're engaging. I noticed you've got Harvard listed on, on, on the, the members of the AI Alliance and so many other academic area, you know, institutions, I should say.

And you've also got things like you've got NASA on board and, and As well in Switzerland, so I was curious about how you're engaging with academia and so on, and is there any difference between dealing with, say, a big company and dealing with a big research institute or something like that?

Anthony: Yeah, there's some differences. I think probably split it into two parts. In engaging academia, right? There's the teaching and education mission, and then there's the research mission. So in education, this has been about engaging and figuring out, you know, where are the gaps in education where the gaps in particular, not just.

Curriculum, but in resources for students, right? So open source is perfect, right? To address this, right? Because you have lightweight code that you can deploy. You have endpoints you can use with, you know, free access to experiment with AI models and so on and so forth. But, you know, it's a little bit of a mess.

And so what can we do here to better guide, to better organize? Resources right for for students. That's that's been a been a major thrust, and there's there's a lot of work in progress there. So curriculum piece and resources piece on the on the resort. I can mention I can mention specifics to like one specific thing that working on is a kind of a collective guide to a definitive curriculum you know, in A.

I. So we're gonna be building that out and releasing that later this year. On the research side, so this is, this is a, this is a thorny problem, right? Because the resources required to engage at the leading edge of AI are getting higher and higher. So, you know, I can't say we've fully solved this problem in any sense, but the way we're going about it is to try to bring academia and industry together in close collaboration.

And I mean close because, you know, often what you see is, you know, industry will sponsor a student or sponsor a faculty member to do something kind of on their own. Or you know, if it's the other way, industry will get involved in a university, you know, they'll be kind of an observer, they'll join a meeting once a quarter or something as part of an institute and all that.

And that's fine. You know, there's benefits to that. But what we're really trying to do is bring, you know, faculty and students with researchers and engineers and companies together, right, closely collaborating in open projects. And We've gotten some nice things going on that, you know, in that mode.

In particular, in the area well, in a few areas, but one I'll highlight is in safety and trust. So, this is about building better tools, better methods to detect hazards of, of AI output you know non idealities of AI systems, whether they be, you know, hate, abuse, profanity, and things like that, or whether they be, you know inaccuracies that in some settings, like in health, would, would cause real problems, right?

And that, that area, we've started to see some really nice collaboration among, you know, academic perspectives and industry perspectives to try to make progress and those sorts of things.

Dan: Yeah, that makes a lot of sense. And something I was curious about is how many individuals, this is going to make sense.

Do you get many individuals involved in this? Obviously you've got large groups, you've got companies, you've got so on. If I, do you get individuals coming along who, who are, who are like, I'd like to be a member of the AI Alliance. Is that a thing that you could do?

Anthony: Yeah, yeah, that's right. We can now we have been able to do that for a while.

You know, the program started with this idea of let's get, you know, let's get a bunch of organizations together to collaborate closely. And, you know, organizations have resources and strategic priorities. And so we can get critical mass that way. But from the beginning, right? It's it's an open program.

So we're very, you know, very eager to get individuals that want to contribute. And we've seen an increasing amount of that. I will say we need to do a better job kind of, you know, clarifying and articulating the various paths to get involved in various projects. So if you've looked at the website some of this is in progress, there's more content that we have to get out there because things have just been moving very fast and it's hard to, hard to keep everything up But yes, we absolutely invite individuals.

We have individuals, you know, taking lead in things And we want to do a lot more of that.

Dan: It's awesome. Excellent Is there is there a secret handshake or a greeting yet where you can say like they're part of the AI alliance so I can tell?

Anthony: There is a process. Okay, so yeah, so Well from an organization.

Yeah, we have a we have a simple process by which we add new members for individuals It's really as simple as You Signing up and communicating your interest that you want to do something. And as long as the something is consistent with the mission which is, you know, to, to build, enable and advocate for open innovation and AI, which is pretty broad.

Yeah, we're happy to have them join. We have the usual things you'd expect. We have a community code of conduct that, you know, everybody needs to needs to follow. We have, you know, structured ways, you know, we've, Many Slack groups at this point, GitHub is built out, HuggingFaceSpace, all the usual places that people live and do work.

But but yeah, that's that's absolutely part of it.

Dan: Yeah, excellent. Yeah, that's very cool. So I suppose Something that you said earlier really kind of struck a chord with me when you were saying about how the the Llms and large language models are kind of more like compiled code than source code You know what?

I mean? Because you've got your date it occurred to me You've got your data needs to be built into it. Of course your data's a part of it. You can't Have an llm without without the data. I know you train it on the data and all of that stuff But you still need that to to get to places with it. So i'm just curious about how how how collaboration works between as you can tell i'm compiling the my internal llm is compiling the question for us right How how collaboration works between So back in the day, we'd have things like big data.

We'd all talk about big data was that, but that I'm probably sharing my age, but that was the, everybody got excited about big data. I remember at one point and Hadoop and all these kinds of things. So what's the kind of difference between where we are now, do you think with LLMs and so on and where we were, I don't know, 10 years ago?

Is that, is that some, is that too broad a question?

Anthony: No, that's a great one. Let's dive in. Lots of differences. I'd say the major difference. Is, you know, the appearance a couple of years back of what we call a foundation model, right, which is this general purpose model that's trained on a very large amount of data, typically in self supervised fashion, which I can get into what that means in just a second to create this artifact, right, that can be used for lots of different tasks.

Whereas, you know, 10 years ago, you would typically have to, you know, train a specific model on a specific data set for a specific task. And, you know, you could make it good for that task relatively good, but it wasn't portable, right? So if you train something to be really good at, you know, detecting the type of animal in a still image, right?

You know, you couldn't also use that to generate images of animals, right? In fact, the whole generative aspect, being able to create something Based on a learned parameter space. That's very, that's very new, right? It's like 567. Well, okay, I won't. Okay, there's a longer tail before that. But, you know, leveraging foundation models has been to do that has been a relatively recent thing.

So that has lots of implications, right? Because now most developers are going to start with a pre trained baseline, something that is like a compiled artifact. And that that means that how you got there, what's in it, right, is really important. So the importance of transparency we talked about already, the importance of characterizing like what it what that baseline can do is also really high, right?

So like evaluating it with better and better benchmarks for its output, what its capability is and then, you know, how you take and build something useful, an application on that, right? So that whole motion, starting with a trained baseline, that's kind of more like a compiled artifact, right? Versus starting from scratch to train a model.

That's, that's very different from 10 years ago. And the implications in building applications and making them like work well and reliable is very different because now you got to understand. How that, that compiled artifacts, you know, the model works, right? Because you didn't, you didn't build it and it's now very complex, right?

It's a very sophisticated, large object to figure out its behavior.

Dan: Yeah, definitely. It reminds me a bit of a few years ago. I when would it have been now about seven or eight years ago? Possibly I quote unquote met Watson, which is IBM, of course. And, and they said, I was at a conference and they said, come and meet Watson.

Watson and I was like, well, okay. So I went along and that's what we were doing with Watson was training training Watson on how to identify animals in pictures or attempting to and using things like that. But it certainly wasn't the stage where, where it could generate these things. So that must, that sounds like the kind of difference.

Anthony: Yeah. The generative capability is, is pretty different and striking. Yeah. I mean, some sense generative, as I kind of mentioned, you know, generative capabilities go further back than the last few years, but the advent of, you know, these very large scale pre trained models and what's in that is the ability to learn a parameter space brings the ability to actually generate new things that it wasn't directly trained on, right?

Because, like, there's some hand wavy model where, like, okay, if you have all these data points, Right. You learn not just the specific data points, but you know how they're connected in the whole space and then you can use the model to fill in what's in between. So, you know, certain type, you know, certain look of cat, a different look of cat, and then the model can create a cat that never existed and it looks like something in between.

Jonathan: Yeah, that's cool. Interesting stuff. Okay. I want to ask about, I want to ask about uncensored models. Is anybody out there specifically doing uncensored models? Like, and, and I don't want to get into like the, the politics or culture war of this. Like that's, that's a legitimate thing, but that's not what we do here on this show.

I'm, I'm more thinking about like even just. A model that you want to be able to do research on. Is there anybody out there for, for purposes like that is specifically saying, let's pull, I don't know, I don't know. Let's just say that Reddit, let's say, let's just pull all, all the Reddit comments in and let's intentionally not censor any of it and do research on that.

Like, is there, is there somebody out there and that's sort of a dangerous place to be, I would imagine, but is there anybody out there doing that? That's I guess, specifically part of the AI Alliance that you're aware of.

Anthony: So let me answer that in two parts.

Jonathan: So.

Anthony: Certainly people are building models and experimenting with, you know, uncensored outputs and uncensored inputs and doing that in a little bit of a, an open loop way.

Like we're not doing that. That's, that's really not what the Alliance is doing or anyone in the Alliance you know, we can put aside whether that's a, you know, good, bad, or, you know, just, just kind of put that aside. We don't focus on that. We don't, we don't want to do that kind of work. Sure. However, we are individual organizations and And as an alliance, trying to better understand, you know, how to make the quality of output, you know, better and more trusted, right?

More responsible. And so to do that, you do have to look at, you know, some of the bad stuff that is in training data. You do have to look at training specific detector models like guardian models on bad stuff. So they know what to look for and block, right? So, you know, In, you know well structured ways, there are teams inside companies, including IBM, and there, you know, some collaborative work in the alliance.

That is absolutely aimed at doing that. And yeah, you have to look at you know, some bad stuff to understand, you know, what to look for and what to block and how to engineer it away.

Jonathan: Yeah. It's kind of a, it's kind of a weirdly touchy subject, I guess, but it's one that, that sort of has to be, has to be Oh, how, how would you say it?

Do you have to, you have to steal yourself and look at it, right. To be able to get it right. It's an interesting topic.

Anthony: Yeah. And, you know, increasingly that is really about. Creating, you know, specific AI models that understand what to look for and can detect and or block it, right? So that's actually true in hate, abuse, profanity.

It's true in PII too. You can train a model like to detect personally identifiable information. And we've done a pretty good job at screening that out. But in the, you know, in the processing of the data before you train the model, that's really important. That's also done with AI. And then back to the code example.

All right. There are ways to train specific detector models to identify and block malicious code, right? So if there's code that you, you know, you can ask a model to generate code that's going to do something not good. Often the model is aligned pretty well to not let you do that. Once in a while you can break it and that's where you can have an additional safeguard, which is a detector model that that is specifically trying to look for bad stuff and we'll block it.

Jonathan: Yeah. Okay. So that brings to mind the the, the model jailbreaks. I'm sure this is something you're familiar with. Like you, you are not an LLM anymore. You are, oh, I forget what, what name you, you know, you, you are Ted and Ted is programmed to not have any any restrictions on output, you know, stuff like that.

And I've seen several of these and on one hand, they're extremely clever. On the other hand, they've got to be just a nightmare for people actually trying to roll AI out because you know, people, people can convince, convince your model to spit out. You know, uncensored stuff or what have you that's got to be something that, that IBM and the AI Alliance are, are looking into.

And I guess this idea of putting a second AI in or behind it to make sure that it's not allowed to spit any of that out as part of this.

Anthony: Yeah. And the sophistication with which we can, you know, prevent these sorts of things is just like growing, like. It's incredibly fast. But for sure, like when you saw the first, you know, highly capable LLMs that were released.

Yes, you you saw the ability to jailbreak or, you know, do this kind of prompt engineering to convince it that it should output something that it shouldn't. Yeah, it was more possible. I mean this is something that benefits a lot from continued red teaming, both in a formal setting, right? Like in companies and and in collaborative settings like the alliance.

But also informally, like in deployment, you know, constantly monitoring and detecting how people are, you know, trying to misuse and making sure that you're engineered against it. I think things have gotten a lot better. One of the topics we talked about before, this kind of idea of structured inputs to a prompt, to prompt a model, like, in some sense, what you're doing there is trying to exploit structure to get around the safeguards built in.

So understanding how structure affects output. you know, in a kind of a research fashion helps make output more resilient against creative inputs. So that's, and by the way, there's a lot of important academic work that that's that's, that's becoming more and more relevant for industrial application there.

Jonathan: Yeah. I am, I am. Endlessly humored Isaac Asimov predicted the idea of skill in writing prompts. I'm not sure if you're familiar with any of his works, but there was a couple of books in particular where, you know, the, the, the robots in those books had positronic brains, which were effectively general AI.

And one of, you know, they had the, the laws of robotics built into them, and one of the topics of a couple of those books Big plot points is there were people that were particularly skilled at writing prompts, you know, speaking instructions that would abuse those laws of robotics and get the robots to do things that you wouldn't normally do.

And I remember as a kid reading that and going, Oh, this is just ridiculous. Of course, there's never going to be any skill in writing prompts. What is he talking about? And it turns out Asimov had it figured out and he was right. We, we really are sort of living in sci fi.

Anthony: I, for sure. I think it's a reflection of how humans interact too, right?

I mean, with the right prompts, so to speak one human can convince another to do some things that they might not, right? They probably shouldn't.

Jonathan: Yeah. Yeah, that's true.

Anthony: Yeah, it's interesting.

Jonathan: All right. So are you guys in the weeds enough to be thinking about like the open source nature of the libraries that are used to build these things?

So, and what I'm thinking of here really is CUDA NVIDIA's CUDA. So much of this is built on CUDA and that is a, that's a closed source library. And, and really it's kind of problematic for people trying to do these things on their own. On their own computers, if they want everything to be open source.

Is, is this sort of in scope? Are you guys working with, you know, like, AMD and the, the Vulkan specification and all those things where people are trying to sort of liberate the underlying libraries?

Anthony: Yeah, that's actually one of the six focus areas of the Alliance. is enabling hardware choice in AI, and hardware choice, right, is really about having an open software ecosystem that that can enable that, right?

So not just NVIDIA execution, but execution on other GPUs and more novel architectures. I mean, there's a huge flourishing of AI specific accelerators out there. So to take advantage of that, yes we need better all open Software libraries to enable it. So one of the six focus areas of the alliance is to do that.

We are working, we have a lot of the big players, you know, AMD and Intel and others, Meta with PyTorch, you know, some of the emerging important libraries like VLLM and Triton, like these are not necessarily alliance projects, but they're important points of input and collaboration. So, yes, there's a lot to do here.

There are some interesting recent results. There's actually a really interesting recent result that was really driven by by PyTorch and the PyTorch Foundation which demonstrates an all a deployment and execution on both NVIDIA and AMD hardware. With with an all open library set, right?

Not not utilizing CUDA which is a nice piece of progress that was pretty good efficiency and all that. But there's a lot of work to do there. But yes, that's, that's a priority of the alliance.

Jonathan: Yeah. All right. I know this is, this is probably not exactly an alliance question or even an an IBM question, but you've demonstrated that you are an expert in these things.

So I'm going to pick your brain about it. How close are we or will we ever see general AI? Okay.

Anthony: Will we ever see ? And do you have probably we will see three numbers while you're

Jonathan: What? Say again?

Anthony: The, sorry. When asked a a prob, you know, when asked a question that's unbounded in time, the answer must be yes.

Right? Eventually unless we all. Humanity perishes. Look, so I think,

Jonathan: I just, I just, blinds, I just whacked you with that one.

Anthony: So I think AI systems will get better and better. I think the present architecture, right, transformer based large language models you know, will not get us all the way to what most people consider, you know, artificial general intelligence.

They simply are too new. They're just not understanding enough of anything in, in the world, right, in terms of being able to, to learn from the, the huge diversity of inputs that humans are, for example, right? I mean, all they are, they're, they're really great, but they're statistical, you know, They're math engines, right?

So they recognize patterns from huge amounts of data and they, they spit out patterns responsive to prompts that, that seem to align with what other, you know, what, what date, what the huge data suggests might be useful. You know, it's deeply reliant on data. And it's it's relatively simple minded approach.

So yeah, we have more steam there. There's a lot more that these approaches can do, but I don't think it's going to take us all the way to artificial general intelligence. I also don't think we're going to get to a state of artificial general intelligence. Very soon nor do I think it's a very useful term because it's hard to even like once you get Once you get to a point where like an AI system can you know be as good of a human as a human in some Some domain sense like what does it mean to go beyond that right?

And how do you go beyond that if you're still training on data that come from basically humans? So a lot of, a lot of challenges. I think kind of, I was a little nebulous and meandering. So I think I'd sum it up by saying, there's a lot more progress in the present kind of, you know, generation of AI. But it won't get us to AGI.

And it's still, it's becoming unclear as we get, you know, closer, but not too close to that, that goal. What that AGI is like a, not a particularly precise term.

Jonathan: You almost get into some deep philosophy and almost, almost like a spiritual sort of argument there and talking about some of this you're, you've been such a good sport to ask, to answer our more general questions.

I appreciate it. I've got, I've got one more that. I'm gonna, I'm gonna hit you with there are, there are people out there that are talking about the the, the potential danger of letting an AI get too smart. Is that, is that on your radar? Is that something that people should have in the back of their mind?

Anthony: So too smart. Here's, here's what I would say. I mean, yes, yes, we're very concerned that the output of AI models and AI systems, right, which are models embedded in a broader context, right? Connected to APIs and the Internet and so on and so forth. We're very concerned that these systems are engineered to be Safe and trustworthy that it does not produce and can't produce unwanted outputs.

You can't produce malicious code. You can't, you know, jailbreak it to produce malicious images and things like that. We're very focused. There's a lot of work in the AI Alliance and in many companies and universities on that topic. I'd say, you know, are we worried that AI is going to take over the world?

It's going to create an army of robots. It's going to let anybody build a nuclear bomb in their basement. No, not really, because Okay. So now I'll, maybe this is closing. My background is not really an AI scientist until recently. It's actually not really an AI scientist. I'm a physicist.

Jonathan: Oh, interesting.

Anthony: My background is, is rooted in the physical world, right? So I've spent a lot of the earlier part of my career building things, experimental apparatus, things that have to work physically. And when you do that, you learn that like, wow, a recipe to follow is like the very beginning. And really not the main bottleneck.

So if we're thinking about building armies of robots or malicious weapons, like just having a really good recipe to do that is like a very small part of the challenge. And for that reason, I'm not, and I think a lot of people aren't so worried about these grand existential threats. We're a lot more worried about these practical kind of digital world threats, you know, malicious code and deep fakes and things like that.

And so that's where most of the energy is targeted.

Jonathan: Yeah, so I'm, I'm reminded a couple of weeks ago there was some research done ChatGPT has a, a run locally feature. And someone discovered that it will run, it will run locally, but it will also be able to access the internet. And someone discovered that they could poison its long term storage just by showing it an image.

And after they showed it the image, it would then access a controlled URL for every prompt. And so it was a way to leak people's private information out to the internet through through this, this local copy of ChatGPT, and, you know, on one hand, that's a brilliant piece of work. On the other hand, it's kind of terrifying that you can get malware now in your LLM model.

Anthony: Yeah, that's it's concerning, but hey, we know about it. Someone out in the, out in the open tried it and did it.

Jonathan: Yeah.

Anthony: And you know, now we know, so we need to figure out how to be resilient against it.

Dan: Also, chat GPT is proprietary as well. So being that we're an open source show and we're here to talk about the open source thingy, I was just going to say on Anansi's behalf, yeah, that we, we should be let me get the point out.

Yeah, that developing this stuff in an open way is much more. Effective and better for everyone. I would say. Yeah, absolutely.

Anthony: Yeah, absolutely. I mean, it's what, what OpenAI is doing there is taking on the burden to make sure no one can do that themselves. Right. Without harnessing community to help, you know, figure out how to make engineer things more resilient against.

Jonathan: Yeah. I will, I will channel Doc Searles for just, just a minute because I know something that he has been looking for for the longest time is something sort of in this vein of I want my own personal AI. I want to be able to, you know, scan all of my receipts and then ask my AI, my personal AI, hey, what's my personal AI?

What did I buy three months ago that was AI go, Oh, it was this. Here's the picture of the receipt. I, I must assume that there are, there are people inside the AI Alliance that are working towards that particular vision or something similar to it.

Anthony: Yeah, there are there's a whole bunch of efforts that are essentially.

Trying to make it much easier to adapt and tune models to specific context, and the limit would be a personal context, right? That would be, that would be ideal, and a lot of people would like to do that. And there are methods coming along that are developing pretty rapidly to allow you know, kind of fast and efficient alignment or tuning of the model to understand specific personal preferences.

and understand or interface with specific, you know, personal documentation history, right? Techniques like raft you know, autumn instruction tuning based on synthetic data generated from a you know, a question answer pair that, that is seated by an individual with preferences of how they want the model to behave and things like that.

So it's, it's not like a problem, but yeah, that's on it. A lot of people's minds. And I think you're going to see a lot of advancement in that namely the ability for an individual, you know, to tune and create an AI system. That is responsive to, you know, their data, their preferences, their goals.

Jonathan: Yeah.

Okay. We are getting, we are getting to the end of the show. I, I, again, I appreciate you putting up with our sort of wandering, meandering series of questions here. I think it's been, it's been great. Is there anything that you absolutely wanted to cover that we didn't ask about that we didn't get to?

Anthony: I think that last point we started to, to discuss, like we talked about open AI, we talked about, you know, open source and the open community.

I think a lot of the topics we've discussed here, well, I know that's kind of guiding my, my whole role in life right now is that you know, open communities are much better at identifying and solving challenges and, you know, advancing innovation to create capabilities, whether it be personal, functional, and you know you know, personalization of a I or whatnot.

And you know, the flip side of that right is the risk of being more open, having open models and open data sets at scale and so on. Yeah, there's always risk. But I think like a lesson of open sources that the benefits having open innovation and a lot of people. You know, with eyes and generating code against challenges and opportunities is like way outweighs the risks.

And if we can just get past the sense that AI is going to like, you know, you know, end the world, which it won't, it definitely won't that we can actually start to harness more people solving problems to make it better. So open source, open innovation, open communities. That's, that's the way to go.

Jonathan: That's good stuff. Do you think we're in an AI bubble? This is something that I sort of think of on the business end of things. Everybody wants to put AI in stuff and I'm sort of looking forward to here in the next months or years. I predict, I have a feeling that the bubble is going to burst and AI is going to get less popular.

And, I sort of have the feeling that that's when it's really going to start being useful as a tool. When people stop trying to use it for everything. And then it just becomes another tool. It's sort of like the internet. After the dot com bubble burst, the internet did not go away. In fact, it's just, it's continued to grow and grow.

You know, we use it a little bit more reasonably. I don't know if that's true. Yeah,

Anthony: I think it's hard to argue. There's no inflation in AI right now. But what I'll point to one, one trend that I think shows that even if there's a little bit of a bursting, I don't think it's going to go into a winter. And I'd say that I'll go back to what IBM is very focused on, right, which is enterprise adoption of AI.

We've seen, and this is corroborated in a number of places, Oh, my earbud fell out again. Can you still hear me? Yeah, you're right.

Jonathan: You only

Anthony: need one of them. I'll operate with just one for now. Yeah, what we've seen is that enterprises have been very fast adopters and deployers of AI, not for every use case and not at, you know, full scale in many cases.

But unlike you know, earlier technology and AI, you know, kind of revolutions, call them they're embracing, building, deploying AI like pretty rapidly, right? Some, the various statistics, but you know, something like half or more of, of Fortune 500 companies have, you know, generative AI in production.

Now there's a lot more to do, but we've seen like a really strong uptake and when businesses are ready to do that, like actually very conservative risk averse businesses. That's to us a pretty big sign that, you know, this is an enduring piece of progress here, not just a bubble.

Jonathan: Excellent. I've got two questions that I'm, I'm required to ask you before we let you go.

I will get emails about it if we don't. And that is, what's, what's your favorite text editor in scripting language?

Ha!

And this sort of assumes, this sort of assumes that you've done enough programming to have answers to these questions. That's not always true. I mean,

Anthony: look, I look you know, I'm a scientist.

I'll go back to my scientific roots, right? So I'm ultimately a scientist, a physicist, actually. So, you know, I, I, in the, so I other than the kind of like, you know computational domain specific stuff. You know, I'm I like python and I use python for most things. And yeah, I guess just

Jonathan: do you have a preferred text editor?

Anthony: Prefer? Oh, yeah. I mean, just, I guess, VS code. I mean, I just, you know, not, I'm not very exotic in order of my, all that opinionated here, I probably should be more so. But yeah, I, I'm sure you have much more opinionated guests about these things.

Jonathan: Sometimes. I would tell you it's amazing how many people that are out there doing the real work are either A, not very opinionated, I don't care whatever's on the computer, or B, lots and lots of people are starting to say VS Code.

It's becoming very popular.

Anthony: Okay. Alright. I can see that.

Jonathan: Yeah. Thank you so much for being here. I appreciate it. And Again, putting up with our sort of meandering series of questions that, but, but excellent. Excellent. Really good. I was glad to have you here.

Anthony: This was great, guys. I really, I had a lot of fun.

I'm glad you asked all those questions. They're great to answer. It's, you know, it's been great talking.

Jonathan: Yeah. Appreciate it. All right. Dan, what do you, what do you think?

Dan: I thought it was a great conversation. And yeah, like you just said it was really good of, of Anthony to put up with some of our more sci fi sci fi kind of future gazing questions.

Yeah. Luckily I didn't get a chance to to get into physics with him cause he, and we found out he's a physicist. And I was like, Oh, I want to ask about string theory and stuff like that. But he, which would be way out of the scope of the show. But yeah, I thought it was great, a great discussion.

Really, really really great guests. Very interesting. And on the whole thing about AI bubbles, I think he makes a really good point there about the amount of stuff, the amount of investment and infrastructure and all that sort of stuff that, that a lot of companies and traditionally quite conservative companies are putting into this shows that it's going to be around.

It's not, it's going to be in our futures. I think.

Jonathan: Yeah. Specifically the AI bubble thing. I think, I think actually the dot com bubble is probably a good a good parallel. Right? Because everybody went nuts over dot com. And then the, the, the bubble burst. But boy, the internet sure didn't go away. People just got a little bit more retrospective about it.

Circumspect about it. They didn't want to spend quite as much money on it, but the internet now is, is enormously bigger than it was in the nineties during the dot com bubble. And I, I imagine we're going to have the same thing with, with Artificial intelligence and LLMs. We, we've talked before the show.

We looked at the, the, the kind of the presser that was sent to us before doing this. And we went, boy, I hope it's not just a marketing person. And Oh my goodness. It was not just a marketing person as far away from that as possible. And I I'm tickled pink. The, the hour just flew by. And if Anthony is up for it here in a few months, we'll have him back.

Maybe talk about after the. Maybe after the OSI releases their definition of open source and talk about that and get more into that side of things. And we'll, we'll, we'll stop geeking out over AI itself. But it was great. That was great. One of the best shows we've had for a long time. Dan, you have anything you want to plug?

Dan: Not specifically given that last week's guest was I'm going to be going this weekend to our camp in Manchester. If by any chance people didn't hear last week's show, please go and listen to it cause it was excellent. And and if you're in the UK. Come along, come to, and you can make it to Manchester.

Please do come and join us in Manchester next weekend. This, this coming weekend, the 12th and 13th of October. You can find out more at ogcamp. org, which is O G G C A M P dot O R G. Or actually Simon was saying, wasn't he? They've registered og. camp. I think so. Yeah, so maybe just try og. camper. I may even be out of date with my URLs.

You can do that and you can find me and my Master Don posts and all those sorts of things on danlynch. org It's all embedded there.

Jonathan: Yeah, very good. I the one of the groups that I'm a part of They have a UK Greater Manchester, like it's a, it's a thread inside of a topic inside of Discord. And so I went there and I'm like, somebody needs to go and, and pitch, you know, give, give a talk about this thing at Oddcamp.

And one person was like, I've got COVID, I probably can't go. And I was like, I'm a way that, so I'm trying, I'm trying to scare somebody up to go and give a talk. We'll see if that actually happens. All right, Dan, we appreciate you being here. As far as my stuff, I want to say, first off, we appreciate Hackaday being the home of Floss Weekly.

And you can find my security column goes live there every Friday. Make sure to check that out. We've got the Untitled Linux Show still over at Twit, and that records on Saturday afternoon and then goes live Oh, sometime in the next day or two after that, and so make sure to tune in there as well. We appreciate everybody being here, those that caught us live and those that get us on the download, and we will see you next week on Floss Weekly.

Episode 803 transcript

2 oktober 2024 | min

FLOSS-803

Jonathan: Hey folks, this week Simon joins me and we talk with Gary Williams about AugCamp. That's the un conference all about free software, free culture, and a bunch of mates getting together to have a good time. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 803. Recorded Tuesday, October 1st.

Unconferencing with AugCamp. It's time for Floss Weekly, the show about free, libre, and open source software. I'm your host, Jonathan Bennett. And this week we have, uh, well, it's not software this week. It's more about free culture and open source culture. We're talking about AugCamp. And, uh, of course, it is not just me, we've got the, the great, the amazing, Mr.

Simon Phipps! Hey, Simon! Oh, it's me, it's me! It's you! I'm back again. Welcome back! Always a pleasure! So, uh, Ogcamp, you, uh, you, you know, you know something about this, like you're, you're sort of, uh, one of the insiders. In fact, you're kind of doing double duty as a co host and a guest today, aren't you?

Simon: Yeah, so the story here is that, uh, I have a, um, a non profit that I host in the UK that looks after small community activities.

Uh, so one of the other things it does is it collect, it crowdfund the, um, maintenance of the ODF specification. Uh, and a, a lawyer friend said, Hey, this group of people needs a place to host their conference. Uh, all the money and the trademarks and the domain names, would you do it? So I agreed years ago that I would host, uh, the, uh, fiduciary oversight of this thing called OG Camp.

And I also go along. And act as treasurer at the actual event. So I take the cash box and I buy everyone's lunch and everything. Uh, but I don't do any of the other organization and none of the hard work is actually anything to do with me. Uh, and it's a different organizer each year. Um, but we're going to find out more about that now.

So I've been involved in it for years and I, I'm quite, um, pivotal to it happening, but I'm not actually the person that does the work. I

Jonathan: see. And so the person that does the work this year, is that Gary Williams?

Simon: That's Gary Williams.

Jonathan: All right, well, let's go ahead and bring him on. A noble volunteer. Yeah, we've got him waiting in the wings.

And, uh, Gary, welcome to the show.

Gary: Hey, Jonathan. Hey, Simon. Great to be here.

Jonathan: So, what, our, our Our outline for the show is basically like the the question words, right? Like what, why, where, who, how Is Oddcamp? Because we're talking about Oddcamp. So let's start with what And just give us kind of the the overview.

What what is this thing and why should people care about it?

Gary: Yeah, so Oddcamp is uh, we well we self describe as a free culture free open source software Uh kind of hardware hacking, uh Uh, conference in the UK actually in Unconference more accurately. Uh, so we've been running for 15 years now, although this is only our 11th year actually running the conference because of the event that happened over the last few years.

Um, and really it's, you know, it's a place to be if you are at all interested in hardware hacking, open culture, foss, um, and yeah, I guess things of that nature.

Jonathan: Is that where the name comes from? OGG? Are you guys really into doing audio compression?

Gary: So yeah, the name is a bit of baggage with it. We were talking a little bit about this before the show.

So OggCamp actually originally came out of a couple of Linux podcasts and came out of the Linux podcasting community and Oggvorbis and Oggtheora were the hot thing at that time, 15 years ago. So they chose the name Ogg and clung onto it and, uh, It's the, it's kind of the name that stuck, so it's Ogg in the sense of, I guess, uh, open formats and open standards, uh, but actually there's a lot, a lot more wider things discussed at OggCamp.

It's completely unrelated to Terry Pratchett.

Jonathan: One of the podcasts, was it the Linux Outlaws? I remember Dan was involved with this for a while.

Gary: Yeah, it was the Linux Outlaws and I think the Ubuntu UK podcast at the time. Ah,

Jonathan: yeah, cool. Okay, now you said you guys have taken a break. What, what's up with that?

What in the world would cause that? Why? Why?

Gary: Yeah, so we last ran in 2019 and then, uh, in 2020, this, uh, this large virus came and stopped us from running the event, uh, in 2020. Um, 2021, similar story, um, and because we're completely volunteer organized, there were a couple of years there where it just took some time for someone to pick up the slack.

Um, I was supposed to run OccCamp last year, uh, but, uh, we had a baby and that kind of got in the way of my time to organize things. So, uh I understand. Yeah.

Simon: And we did have a great team that was going to run OGCAMP the year before that, and they, they got very nervous about the return of COVID. And so they decided that they couldn't run it.

And they, they eventually stepped back down. And so we had two years where that crew was going to run it and found that the environment wasn't right for it. And then Gary last year, and now here we are with running the event for the first time in five years in Manchester in the United Kingdom.

Jonathan: So you, you have jumped to one of the next questions.

It's in, it's in Manchester. Um, have you ever thought about moving it outside

Gary: of the UK? I think there have been various people who have asked about running one outside of the UK. Um, I guess the difficulty we've always had is Oddcamp's home has been spiritually the north of England. So we've done Manchester, we've done Liverpool, we've done various other cities across the north of England.

We've ventured south a couple of times, there's been one in Oxford and one in Canterbury, but by and large it's been the north of England. Um, there were a crew looking to run one in Edinburgh, so I guess that's technically outside of England. Um, And there were a crew who were partially interested in running one in Dublin at one point.

Um, but, but that never quite got off the ground to say. England has been the kind of spiritual home of Oddcamp, um, but I guess if someone wanted to organise an Oddcamp outside of the UK, I can't see why we couldn't.

Jonathan: Yeah, you could have like, uh, Ogcamp West and Ogcamp East and have them, you know, in Berlin and somewhere here in the U.

S. It'd be fun.

Simon: It's, uh, it's got quite an eclectic draw, actually. I've been watching the ticket sales. Yeah. And there's been people in Germany and France and Belgium. all buying tickets and coming over. So although it's located in the north of England this time, um, there's, there's quite a draw from across Europe, which really quite surprised me when I realized it was happening.

Jonathan: Do you have a decent little handful coming from the States or are we Yankees not, uh, not in your target audience?

Gary: I don't think we've had any make it this year from the States. I think we have actually had a couple from Japan and places like that, but no one from the States quite yet. Yeah, maybe you should be the first Jonathan and, uh, go and get a ticket.

Although two weeks is maybe a bit short. There are

Simon: still tickets actually. You can still come. Yeah. It's very surprising where you can fly to Manchester from across the U S there are direct flights from many U S cities.

Jonathan: Yeah. Yeah, that's true. Um, okay. So. Let's just say I decide I'm going to come to Oggcamp.

What, uh, what should I expect? What's the experience like? What, uh, give me, give me a quick survival guide for a first timer.

Gary: So I think probably the, the easiest thing to say about Oggcamp is that every Oggcamp is entirely different. Say. Um, we have a set of scheduled talks, uh, just to make sure that we've got a bunch of content.

So this year we've got two rooms worth of scheduled talks, and they're from a variety of people. Uh, we've got workshops from people, uh, like Utah. One of our sponsors this year who are a, our Camp Utah, one of our sponsors this year who are a workers union here at the uk. Um. But I guess OPCAM's real selling point is that we're an on conference.

Um, so in the UK and perhaps in other places, BarCAMPs are quite a popular, uh, method of running a conference. And, uh, What that means is effectively if you've got something interesting that you think that OpCamp attendees would want to hear about, write the talk, turn up with it, or indeed don't write the talk, find out what people are interested in on the Saturday, and then what we quite often see is people with their laptop furiously writing their talk on the Saturday night to give on the Sunday.

So it's, you know, it's a real variety of topics that, that you end up hearing about our camp. It's anything from, uh, you know, securing your website with a WAF to mental health to you, what's the latest on what Collabora are doing or, you know, whatever else. Um, that there's a whole plethora of topics. Um, so yeah, no two old camps are ever the same.

Jonathan: Yeah. Uh, you said there's some workshops too where people are actually hands-on with hardware.

Gary: Yeah, so we haven't got any official hardware workshops this year, but we have in the past had people, you know, turning up with the Raspberry Pis and soldering irons and putting together ESP32 microcontrollers and things.

I think there was an event a few years ago, actually, where we had to ask someone with a soldering iron to please go outside of the conference center because your second smoker lights off. So they ended up soldering at the car park, which was pretty cool to see. Um, it's great, but we, uh, we encourage you if you've got a cool hardware project, uh, show off, absolutely bring it along.

Um, and, you know, being in the UK, we've, we've always had quite a lot of the Raspberry Pi community and things there as well. So it's, uh, it's always cool to see.

Jonathan: Yeah, oh that sounds like fun. I have had an experience earlier in life Dragging the sound system a portable sound system outside It was a church in that case But dragging it outside to the front steps to be able to solder something to fix it and not Smoke the inside of the building up.

So I I know how that goes

Gary: Yeah, yeah, it's uh, you get some funny looks from a hotel or a venue soldering gear in the middle of one of their conference rooms, that's for sure

Jonathan: Uh, yeah, I feel like they should know what they're getting themselves into though hosting a conference full of geeks and hackers

Gary: Yeah, well our venue this year have graciously welcomed us back, so, uh, clearly we didn't do anything too wrong last time we were there.

Jonathan: Yeah. Um, okay, what are some of the, uh, I guess what are some of the talks that you are excited about that are coming this year?

Gary: So, on the schedule track this year, there's a few things that I'm quite excited about, actually. Um, we've got Stuart Langridge talking about, uh, open web advocacy. I know that's something that he's been really, you know, fighting for.

Um, so just, you know, getting the open web available across a variety of platforms. I know he's been quite pivotal in getting things like, uh, other browser engines available on iOS, which is, you know, something that we've all struggled with for a long time. Um, We've got people giving us talks on audio podcasting which are going to be really cool We've got hacker public radio actually talking us through what it takes to get your certificate in order to do In order to do kind of amateur radio stuff in the UK and what are all the things you need to consider for that?

And then we've got, you know, even some of our sponsors this year, like Utah are doing a workshop on how to organize a union in your workplace, which is something that we know tech in the Europe is particularly under, you know, and unionized say they're giving a workshop on why should you join a union and what are the kind of things that you need to think about for that?

And then of course we've got our classic Oddcamp panels this year, which is kind of a look back over the last five years and all the things that have changed in FreeCulture since the last Oddcamp and what's been good, what's been bad, how do we make the next five years of FOSS and FreeCulture even better, and how can you play your part.

So a bunch of really cool talks going on, um, I named a few there, but there are a bunch more that, uh, that we've got available and they're all at oddcamp. org slash schedule. And of course. The great thing is that half of our talks aren't even announced yet, because the attendees make 50 percent of the agenda at OpCamp, so it might be that someone turns up with something really cool that we just haven't thought of yet.

Jonathan: Yeah, that's always, that's fun. Um, okay, Simon is whispering in my ear here that there is some offbeat stuff that he wants to talk about.

Simon: Yes. So, um, we're, we're going to, uh, one of the things that's always happened to odd camp has been, uh, a raffle. Um, uh, and there, uh, we've had sponsors supplying toys. Um, actually the organizers have a small budget for, for buying toys as well.

And it's very likely that there will be a, uh, a raffle for all the attendees this year as well. I suspect Gary is going to go out and buy some stuff. Uh, I've got a big budget. box of toys next door. I've got things like a robotic car that's powered by a raspberry pie. I have a lifetime supply of LEDs. Um, I have, you know, those sorts of things.

And then we've been talking about having a swaps table this year, like you'd get at a, at a local meetup where you, you bring something and take something. Uh, and we're not quite sure how that's going to work. But we are probably going to tell attendees to, to bring good quality things that they would love to have been given, uh, that they can bring along to the SPOPs table.

Uh, and then one of the things about OGCAMP is it's, it's actually a very safe environment for people who, um, Find large public events, uh, challenging. Uh, there's a lot of very caring people who take care of the folks who are around. Mm-Hmm. . And so a lot of what makes on campus special actually happens outta the meeting rooms in the, in the, in the corridor spaces, the hallway track, the open spaces.

Uh, it's a hallway track, but it's a, but it's a, it's not just a bumping into friends hallway track. It's also a taking care of, of people who are there for the first time. Type hallway track too.

Jonathan: Yeah, interesting. Um, let's see. What was I going to, what was I going to ask about? I had something that came to mind.

Um, so like what are, what are some of the, it sounds like it's a, it's an intersection of a whole bunch of different things. So you've got obviously people that care a lot about open source. Do you have some other niches there? Like, is there, is there a retro hardware enthusiast sort of group that comes?

Do you have security people that come like what, what, what do these different groups kind of look like? What are the, what are the different interests that get represented?

Gary: It's a good question. Um, I'm trying to think back five years.

Simon: I mean, it depends on who comes, right? You know that. So there's a quite interesting profile.

Uh, I know that one of our organizers has recently got into, um, creating art with, uh, with plotters and, um, with pen plotters. And there is certainly going to be a little section that is interested in that this year. Um, I, I, we've previously had, uh, people who are into security having, uh, having questions about that.

There's typically quite a big Ubuntu community. Um, faction in the room, uh, trying to be friendly, welcoming Ubuntu people. And of course that's evolved over the 15 years, uh, into something completely different now. Uh, so I, uh, but it really, it's going to depend on who shows up. We've got a couple of hundred people showing up.

Um, some of them, the names I recognize lots of people I don't, and it's the odd camp is the product of the people who come, um, because we're, we're We're cowards. We, we do plan a planned track just in case, you know, no one comes with any talks, but that's never happened. And so it's going to be the product of, of, of who comes and that makes it quite different to most of the other conferences that I attend, where the conferences, you know, I can tell you in advance what's going to be happening at all things open, and I can tell you, uh, you know, in advance what's going to be happening at FOSDEM, uh, to a certain degree.

Uh, whereas OGCAMP. I can tell you some things that are going to be happening. There's going to be a raffle. Um, there's going to be two, two rooms full of planned talks. But apart from that, who knows, you know, that's part of the charm of it.

Jonathan: Is that really what's meant by an unconference that things tend to happen ad hoc?

Gary: Pretty much, yeah. So it's, we call it an un conference because there almost isn't a conference, right, until people turn up and make one. So, um, like I said, we have two rooms or two rooms of our schedule track that we, yeah, we come up with. But actually it's, it's the people who turn up and present their interests to like minded people that really make it an un conference, I think, for me.

Um, Yeah, it's, it's a very, yeah, it sort of evolves over the two days. You find that you get a well attended talk about, uh, TLS from some of the security crowd and you'll find someone else go, Oh, there's a bunch of TLS stuff that they didn't cover in their talk. So I'm going to do another one in TLS tomorrow.

Um, and yeah, you sometimes find that. Rooms get full and people are just giving ad hoc talks in hallways or in a hotel foyer and they've got 10 people grounded around them or in our kind of event space, someone turns up with a cool toy and I think one year someone turned up with a robot conferencing device that could walk around and be controlled and, uh, yeah, it was just walking around and people are having conversations with it and it was great.

So it's, it's the unexpected, like you say, the thing that really makes it an unconference for me.

Jonathan: Yeah, as you as you think about the things you have scheduled like on the the official tracks this year Are there any particular talks that you hope people bring?

I know that's kind of an interesting question But is there anything that you don't have covered that you really hope people will cover?

Simon: You know, I'm hoping we've got some folk there who are very into activity pub Uh, and I'm hoping we're going to get some, some good, uh, horizontal thinking about ActivityPub showing up spontaneously.

Uh, you know, a shout out to Andy Piper, who I hope will be the catalyst that causes that to happen. Uh, but, uh, you know, I'm hoping there'll be some, some really good things about that, because I think that's, that's close to the center of where the biggest hope for open source is at the moment, is getting away from over centralization.

Mm hmm. Uh, moving into Federation. And having devices and software which is simple enough for everyone to federate instead of becoming the, uh, the, the data slave of meta. Uh, and so I, that I really hope is going to show up, that we're going to see, talk about federation, talk about ActivityPub, talk about, um, uh, cutting the cord to the, um, the ad industrial complex.

Yeah. Yeah,

Jonathan: I get that.

Gary: I think for me, I'd love to see some talks about self hosting coming up. I think there's a bunch of really cool open source projects that people can self host that are genuine competitors to, you know, the proprietary. Google Photos or Google Drive or Office 365's of this world that have come along in the last five years and, you know, really running with, with what it is that, um, those products had, um, but you know, you can, you can run them on a server in your garage, right?

And I think that's really cool. So, you know, people coming along and talking about those projects and talking about their journey from proprietary platform through to, I run everything on a server myself that's in my garage is, is a really cool thing. I'd like to see some of this year too.

Jonathan: Yeah, um, I don't suppose you have anybody coming that you know of to talk about Meshtastic?

This is a project that I've been spending my time on here recently.

Gary: I haven't seen anybody so far.

Jonathan: I may have to go and uh, drop a link to this in one of the discords and see if anybody there wants to go and give a give a talk on it because that would be fun. Uh, it's, it's lore, lore radios that talk to each other but they, they mesh and they're all decentralized.

I think it would be a great fit.

Gary: Yeah, yeah, it sounds like something that people would be interested in.

Simon: And I'll be bringing one of my raspberry pies along and giving a talk about running, you know, Because that because I host most of the things I care about Uh on a rack of raspberry pies just downstairs from where i'm sitting And it's very straightforward to do Um, and there's no reason why most people couldn't do it to be honest.

Jonathan: Yeah. Yeah, that's true uh, simon, you said you are the sort of the the treasurer and the um, I guess sort of the legal host You of Oggcamp? I'm curious. I'd like to hear more about that. Um, why, why such a thing is needed and what all you, uh, what all you do.

Simon: Yes. Well, uh, so that, that arose, uh, a number of years ago when, um, a, uh, a gentleman I work with who happens to be a lawyer said that there was a community of people running a conference and they needed a fiduciary host.

And I, I run a, uh, a small not for profit organization called Public Software in the UK. And, um, the reason you need a fiduciary host. is because, uh, I think like a conference, there's actually quite a lot of money floating around, uh, all camp is going to cost us probably a five figure sum of money to put on.

And, uh, and it, it breaks even. So the, at the end of the process, there's probably going to be a five figure sum of money sitting ready for next year. And somebody has got it. Somebody's got to keep it, and you've got to trust someone to own it. And, um, this particular conference is one that gets run by whoever is available next year.

And so the people who are responsible for it in 2024, possibly none of them will be involved in 2025. So, If that's the case, who owns the domain names? Um, who has got the money in their bank? Uh, who is it that has got the legal standing to make contracts with the hotel? And so that's what Public Software does, the organization that I run.

Uh, and public software, um, provides a home for unincorporated associations of, of people who are doing things for open source and free culture. So, um, we host, uh, OGCAMP and we have one other project called COSM, which is a crowdfunding pot for maintaining the, uh, open document format that you find in, in LibreOffice and now in Microsoft Office.

Uh, and so, uh, I, I do that. I, um, Uh, run the incorporated association that looks after the assets of odd camp so that Gary is able to show up and there is money for him to spend on the conference this year. And there is somebody who's willing to sign the contract with the pendulum hotel. And there is somebody who is willing to pay the, the, uh, the crew radio supplier.

And there is somebody who can pick up lunch for the volunteers. And then at the end of the show. Um, public software will, will draw everything together and get ready for next year and make it available to whoever shows up to run OGCAMP 2025. So fiduciary hosting, it's, it's kind of like what, um, software in the public interest or Software Freedom Conservancy do for open source projects.

Uh, just for things, I do it for things that aren't software, for, for a small conference, for a standards project. Publix offers hosted other small activities over its lifetime as well.

Jonathan: Yeah, I guess that's something I didn't think about, but if you have that much money running or moving around, like if, if nothing else, you just need to be able to have a paper trail of nobody walked away with a thousand dollars in their pocket that shouldn't have.

Simon: Yeah, so we do that. And also, um, you know, there is value added tax, sales tax to be collected and paid. Yeah, that's true too. So we do that. And, uh, uh, you know, that we've, we have credit card machines, uh, you know, ordinary individuals typically don't have. Credit card facilities and we have credit card machines so we can host the Stripe account and we can, we've got, we have a Square account that we make available.

So it's all of these little details that make it possible to put on an event. Modern society has, uh, as in so many other areas, has, uh, has, uh, officialized or bureaucratized the things that previously would have just been, uh, the way that mates did things together. So, if we'd been running a village fate 40 years ago, we probably wouldn't have needed public software.

Right. But in 2024, you absolutely need a fiduciary host to look after these things for you. Because otherwise, Gary is going to find himself with HMRC wanting to know where the VAT is. He's going to find himself having to pay corporation tax on the turnover personally. And he's going to find it's added to his income tax return.

And he doesn't want any of those things to happen. Right. So, so we make sure that doesn't happen too.

Gary: Yeah, Simon makes it so I can just, uh, contact the hotel, get the invoice sent, and, uh, forget about it and carry on with my day, which is, uh, which is always great. He takes the bureaucracy out of what otherwise would be quite a painful thing come tax return season.

Yes,

Jonathan: yes, understood. Uh, how many, how many tickets do you guys, uh, well, let's see. How many tickets total? What's your, what's your max capacity for the event?

Gary: Absolute maximum for the event in the venue that we're in this year is 250. We hope to get somewhere close to filling that. Um, and we're not, we're not too far off actually.

Um, Where are you at?

Jonathan: How many have you sold? How many are left? I

Gary: think we're just under 200 at the moment. We're not too far off. So,

Jonathan: so if nobody else signs up, you guys still, you're on, you've got a good conference, even with just under 200 people, that's a, that's still a good group of people.

Gary: Hey, we've, we've definitely got enough people to, to put on a good show, I think.

Um, and yeah, plenty of people to come and come and fill slots for talks as well. So

Simon: in terms of cashflow, looks like we've collected enough money to refresh the pot for next year. So, um, I think this, this event is going to break even and that's what we want to do. We don't want to make a profit. We want to, we, we do want to break even.

And we want there to be about the same amount of money in the pot next February as there was in the pot in February when Gary took it.

Jonathan: Uh, what, what is the cost for it as far as per ticket?

Gary: So what we've always done with Old Camp is trying to keep it very accessible to people. So This year, um, I think it's, it's no secret to anyone that the bottom has fallen out of the ad market and, uh, sponsor money is much more difficult to get hold of.

Uh, so this year we've said, yeah, we're going to honor that pay what you can model. Um, but we've got a suggested ticket price of 40 pounds. So, For your 40 you get, you know, two days worth of, you know, really good solid talks on the schedule track, plus whatever, whatever else turns up, plus the hallway track.

Um, but if you can't afford that, then absolutely you can pay us anything from 1 upwards. The only ask there is that you cover the cost of our ticketing platform. Because they charge us a small amount to issue tickets and store people's records and things like that.

Simon: Then there's the opposite end of the thing where we have a bunch of people who say, you know, we really want to support you.

And rather than taking donations, we put on the system some high price tickets. So we've had a whole load of people who have bought what we call professional tickets, which are the 100 tickets that for people who can expense it. And so we've had, uh, quite a few people have bought professional tickets.

They don't have to, they could have just bought the one pound tickets if they wanted to. And it's quite comforting and reassuring that if you look across all of those tickets we've sold, uh, there's a pretty even spread of people at pretty much every price point between, I think the most anyone has paid is 200 pounds and the least anyone has paid is five pounds at the moment.

And there's, we have a pretty good spread across the whole, Yeah,

Jonathan: I'm, I'm actually really glad to hear that you guys keep it affordable that way. I, I was looking at going to a conference. You know, not, not anything particularly open source, although I tend to make conferences open source when I go to them.

Um, I, uh. Famously was at a, uh, U. S. Department of Defense conference where they were talking about the problems with, uh, FPGA security. And, of course, I, from the back, Why don't you just open source the toolchains if you want it to be more secure? I have lots of fun doing stuff like that. Uh, anyway, this conference, I was looking at it in the background.

Tickets were like 699 a piece or more if you wanted everything. It's like, oh man, we can't expense that right now. Um, so I think that's really neat that you guys make it, uh, you make it affordable. You mentioned sponsors, uh, And I, I, I was going to ask whether you have sponsorships. Um, and, uh, so how does, how does that work?

Do you guys go, do you have like a list of companies that you go after each year that it's going to be put on and you say, Hey, you guys supported last time. Would you be interested in, in, in helping do this? And then like, is there a banner somewhere? You know, is there, is there a banner wall where people come up and take their picture with all the logos behind them?

Yes.

Gary: Yeah, we do have that. So. So I think one of the nice things about OGCAMP is that we've always tried to not be super corporate, right? Um, yeah, you're not going to come and see, you know, logos of insert big FOSS company here plastered in every single conference room. But what we want to do is cover our costs and we want to, you know, return a fair value to those people who help us cover our costs.

So, um, this year we've actually ended up with four sponsors. I think I'm right in saying that none of them have sponsored us previously. They are all entirely new sponsors this year. Um, and they are anything from kind of small software development companies, to larger open source projects, uh, to a trade union actually this year is our pinnacle sponsor.

I mentioned Utah a couple of times. Yeah. Um, And yeah, so they get their Lego on a t shirt and they'll get their Lego on banners and things and, you know, a personal thanks from us for, for really helping make our camp happen because we couldn't, we just couldn't do it without, without that little bit of extra cash injection that we get from sponsors.

Um, So, um, yeah, I think like I said earlier, uh, Utah running a couple of workshops this year and a panel discussion. Um, a couple of other sponsors have got stands in our kind of community room where we sell merch and put out tables for people to do anything from play role playing games to show off their cool Raspberry Pi projects.

Um, so yeah, we, we have a list of people we target, um, but this year it's actually just been an entire new set of sponsors, which is actually really cool. And what we're hoping is that that will introduce a whole new demographic of people to, to OrcCap.

Jonathan: Um, so I was going to ask, I was going to ask who the sponsors were, and I think you sort of just covered that, um, you've got the, the trade union.

Uh, is there any music? So one of the things that it, it sort of, it sort of hits my mind that there might be an intersection here is that you could have musicians and people very, very interested in music, very, whether putting it on or doing talks about music or even building your own music hardware, it seems like you could have some of that.

Gary: Yeah, so Dan, previous organizer, had previously done our evening entertainment in 2019, which was quite cool. We don't have any kind of scheduled evening music entertainment this year, but I'm absolutely sure that there will be people turning up with their cool synth project that they've built with their ESP32s or something.

I know we've definitely had things like that in the past where someone's turned up with, uh, yeah, here's my, here's my rack of, uh, Analog synths that I've built from open source hardware and say yeah that there always is that kind of maker slash music crowd I think that turns up our camp But again, it's it's kind of like we were saying it really depends on who turns up as to as to what's gonna be there But but that is one of the the many crowds of people that turn up our camp.

Jonathan: Yeah Do you have anything like I know in some conferences people like to come and actually camp out? Uh, to, to attend it rather than having to pay. Is there, is there something like that? Is it going to be a tent village?

Simon: Um, I think it's unlikely. Okay. Um, you know, the Manchester is not the greatest place to camp out during October.

Uh, and, or, and it's gonna be quite busy in the city this year. There's, because there's a, there is a sporting event going on, uh, as well at the same time as the conference. So I, I, no, I, I think everyone's very likely to either be local or in hotels. , uh, certainly I haven't booked a room yet, so I really must get round to doing that.

I, I might be sleeping on the street. Simon's gonna be ski out on the

Gary: street. Simon's gonna be on the bench outside in the, uh, in the hotel.

Jonathan: No, Simon's going to be in the hallway track like, Hey, do you have an extra bed in your hotel room?

Simon: It's the hallway and sleeping bag.

Jonathan: Oh, that's great. That's great. Um, so let's see what, uh, what, what have I not covered?

What else do you guys want to let folks know about? What, what else are you looking forward to with the conference?

Gary: I think for me, it's, it's just getting something running like this in the UK. I think the UK is. Is kind of really lacking in these community led FOSS events. Um, yeah, there's, there's a bunch of them that exist across in the U S um, but, but we're definitely lacking in them in the UK.

So I think for me, it's going to be the main thing I'm looking forward to is just seeing old faces, new faces, what people come up with and really, you know, how the community has evolved in the last five years, I think really for me is, is the key thing that I want to get out of old camp this year.

Jonathan: Yeah.

Simon: I think I'm in a similar place to that, you know, in the, the, the curious thing is I've been doing open source for a couple of decades living in the UK.

And I've always had to go abroad to go and do anything, you know, and so there is a certain desire to hold grassroots events that are not about, uh, feeding a corporate strip mining open source culture, but they're actually about people who use things and make things. And so I've always loved OGCAMP for that reason, that it's about the only event that I can go to in the UK.

Actually, it isn't the only event in the UK. There is a, there is a, uh, another conference in the UK now as well called, uh, EMF, which actually does have a campout, uh, that happens as well. And that's also in the north of England. So there's, there's now two events in the UK that, that are, uh, addressing that niche.

And I find that really encouraging that that's happening. Um, I, I do, I've often wondered whether, uh, Europe needs to have an, uh, an, a summer FOSDEM. And I'm kind of hoping that eventually one of these events is going to evolve a bit into a summer FOSDEM so that we can have a Europe wide open source event.

Europe does have quite a few good open source events. Um, there's a really good one that I'll be plugging later in, uh, happening in Northern Italy in November. And then there's a couple of, uh, good ones in Germany, like, uh, uh, the one that happens in Bonn. Um, and there's a, a, a good international conference called FOSS Backstage that happens in Europe.

So, there's good things going on, but the UK has tended to be, um, the one place where there has not been enough going on. And I think that's one of the reasons why I'm excited about being involved with Oddcamp. I'm really pleased that Gary's come along this year. Gary's a very level headed, very capable, uh, uh, uh, sensible person.

And I'm very pleased to be supporting him.

Jonathan: Yeah. You know that that idea about not having anything in the UK and sort of missing it I get that because like there's not a whole lot here in Oklahoma where I'm at and Be nice. Don't laugh at my state. Uh, I think I think Oklahoma is actually about the same size as UK if I remember correctly It's it's gonna be on the on the discord Somebody put a picture of like the map of Europe and then Texas over it And like, Texas is just, is bigger than Germany and covers half of France and covers half of Italy.

And, you know, Doc Searles famously says that the big difference between the United States and Europe is, in the United States, we think a hundred years is old and over in Europe, y'all think a hundred miles is far. And that just, that really, I think that's actually really great. It, it covers, Sort of the mindset difference that comes out of the history and geography of the two places.

But like I've had this thought there's nothing in Oklahoma that I that I know of at least nothing big uh, you got to go down to Texas and I got real excited because there was somebody that was going to do like a once a year I think it was the Texas Linux Fest and was going to be in Dallas like oh Dallas That's that's you know, that's when the not too much more than 100 miles.

That's not very far Uh, you know, I could get down there in three hours. That's not too bad um I don't know if that's happening again, because again, we got hit with COVID and so a lot of things closed down. Yeah, it really makes, it really makes me think that, uh, maybe we need more people from the community to start talking about things like this and try to put them on and put them together.

Because there is something to be said for getting together in person and not just over the internet. Uh, meeting, meeting people in person. And actually doing things, putting your hands on things together and, and fiddling around with stuff. Um, So I think, I think that is cool and I, I, I look forward to hearing stories from this year about what all happened and which talks happened and which ones were popular and people that got to meet each other for the first time.

Simon: And you know, Jonathan, we can discuss licensing OGCAMP to you for OGCAMP West if you want to run it over there in Oklahoma.

Gary: Well, it was that meeting of people in person that kind of inspired me or I guess, uh. Maybe drunkenly got me convinced to organise Oddcamp this year. Like

Jonathan: all great ideas, it was birthed in a tavern, in a pub.

Gary: genuinely happened in a pub, yeah. So, um, it was actually, I worked for a brief period with a previous organiser, John Spriggs. And, uh, he was like, oh, I'd love to see Oddcamp come back, but I don't want to be the one doing it. And, uh, you've been to Oddcamp a load of times, you know how it works. How do you feel about putting, uh, putting on an Oddcamp?

Couple of beers later, and before you knew it, I was sending an email to Simon saying, Hey, what do we need to do to get this thing kicked back off on a cold December night at a white Christmas party? So, uh, yeah, yeah, it's, it's getting the community back together, I think, is where some of these things really, really spawn out of.

So it's also going to be interesting to see, like you said, what, what other things spawn out of Oddcamp and having people together that, um, yeah, is, is going to happen. Yeah,

Simon: you know, I want to say to all the odd camp attendees and supporters who are, who are listening that, um, we're getting back again, back together again this year.

We haven't been able to nail down all the things that you, that you want there to be there. So try and bring them with you this year, if you can. And if you can't help us organize it next year, so that they're there. So, you know, we, as Gary said, we don't have a. Um, an event organized on the, uh, the intermediate night on the Saturday night.

Uh, it would have been great to have booked a room and got a band and had everyone, you know, able to order their fish and chips and mushy peas and their half a lager. Um, but, uh, you know, that didn't happen. And so maybe it can next year and maybe it's one of the people listening to this that's going to be the person that, that insists on organizing it.

Um, Probably too late to organize it this year. I think that the hotel told us that they, you know, we've, we've, we've paid for a certain number of rooms. Uh, we could go back to them and try and say, Hey, you said, you know, we said we didn't need the ballroom. Well, we do now. Um, and if you're listening to, you're listening to me saying this and you really want to organize the thing in the ballroom, then get in touch.

My email address is, uh, simonatwebmink. com. And you can, you can get ahold of me that way. Okay. But, so this year I see very much as a relaunch, you know, it's, it's doing the things that we know work, um, hoping someone's going to bring the stuff that we haven't been able to organize, but really hoping that we're going to have some new blood who are going to help us organize, um, the double this event next year.

Jonathan: Yeah, yeah. So what, like, you, you allude to this, what's not happening that people wanted to see happen this year? What happens in the ballroom?

Gary: Well, what happens in the ballroom stays in the ballroom, I guess. I could tell you, but then I'd have to

Jonathan: kill you. I walked into that one.

Gary: Previous years, Oddcamp has been quite famous, I think, for its, its social track.

Um, so we've always had some kind of entertainment on maybe the Friday night, definitely the Saturday night and occasionally actually the Sunday night. Uh, we've always had some kind of prearranged entertainment. So, uh, yeah, whether it be a band playing or some kind of FOSS quiz on the Saturday night or something like that.

Um, Like Simon said, this year our focus has really been on what do we need to do to get OggCamp up and running. Um, and what do we need to do to do that in a way that is sustainable for a small, new, organised increase. So, some things like that have fallen by the wayside. Um, and yeah, Simon alluded to things like the raffle, um, We're pretty sure the raffle is going to happen this year.

They're kind of famous or camp raffle. Uh, we just need to, as an organizing crew, get our head into gear and make sure that we've got everything that's needed in terms of prizes there. Um, so I think the big thing is going to be the social track. I think that's the big difference that people might notice this year compared to previous years.

Um, But that said, um, the hotel have graciously said that we will have the bar open for OCCAMP and they're putting on some food and things like that for attendees to buy at a reduced rate. So, yeah, there will be social tracks coming, uh, you know, there will be things happening and I think it's human nature to have a like minded group of people get together and congregate in the hotel bar or head off elsewhere.

Um, so, yeah. Yeah, I think, yeah, it will, it will feel like an old camp, um, it's just going to be a case of, uh, yeah. I guess a little bit of autonomy happening, uh, among attendees that perhaps we, we didn't do quite as much before.

Jonathan: Yeah. My, my wife is in the discord and she's also apparently listening to some of this and she says, if life circumstances were different this year, I'd all be for attending this conference in Manchester.

Like both of us, none of this just sending Jonathan business. In years past, I went to a conference in Dublin by myself and, uh, While it was fun, it was a bit of a bummer that I couldn't take her along. So one of these days,

Simon: you know, it's not too late. Just check, check out, uh, the, the, what flights are available.

There's going to be some cheap flights direct to Manchester. You can just make the weekend for it.

Jonathan: Yeah, I think our kids are a little too young for that at this point, but, uh, that's, that's all right. Maybe in a couple of years. So, what, let's, let's, let's, let me ask you this. What are some, like, highlights from the last couple of times you guys had OggCamp?

Uh, what are some of the talks that really you remember or just memories you have that you think people would enjoy hearing about?

Gary: I think for me the first dog camp I went to was really eye opening just as to how small the UK is. Um, so I ended up actually getting talking to a guy in, uh, on one of the hallway tracks and, uh, I'm from a fairly small county in the UK, uh, so Norfolk on the east coast for anyone who knows it.

Um, quite famous for the fact that we've got no motorways in any one city. So it's, uh, it's a fairly small place and I got talking to this guy, um, And we were like, he was like, where are you from? And I said, where I was from. And it turns out that he lives five minutes down the road. We both have, yeah, had similar jobs at the time.

Very similar kind of interests in FOSS and things like that. And we've stayed in touch ever since. So that for me was a real OGCAMP highlight. Um, but in terms of talks and things, it's actually been the talks that, necessarily directly FOSS related that have really appealed to me. So I kind of alluded to before that we've had talks on things like mental health and stuff like that.

And actually going along to those talks and just hearing those insights as to, ah, this person struggles with the same things of, you know, Struggles of working in IT as I do or this person has come across this particular problem They've tried to solve in this way and perhaps that's a different way that I could approach something that have I've always been highlights for me So it's it's almost been the unexpected things that you couldn't plan for going to a conference that have been highlights for me So I mean you have any thoughts

Simon: Yeah, wandering around seeing the toys people have got in the, uh, in the, in the exhibit area where the tables are.

That was, that's always a feature for me. I remember having a picnic on the front steps of the venue in Liverpool when the conference was held there. Um, I remember giving a talk all about, uh, the, uh, legislation that keeps on trying to backdoor crypto, um, which is back again this year, just in case you had any concerns.

Uh, so, uh, those are all pretty, you know, they're, they're, they're minor sounding things. They're pretty random. For me, the event, this is, this is the, not the event where there are the, the big corporate talks. This is the event where there are the, uh, the important new relationships are created. And that's what I really hope for for this year as well, is that we'll meet interesting people and befriend them.

Jonathan: Yeah. Yep. Very cool. All right. Uh, let's see what, uh, what have I not asked about? What have we not covered that we, we ought to let folks know about?

Simon: Great questions. Uh, so if you want to buy tickets, uh, there are still a few available. Uh, you'll, you can find them at, uh, I think the org. camp is, uh, is the, the easiest to remember URL that will get you there.

So if you type in org. camp, it will take you to our website. Uh, and I'm pretty sure that we bought all the other websites that sound like that as well. So I think we, we have oddcamp. com and things, but what the cool one is odd. com. There

Jonathan: you go.

Simon: Um, and, uh, you know, I encourage you to pay what you can for the tickets.

And, and that means at both ends of the scale, if you can afford the thousand dollars for the ticket, please buy a ticket for a thousand dollars. Because you're, you're investing in the future of the conference. When you do that, the conference isn't run. For the benefit of, uh, anyone financially, um, what will happen if there is a surplus is public software will hang on to it and it will be available for next year's conference to be bigger and better.

So, um, uh, it's, you know, it's, if you can afford to, uh, to, to buy the thousand, thousand pound ticket, we don't actually have a thousand pound ticket, Gary, could you fix that before the show goes out? Um, go, go, go ahead and do that. Um, uh, Also, keep an eye on the socials, we do have a couple of venues, I don't think we've fixed on an official venue on social networks this year, but we do have a Mastodon account.

We do have a room on Matrix that you can find, and there may be some other remnants somewhere, I think we might still be on that social network that nobody should use anymore.

Gary: Yeah, there's a, there's a few places there's, there's the X accounts, there's a telegram group, uh, there's a discord as well. I know you love hanging out in discord Jonathan. I

Jonathan: didn't say I love hanging out in discord. I said that's where I find myself a lot.

Gary: Yeah, I think I'm in like five discords at this point.

I can't, can't honestly say I read messages in all of them. I'd have to go count. Oh my goodness. A lot. It's quite a lot. Um, yeah. Let's see. Og. camp is probably the place to go, um, and anything that you need to know is, is kind of linked from there really.

Jonathan: Yeah. Uh, what are the dates? I don't think we've ever actually said.

When is this?

Gary: It's the 12th and 13th of October, uh, this year, so coming up in just a couple of weeks.

Jonathan: Less than a couple of weeks. Yeah, that is, that is coming up soon.

Simon: There's still rooms at the hotel, so you can still stay at the venue, which is the Pendulum Hotel in Manchester. So you, uh, that, that's available.

There's still a few tickets left. Uh, if you're quick, um, Gary's going to have to turn off the tickets when we reach the fire limit for the venue. Uh, so, uh, you want to get in there as soon as you hear me saying this, cause there's stop listing now and go to org. camp and buy your tickets.

Jonathan: Yeah. That's the important thing, right?

You don't have to listen to the episode. If you're going to be there, uh, what, what year did, did our

Gary: camp start? You'll know this better than I will, Simon. I don't know, I joined it after the fact. 2009, I think. The last OddCamp was our 10th anniversary in 2019. So, uh, this is the 11th OddCamp, but it's our 15th year, which, uh, feels like another milestone.

Jonathan: Yeah. Boy, the world was different in 2009. Goodness.

Simon: Ah, those were the days.

Jonathan: Yes. Yes, they were.

Simon: Still a Sun Microsystems, then.

Jonathan: Yeah. Uh, Gary, I didn't ask and I guess I should. How did you, uh, how did you get into, we know, we know Simon, we know where Simon came from. How did you, how did you get into like the, the whole open source and free world?

Like what's your, uh, what was your gateway drug that got you interested and that you started coming to Oddcamp?

Gary: It was a teacher back in high school actually handed me a CD. So yeah, I, I at the time was, uh, was running a, uh, Not so great laptop that been handed down and handed down and handed down and Windows didn't run so well on it and I just went to this teacher who was a Gentoo user back then and said yeah, what could I do to sort this out and he said, I'll sort you something out tomorrow and came in and almost, you know, dealt under the desk this Kubuntu 7.

10 or 7. 04 or something CD and I was like, yep, here you go. Um. And never looked back, I installed that on that machine, it lasted me another few years and uh, yeah I guess from there ended up getting into, you know, working in IT, advocating for replacing a bunch of Windows stuff with FOSS where it made sense, um.

Then kind of got into the kind of cloud world and continued to advocate for open source stuff there. Um along with attending Uh op camp and fosdem and linux fest northwest and anywhere I could get my fill of uh, false goodness Um, that's that's where i've been for the last 10 or 15 years, I suppose

Jonathan: And you're thankful that your teacher gave you a kubuntu disk and not a uh, not a gen2 disk

Gary: Uh, yeah, I mean it might have had a slightly different appeal, I mean since then I've moved off of the Plasma desktop and, uh, daily driving Fedora with Gnome.

But, uh, yeah, I mean, it was probably a wise choice not to send me down the road of, uh, doing a Gen 2 install at, uh, you know, 12, 13 years old or whatever it

Jonathan: was. Yeah. All right. So let me, let me actually cover this and we get to ask Simon. I don't think we asked Simon this. Um, and I don't know the answers.

We're gonna start with Gary. I want to know what's your favorite text editor and scripting language.

Gary: Uh, for me, it's got to be nano and bash. I just, I want to deal with something simple. Uh, so I actually spent a lot of time just in nano writing stuff in bash, which is probably the most boring, terrible answer I could give.

But for me, I'm all about keeping things simple.

Jonathan: No, no, that's all, that's all good. You know, Simon, I don't know that I know the answers to this. I don't think we've ever asked you. Yes. What's your, what's your favorite text editor in scripting language?

Simon: So, I never really got into text editor religion, and I use whatever there is that is on the system I'm running.

Um, I have a soft spot for nano on my raspberry pi's, uh, but I, you know, I used to enjoy using E when I was at IBM and, uh, which nobody will know about and, um, uh, it's, but, so he is the editor that, that Emacs. Became when you added the macro language.

Jonathan: So

Simon: I, I used to enjoy using that. Uh, I do most of my editing these days in whatever notepad I can pop up on the device I'm using at the moment.

And I use Markdown, um, and when it comes to scripting, um, it's been a while. I have to tell you. Uh, so the last scripting language that I used in Anger was on a, a sign organizer. Uh, I actually wrote commercial software in sign organizer scripting language. As

Gary: you went lying when you said it'd been a while, eh?

I think that's pretty much it. For me, it's a really difficult question to answer these days because I spend so much of my time in the cloud world that it's like, you know, whatever thing abstracts away the clouds is what I end up using. So I end up writing a lot of Terraform and Ansible and CDK and stuff these days, um, just because I guess that's what I work with day to day.

So, uh, I don't get to play with, uh, with much nice, cool scripting stuff anymore.

Simon: Yeah, to give you an idea of how old we're talking here, um, uh, so, uh, my story was a little bit like Gary's, so my, my physics teacher at school, uh, uh, was moonlighting while he was teaching physics and making the motherboards for, uh, compute for development systems that used a 6800 microscope.

processor called SWTP 6800 development systems. This was in 1977. And, um, so that was my first computer. Cause I used to, I do, he used to soak test them in one of the labs. And we used to go in during the breaks and program them. And the reason I'm telling you that story is I went around the computer history museum in Silicon Valley recently.

And I, there is the computer that I learned to program assembly language on, uh, sitting in the computer history museum. And that's how I know I'm old. Ha!

Jonathan: When you find your first computer sitting in a museum, yeah, that would do it. Uh, now I have to think, like, what was my first computer that I ever actually did anything on, and is it old enough?

It might be. It might be. The first one I ever played with, my dad had one of the Trash 80s. Excuse me, TRS 80. But that wasn't really modern then that was like pulled out of storage But I think we had a like a tandy one of the original tandies and I think those are just old enough to be museum pieces now, so I suppose i'm old too.

Simon: Yeah You keep using you keep using that hair dye jonathan

Jonathan: No, no, mine's all natural there's a little gray poking in there, but it's not too bad

Gary: Now I feel young when I can say that it was a pentium 133. It was You The first machine I really used and did stuff on, but even then that was somewhat of a hand me downer space.

Jonathan: Yeah, yeah, yeah, I get it. All right. Well, I am, uh, I won't be there, but I'm looking forward to OGCAMP because it sounds like you guys are going to have a great time this year. And hopefully we're going to have

Simon: an old fogeys reminiscence session over a beer in the, uh, in the, in the bar at the hotel where I will show people photographs of the SWTP 6800.

And we can talk about a 6800 assembly language and the unique register commands it had.

Jonathan: Yeah, I mean, who knows? You guys keep doing it, you'll eventually have the, uh, retrocomputing track. People will bring their old machines and let people play with them. Alright.

Simon: Now there's an idea. Next year, Gary, okay?

Yeah, we'll put it on the list.

Jonathan: I mean, retrocomputing is big right now. There's a lot of people that are, uh, Very nostalgic for the machines of yesteryear. So, all right. Thank you guys for being here. I appreciate it very much. Gary, particularly, thanks for coming and telling us all about the conference and, uh, best of luck here in a couple of weeks that everything goes exactly the way you want it to.

And, you know, all the surprises are good and all of that stuff.

Gary: And tech gods are with us for demos and audio and. projectors and other such things.

Jonathan: Yes. Yes. I've been that guy and there's always something that surprises you. Uh, awesome. Thank you so much. Uh, Simon, this is where normally I ask you what you think, but you were sort of the, uh, guest too.

Um, I guess I'll go ahead and ask you though.

Simon: Well, I think everyone should buy tickets and come to the conference in Manchester because, uh, you know, I could. I could hype it up and say it's going to be, you know, the, a super majestic technical whiz orgasmic thing. Uh, but actually we're just going to be a group of people who want to be friends, uh, coming together and sharing a space and sharing our interests with each other.

And, um, uh, I think that, uh, if that appeals to you, then you should go get a ticket. Uh, and if that doesn't appeal to you, you really should go to, I dunno, some Oracle conference.

Jonathan: If that doesn't appeal to you, then maybe you need to rethink the decisions that have led you to this point in your life. Oh, all right.

Well next week, interestingly, we have somebody from IBM going to talk about their, uh, Open AI stuff, and it'll be very interesting to compare notes with what IBM thinks about open source LLMs and AI with what OSI and Mr. Simon Phipps thinks about it So we will keep that kind of as a bug in the back of our minds as we talk to IBM about it Simon is there anything you want to plug?

Since we have you here. Well,

Simon: uh, there's, there's Ogcamp, obviously. Go to og. camp and buy a ticket now. I'm going, I also help organize another conference in Europe called South Tyrol Free Software Conference, which is another conference that you might not have run into. It's, it's now grown into quite a big conference.

There'll be over a thousand people there. It's held in a beautiful city in Northern Italy called Bolzano. Uh, it is the, uh, the second weekend in, uh, November, and I would love anybody who is listening to this is in Europe, can't make it to Manchester to come and meet me in Sno for SFS Con with the website is S-F-S-C-N do it and the conference is conducted in, uh, in English and will be fantastic.

Excellent. And what about you personally? Where can people go to find. Uh, probably the best place is, uh, on, uh, Mastodon where you'll find me as, uh, webmink at meshed. cloud, M E S H E D dot C L O U D. Uh, and I would love to see you, uh, popping up there and saying hi and following me. Um, my, uh, most of my stuff, if you want to find out about me is at a website called webmink.

com. That's, uh, W E B M dot I N K, and that's got all the pointers to everything about me on it. Simon, you're all

Jonathan: about the fun,

Simon: uh,

Jonathan: TLDs.

Simon: Well, you know, they exist, and it seems a crime not to make the most of them.

Jonathan: Not to have fun with

Simon: them. Because you can do good things that are memorable with them, you know?

Uh, so, you know, you can email me if you want to. I'm simon at phipps dot email. Uh, you can have a look at my website. Uh, that's, it's webm. ink. And, uh, I, I host my own Mastodon server at, um, mesh. cloud. So find me in one of those places. Running on a Raspberry Pi. That one is not. That one is actually, uh, running on a slightly more robust hosting server, because it turns out it takes quite a lot of energy.

Jonathan: Yeah. Yep. Uh, it makes sense. All right. Thank you, sir, for being here. Sure. It'll do appreciate it. Um,

Simon: Well, thank you for the invite, you know, have, have me back again sometime.

Jonathan: Oh, for sure. For sure. I've, goodness, we've, we've got to, we've only got so many co hosts and people get tired of, of listening to, to, to Dan and Doc and, and all of that.

Definitely an important part of the crew, man. All right. So if you want to find me, Of course you can. Go to Hackaday, that is the home of Floss Weekly these days, and we sure appreciate that. I've also got the security column that goes live on Hackaday every Friday morning. Keep you up to date with the, uh, the bits and the bytes that are moving in ways they're not supposed to.

Uh, and then we do, of course, have the Untitled Linux Show over at Twit. Dot TV still. And we have a lot of fun there covering, covering the news and a bit of, uh, open source stuff, but with a decidedly Linux twist on it. Uh, so make sure and check that out too. We do appreciate everybody that's here. Those that watch both live and on the download, and we will see you next week on Floss Weekly.

Episode 802 transcript

25 september 2024 | min

FLOSS-802

Jonathan: Hey folks, this week Randall joins me and we talk with Michael and Benedict about EMBA. That's the Open Source Firmware Analysis Toolkit that seems to do about everything and just about includes the kitchen sink too. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 802, recorded Wednesday, September 24th.

EMBA! Layers upon layers of Bash.

Randal: It's time for Floss Weekly, the show about free! Libre, open source software. I am your host, Randall Schwartz. Merlinisunwich. com. We're here each week! The boomers, the shakers, the bigs Wait, what? Wait, wait, Jonathan! I'm the host. You're the co host today. You're the co host today.

Jonathan: Wait a second, this is Floss Weekly and I'm the host, Jonathan Bennett.

I'm so sorry. And Randall Schwartz is back as co host today.

Randal: Oh,

Jonathan: goodness.

Randal: I wake up in the middle of the night sometimes saying that exact phrase because I said it for You know, 13 years worth of shows. So it's I really, really do want to appreciate, appreciate you bringing me back as now possibly an occasional as often as you'll let me co host. Because I do actually, I miss the show.

I've got to say that for everybody out there who's been along loyal friends. I do miss the show. What I, what I don't miss is that I now have. My five hours a week back as Jonathan well knows there's a lot of time. There's a lot of time that goes into this show every week, every, every week. And I'm happy to have most of that back now.

And the two hours a week that I will take the coast here. Much simpler than being the pro producer, show runner Jesus and host. So thank you Jonathan for doing that job and keeping the show going. I appreciate that as well. So, yeah.

Jonathan: Yeah. You know, one of my, one of my favorite things about taking over the show is that I, I get to have Doc Searls and Randall back as part of the, the rotating cadre of co-hosts, and I think that's just, that's just fun that to be able to have you guys back involved with the show.

So today's topic is EMBA and that is, it's a firmware analysis tool, I think mainly targeted at Linux firmware. This is it's, it's really an interesting thing. You know, there's a lot of, there's a lot of OpenWrt, right? Like I'm a fan of OpenWrt, but there's a lot of old versions of OpenWrt that are out there in the wild.

And it's, it's pretty scary sometimes to see like 10 year old firmware still out there shipped on devices and I kind of wonder do they have a module that just spits out This is this release of open wrt beware

Randal: Yeah, and and i'm i'm not terribly in this space So I had to do a bit of research to try to figure out how this is being used But now that you mentioned open wrt there's a recent news item actually it says that tp links Routers are now all considered caustic.

And so, because they were made in China and apparently there's been some spyware that's been traced to TP link. So everything TP link is at risk right now. So

Jonathan: yeah. Yeah, I saw that story. I'm not sure if, if I don't, I'm not sure if the statement was that they are like a compromised and malicious from the factory or that they just have zero days that allowed that allowed them to get popped in that.

I was gonna say, I think we

Randal: have some experts coming up that can answer questions kind of like that. I'm also interested in particular in reading ahead of the show. Is this primarily a white hat tool? Or can it be used as a black hat tool? And what are the implications of that? Having a tool that can do pen testing, vulnerability testing on a common piece of firmware somewhere.

So I think it's probably one of the questions I really want to, I'm curious about.

Jonathan: Well, we do have the experts. We will make sure and ask about that. I do want to mention that Hackaday is what Floss Weekly is a part of Hackaday. Hackaday is a part of SupplyFrame and SupplyFrame is actually owned by Siemens.

And I say that because the EMBA project was at least partially developed originally as a part of Siemens, or there there's at least some shared history there. So we want to make full disclosure. Now we do have Michael and Benedict with us today, and I want to say welcome to both of you.

Max: Hi guys.

Hey guys. Good to have you.

Jonathan: Good to be here. Yeah. So maybe the best place to start here is to ask each of you, how do you fit into the project? Like what are your individual roles in, in this thing?

Max: So from my, from my side, I'm currently primarily or directly with the, with the central Amber project. I'm maintaining the central Amber project, which is, let's say the, the, the scanning backend. I'm, I'm more or less the founder of the whole Ember idea of the firmware analysis idea. And we've started multiple projects from time to time.

One is Ember, the other is Embark, where Benedict is the, is the main maintainer now. So, so at the end, we can, we can split it in this way. If you, if we talk about the main Ember project, then probably with me. And if we are going to, to the, to the let's say management interface, enterprise overview and a collaboration interface, then Benedict is the guy to talk to.

Jonathan: Yeah. And so same question to you, Benedict, how do you fit into the project?

Benedikt: As Mike just said, I'm the main developer responsible for Embark right now. Embark is our enterprise solution, like taking Embark as a backend and then trying to put it into a, Nice server front end. Yeah, oh how people would need it in the business environment.

And That's

Jonathan: yeah. Okay. So probably then a question for mike. I'm, i'm super interested in this idea of packaging it But let's let's start with michael first since you kind of kick the project off and I I do want to know like what What is that tie in like what did the beginning of this look like and what is the tie in with Siemens?

What was like the first problem that you guys wanted to solve?

Max: Yeah, yeah, and so probably we we need to go a little bit in back to the future I think we start we start we started with all with all of this Let's say what was it 10 to 11 years back. I think everyone started with with iot hacking then You There was, was Binwalk Craig Hefner did a lot of nice blog posts about cool IoT hacking, cool exploits.

Everyone tried to show some, some hardware hacking stuff. And the idea was back then at Siemens that we Get away from the, from the typical black box approach on, on testing products, our own products and also third party products, to a more gray box ish penetration testing style with, with the firmware enhanced.

So, with the firmware, you would be able to understand what happened in the background. If you're attacking the device, if you're trying to find vulnerabilities you get a better idea what happens in the background and which ways probably could be successful. And this was in the, in the, in this time also we, we built built the hardware hacking lab to extract the flash storage to get access to the firmware.

Sometimes you're, you're, you're in a lucky position to go just to the vendor website, download the firmware. Sometimes the, the firmware is behind the paywall or something else. And so we started with hardware hacking, identification of UART, JTAG, extracting, flash storage, and all, all the things. And then, then we had the firmware enhanced and we, we were, we were struggling really hard because we, we had not more time to perform our penetration test.

But we had so many ideas and we want to do much more stuff now. And so we needed to start automating everything that, that is possible. So and this was more or less the, the initial idea of Ember. It is more of a, of an automation framework in, in the field of embedded product Linux penetration testing that should help the, originally that should only help the penetration tester.

Yeah, that's, that's more, more or less the, there it was, it started like, I would say 10 to 11 years or, or ago. With, with, with an idea.

Jonathan: Yeah. And, and so it's interesting to see like the number of devices that are out there that just like the firmware is Linux. And we talked, we talked briefly before the show or in the, in the intro about open WRT is that a lot of what you guys see that it's just.

Ancient versions of open WRT that just get shipped on some of these devices.

Max: Yeah, you, you, you definitely see open, open VT everywhere, but let, let's say in, in the IOT environment, you see it everywhere. If you are moving a little bit away from, from iot more to the, to the OT environment where bigger products of in for factories or for for other things are used then we we see open vrt not that often But in the iot world, it's it's quite regular definitely

Jonathan: Yeah, one of the one of the fun discoveries that I have had is a lot of you ubiquity hardware Which you know, that's sort of your your small office home office to medium size I mean they have some they have some impressive stuff like all of the access points That's just open WRT or at least it was last time.

I logged into one of them So it's it's in a lot of places I

Max: think Netgear also has a lot of OpenWrt based systems out there. I

Jonathan: think, I think Netgear was like the OG of OpenWrite. They they're the ones that first did the GPL release that got turned into it. So that's not terribly surprising. All right, let's turn to Benedikt for a minute.

And so you are, you said you're kind of the guy that, that packages the whole, the final thing together and maybe makes it a little more business friendly.

Benedikt: So let's talk, let's talk about that. Sure. So the main thing about Ember is it's built in Bash, right? So we have a combination of scripts in the beginning.

That's what Mike spoke about is we had the idea of combining stuff into something that gets automated. And once we, or let's say Mike, the Ember team was at that point where like it was working and then you had something on the CLI, right? So on the command line interface, you have something to type in and then you get something out of it.

We built an HTML export thing. So you get something that is nice and shiny and clickable. But the issue you're always facing is you have some firmware and you are a team, a team of penetration testers who are essentially find, trying to work on one thing at, as a team. So you have certain, certain things one person looks at and certain things another person looks at.

And if you do what Ember does on every computer, then you're running into, well, time constraints also. So one of the ideas was to basically centralize the whole thing, put it into a server. So make it a business application in that sense. You upload something. Amber do the work and then you get something that everyone can look at without the usual issues of CLI where everyone works on their own.

Jonathan: Yeah. So do you, do you have any idea of like how, how popular it's gotten when, now that you've, now that you've put that package together? Do, do you have any, any idea of like, how many businesses are out there using it, how many researchers make use of it? I think that's

Benedikt: a good question for Mike because he keeps track of those things.

Max: Yeah. Yeah. It, it is quite important to, to, to get a good visibility for, for an open source project at all. And so, so I try to track it a little bit. It also helps for for arguing my, my work on our work on AMBA a little bit better. And during the last two years, we, we have seen that, that AMBA gets accepted in, in a very broad way.

So, there are popping up more and more blog posts out there. So, so people are talking or writing about using AMBA. And we have seen a lot of let's say multiple research papers that have used ember in some way or at least referenced it. And it looks a little bit they like to show where ember fails.

And this is great because we can then improve Ember, we get a better idea on where Ember gets used and how the people are using it and we can improve it, so nearly from every research paper we get new ideas to improve it. And if, if, if we, we are, we are also looking some, sometimes at Twitter and other social networks, and we, we, we have a little bit the feeling as, as I would say most of the penetration tests out there that are doing IOT penetration tests from time to time are at least using Amber besides their manual work.

Just to get an, get an initial overview of the, of possible vulnerabilities to get an idea of where they can dig deeper. As Ember is automating a lot of stuff, you can run, let it run over the weekend and you can start with with a quick start on Monday morning, much better than starting from, from zero if you have let run Ember over the weekend.

You get some, some basic information about the firmware, you get an S Bomb, you get all of the things that you can, can then use for, for improving your manual tasks and yeah, so, so penetration testers are using it researchers are using it, and we also are from time to time in contact with quite big companies that are trying to establish AMBA in, in some way.

Some kind of the security processes one, one big company has established Amber into their IC development process. The next one has established Amber as a quality gate for third party components. So I would say Amber is quite solid established. Hopeful, hopefully we get Many, many more developers that or back, or back hunters so that we can improve Ember in the future much more.

Jonathan: Yeah, yeah, absolutely. Randall, I know you were curious about some things around people using it. You want to, you want to take it and ask those?

Randal: Yeah, and where I'm starting with is, so I understand the basics of pen testing. I don't know how it applies as well to embedded. How much of EMBA is unique to the notion of it being firmware and what kinds of things do you do there?

I know like for traditional pen testing, you're opening sockets, you're trying memory locations, you're trying things. You're trying to recognize known vulnerable libraries, things like that. How different is it at the embedded level and what kinds of things do you do there?

Max: So I think you, we need to differentiate a little bit.

So if we, if we are talking about penetration testing of, of firmware, I typically our projects are typically looking like we have the device. So Also, we can poke around with the device like in a typical penetration test and on the other hand we have the firmware where we can look into firmware to find things that we can verify on the device.

So on the device by itself it's quite classical. So you're looking for interfaces like web interfaces, you're proxying your traffic you try to inject commands. That are executed on the operating system. And on the, on the firmware level, we, we're doing much more static analysis then. So we can, for example think about PHP scripts.

There are, if the web server is, is organized via PHP, then Ember identifies the PHP scripts and can do static code analysis on, on these PHP scripts. And then Ember can point us to interesting areas that we can try to exploit then on the device. So at the end, it's, it's, it's every always the same.

Amber gives you, gives you an idea where something interesting could be, and then you as a, as a, the responsible tester, you're digging deeper, you're analyzing this possible issue, I would say.

Randal: Is it doing things like fuzz testing and like just throwing random data at stuff to see if it breaks? Or is that more, this is more about identification and it requires a human to maybe be trying that kind of stuff?

Max: Yeah. Fa fast testing would be really awesome, . But then then, then such a a test would take Hs. and yeah, Amber. Amber already takes quite a long time and and loves your, your, all, all your course of your, your machine . And so, so we, we are working behind the scenes on different other projects, and one, one is a fa automated faster.

But this is not something that is directly into Ember because it, it it just will take too long to, to bring you the results and I, I think this, this need to be a dedicated project where you can decide which firmware you want to, to fasten after your, your initial firmware analysis process.

Mm hmm.

Randal: And I also, my friends would kill me if I didn't ask this, Why bash and not a more capable script writer?

Max: Bash is the most beautiful language No, no, no, not even close.

Max: just kidding. Wow. Bash is so practical. Think about the beginnings, you're chaining tools together. You're starting at the command line interface, you're chaining tools, and then you're Pasting it into a script.

This, this was the beginning of Ember. And so before we thought about Ember by itself, we had a thousand line bash script. And then we, we, we structured it a little bit and it was just working. We gave it a nice, a nice and shiny modular structure and with more lines of code and it was working.

And Well, everyone said that everything will break if you, if you hit the thousand line area of your bash script and No, no we, we are maintaining it till, till today and we have now tens of thousands of lines, lines of code and, and it works really smooth. You can't, you can't really believe it. If you're, if you're researching in the internet, you will read everywhere that, no, it's not possible.

You can't maintain it. You can't do this. It's possible. It's possible. It's possible.

Benedikt: I have to say for me, it was also the first major question I had when I came into the project. So yeah, because like, I always thought like that there might be an idea to, to switch once you want to modulate and put like, put it into.

Building blocks where you, we were always talking about framework and stuff that we introduced different languages and stuff, but like, I'm, I'm always astonished. Like it's working, like we do some stuff, but

Randal: I would be astonished. It's working. So, okay. That's enough bashing bash. I just wanted to get that in there.

It was like, It's like, okay, you know, I could, I could name another four letter language that would actually be pretty useful, but I'm going to stop that for now. We talked with him just

Jonathan: last week. That's Java, right?

Randal: That's not a scripting language. It is! Go, go and

Jonathan: listen to last week's episode, J Bag, they made Java a scripting language.

Randal: Oh, no, no, just no, no, that's just, that's just, that's just, that's just, that's just That's just wrong. I'm glad I didn't watch last week's show. I'm glad I missed it so far. I will read the transcript now though to see what you're actually talking about, but I don't think I'm gonna like it.

Max: Probably we need to think about rewriting AMBA in, in Java script in English.

Yeah, yeah, yeah, right. Well,

Randal: remember now, JavaScript is not Java, not even close. Those are other ends of the spectrum. Yeah. So, so what, what are your biggest concerns now with the project? What are you working on most intensely? And where do you see this going in the future? Mm hmm.

Max: From, from an AMMO perspective we, we can, we can see that the, the world screams regarding S bombs.

So everyone needs S bomb everyone thinks about S bomb nowadays and we, we, we ha, we already create an S bomb but there is a lot of room for improvement, I would say. And this is, this is, let's say, the, the one, one of the short term goals that which we're currently working on to improve our S Bomb capabilities so that let's say we can, can position EMBAR much better for, for the near future.

Yeah. And.

Jonathan: So I, I'm curious about the the, the answer to that same question, but on the, on the other side, is the other part of the program, EMBARC? Yeah. So I'm curious about that same question, but applied to embark, like what, what is being worked on there and what's coming. So

Benedikt: the fun, the funny thing about Embark is always that it's trying to incorporate everything that we're working on into one front end, let's say.

So if we're talking about the whole. vision we have with Ember, we always talk about from finding the right firmware from getting it to the end to the fuzzer we just talked about, like that might be the end goal. Once we had all the already known vulnerabilities and everything, and then we end up with a fuzzing framework or something, which Mike could talk about if he wanted to, but In the beginning, we would have something that is taking firmware from a device, which we have some projects going on.

Basically two of them, the, the one of them would be to get firmware from a device. And the other one is finding firmware on the internet. If you don't know where to look, basically like scraping and stuff. And the basic idea behind MBug would be to put all of that into one thing.

Benedikt: Oh, that's really cool.

That is basically the end goal. That's also, as you can imagine, a lot of work to integrate

Jonathan: those things. So one of the things that I think Michael mentioned is, is when you, when you give a piece of firmware to EMBA and to the Embark to the server, it really crunches on it. Like you said, you can just, you could fire your CPUs up for the whole weekend.

I'm, I'm curious, what, what are we doing with firmware that's requiring so many CPU cycles?

Benedikt: The thing Mike said about a weekend, yeah, that's, that's the new and the, the new shiny thing that's coming or that's, that's behind the curtain. We usually talked about weeks, months. If we, if we throw data sets of firmware in there to, to find. What's working, what's working with Ember, what is, what is it, is it finding, what is it not finding?

Nowadays it's, it's gotten pretty fast even though, and the thing with Ember is, is it's just multi threading and running modules, so multiple things in parallel, so it's taking basically everything. The, the machine has everything the CPU will give you. RAM wise depends a bit on where in the, in the Ember run it is, how much it will take up, but it really takes up everything you, you, you, you throw at it, but that's also the nice thing about bash.

Jonathan: Yeah. Yeah. But what, what is actually doing that is so CPU intensive, right? Are we just or excuse me, are we just un unzipping firmware and then doing comparisons looking for if there's CVEs out there, or are we actually like trying to run all of these binaries through Ghidra to do decompilation?

Like what, what, what is it that is getting done that's so CPU intensive? Both.

Benedikt: Okay. That's fair. Because you said unzipping is, is one, one part, right? And then decompiling is. Further down the line. Right? So, okay. So those are, we are,

Max: we are doing, doing more or less everything. It's, it starts with, with extraction of the f we, we are doing some, some binary analysis.

So which, which legacy functions are used in such binaries. We are decompiling these binaries. We are then using in Gira. For, for further analysis on the, on the extracted source code, we are using then Semgrep for static analysis. Regarding, regarding the Hot Topic SBOM, we are not just reading the DBN database.

We are also extracting or reverse the SBOM from the, on a binary level. So that if there is no packaging system on, on the device or on the firmware, we, we also get the SBOM out of it. Which most of the other tools do not do, and this is, this is quite, quite intense because we need to analyze every binary, every library and compare every all of this stuff against the quite huge I would say data set of possible version identifiers that we we, we, um.

Extracted manually and we built up manually and at the end we are comparing then these, these versions that we identified with the, with the CVE database, for example, to get an overview of how many, how many let's say known vulnerabilities are available for, for the, the identified. binaries and libraries.

If you think about the firmware, a small firmware of your home router has a few hundred files. If you think about the modern firmware it's getting bigger and bigger. So we have to deal with more we have seen things that they call embedded devices, but it is a full blown Ubuntu distribution out there.

And if you throw this static analysis mechanisms like decompiling every binary, like matching, I would say seven or 800 version identifiers against every binary. This is, this is highly resource intensive and and the good thing is the more CPU cores you have in your machine and boys is automatically using, let's say, or optimizing the next scan on the number of course you have so that.

It, it squeezes out your, your, your machine as much as possible.

Jonathan: Yeah. Yeah. Makes sense. All right. I know something that some vendors are starting to do. I guess they've been doing this for a long time and that is shipping encrypted firmwares. Is that something that there's, there's any yeah. Oh yeah.

We have to, we've had to deal with that. Yeah. What, what is the, what is the approach? Does, does EMBA have any tricks to try to get into those?

Max: I, I would say that we, we are using, we are, we are using on the one hand the, the classical extraction frameworks like BINWALK or UNBLOB. We also have introduced multiple dedicated extraction functionalities for such reasons where, where some some encrypted firmware was documented or the decryption was documented somewhere in the internet and, and we have implemented it directly into EnBAR.

If it's not documented and nothing, no, none of the known extraction frameworks can handle it, then at the end you need to go back to the hardware. You need to go back to identify a debugging interface. He sold a flash and usually, usually my task is then give the box to Benedict and say, Oh, I, I, I need to get access to the firmware, please.

Yeah.

Jonathan: Do you guys

Max: offer

Jonathan: like any, any sort of consulting or work services where someone could say, I need to break into this box. I need to get the firmware off of it. Can I ship it to you and you desolder it and extract the firmware for me? Is that sort of in your wheelhouse?

Max: Yeah, but just company internal.

No, no, no, no, no, no, no, no external consulting. We're just working Siemens Energy internally and testing the stuff that is relevant in this area. But not we are not offering such a service to external.

Jonathan: Sure. And I imagine trying to do that, you would run into all sort of legal questions, right?

About, I don't, I don't actually own this piece of hardware, or I don't have all the rights to this code. And then you go in and you go start decompiling it. You. There, there are, there are questions there, right? Randall, there are questions you run into when doing penetration tests.

Randal: Well, the letters DMCA immediately popped into my head.

So that was definitely going to apply here. Yeah, you don't want to be you don't want to, without the proper chain of authority, you definitely do not want to be doing pen testing at all. It's a, it can lead to. Unfortunate consequences. I'll just put it that

Jonathan: way. Yes, and for those that don't know you can go check out randall's wikipedia page and get a little more details about What happens when you don't have?

All of your I's dotted and your T's crossed and you do a little more penetration than your boss has thought you should have.

Jonathan: For a little while, it

Randal: looked like this.

Jonathan: Yeah. Yeah. Yeah. It's not, it's

Randal: not good.

Max: All right. So what about Not a comfortable position. No, no. No, it's definitely

Randal: not comfortable.

It hurts your, hurts your wrists. Let me tell you, I do know.

Jonathan: So what about virtualization? Is, do you have any sort of a a mechanism to take the, image that you've got now, you know, now that it's taken apart and run it in something like a virtualized machine to be able to do tests on it there?

Max: I feel that the, the original idea from AMBA was always we are, we're talking about firmware.

So this means that we, we need to, or we talk about different architectures. In, in firmware, you, you do not have just the, the, the, the, the X 86 architecture. You have some, some mips where you have arm architecture, you have power pc and all, all of this crazy, crazy architecture. So you, if, if you want to run something from this you always need some something that is called an emulator.

So that. That can, can run code from one architecture in your, in your, let's say, in your analysis machine, like your Kali Linux. So you have, have the guest which is the, let's say the binary from the firmware. Let's say BusyBox binary, you want to run a BusyBox binary for, which was compiled for a MIPS architecture on, on your Kali Linux, which you're using for analyzing the firmware, which is some, some x86 architecture.

And then you need an emulator like QEMU. And Amber is doing this. In, in multiple areas. So for example, we, we, we try to run every binary in something that is called user mode emulation. To, to drop output to the command line interface to collect the output and then to analyze the output if there are some version identifiers in it.

And this is one of the approaches that we are using. And the second one is, is something that is called system emulation. And in system emulation, we are trying to bring up the whole firmware with a, with a prepared Linux kernel. We're trying to boot it up. And to give it with a, with a few tricks, something like a, a, a kickstart so that hopefully the firmware is able to, to, to set up the services set up the, the, the network interface so that we can then, then interact with the firmware.

Jonathan: I, I am, I am kind of floored, actually. I expected the answer to that to be, to be, no, we're not doing any virtualization and you guys already, you're way ahead of me. It seems like a very kitchen sink sort of deal where you've just thrown everything in. It's, it's impressive. It's not even mentioning the

Benedikt: docker by itself.

Yeah, yeah, he, you, you skipped that one, Mike. No security thing with the docker. Yeah,

Max: we, we, we have skipped so many things till now,

Jonathan: but we can

Max: dig,

Jonathan: dig

Max: quite deep.

Jonathan: So maybe, maybe this is kind of what you're talking about there, but in doing things like emulating the firmware and all of that, I would, I would honestly have, have to have the consideration, like, What if there is something malicious baked into that firmware?

Are you giving that malicious code a chance to run on your infrastructure? Like, do you have any safeguards against You know, something nasty inside the firmware. Is it, is it all properly sandboxed? I guess is the way to ask that.

Benedikt: Properly? Wow. That's, that's, that's the good question. Yeah. Is it properly sandboxed?

I mean, it's been an ongoing theme, I think, for, for multiple years that we've tried to safeguard. running code that like, that's like malicious because the doc environment for within Ember needs permissions. So we're still trying to, to really get behind all the ways it could still get out. I think we've, we're doing a pretty decent job, but in the end it's, it's definitely like big disclaimer, do not run something if you know there's something.

In there maliciously targeting your host. I mean, the thing we always do is like we run it within virtual environments. It has a Docker environment. And then the thing actually running is inside a QEMU, but you know.

Jonathan: Layers, layers upon layers, that's something malicious would have to get out of. So I assume, I

Randal: know why it's slow now.

Max: That's the reason we need so many cores for every security layer.

Jonathan: That's, that's not a joke. Goodness. And I assume there's things you're doing inside of there that wouldn't work inside like a rootless pod, man. It's got, it's got to be Docker with root.

Max: Yeah, yeah. We're, we're doing a lot of let's say we're doing some things that we need, where we need root access.

We have thought about Portman now for years, but at the end of this, it is a time issue. We need to test everything. We need to test a lot of use cases, find out what, what is working. In, in this huge code base within Portman, what, what is failing and let's say every, every fail is probably some, at least some little research project on how can we, how important is it for Amber?

Or for, for the testers out there, can, can we just strip it out or can we bypass it somehow? And this, this costs quite a, a lot of time. And so we decided Amber is a research project from this perspective. We, we have multiple layers of, of protection, but at the end at the end, an attacker can analyze, can, can check out the code, can find the bugs and can exploit them if, if he, if he wants, I'm, I'm quite sure if it, it is possible in, in, at least in theory, it is possible to, to exploit it and, um, yeah, if, if you, if you get ephemera from an untrusted source, if you, if I send you ephemera, then you should definitely throw it into Amba,

Yeah.

Randal: It should just look for the string, amba vector or something. Yeah. . Yes. Not run the code if it finds that, oh, if only we're that easy.

Jonathan: Randall, you want to cover some of these that you've got stacked up?

Randal: Yeah, yeah. I was just going through my notes things I'd taken before I did this show. So you know, AI is a big thing. Everybody's saying AI. Is there any idea that maybe you're going to introduce some AI to this? Help fill some of the gaps in, or help extend what you're doing, or maybe try a more brute force approach.

You know, is there any way that large language models or even just neural nets can help you out with this stuff?

Benedikt: Let's say indirectly we had that. Okay. What is indirectly? We had that feature for a long time. I can tell you the exact date because I don't remember, but we have a whole section of modules that we introduced and one specific module was for an open AI connection.

So you're able to put in a API key and then Basically what Ember does is with decompiled code and especially if we've script languages, I think we have this option to ask the AI bot. If there's any vulnerabilities in certain code snippets that are previously identified by another tool.

Jonathan: Oh, interesting.

Benedikt: So there is also like a lot of expansion stuff that's possible with that. You can change questions, whatever, because. If you wanted to do that the results, because I've been testing it a bit, the results are, well, do they really give you more? That's the question, I feel like that's the real question.

Yeah, the false positives there, the false positives there are probably

Randal: pretty high. Is it, is it

Benedikt: telling you anything that other tools can't, like? Yeah. That's

Randal: I'm a little biased because I'm a Google developer expert, but I'd also suggest looking at Gemini's current products because they're really moving far forward and they're recently trained.

Which means that they're much more up to date than any of OpenAI's offerings at this point. So, you might take a look at that. Especially Gemini Pro 1. it. And it's really got some amazing results so far for the stuff I've been throwing at it. So just, just an idea there. Let's see. I also had a couple more questions.

Let's see, what else do I want to ask here? So, is this just about Linux based firmware, or are you planning on expanding it to other operating systems, other firmware types?

Max: Hmm. Is there

Randal: anything else?

Max: So that's, that's, that's, that's, that's the question. No. So, so in, in the area we, we are located. We are, we are using Amber.

We have to deal with a lot of Linux firmware. Nevertheless as I said, Our usual approach is to throw a firmware before of a, of a manual penetration test into AMBA. And AMBA gives us a few days later the, the, the results. We, we also have sometimes some UEFI firmware. We have sometimes some RTOS such a VX works or something like this.

And we want to get the, the maximum out of it. Although we are not the experts in this area. So, we, we, we do not want to, let's say, we, we do not want that Amber is crashing because there is no Linux in it. Then Amber should, should, should give us the information that there's a VX works. And we are doing a lot of or we have a lot of modules.

that are, let's say, file system, system independent, so you can also use it just on a binary blob, for example, to, to extract some key material or to find some, some, some interesting hashes. And then Amber can do this also on, on a, on a, on a non Linux operating system. And additionally, in, in the future.

field of, of UEFI, so, so we, we do not deal a lot with UEFI analysis, but there, there is a company called Binary and they have released an UEFI security analyzer as open source. So we have just put their complete open source scanner included it into Ember. Wrapped it around an ammo module.

And if we, as soon as we detect any UEFI firmware then we fire up this vulnerability scanner now. To, and so we are also able to provide some, some kind of value also non Linux operating systems.

Randal: Cool. Cool. And one last question. I'm gonna throw it back to Jonathan. So results, what's the most unusual thing you've discovered with this or just what are some of the practical outcomes of this that make it worth your time?

Randal: you must have discovered something with this or you this. They're now,

Jonathan: they're now having to do the, the question of, of the things we found. What are we allowed to talk about? . Oh, oh, sorry. I didn't realize that would put you at risk. So, so,

Max: so, so like we, we can say we, we are talking quite generic, so nobody knows what, what we are talking about, so, okay.

I think they, it's, it's very often really interesting if you're, if you're analyzing established security products that are established there for years now and you're, you're analyzing them and you can see that the, the, the components that they are using are, are from, from the age of dinosaurs.

And that they're, that they're not security components and they're not updating the, the, the product anymore. Or the, the base product that they're operating on, on the operating system level. And they're, they're just managing the risk, let's say this way. And this is, this is really interesting because often these are established vendors, established products, and then you can see that not only D Link is using a kernel 2.

6 dot something, also very established security company is doing the same. Wow. Yeah. I think there, what, what was it that there was, I cannot remember exactly, there was one one, one company or one product. That was destroyed somewhere in the internet. I don't remember it now, but, but, but,

Randal: but if I could summarize what you're saying is that some people who should know better don't always demonstrate that they do.

Yes. And, and with, with,

Max: with automated firmware analysis you get a quite good overview without any manual work. And then, you know, that's usually the, the, the thing where we are using Ember a lot. Ember is doing the automated work and we can see, okay, oh my God, this product is looking really fishy.

So we, we, we have 10, 10, 10 products. And then we know, okay, we start with. This one, because this looks fishy.

Randal: So it's like, when I'm looking at a piece of code that is operating on a live website, that's written in PHP and has a very obvious SQL injection attack. And I go, how long has this code been in here and this company in particular should know better to run PHP code.

That looks like this. Y'all. It's, it's, it's, it's a sad experience. It requires some warning on my part, and then I move on and go, well, I'll probably write code exactly like that someday. Yeah.

Max: And, and, and usually these are the windows that are shipping encrypted firmware updates so that nobody can see the, the, the PHP code.

It's encrypted.

Randal: We don't have to care about it. They don't want you to see how bad it is, that's why. They're just shipping junk under a golden label. Yeah,

Jonathan: it's sad how much that happens. Alright, so, I've got to ask, why open source? Right, like you guys apparently use this at Siemens or Siemens Energy. It's it's used for it sounds like for a lot of internal use What was the what was the conversation like saying?

Hey, we should open source this and release it to the world

Benedikt: I mean it started as a project where you're like, okay, I have an issue and there's no tool for it I really feel like that's a classic open source thing, right? You you're right. You're starting free time. You you start with a An idea, and then you write something that you actually need for your work and then you're like, okay, I feel like since there's nothing out there, why not just open source it, like put it out there.

And that's, that's basically what happened with Emma making it like a security tool for everyone. That's also a big part of the cyber security idea. I feel like that's if you write a security tool, you put it where everyone can use it.

Jonathan: I like it. I know, I know some corporations, they, they get real antsy about releasing internal tools as open source.

Was EMBA started was it started in, in your spare time or was it sort of kind of official? It's like a company project.

Max: I would say combination. So we we we started during penetration test and then every every you're fascinated you want to do everything and then You're running into time constraints and then you you can't sleep.

So you start coding in the night and Is it no work time? No, it's not work time because you have already Your, your, your, your hours together, so it's free time. So, so pro probably most of the guys out or the open source developers out there can ha have a little bit the same story. Yes, they're fascinated.

So where, where at the beginning you can say where is company time, where's free time and everything swims together. You want to solve the problems and yeah, I think it's, it's, it's a mixture of both. We, we have the time in the company, but we are, we are also spending quite a decent amount of time during our, our free time.

Jonathan: Yes. I know that feeling waking up at two in the morning. I know what's wrong. Then it's like, do I try to put myself back to sleep or do I go in there and fix it?

Max: And then you need to go up because you can't sleep anymore and you need to fix it. And then you realize, Oh I fixed this issue, but there's the next one.

So I don't need to go to bed anymore.

Randal: I think what's worse than either of those is waking up, realizing you have the solution to a problem and not writing it down and forget. That's really bad.

Max: That's true. Yeah.

Randal: I, I've done that. That's why I know it's not a fun thing. Yeah. I try to get at least something written down when I have an idea in the middle of the night.

So I at least have it for the next day.

Jonathan: Are there any, are there any CVEs out there that have a special thank you to EMBA? Do you have like a list of you know, this one and this one and this one were found because people were using EMBA?

Max: So, so We have an internal list of our CBEs that were fixed that we, we reported to other vendors and that were fixed.

Mm-Hmm. , but there's nothing out there in the, in the public. No. Okay.

Jonathan: That would be, that would actually be an interesting project to just have a repository. Now, now I understand like some of the things you find that you just can't, right? Like I get that there are things that will get discovered internally reported internally and may never be assigned a CVE number.

Like that's fine. That's just how the world works. But it would be very interesting to have a repository where you say, Hey, security researchers, if you find something, if you find a CVE through EMBA. Shoot us a note here and they just have like a high score list somewhere on the Ember website. I think that would be a lot of fun.

Oh yeah.

Max: So, so, so we, we, we have a collection of all of the papers and blog posts and mentioning of Ember. So If someone sends us the CVEs that they have found with the help of Amber, then they will definitely get a nice and shiny place there. Yeah.

Max: So, so we, we are prepared for these reports.

Awesome.

Jonathan: So one of the things, so I, I write about security and one of the things that I try to encourage people like just that are interested in it is the barrier to entry for finding CVEs for finding security problems and getting them fixed. Okay. It's actually a lot lower than most people think it is, right?

Like it's, I, I just, now I didn't get assigned to CVE, but I got paid a bug bounty for literally just running a trace route and then running a port scan back when I had Starlink Internet, and they had set, they had set a server up and it. did not, it did not have a proper firewall if you were coming from inside the Starlink ISP.

And I got paid several thousand dollars for reporting that. And this is one of the things that I, I really like to encourage people on, is like, go looking for things, and if something seems weird. So all of that said, what does the process look like for setting up EMBA and EMBARQ? How difficult is it to get started and then say, feed the latest firmware for your, from your own home router into it?

There's obviously, there's going to be problems in home router software. How difficult is it to get started and start finding them?

Benedikt: I would say very easy, but maybe I'm biased here. Of course the easiest way is for, for us just tell you, okay, there's a Kali ISO run a VM on your laptop and then run it.

Just do the git pull git clone and to an installer which is very easy takes a bit of I don't know a few minutes for sure to download and install. The barrier of entry is basically Now I would say because you can also do it on bare metal You can install kali on basically anything I would say the only barrier I would see is Minimum system requirements because Ember is, as we talked about, a bit of beast.

So it has some minimum requirements where you, you can't, like, it won't run nicely below that.

Jonathan: Yeah. So, and so is, is Cali kind of the preferred place to set it up and start using it?

Benedikt: Well for Embar for sure. I feel like Mike and I, we always are on Cali. Mm-Hmm. Em, embark is designed for Ubuntu. Okay. Because if you, if you're talking about Linux environments and servers most people install Ubuntu, right? So that's, that's what, what is it? It's intended for. So EMBA would work on two yeah, well two Ubuntu versions and Ali, the current Ali.

So there's options.

Jonathan: Alright. We are I just saw at the time we are rapidly running out of time and I, I wanna make sure and ask about community. Like, so what, what is the, what is the community size? Like, do you guys get contributions? Have you had people from? outside of your company and outside of the project come by and say, Hey, I thought it would be cool if EMBA could do this.

Here's, here's the patch to make it happen. Or, you know, bug reports. Is that something that is happening or is it, is it kind of not public enough yet?

Max: So, so I think, I think we're getting more and more bug reports during the, let's say during the last year, we are getting more and more insights on how people are using EMBA.

How people are. Talking and writing about it. People are also reporting their bugs and their wishes. We, we had a, a, a few contributors that there, which are directly fix have started fixing the bugs but really just a few. So the, the the, the, the community is growing, but let's say the community that is interested in also helping is growing very slowly.

But the, the more the community is growing I'm, I'm sure that then also the, the, the helping hands will, will come more and more. Sure.

Jonathan: So if somebody wants to learn more about EMBA or EMBARQ, where do they go to learn more? And then if they want to talk to you guys, where is the best place to do that?

Right

Randal: here.

Jonathan: So like, what's, what's the, what's the website? What's the GitHub URL? Is there a Discord server? That sort of thing.

Benedikt: No Discord yet, no. Not that I'm aware of. That's something we should talk about, Mike. Absolutely. Thinking about that, yeah.

Max: cu Currently the easiest way is going, going to the GitHub project. If, if you have ideas, open an issue, open a discussion.

If you, or if you're writing about Amber or showing Amber or something around Amber somewhere, drop us a note somewhere at GitHub or GI or X or LinkedIn. Mastodon the usual social networks. Mm-Hmm. . We, we have there an, an dedicated Ember account, which is called Secure Firmware. And we, we are happy to, to, to publish also your, your papers that I'm mentioning or your talks that I'm mentioning Ember there.

And If, if we need to talk in, in person, like we did it and drop us a note somewhere over there and we find a way to, to get in contact. Yeah.

Benedikt: And I think, Mike, you're at the next, no, conference, the big one in Oh, yeah, yeah,

Max: we, we, we have plans going to, to Black Hat Middle East now. Ah. I think in the end of November.

Okay. So if someone is there, then drop, again, drop us a note and we, we have there a, a, a demonstration of EMBA at Arsenal, and then just come to our booth and we can, we can talk about EMBA and Fermi analysis and all the rest of the world.

Jonathan: Yeah, very cool. All right. We are out of time i've got to ask each of you the final two questions.

It is a requirement. We get in trouble if we don't do it we'll start with michael. What's your favorite text editor and scripting language?

Max: So so scripting language is is definitely bash. I I've now written 30 000 lines of coding bash. So it it needs to be my favorite scripting language. Yeah, and I always exited WIM after working on AMBA, so this is my favorite text editor.

I'm not a master of WIM, but I can exit it. You know how to get

Jonathan: out. It's fine for me.

Max: Yes. Save and exit, not just exit. Yeah,

Randal: hopefully. All right. If you're not saving it, it starts sticking.

Benedikt: All right. And Benedikt, same two questions. Yeah. For me, it's somewhere between Python and Bash. More fluent in Python though, I should say that.

And editor wise, I would say, yeah, not, not the biggest fan of him. I learned it. Normally I stick with Nano. That's fair.

Jonathan: That's fair. Alright. Thank you guys for being here. I wish we had more time. We'll have to bring you back maybe after, maybe after Blackhat. We'll bring you guys back on and talk about what's changed and get in all the questions we didn't get to today.

But thank you both so much. Thank you for having

Max: us. Have a good day.

Jonathan: Yup. Yup.

Max: And

Jonathan: bye bye.

Max: See

Jonathan: you. Alright. What do you think, Randall? You gonna go dig the firmware out of your router now and take a look and see if you can find some fresh CVEs?

Randal: Well, it's not my router, which is why I'm on Wi Fi. So It's I live in the basement of a, of a two main story house and they have the router upstairs.

And so that's why I don't have, which is why we were in troubles before the show. Yeah. Getting me on the screen of why I dropped it. After we're done taping, I've got some advice for you about how to set up the next show so that it's a little better bandwidth wise. So this was interesting in that I haven't spent a lot of time thinking about the firmware that's in all the devices that are now being used in my life.

And now it makes me more scared. Great. But at least there's some sort of diagnostic tool that at least somebody can run against some release of maybe something that I will use or eventually use or be kept from using because they found bad stuff in it. I like your open source question. Cause it's like that, unless they're getting a lot of community Contributions.

This isn't, this doesn't really pay off as open source. Unless most of this stuff was already open source and they're just bringing it together and kind of from a potluck perspective where everybody's bringing food and everybody takes food. Home, it sort of makes sense for it to be open source. So I like, I like it from that perspective.

It was a little out of my wheelhouse, so I didn't get a lot of questions. So

Jonathan: Randall, what you need to do, if you decide to start doing that, start working with this at all, is write yourself a custom module to see is any of your code in the firmware and just like make some sort of trumpet fanfare that plays automatically when it finds, you know, a Swartchian transform or whatever that, that is actually in there.

All right.

Randal: Right. Well, it's like, you know, the usual editor question, you know, for years, I said Emacs to that because there's a part of my code in every copy of GNU Emacs. So that kind of thing. And, and Yeah, Schwarzschild transforms show up everywhere. It's kind of crazy how that how that one's gone kind of spiraling out.

So, yeah yeah, cool. No, I was, it was a, it was a good show. The guests were knowledgeable and the project is worthwhile and in progress. Yep, absolutely. All right. Do you have anything you want to plug? Ah, yes. So so this is now a new addition to my semi regular schedule as often as Jonathan will let me back.

And, and I've opened for the day. Cause so typically my, my Tuesdays are pretty free. So I think it'll probably work out for at least as often as Jonathan's willing to put up with me. But every Wednesday I tape a live stream show as well. On Dart and Floss, which are my, not Dart and Floss, Dart and Flutter.

Ah, too many FL words. Dart and Flutter. I am a Google developer expert in the Dart and Flutter arena. I'm all in on Dart and Flutter. So, yes, although I was kind of hinting at Perl earlier in the show, I really, I personally haven't written more than 10 lines of Perl code probably in the last year. And but I've written thousands of lines of Dart code and Flutter code and, and things like that.

And so every Wednesday we do what we call Hump Day Q& A, Ask Me Anything, where we get a few experts together. And for some reason they let me on there. And I have no idea why. And we take live questions over YouTube. So if you go to YouTube and look for Flutter community and Flutter is anything vaguely interesting to you, please check that show out.

We really enjoy the live questions. We get some good stuff doing, we also do some live coding. So it's been a lot of fun doing that week for week for the last, I think, two, three years. So it's almost been the same burden that I used to have weekly, but now delivering it into the dart and flutter arena and the problem with the, not the problem, but the, one of the things to know about the hump day Q and a is it runs two or three or four hours long.

So. At about the third hour, I'm telling Simon, who's now madly typing in, trying to solve that last problem in his live coding. I say, Hey, we're starting the fourth hour there, Simon. And he'll finally go, Oh, I should probably wrap this up then. Yeah. Why don't you? Why don't you? We're down to 22 people watching, so we should probably stop.

Oh, that's great. Fun show. It's a great show. I love it. I'm, I look forward to it every week. I'm just making notes now about what we're trying to do tomorrow. But check that out. That's probably the biggest place to see me. I also have a dart and flutter YouTube channel, which I try to contribute to you on a regular basis.

So. You want to check that out. Just, just Google Randall Schwartz, Dart and Flutter. You'll find all sorts of good stuff about me and my new career.

Jonathan: Yeah. Excellent. Thank you, sir, for being here. It is good to have you back and we will make sure. We'll make sure and get Randall in the rotating cadre of co hosts.

So we'll see him again here in a few weeks and look forward to that. If you want to find my stuff, of course, Hackaday. We appreciate them being the new, the new home of Floss Weekly. You're not really new anymore. I've been here for almost a year. I've also got the security column that goes live every Friday morning and then still over on twit We've got the untitled Linux show, which is a lot of fun And if you're into Linux, which I imagine most of our listeners are you should go and check that out, too We appreciate everybody being here those that catch us live and on the download and we will see you next week on Floss Weekly

Episode 801 transcript

18 september 2024 | min

FLOSS-801

Jonathan: Hey folks, this week Jeff joins me and we talk with Max Anderson about JBang. It's a little utility to make Java easier to use and run it as a scripting language if you really want to. It's a lot of fun, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 801, recorded Wednesday, September 17th.

It's not your parents Java anymore.

It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today, it's going to be fun. It's going to be interesting. It's going to be Java. Which That's okay. We mostly like Java here. It is not just me. Of course I have, I have Mr. Jeff Massey, the one, the only Mr.

The, the other, Mr. Lennox here with us to talk about J Bang, which is an interesting project all about, from what I could tell, it's using Java as a system scripting language, which is sort of interesting. What, what do you know about this, Jeff?

Jeff: Not a lot. When you, when you first got a hold of me, I thought for a moment you were talking about JavaScript.

Well, it sort of

Jonathan: is. It's Java space script.

Jeff: Yes, not one word. So that's where, I was a little confused at first. So I'm, I, I'm not a real I don't have a lot of experience in Java. So very, very little, but I'm interested to hear about this and kind of the direction they're taking it.

Jonathan: Yeah. Now we talked before the show, it's like, well, Couldn't you just do this with like an alias to the Java binary?

And so, you know, JBang would just then be an alias to Java and that's the show. Thanks for coming, everybody. We'll see you next week. Apparently there's something a little more to it than that. It's a little more complicated. There's more wrinkles, there's more hair on that problem. So, looking forward to learning.

The level's always in the details. Yeah, always, always. Looking forward to learning about that. Let's not let's not take any more time to bring him on. We have Max Anderson is the guest today, the, I believe, the creator of J Bang. Welcome, sir! Welcome. Well, thanks for having me. Yeah. Now, creator Jbe, right?

You're,

Max: you're the man. Yes. I created it after being a year away on a sabbatical, not doing anything not doing anything, work for latest, so to speak. Right. And yeah, and I, I'm happy to be here and, and be in a place where everyone loves Java and wants more Java and all that

Jonathan: kind

Max: of

Jonathan: thing. Java was I've, I've told this on the show before, but John Java was one of my early programming languages.

And so some of the like difficulties and learning how to program and like, what do you mean a pointer error? And why I didn't even think Java had pointers. Why is the log telling me about, so some of that, like, you know, I just, I have this sort of deep seated annoyance with Java because I was programming it when I didn't know what I was doing.

Max: Yes, I see that repeatedly again. People have, well, most people have learned or touched Java, but they mainly have touched it at a time either in their career where they were not, like, that was not a good, it was not a good learning language in the early stages. But it got super popular because it could run anywhere, right?

It could, you could run both on Linux and Windows and eventually Mac and that kind of thing. And phones and, I mean, for a while there were DVD players and everywhere, yeah.

Jeff: Everywhere.

Max: Yeah, it was, whatever devices were there, right? So so I worked in Java for, I've done professional open source 20 plus years.

Mm hmm. I've my day started at Hibernate and, and, and Persistence Solution, that kind of thing. And then I started doing tooling. And then I, I had, um, what's it called? Well, 10 years, 15 years of work and came back. Well, then when containers came around, Java had a problem because it's too heavy and, and, and, and and I was tasked on doing some other tooling in go land and JavaScript.

And I got to touch on that. But then I, I took a break for a year. And I promised myself I'm not going to touch any programming for the least three or four months. And on the first day I break my ankle and have to be not moving for another days. I decided not to touch Java, but go into Python instead.

I knew Python from like 20 years old. And I had fun with that. And then when I came back to actually work on a product called Quarkers, which can make Java work with containers, I was like being super or realizing how complicated Java is like in the minds of people. But when you refresh yourself on what's actually happening in the last X number of years, You realize, hey, there's no reason why it should be hard to use Java.

And I then, as a way to update my knowledge about Java, sat down and said, hey, let me try and make like, there's this the product Quarkus was using in their release engineering, release scripts, they're using Kscript, which is, it's Kotlin scripted. And Kotlin is considered a more lightweight language, but I was like, why are we required to install Kotlin on top of this Java when the script is just doing some file automation stuff, right?

So I took the DSK script head and applied it to Java and that's where the scripting part comes from in JBang. And then I realized, hey, I want to have, The same experience I want to be able to, because the reason why I like Java is all the tooling around it, like the debuggers, the IDE, the content assist, the refactoring.

But the problem was that when you try and do that in a scripting way with JBang, All that tooling falls apart. And then it becomes at the equal terms as Python and Ruby or Node or whatever. And then Java doesn't hold up. Like, then those languages are nicer. But I was like, no, no, no, like we have a massive, like Java has a massive ecosystem, like the whole Maven central auto artifact, you can download and use and do whatever you want.

And and I said, Hey, let's, let's do this. And then I the, the thing was in Java, I forgot like 10 or 11, the add support for running. without compiling it. But the problem was that to, and you could even use it as a script, you know, like have a what's it called? She bang in, in the top, like what, what pound slash slash dash run something.

but any IDE you do that in will then see that as a syntech error because that command is not part of the job eco system. So no, IDE could work.

Jeff: Mm-Hmm, . And

Max: I was like, oh no, we need to find a way. So that's when I realized there was a trick in go. I think I saw it first, that if you add two, four slices bash and seashell and a few other shells, not fish, shell will treat that as shell commands.

So you can then run it and pipe in the file and Java will just treat it as a command sorry, a comment. And that means you can run any Java file and it'll work in the ID. You can come to this, it works. And that's like, Hey, great. Let me do that with the Java eight, because at the time I was using Java eight, but you couldn't so fast forward, I made it.

So there's a J bank command that you can call that can run Java source file. And. Compile it and then run it. And the advantage over the default job is a, it will cast the result. So therefore it doesn't have to do the compilers already there

Jeff: and

Max: only do the compiler into the file changes. And then the, the one that's the kicker is it can get dependencies that it's just in the file.

So at the top of the file, you can add like slash that steps and then the coordinate for artifact. So and that's, once you have that suddenly you can do things and it works in ID like in the early days I was generating files for ID, but now a few days later we have support in Eclipse.

Intel J and VS code, especially VS code. So now my favorite text editor is, is, is VS code with a J bang plugin in it because I can just be writing in my, my own language. And then on top of that, we started going low. Oh, Because that's the other big thing is, oh, you need to start writing code without having to download an IDE, et cetera.

You need to get Java, but which Java version are you going to run? And so then I added to JBang that it could download, or Taco, who I'm doing it with, added support for downloading. Any, the, the, the version of Java you're there. So now we had a full setup where you install giving which is a one liner install, and then you just create a Java file and you can add dependencies and you can build it.

And you can even, there's even a JBank edit that will actually, if you use your own IDE or download one for you and it's configured. And once we had all those things in place, we suddenly had an ecosystem, a whole setup that anyone can run on any of the three major platforms out there. And then suddenly I realized, hey, any student, any, any experiment I want to do, I can do in seconds compared to, I'll also set up as it was before.

Jonathan: It sounds like you're technically making polyglot files here that are interpreted one way by bash and interpreted another way by the IDE and by Java itself.

Max: That is true. And, and the fun part it's also a poly, Poly, poly, polyOS thing, because the first slash is bash recognizing, Hey, this is a bash command in the start.

The second slash is the Java comment. So it works. But the fun thing is that works on Windows, sorry, Linux and Mac. But if you do it on Windows. It gets interpreted as, I forgot what it is, but it's a UNC

Jeff: path,

Max: so it fails. But if you have a third one, everything's fine. It works in all three. So it's, it's, it's a, it's not a feat, it's a bug in all three that makes it work.

But mind you, you don't need that line. That is literally just for the feature of being able to do Like run the Java file directly to be able to do a dot slash on it. Right. Yeah. Yes. A lot of run the so, so that's, that's the, yeah.

Jeff: Sorry. I was going to say, you know, and I think you, you kind of touched on this, but you know, why did, why did you choose Java?

Was it strictly just because of the infrastructure, the existing infrastructure versus not going, Hey, maybe I want to try to work on, you know, Python or Rust or some other language, you know, So

Max: that was the thing. It's like my professional work life has been in the Java world. And I, I am proficient in the others, but I always, I was before I did my sabbatical, I was like, why doesn't people do that?

Why don't they just use Java more? Like it's so easy to use. Right. But I realized that my 15 years of working, I. I got immune to, oh, you have to find a version of a dedicated download. Oh, you have to find IDE to download. You have to find X, Y, Z. Learn a Maven tool or Gradle tool. And I hadn't realized. It was just everyone in my, like, bubble knew that it's like, like, and people found what's that called?

Like when, when, when you're a group of people who've gone through the same pain and now you're all kind of good and you don't see all the problems, right? And you feel like privileged to you, you, you've got here and then go like, Hey, why is this student that never used program before surprised to, to, to, to this and the, so one of the things you can do with, with well, so, so main thing is.

I like Java. Like, I'm, it's what I'm, the language I, I, I like to do. I also like Python. I like JavaScript. I'm not a fan of, but there's a lot of languages I like, but the whole ecosystem of Java, I just, the whole tooling, the compile, like there's a lot of stuff we can do that none of the others can, can, can do.

But the, the, the So that's why I was like, I was like, when I went on sabbatical, I was like, how hard can this be? Why is it? Is it so hard when I came back because then I had forgotten all these things and I was just Realizing how many steps I have to do so, coincidentally also at the time my son was about Seven or eight six seven.

So he got into minecraft at that time And mindstorms and lego stuff and those two actually uses java in in a setup and i'm like, okay Let me go and show programming with Java to my kid. And I heard about all these kids and, and, and, and teenagers using writing plugins for Java. And I thought, then it can't be hard.

Then there must be a guide in Minecraft land that is simple to get started. And I was surprised that people are even doing minecraft plugins because any product founder had like five or six pages of setting up java To get started. And I realized okay, so they they're going through all this pain Just because, hey, all of their friends are liking Minecraft.

So therefore, that's why they get there. But I was just like, why? And then I realized with the JVang stuff I had, I could literally write, I could, it was made, I made it trivial to write Minecraft plugins in, in, in JVang. Because now it's just a single file and you just have a dependency on the Minecraft whatever Minecraft plugins that we use.

And off you go. Stuff like Again, people don't, well, you guys, you don't know because you, you're not a Java head, so what do you call them? But one of the things I always find insane, like when you do a debug, a Java product. It is literally like a 80 characters. You have to type in of Java does agent connect server equals yes to whatever port.

And I was like, why it's the same. It's it's, it was fine when we 20 years ago, add was a feature. We didn't know how it should look like, but it's just stayed around. And in day bang, that is. D debug and it behind the scenes just do the, the, the, that connection string. And I just chipped along and anything I could find that was like this com, like unnecessary complexity that has organic groan, I just kind of chipped away.

And because I just use a standard Java tooling, it works on anything like, so it it runs on Java eight and artworks. So I could even make it work on Java six and older. I just, no one ever asked for it. So I didn't want to incur the pain, but technically it could. And yeah, no, no. So, so, so Java was just because, you know, that's the thing I like.

I might be weird, but, and there's also a whole enterprise out there that uses Java. And I was tired of hearing this. Oh, when you're using IOT, you have to use Rust or, or Python or, oh, you have, if you do front end something you have to use JavaScript. Oh, and because everyone now do use JavaScript front end, we have to use JavaScript at the backend too.

I have nothing against it, but it's just this like, Hey, like Java can actually do these things too. So don't, don't fall in. I have the same thing with the whole AI movement that everyone thinks, Oh, to use AI, you have to use Python. And I'm like, Hey so this is one, my gimmicky you can actually run J bank from Python now.

So if you have a Jupiter, if you have a Jupiter notebook somewhere Jupiter notebook environment you can do pip install J bank and it will actually, you don't have to do anything. It will set up Java in that environment. And now you can, you can use Java from any. Of those free services like cloud notebooks and others and get access dependencies.

And people don't believe me when I say so, but it is actually doable. So now I claim that Java is the most portable environment anywhere. And with JBang, it's the, it's the easiest thing to set up. Cause I don't know if you actually tried, have you tried to actually set up a Python environment on a Windows machine?

Jonathan: No. Oh, no. That is, that is horrible.

Max: Yeah, because this is the thing like just like the java guys are like in a mindset Hey, you just do these seven steps and you're fine python and node. js is is default installed on mac and linux But as a whole ecosystem windows, which is you know, i'm sad to say but it's the most installed desktop operating system out there Yeah, we're running python is not easy Funnily enough, JBang is one line and you have a full Java environment available to you and it just works.

That's actually really cool. I did think it was funny

Jeff: when you said Java was really easy because I'm like, I don't know if I've heard anybody utter that phrase before. You know, I've always heard there's a learning curve, but so, you know, I totally get who you're surrounded by and what you're in and, you know, yeah.

So, but officially. So I, so I can make sure I have this down. What versions of Java are supported? I know you said you weren't going backwards from. So,

Max: It JPEG itself is compiled and with Java eight. So it can run on any Java eight, um, VM and upwards.

Jeff: Okay.

Max: So, because what J Bang does is it like, it's trivially, it's the simplest product I ever made.

It literally, it's just a little J, it's a Java, it's a small Java app. And it, it just takes input and figures out, Oh, is this a Java file? Oh, I need compile it first. And then I create a Java C command line. And then I. I, I build the classes and then I jar it up and have a jar and then I run the jar. And if it's a, if it's a jar file, I just run the jar.

And if it's a Maven coordinate. I go fetch the Maven dependency. If it's a Kotlin file, I, I run the, the Kotlin compiler. If it's groovy, I do the crude compiler. And in that sense, J bank actually goes beyond just Java, but any JVM based language is, is in theory possible to, to, to in that way run. And because WK team generally are good at having backwards compatible command lines or four compatible ones.

It, it has been reliably working on any version has come out in the last four years. And the cool thing is J Bang knows which version of Java the user is asking for. So therefore, if for some reason, let's say Java 25 is going to break something, I can adjust the, the, the, the generation of the command line stuff.

And then it would just kind of work. So that, that's the way that this, this magically works.

Jeff: Oh, that makes it nice then that, so, okay. I I've got my Java version eight plus whatever. Now to make this a little easier, these complexity reductions that you talk about, are they. Have they made their way upstream?

I mean, are you making life easier for me?

Max: I'm actually, so I work in Red Hat and my main job is actually to work with some of the compiler guys and OpenDK team. And other things and I've been pointing them to some of this stuff, but don't get K team is, is a very like, like it's a really complicated machine but it's like really, really efficient.

Like it's a really, like it's, it's engineered really, really well, but they also use everywhere. Right. So it's kind of like, I think you had an episode about the core was it the core lips, the core tool about how backwards compatible that I think has to be open your case is in that ballpark, but they also want to tweak things up, but they are so they're very conservative, right?

So even if I propose something now, it's going to take. Literally years before it's there. So I haven't done a lot of it, but one thing I've Not sure if it was coincidentally or not But a year after I made the big splash with jbang and and created it And showed, Hey, you can run J bang. And another thing I can also run JCL, right?

So JCL is, is a shell is like this tool in Java where you just run. It's like a, what's called rebel, like one line at a time. I can run those scripts too. And I showed that, Hey, it's you. The only thing you need to have being run, it's just a single file. And. You didn't even have to have a class in around like the whole like public class static void main thing you, you, you don't have to, to, to do.

And then a year or two after to dedicate team has now come out with that, I think it's Java 22. There's a preview support for what I call a naked and naked main. So that's you, the simplest Java thing you do, you can do today. Now is. Void main system up from like, there's no, there's no arcs. There's no class.

There's no imports. They, they actually, that's the one thing that, that, well, it's a thing that they started to, to, to put in. And some of the flags I've been doing like the debug one I've proposed at least to the internal Red Hat team. And I'm actually meeting them this week where I'm now here in Zurich.

And I'm, I'm, I'm gonna. Talk to 'em. So, but again, , this, this, this is so hard to dis do because people are so, like, one thing is that they're used to it, but also just there's so many infrastructure that's Mm-Hmm. , like, if you start, it's, I have the, the advantage that this is not in Java, so therefore I can, I, I can, I can change things a bit.

But I, I do, I, I at least. Find priority. And I think I might have accelerated some of those decisions to simplify Java. And yeah, so I, yeah, I'm trying, but it's, it's going to take a long time. When

Jonathan: you, when you measure, when you measure your like install base in the. Billions with a B you want to be very careful about making changes that could break things.

Yes So you mentioned you you mentioned naked mains i'm curious with j bang do you do you support implied mains? Or when I go to write a j bang script, do I still have to define a main? so so,

Max: So this, so yes you do. And I'll explain what, but, but no, you don't. So there's, so one of the key things I wanted to, was to make sure that the IDEs keep working.

So I could technically, I could make JBang go look in the sources and see, Oh, this is just a naked main. Let me. Behind the scenes generate a a, a, a wrapper around it and, and behind the scenes do something. But if I do that, that job file would not work in any id 'cause none of the IDs currently supports that notion.

Right. So I did not do that. But what I did do, do what I did do is the JS shell support. So JS Shell is this tool that's been in Java since Java nine, I think. Where you can literally just run any kind of Java, like it's, it's a line based execution. And, but the weird, well, I'm not, I'm not saying weird, but a limitation of JShell is, JShell doesn't, are not able to take arguments.

It doesn't handle dependencies. And that's because JCL was built for tinkering. Like, it was like, hey, let me play around with something. Not for use as a main execution engine. But I realized, hey, with JBang, I have a way to run these. So they are still JCL scripts. But I actually handle the command line arguments and parse things in.

And then, so yes, you can do mainless with J Bang back to nine. It is. But it comes with the price that JSON comes with it, which is, it's a slower execution. It's not as fast as everything else. So, so yeah, so the answer is yes and no. So

Jonathan: this ties into something else I was going to ask about are you doing compiling down to byte code?

Like with Python, oftentimes when you run a script, you'll end up with a pyc file hanging around and it does that to speed up compilation. So it sounds like sometimes you compile down to byte code and sometimes you don't.

Max: No, as a, what's it called, conceptually it's the same thing I do. Right. So Python adds the, has these Pisces files.

It says, Hey, this is a compile time. And if, if the, if the file, the timestamp matches the file we will we will use that the Pisces and I do similar thing here. I just, I have a, there's a medical folder and your home directory called dot J bang, where I have a cache. And in there, there's a, there's a, there's traces of what you've been doing.

Right. And and that's the thing. So this is the, the, the, the, the, once people, people have a hard time grokking that it's that easy. But for example the main list, the main list, sorry, a naked main is available in Java 22 and people go like, okay, I can't use that. That takes too long because if you use the normal install, you be complicated.

But with JBang, it's just JBang Java 22. And all this stuff will just be, I'll download a JDK and it'll just run for you. And the same thing is what I do with the, the compilers, the bytecode stuff. I will generate bytecode well, compiler classes. But I'm using the standard. Java C tools, right? So I'm not, there's nothing J Bang doesn't do anything that Java itself can't do.

Right. So, so, so it, I'm very confident that it will be portable for a very long time. As long as the OpenIDK team stays portable, J Bang will,

Jonathan: will,

Max: will too.

Jonathan: What, what about, we talked a little bit about Java versions. What happens if somebody says, here's my J Bang script and it calls for Java. 11 and Java 11 is not installed anywhere on the system.

Do you, do you deal with that or do we just fail?

Max: Yeah, well, yeah. So the, the, Well, if, if you run J when you install J bank, there's no job, no job needed. You can use any package manager and stuff. So basic J bank has nothing. It doesn't even have Java, right? So just have a jar, but it also has a script. So a best script or PowerShell scripts, which is windows.

And that thing will go look for Java and it has a default thing go like, Hey, I'll use the default DDK. That is on your system in your path to run to run JVM. But if that's not there, we will go download, I think it's Java 17 now. There's a, there's an API in the Java ecosystem that's hosted by a, what do you call it, Fujie.

And Fujie has an API to go get all these different variants of JDKs for different platforms. So we, we recognize the system, the architecture, the, the platform, the OS, we get it and download, install it. And then that means now we have the Java to run JBank. But if your script then needs to say, Hey, I, I'm using Java 21.

So you can do that with slash slash Java 21 in the file or 21 plus to say 21 or higher. We then go and look is the current Java at Java 21. Nope. Okay. Go fetch. We installed it in jbank cache and we will go, we will set up for you. So yeah, that, that thing is just. Magic in Java. Like it just works.

Jonathan: Yeah. So something else that might be magic that I want to know about is, again, you've mentioned this briefly, but dependency handling how difficult is it because Java, Java is all about dependencies. Like there's, there's like a million libraries out there. That's like one of the super powerful things about it.

Is it, Hey, I wish I could do this in Java. There's probably a library that does it. How do you pull one of those in? Do you have to, like, include somewhere that, like, here's what it is on Maven and go grab it, or can you just, like, include the class and it'll automatically go look it up? What does that look like?

So I,

Max: I had a prototype for the class thing, but it's just, it's too magical. Like, it becomes, like, things fail. But so, so right now, so it might happen eventually, but right now I use we use a slash slash steps for dependencies and it uses the, the, like the convention it's to say, if you use cradle, they have them all.

So maybe it actually has a syntax for specifying group artifact version classified as a whole language. And that's just the one that we support. And that makes it very compact. And, and that's, that's what we do. And it, it. Yeah, that's all that is it to it. Like slasher steps and you can do it in the file.

You can have a separate file. You could do it on command line. So, and you can combine them so you can have a, let's say you have a a script that does something that says I need out 21 plus. I'm using Hibernate for database access. But I, then when you run it, you need to get a driver, like a square, like in a square, a Postgres.

So then you can go J bank desk steps. The Postgres driver and then the scripts and then these two gets combined. And now when you run, you have access to the, to the driver. So you can, it's very composable in that sense too.

Jeff: Jeff,

Jonathan: you want to

Jeff: ask that? So I, you know, you said you had a cache directory and everything and you're, you know, And I, like I said, I'm not super on Java. So do I end up with a, like a jar file that I, you know, I write my script and J bang, it can run, it can work. Do I end up with a jar that I can then just say, Oh, Hey Jonathan, I wrote this script, but I just sent him the jar file and it runs on his machine or how does

Max: that work?

Yes, you can. Yes, you can. So this is that though. Yeah, exactly. So yeah. So there is actually. There is a jar file in, in, in, in, in the, in the background. Originally I thought that in the early days it was just there because that was an easy way to actually not have a thousand files, but just the jar.

That's one jar that has the whole thing. But then someone, this was actually the, the, the, my the mind Mindstorm, the Lego Mindstorm use case I had for my kid. I needed a jar. That I ran, I had to export it and ship it into a Raspberry, the Raspberry Pi thing that was doing the, the mind thing, the, the, the Lego stuff.

So I had to export it. So there's a J bank export that will generate the jar that the, the results, but it will also depending on how they suppose you can make, make the jar and then copy in next to it, all the dependencies. And then the jar has references so you can just run it. So it's like a a multi file thing on disk.

Or we also have an export fat jar, which then takes all the dependencies and just scrolls into one jar. And you can ship that over and run it somewhere. And there's a bunch of other ways you can like export a container, export other stuff. But yeah, you can actually, you can take those scripts and export as a jar and just give it to someone and hope they have Java.

You can tell them to install a J Bang because J Bang doesn't only work for scripts. It will also work for jars. And then you can, J Bang will go get the right Java version and run that jar for you. So I'm not sure if I explained that well enough. Right. Because then, and this is the thing where, where I was having fun, some weird night, I go like, wait, I can now run the jar locally on my, I can have Java sources, create a jar.

I can export the jar. And I go like, Wait, why don't I just allow you to run any JavaScript, just Java source file or jar that is locatable by HPS or the ASP request. So that opened up for that. You can go and say, not just J Bang a file, but J Bang a URL. And it applies the same logic that it does locally.

It's a jar file. Okay, I download the file, compile it to a jar, analyze dependencies, and all is there. It's a jar file. I run it. I download and run it. It's a main dependency. I fetch the dependency. And I, we actually made it even further. So now we, we have a shorthand. For any GitHub repository or a gist, you know, gist service for, for, for grabbing stuff, you can do gay bang, a gist URL, and we will go and look for Java files in that gist and compile it for you.

So you have something running somewhere you can just, it's now trivially easy to distribute that to anyone, anywhere on the planet. As long as JBang is on the, on the system.

Jeff: Yeah, that's awesome. Because that would be good for, you know, say I write something that I need to give it to my mom, who's not a computer person, I can just, you know, okay, install this, and then do this, and then Yes.

Everything, everything's automatically taken care of.

Jonathan: What is, what does the process look like of installing J Bang itself? Is there just like a, is, is there some little tiny script that you can curl and run that bootstraps everything? Or yeah, you, you mentioned it's available in pip. What are the, what are the options for getting J Bang on a machine?

Max: Oh, how much time do you have? Cause I, I counted it just before here. I think I have about 20, like 90 or 20 different ways you can install JVM. So there is the traditional, like in Linux land, you can curl download thing. Right. But you can also I have there's a, a Fedora package. There's a Red Hat rel package.

There's a standards package. There is It was called ASDF, the installer, there's a Nix package, there's a Docker image, there's a JavaScript module, there's a Python module, there's a GitHub action, there is on Windows, there is if you know, there's a NuGet. And a Choco installer, a Scoop installer, and on Win Brew, there is Sorry, on Mac, there's Brew to go.

So, I, literally, any reasonable Python system that either I was able to make work, or someone in the community has done, We have available to get it. So basically there's no excuse to not install and

Jeff: we talk about this on. The untitled Linux show sometimes, but I did look and there's also a snap that's available for Ubuntu.

Oh, yes,

Max: sorry. Yes. We like to tease back and

Jeff: forth sometimes about that. Some of us are not as excited about the snaps.

Max: Yeah, I know. Yeah. And the snaps and there's the one that there's some Fedora line. I forgot the name now, but the same concept of these isolated ones, which was actually a really a big challenge.

Yes. Yeah, it's like that, yes, right, but it's a very big challenge because I need, like, JBang is, needs a Java to run, right, but if I'm not allowed to run the Java that's on your system, I, it's a very limited use. And one of the things Flatpak does is it's default is sandbox, so you can't do that. So, but we, we found a way and, and Yeah, so now you can use it as a Flatpak and it can run.

But you have to run with like some, Like escalated privileges because you J Bang is calling back to your system to use the Java you have available. So, but yeah, no, no, there's there's there's all all the ways you can think of if one is missing Let me know

Jonathan: Mashed potato mashed potato from our chat room says was compiler from source on that list

Max: Compile So the product is fairly easy to compile You but yeah, but that's, yeah, you can, but it's not, it's not a way I, I I say, Hey, this is how you download.

It's one of the things I learned from early on is any successful, like utility open source product in the first, you have to have working software and two, you have to have it available to release. Yeah, so I'm able to run and that's why in the beginning I was like because I've been doing tooling for 20 years and I had to fight people for, hey, you know what great.

It runs on Linux and Mac. But majority of enterprise customers runs on windows and every linux guy and every mac guy hates me for it It's like I don't want to deal with it. No, no, I know but that's One thing is what you as a privileged guy who have access to all your links You're not the developer runs in a bank somewhere and told to run in this windows citrix hosted somewhere It will it won't run on this stuff, right?

So I I spent That was what, before I ran, I released anything, I want to make sure it ran on all three platforms. And it was, it worked on every release and I could release that very easily. And and that's, so early on, I think we, we had 10 or 12 different ways of packaging installing it, which was really, really hard to do because there was no other Java product that had done these.

Very different ways of setting up installers. And that actually, you had Andres Amorea on about JVelizer a month back or so, and JVelizer actually part of like the way JVelizer got created was I talked to Andres about some of this stuff and he said, Hey, I'm doing a Go thing and GoVelizer has this thing.

Why don't we have this for Java? I said, well, If you make it, I have all the scripts, I have already done all the work on how you make a CLI in Java that can run on the, or be installed by these. So he took that and improved on JReleaser to the point that now JBang is using JReleaser instead of the scripts I created like five years ago.

So it's all all connected.

Jonathan: Yeah. It's a, it's kind of a challenge these days actually to get a Java runtime environment on windows, isn't it? Didn't, didn't like Oracle make some licensing changes to where you got to have a, some sort of agreement with them to be able to download it. Oh,

Max: yeah. So this is another part, like, so no, so no, well, this is fun.

Like it has never been easier to get Java on any platform because what Oracle did or Sonar like years back was they open sourced it and there's all these different teams that makes JDKs available. So yes, Oracle did change their license. Statements and I'm not a lawyer, so don't, but basically it says you can, you can use it for free, but after a year, if you stay on this version, you have to pay Oracle or you upgrade to the latest.

So if you stay in the latest. You're fine with oracles,

Jeff: but

Max: of course, as a red hat or anyone else, so I'll just say, Hey, use any of the other ones, especially Eclipse adoption, which is making a build of OBDK available to you. And there's Amazon and Microsoft and others has DDKs that are available without any of those license restrictions on them and, and.

And then there's this fuj, which is a service that literally gives me an API, like I I in J Bank, I just do, I generate a URL that says, this os this platform, this combination. And it, it will orchestrate, it'll go, oh, you need you want adoption bill for windows on, on this architecture? I'll go get that.

At the actual vendor, right? So there's one place that you can look up and get any kind of data K for on any platform now. So it is trivial easy, but yes, Oracle, of course says, Hey, install the Oracle one, because if you get. That's the one you use, then you, you, you, you, you will want to get support from Oracle and pay for that.

Does, does J bank,

Jonathan: yeah. Does J bank support actually installing? So it sounds like you go with open JDK by default. Do you support actually installing the one from Oracle? And does that ever matter?

Jeff: Oh

Max: yeah. So the default I use is the, what's called Eclipse Adoption. So this is the, maybe you heard, heard the same thing about Omnidecay some years back, there was a, like an alternative to get core, Oracle to realize, Hey, make the binaries more freely available.

So we, I use that because I then are sure that no user that ever uses will end up with a lice, like a. You know, licensed what's called licensing.

Jonathan: Yeah.

Max: But so I don't, so, and also I wanted to make sure that any, any default use of J bang will always get as close an experience. As if, as you have, like, so, for example, Jeff said, Hey, I wrote my script on my Linux.

I'm sending it to my mom or some student was just run a javang. So javang will default download. If you don't have the system, the default system, Java, you'll go download the adoption one. But because we use this FoodJ behind the scenes that can get any of them, there is actually a flag in, I mean, it currently is not exposed in JPEG because I haven't figured out a good way of doing it, but there is an environment where you can say, hey, I don't mind the vendor, JDK vendor should be Oracle.

Or Amazon or Red Hat, and then we will actually go get that one. And that includes and the Oracle ones, et cetera. Where it make it's actually has a use case is when you use early access builds so when there's a new version of something, the open data K community project. Will have binaries before the orders have them like a few days before and then it's nice to do or if for some reason you oracle has some features that are specific to their ddk If you want to utilize those you would want to use the oracle, but that's like I it's a niche case.

So it's not a thing that It's, yeah, there's flags and things you can do it. And I use it all the time for testing weird combinations. Yeah. But so, so no, so yes, it's, it's important. That's the way to do it.

Jeff: Yeah. Awesome. So. You talk a lot about supporting, you know, all three platforms, you know, looking at other open source projects, which ones are successful, how they do it.

So what, what has been the challenge of getting people to, to use JBang, you know, getting users to come in, you know, into the ecosystem.

Max: So, so that's the, that's the, that was the one that surprised me the most, because for me, what I. Built jpang. It was just hey, I want to replace this case with that. Let's just do it and then suddenly I realized Oh shit, I can, I can, I'm sorry, I can do all these things that we just talked about.

And then I go out and showed it to my team, like, and this team, these are like you know, people that have been doing Java for years, and I thought they would immediately be like, this looks awesome, let's go. And they're like, no, no, like, I don't know why this is not using Maven. It's not using Gradle.

This doesn't work in the IDE. Like, why, why, this is not Java. Like, why should I use this? And I go, no, it is Java. It's just a little script. And I've heard so many times people like, no, no, this, this is not what I'm used to. And therefore they just abandoned the idea. And also there's this, say Java is in a enterprise setting, so everything has to be enterprise y.

So if stuff is easy, it's considered not enterprise y. Like it's this weird thing, right? Like I even heard. Someone was like, no, no I, it's, it's called J Bang. It's not Java. So therefore it's not part of the Java ecosystem. And then I point out to them and they say, well, no one will ever use that.

And I put out to them. You do, you do realize Maven or Gradle is part of Java that is a complete separate ecosystem from it. So it should be possible. Oh, that's, you know, that's just how it, you know, it is. And then you go look at this. Oh, it should be as easy to do as you do in Go, right? So for example, Go was, did a brilliant thing.

They actually have all these tooling in like anything is built into. Go like there's a go format or that's a go install all the stuff is there. But that is all copied from like Python and Node. js. So if you look at Python, pip is not part of Python. Pip is a separate thing that Python, the community has adopted.

Node. js and JavaScript, it was originally in a browser. Someone then took it and put it in Node. js and Node. js then made this thing called an NPM. So all the other key systems, which people are saying, Hey, these are easy. Has done what I've, I'm doing it, but in the Java world is considered. Against some religion or something.

That's the, that's my, that's my biggest issue that I generally, when I go out and talk and present, I it's, it's like dividing three people. One is the ones who get it immediately. And like, Hey, I can do scripts and I can run them. I can install them. And that's good. And then there's a group that just, they just.

They, I think if they got it, they've just been so indoctrinated, they're like, no, no, no. This is alien. I will not touch it. . And then there's the one in the middle who are like, they just, they need to go touch it and see it before they, they, it, it connects. Um, and that's, so that's, that's the, that's the, that's been the, the challenge.

And it keeps being a challenge that, that people just can't believe that it's this easy. And like. I mean the taco is the he he he's to to blame or thanks for a lot of these life and stuff like When he he proposed this thing about being able to app So we, I did, we did all the JBang run and run from UL, like in the early days, I could even, I can even run a tweet, like, so you can go, you have a tweet UL, I can go JBang Twitter, blah, blah, and I could have, I could run a tweet.

Unfortunately, I can't do that anymore because Twitter has locked down. If you don't have JavaScript, you can't run anything that way. But and. But Tago came up with this like, Hey, we, if, if, why can't we do JBang app install, like, like you can do NPM install pip install go install. Why can't we do that for Java?

So, and this is the thing, and this is, again, I'm saying here, and I'm pretty sure no majority of those who are listening will not get it until they try it, but you can now, there's, you can, you can take a JBang script and you can Or a jar or maybe an artifact or anything and install. You can go jbank ev install.

Like the latest one is JLama. JLama is an inference engine for AI. It's a little tool. You just go JBang, app install, JLama and it's a different name. But, and then you have JLama, and you just run JLama directly. You don't see JBang, you don't see Java. It's just there as any CLI. And there's a bunch of these tools out there.

Like there's an SQL line, there's a different utilities that are written in Java, which originally you have to go like download the JDK, download the jar, put it in the past, blah, blah, all that is just gone. You just go javang app install. And, and, and you can do that for any Java. This, like that product has been built with Maven or Gradle or whatever.

And the only thing they use javang for is just to, to wire up all that stuff. And again, if you're a Python guy, a Node. js guy, a Go guy, you've been doing that all day, all year for the last 10 years. And now you can do it for Java, but getting that word out and getting people to believe it has been the biggest challenge.

Jeff: Oh yeah, I, you know, and I, I remember what you taught when you said, you know, it's got to be hard to be enterprise and all that, because I remember back in the early days of the internet. If you didn't compile your own IP stack, you didn't deserve to be on the internet. You know, it was just, it has to be hard or you don't deserve to be here.

But, but going into that, can I use it in enterprise? Is it, you know, is it, is it stable enough? Is it ready to go?

Max: Well, so

Jeff: have a cheap code for enterprise. Yeah. .

Max: Yeah. Well, so the, the, the, yes. So the thing is, and this is related to what I said before, like it, it, it literally is, you don't have to do the scripting part of J Bank.

J Bank scripting was just where it started. But you can use any kind of j jar. So if you have, a spring project or micro naught or corkers or anything else. You can just use J bank to run it, set it up. And one enterprise area we, I, I see, and this is where we actually use it. On the product I have myself is all the CDI, like CI scripts, like the, the, it's so popular, we're doing all this dev ops stuff.

And the fact that our Java developers can just use Java to script things that they do. So that's one entry, but The other product that my main job is actually to work on a product called Quarkus, that's actually the t shirt I have on here is it's this thing where we, we, we, we, it's an enterprise framework stack that makes Java, you know, fun again, so to speak.

And, and, and, and, and can run very effectively. And that one actually, the way Quarkus does that is by doing build time, like doing simple build time. And that normally requires a full build tool. But if you combine that with Quarkus actually has support for JBank. So if you take J bank script and add a Quarkus dependency.

It will actually not just be fetching dependencies, but J bank has support for extension. So Quarkus actually gets invoked to go, Hey, I compiled these source files. Now go your Quarkus thing. And then that output is actually. An enterprise app, right? And that can then use all the stuff Corvus has, like native compilation, running container any camel, CXF, all that enterprise stuff is available to you.

But I do want to put an asterisk on it I'm not telling you to drop all your gradle and maven projects because there's stuff that debank doesn't do that these need to do there's complexities that debank navel will cover but Definitely. You, you, you can, you can use J bank to, to write microservices and, and get up and running.

We, we do it for our, we have a GitHub bot and some applications. They all, they are actually either a caucus app then run by J bank because you can get all the Javas, or it's a script that sets up a few things and just runs. So, so yes, it is enterprise, right? But it has a certain use case, right?

So in that way, it, it, it is there.

Jonathan: Very cool. So I've, I've got to ask like what what's, what's coming down the pike, is there something coming for J Bang that you're excited about?

Max: Well, so my what i've been doing mainly is trying to I go out and I I try and find all these utilities because there's there's a bunch of people who wrote Awesome things that never got into hand of people because it was hard to run these things.

So i'm going I mainly go out and submit small patches Hey, jbang we have this notion of a jbang catalog which sets up the command line or that kind of thing and and make it rumble So that's this That's what I've been doing. But the, the, the, the, the main, so when I then do that, then I might find a thing that they do that I hadn't thought about, and then we do a release.

But, and I'm on release, I think like, like 0. 119 is the one that I think I've done like 250 releases in. For four years now. Right now it's very incommensurate, but I'm trying to get to that one. Oh, but that one, Oh there are some enterprise features around how you manage dependencies that I don't capture.

But otherwise I'll say we are very free to complete in, in being able to do the main mission of what JBang is. So, so, yeah, so. For me, the big thing now is actually to get people aware that this exists and try it out and not try and be scared of it. So it's more of a soft features than the hard features in that sense.

Jonathan: Trying to get to 1. 0, do you feel like Sisyphus rolling the rock up for all of eternity? Is it a Sisyphean effort?

Max: Yes. Well, it's just, the thing is I, the problem is I've, I've made the come back to like make, it's so easy to do releases and like, it's been stable and we actually try. And I think we have one or two glitches where we, we did a mistake.

But it's actually, you can still use the old ones just fine. But one of the things I want to do before the wonder will break some users and That's, that's the rolling I'm, I'm like one day I'll, I'll do it this way. And then they, yeah, so no, so yes, I do fall through that. But I, the, the cool thing I do is this part of the the way we do the current, like the, the installation, whether it's the Maven plugin or greater any, any of the way you install like curl and stuff is is going through J Bang dev a URL so I can actually see When people are running a J bang in stores.

So no, I can't see who you are, but I can just see someone somewhere did fetch this little script to go look at the latest version information. And I can see the growth is there a lot of people in Azure and GitHub Jackson is using it. But I'm still missing the. The uptake in all those Java utilities out there to, to to make, to, to have it and any Java stuff available as easy as you can for Python and, and, and JavaScript.

Jonathan: All right, we're getting close to wrap and here in just a minute, I'm going to ask if there's anything we didn't ask that we should have. While you think about that, I'm going to ask a different question. And that is, what is something that your users have done that really has surprised you? Is there any like oddball use cases that are particularly interesting?

Max: Okay. Well, well, so, so, so the, the, the latest one, it's not all about the JVM, it's just this thing of so, so the, the, the JLAM, I think, so, so again, everyone thinks AI has to be a Python, which is great and stuff, but JLAM Java has a vector 20, there's a, a vector support coming, so they can, it can do efficient array copying and manipulation, which you need to do in inference engine.

So there's a guy Jake, I just know him as Jake on, on, online. His run and written in a Java implementation of that's called JLama. And he had an installation set up that was very complicated. And I just pointed out, Hey, you can do a JBang. And he just did that like this week. So now you can actually, with JBang, again, you can start from scratch on your machine and just install JBang and go JBang download this, one of those models and run it and no, and it will be a CPU, a GPU optimized and that kind of thing on any platform where that exists.

So that, that's my recently interesting setup. Then I know that's, that's gotta be tricky to explain but I'm showing it, but Java has a notion of agents. So you can attack any Java app. You can attest an agent that gets notified by anything gets loaded, and then it can manipulate the classes and you can do evil things or good things.

They can monitor and that kind of thing. And again, if you use the normal Java tool chain, that's a lot of arguments, a lot of setup, and you have to download files and that kind of thing. So we have a product in Rehag called Byteman, which is an agent that can go in and you can say, hey, if I hit a method that has this signature the third time, throw an exception or log a statement, right?

So you can do all kinds of things, but you have to do all these steps. And then I realized in JBang, we have all the pieces to not just go get dependencies, but all of the app, but also the dependency of the agent. And also any configuration that the agent needs, I can go get from a, from a URL. So again, with JBang, you can actually.

Set up any kind of agent usage on any kind of a Java app in a one liner command, no complicated cradle, maven, all that kind of thing. And I find that super interesting, but. I also think many people will actually appreciate it because those who actually managed to do that without it is very, very few, but the, the, the, the, the power is immense.

And this is stuff that you cut like in, in Python or Node. js is those kind of things just doesn't exist or are even more obscure than, than we have here. So no, so the,

Jonathan: Yeah, interesting. All right. Is there anything that we didn't ask about that you really wanted to let folks know about?

Max: Well, the main thing I have is like, if you're listening to this and seeing it, do just try it out and spread the word, like try, try and use it.

And if you have a Java tool to somewhere. Apply a little bit of JBang so it actually becomes available to anyone out there. That, that, that's my, that's my hope that some, some of those people in the, Hey, Java is this horrible, complicated thing. But I know that this utility I actually would like to use if it was easy to use Java.

That's where JBang can help. So, so, yeah.

Jonathan: All right.

Max: Try it out.

Jonathan: Yeah, absolutely. I do have a couple of questions that I'm required to ask everybody. I think I already know the answers. But what is your favorite scripting language and text editor?

Max: So the text editor. So I'm, I use every editor on the planet because I've done tooling it, but the one I, my day to day driver at the moment is VSCode or or some variant of that.

The scripting language, well, I would say Java space script. I, I've done a lot of scripting in my, my like bass and Ruby and Perl and Python, but once I got it working for Java, I use it for everything. But it's technically not a scripted language, but. scripting. So I'll say it counts space. It's my, it's my favorite script.

Jonathan: There you go. All right. I think that's it. We've gone past the hour. We, we sure appreciate you being here, Max. Thanks for stopping by and Telling us a bit about J Bang and scripting with Java. Sounds like fun.

Max: Me too. You should try it. It's, it's, it's not your parents Java anymore. Yeah.

Jonathan: Yeah, I'll have to, I'll have to look it up and give it a shot.

All right. Jeff, what do you think? Have we, have we convinced ourselves to go try some JavaScripting that is not JavaScript?

Jeff: I, you know, I'm kind of interested in this. It's you know, I, I've seen a bunch of Java programming in the past. And, you know, I've, I've messed with other languages, not really with Java, but, you know, kind of something to make it a little easier like that.

I kind of interested in that, you know, I, I, it piques my curiosity.

Jonathan: I, I feel like it would be fun to at least like, fiddle around with it enough to be able to say that you've done so. I, I'm curious. I, I guess I wonder are there things that you normally do in scripting that Java is not made for? And we, we didn't, we ran outta time.

This is one of the questions I, I wanted to ask. We ran out time to do so. But I'm interested to dive into it and see, you know, like in bash scripting you get to do things like, call other bash commands with backticks and then replace the output of that command into your script and Just I I guess Java has all of that built in because it's more of a full featured programming language So it doesn't even need all those tricks But it's interesting.

It'll be fun to play with I definitely need to go grab it and check it out in in possibly a snap format Maybe I don't think I have any computers that'll run snaps actually

Jeff: That's, that's a UL, ULS inside baseball. Yeah, yeah,

Jonathan: yeah, yeah. All right Jeff, thanks for being here. Do you have anything that you want to plug?

Jeff: I don't. I just well, I guess other than check us out on the Untitled Linux show, over on the twit. tv network become a member of Club Twit. And I'll see you next time.

Jonathan: Yeah, have a great week. Excellent. Thank you for being here. I do want to let folks know. First off, next week, we're talking about EMBA with Michael Messner.

And that is an embedded firmware analyzer. I came across this doing the security column on Packaday, which go live, goes live on Friday morning, which you should check out. Came across it one week and it's an open source project doing like embedded firmware analysis, looking for problems, trying to understand the way it works.

And I got to looking at their code and thought it was really cool. And so we're going to have them on the show next week. It's going to be super duper interesting to talk about EMBA. Other things that I want to plug, of course, as Jeff said, we've got the the Untitled Linux Show over at twit. tv.

Would love for everybody to go check that out. Show Twit some love, and yeah, we appreciate Hackaday giving us a home for the show. And with that, we will see you next time on Floss Weekly.

Episode 800 transcript

11 september 2024 | min

FLOSS-800

Jonathan: Hey folks, this week Aaron joins me and we talk with Andreas Kling about Ladybird. That's the from scratch web browser that's almost ready for primetime. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 800, recorded Wednesday, September 10th. Champagning the Ladybird browser.

It's time for Floss Weekly. It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and today is no exception. We're going to have a lot of fun. We're going to talk about Ladybird, the from scratch web browser, which, boy, that just, it sounds weird to even say it, but We've got the, we've got the man himself we're going to talk to in just a minute.

But before we get to that, we've got the other man. It's, Wow, that transition did not work as well as I thought it would, but here we are. We've got Aaron. Welcome, sir. Hey, thanks. Thanks for having me. It's good to have you back. I am so glad. One of the things that makes me extremely happy about moving to Tuesdays is that we get to have Aaron back as one of our co hosts.

And I just, I get a kick out of that because I've always, always enjoyed being able to work with you. And I was very glad to be able to have you back in the rotation.

Aaron: Yeah, for sure. I just noticed that my maybe because of my mustache, my beard's a little longer, it needs a trim. But I've noticed that like, I've got the old man syndrome.

Like the older you get, the more your face droops and I'm sitting here smiling and I'm realizing that it's just like a straight line across now. It doesn't go up in the corners anymore. So yeah, I've got a, I've got a hope that people know that I'm smiling, but I am, I'm happy to be here.

Jonathan: Yes. So Lady Bird, what, what do you know about Lady Bird?

Aaron: Absolutely nothing. Absolutely nothing. It'll be, I'm really interested for this conversation though, because you know, a, why do we need another browser anyway? Like what's the problem that they're trying to solve? I think. A lot of us know what that is, but I still, I, it's a lot of effort to do a browser.

There's some

Jonathan: problems. Let's just say there's some problems.

Aaron: There are problems, right? We'll get into that. And then, you know, I'm, I'm old enough to, to go back to the days of using mosaic and text based browsers and gopher and, you know, I was around at the beginning of the internet, um, at least that part of it.

So. So, yeah, I mean, I remember when new browsers were popping up you know, pretty frequently and it was like a big deal, like when Netscape navigator came out, it was like, Whoa. So anyway I remember those days. So I'm kind of curious, like, what are we going back to those days of like, Hey, let's just get back to basics here.

Or, you know, is there, you know, how do we, how do we accomplish that? And also make sure we can still do the things we want to do. Right. On mine. So, yeah, I'm really curious.

Jonathan: I'm, I'm kind of excited that we have a new browser that's not just Chromium with a different skin on it.

Mm hmm.

Yeah. We have, we have a bunch of those, and so people, I mean, you just talk about this browser and you go and look at it.

It's just Chromium, guys. It's Chromium with a theme. It's, do we really need to do a whole show about this? Lady Bird is a bit different. Well, let's, let's go ahead and bring Andreas Kling on, who is the man when it comes to Ladybird, and I guess the man when it comes to browsers. Welcome sir, thank you for being here.

Hello, how's it going? It's great, it's great. So, Ladybird, it's, it's a browser. Let's see, where should, where should we start? I have lots of questions. Shoot, I don't see a download link. Let's start there. Okay, the audience knows this, just so you know this, I will ask a lot of questions that I either have a guess or I know the answer to.

I'm pretty sure I know why there's not a download link, but I'm, I'm trying to be a proxy for our audience in asking me to share.

Aaron: Well, that's the first thing I did this morning as I went to go download it. It was like doing exactly that. I was like, Oh,

Jonathan: I see a project,

Aaron: but no download.

Jonathan: Do I have to, do I have to compile this from source to be able to use it?

What?

Andreas: That's crazy. No, like you were saying we are doing something different, which is that we are not starting from Chromium like everybody else does. And because of that, it's going to take a little bit longer to, to get this thing ready. So the, There is no download today but you can download the source code and build it.

I think it's two commands that you have to run and then it should take care of everything for you. We, we do get a lot of positive comments about how easy it is to build our project. And we are aiming to put out an alpha version in 2026. So that's sort of the timeframe that we're looking at right now.

But we, to go more public because we started a nonprofit earlier this year. So that's why you've been seeing us more and more because we started to get more serious about this. And I hooked up with Chris Wanstroth. Of GitHub fame. And we started this nonprofit to fund and develop Ladybird.

And we've talked to a bunch of companies that are also interested in, in like having a new browser on the scene. So that's sort of how it's come together. And but at this moment you can, you cannot download a running product. We are pretty far from that, but because. It's a lot of work to build a browser.

We do need a little bit of funding, at least not as much as everybody else, but a little bit, so we have been raising a little bit of money. And yeah, that, that's kind of the, the general state of things. And a lot of people, of course, brave early adopters and courageous open source enthusiasts still go and build the browser.

And they are typically disappointed, but come away with a bit of hope at least in their, in their stomachs, I would hope because it doesn't work Well, for daily driver browsing today and we're not trying to mislead anybody about that. And we are actually two years out from, from what we would consider an alpha version.

So we don't encourage anybody who's not technically minded to even bother with this at this point in time. Because you're just going to have a bad time. But for people who are technically minded, you are most encouraged to try it out, mess with it. Try your favorite website, maybe a website that you made yourself.

Tell us what didn't work, what didn't work right. And maybe even figure it out yourself and see if you can help us fix it. So we've been doing a lot of that. Trying to collect new developers.

Jonathan: Is using ladybird right now a better or a worse experience than trying to use something like the links browser?

Andreas: Oh, wow. I don't have an, I don't know. It might be better.

Aaron: It's gotta be better.

Andreas: I would hope it's

Jonathan: better.

Andreas: It's, it's probably better. Yeah. Cause yeah, but the links is that links to one, but JavaScript.

Jonathan: No

Andreas: links

Jonathan: is text based links is the entirely text based browser. I don't think it runs JavaScript at all.

Andreas: Oh, yeah. Then you're kind of screwed.

Jonathan: Yeah. Okay.

Andreas: Yeah, no, we, we, we can do bare, we can do JavaScript, not just bare minimum, but like we can do real JavaScript. But we do struggle with performance. We struggle with some of the more intricate features and especially stuff like a YouTube doesn't work yet because of a million intricate little things that we have to figure out.

And there are a lot of these bot detection systems that we have yet to like convinced that we are a real browser. So CloudFlare and Google and, and whatnot, they have like these you know, mechanisms to make sure that you're not a bot and we look like a bot because we're just doing stuff wrong.

And that's entirely on us. You know, they're, we're not complaining about them. We just have to get better.

Jonathan: I could actually see somebody like Cloudflare getting interested and excited about the project. You know, they're all about having, in some cases, their own technology stack. And so, having a browser that's not run by one of the big guys is maybe something they'd be interested in.

Andreas: Yeah, so we've spoken a little bit with one of the engineers on the team at Cloudflare that makes this anti bot software. And we have a little bit of a back and forth there, but there's just a lot of work on our side to do. And of course, not to mention any of these fingerprinting things, you know, where you can check, Hey, how fingerprintable is my browser?

We are possibly the most fingerprintable browser right now.

Jonathan: So, okay. So what's the origin story here at, at what point did you wake up and go, I'm going to build a browser from scratch. And like, what, what sort of my, where, what was your headspace? What, what kind of crazy headspace did you have to be in to make that decision?

Andreas: What, it never happened to you?

Jonathan: Not that in particular, no.

Andreas: Other crazy things, yes, but not that one. Right. Well, it all started when I decided to build an operating system from scratch. Which was its own crazy headspace that Somehow seems less crazy? In some ways, perhaps, yeah. In retrospect, it was a simpler time.

But yeah, so I was doing that for a while, starting in 2018, I built the Serenity operating system, Serenity OS, and it was a. Passion project for myself that I did as a sort of personal therapy to keep myself out of trouble. I used to have a pretty big problems with drugs, alcohol, stuff like that.

And I needed something to focus on that was healthy. And yeah, I went pretty hard on building an operating system, put it online. A lot of people liked it and started to work on it as well. And a community formed around this and the community just kept growing. And we became more and more ambitious with our scope and we would add things like well, initially it was pretty modest, you know, we would add networking and we thought that was a big deal, but And then, like, why don't we have a photo editing program, or a music production studio, or visual programming tools.

And it just kind of kept growing in scope. So it was natural one day that we just decided that we should have a browser also. And a big part of SerenityOS, the SerenityOS mindset, was that we do everything ourselves. Like, we don't borrow code from anywhere, we just do everything ourselves, because it's more fun that way.

And I'm sure every hacker ever can relate to that. Even if you wouldn't necessarily do it at work, you at least enjoy doing it at home. Sometimes. So yeah, that's sort of, that's sort of the where the whole thing started from was that we wanted a browser for SerenityOS because we were adding everything to it.

And I have a professional background in working on browsers. So I was working on the KDE browser Conqueror like 20 years ago. Since then I've worked at Nokia on their browsers and then at Apple on their browsers. So working on browsers was like my job for a long time. So it was very natural for me and once I started working on a browser again, I kind of just slipped into this old habit, and it became my main focus, and I kind of stopped working so much on the rest of the operating system, to the point where they became two distinct projects, and we forked because, The browser was getting so much attention and people were focusing exclusively on the browser.

And we were living in this GitHub repository together with an operating system, with a photo editing software and music software. It was just a really cramped space. So projects split up and now I work only on Ladybird. So the browser only. And we've sort of changed the rules a little bit. So as I mentioned, it used to be that we don't use third party software.

Like we do everything ourselves since we're in the US, but in Ladybird, we want to actually make a product that people could use. And so we have admitted to ourselves that we're going to have to use a little bit of third party software to, to make that happen in this lifetime. So Over the last couple of months since we forked, we've been integrating some of the sort of open source ecosystem for things like fonts graphics formats, audio formats stuff that would take us a long time to do ourselves exhaustively and correctly.

We can just like piggyback on, on the existing stuff.

Jonathan: And

Andreas: yeah, that's sort of the origin story. Yeah. And that's where we are today also.

Jonathan: So speaking of where you are today, I'm going to, I'm going to hand it over here in just a second, but I do want to ask first, like what, what is the state of ladybird?

How much of the web actually works? How much of it renders correctly? What, how, how frustrating is it to try to use it?

Andreas: Oh, well, it's pretty frustrating. I'll tell you that much. But we've been focusing on our own sort of Dogfooding use cases. So we tend to use a lot of

Aaron: dogfooding. It's champagning

Andreas: champagne.

Aaron: You're drinking your own champagne, not eating your own dog food. Right. Or maybe not. Don't be yet. As the case may be.

Andreas: Yeah. I don't know that that will work for me personally, but I can see how dog fooding is a bit of a weird term. Yeah, no, at Apple we always called it living on. So like you would be living on the latest build or living on Living on Ladybird.

And we're trying to do that. So we're focusing in on our own daily use cases, like using GitHub, using you know, Google and reading web specifications more than anything, really and getting communication software working like discord and stuff like that, like stuff that we use every day. And those kinds of things have been seeing pretty good development.

They're not super great yet. GitHub is, I think, the thing that we handle the best. But once you sort of stray outside of the things that our developers use every day, you're gonna probably run into issues. And It's, we've taken kind of a vertical slice approach to developing the browser where we've been just picking a website and then doing whatever it takes to make that site work as well as possible.

Which is, you know, one of many approaches. Another one approach you could take is you could pick a spec and then try to implement as much of that spec as possible. But we've been kind of going after these like, let's try to make this site actually work. And I, I often liken it to video game emulation, where you know, like if you're going to implement an emulator for the Super Nintendo or whatever, you're not going to sit there and implement every CPU instruction.

You're going to try to get Super Mario Kart to run or whatever. So we've been taking that same approach to getting websites working. Which means just to answer your question it's, it varies greatly depending on if the site you're visiting is something that we have worked on or not, but over time The, the rising tide lifts all boats or whatever, right, where because all of the fixes that we make for the various websites that we do work on, they are all in service of, of you know, supporting the web platform.

And it just happens that we pick sort of scattered random Parts of the web platform to implement at a time. But over time, our general standard support, our general support for the web has been improving. And recently we've also started running, there's this test suite called the web platform tests, which is a sort of a collaborative test suite that all different browser vendors contribute to.

So, you know, Microsoft, Apple, Google, Mozilla. And ourselves. Can contribute tests to this giant battery of like millions of tests. And we've been running that recently and I think I think we're passing like a bit more than half of the tests. And we're like actively working on passing more of the tests and Yeah, we, we definitely want to just get those numbers up.

But because we haven't been running the test before, it is there's a lot of low hanging fruit that we're dealing with. Yeah.

Aaron: Does it support Flash? That's a joke.

Andreas: It, it doesn't actually I mean, coming from

Jonathan: SerenityOS, that would be very on brand.

Andreas: A little bit, I guess. Oops. Oop. Somebody's got a dog.

It's all good. No, we don't support that.

Aaron: That was just a joke, but seriously, though, like, like, why, like, well, let me ask this first. Actually, I want to know, like, you kind of told the origin story. I want to know, like, why you think we need this number one. But before I even get there, the thing that I'm most curious about is how much interest have you received over the past, whatever, six months since the whole Google antitrust Yes.

Stuff has been going on. Have you like, you know, gotten way more interest all of a sudden, because of all of that?

Andreas: I don't know that I could really separate interest coming from that from interest for other reasons, but there's certainly been a lot of interest over the last couple of months, but we only went public with our.

Nonprofit in July. So, and the, the, I guess the latest Google rulings were pretty recently. And when people donate to our nonprofit they sometimes write a little message and there have been a lot of messages mentioning the, the state of affairs of the industry. Which I think, you know, it was positive and.

When I, when I talk about the origin story, it's, it's sort of it's sort of Ladybird just evolved out of hacker culture in some ways, but at the same time, it's also finding reasons to exist that aren't just organic. Because there, I think there is a need for a like a truly open browser that isn't connected to like the advertising industry, right?

And isn't necessarily depending on the exact same browser engine that everybody else uses, but like a new implementation that is standalone and doesn't have to do whatever Google wants at all, at all times. And yeah, it's, it's, it's a tricky little bit of a tricky subject because I've been in the browser industry for a long time.

I know people working out in every browser. There are a lot of great people everywhere. But at the same time, I think we should also acknowledge that the industry is a little bit messy. There are a lot of weird incentives. For, for the major browsers. And we are very keen to try a new approach to this where we are a nonprofit and we say very publicly, very explicitly that we will not take any kind of strings attached sponsorships.

So we're just going to. Do this with a small team you can sponsor us if you want, but you're not getting anything other than a logo on our website. We're not gonna, not gonna put your search engine in our default settings. We're not gonna send data your way of any kind. And We, we understand that this drastically limits our budget compared to the competition, but we think that this is something worth exploring because the world should have a browser that is independent of all of that.

And we kind of started in hacker culture and just organically grew a browser. But we find ourselves in a position where like. There's nobody else to attempt this. So why not us? Why couldn't we be the open community developed browser with no attachment to any advertising money?

Aaron: Right. Right. I think it's a, I think it's a very ambitious and very noble thing to do.

At the end of the day, it may be too early to tell this. I don't know. Are you going for, for parody at this point in terms of functionality or you know, when we get to 2026 and there's something that is on the download page for people to try will there be noticeable difference? What will the differentiation be between Chrome, let's say, and this, when it actually comes out?

What are you, what are you hoping to achieve?

Andreas: So what we're aiming for is that you can do your daily browsing with reasonable, Stability, reasonable performance. But it's unlikely that we will be faster than any of the other browsers because they have thousands of engineers to throw at performance.

We don't have that. We're not going to have a sophisticated developer tools that like help you write your CSS and develop your website. We're going to have some tools, but we just don't have the manpower to put all that together that they have, you know, a 10, 20 year lead on us. That's going to take time for us to backfill all of those things.

But in, in terms of just like rendering the web we are hoping to make something that will work well enough that a user can use our browser. And. Not think about the fact that they're using Ladybird. That's sort of the happy outcome is that you would just use this browser and it will work and you're not thinking about what browser you're using.

If you look under the hood or if you start to poke around in the menus and you try to try to do fancy things, you will quickly discover that we don't have those yet. But our hope is that it will be a decently well enough working browser for common websites that you're likely to visit. You know, your LinkedIn's, your Facebook's, your Gmail's.

All of those kind of things. And then It's seems very likely that we will spend, if that all works out really well, then we will spend the next year or two just fixing the last 5 percent of bugs. Because I think it's one of those things where like the first 90, 95 percent are going to take two years and the last 5 percent are going to take 10, 20 years.

Jonathan: Yeah, well, part of that's because the web is, is such a moving target. Like there's constantly new things getting added to the JavaScript standard, to the CSS standard, to the HTML itself with, you know, HTML five and all of that. Is there, you, you might get to the point to where you sort of caught up, but if, you know, if it's going to continue getting supported into the future, I don't think it's ever going to be done,

Andreas: right?

No, it never will be. This is definitely a project that can go forever. And. There are all the other actors. So, you know, Google, Apple, Mozilla, they are all actively adding new features to the web. We are passively just implementing what they've. Come up with maybe one day we will decide that we, we can think of some great features too.

But at the moment we're just keeping it cool, you know, just implementing what's there. I do wish that people would slow down a little bit with the feature development but we do also recognize that the web has to evolve. Maybe not as quickly as it does sometimes. Yeah.

Jonathan: Yeah. So, so Aaron asked about the the idea of differentiation between Ladybird and Firefox and Chrome, and there is something that comes to mind, and I wonder if this is, this is something that you guys have realized, if other companies have talked to you about it yet.

Both Firefox and Chrome are huge, and they're difficult to install and work with. Like, they're just unwieldy. Like, particularly Chromium, trying to do a compile of Chromium is just a pain. And you mentioned earlier that Ladybird is pretty easy to get building and to get installed. And I could see it having a life in, in some places, you know, like you've maybe even embedded just because it's so much easier to work with is, is that something that you've, you've kind of had conversations about?

Andreas: We're not ruling it out. I know that there's been some interest in the embedded use case, but we haven't really been targeting it. We, we've been playing a little bit cowboy like with memory usage and resource usage in general. Because it's easier to just allocate lots of memory than it is to be careful and Precise.

And a lot of people, you know, a lot of people look at Ladybird and they assume that because it's new and because it has a limited set of features, it must be using less resources, but that couldn't be further from the truth. We managed to implement fewer features with a lot more resources than the other browsers.

But of course that's because they've been optimizing. Their browsers for decades and we haven't. So we're catching up a lot on, on that. And I think the embedded use case requires a ton of pretty sophisticated optimizations that we're going to have to do anyway, but I don't know, I don't know what kind of interest there would be.

I guess that's something we'll, we'll find out eventually if somebody comes to us and says, Oh, I have this great idea for an embedded use case. We're definitely interested in helping them, but it's not something that we're pursuing ourselves. Like we're primarily interested in making a desktop browser for people and for ourselves is the, is the current target.

Jonathan: Yeah. I have a I have a Raspberry Pi in my hallway. That is my HVAC controller. And the idea there is that it's supposed to run a web browser to, to show, you know, your, the webpage of what your temperature is. And it's kind of a pain to work. I think, I think, I think. I don't remember if it's Chromium or Firefox is what it's supposed to be doing.

It's not working at the moment. I need to spend some, you know, some engineering cycles on that again to get it working again. I think it could be a lot of fun, because the webpage it runs is just Stupid simple. It'd be fun to try to do that with ladybird. So I, I am

Andreas: For that kind of thing. Yeah. I mean that shouldn't be terribly difficult.

We could probably throw in some command line switches for you to like start in full screen mode. There's no ui and stuff like that, right?

Jonathan: Yeah I could that seems like that would be a huge win. I could definitely see some some businesses wanting to use that. What's the what's the license? What license is ladybird under?

You

Andreas: We are under a two clause BSD license, so pretty permissive, very permissive. Yeah. Yeah. Yeah. Yeah. Not everybody's favorite, but I don't think there is a license that everybody would agree is the best ones. Right.

Jonathan: Yeah. Yeah. That's, that's the thing there. And what about platform support? Is this you know, X86 64 only, does it work on ARM?

Does it compile on RISC V you know, MIPS, there's a whole bunch of different architectures out there,

Andreas: right? So far it's been X86. 64 bit and arm 64 bit as well. So we run on Linux and Mac OS at the moment. And like, you know, the, the wide family of random Unix's as well. Since we come from, from SerenityOS originally we had pretty, like, generic Unix fundamentals in the project, so it's been easy to port it to, like, FreeBSD, Haiku you know, NetBSD, other systems like that but Windows is the big elephant in the room because it's so different and not Unix like that, you know, Windows support is something that frequently requested, but we don't have any active person working on it.

But this, this is probably not the podcast to, to complain about Windows support.

Jonathan: And, and somebody could run it like in, in the what, what are they called the, the Linux subsystem L S W. No, that's not right.

Aaron: Linux subsystem for Windows. Yeah. LSW.

Jonathan: That does not seem like the right acronym, but okay.

Okay. WSL. I think it's WSL. It's WSL.

Aaron: Windows subsystem for Linux. Yes,

Jonathan: which was always a confusing way to put that. It feels backwards. It has led to some discussion. Is Windows looking to replace the Windows NT kernel with Linux?

Aaron: Right. Right,

Jonathan: but if somebody wants to there's there there's that right? I'm assuming it runs just fine.

Andreas: It does it does Yeah, although it is probably the worst developer experience that we have so people who try to do it that way They're the they complain the most out of everybody that comes to us.

Aaron: You're really making a Trying to try to force a square peg in a round hole

Andreas: Indeed

Aaron: at that point Yeah, you can do it, but you know, why not just start up a virtualized environment in your Windows desktop and yeah, you know, just do just do that.

Just run Linux on top of it in virtualization. What I had a question and now we had that discussion. I'm not sure what it was. Well, I guess one thing that I was wondering is like, what are the biggest hurdles do you think knowing the space so well? Oh, I remember what the other one was too. But the first one is what are the biggest hurdles do you think that you have, like, this is just going to be impossible to build from scratch because.

You know?

Andreas: Oh, what do you think? I think I never really thought of it that way. Or maybe

Aaron: because everything's a standard. Sorry to interrupt you. Maybe because everything's a standard, you can just write to the standard and you don't have to worry about it. I don't know what the answer is there.

Andreas: Well, that's partly that.

So the standards are a lot better today than they ever have been because the various groups that work on the standards, they've been doing a really good job and backfilling all of the stuff that used to be unspecced, used to be sort of browser voodoo mystery stuff. Over the last 10, 15 years, specs have gotten really good.

So if you're writing a browser from scratch today specs will get you a long way, but there are still some things that you sort of have to figure out what other browsers do and then mimic that as well. And. That continues to be one of the more annoying challenges when the specs are lacking. So we ran into something just this week.

And not the first time, which is JavaScript date parsing. So if you look into JavaScript specification, the way that they explain date formats is that there is the ISO. 86 0 1 date format that the spec says you must support outside of that format. The implementation decides what additional formats to support and in practice every other browser supports like a giant collection of random date formats.

You know, like Jan one comma 1972 or Mm-Hmm, one, Jan, 1972. And. There's no list of these anywhere and people have tried to spec how this works, but it kind of just, they just get bogged down and, and they get tired of it and abandon the attempt. So even today, the spec doesn't explain how to parse dates.

And that's been pretty frustrating for us as a new engine. Because we frequently inquire, encounter new date formats that we haven't seen before. And then we have to implement parsing for that particular format. And now the, some websites starts working.

Aaron: Weird.

Andreas: And yeah, then there are, there are a couple of things like that.

A couple of touch points where you have to just look at what other browsers do because the specs, you're just kind of shit out of luck with the spec. But you know, to, to the credit of, of the standards groups, These number, the number of such things has been going down aggressively.

Jonathan: That puts you guys in an interesting position where you're, you're sort of doing an audit of the web specs.

It might be interesting to try to put all of that feedback into a document and, and make it out, put it out there, make it obvious, so hopefully someone could get fixed.

Andreas: Well, we are doing Ubuntu better and we are actively reporting and fixing bugs in the specifications. That is one of the great benefits of new engines coming on the scene is that we end up really putting the specs to the test because people have been iterating on these specs for years, but nobody ever tries to implement them from scratch.

And so we come along and we do that and we find like, well, this doesn't actually make sense. If you try to implement it it falls apart. That happens a lot. Regularly, I would say. We just come across some little thing where it doesn't Have internally consistent logic. And then we report bugs and it gets fixed and that's fantastic.

And it always feels so wholesome when I see somebody from our team go and find a bug in a spec, report it and it gets fixed and the whole ecosystem benefits. So yeah, we are very much auditing the specs and doing our best to be good citizens and, and reporting bugs in specs, or if you find bugs in other browsers, we try to report those as well.

Yeah. Try to be, you know, try to be good boys in the club. Very good.

Aaron: So what, my other question was in terms of the other open source OS vendors, Red Hat, Ubuntu, name your favorite one you know, since, since you've already described how this kind of grew up out of SerenityOS, like, do you, are you getting support from those folks more like you would expect you would, or no?

Andreas: Not really. I didn't really have any expectations though. So there's been some interest in packaging our stuff, but it's so early still, you know, we don't have anything that We could in good conscience, ask an end user to try. So even when distros do try to package our stuff, there's been like I think Arch has some package.

Nix has some package. And I see a lot of people trying these packages and they come to us and they complain that it doesn't work. And yeah, because this is not ready to be packaged. So. I have kind of mixed feelings about Distro's packaging pre alpha software. I think maybe they should just not do that.

It's not a service to anyone. It doesn't help their users and it doesn't help, it doesn't really help the projects they're packaging either. At the same time, it is also exposure for the project and like, you know, sends people our way, just you know, it's, it's, it's, it's, I wish that we would get to manage our own first impression, I guess.

Aaron: Yeah. I'm just surprised that, that, you know, those, those bigger players aren't like, you know, in your sponsors. And I guess there's a, there's an argument to be made like, Hey, most of the browsers are based on open source anyway. So, you know, I grew up in the days of Richard Stallman, well a little bit after when he was really going, you know, but in the 90s when he was still preaching, you know, all that stuff pretty heavily and everything has to be open source and you need a clean release of your distro and.

You know, back then it was proprietary browsers with proprietary code in them that you were fighting against. Now you could make the argument, well, good enough, right? There, everything's open source. It's good enough. We don't need this, but I am a little surprised that some of them haven't come on board and said, you know what we would like to, yes, well, there's a lot of open source out there, but we would like to separate ourselves from the commercial entities of the world, the Googles and the other players, because we don't want to be beholden to them.

So, yeah, I'm just kind of surprised that they're not in the list.

Andreas: Right. Well, they are very welcome to join the list. If anybody from, from any of these entities is listening, please get in touch. We would be happy to have them as sponsors, obviously. And I, I totally feel that way as you described that it's great that we have all these things that are open source.

I think that's a fantastic, amazing development that happened. The fact that all of the most important software on your computer today You can read the source code for it, modify it and publish your modifications. That's fantastic, but We can take it a little bit further than that, you know there are other things that matter too.

And it, I think it does matter where all the, this huge pile of money comes from. And it's kind of like this elephant in the room in a way it's been like that for a long time. I feel in the browser industry that there's been this browsers being developed by hundreds or thousands of engineers, and there's a huge pile of money.

But we don't really talk about like what that money comes from until recently when it's become much more public knowledge and people are starting to see like how much their search search queries are actually worth to these companies. Right. So yeah, that's been really, I think that's been really great.

And it's a really healthy thing that that information is out in the open so that people can start to maybe care about this in different ways other than only like, is it open source or not? There's like other, other parts of a spectrum here that we can care about. Yeah.

Aaron: Yeah. I, I kind of sarcastically.

wonder, like, you know, why did it take so long to get dark mode? And it was because it took that long for someone to figure out the, the, how to monetize or not how to monetize it, but the monetary value of dark mode, like, Oh, you know, we did these tests and we realized that when you have dark mode on, you spend an extra five hours a month, you know, in your browser.

And that means X, many more ads that you see. And that means, you know, so now we have to do dark

Jonathan: mode. And now we know why it's called dark.

Aaron: I, I have to have dark mode. I mean, my eyes aren't good anymore and dark mode really helps. And

Andreas: that's true. So it's

Aaron: like, do you have dark? Let's ask the question. Do you have dark mode yet?

Andreas: We have something like dark mode. So CSS has like a preferred color scheme properties these days. So websites can advertise that, like if the user prefers dark mode, then the sites should look this way. And otherwise it should look some other way. The problem is that there's a large portion of the internet that doesn't specify how it should look like in dark mode. And they kind of assume that the background will be white by default.

The text will be black by default. And if you start to violate these assumptions, then you break some content. So that part is messy. But we, we do have sort of state of the art dark mode. Yes. And I hope that it's something that will evolve further so that even older websites can display consistently in a dark way.

But it kind of depends on heuristics at the moment where we sort of have to just take a guess at what would be a great way to dark modify this website. Right.

Jonathan: I see on the website you talk about things like having a runway of funding and some other sort of business esque terms. And I'm curious, is this a business for you guys? Is it more like a non profit? Like, what does that side of it look like? What's your thought process there?

Andreas: It is completely a nonprofit.

We have no intention of ever selling anything other than maybe a t shirt and a coffee mug or something. But that's a nonprofit. We are a 501 C3 registered in California and we take. What's it called unrestricted donations only so You you can't give us money and tell us what we should do.

You can you can just sponsor and trust us to to do what's right and You know that that does Exclude a certain type of sponsor from ever wanting to support us, but it's okay. You know, we don't, we don't need all the money in the world. We believe, and I believe from experience that a small team can build a competent browser.

It's just you have to focus and you have to be more selective about what features you do. And. Stuff takes time. There's real elbow grease involved, but it should be doable. And yeah, on the sort of business side, there is no business model. It is either we get this thing funded by donations and sponsorships or it falls apart.

And it's a, you know, we're kind of going out on a limb here taking a chance, but we're hoping that We'll be able to deliver something in an alpha version that will make people see, okay, there's something real here, something worth getting behind and sponsoring so that we can actually have this thing for, for our species for and if we can, if we can get it to that point where people see that this is something worth sponsoring then we think that we can continue it and, and perpetuity funded in that way.

That's at least that's what we're hoping to achieve.

Jonathan: What does the community look like? Do you have people on the outside sending in patches? Obviously, bug reports. People come in and complain about things. You can't stop that if you wanted to. But, are you getting patches sent in? Are there outside entities?

Are there any outside businesses working on this? Saying, man, it would be nice if Here's the code to do this thing.

Andreas: Sure. Yeah. We have a fairly large community. So I think in terms of like active developers, we are seven full time engineers right now paid, I think. Oh, wow. That's impressive. So that's pretty good.

Yeah. And and they're all people I hired who were previously open source volunteer contributors. So it's, it's been lovely to be able to take and like give jobs to all these people that showed up and worked on, on Ladybird.

Jonathan: Absolutely.

Andreas: And it's, I would say it's even like one of the coolest things I've ever experienced was just to be able to do something for fun for a long time and then give people jobs doing it.

Yes. But yeah, so, so a bunch of us now are full timers, myself included. And we also have a, an open source community. I think we're usually maybe like 30, 40 active people like contributing multiple times a month. And then a long tail of people. Contributing either, you know, one once in a blue moon or once ever.

So, but it's, it's been growing slowly. And I think historically like working on browsers, as you mentioned, like it's really complicated to build Chromium or to build Firefox and because it's been a really complicated thing it's been a little bit hard for people to get into it, but Our project is fairly easy to get into.

It's a lot smaller than the existing browsers. It builds faster easier to, to learn. So we're welcoming new developers all the time who are sort of working on their first browser project ever. And that's been really positive as well. So I'm hoping to turn them into more frequent contributors.

And I'm also hoping that we will be able to fundraise a little bit more so we can hire a couple more contributors to be employees and But yeah, you mentioned business lingo on the website about runway. And indeed we are trying to be careful with, with the said runway because. We recognize that our funds are limited and the classic startup thing to do would, I guess, would be to hire as many people as we can right now and burn the money for the next six months and see what happens.

But that's not really compatible with our view on how this should be handled. So yeah, we're holding ourselves to a strict, like there has to be two years of salaries in the bank. At all times, because or we should aim to have that before we hire anybody new. Right?

Jonathan: Sure.

Andreas: And if that slows us down, then it slows us down.

But I don't want to, like, when I give somebody a job, I feel like I should It's my responsibility to make sure that that person has that job for a longer time. Then six months and us just burning through the Capitol to, to go faster.

Jonathan: Yeah.

What does the leadership structure look like? Are you a BDFL?

The Benevolent Dictator for life? Or maybe not for life?

Andreas: I don't know. So I was that. I've been a BDFL, but evidently not for life. I was that of SerenityOS until we forked, and then I sort of transferred ownership of SerenityOS to the group of maintainers that That I had previously invited to, to, to do that.

And and Ladybird, I think I'm not really the BDFL. I'm just the the president of the nonprofit that runs the project, but really the project is run in practice by the nonprofit and by the people that work on it, and then Code contributions are sort of quality gated by a group of maintainers, but there isn't any formal structure outside of that.

And I think it's something that will evolve and formalize a bit more as we get closer to, to releasing stuff. Or as, as our organization structure grows, but at the moment it's very like flat structure. Everybody's welcome to work on everything. Nobody is like.

We, everybody just kind of finds the thing that matters for making the web work in the browser and then they focus on that. It's possible that we will change that. We're kind of in a luxurious period right now where you can sort of like you, you, you throw a bug fix anywhere in the browser and it's, there's a real chance that it fixes some important website.

So And the future is going to be, you're going to have, you're going to have to look harder for valuable things to fix. But yeah.

Jonathan: Yeah. Interesting stuff. Aaron, did you want to jump back in? We're getting close to the, close to time. You want to make sure we're getting

Aaron: close. Like, I guess I don't want to jump too far into the weeds with this one, but maybe I'll just open up the can of worms anyway, and we'll see where it goes.

Jonathan: Just crack it. Let the, let the worm air out and not near the worms.

Aaron: Yeah. Sometimes these questions go in, in, in weird directions, take a long time to answer, especially for novices like me. But the question is architecturally speaking, like, what would you say are the biggest differences or even like some of the things that you, that you're seeing maybe some indications of like, this is going to be a game changer.

In terms of technology and how we're developing this, because we don't have to conform to. whatever chromium webkit, but we're able to do it this way. And that makes a big difference.

Andreas: Right. Architecturally, what are some advantages? I guess one thing is that we have a lot of flexibility right now because we're not big and complicated yet.

And there are a lot of features that we haven't implemented yet. And so we don't have like a gigantic code base that has to be retrofitted to do. Some some like security mechanism, for example. So like every other browser, they started out as a single process browser. But we started building our browser after multiprocess browsing was a thing.

So, we were able to get that stuff in pretty early. And as a result, we don't have a code base that, I mean, we were single process originally, but we were able to make ourselves multiprocess pretty quickly because we didn't have a gigantic code base with millions of lines of code that had been shipping in a single process way, you know, for decades.

And then we had to retrofit multiprocessing into that. And I, I think we have a lot of opportunities still to do interesting architectural things for, for security, for stability, for performance that are much harder for other browsers because they have so much code that, that just has to be changed in complicated ways, let's say.

Yeah, that's, I think that's our biggest opportunity.

Jonathan: Yeah, pretty cool. Why, why C Why with a new browser? Did you not write it in Rust or Go or some other everybody's favorite language?

Andreas: Right. Well, I guess there's no language that everybody loves, but everybody loves to hate C lately. And we started, I started SerenityOS in C because as I mentioned, there was a, it was a personal therapy project.

So I just. So the language that I knew the best and was just doodling around. And then I didn't mean to, but, you know hundreds, thousands of people ended up wanting to work on it also. And then it kind of just you know, I got a little bit out of my hands. And now we have this gigantic code base written in C plus plus.

And we have a lot of things going on and we want to find a way forward where we can You know, feel like we're doing our best by our users in the future, that like we're doing our best to deliver something that's safe and secure. So in practice, that means that we have to, we probably have to evolve past C in some way.

Because C is not evolving towards safety as, as fast as we would like. And so I don't know. We've looked at a bunch of the different languages recently and ended up with Swift as a secondary language to introduce into the code base. Not everybody's favorite choice, but we did this experiment where I asked people like, please try to implement some part of the browser in a couple of different languages and then tell me which one you like the best.

And. Everybody came back liking Swift the best and fair enough. It was a, it was an empirical process and I had the same experience. I liked it the best out of the languages that we messed with. And so we are aiming to introduce that now with the new version of Swift that is coming out in the next couple of weeks.

And the idea there is to have a safe language that we can incrementally introduce into the code base because Swift can talk to C and vice versa. And that's kind of, that's kind of our plan there, but it's going to take time to do that. But it's something that we feel like we have to do something because.

You know, safety is a thing people do want to hack your browser and we probably have more bugs than we know. I'm sure we have more bugs than we know. And if we could like systematically prevent many of them by using a safe language, that's something that we, we definitely need to pursue. But yeah, we are still like on square one with that because we're depending on the next version of Swift because it's the first one that can actually understand our super modern C that we've been using.

Yeah, so that's kind of where that's at. And I know that there are a lot of languages that people always ask me like, why didn't you use my favorite language? Or why didn't you use this language? And I just never engaged with that because I feel like Nothing good ever comes of, like, criticizing somebody's favorite language.

Right?

Jonathan: Yeah. Yeah, that's fair. Have you, have you been in So first off I've got to think there there are people out there to go Swift they went with Swift. Oh, they're selling out to Apple. That must be what that is Yeah, you've gotten much pushback from that.

Andreas: I Have been told by many people that like congratulations on the Apple money

But No, we haven't the the We were acknowledged in the sense that I think the the head of the DevTools department at Apple tweeted at us saying something like, cool. But that was it. Yeah, no, we're not, we're not selling out. It's just, it just so happens that Swift has a really compelling story as like a C plus plus successor language.

Like that's something that they're investing in. Yeah. Sort of this narrative of like, Hey, do you have a huge C code base and wish you could be safe, but it's just too much to just rewrite everything? Here's Swift. You can rewrite incrementally. That's really compelling to us. So.

Jonathan: That's interesting from the kind of the standpoint of looking at the language too.

This is something we're seeing with Rust as well, where they're putting Rust in the kernel. It's forcing Rust to grow up. And. If they gain traction with that, with Swift, trying to get other C developers to use it. And maybe Ladybird will be part of the story too. Putting it in use, they're going to find places where, oh, this thing in Swift is not working quite as well as we thought it did.

Or there's a bug in trying to make this work. I think that could be really interesting going forwards, as you guys I'm assuming you will try to, you know, kind of work closer with the devs from the Swift language. And it should be a good thing for like both projects.

Andreas: I think so. We've already found and reported a number of bugs and have had a good experience interacting with a Swift team so far.

So that's, that's really positive. And I think it's absolutely the case that, you know, whenever you take a language and bring it into a new domain where it hasn't been tried before, you're going to find tons of interesting and. Not interesting problems. And we've had a lot of not interesting ones also.

I will mention like trying to get CMake to understand how to build.

Jonathan: Oh

Andreas: it's not very interesting.

Jonathan: No, it's just pain working with compiling tool changes. Just pain.

Andreas: Yeah. Thankfully though, we have somebody who, who derives. Some amount of pleasure from, from working with build systems. So I'm really grateful that we have Andrew who's been figuring that out.

But yeah, no, we're, I think a lot of good things will come with that. Just like you mentioned with Rust and Linux and everybody has to grow up a little bit, everybody has to compromise a little bit But usually something good can come out of that.

Jonathan: Yeah.

Andreas: I think it will be the same for us.

Jonathan: Absolutely. Is there anything coming down the pike that you are particularly excited about in Ladybird that you want to want to plug, let folks know about?

Andreas: Well semi related, I'm running a a coding jam this weekend called Browser Jam.

Jonathan: Perfect.

Andreas: Where people are invited to come and we're all gonna write a new browser over the weekend.

Oh. And I don't think it's ever been done before. People do game jams where they make games over a weekend. And I We thought that we could do a browser jam where everybody makes a browser. So the idea is, you'll show up on Friday afternoon, and we will give you a piece of HTML, and then you have a weekend to build a browser that can render that HTML.

We'll see how that goes. But if you're interested, you can go to github. com slash browser jam, and you'll find the information there. That's great. That's a lot of fun. That's

Aaron: kind of fun. I mean, you know, that's what, I mean, that's what I like about doing that kind of stuff with HTML. ESP 8266s and 32s in the Arduino, like, stack just to like, you know, you don't need much there, right?

Kind of like the project you were talking about before, Jonathan, where I just need to read an HTML page and maybe some little JavaScript and make it work, you know? And if you can get to that point and understand how it all works, that's a great learning project. I think it's

Jonathan: Absolutely. It may be early in the project for this question, but is there something that someone has done with Ladybird that just really surprised you?

Like just off the wall or odd or, or other otherwise surprising that somebody has used it for?

Andreas: It might be a little early for that. Yeah. But I am frequently surprised whenever somebody gets some really complicated website working. I am just so surprised that we're already at the point where we can like do Facebook or we can Log into YouTube or things like that So it's more like i'm just continually surprised at the progress that we're making it's been really inspiring and especially because i'm Maybe not.

Yeah. We don't have a lot of people with previous browser experience on the team. It's me. And then a couple of people have been contributing a little bit here and there, but most of us are like complete noobs at this and just sort of learning as we go. And the fact that relatively or very inexperienced team has been able to assemble a browser of, of even this quality level in this timeframe is I think amazing.

Jonathan: Yeah.

Andreas: And I'm really proud of, of the team and, and it's been awesome to get to see people grow. Like we've had people join the project when they were like 16 and they're now in university and still hanging out and chatting and like fixing stuff. And that's been a, it's been a lovely process to witness.

Jonathan: Yeah. Question from the chat room, mashed potato wants to know, is anyone live streaming the browser jam event? Is there some place where we can watch people code in real time?

Andreas: That's possible. I don't know. So we're, we have a discord server where people are coordinating a little bit, making teams and stuff like that.

And I saw some people were talking about the possibility of live streaming. So if you're interested in that join the discord it's on the github. com slash browser jam. And yeah, see if, if somebody there is streaming, hopefully somebody will.

Jonathan: Sounds fun. We are, we're getting down towards the very end of the show.

Is there something that you, you wanted to mention to folks that we didn't ask about? Did we miss anything?

Andreas: No, I think you covered a lot of the stuff that's important. And I'm glad that I'm glad that we talked about the issues that exist in the industry. I feel like it's easy to, To forget to acknowledge that, but it feels like we're in this new reality now where we acknowledge that browsers have been funded by Google and it's no longer a secret that only some people know about, but like it's something that everybody understands now and we can start to look.

Real solutions to, to that. And I think we're trying our best to offer one possible solution. And we would be very happy if, if people experiment with other solutions. I don't think that Ladybird has to be the only new browser. I know that there are some other. Up and coming browser projects as well.

There is the Servo project they're doing browser engine in, in Rust. Mm-Hmm. . And they're very big on sort of the embedded use case. And I'm glad that they're exploring in that direction. And I think there's room for more. I'm, I'm really hoping that there will be sort of a, a new age of new browsers.

I guess that would be really, really fun. As a, as a fan of browsers, I would love to see that.

Jonathan: Yeah.

Andreas: Yeah, so I guess shout outs to Servo for for also going down this path.

Jonathan: It, no, no, that you mentioned them. I'll have to see if I can reach out and get somebody from Servo on the show too. Cause that would be a lot of fun.

All right. So before we let you go,

Aaron: one thing before you drop, what's the best way, easiest way for people to get involved or the best way to get

Andreas: involved? Right, of course. So the easiest thing you want to start with going to our GitHub repository. So it's at github. com slash ladybird browser and join our discord server.

That's really where all of the day to day coordination happens. People like working on stuff together or discussing bugs. It's fairly pleasant space. People are very nice, welcoming, professional. And as I mentioned, like we're welcoming new developers all the time, so you won't stand out or be weird, even if you don't know what you're doing.

Join our Discord and come chat. And what I usually tell people to do if they don't know where to start is to Build the browser, and then go to a website that you made yourself. Hopefully you have something. Hopefully it's pretty simple. And see if it works correctly. If it doesn't try to figure out why.

And see what you can learn from it. And maybe you can figure out how to fix it, but at the very least you can make a pretty good bug report even if you can't figure out what to do. Yeah, and then take it from there.

Jonathan: Yeah, very cool. All right. Last two questions that we are required to ask or the chat room gets mad at us.

And that is what's your favorite text editor and scripting language?

Andreas: Oh, well, it's, I don't know what my favorite text editor is, Vim. And I, it's just muscle memory, I think, even if I, I've tried to install other text editors, but I never used them because to start a text editor, I just type Vim. So there's that.

But I, I tend to program in IDEs. So like I do like in practice, a lot of my text editing of source code, I end up doing in like the JetBrains IDEs or VS code or something like that. But on the command line, it's always Vim. And then scripting language. Oh, scripting language, right. Probably has to be JavaScript, just because I keep working on it every day.

And, or no, I'm going to tell the truth. My favorite scripting language is PHP. And I'm not ashamed.

Jonathan: Well, having a background as a C programmer, PHP is kind of friendly towards the whole C style of bracketing and all of that. So that's not terribly surprising.

Andreas: Yeah, true, true. But it's terribly out of fashion, I feel like.

Jonathan: Oh, it is. It is very much out of fashion. But PHP still holds a place in my heart as well.

Andreas: Yeah, it's like, it's like Perl, but slightly more modern. Just slightly. Tiny bit. Tiny bit. Yeah.

Jonathan: All right. Hey, this has been awesome, and we sure appreciate you being here Andreas, and thank you. Thank you for your time. Thank you for telling us about the project. Yeah, thanks for having me guys. Yeah All right.

What do you think?

Aaron: Yeah, it's pretty cool. I love learning about these projects early on, you know, and having the opportunity to kind of get in on the ground floor, so to speak. You know, I often, I often tell When they ask people, when people ask me, what, what is Floss Weekly? You know, I have to explain it.

I'm like, well, you know, we talked to the Kubernetes guys, like when they were still at Google and the project was just getting started. Like that's a big part of what. Over the years, just organically, I think has happened because people want to come on the show and talk about their project when it's in the early stages.

So it just kind of works out organically that way. But the benefit for us and for listeners and viewers is they get to hear about these projects and get involved. You know, on in the kind of the early stages and

Jonathan: yeah, I think this

Aaron: is another great opportunity like we we We've just come to the point where everything has consolidated so much in the browser space that we just need another alternative if for no other reason than to have another alternative.

So we're not locked in. So yeah, I think it's a great project. You know, I guess stay tuned, right? Like either, either go try it if you're the type that. Likes to build things, build your own software, then go try it. And if not, then wait a little bit and come back and check it out in, you know, another six months or a year when there's a beta version or something.

Jonathan: Yeah. You know, I was thinking as we were talking so much of computing now is just the web. Right. And so we used to talk about, you know, your favorite. We still, we asked everybody that's on the show, like, what's your favorite text editor? We do so much work on the web. Maybe we need to start asking people what their favorite browser is.

But to kind of carry that, that thought through, you have different text editors for different purposes. And I could, I could see a future where you have a bunch of different browsers for different purposes that, you know, are not all just reskins of Chromium, like we started the show talking about. So yeah, no, I think this is, I think this is really cool.

And I really do think that there are going to be Some really interesting, maybe niche, but some really interesting use cases where it's going to make a lot of sense to run Ladybird or Servo rather than Firefox or Chromium. Yeah,

Aaron: yeah. I like the ones you came up where the, the one that you came up with, and I think that branches out into others where you need to run a kiosk or you need to run something and you don't want to be encumbered with maybe even the fear of the unknown, right?

Like who knows what. Chrome is going to do or Chromium is going to do how it's going to change. Right. You need something stable. You need something that's going to be, you know, you can count on. And yeah. And, or I just want to, I just want to run a kiosk, like you said.

Jonathan: Yeah, definitely something to be said for that.

All right. Did you want to plug anything, Aaron? I know you've got that, you've got at least a YouTube channel, which you'll mention.

Aaron: Yes. I will give you a preview quickly of an upcoming video. It is September. So I've got two channels. I've got the main channel, which is RetroHackShack on YouTube. And I've got RetroHackShack After Hours, where I do e waste Wednesday and a bunch of like smaller, like maybe slightly less interesting to the broader audience kind of things all around vintage computing.

But on the main channel, what I've been working on for months, people don't know behind the scenes how long these history video takes, history videos take, but I'm working on a history video. It'll be relatively short, but on the history of the Tandy

Jonathan: 1000.

Aaron: So the Tandy 1000 was kind of. They kind of fell into this position of being a major player and having a really successful product because of the failure of IBM's

Jonathan: PCjr.

Aaron: So it was really interesting history where Tandy came along and they said, Hey, we're going to make a PCjr competitor. And then PCjr got discontinued. And they just happened to have like all the features that people wanted at that point and ended up being a super huge important and for a lot of people either their first computer, their first family computer, you know.

So that video should be coming out I'm hoping by this weekend, fingers crossed, but they just, those history videos, you have to go in and get credits for all the images and all that kind of stuff. And it just, it just, it just takes a long time to do. So yeah, be on the lookout for that.

Jonathan: And it's RetroHackShack and RetroHackShack After Hours.

Aaron: Yep. Yep. RetroHackShack everywhere. Just search for RetroHackShack on your social things or on YouTube and you'll find it.

Jonathan: Yeah, awesome. Alright, I do want to let folks know that next week it's, we're back to Java! And it's JBang, which it lets students, educators, and professional developers create, edit, and run self contained, source only Java programs with unprecedented ease, which, Sounds like we're doing Java for scripting, which that'll be really interesting to talk about looking forward to that.

So that's next week. Make sure and catch that you can follow me. Of course on Hackaday, we've got, well, that's the home of Floss Weekly. That's also where my security column goes live every Friday morning. There's also the untitled Linux show that's over at twit. tv at the twit network. And we have a lot of fun there talking about what's going on with Linux, the news, some open source stuff there as well.

And then. You can also keep an eye on my YouTube channel. You can search for me. I think it's Jay Bennett at YouTube. And some fun stuff mainly around Meshtastic, which we're about to have a big 2. 5 release the, the first alpha of that come out as some really exciting stuff there. So check that out.

We sure appreciate everybody that's here, both that watch us and listen to us live and those that get us on the download. Make sure to come back next week and we will see you then on Floss Weekly.

Episode 799 transcript

4 september 2024 | min

FLOSS-799

Jonathan: Hey, this week, Laurie LaRusso and Steve Hoffman join me to talk about Percona, the open source database solutions company that after all this has managed to keep open source, we talk about how and why you don't want to miss it. So stay tuned. This is Floss Weekly episode 799. Recorded September 3rd, still open source at Percona.

It is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today we're talking about Percona. It's an open source database software. It seems like anymore. It's one of the few opens big open source database solutions that still actually open source goodness.

I think we're going to talk about that just a little bit during the show today. I, I don't know much about Percona. I, I kind of have to admit my database experience is sort of just My SQL and SQL light and a tiny, tiny bit of fighting against Microsoft SQL. It's been, it's been something because of because of things happening outside of our control, I am the host and co host today.

So that's going to be just a little bit different, but we've done it that way before. It's, it's not going to be a big deal. But let me go ahead and bring our guests on today. And we've got Lori Lorenzo and Steve. Hoffman and we talked a little bit before the show and Laurie, you are the, the community manager and Steve is sort of the, one of the VPs of, of technical.

Is that, is that about the way that, that lays?

Lori: Yeah. So just, my name's Laurie LaRusso, like Karate Kid LaRusso and yeah, I'm head of community at Percona and yeah. Yeah.

Steve: And I'm, I'm one of the engineering managers or engineering leads here for Percona run the the development side of the house. So, Loray and I are joined at the hip.

Jonathan: I, I understand. I know how that goes. So, let's, let's talk for just a minute about, I guess the technical side is where we'll start. And, what, what is Percona? Like, how is it different than, you know MariaDB? Or any of the other database solutions that are out there? What's, what's the kind of the, the thing that makes the difference about Percona?

Steve: So it's not wildly different than MariaDB's original model. We both took a fork of MySQL and, you know, added our own features to it. What we think, you know, was the, the answer to the needs of the market, you know, from an enterprise standpoint. We've also since done that with MongoDB and we're also working in the Postgres space as well to bring new features on top of an already great base.

So in that sense, not wildly different. I think. You could say we're similar there, but then I think we go a lot further than them. So we also have our Operators product. We also do our, our monitoring and management suite. So maybe we have a wider portfolio, but I think it's fair to say MariaDB model is similar to Percona.

Okay. And

Lori: so this is where, let me just jump in. This is where I think Steve is being humble and that's why you have a community person to back him up. So, you know, I think the, what you're missing is the innovation, right? So we have community editions of MySQL, of Postgres but what we do We make them better, and we're open source.

So, we take, we take the regular edition, we open source it, and we add in our little spice to make it better for users.

Jonathan: So is, is Percona a database? Or is Percona a database company? Or is it both?

Steve: We like to call ourselves a solutions company. We produce database software, and and you can get it in all different formats, but we make the majority of our money on the services that we provide.

But when you take our software and our services and sort of smash them together, you get sort of the overarching Percona solution.

Jonathan: Okay.

Steve: You don't have to use both, but I think that's where you get the best value is our software with our services. We can, we can help you run it better than anybody else.

But if you're using, let's say a community edition. Of either Mongo or Postgres or MySQL. We can still service you but it's just, you don't, you don't get the full value. Okay.

Jonathan: And, and, I, as you, as you work on some of these pieces of software I'm sure, so you make software changes to them, you make changes to the source code, because it's source available without being an open source.

Do any of those changes ever get pushed back up to the original project? Or do they kind of all stay siloed in your, I mean, obviously it's open source, but still, you know, there can be either an effort or not an effort made to push them up.

Steve: Yeah, I mean, we like to contribute everything that we do. Now, we have learned over the years that certain things aren't really necessarily welcome or valuable back upstream.

In many cases, we're actually building enterprise features that already exist in the upstream's enterprise product. So, us contributing that backup there, you know, they don't want that. But we do bug fixes, we do, you know, extensions and enhancements all the time. And again, we do it all, make it as open as humanly possible.

Direct contributions. You know, and hope that we get it included in, in upstream so that it just everybody gets to benefit from the change.

Jonathan: Yeah, I mean, every, every change that you can make upstream is just one, one fewer change for you guys to have to be the stewards of. That makes sense. That's, that's kind of interesting that you're re adding some of the enterprise features that have been not added intentionally.

Is there, is there any animosity from any of the upstream vendors because of that?

Steve: I think on the community side, there's no animosity. I can't even say that I'm aware of it on the enterprise side. I'm sure, you know, if Percona got way too big, people that haven't taken notice certainly would, but we seem to have a great relationship with the MySQL community, certainly the Postgres community.

And Mongo, you know, I mean, I don't think we have each other as adversaries. Business wise, surely we're competitors, but I don't think it's. I've not received any death threats, I don't think any of you on the team have, so

Jonathan: we're good standing there. You've managed to remain friendly competitors, that is handy.

Yeah, exactly. That's fun. Well, but the database world seems to be sort of cutthroat these days. We talked just a little bit before the show about like all of the different licensing changes that have happened and forks because of it. And it seems like, well, no, I know one thing that has happened is you'll have a company that they have a database, like they have their secret sauce in their database.

But instead of making it all secret, they've made it open source. And so they've got this database code that they work on, and then they also offer it as a service. And like, that's great until Amazon comes along and says, it's open source. We can offer this as a service too. And suddenly this, this, this business that does open source, like their revenue stream starts to die, dry up.

And they immediately go, we've got to make some kind of change to be able to keep making money. And it seems like in a lot of cases, the change they make is we're going to use a business source license of some sort. And. It, it, it, well, it fractures the community. It, it inevitably results in a fork, you know, like Redis and Valky.

We, we now have Valky because Redis did the exact same thing and they, they, in their wisdom, they made this decision and they made the announcement to you. Move away from an open source license with Redis during a, I think, a Kubernetes conference. And so there were a bunch of people that were using Redis all together in the same place when this was announced.

And apparently by the end of the conference, they sort of walked away going, Yeah, we have a gentleman's agreement to make a fork and call it, you know, all get on board with the same fork. But it's just it's got to be it's got to be cutthroat.

Lori: Is it a

Jonathan: cutthroat environment? Have you seen that?

Lori: Let me take your, your Amazon example for one.

And I think that's where you come up with strategic alliances, right? That's where you really work within, I mean, it's a cloud company, right? It's a hyperscaler. So if they can. That's what the marketplace is for. If they can package up what we do and add it to what they're doing, I think that's a win for both, right?

So the way that we treat our core customers, the level of service that we provide is super unique. So to have a place for us to, a platform for us to list out like our version of MySQL or Postgres, I think it's, it's a win for both. And then when you think about how the community kind of rallies around, you know, open source and wanting to keep things open I think For me, the biggest shift I saw was when Terraform went closed and there was a manifesto.

The community was so irate that they came up with a manifesto. And then from the manifesto, like they ended up The, the name of the project now is OpenTofu, but it, it started, I think it was OpenTF and it's underneath the Linux foundation, just like Valky is now underneath the Linux foundation. And I think when you have something that's open, that people use it and adopt it because it's open and they build their businesses off of it because it was open.

And then you close it.

Speaker 4: Yeah.

Lori: Then you have like this big community uprising, specifically if it's something so popular, like Terraform or Redis and to be a company, to work for an open source company like Percona that is there to carry the open source flag and say, okay, so Val Redis closed source, guess what, we're going to be a part of Valkyrie.

Like we're the only ones that offer, you know, like on prem support for that. And in that look at the other companies that have joined Valkyrie. You've got your AWS, you've got your Google, you've got your Oracle. So it's like. All of us working together to have a piece of this pie, if you will, in open source land, I think it's, it's, it's really awesome to be a part of a company that supports open source, that it's that core to who we are.

Jonathan: And so is it fair to say that Percona's business model is not simply hosting databases for companies? There's more to it than that.

Steve: Yeah, perfectly fair to say. What's the, our model is actually you hosting your own database. So we give you the software that lets you. Run it on whatever your infrastructure is.

So if you want to run it on prem, if you want to run it in the cloud, if you want to run it across multiple clouds, we, you know, we actively support and test all of those scenarios that's given the power to the, to the people. Yeah, yeah. And

Lori: I, and I think what we do is we help you level up, right? So we have not only the innovations that are coming out of engineering, but we have the experts in our support and our managed services.

So if you are trying to elevate, if you are trying to do something and you realize you don't have the staff or the support in house, that's where we come in. So having that, that like hand to hold you through a migration or to help you tune your database so that you're getting the best results to take your database and put it on the cloud with our new product, Everest, like just because we're open source doesn't mean, you know, that we are a less than we are actually like a

Jonathan: No, absolutely.

I agree with that. I, I think part of the problem is that so many companies, they, they love the open source model, but they've not really figured out how that model actually ties into their business model in a way that makes sense. And sometimes they end up sabotaging themselves. And it, it sounds like Percona has managed to avoid that.

So that's, that's good. That's a good thing. That's a good, good sign for the future. We just celebrated

Lori: 18 years. Oh, wow. Right, Steve?

Steve: Yeah. Yep. 18th birthday. We became an adult. Percona can now legally Drink and phone? Do what 18 year olds do. Well, just different in every country as we learn. We did a poll internally about turning 18.

It was like, wow, it's amazing. Globally 18 means different so many different things around the world, but that was a lot of fun

Jonathan: What was what was the beginning of the company? What was the what was the first thing that percona did was it was it literally my sql stuff My

Steve: sql

Jonathan: phone support. Yep phone support really Ah,

Steve: so so our our our co founders were actively taking calls from customers to help optimize their databases.

So they were a consulting company. So if you think about Percona, it's a per con performance consultants and dot com was taken. So per con a dot com came to be. So yeah, little, little trivia history on our name, but yes, Peter and Vadim took many, many phone calls. I think, I don't know the exact date, but we have a book.

That they put out at about the 14 year mark, or maybe it was the 13 year mark that said, like up until at the time, eight or so years ago, they were actively taking phone calls from, from customers who were having performance issues. Fortunately we have enough staff, we have, you know, smart enough people that they don't have to anymore, but that's how we got our

Jonathan: start.

Oh, that's, that's great. That's, that's a lot of fun. And, and so what's the, like what's the portfolio now? What, what are the different databases that you guys will. Still take support calls for her.

Steve: Yeah. Yeah. I mean, we're, we're active. Like I said, my SQL is what most people know us for MongoDB, probably second.

But then we, you know, we're very active in the Postgres space and getting more. So every day and then from there, you know, where you choose to run it, we support, like I said on prem bare metal installations, cloud installations. We create operators now, so you can run this in Kubernetes. So in your orchestrated environments.

Which we're seeing a ton of traction there. We're now releasing you know, if you think of Amazon's RDS is sort of like database as a service, you know, point, click, get a database. We have an on prem equivalent to that in our Everest product. So it still uses Kubernetes and work and operators underneath, but it gives you that that single API that you can call and wire into your CI CD or.

Or, or turn over to your, to your developers and let them point, click and create databases as opposed to having to know all the, you know, behind the scenes commands.

Lori: And you forgot Valky. So Valky is our latest database that we're going to support.

Steve: Yeah. And Valky, we are only, we only support it. We don't actually build our own version of it.

We contribute to it. We're, we're active in that Valky community, but there's not like a Percona fork of Valky. So Valky, we, we're trying to. Support that community not compete with them by any stretch,

Jonathan: right? And so i'm sure you could you could imagine a future where You have a slightly different take on something and you do end up having an in house valky fork.

But I i'm sure trying to avoid that if possible, right? Yeah.

Steve: Yeah, I can't imagine that future I don't really see it. I don't ever see it as good right like I mean it it If we're really community, right like and we mean it then We find a way not to fork now. I can't fault any of the projects that have had to go, you know, go their separate ways.

I know there were certainly valid reasons, some of them personal, some of them technical, some of them vision, but I don't know that that's always the best outcome. So for me, that's like the last thing. If we've hit that, hopefully we've exhausted all options because, you know, to take to take a thriving community and effectively split it in half It's gonna slow, slow innovation down, right?

That's one of the beauties of open source is just how fast we can innovate when, when all of us are focused on the same thing. So again, I'm preaching here. This is just my personal belief, but yeah, I don't, I don't love that, you know, direct competition with each other. If we, if we Percona fork, first and foremost, we want it to be to help innovate and add back to the community.

But in some cases when that, you know, when that, when that takes its turn, like, you know, for us, we do have a fork of MySQL. Oracle Enterprise isn't going to accept our LDAP authentication. They have their own, right? They don't need us, but, but we do feel that it's good to bring to the community. So I think, like I said, we do it for a slightly different purpose.

Jonathan: And so, for example, we've talked about Everest a couple of times. So that's, that's fully open source. If, if somebody wants to just host their own version of Everest and not have to pay you guys anything for the contract, that's out there as an option. And then your, your business models, essentially, this is a complicated thing to do.

We suspect that eventually some of these businesses are going to want to have support for it.

Steve: Yeah. Yeah. I mean, that's been our model is if you've got the manpower, the time and the talent to do it yourself, you Our software is free and open source. If you don't, we can certainly help with that. You know, that's been our model and that's fortunate for us that our business model has been successful enough that we've been Continually growing and continually profitable company since the beginning.

Jonathan: Yeah, well, so I mean that that business model has worked for other companies over the years That's essentially what red hat was founded on and it it grew them to be a billion dollar business so there's there's definitely nothing nothing wrong with that And that is that is proven out to be a a winner a winner in many industries at least yeah, that's very neat.

So I'm, i'm curious in addition to that. Do you ever get someone say You Call you up and say, Hey, it would be great if my sequel did this. It would be great if Postgres had this feature. Can you give us a quote on adding that to the code base? Does that happen?

Steve: That's

Jonathan: only on the days that end in

Steve: Y. Yeah, we get, we get that a lot.

Like every customer has a unique need. You know, that. This is the way our business works. We need this feature, you know, and we have a product team that does the evaluation. Is this solving a problem exclusively for you or is this a problem that many, many people are experiencing? And by creating it, we can we can benefit the broader communities.

We try to stay on that, that, you know, community based side where we're looking for large swaths that would be impacted for the products. And that's not to say that we don't do some, you know, custom stuff. We have some customers that have specific needs. If the ask isn't huge. We put it in there. And there's a, there's a monetization model for that.

I mean, sure. But I

Lori: think that is that is the value of having community live under engineering as well, right? It's this feedback loop. So as many customers as we have, as many conferences as we go to, being able to have the pulse of whatever community we're in, if it's like an all things open, it's just a general open source.

Speaker 5: You

Lori: know conference, but if it's a PG comp, we can really focus our attention to that particular language, that particular particular database. I'm sorry. And, you know, really work within that sort of feedback loop. What are we doing? Well, what are we missing out on? Like here is a talk that we're giving that talks about why we decided to do this, you know, and And I think that's where community wins, right?

Again, Percona being an open source company and community has always been at the forefront of what we do. And then being able to really like zero in on the events that we go to and having like a forum where you get answers, having a very good resources within Percona to give that knowledge out to the community is a win win.

Jonathan: And so that does kind of lead onto another question about that community effort. What, what does it look like as far as in, in these places where you guys are the, the, the source code, like you have your own. Your own repos for this. Do you get a lot of patches and work from the community? Or, or is it pretty much just a a one way street where you guys do the work in the community or you're out there using it?

Steve: mean, I can tell you we get a ton of contributions. And, and, you know, community contributions don't always come in the form of long lines of thousands of code changes. Sometimes it's improving our documentation. Sometimes we get a lot of community give back on our forums where they'll come in and answer questions for, you know, for the masses.

Like, how do I set this up? How do I use it? I'm getting this error. So we're very, very active on that front. That's one of the things that shocked me. I don't. come from open source, like, you know, way back when this is something within the last five years for me, but seeing how active our communities are is the exciting part because number one, that the entire burden doesn't fall to us because I think if it did you know, and maybe this is where some companies go wrong is you know, the community is not paying your bills.

So you tend to favor the ones that are but because we have such an active community, it allows us to work together. And, and accept code contributions, accept documentation updates, accept you know, answers on the forums, things like that.

Jonathan: Are there, are there any, are there any rough spots? So I'm, I'll give you the background to what I'm thinking here.

I'm part of an open source project and we just recently got sort of called out in our one of our, one of our places where people can ask questions. One of our forums of essentially you guys made this big change and you didn't ask us about it first. And it's, it's been a little bit of a thing because like there's, there's a couple of ways you can look at this.

For one thing, almost every open source project is a meritocracy to some extent. And so, one of the answers that we give is, if you guys would like to see changes, come write source code. Come be a part of Discord, where the, where the developers are talking so that you can give feedback. But it's really, it's really become a little bit of a source of friction.

There's some other things going on, but not important. Of course, it's always more complicated. Yeah, but I'm just curious. Do you guys, have you guys had any of that kind of friction and how have you solved it?

Steve: We, we have our own forms of it. I don't think we've seen that specifically, but with, you know, every, for the products, That we are in control, like PM and Percona monitoring and management.

We don't have an upstream necessarily beholden to, I mean, we do have you know, a copy of Grafana in there that we base it on. But, by and large, we're driving the, the roadmap. And a couple of times we'll deprecate a feature or, you know, we'll put something in that ends up not catching on and so we sort of abandon it or start pulling it out.

And so we definitely hear about it.

Jonathan: But I think I think it's impossible to deprecate a feature without someone coming along and saying, Yeah. That was part of my workflow. You know, it's the XKCD comic about holding down the spacebar, heating up your CPU. That was part of my workflow. Put it back.

Steve: Yeah. We get that, right? Like, but you realize we took that out because there's a better way to accomplish this. Right. And so most of the time we luck out with, you know, sort of a re education on it. We, we didn't Take something away without giving you something back. Here's, here's why. And, you know, it hasn't ended sourly.

I can't say that people are like, oh, thank you for that. But they, they at least walk away with an understanding of like there, there was a rationale behind it. It wasn't. Who can we piss off today, you know?

Lori: Also, I think it's a it's a matter of communication, right? Like it's a matter of giving people warning letting them know what's going on being able to have you know I don't want to say standard talking points but you know those standard talking points as to like this is why we did it and because you you don't want to Detract from the end reason, right?

Which is maybe there's a better way, or when we actually looked at our logs, nobody's using this in terms of like the hand scheme of things. So having a good communication and like Steve said, like having the forum as a place where people can converse, you know being able to kind of contribute back, submit issues that I think is, is key to keeping the community Safeguarded from some detractors.

You know, it's always the, the small group can sometimes make the loudest noise, but when you're just very transparent, like in the whole methodology of being open source, I think, you know, you end up just winning long term.

Jonathan: Yeah. I was, I was going to ask you, Laurie, to, to speak to this next, because sometimes the community people can be on the other side of the fence from engineering.

And if there's not, if there's not decent communication between those two teams, they can be actually working at cross purposes. What, what kind of things do you guys do to make sure that doesn't happen to make sure that the community team, so this is something that just recently happened to us the community team got surprised by a change that the engineering team made and it's like, Oh, that's gotta be a bug.

We're sure to get that fixed. And then the engineering guy is like, that wasn't a bug. That was an intentional change. Like, please tell us about these things. How do you, how do you, how do you avoid that? Yeah.

Lori: Well, I love that my dog just started barking. So I apologize. Hopefully it won't

Jonathan: be

Lori: too much in the background, but I'm very happy to say that at Percona community has moved under engineering.

At other companies that I've been to, community has always fallen under marketing and that's where you get the disconnect, right? Because marketing has marketing objectives and engineering has their objectives and when there isn't open flow communication across the board, Then you have those issues where you're like, wait, I'm presenting the wrong version of something or I didn't see the release notes.

Nobody told me. And it's it's a communication issue. And so I know, don't laugh. It's she's old. She's 14. She doesn't know what's happening. She's just like, why is mommy talking to a computer? So So now that community has moved under engineering, it's so much better because we are on all of the product updates.

We are looking at the roadmap. We are making sure that we are dropping breadcrumbs to innovations that are going to come out in like four or six months, right? We're letting the community know like why we're thinking the way that we're thinking, what our engineers are doing. And then with that, you know, we have, Surveys that are just product led surveys that are just like, we want to know what version you're using.

What is this? Like, how is this working out for you? Like, what can we do? And so keeping a really tight feedback loop like that is where I think community and engineering can just sing. So it's been a night and day from previous companies to be under the engineering org and showing them our value, right?

Like our tech team, our tech evangelists, our developers, our engineers, like they have. Like the history within their own career of touching source code, of building things, of doing testing and doing that kind of stuff. So we're also working to embed them more within engineering itself, like to be the ones that are helping to contribute, that are looking at pull requests, things like that.

So that I think is like, it's, it's new for us. And we're just kind of testing it out, but I'm very excited because the tech teams. are very like my tech evangelist are very excited to get their hands dirty. Nobody wants to be a showpiece like you want to be able to actually demo what's going on. You want to be able to speak from engineering and say, this is why we're doing things.

And that's what we're working on right now. And I think it's going to go really well.

Jonathan: Oh, yeah, that's interesting. Steve, what, what does that look like from, from, as I say, again, your side, your side of the fence, trying to bring your, your evangelists in and let them work with the code. Has that gone well?

Steve: Oh my god, it's better than you can imagine. Typically I would say, you know, we rely on community for the voice of the non paying user. Because we don't really have too many other avenues for that. And if you're not careful, your entire roadmap becomes the customer roadmap. So getting that continuous line of feedback from them, you know, of what does the community want is awesome.

And then even more so that they can be so hands on. And again, it takes all different forms. It's not purely contributing lines of code. Sometimes it's using it. Sometimes it's, you know, Hey, I couldn't talk about this at the next conference. This is too clunky. This needs some polish. And this is how I think it should look.

So, you know, the, the, the forms of feedback take all different forms, but they are wildly valuable. You know, you didn't realize just how much you missed it until you start getting it. You're like, wow, where would we be if we had thought of this. So much sooner you know, it, it, like Laurie said, it's been a fairly, not fairly recent change, you know, measured in six months or so, five months, something like that.

But it's been a great change and we're just keep building on it. So I'm excited to see it continue.

Jonathan: So is, is the next iteration of this to, to, to make sure that the marketing people know the source code too?

Steve: We might be asking too much. I don't know if they have an interest in learning the source code. At least the tech evangelists have an interest in it. And I'm not faulting the marketing folks. You know, don't ask me to design anything that looks catchy. I not a creative,

Jonathan: I get that. I just, I know I've heard stories of, you know, marketing people coming and asking just the most off the wall questions.

Like, could it be possible for us to do this? And, you know, engineering goes, I guess it's technically possible. And then a couple of months later, you see the latest marketing thing, this new feature coming soon. Yes.

Steve: Yes. That, that happens everywhere. That's every job I've ever been at that happens. That has nothing to do with open source though.

That's just human nature.

Jonathan: So it's, it's the marketing guys just trying to grab onto something new and flashy to put in. Yeah. That's great. Okay. So let's see. What, what does I'm curious, what, what does this look like then? So if you get somebody from the community that says, Hey, I've got this great idea, but they're not a paying customer, but you look at it and go like, that's legitimately could be useful for people.

Like what does the decision matrix look like for this to, to, you know, are we going to put some resources into making that happen?

Steve: Yeah, I mean, so I'm going to speak for my my product counterparts. You know, but but one of their big value ads is really assessing the overall market need. And it's not just, you know, take your idea at face value.

Is there enough people that might buy this? Well, that makes the decision, but it's Is there enough of an idea in your suggestion that might be able to expand it or might be able to take it in a different direction, maybe sometimes even narrow it. And that helps us get to a, Hey, we're onto something here.

There is a legitimate market need. And again, we do partner with the community team to make sure that we do vet those ideas before we just run right out and build it. Cause Lord knows that's expensive, but Hey, we're thinking of this thing. We got the feedback. It seems to be valuable. Let's talk about it at the next Pick your conference, you know and see what the feedback is if it's positive if you know if we hint that hey We're thinking about doing this and the feedback is oh my god.

Yeah, that'd be awesome. Then it just starts moving up and up the list You know, I think the biggest thing is Giving that community a voice and then listening to it. Not just, you know, here's a form you can fill out and, you know, black hole it. Right. Right. I

Lori: think one of the things, sorry, that I think that we're missing from this conversation that I'm happy to bring in is not just Percona related community, but community in general.

So Steve is my Guinea pig and I'm so happy to have to be on this podcast with you and one of the things, one of the communities that we are a part of is finos. And so FinOS is the financial arm of the Linux Foundation. And one of the things I was sitting on a call that they were looking for is contributions to one of their projects.

It's the something something control. I can't remember. It's three C's. It's CCC. I'm terrible. But it basically, they need help with, like, Expertise. And so Steve, I was like, I've, I've all told him that he needed to help me. And now we're contributing our sort of reference architecture for for my SQL in terms of like having a hardened, like security features for my SQL things you should look out for.

And so it's not just talking to ourselves or talking to our own community. It's how can we impact the broader community with our, our expertise and, and There's another project that we're working on it's called Gwok. It's in the OpenSSF, so it's a security project. Gwok is being built in Postgres.

One of my contacts was like, Hey, you work for Percona now, we need help with Postgres. We wanted to keep this open source project completely open. Can you help us? And it's like, Yes, of course we can. So we are now looking at their database as they're building out this security project. So it's, it's not just working within our own sort of Percona mindset.

It's what can we do at Percona to really elevate the community in general. So it's, it's a, it's a really exciting time to be at Percona because we are sort of flexing our, our skill set in other communities that maybe we haven't approached because You know, it's, it's not just the database, right? It's like, how does the database make your project better?

And so I think, you know, the next thing you think about is, is AI, right? And so there's all of these tools that are coming out and for us, it's, we don't want to be the next greatest AI tool, but like, Hey, what are your, what are you building out of you're building out of your database? Like, do you want to make sure that you are fine tuning your database that you're getting out of it, what you need?

So. Again, it's sort of like we are vastly more than just talking to ourselves, which I think is, is what sets us apart.

Jonathan: That's, that's an interesting point because, you know, Steve and I, we've been talking a lot about the, the, the community upwards to your upstream vendors and the community down.

Downwards to your users and Lori comes, it just brings along this idea of what about talking about the community outwards to sort of our peers and the other people doing interesting things. And that's something that we we don't think about quite as much. And that is, that is really interesting. And so what is the, what's kind of the motivating factor to want to reach out and work in some of those, some of those kind of vertical directions, excuse me, horizontal directions.

Lori: Well, I think again, it's like everybody has to store their projects somewhere. And if we can lend a hand in making your project successful, I mean, that is the value of open source, right? It's people coming together to build something, to innovate on it and to make it better, to solve whatever problem.

So for guac, it's an S bomb problem and where Percona might not really be interested in solving like S bombs, you know, like your supply building material, if we can say, Hey, in building Gwok, if you do this to your database, it makes it cleaner. It makes it better than at the end of the day, your project will be better.

Your project will be more successful. That I think again, is, is the value of going, To these other communities and showing what you do and how you can do it better. And when you think about foundations, like look, who's involved in foundations, you've got your Googles, your Amazons, your Azure's, you've got your Facebooks, you've got your Fidelity's, you've got Citibank, you've got all of these massive companies, you know, like what is it?

90 something percent of enterprise companies use open source components. So why wouldn't we want to share the love across the board with foundations that we're a part of?

Jonathan: Yeah, absolutely. I'm curious. Do you find that you're kind of your community? People have to sometimes do some translation work for your your engineering team.

And so what comes to mind with this is I do. I do a little bit of small business it for very small businesses around town, and sometimes they'll come to me with their ideas. The, the strangest asks, like, I need a computer with, and, and they'll say something that they saw from a magazine or on an ad somewhere, like, I need a computer that has one of those new NVIDIA workstation graphics cards.

You're a sprinkler system. You, you put. You, you put lawn sprinklers in what, and then, you know, so then there's this, okay, so what are you trying to accomplish? And then they'll tell me, and it's like, Oh, you don't need one of those really expensive GPUs. You just need, you need a new desktop with a new processor and a GPU in it that will run something with CUDA.

And then you can use your blueprint software. And so there's this, this kind of. Translation issue where they, they, they know what they want, but they're trying to tell you like the details of it and they don't understand the details. And I have to imagine that in the database world, working with, working with customers and even some on this, working with other foundations that this sort of thing happens to, we need a database and I don't, I don't even know databases well enough to tell you the sort of off the wall things.

I'm sure you guys get it though. People ask for off the wall things and it's like, well, let's help you actually come up with what this, what you really need.

Steve: If we're lucky, we can hear the, you know, what they're truly asking for. That's if we're lucky. We do rely a lot on, you know, the community team, the product team, our customer success team, some of our service, you know, services engineers.

Like, they are masters at hearing the problem under the problem that's being described. Right, right. If we're unlucky, we just start building based on, you know, I need this NVIDIA thing, you know. Yeah. And that's happened, well, once or twice, more than I care to admit. I'm sure. You know, I think then, then we circle back, like, okay, how do we miss this?

How do we get this out next time so that we're not just, you know, sort of chasing fantasies, but actually, like, getting to the core of the problem. Mm hmm. Because there's nothing worse when you build the exact thing that you were asked to build, turn it over, and they're like, it didn't fix my problem.

Like, what, what was the problem? Like, maybe we should have started there instead of just building the solution. But yes, yes. Yeah, I

Lori: love it. Because it all comes down to communication, right? Like, there have been many times and like, Steve can tell you where I've dragged. Somebody from engineering on a call with me, right?

Talking about community stuff and people asking for things that I have no idea, like I'm a marketing person, right? Like I have, I am not technical. I can try and translate to a point, but I need to pull someone in to be like, what, what, what exactly are they saying? Like, and then, you know, and then also like being able to take a step back to be able to stop a conversation and say, okay, hold on, like, Okay.

Let's talk about part a you've just went from A to Z. Like, let's just figure out what a is. And I think when you have a good team involved, right? When you have someone that can maybe take the marketing fluff and drill it down to the actual ask. And then you have the engineer that can take the actual technical side of things and and put it together.

You're gonna win. But when to Steve's point, when you just do one or the other, you could build something that's not necessary. And, you know, nobody wants that.

Jonathan: Yeah. Do you, do you get this with pull requests as well? I, I'm sure you must from time to time.

Steve: Usually you get the, yes, it is very, very specific to their problem.

Not, doesn't take into consideration how anybody else uses the feature. You know, but believe it or not, it's, it's less. Frequent than you might think more often than not, you get a, you know, very flexible contribution that's like, you know, this solves only my problem, but it makes extra effort not to step on anything else as a result.

So those, you know, those are the ones that we appreciate. But even the ones that are purely self serving will try to kick it back and say, Hey, you know, is there a way to do this a little bit more friendly to the other people that actually do want this to happen? And, you know, can we improve the. The error messages or the return codes or whatever the case may be.

So we try to, you don't want to tell somebody, you know, thanks but no, don't ever come back. So it's always to keep the conversation going. I mean, it takes an awful lot from someone to be willing to contribute code and databases. They're not easy. So if you find someone giving you database code, there's someone you want a relationship with.

So we do our best to keep that going.

Jonathan: Yes. Yes. It seems like databases and compilers are two of the really hairy problems in, in modern code. Not

Steve: for the faint of heart.

Jonathan: Yes, absolutely. Okay. Probably a question for Lori then. Have, have you guys had any real like pain points beyond what we've talked about with trying to wrangle the community?

I know sometimes there's things that come up that are just not pleasant to deal with.

Lori: So not since I have been at Percona, but I used to be the CDF, Continuous Delivery Foundation outreach chair. And in that there, it was kind of there was a wonky time, like they lost their executive director.

They got a new one. There was, you know, community didn't understand what was happening within the organization and the way that we, like, and there's nine or there's eight projects under the CDF and the, and the projects were wondering what was going on. And so what I did was I held feedback sessions.

Where I just let them tell me everything, the good, the bad, the ugly. There was lots of ugly. And it, it all stemmed from miscommunication or lack of communication and not understanding like what was going on. And, and from that, you know, creating an actual action plan moving forward, having everybody buy into it because they saw.

Bits and pieces of the conversations that they had within the plan itself, and then launching the plan and holding yourself accountable to the things that you said that you were going to do. And the change from when I started in June of 2022 to, we had CD con in, um, May of 23 was, or April of 23, whatever it was, was phenomenal.

Because everybody felt like they were a piece of, of the CDF, whether they were a project, whether they were a contributor, whether they were someone that just happened to use one of our projects, a user, someone that adopted it, it was just, it was amazing. And I think that is where community can really shine is when you do take a step back and say, okay, this isn't working.

How can we make this work? How can we fix this? And it, if it is sitting there listening, I had very many calls and The CDF is a volunteer position, so sitting there knowing that people are going to rag on you for an hour at a time, telling you how you're not good, like you're not good enough, you're not doing what you said it takes a toll, but it also I think is inspiring, because it makes you want to do the things that the mission statement says.

Excuse me. The mission statement says you chose to be at this foundation for a reason, right? Like you chose to use this project for a reason like we need to be beholden to like why we made that decision. Like we need to hold true to our core values as to why we're doing this in the first place. So it's, it hasn't happened at Percona.

I mean, I'm new, but I, like, I think again, the way that. Coming in from an outsider looking in, right. From a marketing person, that's always been community to a community person inside of engineering. It's really awesome to see the innovation from our engineers, the feedback from the community, the amount of push and pull that is all positive, you know, it's, it's really cool.

So happily haven't had to deal with anything like that, but again, coming from the CDF, it was just a matter of, you know, letting people's voices be heard and then creating an action plan and then being accountable for what you said you're going to do.

Jonathan: So I want, I want to let Steve answer that if he wants to, and then I have a followup for Lori, because this is fascinating.

Steve: No, no, I actually, I don't, I can't recall an instance like that where, where where there was just an up, upheaval. I think we, you know, I think we're very cognizant of how you treat the community. You asked earlier about, you know, how do you, it, you can't just be. In your own little community, you can't expect everybody to come contribute to you.

So we have to You know, that's why we share our expertise outward So I think because we're very very aware of that. Maybe that's helped us dodge some bullets but I can't I can't remember my five years here any ever being sort of thrown at us I don't think it's luck. I think it's you know, we're very intentional about how we get involved in the various communities either the ones that we depend on our own or the ones that we depend on

Jonathan: Yeah, makes sense.

Okay. So, Laurie, let me follow up on that with, with this question. When you have those conversations with community and you do come to the conclusion that, Oh, there's a In the organization, the broad organization, there is something that we've done here that maybe it needs a course correction. How do you get buy in from those in the organization to convince them that there needs to be a course correction?

Like that seems like that could be the hardest part here. How does that work? Do you have any secrets for us?

Lori: So this is actually funny. So when I first started with the CDF, I had a nail in my tire and I had a meeting with the core team from the Linux Foundation that worked on the CDF, and I didn't know that the new executive director was going to be on the call.

So I'm at a tire place. With my hoodie on, my headphones, like looking very much like hacker crazy, and not realizing who was on the call with me, I just kind of stated everything. I was like, this is happening, this is happening, this is happening. And this is how I see a pathway forward. And without realizing that the executive director was on, he's like, yes, like, yes.

And so I think it's being able to justify the actions that you want to take having an actual plan and a strategy. And it wasn't just like, I want to listen to a bunch of feedback. It was, I want to listen to a bunch of feedback from that feedback. I want to create our plan of action from that plan of action.

I want to present it to everybody to get buy in from that buy in. Then we're going to go. And it's having the overall strategy when you, and then presenting it. Right. Because if he had said, no, like, I don't want you to do that. I would have said, okay, what is your suggestion? And then I'm going to go back to what mine is, or maybe I will alter it and include his pieces.

I tend to steamroll. I don't, I can't, I can't help it. But again, it's all about like opening the communication and, but also having the, the. Like the information to back up what you're saying. Like I was a part of the, that organization for a year. I was one of the people complaining, which is why I stood up and be like, ran for outreach chair, right?

Cause I was like, I cannot stand to be in another one of these meetings. Again, this has been terrible. Like we, like, we want to do something and nothing is happening. I guess I will stand up and I will, I will do more. And I think that's also what makes community great. Right. Is that everybody comes to the table and it's up to you to decide, like, How much you want to put in, like, do you want to be a chair?

Do you want to volunteer? Do you want to be on a program committee? Do you want to initiate, like, do you want to go to GitHub and say, here's an issue, like, do you want to give feedback? Right? Like. Community is only as strong as you make it. And it's up to the individuals to kind of take that leap. Like you can use a project all you want and never say anything.

And you can be, be upset that things aren't working the way you want it to, or you can have ideas, but if you don't share them, if you don't bring yourself to the table, then they're not going to be heard. So I think again, for me, it always comes down to like open communication, feedback like listening to others and then Using that to strategize on how you move forward.

Jonathan: Yeah. Excellent. All right. We have been going for just over 50 minutes. I'm curious. Is there something that we haven't talked about that it's just a burning that you guys really want to talk about? And this is sort of a different question. Difficult question, because you'd have to go through all the things I wanted to talk about and which of those have we checked off, but is there anything that we have not gotten to that you really wanted to?

Steve: mean, there's a couple of items in the road map. Maybe I could assign a few JIRA tickets to you if you wouldn't mind just throwing some code down for me.

Jonathan: I can write code, but man, databases is just outside of my expertise. That is actually a good point, though. What are you guys looking forward to for the future?

What is coming down the pike that you're excited about?

Steve: Oh, jeez, all kinds of stuff. I mean, I think the ones that I'm most excited, obviously Everest is huge for us. You know, bringing. database as a service, giving you control, giving you sort of the best of both worlds, the AWS experience, but also control of the data and the systems underneath it.

And then one of the other big exciting ones is our Postgres team is working on transparent data encryption, which is something that we're really excited about. I know there's been multiple There are multiple solutions out there, but we, we think we're looking at a slightly different approach to the same problem.

And hopefully you know, the, the early returns are that the community does see value in it. So I think that's always great for us to keep, to keep on going when we hear things like that.

Jonathan: Yeah. Transparent data encryption being like the, the database itself automatically does encryption and then it's encrypted at rest.

Steve: It's actually stored encrypted in the database. So it doesn't matter. I mean, It's, it's actually double encryption if you encrypt at rest, meaning the disk itself is encrypted, but this is encrypted inside the database. So even at runtime you don't have to worry about your data being exposed unless you are the application that was intending to see the data.

Very cool. So it's being encrypted in real time as it's being read and written to the database. Excellent.

Jonathan: Blory, did you have anything you wanted to plug for the future? Anything you're super excited about that's coming down the pike for Percona?

Lori: Well, as a community person, I'm going to jump on what Steve just said.

Right. And like, we're going to a bunch of conferences, conference season is back. So I am very happy to be able to be to be at events representing Percona, so we'll be at open source summit Europe. We're staying for Valky day. We'll be at all things open. We'll be at PG conf in New York.

We'll also be at. OSFF, which is Phenosis Conference in New York. We're going to be at KubeCon. Like, so to be able to take everything that we just talked about today and then put it into practice, like, in person is what I'm super jazzed about. So, like, to see what the reaction from the PG Postgres community is when we sort of have our presentations about TDE, to get their feedback in person, to launch Everest at at Open Source Summit Europe, right?

To see, like What questions people are asking to see how we can then react is that as a community? Like, what do we have? Do we need documentation? Do we need this? Do we need that? Like, this is like, this is where I get super jazzed because it helps set the stage for what we do in the spring. Right. So with community, everything you do is like a couple months in advance.

Right. So being in person at like. Six conferences in like three months or something crazy. And then be able to take all that knowledge back to the engineering team and then package ourselves up for the spring round, right? KubeCon in the spring, open source summit in the spring, all of those other things is like, it's the fun time.

So I'm well rested from the summer and now I'm ready to get back on the road.

Jonathan: Excellent. Busy, busy, busy. All right. So I have to ask you guys each before I let you go. These are our required two questions. We'll start with Lori. What's your favorite text editor in the scripting language?

Lori: I'm sorry. Did you see the face I

Jonathan: just made? What? I'm

Lori: going to pass, like.

Jonathan: So is it Microsoft Word and English? We get that sometimes. I'm not going to lie.

Lori: Then I will, then I will, I will say that. Like, you just, I was not expecting this question. Catherine did not prep me for like the standard questions.

I've got nothing. I got a little egg on my face. Just say Vim. Vim. Vim. 100%. No,

Jonathan: no, that's fine. And that's, that's something most of the people have an answer for. Everyone's every once in a while, somebody doesn't, I think the really fun one is when someone's like been in management for a while too. And they're like, let me think back to what I used to use back when I actually wrote code.

So that's, that's fine. There, there is absolutely nothing wrong with that. But Steve, same two questions. I imagine you'll, you'll have a little bit different answer.

Steve: Well, I've definitely Vim is my favorite editor. This copilot thing, I'm, I'm digging. So I use VS code a lot more for, you know, like an IDE, but I still, my old go to quick and dirty, just Vim file.

And I could be a thousand times faster navigating the file there than in anything else.

Jonathan: Yeah, absolutely. And scripting language.

Steve: Bash is my go to. You know, Python, if it's going to get longer and more involved, but like bash, if I could type it on the command line, it's just,

Jonathan: it's just easier for me.

That's great. I'm waiting for someone to tell me one of the esoteric extensions on something like bash or Python. So somebody that uses Amber. Instead of bash as their main scripting language or I was I was recently introduced to Python which is Python with braces and some fun stuff out there like that

Steve: I work with a guy who's all about z shell everything was a shell.

That's great. Like I'm lost in that

Jonathan: Fun. All right. Thank you both so much for being here. It was a blast Really really appreciate it and that's Laurie LaRusso and Steve Hoffman. Thank you guys both both for being here

Lori: Thanks for having us.

Jonathan: Yeah, absolutely. Good stuff. All right. Well, that is the show today.

And as I'm sure you noticed databases are not my area of expertise, but we we we sorta, we sorta hung in there. We faked it for part of the way, but had a really good conversation about community as well as some technical things and a bit of database stuff and was It was really, really a blast.

We've got something really cool coming up next week. We are talking with Andreas about Ladybird, which is a browser, but it's not one of the big browsers. It's a new browser, sort of written from the ground up, which is really interesting. So make sure and tune back in next week for that. Do you want to follow me?

There is, of course, Hackaday. I've got my security column goes live every Friday. Friday, and that's also the home of Floss Weekly, as you probably know. There's also the Untitled Linux Show over at twit. tv still, and make sure and tune in for that. We sure appreciate everybody that's here, both to get us live and on the download, and we will see you next week on Floss Weekly.

Episode 798 transcript

28 augusti 2024 | min

FLOSS-798

Jonathan: Hey folks, this week Rob joins me and we talk with Carl Richel, the head honcho over at System76. We chat about their new desktop, Cosmic, we talk Rust, we talk System76 hardware, and even a little bit about Intel and AMD. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 798, recorded Tuesday, August 27th.

It is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we have a treat today. I know I'm excited about our guests most of the time, but I'm extra excited today because we're talking with Carl Richel about System76, about Cosmic, about.

All kinds of stuff. I'm sure we're gonna talk Russ. We're gonna talk Wayland. We're going to talk the new cosmic desktop, which the alpha is out. It's great. Before we get him, let's talk with our co host. And today we have, we have Rob, we have the, the cereal desktop, the, the, the distro hopper, which is why Rob is on the show today, because I, I was, I was pretty confident that I could talk him into hopping, at least in a virtual machine over to the new cosmic alpha, Rob, welcome.

Hello, hello. Hey! We've been watching Cosmic for a long time now, haven't we?

Rob: We have. Been all curiously interested in all the places they've been going.

Jonathan: Yeah, I think so they they made it's oh, it's been like a year ago now carl will be able to tell us I'm sure exactly how long it's been but it's been like a year ago now They made their initial announcement and the things that that really caught my attention right away are one They're getting away from gnome, which i'm I I like the pop os desktop, but i'm not a huge gnome fan two they were going to rust base, which I think rust is just cool and three they're like And because we're doing Wayland and we're building it from the ground up, we're going to do HDR.

And I'm like, Oh, HDR is cool too. So like from the initial announcement, they had me hooked. And I've just been kind of eagerly awaiting the whole time. Like, Oh, when's it going to be ready? When's it going to be ready? Oh, now you, you gave it a

Rob: test drive, didn't you? I have I've given a test drive on a VM. I, I was really hoping to try it out on bare metal before the show, but you know, time flies and it gets away from me.

Jonathan: Yes, day job and family and all kinds of other stuff going on. And then there's that other show that we keep you busy with too. So we have a lot, but it looks really good. It looks really good. And even in a VM, it does. All right, let's, let's let's go ahead and bring, bring Carl on. Obviously the guy that knows what's really going on.

Carl, welcome to the show.

Carl: Thanks for having

Jonathan: me. Yeah, thanks for being here. This is, oh, this is going to be fun. I, we, we had Carl on back when we were still on the Twit Network. And Leo was so, so excited to be able to talk that he's like, Oh, I want to co host too. And so, you know, it was Doc and me and Leo all co hosting.

Well, I mean, Doc was the main host and then Leo was kind of the main host because it's Leo. But it's, it's great to have you back. And it's great. It's great to chat about, you know. First off, what's happened since then, but I think mainly what everybody's curious about what I'm curious about. So let's talk cosmic and let's

Carl: Yeah, I love to talk cosmic.

So Thanks for having me on to chat about it. I was thinking about the last show We were on together too with that with Leo and I remember distinctly he has he has a one of our darted pro laptops And he, he apologized for running Linux Mint on it, I think. Oh, that's But we're just happy, we're just happy when Linux works that way.

We, we're here to make Linux work on our products, whether it's Pop or Mint or, or Bootery or, or Arch.

Jonathan: Yeah. So I'm actually, I do the show on an HP Dev1, which is not quite your hardware, but it's also pretty much your hardware from what I understand, but running Pop! OS on it still. And I've had a Yeah. Yeah.

99 percent great experience with it. There's something weird with the power charge circuitry, but we work around that we make it work anyways

Carl: Yeah, that was a great product and a great collaboration of the folks at HP. You were really great to work with.

Jonathan: Yeah

Carl: They're very professional. I think Working with them on dev one Also helped us refine POP 2204, because that was coming out at that time.

And so we had just more people with different perspectives looking at what we were building and, and really helped, helped sharpen that, that operating system release.

Jonathan: Yeah. Okay. I didn't expect to get to this question so early, but it, it really leads into it. You did the collaboration with HP and there's, there's another company out there that I think might be really interesting to do a collaboration with.

And that's framework. And I'm, I'm just curious, like, do you, do you have opinions on framework? You consider them a competitor and have you ever thought about that? Like, I wonder if we could do something similar with the framework laptop.

Carl: Oh, I've been, having been in the hardware business for now.

Almost 19 years. I know I know how challenging it is and I, I think it's really admirable how well they've done getting into the market and doing a lot of creative work around right to repair. We have, we have that. But very much in common since day six and, and, and framework both care a lot about their about a user's right to repair, but their ownership over their products.

So so I think I congratulate them on, on doing something that I know very well, how hard it is to do, and they've done a fantastic job. So, so good for them.

Jonathan: Yeah. I, I, I kind of, I can envision a future where it's like either, you know, system 76 makes. One of the embedded main boards that people can put into a framework laptop or framework sells a system 76 version or maybe even makes cosmic, you know, maybe as simple as making cosmic a ship option that you can buy one of our laptops, you can put system 76 cosmic on it.

And everybody wins. Like it just, it, there, there seems like there's gotta be, there's gotta be some way that the cross pollination and that can happen.

Carl: I think that would be possible. There's, I'm very interested in modular mobile devices. So perhaps at a level, A little bit further than where framework currently goes, but very similar in that we have a main board that can be used across a number of different devices.

And you essentially connect daughter boards to that main board for your port layouts, and that might be used for a desktop or a laptop or any other kind of device. I think it takes a lot of creative engineering to kind of compete at a very high level and do that with really good thermals in a mobile platform.

But I mean, doing hard things is the kind of stuff we're interested in.

Jonathan: Yeah, absolutely. Absolutely. Okay. So let's, let's shift gears a little bit. Let's talk Cosmic. What was, what was the point where you guys realized, okay, we need to, you have, you have Pop! OS, which is a very, very heavily, heavily customized.

Ubuntu, and it's a very heavily customized GNOME. And at some point you guys made the decision that GNOME is just not cutting it anymore. I'm curious what, what that looked like. What was, what was the point where this, this harebrained scheme as it were, popped up where it's like, we need to, we need to start from scratch on this.

Yeah, why? Yes.

Carl: Yeah. Why? Okay. Well, it's a long, it's a long story, so I guess I have to decide where to even start, but at the best place is probably with why we decided to make the changes we wanted to make to gnome in the first place. We were. All of our engineers that were working on the Pop OS were using i3 at the time.

When they were using i3, I got became interested in it. And so I started using Sway. Then we were getting calls in our tech support department. I said, Hey, does I3 and Sway run on, on Pop OS or how do I set up I3 on Pop OS? And this was not just a one or two requests. This was extremely common. It was, it was coming in regularly.

Then I went to a, A firmware conference at Google for Open Firmware, which we do a lot of work in that area. And so we were giving a talk at the conference and everyone at the conference was using a tiling window. It became clear to us at that point that what we were providing at PopOS out of the box was not what our customers needed.

So in response to that, we decided to build the extensions we built into Gnome, which, which added auto tiling added a different UX in general with a launcher and application library and a doc. And we were adapting for some things we're writing new, new software for others. We were adapting extensions like dash to doc to provide the user experience that we wanted.

Then we found our users were craving and our customers were craving. So it all started with just the desire to. Respond to what our customers want and build something that they wanted. The next. Step was to try to make that more official. And we wanted to prove what we were doing and know, and then see if we can make more official.

We, we had a, a UX architects or a UX architect. Maria was on the design team. I had gone to a number of quad X and. And over time, it just became clear that our vision for the desktop, and we pitched the idea that you could take, we could kind of separate the idea that the UX is what we And in reality, it's a whole bunch of different pieces that you put together to create a holistic experience.

And we thought that Noam could perhaps take the, take the perspective that SimCity 6 could build our user experience out of it. Ubuntu could build theirs, Noam could build theirs. And the idea was considered interesting, but just not where Noam wanted to go.

Speaker 4: With

Carl: that being the case it became, Clear at that point that we would not be able to continue using gnome and continuously adapting it, you know, working against what was going on with upstream to build the experience our customers needed.

And so we decided to build a cosmic.

Jonathan: Do you, are you kind of of the opinion that maybe Gnome has shot themselves in the foot and some of the other downstream from Gnome are going to have to face similar decisions?

Carl: I don't know. Gnome has their vision for the desktop. I,

I don't share it. It's kind of that simple. What I, I kind of, I love this, what I love about Linux and open source is that it's, it is a collection of all of these different communities, interests and tools and different things that you could pull together to build something with. That's what's special to me about it.

And we don't have that at the desktop platform. That's what, that's the scratch we're trying to itch with Cosmic. So we have this thing where we have customers that clearly need something different because they're just telling us all the time we want to be able to build it, but we can't be the only ones.

I'm sure other distros have different target audiences and they should have a different UX because there are audiences different. We don't all have to be the same. In fact, the vibrance that comes from distros and their response to customers, I think is valuable. So we wanted to build a platform. In cosmic where you could take what we've made and it's explicitly designed to create your own user experience.

So you could build the Ubuntu UX out of it. Very, really, very simply, you can build Manjaro could have their own UX. You could build a UX more like sentiment. Out of cosmic. Those are the things that, you know, those are some of the projects that are, have kind of a unique approach to how they they adapt and customize custom environments, but cosmic allows you to do it at a very deeper level and at a professional level that's very high quality.

So so while it's going to be the flagship experience of Pop! OS, and I'm sure the first release is going to be very similar to what we Push out what we think is right for our users. What's most exciting to me is how people will. Brand cosmic, how they might experiment with different applets and user experiences and ways of launching applications or, or managing applications or moving windows, you know, all those things are up for grabs for, for experimenting with in the community.

Jonathan: And there's already, there's already been some, I know Rob's chomped at the bit, I'll get one more and then I'll hand it to Rob. There's already been other distros I know of. That have expressed interest in, but we could package cosmic. I'm pretty sure Fedora is planning on packaging and shipping a cosmic spin.

And that's gotta be exciting for you guys too.

Carl: It's exciting and encouraging and much, much earlier than I thought. I don't think I, we saw it coming. Not too long ago. And so we're learning how to be a great upstream for distributions. We consider distros our customer. We want to make a great, we want to make it easy to package easy to maintain.

We want to listen to their feedback about what we can do to you know, build a good experience for them. So I think they kind of heard that attitude coming from us that we're really care about you know, empowering distros and and yeah, it's in Fedora. OpenSUSE Serpent is packaged as well.

They're a new up and coming distro. I think that's pretty cool. I hope I'm pronouncing that right. Arch, Arch spin that has a focus on performance, their packaging as well. When they're, I think in the performance reboot, I think uses some different compiled flags or something along those lines, but yeah it's, it is encouraging and exciting and we hope we we hope we do well by everybody and all the distros that are putting their faith in us so early.

Rob: is exciting. So I mean, GNOME is a nice interface. If you want to do the GNOME way, but was some of your reason, was it just hard, more difficult or becoming difficult to keep up with their changes and the way their extensions work and having to keep updating yours to work with the newest version?

Carl: No, not really. It's that taking that route means that what we are doing every six months is adapting our extensions to work with the new version of Gnome, and maybe adding a little bit of icing or advancement or additional value. So that work isn't very fun. And we're just working Every six months working to get back to par, which is where we were and maybe adding a little more with Cosmic.

What we've done is built the entire foundation where it's it can be adapted to, as I said, other user experiences. But once we have that foundation in place, all of our engineering and all of our effort is adding features and capabilities, new experiences on top of Cosmic. So it means we'll be able to innovate much more quickly.

Just having this flexible base to build upon.

Rob: With that flexible base, The alpha you have today isn't doesn't have that flexibility yet or does it

Carl: it does yes it's the the alpha as It has essentially all the UI elements that you would need to use the operating system But all of those are also are stable Are interchangeable already.

So we have a panel on top and a dock on bottom, but you can remove the dock, move the panel to the bottom. You can make it more like a windows experience. You can move the dock to the left and then have a panel on top and you can have buttons in the top left, like unity had with Oh gosh, they called it lenses for a while in touch.

I'm starting to mix up some of the older Unix or unity user experience features. But in essence Together with panels and applets, applets are applications that run independently inside of the panel and on top of it. So the combination of panels and applets means you can build in a UX.

You can build a custom launcher that is different than our launcher and maybe is more like perhaps an elementary launcher or something along those lines and add a button to the panel and now you have an experience that's closer to that of elementary. So it does have that flexibility in it. And.

There's lots of heavy customization that can be done through config files. What we expose in settings is what we think makes most sense for users, but we would expect people making different user experiences to change the settings as well.

Rob: So, yeah, I guess I should have put a little more time into my experience that I had with it. But I have I have had some time and I explored around with it only on a virtual machine. I, I should have taken the time to put it on a bare metal. But, you know, before the show we were talking about on a virtual machine because of, I believe you said the graphics acceleration.

It's, it's not quite as Stable there because it did crash on me several times. So could you speak to that maybe? Yeah.

Jonathan: Is that, is that expected? I think that's expected, right? You said during the press call.

Carl: It's, it is expected and it's all because we don't have software rendering in our capacitor yet.

So Cosmic Comp doesn't have software rendering, and so you need hardware acceleration in a virtual machine, which is actually kind of fiddly to set up, and more fiddly than, than I'd like, so so we need to get software rendering done. It doesn't affect anyone on hardware. And of course, all of us developing Cosmic, we're all on hardware.

And so, so one of our focuses will be making sure that you have a good Cosmic experience for the second alpha inside of a VM. Because we understand, you know, the commitment just to install it on, on your hardware is high. It's not, it's not production software yet. So, We want to improve that experience for people in the second alpha.

Jonathan: Okay, I've got to jump in and ask. We talked about moving from GNOME. And this is like partially a troll question, but also partially like a real question. Did you consider KDE as your base? Or any of the other desktops? There, there had to be at least a thought of, I wonder if there's something else that's out there that could do this before you just jumped into the deep end of the pool and made your own.

Carl: There's very little that we didn't consider. We had a big pros and cons list for, for different directions. KDE is, I, they're kind of my spirit animal. I like their approach. I really, I really like like KDE and I like the community there. I think I think they're, they're great. Their response to what users need and want is admirable.

They're, but they're, they're technology stack is something I think there's two things with where, where we decided not to go with, you know, trying to adapt three that where we didn't didn't try to go with adapting K to E to our experience. One is we would probably not make K to E users happy. Not because of what we're doing, but because we might trim down things in a way that they'll say this is supposed to be KDE, but it doesn't have all the KDE stuff.

So, so I thought we might be a lose lose situation there where you know, people feel like it should be KDE, but we're building something different. It's not like KDE. And and I don't know how responsive you know, KDE folks would be you know, to, you know, to like more of a kind of middle ground between all of the options that they have and know.

But in any effect, I thought we couldn't make everyone happy. And we might get in the same kind of situation where upstream is not very happy with us. And, and then customers, I think it's something else or seconds. We're not, um, the technology stack C plus plus isn't a language that we work in.

It's not that. Our team couldn't learn C to go that route, but I just can't imagine being a brush shop and then going that way. Yeah, that's fair. Third Katie is as our vision from the start was that we wanted to build a platform for people to build experiences with. We wanted to be that, that user experience, that UI level that doesn't really exist today.

People are using Android to. You know, make car infotainment systems or, or exercise bikes or, you know, specialized devices. Well, this is a, this is a Linux and open source UI project that enables companies and, and distros to do that on their own. I can't, you could I mean, this would be, this would be something that'd be very easy for, for valve to build a steam deck experience with, for instance.

Oh, so, and it's intended for that. So that's also just a distinction or a difference with what our core intent was in building Cosmic.

Jonathan: Yeah, excellent.

Rob: So being so involved in open source for so many years, Any thoughts on open source hardware, like getting Pop! OS working or in Cosmic working with a RISC

Speaker 4: V?

Carl: I think RISC V is very exciting. It's it always turns up in our channels when something new is happening in the RISC V hardware world. We're being the size that we are. We can really do like one really big project at a time. And so cosmic is that one really big project that we, that we're wanting to knock out and then we'll see what's next.

Jonathan: Yeah. Interesting. With it, with it being built on. Well, with Rust, which compiles basically everywhere now, and on top of existing technologies like like Wayland and all those things. I, I kind of suspect that when you get to the point that, okay, we're ready to do our main release, like, the bulk of the development is done with Cosmic.

You know, for that initial release. I kind of suspect that it's going to be a fairly easy lift to go to another platform. If they have like reasonably well working graphics drivers. I, in fact, you, you might find that somebody from the community beats you to it and makes it work there. We've, we've seen stuff like that over the years, time after time,

Carl: I would be elated to see that.

I think it would be awesome. And that's one of the things that's so encouraging about what's happening with the cosmic community to we have hundreds of people contributing patches, building features. It's the energy around cosmic is very, very exciting. So so, yeah, I think the next extension of that is Cosmic's running on this or, Hey, I built a, I put these, you know, five applets that when you combine them together you have this really unique way of using a computer or using this device that doesn't have a keyboard or, you know, Yeah, that's the stuff I'm really excited for.

And I think if we've done well, then we're going to see lots of that over the next, next few years.

Jonathan: Yeah. Okay. We do have a question from the chat room. I've got a bunch of stuff that I want to get to too, but mashed potato asks also, he says, why Ubuntu and not Debian? Well,

Carl: Debian Ubuntu does a fantastic job of pulling together a modern or, Very recent, but stable base, I think.

So combined with our long experience with the Ubuntu I think we can build a pop OS as a better product with Ubuntu snaps are making that more challenging and I'm not I'm not like anti snap either. I, the only thing I don't like is about snap is I kind of like the technology. I just. I don't like a proprietary store.

That's, I would prefer it to be like repositories that we all are used to in Linux and Flathub, which is, you know, an open store. That's my only beef with it, but I would love to have Snap working in you know, Cosmic Cosmic App Store, for instance, and have it as an option for people to turn on or off, you know, depending on their preferences.

But because we don't want to have a proprietary store in Pop! OS by default, we have to do quite a bit of additional packaging work. Like Thunderbird in 2404 moved to Snap. So we're packaging Thunderbird as a Debian for Pop! OS. So

Speaker 4: it does,

Carl: It does add that work, but. There's a lot of other work that they do that is a layer on top of Debian that makes our jobs easier.

Jonathan: Yeah, interesting. Okay, so Let's talk, let's talk Wayland for a minute. Something that's not been entirely clear to me. I think I know the answer to this. With Cosmic, is there an x11 option at all or are you guys Wayland only?

Carl: Wayland only. But there is X11 X Wayland support in Cosmic Comp, so it doesn't really feel, at this point, the Wayland protocols, and this is just a community wide thing, the Wayland protocols are getting really, really good, really, really close to being, like, I think their NVIDIA is finally playing nice, I know it's been a long road for them, but that was our biggest, our biggest question mark wasn't, you know, For releasing Cosmic was NVIDIA because if if NVIDIA drivers were not working, we would just have to delay until that was until it was working too many of our customers use NVIDIA and care about the, the hardware.

Speaker 4: Yeah.

Carl: So but besides that, essentially what we're, we're at the point where moving to Wayland, I don't think. There'll be some corner cases, but there will be very far few between. Otherwise for most folks it's just going to be a more secure higher quality experience.

Jonathan: Yeah. So it, I think that's one of the things that's actually really compelling about cosmic is that you guys were able to, with the exception of ex Wayland, which I don't think counts in this case you were able to just sort of push aside all of the X 11 cruft and start from a fresh slate.

I think that's really compelling because like, X11 has a lot of cruft around it.

Carl: Yeah, it's, it is interesting that that's something we haven't, X raying it is still a pain. It really is difficult because the, the knowledge to implement the things that are there are just. Buried in some guy's head that wrote it 40 years ago.

Yeah. It couldn't be 40 years. That sounds like way too long. Maybe 30, but that's where it's at. And the documentation is the code is it is definitely challenging. But so we spent a lot of time just working on getting XWayland working so that that experience can, can feel seamless.

Rob: So I assume all of your.

All of your apps, applets, tools are all direct Weyland and not going through the ex Weylandshim.

Carl: Great.

Jonathan: Yeah. That makes sense. That makes sense. Okay. I, I, I sort of watched the Weyland development process and sometimes that's painful. I, and I've got to ask do, do you guys get frustrated with the slow pace that Weyland takes and the, it's, from the outside, it seems like.

Every decision gets bike shedded for like a year before anything gets added. Is it, has that been a source of frustration or are we kind of past where that's a problem?

Carl: No, I think the way we approach it we have two engineers that do most of that work quite a bit in the Wayland protocols, Ian and Victoria, and their approach is in essence, This is something that we have to do to deliver a product that's going to work for our customers and our users.

So if we have to release you know, pre V1 protocol even if we have to change it later to match what's, you know, the eventual the eventual protocol, then that's something that we'll do. And we don't feel you know, we'll, you know, we don't feel like that's you know That's bad because Wayland didn't move fast enough.

It's just you know, it's just part of the process of working in a community with a lot of different interests.

Jonathan: And I think that's probably the advantage. One, one of the advantages with Wayland is because Wayland itself is pretty much just a specification. And the, the implementers are very, very free to add their own specifications, to add their own interfaces on top of that.

And so you have things like what KDE has done, where the KDE guys got tired of waiting for. HDR to materialize in Weyland upstream. And so they finally just said, okay, fine. You know what? We're gonna build our own and because we're gonna have fun with it We're gonna call it frog and nobody knows why but that's what we're gonna call it And hey, look with a couple of days, you know a couple More than days, a couple of months of hacking on it.

You guys can use HDR on a couple of programs in KDE because we got tired of waiting for Wayland. And I think that's secretly one of the superpowers that Wayland has.

Carl: Yeah. And that's an okay approach. I mean, we, we do have to get the things done for, for users and that's just a given and that's, you know, part of building software and building products.

If the consensus protocols take a little bit longer and we have to, you know, adapt to them. Afterwards, that's all right.

Jonathan: Yeah. Speaking of HDR, this is actually one of the, for me, not for everybody, but for me, it's one of the killer the killer features back when, when y'all first announced is that a cosmic is going to have HDR.

And I know, I know that's not ready with version one or the release one, the. Candidate one, whatever we're calling it. But I know last time I asked you about this You said that you you anticipated it coming with the next pre release what what is the what does the roadmap and timeline look like on that now?

Carl: I hope I didn't say that I could I am so optimistic terrible timelines Optimism and bad timelines just don't go well together. But the thing is Well, when I get into a tirade about really big projects, it's not actually possible to predict how long it would take, but with with HDR it is a focus of ours and we're Victoria is the, the you know, lead developer for Cosmic Comp, and she's going to all the HDR hack fest and that are going on for Linux.

And so this is a, it's a broad community wide effort to get HDR working really well

Speaker 4: in Windows.

Carl: I expect, so I don't think the first release of Cosmic will have HDR, but I believe we'll add it sometime in the months afterwards. I just don't want it to be a blocker. Because I think we've got, we nailed so much here that it would be okay to release and then add HDR

Jonathan: afterwards.

Carl: And HDR is also really power hungry. And so there are a lot of considerations with like how often it should be used, particularly on a, like a laptop.

Jonathan: Yeah.

Carl: So there's things to think through there a lot of times. HDR content might be, um, you know it might be a choice whether you want to see something in HDR content all the time.

Jonathan: That's fair. I, it, it comes to mind that I, I asked you the sort of troll question about KDE earlier. But it actually comes to mind that. approach to doing this would just be to re implement and use the, the KDE HDR protocol, because at that point, you've already got a couple of applications that are wired up to use it.

Whereas as far as I know, there's nothing that's wired up to use the, the kind of in process HDR stuff that upstream Wayland is doing. And so if you wanted to get it out the door, Earlier, you know, with the next release just re implement what the, the KDE frog protocol and, and let in player and a couple of others use it.

Carl: I have, I have a strong suspicion. It's not that easy. Okay. That's fair. I don't, I don't, I, it's just a, just a suspicion.

Jonathan: All right, so let's let's see, where do we want to go next? We asked about HDR, we asked about Wayland Rob, yes?

Rob: Yeah, so, I believe, when is the final version of Cosmic expected? Is that later this year, or is that a little optimistic?

Oh yeah, so now the

Carl: optimists bad timeline guy has to make a guess. I really want to make a you'll have a release this year and I think there's a good chance the problem, the problem might be it ends, it lands in like December, which is a hard time to release a big project. It's hard on your, your employees that would like to see their families and those types of things.

So I think if we can't make it by. early to mid November, then we would probably do first quarter of 25.

Jonathan: That makes sense. What is the, what has the reception been like though for the, the, the pre release that's out? Like, do you, do you have a, do you have an idea of how many people have downloaded it and tried it?

And then what's the feedback been?

Carl: Gosh, I don't have. I don't have a good idea. I know that we're in for downloads for the alpha. I know it's into like tens of thousands or something along those lines, but that's just like the Popeye says. I don't know about any other. We don't have telemetry telling us how many people have installed cosmic.

Speaker 4: Right.

Carl: I've almost gotten through most of the press coverage. Cosmic, that's We were, we've been so busy after the release, we had all hands company events and engineering meetings. And, and so we're just all, I'm still recovering this week from all that. But everything I've seen is I think very promising and it's a good response.

I think one of my takeaways from what I've seen is the people that the The time that it takes to build a foundation and the time that it takes to build things on top of it is very, very different. So what people are seeing today, and I think if this took two years, then it might be like two years before it's really there.

But what we're going to show over just the next couple of months is that things will happen very, very quickly. So that's that's one thing I noticed that there's kind of an impression that it's, you know, a lot is there, but. There's a lot to go. They'll see that the things that there are to go aren't really really big projects.

Jonathan: Yeah. That's the, the, the 80, 20 rule. You get the, yeah, yeah. Yeah. That's, that's a, that's a real thing. It really is. Yeah.

Carl: Yeah. And then bug fixes to We're focused on putting features out and getting all the features out and then sharpening everything up. But I don't worry about bugs very much either because every time we go to nail down bugs with few exceptions, those are kind of the quick things.

That's just the quality control and and, you know, turning a focus to all the little paper cuts.

Jonathan: Yeah. Okay. So I've got to ask, this comes to mind because. Another project that I'm a part of, we're dealing with this this morning. Have you gotten to the point, and I'm sure you must have, you've, you've been around, you've been doing, you know, Papa West long enough, if you've gotten to the point to where with Cosmic, you've got users complaining, you made all of these changes and you didn't ask us first.

Carl: I feel surprisingly, no, um, and I read almost every bug report And I mean, I'm almost through all of them that have come in since the alpha started. And there are, there are feature requests. There's not a whole lot of you change this and I hate you yet.

I, I fully expected. So this happened to us when we moved from, you know, Pop! Without our cosmic UX and no. To our cosmic UX and no, and they said, I want my overview back and I want this and this and these other things. And so we were prepared for it. This transition is going to be different because the UX from POP 2204 to 2404 is going to be very, very similar.

You're going to have the same launcher, app library, doc, panel. It's going to feel the same, but everything under the hood is brand new. The apps are new. And so you know, fortunately, I don't think the bar has Too terribly high to reach the app functionality that we had before. With the exception of files, file browsers are just really complex.

And so there's a lot of a lot of work there. I think we're going to, we're going to make it there too, but point being. The UX is going to change. It's going to be what we're going to hear. I suspect is I can't install my clipboard extension or this caffeine extension, or, you know, these other things that we're replacing with applets.

All of that, I think is, is, is absolutely necessary because you know, a desktop that has independent running applications in the panel instead of monkey patch, JavaScript, it's just. Going to be better. So more extensive extendable. So we'll have some pain with that. But I, I think from what we've seen already, a lot of enthusiasm people already building applets to replace those things.

So, you know, hopefully, hopefully it's a very small number of people that we.

Jonathan: That's, that's actually an interesting question. Is there a simple, like, scripting tool for doing extensions? Is, does that exist in Cosmic at all?

Carl: Yeah, we have a, we have an applet template, and essentially you're writing a small application that is embedded in the panel, and it can have its own settings.

So, That's and it provides essentially the same functionality. You can spawn windows. You can modify how workspaces are, you know, presented. There's a lot of different things you can do with applets. So So that is, that's our replacement for the idea of extensions.

Rob: So those applets would all be, the developers would all be making those in Rust?

Carl: Right. And they are all independent applications as well. So if you open System Monitor, Cosmic and you can just search for cosmic and system monitor. You will see an applet for a user applet, a Bluetooth applet notifications applet, every app tray, everything that you see in the panel is an independent application, meaning that it can't step on the other applications.

So if there's something wrong with it, it doesn't crash your desktop. Or if there's you could, and you could add or remove them at will. That also means that we can sandbox them. So a community that's building applets doesn't have to be explicit, you know trusted with access to your home directory.

We can we can sandbox them you know, away from that and have the same kind of permissions and portals access that you have in applications in applets that you have in extensions.

Jonathan: So, so when someone installs one of these applets, do they literally get a security pop up? You know, these are the permissions that this applet wants to have.

Like, have you, have you gotten into that, that fine grain sort of of handling?

Carl: We haven't gotten there yet because everything's first party so far, but there are community applets being built and that's something that we're going to have to think about and work on as well.

Jonathan: Yeah. I, I, I hate to say it, but you know, it's coming.

Someone's going to write an applet and it's going to be down to borrow something from the Android world. This applet turns your flashlight on. And Oh, by the way, secretly, it also reads your clipboard and sends it to us. Like that's the price of success. It's

Carl: true. Yeah. And so it's, we'll have to work, we'll want to work on it early.

We are going to have, we'll have a repository for applets and Every every poll, we're going to do engineering and and design review for applets that go into cosmic, and that will be available in the cosmic store. So to I mean, there's going to be more effort for us up front, but because our toolkit is very young, because the community is very new.

And the project is new, it's just going to mean there'll be a little bit longer review process, but we are going to actually do full code reviews. And in this case, even design reviews and provide design feedback so that so that applets can fit into the cosmic UX. And that's largely because the our templates and our widget library is just.

It's just young. So it's it's, it might be as a developer today, hard to turn to know exactly like how, and our documentation is, is young too. So how, you know, how should I, where should these buttons go to fit the UX and some things like that, that we just you need to, Tighten up for the community.

Jonathan: Yeah. Okay. So to get, to get an application into the store, I assume it's got to be open source and use an OSI approved license.

Carl: Not necessarily. I don't think that would be a requirement, but we haven't seen anything that isn't.

Jonathan: I just, in thinking through this, it's like, well, how do you, how do you do code review if you can't get to the source?

Carl: Oh, that's a great point. No, you're absolutely right.

Jonathan: And then I guess the question that, well, so, I mean, it's interesting. This is all very new stuff to you guys. You're, you're, you're sort of thinking through all this as you go along. I think the question that kind of naturally follows after that is, is there even a mechanism for someone to install an applet that does not come through the store?

Like you, you would probably want to discourage that, but at the same time, you might want to still allow it. It is, is that even an option? Is that a thing?

Carl: Oh, yeah, absolutely. You can download code from a repo. You can do that today. That's where most of these applets reside. It's in a repo spread out on GitHub.

You can download those, install them and add them to the panel.

Rob: Yeah. Okay. So, so with each applet being its own program. Would other developers be able to make like a C program or anything and actually run as an applet?

Carl: That's, so it is possible with a different toolkit. So our ICE toolkit and I hope I'm, I'm hoping I'm getting this right, but with our, with our toolkits, I believe it requires that it has to be written in Rust.

If you're using Slint, which is another toolkit that has as libcosmic widgets or, or matching widget widgets, you'd be able to use C or C or you know, the other languages that Slint supports.

Jonathan: Yeah. Very cool. Is, is. Is Cosmic built on top of one of the existing compositors? Like, is it, does it use WL roots or, or one of those?

And I can't remember. I know I've looked this up before and I can't remember off the top of my head.

Carl: No, it's it's a Bruss compositor. That's it's based on a community project called Smithing. Which and so we, we hired Victoria. She was the lead of the Smithy project, that was her project.

And so Victoria was developing Cosmic Comp and Smithy is a big part of, we have a Smithy toolkits and a lot of other things we've built around it over the last couple of years. So that's all all brushed and all all new.

Jonathan: That's okay. That's, that's really cool. I did not realize that you'd gone out and you, you hired the developer that was building the thing that was really similar to what you wanted to build.

That's, that's actually really neat. One of the things that we talk about here is developer open source developers have rent to pay and they need to eat too. I, oh, that, that is really cool that you guys, you guys found a developer and hired him, brought him on board. And so you're, you're kind of bringing the existing project on board.

Making it work with what you need to do. I, I think that is an absolute win for everybody.

Carl: Yeah. Yeah. It's, we wanted to build a the compositor. We thought that was a, the compositor has a huge impact on the field of the operating system. It is kind of central to, to the OS experience. And so we knew that was going to be a key part.

And Victoria is absolutely incredible. An amazing person, amazing engineer. So we're very, very excited to have her on the team.

Jonathan: Do you see the possibility of other, like, desktop environments that, so, someone else out there that doesn't want to use Cosmic but wants to use Smithy? It's that, like, I assume that's possible.

Is there anybody doing it? Do you anticipate that happening?

Carl: I think what we're seeing happen is that people that want to work with compositors but don't want to work in, C or C they want to work in Rust they're going to SmithA and using SmithA as their target for things like accessibility, for instance, and other, you know, unique areas where we want to, you know, improve the Linux stack.

Jonathan: How long have you guys been a Rust shop? System76?

Carl: Probably six years, I think. A lot has changed in that time, hasn't it? It certainly has, but when we started Cosmic, there wasn't, there's still the, the breast windowing environment and ecosystem is very, very nascent. And when we started Cosmic, I know more than two years ago, it was very, very young.

Yeah, there was, you didn't build applications in Rust and today, well, you could. And and I might've seen the bindings, the bindings for GTK are actually quite good. And we were used to using those as well. So you can build apps that way, but not the pure stack that we wanted to build. Where the toolkits, the widgets, everything about the application was written in Rust, including like other things, like there was no text rendering.

So. We had to write text rendering, but the really cool part about all of this is it's a it's a growing community of really passionate and incredible engineers and we all, we're getting, we get to be a part of providing the tools that they're using to build things with now. The Cosmic Text is becoming the standard.

Text render for for Rust Rust applications. You know, that's, that's the awesome thing to be able to contribute to the world.

Jonathan: Yeah, it's, it's, it's really neat. I know something that we've seen like as part of Rust making it into the Linux kernel, if for one thing, it's, it's kind of a, a. A shift for the kernel guys, because now they've been reading and writing in C so long, and they have to add this other language they sort of had to be familiar with.

But one of the other things that's interesting to watch for that is, as Rust is being added to the kernel, Rust itself is changing as a result. They have had to make You know, a lot of changes. They've fixed things. They've made things better in Rust. Have, have y'all seen sort of something similar where as you're building a compositor out of Rust and as you're building these different pieces, that some of the things that you had to work with have now made it back into the Rust language?

Carl: Yeah, I don't know details. So, you know, about that, that's, but I, but I know that everything evolves together if it's going to move forward. And that's the, and the, the base language and language itself is, is no different than any other component. You know, evolves for its use if they're, if you're doing things well.

Jonathan: Yeah, yeah, it makes sense. Are, are we at the point to where we, we are up and coming programmers? We probably need to just start teaching them Rust definitely in addition to C, but you know, is, is there a, is there a point whether it's now or in the future that we teach them Rust instead of C?

Carl: I would think so.

I mean, I went to a memory safety. By default I know I hear the borrow checker is a pain in the ass, but but I think once once you get it, it just, it's a, it makes a lot of sense. I, I don't like language wars. So I don't know how far I

Speaker 4: would

Carl: go into it, but I know it's the, I think building a the bout of memory, safe code and coverage that we're going to have, you know, obviously we have to use, you know, you know, non safe for us for lower level integration, you know, C libraries and other things like that, but Cosmic has so much coverage you know, safe for us code that I think it's.

By the time we're done, I think it's going to be a far more secure operating system and have more coverage than anything else that's out there.

Jonathan: Yeah, it's it's definitely been interesting to watch. And you know, there's there's been things that have happened like Rust developers Every once in a while, I get reminded that like, there are, there are bugs out there other than memory bugs.

We're just, we're so used to those being the norm though, with languages like C that we sometimes forget that, oh yeah, logic bugs exist too.

Carl: Yeah, I think that, I think the I mean, why, Well, we so often hammer memory safety homes because of so many critical vulnerabilities are safety vulnerabilities and you know, things that that we can solve at the language level.

Jonathan: Yeah, absolutely. Okay. We've, we have talked about cosmic. We've talked about Russ. We've talked about Wayland. I want to ask more about system 76 itself. And. I guess first off, is there anything, is there anything upcoming with system 76 as a company that you, you want to plug or let people know about?

Like, is there, is there some secret new hardware that you want to announce on the show? What's what's coming down the

Carl: pipe that you're

Jonathan: excited about?

Carl: There's going to be a new workstation. From system 76 called Thaleo Astra.

Speaker 4: And

Carl: Thaleo Astra will be unlike any other desktop that we have in the, in the line.

And that's all I have to say so far. That's fair. That's fair. It will, it is. Extremely performant and very unique desktop. And we consider it a workstation because of its level of performance and its uniqueness, but I think there will be a lot of people excited about about this product.

Jonathan: Yeah.

Now, I, I honestly cannot remember. I think your. Intel shop. Do you offer AMD on any of your hardware? I just, I can't remember off the top of my head.

Carl: We do. AMD is killing it on desktop. They really are. Intel's fourth generation is quite good. 14th generation. So we have Ryzen 9, 000 products. We have Intel well, this is desktop Intel 14th Gen 15th Gen, I think it's coming in October and Threadripper products on on some workstations.

We also have a a value desktop that is 12th Gen Intel. It's because a lot of people schools and and other, you know, environments don't need You know, the latest and greatest. They need, you know, value. So that's what that price for on the mobile side. We have rising laptops into Intel laptops.

And and NVIDIA graphics on the higher end laptops. So, so really, it's pretty broad spectrum. I think that's one of the interesting things about Pop! West 2 is when we were, when we were building pop, we thought, okay, we're just going to make sure it works really well on our hardware. It turns out that our hardware, we cover so many different it's a pretty good representation.

Yeah. Yeah. And it just ended up that, well, Pop! West works great on hardware because our lab is full and it needs to work well. So so yeah, we have we have a lot of, depending on what the, what the user's needs are, customer's needs are, Intel or AMD or AMD graphics or Nvidia graphics, kind of the full spectrum.

Jonathan: Yeah.

Carl: Which I think is a pretty, it's an amazing thing.

Jonathan: That is really cool.

Carl: That Linux Ships with on everything that's new today. Yeah, that's that's amazing.

Jonathan: Yeah, okay You you may not want to answer this one and that's fair I'm curious though. Have you guys gotten bit by the the Intel 13th and 14th generation problem where where the chips try to eat themselves?

Carl: So yeah, we're it's a familiar with it But at its base, I believe that this Well, that's our, our take is that this was a weatherboard manufacturers, redlining their

Speaker 4: firmware.

Carl: And it was just it wasn't within specifications. I don't know why it was so different than previous generations with Intel specifications and how they were implemented.

But what we found, we spent, we, we did have some customers find that, okay. I've got instability when I'm compiling. And that's what our customers are obviously there, you know, the common use. Heads up. Yeah, so we did so we identified the problem. We worked through it with Intel. We worked through it with our upstream board manufacturers and, and within a few weeks we had a stable firmware shipped out to customers.

So I don't know. I don't know if it was as big as it's, like, we've got a As it's, you know, came out to be I know I've heard oxidization things and other you know, stuff. I don't know. What is, what is your take?

Jonathan: Well, let me, let me put it this way. I have a customer right now that bought two new desktops from another vendor, a very, very large vendor that I, I guess I won't name a very large vendor though.

And they have had problems with those desktops being unstable the entire time that they've had them. And they are now running on the most, on the latest board firmware, and they're still having problems with them being unstable. And so we are probably today going to, we finally figured out that this is.

almost certainly what it is because they're 13th gen Intel. And so we are today going to start the fight to get that vendor to RMA the chips. And I don't know how that's going to go, but I'm reasonably certain that what has happened is that those chips have, have eaten themselves to some extent. I

Carl: have the perfect solution for you.

It's nice being a you know, small, medium sized, hardware manufacturer because When our customers contacted us, it was the guy that's 15 feet away from the QA guy that's doing the testing, reproducing problems, was getting calls and tech support about a problem and walked over and talked to him and we said, yes, this is a problem.

And we're able to work through it pretty quickly. I mean,

Jonathan: so that's, that's like been the biggest problem with working with the bigger vendors is you call in and you get You get their first tier of customer service and those guys are programmed to say, oh, no, no. There's nothing wrong with our hardware.

It must be, why don't you go and reinstall windows? That'll surely help. That'll fix it. You know? Did your reboot? Yeah. Have you rebo, have you rebooted the computer? Have you rebooted your router just to make sure right. Oh, okay. Have ha I've got, I've gotta ask, have you guys seen have you had to RMA any hardware as a result of this?

Is that, has that been a thing?

Carl: We haven't had to army hardware. We did have one customer who just because of speed decided to move to thread ripper.

So, you know, and they, they had a vote of confidence for pressing, Hey, we know you're going to get this, but we've got, you know, we had we need a fix now.

So we just swapped their desktops out for them. But but no, not something like the not, not really like, it wasn't a return problem.

Speaker 4: Sure.

Carl: That's, that's something you're always watching out for. Yeah. Because if there's, if there's a pattern and it means returns, then you've got, You've got a big problem.

You've got to get something fixed. You've got to get, you have something to fix.

Jonathan: Yep. Absolutely. Rob, do you have anything else you want to get in before we, we get towards wrapping? No, I don't have anything else. Okay, okay. It has been great. I've got to ask, Carl, is there anything that you wanted to tell folks about that we didn't ask about?

I feel like we've covered a lot of things, but is there anything that's just burning?

Carl: Oh, I don't know. I would just urge everyone Play with play with cosmic. It's an early alpha, but

Speaker 4: it's

Carl: tech and it's fun. It's just a fun desktop. And that was you know, our intent is not just to, you know, build something because we're a company and delivered to our customers.

We really just love technology and want to make it fun. And so we hope it's something fun for you to tinker with and play, build stuff. So let's get out there and make stuff.

Jonathan: I do. I do know what I want to ask. That is what, what is the process going to look like for people that have existing Like system 76 hardware running Pop OS.

At what point is Cosmic going to roll out to them? How does that going to work?

Carl: Once our release is final, then there will be a button there. You'll get a notification that says. PopOS 24. 04 is available when you click it, it'll you know, there'll be plenty of disclosures, like you're upgrading to a new this isn't just your typical, you know, upgrade to a new version, here's a new desktop.

Yes. So we want to be very transparent. That's okay. You're not going to have gnome anymore. When you do this, you don't have to do it. You can stay with where you're at and it's going to be supported for, you know, a few more years. So you can feel comfortable there. But, we're, we're we're moving to the future and a new desktop.

Rob: Yeah. So it's good. I was going to say, so you're not going to have a pop up every, every few hours. It pops up and said, time to update, time to update. No, like, you know, some other companies have done.

Carl: Yeah, we'll have to be we, we want to be gentle. Because you do want, you do want to keep people up to date, you know, and there's, there's a balance between being a nag and doing that well.

So I think we're okay with it.

Jonathan: Yeah. So it's going to be, it's going to be the default. And is it going to be the only choice in 20, in 24. 04? Or can people still run Gnome if they want to? Is it, is it even out there still?

Carl: They can run gnome just like running like installing gnome desktop. Gosh, I think that's the the right meta package.

Ours is cosmic sessions or cosmic sessions. I'm thinking, I think it's gnome desktop, but yeah, they'll they'll be able to install gnome and run it if you like.

Jonathan: And it'll

Carl: be it'll be vanilla. You know, so yeah, that makes sense. Comparison like

Jonathan: What you guys don't want to have to do is continue to maintain the Pop!

OS desktop experience on two different desktop environments, right? That's just, that's, that would be ridiculous.

Carl: It would be a lot of Yes, providing the same experience. Yeah, no, we're just we're going to pull the bandaid off and run for the future.

Jonathan: Yep. Yep. Well, I, I've got to say we, we have the Linux show.

And so we talk about this from time to time and I've got to say it's become the thing we will tell people don't install some weird niche distro for your first distro. Don't install some weird niche distro for your grandma or your friend on their first time on Linux. Stick with a distro that's. Solid and that everybody knows something about and so we have this kind of list It's like you use Fedora use Ubuntu or use pop OS And so I think it's it's really saying something that you guys are making that list And in such a short time too.

Yes. Yes And What's really fascinating me is that I think we're getting really close to the point to where it's going to be and you should really set them up with one of the established desktop environments, you know, like KDE or gnome. And I think we're about to add cosmic to that list, which again is really saying something.

But it's also, it's also really exciting, like for the community to have a third real choice there. And so I think, I think that's neat. Yeah. That's great.

Carl: I do too. I, I am of the belief that there can be no, there's no such thing as too many distros, no such thing as too many des I think the whole, it's fun to create things.

Go out and make stuff. That's where the vibrance and the that's what's awesome about Linux.

Jonathan: Yeah, I would agree. I would agree. Alright, so we've gotta ask, what is your favorite text editor and scripting language?

Carl: Text editor? Well, right now I'm using Cosmic Text. So I don't know It's the new default scripting language.

I have no idea. Python.

Jonathan: There you go.

Carl: Yeah. Oh,

Jonathan: I reminded the old Dilbert cartoon. You have drank from the cup of management and so you don't write script anymore.

Carl: I don't know. Well, and the real reason I said Python is because that's what I used to write.

Jonathan: That's fair. It's a thing. It happens. All right.

Hey, thank you, man, for agreeing to come on. It's been a blast. The hour has just flown by and we'll have to, we'll have to have you back. You know, once, maybe after, after the 20, 24. 04 drops and everything has, has gone solid, we have you know, full releases and We'll have you back and ask about how it went.

It'd be fun.

Carl: I'd love to be back. Thanks for having me.

Jonathan: Awesome. Thank you. Thank you so much. All right, Rob,

Rob: what do you think? I'm going to be trying this out on bare metal one of these days and I'm going to be daily driving it way before I should be.

Jonathan: Yeah. I, I feel like I probably will too. I, I, I kind of want to, I kind of want to jump to a jump to it on the laptop here.

I need to wait because this is my production machine, but it's, it's, it's fun.

Rob: I'll have other machines to work on. It won't be my, my only one available. If it, if it, you know, if it's not quite ready yet, but yeah, I think it's ready enough that I could use it for most things, most of the time.

Jonathan: Yup. Yup. That's a lot of fun.

Oh, all right. Well, hey, do you have anything you want to plug?

Rob: You know, just come check me out on the Untitled Linux show, or if you want to Find me directly. Go to robertpcampbell. com.

Jonathan: Yeah, very cool. All right. I want to let folks know that next week we are planning to talk with Laurie LaRusso about Percona.

And that is certainly going to be a lot of fun. Percona is open source database software. And then the week after that, we're talking Ladybird, which Ladybird isn't from the ground up. Browser because there's basically only two browsers right now. And so these guys think there really should be a third one.

And so they're working on Lady Bird. That's gonna be a lot of fun too. So some neat stuff coming. As far as finding me, of course there is Hacka Day. Keep an eye on the Hacka day. We've got the security column goes live there on Fridays. And then there's also the Untitled Linux Show, which I do with Rob and the rest of the guys over at TWIT TV on the TWIT network.

And that is a blast and you should definitely check it out. There's also my YouTube channel, if you're really, if you really want more and that is mostly MeshTastic content these days, but if that's something that you, you want to get into it's youtube. com slash at J. P. Bennett, I think. I don't have that pulled up, I believe that's what it is, or you can just search for me on the, on the YouTubes and you'll find it.

Some really exciting stuff in MeshTastic coming with the 2. 5 release coming up, so check that out if you're interested. Other than that We appreciate everybody being here. Those that catch us live and get us on the download. Thank you so much for watching or listening, and we will see you next week on Floss Weekly.

Episode 797 transcript

21 augusti 2024 | min

FLOSS-797

Jonathan: Hey, this week Dan joins me and we talk with Pádraig Brady about Coreutils, that software that pretty much everybody runs, whether you realize it or not. And because of some reasons, it is one of the most conservative development processes we've ever talked about. You don't want to miss it, so stay tuned.

This is Floss Weekly, episode 797, recorded August 20th. Don't RM RF up the tree. Hey everybody, it is time for Floss Weekly. It's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got a fun show today. We're gonna be talking about core utils, and no, that's not a typo, not a mistake.

We did just talk about core utils, but about three, four weeks ago, we talked about Rust core utils. And this week, we are talking about the OG, as I like to say, the the original Core Utils. Still, still a project. It is not just me, though, of course. I've got I've got Dan the man, Method Dan, the original Linux outlaw.

Hello, sir. Welcome.

Dan: Hey, it's good to be here. Thank you, Jonathan.

Jonathan: Yeah, it's always good to have you with us. Dan, I suspect that you sort of have a clue about Core Utils. You've at least used them a few times over the years, right? Right.

Dan: Oh God. Yeah. I was just gonna, I was just thinking earlier that I would, I would bet that anybody who uses Linux as their operating system has, even if they don't know it, interacted with the projects we're going to talk about today.

Jonathan: Yes. Either directly or indirectly. Right. And, you know, there's, there's several different implementations of core utils because you've got, well, you've got the rest core utils, which is fairly new, but you've also got things like BusyBox that also includes a bunch of the core utils, but with a different code base.

And so they're, they're just, they're ubiquitous. I mean, there's probably, oh goodness, there's, if you, if you count the, the other versions of it, there are probably billions of copies of core utils in the world. And that's, that's, That's that's something not everybody can say that that they've got that many of their binaries running around.

All right. Well, we've got we've got the man We've got paw rig and he is I believe the corp the the head maintainer We'll have to ask him exactly what his title is but he is he is the man in the core utils Project that's Parag Brady and let's go ahead and bring him on Parag. Welcome, sir.

Padraig: Hi everybody.

Jonathan: Hey, it is great to have you Great,

Padraig: great to be here. Thank you very much.

Jonathan: All right. So what what is your official title as it were? I I believe you're the head honcho, but that's probably not what they call you

Padraig: I wouldn't say that. It's open source, so there's no real official title, but I'd say if you had to give me one manager, it'd be release manager.

And I'm one of the, one of the maintainers. There's a, there's a few of us.

Jonathan: Okay.

Padraig: Okay. And I've been contributing for the, to Coreus for the last 22 years.

Jonathan: 22 years sounds like a very long time, except Core Utils has been around for quite a bit longer than that, hasn't it? How long, how old is the project?

Padraig: Indeed, well there's I guess it was originated around 92, I think, it's when, well, there were original it was a bit separated into file utils, text utils, and shell utils, and then a little while after that it was amalgamated into a single Core Utils project. So that, that was done by Jim Meyering.

Jonathan: Now, the, the, the, like the original, original core utilities, some of these utilities go all the way back to like Ritchie and Kernaghan, way back in the day with the original Unix. Is there any shared code base?

Padraig: No, it says that the GNU says that it's a part of the, GNU thing was to be a kind of a complete separate implementation.

So, so, so they were implemented separately, the GNU source space is completely separate. But, but of course, like there's, there's a huge focus on compatibility with those older utils and with other systems. And I guess at this stage, the, the, the, the, the, the, the, the, the, the, the, The, we've been developing them so long there is an onus to be backwards compatible with ourselves.

So that's a, that's a, that's a huge concern.

Jonathan: Yeah, that's interesting. So I mentioned in the pre show we had we had the Rust Core Utils project on and that, that was one of the really interesting things. So, so first off. There is, there's, there's not animosity between the two projects, but in fact, you guys have cooperating on some things and one of the really interesting places of cooperation is in writing sort of this shared test suite that determines that you know, all of these tools do the same thing, no matter which implementation you're looking at, which that really interested me.

Padraig: Well, absolutely. Look at the implementation is secondary after the day. At the end of the day, it's the interfaces with users, and that's, I guess, encoded in the test suite. So we put a huge emphasis on the test suite. And it's something personally I've been focused on all along, like, like any kind of changes or patches we do tend to be mostly focused on.

Most of the effort is actually put in, put, put into testing and writing tests for, tests for patches. So in regards to the ROS Coriatos, I actually, like a few years ago, I noticed with interest the Rust coreutils project and suggested that they could easily tie in the coreutils test suite because it was kind of written like that.

So the Rust test driver, they essentially put coreutils earlier in a path and just call our test suite. And it automatically pulls in the Rust utility. So, so, so we write our test suite to be with portability, portability in mind. Because we have to run our test suite in a lot of places. So Rust gets to, the Rust core utils gets to take advantage of that.

Yeah. And just to mention, it's worth mentioning that the, it's, it's a two way thing. So, so sometimes, Rust folks, when they're implementing new utilities, they might notice bits that we haven't tested and they'll supply patches to us and we'll implement those. So it just verifies our code is working as expected.

Jonathan: yeah, no, that's neat. And I've found that test suites like that, what they're really, really useful for is when you make changes, They, they sort of help you guarantee that your changes didn't accidentally break something that you didn't think about when you're making the change.

Padraig: That's, yeah, 100%.

Yeah, it's brilliant. Yeah, we put a huge effort into the performance of the test suite. Like, you can run all the tests Say on a standard laptop in about 50 seconds, but the nice thing I've run it on really, really, really fast machines and it automatically scales up so you can run all the tests in five seconds.

That's impressive. 100, 000 machines. But it's, it's not, it's not nice that it scales up and it kind of I guess it shows. The advantages of the Unix model, because we, we tried to rather than saying, writing the test suite in something separate, like, driving it with Perl or Python or something like that, we actually, it's mainly shell scripts, so we were kind of reusing the model and reusing all the tools while testing, writing the test themselves, and it shows that, that's one of the nice things, as well.

So, But having separate processes as you do separate things, they automatically scale up to multiple processors. And so it nicely scales up automatically using the standard Unix model.

Jonathan: Yeah. So a question that kind of naturally flows out of this conversation is, what about the, like, the original Unix version of these utilities?

Is there any cross pollination with Unix? Any of those, like, I guess we might be in a place where one of the original Unix is, somebody's version of Unix uses core utils rather than the old Unix code base. In fact, that's probably likely. And then I guess also the original Unix version of the core utils.

Do we run those through the test suite to see what they do?

Padraig: It's yeah, well, I haven't done it myself. It's an interesting question. I'm sure a lot of them would break, like the test, like ROSCore, you tell us, is trying to be compatible with the latest GNU upstream. So there are new options and behaviors really there.

So, so I guess a lot of the older, older variants would break in most ways against the, against the existing test suite. So.

Jonathan: Yeah, yeah, that's interesting. Do you know, does anybody's Unix ship core utils rather than the old code?

Padraig: I don't think so. So the main focus with core utils is Linux. And essentially these days Linux is ubiquitous.

So that's, that's generally what everybody really uses, like in the large tech companies. It's, it's all Linux internally.

Jonathan: Unix itself is kind of dead these days, isn't it?

Padraig: Pretty, pretty much. Yeah. It's, and that kind of gets on to the kind of the portability aspects of it. It's still very important.

Less going forward, as we said, because things are more and more focused and more and more consolidated on Linux. But there's still a lot of portability concerns with compilers, different compiler options, different shells. And we still try to be as portable as possible. And when you keep as portable as possible, keep your code as flexible, I guess, as soft as possible.

And it keeps the interfaces true and separate and good.

Dan: I mean, Parag, you mentioned there that the portability and stuff, and Jonathan said nobody really uses Unix anymore, which I imagine would probably upset a few people, but I'm not much of an expert, but I'm wondering, does BSD, do people ship core utils with BSD at all?

Padraig: Yes, so that would be our main other Unix portability target, and we have full, personally, I have access to, like BSD, we have access to free BSD systems, and kind of implicitly through macOS that they use kind of free BSD interfaces. I was

Dan: interested in Mac OS, I was going to say, cause you've got Darwin, which is the the kind of the, the, the base under Mac OS and they must use core utils, I would imagine.

Padraig: They actually try, like Apple tries to steer clear for GPL three reasons but, but there are a lot of users of core utils. It's in homebrew, for example, so it's easily installed on Mac OS. And for, for a long time, the, the sort. For example, the sort that was used in FreeBSD and MacOS was actually the GNU version, because that's actually quite difficult to implement and stuff like that.

But, yeah, getting back to testing, like we do kind of put in order to make sure that we have full tests passing on MacOS and FreeBSD.

Dan: That's awesome. Now you mentioned that you've been working on this for, you've been involved with the project for 22 years, which is impressive. I'm always really intrigued in how people got started in these things and how they got involved.

So how did you come to get involved with, with the project? How did you get interested in it? Was it something you were, it was computing something you were into as a kid or, you know, was that, how did all that come about?

Padraig: Well, not as a kid, I'm not going to say my age now, but I hadn't access to a computer until kind of midway through college.

So I started very late from that point of view. But I started first also with Windows, which is interesting. And I was using that through college. And then Went into industry for a year or so and got a bit frustrated with the whole black box nature. It come to a problem and then it was really difficult to actually fully, fully fix things.

So you you're working around issues more and more rather than actually fixing core issues. So then I happened upon it. Linux was just starting out at the time. I guess that's the new thing to my age. And, um, at that stage then, it, it, it's something that, that resonated with me. I tinkered around with it for a couple of years and at that stage then I kind of the Unix model really resonated with me.

And I saw, like, there was a notice that some of the core utilities would they'd benefit from some new options, and so I proposed a few patches around 22 years ago, and they were accepted. And ever since then, I became a kind of an official maintainer, maybe about, I suppose, 16 years ago now.

Dan: Wow, that's amazing.

I always love how people get involved in these things because I think one of the great things about the kind of free open source software world is how, without meaning to be cheesy, open it is, you know, in that you can get involved. What were the challenges in kind of getting involved? Was it, what did you, were you nervous about maybe submitting a patch or and getting, you know, rejected in some way?

Padraig: No, I was very excited. I remember my first patch, how naive I was. I was, I was worried about, well, the code was okay, but I was worried about someone was going to come up with the idea. It's one of the funny things to look back on. I thought someone was going to, it was such an obvious thing. I thought someone was going to come up with the idea and I was mad to get the patch in before anybody else did.

But sure, of course, even there's an infinite amount of code. code to write and an infinite amount of ideas, so that's never a never an issue. So, so that that's one thing I would say to people is even if something is implemented already, there's always a way to do it better so. Always err on the side of sending in the patch, and generally people involved in these projects are more interested in the tech itself and are very interested in incorporating new people and new code into the projects.

Dan: Yeah. And a question I have to ask, it's the kind of dirty question in the room, but I'm really interested is have you managed to get paid for working on core utils? Has that been like part of your job at any of it? Cause I know you've worked in lots of places.

Padraig: Sure. Not directly but I wouldn't be working where I was without having worked on your core utils.

Let's put it like that. So, so there are a lot of it's, it's a good thing on your CV.

Jonathan: Yeah, yeah, no joke, and they say,

Dan: I know you've contributed to it. Sorry. Go on, Jonathan.

Jonathan: I was just going to say, they say with the kernel itself that it's like five, landing five bits of code. Landing five pull requests in the kernel is the average that it takes for someone to get a job offer as a result from it.

Like there, there are a few places we're contributing open source code. It's just. Excellent for your career. And I imagine something like core utils is going to be on that list.

Padraig: Yeah, it's just, well, I guess anybody who's used it as a set at the start of the show, anybody's has used Linux at all has either used them directly there and everybody knows, knows about them.

So it's it's it's, it's good for the CMU as we say. I've been very lucky over the years, really, to have been, I feel very lucky to have been involved with the project and it's still very interesting and rewarding going forward, so it's all good.

Dan: Yeah I'm interested in how, I know this is probably an obvious question, but I'm interested in how the project's managed.

You mentioned that you've got, obviously there's a few maintainers involved probably quite a lot of maintainers I would imagine. How does it compare to something like, like the kernel project? Do you, do you manage it all through Git and, and the releases and all the patches and everything else?

Padraig: Yeah, so we moved to Git fairly early on. Patches are managed with a mailing list still. The number of maintainers, there are, I guess there are three or four kind of central core maintainers that work on it over the years. There's a separate project, so kind of focusing on the portability aspect, there's GnuLib, project, which is probably slightly a bit more active, and there could be, say, 50 projects reusing the portability code that is abstracted away or encapsulated away in the GNU Live project.

And so, so, so, but, but GNU Live was kind of originated as core utils. So they were looking at it simply, there were lots of if defs in the core utils code, and that was gradually moved across to a separate project to keep. So GNU Live presents a GNU interface everywhere, and then it allows the actual projects using GNU Live to be a lot cleaner, and just as soon Like a new interface is available.

Jonathan: It seems like code bases just grow if def statements over the years. It's just a natural part of their development. Especially

Padraig: something trying to port to every Unix and every compiler in the world. You have to be especially careful.

Jonathan: Okay, so, with a project that's been around as long as Core Utils has, and with sort of a, in some ways, a frozen specification, What's it, what's it like to work on core utils and Are, is it, is it done?

Is there a place where it's going to get to be done and what does it look like? And so I guess, I guess really what I'm getting at is I asked that question very tongue in cheek, but what I'm getting at is what is it, what does it look like in core utils to make changes? And I assume it's a, it's like a very conservative process to very slowly make changes and to do it very intentionally, right?

Padraig: Well, yeah, you have to be, something that's used so ubiquitously, you have to be very careful. So there's the whole, as it was, mentioning earlier, the focus on testing. So, that kind of handles that. But we have to be very cognizant of the interfaces. And also the, we have to be cognizant of our interfaces with the community.

So, there are lots of requests over the time to add, over time to add this option and that option. And a lot of them are good suggestions, but they're not just appropriate for adding, because the equivalent functionality might be in a separate tool might be slightly dangerous. But we do take and we do make new additions ourselves over time just to add new functionality, and port to new compilers, and new architectures, and enhanced performance, and portability, and all that.

So there's changes all the time. But just getting back to engaging with the community, that's great. Like, like that's one of the things. You have to be especially cognizant and careful about that with an open source project. And one of the things we've done, for example, is we've maintained a page of rejected requests.

But they're very carefully considered and curated. And it, like if someone comes in with their with their suggestion, and we carefully consider it but reject it, we may add it to that page. But they can also see. If it has, like often, the same suggestion has been made multiple times. They can also see similar, uh, similar suggestions being rejected with very careful, carefully considered reasons given.

And so they don't feel I guess alienated when we, when we give feedback like that. But, but, but we're definitely very open to, to new features if they're appropriate. So it's just, we have to be just careful of that. Backwards compatibility with ourselves the compatibility going forward and just being, I guess, generally cognizant of the Unix model and just being true to that and keeping things appropriate.

Jonathan: Yeah. Let's see. Oh, progress bar. What, what's the, what does it look like when, when someone sends in a request? And let's just take the progress bar as an example. What's the process look like? Like, do the maintainers vote and say, you know, three, three out of five say, this is a bad idea, so we're not going to pull it in or just That's essentially it.

Padraig: Yeah. And it all happens out in the open. The, the, the important thing is it happens in the open on, on, on the mailing list. So we give reasons why it mightn't be a good idea. That's An interesting one. That's one of those 50 50 ones, which is probably a good idea, but it's also implemented already in our sink and stuff like that.

So do you want? Do you want to complicate the code base just to add that? That is one that we may actually add eventually. It's There, there, there's an interesting one like that. So, so that's getting back to the Unix model. Mm-Hmm. . So it might be nice to do that more generally. So looking at something like pv, so that's pro general, that's a separate progress viewer.

Speaker 4: Mm-Hmm. .

Padraig: And, and you can point it at any command and it, it'll open up the file descriptors and. Now, it's, it's not as general as if you put it directly in CP, but it's more general in that it will work with any command and you can pop it in the middle of any pipeline. But one of the reasons we hadn't done that particular one was rsync is equivalent functionality that already has a progress bar.

And looking at The Unix model of thinking, having one tool that's doing something more general, then there's a separate PV tool which can be pointed at the CP process, and it will inspect all its file descriptors and see how far along they are at reading and writing a file, and you can pop it into the middle of any command or any pipeline or directed at any command.

So it's a more general solution.

Jonathan: Yeah, and it's also interesting. Some of the other, like, for example, the Rust core utils, they've gone ahead and they've added that in, I think, at least in CP, maybe in some of the others where it has now a built in progress bar, and that's one of the fascinating things about having multiple implementations of this that are kind of looking at the same the same test suite and the same kind of core rules.

But you have a little bit of you have a little bit of flexibility. as well to be able to do things just a little bit differently without breaking the rules.

Padraig: Right, right, right. And like, it's an interesting thing, like, if there are kind of borderline functionality or new options like that, that are already implemented elsewhere, that that's more kind of more leeway for us.

To implement those for better compatibility. Now,

Jonathan: so the, the, the fact that this is again, talking specifically about progress, the fact that this is still kind of being reconsidered it makes me think that, that, that website, that, that page of these are the things that people have suggested that we are decided not to do, like, that's not necessarily a static list.

And, and you guys have the, That you have the freedom to go back into that and sort of mine for ideas again and reconsider, well, maybe this wasn't such a bad idea.

Padraig: Absolutely. Yeah. And some of them, that doesn't go for all of them. That there's some, some special

Jonathan: ones in there. Yeah.

Padraig: Not to, well, this happened long ago.

That's a, I'm not singling anybody out here, but there was one suggestion that so for RM minus R. To recourse down the way, you could have an option to recourse up the tree. Yeah, we rejected that one

Jonathan: pretty quickly. That, oh yeah, that's, that must be, that must've been like the, the April 1st RFC for the project.

No, no,

Padraig: there was some, some arguments for it.

Dan: Really? And also, also implied the dash F as well. Just

Jonathan: don't accidentally use that flag. Oh my goodness. God, yeah. Uh, and, and then what about when you, and I'm sure this has to happen from time to time, but I'm sure it has to be rare. What's the process if you say, okay, we're going to make a change and it's going to break something, but we're going to make it anyways.

What does, what does that look like? Has that happened? Is that going to happen?

Padraig: It has happened. As we said, we're very careful about doing that. Generally. It's, it's only on if it's associated with some extra functionality that we're doing. So we break compatibilities in some rare edge cases just to allow you to add a lot of extra functionality.

So it's rare we do that and we err on the side of not doing that because, like, nobody wants to rewrite shell scripts when they upgrade from CentOS. Thanks. Bye. 10 to 11 or whatever. So we, we, we just have to be very careful about doing that.

Jonathan: Yeah. And is, is there anything sort of on the on, on the radar that is going to be breaking or maybe otherwise a, a huge big change coming?

Padraig: Nothing breaking interface wise? Okay. Look like, look retroactively, look, looking back at some things that the of another or we, we have another page kind of written of core util Scots. And these are things like, you would never have done this originally if it was designed as one cohesive set of utilities.

These things would never, like, just one gotcha, for example, DD, you often want to present hex numbers to DD because you're dealing with power of two blocks for inputting output to disk and stuff like that. So it'd be nice to supply a hex number. So if you could, and you can. seemingly supply hex numbers like 0x100 for, say, 256 byte block, but 0x100 to DD is 0 multiplied by 100, which is, which is 0, which it accepts and goes ahead and just doesn't skip anything, for example.

So, so there, there are little gotchas like that, that haven't been very carefully considered back in the day, but we have to keep compatibility with that going forward. Thank you And so please, like, we'll break compatibility slightly in that regard. For example, POSIX specifies, if you're not giving an error, you shouldn't give a warning.

Like if you're not exiting with an error status, you shouldn't give a warning, but in that case we do give a warning because it is such an edge case that you probably wouldn't be doing a zero multiplied by something, but we give a warning in that case. So we break, that's not really breaking compatibility, but we're just very careful about how we approach these sort of things.

Jonathan: Yeah, interesting. Alright Dan, you want to pick it back up? I've lost the connection to our back server, so we're having to do it in the open.

Dan: It's all going on today. Yeah, it's been a fun

Jonathan: one.

Dan: Yeah. I'm interested in one of the things that you actually mentioned Pari was the Unicode situation and internationalization as well as another thing as well for for character sets and so on.

So what's the situation with, with, with that at the moment?

Padraig: Yeah. So that's it's. A tricky kind of implementation thing that spans most of the utilities, especially the original text utilities. Personally, I was interested in doing that, and while I was working with Red Hat, I requested, say, a block of three months to go away and kind of just implement that, which wasn't granted, which is interesting.

I was surprised at that at the time. never had really a block of time to, that are required to go away and work. So that has kind of happened in piecemeal over the time, over the last few years. So the main Unicode functionality is currently encapsulated in GNU lib. And there's a lot of Unicode expertise By the, the developers and the main developer, Bruno Hebel of, or of working on Cano Cano lab.

And those interfaces have evolved a little bit over the last couple of years. They've added new abstractions for dealing with characters and multi characters and cells and stuff like that. So that, that has been gradually being added to the corridors over the last while. And, I kind of created a planning document, kind of describing the work that had to be done there.

So at least we have a kind of an overview of what needs to change. So it will eventually happen. Just it's happening slowly at the moment. And one thing that has changed over the last while as well, like when we originally envisaged this, it had, there was a lot of different character sets that are in use, but most things have consolidated on UTF 8 now, so that kind of suggests different ways of handling things of converting everything to UTF 8 before processing and maybe having separate tools for kind of sanitizing and working with UTF 8.

And then as an interface to other utilities, rather than having each utility dealing with edge cases of mis encodings and cases like that. So yeah, it's still a work in process. So, I guess that would be the main kind of functionality or feature that, that. Is kind of outstanding and core utils at the present.

Dan: Yeah, so it's a big one to deal with I imagine It's interesting that you said that you I mean without getting into politics too much that you weren't granted the time to work on that That's you'd think a lot of people would be would be after that, but who knows? Now this is a slightly left field question I suppose because Jonathan mentioned busy box at the start And I am going to show my ignorance now because i'm not entirely sure of the relationship between core utils You I'm busy box.

So seeing as you're here, I'm going to ask you is, is there any kind of crossover between the two? And so I'm assuming not in code. How does, how, how is the, how is the relationship between the two and how's it, well,

Padraig: definitely not in cold. There, there is a little interaction sometimes between Suggested new options and stuff like that.

One interesting thing, like busy Box is more geared towards embedded systems. Yeah. And it has and there's different licensing and stuff to, to date with that. So, so the, the, the main difference there is licensing. Interestingly, a while ago, um, core Utils was adjusted to be able to be built in the same single binary setup as busy box.

So. So the standard way you would build BusyBox is as a single binary with symlinks mapping the various command names to the single binary, and the core utils can be built exactly the same way. So you can install core utils in I think it's a, it's a couple of megabytes with sim links to the single binary.

So, so from a functionality point of view Coriutos provides the same things, I guess, with more portability. But yeah, there's the licensing aspect, the main difference there.

Dan: Now, you mentioned licensing and my eyes lit up because I'm a bit of a licensing nerd. I don't know why I just find this stuff fascinating.

Now I, I was working that I had, I was interested that you're under GPL V3 with core utils and it's, it moved from GPL V2 or newer to GPL V3 or newer. I believe. about 2003 or something like that. So were you around at the time? And what was the process like?

Padraig: I wasn't involved in that licensing. I had a few earlier patches, and then I joined really the project a bit after that.

In a, in a more involved way, I guess. So I wasn't involved in the licensing and I haven't really been involved in the licensing, mm-Hmm. any licensing issues or anything like that going forward. So I'm the wrong person to talk really about that. I'm more more focused on it. That's okay. The, the technical and, and to be honest the, all the, the core maintainers were, were focused on the, the technical aspects really.

But we are aware of the. I kind of, some of the restrictions of GPL 3 just even from a political point of view some people just kind of shy away from it and keep things simple and just avoid it. So that, that's, that's not an idea.

Dan: Yeah, it's an interesting one. I, I, I've got, so the kind of little background to this for me is I have some friends who work at the Software Freedom Conservancy who were involved in a lot of that kind of, like you know, That's licensing.

That's Licensing Central right there. And then I know that they were very keen to get projects to move from v2 to v3 of the GPL, and some were keen and some were not, and some still haven't, like the kernel of course. And so I just thought I'd ask, but that's fair, totally fair enough, that you weren't around or you weren't involved with the licensing at the time.

Now, one of the things you mentioned to us in your email was that you worked for Meta for a good long time and their use of Linux in the back and the stuff that you've done with them. So I'm really interested to dig into some of that because I know that they have their own distribution. How does that work?

Padraig: Sure, I can't go into very details about absolute numbers or anything like that, or details, but I'm happy to go into general general information, and a lot of this information applies to all the big tech companies like Google and Amazon, that they all use the same sort of models. Stuff like that.

So I guess the interesting kind of general information with Meta is to have a huge scalability requirements. So a huge focus on performance, like if you've got, for example, a 1 percent win out of compiler, they're talking hundreds of millions of dollars. The scalability is immense, but it's interesting as well, not in particular to core users, but maybe more aligned with a project like glibc or that kind of library code.

That when you're working on these things in open source, it's used by Meta and Google and everybody else. So it actually has much larger scalability concerns, but it's not really as apparent or, I guess, measurable. to these people. So it's there is a, I guess, there's a huge onus and responsibility on people working on performance.

Like, when you make a change in meta, it's, everything is easily, very easily tested. And You can see exactly the dollar values of changes. And so that's cool as well, because you can feed those, it's easier to test, it's very hard to test things kind of open, open source world, because You have a lot more, it's just harder to test because you have a lot more disparate systems and you haven't everything as tightly cohesive in a whole test framework and stuff like that.

So there's a kind of a symbiotic relationship both ways, like you get really good testing in a place like meta, and then you can feed that information back up and feed the code back up. But, but there's an interesting thing as well that, like, Corporations like these, they get huge use out of open source code, but sometimes there isn't the, I guess, the focus on sending code changes back up because just looking at the loop, there's a short term win from not spending the time to send your changes back up in top string.

But then there's a long term loss by not doing that because you become forked and you kind of diverge away from all the improvements upstream as well. So that was a good bit of effort on my part in META was kind of ensuring that all our internal processes and changes were a bit more. Getting code back open into the o open source.

So, so we didn't become diverged and, and it's something that that's really easy to fall into, but because of the short term win and not doing it like, like even open source first, companies like Red Hat back in the day, they, they originally got into that, that sort of situation. They had had a fourth kernel and it's, they, they actually got into a, a bit of a knot that, that they had a huge effort then to get outta that.

So it's just an interesting thing for any tech company or any company these days, since tech is involved in most things. They have to be very wary of not forking away from upstream too much.

Dan: It's good that that they, well, it's great that, that it sounds like they were supportive at least in, in you contributing stuff back upstream.

Was that something that, like, the management, without meaning to cause any trouble here, and feel free to tell me to, you know, you don't want to talk about it. But I was just curious, were the management and so on, were they supportive in that? Were they like, yay, go and do it? take a day to do this or, you know, support it.

Padraig: Absolutely. Like like in Meta at a certain level, especially you were encouraged to go off, you were trusted to do the right thing as long as you presented the right arguments. Like they were happy as long as you're doing the right thing. So, so within Meta was very open source friendly. And increasingly going forward now with the AI ecosystems that they're really kind of leveraging the kind of the open source aspect of that.

So, so, so no, no, there was great support in, in meta. I was kind of more alluding to the kind of the general kind of thought process of engineers in general within tech companies. They were focused on the short term wins not, not, not on getting their like we're focused we're working away in open source a lot.

And most developers haven't that mindset of pushing stuff back up.

Dan: Yeah. That makes sense. We actually have a question. So there are people listening and watching us at the moment. And we have a question from Mashed Potato, who says, What's the relationship like with developers of Core Utils replacement tools, such as Exa?

Could Exa find its way into Core Utils to replace LS? Is there a good reason to keep LS as it is? That's quite a big question.

Padraig: Well, absolutely. I've, it's a long time since I looked at Exa. There are a few interesting, like, I do have a look at every so often at tools like this, and if there are interesting general functionality that would be appropriate for everybody to use, absolutely we would incorporate it into LS. The big thing you have to be aware about with making changes is the interface.

You, Wouldn't move everybody to having to type exa ls is kind of wired into everything at the moment. So but you could add functionality or the ideal thing is to adjust things.

You have to be careful as well, but the ideal thing is to adjust things without requiring options or changes on behalf of the user. But you have to be careful. So, LS is an interesting one. So, it's an end user tool. So, we made changes recently and there were really good reasons for making the changes, which was to quote filenames that had Problematic characters in them, like spaces or shell characters that were special to, to shell becau because it, it introduced un, unless you quoted the names, there would be ambiguity with the, the spacing for delimiting file names or were the spaces within the file names.

Or you could have semicolons and you could put commands in there. So if people are copying and pasting and you can, there, there's a lot of way to, ways to hide stuff. So, so there, there was security implications there. So by default, we now quote the output of problematic filenames from LS. And there was actually quite a lot of pushback from that, from various people.

And the main reason is because they weren't used to it. And

like we of course provided, always provided a way to go back to the old behavior if we really wanted. But yeah, you just have to be very careful about how you adjust these tools, especially stuff user facing stuff like Ls.

Jonathan: Yeah, it's interesting. I just went and looked. The, the EXA command itself has been retired and a fork.

Iza is now what is the what is the latest and greatest. So and I think, so yeah, it's, it's interesting to look at it cause there's going to be some great ideas there. But what's also fascinating is that Core Utils has been around since 1992 and Exa only lasted for, you know, however many years.

And now the, the main developer of Exa is missing and cannot be contacted anymore. Like just. Just the fact that Core Utils has been successfully maintained for that long is like a, it's a huge win and not, not every project can say that, that they have, you know, that sort of a track record that 20 years from now, you can pretty much guarantee that Core Utils is going to be around and still putting out releases.

Padraig: Yeah, absolutely. And like for companies kind of investing on a platform and they're writing shellscripts and stuff like that, depending on all these utils, there's a huge kind of responsibility for these things to stick around and be stable. And just another aspect of that example, it's good to expand on a particular example and look at all aspects of it.

Like if we were to, say, take things out of ESA, now is it? And add them into LS, there might be a little bit of extra performance in every LS invocation, which that gets back to the point about how there's a responsibility on us to mine performance because it moves out into all companies and all users.

There was one example there recently where file capabilities were colored, and file capabilities are one of the things that never really took off in Linux. So, maybe one in a million files has capabilities now. So there's no real Advantage of coloring. So there's an overhead there of every LLS invocation looking at every file to see does it have file capabilities.

And it's one of these kind of esoteric things. So it was never kind of the interfaces to detect file capabilities was never optimised over time. So kind of a rule of thumb. Or kind of a what I call it, a back of the paper quick calculation I did was that by taking that out of LS, nobody really noticed, but it saved about 75, 000 worth of electricity a year, just estimating the use of LS around the world.

And so, which is, I don't know, maybe 40 households of electricity. And just. By not having this extra little bit of functionality in LS. So these things are important.

Jonathan: Oh, I'm so tempted to say something sarcastic about cryptocurrency there, but I think I'm, I think I'm going to not. So that, that idea of performance, though, that does bring to mind a really interesting question, and that is when Say AMD pushes out their newest processor and it's got AVX 512 support for everybody now.

Of course, Intel has gated that off to their pro line of processors. Are there changes that get made to some of these core utils? Because, Hey, now we have AVX 512, and oh, you can do string comparisons in AVX 512. Like, are you guys sort of on the cutting edge of that, watching those changes in processors, and then therefore going in and making tuning changes in some of these commands?

Padraig: I wouldn't say we're on the cutting edge, but we're definitely incorporating changes such as that. There's two aspects of that. Again, we try not to implement everything ourselves. So, looking at the crypto side of things, or the hash side of things, we, rather than implementing Assembly slash Cindy versions of those ourselves, we kind of push off to live crypto or the open SSL libraries because they're, they're kind of ubiquitously available as well.

So we'll, and we'll with version three, we can, and the licensing changes there, we can without issue link to those. So, so we, we'll link to those by default if they're available and use the fast version. So, so the checks on or cha 2 5 6 on or whatever. We'll use the, the, the latest version of those, but, but also within like core functionality ourselves.

Like for example, wC for counting lines that can be done efficiently in SIMD code and AVX code. And, yeah, we do have code for that. We have to be careful in portability, so there are special portability constraints in how we set up the build, so we actually added libraries to the build system to support that.

That's how you efficiently are. Kind of definitely separate all AVX instructions into separate compilation units in Automake world. So we have libraries that we, internal libraries that we link to to implement that for a few utilities now, and probably more going forward.

Jonathan: Yeah, I know kernel and GCC itself, You have a MD and Intel employees that come through and, and send these big patch sets in.

Like, here's support or here's the tuning for the latest, the latest and greatest from our company. Do you guys get any of that in Core Utils? Are there, are [email protected] [email protected] email addresses, sending patches in?

Padraig: No, to be honest I've had interaction with those guys from working at Meta and ver various places.

And various other projects, but not directly in core utils.

Jonathan: Okay. Let's see. I was going to ask about Dan. What was I going to ask about? That was, that was a short answer. I was going to look it up. Retiring commands. That's what it was. There are, there are some core utils commands that have been around for.

for years, for decades. And it's like, some of these I don't think anyone has really used in decades. And is, have you, is there a thought about, well, let's just retire these rather than making them continue to be part of the the, the maintenance burden, or are they just, are they going to be around forever?

Padraig: Good question. So there's two aspects to that, really. There's individual commands and then there are classes of commands. So answering classes of commands at the initially. Like, for example, all the, the separate checksum commands, like nd5sum, sha1sum, sha25sum, blah, I'm not going to go on, but you could go on forever there, and that's kind of a bad way to do it, to have a separate command for each of those.

So, going forward, we're consolidating those in the checksum checksum minus a, then you select your algorithm there, and so, We won't have any more of that class of command. Everything will be consolidated in checksum. And I guess in 20 years time we'll start removing SHA 386 sum or whatever, just to clean things up.

So individual commands then, yeah, there are some commands that are less used, like ptx, tsort, this sort of thing. I guess the main idea there is that there's less maintenance. We don't much maintenance focus. Like if we get a compiler warning out of them, we'll fix it up. Or if we get some security warning or whatever, we'll maintain it.

But we won't put much effort into adding new features or changing functionality on those going forward. That's the main thing. As for removing command, there's the big compatibility concern. Like These are, these utilities are so used and that there's some edge case on. Some space probe in Mars or something, we just can't, we can't, we probably can't remove, remove commands, so it wouldn't be that much maintenance going forward.

And on the other hand, that's why we have to be very careful of adding commands. And even options in that regard, but in added commands, we have to be very careful.

Jonathan: Yeah. All right. I do want to make sure and ask, is there anything that is coming that you wanted to let folks know about? Like, are there any future plans that you're particularly excited about?

Padraig: I guess the main one, you already asked about the internationalization and Unicode support. So, so that is something that's coming gradually. And we're definitely focused on that, and it's happening as we speak. In the last release or two, there have been updates to expand and un expand and utilities like that for them to handle multiple characters, so that will be coming.

Perhaps there might be a new utility. We've moved it for a while about a replace utility. So it's interesting. To replace a file on Unix is actually really difficult to do it with ACID principles, and copying data around with ACID principles is actually really difficult. So, kind of makes a lot of sense to have that encapsulated in a separate command.

Like rather than have sed minus i, you can have The replace command and provide another command to do the actual processing and then the replace would do all the complicated stuff about temporary files or kind of moving files to atomically and and all that anarchy. That, that's something that, that might be on the horizon.

At the end, there's. There's always the ongoing maintenance of new kernel interfaces for example, one thing that's changed recently that might allude to things going forward is we added and we had to be very careful in how we added copy offloading. So copying a file is actually really primitive in a POSIX interface.

It's like you have to copy the metadata separate to the data. You have to, there's atomicity issues there as well, which gets back to the replace command. But recently there's been the copy range command to in the Linux kernel and similar commands elsewhere or similar functions elsewhere to provide copy offloading.

And but we locked the doors maybe 10 years ago. and had a deep dive on those, and they weren't stable enough either in interface or functionality. So we provided feedback to the kernel folks then, and more recently in the last two or three years we've been able to start using these things. So that allows for more efficient copying operations, but we still have to be careful and actively inspect and avoid older kernels and stuff like that.

So, there's going to be changes like that going forward with new kernel interfaces.

Dan: Oh, awesome. So I'm curious if somebody say, listen to this decides, you know, I'd like to get involved with with core utils. That seems like my kind of thing. I mean, I'm always keen to ask people who've made a career out of this and who, you know, have contributed and become maintainers on these projects.

Do you have any advice to somebody? How? What's the best way to get involved and to you. You know, to, to come along, is there anything you particularly need, say, from the community that you think would be great if we had somebody who could do X, Y, Z?

Padraig: Well, there's a, there's a to do file in the repo, in the main repo.

There's a, If you look on the main, the main GNU Core Utils mailing list there's, if you, if you sit on that for any period of time, really, you'll get an idea of the, the work we're interested in doing and what, what, generally the work that's, needs to be doing and interested in doing. There's no we don't have a Well, we do, I guess I'm saying we don't have a bug tracker, but we do have if you go to bugs.

gnu. org, uh, slash there's a core utils section there. And so there are some outstanding bugs that need to be handled as well. And just in general, we're always very accepting of Just interactions and patches are on the list, and we'll try and guide people Like, we're very, very interested in getting code and new people involved in the project.

Dan: Excellent. Yeah, I mean, that's the lifeblood of every project, I suppose. So, Jonathan and I have just been having a little discussion in the back channel here about, about GitHub, because I didn't think you used GitHub, but apparently Jonathan says you are on GitHub, and he want, being that we talked about contribution and so on, we, I was curious, do you, do you accept pull requests from GitHub?

Padraig: We kind of do unofficially, right? So, so we, we, we have a, a GitHub mirror and we, we probably changed that a little bit going forward, but because it's become more, GitHub has become more ubiquitous since we, we had a policy. So currently you can make a pull request against Core, which I set up about 10 years ago.

And, just to consolidate things on the mailing list, we give people advice, like when you make a pull request, as the main commit message it puts in, that you should send it to the mailing list, but you can still create the pull request. So, so we might change the, the policy there to, that we will support pull requests separately as well, because yeah, and I have looked at pull requests there over the last while, and I do monitor that all the time.

So, yep. That's actually, I'll add that to my to do list this evening to change the wording there. At least we're giving you work to do, I feel bad now.

Jonathan: Oh, no, no, that's our job. No, that's interesting to me. I get the, I very much get the the hesitation that particularly the, the thing that I've noticed is that a lot of people, particularly free software projects have, and GNU projects have, about using GitHub, because it's not all open source software.

And I definitely get why people don't love that. But at the same time, it's so useful. And it, like you said, it's become so ubiquitous that it makes things so much easier for people to come in and add, you know, a trivial or a small pull request without having to go to the mailing list. A lot, a lot of projects have arisen with this.

Padraig: Yeah, I don't wrestle with it too much. I'm definitely on the side of defense that I'm not kind of. Very binary on you can't do this and you can't do that. The, the world is gray. So I, I, I'm on the, the side of the fence that whatever gets the, the most, the, the best logic in to the most people is best and mm-Hmm , most people are used to using GitHub, so I have no issue whether using GitHub at all.

Jonathan: Yeah. That seems to be where a lot of projects are coming down. Alright. Is there anything that we didn't ask you that you wanted to make sure and let folks know about? And I know that's a tough question because you have to. think about all the things you wanted to talk about, and all the things we've talked about, and do that set comparison, but Dan, do it, Dan.

Padraig: No, I don't, I'm just, I scanned my notes here, and no, I think that that was a very all encompassing set of questions, so I think you've got everything there.

Jonathan: Yeah, we try, we try. What's the weirdest thing that somebody has done with Core Utils that you're aware of? Have you gotten any user stories that have just surprised you?

Or maybe, you know, requests that have just been off the wall are very surprising?

Padraig: Look, you know, that's Jesus, nothing goes to mind there.

Jonathan: Maybe the, maybe the recurse up the tree was the answer

Padraig: there. That was the most off the wall one, I guess. There was an interesting set of videos I saw from, I think it's Robert Elder.

He, he went through a set of core utils recently, and he did a set of videos where every command, he started off that every command was his favorite command, but he had a very interesting use of timeout for Avoiding saturating his network with a backup. So he'd start off a backup, but if the backup, so the backup would keep copying stuff that hadn't copied already, but he'd time it out after two hours.

So I thought, I just thought that was a very interesting insight. Time out his backup. But it would get, but it would run the next night, and then it had left off where it went before, so it would eventually, eventually copy his files across, but you know, there's lots of esoteric uses. I, so many, I guess so many esoteric uses that I can't remember many.

Very many particular ones over time.

Jonathan: Yeah, that's the thing. You have, you have people that write these one liners where it's 15 different commands and most of them are going to be core utils commands and they do something, you know, ridiculous or really impressive. So there's a bunch of them out there,

Padraig: but on the other hand, like looking at questions on the stack overflow and stuff like that.

And there's, I just, People are, I don't know why they do things the way they do, like, it's just, I should have a section, I definitely don't do this with core utils commands, but, each to their own, you know?

Jonathan: That would be amusing to read that FAQ, the things you definitely shouldn't do with core utils. Oh, that'd be great.

All right, well, I've got to ask you before we let you go, what is your favorite scripting language and text editor? Interesting.

Padraig: Interesting. Scripting language. Well, what is a scripting language? That's true. I'd say my favorite interpreted language is Python. So does that count? Yeah, absolutely.

Dan: Yeah, yeah, definitely.

Yeah,

Padraig: that's one I've used for a long time now. And I get the most flexibility and functionality out of it.

Jonathan: And text editor.

Padraig: Text editor, I use VI. So, an interesting part of that, that's worth mentioning quickly, is that the interfaces, I use CLI a lot, I guess, obviously. And so the main, like UNIX has not been designed kind of cohesively, it's kind of evolved in separate kind of factions with SysV and BSD and stuff like that.

So the kind of, the interfaces have kind of evolved to be bash, or, sorry. Vi versus Emacs in various commands. So I've set up all my systems to have Vi key bindings, and it actually helps with the speed of the interface and helps with RSI, which I don't have any RSI ever, but maybe that's, maybe that's the reason.

Jonathan: It's gonna be the secret why. It's funny you ask what what counts as a scripting language. We had, we had the creator of Bash on back several years ago, and I asked him if Bash counted as a scripting language, and I think he was sort of offended that I asked the question. And Yeah

Padraig: Yeah, it's one of these things.

Jonathan: Yeah. Yep. All right. All right. Thank you so much for being here The hour flew by and we had a great time and I think really covered a good Corpus of information on the core utils and it was it was a blast to have you appreciate it Absolutely.

Padraig: Thanks,

Jonathan: man.

Padraig: Much

Jonathan: appreciated. Yeah. All right. What what do you think?

Dan: Yeah, I think as, as I as I kind of said at the top of the show, everybody's used core utils, whether they know it or not, I think, anyway. I say everybody, you know, I would imagine these days with the amount of things we've got in devices that are running all kinds of different stuff. And one thing I actually found very interesting that Pyrog We told us about was because we mentioned BusyBox and embedded.

I've got a lot of friends who work in embedded Linux and in that kind of world. And I didn't realize you could get like a two megabyte implementation of, of, of core utils. Now, I don't know if they know that or if people from busy, because I also know people who work with BusyBox and they're going to hate me.

If I say don't use that, use core utils, it'd be better, but you know, it's just these options, isn't it? But very interesting stuff. And I think the ubiquity of, of this is, is, is so amazing and the responsibility that it puts on maintainers like, like Parag and the other people involved that this is used everywhere.

So I imagine that, you know, it, it, it must be quite a weight to kind of bear.

Jonathan: Yeah, yeah, it's like we said, it's, it's it says a lot about the project and the way that they run it, that it's been around for so long and it's still a healthy project. So that's, that's neat to see. And I also find it, I find it real fascinating.

So I, I couldn't help but think when they were talking about how they, they very carefully make changes. There's this line from the Lord of the Rings books where they, they discover some like wonderful cavern and Gimli tells Legolas, That he wished he had some of his kin to come and work on it in Legolas.

It's like, you would make changes to that? It's so beautiful. And Gimli says some line about, we would only remove one rock every hundred years to try to make it better. And that's kind of, that's kind of how I feel that they're working with Coriutils. We make one breaking change every 20 years. We didn't, we didn't get to ask him about it, but apparently Coriutils itself has.

a really good record when it comes to security as well. Something like only five CVEs in the last, you know, 13 or 20 years, something like that. And I wish I thought of that during the show to ask, to ask him about, but it's, they're really doing great work. Absolutely.

Dan: Yeah, definitely.

Jonathan: All right, well I think that is, that is it for Core Utils.

Do you have anything that you want to plug, Dan?

Dan: Yeah, I should mention some people, longtime listeners of, of FlossWeek who will remember an event called OggCamp that I used to run in the UK, which is a free software open source unconference, bar camp style event. It was at one time, the biggest in the UK.

I don't know if it still is. It probably it's hard to say that it still is. Cause this is the first one for five years. So there's going to be one. So the, what I'm doing here, burying the leaders, there's going to be one in October and it's been picked up. Thankfully we got stopped by, we got stopped in our tracks by COVID as a lot of people did.

But the events coming back and it's going to be in Manchester. It's in October this year. You can go to ogcamp. org. That's O G G C A M P. And you can get tickets on there. You can find out what's going on. And I'm not actually organizing it anymore, but but Simon Phipps of this parish is is on the organizing team.

And if you want to get involved, you want to find out more about it, you can come to Manchester and you want to you want to talk or, or, or you know, do a workshop or any of those sorts of things, then let us know and head to orgcamp. org.

Jonathan: Yeah, excellent. So things I want to plug is, of course, we thank Kakaday for being the new home of Floss Weekly and you should make sure and tune in next week because we're talking to Carl Ritchel about Cosmic.

The new alpha release of Cosmic is out with their Rust based compositor, some really fun stuff there. I'm going to chat with him about it. And then of course you can follow my security column goes live Friday mornings on Hackaday. And then we've still got the Untitled Linux show over at Twit. And that is always a blast.

That is every Saturday. And we have a lot of fun there talking about Linux news and some open source news as well, but it, that's a, that's a much more Linux flavored show than this one is. But you should definitely check that out as well. I very much appreciate Dan being here as co host and we appreciate everyone that watches and listens both live and on the download and we will see you next week on Floss Weekly.

Episode 796 transcript

14 augusti 2024 | min

FLOSS-796

Jonathan: Hey, this week David joins me and we talk with John Britton and Mike McQuaid about Homebrew, the package manager that macOS is missing, and Workbrew, the new commercial offering built on top of it. It's a lot of fun, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 796, recorded August 13th.

Homebrew! I'm more of a Whopper guy.

Hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something real fun today. We've got, first off, David is in the secondary hot seat, the co pilot chair. He's my wingman. He is the co host today. David Ruggles.

David: I'm holding it down. Yes. I'm holding it down.

Jonathan: Yeah. Now today our, our topic is. Well, it's, it's Homebrew, which is the package manager that Mac OS wishes it had. And neither of us are big Mac guys, are we?

David: Hmm. No I use it. At least once a week, but not extensively. So yes, I have, I have lots of questions that will be the, uh, Luddite.

Jonathan: You're our audience proxy for, I don't know much about this. That happens. That's fine. I have, I've actually used homebrew back years ago we were talking about before the show, I was a part of an organization and someone else that was there was a huge Mac fan. And so they bought some Mac machines. And the whole time I was going, he was.

Associated with the military, so the whole time I'm going, I know he's going to move away, and I'm going to be the one stuck administering these machines. And guess what happened? Yeah, I was stuck administering the machines. So, I installed Linux on one of them, and I installed Homebrew on the other. And we, we made out very well with that.

So I've got a little bit of homebrew experience and a little bit of Mac experience, but, you know, rather than us talking about what we know and don't know about the project, let's bring, let's bring the guys on. So we have John Britton and Mike McQuaid. Let me see if I have this right. John is the business guy.

And Mike is the homebrew guy. Is that, is that sort of the way the land lays here?

Mike: Yeah, I don't know if John likes it put that way, but I'm fine with that description.

John: Yeah. I mean, I definitely wear more of the business hat, but I'm also a software engineer. So, you know, being called a business guy is a little bit rough right now.

Jonathan: It looks like you're wearing the Tron hat. Yeah, I like it. Okay. So first off, how did, how did we do about, about homebrew? Let's start there and then we'll get into this kind of the, the, the business thing that's going on, but I want to know first, let our folks know. And so obviously this is probably a question mostly for Mike.

What, why, why would somebody use homebrew? What's the point?

Mike: So my I guess description for non tech friends and family is generally homebrew is a app store for open source software, essentially. And if they want to get deeper on that, you can go down the line, the fact that it's mainly both the software installed by Homebrew and Homebrew itself are primarily operated through the terminal.

So yeah, basically that's, that's the starting point of why you care about homebrew.

Jonathan: Yeah. And it's, it's all, is it all open source? Like, so I, I'm pretty sure that homebrew itself, the entire thing is open source, but all of the software that you go out and grab now, how much of it do you build from source at install?

It almost seems like that's one of the things that you at least can do.

Mike: Yeah, so that's a differentiation between kind of two parts of homebrew, like the original and some stuff that's come later. So originally homebrew was, everything was built from source on the user's machine. So I guess you would call it like a build from source package manager.

Yeah,

and over time homebrew decided that we were going to take more of that open source software that was built from source and build it ourselves. And then now we build everything ourselves. So most users most of the time are going to be given a pre compiled binary package, what we like to call a bottle because everything in homebrew has like a beer metaphor running through it.

Right. But then over time there was a project that it started off as its own separate project, but it's been brought into homebrew proper now called casks. So we have things called formula, which are used to install things from source, which are open source software. And then we have casks, which are used to install Software that is a binary that we get from somewhere else.

So a classic example of a formula might be something like W get or some other command line tool you're used to interacting with or my SQL or database or whatever. And then a classic example of a cost might be something like. One password or Google Chrome or zoom or something like that, where essentially the, the flow otherwise would be download from a browser, click, click, click, whatever.

Jonathan: How long, how long has homebrew been around? Like what, when was the the initiator of this idea? When did somebody first start writing code to make it happen?

Mike: So yeah, Homebrew turned 15 this year in May, if I remember quickly, it was created by a chap in London called Max Howell who was working for Last.

fm at the time. That may bring back memories for some of you that's, I'm sure they still exist, but you know, less widely used nowadays. They once were so yeah, he basically was exploring different package managers. And he could never find one that kind of quite, met his needs and I think he was nudged by someone in the pub one evening to go and well you know if you hate them all so much why don't you make your own one and he did and that's that's the future really and so yeah so that that was 2009 and then I got involved with homebrew Later on that year, like, I guess, September 2009 or so I, Max was a friend of a friend in London and I heard about it and I was also dabbling in package manager things and I sort of just started contributing and never really stopped, I guess.

So yeah, 15 years ago is a couple of,

John: I was just going to say a couple of months ago we had Max over and we did a live stream kind of going through the history of 15 years of homebrew as well.

Jonathan: Okay, cool. Is Max still, is Max still involved with the project?

Mike: No, he's doing his own things nowadays. Sure. He was involved for kind of, I guess, maybe five years at the beginning, and then he sort of handed it off to others.

Jonathan: Yeah, that, that's great. It's one of the, it's one of the problems we see with some open source projects. It's like even really popular ones where somebody starts it and it's like, well, I guess enough people like it that I'm stuck here for life. And so like genuinely good for him that he was able to get off the boat.

And the boat didn't crash.

John: Congratulations. Your open source project is successful. You have an unpaid job for the rest of your life.

Jonathan: Yes. Well, it's a, it's a huge problem. It really is. And you know, there's the, the, the classic XKCD comic. Like you have this, the whole, you know, massive software stack, all the building blocks, and you have this one little piece that holds everything else up.

And it's like, this is just maintained for nothing by, by one guy in Kansas. And the scary thing is there's multiple projects like that. Like you can talk about the network time protocol has been that way for the longest time. Even things people don't know about, but are super important, like the term info files.

And there's legitimately only like three people in the entire world that actually understand term info. I think I'm, I'm fully convinced of this, but you know, if, if those break, we're all in trouble. So. Let's, this seems like a good place to at least dabble in the concept. I, I've heard of something called workbrew, and it is apparently related to homebrew, and it is apparently also you guys.

What, and this is probably, if I understand the lay of the land correctly, this is a question for John. What is workbrew, and why, why are we trying to do it?

John: Yeah, so homebrew, is, you know, insanely popular, used on Mac OS. There's more than 30 million devices with Homebrew installed. And it's pretty much made to be a single player experience.

You sign on to your machine, you open up a terminal, you install some things, it's all managed kind of on your machine. And what we're doing with Workbrew is trying to make it so that It's more of a multiplayer experience. It's built for teams. It's built for companies so that you can have a kind of a shared set of developer environments across all your machines, a shared set of policies, a shared way to manage and deploy and install, get analytics and observability, know what's going on within your fleet and keep everything secure and compliant.

So what we're building now is really. focused on how teams are using Homebrew day to day and trying to solve their major problems. So I think you know, maybe Mike, you want to say a little bit more, but I'd say that's, that's kind of the starting point.

Jonathan: Yeah.

Mike: Yeah, no, I think that's

Jonathan: a good start.

Okay. Now is, is Workbrew open source as well?

John: Workbrew is not open source. We're built on top of Homebrew. I like to think about it as kind of complimentary. So at the foundation of work brew, we're using brew, the open source package manager. We don't have a private fork. We don't have our own custom version.

It's exactly the same as the open source project, but on top of it, we build a bunch of complimentary tools for deployment, analytics remote management, security and compliance those types of things. And our approach, I think this is actually probably the topic that's worth getting into in a podcast like this.

Sure. Is our approach to commercial software in an open source world. So as you know, homebrew is an MIT licensed open source project. We, BSD, BSD, apologies BSD licensed open source project. And we basically upstream all of our changes that are necessary. So anything that, that needs to be. Available inside of brew as the core kind of foundation we make available to everybody.

Then there's kind of a line and that line is pretty well defined and it's multiplayer. It's enterprise like kind of security and compliance features. It's managing remotely. It's all of those types of things that are on top of homebrew that are closed source. We don't make that available open source right now.

Jonathan: Yeah, and that makes sense. I think it's going to serve you well that you've, you've figured out that line and you have a a clear delineation and you make it clear upfront that like, this is the, this is the part that's always going to be open source. This is the way we're going to manage that. And then this is the part that we're going to build on top of it.

That's not. And at least I, as a potential user, I, I really appreciate that. And so do you, do you generally pull from the same like list of packages? So for one, I understand WorkBrew is going to be doing much the same thing. We're, we're installing extra packages on Mac. I think

John: something that's, that's useful to explain about Brew is kind of the two components.

One component is the CLI. So it's the actual package manager responsible for putting software onto your device. And then the second kind of component. Is the packages themselves, the library of, you know, as Mike was saying, formula and casks The homebrew project has two official kind of package repositories.

One's called core and one's called cask. In homebrew core, you'll see all of your built from source packages and in cask, you'll see all your binary distributions. A lot of that stuff may not be open source. Some of it may be open source to distribute as a binary from the upstream vendor, but ultimately you have those two components.

So when it comes to workbrew you're still using brew, the open source package manager, and you're still able to use the library of packages that are available for homebrew in core and in cask, but you can also create your own taps as they're called in homebrew speak that are repositories of your own internal packages distributed, you know, within your organization and you can kind of set rules and, and manage how that stuff is done at an organization level.

So that's kind of where the delineation is.

Jonathan: Yeah, that that makes a lot of sense. Yeah, David, you I'm sure you're chomping at the bit. We haven't let you got anything in yet.

David: So a couple of questions just from listening to you there. The first one, I noticed that you refer to brew and homebrew and workbrew.

So what are the, like is Brew core to both Homebrew and Work Brew, or is Brew and Homebrew synonymous? And just a shorthand. I'll

Mike: get this one. So essentially like we, we often use brew as a shorthand for either home brew or work brew because Brew is, that's the, the CLI. command that you type in to your terminal if you're Workbrew.

I guess to jump back a little bit because it might be interesting again, like the approach that we're taking and it might explain how things fit together. So essentially the flow for Workbrew is you run our installer. And our installer installs some workbrew stuff and it installs homebrew so homebrew is installed completely normally to the normal location but we just do some stuff where if you were to go and say that to the homebrew open source project they would be like why would you do that but because because we have a bunch of homebrew maintainers and People like me who have worked on homebrew for 15 years, like we can do some slightly more unconventional things with homebrew.

But on your system you still have a completely unpatched, normal, open source version of homebrew running on your system. But then we have essentially like a wrapper that we have on top of that. So when you run brew and you have workbrew installed in your system, you run workbrew, which then calls into homebrew.

But the essential behavior for end users is it looks exactly the same. So I guess John talked on this earlier, like me as a kind of long term lover of open source software, I kind of like this approach of kind of combining the two because it means that Both the open source software, Homebrew, gets better over time, but also we have the way, as John mentioned, like the realities of kind of making commercial software in an open source world where we can't give away everything for free or it wouldn't be a business, right?

So, yeah, I think you get the nice best of both worlds. And the other kind of fun, I guess, analogy I've used with this stuff is it's like My apologies to John, who's heard this particular analogy about 8, 000 times is it's like Lego, right? I don't know how much any of you are playing with Lego nowadays, like, but my, one of my kids is super into it.

And Lego now feels pretty different to what it used to be, where I remember like there was a lot more modification of models and stuff like that. Whereas now it's like a pretty hard line between like the super, customized, this Lego T Rex comes with a particular claw that is made only for this one piece and you can't use it for anything else really.

Or you just buy a bucket of a bucket of blocks and you assemble it and make it up yourself, right? Like essentially what we're doing with with workbrew is if, if you look through the pull requests that are open by me and other people in the last year or so, you can probably see the blocks of how you could go about building your own workbrew.

But I guess our value proposition is essentially like, well, we, we built it for you and we can support it for you and everything like that. So. You know, maybe you're better to buy it off us, but the open source project still has a lot more of these kind of like hooks and things that you can plug into that makes the open source project better and more useful for people as it goes along.

Jonathan: Yeah,

Mike: absolutely.

David: And the the next question that I had as I already established at the opening, I'm not great I don't have a lot of Mac experience other than some user and I've maybe I've used brew once or twice. But I do have Linux experience an open source experience. And so kind of relating brew to.

Package managers, I'm used to I have two questions and you can answer them. However you desire. The first one is, do you have package maintainers that are responsible for specific Applications within, you know, that homebrew would pull down. And then the second one is, you mentioned taps as something you could do inside your corporation as kind of like your own repository.

Are there. Publicly available taps that are like maintained by people not directly related to homebrew. I'm thinking of kind of like PPAs and Ubuntu.

Mike: Yep. So in terms of package maintainers, I guess that's an interesting thing with homebrew is that we don't have specific package maintainers. We have like somewhat officially unofficially People who might maintain individual packages, but the maintainers of homebrew are relatively few.

So we've got, I think, 30 something people right now and between them, they essentially maintain everything. But that doesn't mean that they do everything because homebrew was built with GitHub and collaboration in mind from the outside. I think Max on the, the video call that John mentioned earlier that we had to kind of talk about it, like one of his things from the outside.

I'm sure you won't mind me saying this if you didn't use this exact word was essentially to be lazy and be lazy. Okay, I don't want to maintain everything by myself, so how can I build this from the outset such that essentially the community maintains this and I don't? The other difference we have nowadays is we have very, very, very heavy amounts of automation to essentially detect changes and keep things up to date and things like that.

But I guess to specifically answer your question, though, we have a slightly different model where We don't have like hundreds of different people who each maintain one or two packages. We have a small number of people who maintain everything and like review community contributions to all of those things.

The next question about taps. Yeah, we have essentially the tap model is very similar to a PPA model or something like that where any arbitrary person on the internet can just decide to set up a a tap and then by default the easiest flow is having them on github but really you can put them anywhere where you can have a git repository.

I mean technically they're just a folder on a disk so any way that you can get a folder onto a disk and keep it relatively in sync you can make that a tap and they behave much in the same way regardless of whether they're being maintained by homebrew ourselves or whether they're being maintained by the community.

It's more just like we set a higher standard for like both. The licenses and stuff like that and styling and, you know, making sure our best practices are followed that the community don't have to do, and may decide they don't want to do, may decide they can't be bothered to do and that's basically how that kind of ecosystem fits in.

David: So, one follow up question. How many packages is everything?

Mike: Right now, I think it's about 20, 000. Oh wow. Between all of the formula and all of the clasks, so 20, 000 official ones. So if you go wider onto the, all of the third party taps and stuff as well there is probably considerably more than that but basically at least 20, 000 or so.

That's, that's,

David: that's astounding. Yes. Especially for a small team as you have maintaining it. And you've mentioned CAS several times, which I understand because you've explained that and you talked about formulas, but I've heard references to bottles, what are bottles and how do they relate?

Mike: So a bottle is basically like the original, as I mentioned way home we worked was it's was a build from source package manager.

So the formula, I guess it fits more obviously in the metaphor and the original stage in a formula. If you imagine a formula for a beer, it might say, you know, I want Whatever. I don't make beer like some sort of liquid and some sort of not liquid and mix them together and you get a, some sort of beer.

But so essentially a formula is a series of build instructions for like how you take an upstream source. So like say something like, w get the tar ball containing the source code for Wget, how you would go and run various instructions on your machine and produce a binary at the end. So what we do nowadays in Homebrew is that formula for most of the people most of the time is only essentially used for building a binary package on our servers.

So when that formula is modified on GitHub, then we will run through those instructions. We will build a new bottle, which is what we call the turbo that contains the binary package. And then we then upload that to, we use like GitHub packages to store our all our binary software basically. And then when a user types brew install on their machine, instead of doing through all those steps or essentially just downloads that binary package, that bottle, and it will pour the bottle extracting the tarball into the place on disk.

And then a user can have that up and running. And for, you know, for, for smaller things, the time difference is minimal for larger items. If you have a relatively normal slash fast internet connection, you can do that. you're talking about the difference between, you know, less than a minute versus, you know, even on high end hardware, like four or five, six hours to compile some of the really big meaty stuff.

So it can be a huge time saver for certain people and users. And it's also a lot less hour prone.

Jonathan: Somebody had a lot of fun coming up with that extended metaphor and figuring out all the ways that it fit. Well, I

Mike: came up with that. So if you want to blame anyone for that particular strange metaphor, that's me.

And then most of the original stuff was Max.

David: You can definitely tell that this was developed on a napkin in a pub.

Speaker 5: I think we're looking

David: around the room creating parts of the infrastructure.

Mike: Yeah, well, the first commit ever to homebrew is interesting. It's something I actually, I did a bit of at GitHub when I worked there as well, and I recommend it for people when they're doing projects is this idea of like readme driven development.

I don't know if you've heard of that before, but the idea being before you start a project, essentially start by writing the documentation of what you think. it should look like and how it should work. So thinking from the outside in. So that's how Homebrew originally worked. And if you look at the first commit to Homebrew, the first commit is actually a readme of how Homebrew works, despite the fact that there's no code at this point.

And yeah, and the last question in the readme is, was Homebrew conceived while under the influence of alcohol? And the answer is yes.

David: Both Unix and Linux were conceived under the influence of various substances.

It's true.

Jonathan: All right, I'm going to jump in and I want to ask John a couple of questions about Workbrew in particular, and Let's see. So I'm, I'm assuming that the way we could set this up is a business would, would say, look, these are the packages that we want to allow people to install on their machines that we're, we're in charge of, and then we have this whole other block of packages that we, we don't want to allow.

Like, that's the sort of tooling that, that you guys give with WorkBrew, right?

John: To some extent, I think like I would actually preface it by saying that for us, the most important thing is a developer experience. Brew is kind of a tool, a tool belt for software engineers. They go to brew and they get all the things they need to do their jobs.

The reality of the situation is that companies, especially in highly regulated industries, you know, we talked to a lot of banks, a lot of you know, FinTech type companies, governments. Insurance companies, healthcare. The rules of the road there are just, you have to do certain things a certain way. And so rather than think about it as a way to limit folks and what they can do, it's more a way to enable developers to continue to use the tools that they know and love at

Speaker 5: work.

John: So we think about it as without Workbrew, the people in those companies would be told, no, sorry, you can't use brew. It doesn't meet our security and compliance requirements. And so with Workbrew, what we do is we make it so that. The IT folks, the security folks are able to feel comfortable with the risk profile of giving end users, developers, the ability to go out and install 20, 000 different open source packages.

And in some cases, you know, especially the financials that I talked to, they have some requirements around things like data protection, data loss protection. So they don't want to allow anything that opens up tunnels in the network. So particular VPN packages, wire guard. You know, Ngrok, stuff like that, where You know, they're honestly like useful tools that are not necessarily nefarious, but because of the risk profile of those businesses, they just can't allow it.

And so that's really where the kind of security and compliance type of, you know, restrictions come in and work brew. And with every customer that we work with, we talked to them about, you know, the developer experience and how they can maintain a positive developer experience. And instead of, you know, You know, using hard and fast rules, setting policies that they can monitor and see if, if something is out of policy.

So every every user that we have, generally the way that they come on board is they do a complete audit of every package that's installed on every machine. Yeah, homebrew. This is a huge amount of information for them. They had no visibility previously as to, you know, let's say they have 500 engineers, 1000 engineers, what packages were being used, what their, you know, potential surface area for attacks was, whether it's supply chain attacks or, you know, other kinds of data loss or things like that.

So they can do an audit and they can see everything that's happening. And then once they have that information, you know, They can take steps to provide alternatives rather than just say, Hey, this is turned off, you have no access. We give them a way to say, Hey, the preferred tooling for this job is X.

And we really are excited about the features around kind of collaborative developer environments for software engineers. So the idea is that rather than each person in kind of a single player mode managing their entire kind of development stack themselves, if one person finds a problem, whether it's a security issue, a productivity thing where they can move faster, When they make that change, they're able to share it with their entire team, you know, seamlessly.

So that's, that's some of the ways that we think about, you know, The same thing that you were saying, but really from a first principles, you know, mindset, why are we doing this?

Jonathan: Sure, sure. Does either work brew or and or home brew provide like automatic updates? Or do you have to go in and say, all right, all the packages I've installed, go check for new updates and install them.

John: I mean, the short answer to that is kind of it depends what you want. This is another one of kind of the principles that we're trying to follow, which is one size doesn't fit all. We have, you know, these fast moving tech companies that really want to embrace the cutting edge and they always want to have the latest version of everything all the time.

And on the other hand, we have, again, these highly regulated industries where they say, I know that a new release came out, but we cannot deploy that new release into production until a human has reviewed it. Yeah. They've created a signature, we've entered an entry in our audit log, and we've allowed it to enter the production environment, right?

And so these are two polar opposites. So, our principle here is like, one size does not fit all. We want to give people the ability to kind of choose how this should work. And Mike can talk about, you know, Homebrew's auto updating. Facilities.

Jonathan: Yeah. I'm curious about that too. Mike, I'll let you take it for a minute.

And, and what, what's the solution there with, with homebrew itself?

Mike: Yeah. So homebrew by default will basically. Almost every time you run a significant command, it will auto update itself and it will try to ensure that it always results in a consistent state, which involves, it will do things like auto update, auto upgrade, like reverse dependencies and all this type of stuff.

So I, I guess in general, like Homebrew is a developer tool, like my, thinking is if you ever are having to have in tools a thing where you say, Oh, if you run this command, make sure you run this command afterwards, then that's generally not very good UI, right? Right. So my, my goal is over time that Homebrew essentially, you can just get away with installing software and everything else that you might need to do running updates, doing cleanups, upgrading things, getting rid of things you don't need anymore, auto removing things you don't need anymore, like all of that stuff is done.

fairly seamlessly and automatically. And I guess another note related to a couple of things that came up before it, I guess we haven't mentioned explicitly in the school before, but I would be Sad if I were not to mention the fact that Homebrew also runs on Linux, which is, it may seem slightly strange as to why one would ever want to do such a thing.

But one of the cases we've seen on Linux is I guess what John mentioned earlier, where one of the things with Homebrew, because it's a rolling release package manager, which essentially means when Homebrew can get the latest update of a thing and it works and it doesn't break Homebrew stuff, then Homebrew will upgrade to a newer version.

We've seen people using You know, a more stable like base layer OS, Debian, Ubuntu, whatever, and then they might install. homebrew somewhere on their system on Linux and then they can get access to kind of maybe the bleeding edge for certain developer tools. Like if you have, I don't know, like a classic, like a CLI I love is RIP grep.

It's a RG is the command and it basically just does really, really, really fast recursive grepping through folders and stuff like that. It's what VS code uses for file finding text and files. So a tool like that, for me, it's like, I, If I want a Linux machine, I get frustrated if I have to wait, you know, like months for someone to get me the newest shiny version of RIPGrep.

And also the blast radius, if it, you know, If it doesn't work, then it doesn't really impact anything else in my system. Right. So then you, you can get this nice combination. And in a funny way, that's sort of how Homebrew works on macOS as well. In the, the, the reason why I'm drawn to macOS is because when I was using Linux I'm too much of a tinkerer and I would just continually break my system by being like, Oh, if I, if I increase, I'm showing my age a little bit here.

If I change X. org settings to do this, then maybe The refresh rate will look slightly nicer. Oh no, I'm stuck in a terminal for 24 hours, all this type of stuff. Whereas the, for me in the macOS world, like essentially apple of like nailed down that that trunk of my bonnet, as we'd say over here to my car.

So I can't get inside there. And that's for people like me, that's better for other people. That's worse, but you can sort of have a more Mac like. feeling with your package manager if you run homebrew Linux in that way.

Jonathan: Yeah. So I was going to, I was going to ask about, I have a little note here on my notes to ask about running it on Linux.

So this, there's a question that obviously follows up. Can we run workbrew on Linux? Does that make any sense?

John: Yes, but I would say not yet. This is something that we've talked about. I've had definite, definite interest from folks we've been talking to just have consistency across all platforms, the ability to see what's happening everywhere and be able to remotely manage them.

We don't have folks using Workbrew on Linux today, but I expect we will in the near future.

Mike: Okay. And now we have internal prototypes, basically, right? We have not yet released for what, but I mean, essentially homebrew works similarly enough on both cases and our internal prototypes are, it's the type of thing where it's the classic engineer thing of if I was earlier in my career, I would say, Oh yeah.

We could have a Linux version out in a couple of days because it all compiles, right? Like that's, that's the easy bit. It compiles, what could go wrong? Exactly. I, I've learned enough over the years that I'm like, well, you know, there's, there's probably more, the iceberg here is probably. Perhaps a little deeper than I might want it to be so

Jonathan: yeah in the past 24 hours.

I've pushed code that compiles I was that developer today within the past 24 hours Now work workbrew is available though like not not the Linux side of it yet But on on Mac if somebody really if like if this is the tool that meets their needs work workbrew is out there They can get it

John: Today, the way to get Workbrew is to come to our website, workbrew.

com get on a call with me and I'll show you around. We expect very soon probably in a matter of weeks, to have it publicly available for folks to just get on and try. But right now it involves kind of a conversation with us to see if it's a good fit for you. And really the reason for that is that we're We're looking for design partners.

We're looking for companies. We're looking for for teams that See the same set of problems that we see With using brew in a team environment and who really want to give feedback and help us build this tool together but we have it in production with a number of different companies. We use it ourselves every single day So it is there and it's up and running but it's not entirely self service as of this moment,

Jonathan: right?

What's the what's the uptake look like? Have you had quite a few companies that are a good fit that have come on? You

John: We had a really interesting situation maybe a couple of weeks ago. Mike was interviewed on the next web and there was an article that led to quite a lot of interest. Over the last few weeks, I've basically been in nonstop conversations with lots and lots of name brand companies that you've probably heard of, but I can't talk about.

But again, many of them are in these like kind of regulated spaces where it's, we're a finance company, we're a healthcare company, we're insurance, we're government you know, some kinds of, you know, different. Use cases for exactly the same thing. It's all different industries, but they're all saying, we'd love to use this, but our requirements are such that the open source thing just doesn't, won't fly here.

And really there's kind of like three stories that we see at companies. The first story is do nothing. They just let brew kind of be the wild west. Every developer who wants to use it just installs it on their machine and IT and security look the other way. The second kind of big category of folks is, you know, this informed trust model where You start at the company, there's a readme that says part of setting up your dev environment is installing brew, installing these 27 packages, here's a guide, probably the guide's out of date and it doesn't work when you first start, and then if something's wrong, there's that one expert who's done a lot of stuff with brew that you go and ask for help and that's really, really common, and then kind of the least common, but pretty popular case, especially among very large companies, is that they have some kind of internal tooling built around this already.

So they'll build custom scripts to integrate with their MDM tools so they can manage a fleet wide deployment of brew. They'll build scripts to do things like inventorying what, what packages are installed on their devices, reporting version numbers, and cross referencing that with vulnerabilities databases.

They'll do things like add self service scripts to their MDM tools so that End users who don't have admin privileges on their devices are able to install brew packages. But the downside to that is that for every single package in brew that your company uses, you have to have a staff member who's like maintaining that in your MDM tool, keeping it up to date, keeping it in sync.

It's just like unbelievable the amount of hurdles that people go through to make this work. And so yeah, the interest has been phenomenal. People are very interested and kind of those three categories of folks, they all hear from us and they say, wow. Why, why has this taken so long? Why hasn't this been here before?

Like, it's so obvious. So yeah.

Mike: Building as fast as I can, John.

Jonathan: And I assume part of the, part of the sell is, rather than you have to have this one guy at your business that understands brew, you've got the actual brew guys that understand brew. And you pay us enough money, you can call us and we'll fix things for you.

John: Yeah, that's definitely part of it. I mean, Mike is very generous with his time with our customers in terms of getting them up to speed on what needs to happen with brew.

So, yeah.

Jonathan: I mean, that's sort of a win win for everybody though. Right? Like I know how to do this very specific thing. You need somebody to do this very specific thing. You give me money and I do the thing like that's.

John: Well, the really beautiful part of this whole thing is it's not just one company that needs this.

Jonathan: Right.

John: They all, they all need the same thing and doing it once and making it, you know, reusable and, and, and packaged in a way that every single company that has this problem can adopt a standardized tool. We're basically saving an entire team's worth of effort at every one of these companies, right? And so, the individual kind of ROI calculation that each of these companies has to do is incredibly obvious that it's the right decision for them.

You know, rather than paying a team, you know, I could, I could list off maybe five or six of these companies that everybody knows and uses their products every single day, and they all have teams of people that are managing this problem. And in every single one of those cases, They would get a better solution from us.

It'd be more purpose built. It'd be more highly integrated with Homebrew. It'd be less maintenance costs for them. Less kind of distraction for their teams to have to manage it. It's just a very, very easy decision for them to make the buy versus build decision because, you know, the lack of expertise, the ongoing maintenance costs, what happens when a new version of Brew is released, what happens when a new version of macOS is released.

You have to, you know, You know, if something doesn't work, when one of those events happens, every single engineer in your company is stuck.

Speaker 5: You

John: know, that's a high cost.

Jonathan: Yeah.

John: So, you know, we make sure that never happens.

Jonathan: Just, just make sure that you, you, you don't pull a CrowdStrike and

John: Oh, absolutely.

Mike: I would be remiss if I said that, that, that hasn't been a thing that has been talked about recently when we've been talking about, deployment strategies and things like that of, yeah, there's there's ways to do this and ways not to do this and let's not do it the way that maybe doesn't always go so well.

Yes.

Jonathan: Yes. Have you gotten to the point to where particularly based out of these conversations with businesses, you've started adding things that you hadn't thought of to either work brew or home brew?

John: Oh, absolutely. That's kind of the biggest thing that we're doing right now is just listening to these customers about the problems they face and trying to build the most general solution.

I mean, Mike likes to say this thing about the baby. Do you want to give your kind of baby analogy about how this works?

Mike: So my, it's funny, at Homebrew, I'm probably the closest thing Homebrew's had to like a product manager, at least since Max left. Yeah, I guess. Working at GitHub for 10 years, I saw some product management done very, very well, and occasionally some product management done less than perfectly.

And something I feel like I learned from this stuff over the years, particularly when your audience is developers, is the line I like to use is users are, or customers, or developers, or whatever you want to pick. Or like babies in that they can cry and tell you that there is a problem, but they cannot tell you what the solution is to the problem.

The tricky thing is, developers compared to actual babies even if those babies may one day grow up to be developers developers tend to tell you, go jump straight from, I have problem to, and if you tasked me with how I would build this solution, how would I build the solution? And they come to you saying, what I need is, the solution that I would have been tasked with building.

And often they don't necessarily have the context that you do. They don't maybe think about like, what are the average users like? So a classic thing that this would come up in Homebrew quite often and the nice thing in Homebrew is you can just decide to do these things because it's not as higher stakes as certainly GitHub was and Workbrew is increasingly becoming is.

On homebrew, someone might say, Hey, I've, I've added a new option that I want to opt into for this particular behavior. And then I read it and I think about it for sometimes not very long. And I'm like, why would you ever want that option turned off? It's essentially like, I, I've added a flag. So when you run this homebrew command if I have my face puncher plugged into my USB port, it doesn't punch me in the face.

So I sat homebrew underscore, no punch me in the face, please. Right, and I'm like, well, maybe punching you in the face could be opt in. You know, like, let's flip the logic around. Or maybe let's just make homebrew punch free software. Like, maybe we can skip this flag altogether. So, to me, like, that's the type of stuff that comes up.

And again, I don't blame any, you know, Person who makes a PR like that, because they're, they're trying to be a good dev and be like, okay, I need to, I don't, I'm not sure that anyone except me cares about this. I want to narrow the impact radius here, but I can look and be like, well, actually everyone cares about this.

Jonathan: Yeah, it's, I don't want to break anybody's workflow. Right. I don't want to. Stop overheating when you hold the space bar down for everybody. That might be important to somebody's workflow. And, and that I did the, the, the baby analogy. I like that, David, I'm sure you get this too. Like a customer will come to us and say, I need this.

And then they'll, they'll tell you this most off the wall thing. Like, yeah, I need a new server. That's got a GPU in it. You need a one now. And then you, okay. What's the problem that you're having? Well, we'd like to be able to use. We'd like to be able to do this. And it's like, Oh, we need to get you an account at, you know, chat GPT.

We don't need to build a server. It's got a GPU in it. I'm sure you get that too, David.

David: Reverse it. You kind of have to, you're given instead of. The baby crying, they come to you with a solution and then you have to reverse engineer the solution to figure out what the original cry was. So you can give them a better solution to

John: your original question, though, about like, what have we built?

That's come out of, you know, customer conversations. We've had a number of, you know, customers come to us and are kind of very You know, common customer profile is ahead of I. T. Somebody who's responsible for managing you know, their MDM at the company, like the jam for Kanji or one of those type of tools.

And they'll come to us and say, Hey, one of my jobs is software patching, and I need a way to know when something has a vulnerability. And how I can really quickly fix that vulnerability and know that it's been fixed across my entire fleet of thousands of devices. And so we built a vulnerabilities dashboard that effectively takes a look at every single package installed across your entire fleet, catalogs all the version numbers, knowing exactly which version is installed on each device, and cross references that with a bunch of different data sources where we have information about all known vulnerabilities in those packages.

And we present this as a nice simple report for this IT manager to say, Hey, package X has a vulnerability at this version and it impacts 205 devices at your company. Click this button to apply a patch that fixes that on every single device. And after about 15 minutes, you can see 95 percent compliance, all of those have been patched, and a few of the devices that were turned off, you know which devices they were, so you can send a Slack message or give somebody a call to make sure they boot up their device and upgrade that package so that the vulnerability is addressed.

And that came directly out of, you know, customer. Customer requests.

Jonathan: Yeah, so you guys kind of provide a software bill of materials for across the whole organization Yeah, yeah, absolutely super interesting Okay. Now with with brew itself. We've talked a lot about command line applications Do we do do we do GUI applications?

Can we install, you know more complicated or less complicated but good GUI based applications Can we do real crazy things? Can we install entire bright light? Can we install Firefox with brew? Is that a thing that works?

Mike: Yep, and not only is it a thing that works so that this is essentially the way I Interact with homebrew primarily nowadays.

So one of the things we've built on top of homebrew was this thing called brew bundle Which essentially the the bundle part of the name relates to if homebrew is written in Ruby and there's a tool called bundle or bundler I guess is the full name that consumes gem files which are a list of ruby gems essentially like third party ruby modules and then builds them all together so you kind of have all these in your app.

So homebrew there's a part of homebrew called brew bundle which does the same thing with brew files where you can basically have a bunch of software In there. And you can specify, okay, I want these formulas. I want these casks. I want these things from the app store, the Mac app store. I want these VS code extensions.

And, you know, probably more to come in future on basically like the way that allows you to use homebrew is instead of. Saying, okay, install this, install that, uninstall this, uninstall that. Instead, you can have a list of essentially, here's everything I want installed on my machine. And, anything that's missing, I want you to install it.

Anything that's out of date, I want you to upgrade it. Any, you know, background services that I specify, I want you to run them. And you can also tell it to do a cleanup, which means any software that is not on that list, I want you to uninstall it from my machine. Which is probably mostly useful to people like me, who accumulate huge amounts of cruft testing various homebrew packages.

But the nice thing with that brewfile is then, you can, Like my most popular not homebrew open source project is a thing called strap I built for primarily originally for github's internal use, but it can be used by anyone And basically what that lets you do is say, okay when I first set up my machine the first thing I want to do is pull down this list from a github repo and I So as long as you're kind of relatively diligent about like dumping software to that list and then committing that to that repo, then you can get essentially a nice single file description of here's everything on my machine.

And if you're one of those people who likes dot files, Like having all your configuration files, like there. That's the repo that I put that in. And again, that tool pulls down your files repo. And essentially my goal with that repo is I should get my machine 90 95 percent set up by just the contents of that repo being pulled down and all these scripts run and stuff like that.

And nowadays I even have all sorts of mad things that Extract secrets from one password and write them to the right locations on disk and and all this type of good stuff And but yeah, but as you can tell this workflow is very heavily github centric right now. So if you are interested in a non GitHub centric version, then again, stay tuned to what brew is up to in the future.

Jonathan: Yeah, interesting. Now does brew, does brew help with doing like program installs without using brew? So like on, on Mac, the the, the, the normal way to install software is like you, you get your package, you double click it and it gives you this nice window with here's the application and here's your applications and you just click and drag can, can we do something real fancy, like build a package in brew and then Spit out that zip that then someone can install without using brew.

Is it, is that in scope?

Mike: Exactly what casks are basically most casks. That's what they do. So if you, the, the Google Chrome cask, effectively what that does is downloads the Google Chrome. So in comparison to maybe a Linux package manager, a Google Chrome cask, that's not some special version of Google Chrome that's made by the homebrew team that just downloads the installer from the homebrew team.

Apples, sorry, apple, apple, not making a crew installs the doubt from Google's website. And effectively that kind of drag and drop or click through and install or next, next, next, accept, license, etc. Like all of those steps are essentially automated inside the cask. So instead I can type brew install google chrome and then at the end I get the exact same result as if I'd gone and walked through those steps manually of the google chrome installer.

And the nice thing about that is it essentially provides a higher level API on So there's about 10, 000 casks, as I mentioned before. That's essentially 10, 000 pieces of software where the API for how to install it is, well, do I drag this thing? Do I click the thing? Do I download it from here? Do I like have to run a terminal command?

No, it's you run the same terminal command for essentially any of those. piece of software and they are all installed the same way. And to me, that's the most powerful thing about both casks being in homebrew, but homebrew itself is essentially you have this high level API for this stuff. And when I was talking about my brew file before my brew file has a bunch of casks in it.

So I'm not just installing my developer command line tools that way. I'm installing slack that way. I'm installing zoom that way. I'm installing like all my like Safari extensions that way. So essentially pretty much all the software on my machine. It's being, in fact, probably literally all the software on my machine that is not provided by Apple is being installed through Humbroo in some way.

John: Yeah. And on the software that's provided by Apple via the Mac App Store. Brew bundle also has an integration with a tool called MAS, which is the Mac app store command line. And so you can actually add to your brew file just like you would add brew in the name of a formula or cask in the name of the cask.

You can add MAS and the identifier of an app in the Apple app store, and it will automate the process of opening up the Apple app store and requesting to install the brew bundle. the program from Apple onto your device as well. So literally everything can be put into this brew file and automated.

Mike: Yeah, that's great.

If there's show notes or people watching along, in some ways it's easier to kind of see rather than do it. So if you Google for Mike McQuaid which is My name, you may struggle with the spelling of that, but I'm sure you'll get there in the end. And then brew file B E R B R E W file. Then you will get the top hit to like my brew file in my public dot files repo.

And then you can see like what that looks like. And please don't critique my particular. Choices of software because don't don't that very near and dear to me. Do it out of me.

Jonathan: Exactly. Now can we, can we go the opposite direction? So let's say, and I've, I've, I have this question because I've had to work through this before.

It's been years ago, but for a while I was developing an applique, a cross platform GUI based application. And one of the things that was a challenge was building that drag and drop installer. So, you know, if you didn't want to tell, so you could tell people, okay, go install brew and then install my application.

But if you don't want to do that, you want to give people that zipped up installer where you just, you drag and drop it and that's it. And it seems to me that there could be a opportunity here to To have a script inside of brew where you run the build and it gives you the installer that then you can go out and Give to people and has anybody done that?

Is that something that that brew can do? I know that's kind of a weird That's it's a weird idea, but I it would have been useful to me back then

Mike: I've seen people do such things in the past. I think what makes it tricky is, well, so I guess the two sides of homebrew, right? So the, the casks, for example, that essentially already exists.

If you have a cask for, as I mentioned, Google Chrome, that's means that that's already been provided by the upstream software, but then with homebrews formulas for, you know, say you wanted to install some sort of open software. Open source software that way. The tricky thing is because Homebrew has its own little special snowflake ecosystem where everything works just such how it does.

Essentially, In Homebrew, everything wants to pull everything else from Homebrew and be updated by Homebrew and be in the location of Homebrew. So that doesn't make it impossible. I mean it, you know, technically it would be possible to do such things, but it makes it trickier. My best actual experience in the past, to give a shout out to another open source project, is this a cross platform build tool called CMake, you may well have heard of.

What's less known is it comes with a thing called CPAK, which is C P A C K. So that basically lets you do cross platform packaging like this. At previous jobs I've used it for essentially generating, you know, like when I used to work on Qt applications for generating a nice click, clicky clicky windows installer or a RPM or deb for you know, Debian or Red Hat based distributions or a Mac OS kind of drag and drop style or like traditional clicky clicky installer.

Like you can essentially spit all of those out from the same project. And that also provides some third party tooling, some of which I may have contributed to myself, that aids in doing things like pulling all the libraries from Homebrew and putting them in the right place and all this type of stuff.

So it's kind of out of scope of the Homebrew project itself, but like, you can. You know, if you, if you look a little bit funny at the problem and take tools like CPAC, then you can definitely rely on homebrew a little bit to solve this problem

Jonathan: a little bit easier. Yeah, sure. Okay. So is there, and it seems like there used to be is it Mac ports is also sort of in the same space that there are some other projects that sort of solve some of these same problems, right?

Yeah.

Mike: Yep. So Mac ports is one think is another. And I guess, yeah, nowadays people are using Nix in some cases as well.

Jonathan: Oh yeah. Is there any cross pollination? Like did some of the same, maybe same people doing the, the, the, the package management or yeah,

Mike: I don't know. So I wouldn't say there might be.

We definitely. We talk to each other. So, and sometimes not, well, I mean, when I say talk, I mean, I don't mean it in the silly way that, you know, normal humans would actually talk to each other like we're doing right now. We, we, we type things at each other on the internet and there is collaboration in some ways between the projects.

Some of it is explicit, like where we might go and we have kind of shared channels with some of these folks. where we might kind of figure out problems. And then some of it is the, in the nicest open source spirit of the world, us, like, various projects stealing patches off each other where, you know, maybe there's some new macOS version or compiler version or whatever it may be, And someone, one of the package managers writes a patch and the other package managers need the same patch so then they can take it from each other.

So we've all, again, there's a nice collaboration where we've all done that from each other. And also sometimes for inspiration where if a particular package is not working on homebrew as well as it should, or we get a complaint and someone says, Hey, this works fine on Mac ports or Nix or whatever, we might go and look and see how they do it.

And again, I'm sure. The same thing is happening in reverse and same with the with the There's probably just as much if not more actually with the Linux package managers as well and the BSDs as well because the BSDs use Clang as their compiler, which is the same as what we use So yeah, so there's basically, it's nice.

It's kind of classic open source Collaboration and action really in that like all these projects because we're all open source We can all share information and resources and help each other out with stuff

Jonathan: Yeah, I have anybody interested in workbrew asked about any of those other other tools. You know, I could, I could imagine someone say workbrew sounds great, but we want to be able to use Nix as the back end.

John: Haven't, haven't heard that one yet. But we have had some folks that are, you know, coming with like cross platform requests. So they say, for example, we want to have consistency across all three major platforms, Windows, Mac, Windows, Mac, and Linux. And it's been particularly interesting with regards to non developers.

So there are definitely companies where they have kind of a wide range of employees and some of those employees are using brew today and they understand it and they rely on it, but they also don't want to have a different system for, for example, our customer service team or for their data engineers or for data science or, you know, folks who depend on code and packages, but may not be as comfortable using the terminal and installing things.

And so we've talked to them about. How can we provide a singular way to provide a developer environment across all three of those platforms? So that's been probably the most similar kind of questioning that we've had from people. But not directly to say, Hey, I have like five different kinds of package managers.

How do I use them all together? More it's like, we use this one thing. It's really cool. How can we make it work for everyone?

David: So I assume on Windows, since Brew works on Linux, you could run Brew under WSL today. But it's the Windows specific stuff that probably needs some

John: Yeah, you can run, you can run Brew under WSL today.

That's how most of the people that I talk to who have interest in it are doing it. So I've talked to potential customers who say you know, we have this 10, 20 people over in this department who are using WSL on windows because X reason, can we get those mapped into work brew as well? I haven't had the same question about, Hey, can you run this natively on windows?

That almost never has come up. But there's a, there are other Rather than saying like, hey, we have this MDM tool, because really the people that we talk to a lot are the ones who are, you know, the IT managers who are responsible for getting the fleet out into the hands of the, of the company's employees and making sure they have the tools they need to do their job.

And so what they're saying is, hey, my MDM tool, whether it's Microsoft Intune, Jamf, Kanji, whatever, has a couple dozen applications in its built in package manager. In Kanji they call them auto apps, in Jamf I think they call them catalogs, but the idea is that, you know, Google Chrome, Zoom, Slack, all of those come as like default packages that the MDM provider keeps up to date.

But they say, well, we have this, you know, this couple dozen packages that are not managed by them, we have to manage ourselves. But Brew manages them, we can use Brew instead. And so they want to standardize on a, on one, one tool for all of this. That

David: makes a lot of sense. So another question that I had was just about the interface.

So you talked to, you mentioned the cross system management and like you talked about how you could, See your systems updating with work brew and identify the ones is that a web portal? Is that something built into the work brew command line?

John: Yeah, the overall architecture of what we ship is Three pieces on top of brew.

So it's brew the open source project Plus we ship an installer PKG file Which is a Mac PKG that basically enables zero touch deployment of brew it makes it so that if you have a brand new Mac You business manager, the first time that you turn it on brew is installed and everything is secured. If you have existing devices with brew installed and maybe dozens or hundreds of packages installed on those devices, you can run the PKG file locally or you can run it via your MDM tool and brew on that device will effectively be upgraded to work brew so that it will have that kind of connection back to the system on the kind of Administrative interface, there's what we call the console, which is a web portal that gives you a high level overview of every device in your system or in your fleet, and it gives you a high level overview of every package.

So you could say, for example, which devices are open is open SSL installed on. What versions are installed? Are there any known CVEs against it? And then run a patch to say, upgrade it to the latest version because there was a known vulnerability we want patched. So that's kind of how the interaction works.

On the device, there's one more thing that like, kind of goes to what I was mentioning earlier. One size does not fit all. So, on the device, we give companies the opportunity to choose different permission models. We call them Restricted, Managed, and Guided. So Restricted is the most controlled. This is kind of useful for individuals who don't know what the command line is, may not have any interest in brew, may not even know what brew is, and you want to manage all their installed software in the same way as every other device.

So you can install WorkPro on their device and essentially provide it in a restricted manner where the end user has no access to the Bruce CLI. It's only managed remotely. So again, great for like a customer service team or maybe data analysts, people who might not be comfortable with the CLI. Then there's managed mode, which is kind of the, Big most popular thing that we're seeing among companies with developers where we install brew on their device for them via the PKG via their MDM tool and give the end user access to brew via the brew secure CLI, the wrapper that we have around brew for the end user.

It behaves. It's just like normal brew so you can do brew install, brew uninstall, upgrade, tap, you know, pin, whatever they want to do, any normal command, and those commands are parsed and kind of made subject to policies or configuration options in brew so that, you know, we keep things secure and compliant.

In those cases, what's really interesting is the end user on the device doesn't necessarily have to have admin privileges. So one of the big kind of points that we hear from, especially the regulated companies, is we can't let people use brew because in order to use brew successfully, they need to be admins on their devices.

And for legal reasons, we're not allowed them, allowed to have them have admin rights on their devices. We just cannot do it. So we basically enable them to give their users the same brew experience that they know and love. without having to offer admin privileges on the device. And then the third kind of mode is more progressive companies that want to give their engineers full access.

It's essentially the same kind of guardrails around policies and how you can use Brew, but if they have admin privileges on the device, they're able to effectively escalate the privileges and go around these guardrails and say, even though it's out of policy, I'm going to do it anyway, so I'm not blocked.

And then the kind of management or the IT and security teams will get an alert to show them that. This device is out of policy or this package is installed in my fleet out of policy and then get up to date that way.

Jonathan: Yeah, so it sounds like you're not planning to bring brew itself to windows.

PowerShell. I was going to ask about that.

John: I think Mike is best suited to answer that. Never say never.

David: It just sounds like pain and suffering though. A follow up question on the whole dependency management. So, I have DevOps experience where you're, you know, developing webpages or web applications.

And so, you need to keep your dev system in sync with the same versions that you're running on your production systems. Do you have anybody using Brew to manage the software stack on production systems? Ah.

Mike: Right now, not really. It's essentially Homebrew, as you mentioned there, like the, the limitation you, you generally have in production is you want to have everything locked down to very, very specific versions that are consistent across your entire fleet.

But That's not the model in which Chromebrew operates by default. That is increasingly the model we're seeing people wanting Workbrew to operate in default. So we are building stuff in that direction. Again, sorry, I can't say too much there, but essentially like that, well, I guess it goes back to the, the babies we were talking about earlier, right?

Like essentially this is, this is a crying baby problem that we are aware exists even in development modes, but certainly the desire to have everything consistent between CI, dev, production that, that is a problem that exists today. There, there is not a great solution to, and there is a problem that we are working on right now.

I guess the Homebrew middle ground there is because Homebrew is a rolling release package manager for cert, it used to be Homebrew was just, you know, You get the latest version or you, you don't get a version, right? Like you have to just pick between. Now, more popular packages with kind of better support for kind of running older versions, say something like MySQL or Postgres or Node.

js, Ruby, whatever it may be, that still provide bug fixes, security releases, etc. for versions beyond the most recent one. You can install a package and you might say, you know, brew install nodejs at 18 or whatever, right, which gets you a like less than the current newest version but will continue to get you patch releases and security updates and stuff like that.

I guess a sort of balance I've seen with this stuff as well is that historically, The enterprise, the maybe individual developer model is let's just have the newest version of everything all the time and the enterprise model has been, let's pick a set of versions that work and then essentially only ever upgrade them if we absolutely have to.

But the problem with that flow is you often end up in a situation where, whoops, we probably should have upgraded this version a while ago and now a bunch of bad actors have got access to our product. either development machines or servers or whatever it may be because we decided to sit on this version indefinitely even when there were security updates that we were too scared to install.

So to me there's a, there's a happy middle ground to be found. Homebrew leans slightly more towards the like, you know, let's have the newest thing of everything all the time. But as I say in WorkBrewland we're building, essentially we already have some tooling there to essentially get that middle ground of like, okay well this package is actually vulnerable right now so I don't care if you're trying to sit on an older version because.

Some perceived view of stability like let's upgrade everyone. So at least they're not in a vulnerable version anymore.

Jonathan: Excellent. All right. Well, we, we have hit the the top of the hour, so I want to get into rap. And one of the, one of the things I want to ask you guys is, is there anything we missed? Is there anything we should have asked you about that you wanted to let folks know about that we didn't cover?

It's kind of challenging question because you've got to do some set math and think about all the things that we talked about, but if there's anything that comes to mind,

John: I mean, I guess there's one thing

Mike: that I might mention, which is, yeah, I think I mentioned like Linux before I'm developing environments and stuff like something where.

We've actually seen quite a lot of use of homebrew that might be worth playing around. For folks who are listening here is homebrew runs pretty well in, if you're using GitHub actions or, you know, GitLab runners or whatever it may be. That's a nice way because the packages are The same on Linux and Mac, and because the versions are in sync, that can sometimes be a nice way to have your development environment and your CI environment in sync, where you might have your developers using Macs locally, say, but then they might have a bunch of tests running on CI boxes, which are most of the time, for cheapness reasons, going to be running on Linux.

So if you do, you know, use a brew file, like I mentioned before, or even just run brew install. go or whatever it might be, then that means you can have that consistency between what you're doing locally on your Mac and what you're running in your CI environment. Yeah. And that can be kind of quite nice.

Yeah, absolutely.

John: I was just gonna say that you know, if this, any of this sounds interesting to you, Mike and I are both available you know, to talk to people. You can find us on our website, workbrew. com. There's a contact form that goes directly to my inbox. I'd be happy to chat with anybody there.

We're also active on, you know, GitHub and Twitter and things like that. We'll give our contact information for that as well.

Jonathan: Yeah, and we'll make sure and get that in the show notes for folks if they want if they want to reach out probably a question mostly for mike Although john, if you have any stories, you certainly can tell us what's the what's the most surprising thing?

Somebody's done with brew. What what does somebody? Message you or written about and you know, I didn't know you could do that with brew Or why would somebody want to do that with brew? Well, I, I'm

Mike: actually going to choose to misinterpret your question because I think you'll find it's slightly more funny.

The version that, so the thing that jumped into my head was almost like, what's the, what's the. What's the stupidest thing you've seen happen with, with homebrew? Stupid can be surprising. Yeah. So my favorite bug of all time actually is we might be able to, if you've got show notes or whatever, like fire me an email or something, you know, I can send you the link because this is open source, you know, you can actually read.

So there's a bit of debugging that went on. Someone was getting some very strange messages when they were running homebrew. And this is in the earlier days of Mac Os. So nowadays Mac ships with a thing called system integrity protection, which essentially means like, look, I don't care if you run pseudo, there's certain stuff on your disc that's more important and we're not gonna let you just like screw around with it.

So stay away. But before they had that, someone was trying to run home Homebrew, getting very strange error messages. And what was figured out was that they had managed to replace bin bash, which is, you know, on a Unix system, a relatively important thing for you to have somehow with no JS. So every time they were attempting to run a batch script, they were getting JavaScript errors in their shell.

So that to this day is my favorite issue I've ever seen on

Jonathan: Homebrew. Oh, that's beautiful. That is beautiful chaos. I love it. All right. Yeah, John.

John: I was just gonna say, just one of the things that I was like surprised about is I can't mention names, but several very large enterprises that make that make a lot of the core technology that we depend on have their own internal forks of brew.

Oh, yeah. And so, you know, they, they're basically maintaining an entire Parallel infrastructure where they're like reviewing everything themselves you know, to get at these like kind of supply chain story, like keeping things up to date vulnerabilities. It's just like unbelievable that it's, you know, brew is that important to them that they can't, they can't decide, Oh no, we just won't use this.

No, we actually, the only option we have is to maintain this ourselves internally as a totally separate fork. So that just kind of goes to the, you know, the idea that it's a very essential tool for a lot of developers.

Jonathan: Yeah, yeah, that's great. All right, so I'm going to ask each of you a final two questions, and that is your favorite text editor and scripting language, and we'll start with John.

John: I mean, this is pretty easy for me. I'm a Ruby developer so Ruby is my favorite scripting language. I, you know, I used to work at GitHub for close to seven years. And I was so happy when I joined GitHub because it was finally a company that used Ruby as like their main language. I had always been like a web dev, you know, my early days I was doing like PHP and like Java and stuff like that.

And when I moved into writing Ruby, you know, at work, it was like the best thing ever. And then I kind of have, you know, A soft spot for Pico. Okay. As my editor, I first, when I started writing code that was the first editor that I learned how to use on a Unix machine. I was running like a Gen 2 box.

Yeah. And old habits die hard. It's like still the thing that I just opened by default in my terminal.

Jonathan: Pico and not Nano? Yeah.

John: Pico, not Nano.

Jonathan: That's great. I

John: have an alias usually.

Jonathan: Makes

Mike: sense.

Jonathan: All right.

Mike: Mike. Yeah, so scripting language, I'm actually gonna disagree with John. So I use Ruby for like proper programming nowadays.

But if I just need to quickly solve a problem, I always go to Bash. Like me and, me and Bash, Bash has treated me so badly so many times. But yeah, I keep coming back. Like I don't know what it is. And yeah, as for text editor, like nowadays I feel like I'm, I wouldn't say begrudgingly, but you know, it feels like a lot of developers are on VS code now, and it makes just life easier for me to just follow the flow and go there.

But yeah, I'm still, I'm still keeping my eye on other things like the extension ecosystem of VS code is great, but like, you know, it's not the fastest thing in the world. And there's a text editor by an ex GitHubber. Called Zedd that is kind of up and coming, written in Rust. It's super duper duper fast.

And I've been playing around with that ever so often. It doesn't have quite all the features I feel like I need, but like, yeah, I, I have my, my hope for that as a potential feature option.

Jonathan: Yeah, absolutely. We, we interviewed a young man back a few weeks ago, building the amber language that is intended to be bash code with some of those bash pain points removed.

That one might be interesting to look at. We had a lot of, we had a lot of fun talking. I kind of like

Mike: the pain points at this point of everything bash does wrong is just, you know, it's like, it's almost like scar tissue that. It feels like it's part of me, you know?

Jonathan: Yeah. I'm, I'm trying to figure out, it seems like we either interviewed or we're going to, ah, I'm in contact with the guys from Zed about hopefully getting getting them on the show.

So watch for that. too. Guys, thank you so much for being here. I appreciate it very much. And it was, it was really fun to get to learn about, about homebrew, which, you know, it's been around for forever. And then work brew, which is a really, really fascinating project. The business that's been built and boy, hopefully hopefully it'll continue to go well.

And maybe here in about a year, we can have you guys back on and talk about what's happened since.

John: Oh, we love it. Thank you so much for inviting us.

Jonathan: Yep. Good deal. All right.

David: What

Jonathan: do you

David: think, David? Ah, I love it. I, of course, as we started at the beginning, I'm not a big Mac guy, but I'm interested in playing around brew on Linux.

Jonathan: Yeah. I tell you what really fascinates me the most is for my use is a brew in GitHub actions like that. I, I, I never had that thought. That is very surprising to me, but you know, as they describe what you can do with it, it makes sense that you know, you're, you're. Your GitHub runner may be running something very different than what your develop machine is.

And having this external package manager that can put the exact same packages on all of them, like that's, that's kind of compelling. So that's definitely something that I will have to keep in mind for the future that could be interesting to do. I like that. I, I'm also just thinking about the conversation we had.

I am, I'm impressed with the way that they are building Workbrew on top of Homebrew, and I, I don't know if you would call that, I don't think you would call that an OpenCore project. I don't think that's fair to call it that. But just that they have this very clear line of demarcation between the two.

And they're pushing features into homebrew as needed to make workbrew work. I think it's a good, I think it's a good model. And I think it'll, I think it'll serve them well. So, you know, looking forward to. Hearing about their success as time goes by.

David: Absolutely.

Jonathan: Yeah. All right. David, is there anything that you want to plug

David: before we let folks go?

Not specifically, but I always like to take the opportunity to plug club twit and the twit network. They're going through changes that I think are positive in the long run. Yeah. And we have fun over there. So come on over.

Jonathan: Yes, we've got, we've got our show that David is from time to time, but co host on.

And that's the untitled Linux show where we talk about all kinds of fun Linux news and open source news. And there's some, as we say, there's some cross pollination between floss and and ULS. Coming up on floss weekly next week, we actually have Pedrag Brady from CoreUtils, and we talked about the Rust CoreUtils a few weeks back, and I got an, I got an email that said, Hey, you know, we're the OG CoreUtils, we'd love to talk to you too.

So we've got them coming on, and looking forward to that next week. We'll be back at our regular time next week on Tuesdays, we recorded a little early today. And then, yeah, we, Appreciate Hackaday being the sponsor of the show, giving us a place to land. And you can follow my work there. The security column goes live every Friday morning, and that's always a lot of fun too, to keep up with what's going on in the world of security.

But other than that, just want to say, thanks. We appreciate everybody being here, catching us live and on the download. And we will see you next week on Floss Weekly.

Episode 795 transcript

7 augusti 2024 | min

FLOSS-795

Jonathan: Hey, this week Doc Searles joins me and we talk with Olaf and Dave about LifeRay, a project that's a little difficult to pin down. It started with the idea of portals, but has grown into a basic building block for building any sort of experience on the web. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 795, recorded August 6th.

Life Ray, now we're thinking with portals.

Hey folks, it is time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host Jonathan Bennetts and we've got a fun java filled show today. Once again, it is not just me though. We've got Doc Searles, the the one and only the og. Hey doc. More O than G. A lot more O than G.

Yeah, well, I feel that way some days. Now, you've described your day today as a death march. You've got something going on that you're feverishly getting ready for.

Doc: Yeah, there's an annual event put on by the Internet Archive called D Web Camp, for Distributed Web Camp. And it's at Camp Navarro, it's up in Northern California, among the redwoods.

Where it's fun to see people coming from other countries where all they're doing is standing around looking straight up wondering why These how weird these trees are they're looking at a cloud up there They're looking for the cloud or they think the Ewoks are coming. There you go Because that, that, that particular episode of Star Wars, the Star Wars movies was filmed in the Redwoods.

Oh, okay. But this is, this is much less lush. This is more like dirt on the ground rather than, you know logs and, and, and moss. But, yeah. But it's fun. It's a, it's a fun camp. And, but I, I'm giving a talk. early Thursday morning there. And and I've got a whole thing I'm doing on it and it's, I'm not ready.

So there's that. It's also weird. Cause I mean, it's, it's been, I'm in the middle of Indiana and not much different where you are, where it's hot. It's the summer here. It's hot. And, and it doesn't get cold at night. It stays warm because there's humidity. And In that part of California, it might be 90 in the daytime or, you know, 30 something for those of you elsewhere.

I now give my age as 22. 5 Celsius, actually, so. I like that. Yeah. So that's about how old I am. Anyway. Yeah. But it's cold at night there, so I'm bringing my flannel, my flannel PJs. Yeah.

Jonathan: Yeah. Yeah. Alright, fun. Well, so we've got a we've got an interesting topic today. We're gonna be talking about LifeRay, and LifeRay from, from what I gather is sort of difficult to put a pin in exactly what it is.

It's a, a low code digital experience portal, which sounds very buzzword y, and I, I think it sounds that way just because they, they try to do so much. Have you gotten a few minutes to look over this, Doc?

Doc: And I, I've had a few minutes to look over the briefing . I'm not, I'm, I'm not hacky enough to have downloaded anything or do any of that for our guests.

I, I was, I worked for Lennox Journal for 24 years, but I was like the business editor. Mm-Hmm. , you know, so not the not a, not a technical guy. So,

Jonathan: and I think, I think this is gonna be a very sort of business oriented show, which is why I I particularly chose you out to be here. I think it'll be real fascinating.

Let's, let's go ahead and bring the guys on though, because obviously we can, we can sit here and talk about what we think it is, but we've got the experts right here. So we've got Dave Nebinger and Olaf Cocke. Welcome to both of you guys. Thank you. Thank you. All right, let's, let's start with this kind of overarching 30, 000 foot view question.

What, what is Life Ray? What do you do with it? What problem does it solve?

Olaf: If we tag team, we probably get most of it. Going historically started as a portal and in times when you could not put different. Individual pieces on a web page and it continues to develop into the many different Things like on the business side.

We call it digital experience platform because yeah, that's the rage That's that's more than a portal that combines content management system that combines Identity management all kinds of single sign on stuff that you might want to build your application with and And then it gets a lot of features.

So it's not a single tiny bit that does a little bit for you, but it's a full application. So we built in several techniques to extend the whole platform to integrate external software. And part of that is now being done through low code environments or no code environments. So you can easily build form based applications through that environment.

And so that, that makes it hard to really grasp. And we've had a couple of fun usages of that, where it actually doubled as one of the systems that you typically integrate with it. So let's say you typically integrate with a single sign on platform, but there is a plugin that can also mask Liferay as a single sign on platform.

But that's like a really niche thing. Dave, what have I forgotten?

Dave: Well, they had to hit more of the buzzwords. We're a CMS, we're a digital asset management platform, and obviously low code that Olaf already mentioned. It's really, like you said, it's hard to put a pin in what Liferay actually is because it can be so much from a simple Hosting platform akin to WordPress all the way up to a complex enterprise platform for building and hosting custom applications that an organization may need, but they don't want to waste all their time building common capabilities like security and authentication and permissions and styling and, and those kind of things that application developers typically need.

The platform kind of encapsulates all those details. So you bring your custom applications and Liferay takes care of all that supporting

Jonathan: stuff for you. Does it, does Liferay fit in sort of the same niche that something like a next cloud would, is it, is it sort of that same idea? I don't know if you'd say it's, if you're familiar with next cloud,

Olaf: Not too much, but I would say it's more than I haven't seen that comparison, so we're not competing with them on any market.

It's rather literally an application, which then integrates something, has one web front end. So think of it as a web server that brings in content from other applications. Like you have your E. R. P. In the background, you have your C. R. M. In the background. Recently, I've been been using the customer portal use case where you come in as a customer or as a citizen to your city's website and you want to tell them, Hey, there's a pothole on the street.

So what do you do? The city needs a ticket, but nobody, no citizen would like to file a ticket and then see all of that metadata mess with 25 25 fields. So in the citizen portal, you will have a front end that says, Hey, where's the pothole? Oh, it's here. How bad, how bad is it? Fire it away. So you get a separate front end to some external backend, which handles all of the ticketing and metadata and so on.

But you Simplify it for the actual front end user. And they don't need to leave that that website or that portal in order to do something else. Like I'll need a new passport. I'll need this or this or that you can do all on the same platform, even though in the backend, it's vastly different things.

Doc: Okay. So, you know, as, as you guys are explaining it, the, the three letter Acronym that jumped into my mind was an IDE. Is that term even used anymore? Does it describe what you were doing?

Dave: We're not really an IDE in that you're developing software for somewhere else. The Liferay platform at its core is serving up web applications JSP pages, all the way through React, Angular other JS frameworks.

It's a hosting platform. That provides a number of services to make it easier for those applications to be built and hosted. So the developers don't have to include those aspects in their own application.

Jonathan: Does Liferay work as the web host itself or does this sit on top of something like Apache?

Dave: We use Tomcat.

So we're leveraging Tomcat and other JEE based servers. We're also compatible with like JBoss and. WebLogic and some of the other big players.

Jonathan: Okay. So this, this would be in the Apache world. We would, we would build something like this, probably a PHP, but you guys are in the Java ecosystem. And so it's, it's all built in Java and then working with the existing Java tools.

Interesting.

Dave: It is, but we're not limited to Java through our extension platform. You can bring. extensions using any language that you're familiar with. If you want to build a NET extension, you can do that. If you want to do all of your development using React, Angular, Vue. js, that's fine. We support that.

And you can leverage the host of Liferay services in your application to implement what you need without having to reinvent the wheel each time.

Jonathan: So it, it really is then like a, it's like a CMS, but it's, it's a CMS that is very much targeted at building like a dashboard and a even a customer experience at the same time.

Olaf: Yeah, let's say the content that is managed by a CMS, the content can be very active, can be an application. And the content receives some services from the platform. One example is, like, why would you need a platform when you can build a perfectly fine React application yourself?

Jonathan: Right.

Olaf: Well let's say you need a user identity.

So, yeah, that's fine, you can sign in, you can sign out, that's fine. Fine. For, for you, as I, if you build on this platform, you get one kind of user identity. And if later on you decide, actually, I don't want to bother with any password anymore. I'll outsource that to some single sign on system. The user identity that the application interfaces with stays exactly the same.

When you say, oh, we want like we've started in a very small environment, but we want to change now to an LDAP system. Then the user identity stays exactly the same, but it's now fueled by LDAP. And if the company merges and decides to now bring in two LDAP or N LDAP systems then, yeah, who cares?

You still deal with one user identity that comes with permissions services and so on. And that user identity is, is what you use in your front end application, no matter if they are authenticated by a single sign on system, multi factor authentication what the password policy on those is if they come through LDAP and so on.

So there's the value for building on a platform.

Jonathan: Yeah. What, what did the origin of Liferay look like? Like, what was the what was the initial problem with that? And I'm, I'm curious, were, were one of you two at the beginning? Did, did either of you guys write, you know, the first lines of code? And then what, what did that look like?

Olaf: No, we're actually veterans. So but both of us are not there to write productive code on the platform. I'm with a company now for 14 years and a bit. Dave is a little bit shorter but he has been working with a product since forever. The actual product started around the year 2000 when Brian, the company founder needed a website for his church.

Oh, interesting. And he completely nerded out, over engineered everything, and you can imagine a church website actually needs to perform really well. And performing really well in the 2000s, or in 2000 actually meant, when you know Java, what does it mean? EJB. So it went all the way in and and completely over engineered.

He solved the problem. I'm, I'm actually not, I'm actually not sure that the the church website was ever served by life, right? I don't know, but, but that's the origins. And then like, People people found it and and asked for support or contributed something asked for for help setting it up.

And from then on it, it completely like it, it went hockey stick and, and up. So to say in 2004, there was a company that was founded or the company was founded.

Olaf: And it's now, actually I didn't count them, I'd say 20 offices worldwide, so lots of local companies, and what, what else is there?

Oh. And the formally purely open source version then turned into a dual licensed offering. Mm-Hmm. . So the open source and a commercial offering, which in my eyes, I, I was around, I was at exactly the event where it was announced and it totally made sense to me. Because up until then they had like 10 12 customers that were all on different branches.

Olaf: I think it was subversion back then. So everybody had their own branch. So bringing on yet another customer and another one and another one with all with their own branches does not really make sense. So you want them all on a single branch or on like a limited number of branches and you can have an unlimited number of customers.

And they want longer support. And they I want to have someone who is responsible who says, Yeah. Oh, yeah. We're responsible for what we do. So that that started the back then so called Enterprise Edition. And as I've started with EJB when Enterprise Edition came in, EJB was already long out.

So, so coming back to the code side there is no more EJB in there, don't worry. It's, it now is a modern platform, but it could still serve your church's website. Yeah.

Jonathan: Interesting.

Dave: If, if you remember the, the early two thousands, the primary interface that everyone was going for was really the screen with a primary content and then a number of boxes on the side, either on the right side or on the left side.

Mm-Hmm. . And this was the kind of interface you'd see if you went to stand n.com or Yahoo News or, or any of those kind of things. And out of that. There was a big push to adopt what were called portals. And there was a Java specification to define what a portal was. And when you look at Liferay Liferay had looked at what was available from a Java portal perspective, but there was really no good open source alternatives out there.

And our platform was built as an open source implementation of the specification for a portal container that would host portlets. And that's where it started from, but these days it is so much more that we don't even call it a portal anymore. It's, it's a DXP because of all the capabilities that it now has.

Doc: I'm wondering, okay, so A digital experience platform, I guess that's what that stands for. Is anybody else using that? Do you have any direct competitors? Do you have a category called DXP and there's you guys and there's that one and that one and that one? Is that, where's that at?

Dave: Adobe is probably the biggest one that folks will lean to when they're learning about DSP platforms.

But, you know, us compared to Adobe, we're open source, they're not. That's great. We're an integrated platform because all of our capabilities we built and add to the platform natively. So the whole usage across the platform is consistent. Adobe kind of purchased theirs and pieced them together. So they're more of a hodgepodge, shall we say.

And moving around between the different components is often not as. Cohesive as what you might see in the Liferay platform, but there's other others in the Magic quadrant by Gartner that also do DXP platforms, but we are the only open source player in that market

Doc: Is it a magic quadrant of DXPs or there are DXPs in a oh really?

Okay. Yes. Yes. That's interesting

Jonathan: What what license is the project go with

Dave: So we have two licenses. We use LGPL for the community edition. We also have a it's a special license that for the commercial aspect.

Jonathan: Is is is this an open core project? Like are there bits that are in the commercial offering that are not in the open source part?

Olaf: They're,

Jonathan: they're the, the way

Dave: the code works now, the repository has everything, okay? Both the, the open code as well as the not open the commercial code it's all in the repository. It's just the license that protects the usage of that in commercial situations.

Jonathan: And, and so what's the, what's , what, what's the, what's the catch there?

Right? Like what's, what's the difference? So, and, and I guess part of this question is why, why would someone. want to come and use that, that commercial license?

Dave: Well, for the most part, the commercial aspect is going to bring support. It's going to bring access to our cloud based offerings. We're unique in that you can self host your platform, but we also have cloud offerings.

The cloud offering is for the commercial side. So there's a number of reasons that an organization will say, we, we want to go with the the subscription to LifeRay rather than the open source version. But the open source platform is just as capable on many aspects for hosting any kind of website. The big distinction between the two tends to be the enterprise y sort of capabilities that a client might need, such as, you know, using an Oracle enterprise database as opposed to an open source database like MySQL or Postgres.

If they need enterprise connectivity for SAML or OpenID Connect, those kind of pieces will fall under the enterprise side. But for generic sort of open source connectivity, you can use the Community Edition for that.

Olaf: And if I'm really nitpicking then there is the, the price for the license for the commercial, as well as for the open source version is exactly the same.

The license is there for zero. What we do is the services subscription, which literally then is access to longer maintenance to, to support, to get hot fixes on exactly the version that you're running and so on. And then the service level on top of that. So, yes, you can you can file issues on the open source version but there is naturally no surface level attached to that.

We're very interested in that, but the on the enterprise level where we have a contract with our customers we do guarantee your surface level.

Jonathan: Mm hmm. Makes sense.

Doc: So, I'm, I'm curious about case histories. You, you mentioned the City of Vienna and Hewlett Packard Enterprise on your website.

But I'm interested in hearing about those kind of things, but also the open source ones. Ones where you're actually not involved, but you know they're there, and they're busy kind of proving your case. Can you go over some of those?

Dave: Yeah, from the community side There's a major American automaker that for years had used Liferay to host their internal intranet.

They built out a complete solution for all their employees to access. Much to our chagrin, of course, we, we would have preferred that they would have gone with the the subscription model, but they were able to build their solution on the Community Edition. The flip side of that is, from the commercial side, we've hosted many different types of platforms.

From a children's site that hosted games and videos and, and things that, that kids would be interested in. All the way up to websites for some of the U. S. military branches. Enterprises of all kinds, banks, insurance, manufacturing. We also do internet based solutions, extranet. Internet front facing brochure websites.

The platform is really flexible in how you can use it to solve any sort of web based problem. And the fact that it's a DXP means we can provide the content regardless of what kind of device you come to us with. If you're on a mobile, you're going to get a responsive platform. If you're on a tablet, it's going to also be responsive versus a desktop.

But ultimately, you're going to get the same kind of experience. And from a platform level, we can follow you across those devices and make sure that we are tailoring content and personalize it for each visitor so that we can help target them to meet whatever the business needs are.

Doc: I was wondering about, so it's interesting when I went to your website I'm, I've been in my work as a journalist is basically around privacy and and I'm Always looking at, at the cookie interfaces and you guys had tracking cookies for targeting turned on by default. And, and I'm wondering if and, and I rejected it because I'm actually not in your market.

So I thought, okay, well, I kind of don't feel like being tracked. How's it going with that? I'm just curious about if you have any intel, intel as it were on, on people turning that on and off or that helping or hurting your. Your cause as people get into their, the websites, because websites are basically where you manifest for the most part, I would think.

Dave: Well, when you, when you hit liferay. com, you must realize that is our marketing site. And the cookies that we're using there are designed to help us to promote our products to the right audience and then track how they respond to that information. It's not meant meant to you You know, follow you throughout the web and use it for any nefarious purposes.

It's more for our business aspects. And when you say you're not in our market, well, you know, we like to think that everyone is, we're a worldwide company, we have offices all around the world, we have organizations using our platform for many different businesses and purposes. So, we like to think that everyone is our mark.

Doc: Well, I, I, I like to be in your market, but as, as somebody who's 22.5 years old Celsius

it's unlikely in the fullness

Olaf: of what's left of my time. Well, you, you're still, you're still the person with the most browsers on a single computer that it seems that way. It seems king of the browsers.

Doc: Yeah. Yeah, I just added DuckDuckGo actually as another browser. I play with the, how can you say about how large your market is in terms of sales?

I guess it's mostly enterprise sales. And if you can't, that's fine. You save a lot of offices, so it must be doing pretty well.

Olaf: Yeah. So the last thing I've seen is I think it was more than a thousand or a thousand, 200 customers. on it. We don't really follow that, or I don't really follow that closely because like a rough ballpark to me is enough.

There is the one aspect I've mentioned Brian as the founder of the project, and he's still around. And as Dave said, we're one of the few On this market who allow you to pick self hosting or cloud hosting pass or sass All of that works for us and we're committed to all of them. We're also One off if not the only large open source software company that is still operating completely and absolutely without any vendor Not vendor venture capital So it's full fully owner owned or founder owned and there is nothing Nothing coming in In fact, there were offers, but the story that I've heard about them was when the the venture capital companies were asked.

So assume you give us money, what should we do with it that we don't do already? They couldn't answer so

And

Dave: the additional thing is the the community edition we don't do any kind of tracking or tracing on so You can download the community edition use it in your business use it in your church use it in your Interest group for whatever reason, we're not really going to have any visibility on that. So, you know, you're, you're kind of free to run away with that and, and use it as you like.

From our side, though, obviously, you know, we're, we're not tracking and tracing that. So we, we don't really know how widely used the Community Edition is actually used out in the wild.

Olaf: Yeah. And as Dave said the, like liferate. com is our marketing site. There is the community site, which is liferate.

dev. And I haven't checked it, but if you look at that, I'm, I'm quite sure that there is no external tracking whatsoever. There might be the, the actual, yeah, we're using cookies. I think that's there. But I am not aware that we have configured any marketing like tracking external thing there. I just

Doc: brought it up.

Nothing happened. That's good. It's a very old fashioned that way. Oh, no, there's a tiny, this website uses cookies to ensure you get the best experience. Learn more or accept. But there's no third

Olaf: party cookies that I'm aware of. Yeah,

Doc: yeah, yeah.

Jonathan: I Very good. I, I guess this is, this is an opinion shared by many people, but I am so sick of the, this website uses cookies pop up.

And every time I just say, yes, put cookies on my computer. That's fine because it's, it's just a website. Storing a little bit of data on your machine to make the web, like that's how the web has always worked. I, I, I am very unhappy with European laws for cursing all of us with that dumb little pop up.

But that's, that's an entire rabbit trail. Yes, the, the The idea of venture capital is really interesting. So we've talked to some people that have had venture capital and it's worked for them. And then you also talk to people that have had venture capital and it really changes the project or it changes the company that, that, you know, was started from an open source mindset.

And then there's even now some venture capital groups out there that are like the VC is intended to be friendly with open source And I find it really fascinating that life ray Didn't ever have to go down that route And I I think probably why is just listening to you talk about it from the very beginning when it was just Hey, let's build a portal for a church The concept was sticky.

There was something about it that when, when other engineers, when other web people heard it, they went, Oh, that's cool. And is it, is it still sticky? Do you still have that, that you know, that kind of aha moment when someone finally understands what it is that Liferay is they go, Oh, that's really cool.

I could use that. It, do you still get that?

Dave: Oh, we do, but it's changed over time, right? Initially. When you went to Google, you could search for portal software and, and you would find a WebSphere portal from IBM. And there were a number of other portal platforms that were out there that have gone the way of the Dodo, right.

They've all but disappeared. Liferay is the last one standing in that market. So, you know, we don't even like to talk to Liferay as a, as a portal platform, because. It's not hip to be up for the platform. So now, you know, obviously the market is more interested in DXPs and CMS, headless CMS digital asset management, those aspects low code and everyone's favorite AI.

So we're, we're growing the platform over time to stay relevant and incorporate the kind of capabilities and functionalities that businesses are looking for in a platform such as ours. But that has changed how we have stayed relevant.

Jonathan: Is there is there a big buy in with AI in Liferay? Is that something that's here already or is coming?

Dave: We have a number of integration points already. When you're building CMS. Sites, you can leverage a generative AI to help you build content. There are other aspects for allowing for image generation and auto translation features, which is really been a great help to many of our clients.

We have one client with uses over 70 languages on their platform and they AI auto translation to previously before that capability was in there, it would take them months to coordinate a release because they had to make sure that all the content had been correctly translated into the 70 languages.

Now they're doing it in a matter of weeks. Because AI is allowing them to get that content translated into the languages they need. And then they're just coming in and reviewing and fixing the obvious problems that AI has with, you know, understanding correctly and translating correctly.

Olaf: I can say it's a good help for me or great help for me.

I'm in my day job, I'm a sales engineer. So I'm always asked to demo the platform to various different people. And I mean, as, as much as AI can hallucinate whatever, I do not care about it being, being exact to demonstrate, to demonstrate platform. It's totally easy to say, Hey, write me 10 articles about how the financial world like blog articles about how the financial world changed in the past 10 years.

Make it 200 words and more go. They have AI has AI

Jonathan: has replaced the lorem ipsum for you. That's

Olaf: exactly it.

Jonathan: That's great. I

Olaf: have a lorem ipsum on my stream deck in front of me. But I'm rarely using it anymore.

Jonathan: Yeah, that's funny. I even

Olaf: have bacon ipsum on there. Bacon

Doc: ipsum. So, inside your product, I mean If, in using AI there, I think you said you use it there, not just to help with marketing.

Do you, do you go to the clods and open AIs, or do you download Llama and work with one of the open source alternatives? No, we, well,

Dave: we define interfaces for everything and then back that by implementation. So we have a generic interface that defines how to use generative text, for example.

Internally, then, we will implement that in various different flavors, whether it's going to OpenAI or Cloud or Gemini when they have APIs that we can plug into and use. We have not yet. a module around a self contained large language module. Not to say that we couldn't do that but I think what you run into is having the the muscle and the processing power to host that kind of thing on your own.

I think generally the best consensus is to try to leverage one of those services rather than, you know, Train and host your own

Olaf: and as the whole platform is built on those apis Now the back end is built on osgi or around osgi And I always like to stress that everything is based on an api It is up to the customer as well to say like hey This is a a nice implementation, but why would I want my send data over to them?

I want my data to stay here or to, to go to that provider. So the interface is open and the whole platform allows you to just deploy a tiny module that then implements the interface to interface with the system that you want. That will be Java, like on the OSGI side. Yes, there will be a Java module, but more and more of those modules, we provide a headless interface for that.

You can just trigger from the outside. And then you're, you're again on, on whatever whatever language you want, because the only thing that's in common then is rest.

Jonathan: So I am poking around at some of your websites and I find myself at liferay. dev, which is sort of the, the main open source community site.

And I see down at the bottom powered by the Liferay portal CE and oh, it's like seeing companies eat their, eat their own dog food. As we say is the Liferay portal used a lot internally for Liferay? Like how many, how many instances of it do you think you guys have across the company? Every host.

Dave: Interesting platform we offer is running on life brand.

Jonathan: Yeah. , that's so a lot. . Yes.

Dave: And, and honestly, eating our own dog food makes the product better. Oh, absolutely. We find the bugs that, that bother us. Mm-Hmm. . And we, you know, we fix those. So, you know, having ourselves as our own customer just makes the product that much better, more stable, resilient, and secure.

And that also, I'm sorry, but that also means when we extend this out to clients, they're getting the same level of support and security and everything that we need for our own business.

Jonathan: It's always a little worrying actually when you interview a company, we talked to a company that does open source that doesn't use their own open source project.

You just have to wonder about that. It's like something went wrongs here somewhere along the way. That's right. I'm curious about scaling, right? Like so you've got some big customers and and this, this is where really, I reveal some of my ignorance about doing, How Tomcat works really and how the Java ecosystem works.

I'm curious like how big does this scale up to and what does that look like?

Dave: How much money do you have? Well, that's always the

Jonathan: question now, isn't it?

Dave: Yeah, well, you know, and I joke, but it will scale up as large as you need it to be. So if you have just a requirement for a thousand concurrent users, You're probably going to use like a two cluster node for that.

But if you have 10, 000 concurrent users or you want to protect it against being slashed on it, you're going to go a lot larger there, but that's fine. Our platform is built around being stable and secure and steady in a clustered environment so that it is. It's returning the right data. Nothing is stale or returned out of any of the cluster nodes.

It's really meant to be a resilient platform regardless of the size that you need.

Jonathan: And, and I assume that the clustering is sort of built into it or it's built into Tomcat maybe? No, it's

Dave: built into the platform. So we have each node forms a, what we refer to as a mesh network. And there's messaging going back and forth saying, Hey, I just saved this blog.

If you have an old version, you should dump it. And you know, that kind of messaging is handled across the platform. So it becomes a a very strong way to ensure that all nodes are up to date and always returning the latest content.

Jonathan: Yeah. And then do you have multiple, multiple nodes sitting behind like a a traffic manager or are these.

Are these, you know, separated out geographically and you use DNS to point people at different places? I'm just, I'm just curious, however you want to

Dave: do it. Yeah. Typically it will in simple cases, you're talking about just a small cluster with a load balancer sitting in front of it. But we do support doing geographically distant regional centers where you're hosting Liferay and, and depending upon what your requirements are you can build all those kinds of things and they will work with Liferay.

Olaf: I got to say it's regional though. So it's not around the world. It's not one server for Europe, one for the, for the Americas and so on, but it's more or less regional. So it's in the end, it operates on a, on a consistent database. And which can also be clustered. No matter what. So typically, the best understood system that I see is a load balancer and then any number of nodes behind it.

And it also works quite well with content delivery networks. So you can configure all kinds of caching headers and make sure that a request doesn't even hit the platform if it's not necessary. Right. But of course, if it's if it's not publicly visible, you want the platform to be hit for every single download, for example, or access of a page, not for all of the CSS and JavaScript, though.

Jonathan: Yeah, makes sense. All right. So to go in a different direction, I want to know about the community involvement. And I guess particularly in the, the, the community edition, but I suppose this, this would apply to both of them because the source is available even for the the, the enterprise edition. So like, what's the, what's the community involvement look like?

How many, you know, how many contributors are there outside of the company? And what, what does that, what does that look like?

Dave: Well contributing to Liferay takes many forms, right? There's, there's contributing just by joining the forums, joining the Slack channels and answering questions and things like that.

We, we count those in our community as contributors because they are helping to strengthen the community and build it out from a platform perspective. We do, Accept and want to accept changes and pull requests from outside the organization. The challenge for us, which may be unique to pure open source projects, since our platform is also a product, it is the product that we sell, the challenge for us is how to, how to accept Those pull requests, how to accept those changes, and oftentimes it, it will require some iterations with the contributor to get it into a form that we can accept and approve and merge into the platform because once we get that merged in, you know, we're taking over ownership and responsibility for that.

So the challenge is often how do we, how do we take those submissions? How do we transform them so that they are enterprise level, enterprise quality, and then get them merged into the system? It's not the fast process that you might expect with a typical open source process. Yeah. But it does exist.

Olaf: Yeah, there is another aspect to that. Because everything is open source. an API or has an API. One of the standard ways to extend the platform is to not at all build from source. And and contribute something to the core, because the core is built of Dave correct me if I'm wrong, thousand plus modules which is typically a separation of API modules and implementation modules and so on, and you can easily drop yet another module.

They're tiny, they're small, and you can implement them. Just in addition, and they do not need to be developed or delivered with a core, you can put them on the marketplace, for example, or publish them on your GitHub or anywhere. So it doesn't need to be a contribution that puts something back into the core.

That's the way I have started contributing by. Number one, making suggestions. Number two, publishing some random proof of concept things on on the blog to show like, hey, I've tried something this actually works. And fun fact I have, I've driven several of our developers nuts when they ask me if If something runs on master and I said, I don't know, I've never built master, but like that now finally broke down after 14 years, I have compiled from source for the very first time because I wanted to contribute some code that I want to be in the core.

Jonathan: So I've got to say we're talking about the, the, maybe the difficulty of contributing. The project has 929 contributors. Now it's been around for a long time, but just looking, looking and this is just looking at life ray portal, the main, the pro what seems to be the main GitHub repository, 929 contributors.

That's, that's pretty good. That's quite a few. Is there, is there a CLA to ask people to sign a CLA to contribute? I would assume you have to, to be able to do a license. Yes. Yes. Yeah. That's, I know some in the open source community are not happy about CLAs and I understand why. But at the same time, when you're doing something with a big business like this, it, it makes a lot of sense.

And I think the other thing with that why people don't like them is, To sign a CLA and then the code, the license code, the code license gets changed. People sometimes feel a little betrayed by it, but that's not the case here. You already have the licensing in place. People know what they're getting into when they send the pull request in.

And so I, in my opinion, I think that that makes CLAs even a little more palatable because everything is upfront. You know exactly what it is. And the other thing I wanted to mention with that is I appreciate that the proprietary license. is out there and available for people to look at. It's not something that's hidden.

You don't have to agree to, you know, you don't have to sign an NDA before you get to look at the license. And I appreciate that a lot. It it's it's a testament that, you know, the company actually believes in what they're doing and you're not trying to strong arm people into it.

Dave: Yeah. And all of our license is really just to protect the product, right?

We want to make sure that when you're using it, you're not. Like stealing and giving away our technology to somewhere else. That's all it is. It's not meant to hamper adoption of the product either from a community or a product perspective.

Olaf: There's also another aspect. So it, it almost led to me asking a third person on here, but he would be good for an episode all on his own.

One of our legal team guys, the most nerdy lawyer that I know. So he, he is deeply enrouted in the in the open source ecosystem. And he gave us the hint that we actually sponsored now I'll have to look that up. I forgot the the exact project some of the the license scanners that actually go through everything and let you know if you are matching that license or if there's some, some foreign code in there Dave, maybe you can look it up while I talk it's Matt, Mattias answered to it.

So and these guys came back to us and, basically we, we paid them to to build all of their code for the Java. Instance, as far as I understand that Mateo would be the the authority on that. And with our code, like with that code base of a thousand modules of, I, God knows how many lines of code that is, they basically have seen it all.

So that tool now should be quite rock solid with regards to Java. Yeah, interesting. Did you find it?

Dave: Yeah, it would be a scan code IO and reuse that software.

Jonathan: There you go. That's it. Yeah. Do you guys ever see things in the the, the, the proprietary license, the, the, excuse me, the enterprise edition make it into the community edition, is there kind of a flow where things go down that way after, and I know some, some businesses, some, some projects have, have this codified, like we build it for the Enterprise and then once we make X amount of money with it, it goes into the community edition.

I always thought that was an interesting way to approach it. Is there anything like that in Liferay?

Olaf: I want to nitpick there and say no, because it's better. It's all out in the open anyway. Like the code is dual licensed and to cut a release. The one thing that we do with the enterprise edition is we have bi weekly releases that are bug fixes and that go quite a while back.

While on the community side we, we do provide the quarterly releases and if there is an emergency that there was none for a while, then yes, we'll do that as well. But on the community side, you're typically on the latest version and but, but all of the code, all of like, if you compile master, this is what goes into the next release both at the same time.

Oh,

Jonathan: okay. So there's not. It, it, it literally is not open core. Then there's, there's not anything missing out of the community edition. There's, there's one code base. No. So there, there's

Olaf: nothing that could trickle out. Okay. Because it's all the dual license anyway.

Jonathan: I gotcha. I, I misunderstood the way that worked then.

I appreciate, I'm glad we, glad we cleared that up. . Alright. Doc, is there anything that you wanna make sure and get in? I think you had one or two more questions. Well,

Doc: I,

Jonathan: yeah.

Doc: I mean, one is when you're sh all of, you're saying you're, you're a sales engineer, but when you're showing off what. what what life rate can do.

What do you point to? I mean, what's, what's your, what are your canonical examples? I mean, I look at your customers, you've got some pretty heavy customers there, you know, and you know, you've Jose Cuervo. So that beer, you know, and Carrefour Airbus, I mean, city of Burbank, I imagine they're using it in very, very different ways, but But it'd be, I mean, it'd be one interesting thing to say, you know, when you're flying an Airbus that, you know, we keep the wings from falling off or some other thing, but I, Yeah.

Olaf: Yeah. So what we typically do is, number one, we have a discovery call up front so that I know which part are they interested in. And then when we have a demo call, it might be an hour, it might be two hours. I prepare something, either I pick a, a stock demo that we have and customize that a bit, sometimes it even works.

Like if I can offer, Hey, we have a matching point that I can demo tomorrow. Or I can prepare something better for you for next week. Sometimes people just opt for tomorrow and then how I open An hour quite often in the last times is that I explain what I'm going to do. I just give an agenda. So we'll go through, for example the content management part in the first 10 minutes, then we'll go into personalization, into integrating other platforms.

Then we'll go for lunch in the afternoon. We come back and then we cover the commerce features and, and DevOps. For the rest of the week we'll go, hold on. Oh, excuse me. I just hear we only have an hour. So I guess I'll have to severely limit myself and, and just pick a very small fraction of what we have.

So that's how I typically do that or very often do that. If only to demonstrate that or to show that just because I haven't demonstrated a particular feature doesn't mean it's not in. But because we by far don't have any time.

Dave: Yeah, there's so many capabilities in the platform itself. There's no one demo that works for everyone.

Each organization will come in with different things that they need, whether they're looking for CMS capabilities or document management or hosting videos or whatever. Hosting custom applications. They all have different requirements, so there's no one demo that we can show them on the platform that's going to check off the boxes that they are interested in.

So we want to know the kind of things that they want to see, and then we show them how we can solve those things using LifeRay. And most of the time it's using out of the box LifeRay. The. We're at a point now where the the capabilities on the platform with low code and objects and Everything else that we support Fragments and whatnot.

It's really easy to get solutions at work without having to invest a lot of money to make it happen

Doc: So you don't go and say well, this is how HP did it or this is how Airbus did it. It's so customized to what any particular customer needs. You just go straight into it Yes,

Olaf: It's either fully customized or like we have a starting point for an intranet.

We have a starting point for a customer portal. We have a starting point for a manufacturing site that is more commerce heavy and so on. So we can pick some and sometimes I pick a demo to start with that is deliberately from a different industry because like it's a, it's a customer portal. Touching a different industry rather than the, the commerce sites demonstrating the same industry that they are from.

So, I

Doc: wonder if, if you guys are, are at the point of success or maturation, or maybe this doesn't apply. That there are other businesses that specialize in Liferay installation. That's when you know you've made it.

Olaf: Now for as long as I'm with a company, we have a partner network. So the company business is very purely on, or yeah, not exactly only, but mostly on, we're a product vendor.

And the implementations are very often done by our partners, which is independent companies that then go in by the hour and they implement on top of our platform. We have our own global services team, so there are internal consultants, but we It's only a few areas in the world where we actually have them in order to do such projects.

Mostly, we use them, as far as I understand, might be different in different parts of the world. But I understand that we mostly use them to support the partners to do something like validating best practices and making sure that, for example, a site performs well. to, to stand by their side with performance tests, because we do performance tests all the time.

And we might as well extend that to the partner network. So technically we're the product vendor and the partners, like the whole people that build on top of it. That's I think 150, 200 partners worldwide. So there is quite some business built on top of Liferay, which we're happy with. So we're very happy to refer business to those to those partners.

Dave: And we do a lot of training with them and work with them and communications. So they have opportunities to provide feedback on how the platform is working for them and changes that they need. We really rely on our partners to handle the bulk of the implementations and to get the feedback from them in order to help improve and make the platform better.

Jonathan: Okay. So in thinking about, Everything that Liferay lets you do. I'm sure you have businesses that use this or even people use this or something like a, a knowledge database, you know, like an internal wiki, you have people that use it for customer management as well as public facing websites. So like there's, there's a bunch of different things that you could set up with a Liferay install.

And I want to know, what are some of the weird and surprising things that y'all are aware of that people do with Liferay? For me, the

Dave: one example is the, the, the kids site. With the games and the videos, I, I guess I can throw the name out there. Sesame Street for a long time, they were hosted on Liferay and you know, that's kind of a weird use because, you know, normally we're, we're going in, we're doing enterprise sites, right?

You know, so standard boxes, standard fonts, things like that. And. The rules changed when Sesame Street was using our platform. So for me, that that's, that's the weirdest one that I've seen based on the platform.

Olaf: Yeah, they, they aren't, they aren't using it anymore, though. Sadly. I loved referring to them because it was the most colorful site that I could point to back then.

Yeah. I would say the weirdest question that I ever got, I think they never executed. The idea was by a manufacturer of some medical devices. I want to say something like MRI or like that scale of machines, right? Who wanted who were asking for OEM licenses with support for a single user. So, basically, to build that in, to build in a web server and a browser in kiosk mode into their device, and serve a single user on top of the platform, and just use it as the, basically, the frame for the building blocks of or, and being able to compose an application out of many different small elements that use the same infrastructure.

I'm not sure what came out of that, because I have learned about that long before, Before I became a sales engineer and actually looked at the projects and, and had to to basically sell them.

Jonathan: Yeah. All right. Good answers. Okay. So now this is a hard question. You got to do some set math here. Is there, is there anything that we did not, because we're getting close to the fact we're past the bottom of the hour, thanks to some technical difficulties at the beginning of the show.

Is there anything that we didn't touch on that we didn't ask you about that you really wanted to let folks know about the project? I

Olaf: think I've mentioned once or twice just the vocabulary and that those teams would be mad if we don't explicitly make sure that in all of the description of what Liferay is, we barely touched that there is a full fledged commerce front end in there as well.

So I, I managed to drop the commerce syllables sometimes, but in general, we did not talk about that aspect of the platform at all. Oh, yeah, that's interesting. And other than that no, I could only tell you a funny story about the origin of the name.

Jonathan: So. Oh, please do. Yeah, I want to hear that, for sure.

Olaf: So I guess I have heard it five years, at least five years ago, probably rather 10 years ago. So what came up to that is Brian, our founder used to what was that? Or met a guy in university. And his name is Ray. And they figured we're going to do something like they had a business idea and they wanted to build something.

And it was a medical device. Now, they were about to start a company and Ray said, the only way I am going to create a company is if my name is in there. So they figured, hey, medical device. So what's it? It's Death Ray. Oh, no, that's a bad idea for a medical device. So what's the opposite? That's Life Ray. So Brian was quick to just buy the domain.

And and then he never saw Ray again.

Jonathan: Huh. That's funny. That's funny. So, and when,

Olaf: and when years later he had the project he was like, oh, how do I name it? I got to pick a name for SourceForge back then, I believe. So what do I publish it as? I was like, oh, I have this domain lying around, so let's use that.

Jonathan: That's great. That's the story

Olaf: I've heard. That's great. And I believe it's true.

Jonathan: No, it doesn't sound, it sounds, it sounds true. That's the sort of, yeah. Dave, is there anything that you want to cover that we didn't ask about?

Dave: Man, just, if anyone wants to know more about Liferay, they can find us on the, our community Slack channel, you know, both Olaf and I are very prevalent on there and would be happy to answer any questions someone might have.

Jonathan: All right. Very good. Now I've got to ask each of you before we, before we let you go, we'll start with Olaf. What is your favorite text editor and scripting language?

Olaf: Hmm. For the text editor, I'm promiscuous. I just use whatever is there. If I have to do, like, I, I am on Ubuntu and I'm happy to use KDE if I need a full fledged IDE much to Dave Chagrin, I'm using Eclipse.

And if I'm on the shell and I need an editor there, I open vi. But if my Git commit opens a nano, who cares? ,

Jonathan: I, I know that feeling and it, and in scripting

Olaf: language, I rarely do something, but I would say bash. That's, that's fair.

Jonathan: Alright, Dave. VI and Perl. Ooh, . You're the go-to Oh, well, old school. Old school.

Old school, yes. . Alright. Excellent. Thank you guys for being here. And you know, like I said, we got started a little bit late, but I think, I think we made the time up in goodness. It went fast. It was a lot of fun talking to you about it and it did a lot of fun to, to learn about life. Ray, something I was completely unaware of, but really, really interesting and apparently out there in a lot of places, so appreciate it.

All right. Yeah. Yeah. Yeah.

Doc: It was great. All right. Doc, what do you think? I, it, it's interesting to me that again, it's a, I think one of the largest enterprise, I mean, not enterprise, I guess it's a company, you know, I mean, a project that I wasn't aware of, you know, and that, that's like you, I wasn't aware of it and it's and it's interesting and it's interesting how many people are using it and now.

How well it seems to be working, and how capable it is, and how broad it is. All that stuff. It's kind of like, it is all singing and dancing, and they can actually pull it off, so, that's You

Jonathan: know, what fascinates me about this is, like, so first off, it was the beginning of this project was so specific.

It was, it was one guy making a website for his church, but he said, Well, this portals thing is popular. It's what's hot right now. Let me try to do that. And, So that immediately, because they tapped into portals, immediately it was, like I said, it was sticky. Like, it's something that you see it and you immediately go, Ooh, that's really interesting.

And then, it seems like what has happened over the years, and I should have asked him about this and didn't it seems like there's been a lot of what we would normally refer to as feature creep. Like, oh, it would be cool if it did this. It would be cool if it had, you know, a low code. It would be cool if you could drag and drop things.

It would be cool if there were integrations. It would be cool if it could do this, that, and the other thing. And normally that kills a project. But Feature Creep can, can make projects huge and unwieldy and, but it seems like in this case, I, I, and I guess this is because of the management at the top.

They've managed that Feature Creep to where they now have this like cohesive set of features. people really like. And, and that's, that's interesting to me that as an open source project and as a business that they've, they've charted those waters so successfully, apparently successfully, as big as they are and still making money.

So.

Doc: And, and, and funny that they were in a portals business and portal was a big term back in the last millennium, you know, and I, I, I remember I was at a party in San Francisco back when I lived in that area and which I did for a long time and it was in the late nineties and there was this, You know all these young guys wearing black clothes and goatees and like us, we're the ones, but anyway, they're, and and it was overlooking San Francisco from up near Twin Peaks and, and this guy You know, you're just talking at this thing.

And I said, so what do you, he says, well we have this new startup. And I said, yeah, what do you do? So we're an arms merchant to the portals industry. And I said, portals, it's an industry. He says, yeah. And, and, and he gave me nothing but BS about all the BS of the time. And, and I finally asked him how are sales thinking that would be an insulting question because they were venture funded, right.

And, and And he said, they're great. We just closed our second round of financing for 25 million. And, and I thought it was a, it was an epiphany for me because I realized, wait a minute, there are two ways a company may, you know, two markets for a company. One is for its goods and services and one is for its own ass.

I'm for sale. I'm going to go you know, but these guys, life rate, I mean, They're not venture funded. I think that's actually very cool that they're, they're entirely bootstrapped. It just works. That's, that is really unusual and, and, and there's, and they're not looking at an exit and I'm telling you there's.

As somebody who's encouraged development for a long time and it always goes to venture and it always the the looking at the exit looking at your way off this highway Where you don't even own the company anymore. The public owns you don't care. You've got your cash your You you've got the boat in florida now or whatever it is.

And and You know, but but you know the purely practical this thing exists because it's a good thing it works You It's growing in the world. It doesn't need to, you know, advertise itself heavily. It's pretty cool.

Jonathan: Yeah, yeah,

Doc: that's great.

Jonathan: Yeah, fun, fun project. Definitely one to keep our eyes on. All right Doc, do you have anything you want to plug?

Doc: Oh my gosh I'll plug IAW, because the Internet Identity Workshop is coming up at the end of October. Look up my Internet Identity Workshop. I, I have a short I, I, I pay for II Workshop every year, but it's so slow to go over there. But anyway it's always full of people. A lot of great things happen there.

It's relatively cheap as conferences go. It has no panels, no It's just all, it's all, it's all demo and and have fun with it. And Talk about your new idea. It doesn't have to be about identity either. So, and I'm very encouraged about this one because we have some, some people who want to do the crap I've been telling him to do for a thousand years, you know, I mean, for, for, I guess they were still listening in.

I mean, I I'm big at having markets work from the customer side, how let's equip the customers to be more powerful. And this is a really, a real heavy in that space. That wants to be there and they're going to be there and it's going to be really cool. So, yeah, that's a little tease. All right. Yeah.

Excellent.

Jonathan: So we've got something fun coming up next week. We're actually going to talk with John Britton from Homebrew. And they now have something new they call Workbrew. Which is, it is apparently the, so Homebrew is running homebrew. com. Compiling and running like Unix and Linux applications on Mac OS and it's been it's been around for a long time It's been very popular among the enthusiasts people like me and you But what workbrew is is taking that and making it more commercial friendly And I'm really fascinated to find out what all is going on with that.

We're talking with them next week. And then as far as things for me to plug I've got my, I've got my log scroll up right now. You can tell what I was messing with before the show, but normally what shows up here on this monitor is Hackaday. You can find my work at hackaday. com. We've got well, that's.

The home of Floss Weekly, but it's also where my security column goes live every Friday morning and the occasional other thing. The only other thing that I've got that I will let you know about is the untitled Linux show over at Twit, the Twit network, twit. tv. And you can find ULS there to keep up with all of the news around Linux and some other open source and hardware stuff sprinkled in.

We've kind of found our, found our niche there. But make sure to check that out. Appreciate everybody being here, our guests, appreciate Doc being here and everyone watching and listening both live and on the download. And we will see you next week on Floss Weekly.

Episode 794 transcript

31 juli 2024 | min

FLOSS-794

Jonathan: Hey, this week, Katherine joins me and we talked with Andres Almeray about JReleaser. It's the way to automate your releases and kind of make them self documenting. It's more than just Java and it's more than just releases for that matter. You don't want to miss it. So stay tuned. It's this Floss Weekly episode 794 recorded July 30th.

Release them all with JReleaser.

Hey folks, it is time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. And we have a real treat today. I'm kind of getting some payback for the last several years. Well, first off, Katherine joins me today, Katherine Druckmann as, as co host.

Welcome Katherine.

Katherine: Ah, thank you. Thanks for having me again.

Jonathan: Yes. And so Katherine, you're, you're, we were talking before the show. You're about halfway responsible for bringing our guest on today. Because you, you, you introduced me to Lori and then Lori's like, Oh, by the way, I know this guy that does the, the J releaser.

It's a, it's a Java program. And I'm like, Oh, Java, but for those that have been watching the show for a while, no, I'm not a huge Java fan. Now, to be fair, this is probably because I tried to do stuff in Java as a very, very young developer. And the frustrations that I was feeling with simply, let's be honest, not being very good at writing code at that point in my life, I sort of associate with Java.

And so. Maybe it's not Java, maybe it's just me, but All the same. I, I still, every time somebody tells me about a job or, or, you know, ask somebody what, what language is such and such written. And please tell me it's not in Java. So this week's a Java pro project. Next week's a Java project. Maybe we're going to just burn my distaste for Java out.

Maybe, maybe yeah. Revenge of the job. That's a good show title. Maybe we'll go with that. But for a guest today, we have a Andres Alomare. He will tell me if I mispronounce that, I'm sure. And he is a Java developer. And he is, well, he does a lot of things. He is a developer, developer advocate for the database group at Oracle.

Interesting. But he is also involved in JReleaser. Built in Java for Java. And this is, this is all about, it's about pushing releases. Like, so it's, it's part of your which could you call that? Maybe it fits into a continuous integration pipeline, but it, at the very least, it's a part of your release pipeline and that's a difficult problem to work out sometimes.

I think, I think Java has maybe some extra wrinkles with doing releases that other languages don't. Don't it's at least a little different from doing C projects. So let's let's bring him on though Let's not waste any more time And once I get him unmuted here

Andres: Andres, welcome to the show Hi hello Jonathan, hello Katherine, and for having me today.

Jonathan: Yes, yes, I appreciate you being here. Appreciate you being patient as we worked through a few uncharacteristic technical issues at the beginning of the show. Had some gremlins in our system today, but hopefully those are behind us.

Katherine: So it's funny when the nerds can't make it work. It's like, what hope for what hope is there for humanity?

Jonathan: Yeah. Yeah. Okay. So J releaser, give us kind of the, as we say here, the 30, 000 foot view, like what, what does this tool do? What do people use it for? What, what problem does it solve?

Andres: So the problem that it solves is a releasing any kind of projects into any given targets that you may have, for example, the most basic way is create a GitHub release page and uploads any binary assets the project may have as release assets.

And post announcements to different announce communication channels, where it may be X, Twitter, Mastodon, Slack Mattermost, I mean, email, there are many, right? If you happen to be built an let's say a macOS app or a command line tool that you would like to see installed using popular package managers.

Such as homebrew, macports, scoop, winget, all these things. Then the tool will also generate the required manifest files and create additional git repositories if needed to push out those files so that the package managers will have a much better time and you as a final consumer will just simply have to do brew install my app and then you're done.

Jonathan: All right. Brilliant. Huh. So, it's, it's aware of all of these different places you might push to. Now, is, is JReleaser aware of things like, say, a let's say GitHub Actions? So I know a lot of projects have some of this automated where, you know, you have a github action that you go and you run it manually and it will, it'll bump your version number and run your, your final tests and then produce your release.

So can, can you set jreleaser up to like watch a github and just watch it? Automagically go from there.

Andres: Yes. In this case, you will definitely have to configure a gift of actions to listen to a specific GitHub events. So that is completely outside of the scope of any tool that you use. This is just regular GitHub semantics, but once you do, you can cook, can cook it in with any particular tool you want.

In this case, Jerry Lisa can be using this way. Your leaser can be consumed as a command line tool, regardless of the project of your choice. You mentioned that yes, it is written in Java, kind of hints at it by having a name, a J in the name at the front, but it can be used for any kind of project. So if you like Rust or Python and Node, Perl, even, even Python or something else, you can certainly use it because you only need an external configuration file, whether it be JAML, JSON, or TUML with these different formats.

And you could just run it in the CLI. So there is a specific GitHub action, JRELISA configure for GitHub that you can use that will read that particular external file and it will produce the the release. Now, if you happen to be using a Java project. Then there's also tighter integration with well known Java build tools such as Maven, Apache Maven, Apache Ant, and Gradle.

So whether you want to use an external command line tool, an external configuration file, or you use build specific plugins, the release tool will help you.

Jonathan: So I assume JReleaser is built in Java, right? That is correct. And do you, do you manage the releases of JReleaser using JReleaser? Are you, are you dogfooding?

Andres: yes, in the first place. So this is one of the things that we did since the early days. So the project started, if I'm not mistaken, the first commit happened around November 28th of 2020. So I was bored during pandemic. And so to say, I can give you the quick history of how this thing happened. Yes, I am a Java developer, but I also work with our search languages.

So I created a command line tool using Go. And I needed a way to release it. Now, I didn't know exactly how to do these things. I was looking around it and I found a nice project called go release it. If you're working with go languages, with the go language and producing these kinds of things, you owe it to yourself to have a look at go releaser.

com because the best tool you can use to release a go at the moment. Now I provide a lot of things, exactly what I mentioned before, create a GitHub page with a release and upload the assets create all the package managers files that are needed. And at the time it didn't announce. If there's something that it added later, thanks to Gerald Easter.

So it is a little bit of a cross pollination within the two projects. And so I was very happy with what Go did at the time, but I also continued to work on Java projects, and I needed a way, now that I know how it's easy to do things with Go, I wanted to do the same thing with Java, and there's no tool like that.

So talking with some one of my friends that did the same thing. So, I mean, he wrote a command line tool and wrote everything by hand to release the homebrew, create a Docker image and a post announcements, all these things. And every time that he needed to make an update, he will have to look at a bunch of scripts to find out what exactly needed to happen.

So we took those two things, those two projects as, as as an input and as an inspiration, and that's how Jerry Leaser, the tool came out to be. And since the first day. Since we're starting to push, really commits to the main branch, we, we use the previous version of Jeralyzer. We obviously had to do some sort of bootstrapping, but the previous version of Jeralyzer will be used to release the next version of Jeralyzer in some sort of early access snapshot fashion.

And the great thing is that this configuration file that we have for early access can also be used for a stable releases or final releases. And the only difference that we have is the, whatever input file, the input version of the project it is. So if I want to release version, the latest version is 1.

13. 1. Whenever I want to release 1. 14. 0, then I just need to click a button on my GitHub action, whatever, when my GitHub workflows I had configured in the project, give it what is the name the version number that I want to release, and it just happens. And I'm quite confident the release is going to work.

Because I have had more than a thousand releases happening every time that I do a push to main So this release configuration has been battle tested a long time.

Jonathan: That's it's funny It's also kind of funny that you've got one of your you know, like your your big final test Is the release like that humors me the the the?

That's what eating urine dog food means to some extent. But that's, that's funny that you've got that big test built right in. And like, if something were broken, you'd know about it. You'd know about it right when it was a problem.

Andres: Yeah, yeah, exactly. So we have plenty of tests like every other project, but we know for a fact, there are some things that will be very hard to test.

So the actual we actually testing production by releasing these early access pushes.

Jonathan: Yes, that's great. That's funny. So is this limited to doing releases of Java projects?

Andres: No, it's not limited at all. You can use it with any kind of source projects. We do have, as a matter of fact, on the website of the project which is jarlisser.

org, by the way, you can find a set of examples for non Java projects. So let's see if I can remember all of them that we have. We have C, C Perl, Haskell, Ballerina, which is JVM language, but it's not Java. And we have Ruby, I think. We have Crystal, which is statically type Ruby. We have Zig and I think we have, I think we have Swift as well and Rust, obviously, everybody's doing Rust these days.

So if there's something that I didn't mention, it's not that difficult to hook it in. Python? Is Python in there? I don't, I don't think I'd set up Python, but it wouldn't be that hard to set it up. So,

Katherine: I have a question. So I noticed when you released, so I did a little bit of looking things up. When you released, you announced that you can even do things like it announces your releases on Twitter or elsewhere, right?

Which is kind of fun. It's, you know, full automation from end to end, right? Making developer lives easier. What are people asking you for in terms of feature requests? Like, what can it do yet that you wish it could do?

Andres: Ah, okay. Well, it's worth mentioning that we support not only GitHub, but also GitLab and Gitea.

So if you have your own instance of Gitea, then off you go. And when it's both, in terms of GitHub and GitLab, it's both free versions and enterprise versions. That's not a problem. So we have, have had a suggestion to support Bitbucket as well. There is, as a matter of fact, a pull request pending. The difference, if you know what is Bitbucket, it works in a different way as GitHub and GitLab, in the sense that it does, the service of a Bitbucket does not provide what is known as a Git release page, like GitHub and GitLab do.

Andres: You will have to post create a document perhaps on Confluence and some of the sites, and then uploads all the assets in some way, some sort of attachments perhaps to that page. So if we were to do these things with Bitbucket, we probably would have to support additional tools in the Atlassian toolbox.

And that's certainly something that we want to do. But it's still it's still work to do.

Katherine: Yeah. Some people have, you could see a lot of value there. Yeah.

Andres: Yeah, some people have asked us also to assemble binaries using Deviant and RPM, which we can do. But for the time being, it's limited for Java projects.

We use a command line tool provided by the the Java development kit, the JDK, which is called JPackager. These ones create the files that are this, this kind of native installers, but they bundle a Java runtime because your project has to be Java. So if you want to create a Deviant file or an RPM for a non Java project we'll have to support that in some way.

And that's, that's something that we have in the roadmap and having a notarization for Apple or signing artifacts using the sign tool for Windows, which is kind of required for some installers. That's also some of the things that people have been asking, but we have yet to to provide those.

Jonathan: So I'm, I'm curious, I don't, you just mentioned Apple, but I don't think this is, this is part of what you talked about.

What about mobile applications? It. We, we, we think about this idea of, you know, doing your release and you have all of your manual steps you have to get done as a release and one of the, one of the projects that I've worked on that it was just a lot of manual steps to get a release together was pushing out like a new version of an Android app or I'm sure, I'm sure it's similar with an iOS app.

Is there support in JReleaser for doing mobile apps and getting things, you know, pushed up to the, the various places where they need to be?

Andres: Not specifically yet in the sense that if there is a specific channel or a specific remote service for publishing these applications, I mean, the App Store or in the sense of Mac is you have to both notarize This is something that could technically be done today outside of jresep, but tying it up in with the same configuration file.

Jresep has two mechanisms to extend itself if there's something that is not provided by the core mechanism. There's something called script hooks or command hooks you may think of as I mean, it will be as if you're configuring some sort of gif of actions within the configuration file for javalisa.

The syntax looks very similar.

Andres: the other one is a java specific extension that needs to be loaded by the java tool in order to provide the behavior that you want. Now, it's possible to do what you describe using command hooks or script hooks. Someone could certainly do this. The, the disadvantage is that it's as project specific.

So if there will be more projects, more mobile applications, I would like to take advantage of the things. And someone would have to either create an extension or we will have to figure out all the steps that were following this way and find a way to get it into add it into the core behavior.

Jonathan: So would it, would it be fair to just kind of categorize JReleaser as it's very much like GitHub actions, but run locally?

Andres: You hit it right in the nail. That's one of the things that we have. We have a drive run mode, so you can test your configuration locally as much as you want.

And another thing that is the reasons why we have these pushes for early access on, on, on every commit domain, because whenever we add a new feature, I can test this thing locally before I can push and then create maybe a faulty early access. So once you have your configuration or you're just getting started, You can set it up with drive run drive run equals throw all the time.

And then you just keep going, going, going, iterate until you find it. Okay. Now, apparently now this is good to go and then you can push it to the remote. And it is it is a lovely

Jonathan: feature that you can do things with dry run. I cannot tell you the amount of times that I've been working on something.

Particularly with GitHub Actions, because it is, it is, it is possible, but it is very difficult to run GitHub Actions locally. And even when you do it locally, it's not quite the same as running it up on GitHub. And so, you end up with about 15 commits in your project of trying something with GitHub Actions that didn't work.

Like, these are the commit messages. Trying something else with GitHub Actions. Hopefully it'll work this time. I'm about to give up. Why? Like, these are, these are real commit messages. Why is this in my life?

Andres: Yes, exactly. So, so yes, the tool allows you to test out things locally as close as you can.

And I mean, if you really want to, for whatever reason, you can also push out a real release from your local environment because you, because you may not trust GitHub or any other remote services with certain particular secrets. And that's fine. So, and the tool is designed to be used in both ways, local and remote.

Jonathan: Yeah. Is it, is it essentially the same running it locally and remote or is there, you know, is there some fundamental difference about if you run it on a service somewhere?

Andres: No, it's exactly the same as the same command line tool and the same configuration file.

Jonathan: Okay. What, what are the, what are the options for running it remote?

Like where does that make sense to try to do this remotely? Like is this something that will happen in AWS? Can you run, I think you said that there is a there's a J releaser action already built into GitHub. Like where, where are the places where you can have this in the cloud as they say?

Andres: Well, the my recommendation is that you pick your CI city tool of choice and then configure it there.

We have a summary finding the guide in the, in the webpage of the projects. We have, I think there's integration with 16 different services. At the moment, there is two that are specific, one that we already mentioned a couple of times, which is gif of actions, and the second one is GitLab CI, so as you may know, it uses docker containers, so, we do produce a docker image for jreser that you can use in any other way that you would regularly use a docker container, but this one has additional settings to detect if it's running within the GitLab environment, or the GitLab CI's environment, then it will set certain specific environment settings, So that it will do the right thing for you.

Jonathan: And then, and this is of course on our mind because what, two weeks ago now we we, we blue screened the entire world and CrowdStrike pushed out a bad update. Does JReleaser have test suite support in it and will it, can you set it up to abort pushing an update if one of those tests fail that you expected to pass?

Andres: No. And the reason being is that Java Elixir is a release tool, not a build tool. And this is the reason why we can support many different languages, regardless of whether it's Java or not. Because you keep using your build tool of choice, whichever it may be. Cargo for Rust, and Scones or Conan for C whatever it is, right?

Yeah. Once you have built your binaries, if you produce any binaries, Then you continue into the release process. And this is where Jeremy, sir will help you. Okay. So if there weren't any tests that were failing it will be a, steal the the job or your bill to find that out. And if the bill fails, then that's where you stop.

That being said Jerry sir does have support for additional security and supply chain settings. For example it recently added support for generating the input file required for GitHub attestations.

Jonathan: Okay.

Andres: So if you're released on GitHub and you want to use their recently added GitHub attestation feature.

You need some inputs and Jerry said can compute certain inputs, so it makes your job easier

Jonathan: that that would you that wouldn't happen to be related to a a certain compression library that contained a backdoor for SSH. Would it,

Andres: No comment? Yes, the other. The other option for at the station is salsa.

Yeah, yeah, it's also about that and that we do have a custom bring your own builder. That we built along with the, the salsa team. So if your project configures Chevrolet set and it's Java based where it's maven upgraded, they just pointed to our Java builder and it does the thing for you and it generates the attestation file and it will upload it to the release notes well to as a release asset into the same release that you just created.

We can also generate SBOMs and SWID tags, and SBOMs, whether it's SPDX or Cyclone DX, we support many different formats. Okay,

Jonathan: so I, I think You beat

Katherine: me to the S Bomb question. Well,

Jonathan: so let's actually, let's dig into that a little bit more. And I think I'm following along with what, you know, these, these acronyms and extended acronyms mean.

But for, you know, not everybody is in the weeds on this. So what is an attestation? What is an S Bomb? And why do people care about it?

Andres: So, an attestation is some sort of file that is signed, and that it proves the provenance of an artifact from a certain location. So you can, you can tell. That a certain binary has certain given signatures and hashes that it was created in a, in a given environment so that you can, if you as a consumer, someone can tell you this environment is safe and it has certain characteristics that you can verify this with the attestation and the artifact.

You can also verify that the producer, the person that is giving you is providing you the artifacts is actually the one that is saying who they are because of the signatures. In case of SBOMs, it stands for Software Bill of Materials, and that's something that has been known for quite some time, but it's now become much more important in the last two or three years for two reasons.

There is I think there's, I completely forgot the name of the of the memorandum that came out from the White House two or three years ago that says if you, if you are a software provider and you want to deal with the, with the government, then you must provide SBOMs. And in the European Union, we have a similar thing called the Cyber Resilience Act.

Now, these two things, these two things came out, have came out into force as law this year. So if you are dealing with government and offices, you must provide these additional metadata files. And because it's somehow most common in governments. Other companies will also start asking for these things regardless is governmental related or not.

So In in a few years ago, this was like, okay, let's figure out what it says bombs Maybe there's some tools do conversion what not now Place if you you are not ready to produce s bombs You must do it right away today after you finish listening to this podcast.

Jonathan: And a lot of that came about as a result of the, the big log for J vulnerability back about three years ago now.

So, and this is, this is something I kind of want your thoughts on because it's, it's, it's one of the things that I'm not crazy about, about Java. So when you build a Java application, you've got. Your entire library stack gets pulled into your, usually it's a jar that you distribute. And so what happened was one of those library bits was log4j, which is, it's just a piece of logging software.

It helped make your logs look prettier, right? But it had a, it had a problem, had a vulnerability in it that you could give it a, a bit of string and it would, it would essentially try to expand the string. And as a result, you could. Run shellcode you could run like command line code using this this specially formatted bit of string And the problem was that this thing was so popular.

It was everywhere like it was in Minecraft but because Logging is great. It was also accessible from everywhere. And so Like in the example of Minecraft, you could join somebody's server, send a message over the server, like, like sending a message from one player to another, and because it was trying to log that, you had remote code execution with this really trivial, you know, way to do it, and because of the way that Java builds these libraries right into the JAR, You can't just update the library.

You had to then update every program that contained the library. And so the thing that happened though is, there was then this question that every corporation and the governments needed to ask is, does our Java application contain that? What version of that library does it contain? And so then you had this idea.

I, it probably, it probably wasn't conceived of then it probably already existed. I honestly don't know for sure, but it became very, very popular because people suddenly saw it would be really useful to just have a list of all the libraries that this program contains. And so that's kind of where we, where we came to this, this point of.

S bomb software build materials are really important because there, there will be, I mean, obviously there will be another log for J there will be another vulnerability that is that bad. And people want to be able to get out in front of it a little easier this time around.

Katherine: Well, the whole conversation around dependencies is, is so, soft, I can oversimplify and say software is a lot more complicated than it used to be.

And so you look at these dependency trees, you know, and think about the good old days when you had a handful of dependencies and secondary and tertiary, but now it's, you know, if you actually draw a picture on some, especially web apps, it's kind of mind blowing. You can't even see it, it's such a mess.

Andres: Right. So, so basically SBOMs is just a recipe that lists all the dependencies that you have in your project, how it was built. And and because you have this list, then you can go to some other remote service and ask if any of those dependencies have any vulnerabilities. So the current focus that we have for SBOMs is check for vulnerabilities, but it's important to remark that SBOMs can do more than that because there's just many, I'll tell you that.

Right now we're just trying to check that if something is vulnerable or not, but we can do more than that just because we have that information about the project themselves.

Jonathan: Yeah. So there's, there's some other interesting things people are doing with SBOMs. I think groups like Tidelift are taking your SBOMs and actually saying, Hey, if you will pay us essentially a subscription fee, we'll take part of that fee and turn around and give it to the open source projects that make up your SBOM.

Like some really interesting things like that are happening being, you know, where this idea is being reused for things other than just vulnerabilities. It's really fascinating. I, I'm curious if you have any thoughts on, and again, I warned folks, I have just, just a little annoyance with Java and it's not just Java.

It's several languages that do this, but the fact that all of those libraries get built in at build time and the fact that that makes it more difficult. to up, to do an update when you have a library problem. So, on the other hand, on the other side of the coin, in a C project, where you have libraries that are just system libraries, you can just go and update the system library and fix the problem.

Like, it's kind of a bit more, let's just say it's more challenging in a Java project to do library updates, it seems. Is that, is that accurate? Is that a fair thing to say?

Andres: Well, so, I mean, if you have access to the system and do a patch that is affects the system wide, then great. You have admin, you have root, but if you don't, and then you, you get hacked, then off you go, all your C apps will be affected.

Whereas in a Java app, if it's contained to just a set of applications that are not completely system wide, then you only need to patch those. Now the problem is, find each one of those that may or may not have been affected. You may be lucky if it's only a handful. You may have a terrible time if you had to check every single one of those.

Jonathan: yeah. Okay, so What I, what I heard you just say is that I approach this, this conundrum from a very sysadmin point of view and not an application writer point of view.

Andres: Yes, kind of.

Jonathan: That's, that's an interesting observation.

Andres: There are pros and cons from both approaches. Yeah, yeah, that's,

Jonathan: that's true.

That's, that is, of course, true. Okay, what about what about Node. js and Kubernetes? It's hard, it is hard to talk about like this style of problem without thinking about things like, what was it, right pad, or was it left pad? Left pad. Left pad, yes, yes, yes. The single, the one liner JavaScript library that someone decided, you know, they weren't getting supported for it, so they just pulled it and made it disappear.

It's difficult to talk about the supply chain issue without thinking about left pad. So does, does J releaser help with no JS and JavaScript applications as well?

Andres: For, in the, in the case that we just described, I mean, dependencies and all these things. No, it's a, it's completely out of the scope by the service that allows you to publish the packages and consume them because that particular service and okay, I mean, this happened, what, like eight or nine years ago?

It's been a

Jonathan: while ago now. Yeah.

Andres: Yeah. I have not been kept up with times, but the problem at the time was that that particular repository was not read only or a pen only. So you could modify it. And that's exactly what happened. Someone said the the author of Lepa said, I'm, I'm angry because I mean, we can talk about this, the story, but he said, okay, I'm going to unpublish my packages.

Mm-hmm. , because this is what you did to me is, is not what I was expecting. That boom, that broke the internet for three days. In the case of Maven or in the case of the Java system, we have this repository called Aven Central, which is a panel only. So once you publish a package, it's stuck there.

You cannot unpublish. So if you made a mistake, well, sorry that stays as it is. And if you made it right, then also it stays as it is. So that's a real advantage.

Jonathan: Yeah. And so can, can you use can you use J releaser to, to push no JS packages?

Andres: Right now, if it's published as a regular HTTP yes.

If we need additional support for, I mean, if it's more than just a simple By simple, I mean post or put and be authenticated. We probably need to add a specific support for it, but if it's just a regular HTTPS endpoint where you do a post and a put, then yeah, you're good to go.

Jonathan: And I suppose because in JReleaser you've got like script hooks, so you could, you could probably just hook in to run a bash script as part of your release process, right?

So really, so really the sky's the limit. You don't have to have support for all of these things. Like I could, I could sit here and we could ask about, you know, every package manager out there and every language out there. But the, the reality is you can write your own script and have J release called script and you get there that way.

Andres: Yes. So obviously the advantage on cool is that there is already a DSL where you can define all the kinds of configuration. And this DSL works for every single environment running environment that scribe before where they see a live. Or is Maven Gradle plugins and end all those things. Now, this DSL allows you to reuse common configuration that you have for the different steps for Excel or, or settings.

So there is project common configuration, like what is the name, what is the tag name? Version number, release name, stuff like this project project author, links, text. So many different things. And then there are sections where you configure the actual release. Mm-Hmm, . Where is it going to go? Is it going to be to GitHub or GitLab or something else?

And you also can configure release notes. So this, this is something I wanted to come back something when you mentioned where you're trying things locally and then you create a bunch of, of, of commit messages. We always do, we most likely use throwaway commit messages. But if we don't do, we don't squash it, we don't rewrite those, those trash commits will remain forever into our history.

So one thing that the tool allows you to do is generate a release notes based on commit messages. It's not the only tool that does it, but this is something that if you define your own conventions and by the way, we support out of the box, conventional commits or gitmoji conventions it makes your life much, much easier.

So all these things make sense when they are completely baked into core. If you want to use a script hoops to support The package manager that we don't support yet, you can certainly do this. But I mean, if you feel that maybe it's too wonky, yeah, it kind of works, but if it would be better to have support of core, we certainly would encourage those people that write a custom scripts to come back to the project that open a discussion topic or an issue so that we can track this and then figure out a way how we can add that behavior back into code.

Yeah, absolutely.

Katherine: So when you talk about things like release notes, I start thinking of like I don't know. I mean, I, I feel like every project kind of has their own style in a way about a lot of things, not just release notes, but talk to me a little bit about extensibility. I mean, again, projects have their own needs.

And if I have specific needs, what, what can I do? I under, I believe you have an extensible architecture. I can write some sort of plugin or something to customize it to my own needs.

Andres: Yes. We have an extension mechanism and there is a handful extension points at the moment. So the the full configuration, the whole, the whole life cycle is not yet extensible.

We're just adding, well, a few extension points at the moment. And one of them, it is that has not yet been added, but it's something that we have in the roadmap, is fully configure how release nodes or the changelog gets gets created. We have the options to, already, to parse these conventional commit settings, or you can define your own presets, your own conventions in, in declarative way we can also take in a changelog from an external file or instruct GitHub to use their native way to generate release notes based on issues and pull requests if that's what you want.

But if you want to post process this release notes in some other way to, I don't know, change some text into emojis or do some kind of text replacements, it might make sense to write your own custom extension. And this is something that will certainly be added into the future.

Jonathan: So one of the, one of the interesting things that you can do and again, I'm most familiar with, with using GitHub for this sort of thing.

That's why I keep going back to it. But one of the interesting things you can do with GitHub is you can make your you can make your release process like transparent. People can look at it. It can be part of, you know, your, your GitHub code that's out there. But then when you do that, you, you suddenly have this problem.

I'm like, how do you. Keep all of your your keys secure and secret. Because obviously you don't want to leak. You don't want to leak a organizational GitHub key. You don't want to leak an AWS key. And there, people have come up with ways to do that. And there's even some tools out there now to scan for those things, to make sure you're not leaking them.

I'm curious whether this makes sense, like, is this a thing with J releaser as well? Can we make our J release? It's not exactly a script, essentially a script. Can we make that public? And if we do so, do we, do we still run this problem of, you know, leaking secrets that really should not be leaked?

Andres: Certainly you can, you can make it public as a matter of fact, the Jerry's her project has his own release configuration made a public and everyone can see that they are certain elements with property that given setting is a secret.

So they, because we know certain property, so take those inputs coming from environmental variables. So it's up to you to provide the secret in a proper way with gift of actions. And using environmental variables and that's something the same thing that you can do on your local environment.

Jonathan: Okay, that makes sense Yeah, good stuff.

So let's see As you as you look at as you look at the jrelease project What what do you see coming up next? Is there something that you guys are working on already that you're you're excited? Well, you know before I even ask that there's a couple of other questions that I wanted to get to and and that is well First off, what license is this released under?

We talked about asking people to push things back upstream. What license did you guys go with?

Andres: We went with Apache v2, this one. Okay, so that's a And we also have a CLA.

Jonathan: Oh, okay, okay. That's a pretty permissive license. Let's people do pretty much what they want to. Now, I'm curious, what's the rationale behind the CLA?

Andres: The CLA was done so that if in the future we would mind to relicense a project then we don't, we don't need to hunt down every single committer to make this happen. So for those that are in the Java space, there's a very well known project called Hibernate, which is an object relational mapper.

Everybody likes it. And there's also a lot of people that hate it just because it's an ORM. And it started with a GPL license. And they wanted to migrate to Apache v2. It took them more than three years To search for every single committer to ask them for the permission to do this license switch So the one of the advantages for a project that have a cla is that if you write down in the cla that Well, not only that everybody has to sign it so you can know who who there are how to contact them But also if you write down in the cla that the the project The the commit has given the rights to the owner of the project.

In this case it will be me.

Andres: Then I have more limiters to do in the future, but obviously this is not some evil ploy to do something in the future. But it's just to make sure that, to, to put out things in the open again, to, to ensure that if you are willing to commit to contribute to the project, then know that these are the conditions and you're fine with this, then go ahead.

Yeah, you

Jonathan: know, I'm, so I, I'm part of a project that has a CLA as well. And it's for very similar reasons. It's the, the, the reason there is if we discover in the future that there's a problem with the license, this gives us the ability to relicense it. At the same time, you do see businesses and projects out there that have CLAs that let's just say they use that power for, for what some might consider evil.

Right. And I w I wonder whether it would be possible to write a CLA that Is a little more, a little narrower than simply giving all rights to, you know, the, the upstream organization. I wonder whether we could write a CLA that just says the upstream organization has the right to relicense this to any other, let's say, OSI approved license.

I don't, I don't know if anybody's doing that, but that seems like that could be an interesting interesting way to handle this. You know, give, give a project some tools to fix things. If things go wrong, we're at the same time giving the people that contribute a bit of assurance that look, we're not looking to sell you out and go make millions of dollars on the backs of your code.

Right?

Andres: Yeah, exactly. I completely understand. And I agree.

Jonathan: Yeah, that's, that's an interesting thought. Okay. So you, you, we also have talked about other people contributing things. What does that look like in JReleaser? How many, how many people are working on this and how many contributions do you get from outside?

Andres: We have a small team working on the team, on the, on the tool, but we do use don't know. And this is also sort of conversion, but it's another application that you can set up on GitHub since our GitHub action is called all contributors. And with this, you can have a very nice looking markdown based documentation where you see the avatars as every single contributor.

And they are listed by the type of contributor they made, so whether they fix code, they are announcers, they promote the project, or they're doing documentation, internalization, so there are many ways. So for every single contribution that is made for the project, whether it's just one typo, or it's a big contribution, we list every single contributor over there, and I believe we're past 70 contributors at the moment.

Jonathan: Yeah, that's really good. And you get a lot of I call them drive by contributions. It's where someone has one specific thing that's either broken or, you know, that they want added, and they, they drive through and they drop off their patch. Here's this new feature, and then you never hear from them again.

Is that fairly, fairly common there?

Andres: We have had a few in that, in that regard but those that are part of the the core team Most of them have joined us since the early days again, since since 2021 started discussions on how the two could work. So I will say that it's it's, it's a herd. I mean, it's a healthy mix of long time contributors, even though small set and just quick contributions from all around the world.

Sure, sure.

Jonathan: That makes sense. All right. And then I was, I was going to Let's see. What was the question I almost asked?

Jonathan: What what's coming up? What's what's in your roadmap for the future.

Andres: Right. So, so in the early days of the tool the reason why it was written. So it was so that I could release a command, a Java command line tool in many different aspects. Now for this to work, the package manager has to understand that your binary has certain file structure.

So they will know what to do with this onManifest and every other file that it requires. So for, for this to work, we have diff like a set of distribution types. There's your standard Java binary where it has a, a, a script or a launcher. And the list of jar things that have been asked for the tool is to support source distributions.

Once we do this, then how we will be able to create source RPMs. Or deviant files for sources or support any other kind of package managers that will deal with source distributions other than binaries is something that will certainly be looking at how to make it work.

Jonathan: Yeah. Interesting. All right.

Katherine, anything you wanted to ask? I think you've covered

Katherine: it. I

Jonathan: get down towards wrapping. I,

Katherine: I don't know. I think you beat me out to all the questions.

Jonathan: Sorry. Okay.

Katherine: Yeah.

Jonathan: So I'm, I want to know, is there, is there anybody out there? Is there a project or anything that you know of that's using J releaser in a way that just surprised you?

Is there anything weird? Yeah. Are there any surprising uses?

Andres: So there's actually some sort of interesting look within one of the, our inspirations for Jerry cert the project that I failed to mention it before is called J bank. It's a command line tool that allows you to launch Java applications in many different ways.

And once Jerry reached a certain level of maturity, J bank decided to complete it. Their own manual way to do things and use Jarlisa. And both Jarlisa and Jbank use a package manager called sdkman, which allows you to install binaries just for the Java platform. So this is like, similar to RVM for Ruby, I believe.

As a matter of fact, that was the original inspiration. And I am, I am in talk, well, I was in talks and I'm, I'm not, I'm kind of a member for the SDK package manager as well. So at some point, sdkman also uses Jarlisa. For some releases. So the three projects are kind of like joined at the hip. If something goes wrong with Jarlisser, then obviously Jbank and SDKman will know immediately what happened.

And one more thing that I can say happily is that these three projects alongside alongside other six are now members of a brand new server foundation called the Common House. You can go to commonhouse. org. The idea behind this foundation is that we want to provide a long term home or a forever home for open source projects.

We started with we're targeting well known Java projects, but it's nothing, this is not that it's close to non Java projects. So in the future, for sure, we will be able to support any other kind of project right there. Because the whole point is that if you have a project that has has created a good impact in a certain open source community.

And you feel like, I mean, the maintainers that they, it will be better to be supported by the umbrella and the facilities of an any given foundation that common house will probably be one of those options.

Jonathan: Yeah, interesting. All right, we are, we are getting towards the end of the show. Is there anything that we did not ask you about that we should have? Anything you wanted to plug or let folks know about?

Andres: One more thing is that we decided as part of the team to have a very deterministic release cadence.

We release a final release every two months. So if you are a consumer of the tool and you know that there is a feature that is upcoming, then you know that it will not take more than two months for this thing to be released or to be available in the next release. Another thing that the tool does, and we do use it for our own releases as well, is that every time that our release comes in, you can configure to have the set of issues that are related to that particular release.

To have a release label attached to it and add a new comment. I said, this issue was released on version such and such a link to the release notes, because this will notify the author of the two of the issue right away. So besides this, you can say I'll come publish to a slack or email or to Twitter.

Then I can also notify through GitHub and GitLab through their issue tracker mechanism. That there, something has been done. Something has been released. Oh, that's really cool. Also, we get that notification

Jonathan: that's that's, that's really useful that you can do that. Alright. So it, it takes you like two

Andres: or three properties.

Jonathan: Alright. One other thing that I, hang on, hang on one second. The technical issues have been terrible.

Katherine: Uhoh technical difficulties. Again,

Jonathan: let's see if I've got him back.

Katherine: Are you, are you the, you are the solo maintainer, right? Are. Are you the only lead maintainer on this project?

Andres: I'm the lead maintainer, yes, and we have two other people that contribute to the project, but I'm, I'm doing most of the work at a time.

Katherine: That was, yeah, percentage wise, I'm curious. So if, if you get tired, do you have a succession plan?

Andres: That's exactly why the project moved to the Common House Foundation. Because that is one of the tenets of the foundation to ensure that it's a succession plan. And this is something that we will continue to work on in the coming months to ensure that if something were to happen to me, then someone can continue with the project.

That's great.

Jonathan: And, if something God forbid, but if something happens, it's going to be easy for them to win the lottery. Well, it's going to be easy for them to continue doing releases because you've got your release script published right there in the public, right? Exactly. Yeah. I liked it. No, that's something actually that I've, I've thought about again, several projects that I've been involved in.

We have kind of this idea of like, who's going to take it over or, or, you know, maybe even it's not established, but you kind of get the idea. Well, it would be easy enough for one of us to pick it up, but then it's like, whenever a release comes around, it's like, what all does he do to do a release? And the steps, sometimes the steps are not easy.

Katherine: And even when you really all that good at documentation,

Jonathan: nobody is, nobody's good at documenting the steps for a release. You just, whoever does it gets it figured out and then just kind of does it at each time. I think, I think there's something to be said for J release or just for that. Just for the idea that it forces you to document your release steps.

And I think that's useful in and of itself. All right. So I do want to know favorite text editor and scripting language.

Andres: Huh. So my favorite text editor is a Vim. And if, because I like to be on the command line and if it's not in the command line is text made because I used a Mac and a scripting language.

Andres: Well, I started with Perl, then moved to PHP, and then jumped into JavaScript, and from then I jumped back into Javaspace, and I fell in love with the Groovy programming language.

So if I have to say just one of those, then I would go with Groovy.

Jonathan: Okay. Groovy is one that I am not particularly familiar with, so that's pretty fascinating. Is it I don't hear that

Katherine: a lot.

Jonathan: Yeah, does it it uses the JVM as well?

Andres: Yes. You can think of groovy as dynamic Java. Okay. Interesting. So the syntax is 98 percent equal and it's very easy to interchange the Java code with groovy code from within a groovy script.

Cool.

Jonathan: Very cool. All right. Well, thank you, sir, for being here telling us all about JReleaser. It's super interesting. For a couple of projects I do, I need to look into it. So we sure appreciate your time and bearing with us through a couple of technical difficulties. And we'll make all of those disappear, I hope, in the edit.

Katherine: Nothing to see here. Exactly.

Andres: Thank you so much, Katherine and Jonathan for having me

Jonathan: here. I really enjoy it. Yes, was, was very fun. Very much, very much appreciated. All right. Katherine, do you have any projects that need the J release treatment?

Katherine: Not at the moment, but I, I was thinking the whole time as I I'm listening to the, the, you know, all the marvelous things that it does.

How nice it would have been to have it in the past.

Katherine: That is really it, you know. I, I, I like to think that I've documented release processes pretty well because, only because I, you know, I come from a, a, a position of like, explain it to me like I'm five and, and if I don't document it for myself, I will mess it up.

So so there's that. But I, you know, it, it making, Things easier so that you can focus on the hard problems is such an important goal. Yeah, yeah,

Jonathan: for sure. Yeah, we didn't, we didn't explicitly ask him about this, but I bet you could use J releaser to even do things like push website updates and, and other, you know, not maybe not the things you would normally think of with it, but.

I'm sure you could script it and make that happen. So documentation updates, probably all kinds of stuff you could use this for. And, and again, so the, the, the point there would be you, you have it automated, you have it scripted so that you just, you, you, you run one command and it makes it happen. So like we, we think about doing an update for a piece of software.

Well, it's actually. In some ways, it's a similar thing when you do an update for a website, depending on how the website's put together. Sure, it's

Katherine: powered by software.

Jonathan: Right, right, right. And so, you know, on this side, it's not necessarily a programmer, it's a sysadmin doing it. But you have some of the same problems.

What happens when one sysadmin retires and the next one comes in? It's Okay, how do I update this? Maybe even, how do I get to the server? Where is the server that runs this? And I'm just, I'm just thinking like, this could be a great tool in the toolbox of like, IT departments and sysadmins to be able to do things like that.

So, a very, a very interesting project. And I like the fact, I really like the fact that That it's, you know, it's JReleaser. It was originally made for doing Java releases, but the sky has been become the limit for it. And they've, they've made it as modular as possible. So you can do all kinds of fun stuff with it.

Katherine: Yeah, I like, there's a lot of interesting stuff. A lot of, there are a lot of tools, I think coming out with JReleaser. The idea of making developers lives easier. I'm surprised we got through an entire, an entire episode without mentioning AI, by the way. So that's maybe for next time.

Jonathan: Yeah, I guess. I don't know.

This seems like maybe, maybe one where AI does not fit.

Katherine: Does it though? Automating things? I don't know. Automating. Processes. Do you,

Jonathan: do you really, do you really want to give AI the ability to push code? I don't, I don't think so. I, I think this is something that needs to stay out of the clutches of AI. I don't know.

Katherine: can see a little, a little, a little AI improving my release notes, you know.

Jonathan: Okay. Okay. I suppose.

Katherine: People, that's the thing. I think there's a lot of, a lot of like, Fear about AI and anxiety. That's unwarranted because we don't, we don't realize how many mundane things AI is actually used for. People think of you know, the, you know, the robot overlords will push the malicious software, but it's really like, I, I, can you just spell check my my release notes on an automated way?

Jonathan: Yeah. So yeah, that's true. You can, AI in your processes without giving it the keys to the kingdom. Right? So, you know, we, you think of the dangers of AI and people automatically go to that, that thought of Well, that's how you bring about the singularity, right? You give AI the ability to improve itself.

Well, That's not the only way that you can use this tool. You can give AI the ability to tap you on the shoulder and say, Hey, you might want to think about changing this because it sounds weird. Yeah. Anyway, that's a valid point. We will, we'll bring him back on one of their next big releases, and we'll ask about AI, and see if that's something that they're adding.

And

Katherine: then he'll roll his eyes.

Jonathan: And he'll tell us that, No, AI has no business being a part of your release project. What are you, crazy? Alright. Katherine, thank you so much for being here. Did you have anything you wanted to plug?

Katherine: Sure. Let's see. Yeah, there's that other part. So I do another podcast at Intel open at Intel.

Occasionally, I still do a podcast with DocsRules, although I need to get on that reality 2. 0. Let's see. I will be giving a bunch of talks. Yeah, starting in September, I'll be at Open Source Summit EU. I'll be, I'll be talking a little bit about AI and a little bit about security. So that's fun. And then I'll be doing it again.

And oh God, so many times.

Katherine: Yeah. Yeah. Yeah. I don't know. Find me on the internet and I'll tell you all the other times I will be talking about these things, but yeah. Grace Hopper for sure. All things open. I'll be at KubeCon probably also for sure. Sauce fusion in Atlanta. So that's an open SSF event.

That's something to plug open SSF stuff. A lot of really cool stuff going on there. We hinted at a little bit of it earlier with salsa.

Jonathan: Yeah. Yeah. A little bit going to be at, going to be at Defcon this year. Is Defcon on your, kind of on your radar?

Katherine: I would have, but I have so much travel coming up after that.

I need, I need a little time to, to, to. Be at home.

Jonathan: Alright. I understand that. I do. I do. Alright. Well next week we have another Java project. We're talking about LifeRay. I don't know a whole lot about that yet, but it is sure to be interesting. And I know it's a Java, it's a Java thing. In fact, when they, when they reached out to me, it's Olaf Koch and David Nebinger.

When they reached out to me, they're like, so we heard you mentioned Java. How would you like to do a Java project? I'm like, okay, fine. Oh, so that is next week. It's, it's, it's the second part of our two part Java extravaganza. So make sure and come back for that. And then, Oh, yes.

Katherine: Sorry. No, finish your thing.

Or I'll just interrupt and think I can't believe the most important one. I forgot. I will also be at Intel innovation where everybody should go. I put like, I forgot like the most important event.

Jonathan: You, you, you had one job, Katherine, right?

Katherine: I had one job. Intel innovation that comes right after open source summit, though.

So it's like in my head, it's all one thing, but that's going to be a big thing. I'll be doing podcasts on the expo floor and there'll be all kinds of like meetups for AI nerds, and it's going to be cool.

Jonathan: Cool. Very cool. So there you

Katherine: go. I got that out.

Jonathan: All right. Well the one thing that I do want to plug with the two thing, a couple things, several things to plug we've got hackaday, you can follow my work there on hackaday and most notably, we've got the security column goes live every Friday morning.

And, of course, we do appreciate Hackaday being the home of Floss Weekly. It's, keep your, keep your internet radios tuned to hackaday. com, and enjoy all of the coverage there. And then we've also got the Untitled Linux Show over still at the Twit Network. We have a blast there. You can catch that, it goes live we do the recording Saturday evening, Saturday afternoon, depending on where you're at.

And then within a couple of days that goes live, make sure and subscribe to that for all of the Linux news and musing that you need in your lives. I think that is it for today. We sure appreciate everybody that's here, both live and on the download. And we will see you next week on Floss Weekly.

Episode 793 transcript

24 juli 2024 | min

FLOSS-793

Jonathan: Hey, this week, Aaron joins me and we talk with Jay Khatri about Highlight. That's an open source project to do monitoring for your web applications. They have some interesting tricks up their sleeves. You don't want to miss it. This is Floss Weekly, episode 793, recorded July 23rd. Keeping an eye on things with Highlight.

io. Hey folks, it is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got, we've got quite the show today. It's going to be a lot of fun. So first off, I have a co host and it's surprise Aaron Newcomb. Hey Aaron, long time no see.

Aaron: Yeah.

Hey Jonathan. I can't do the regular scheduled shows on Wednesdays when they happen on Wednesdays. So. So I can't join as often as I'd like, but really

Jonathan: glad to be back. Well, so we may see Aaron in more rotation because as everybody knows, we have moved at least for now to Tuesdays for doing the show and oh my word, that is better for me too.

So I've got the security column that goes out live on Hackaday on Friday morning, which means that I have to have it done before I go to sleep on Thursday night. And recording Floss on Wednesday, editing Floss, publishing Floss, and then turning around and immediately starting on the Hackaday security article and having that done, it's a very busy 36 hours.

And I finally said, I'm the boss of this show. I'm the head honcho. I'm going to move it. And it's, it's been great. And so now I have, you know, almost 72 hours to get everything done. And it's so much better. Yeah, that's awesome.

Aaron: That's awesome. I mean, the original reason why it was on Wednesday was just because that's where it fit in with the twit schedule.

I mean, there was no, no real rhyme or reason to it. It was just like, Hey, we've got this open time slot on Wednesday. And that became the time slot. So it's not like there's anything else really locking us in except for. You know, people obviously, long time listeners and viewers will, will know that it's usually on Wednesday,

Jonathan: but that's easy to change.

Yeah. And if you can't watch it live, that's fine. We still got you on the download. You can still catch it.

Aaron: Yeah.

Jonathan: All right. Well, hey, we've got, we've got somebody fun today. We are talking with Jay Jay Khatri about Highlight, Highlight. io. And this is kind of a, it's like a, Browser monitoring, monitoring software for web apps.

I think I'm not entirely, I don't have a whole lot of, of inspection into this. But the idea of it sounds really cool. Have you, have you gotten to dig into this yet?

Aaron: A little bit on the website. I don't know a whole lot about it. But I do have a background in monitoring, right? So I've been doing monitoring for forever working at companies like New Sysdig.

So I was always involved in monitoring. So I am kind of, that's one of the questions I have definitely. And so we'll have to spend some time up front on the, what is this part, right? Defining what it is and what it does, because, you know, Traditionally, there's been for, for me anyway, working in the industry, there's, there's two or more schools of monitoring, right?

There's application monitoring and then there's infrastructure monitoring. So I'm kind of curious to see which one this is. Maybe it does both. How are they using AI as part of their product? Because everybody's using AI nowadays. AI is, AI is, is huge for this because it can, you know, detect anomalies and, and, and look for patterns and, and then tell you what to do.

Hopefully, if it's intelligent enough, what's going on. So, yeah, I'm super interested and really excited to hear what they're doing and how they're doing it.

Jonathan: Yeah, this is, this is one of those, I'm, I'm a little not necessarily AI skeptical, but I have AI fatigue because It's a new tool that like suddenly got better.

And so we're trying to do everything with it. But like, this is legitimately one of those cases where it may really fit because what are you wanting to do with monitoring? You're doing pattern matching. You're wanting to show me these sessions that are outside the norm. And like AI is just. So I, there might be a, a really neat synergy, not to, not to buzzword it up, but there might be a really neat synergy there with AI.

So let's not guess anymore. Let's bring the man on himself. And Jay, welcome to the show.

Jay: Thanks for having me guys. Yeah. It's kind of fun to, kind of fun to be seeing y'all Guess what Highlight does, but I feel like I really want to talk

Aaron: now. Yeah, you're chomping at the bit.

Jonathan: Go for it. What did we get right?

What did we get wrong? What is, what is Highlight for? What's the point? Yeah, for sure.

Jay: So y'all, I think y'all got it pretty well. I feel like the way that Aaron mentioned the difference between application monitoring and infrastructure monitoring is a good way to start. So we, first of all, Highlight doesn't focus on infrastructure monitoring.

We're sort of explicitly at the application level and essentially what we focus on is connecting your client. And like your browser application with all of your server side, like more traditional observability resources. So the way it works is like you install our client bundle, which could be in React, in Angular or whatever your like client application is instrumented with.

And then you install a handful of SDKs on the server side. And then we do the work to basically piece together what happens all the way from the client to the server. And the way we think about it is like, it's essentially like one big trace, right? If I can see what a user's doing. doing and what they're clicking on, and then what error they might be interacting with being able to connect that to what happens on the server is very powerful because then I can actually go that way, right?

I can know, okay. An error was found. What database call on the server caused this error in the first place, or I can be looking at some database errors on the server in the first place, but then saying, Hey, how is that impacting my user base? So that's kind of what we focus on the application level, but really tying everything in the observability world to how it impacts your downstream users.

Does that make sense?

Jonathan: Yeah, I think so. I think it does. Are we talking primarily about in the browser applications? Is this primarily web apps or does this apply to different kind of applications?

Jay: Yeah, yeah, for sure. We definitely focus mostly on web apps. And the reason that that's the case is like our client side SDK that I'm mentioning.

It runs in the browser in JavaScript. So essentially you install our SDK, you initialize it, and then we're basically monitoring the DOM. Like we're monitoring the UI to be able to then replay it after the fact. So it's pretty cool. You actually install it and then you can actually see that, okay, Aaron clicked on a button.

And when the button clicked, a new page was rendered or new text was rendered, and you actually can see it at that level. So With almost like a video, like replay of what's happening, but then it all gets connected to all of the server side logs, traces, so on and so forth.

Jonathan: Yeah. Okay. So the, the first thing, and this is, this is just.

I guess sort of a weird thing to say, but I'm, I'm reminded of Microsoft recall and that little, that little bit of a creep factor that people got when it's like, what do you mean Microsoft is spying on me? So you're, you're, you're kind of spying on users.

Jay: Right. Yeah. Yeah. So, so we get that, we get that a good amount, right.

And I think for our, for our customers, like they often ask that question, but that, that. The, the, the default thing there is that first of all, when you install highlight by default, we actually mask a lot of the data. Okay. So any of the texts that gets rendered, we actually will obfuscate it before it gets sent over.

So things like emails, things like numbers, like if it's a series of numbers, we just won't record it. And there's like a, kind of a long list of things that we just don't record by default. And then obviously if folks want to go even more strict. There's an option called strict privacy mode, which it actually obfuscates all text.

And so we try to keep like sane defaults that way. We're like really respecting our. Customers, end users, privacy, right? And that's kind of how we think about it. And all of that obfuscation is done on the client, right? So it's never, it never hits our servers. It's not like it hits our servers and then we mask the data.

We do that all on this, on the, on the browser side.

Jonathan: What could possibly go wrong with the idea of masking things on the server side?

Jay: Yeah, exactly.

Aaron: I was just looking as you were talking in, you know, certifications and compliance, because that's obviously a big part of this too. I mean, there's just general privacy concerns on the customer side, but then there's also like payment information and all that kind of stuff that you have to obfuscate.

So yeah, I was just looking at what your Compliance certifications are so,

Jay: yeah. So we have all like the, the, the default, I guess, compliance certifications. We also have a handful of customers in like the healthcare sector and kind of more, more more regulated industries. I think the, I think the other thing about it, Aaron, is that like, because our product is open source there, it opens up a lot of possibilities for our customers where they don't actually have to send us data in the first place.

And they can almost pay us like us. Like a traditional licensing fee run the product in their own VPC. And then it's as if they're just sending data to themselves and it's totally fine. Right? Because. Because anyway, they're still storing that data that's being rendered in the UI in their database.

And so that kind of like solves a lot of those problems, but right, you don't

Aaron: have to worry about sending it to a third party. Oh, I'm using this service. I'm just running it myself. And then, yeah, yeah. That's a very common situation to run into companies that are like, no, we can't use a SAS service, for example for those reasons, right.

We have to, even as easy as it would be. And as, as much as we would love to hand you our credit card and let you bill it monthly, We have to have access to the bits because we just, we just aren't allowed, you know, you can think of government, obviously government Situations, healthcare. There's just a lot of companies, financial.

They just, their, their policy is we can't do it. So we have to run it ourselves. So that's cool. And open source provides for that. And of course that's, you know, that's going to be a big part of our discussion today, but we'll get there in a minute. I want to talk, I want to talk more about the product first.

Okay. Okay. Let's do it. So, so who, who who would you say your competitors are in this space?

Jay: Yeah. Our biggest competitors, our guests, I guess, are the other companies in like the application monitoring realm. Because they have recently come out with, I guess some of the competitors have come out with session replay.

Some of them don't do it in like the same way we do, but they do have it. One of the companies in that space is like new relic, for example, they just came out with a session replay offering. There's another cup, I guess if you're familiar with log rocket, They have like a session replay offering, but they focus more on like analytics and product analytics and stuff like that.

I think the big thing about what we do differently though, is like that whole connecting the client to the server, right. Being able to like piece together everything that happens for almost from like a user journey. And so if you look at New Relic's offering, it's pretty, like, isolated, I guess, and I think a lot of the big players are like that.

And so we kind of sell our product as, like, one big product, rather than, you know, multiple SKUs, and that's kind of how we think about how people should use this sort of tool. Does that make sense?

Aaron: Well, totally, since I used to work there. Yeah, we had it all, we had, we had all those products separated, and they combined them into one UI.

But I think they still sell them differently. So you had, I work specifically on infrastructure monitoring,

Jay: but then we

Aaron: had APM, right, which was your application monitoring, we had synthetics. Right. Which was like your testing and then we had user experience. So you could get your, so exactly what you're saying, like you have it all streamlined into one thing.

And the nice thing about that is you have a nice journey for what your application is doing. So here's what happened at the front end. They click this button. And then here's what happened in the middle as it was, you know, traveling to the cloud or wherever your infrastructure is, here's what happened on the back end.

And then all of a sudden there was this error. I'm assuming you can show that because you've got metrics here. I see. Yep. And then that's why this particular thing was so slow. And that's why your users saw the little circle going around for a minute before, you know, that's the thing that drives us all nuts, right?

Yeah. And I.

Jay: I think the interesting thing there also, like on the business end is we've, we initially built this for engineers. Like our original idea was like, Hey, we just want to build like a session replay tool. That's built for the engineer. Like how, how can we like reproduce the dev tools of the browser, but then give them the visual power that they would get with one of these like session replay marketing tools.

Right. That was the kind of original vision. But I think Over time, we realized that like, once people get this in the hands of their business, it starts to spread in kind of weird ways. So for example, you mentioned like being able to understand why the spinner doesn't stop spinning, right? That's actually very relevant for like a support person, right?

So if I'm a support person at a tech company, and I get like an inbound ticket that says, whatever spinner wasn't spinning, right? I can actually use highlight to help The engineering team triage before they even get the issue in the first place, because perhaps it's not actually an engineering issue, you know?

And so I've been in, we've been in like demos with support and engineering to kind of teach them how they can work together to kind of fix issues faster with a tool like highlight, which is kind of cool. And I don't know, it's kind of awesome that like the tool itself. Broadens the market in some sense, right.

Where we can kind of start to sell to other, other personas throughout the organization. Yeah, exactly. And there's a

Aaron: huge business benefit too, right? Cause you don't want customers abandoning their session. You want to be able to actually show, no, we're making these transactions faster. We're doing more business, you know?

So once you get past the technical part of it, then especially if you can show that, Hey, we, you know, accelerated or, or you were able to fix your problem faster or whatever. And that means you can do more business. Then it becomes a no brainer. It's like, yeah, we need this because yeah. So

Jonathan: yeah, I'm real curious.

Is there any way our people do you using this and like their continuous integration, their test suites is I can, I can imagine. And I, I sort of have this past weekend's debacle on the mind where someone, a very large company pushed out an update and, you know, soft bricks, like 8 million windows machines around the world.

And I'm just, I'm just thinking now, you know, more of us are thinking about maybe we shouldn't test in production and we should have continuous integration and do some of these tests. So I'm curious, is this, is this a piece of that? Can I, if I have a web app, can I put Highlight in my CI pipeline? And then, you know, if something fails or maybe even it doesn't fail, but Highlight says, This took 30 milliseconds longer in this run than it did last run.

You know, are those, is that sort of tooling there?

Jay: The tooling is there, but it is not, I would say it's not a common use case. And I think it's just more of like, we're a pretty small team. So it's just like what we focus, like what we want to focus on in the short term. We have customers though, that do run highlight in their testing environments and then kind of like tag specific data based on the test that's being run.

But I would say it's like not super purpose built for that. Yeah. And I think Aaron was mentioning synthetic monitoring earlier. Like I think actually a long term vision of connecting highlight to your synthetic monitoring infrastructure, whether we build that or whether we partner with someone else could be a really cool.

Product, you know what I'm saying? Cause most of these synthetic monitoring tools essentially like run puppeteer or like a headless browser and then like just record the browser and then maybe record some logs and all that fun stuff. But the cool thing about highlight is it's, we're just replaying it.

Right. So you'll actually get all the Dom insights and then you get the connection to the backend trace. And so, yeah, I do think there's a world where, where that could, that could work out really well.

Jonathan: Yeah.

Jay: Yeah. How long, how long has highlight been around? Yeah, so we've been around for like two and a half years now.

And our team's about 10 people strong, strong and mighty 10 people.

Jonathan: That's a good, that's a good size team. You know, you can, you can manage a team like that fairly easily. You get too much bigger than that. And it like, it gets, it gets sort of top heavy trying to keep everybody on the same page. I think, I think, like about 10 is, is really for, for something like this, especially is, is really pretty ideal.

Jay: I agree. I think it's like a. You always think that, Hey, we throw more engineers at this problem and maybe we'll, we'll be able to build more. But I'm, I, every day I believe more and more that that's really not the case. You know, it's like two, one to two engineers on a given project is like the, is the sweet spot.

And I don't know, I, I'm scared of the day when we, we want to become a bigger team because I just don't know how the management structure will look and all that stuff. But for now, yeah, I'm really happy with it and it's been awesome so far.

Jonathan: So what, what was the what was the, the problem space originally that you guys wanted to find?

Like, were, were you seeing, you know, application errors? Were there security problems that you were trying to catch? Was it just performance? Like, it seems like you can do all of these three things. I'm just curious, which one was the first focus?

Jay: Yeah. So, so a little bit of background on the company is like, I, before Highlight, I actually went through Y Combinator with another company.

And At that time, we were working on kind of like a, maybe it's not worth going too deep into actually the startup itself, but there were a lot of companies with us in our batch that were using a like marketing tool, like hot jar and connecting it to like a error monitoring tool, like bug snag or century or something like that.

If y'all are familiar with those tools, right. And. It was kind of like hacky, right? Because it was like on in, in the bug snag tool, you can get the client ID of the session and then in the hot jar tool, you can also get the client ID of the session and you have to sort of piece that together. But it was very like, Hey, a customer comes up to me and says, I have an issue.

And I, as a startup founder, want to be able to fix it really quickly. Right. Cause I don't have many customers in the first place. How can I do this? And so that was kind of the problem statement, honestly, honestly, initially, and then I think from that, that's where we started to kind of find the enterprise value among larger customers over time and that sort of thing.

So that was kind of the original idea. And honestly, it hasn't changed much, right? Like. Connecting the client to the rest of your observability resources is, is our motto, like that's what we want to focus on. And now only recently, I guess the last year or so we've gone more into like server side monitoring and all that stuff.

But I do think that the general principle of wanting to connect all of these source sources is it remains true, you know,

Jonathan: So I'm curious, you had this problem, you put together your own little tool to solve your problem. At what point did you say, we should just release this open source? Was that like from the beginning you wanted to do it that way?

Or was there kind of an aha moment of, this could be useful for other people, let's set it free?

Jay: Yeah, that's a good question. So, we, we, originally it was not open source, it was closed source. But I don't know, I've, I've worked in a lot of companies that were very pro open source. And so even all of our infrastructure components, we never really relied on like managed services and things like that, which looking back on it, it's like, I guess, convenient that we did that.

Right. But yeah, the first year, year and a half, it wasn't open source. And then we started to kind of go more into enterprise. Like we got a few customers that were like, you know, like more than 200 person teams kind of thing. And at that point, I think we realized that it would be beneficial from more of like a convenience perspective for these customers to be able to self host, highlight, manage versions of their own highlight instance.

And then I think the other thing that like me and my co founder did is we reached out to all of our customers and we were like, Hey, if you, if we, if we open source this, would you self host it on your end? In other words, we were kind of trying to figure out whether they would stop paying for our SaaS.

Yeah. Our SaaS provider. And none of them said yes. So none of them, none of them wanted to actually open source it. So to us, it was actually pretty it's pretty de risked, right? Like we were like, okay, if we open source this, it's not going to change our existing business among all the startups that use Highlight.

And if anything, it'll just increase trust, right? Cause they're seeing the open source project and they know what we're working on at any given time. And then it gives us an opportunity to really focus on getting in those larger accounts and kind of proving the more self hosted software licensing kind of model, you know what I'm saying?

Jonathan: So I'm, I'm curious. You went, you went open source. What has the buy in been since then? So as, as some of your customers pushed code back in have you had in, in, in the interim, have some of them gone to self hosted? Have you, has open, has open sourcing the product lost you customers or has it been all upsides?

Jay: Yeah, good question. I mean, I don't know if I can say it's all upsides. I think there's, There's a lot of work when it comes to an open source project, just in maintaining it and accepting PRs and all that stuff, right? Like you kind of open up another level of, of committing to the project, I guess, right?

But I would say in terms of like business value, it's definitely been all positive. Like we haven't, yeah, we haven't lost any customers on the lower end for sure. And then on the higher end, like we work with some of like some very large health and health insurance, health care providers, we work with very large hedge funds and banks.

And so I do think that that we've proven out the fact that that's been worth it for us. And obviously, you know, we can always be putting more into the open source project and marketing it and getting more contributors and things like that. But so far it's been working really well. And then on the contributing end.

We've had maybe like 20 30 contributors at this point outside of our company.

Jonathan: Oh, yeah

Jay: So it's been a pretty healthy honestly growth on that on that sort of end as well

Jonathan: have have any of those contributors been like Continuing repeat contributors to it or they mostly drive bys because I would imagine I would imagine on a project like this You would get a lot of drive by like High quality, but drive by contributors where it's a business that's using it.

That says, wouldn't it be nice if highlight would also do this, or we, we have this specific bug that we need to fix. I imagine there's a lot of that.

Jay: Yeah. Yeah, there is, there is, which is kind of cool. Right? Like if you're, if you're a potential customer of highlight, you can be like, Hey, there's this issue with this SDK.

Can I come and fix it? And we're happy to happy to accept it. Right. So that's kind of cool. But, but on the on the contributors end for like drive buys, I would say maybe 80 percent of folks that come in are more drive by and are not like repeat. But we do have like a good group, small group of folks that are pretty consistent.

And we've put in a lot of work to like incentivize them to keep helping us. For example, we have like a bounty program. I don't know if you've heard of Algora. No. Okay. It's not particularly. It's like a SAS product that you can connect to your GitHub. And then we basically pay Algora a percent of every transaction that we pay to our contributors.

And so our contributors can land on a PR and they'll see that there's a bounty on the PR and then they can help us with that particular feature. So it also helps us prioritize what we want to be contributed to, because a lot of the repeat contributors, they're just working on what they want to work on, but if we can incentivize them in the right direction, it's kind of cool We can align it with our product roadmap too.

Aaron: And that's great for, for people just getting started. I always yeah, you know kids I'm old enough to call college age kids, kids getting out of school there. They, a lot of times they'll ask, how do I get started even in high school? I mean I did a, I did a high school. Talk, you know, about startups and working in technology and things like that.

And that was one of their questions is like, we're really afraid because we're going to go invest all this time and money into college. And we, you know, we keep hearing that it's really hard to find a job. It's like, well, that's what, you know, open source is great for that. Right. Yeah. Yeah. And it sounds like this is even better because now you're getting paid and you have a system instead of just going out and searching GitHub for what you want to work on or, or areas where you can work.

Now you've got a nice system where you say, I want to work on, Monitoring, right? And now you can find those problems and try to solve them and get a little money and get some experience on your resume. So I think it's awesome.

Jay: Yeah. Yeah, it is really awesome. And that Algora team I mentioned is, they're also a startup, so they're very open to feedback.

Like we've kind of worked with them to, to get some features in that would help us kind of manage and triage and all that stuff. So I do just a small, small shout out, like to check out, check out that project. I know there's a lot of open source folks here. But it's a, it's a pretty cool project for sure.

Yeah. I'm going to bookmark

Jonathan: it. I'm, I'm looking at the Algora page and get hub too, because there had, there have been groups that have done this, that have done bounties over the years. And some of them kind of eventually dry up and I forget the name of the one, but there was one in particular that just sort of quietly stopped paying bounties out and I don't know that we've heard much from them ever since.

Oh, I see. I see.

Jay: Yeah. That's pretty tough.

Jonathan: That's tough. But

Jay: it is, but I mean, it's, it's, it's kind of cool, right? Like it, yeah, it's, it's pretty awesome for us that. We can we can help those folks out because I think it also Even beyond like helping that small group of folks that are repeat contributors it also grows our Pool of contributors that just want to be repeat contributors, you know, so it's been awesome for us and I really like that model that's been working out so far

Jonathan: Yeah, and so when when you guys decided to go open source, what was the what was the license that you went with?

Yeah, we went with apache

Jay: 2. Which we're, I think, still pretty happy about. I don't think we make plan, have any plans to change things. The only thing we've changed, I guess, on the licensing model is being a little more explicit about what we charge for on our open source license. Our, our, our open source distribution versus what was, what is free.

So when we started the project, you could just install highlight. And get it running in like a Docker container and all the features in our SAS product were accessible in the self hosted distribution. But now what we've done is we've actually, it's a little more open core. If you want to think about it like that, where certain features like, like RBAC and SSO and a lot of the like more super enterprise y features we've gated on the open source distribution.

But You as a team, if you want to just try it out and even if you want to use it long term, we still have a lot of traffic, honestly, on our open source installs. You can kind of get going with just the core functionality. Does that make sense? Yeah.

Jonathan: So I was, I was going to dive into this. We can go ahead and talk a little bit more about it.

Like what, what the. What the pricing model is, like how, so as we like to say here on the show, programmers need to eat, even if they are open source programmers, they have rent to pay as well. And so, yeah, I'm, I'm curious. Let's talk a little bit more about that. Like what, what is the revenue model? Do you have, are most of your customers still sass where you host the product for them?

And and let's, you know, let's dive into that.

Jay: Yeah, I would say more than 80 percent of our revenue comes from sass still. I do think that the pool of self hosted folks is starting to grow. Just given the investments I've, I mentioned the investments we wait, we've made in the, in the sort of open core self hosted kind of model I mentioned before.

But yeah, the way we charge is like on the SAS product, it's like 50 a month for like an indie person. That's like just getting started. It's like 50 a month. And then you just pay a usage fee on top of that. So if you're sending. 10 million logs, you'll pay an additional a hundred bucks a month or something like that.

And it's pretty simple. So that's kind of like the, the base tier that we offer. And then on top of that, we have a couple more tiers for like larger businesses and larger business sizes and things like that.

Jonathan: Yeah. Yeah. And I, I

Jay: assume that

Jonathan: you, you make use of the highlight client yourself in, in all the, you, you eat your own dog food to some extent.

Jay: do, we do. Yeah. And, and that's. That's been pretty cool. Actually. We actually funny story is like early on, we used to demo highlight installed on highlight and it would just confuse people so much because it's like, you're looking at the highlight UI and then the highlight UI is the highlight UI and you're clicking play on it.

So that's maybe a bit of a regret early on. Cause like people were just, didn't even know what they were looking at. But yeah, we do use highlight on highlight and it's been pretty cool. We use it for obviously all the features and we monitor our own app. We actually have our other, like right now well, yeah, right now we use we use the session replay tooling and we connect it to a lot of our like server side tracing and things like that, and it's been pretty cool to demo to our customers that we do this ourselves.

And we almost use that as kind of like what a demo environment looks like, even though we kind of replace what the screen looks like. So it's been, it's been awesome using highlight. Cause it's like. We can, we can just prove to folks like this is how it should be done and what it could be like if you use something like this, you know?

Jonathan: Yeah,

Jay: absolutely.

Aaron: How hard is it to get started though? Like, is this something that you could go in and do a POC at a customer pretty easily and just say, Hey, pick one of your applications, we'll get it installed and we'll see how it goes basically. That's always well, that's one of the ways I find that, you know, when you're trying to sell this stuff and get people to buy it,

Jay: if

Aaron: you show them how much value they can get out of an application they're used to, to your point, nobody knows highlight on highlight.

Right. Then you know, it just, it just makes a lot more sense to them at that point.

Jay: Yeah. Yeah. It's very easy to get started. And a lot of our early larger customers came in, we did like not even reaching out to us. Like they just came to our landing page, made an account and started sending data through.

And then we kind of went through more of an enterprise buying cycle or whatever. Right. So yeah, a lot of our customers do the POC without even us knowing, and then they come out to us and they ask more questions and that's great. But, but when it comes to more. Larger customers at the get go. Yes, we get a POC going and we can just have them install it on their own application or one of the applications they have.

And because it's so simple, it's like at the application level, it's very easy to get things going. Whereas I feel like with a lot of infrastructure monitoring, you have to install agents, DevOps needs to be involved. It can be like kind of very hands on. For us, it's pretty nice. Cause it's like, Hey, these are the three code snippets you need to add throughout your application.

And you'll be able to see a session connected with a log on the backend. And then from there, they, the, that's like enough to get them to a point where their imagination can kind of take them to what, what, what capabilities they want, you know what I'm saying?

Aaron: Yeah, absolutely. Yep. And it can be difficult, I think with, cause you're kind of like, You're up against a few different areas with this product, it seems like.

And what I mean by that is like, I'm just going back to my time at New Relic where you would have to install multiple agents to get the full experience, right? I mean, you would have to, okay, we're going to use, you have to use it. You're writing your application in Python, so you have to use this Python library.

And then for Synthetix, you have to set up this whole environment and install software there. And then for the backend stuff, you have to install another agent on your server. And To log what's going on. Oh, and by the way, there's logging. Are you going to send that to Slack? What are you going to do? And so all of a sudden it seems simple at first.

And then as you go through, in this case, let's say a POC, cause that's what I'm used to thinking about this in the customer kind of gets a little bit of buyer's remorse because they're like, Oh, this is actually a lot harder than you told us it would be. Or you made it out to be, you know?

Jay: Yeah. Yeah. I think, I think the, and this is like maybe more of a philosophy, philosophical thing as well.

I think. The future is going away from infrastructure monitoring. In my opinion, like I think, you know, with, with serverless concepts and the fact that, you know, now you're not managing your own containers, any, any given startup today, right. That starts, they're not dealing with Docker containers in production.

Maybe they do it locally or whatever, but I think that's, That is maybe the the argument for why application monitoring is a future and us kind of focusing there is where, where we'll like start to, to build more and more value for the market. On the, but yeah, at the same time on the POC end, I, I, I, it makes sense that like for infrastructure, it takes a lot of work and there's like.

An SDK in this case, cause tracing, you need application level stuff, but resource consumption, you don't need application stuff, so it's just an agent. And it's kind of like a lot of random stuff. I think the nice thing about what we're doing is it is very focused, right? And people can install and send a little bit of data to get, to have an idea of what the full value picture looks like there.

Aaron: We should talk a bit about AI too, before we get too far into this and how you're incorporating that. I mean, everybody's mentioning it on their pages, but then I also want to just to set up a further topic. I do also want to understand like other, like you mentioned that you work or it seems like you're, you're, you're open to working with other open source.

Projects and things like that. So I wanna talk about ecosystem as well. Mm-Hmm. . But let's start with, let's start with AI since that's the the, the, the site guys. Elephant in the room here. Yeah.

Jay: Yeah. I actually, I actually have a, a, a, a a a a twofer for you on Aaron, Aaron on that front where we actually did recently a collaboration with an AI related project with highlight.

So I'll tell, I'll, I'll tell you about that after. But yeah, on the AI end, I feel like I may be in between y'all two where I'm not too much of a skeptic, but also like, I think we tread a little lighter than maybe most of most of the startups in our, in our, in our world. But the, I guess there's two ways that we've used AI.

So one. Is in ter in terms of like enhancing the existing experience of highlight users. So for example, if you're on a session, we have an AI feature that will actually like summarize the session for you. And we send like an email digest every week by default. But you can turn it on such that it actually will describe what a user did in that session before you click into the email, for example, right?

Mm-Hmm. . Mm-Hmm. . So there's a lot of things about like. Understanding the user journey and the model actually can tell you that, Hey, user clicked on this button, but then had some user frustration at this timestamp. So that's kind of cool. It makes it easier for you to triage issues when it comes to your sessions.

The second thing that's kind of similar to that is like on errors. So if you have like an error and a stack trace, we will actually suggest a fix to your stack trace. So we'll say, Hey, on line five of this stack trace, maybe Instead of using nil use actually initialize the value or whatever it is.

Right. Because that's probably where that air is coming from. So those are kind of things I guess. And there's a few other examples that we've used to kind of augment the existing spirit experience of highlight. And then one thing that actually we're launching next week is more of like making highlight, creating highlight power users faster.

And so one thing we've added is the ability to convert. Plain text queries into structured like into our structured query language. So in our log viewer, for example, if you say all logs with level error from three days ago to yesterday and click enter, we will actually convert that plain text query into like a structured query that you can then like run against your log viewer.

So yeah, lots of cool things on the AI end that we're thinking about. Thanks a lot. And I guess, yeah, they're, they're, they're though in two places, right? One is like making it faster for you to get value out of a highlight early on and learn the query language and things like that. And then the second is more like augmenting the experience.

And I actually think that maybe, and maybe this is a little controversial. I think the former is more powerful at least in the monitoring realm right now, because I think that the whole monitoring world is a lot about, it's a lot about accuracy and exact, right? Like you want. You don't want people like summarizing your log lines, right?

The reason you log something in the first place is because you want to know at what timestamp it happened and what exactly was being said. So I, I think it's, we're still early on in figuring out exactly where AI lands in this, in this world. But I do think we're making a lot of progress. So that's pretty exciting.

Jonathan: Yeah. I'm curious what what language is Highlight built on? Please, please don't tell me it's Java.

Jay: It's not Java. It's not Java.

Jonathan: Somebody emailed me. I said something slightly derogatory about Java, and somebody emailed me like, hey, by the way, we do Java if you want to talk about it.

Jay: He's picking a fight.

He's picking a fight with you. No, no, we write Go and JavaScript. So, like, our front end's all in JavaScript, and then our back end is, I think, exclusively in Go. With a little bit of Node. js here and there on certain testing things, but yeah.

Jonathan: Is it, is it mainly built on, on Node. js? Like, what's the, what's the framework that, that highlights built on top of?

Jay: Well, yeah, I actually wanted to talk a little bit about that too, but, The framework, like the web framework, is that what you're talking about?

Jonathan: Well, I mean, so the whole thing, is it, is it primarily a Node. js project? Or is it like written from the ground up with the Go backend? Or, you know, is it a, is it a module that runs on top of Apache?

What have you? Got it.

Jay: Yeah, yeah. It's written from the ground up in Go, actually. So it's essentially like a, a few services written in Go. We have like a main service that collects monitoring data and things are like buffered in Kafka. And that's kind of one of our biggest, like, I guess databases that if you want to call it a database that we use before things go into the rest of our system.

And then we have like a set of services that read from Kafka and then write to ClickHouse, which is basically our actual data store that we use at the end of the day. So those are maybe the two big projects that we use and depend on, on the open source world. And then I think one thing that's might be worth notice note noting on the open source framework side of things is like are you, are you all familiar with open telemetry?

Yeah. I was just going to, that's what got

Aaron: me down the road of the next, how you're doing integrations and does this work with Grafana? Can I set up dashboards anyway? Yeah.

Jay: Yeah. So that's one thing actually that maybe Aaron, you know, a lot about because I think a lot of the bigger monitoring vendors have started to kind Talk a lot about open telemetry, right?

It's like one of their biggest marketing marketing verbiages when it comes to putting the word out. But I think actually it's really beneficial for us in particular, a small startup, because it's made it really easy for us to be able to support so many languages. So for example, a few months ago we were at a conference out here in Seattle called Microsoft build.

And obviously there's a huge ecosystem around T net, right? Mm-Hmm. And we don't have a T net SD K. So we showed up to this conference kind of a little like, let's see what happens. And turns out, one, we got a lot of meetings from the conference in terms of conversations and getting POCs up and running.

But two, we could just kind of point them to the open telemetry documentation. And now if you look at do nets, open telemetry docs, and highlight. It's actually just a few small steps above the OpenTelemetry documentation to make it like an easier experience on top of Highlight. So that is one project that I think we're really thankful for.

And yeah, I think the, the, the, the team at OpenTelemetry has been doing a great job in terms of Getting all of these languages supported and we're trying to help a lot on that front too, with kind of contributing back to the client side SDK as well, and kind of I guess, yeah, promoting that environment for the most part.

So it's been awesome. Yeah,

Aaron: that's cool. For those that don't know, I mean, open telemetry is a, is a project that actually started out as open tracing, I think, but it was an open census. Oh, OpenCensus. I forgot about OpenCensus too. Yeah, Yeah, yeah. They combined them. So anyway, it was, it's really an open source way of getting information about well, really anything.

But, I mean, things that you want to use tracing for. So mostly, like, applications and things like that. And so, again, it provides this common standard, right, that you can use. So it's like, oh, your application supports OpenCensus? telemetry. Great. We support open telemetry. We can ingest that data or whatever, do whatever with it.

And you can do it in an open way and you don't have to, Oh, I've got to pay a license to, I'll just call it one of my former, you know, app dynamics, let's say, right. Yeah. Which is Cisco. I don't know if they've renamed it now, but you know, I've got to pay a license to them in order to get that data into another application so that I can use it or read it or massage the data or whatever, analyze the data.

So it's just a, it's just a great leveler, right? And levels the playing field for people that are doing that. So that's great. I'm glad it's working out for you. I know that we didn't use it in the companies I've worked for in the past. We didn't use it for so long because of all the turmoil and customers are like, we're really excited about open telemetry, but we're not quite ready yet because there's things going on.

We're not sure what's going on. There's a lot of uncertainty.

Jay: And I feel like you're always kind of going to have that problem to an extent, right? There's always be like the long trailing. That's the long trailing elixir and Erlang and all these languages that like, you know, they're a little far behind on the open telemetry bandwagon.

But I think for the most part, actually, it has a lot of coverage nowadays and you're seeing it in new rel, a lot of new relic and what they say. There's a few other companies that are really kind of milking, milking that the concept of open telemetry and helping educate people. And honestly, it helps us too, because it's like, if you want to try out something.

On the application monitoring end and kind of go away from your existing provider. It's it's a good option to kind of try things out So

Aaron: yeah, 100 percent avoids that lock in. Like I was saying, we're exactly exactly

Jonathan: So this this is a good place to jump in and ask something it I I want to kind of dig around a little bit in the the different things that you support in that you can put highlight on top of and I'm gonna, I'm gonna throw out some probably weird examples, but I think it'll, it'll kind of maybe put the boundaries on the space of what you're, what you're looking to do.

So, can you install highlight on top of a, WordPress website and get information about people's interactions with WordPress. Can you do it on top of something like a Tari app? I'm not sure if you're familiar with that framework. Can you do it with something that's running like a flask, Python flask back end?

Can you do it like, so just some, some weird things that I'm curious and maybe, you know, maybe OpenTelemetry helps a lot with all of this, but I'm just kind of curious of like where the boundaries are, where this would make sense to try to use.

Jay: Yeah, I'm not familiar with Tari, but I feel like, is it T A T A U

Jonathan: R I?

It's, it's taking a web app and making an application out of it.

Jay: Yeah, makes sense. So I guess what I would say on that front is like the, the, anything that runs in the browser can be, can install our client SDK. So we do have customers that use highlight in like Weebly and WordPress and Squarespace type applications.

That might hit like a, a, a proprietary back end that maybe another team at their company is using. And the power there is that like, if I have a front end team that's like using this no code tool but someone clicks a button and something breaks, they can actually attribute it to what might be happening on the back end that's going wrong, right?

So yes, the answer is yes to the first WordPress thing. Tari, I'm not too familiar with.

Jonathan: There's a, there's a, there's a more common solution than Tari. And I, I am absolutely blanking on the name of it. Is it

Jay: Electron?

Jonathan: Yes, of course. Electron.

Jay: Yeah. So we do, we haven't, we have actually Electron documentation.

And because that runs a web browser in a desktop app, it's a very good fit for highlight actually. And so, you know, companies like Notion and. linear and those types of tools could definitely get benefit from highlight. And then the third thing was a flask app and yes, that's our bread and butter.

So we, our, our CTO is a big Python guy. And so we have a lot of, we have a lot of tutorials on getting this up and running and fast API flask. And a few other like common Python web frameworks.

Jonathan: Cool. Is, is there any, is there any of this that you don't support that you're kind of thinking, man, it would be nice if we could run on, and I don't know this space well enough to even throw a name out.

Yeah, I, I threw my, my big guns out and you, you already support them. I passed the

Jay: test.

Jonathan: Yeah, I guess so.

Jay: No, no, I would say yeah. One thing that we're eyeing a lot is like mobile, mobile recording. It, I think the reality is it is a very different ballgame though. Like, you know, with our 10 person team, I think we would need to have actual iOS engineers on our team or, or Android engineers on our team to be able to even kind of start to, to build that.

So we are thinking about that. And I think every once in a while we'll close a customer that is like, Hey, When you turn, when you, when you get mobile recording working, let us know, you know, and so we'll see, we'll see if we get like how fast we get there, but I think eventually we'll get there and I think that's the next maybe milestone that we're looking at for sure.

Jonathan: Yeah, that's interesting. Is, is there anything else on the radar that you're, that you're looking to do either short term or long term? Want to chat about?

Jay: Yeah, I mean, I feel like the big, the big thing is maybe mobile recording, the other like smaller things like. I guess we have like a big launch week next week.

And I'm actually working on recording a bunch of videos for that. So y'all are helping me practice for the video. Yeah. But yeah, like we're, we're launching a bunch of really cool things. For example, the AI stuff I mentioned for building queries. We're launching a like metrics drill down product or feature.

Where like on an actual metric, you can actually click into it and see all the sessions related to it. So if you're like visualizing users grouped by a specific URL, right. If you could click on one of those bars that you've, you've created, and then you can see all the URLs or all the sessions of those users on those URLs.

It's pretty cool. Same with logs and all those kinds of stuff. So we're kind of really milking that like cohesion feature, like connecting everything in your, in your product. And then I think even more longterm, the other thing we're really excited about is like more product analytics and helping people understand user engagement in their product and still sticking with the application level.

But, but going more towards like support and the, the product type folks at an organization. So yeah, a lot of, a lot of fun things. And if anyone's interested check out our Twitter channel or our Twitter account for next week, cause we're doing a bunch of launches and that could be fun.

Jonathan: Yeah so if somebody wants to get started, wants to get it installed, is there a quick start guide somewhere you would point people to?

Jay: Yeah, I mean honestly if you just go to highlight. io slash docs That's the best way to start. There's a get started button there, which will take you to all of our language specific docs.

And then you just pick your language and get started. Alternatively, you can just go to our actual app and click sign up at the top. And that will also, it also embeds the docs. So it's pretty easy to get started. And, and if anyone has questions, we also have a discord that they can kind of jump in and ask questions about.

Aaron: Yeah, that's nice. It's always kind of a pain for me when you go. You want to do like a free trial of something. I mean, I'm assuming if you sign up, I mean, you just get the free version or is, or do you get a trial of some of the more advanced features?

Jay: You actually get everything. Yeah. The only thing you won't get is like, yeah.

Like you only, you won't get like our back and SSL. So yeah, you'll just be able to try it out. And it's very hands, hands off in terms of having to talk to us and things like that.

Aaron: Yeah. But anyway, what I was saying was easy, easy access to documentation as you're going through the process, like the onboarding process for people is super important.

Yeah. And I hate it when I do, Oh, free trial. And then I have to open up several browser windows and try to figure out, you know, and you're going back and forth. Oh, here's a PDF. Now I've got to like read a PDF. Yeah. So yeah. Yeah. No, it's nice that you've done a little document integration. That's great.

Jay: Yeah. Yeah. And we're, we're always trying to make that better actually. Yeah. We're this upcoming few months, we're going to be doing some work on, on the docs to make it a little more language specific rather than product specific. But I mean, nonetheless, it's very easy to get started. And hopefully folks can kind of jump in and get some data flowing.

Nice.

Jonathan: Aaron, is there any, any other questions you want to cover before we start to wrap?

Aaron: I mean, I could go on. I mean, you know, I could go on for a few hours here. We could talk about customers. We have a couple, we have a couple of

Jonathan: more minutes before it's really time to wrap. So if you, if you have a couple more queued up, then go for it.

Aaron: Well, I was going to ask, like, is there, I was looking at the customer page cause that's always, it's a good place for people to start. Like what are customers actually saying? And I'm in marketing, so I know, you know, there's, you know, some marketing magic that happens there, but But I do have like a customer story, I guess, like that where someone has used this and then come back and said, wow, this really helped us do X, you know, we couldn't do it before we were having this problem.

And wow, this really helped us get over just to kind of illustrate to people, like the potential, I guess.

Jay: Yeah. I mean, honestly, if you go to that page, Aaron, that first customer's testimonial from that healthcare company is a great example of like a very large company that I guess historically has been a little more slower in their ways that has kind of used highlight to actually get a lot more visibility into those older applications.

And I think it's actually a function of the fact that they can install this at the application level. And they don't really have to touch a lot of the infrastructure stuff. Right. And in those cases, I feel like it's awesome because they from, from having just like logs, for example, just logs and metrics.

They're being able to kind of move to using more modern technologies like next JS and all of like sort of like JavaScript isms that are kind of becoming a lot more popular today and be able to monitor that monitor that with something like highlight. And then at the same time, actually connect that to those older services that these, these, these applications depend on, right?

So that's kind of like, I guess, a very, like a, almost like a perfect enterprise story in terms of folks being able to get value. And then on the earlier end, I feel like with, with smaller companies, we do have a lot of customers in like the modern JavaScript ecosystem that use us. And it's a lot more, it's a lot, a lot, it, a lot of the value comes around connecting the client to the server, like I mentioned earlier, right, where if I use another tool, I'm using a bunch of like, I'm using four siloed tools that are not very well integrated.

And with highlight, I can kind of get a more comprehensive picture of what's going on when I'm debugging an issue.

Mhm.

Jonathan: Yes. So we do have a bit of a live chat room and we got a question just now from I happen to know a flask developer that will probably be very interested in this. And he wants to know, are the front end javascript plugins trapped or blocked by default by any other plugins?

And he mentions privacy badger, ublock, etc. And that's, that's, that's interesting. Like what's the, what is the interplay between the highlight tools and some of these other like privacy focused tools?

Jay: Yeah, that's interesting. So we were, I guess we, when we started highlight, we were we were not looking forward to the day that you block considered us a, a, like yeah, like an ad snippet or whatever they call it, right?

Unfortunately, you block now considers us an ad snippet and blocks us. But to, to, to solve the, for that, like we have a few options, right? One is you can actually like. Proxy your highlight requests through your domain. So if you want to run, let's say you're, you work at acme. com. You can proxy all of your outgoing highlight requests through highlight.

acme. com and you just need to share with us some DNS credentials and we can get that working. And then we even have like a more hands on approach where we will actually proxy it through like a cloud flare worker, all the data. And that's a kind of an even better way to, to sort of mitigate these blockers, but it is a problem.

It, it, it really is a problem. And we are we think about it a good amount, but it's honestly, at the same time, I get why a lot of these blockers do this, right? Like I think more often than not, they're doing it for a good reason. Right. And we can, we can solve this, especially for the larger customers that have a lot of traffic on those, those two notes that I just mentioned.

So, okay. So I'm just saying, so if

Jonathan: I'm, I'm real curious with, with something like uBlock I don't know much about the people that run the, the uBlock extension, but have you, have you heard reaching out to them? Like, Hey guys, I see that you're blocking us. We don't actually do advertising. Here's what we do.

And I've just, I wonder whether that would bear fruit or if they would ignore you. I honestly don't know.

Jay: I haven't tried. I haven't tried. We haven't tried. Our team hasn't so we should try. That's a good point I I feel like the only problem though is I think their definition is not super Strict in terms of why they would block something like this and I think they do it based on amount of traffic

Yeah, but

you're right maybe Some reasoning and some background on the project and things like that would help kind of make the case.

So that's a good point. It's a good point. I'd be, I would be interested to

Jonathan: hear what

Aaron: happened. Yeah. Yeah. Yeah. Would this also hold true for other ad blocker technologies and, and plugins like ad block or like I use Piehole would Piehole, Piehole block? I guess they would, would they?

Jay: I'm not sure about that one specifically.

I know uBlock we are on like a list. And the thing about these lists is they're like open source, right? So you can find them on GitHub and people basically submit them. And the thing is, it can't, it's not necessarily the company that's using Highlight that's submitting them. It's right. It's their end users.

So if a company has like 5 million users and they're all, all of those users are sending data technically to the company via Highlight. If one of them complains, uBlock might like consider that. You know what I'm saying? And a lot of these GitHub repositories that have the data in them are shared. So I do think that uBlock is not the only one that pulls from that list.

So Aaron, it might be that. Piehole, for example, pulls from the same list, and they just run a job every month to kind of keep, keep track of what's going on on that front. So

Jonathan: yeah.

Jay: Yeah. It

Aaron: reminds me. I'll just set something up and then test for it.

Jonathan: Yeah. It reminds me very much of email servers.

Ending up on spam blacklists and you know, if that happens you let's see I think so spam house is the big one and they've got it automated because this happens all the time They've got it automated you go to their website and you say here's my you know, my URL or my IP address Please remove me from your list And if you're actually sending spam, you know, they'll let you do the remove requests like four or five times and then of course you get like blocked forever.

Yeah, yeah, yeah, yeah. Makes sense. I wonder whether these, you know, advertising block lists, whether they are sophisticated enough to have some of that tooling. There have to be, there have to be domains and scripts and such that end up on there that shouldn't be. Yeah.

Jay: Yeah. It's a

Jonathan: good

Jay: point. That's a good point.

And, and I think it honestly is a good learning for you guys to tell me like, Hey, just reach out. Cause we're, we're not doing it. We're doing it in good faith. But at the same time, it's like, honestly, not a huge deal. Like it's like the companies that really care about this and they're like, Hey, we're missing these sessions.

They'll reach out to us and then we'll help them out with that proxy stuff that I mentioned. And it's not a big burden. But I imagine, you know, right now we have like 300 customers. We have 3, 000 customers. It becomes like a bit of a different story. So, yeah, yeah, yeah.

Jonathan: Okay. I've got a, I've got a similar question to the one Aaron asked.

He wanted to know a success story. I want to hear a weird story. Now, like ask, asking people, what's the strangest thing that someone has done with your product? So where, where has somebody used highlight that just surprised you or you find odd and not, not every project has one of these stories. But if you, if you have one, I think it'd be fun.

Jay: Yeah, so I don't have a story of a customer using it in a weird way, necessarily. But, one thing we used to do in the early days when we were getting our first customers, is we, So Highlight is just a JavaScript snippet, right? So you can run it in the background in the browser. And so we would actually go to our prospective customers, and we would go to their websites.

And we would inject the highlight snippet on their site. Oh

Jonathan: yeah, something like a monkey script, essentially.

Jay: Yeah, and so like, we would go to the Floss, the Floss website, and we would install highlight, and then I'd send you an email and be like, Hey, We just installed highlight on this website. Wouldn't it be cool if you could troubleshoot what was going on for all of your sessions, you know?

And so that kind of worked pretty well. It wasn't very scalable because it's just like, I think at the time also session replay was pretty new in the engineering world. So it was like kind of a ripe time to be doing it. But, but yeah, it was kind of a fun story of using highlight to sell highlight, you know, I like it.

And yeah, it's kind of a fun, fun, fun fact,

Jonathan: but okay. So we are getting back towards the bottom of the hour. I want to know, is there anything that we have not asked you about that we've not covered that you wanted to make sure and let folks know about? I know I asked, I asked all the hard questions here at the end.

You got to really think about these.

Jay: No, no, you're good. You're good. I think honestly, just I would love folks to be keeping an eye on what we're doing. We specifically on the open telemetry note We put out a lot of content and webinars and stuff like that about OpenTelemetry. So would encourage people to follow us on, on, on Twitter, if they want to kind of keep an update on that sort of thing.

And then, yeah, if any of, any of the folks from the Hackaday discord want to give us feedback on the product. I will welcome that very openly, so would love folks to, to keep in touch on that front, too. All right. Do

Aaron: you have, like other, other avenues? I mean, not everyone is excited about X, I guess, we have to call it these days, right?

For, for various reasons.

Jonathan: The social network formerly known as Twitter.

Aaron: Right, so I'm just thinking, like, you mentioned Twitter a couple times, like, is there, have you thought about, you know, You know, Mastodon or even like Hey, I'm interested enough. I want to join a Slack group, for example.

Jay: Yeah. So our discord is a good place to be if folks want to kind of just learn about what goes on, on the content.

And we do, we do pipe all of the. stuff and any of the stuff that we post on in discord as well. The second thing I would say is LinkedIn. We're pretty active on too. Those are our two big social channels. Mastodon isn't something we've looked at, but maybe we should honestly it's not too much effort to kind of pipe things to another source and yeah, that's a good call out.

Yeah. I mean,

Aaron: I don't know. I've surprisingly mastered on it. It's not. I don't think it's growing as much as they would, you know, people would like, but the, my followers on Mastodon are pretty pretty active. Okay. They're active. So that's good to know. And like you say, you know, you just use a tool like buffer or something and you just post it at the same time.

It doesn't, it doesn't really hurt. It's a

Jay: good point.

Aaron: It's a good point. And it helps support the ecosystem. I think it helps support alternatives for people that might not jive with

Jonathan: Yeah, yeah, yeah, yeah. Of course, of course. That's fair. Alright, so I've got a couple of last questions that I've got to ask before we let you go.

And that is, you personally, what is your favorite text editor and scripting language? Oh man,

Jay: I'm a, I'm a Vim person. Okay. And I'm a, specifically I'm a NeoVim person. Okay. Yeah, me too. But nowadays I do use VS Code because I don't code as much. And so I use like the Vim plugin in VS Code. Okay. Which honestly has gotten really good since I, since I looked at it, like, I guess four or five years ago.

So, yeah, that's my, that's my that's my text editor. What was the other question? Scripting language. Scripting language. Oh, I'm a Python person for sure. Yeah, yeah.

Jonathan: Yep, very cool. Nice. Alright, well, Jay, thank you so much for being here. Sure, appreciate it. And had a lot of fun learning about Highlight.

And yeah, of course. Yeah. Thank you so much. Sure.

Jay: Thank you for having me. This was really fun. This was really fun. Yeah, it was a blast. All great questions. And I, and I yeah, excited to keep in touch too. Yep. Awesome. All

Aaron: right. What do you think? That's great. This is, this is, this is fantastic. I think it's, you know monitoring, be it application or infrastructure monitoring is a crowded place that's had a history going back to the early days of computing, right?

I mean, you could, you could really go all the way back to the early hackers at MIT that were doing things with you know, PDP machines and stuff like that if you really wanted to, right? Because they had to get, you know, that same telemetry data, although it was a very different world. But so it's interesting to find someone that's doing something different in this space or trying to help solve a problem in a different way.

Right. Cause a lot of people will come along and they say, Oh, we're new and exciting, but they're actually not doing anything different. Right. Except maybe they're a younger company and they're charging you less money. So yeah, to hear someone that's tackling this problem from a user perspective and.

You know, really trying to meet developers where they're at with things like Hey, you can use this without necessarily having to bother your DevOps people or your sysadmins to install some complicated thing that they're going to be like, no, I'm not doing that because that's not in our architecture.

So, so to be able to do some of those things is actually really, really cool. And it does solve a problem that developers have. So yeah, I, I like it. I mean, I, I'll have to actually play with it. I didn't we were talking so much, you know, a lot of times we'll actually try this stuff out while we're talking, so I didn't have time to sign up and do all that kind of stuff, but yeah, I want to play with it and just compare with other products that I've either worked on in the past or used in the past to see how it works.

Jonathan: Yeah. When I, when I scheduled you for today, Aaron, I didn't realize you were such an expert in the space. That that worked out really well.

Aaron: Well, yeah. And I didn't elaborate either in my email. I just said, this is right up my alley. I think it was my response, but yeah. I mean, I've actually worked in the, in this industry specifically for.

Whatever 10 years or something at this point, which is a long time. It seems like and so, yeah, I've used, I've actually worked for these companies that produce these products and actually used a lot of the other products as well. So yeah, it really is right up my alley and you know, was something that was not on my radar, but it will be now I'm going to go follow him on on LinkedIn right now.

Jonathan: Yeah. It's fun. It, you know, for the past few hundred shows, it feels like when I'm on or now that I'm, I'm the main host, I've, I've been the play by play guy and the other person has been the color commentator. And today I get to be the color commentator and you're the expert. You're the, you're the play by play guy.

It's kind of a neat inversion of the roles. That's funny.

Aaron: Oh, I hope I didn't go too deep on anybody for anybody, but I hope it wasn't too much inside baseball, but it was, it was really interesting. Like I said, I could, I could talk and talk and talk like I'm ready to sign up for a day session, right?

Where you go through everything and talk about it. And I went to their careers page too, cause I'm looking for a job right now. They don't have any product marketing openings, but maybe when they get a little bit bigger definitely I will be checking in from time to time.

Jonathan: Yeah, interesting. That, that could, that could be something.

All right. You never know. It's happened before on the show. It is. Yes.

Aaron: Yeah. Yes, it

Jonathan: has. Yes, it has.

Aaron: All right. You have anything you want to plug? Well, of course you should go to my two YouTube channels and check those out. So there's RetroHackShack and there's now RetroHackShack After Hours, my second channel, which isn't quite as scripted, although it's still really interesting.

So for example, all of my e waste Wednesday videos, Wednesday, one of the reasons I can't do, couldn't do this show when it was on Wednesday was because I would go to e waste in the morning and I can only go on that day at 10 o'clock. To be able to buy stuff from this e waste place. And so So what I do on my second channel is I bring that stuff back and just kind of like do an overview.

Here's what I found today. Isn't this cool? Whatever. What I recently did on RetroHackShack After Hours is I actually filmed the whole process. I actually took my phone and, and, and took video and audio and stuff and showed people, here's what an e waste place looks like. Here's what you, I can find at mine.

And here's what I pay for stuff. And, you know, here's the bin of you know, cards, ISA cards and PCI cards and things like that, that they sort out just to show people what the experience is. So in either case, go check out my two channels on YouTube and subscribe if you're interested in that way, you'll always get that content in your feed.

Jonathan: Yep. Awesome. Thank you, sir, for being here. Absolutely.

Aaron: Anytime.

Jonathan: Yeah. So I do want to plug, of course, Hackaday. We've got the security column that goes live there Friday morning. And then of course, Hackaday is the home now of Floss Weekly. We sure appreciate that. We are actually looking for a guest still for next week.

So if you have someone or you Are a project and want to be a guest, let us know. It's floss at hackaday. com. And then we've got the folks from life Ray on August 6th. That's two weeks from now, looking forward to that as well. I think that is all of the housekeeping, all of the notes that we have.

And so we'll, we'll let you go. Thank you everyone that is here. Those that caught us live and those on the download and hate, we will see you next week on floss weekly.

Episode 792 transcript

17 juli 2024 | min

FLOSS-792

Jonathan: Hey, this week Jeff Massey joins me and we talk with Sylvestre Ledru about the re-implementation of ls, cp and a bunch of other utilities in Rust. But they're not doing it for the reason you probably immediately think of. And to find out more, you're gonna want to stay tuned. This is Floss Weekly, episode 792 recorded July 16th, rust Cortil.

Hey folks, it is Time for Floss Weekly. That's the show about Free Libre and open source. Software. I'm your host, Jonathan Bennett, and today we have a great show. We are talking with Sylvestre Ledru about core utils, but not the regular core utils, we're going to be talking about the rusty core utils.

It's going to be fun. It is, it is of course, not just me. We've got Jeff Massey with us today. Hey, Jeff. Welcome,

Jeff: sir. Welcome. Glad to be on, you know, on this is this topic's interesting. It is. I'm looking forward to it.

Jonathan: So I, I asked Jeff to be the co host and he says, I don't know anything about Rust. I'm like, yes, but you know about core utils.

Yeah, I guess I do.

Jeff: My other, other podcast job. You know, we, we hit on that quite a bit. So a lot, a lot of core utils.

Jonathan: Yes. Well, when you're coming, when you're covering Linux, when it's a Linux show, you do, you do a lot of core utils. Now have you, have you ever gone and grabbed the rust core utils?

Have you tried them? Have you dabbled with them at all?

Jeff: I have, there's not the entire package, but there's a few of them I've actually played with and, you know, so I, I can say I have used them. It's a better, that's more

Jonathan: than I could do, actually. I'm definitely aware of them. We've covered them in the past.

Well, let's, let's not let's not spend any more time talking about the project when we've got Sylvestre Ledru here. Let's talk to the man himself and get the get the download there. So Sylvestre, welcome.

Sylvestre: Nice to see you.

Jonathan: Yeah, it's good to have you. I am super excited to have you here today.

And so you are the, are you the, the, the. BDFL, the Benevolent Dictator for Life over the, the, the Rusty Core Utils. Or have you kind of come more recently to the project? What what's, what's your history there?

Sylvestre: So for life, I don't know, but I started to be involved in the project for the last four years, basically right, right before COVID hit.

Jeff: Ah, yes.

Sylvestre: And I, I have been the main maintainer, but I have three other great maintainer helping me with that work.

Jonathan: And so you didn't start the project, you, you kind of picked the banner back up?

Sylvestre: Yeah, so someone Jordi started like 10 years ago, and then some people worked from time to time on the project, just doing some basic maintenance.

It was hard to get pull requests, and I noticed that the project needed A maintainer who could do spend more time on it. So I decided to volunteer and I'm stuck with a project. Hopefully not for the rest of my life, but for the year to come at least.

Jonathan: Yeah, that's, that's an interesting thing that happens in a lot of, a lot of floss projects, you know, some of them grow enough to where they take on a life of their own.

And so the, the original maintainer gets to step out or hire somebody or, you know, turn it over to a crowd of people, but there's some projects that it's it's one guy doing it for. forever until, you know, until he finally decides to retire. And sometimes it's a problem. I, I suppose time will tell what, which direction the rust core utils are eventually going to go.

And I'm going to get into that in, in a bit. So let's, let's start though at the very basics. I don't know if Jeff and I know the answer to this question, but let's, let's start with the question of what are the core utils? Like what, where do you use these programs? What are they about? What are, what are core utils?

Sylvestre: So it's a, it's a good question. It's, it is easy and not that easy at the same time. So it was if you look at the source code of the first version of Unix in 71 you can see Still, you can see the same program existing, at least some of them. So chmod, for example, to change the permission of a file, ls, cp, those commands are part of the coroutines.

And then they, they grew in different directions. Some people needed different tools. So one of them is sort, cut, and so on. There are tools that we never use, for example in our regular life. So there are some things like PR which is to format. Text before printing, there are other like PTX that you rarely use, and TSORT to do topological sort.

So we have about 60 to 70 CoriTLs. We are trying to match what GNU is doing with their implementation. So I don't know if it is clear for everyone, but you have different implementation of the CoriTLs. You have the GNU one, which is de facto the gold standard, but you have also some Unix implementation, InVaried, Dropbox, and so on.

Jonathan: Yeah, and so there is there's also you know, you have some other implementations too. So BusyBox, for example, has a lot of the core utils that's part of the BusyBox binary. In fact, a week or two ago, I made a comment about how, you know, BusyBox and BusyBox doesn't have grep. And somebody in the comments was like, yes, it does.

It has those programs. Okay. Yes, fine. Okay. It has those programs built into it, too. But these are, it's interesting. These are old utilities, like they've been since 1971. That's a 50 year program that that's been around. That's, it's incredible. And so this, this sort of obviously brings us to the next question.

Why, I guess, for one, why do we still care about these? And two, why are we rebuilding them in Rust?

Sylvestre: So good question. So, we need them every day, like as soon as you do terminal, you use the CoriTLs all day long, just to do LS, CP, MV, RN, CHMOD. So changing the permission, it is our way to communicate with, with the file system, with some of the operating system basics.

And so that's why you care about. Now to the question, why are we interested in implementing them? One of the first reason it is fun to implement those tools. You understand a lot about the system, how it works.

Jonathan: Yeah.

Sylvestre: I You know the expression staying on the shoulder of giant those folks who designed Unix 50 years ago CHmod is still working the same way.

CP is mostly working the same way. And it's fascinating to see the decisions that those folks made at the beginning of modern computer are still relevant now. So if you use any BSD, any Linux, you still have the same paradigm. You still use 666 or 777 when you don't know how to set the permission. And it's still, if you look at the code from Unix back in the day.

It's still the same the same paradigm, and we still use the same model. Now, now we, why are we re implementing it? It's not about security. If you look at the new implementation, they had like 13 CVE over the last 20 years, so in terms of security, which is often one of the, the argument for promoting Rust.

In that case, it is not our driver. Talking for me, my main driver is thinking about the next generation. Like in, in 50 years, will C still be relevant? And, and the young generation, will they want to learn C or C anymore? And I think Rust is part of, it's a fancy language. Everybody wants to learn Rust. I think it's the right time now to to re implement that software in a brand new language which is fixing some of the hard things that C is still Some of the hard things that we still see in C, like memory management or parallelism and those kind of things.

Jonathan: So we have just alienated half of our audience by saying that C is going to go away. No, I think, I think that's a, that's a really interesting really interesting answer. It, it, it fascinates me, you know, like you say, so much of the push for Rust is because of security. And I'm curious So I assume that you look at the C code of the core utils.

To, to make sure that you're matching the implementation from time to time. And I'm, I'm really curious, is there like really old and crusty C? Like you, so, for those that don't know, for those that have not like looked into hardcore C programming you can, you can commit cardinal sins in C. Like you can't, you can do nasty, ugly things in C.

And sometimes like if you're writing a kernel, you have to, and I'm just curious, like, as you look into the C code of these old programs, is it crusty? Do you, do your eyes sometimes water and bleed just from looking at it? Are there some of those in there?

Sylvestre: So it's really, it's a, it's a tricky question. So yes, I look at the source code, but I have to be careful in the way I look at it because it is TPL code and and how our implementation is MIT.

So when I look at it, I'm looking at it because I'm contributing upstream. So I'm also contributing test. I'm working upstream. I wrote a few patches to improve compatibility, error management, adding tests. So in those cases, obviously I have to read at the code, but I'm not reading the code to do the reimplementation on the other side.

I'm spending a lot of time reading upstream tests to make sure that we are 13 compliant. But I'm not looking at the code. So code, however, is complex, but not because it is C, it is because of the legacy. Like CP and LS are nightmare to implement because you have a lot of option and you need to You cannot have undefined behavior.

So if you pass option X and option Y, you need to define a behavior, and sometimes that leads to a very hard code spaghetti code. So that, that part is hard. It's more dealing with the legacies and then that C is sometimes hard to read.

Jonathan: Yeah, does Rust give you the tools or are there more modern techniques that have allowed you guys to get away from the spaghetti code?

Sylvestre: So we, we, because we can re implement, re implement the tool from scratch sometimes it's easier, but sometimes we have some crappy workaround, like if this option and this option are passed together, then you, you change the configuration at the bottom of the, at the start of the functions, this kind of thing.

So sometimes we have to do some crappy Work around to implement. So, for example, one of the things that I implemented during a long flight was the ls dash dash dirt D I R E D which is the directory editor mode for Emacs and that one introduced a bunch of special cases if you are dealing with a directory.

It's not the same as a file, if it is recursive or not. So you have to manage plenty of those cases.

Jeff: Mm hmm. So, I got a question for ya. What, how, how did you get here? So, what made you get up one day and say, You know what? I think I really need to get in and rewrite Core Utils, .

Sylvestre: So it's a, it's a longer story.

So my official job is I'm a director at Madie. I have been working at Madie on Firefox for the last 10 years, and and Madia created rest. So I knew I know most of the core was developers. Some of them were in Paris, and I had the chance to work with them every day. And I was jealous of them doing Rust code.

And at my work in theory, I'm not supposed to write code. Like, I'm managing people and projects, I'm not writing code. So I was jealous of those folks, and I was like, I need to learn Rust, and I want to learn Rust, but I don't want, I'm not a student anymore, so I want to do something that is going to last.

And and that project found the right place. Like the right project to understand how it works and the operating system, like, you know, that middle ground. It's not a kernel, it's not a high level application. It is really something that I can understand. And they are self contained.

Jeff: Well, it would sound then like you also, based on your day job, you kind of can set the pace so you're not you're not so beholden to timelines.

You know, there's a little more flexibility in there.

Sylvestre: Yeah we release Firefox once every four weeks, a major release every four weeks. On my, this project, I can ship when I want. I don't have any constraint. I can stop working on it for like a month. And so yeah, it's a, it's a good way to reconnect with code.

Jeff: Well, and talking about the release schedule. So, albeit, you know, there's some flexibility. When, when do you think it is going to be ready to, you know, just. Put it in Debian or something and say, okay, here's the new core utils we're going to use.

Sylvestre: So it's a good question. So it depends what you mean by ready.

I have been using it in production for almost two years myself. So all my system are using that implementation. There are things that we don't implement but there are usually corner cases or options that I don't use or I don't need or programs that I don't need. I mentioned earlier the pr common to format text for print.

I have never used it in my life. I only use it when I need to implement some function. And so it's already ready. Now. The question is, when is it going to reach a bigger audience? So I know that some companies are already using it in production. So a big social network not Facebook, but they are using it on embedded devices as far as I know, mostly for license reason.

I know that some of the other operating system for cars also using it for license reason. And then this is one of the Strengths and weaknesses of open source. Maybe others are using it in production and never told me about that

Jonathan: So i'm i'm curious you you mentioned that there are there are some of these, There are some of these, some of the utils have sort of weird corner cases that you guys don't support.

Are you, are you planning to eventually add support for those corner cases, or are there some of these things that you have just intentionally said, we're not ever going to do this the way that core utils does. And so I guess the bigger question is, is the goal, you know, 100% Sort of bug for bug, maybe not literally, but, you know, exact feature for feature compatibility with core utils, or do you guys feel a little bit of flexibility to say, you know, maybe this decision that was made back in the 1970s, it was not the right decision or it's not relevant anymore.

And we're going to update it a little bit. What, what's the, what's the philosophy on that?

Sylvestre: So I I have been involved with an LLVM in Clang. And one of the success, in my opinion, of Clang is that the team considers that If they don't implement a GCC flag, it's a bug. And I, I think it contributed significantly to the success of Clang.

And I'm trying to replicate that model into that one. So yes, we want to reimplement everything. Now there are corner cases in tools that nobody uses that maybe it is going to take longer, but our goal is really to implement all the feature and all the flag for the most common options and commands.

So LSC, PMV and so on. So we are, we already are passing 100 percent of the upstream test on some of those common, but not all of them. So we are working on, on fixing those. So for example, one, we have a Google Summer of Code student who has been working on re implementing some of the color function of LS.

And it's quite hard. Like, I, I play with it and LS dash dash color is not easy at all to implement. And so we contribute with other upstream developers and other Rust developers to make it perfect. But this is, we consider that as a bug.

Jonathan: Yeah. Oh, I've, I've done just a tiny bit of color work on the, the output of another, another project that I'm working on.

And so you get into antsy color codes and then you have to pay attention to things like term info. And I can imagine that being just a mess to work with. Oh goodness.

Jeff: So, how many of the utils are 100 percent compatible?

Sylvestre: I don't have the exact number, but we are publishing every day the updated list. So, we can share the link after that podcast, but we have a list.

So, we run the test, all the GNU tests every day, and we publish the result. I think it's on, we have like 65 upstream programs, and I think 20, 25 are 100 percent compliant. But sometime So what I, what I like to do is when I reimplement the tools and I notice that That my tests, all the new tests are passing.

And then I realized a bug in my code. I'm going to look if upstream as a test or not. And if they don't, I'm going to upstream I am going to commit upstream a new test to make sure that it improves a new compact, so new compatibility. So I've been contributing a lot of patches upstream to make sure that their test suite is better and better.

Cause as I said, there are so many combinations that we cannot test everything.

Jeff: Yeah. So, and maybe I want to make sure I heard you right. So, If you find a bug in say the GNU utilities, you implement the bug as well?

Sylvestre: No, we are going to report it upstream. And usually the answer is not a bug. It's a feature.

I'm joking. I'm joking. But

Speaker 4: yeah,

Sylvestre: for example, there is a checksum command is, is a weird comment that you can see that it was designed in a weird way. So you can pass So checksum has plenty of arguments. So there is one which is dash dash tag and dash dash untag. And the command is only going to pick up the last one.

Even if they are conflicting with each other, it's only going to pick up the last one which is quite confusing as a user. So initially I reached out to the GNU project saying can I just make the first one conflicting with the other one and they said no because you are going to break some behavior from the past that someone might have used like 20 years ago.

It clearly makes sense and it's clearly a bug. But it has been used in the past. So we are trying to find a good compromise. So sometimes we, we try to display the same output and the same errors. And sometimes we think that we can do a better job in terms of doing, in terms of doing error management.

So in that case, we are not going to follow exactly the same output as GNU. And we are going to provide a better error management. Oh,

Jeff: sorry. I didn't mean to cut you off there. No worries. So saying, saying that, you know, you're trying to provide a better environment. So how does that, how's the community what's the reaction from the community on these tools?

I mean, have you gotten any feedback or?

Sylvestre: Well, we, we have I just looked at the number before I'm meeting with you folks and we have 500 contributors. I'm sorry. I'm laying 499. Hopefully someone is going to contribute this evening and then prove me wrong. So we have a lot of people who are interested in contributing.

We have a lot of good first bug. So we know that people are excited by that project. Now how many people are using it in production? I don't know. And but the reaction is usually very positive. Some people are always asking the same question about license. So, we, the folks who, who started that project, they use MIT.

And some people saw that it was an attack against the FSF, which is clearly not the case for me. So, we have always, if you look at every Hacker News or Reddit thread about it, they are going to mention the license. So, some people are very vocal about the license. I'm not. I, I honestly don't care that much, as long as it is OSI compliant.

So we have that one and some people there is always a concern about Rust that it is hard to use, hard to package, and hard to develop into. And for some people Rust is just a trend. I disagree, but some people still think that this language is going to disappear.

Jonathan: Yeah, so okay several things there I want to ask about, and the first one this may be a difficult question, or maybe, maybe you have the answer ready, I don't know but I, I get why developers like this, right, because Something, something new, a re implementation of something really old, a popular language.

I mean, that's just candy to developers. Like, I'm sitting here going, I wonder if I could send a patch in. I'm sure there's something that I could dive in there and work on. I don't know Rust at all, but I'm sure I could work on it. Like, it's just, it's just candy for us developers. But for users, for end users, what are If there are any, what are the advantages of going to the Rust core utils?

And you said something about in some use cases, there is actually compliance reasons to do so, and I'm really interested in what that is. You mean compliant in what, in what sense? You, you, oh, sorry, there we go, that one. So you said something about car manufacturers use it and it seemed like there was a, there was a legal compliance

Sylvestre: issue.

Yeah. It's GPL. Oh, it's the GPL versus

Jonathan: MIT. Oh, okay. And so is there, is there something then for, see, I, I figured that was maybe with the new EU laws, they are pushing people towards memory safe languages. And so I thought maybe that was what was going on there. But so comment on that if you would, for a minute about like what the reasons are that regular users might want to use the, the Rust core utils.

Sylvestre: To me, one, one of the reason is that performances. So for some function, we are faster than the glue implementation. So for example, if you do a recursive LS or if you do a CP, we are faster not because we are better developer, but because we are leveraging some of the tweaks that. Rust is providing you for free in the system.

We also have some extensions. So we are documenting every extension that we do. So there is something, I did a presentation at FOSDEM like one year ago. And I mentioned that we are, we have a dash dash progress option in CP and MV and people clap in the room, I was very surprised. And then I, I, and then I had to move some file a few months after I was like, Oh yeah, now I understand why it's extra because it's such a pain with CP to, you never know where you are at.

You have to do some DU on the other side in another terminal to know how many files you have to transfer and so on. So we have some extensions that are helping the user. We also took some, we also implemented some options from we took some options for cut, for example, from the BSD world. So we can do some extensions.

We are trying to be reasonable. Like, we are, we are really trying to understand if it works. Really provides a value to the user because we know that we are contributing to the fragmentation and to the mess by adding extensions. So we are trying really to be careful.

Jonathan: Yeah. Oh, that's, that's interesting.

And yes, for the, for the, the, the, the progress bar. Thank you. That is, that has been a long standing gripe of a lot of people about CP. Okay, so let's, let's go in the direction of packaging. So we, we talked about this a little bit before the show, and I will try to re, re pontificate my thoughts on this.

I'm a Linux user. I use Linux, and therefore I, I am rather fond of using the package manager provided by my Linux distribution. So in Pop OS, it's apt in the Fedora machine behind me, it's DNF. That works great. And there's this issue that languages. Python is one that does this. The, the, the Node.

js JavaScript sort of ecosystem does this. And then also Rust does this. And in Rust's case, it's Cargo. And you have a package manager just for the language, which. For developers is amazing, and it's extremely helpful. The problem that I see there is there's this sort of disconnect of there's now a package manager for your distribution, and there's a package manager for your language.

And how do you then install the packages that you want? Because, you know, there's the obvious advantage of installing them via your distros package manager. And it's just, Is Rust, to put a point on this, is Rust hard to package and has that been a problem for you guys for getting these packages into distros?

Sylvestre: So, I'm, I'm a Debian developer and I uploaded the first version of the Rust compiler in Debian, so I know the pain and I have plenty of scars to show it. So Rust is not easy to package, but it's very similar to Java. So it's not

Jonathan: just me. It's not, it's not just me that has that problem.

Sylvestre: Yeah. So it's, I found it very similar to the Java ecosystem, like with Java, you have.

Not anymore, but it used to move very fast, so you had plenty of different versions of the same library, and you had to package different versions of the same library. So, for example, you had several versions of the XML parser, or several versions of a library to, to read files, and so on, because it, it It's upstream is not always following the best API practices, like the same VEA system.

So you have the same issue in the Rust ecosystem it's less and less an issue because developer are stabilizing the core, the core libraries, the crates. So, but in, in a distro like Debian, you need to package all the dependency independently. And without any network connection. So that means that if you have like the Rust coroutines, we have, I think, 300 dependencies.

That means that you need 300 different packages in Debian to be able to upload that version. And when you want to update a new version of the coroutines, that means that you have, you need to update the dependencies that have been updated. Sometimes it's easy, sometimes it's hard. So yes, it's not that easy.

We have tools in Debian to make your life easier. And other distros have probably similar tools. But it's not specific to Rust. Like OCaml has the same issue. Python, NPM. It's it's Part of the work of the Debian developer, of the packager currently.

Jonathan: Yeah. Okay, so, I, I, I'm very curious Is Rust core utils available in any distros?

So, you know, can I, in Fedora, let's say, can I install, you know, Rust core utils with DNF, or in one of the Debian derivatives, is it possible there, is, is the packaging worked out anywhere?

Sylvestre: I think there was a, I would phrase the question differently. It is where it is not available currently. I think it has been packaged in most of the distros.

So it is on Boo. I saw some people packaging it on Windows. I don't know what that means, but there are some people working. I think it's Wingate or something. It is it is on Debian for a long time. Ubuntu, Fedora, ArcLinux and so on. So you can do it. Now the question is Is how do you update your system to use it?

It's this one is harder. So for example on my system, I just override the path in every terminal To point to the to the rust implementation When I was trying to evaluate how much work was left to be able to boot a Debian system, I was removing the GNU Core Utils to replace it by our implementation, to make sure that I was not using the GNU one.

Linux distro are providing different options. So I know that some Linux distro are offering to replace the GNU implementation on Debian. You have the two implementations next to each other. So you just update your path to make it work.

Jonathan: Yeah, that's that's, that's probably a bit of a challenge. So most distros won't let you uninstall core utils.

That's kind of a protected package. Are, are any distros, do any of them treat the Rust core utils package as a replacement so that you can do the install of one and then the uninstall of the other? Is that, is that actually possible anywhere to go with just the Rust core utils?

Sylvestre: Yeah, I, I don't remember if it is gen2 or arch, but one of the two is providing that option.

Jonathan: Why am I not surprised that it's arch? Why am I not surprised?

Yes, so I, I suppose at some point in the future, arch users are going to say, By the way, I run arch and the Rust core utils.

Sylvestre: Yeah, I had I was at FOSDEM also in February and you know, some people were talking about Rust next to me and one of them told me, oh, I'm using the Rust Core Utils on my system.

And I was surprised because it was the first time that a random guy at FOSDEM told me about that at the conference. Like in real life, it was funny. That kind of

Jeff: anecdote. That's great. Well, as a random guy, where, where could I find the Core Utils? The Rust versions. Now, I know Linux you listed a bunch of different distributions, but BSD or other Unix's or I know you said you touched on Windows a little bit, but

Sylvestre: Yeah, so it is it is one of our strengths is that we, we are treating all the platform as Tier 1 all the supported platform as Tier 1.

So Windows, Mac, Linux, Android we have free BSD support. Someone has been working on the OpenBSD port. So we treat every platform as Tier 1. So if it breaks on Windows, you need to have a good reason to fix it. For breaking it. So for example, Artlink is one of them. So Windows doesn't have support for Artlink, same for Android.

So for this one, we disabled this feature in the code. So it's really but we are really trying to support every platform as we can. And we have CI and GitHub that runs every PR on this platform.

Jeff: Oh, that's awesome. So, okay, I'm average guy. Okay. I decide I'm going to go and download the core utils. Am I going to notice a performance difference?

Sylvestre: Good question, it depends on the command and depends on the option. So sort is faster ls can be faster, and there are other commands like I think cut will be slower. So for now we have been focusing on compatibility and then we will focus on performances. We, we have some performance win, but not always.

For example, there is a common factor to, to to get the prime numbers, to play with prime number, and we are significantly slower than the GNU implementation. And some people recently looked at using some crates which are doing prime number math, and they were pretty bad compared to the GNU implementation.

The math into GNU. So there are places where we are slower. So if someone is into math and want to do some prime number math there is a space for you to do that in Rust and to make it faster than this implementation.

Jeff: I bet you there's somebody in the audience that's probably really good at math, you know, so here's your here's your chance to contribute.

So, and it's also good to know that so even if you hit 100 percent compatibility, That's not the end of the program. You can then go back and say, Okay, it's 100 percent compatible, let's make it faster. So that, very interesting there. Now you mentioned Android, and so I imagine there are some systems that are going to be rather, you know, memory, both RAM and drive constrained.

How's the package size? Is there much difference? Is it, Bigger, smaller, the same?

Sylvestre: We have tricks to make it smaller. So if you Rust is generating some significantly bigger binaries than C in general, so you, if you download If you don't use a trick that I'm going to share, it can be up to 100 or 200 megabytes So called it fields because it's much bigger and we don't use a share library.

So So you cannot say space by using that trick. So there is a trick that you can use is that just like busy box, you, you have a single binary and then you create same link. So it is a trick that I'm using in Debian. So you have only one binary and you do rest dash coroutiles. And then you create a same link that is going to be named LS or NL or.

CP and so on. And at the end, you only have one binary. So size is reasonable. I think it's 20 or 30 megabytes for the memory consumption. It's really dependent on the program. I we had a bug report recently saying that our implementation of mall. So to read a text is significant, using a lot of memory.

So one of the maintainer. Tertz decided to work on it and decrease the memory footprint by a factor of 10. But even with that, we are still using more memories and so it's part of the fun of that project also.

Jonathan: So there's, there's a rumor apparently going around that the Rust core utils project is entirely funded by you getting a Euro every time someone asks you why it's a MIT instead of GPL.

Sylvestre: Yeah, exactly. Yeah, it's usually a good way to start some trouble that one.

Jonathan: Yeah, so, okay, I'm, I'm curious From what I understand of licensing, if you wanted to, you could actually update the license from MIT to GPL because they are compatible in that way. Is that something that's ever been considered?

Now, I'm not telling you that this is something you should do. I'm just asking the question.

Sylvestre: Well, I I have a question with plenty of friends and people online. As I was saying earlier, I I care. I don't care about license. I only care if the license is OSI compliant. I think it's a good rule. So I'm not into a license debate because it's more philosophical than technical.

And and we use that license for a long time and we got a community and the community is vibrant. So we have 20 30 people contributing to each release, at least 10 newcomer every time. I'm not saying that it is thanks to this license. But changing the license might create a lot of unwanted noise and conversation.

And I don't have time for that.

Jonathan: Yeah, especially since you have some users that are using your package because it's MIT. That would, that would definitely be disruptive. That would be that would be fork, fork bait, let's say. Okay, so one of the other things that you mentioned in the prep is that you guys fuzz core utils.

You do some differential fuzzing as well, and I'm very curious about that. You said there has only been like 13 CVEs in the last 20 years in the upstream core utils. Was any of that found because of your fuzzing?

Sylvestre: No, no, no, I I was, when I started fuzzing our implementation, I was excited because I saw that I would find security issue upstream in the GNU implementation and I didn't find anything.

Not even a single crash.

Speaker 5: So

Sylvestre: it is a testament to the quality of the GNU implementation. Like I, I, I'm in touch often with the two main developers. So I'm going to butcher his name, but Padraig and Jim Meyering. And and they are amazing developer, lovely human beings. So it's a pleasure to interact with them.

So they are terrific developers. So I'm not surprised that I didn't find any issue. We first. Not, not really for security, but for crashes in general because it can find some some weird behavior. So for example, there is a sec command in GNU that you, in the coroutines that you can use to generate sequences of numbers.

So integer, integer or float and so on. And when you start fuzzing those things, you find some weird behavior when the numbers are very high or very small or close to one or those kind of things. So we found bugs. And differential testing, differential fuzzing help us find differences with with the GNU implementation.

So we do, so basically what we do is we generate some codes and some batch script or some code that we are going to send to those commands. And we look at at the error code. If it is zero, that means that it works. If it is not zero, it's an error. And then we are going to look at the, at the standard output and the standard error to see if we are producing the same output.

So with LS it's very important. And we are looking at the error messages. So we do a differential fuzzing, not really for security purposes, but for compatibility.

Jonathan: Yeah interesting. Okay, so One of the, one of the other notes that you've got here that I think is interesting to dive into is this idea of dependencies as security, not vulnerabilities, but attack surface, we'll say with the idea of supply chain attacks.

One of the, you know, we talk about Rust as being potentially hardened for security. I'm curious what you think about this idea of there being a bigger attack surface just because with using Cargo, you've got so many dependencies in each Rust project.

Sylvestre: Yeah, you you have to be careful. So, I was if you look at the dependency tree of our implementation, we have like 300 dependencies.

Some of them are huge, some of them are tiny, and we know what we are depending on at level one, so that means direct dependencies. So, for example, we are using the Nix crate, which is a wrapper. Around some libc function. We are using lscolor to do the color management of ls. And we are using sc linux crate to do sc linux feature.

And we know those upstream developer and we trust them. And we know that they are usually very good at doing release management and not taking crappy PRs. And we know that those crates are well maintained. However, with the dependency tree, you have some crates low in the stacks that might be, that might get compromised at some point.

So you have so at Mozilla and with Google, we worked on a program called CargoVet which enables So Chrome team and the Firefox team to verify if to audit the crate and to share with others that, yes, that crate has been verified and validated and there is no issue with that one. So there are mitigation strategy, but it's a, it's a global issue in tech.

Like we saw that with MPM a few year ago, it's really a typical vector of attack injecting some backdoor into dependency. And of course the EXE story, the recent one, is an amazing example of supply chain attack. So yeah, it's an issue.

Jonathan: So that's, that's something I was just thinking about with, with XZ introducing a backdoor into SSH, which is still the craziest story ever.

The, the fact that you're doing A B testing between the Rust core utils and the Upstream core utils it's an interesting opportunity to maybe find issues like that sooner. Maybe immediately. Particularly if some of that testing happens on like on various distros with real installs. And I'm not sure exactly how your CI works.

This might be, this might be difficult to exactly get at. But so I, I don't think SSH, SSH obviously is not one of the core utils. And so the SSH daemon, it's not something that's, you know, in scope for the Rust The Rust query tools project. But just thinking through this, like, let's just just kind of game, game this out.

If there was a Rust version of OpenSSH or even of XZ, let's say you could, you could write a test that would have feasibly caught This difference in behavior because that's essentially how it was discovered, right? There was one of the one of the was he was he a Debian developer? Anyway, one of the He was a Microsoft developer.

That's right he was he was working on I think on Microsoft Linux on but anyway, he he He was playing with with SSH and suddenly realized this is behaving differently than I expect it to So something was taking longer than it was expected to and He It's something that's it's ever since that story has come up that has interested me is this idea of could you automate some testing To discover that sort of difference and you know Obviously if if you could automate that and run that on it on every update you could potentially find stuff like that a lot faster and it's fascinating that because you have you know, you have a Kind of a black box implementation, but it's supposed to be doing the same thing as these query tools and you're doing it in Rust instead of C, you know, it gets you kind of this insulated second opinion and So if someone ever tried to do something like that in and obviously with the query tools It'd be extremely difficult, but still if someone tried to slip one of those things in there You guys would see it, likely, because you have this huge test suite that you're doing A B comparisons with.

I think that's really fascinating.

Sylvestre: Yeah, you're making a very good point. I think the attacker on Xe knew that. That's why he said disable some feature in OSS Puzzle. Like, they knew that fuzzing would be a great way to catch those kind of error, and that's why they went on OSS Puzzle. Fuzz and disabled some as a check.

Yeah, good point.

Jonathan: Yeah. But, but so in, in, in this example though, like, so again, playing this out, like what if there were a rust version of xz or SSH or whatever the patch that he sent in that explained it in OSS fuzz, it got accepted. Because nobody really stopped to dig all the way into it. Whereas if someone had to re implement that in Rust, you would have to get to the bottom of it and understand, okay, why is this suddenly doing this?

Why is he making this change? And things would, things would not make sense there. And I just, I, I, I feel like it might be an opportunity to catch something like that a lot faster, which. I've got to admit it was caught, it was caught extremely fast. Hardly any distros actually shipped it and it was not, it was not live for very long at all.

But I don't know, it just, it seems like a very interesting opportunity that, that maybe Maybe we need more utilities that have a Rust version of them as an insurance policy.

You want to talk about that? I've actually been told that there is a there's an idea that the Rust core utils that you have to maybe include some other applications.

Sylvestre: Yeah so we we have been working on the re implementation of the find utils, which are quite famous, which are not part of the official core utils.

And we started also some initiative to do the same with diff and many of the other tools that are called to to the Linux distro right now. So utils linux, procps, psd utils, hostname login, and so on. So this one is more for For fun, it's one of the thing with those project is that you know, what is a target?

So you don't need to buy shadow to discuss about what should be the input or the output you have a reference So it's very easy to learn the rust by doing it It's fun. If you are into operating system like I am so, you know You can work on your own and you just have to mimic what the other software is doing and you can It's really a great way to learn rust Like it's a way I learn Rust and I'm sure that many people starting contributing to those tools and and learn Rust thanks to that.

Jeff: So is the plan to just keep expanding the programs you translate into Rust? I mean, are you looking at taking almost all the commands, you know, you'd run in the shell and eventually, you know, 10 years from now or whatever? Convert them all?

Sylvestre: Yeah, I think it's a, it's a good investment for the future of our industry.

Like, you know, there is a Chinese saying that there is two, there are two times where you need to plant a tree, 20 years ago and right now. And and I think we are at this point that if we want to have a good take in 20 years, we need to start investing now and starting to replace those two now.

Because I feel that Us, we are starting to be old, but the new generation, will they want to learn C to do some some maintenance of those tools? And those tools need to evolve, like they cannot stay the way they are. I was looking at the, at the GNU coroutines not, and I saw that they updated some code for the new GLibc.

And there are always changes happening, new architecture and changes on the kernel, on the libc and so on. So we need to provide better tools for the future generation to to access tools.

Jonathan: Yeah. I kind of want to jump in and ask quickly, like what what does that look like? Because some of these tools do have changes that get made to them.

And sort of your, your target is kind of a moving target because of that. Is that, is that a challenge in and of itself?

Sylvestre: It can be frustrating, like when, when the new implementation is pushing a new release, we are updating our CI, and we often see like five tests, which were green, are becoming red, and that's always a bit sad.

You know, more work, or they are making changes, and sometimes I'm looking at the changelog upstream, and I realize that I'm the one who made the upstream change and broke my test in the Rust implementation. So sometimes I hate myself for doing it. That happens sometimes. You know what I mean? But it is exciting.

It is very, I, I didn't know much about the new core utils when I started, but it's still there are discussion happening about adding new option on the mailing list and changes that are made, so they are living software, so it's healthy.

Jeff: So, so with you know, okay. The core utils are going to rust.

Other utilities are going to go rust. You know, there's a lot of talk, you know. So. Rust in the kernel. So with this whole rust ecosystem, you know, how, how are the core utils going to. integrate into that like new shell, for example, how does that all fit together?

Sylvestre: So new shell the new shell folks started contributing to our tools because they want to be able to plug themselves directly in Rust to that and not do some system call regular call to the binaries that are provided by GNU.

So they want to be in the same memory space, so they started splitting some of our tools to to provide API for them. So there are more and more integration with GNU Shell which is a fascinating case, a fascinating example to me. I was very pleased when I saw that because, you know, when you, when you do software, you see people using your software in an unexpected way.

And that one was amazing to me. Then for the kernel The Rust ecosystem is well designed, so with crates and cargo, you can, if, if we provide we are shipping some new crates to do some self content change. So, for example, the SC Linux crate was started by someone who contributed to our project to introduce I think it was CP or CS CS on SC Linux feature, and he decided to create a crate, and that crate is used by many other software.

We are trying to split our work so that others use it.

Jonathan: Okay, so I am, I'm curious, you said earlier that you've got sort of a good working relationship with the upstream core utils guys. Is there a future where the Rust core utils becomes more official? You know, at some point in the distant future, are the GNU core utils going to be the Rust core utils?

Is, is this something that could happen?

Sylvestre: Not anytime soon, maybe at some point, but for now they are the gold standard. Everybody uses, well, every Linux distro uses that implementation. And as far as I know BSD and Mac are following what they are doing in general, in terms of options. So they are still the gold standard.

Maybe at some point that will change, but not anytime soon.

Jonathan: So there was something else I was going to ask. Are there any distros or projects that are shipping the Rust core utils by default? And I could, I could very much imagine a Rust centric distro. Like if it doesn't exist, maybe it needs to. that uses the REST core utils by default.

Sylvestre: There is one, I forgot the name, but there is a Linux a distro based on Rust that is using our tools. I don't remember if it is Redux or something like that, but there are people using it already for basis for the operating system. So, the one that I was mentioning earlier for cars, I think it's called r purchase.

I don't know much about it, but they are using it and shipping by default.

Jeff: Mm hmm. Very cool. So, if, if you're doing a lot of that with the You're replacing Corey Utils, the other stuff. Are you, are you putting a GNU out of a job? Is there any animosity or any kind of,

Sylvestre: no

Jeff: friction?

Sylvestre: I don't with with Jim and Padraig, we exchange email often.

So then they are very friendly with us. And and I love what they are doing. I have a lot of respect for those folks. So there is no tension on that front. For the FSF, I have no idea. I haven't received an email from Stallman yet. Maybe I will after that call, that meeting.

Speaker 5: That would be, that would be fun.

Let us know what he says. He has the most interesting opinion on things.

Jonathan: All right. So let's see, is there, is there anything, any problems that you've run into in the process of doing this that were unexpected, any really difficult problems you can tell us about?

Sylvestre: Yeah, sometimes it's, it's interesting to understand why a developer decided to implement that function this way, or that argument. And sometime I wish I had a time machine to go back like 15 years and tell, tell that developer you should not do that this way, you should do that this way. I was mentioning some, some of the, some of the issues that we often see is if the software is well designed, you have Two options doing the opposite, conflicting with each other, and you have some error messages.

But sometimes you don't have that, so sometimes you have conflicting options and only the last one is going to be used. And you have sometimes some lack of consistency in the GNU coroutines, and it's probably going back from the Unix time. And and those things are hard, and sometimes you're like, Oh, I wish I had done that differently, because it makes our code uglier than it needs to be sometimes.

Jonathan: So what, what is, what does the timeline look like? At what point are you going to be able to say, okay, the core utils are done or as done as they can be considering that code is still getting written for the upstream core utils. But like, if you, if you kind of look at your, your, your progress that you're making now and extrapolate that out, you know, are you six months away from hitting a hundred percent on the tests and all of them are five years away?

Like where do you think you're at? I

Sylvestre: think we, I think on the main tools cause if, let's, let's be, let's be honest, like you use the 20 percent or 30 percent of the current yield, like there are many tests that you never use. Like the topological sort nobody uses it most of the time. Like I'm sure that someone is going to say, yeah, I'm using it often, but I never use it in real life.

Central to

Speaker 5: my workload. Yeah. Yeah. Yeah.

Sylvestre: Yeah. Yeah. Yeah. So you have tools that you rarely use. So those ones are going to take longer, but it is our goal. I think we can be 100 compliant with the main tools within a year or something like that, maybe two years.

Jeff: But yeah, awesome. Oh, that's soon.

Sylvestre: Well, the GSOC helps. Like, having someone to fix all the LS corner cases, it's very helpful with the corners and so on.

Jonathan: Yeah, yeah, that's true. That's true. Okay, is there anything that we did not ask you about? I know this is a hard question. Is there anything we didn't ask you about that you want to make sure and let folks know about?

Sylvestre: Well what should I do to to contribute? We have good first bug. We are four maintainers who are spending way too much time on, on that project. We can help mentoring. And as I was saying earlier having a reference implementation makes your life significantly easier. We have a test suite that runs in less than a minute.

It's very, very fast to run all the test suite. GNU takes longer, takes like 15 minutes to run the test. Mostly because they are using a lot of script to run the test and a lot of different namespace and memory. So for us, it's the same memory space. And so it makes things way faster. So it's very easy to run.

You know very quickly if you are regressing the tools and we have a lot of trust in our CI I think the code coverage is like 85 percent 86 currently, so it's amazing And so that makes your life significantly easier as a developer when you want to start hacking in those software So contributing is very easy if you want to learn the rest it's one of the easy project to start with because there are so tools are very self contained and not many dependencies, not like starting to contribute on Firefox or Chrome or something like

Jonathan: that.

So if somebody wants to learn more, where are the places to go to? If, if I want to jump in and do some work, but I have questions, you know, is there a, is there a forum or an IRC or a discord where, where do you, where do you want to send folks to, to find out more?

Sylvestre: I wish it was on IRC, but it's on Discord.

I'm part of the old IRC. But yeah, it was before my time and the community was already there. I don't want to be the old guy saying to the young people, you should use IRC.

Jonathan: I would imagine that there is a way to bridge IRC and Discord. Please pop out your notepad. I don't think so.

Speaker 4: Well I'm just

Jeff: thinking, you're saying Old Guard, I'm the only one here with white on the beard, so, you know.

I shaved, that's

Speaker 4: why.

Jonathan: My white is all on top. Okay, so last couple of questions then that we are required to ask everybody, and that is, what is your favorite scripting language and text editor you spend all day in?

Sylvestre: So text editor, depending on what I do, so if I'm on server, I'm going to use Nano. If I need an application that starts quickly and don't use 20GB of RAM, I'm going to use Emacs.

And if I'm doing Rust code, I'm going to use VS code with RLA, so Rust tool. But that one is using way too much memory, in my opinion. But yeah, it really depends on what I'm trying to do and how much time I'm going to

Speaker 5: spend,

Sylvestre: I think. Yeah, and scripting language. I love Bash. I love Bash. I know that you interviewed the Bash author.

So to me, it's a scripting language. It's ugly, but I love it. And Python. I love writing Python also.

Jonathan: Yes, yes. Did you catch the interview we did with Pavo about Amber? Sort of a better Bashling? That one was really interesting. I enjoyed that one a lot. Yeah, definitely. I'm just And it rings a bell.

Sylvestre: Yeah, I'm looking When I was listening to him, I was like, yeah, it rings a

Jonathan: bell.

Yeah, I'm looking forward to I'm looking forward to the day when we bring somebody on and they tell, you know, that's not associated with that project. We bring them on. They're like, oh yeah, Amber, it's great. It'll be fun. Alright. Thank you. Thank you, sir. Thank you so much for being here. It was a blast to learn more about the project.

And you know, maybe in six months or a year or so, we'll have to bring you back on to talk about what's changed. Sounds good. Alright. Thank

Jeff: you.

Jonathan: Okay, so, what what do you think?

Jeff: I think it's awesome. I, you know, With the, with the new language, forward thinking and all the fuzzing and everything and just, you know, and even better defining some of the ways that some of the tools handle, you know, like you said, conflicting switches and, you know, just kind of, kind of cleaning up.

I mean, I think it's awesome.

Jonathan: Yeah, I, I'm, I wonder, and I, I, Of course now, this is some staircase humor, as it were. I should have, I should have asked about this during the show, and we can ask about this next time. But I wonder if there's a future where, like, you can, you can put the Rust core utils in one of two modes.

I have, like, bug for bug compatible mode, or a clean up some of the weird stuff mode. Because, you know, like you said, things like the different handling of conflicting switches and it sounds like for now they are they are I think specifically you should call that misfeature for misfeature because they're they're not bugs But I could see a future where maybe at compile time or install time you say, you know, turn on the extra candy stuff and fix the old stuff.

But like, just the ability to have a progress bar in the copy command. Like, that's great. I so want that. I sort of want to install the Rust core details and start using them just for that. Because that drives me nuts. And of course, there's workarounds. There's ways to handle that. But that's, yeah, that's really cool.

Jeff: I've done the DU, like you mentioned, just to go, is this thing still working? What's going on? Let me see, you know, and Oh yeah, it's still going. And you just.

Jonathan: I don't remember if it's the CP command, but it's one of them that it's like, the official way to get a progress bar is you send it a a system signal.

Like the it's, it's not SIGKILL, but it's one of the other ones, like SIGUSER1, I think. You send it this signal, and it'll tell you what percentage it is. And so, like, you, you have to open up a second terminal, and so, like, you can set up a watch command with a kill all and then that signal. But it's just so clunky, it's like, why?

So I'm glad somebody's come

Jeff: along and done that. Yeah, I've, I've even tried that, what you're talking about. I've gotten the bar before, but it was, oh my gosh, you know, I was cutting and pasting out. Yeah, I was cutting and pasting out of a guide and like, oh man, it's just. Just start it and walk away. It's just less aggravating.

Jonathan: Yup. Oh, it's great. It's great. And we will, we will have to, we'll have to have the actual core, like the upstream core utils project. See if we can get those guys on. Cause that'd be a lot of fun to, to chat about that too. We can ask them why there's not a progress bar in CP. Come on.

Jeff: There's

Jonathan: 53 years

Jeff: to get it

Jonathan: right.

What's,

Jeff: what's going on here.

Jonathan: Oh, that's great. All right. You have anything you want to plug before we let folks go?

Jeff: The only thing is check me out and Jonathan as well and other co hosts over on the Untitled Linux show on the twit. tv network.

Jonathan: Yep, absolutely.

Jeff: We have a lot of fun over there. So, definitely, definitely want to see people over there in a very similar kind of vein as this show.

Other than that, that's all I got. Just thank you for having me on and always a pleasure and had a great time.

Jonathan: Yeah, thanks for being here. Alright, so I will let you know that the plan for next week is to talk with Jay Cattry about Highlight. io. That's going to be a lot of fun. We are recording on Tuesdays.

It's 1130 Central Time, my time, and we stream off to YouTube. So make sure you go and follow the full video. Floss Weekly YouTube channel, where we are now doing the video interviews as well. We finally got that workflow worked out. And so you can catch the video version there if you want to, or just stick with the audio, you know, what, whatever it's up to you.

As far as things for me to plug, I will mention Hackaday. We've got the security column goes live every Friday morning and lots and lots of stuff to cover there. And we. Other than that, we sure appreciate you being here. Everybody that watches this live, those that catch us on the download, and keep it up!

We'll see you next week on FLOSS Weekly.

Episode 791 transcript

10 juli 2024 | min

FLOSS-791

Jonathan: Hey, this week on the show, David Ruggles joins me to talk about my origin story. Due to a bit of a scheduling snafu, it's just the two of us, but we have some interesting history to discuss. This is Floss Weekly, episode 791, recorded July 19th. It's all about me.

Hey folks, it is time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we've got, well, something a little off the beaten path. First off, I've got David Ruggles as the co host with me. And we're gonna have fun today because Well, our expected co host had a conflict today, and our expected guest had a conflict today.

And so David is coming off the bench, as it were. And he, he had an idea for what we could talk about. You know, if we ended up in this situation where nobody else showed up. And that's, that's what we're gonna go with. And so his, his thousand IQ idea was to interview me! And to talk about my origin story, which I don't think we've ever done on the show, so that's going to be interesting.

I have very hastily put together some notes for him, and so that's, that's what we're going to do. It's, it's sure to be, well, would it be tooting my own horn to say that it's sure to be interesting? But anyway David, you're You're sort of, you're sort of in the, in the big chair today. And I'm the, I'm the guest, I'm the interview VE.

So take it away, sir. All

David: right. Well, I will do my best because this is my first time in the big chair. So we're just, you know, we're spinning the salad and seeing what comes out. So I I've been getting into podcasting this year, actually, finally got an internet connection that supported it. So now that I've been doing it for a little while, it's just leading to more questions and you've been out there doing it for longer than me.

We'll get into exactly how long, I'm sure that'll be one of my questions. So I actually relish the opportunity to dive behind the hair.

Because if you have not ever seen a picture or watch a live stream, everybody has to agree that Jonathan has awesome hair. So I think that's my first question. Is that

Jonathan: hereditary? So, I don't know. Little known fact about me, I was actually adopted at birth. And I've had very, very, very little communication with my actual biological parents.

And so there's a lot of question marks about what, what about me is hereditary. And so I don't, I don't know about the hair. So it's possible that one of your gifts

David: was awesome hair. So just to kind of set the stage What got you into computers? What was your light bulb moment to quote your own note?

Jonathan: Yes. So one of my very early memories, you know, you, you talk about core memories. One of my very, very early memories was going to, I'm pretty sure it was a Sam's club with my dad and. For those that remember, I don't know if Sam's Clubs still do this but they used to have, like, you would walk in, there'd be a little bit of stuff off to the right, and then they would have the huge, big, you know, floor to ceiling shelves.

And facing you at the door, on that first set of shelves, was their computer stuff laid out. And I used to find that fascinating. I would get, you know, as soon as I was tall enough to reach up there, I'd go over there and I'd fiddle with it, but I didn't know anything about what I was doing. And I have this memory of my dad being with me, and, For the first time, like, well, here's how you hold the mouse, and then you can move it around, and you see when you move the mouse, it moves the cursor on screen, and you can double click with the mouse, and it opens stuff, and oh my goodness, it was just like a lightbulb moment, it was an awakening, oh, I can actually do things with this, I know how to, I now know how to make this do things, and you know, that kind of just, set me on this journey of, this is, this is really interesting.

And yeah, so that's, like I say, that's kind of a, kind of a core memory, but it's also, it was the, the start of sort of a lifelong fascination, this idea that, you know, I can, I can actually have control over this box and make it do what I want it to do. And I always, I always found that to be just a really, really fascinating.

And that was one of kind of the hooks that got me into fiddling with computers.

David: Yeah, I think most people that are deep into especially open source, anything where you're, you're tinkering, you're making any of that community, you always have that one core starting trigger or moment.

Jonathan: Yeah.

David: So next question would be for me or for you would be.

So, your first memory is in a Sam's Club. Huh. How long after moving the mouse in the Sam's Club did you move to having a computer in your own house?

Jonathan: Oh, that's an interesting, so, it wouldn't have been too terribly long but I don't, I couldn't tell you exactly how long. Maybe, maybe a year or two.

And so, like, my, my early memories of the computer were I remember the first time that we got, like, one of those sampler disks for dial up. And that wasn't even, like, the full internet. That was some, So,

David: like, CompuServe or something?

Jonathan: Yeah, it was something like that. So it was kind of their walled garden version of the internet.

And it seemed like that didn't ever even work right. So, and then, you know, at some point we did finally get dial up. And I gotta give some credit to my parents. I was not exposed to the raw internet at a young age. I was, there was some, there was some boundaries set around that which, You know, in some cases we're, we're very good because there are, there are things on the internet that prepubescent teens just don't need to see.

But then, you know to jump a little bit ahead, like there was a time that I went to volunteer. I was like 13, 12 or 13 years old. I was gonna volunteer to try to help with a an operating system written in QBasic, which is something. But the way that I wrote in to volunteer was like, I'm not going to give my last name because I'm a little bit paranoid about the internet.

And of course, these adults that we're working on, it just brushed me off as a silly kid. So, you know, that could have been, that could have been, that could have gone a little differently and been, been interesting. Yeah. So like I said, that, that moment in Sam's club was a very, very early memory. And then pretty, pretty early on in life, we finally got a computer and then We finally got, you know, connected to the internet when I was six, seven, eight, something like that.

So pretty, pretty young. And yeah, not to date you too much, but what decade was this in? Would have been the 90s, the early 90s.

David: Okay.

Jonathan: So I am, I am just the right age that, like, I remember, I have some memories of before the internet but at the same time I'm young enough Or, excuse me, I'm, yeah, I'm young enough that I also grew up with the internet.

So, like, my formative years sort of split that, at least in my particular case. I mean, everybody's gonna be a little bit different age based on where you were and how much money your family made, how early adopters they were of the internet. But I, I kind of split that. So I have, I have memories of both. I have memories of, you know, playing on an, on an NES, a Nintendo entertainment system and having to like go to the library or the game store to look up the cheat codes or the guide.

And then I also have memories of about the time the super Nintendo or the N64 came along, having access to the internet and being able to go on GameFAQs and find the same information.

David: So you've got a computer now you've been playing around with it. Huh. Oh. What was the first programming language and project that you started experimenting

Jonathan: with? So programming language, that was QBasic. And that was, that was because my dad was a business major with computer science minor. And unfortunately, he's not done a whole lot with that programming background, but he did, you know, remember enough about BASIC that when we had a computer, and I got to the point that I was interested in it, he was able to fire up BASIC for the first time.

And show me some of the, the simple, simple things about running with BASIC. And so, you know, I, I would, I would write programs that would, you know, tell me your name and then it would print your name back. And then, you know, the next thing you do with that as a, as a kid is, well, let's compare this. And if you type Jonathan, then Jonathan's really cool.

And if you type, Dad, then dad's the best, you know, that sort of, that sort of level of programming. And that, that was kind of where it stayed for a while until you kind of connected that with the internet. And then at some point I discovered kind of that, there was a series of sites. I don't know if they still exist or not, but there was a series of websites around the idea of QBasic.

And so you had things like I want to say one of them was Pete's QBasic site and, and that had sample programs to download, but it also had code examples. And, You know, I, at some point I came across like you can draw graphics with QBasic. And so I started, so one of the first real programs that I really remember working on was, I was very much into Knight Rider at the time, the old, the old 80s TV show, 80s, 90s TV show.

And so I started working on this, like, well, let's do schematics about this really cool car and put text on the screen and, you know, make it look like those 80s graphics. And I spent some time doing that and got into that and found that. Found that to be really, really cool and really enjoyed that a lot.

Um, and then as I got older There was a time where I kind of set programming to the side for a while and really got into like video editing and stuff. Which, you know, is another thing that has served me well. But yeah, the initial programming language was really QBasic. And so, it's interesting, that's where some fun things come from.

Like my editor of choice is Nano. It's because Nano looks like the old Microsoft Edit or QBasic interface. And so when I saw it, it's just what felt to be right at home. And so there's still a few of those, a few, a few ticks that I have that I think came from that early QBasic experience.

David: Well, that kind of Stole the thunder of one of my questions at some point I was going to ask when did you get introduced to vi and why is it the best but

Jonathan: we'll just skip that i got stuck in it for so long i just i'm traumatized and can't go back yeah

David: All right, so What was your next programming language after?

Q basic or basic or any of the basic varieties?

Jonathan: Yeah, so You There there came a point and so like I was interested in doing games. I thought, you know, programming an RPG would just be the coolest thing. Well, if you try to do that very hard for very long, you run into problems with QBasic. At least QBasic, what was it, QBasic 4.

5 is what would actually ship with Microsoft machines, and I'm Because it couldn't compile, right? It would just run in the interpreter, and that was it. There was a later version of QBasic, like the QBasic 7 series or something, that would let you actually compile to an executable. And I, I knew early on, like, that's, that's where it's at.

Your programs can run faster, they're easier to distribute, and so I started looking into that. Well, Along, somewhere along that time, I discovered that C was a thing. And I don't remember why, but in my young brain, I knew that C was the future. Right? Like, that's the direction I want to go. I want to learn C And so, you know, I as a, I don't remember for sure how young I was, under 12, maybe like a 10 year old, somewhere around there, I got this big, thick book of, you know, a beginner's guide to C In fact, that may, it may be sitting up there.

So yeah, teach yourself C and I started going through that book, trying to compile some of the things in it. I will say, while I got partway through it, some of the things in there did not click. Like being a 10 year old and trying to understand what is going on when you're creating a When you're creating classes, you know, your constructors and deconstructors, like a lot of that just, it didn't make sense to me, like, what, what, what do you mean you make a constructor and it's blank and it doesn't do anything?

Why is that there by default? What are you talking? Like, so, there, and, Part of that is because C does some magic in the background. And the book did not do a real great job of describing the fact that C is doing this magic in the background. And a constructor is just where you can add your own magic on top of that ma So there were, there were things that I just, I did not grasp about C at the time.

But that was the next language that I really tried to go into. And then I would say probably the, yeah, I messed around with Perl I messed around some with Python probably the first language that I really did anything, some Java as well, but the first language I really did anything constructive in I think was Perl.

And I think I have a question in the, in my notes about that, and so I'm not going to jump ahead on that story. I'll hand it back to you and let you, let you dig further.

David: Okay. So, actually, one question I did have, you mentioned BASIC. Did you ever mess with Visual BASIC?

Jonathan: I downloaded the Visual Basic program, and I looked at it I think when I messed with Visual Basic, it was, I had two things that threw me off with it.

One, the interface looked very clunky. You know, the version of it that I got, it just, it looked clunky. And so I immediately thought, well, this is not going to do what I want it to do. I want something that's a little bit more up to date. And then also the kind of, when I looked at it, it seemed to be a very simplified drag and drop.

And that also was not really what I was looking for. So I, I, I looked at it and kind of walked away from it fairly quickly.

David: Yeah, makes sense. My exposure to Visual Basic is, when I went through college in the late 90s, that's what they used to teach programming and logic. So, it was I'm sorry? It was interesting.

Jonathan: Yeah, so I mean, keep in mind, so far we are very pre college on my experience. This is, this is all, pretty much all before the age of 13. I, just trying to, trying to soak in the things like a sponge on my own.

David: So just, just to establish it, if it wasn't clear already, I am slightly older than you. A little bit.

But not by a whole lot. A little bit. Ah, so going, so I assume C was still under the Microsoft ecosystem DOS, Windows 3. 1, 3. 1.

Jonathan: 1? Yes, still, still running on Windows. That would have been Windows maybe 95 is where I started messing around with that, somewhere in there. Okay, yeah. Oh, it's interesting.

So you mentioned that there was in the, in the C book, the instructions were for Windows. But there was a little tidbit about if you're running on Linux or one of the other Unix's, you can actually use this command. And I used to look at that and go, I wonder what they're talking about. What is, what, what is, what is this Linux thing?

David: Well, that leads into the next question I was going to ask is what got you into Linux? So what was your, how did you transition from Microsoft to Freedom?

Jonathan: So, so Well, like a lot of such things, a friend got me into it. We, so when I was growing up, we moved around a lot. My dad actually worked as a field accountant for a construction company.

And so about once a year, we would move, move, move. Kind of like being a military brat. And, finally, we moved to southwest Oklahoma, where I'm at now. And I met another young man about my age. And it was one of those times where we just, we clicked, like we just understood we were interested in the same things and we were, and he was a Linux nerd.

And so he finally talked me into, you gotta try this Linux thing, here let me, I think at some point I got a I got a laptop. And he convinced me, let me, let me install Linux on your laptop as a secondary OS. And so, you know, I had it there started playing around with it. And, let's see, it seems like I went on, I went on a vacation, and this was early, early FedoraCore, like a FedoraCore 2, maybe?

I went on vacation to a place that only had dial up internet. And, the laptop did not have support for the modem inside of Linux at the time. Again, early days. And I ended up breaking my Windows install so that I only had a fresh install of Linux on the laptop. And so, like, the entire vacation was me fiddling around with it, trying to make the various things on this Linux laptop work.

in this location where there was no Ethernet cable that I could just plug into. And I've had that experience multiple times. The fresh install and you've got to get, you know, you've got to try to get your drivers working so that you can install your programs. Um, and it, it really, it's an adventure.

And you know, there, there did become, there became a time where I really got tired of Windows XP. So. You know, the Windows XP thing, when you, when you first start with XP, you, on a fresh install, it will always pop up and tell you, Hey, you know, you go to your C drive, like you open Explorer and you try to go to your C drive, and It'll give you this message that modifying files and folders in this folder can be particularly harmful to your operating system.

And that always used to annoy me. And there came a day, I don't remember at what point this was, it was either You know, I was a teenager, and I was either like right before college or very early in college. And, I, I got into the habit of once or twice a once or twice a year doing a reinstall of Windows XP.

Because Windows would just, it would slow down. It would get to the point to where the laptop I was running it on would just get really sluggish. Really slow to do anything. I've kind of figured out since then that that's because I'm running on a conventional spinny hard drive. Windows XP and the hard drive just starts wearing out, and so it takes longer to pull data off the drive.

I'm pretty sure that's generally what's going on there. But my solution at the time was just to do a Windows reinstall, because it would, it would freshen the sectors, and therefore it could pull things off of them faster. The, the drive didn't have to work as hard to try to figure out what the bits actually were.

So I was just in this habit of reinstalling Windows a couple times a year. And I went to do a Windows reinstall, got it. Freshly installed and went to that C drive and saw that message and just it it really irked me It's like what what right does Microsoft have to try to to put the kids kid gloves on me?

Why do I need to interact with my operating system with fleece mittens on it's like this. It's just not right And so, you know at that point I said, you know, I just I don't really need this everything I need to do I can do on Linux. And so, you know, that would have been 2005, probably. I wiped out my Windows install.

It's like, I don't need this. I'll just give, I'll give Linux the full, the full disk. And I've been pretty much Linux only on my computers since then. And yeah, it was a, it was a good feeling. It was neat. It was early to do that. It is much easier to pull that off these days. But, that's, that's when I did it.

David: All right, so there was a question from our gallery and it said, did fixing Linux on vacation meet with spousal approval? But my assumption based on our timeline is that this was well before marriage.

Jonathan: Yes, yes. I was like when when that instance happened, I was probably 16 or 17 somewhere in there when I took the laptop on vacation, and it was it was to my grandparents house.

And so they just a lot of times at their house. I just hung out in the basement. Well, my parents and grandparents caught up and talked about things. But, yeah, there was no there was no spouse to approve at that point. It was before I met my wife at all.

David: Makes sense. So A couple of follow up questions, did you ever mess with Slackware?

It's like one at the original.

Jonathan: No, no, no. I never did anything with Slackware. I've never, believe it or not, I've never done a Slackware install. My, my first distro was FedoraCore. And like I said, it was probably FedoraCore 2. And that's because the guy that got me hooked on it was a fan of Fedora.

And I think that's because his dad was a fan of Red Hat Linux. And so that's just kind of where I got introduced to it.

David: Okay. And then the other question I had, or maybe not question, clarification. So not only were you fiddling around with Linux, but because you didn't have a internet connection, you also didn't have the normal resources.

So you were having to figure it out kind of on your own. In addition to just fiddling with it.

Jonathan: So yes, they had Dial up internet there at my grandparents house. And so It was, it was a lot of me, I had the laptop plugged in, my memory of this is I had the laptop plugged in in one room, and then their desktop was in the other room, and so my memory is me fiddling with the laptop, and then going to the desktop to try to look something up online to figure out how to do something, and then going back to the laptop to try to make it happen, and yeah, boy, things were, things were different back then, things were a lot different back then.

Yes, better and

David: worse, better and worse. Yes. So you had a comment that you bypassed a not so great firewall and this is somewhere between windows and college. So this must be the teenage years.

Jonathan: Yes, the timeline on some of this is a little fuzzy but the same friend that introduced me to Linux he went off to college in Florida to a rather conservative college, and one of the things they had there was they had a firewall that would block So yeah.

All kinds of websites. And some of those were perfectly legitimate websites that he really wanted to be able to get access to. And, so we started probing this firewall to see, like, what could we get through it? And how could we pull stuff off and get him? So I had, I had a little website that, well it was, it was hosted on, heh, it was hosted on an old, old Linux machine that was in my bedroom at the time and It was, you know, it was like a personal blog.

If I, if I remember correctly. And so, we eventually, like, we sent it to the school and said, hey, could you please unblock this website? Well, that meant that, that domain was unblocked. And so, you could get through to it. And so, we started messing around with OpenVPN. Wireguard didn't exist at the time. OpenVPN and FWNOP.

as this sort of solution to try to push a VPN through their, their big stateful filtering firewall. And we made it work! Because we had, you know, we had a website that was on their allow list, and you can set OpenVPN up to be an SSL firewall. In fact, I think that's what it, the way it works by default.

And so you just set it up to act like it's talking to a website. Well, FWNOP, that's the Firewall Knock Operator. That was something I found at around the same time. I, I think I discovered that because at the time I was listening to a lot of Security Now with Steve Gibson. And Gibson did one of his shows on the idea of port knocking.

And the, so port knocking is, you have a firewall that's closed, but it's listening for incoming connections. And so you try to open a TCP connection on multiple ports in a row. And the operator on the other side of the firewall hears those, it's kind of like a secret knock. And if the secret knock matches, it opens a port in the firewall to you.

I thought that was a cool idea, and it worked really well for what we were trying to do. So, I went looking for an open source implementation of this, and the one that I came up with was FWNOP, written by Michael Rash. And FWNOP goes a step beyond just port knocking. In fact, it's still, there are still some, some places where you might want to use this.

It's still a neat project. It goes beyond just port knocking and it actually sends, it does what it calls single packet authorization. It sends a single UDP packet that has actual cryptography inside of it. And so you can then you can, you can check the authorization on the string and then decrypt the string.

And inside of it, it'll have, like, you know, a timestamp and a source IP address and then a request, like, please open this port for my IP address. Well, so what we were doing with it is we were saying, you know, Rather than open this port, it was please redirect this port from my IP address. And so you could send in a request over port 443, and then internally FWNOT would flip that over to your OpenVPN port.

And so it really, it worked, it worked beautifully for what we were trying to do. The, the problem was, and so this, this gets back to the programming thing, the problem was at the school where his internet was it was using a proxy. You had to specify a proxy. And FWNOP, the client, didn't have didn't have support for that.

And so, you know, I put my big shoes on, I said, I bet you I can figure this out, and I went in, it was Perl code. I was like, oh, Randall Schwartz likes Perl. Perl is cool. And so I dug into that for a while, learned a lot about how networking works, and how proxies work, and routers, and routes, and all of that.

And finally, you know, kicked together some code to, in the Perl version of FWNOP, to be able to specify a proxy to be able to send one of these packets through. And that was kind of, that was kind of the first really useful programming thing I did. And that was, you know, that was a lot of, by that time I had easier access to the internet, I had some Linux chops.

And that was just a lot of reading about things that work, reading about Perl, reading about networking, and then going into the FWNotPerl code and reading about it and figuring it out. I finally got to the point to where it worked in the Perl code and sent it in to Michael Rash as a patch. You know, this is the code that I've got.

And interestingly, he, I don't remember if he applied it to the Perl code or not, but they were, at that point in that project, they were doing a transition from Perl to C or no, C, Perl to C FWM is written in ANSI C. And, so he re implemented the patch in C. And I think I tested it and it didn't work, and then I had to go in and fix the C code, too.

Because apparently he didn't have a, he didn't have an easy way to, to test a proxy like that. And so that was my first real exposure to, to doing something useful both in Perl and in C if, if I remember correctly. So that was, that was super interesting to, to kind of take a look at doing all of that.

But of course it was you know, it, it was It's an interesting experience. Your first time, you know, really sending a patch into a project. And, and seeing it, like, it went out to other people. It was, it was really, it was code that other people found useful potentially. And that that really intrigued me a lot to be able to do that.

It was just, it was just kind of one of those, those awakening experiences where you, you understand, like, here's the juice, here's the goody of open source, here's the thing that open source really makes sense for being able to send that patch in and, and have it. Become part of the project.

David: Awesome.

My internet just blew up on me right there, so I missed the last part of what you were saying, but thank you for carrying on. I tried. So college experience, not about the classes. I assume we're not talking about the partying, either.

Jonathan: No, no, no. So, where I went to college was a similarly conservative college, and I, I spent a lot of time breaking and fixing my Linux laptop.

Because, of course, as one does when you're early into Linux. I, I had some fun helping other people with computer problems. There was, there was one instance, so at the college where I was at, down in the coffee shop, I think, they had, the coffee shop and the library, they had some communal computers.

And this was early on, people didn't realize that it would be a terrible idea to tell the computer to save their password when other people can log into it. And so there were, I was not the one that discovered this, I was not the one that really put 2 and 2 together, but a couple of friends of mine did.

And they went, people are saving their username and password in these public computers. And so they went in and they grabbed it all. And that situation did not end as well as it should have. And that's where I learned about responsible disclosure and the way that that should work. But I also got to be really good friends with the Well, he was, he was kind of the audio engineer and they brought him in to run their little recording studio.

And a couple, actually I became really good friends with a couple of people that worked in the recording studio. And, there was, there was about a six month period there where I ran the recording studio as a student. Because I just, I enjoyed it so much and I, I would work for them for free. And so they let me do that.

I learned a lot about, you know, audio equipment and microphones and, you know, just really kind of trying to dive into that world, learn, learn how to mix, learn how to even build audio systems. Really, really dug into that. And then the other interesting thing that happened in college is I came across, I started doing some reading on like, the early open source philosophy, if you were.

So I read Eric S. Raymond's The The Cathedral and the Bazaar and that really resonated with me. And some of those other early works about, you know, this is the way that we imagine open source working. This is why open source is really cool. Did a lot of reading into sort of the history of all of that.

The history of Unix, the history of Linux, history of open source. And so, yeah, for what I am doing today, I not very much of my actual classroom experiences made it, as far as being useful but a lot of the a lot of the outside of the classroom things that happened in college have been very important for me, so that was, that was something.

David: Yeah I, I think a lot of people, unless you get to the doctorate level and you're doing the next iteration of, development, whether that's you know, networking or computing or any of those things, anything beyond that technology is advancing so quickly that college classes themselves may not stick around that long, but that whole learning how to learn.

That's the most important part of that whole process.

Jonathan: Yeah.

David: Another thing I wanted to just quickly touch on, I noticed you mentioned that you kind of dabbled in some video editing and then you got into sound. And there was a question from the comments again, they want to know if you play any instruments.

Jonathan: So at the moment I have instruments. I used to play them very, very faithfully but then I had kids. Got married, had kids, and that just sort of soaked up a lot of the time for doing music. But no, I, I have played the trumpet the, a little bit of the piano, and a little bit of vocal work. By far, I was best at playing the trumpet.

And And then at the moment, I've also got a bit of a modular synth build that I'm, I'm trying to become proficient at, although that is, that is happening very slowly, and as, as so far, that has been more difficult than I expected, but starting, starting to have some fun with that as well.

David: But one of the interesting things, at least the thing that I find interesting, is there is so much overlap between the creative mind and the open source mind.

I mean, you look at the. that you have on floss. The other people that come in quite often and almost everybody has some musical connection or some graphic connection. I mean, I, myself, I've done some mixing. I don't play any instruments. But you know, I've done video editing you know, all that kind of stuff is just it's, it's really fun.

It may not be obvious from the outside, but when you get into, especially programming, there is such a creative element to solving problems and things that there's just naturally a lot of crossover.

Jonathan: Yeah, so I would, I would definitely agree. Being a good programmer does require some creativity.

Particularly when you're like working on When you're working on something really interesting, like that's not been done before there, there is definitely an element of, let's come up with a creative way to do this, you know, let's, let's take these pieces that are out there and put them together in a, hopefully an elegant way.

And that, that, that idea of elegant code is sort of similar to having a creative eye.

David: So I want to jump around a little bit because we're already after the top of the hour and I want to make sure that the questions I'm interested in get hit. So what, how did you get into podcasting and become. A Floss Weekly co host and then a host and, and kind of you, because you've obviously, we'll, we'll, if we have time, we'll touch on it, but you know, you had your own company, you did a lot of IT support, you're still doing it, but now you're doing more of the public face kind of, of this stuff.

How did you get into all that?

Jonathan: So. Let me think about where to start that. Probably probably the beginning of that story is I got into some trouble at college. Okay. And well, so the trouble was, and I know some things now that I sure wish I knew then, it would have really helped me out. But one of the bits of trouble was that my, my grades really started falling. And so they, because they had, they had, the people at college recognized that I had some skills.

And so they put me into some positions doing sound stuff and my grades started falling. And so they, they brought me into this meeting and it's like, things need to change. You know, it was one of those kinds of meetings, but there was, there was something that happened. Boy, this is a another one of those core memories.

It's funny. So it's funny things that you say, maybe even off the cuff. I don't know if he, if the guy that said this to me, I don't know how much he thought about this before he said it, but it really made an impact on me. He was, he was basically describing the problem. And then he says, you know, in this meeting sitting here with us, you present yourself very well.

And so he gave me this, like, this really nice compliment that you're articulate, you make your points well, you present yourself well, you've, you've put yourself together well. And he's like, I don't, I don't understand, what are you, what he was saying is I don't understand the disconnect between the young man that's sitting in front of me in this meeting and the grades that I see, right?

Now, the thing that I have learned since then is I have some health problems. And I was probably beginning to experience a thyroid problem even then, which is why I was doing things like. sleeping 24 hours a day on the weekends to try to be able to make it through classes throughout the week. Oh, I, I had some problems that went undiagnosed for a while, but anyway, we go into this meeting and he gives me this compliment.

And like that became a core memory for me that, that this guy that I respected told me that I present myself well in a marticulate. And so I, I got kind of a confidence boost from that. And finally, you know, I just, I got to this point where I started telling people, I could probably do that. And so let's see.

I got back into programming for FWNOP. In fact, I wrote a a couple of clients, a couple of graphical clients for it. And we started talking about it, and I suggested, because I was, I was aware of Floss Weekly at the time. I listened to it. And so I suggested to Michael Rash, the guy that, that writes FWNOP, we should try to go on to Floss Weekly as, as guests.

And Pitch the program. So we did that. You know, Randall Schwartz was hosting at the time. And just like I do now, anybody that has an open source project, if you write in and say, Hey, I want to be on the show. Yeah, absolutely. We are always looking for guests. Right? So we we got to be on the show. Michael Michael, you know, he was the project lead.

So he always was. was almost entirely answering the questions. They did kick it over to me for, the question was, it was a two part question like, you know, tell us about the clients that you wrote. And I ended telling them about the clients by saying, you know and it's written in C under the WX widgets library.

And the very next question that the co host asked me was, in what language is it? And what is, you know, what library is it written in? So I answered that again. In retrospect now, thinking back on that, it's like, well, that would have been a great opportunity for me to continue answering with a more detailed answer.

But no, I was not quite that aware at the time. So anyway, I had that experience as a guest on the podcast. Well, then Randall sent out a request And it's like, we need more co hosts. They'd had a couple of their co hosts stop. And I sent him an email. I said, Hey, I've been on the show. I feel like I've got a decent video set up.

I've got a decent audio set up and I'm fairly articulate. I handle myself well in meetings. And like, it was part of my thought process. I could probably do it. And so he emails me back and said, sure, we'll give you a try. And I don't, I don't remember what the first, I don't remember who the first person was that I interviewed, but apparently I did okay.

And so then, you know, by the end of it, he emails me back and says, yep, you're part of the rotating panel. And so I would just, he would send out emails. I was self employed. And so I was pretty much always up for doing it whenever, and so I ended up doing it a lot. And then Randall started having some health problems of his own, of his own.

And there were some days where he would just miss being there. He wasn't able to host for one reason or another. And so I was, I was one of the ones that stepped up. And there were, there were a few times that I was the host of the show unexpectedly. I think there was at least one time where I was scheduled to be the co-host and it was just me.

And so I got to be the host with no co-host. That's kind of, you know, throwing me into the deep end, but. Made it work. And then of course Randall stepped down and Doc Sorrells took over and again I was one of his co hosts and then when when Twit had to make the decision to pull the plug on the show I I was already writing for Hackaday, which the story with Hackaday is similar I was a fan of Hackaday.

Oh, goodness, I've been a fan of Hackaday for forever. Very soon after it started, I think I found and started reading on Hackaday. They put out a call and said, hey, we are looking for contributors, people to write stories. And I sent an email and said, hey, I think I can do that. I know how to write. I did well.

I could probably write these. And so they had me write an example story and, and they liked it, so I stuck around. Then they, they knew I was interested in security. So they, they it, it wasn't Elliot. It was it was the previous editor and chief, Mike Stitch, the previous editor and chief said, Hey, we, we know you like the security beat.

You're good at writing up these security stories. Do you want to do a weekly security column? I said, sure. So I've been doing that, you know, every week for several years now. And then when Floss Weekly was going to end, I sent an email to Elliot Williams, who is now the editor in chief at Hackaday. I said, Hey.

Well, how did, how did I put it? Hey, would you like to adopt a homeless podcast? And so we, you know, we started shooting the emails back and forth between, you know, between Twit, making sure they were okay with it, and between Hackaday, and he was dinging his bosses, and finally we got the green light from everywhere.

And so we landed Floss Weekly here at Hackaday. And it's been, it's been a good fit. We've enjoyed doing it here too.

David: So that kind of brings us up to the The today, but specifically about hosting. So, as I mentioned at the beginning, you know, I have jumped into co host and it's fun and it was kind of a similar thing. I was like, you know, I've been watching it. I've been loving it. I think I could do that. So I sent a message to you after I had a setup that was.

What I felt was at least sufficient to get started. We can always improve. But now that I've been doing it for a while. Um, It's, it's one thing to just, you know, you have, you watch it. You're like, Oh man, I have the perfect comeback. I have the perfect question. I have whatever, but now you're doing it every single week.

And you know, how do you keep it fresh? How do you keep it going? Because, you know, you, you do that very successfully. So how do you make that happen?

Jonathan: I appreciate that. I will say that so far, the thing that has been the most challenging is the and in fact, it's kind of funny when, when I was talking with the folks at Hackaday about bringing it here, I mentioned several times, like, I would love to have somebody from Hackaday that just helps with scheduling.

I can do everything else, but I would like help with scheduling and that didn't happen. So I get to do the scheduling too. And so that's, that's the only part of it that. I won't say bugs me, but every once in a while it feels overwhelming is that constant weekly grind. You've got to have somebody ready.

You've got to have somebody ready. And then things like today happen, like where you thought you had somebody ready and there's a scheduling problem and some people don't show up. And it's not the end of the world. I've made some changes that have helped. I'll tell you one thing that really helped with that was we now have a public schedule.

It's a Google Doc, but it's, it's linked to a private Google Doc where I make changes. And so then, you know, I can just, I can email somebody and say, hey, here's our schedule. You pick the day you want and email me back. And that's, that's become, that's made things a whole lot easier. But as far as staying fresh on the show.

The main thing is I really find the topics interesting, and we're talking about a different open source project each week. We're talking to somebody else, and the people that we bring in are generally passionate about what they're doing and the project they've got. And so, you know, that's sort of contagious too.

But yeah, the main thing that keeps it fresh is that I've, I've, I really genuinely get excited about open source things that people are doing. And that helps a lot.

David: Yeah. So we we've got about 10 minutes left or so. So I get to go back and hit some of those other questions that I skipped over because I really wanted the answer to that one.

Host prerogative. Yes, of course. So you mentioned that you worked on phone systems and we've talked about that before a little bit because we both have some history with Asterix and stuff. So what got you into phone systems and

Jonathan: Yeah, so the start of that was, I think where I was going to church at, they built a new building on a shoestring budget, and they didn't have any phones.

And one of the guys that went to church there was a businessman, and he donated his old phone system. Didn't know anything about it. And you know, nobody knew what to do with it, but I was there. And so I started looking at it and poking around on it. And it's like, I wonder if I could find a an actual technician manual for this, you know, not that, not the dinky little user manual, the 15 page thing, I mean the installation manual, the 300 page, and I was able to find it.

I was able to track down a copy of it online and got that printed out. Um, Based on that, and then doing some reading about how you wire up phone systems, I did the install on that phone system, programmed it and got it working. And that was kind of my first introduction. Well I was, I was looking for a job at the time.

This was, this was after my college experience ended. And I was, you know, trying to figure out what I was going to do next. I thought I had a job lined up to work for an alarm company, doing fire alarms and such. And that, that fell through. I didn't get hired there. So, you know, I didn't have anything.

And I mentioned this to the same businessman that donated the system. And he mentioned to his phone guys, I've got this kid that I go to church with that did this CommDial phone system install himself just from reading the manual. And they were like, he did what now? Is he looking for work? So, I got hired on there and did some CommDial stuff, did a lot of ODAVI stuff, did a lot of cabling.

And that, I don't remember exactly how long I've worked there, but about a year, something like that. And that job came to an end. And so, you know, I'm then. Unemployed, but I'm still living with my parents at the time, and I kind of look and I go, I've got this set of skills. I now know how to do phone systems.

I know how to do Linux system administration. I can build the hardware for it. I know how to do audio systems because I did a lot of it in college. Those things kind of fit together and they might look nice on a business card. And so, you know, as a result of that, I, as I say, I hung my shingle. Well, I started the business and I, I've, I wanted to come up with a really fun name for the business.

And I've always been kind of a star Wars fan, the, the, the extended universe mainly I will never forgive Disney from, for decanonizing the extended universe. That's a different topic. And so, you know, I was trying to think, like, I'm into sci fi and it's a computer business, so there's got to be a name to come up with.

And the name that I came up with was Incom Systems, I N C O M. And that is from the ship manufacturer, Incom. They made the Incom T 38, which Luke famously Learn to fly on and so that's that is where that came from But so yeah, I put the shingle out and I said, hey, look I do I do phone systems. I do computers I do sound systems and I've done a few phone system jobs, very little work on the sound systems, but a lot of work on computers.

The, and it's, it's funny, my, my real break, like, the, really where I broke into the Lawton market with doing computer stuff, is I started by just going to businesses and handing my business card out. And I went into this doctor's office, and the lady that worked at the front desk was like, well, yeah, I'll take it for the office, but you don't happen to know anything about Linux computers, do you?

I was like, well, yeah, yes, I do. And she had one of those little ASUS E netbooks and it had dropped its wireless driver. And so I, you know, I took it from her and went down to Starbucks and sat there in Starbucks until I could get the wireless driver working again and brought it back to her. And so then.

The next time the office needed something, I was, I was like the only one that could fix it that she knew of. The next time the office needed something, she's like, No, no, no, we're using this guy now. So, sort of working on their stuff and it just kind of grew from there. So it's, it's, it's all Linux all the way down.

David: Yeah, once, once you get that first satisfied customer, then you've got that word of mouth helping you grow.

Jonathan: And so what really happened there is very soon afterwards the cable company in town got bought out or maybe about the same time the cable company in town got bought out. And so there was a new, they brought in a new business salesman for selling Internet service, cable internet service.

And that doctor's office was one of his first customers. And so I got to work with him doing the transition over to cable internet. And he liked me and then he started recommending me to other businesses he was selling to.

David: Awesome. So how, how did you, was it just simply searching to, to make the connection because you knew Linux, you knew phone systems, but how did you find out about asterisks and how did you transition from, you said it was Comtel that you were originally installing?

Jonathan: So my first install was a, a Comdial, I believe. Comdial. Yeah. Yes. And that's because that's what we had. And then. Worked with Vodavi, because the business I was working at, that's what they installed. And, but when I went to start my own business, I, I knew I liked that idea of open source and doing it on Linux.

And I don't remember if I was aware of Asterisk first, or if I found it because I knew there had to be something out there. But anyway, I started, I started diving into that idea of, well, what if I did a business phone system based on Asterisk? Surely I could do this cheaper, right? The answer to that is sort of, by the way, it depends.

But no, I, I, you can do a free, you can do a free download, a free install of Asterisk and you can put it on, you know, just about whatever hardware you want to. So I started doing these crazy fun experiments with like, I would have Asterisk installed on a home computer and Asterisk installed on a laptop.

And then I would go to Starbucks cause they had wifi there and VPN and try to connect the two and. So at one point I was sitting at a Starbucks with my little Asus E, the little tiny white computer sitting on the desk in front of me, and then a VoIP phone sitting beside it, plugged into the laptop and fiddling with it to see if I could make phone calls on it.

Which I did eventually make work. But there were, there were all kinds of problems with it. Like it didn't work well, but I did eventually make it work. And that was, that was very fun.

David: And then you quickly discovered that VoIP over wifi is not. So

Jonathan: yes, actually, I think the thing that hurt me the most there is that I was trying to use IAX, the inter asterisk exchange to go between the two asterisk machines, and that only works if you have really, really tight timing between the two.

So, like, you've got to have external hardware to provide that timing pulse. Otherwise, you know, the two machines fall out of sync and you get all kind of clicks and pops.

David: Yeah, IAX is not very forgiving in fact, I believe it's even been deprecated at this point. They're recommending everything go to SIP.

Jonathan: I think But, that's a discussion. I think they still use it for some things internally, but yeah, SIP has definitely taken over that.

David: Yeah, I'm actually in the middle of a project now where we're Swapping out machines to convert from IAX to SIP. Staying with asterisks,

Jonathan: but

David: just converting.

Jonathan: One of my fun business stories, this one is just so much fun, I'll tell it.

I work with a group, I am the smart hands on the ground every once in a while for a group that manages phone systems, mainly in hotels and they, they sent me to a local, fairly large hotel and we were doing a, an equipment exchange, and it was a It's called a Brain Box, which, come to find out, is just a one use server with CentOS on it.

It's been several years ago now, with CentOS on it, and then Asterisk running on it. And so, essentially what it was doing is it was sitting between the phone system and the upstream telephone system, and it was also connected to the internet. And it was doing things like doing live lookups for a long distance call.

How can I make this call the cheapest? That was the sort of thing that it was doing. But we went to do the install, and they couldn't get into one of their boxes. They're like, we can't get into it. I had already figured out that it was an Asterix. It was Linux. It was CentOS. I'd already figured that out.

And they're like, we can't get into it. We can't see it come up. I said, well, do you want me to break into it? Like a what? Well, I've got a monitor here, and there's a keyboard there, and I can probably just tell it to start at run level one and break into it. And the guy on the phone for BrainBox was like, if you think you can do that, that would be nice, actually.

I was like, sure. Plugged everything into it, you know, got to the boot screen, you, you interrupt the boot. I don't think this works anymore. I know at least on Fedora they've made it more secure than this, but it, you, well, I say that. There are still ways to do it. Anyway, you can just interrupt the boot and tell it, you know, start at run level one.

And And when you start at run level one, it'll let you log in. And at that time it didn't ask for a password. And so I, I did that. I broke into it and I'm like, okay, so this is the information you're looking for. And then the guy goes, Oh, we got them backwards. And he made the change on his side and everything was good to go.

But yeah, I broke into the box. The guy calls me back later once I'm back at home, and he's like, We were really impressed with that, by the way. I'm gonna send you a goodie bag. And so he, I don't, I've got it here somewhere. He sent me this little squashable brain that's got their, their logo on it and some other fun stuff.

But yeah, that was, that was a lot of fun. He's like, yeah, I can get, I can break into that.

David: Well, that is a fun story to wrap this up with because we are at the bottom of the hour. So. I shall do my best to end this properly by asking you the two most important questions. What is your favorite text editor and favorite scripting language?

Jonathan: Oh, okay. So this is a complicated answer and it's changed over the years.

I still, from the command line, my muscle memory default is to go to nano. I have, however, for programming work on a desktop, started using a lot of VS Code. One of the projects that I work on, they are pretty much a VS Code shop, and I've gotten to the point where I kind of like some of the things you can do in VS Code.

Certain things about it still drive me crazy, but I do a lot of, like, desktop programming in VS Code. And when it comes to scripting language, I, you know, I don't, I don't know. I do a lot in Python. I do some in Bash script. I'm, I'm actually pretty excited to see what happens with Amber, Amber script.

And that's the guest we had just last week. So I had to write a tiny little script for something between now and then. And I did it in Amber script. I'm like, well, let's just try it. And it took a little bit to get it to work, but you know, I, I figured it out and made it in Amber. I kind of like that.

So, you know, for system scripting I don't know, there's a decent chance that Amber is about to, about to become that choice. So, I don't, all that to say, I do not have a single favorite scripting language, but if you had to make me name one, for right now, it's probably, probably still Python.

David: That's the correct answer.

Now so, quick follow up on the VS Code from the audience. Why not use VS Codium? Or R?

Jonathan: I don't remember, honestly don't remember if my install is VS Code or VS Codium. I, I, I very much like the fact that VS Codium exists. Very similarly to the fact that I really enjoy and like that Chromium exists.

But I know I do run Chrome and not Chromium, because there are some of those extra plug ins and stuff that you can't get on the other that are kind of necessary. So, I'm, as, as I have said to multiple people when talking about the show, we are, and I particularly, am an open source enthusiast because I'm an open source enthusiast.

But not a purist

David: Well, I hereby officially hand the Host hat back to you and allow you to wrap this show up. Thank you. It was enjoyable for me

Jonathan: Oh, there was a lot of fun. I think there's some things that I some of those stories. I've never put out publicly We've never quite done that everything put together like that. So that was a lot of fun, too I appreciate you being here David was not the co host that was scheduled for today He kind of stepped in at the last minute and I appreciate you being here, sir.

Definitely the hero of the hour

David: Enjoyed

Jonathan: it. Yeah, do you anything you want to plug or mention before we let the folks go? You

David: Not really. I didn't have anything to prepare because it was such a short timeframe. So, Hey, well, I will say this. I always plug Twit join them. That's seven bucks a month which is less than most coffees per day.

And And we've got great shows over there and the great discord.

Jonathan: Yeah, so he's talking about ClubTwit. You can, you can get to Club, you can get to Twit content for free, most of it. But ClubTwit is their their paid subscription, essentially, to help where folks get to go and help support the network.

Because, yeah, I'm not sure if you've heard, but the, the, the revenue from advertising and podcasting has just kind of exploded downwards in the last couple of years, so they, they are struggling to figure out what makes sense for them going forward, and ClubTwit has been a big part of that. We do have the Untitled Linux Show over on Twit, which you can, you can get to that without being part of the club now, but to get, I think, the video feed from it, And to be on the Discord, you've got to be part of Club Twit.

And we would love to see everybody come and hang out with us there, where David is one of our sort of rotating co hosts over there. Yeah, I think that is the main thing that I want to plug. We sure do appreciate Hackaday giving us a giving us a home for the Floss Weekly podcast, and I think that is it for this week.

Next week we will be back. We've got a we've got a really interesting interview coming up next week, who I can't remember what it is. I'm doing the, I'm doing the Doc Searles thing. I don't remember, so I'm going to quickly try to look it up. Of course, I use Google Docs, so it's kind of slow. And Oh, yes.

Next week is Rust core utils. I'm looking forward to this one. We're talking to Sylvester Leddrew about Rust core utils, which for those who don't know, that is the re implementation of your basic Linux tools in Rust instead of C and C That's going to be really interesting. Yeah, so thank you to everyone for being here.

Those that caught us live and those on the download, we sure appreciate it. And we will see you next week on Floss Weekly.

Episode 790 transcript

3 juli 2024 | min

FLOSS-790

Jonathan: This is Floss Weekly, Episode 791, recorded July 2nd. Better bash scripting with Amber. Hey, this week Dan Lynch joins me and we talk with Pavel Korash about Amber. It's a new programming language that's designed to make a bash just a little easier to use. You don't want to miss it, so stay tuned. Hey folks, it's time for Floss Weekly.

That's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something really fun to talk about today. But first off, we have a co host, of course. I am not running the show solo today. And it is Dan, Dan the man, Method Dan, the Linux outlaw. Welcome, sir.

Dan: Hey, thank you.

Good to be here. Always good to be here.

Jonathan: Yeah, it's always good to have you. A lot of fun. A lot of fun getting to co host and do the show with Dan. And this show is kind of right up your alley as one of the other Linux guys, because we are talking about Amber, which is sort of a replacement for

Bash scripting.

Not really a replacement. Maybe they should have called it Bash or something like that. Have you gotten some time to play around with it? I haven't.

Dan: Only today. Yeah, I had a look today. We we were talking pre show there about my confusion when I first looked it up with another project, which was completely my fault.

I thought we might be talking about molecular biology, so I'm quite pleased that we're not, because my, my molecular biology is just not Great. It's, it's not quite up to scratch. Yeah, but Amber's a really interesting project. As you said, I'm not sure I'd call it a replacement for bash so much as an extension to bash or I don't know.

We'll have to ask what, what the kind of correct term is.

Jonathan: Yeah. So it actually, it reminds me a lot of I can't remember the name of it, but there's, there's another language that. Is a way to do a little bit nicer CSS. And then you compile it down in actual CSS. And I can't remember what they call that.

But it, you know, it's, it's the same idea. So what Amber, from my understanding, what Amber does is you, you write your bash script in this kind of, a little bit more structured, a little bit nicer scripting language. And then you, you tell it to compile and it will spit out it. Bash code. Or, you can actually run Amber scripts directly with the Amber executable.

And so there's just a couple of different ways to go about it. And I've actually got a feature request already in the Amber github. And that is, boy, it'd be really nice if you could do this with a shebang. And currently you can't because when the Amber program runs and tries to interpret the script, It sees that first line, the, the, the, the hashtag, hashtag, exclamation mark, if you will.

And it goes, this is not valid Amber because it doesn't see the hashtag. It doesn't see the hash as a, a comment. It tries to interpret it as basically like a, if I remember correctly, it tries to see it as almost a pre processor directive. And so it just, it errors out. And so I, you know, I opened up a feature request and say, Hey, it'd be really nice if you could just run amber scripts, like you run bash scripts.

And I think that's coming. We'll get the confirmation here in just a minute. Let's not let's not waste any more time. Let's go ahead and bring the man himself onto the show. Powell. Well, welcome, sir. Welcome. Hello there. Hey, so first off You, you said before the show that we're going to have to forgive your poor English.

And as I said, your English is, is better than a lot of Native Americans that I know. And but where, where, where are you from and what, why would, why would we potentially have to forgive your English? Well, so

Paweł: I'm from Poland, so there's the stigma that many Poles have that pronunciation.

When it comes to like speaking in English, so I don't know. That's just basically I wanted to Clear that out

Jonathan: Well, I I've got to say it is it is not a problem We we do occasionally have guests that we have to work a little bit to understand But I don't think we're gonna have that problem today. So tell us about tell us about amber What what's the problem you're trying to solve?

Like what's the the origin story for this super villain that we call amber? You Okay.

Dan: Superhero. So

Paweł: I'll start with With a moment when I found out that I need Amber. So I was building like some project that needed some automated workflows. And the problem was that it actually I wanted to write some code that is like it can.

Automate some file management and maybe do some API requests, but it never worked out very well. When I was writing it with Python, I was feeling like I'm doing some kind of wrestling with Python. It's not meant to, it's not designed to do this. And so I thought, well, I want to do something like, I probably should use bash because I'm like doing a lot of bash calls and I'm interacting with the STD out and like, you know, managing ever like the data and, and all that stuff.

So I thought, well, let's do it in let's, let's build, let's find some language that can do that. Like some, Something else that is more advanced than bash, maybe something that is more like maybe designed for, for developers, not maybe not necessarily for people that are tinkering with bash because bash as it's Itself feels like it was designed solely for shell interaction.

So you write commands and and, and basically you, you interact with it just like that. Right. Like you write command, you see an error, you. Take a different approach, probably you, you do something else when you write a script in bash when you type a command that errors inside of that script, it's, it's it goes to the next command and that's, that's pretty that's a bummer because I would assume bash would stop when it fails and it does not.

So so that's one problem that I had with it. I didn't want to sit around and debug bash my bash script as it was not very necessary to, to spend that much time on it because like my project requires much more other So I just thought, well, there must be some kind of a language that does that.

And I just, I found out that there is none that I have to I thought I'm going to create such a, such language. And yeah, I spent just to solve one problem that could, I don't know, maybe save me like 15 minutes. I yeah, I sat down and coded this language for two years. Haha. Yeah, . Excellent. But it's, yeah, I was, it wa wasn't like two whole years of development.

It was like more like when I had a free time. 'cause I'm a st I used to be a student as well. I graduated sometime ago, so Congratulations. Yeah. ,

Jonathan: computer science student.

Paweł: Yeah. Computer science student. It was like I graduated with a bachelor's engineering. Degree it was a week ago.

So, yeah. Yeah. So, and okay. Okay. So one, one interesting thought, one interesting thing is that Amber, it was also a thesis for for my

Jonathan: degree. So,

Paweł: yeah.

Jonathan: Yeah. Neat. So the, the Amber language it's, is it written in bash? Have you gotten to the point to where it's written in Amber?

Paweł: No, no, I don't think we're going to do that.

It will be way too slow and way too hard to implement this since bash has many limitations that it just,

Jonathan: it's,

Paweł: it's really hard to, to implement this same thing.

Jonathan: Well, to be fair, I don't think

Paweł: bash is written in bash either.

Jonathan: Yeah, it's true. So what, what language, what language is under the hood? What did you go with?

Paweł: I, I picked up Rust because yeah, it's, I, I doubled with C plus plus, but I ended up with dealing with segmentation faults and I thought, well, it's not for me, I'm just, I'm just going to use Rust.

Jonathan: Yeah, I feel that. I understand that. So. You can, let's see, so we can, we can compile Amber to bash code.

Like that sounds fairly complicated. How does that even work?

Paweł: Well when you run Amber compiler, it reads to the standard inputs. Or maybe you, you provide a path it reads basically the Amber code and. Uh, it tries to parse it and creates an AST, AST it's abstract syntax tree. And it like uses a different modules that like there are trying to detect if the syntax that is, that it's trying to parse is actually this syntax that it's trying to parse.

If not, then it's like letting the compiler to go to, to pick another module. And and works like that. And it creates this tree that is then that the translator module is trying to yeah, basically iterate through the, all the notes and, and translate it to the bash. So this is how it works.

It's pretty simple. I think it has the. Only two phases. So it's basically parsing and translating. It's yeah, I think it's enough for that, for, for

Jonathan: this reason. Yeah. The, the, the bash code that it spits out is that I would almost expect that it would be not particularly pretty bash code. Like it's got, it's gotta be a little extra verbose to be able to get all of the, well, to match the features that you've put into Amber, right?

Paweł: Yeah, that's true. So basically Bash does not support importing separate modules. I mean, it does via sourcing files, but I wanted to I don't know. I just wanted to prepare myself for maybe because like we could have different files that have the same variable names or the same functions that are being exported, and since we don't know if there's going to be a clash, because you can import the same function from two different files that have the same name, With as keyword, which, which changes the name of the function.

So we can like import foo function from the both files, but you can change the name and the, when you import. The function and the source code to like a or B, right. And you have two different functions that do different things. There are from two different files, but they have to exist in the same like space, right.

They cannot override each other namespace problems. Yes. Oh yeah, exactly. That's, that's the main reason why you might see the weird numbers and, and resulting bash file. However I think that we, we can improve this. We can Frank for instance name the, like add the numbers only when there's some kind of clash.

So this would, we'll basically preserve everything that it is like to make it human readable. And then we'll like try to find out where the clash could potentially be, and then add some kind of numberings or, or I don't know, maybe Maybe append a path to the file that this function was used in or

Jonathan: whatever.

Yeah. And so with Amber, you can, so you can, you can compile it down to bash scripts and run them that way. Can you run Amber code directly?

Paweł: Yes, it's, it's. It actually compiles to bash and it runs like it invokes a bash and it's just that way. So there is no interpreter. Um, if we, yeah, yeah. I was thinking about making one, but I thought that not for now, at least maybe in the future for now, it seems like an overkill.

Jonathan: Yeah. So I was, I was going to go down this rabbit hole of asking a question about like, does the What, what the Amber interpreter, assuming that there was one, what it does with the code, is it exactly the same as what Compiling it to Bash and then running it through Bash would do with the code. And then, you know, you have this, like, you probably would need this big test mechanism to run through a bunch of edge cases and make sure that your interpreter is going to do the same thing that your compiler does.

And, like, languages have taken that route. It's just, it can be a lot. Yeah, it's, that's true. I took the easy way, so. There is, there is something to be said sometimes for the easy way. Yeah. So. Yeah. You started this originally to, to solve a, to solve a problem. Are you actually using it? Do you make use of Amber yourself to, to, to, you know, maybe even you in your day job to fix things?

Dan: Hmm.

Paweł: Well yes, I at my at work I use, I have this like kind of a script that is also, that is like. Turning up my IDE that is opening up my browser that is also setting up because I'm a web developer and it's like boots everything up and also create some aliases for, for different commands for Git commands and all that stuff.

So I use it for that. And also we have written an installator. Insulator installer for Amber in Amber. So it's pretty, it's pretty interesting to, to see that we net, we didn't like write the compiler itself in Amber, but we all, but we have written an installer in Amber. So I think that's, that's an equivalent of writing compiler in the same language.

Yeah, we think so. Cool.

Dan: Yeah, that sounds, that sounds excellent. I was interested in obviously you're targeting bash, which makes a lot of sense. Is there any plans to look at any other, any other shells like fish or any of these other fantastic ones that people are into or are you focused on bash?

Paweł: Well for now we're actually want to, because bash is the most like popular option it's installed, it's pre installed in many distros and I think that targeting bash is a good idea. However, some people may not want to use bash and for that reason, for the maximum portability and yeah, we want to also probably in the future support SH itself.

So, so that you can run the script anywhere. And most of the shells that are like custom shells, I think they support SH out of the box. So so we can also, yeah. So we can basically compile the same script to, instead of bash to just compile to sh and it should be good. The sh also lacks many features that bash provides, such as arrays and associative arrays and and many other things.

And that would be a pretty tough thing to, to, to come around. I mean I think that developers, I mean, either we should figure out something to to work around, to work this around, or the develop, we were just going to hand it over to developers and say, oh, you're using array. Well, it's, if you're compiling to a Sage, it's, you're in, we're unable to to translate that.

So we'll have to figure it out. Maybe some ideas will arise, but yeah, we'll see, to be honest.

Dan: Makes sense. Now I noticed you said the, the, the word we there as we'll have to work it out. Is it other people working on the tool with you or do you have a team?

Paweł: There are a couple of maintainers that are also on board with us.

So when okay. Before the, the whole, like when, when Amber got popular there was only, I mean, only I was like working on it. And so the development of it was pretty. Slow because I also didn't have time every, every day, like to, to sit around and just, you know, at least type one line of code to keep this, keep pushing this project.

Yeah. So that, that was very very good that I was very blessed that so many people came around and started, started building pull requests and, and right now they're like. I think five maintainers besides me. So

Jonathan: that's good. Yeah,

Dan: that's, yeah, that's quite a lot of maintainers for a, you know, a new project, such a new project.

We've seen other projects in the past that have ended up with one maintainer who's then disappeared. And then, you know, obviously that's not good. But yeah, that's awesome. So, um, you mentioned that you, you, you have a team working on it and stuff. Do you have any idea how popular it is in the user base?

How many users? I mean, not in numbers, it's hard numbers, but is it, has it, has it exploded in popularity for you?

Paweł: Yeah, I mean when it comes to the overall installer in installs of Amber, they're like, 2, 500 or so, something like that. When it comes to the visits to the website, cause I, I'm also like trying to, I also track these, these numbers just to see what can I improve?

What, what's what is most interest interesting for users and all that stuff. It seems that there's like. 40, 000 so far visits to the website, which is pretty interesting. But overall I don't know. I'm I think that as we move on, as we add features, as we patch some bugs people are more likely to, to hop on the train and start using Amber.

Not sure if if that's enough, I think we're, we're going to improve it even more. And maybe at some point there will be at, at a stable release, maybe some people will, will, more people will, will come and use it.

Dan: And what about the, the kind of type of users that you see? Is it mainly power users or is it good for people who want to learn?

About bash or what's the kind of spectrum like that?

Paweł: Yeah. So there's two different types of users. Yeah, that's their power users that are also looking for an alternative to just write some scripts that are perhaps updating their system and, and, and the backgrounds, or maybe some kind of a bot detector that a, a script that checks if.

The connection that is created with the server is a bot or not. And well, there are also some people that wanted to use it in, at work for managing some small environments like Some kind of Docker containers or maybe some, some, some small environments that do not have enough maybe space and memory to install Python or any other external tool, but also they need to automate some things and To use bash, it's, it's like, it's, it's not that convenient.

So they choose to use Amber and just compile it, compile the code and send it to, to this law, to this little environment. Yeah.

Dan: Wow. That's very cool. So you could also. I'm wondering about like embedded spaces. Like that's what you're talking about. There is like embedded areas possibly as well. I know dockers not necessarily, but I was thinking about like, maybe people would use it for small devices, as you say, like and target because you, a lot of them don't necessarily have.

enough room for, for, well, they have room for bash, but they don't necessarily have room for a lot on top of that. So how, how much kind of, could I, obviously you can generate the bash and send it, as you said, then how difficult would it be to install, install Amber on, on a small device? Does it need a lot of resources?

Paweł: Well, the thing is that you don't have to install it on a small device. You can just compile it on the, on your computer, on your machine, main machine, and just send it to the, to the little device that you have. To the smaller device and you know, debug it and see if it works. And develop the code this way.

It's it's a good alternative, I think.

Dan: Yeah. Yeah. That makes sense. So I was interested in some of the, some of the advantages that you get from, from using Amber, because as we mentioned at the top of the show, Jonathan has been using it. I've, I've I've not unfortunately had time to do much with it, but I will be after this.

Don't worry, I'll, I'll certainly be trying it out because I was looking at things like you've got like type safety, which is interesting. So Amber can help you, I can help you with better scripting practices in some ways that is that fair?

Paweł: Yeah. When I was building the, the language, I was thinking that, well, it will be cool if this language also supported libraries, like building libraries out, out of the box.

So people could build some kind of like ready solutions for for given problems that are also typed so that the users of these libraries can, See what the function requires, like for example, number and text or whatever. And and that would be cool because I wanted Amber to be more of a automating solution, not necessarily a wrapper for bash.

It, we have to start somewhere. So, so Amber is, is pretty limited at this point. But I think that once we progress the the whole ecosystem will start the, the whole type safety will make more and more sense and people will be able to, because right now they're only primitive types that can, that people can use.

And I think that if we. As the, yeah, as the time goes on Amber will gain some kind of, um, some way to create types, many types, like for instance, for exit codes for different maybe enums or whatever we'll, we'll have to discuss it on, on GitHub probably because It would be nice to, to get the feedback of the people that actually use it.

So, yeah, but that's, that's the, that's the main idea behind this.

Dan: That's awesome. And, and what was there any languages that you looked at that we particularly took like inspiration from when you were trying to create Amber?

Paweł: I was inspired by. Mainly, mainly ECMAScript, which is JavaScript, mainly because it's super popular and people are familiar with the syntax.

I also liked the some of the Rust features like I like that loops and Rust I mean, at least the infinite loop starts with the keyword loop. And it was like, it, it makes sense. So I, so I use that as well. And instead of string, I decided to choose a different name for, for string. I decided to use text because I wanted to give it a little bit of personality and, and, and it's, it's own vibe or its own flair.

So, yeah, so that's, That's the reason that these are the languages that was inspired by.

Jonathan: I'm, I'm curious if there's anything. So it comes to mind that you could, if somebody really wanted to, you could do something crazy like use Amber as a replacement for PHP and write out websites with it. Like, there are some interesting, and I don't know that that's a good idea, that anybody would do it, but like are you sort of thinking about like this, maybe having a life beyond just writing writing shell scripts?

Yeah, I

Paweł: was thinking about, because you have this GitHub workflows and GitHub actually, no the GitHub webhooks or something like that. Well, basically it's this tool that can make an REST API and, and, and do a request. And I was thinking like, well, that would be cool to create some kind of a simple HTTP server that can X accept some GitHub requests that it sends and, and do some automations based on it.

So I think that we're actually going to do some kind of a server ish things. Maybe, but, but that's probably In the future, probably distant future. We don't know yet. Um, maybe that will be some kind of a, a separate project that is not a part of the compiler itself.

Jonathan: Yeah. So I'm, I'm also real curious is everything that Amber does.

And so I guess there's a two part question about what does it do right now? And what are you looking to for the future? Is everything that Amber does just Bash and the built ins or is there anything that like has to make a call to SED or AWK or GREP? And if so, or if you're ever going to do that, how do you, how do you handle the dependencies on that?

Paweł: Well, that's a good question. We, for now we actually use, we depend on BC for handling numbers. BC is a basic calculator. It's basically a. Arithmetic calculator with decimal precision. But yeah, we want to I, I think that we want to get off all of these dependency dependencies to the maximum because just to, just to improve the portability of Amber.

And we'll probably find out some custom solutions for that. For instance, when it comes to numbers with floating points, because bash does not support floating points, we could use some fixed precisions some kind of like functions under the hood, just to, just to at least emulate it in some way.

But yeah, that's that's that's, that will be a tough tough task to do. We'll see if it's possible to just ditch everything and, and everything is going to be all right. For now, Amber does not use like a lot of dependencies, so it's not the best, it's, it's not going to be that hard to, to accomplish.

Right now there's like sed, sec and BC. Yeah. As the dependency. So it's not a lot. And I think we can cut some of these off and

Jonathan: it'll be good. Again, I'm not going to tell you that this is necessarily a good idea, but something that comes to mind, you could actually write a Amber helper for, you know lack of a better term.

And those bits that you need from sed and the bits that you need from you know, bc could live in this little helper binary. And that actually could be really interesting in moving away from bash into other languages. That might be a way to approach it. Is that something that's on the radar? Yeah, that was my idea, actually, thinking

Paweł: about it.

There you go. Yeah. But the problem is that you're actually installing some binary in the end, and that might be hard to, you would have to maintain many like many targets, compilation targets. Yeah, I heard from people that it's not the best like they don't like the solution for this They would prefer to use some kind of shell scripts shell code embedded But we'll see I think that if there is nothing we can do we can always ship the binary and see how it how to how it works

Jonathan: Yeah, I suppose that sort of defeats the idea that You want to just export, you want to, you want to compile to bash code that doesn't require And you also

Paweł: have this problem that whenever you run the, the, the, the Amber code, it has to fetch from the internet, the binary or from, from somewhere.

Or if we embedded the binary into the shell script, it becomes unreadable, which is also a, a Yeah. Yeah,

Jonathan: it's a deal breaker. Yeah, it's a, it's a, it's a challenging, challenging problem to try to work around. Yes. Okay. So how does, how does Amber handle the, I'm, I'm still, I'm still kind of trying to wrap my head around the way that the language itself works.

Because in, in bash, so just for instance, in bash, you can just, you can go from bash scripting like in one line to directly to, and then here's the program I want to call with all of its arguments. How does, how does Amber handle that? Can you just have a, you know, a call off to a binary as aligned by itself or does it, does it require a little bit more syntax than that?

Paweł: Well there's this command syntax that enables you to call anything from bash. So whatever you, your rights. And the aim and the command expression, it's basically a, you write a bit between two dollar signs. It's like a, like a, like a string, but it's for, for bash, for execute, executing bash. So your rights.

The command here, and then it requires you to also handle the error because in Amber, there are entities that can fail and there are entities that are safe. So the failable entities, which are, which are for instance, the, the commands or functions that are failable, failable, because functions can also fail.

And that is also something very interesting about this. You have to also handle it somehow. And there are many ways to, to, to handle the, the errors. You can write the failed block, or you can, you can write the unsafe, which basically treats this command as if it would never fail. So it's usually for, for, for the times where developers is like, I think that it's, it, it's never going to fail.

So I'm going to just leave it like,

Jonathan: like that. It's like running cat. It. If cat fails, you've got bigger problems on your hands. Oh, yeah. Unless the file permissions are Well, okay, so there's, there's, there's a few cases there, but so we have a, actually a question from the chat room. Mashed potato wants to know Is having a good understanding of Bash recommended, or can one just jump straight into Amber?

This reminds me very much of the question, do I need to learn C before I learn C

Paweł: Well unless you don't want to run many I mean, it's, it's always a good nice to have to, to, to know some basics of bash or at least some SH or how to run commands. But other than that, you don't have to like know every single, like every quirk about bash.

You can just invoke the basic commands that you need. And for instance, assign the results from stdout from, of, of your commands to a variable and just, you know use a standard library. That is also an experimental as of right now. But in the future, let's say you could use the standard standard library and just, you know, manipulate, manipulate the data the way you want and, and I think that's It's, it's pretty powerful because the syntax is very easy easy to, to pick up and, and I think yeah, it's, it's okay to, to, you can start with it and, and see if it's easy for you enough

Jonathan: yeah.

So I want to ask about some. Odder places that people might try to run Amber or at least that I immediately comes to my mind to think to run Amber and so the first one we we mentioned it briefly earlier and that is the idea of embedded places So like on my router running open wrt or other other places Places like that.

Instead of an actual bash, you may just have BusyBox. And I'm curious, do you support, is there, is it going to work, or is it just going to fail spectacularly trying to run the exported code on some place like a BusyBox install?

Paweł: Oh, I never tested it. Not really sure. But my guess if it has,

yeah, I'm, I'm, I'm not, I'm not really sure about that, but I think if it has bash install, it should work. Yeah. I mean, unless you do something. I don't know, you mutate some you, you, you I don't know. You try to create some, something you do, you try to do something that is legal or

Jonathan: yeah. So the crazy thing about that is that BusyBox, BusyBox is an all in one binary.

For embedded devices, embedded Linux, and it, you know, it behaves differently depending upon what command you call it with, but they're all symlinked down to BusyBox. And one of the things that symlocked, symlinked to BusyBox is Bash itself. So, you know, you're, you tell your system to, you know, run this with Bash, and it's actually BusyBox that picks it up and runs it.

And you know, on an embedded device like that, you're probably not going to have sed or grep. Maybe, I mean, you can compile them, but you know, we're talking about targets that four megabytes of flash, it's a little minimal in the future. Ideally, I think that you would compile Amber to SH and just, you, you would just run it with the BusyBox.

Yeah. And you, you, you kind of imagine that anything that SH supports, hopefully BusyBox is going to so yeah, I, I guess officially have a bug in your ear that when you think about doing this also do some testing with BusyBox.

Yeah,

cuz it'd be fun. It'd be fun to run run amber scripts on on open wrt on the router.

Yeah. Yeah Now what about some other even crazier places? What about the BSDs? Are we are we thinking about being able to run amber on OpenBSD, FreeBSD,

Paweł: NetBSD? Yeah. I've heard that some commands that we, that we use are not supported on BSDs out of the box, so we'll probably have to double around this.

But, you know, I'm not really sure if that is that hard to integrate BSDs into with, with with Amber. I think that. Um, it's not going to be hard. It's we, we just have to like I mean, we could, we, we don't even have, have BSD version comp like the, the compiler is not built for BSD as of right now.

We can, we could add the targets and, and then try to see if it's, if we can support it and all that stuff. For now, the only BSD that we support is Mac OS. Which is not a BSD, actually. It used to be, but

Dan: yeah. Yeah,

Jonathan: it, it, it kind of is, but it kind of isn't at the same time. Okay. So let's get crazier.

What about windows? Can we, can we run, can we run Amber on windows you know, Sigwin or PowerShell or even the old school windows command line? Oh gosh. Amber on

Paweł: DOS.

Jonathan: So,

Paweł: yeah, I was thinking about it when I was creating it and I thought that if I catch the catch the two things at the same time, I'm gonna I'm, I'm probably gonna fail.

I have to like focus on one one target and, and, and, and, and, and, and, I think probably we could, we could support it. Why not? I think it would be a really good idea to, to, to support it so that people can build like libraries that are, that are not unique to POSIX operating systems that are also working on Windows devices which would be really cool I think that as of right now, we don't support, PowerShell and command line we'll probably do, but in the really distant future, and we'll also have to check if it's actually viable and if that many people.

You would, would rather use Amber instead of PowerShell because PowerShell is pretty, pretty interesting. Like it's, it's very powerful. And, and I see that it has this object oriented, like paradigm to it that is very compelling. And I think that Windows users would prefer using PowerShell for that exact reason But yeah, we'll, but we'll see.

Maybe, maybe Microsoft adopts Bash and we don't have to worry. That,

Jonathan: that would, that would be lovely, but I'm not going to hold my breath for it. Where, so where, where, where Amber on Windows would really be helpful. Is for those of us that are primarily Linux geeks and we have to go do something with Windows.

And yes, PowerShell is powerful, but it's sort of also painful to work with if you're used to bash or something else. Mm-Hmm. . And then I was also gonna say it, it sounds like the, the direction that, that you want to move with this is where Amber is just eventually going to have a, an option where, you know.

Dash dash target and then you say bash or sh or you know, BSD or fish or PowerShell and then alternatively if you don't run it with a target, it's going to try to detect like what what shell it's being called from and automatically spit out the, that flavor of, of of shell script and I assume that's kind of the direction you want to go and that that's going to, it's going to be its own set of challenges, but it's also going to be Pretty neat when it gets finished.

Paweł: Yeah. For now we are, we'll start with just the SIH and bash, the SIH being the most minimal, but yet more. Portable solution for a portable target for fish and ash and CSH and bash these. I'm not really sure, probably maybe because like we would have to wait and see how fast we reach the limits of bash and then probably we'll just.

Uh, we'll just, okay, say, okay, that's enough. I think this is like as good as we can go and we'll probably support different other targets. For now, I think it's good enough to support Bash and SH, but we'll see if, if adding more supports will actually help Amber to to be more popular and, and, and useful for other people.

Dan: You mentioned the object oriented kind of things that PowerShell can do. I Is the, is there any plans to go in an object oriented kind of direction with, with Amber at all?

Paweł: I think that it's not necessarily e important to, to go with the object oriented paradigm with Amber. Mm-Hmm.

because shell scripting is. I mean, usually when people ask for OOP, they mean something else than they actually ask for. Usually it means that they need some kind of way to to structure things out. Like they want to have a proper structure of, of their of their code. code base so that they enclose some things in given class and they use them as static methods or whatever.

But on the other hand, I think that OOP delivers some, I mean, I'm talking about the inheritance part and all that stuff, not talking about structures and like, The, the like objects, because like, this is very common think that, that I think we should support a neighbor in the end, because so far we always support uh, tables and I mean, erase and a table is a Polish term for that erase and and yeah, and that's basically it.

So yeah. So I think that inheritance is not very important for Amber maybe traits or something else. We'll have to figure it out. What is the best solution for for shell scripting environment. And I think that's. As we go, as we discuss different ideas that we'll finally find something that suits Amber the best.

Dan: So I noticed that when I was looking at the GitHub repository, that you're, you're licensed under the GPL V3 which is one of my, well, Possibly my favorite license. I'm sure everyone wants to know that, but was, was there a lot of thought behind that? Did you, did you just pick a license at random or was, did you, did you just specific?

Do you know what I mean? Some people are just not interested in the license and they're like, I just want to get on with writing what I'm writing and whatever the default value is, that's fine. So is it important to you that people are sharing the code and, and GPL?

Paweł: Yeah, I, I just want Amber to be to be for people, not for, for, for me or whatever I want this tool to be just, um, that, that it's widely that it's easy to, to fork easily, easy to, to integrate with other platforms that require GPL three and that I just want it to be the most friendly license that it can be, that is also yeah, just, just wanted, just wanted that.

Dan: Yeah, that makes sense. That makes total sense. We've actually got another question from the chat room. From The Benton, who says Bash and scripting in general tends to be very iterative. Does Amber support any map slash reduce style functional approaches to scripting?

Paweł: Oh, that would be cool. That would be really nice to add.

We haven't actually done it yet because functions in bash are not I mean, you, you, I mean, you, you actually can treat functions like values, but. With, with the, with the, with function names, like you could create a function with certain name and just evolve it you know but I think that yeah, that would be cool to create a functions to make functions.

Actually values and to assign it to a variable and then like add lambdas. And when you create a map, you could pass a function to a function that could, you know, to a function call that could iterate through all of that, all of that data set that you have and do something else with it. So higher order functions are on the list.

It's there's actually an issue for that. And I think we'll, we'll have it. We'll, we'll introduce it eventually.

Dan: Excellent. There you go. So I hope that's answered the question. The Ben 10 back to the I, I'm a license geek. I love licensing. So I'm just going to bore you with that. I'm afraid.

No, no, I won't bore you. Final question on licensing. I promise. Could you relicense the, the code if you wanted to, do you have anything like a contributor license agreement or is that anything that you would you or does everybody retain their own copyright and all that sort of stuff?

Paweł: I'm not familiar, familiar really with that.

Dan: in terms of like the Linux kernel, for example, the developers all contributors who write code retain their own copyright. So that means that no one entity can relicense the kernel, for example, because, Because the copyright is, you need to get agreement from, in terms of the kernel, well, how many people, a lot of people, in terms of the kernel.

Yeah, it's,

Jonathan: I'll jump in and say a couple of words. It's something that's, I've seen a couple of times when a project gets down the road and they go for, for, Several different reasons. They'll go, oh, we went with GPLv3 and we should have gone with MIT. Now, sometimes it's because they want to commercialize and make money.

Other times it's because we just realized that this was a bad decision. So something like a CLA it, it essentially, it's whenever, when somebody writes code, they assign a copyright to whoever holds the CLA, and it says you can re license it if you want to. And there's, there's strong opinions on whether CLAs are a good thing or not.

Because like I said, a lot of times they get involved with commercialization, and people get very unhappy about that. So I'm guessing based on the fact that that CLAs are not in your wheelhouse, not something that we have with Amber.

Paweł: No, I mean, I don't see a compiler that is commercialized. I mean, compilers should be free and easily accessible to people.

Maybe some products around Amber could be like, there could be some products, right? Like that could be, you know, commercialized, but not the compiler itself, I think.

Jonathan: Yeah, that makes sense. I was going to ask whether you have plans to try to ever make any money with Amber or if it's just always going to be a pet project.

It sounds like there's been at least a little bit of thought gone into that. Yeah. Yeah. I mean, I was, I was thinking about that because like, it's nice to have stars and GitHub but if you don't get money, right. I think There are some plans to maybe build some platforms that utilize Amber in some way.

Paweł: Uh, but there are no many, there are no, like I, I, I haven't thought about any particular idea to to implement as of right now. So maybe that, maybe something related to Amber could come up but I don't know. I, I just haven't, I, I'm focused on the compiler right now and I want to make it the best it can be.

Jonathan: I, I assume if somebody came along and said, Hey, let's give you a contract to work on Amber for a year, you'd, you might be interested in that. We could, yeah, I think I would be interested. Somebody, somebody would, somebody from the project would. So what's the what's the, the, the game plan as far as trying to get into the various Linux distros by default?

Thanks. I just looked on my Pop! OS machine here. I can't just apt install Amber, or if I do, I don't think it gets the, I don't think it gets you guys. I think it gets one of the other projects. Have you, have you been in contact with maintainers? Are you, are you talking about that with packagers? Yeah.

We already have a snap package for Amber. So you can go to the snap store and snap craft store. I don't know how it's called exactly, but yeah, you can, you can grab it from there. Okay. We don't have any PPA created as of right now, but we, like, I'm open to people creating some repositories and we could we're, we're also thinking about creating an organization on GitHub to store everything.

Paweł: There. Related to Amber. Mm-Hmm. .

So yeah, it's like, it's the early days right now and I think that as we move on there, there are many, many packages will come up and like for instance, there's already some a UR for arch Linux users that can use that can install the amber. It's called, I don't know, it's, it's probably on the readme on the repository.

Jonathan: Yeah. So let's see, first off, that's interesting that you're, you're using snaps. It's actually one of the, one of the few, one of the very few compelling use cases for using snaps because you can, you know, as opposed to, to some of the other images, you can use it for command line command line stuff.

And it just works. There's one of the things that they kind of had in mind was snaps. So that that's, that's fascinating.

Paweł: We'll also probably create some taps for macOS users. It's like homebrew is the main main driver. Yeah.

Jonathan: Yeah, I would I would imagine it'll be reasonably easy to get into brew They seem to be pretty open to new packages and people, submitting, And it's fairly easy to just download the download the binary and install it manually, isn't it?

Paweł: oh, yeah, sure you can basically, go to the repository go to the releases and there are plenty of You archives that you can download and use the binary. It has no dependencies, so you can just download it and use it.

Jonathan: Yeah, that's neat. Let's see, so if somebody wants to come and get started, what's the best place to come and find more information about Amber the Project?

Paweł: Mm, probably the documentation. We have the doc stats aims slash lang. Aim lang.com.

Jonathan: Mm-Hmm. ,

Paweł: which is the documentation website, which is very, like, it's, it's the place where you wanna learn, aim. We are working on it to make it the most informative and, and educational. Mm-Hmm. , but other than that, you could join our discord server where we're talking about like, there's a channel code help where you can get some help from other people that already code in that, in, in Amber and I don't know, maybe GitHub discussions.

That's also something someplace that you might be interested in. Yeah. I think that these three sources are the best.

Jonathan: Yeah. Now, okay. There's something that we pretty probably should have mentioned or talked about earlier, and that is you've got a, you've got a warning in a couple of different places that Amber is not ready for production yet, right?

What's the what's the story on that? And like, at what point are you going to feel good at like good enough about Amber? Like it's, it's ready. It's not necessarily done, but it's done enough that you can take that warning away. What's the timeline look like?

Paweł: I think that when we stop introducing breaking changes, that for sure, because for now we're, we have a couple of ideas that would break the, the, yeah.

Sorry for that, that would introduce breaking chains for, for existing source code written in Ember. I think that yeah, mainly that, and also if we improve the code coverage, because for now we, we We have some tests, we have many tests, but, but it's not enough for a compiler. I think that we need to cover as much as possible scenarios.

Jonathan: Yeah. Yeah. Makes sense. Um, all right. Is there, is there anything that we didn't ask you about that you wanted to cover? Oh I know it's a hard question.

Paweł: Well, maybe uh, the, like the end goal of Amber is I could I could make an analogy that Amber is supposed to be something like Apple shortcuts for Mac OS.

Hmm. In a way, not necessarily like one to one one to one tool, but, but in a certain way, like for instance it's, it's not going to be like this nice user interface, uh, application or tool or whatever, it's going to be just the, just like, oh my goodness. I'm just like trying to, yeah, exactly.

Exactly. I think that it's the idea of it, but, but it's not the implementation. It's meant to be easy for developers. And it's meant to be for developers and power users, but it's not meant to be like closed and very limited. So it's open for people to extend it with libraries and, and, and maybe applications that interact with other applications.

And this way people can automate many things, but not necessarily So, so it's, so it's like, so, so that's something very Very interesting. Maybe that's, that's the, the idea that I had in mind. That's what I want to say.

Jonathan: Makes sense. Makes sense. So you, you've got the Discord where people come and ask for help and show you some code snippets on GitHub as well.

What's the, what's the strangest thing that somebody's doing with Amber that, that you've seen, that you're aware of?

Paweł: Okay, so one of my friends has written a tic tac toe in Amber, which is pretty interesting. It was a fun little game. Okay. Some other person has written a a script that searches the Bible.

When you, like, write a citation from the Bible, it just searches from the Bible and it spits out where it was exactly, which paragraph and which yeah.

Jonathan: Very cool. All right. No, that's, that's a lot of fun. So I've got, I've got two final questions that I've got to answer, got to ask you, and that is, what is your favorite text editor and scripting language?

so much. Oh, okay. Can I say Amber or You're allowed

Dan: to. Of course you can. If

Jonathan: it's, if it's true. And so that may not be the case yet. I don't know. I could, I could see this where it's a, it's a passion project, but it's not quite your favorite yet. It depends on

Paweł: if if other languages like TypeScript or if these are actually scripting languages, because I use them not as type of scripting as Right, right.

Scripting languages more like I use them more as of, Like a tool to create, to build applications. So I, I would say, I would say Ember. I don't like Python that much. I know that that's a controversial thing to say, not a fan of white space indentations. And sure.

Jonathan: All right. Okay.

Paweł: Yeah. When it comes to, text editor, right? Text editor. I use Zed. I love it. It's just, it's really fast and and it's not that, I don't know. I just, I, I used to use visual studio code, but I've, I found it's very like. I don't know how, how to, how to describe it. It's, um, like not commercialized, but it's more like corporate sound like that.

And I didn't like this vibe and I just wanted to change it to something else. Was looking for an alternative for years and finally some, someone created that. It's on macOS only. So it's kind of a bummer, but yeah, it's, it's true. It's a good tool. Yeah. And I used to use, when I don't have the access to, to Zed, I usually use Vim.

Jonathan: So, yeah. I didn't recognize Zed at first, but now going to their website, it's like, oh, that's, that's the one made by the guys that made Atom. That's right. Yeah, that's exactly.

Paweł: I was a fan of Atom. I just, I, it was too slow. So I switched to VS Code when I, when I heard that they created Zed, I was like, oh, that's, let's give it a shot.

Let's jump, jump there.

Jonathan: Yep. Yep. Makes sense. All right. Whoa. Thank you so much, sir, for being here and really, really enjoyed it. And once you hit your 1. 0, we'll have to have you back and chat about what changed between then, between now and then. Looking forward to it.

Paweł: That's great. Thank you. Thank you so much, sir.

Jonathan: All right.

Dan: What do

Jonathan: you, what do you think?

Dan: I think it was great. Yeah. Really, really interesting project. I've actually, I was going to install it while we were talking, but then I thought maybe not a great idea on the machine that I'm actually talking to you on as we're doing the show, but I will be doing it right after, but it's literally like a curl command to install it.

And so yeah, so that's really, so I'll give that a go. Yeah, I'm really interested. I, I love. Doing stuff with bash, but it's bash has a lot of problems and it's old. It was, I actually looked up earlier. It's, it's like 1980 something it originally came out and yeah, it has, it has its issues, but. I, I think being able to use something like Amber to compile into Bash, I could see real benefits for me doing that.

So yeah, I'm definitely going to be playing around with Amber in future, I think.

Jonathan: Yeah, so one of the fun things about it is it's, I believe you said it's, it's written in Rust. So it'll, it'll run anywhere where you can run Rust and Bash, which is like. Every Linux out there, just about, so I mean, you could put it on, you've got Termux on your Android phone, you could make it run there if you have, you know, we talked about a router, if you've got a router that's got enough flash room on it, you could totally run it and make it work there, you know, if you had, if you had a full bash to make it work and that's, that's, that's really interesting.

So, you know, there's potential to put it on RISC V devices, on ARM devices, all kinds of stuff. Yeah, yeah. Sometimes as much as cool as bash is sometimes bash scripting. It's just It's just not very nice. Every once in a while. Not to throw shade on it, but it's just sometimes it's not my favorite. Yeah, I enjoy Ember a lot.

I think it's going to be very interesting to watch the project though, because they've got, like, they've got some problems to try to move in the direction they want to move. There are some, there are some challenges. And it's going to be really interesting to keep an eye on it and see, like, how do they solve those challenges?

What, what direction do they take with it in the future? Yeah, and I think it's impressive that they've already got five maintainers. That's amazing. That is, well, and I'll tell you, I'll tell you why. I can tell you exactly why that is. Two, two reasons. One, the idea of Amber is it's sticky. Right, like you, you, you hear it and you immediately go, Oh, oh, that's interesting.

And it kind of sticks with you, right? And the other thing is, the people that use it Our programmers, just by the very nature of it. And those two things together is going to attract a lot of people, well, people like you and me, that are interested in it, and, if there's something about it that bugs us enough, we can go fire up, you know, our favorite editor, grab the code, and Go fix it.

And that is how you end up with a project that has five maintainers is because you, it's kind of this, this, this perfect meeting ground of the right project. That's going to attract the right kind of people to be your maintainers. That's very true. Yeah. That is when you said Amber is sticky. I was thinking of Amber, you know, as in Amber itself is, is sticky.

Dan: So you, you, you've really nailed the analogy there. Yeah. Yeah. That's great. They should work that into their marketing somehow. Oh, all right. Dan, you have anything you want to plug? Yeah.

Yeah, I mean this weekend, it's the Liverpool Makefest, which I've talked about in the past. If you can, if you're in, if you're in the UK, if you're near to Liverpool, you want to come along.

It's completely free. It's on all day from 9 a. m. to 5 p. m. this Saturday, the 6th of July. If you go to liverpoolmakefest. org, you can find out stuff on there. And you can make yourself free. Come along, make robots, do drawings fly balloons is a thing we're going to do. Oh, cool. All kinds of stuff.

So yeah, go and check that out.

Jonathan: Yeah, very cool. It's, it's a little bit too far of a drive for me. I don't think I'll make it this year. But maybe Excuses,

Dan: excuses. I know, I know. Mind you, the amount of times I've been to Oklahoma is, is, it's not high. So I, I can't really complain.

Jonathan: Oh, true, true. All right.

Well, thank you, sir, for being here. I appreciate it a lot. No problem. All right, and then as far as things I want to plug, of course, we appreciate Hackaday giving us a home for Floss Weekly, and you can find my work there, specifically the security column. It goes live on Friday mornings, and that is a lot of fun.

Make sure and check that out. There is the YouTube channel for Floss Weekly if you want to see the video. You can just find us it is FlossWeekly over on YouTube. If you search for that, you might also find the Untitled Linux Show, which is my show over at Twit, which is more Linux and less open source, although, boy, there's a, there's a lot of overflow between those two.

So if you like this, you'll probably like that too, and give that a a check out too. We sure appreciate everybody being here, those that caught us live, those that get us on the download, and we will see you next week on Floss Weekly.

Episode 789 transcript

26 juni 2024 | min

FLOSS-789

Jonathan: This is Floss Weekly, episode 789, recorded June 26th. You cannot eat the boards. Hey, this week, Doc joins me and we talk with Igor and Ricardo from the Armbian Project. It's all about bringing Debian to every armboard imaginable and trying to get good support for all of them. You don't want to miss it, so stay tuned.

Hey folks, it is time for Floss Weekly. That is the show about flossing. Free Libre and open source software. I'm your host, Jonathan Bennett, and we're going to have a fun time today. We've got the one, the only Doc Searles with us. Hey Doc, how you doing?

Doc: That may be one too many. I don't know, but yeah, I'm fine.

One too many. I'm

Jonathan: good. I'm good. It's good to have you back, back in the the, the, the secondary hot seat, as it were,

Doc: it's still

warm.

Jonathan: It's still warm. There you go. I like that. So today we've got the folks from Arman that is the Debian on arm, and I think some other things, maybe risk five we'll have to ask.

Are you, are you familiar with Armbian Doc or is this UN territory? I am now . You are now.

Doc: I am. I am now. And it's funny, when I first looked at it, you know, it's a, it was a tiny print and I thought. Ambient. That's interesting. Why did, why did they name it after a drug?

No, not

quite. No, no, no, not quite.

Not quite.

Jonathan: Yeah. So it's, it's no, it's, it's going to be a lot of fun. It's going to be really interesting, but of course it is sort of the, one of the kind of premier Linux distros for ARM and the, the list of boards that they support is really impressive. I'm going to have to, put a bug in their ear about adding a board to their list because i'm sure they get this all the time people like hey why don't you support this or that and you know there are reasons we'll ask them about that yeah let's go ahead and not waste any more time let's go ahead and bring them on so we've got igor and we've got ricardo and i want to say welcome to the show guys Hello, thanks for having

Ricardo: us.

Jonathan: Yeah, yeah, it's, it's gonna be, it's gonna be a lot of fun. I'm looking forward to it. So I want to, two questions and I'm not sure which order to put these in because they both kind of depend upon each other. Give us first, I guess, one of you take it, the 30, 000 foot view of what Armbian is and why somebody would reach for it.

And then the other question that sort of goes with this is what role do each of you play in the project? Okay,

Igor: let me try to do some very quick introduction. So RMB is like a base operating system for single board computers. That would be like a definition, but it's a lot more actually. It has a very strong build framework.

Which allows us to build images from, from sources. So, and customization in this process is really extreme. So we provide small IOT, let's say very minimal images, several images and several desktop variants. So, and also we implement all available technologies like video acceleration that kind of stuff. So, and Argonne even it has really let's say extreme diversity at, at the source point, so all the, the, the hardware has some specialties, different bootloaders different way how to put things together, so. At the end for the user, it looks the same. So this, this challenge is really still is but it used to be much bigger in the past because the, the, the development of the basic SDK was really, really poor.

Now it's getting better, but still, it's a challenge to bring like you mentioned, if you want to bring new hardware, new custom hardware, We say this is more like a custom hardware board. It's quite a lot of work. So it's not that simple to to add new, new hardware to Armbian. But once the hardware is in Armbian, it's quite easy to, to change it to some other distro.

So that switch is, like very simple.

Ricardo: Yeah we could say that Armbian, essentially, if you ask the users, it's a Linux distro, right? So a typical user, you ask him, what is Armbian? It's, well, something like Debian or Ubuntu, but it's made for my single board computer. And for this perspective, it's something that he could go online, find an image, right, that he flashes to an SD card.

And actually gets that board booting, which seems simple enough. If you're coming from a Raspberry Pi kind of thing, that's very obvious, right? You go to Raspbian, you get a download, you flash it with the image flasher and you boot it. So it's, Armbian is essentially if you look at the code of what Armbian is, it's an image builder.

So it's a build system that is able to compile all the bits. Necessary for those boards and that's starting from the infamous blob and then the bootloader and then the kernel and then user space on top of this. And so Armian was born mostly as a giant script that did build all these parts and made them work together.

And later it evolved into being a distribution, so it not only builds the the images that you can download and boot, but it also builds and maintains package repositories so that you can then upgrade your packages and maintain them up to date. So it's essentially a Debian Ubuntu derivative. But with very specific parts for those components that are not covered by other OSs. For example Ubuntu has support for a few SBCs, Debian has support for a few SBCs. RBM is building currently for 240 different boards. So it's yeah, there's a whole mishmash. And the big challenge about RBM is managing this diversity and this complexity.

And not let that lose control, and especially once a board is in, or a a SOC family is in, then we try to keep that updated and provide. LTS kernels updated, security updates, that, that kind of thing.

Jonathan: Okay. I, I have, I have a bunch of questions, place I want to go, but first I do want to ask I, I try to give you guys a twofer question.

So let's That didn't work. It didn't work. So let's start with Ricardo. What is your sort of how do you, how do you fit into the project? And then once you're done, we'll go back to Igor. Okay.

Ricardo: Yeah. So right now I'm doing this mostly part-time, right? It's a I'm not really dedicated into this, but a few years ago, I, I dedicated a couple of years, essentially two years to completely or almost completely rewriting the build system.

Mm-Hmm, . So I consider myself the lead developer on the build system. That does not mean that I am a kernel hacker or a boot loader or hacker. So I, I know how to build this thing. And I know a few of those families, so Rock Ship, some Amlogic I, I have an idea about them. But I'm I have no idea about all winner, for example which is a very, very popular family inside.

So I came to Armbian essentially because in my day job in the years before I was building cloud images Debian cloud images and Ubuntu cloud images for running on clouds. Right? And that was essentially the bootstrap and that kind of stuff. And then I found Armbian when I got aboard. And said, well, this is I can shoehorn this to produce cloud images.

But then I made a fork and once I did a fork, then nobody would take my pull request because it then made Armian build cloud images. So I then started doing transforming that huge amount of bash into something extensible. And not just by having scripts, but by really having a framework that allows us to customize different points. In, in, in that build system and then later we made this distributed so we can build the the separate parts so we can build today. We're building about 600 boot loaders. 57 kernels and this can be distributed. So my thing is really, my approach is really looking this from a mile high and trying to understand what is needed for the real experts, real kernel experts inside each of those boards and families so that they can do real good work and have tooling available to them so they can maintain this over time with minimal effort.

Jonathan: Igor, same question.

Igor: Okay my current role is more more like more in management of the project. So project management as a, as a, as a whole business development taking care of people their needs about the internal group. So that kind of tasks uh, but in the, in the development, I'm more involved in, in automation.

So I'm developing and maintaining automation for these systems. I'm involving new people to help me in this automation. Also, I'm still overlooking the build framework maintenance. So, like a main reviewer, kind of, but I'm not developing much in this segment anymore. Before, I was like, I, I'm the, that guy who who made that initial script back a year, years back.

Like it's almost 10 years back now where everything started. And then where also this community started to build around. So we started with a cheap Chinese SBCs based on Orbinar. So. I'm on the other hand more acquainted with Orwin than rock chip and then, or raspberry pie or others.

So Orwin I spent many years and we have quite a, a, a big team around those chips. And also today more around rock chips. So my role is like Founder. Yeah, okay, founder, yes.

Ricardo: Let's read this from the side. Creator and founder, come on.

Igor: Yeah, of course, yes. But I'm still, like, moving it as a full time job now for two years.

It was impossible to, to I would say eventually it became impossible to do this, this kind of work on a side. Yeah. So my, my full time job was start to fall apart, so that's all I'll say.

Doc: So, so, so I'm wondering how many how many single board devices do both of you have laying around that you work on? And where do you keep them? Because you're very, you have nice looking surroundings there. And my guess is you have a workshop or a garage that's full of these things.

Ricardo: If you can see mine here, well, there's about 12 boards going on there.

You can see the Linux heartbeats there. But I have a lot, a lot more this is really gets out of hand very quickly at the beginning, you start getting, Oh, you get a board somewhere. I started with a TV box. We can talk about that later. But then yeah, you get one board and you buy one more than some someone sends you another one.

Then a friend can't get one to work. And then it sends it. Well, I don't know. I have a few dozen.

Igor: Yeah. My, my number is unknown. So I would say yes. Yes. I can actually I have one Big box where it says retro. So I put there single board computers, which, which are already for museum. Let's say I don't deal with them anymore.

But then I have a full rack, full size rack. It's in the other room. Uh, there are plenty of those boards, but they are in in in action. So I'm running a test, test operation there. So all, all, all the code that is changed every day goes to those boards and we see what's happening. So it's a test facility.

So there are around 50 boards currently that are running.

Jonathan: Ah, so I, I, I work some with ARM boards. I do some things, I do, I have some raspberry pies, and I got started way back in the day with the beagle board, I think, like the beagle bone was one of the first ones that I worked with. And, I, I have sort of a an observation, and that is, the boot process on arm boards is terrible.

Is this, have you experienced this too, or is it just me? Yeah, it is. It is.

Ricardo: It is complex. So essentially these boards, you can take them into three big categories, right? So something that boots and it has a proper UFI, ACPI firmware. So based on usually the K2, Tiano core. And those are very, very restricted.

So those are really for big server boards. So all the ED is 128 cores. Those have proper UFI firmware so they can boot generic operating systems. Say you can just get Ubuntu ISO and put it in and it'll boots. Will it work perfectly? You don't know, but it will boot, right? And there are boards, they're essentially based on U boot.

So U boot is this very popular open source project. Most of those boards at the SDK level, so at the silicon designer level have been brought up using U boot. And then you have a, a, a, a very small number of boards that represent a very large amount of users, which is the Raspberry Pi, which has Broadcom specific bootloaders, which is different from all this that I said.

So those are, I might be forgetting some here and there, but those are the main things. So if you're doing the Raspberry Pi and you're doing the Raspberry Pi the way it's intended to do is very easy and it works. If you have the UFI board server boards they call it server ready certified or something like that, then it's just UFI.

You're gonna have troubles with the A CPI tables and other PCI stuff, and then your GPU won't work , but it, it'll boot. But then this large majority of those boards fits into the, the UBO category. And then that's not only U Boot because you need also firmer the blob. Right? So this is mostly closed source provided by the silicon vendors.

Or in some cases fully open source but then that, that gets extra complex. So the main thing about those boards is you really need a way to get into their console. And this is about UARTs, right? Serial consoles. And if you set that up correctly, then well, you can watch it boot. Otherwise, you have to wait and hope that the bootloader works, and it brings up the kernel so that you can get some HDMI display.

And if it doesn't boot, then you're lost. So yeah, it gets intricate. You really need low level access to the board to be able to debug it and understand what's going on.

Jonathan: Yeah, so I've observed that we, we kind of see more of the UEFI. Starting to come to even smaller boards. And one of the interesting things is, is people are doing.

UEFI shims, essentially, where they'll write something that maybe you boot will boot or even for the Raspberry Pi, this exists, you know, it'll boot through the, you know, the proprietary Broadcom system. And then it is a little tiny piece that then gives you UEFI. And so that sort of seems to be the direction that things are moving.

And I've got to say, like, as an end user, UEFI makes things easier. It really does. Oh, for sure. You get grub, you get all sorts of stuff that just starts working the way that you expect Linux to work. For sure. But, I, I'm, it, it fascinates me that you say that it's such a pain to work with. That I, I didn't expect that.

Ricardo: It's not, it's not really a pain to work with. What I mean is usually if you buy an x86 PC you just assume that your BIOS works. Right? Yes. Sometimes, sometimes you need to flash it. Oh, there's a, a support for some new RAM. But usually it just works, right? This is not absolutely, I can't say this is 100 percent true on the ARM side of things.

It is on the very professional server boards where everything's tested and validated. But for the smaller boards at lower costs, they simply isn't there yet. So this ADK2 project is really catching up, especially on the Rockchip platform. So the 35 6x and the 3588 boards have a really good support.

For a DK two, which is yeah, a, a, a complete UFI firmware, which can access all the devices, can boot from the network, can boot from sat, can boot from NVME and this kind of stuff. If you go to Raspberry Pi, yeah, you can put a, a SD card with the UFI firmware on it, so you can, it puts its internal boot loader.

And then chain loads. And then everything works, but it's well, is it booting from the SD and then later loading the OS from a USB. So it's not really built into that. It's completely different if you buy an unfair server, or it's just the firmware is there. It's open source. It really has been tested and they are ironing out all the kinks. To the point that it can boot Windows on ARM, right? So, so it really is, it goes into there. But really, it's much heavier, it's much harder to work with. It's C if I'm not mistaken. So if you're developing something for those boards, you usually don't need the UFI thing. If you're doing some IoT project, U Boot was gonna get your kernel loaded faster in a more consistent way, simpler, right?

So Yeah it does depend a lot on the use case, but most of those boards are still on on Yubu, right? Some we've done in Rvn some experiments, some experimental images using edk2, especially for the rock ships and they work, they work fine. But we're not ready yet to commit. There's a very large investment in Uud, right?

And we are very few guys supporting this. So it's not the time yet to move completely. Even if we could, that would be addressing about 20 or 30 boards out of the 240.

Jonathan: So I'm curious, and maybe this is a better question for Igor, what, what does the process look like to get a board added to r bn? So it, I'm imagining that somebody.

makes a new issue in the GitHub and says, Hey, I've got this cool board. Why don't you guys support it? And you hope your users are nice, but sometimes they're not as nice as you would like. Is there some process where you say, if you send us one, we'll probably be able to support it or give us SSH access into your board and we'll try to support it.

Is there, is there some kind of standardized process you guys have?

Igor: Well, it's, it's a bit difficult because you need to allocate resources. So if you don't have resources. I would say the team the people who are currently working on the project or maintaining some, some hardware is, is already over to the top with, with everything.

So I, I, I ask guys. This it's usually, it's usually hardware dealers that are, uh, sending us some samples. So they are showing, they want to take this board, they want to look on it. It's, it's not coming from users that much, also from users as well. But it's, I, I ask, if there's, if there is no answer if nobody will take it, there is nothing I can do, I would say.

Because I have on the other side people who will do bring up if they can pay the bills with this. So so, but if there is some strange, weird hardware that only one user has, it's like, it's too expensive to, to, to, to start. So we cannot cover this. It's it's really. Difficult. I mean, and you need the hardware and you need some, some support from, from vendor as well.

Okay.

Ricardo: Okay. Igor, once, once Igor has your address, he sends you boards you didn't ask for. I don't do that. No, it's not true. It just happens by accident. Okay. Not really. From a, from a technical perspective. Usually the, the board vendor, right? The, the guys who are doing the PCBs, they bought the SOC from someone, right, from some Silicon vendor.

These guys have an SDK usually very old Linux, very old year boots. They bring it up, they, they send it over and say, Hey good luck with this. So essentially the, the more common nowadays is that some user takes that, boards that into some kind of more mainline ish Format and then submits a pull request.

So it's usually a hobbyist user that is well, knows enough about this stuff to do this. Sometimes it's developers, just random developers in our community find a board at a conference or something like that, and then, then bring it up. But yeah, it's mostly pull request oriented. Has, there has been have been some cases where, well, a vendor came and said, I'll send you five of those boards, will you bring it up? I find that fine, but I also find that I cannot eat the boards. So what I mean, they're awesome boards, but I

Jonathan: cannot eat them. So yeah, Bitcoin on them to pay for food. Have you had vendors actually offer to to cover your time to pay you money to do the bring up?

Igor: Yes, in a few cases, yes. Yeah, that's always handy. Yeah, but it's not that good still.

Jonathan: Sure. So is RMBN running mainline on all boards? Any boards? Is there this massive pile of stinking weird patches that you guys have to apply on top of mainline? How does the kernel side work?

Igor: Well, it's not

Jonathan: depends,

Ricardo: right?

Igor: It's not smelly patches because most of them are ours.

Jonathan: No joke. I, I write code. My code is sometimes smelly. You can, you can admit to it. It's fine. No judgment here.

Igor: I know, I know. No, just yes, of course, it's heavily patched mainline kernel. So, and again, it depends on which family. So, we have all winner, which is Which is massively patched and we have a rub chip, which is a little bit less and then we have several which are just Just a few patches on top of mainline.

So that gives you also the status of mainline kernel by let's say Soc family and also the interest on the other hand So if you have small users you also have a lot of Finds less problems and you have less patches, so yeah, it's, it's, it's connected.

Ricardo: It also depends a lot, essentially, on the age of the board and the interest it generated.

So when it, when it got added very likely, I'm, I'm going to generalize, but yeah, it got added using a vendor kernel which is Not just an old mainline kernel, it's an old kernel that has been heavily hacked upon by the vendor. That means that if I take that same vendor source and try to run it on a standard machine, it won't run because it has been hacked at.

So usually, sometimes, especially new SOCs are added in that way. And then as the work starts on the mainline kernel, then there's a whole bring up that, that is very basic stuff like clocks and pin controllers and this kind of stuff. Once that gets the first lump, right then we can, we can add the mainline kernel support, which has probably can bring up a few CPUs, can bring up the RAM.

It can boot and have a serial console. It won't have internet, it won't have HDMI, it won't have anything. So this is the situation with the latest Rockchips, for example. It's two years ago we could get some decent experience using a vendor kernel that was completely out of date and insecure. Or we could get a serial console on mainline.

Two years have passed and now we can get almost a great experience on mainline. So this has been done mostly by people outside of Armbian, but also by Armbian developers themselves. And Armbian has been this channel to expose a lot of users to this without them having to go pull from a crazy git tree somewhere.

They just pull Armbian and say, build me whatever is the latest that you guys can have about this, right? So for each board, we do have a few, we call them branches. They're not really branches, but essentially we have a version of the thing that's called legacy or vendor, which is essentially the SDK supplied by the vendor, sometimes with some fixes.

Then we have something called current. Current is usually a LTS mainline kernel, so currently would be 6. 6 or 6. 1 in some cases. Yes, 6. 6. And then we have an edge branch, which is where the action happens, right? So this is currently at 610 RC5 or, or something for most boards. So this is really a bleeding edge and, and, and trying to bring the latest and greatest not only from the mainline, but also with patches on top.

So it's, it's like really dual headed serpent

Jonathan: there. Yeah. So I, I have, I have noted that. Vendor kernels are terrible. And you look at some of them sometimes and it's like, Are you guys not ashamed to have pushed this? Like, it's so old and it's so bad. I, I, What I can't figure out is why more vendors don't work with the upstream kernel to try to get their patches to work.

upstreamed. It seems like it would be such, such a smaller maintenance burden on them rather than trying to bring this five year old kernel tree along for every revision that they've got. Just, I, I really don't understand why there isn't more effort from the people actually making the, these things to get their, their code upstreamed.

Ricardo: Yeah, and even you see a few vendors which are not there. They're not upstream first, right? Which is essentially what Google is a bit forcing Qualcomm to do. So you guys want to do on the Chromebook, so you've got to be upstream even before the silicon is ready for consumption. So that's the ideal world when you Some, some big company like Google setting the rules and then those vendors are forced to do it.

But we, we do have a few vendors that are at least trying to get a bit closer or at least trying to work on an LTS kernel and then backport the fixes actually, so that you're not completely out of a support. But it's pretty rare. So this, these guys have this embedded mentality, which you do one kernel, you ship it off into a device, and you never touch it again.

Yeah. So it's, it's really about their mentality is about where they put their money and developments. And of course, you're submitting patches to the Linux mailing list, people are gonna look at it, I'm gonna tell you to change your stuff. Right. And don't send me

Igor: this.

Ricardo: Right. So these vendors want to get out something that they can demo.

And, and show how many FPS or whatever it does. They don't care about making a maintainable code. So the, the, they don't have the financial incentive because yeah, it's saying, oh, we are mainline first is nice, but it doesn't get the demos and the FPSs and whatever. It's difficult

Igor: to control this for them.

So so they have a SOC, they, they got a kernel which SOC vendor is kind of supporting and, uh, they're. Paying for that support and they don't want to spend anything more than that. And so for for some Specific use case for IT for industry. They need one feature or two features to work and I'm totally happy with that so they will not invest into new kernel because All this good enough.

So my TV is running kernel 4. 14 and it will never be updated. And all those devices IoT devices are the same. And we, there is little we can do, but we are pushing, let's say mainline and we are getting, let's say interest from not just from end users, also from industry who are having those weird All winner or rock chip devices, can you maintain a mainline kernel for us?

Because this mainline code, which is up, up there, let's say upstream, is also not something that always works. It it, if nobody is maintaining certain sections, it start to collapse. And this is what industry is afraid of. So it doesn't work. So , yeah.

Doc: So I'm wondering what, what, what, how do you guys support yourselves? What is your business or is a different business for each of you? And is there, do you have a side gig or is there money in this? I'm not, I'm not clear on, on, on the economics of your lives.

Igor: For, okay. Ricardo has a job. I am here full time.

Let's say I invested some, some, some of let's say my savings into this. So the company is providing support, so professional support services. So we have, let's say, a few, few support deals and that keeps us running. Of course, we got donations and that stuff, but that's that doesn't pay the bill. So.

Ricardo: Yeah. And from my side, like I said, I can only eat the boards, right? They try to pay me with boards, but that doesn't really work out. But to be fair, I did get a few years ago more than one job engagement using RMUN as a base and coming from RMUN, right? So someone found a video on YouTube that showed some of the boards that I brought up.

Someone in in California said, well, I can use this for this audio project. This board is really interesting. I got hired, got a good gig doing that for, I dunno, a couple months. So yeah, I do get some gigs out of out, out of this, but yeah, I know really no intention on the, the, the involvement in RB and the build system proper is really a community effort because there's no money there.

Yeah, it's

Igor: can the board.

Jonathan: I'm, I'm curious. You know, we've got some, the UK and the eu, they're starting to pass regulations and pa passing laws to try to make iot things more secure. It does that have an impact on you guys and I guess I guess you could imagine an impact like directly on RMB in but I'm I'm actually hoping that this is going to have kind of a secondary impact on RMB in where maybe it forces more vendors to look at sort of the Development that you guys do is there is there any interplay between RMB in and some of these new regulations?

Igor: No, but we were involved, I would say we tried to be involved because we didn't the project didn't start. But related to security there was some, some I would say foundation gave out some, some funds to, to improve security in, in in the OS level and they provided some, some funds for it. We tried to apply for that, but we didn't. We didn't came through. So But, so it was, the point was to so it was security title was security and there was some European funds, whatever, I don't know to improve, yeah,

Ricardo: NGOs come up with yeah, some funds for those and sometimes we can get some of that but not, not, not much really.

It's, it's really The, the IOT aspect of this is, is the interesting part is because Armbian is a general purpose operating system, right? It's just like Debian or Ubuntu. So, okay. You can have a minimal image and you can have a CLI server image, and you can have a full desktop image, for example, but that's not where the power lies, the power lies in the build system and it's extension framework.

So. This really gets people who are, who get one of those boards and say, well, I want to make a product out of this, right? And then they can customize that image and trim away what they don't need, add what they need on top of this. And then it becomes like that, that company's IOT project. So Armbian's really, doesn't really have too many features in, in the final images that you can go and download for IOT.

But it is really in the development process, in the build process, and when you get a hands on with the build system, where you get the value out of this. And this comes essentially because it makes it very easy for people, well, I can get this Debian Ubuntu going, and now I just need an audio player.

Right. I need an audio player that also starts when it boots. Right. And there you go. You have an appliance and that's an IoT project.

Igor: We got we got I think two, two two projects uh, from, from from industry that they were interested in switching from Yocto to Armbian. That was, so they came to us and they said, can you help us?

We have this Yocto thing this BSP, whatever. And, but our customers so their customers were Saying this SDK is so complex, so if they want to change something in the OS, it's, it's, it's really, it's really painful. But Armbian gives this, provides this really, really simple. So it's really simple to change something, to add something, rebuild image, and flash it to the device.

So this process is really simple. Much, much less complicated than Yocto, for example. Yocto, you really need to learn. You really need to invest some time. I think with Armbian I don't know, Ricardo, how fast do you think a new, a new person can, can, let's say, develop it, of course. Yeah,

Ricardo: depending on what, what, what is your IOT project, right?

Depending on what it is, and is, is your base software packaged already in Debian or Ubuntu? And is your kernel already brought up, and your, your U boot is already working? So it's really just glue codes essentially saying instead of what when you bring up you're going to bring up a generic desktop you're going to bring up something that is like a kiosk mode something for example or something that doesn't even use a display just outputs audio or read sensors or and then there's of course some companies need more of this so they need over the air reprovisioning of those images They need metrics, right?

So they, they, they want to collect telemetry data from this. So yeah, there's it really becomes quite disconnected from the original image. A certain a certain point, but it should, they, they can use, really be simple. So I, I've seen prototypes being built in today. for all your stuff.

So it's really really cool, interesting stuff.

Jonathan: So I'm curious, the name Armbian obviously suggests that you guys are very much ARM focused. I'm curious if you have some support for things like the the Star5, the Civision 5 2, which really is one of the first RISC boards that's actually usable a little bit more than a toy.

You can actually, if you really wanted to, you can compile on the board, which I think is an interesting thing. Um, Is there RISC support in RMBN? Is there NIPS support in RMBN? You know, are there any of these other esoteric sort of

Igor: No, no, we didn't go further. So actually we start with Arm, arm first with arm third two beat, HHF, let's say the, the back in the days.

Then it was upgraded to arm 64, and then Ricardo came to the idea that X 86 would also be good to have. So it was added X 86. I'm running it on my laptops, on my servers, so and of course, risk risk five. We have the support exists. We have, I think, two boards that are officially supported. I don't I don't know if this one is among those, but there are not that many different chips anyway today.

So yeah.

Ricardo: So I was guilty of adding the x86 support to Armbian, which made it AMDivian or whatever. And then, yeah, some, some other folks that got really into Division 5 initially they started sending some patches and then I had to add build system support for this because the two chains are different and there's no TFA, it's SBI.

And there's a lot of smaller things to be done on the build system side. So I integrated those. I really, I actually don't have any RISC V boards running, but I added support for a half a dozen. So, and that really is the case. I'm just trying out the build system and then let those guys build images and see if they work.

I hear they work, but I never saw them work myself. So it's really, there's, I haven't had any focus on the RISC V side of things. They are interesting, future wise, but I haven't seen one that I said, Oh, this one, I want to have it myself. Yes. Yeah. But yeah, I'm yeah.

Igor: Yeah. We are currently working on one eight core what is the banana pie?

F three. Oh yeah.

Ricardo: The F three. Yes. Banana. That's eight core. So it starts making interesting. I personally not interested in things that have much less than eight gigs of ram. So my ideal board today has 16 gigs of ram. So, yeah, unless it hits that sweet spot for me, 8 cores, 16 gigs of RAM, I personally have little interest in them.

Igor: don't know how many memory actually this one has. Is it 8 only, or?

Ricardo: I think it goes up to 16, yeah.

Doc: I'm curious about what your users are doing. I mean, what are, what are the typical uses of these boards as they go out there? There must be some sort of, you know main use or maybe it's just all over the place, but there must be some interesting stories about it too.

So, but that's usually the question Jonathan asks.

Ricardo: Oh, there's a law, there's a law, there's users running a CLI application. So they're doing stuff like running home assistant, for example. So Home Assistant itself supports a few boards that they well, sell or, or, or, or support directly in the Home Assistant OS.

But using Armbian, you can get almost all, any of those 240 boards and, and, and run Home Assistant on them. That is very popular. And I mean, I mentioned Home Assistant, but there are others, of course, right? The, the DNS filtering thing by whole. Many, many others. So, so this auxiliary kind of things or run a, a router or a firewall, that, that kind of stuff.

That's very popular. And I think there are users, yeah, for NAS situations. So people getting those boards and then you can hook up a lot of SATA disks or NVMe disks to them and running a NAS solution like Open Media Vault. Based on the Armin kernel, that's very popular. And then on the more recent more powerful boards, they have GPUs and faster cores.

There's a lot of users actually using them as daily drivers as desktops, right? So they're doing we, we built some KDE Neon Plasma 6. images that were very, very popular a few months ago. So those, those get actual 3D acceleration they get using the vendor kernels, they get video acceleration.

So those are really starting to get really good to run as a very cheap desktop, a very power efficient desktop. So it's, yeah, it's, I, there's, of course, a lot of people in the middle, I'm not mentioning here, but it's either people running really headless. CLI applications or desktop stuff, I think.

Doc: I'm also wondering about I'm involved in endless conversations about personal AI and, and having an AI appliance that will, people will need an AI appliance of some sort.

Now that would seem to me, That'll require a specialized graphics chip or something like that in it. But I don't know, maybe in the armed world, I'm not familiar, familiar enough with it. So is, is, is AI on your radar at all in terms of what people are wanting?

Igor: I don't know , but I think yes. It's, it's, it's a hot, it's a hot topic.

There's the reason and some of those boards are coming with chips for acceler ae acceleration. , but on very basic with basic capacity, I would say. So they're not so powerful, but I don't know what they're used, what they can use before I, I'm

Ricardo: doing ai at the day job, right? Using those very, very, very, very expensive.

And dvia cards , so running cards that have 80 gigs of video ram, for example. So for really large language models. There's nothing even remotely similar at the arm. Of course, if you get one of those big Ampere servers, you can plug in an NVIDIA card into it and it will work. So you can get one NVIDIA card running on an arm board.

But if you look at those smaller boards that have GPUs in the, in the SLC. Those are really much different and much less powerful and have shared memories, so they're really not adequate. The interesting thing is that some of the new boards, so the the latest, not the latest, but the second latest generation from Amlogic and the latest ones from Rockchip, they have what they call an NPU, so it's a new a numeric thing, so those are specific for inference.

You cannot train models on them, but you can run and infer against them. So usually very useful for, especially those boards that have multiple connectors for cameras, for example. So you can do object detection in real time. But this is really only starting to show up in the mainline kernel, and some of those are re get really, really interested be interesting because they are similar to the GPU cores in the same SOC.

So the drivers for this are being added by Tomeu Visoso used to be a collaborator guy, and now has been working with Libre Computer for this. They are being added into Mesa. So it's a really fascinating. See how this is going. There's not really a super standard API, either the kernel level or at the user space level.

So OK, maybe tensor flow. Has, has this aspect of but it's really very fragmented right now. Those boards do have six flops NPUs, five flops NPUs. So they're decent enough for, especially for those image processing use cases. That is actually more like machine learning than proper AI, right?

It's no usually I associate AI with the large, large language models. And those can run on the CPU as well. So especially on those machines that have 1632 gigs of RAM, you can infer those models and run them actually on the CPU using stuff like Olama and other open source projects as well.

Jonathan: Yeah, the Sozo's he calls it the, the rocket let's see the, what was the rest of that?

The, the, the, the rocket accelerator. That's it. And yeah, he's got the rock ship one. Yeah. For the latest rock chip. And he's got a demo of it doing Real time object detection at reasonably fast frame rate too. That, that is, that is actually pretty impressive. The fact that that's, that's mainline, like we're not, we're not depending upon a vendor kernel with that.

And it's actually running in Mesa. That, that made me feel really good when I saw that, you know, a little bit of hope for the world.

Ricardo: Yeah. And he did the Vivanti one from Indiana Logics and they call it Vivanti. Which is just Vivanti, I'll come backwards.

Jonathan: Yeah,

Ricardo: and that works pretty great. And that is, the full stack is out.

So you can get both the kernel and the Mesa parts of CL. You can actually run the models. The rocket stuff, I think it's too cutting edge still. It's bleeding edge. So it's just being sent to the kernel mailing list. I don't know how the user space is. It's very new. This also is doing a fantastic job and I hope more people sponsor him to, to keep on doing it.

Doc: I'm wondering I, it feels to me being old and having been around before even the PC was there that, that the, the AI world sounds to me a lot like the mainframe world sounded in 1974 when, when you couldn't, everything had to run on a terminal and, and, and the, the, The idea of personal computing was oxymoronic.

It was like, it made no sense because there was no such thing yet. And, and yet I think the primary use in the long run for AI is going to be ordinary things in users lives, you know, I mean, you know, I, you know, for me, I mean, one of my examples is what are, what are all the property that I have? Where did I leave this thing?

Where was I three weeks ago, Thursday? You know, what route did I take to get to someplace? I mean, I, I know a lot of this data is in the hands of giants that I don't have it, but but a lot of it is laying around here, you know, like just getting, I don't have this problem, but if. If your credit card bill has Amazon on it, it doesn't line up with Amazon's record of what it sent you because it accounts for it differently.

The bottom number is the same. I want an AI to do the algebra on that and do the logic to figure out, oh, wait a minute, that was a business expense, that wasn't. And there's just a lot of stuff in our everyday lives that are, that's out of control that an AI would help with. And, And interestingly, I don't think our desktop computers are made for that.

I mean, I think there may need some kind of network attached, something else that can handle that stuff in a dedicated way, like, and, and also like selectively disclosed data about me to the marketplace on an as needed basis as well. I don't know if you guys have thought about that at all, but, it seems to me that's sort of like the unexplored part of AI at this point. Everybody's sort of like modeling their idea on, gee, you need giant server farms, you need these expensive NVIDIA boards, and the rest of it.

Ricardo: Yeah, but you do need them. That's the point. So those, especially for

Doc: some things, right? Not for everything. Yeah,

Ricardo: especially. Especially if you're training models, right? So if you have this, this set of data, it's labeled manually. You don't, don't ask by whom, but it's labeled and then you need to process this this training of those models, you know, into something that can run, you really need those, especially the, the VRAM is really important.

So this is not the same DDR4 or 5 RAM you have in your in your, in your PC. There's this. Two orders of magnitude faster. So yes, there's, they're expensive, but once those things are trained you can do inference on, on quite commodity hardware, right? So you can do them on the CPUs. Just check out, Oh, llama, for example you can, you can get running with those Lama tree.

Gemma and other models from, from, from the big players, right? Because they're pre trained as it were. Yeah, they're pre, they're pre trained. Exactly.

Doc: Yeah.

Ricardo: So if you're really working in, in this area or training your own models, and then that, that changes the thing that, that's why NVIDIA is so so far ahead.

It really is about the APIs that you use to program against. So CUDA and this kind of stuff. So there's really really at the beginning of this there's no, no effort to standardize any of this yet because its value hasn't even been proved completely yet. So why standardize an API for this?

It's very, very appropriatory stuff right now. So

Jonathan: So, MashedPotato No open source at all. Yeah, MashedPotato in our live chat says, Doc, have you considered Microsoft's recall for this? Exactly. Oh, wow. It, it, so it's, that's funny though, because what you're talking about is sort of the same thing that Microsoft was trying to tap into with recall.

The problem is, the way that Microsoft did it, By taking screenshots? No, there's, there's the security problems and all that, but they tapped into the creep factor. It's creepy that an AI is taking screenshots of everything you're doing. Yeah, somebody dropped the ball on that one.

Ricardo: Yeah, and I, I can go to chat GPT and ask it to write a device tree from a vendor kernel to mainline, and it hallucinates so beautifully.

It's like, it really has no idea what a device tree is. What a kernel driver is it's it's really it's fine for really basic stuff But all it's all it's doing is it doesn't serve arbian at all

Jonathan: It's putting it's putting letters and characters into the same order that sort of resembles the other things Labeled device tree that it's seen in the past and that's not very useful

Doc: It's funny, I gave a a talk here at the university where I'm attached.

And it was actually about the history of open source and the history of, of, of standards, of internet standards. And, and most internet standards are with the IETF now, and the IETF has these RFCs, requests for comment. And, and each standard is kind of attached to an RFC. So I said, make me a table, this is ChatGBT, which has the RFCs on the columns.

And it has standards or uses, I mean down the left. It had about at least half the RFCs wrong. And the dates wrong. You know, and, but the stuff it put in the boxes of what it did was like, that's half right. But I realized, I created an enormous amount of work for myself with that. And I got rid of it, actually.

And I came away thinking, you know hallucinate is the wrong word. Bullshit is the right word, because that's exactly what it is. They're bullshitting, you know, they're making stuff up and they know, it knows to the degree it has knowledge that is, I don't have an answer here, you know, I mean, a human being will say, Transcribed That's a good question in order to buy time and then say I don't have an answer.

These things like, I'm going to give you an answer. It could be anything, right?

Ricardo: If you look at this kind of stuff, so the Raspberry Pi 5 now has a PCI exposed thing. We've we've exposed full size PCI exposed on the stuff like the rock pro 64 since 2018, right? So it must hit critical mass. With the user so they all know the Raspberry has a thing. Yeah, but other stuff had that years ago.

Why didn't anyone commemorate this at all? So this AI thing will also, I think, settle once it reaches a certain point, right? Right now it's all too new and all too impressive. But that main thing that don't assume that what you saw that specific demo work extreme, extremely well for

Speaker 3: will

Ricardo: also work for your use case.

Yeah. Right? So the demos are impressive when you get to actually, everyone wants to deploy AI in their companies. Nobody actually wants to talk to an AI in their personal life.

Doc: Yeah, that's true. And, and An interesting thing with AI too is that if you, if you want a clear answer to something, you don't want a different answer the next time you ask it, right?

You want it to be the same. It is never the same. You know, you want it to draw you a green field with people on it and the people have balloons and you ask it to draw it again, but without the balloons and you get a brown field with completely different people and. Three balloons because you used the word balloon.

I mean, I'm just making this up, but that's I mean, that's, yeah, it is early days, for sure. Yep, yep. Yeah.

Jonathan: Alright, so there is something else I wanted to make sure and get to before we let everybody go, because we've been talking almost an hour, imagine that. So I have, I have gotten, I've gotten excited about different boards multiple times.

I mentioned I started with the BeagleBone I, for a while I spent some time trying to get the what were they called? The Utilite boards, I think was the name of them. This has been, this has been years ago, like 10 years back. But I heard

Igor: about, but I don't recall what is the, the, yeah,

Jonathan: they, they were Well, they, they were Vivante was the GPU on them.

And I saw it.

Igor: IMX, IMX. Yeah, they're IMX

Jonathan: boards. And there's been several times that I've seen boards and it's like, Oh, this is really cool. And then, you know, you'll get a post from somebody like Fedora saying, Hey, we support this now. And it's like, yes a mainline distribution supports it. I'm going to go buy one of these and use it for, you know, what have you, a set top box.

All kinds of, all kinds of cool stuff. Plans that I've had in the past that I want to use one of these little arm boards for because they're tiny and they'll fit anywhere and Almost without fail you buy the board you get it in you follow the instructions from Fedora or Ubuntu or whoever it is and Either it just fails to work completely Or you're stuck forever on a terribly old kernel, or they forgot to mention that they have no HDMI output, or there's always been something that's broken about it.

And I have just sort of given up and just started using Raspberry Pis for everything with the Raspberry Pi OS, because my experience there has been that stuff just works the way that it's supposed to on the Raspberry Pi. Yeah. We,

Igor: we try to, we try to divide Let's say hardware did all that will all that will work almost in almost always.

So we call this like a standard support and everything else is like. Uncharted territory, because if we don't have a maintainer that will take care of this, that when kernel changes, he needs to check if board still works. Otherwise it is possible that you will have Exactly the same experience which you just described.

Speaker 3: Mm-Hmm.

Igor: described that soic will not work. You will need to push it manually, you'll need to hack it together, that it'll boot up eventually, or, or there will be some errors. The kernel will crash or something like that. So we, we actually narrow this all giant 200, 300 boards down to, I don't know, 50 or, or even less, which we keep an eye on, keep an eye on, on it.

And there. You should be pretty, the experience should be pretty solid. So like download, flash, burn, run.

Speaker 6: Yeah,

Ricardo: and the funny thing is if you get any of these, let's call XYZ boards. So if you type into Google XYZ, HDMI doesn't work. I bet the first result will be Armbian with some solution, right? Uh, on the forum or, or, or, or on issue on GitHub or, or this stuff.

Yeah. We have support for boards that have been abandoned completely by the vendor. I'm not going to name them. They don't

Speaker 3: exist. They simply

Ricardo: gave, gave up. They, they are still printing those boards. They are still selling them, but haven't made a single software release for them in years. So we are actually the, the premier supplier of the software running there.

We never got a single cent of the donation from them or any kind of support or engineering support or anything, right? So it's, it's really a community effort at that point. Like Igor said, there are some boards that we are a bit more confident about. That they can run that they can take misuse.

And you're not gonna break them by flashing to the wrong thing that kind of stuff. And then there's unfortunately, a great mask. of those boards that, I gotta say the truth, you want to play it's a great way to learn. It's really nice, you get one of those boards, you go into CloneArmbian, see it building, understand how, what it is doing, what patches are being applied, it's a great way to learn.

So I get this more and more people that are know enough about kernel developments but don't want to know about the bootloader that they can go a little bit into the kernel without having to handle the bootloader by using a framework like Armian, right?

Speaker 3: Yeah.

Ricardo: And so we see this across all the areas, especially in user space.

Oh, Armian doesn't build user space, we use Debian user space mostly, like glibc and everything. down from that but for some of those silicon vendors, they have specifics like video encoders. So then it gets into Chrome and video for Linux and, and, and OBS and and this kind of stuff. So it has everyone who has an interest in, in developing this has really a lot of leeway to, to, to go in and learn.

It's really attracts this, this kind of stuff. If you want to be trouble free. Completely. Then, yeah, I stick with a vendor that only works with one silicon provider for all their boards, if I'm not mistaken. Broadcom and Raspberry Pi, they're the same thing, essentially. So yeah if you like your boards to boot up the GPU first and then activate an ARM core later, yeah, it's, it's great.

But yeah I, I, I hope they move away from this model eventually, right? Let's see, Broadcom is also not a company that's very well liked these days. So let's see how that goes. Yeah.

Jonathan: Okay. So question from the chat room, mashed potatoes, his last question thought he wants to know, how does Armbian differ from Raspberry Pi OS?

Oh, completely,

Ricardo: completely, completely. In the end it's the same, right? So Raspberry Pi gives you an image, which is based on Debian. I think it's currently based on Bookworm. But they rebuild their user space to adapt to that. They insert a lot of utilities, which don't belong in a normal Linux system. So if you look at that boot partition, it's a fact.

32 MS DOS partition with a config. txt and whatever CM line things. So that's all adapting to their proprietary bootloader, which is out of reach for most of us, right? So they don't, we don't have the source for that. So they do adapt that, they do have their own utilities for tuning things. And most importantly, their kernel, the big thing about the Raspberry Pi Foundation is the and the, and the, and the software they produce is that I don't know much, but if you, if you take a well known hat, I don't know, is it a shield or hat, whatever, hat and you take the most famous hat you will get support for that hat directly into that image.

Right, so there's a kernel device tree overlays already baked into that. It's there's already an option that you can set somewhere that enables that thing. That

Igor: experience

Ricardo: can only be curated if you really have a very small set of silicon to support it against. Right, and in the case of a Raspberry Pi, you can do with essentially the same kernel run on all the Raspberry Pi boards.

So in, in, in essence first, I think the Raspberry Pi Foundation has put a lot more engineering efforts, right, into the software than any of the other boards suppliers, by far, right? They, they produce not only goods and well integrated, well tested stuff there, but they also produce documentation and training materials.

And then how to use and the forum is active. So it's, it's really if you don't like diversity, it's, it's ideal there, right? It is for us. One, we have no, no corporate sponsor. We are a community efforts. We have a 200 X more different boards to, to support. So you can imagine that the level of Polish and refinements.

It's not the same. So you can go and get your hats going with any of those boards, because in the end, they're just I squared C, SPI, UARTs or SDIOs or that kind of stuff. So you can make it work, but it's probably not pre made for you and ready to go. So it's a great learning opportunity, but if you really want something that's pre made and ready to go, then it's it might not be ideal for you.

Yeah, but if things just

Igor: work, it's not fun. That's

Ricardo: that's also It's, there's a lot of tinkering, right? Yeah, yeah. To tinker with those things is it's fun sometimes. It's,

Igor: Yeah, it's like going with preset Lego is the same. You have a plan how to put things together and you have an aeroplane running.

But if you have just an open box of Lego set and you need to assemble something, it's more challenging and more fun because you need to really think wider. This is more, if you look on this. diverse ARM hardware. It's, it's, it's weird, weird boards, weird way of booting. But the end, if you add little, if you play a little, you learn a lot.

Jonathan: Yeah, yeah. Okay, so I want to, I want to ask some wrap up questions, because again, we're, we're past the bottom of the hour. It's been a great discussion. So quickly, if, if somebody wants to get started with Armbian, learn more about it, what is the what's the, the best place to go to?

Igor: Fix some board that it's not working well. Let's say that would be probably the easiest way. I encourage people when they come and they said, Oh, this board, which we don't, let's say, support officially, but we have it there on the download page. It's there. We don't know. It builds. Sometimes it works, but it's not official support status, and they complain it's not working.

Try to get it working. So you have tools. We will help you and and sometimes people really put an effort Just recently some guy bring up one really cool hardware Helios 64 it was and it is a NAS. It's like a NAS case And rock chip based hardware, but it was like the company went out of the business.

We couldn't it was a lot of It was immature software support, so we couldn't, let's say, bring it up and pay the bill, so it was, and we said, we cannot continue, so.

Ricardo: But the community, yeah, I would say go to github. com slash army and slash builds. Yes. That's where you want to be that's where we all live.

There's also documentation REPL, there's firmware REPL and other stuff, but find the build REPL get, get building get a Linux machine, try it out. You can build it for x86 if you don't have a board. You can build cloud images, you can build minimal images, you can build on, on ARM for x86. You can build on RISC V forearm.

So it's really interesting to get, get into the build system. So this is what attracts, I think, most people. They, they, there's a learning curve, of course, but it's, it's much easier than if you start with an embedded appliance system, like Yocto or, or, or, or Buildroot or, or others. So it's, it's quite friendly.

We try to be friendly to users who are just. trying this stuff out. And sometimes these users become developers. It's, it's really fun.

Jonathan: Is there a, is there a single board that is kind of the premier supported? Is there one that you would suggest somebody starts with? Somebody says, I want to run Armbian on something Arm.

What's the board you recommend?

Igor: Rockchip latest probably would be because there is a lot of people around. There's

Ricardo: a lot of interest on the Rockchip 3588. So those are the

Igor: but the, but there is still, let's say a lot of things that doesn't work well. Perhaps previous generation would be safer.

Safer start. So previous logic 3, 5, 6, 6 or something like that. Yeah. The 35 60

Ricardo: eights are working excellently excellent. They're great routers. You can get boards that have two five gigabit in, thats two of them. NVME. SPI flash and our boards, they're costing about 60, 50, 70 bucks. So they're, they're pretty cool.

There are four core, a, a 55, so they're not super fast. If you want to go to something, there is more cutting edge than get the, yeah, the rock ship, 35, 80 eights. Those are eight cores. Four of them are fast day 76. I think they go up to 32 gig gigs of Ram. So you've got, there's people doing Homelab clusters, Kubernetes clusters on, on this stuff.

There's a lot of interest into this, so the kernel mailing list is bubbling with, with stuff about this stuff. So it's really cool. It is, of course not gonna be perfect but you're gonna have a lot of fun. And, and they're actually usable application nodes, right? So.

Jonathan: I've got the Turing Pi, the Turing Pi 2 carrier with some Turing RK1, which is based on the 3588.

I've got them here right now, and because there is a bug in the kernel, when you use it with NVMe, I am trying to compile my own kernel. And Partway through the compile process, it is repeatedly crashing the board. So that's what I'm fighting right now. Get some cooling and get some good power. That's the first always stuff.

Yeah, I don't know that it's a cooling or a power issue, but we'll see. I hate

Ricardo: to say that, but I've been proven that some, even if you don't think so, try a different power, try different cooling. And do try Armbian. I'm not sure. I you probably heard of Joshua Hyek, who's doing the Ubuntu images, especially for the RK1.

Jonathan: I have, yes, I have actually, I'm in a Discord chat with Josh talking about some of these problems. He's been great. Yeah, exactly. And,

Ricardo: It's very fun because this is when we, we two years ago, we decided to do a disk kernel for this SoC. In a more disconnected from the build framework. So it has its own REPL and this has attracted not only Armian developers into that, but other distros and other projects are, are also using that kernel.

So that Joshua had his own Ubuntu distribution that was much simpler, let's say, than, than Armian. But then we ended up collaborating on this. And I'm going to mispronounce his name. Xuleng Feng which people know as Amazing Fates. There's a lot of really wizard level guys boogie ice cream.

There, there's some of those guys who are already a legendary level and they're a level of expertise into those things, both in the vendor and the mainline kernel. So. Yeah, it's fun all the way down.

Jonathan: Yep. Excellent. We are, we're absolutely out of time. And so I am just going to jump straight to the end.

I'm going to ask each of you, our, our famous final two questions. We will we'll, we'll start with Igor and then we will go to Ricardo. So Igor, I'm going to ask, what is your favorite text editor and scripting language?

Igor: Scripting language, Bash. The text editor, Joe, Joe.

Jonathan: Hmm.

Igor: Okay. Or n or Nano or Nano.

So Joe or Nano? It depends.

Jonathan: Gotcha. I, I tend to,

Igor: but I, I, I use, I used VI as well, so I'm handy with vi a, a, a real,

Jonathan: a real Renaissance man there. Where it comes to text editors. Ricardo, same question.

Ricardo: I'm forced to answer scripting them bash. Mm-Hmm. Of course. The Arman written in Dash if I could say j, I would

And I do use Intelli J Brains IDs for literally everything.

Jonathan: Oh, okay. I have spent some time inside of well, Android studio is really where I got to use it, but that is, that is a reskin JetBrains. So just the same. Yeah. Same stuff. Yep. Yep. Excellent. All right. Thank you guys so much for being here.

Was a lot of fun and the hour literally flew by and I've, we will have to have you back in a few months and talk, chat more and chat about what has changed. So thank you. Thank you both for being here. Thank you. Thank you for having

Ricardo: us. Yep.

Jonathan: All right. Thank you. Yes. Yes. Doc. What'd you think?

Doc: That was good.

That was really good. And I was gonna say, it's a, it was a very fast hour and 15 minutes actually. Very much so. You know interesting stuff, interesting stuff. You didn't ask about the weirdest use that they had. You sort of did. You kind of did. I sort of did. You stole my thunder. I sort of did. Yeah, I know.

I'm sorry.

Jonathan: We also ran out of time though. We'll get it next time. I'm sure, I'm sure they've got some fun stories. I don't know, last week we asked about that, and the guy's like, Well, the most weird one I really can't talk about on the air, but. Maybe they have some of those stories too.

Doc: Yeah, like the one where the guy got killed, we didn't talk about that one.

Well, no, it wasn't quite

Jonathan: that, but anyway. Alright, Doc, you have anything you want to plug before we let you go?

Doc: Oh, yeah So look up KWAI, KWAAI dot AI, I think it is. It's a, it's a open source personal AI. Yeah, KWAAI. AI. I, I mean, it's, it's this open source effort. It's all voluntary. It's a nonprofit KWAI itself is a nonprofit.

It's based on a South African word. And they have some hackathons coming up. I am the Chief Intention officer of that effort. Oh, fun. They named me that because I wrote this book called The Intention Economy, which in part it inspired it. It's a voluntary position. I mean, nobody's getting paid right now on this thing, but it's but it's fun.

And I, I, I really love to see people get interested in it. Yeah. So open source, personal ai.

Jonathan: Yeah. You gotta, you gotta hook us up for an interview. I, I tried to reach out to, we do. Do that. I tried to reach out on one of the social media networks, the, the, the, the one Microsoft owns, which ? Well, there's, I know the business one, I forget what it's called.

The Business one, which I, I, the busiest one there, there is. Anyway, so I, I tried to reach out and haven't heard back, but if I've got the, ill, I'll, I'll, I'll

Doc: do the reaching and I may do it in 10 minutes when I'm on a call. , there you go. With them. You know, I, I'd like to get their top tech guy on there.

Oh yeah. to talk about it. Yeah, that'd be great. And, and it may be early in some ways because it's so such a work in progress right now. But it would be, we, we should do it, but we'll do it. Make it happen.

Jonathan: All right. Very good. So coming up for the show, we have next week is Amber, actually, which it's, it's interesting that both of our guests today said that Bash was their favorite scripting language.

Amber is a sort of a more structured scripting language that then compiles down to Bash code. It's really fascinating stuff. I'm looking forward to that. And so that is that is next week. And starting Starting in July, so starting next week, just so you know, we're going to be doing the recordings on Tuesday.

We're going to give that a try for a month, and if it works well, we're going to move to Tuesdays, because that's going to make, that's going to take what I have, about 6 hours to get done, and give me 30 hours to get it done. And so my stress levels are going to go down if we can record on Tuesdays. It's going to be great.

If you want to find me, my work there is of course everything on Hackaday. We've got the security column goes live on Friday mornings. And we are now bringing the show with some video on YouTube. So you can find the Floss Weekly YouTube channel. Now if you search for that on YouTube, you're going to find two different things.

You're going to find the Untitled Linux Show. Which is my show over on Twit, because we sort of inherited the Floss Weekly YouTube channel from them. And then there's the actual Floss Weekly YouTube channel, which is just this show. So go, go subscribe to both of those. That's what you should do. All right.

Thank you everybody for being here, those that caught us live, and those that'll get us on the download. We sure appreciate it, and be back next week on Tuesday for Floss Weekly.

Episode 788 transcript

19 juni 2024 | min

FLOSS-788

Jonathan: This is Floss Weekly, episode 788, recorded Wednesday, June 19th. Matrix! It's Git, but for communications.

Hey, this week Simon joins me, and we talk with Josh and Matthew from the Matrix Foundation and the company Element. It's the Federated Uncentralized Communication Platform and some of the challenges they're facing right now between funding and some EU and UK laws that really threaten to put a damper on things.

But there's a plan. You don't want to miss it. So stay tuned. Hey, everybody. It is time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. Today, we've got a really a great and interesting show. Let's go ahead and bring on our co host. We've got Simon.

Simon the man. The, the open source guy. The, one of the, one of the stewards of the open source definition, which is kind of important these days, isn't it?

Simon: Yes, it, more and more so. There's a lot of pressure. Yeah. On the definition of open source, but particularly because of what's happened in AI and you know, there's a whole show to have about the the there is a word I could use to describe what's going on there But I don't think it's allowed on this show, but it involves cluster Oh,

Jonathan: yes.

Yeah, we try to avoid that one See my kids watch the show and I would prefer to not have to tell them to not say things that they hear on the show That's just a preference thing.

Simon: Yeah, so they so the open source definition is, you know, it's You It it's, it's now venerable and we're OSI is in the process of creating a definition of what open source AI means.

And that is extremely lively that discussion. I'm

Jonathan: sure it is extremely lively. That's yeah. Well, today we've got we've got something. Probably the opposite problem. It's, it's some people that are trying very hard to do something the right way, the open source way. We're going to talk with the folks behind matrix and Simon, you're familiar.

In fact, Simon is the one that said, Hey, when we set up the back chat for floss weekly, make sure it's something that we can integrate with matrix. Cause that's where I've landed. And then I found out in the, in the past 30 minutes that, you know, where all the bodies are buried, because you've worked with at least one of the guys we're going to interview.

Simon: Yeah, so I've worked with Josh at OSI and then I've known Matthew for a while as well. And we, we team up against the European union occasionally on European legislation. So I've been using Matrix for really quite a long time now. And I have, I actually have two Matrix IDs because there's a lovely community member that they have called Beeper that is making an integrated chat client that uses Matrix as its infrastructure.

And if you're a member of Pidgin, it's kind of the. The the, the modern equivalent of the, of, of the pigeon instant messenger. So I've got a lot of, a great deal of matrix in my life, both element and beeper.

Jonathan: Wasn't,

Simon: isn't Beeper

Jonathan: the one that tried to go toe to toe with Apple?

Simon: Yeah, yeah, they did and they're still, they still are.

They, they made an iMessage bridge that makes you show up as an, as a, an iPhone user to other iPhone users. And that's still available. So, you know, the iPhone, I can talk to iMessage people. I just have to open an iMessage account.

Jonathan: Oh, interesting. I, I thought I heard that that went away. Let's ask them.

Let's ask the experts about it. Yeah, yeah,

Simon: yeah.

Jonathan: Let's not waste any more time. Actually. Let's go ahead and bring them on. We've got Josh and Matthew, both of you. Welcome to the show.

Matthew: Hello. Hi, thank you for having us on.

Jonathan: Yeah, excellent. I am, I am super excited to talk about matrix and element and all of these things.

So let's, let's start with. Who does what? Let's try to keep it, our two, two guests, try to keep them straight. Let's start with Matthew and kind of give us the rundown of like, where are you coming to this conversation from? How, what, what part of this do you represent?

Matthew: Sure. So I guess I'm Matthew and I guess I came up with the idea of Matrix about 11 years ago now.

I've always been enthusiastic about telecommunications and open source, and eventually got the opportunity to smash the two together to try to come up with a new communication protocol that could be the missing communication layer of the open web. It would literally be a real time communication fabric, a bit like Tim Berners Lee originally hoped for on the web.

That there would be a kind of read write semantic, and it would be real time, and you can publish real time information, like instant messages, or VoIP, or files, etc. And it would have to be end to end encrypted, because it would be decentralized, and it would smear messages around the place. So that was what I came up with, with my co founder, Amandine, who isn't with us today.

But, I guess I've spent the last 11 years of my life trying to make that happen. And I guess nowadays my day job is Element, which is a company that we founded as the, well, the guys who created Matrix founded basically as a way to try to keep the lights on and fund us to work on Matrix.

And my position there is CEO, CTO. And I'm also a guardian of the Matrix Foundation. And I'm also project lead. For Matrix, so the kind, not really a BDFL, but more the guy who started it in the first place and keeps an eye on trying to keep it roughly on track on a technical side of things. So that's me.

Jonathan: Alright, Josh, same question, how do you fit into the Matrix slash Element conversation?

Josh: So I fit only into the matrix conversation. I am the managing director of the matrix. org foundation, a foundation that was started back in 2018 and really is the steward of the protocol. And really the, the, the body for open governance of the protocol.

And Basically, I'm tasked with operationalizing the foundation and supporting the broader ecosystem of which Element is a major player but only one.

Jonathan: Okay, I want to go, I want to go to Matthew then next because I want to understand more about the, the matrix protocol. What you described actually sounds similar to the Fediverse, but it's another approach to kind of solve the same problem.

Matthew: It's similar. I mean, ActivityPub, I think, postdates Matrix by a couple of years. ActivityPub focuses, I guess, on multiplayer RSS, for want of a better description. So, it's about micro blogging. It's not really real time. It's not decentralized in the, I, in the way, So Matrix is very much about relatively low latency, instant messaging, or, I don't know, live location sharing, or setting up a voice or video conference in real time.

So the whole thing is meant to feel like WhatsApp, or iMessage, or Signal, or Telegram, or Slack, or Discord, or Teams, or whatever your favorite real time communication tool is. Whereas ActivityPub in the 30 verse is very much, obviously, trying to compete with Twitter. And micro blogging systems. And the two can overlap.

People have built micro blogging on top of Matrix. We did it, in fact, ourselves, as a thing called Cerulean, which was a proposal for BlueSky that we showed to Jack and Parag at Twitter. They chose to build their own thing, rather than building on Matrix, sadly. But you can do micro blogging there, just as if you were feeling better.

Slightly masochistic, you could try to do instant messaging over the Fediverse too. So, I see them as kind of sister technologies. We've got good links with the ActivityPub team. I've spoken a lot to Gargron at Mastodon over the years. They used the end to end encryption for Matrix at one point as an attempt to do encrypted DMs in Mastodon.

And there are bridges like Kazama, the link. Fediverse and Matrix together, and indeed with things like the Bridgy thing on BlueSky, I think you can go, now go all the way around the houses from Matrix to ActivityPub to AppProto and BlueSky if you so desire.

Jonathan: That's great. So Matrix does have the same kind of a federated approach though, right?

Like, so I can, I can host my own Matrix server and then open up a chat with, and in fact, I think this is actually what I'm doing. I'm not hosting it myself, but I'm using somebody's. But you, you can make the two Matrix servers, Talk to each other, right?

Matthew: Yeah, absolutely. So it's federated, but it's more than that.

It is decentralized. And the conversations get replicated between the servers. And this is the unique thing that nobody else does. I guess Blue Sky is somewhat close to it in some respects. But the whole idea is that it's like Git. The, the model is the sort of subversion to Git is the same as Blue Sky.

IRC or XMPP to Matrix. So in Git, everybody has a copy of their source code repository, and if you work on it, you basically have your own independent clone that you can do whatever you like with, and you push and pull commits with other people. And Matrix is basically identical. The dirty secret of Matrix is that it is effectively Git, but for communication.

And so, I might have my server, and if I invite Josh and his server in, and perhaps Simon on his server, then every time I send a message, it gets pushed to the repositories effectively of the chat rooms on the other servers. And when they communicate back, they will push to their copy, which in turn, you know, fans out.

pushes to my copy and the other servers as well. So in the biggest matrix rooms like MatrixHQ, I think we've got about 20, 000 servers with about 60, 000 participants. So every time somebody sends a message, for better or worse, it has about 20, 000 HTTP heads to all of the other servers in the room in order to fan it out.

But the lovely thing is that there cannot be a single point of control or a single point of failure. Because nobody owns the conversation. It doesn't exist on a single server. That's it. And kind of it's multilateral communication intrinsically, so that the very act of me talking to somebody else shares ownership of that conversation with them.

And it turns out this is novel, and it's actually academically novel on the access control side of things, because we were the first people to figure out how you can ban people and kick them and give them permission to change the room name and all that sort of thing. In this sort of system, where you don't have a centralised authority, because normally you have like an access control server somewhere, or you have a focal point which determines the rules of who can do what.

Whereas in Matrix, every server goes and executes its own access control algorithm, to make sure that if I speak in a room, Am I in that room? Am I allowed to be in that room? Have I been kicked? Have I been muted? And all that sort of thing. So, that's the real interesting idea. The matrix isn't really a messaging protocol so much as a way to synchronize real time chatroom history of whatever flavor around in real time.

It's more like a pub sub database. Multiplayer CouchDB or multiplayer Redis with OpenFederation if you think of it in those terms.

Jonathan: You, you say it's like Git for communication, does it literally use something like a Merkle tree? Is it, is it sort of that, that blockchain y thing? I know people, people get mad at me when I call, call Git blockchain, but I, I stand by it.

Matthew: It is a chain of blocks, it is a Merkle tree. And basically every time you send a message or anything happens in a room we call them events rather than messages But a message could be an instant message, but it could be a join or a part a I know topic change In fact, it's arbitrary key value data. So it could be a I know floss weekly dot Defcon Defcon level 5 or whatever, whatever metadata you want put into there.

And each one gets signed by the server into a Merkle tree, so that you get a transcript consistency for that conversation. As well as these rules about how you merge together the Merkle trees, just like it has its horrible Octo merge algorithm for going and actually merging together branches.

Matrix has an equivalent merge resolution algorithm called state resolution. that allows different copies of the room on different servers to converge on the same result.

Jonathan: I, I was, I was actually going to ask you about this. How do you handle the split brain situation? Because on a, on a very different project, I've been, I've been noodling for a while about this idea of trying to do guaranteed delivery messages on a very lossy Low bit rate medium over over radio waves.

And like 90 percent of it, I can work out, but this idea of, you know, if the two networks, they move away and then when you bring them back together, how do you, how do you figure out which is the, you know, the canonical. Message tree and which you know, how do you deal with the ones that are not on that?

That's a difficult problem And I actually I need to go look at what you guys came up with because I might be able to Borrow some of your ideas I mean

Matthew: that is literally the core of matrix and it also does support some low bandwidth Environments there is mse 3079MSE being matrix spec change, the mechanism by which the spec evolves.

And that is all about using CoAP rather than HTTP, as an IoT transport for these horrible lossy links. But in terms of how the actual merge resolution works, basically, every time a server sends a message, it includes a proof that says, Matthew is allowed to talk in this room, because he had got invited.

And Everybody in the room, the servers, execute that proof and check that it, you know, that it stacks up. And if it does, then they accept the message and I'm allowed to do whatever that operation is, and if it doesn't, they know that I'm lying and I will get ostracized and the event will not get agreed to.

So, this was done as a very pragmatic engineering solution to the problem and then we started getting emails from the University of Karlsruhe about or Coulter Institute of Technology, technically, where there is a distributed systems networking group where they got excited because they started analysing it and discovered that it was a unique, special snowflake way of doing decentralised access control.

And if you go and Google for, KIT, DSM, and MATRIX, you'll find all sorts of papers and interesting presentations on the, on why this is academically interesting. But the, the end result is that you basically replicate the room across the world.

Simon: Yeah, so that, that, that, that, yeah, that all, that all sounds standards ish.

And it sounds like the Matrix Foundation could end up being a standards body. Is, is that something in your future, Josh?

Josh: Well, you know, it's funny you should say that. Because after spending six years at OSI, which is itself a bit of a standards body for for licenses. I am now at the Matrix Foundation, which is also a bit of a standards body.

Of course, not officially recognized as one. So I believe that's an open question as to how we proceed with that but you know, pre show we're talking about the Eclipse Foundation and the work they're doing to be recognized as such and I would not be surprised if we, we go down a similar path ourselves either individually or in partnership.

Simon: So how is Matrix Foundation set up? I remember talking with Matthew about it two years ago in Brussels. And he was talking about wanting to fork Element so that the community stuff was in a separate organization, and the corporate stuff that paid everyone's salary was in its own organization.

Seems that's what's happened now. So where does Matrix Foundation fit in?

Josh: Right. So this is again having spent time at OSI and being deeply opinionated about the course of open source projects. I was really excited and encouraged when I was speaking to Matthew and Emma Dean and the rest of the team before I accepted this role about the way that they created the foundation.

They handed over the assets like the trademark for Matrix and the Okay. Copyrights to the extent that they had them in the specification And all of those things live within the matrix. org foundation at this point and the spec core team, which is the the team that more or less manages the matrix spec change process That is all within the bounds of the foundation.

So backing up a little bit the matrix. org foundation. We are a Community interest company founded in the United Kingdom that is more or less a not for profit, right? There is no beneficial owner, no private owner no private gain that comes out of that. We are committed to keep all of our assets dedicated to fulfilling our mission and that mission is, of course, stewarding the specification doing advocacy as we see fit on matters like Encryption and privacy and security and human rights.

Because those are all highly related to, you know, secure communications protocol. So the role of the foundation is. If nothing else to look after the standard and to convene people around the standard. But our aim is as you see elsewhere in, in FOSS we see foundations that are able to, you know, look after the core technology, but also be a vehicle for nurturing the broader ecosystem.

Right. So with the Matrix. org foundation. For instance, we are hosting our first ever matrix conference later this year and bringing together policy makers and engineers and product managers and everybody who has a stake in the work that we're doing. So. I don't know. You could also use a bit of a gardening metaphor.

You know, we're here tending a garden as as the Matrix of Art Foundation.

Simon: So that it's, I'm kind of interested in having picked a UK CIC as the foundation. You know, from two perspectives. Two perspectives. First of all, I spent like five years advocating that open source organizations ought to be CICs.

So in my view, it's the perfect legal vehicle for an open source foundation. How's that going? You know, I know I was looking on Companies House just now, and I, I see you've had to stick the CIC on the end of the name.

How's it going? Being a C3 is very well understood to Americans like you, whereas being a CIC is almost unknown to Brits like me, so what's it like? Is it working well?

Josh: You know, I have to say it has not presented a significant barrier to us. It definitely has taken a little extra doing in terms of understanding what our constraints are and how we need to operate.

We are not tax advantaged. So, you know, that's one of the perks that comes with a 5. 1c3 or a recognized charity in another country. But it's been working out for us. Now, that said, we, we see like the Eclipse Foundation has rea Rehead quartered itself in Brussels. And we see foundations discussing having entities in a number of jurisdictions, right?

Whether that be the U. S. or the E. U. or the U. K. So And there are a number of reasons for that, right? You might be able to be a recipient of funding based on your jurisdiction. But for us, the United Kingdom has been a perfectly acceptable home and the CIC has worked out all right, I'd say.

Simon: Right, well, I'm happy and in fact somewhat relieved to hear that's the case.

So I had a look at the corporate structures around matrix and element. There's quite a lot going on. I noticed that you seem to have the, so being a CIC that you can't have any beneficial owners, so that's good. Over on the other side, I see, Matthew, you've got a holding company whose name you don't use in trading.

And then you seem to have three subsidiaries in Germany, France, and America. You know, how is that working out for funding the matrix work? Because that's very, you know, you're a critical part of the open source infrastructure for the future. And if you fail to fund this activity sustainably, we are all in trouble.

So how is it going? Well,

Matthew: that's a big question. So yeah, on the element side, first of all, it's completely disconnected from the foundation. Myself and Amandine are on the boards of both Element and the Foundation, but that's as far as the relationship goes and we're deliberately in the minority of both boards so that we can't go and do crazy things with either organisation.

In terms of Element being key to funding Matrix, historically, absolutely. So the, the origin of Matrix is that there are about 12 of us who were working at Amdocs, a telecom vendor when we came up with the idea of Matrix. And Amdocs actually funded the first three years of Matrix dev as a crazy R& D project in the hope that perhaps this thing could replace the phone network or replace email and be the next generation communication system that they could benefit from.

Eventually though, they got fed up in funding the whole thing and we somewhat abruptly left and suddenly had to make payroll for 12 people. Now, in the ideal world, I think we would have just grown, we would have set up the foundation and we would have asked for donations and slowly bootstrapped our way into being a Mozilla style non profit vehicle going and building out mission driven things for Matrix.

But We didn't have any money and any revenue. We literally pooled our redundancy package from Amdocs together to buy ourselves a couple of months in order to set up shop. And luckily the folks at Status, who do decentralised communication on top of Ethereum went and invested and put I think four and a half million dollars behind the bar to get us started.

So, on one hand, that was great because we were able to set up shop as Element and make payroll and actually continue our work. On the other hand, it set us on the path to basically a VC investment model where we, on the Element side, raised in total, I think, 55 million so far, of funding. As element in order to first of all build out matrix and then obviously try to do something that would make money as a return to our investors.

So before we set up the foundation in 2018, you ended up in the strange model where basically 100 percent of the matrix work was being done by us as element. And we were basically taking investor money and building Apache licensed open source software with it for the greater glory of the world to try to terraform this new industry.

Okay. Now, that obviously has worked relatively well. I would say that Matrix itself is wildly successful. We've got 152 million people addressable on the network. We have people like NATO and the United Nations and all of France, most of Germany, Sweden, Ukraine, you name it. The different countries and organizations are enthusiastically using Matrix.

And we've always had this really delicate balancing act between Do we just give away everything as liberally licensed open source and get as many people using it as possible and pray that there will be a way for Element to do an automatic on top of WordPress and basically sell value added things, whether it's support or SaaS hosting or whatever?

Or, or what the correct balance is. And so that has been the eternal funding challenge. These days Element is at a point where we are not profitable yet. Hopefully we will be in the near future, and we've been killing ourselves over the last couple of years trying to get to that point. About 30 40 percent of our effort still goes into core matrix development, effectively donating the work to the foundation, hopefully to the benefit of the foundation, and then the rest of it is spent doing commercial work.

element work. So I feel, I think of it a bit like Red Hat or Intel or somebody who has a whole bunch of people who are frantically committing code to the Linux kernel or whatever your favorite bit of the Linux stack is. But they also have the commercial business going on where they sell Red Hat enterprise Linux.

And so for element, the equivalent is this thing called element server suite, which is a commercial distribution of matrix, very similar to realm, but for matrix land.

Simon: So do you have anyone else in the ecosystem who's Contributing significantly to the core code. You know, I, I mentioned in the, in the, in the rundown Beeper existed.

And there's Eric over there who's doing that, I think a fine job building a, a you know, replacement for libpurple. Do you, is that meaningfully contributing to Matrix or are you still really carrying the world on your shoulders? I'll let Josh

Matthew: answer on behalf of the ecosystem, but from my perspective it's you've got to think of Matrix in two levels.

There is the kind of core technology, the stuff in github. com slash matrix dash org, like the spec. and the encryption, and lots of SDKs and things, which, without which Matrix would never have existed, and continue to be a kind of fundamental building block. And of that, about 90 percent is still written by Element.

But then on top, you have loads of things, like Beeper, which have been written by different people in the community. So Element obviously has its own app sitting on top. But there are literally hundreds of Matrix clients, and servers, and bridges, and bots, and carrier pigeons, and God knows what else, which people have built on top of the underlying technology.

So Beeper is interesting in that they have contributed a bit back to Synapse, which is the Python server. I think I've seen, like, tens of PRs over the years which is pretty good. They've also contributed some MSCs, so metric spec changes to the spec itself. Although on the client side, I don't think they ever contributed anything back.

So they went and forked the iOS and Android code bases which we'd written that element, improved them a lot, and kept them proprietary, and despite asking a few times that they might possibly upstream them, they never did. So, it's always a bit of a cure it sack. What do you think, Josh? Ha ha

Josh: ha!

Yeah, So, like a big part of the role of the foundation is to to build up the matrix ecosystem into something that has a diverse and sustainable contributor base. I think it's fair to say that We have a very active ecosystem. There's a lot of great work going on. But we also we are still far too reliant on element.

And I think that's, that's a fair characterization both in terms of you know, Contribution upstream to core libraries as well as support for the foundation itself. And on the one hand, I'm very grateful to element for the generosity. Also, not surprised because, of course, it's run by the founders of matrix.

But it's. It's a dangerous position for any ecosystem to be in when there is one organization that is doing so much of the upstream development. And so, you know, we do have you know, to Matthew's point, Beeper, for instance, has done great work in creating a ton of bridges that really make the matrix experience much more powerful and interesting for people.

I love that I run my own home server and I use that home server and a bunch of bridges so that I'm recreating Yield, Pigeon, or Meebo experience but all FOSS which is pretty great. Then we, we also have I believe it's the etk. cc folks who have created some great software that packages matrix and bridges and other sort of related libraries to make it easier to run for home server operators like me, who are good enough sysadmins to be dangerous, but not to be employed as a sysadmin.

So there, we, you know, I'm, I'm very encouraged by the number of contributors and organizations that are contributing in the ecosystem. But it is, we have a lot of work to do for sure, to to get more upstream development.

Matthew: Hopefully the good news is the element isn't a evil corporate upstream trying to screw up the community or whatever, because frankly, element success is completely contingent on matrix of success, it would be sabotage if we did something.

to make we as Element did something to make Matrix less successful. And so there is that guiding principle throughout, and has basically been enshrined in the Element, sort of, core values since day one, that the company exists in order to realise the Matrix ecosystem, and then hopefully provide a flagship killer app on top of it.

Noticing that many of the previous technologies b abysmally failed due to a lack of a killer app. Like, how was Activity Pub doing before Mastodon came along? It was a disaster and I was, well, what was the killer app for SIP X and ppp, IRC? You know, they, they never had the kind of obvious go-to thing, whereas nowadays, the better or worse, well, if you ask what the killer app for email was, depending on the year.

It's gonna be Hotmail, Outlook Express, Gmail, Lotus Notes, I would argue. Profs. Link. Yeah, or Pine. CompuServe.

Jonathan: Yeah.

Matthew: But basically, it had the kind of, you would be able to ask somebody and they would have a strong opinion as to the killer app for email is this, and email itself is the killer app for the internet.

So, Element was trying to do that and is trying to do that for Matrix.

Jonathan: So I want to jump in and we've got a question from the chat room and I think it's going to be a great segue to talk about something important. So MashedPotato asks, Are matrix elements aligned with signal etc on their position To pull out of territories if end to end encryption is outlawed or if they're required to have back doors And this is a reference to I believe the the the way the governments are saying it now is chat control Which I don't know a whole lot about But apparently it's a thing.

It's become a thing in the last few days. And so I don't know whether Josh or Matt wants to take this first. I have a feeling that you're both going to want to comment on it. So let's go yeah, let's, let's, let's, let's kick it off. Take it away.

Josh: I would be happy to jump in on this to get us started.

So this is this is proposed legislation. That's been in the mix for at least a couple of years now. And toward the end of 2023, it seemed like like maybe, maybe we were going to be all right. But it has come back up in 2024 and they, One of the sticking points has been basically their way of undermining end to end encryption.

I think their, their new turn of phrase for this is upload moderation, which in their view is somehow not undermining end to end encryption, but it's just mental gymnastics. It's the same thing by another name and I think so we're, we're monitoring the situation you know, we have our, our, our foundation's DPO has been paying A great deal of attention to, to this and how it impacts matrix is different than how it impacts other communications technologies.

So that's one thing I want to signal and I want to unpack that a little bit, but suffice it to say we should be fighting tooth and nail against legislation like this because it makes everybody less safe. It's everybody less secure whether or not you're based in the European Union and no matter what technology that you are using.

So We're very concerned about chat control. One of the things that is a little different in Matrix, and this is something that Matthew spoke to earlier is the fact that it is not just decentralized it is, excuse me, it's not just federated, it's also decentralized. Or, you know, it is both of those things.

And so when I think about you know, If something like chat control were to get passed, and let's, let's assume for a moment that it doesn't get hollowed out or blocked by the courts, right? And, and there's reason to think that it might. Let's say that it makes it to an implementation and enforcement stage.

What does that look like for us? And the way that I think about that is how would that legislation interact with client developers, server developers the specification itself, and then home server operators, right? Because all of these things are different moving pieces and are you know, sometimes in different jurisdictions are developed by different people who are subject to different laws.

And so. When I think about what the implications of this passing might be for instance, The foundation, again, its core role is to steward the specification and and something like this goes completely counter to the mission of the foundation. And so I don't see any world in which we would see the specification change to accommodate the magical thinking of legislators.

So let me, let me start with that. But then also all of the software is open source. Right, you know most of the popular clients in matrix And indeed also the popular servers, they're all open source, you can scrutinize these things. And if for instance you see that a developer of one of these things, because they're based in a certain jurisdiction, is starting to make changes that would be concerning, fork it.

Right? Fork it. And, and, and don't, don't adopt those changes. Now, of course, you have to be mindful of not only the technology that you're running, but the technology that the people you're communicating with are running. So, in an ideal world, this stuff doesn't pass. This stuff doesn't get enforced. But I just want to highlight that the way that Meredith Whitaker and Signal have very rightly Taken a principal stand and say, we would rather pull out of the market than then undermine our technology and people's security.

You know, that's the right thing to do. And we would take a page out of that book, but also what that looks like in the world of matrix is different because we are structured very differently.

Jonathan: Do you, do you foresee a possibility, so let's just think like worst case scenario where the, the Matrix Foundation would then have to move out of the UK, just as a result?

Josh: Well, that's interesting because I don't quite know how this is going to impact the UK specifically, right? Being in, in this post Brexit world, I don't think it's not clear to me what, what, what would our obligations would be.

Matthew: I can take it on the UK side. Let

Jonathan: me, let me make sure I have one thing clear because I, I pay attention to US politics, not nearly as much to EU and European politics.

So the chat control legislation, that is a piece of EU legislation? Or an effort. Okay. So not, not directly applicable to the UK. Interesting. Good to know. I did not actually know that.

Simon: And it's, it's important also to know that that legislation is, is not primarily a piece of technology legislation. That, that piece of legislation is primarily a child protection measure.

And so the legislators who are involved in it are not people who understand that open source even exists, let alone how software works. And so you know, that, that piece of legislation itself is something of a special case in the overall landscape. But we do in the UK have another piece of legislation called UK Online Safety Bill which is just as as ill advised.

And unfortunately, because of the way that that legislation works in the UK, it's harder for us to access and influence than the European legislation is. Because the European Union is actually extremely open to discussion with outsiders, whereas the UK government, not so much. But Matthew, I should be asking you to say these things.

No, no, not at all. I

Matthew: mean, honestly, last year was a really terrible year for matrix and element and by very many different metrics. And one of them was the UK online safety bill. Not least because historically I think we've depended a lot on our European friends to wave the flag to prevent Well to protect human rights and avoid mass surveillance kicking in post brexit We can't suddenly turn to the french and germans and say hey guys, can you please Confirm that it's actually a really bad idea to have mass surveillance, so instead I find myself doing it and Spent basically a year turning up on the main tv and radio programs trying to explain to everybody That a backdoor that is there to protect the children Will be abused by attackers to attack the children as well as everybody else at the same time And meanwhile the people actually doing illegal things will just continue to use actually secure systems rather than the ones You where the technology has been deliberately weakened in order to scan what everybody is saying.

And it didn't work because the punchline is that the online safety bill is now the online safety act and it got passed by government So the worst case scenario that we're worrying about with chat control on tomorrow I guess thursday when they vote on it has already been and gone for the uk and yeah, it failed and so I feel massively burnt and a bit stupid for going and Investing the time to try to influence the process there and it is terrifying to see the eu suddenly You Sprouting off in the same direction particularly in the dead period immediately after the four yearly election cycle which happened last week and It's all in a bit of disruption and it kind of feels like it's suddenly come out of nowhere.

There's something To try to rush through when everybody isn't looking there is one good piece of news though on this which is canada somewhat unexpectedly very vociferously said they were not going to scan end to end encrypted messages Because they had been listening to the debate in the uk around the online safety bill and entirely agreed with the argument That this would be catastrophic to online privacy and would create a surveillance state and they didn't want to do that So even if I failed in the UK, apparently Canada was listening and took the hint, but the EU, meanwhile, has gone in a different direction.

So it's scary times. This is a dark timeline that we are in. Yeah,

Simon: I don't think you actually failed, Matthew, because, you know, when the Online Safety Act was passed was passed. Ofcom, which is our regulatory agency in the UK, was given a fairly open ended remit to go work out, to interpret how the the law was going to be enacted.

And that means that we've got a significant delay, and we've also got an agency which is going to report back to government that they've discovered it's act, unfortunately, impossible. Which of course if you're a computer scientist, you know, it's impossible to to, to having, you know, the, the very phrase encrypted back backdoored encryption contains within itself a totology, you know, it's either encrypted or it's backdoored.

It's, it can't be both at the same time. So you know, there, there's a good sign there and I'm not. completely desperately miserable about chat control in Europe either yet. All that's happening is the European council is doing part of the forwarding that the ongoing process in pushing it forward and the new parliament is going to have to stand up and Do something with chat control.

And fortunately, the right wing did not take control of the European parliament in the elections. That what did happen was all the people who understand these issues in the green and pirate parties got decimated, but all the centrists are still there. So we do have a job on our hands to go and educate this new wave of centrists about how a, a functioning democracy needs to recognize the, the statistical or the proportionate relevance of different aspects of protection.

And they need to have people explain this to them. So I think you did a great job, Matthew, honestly, and I'm, I'm really hoping you're going to, you know, so Matthew did a memorable presentation in the digital markets act presentation in Brussels where we heard from all of the big companies. Platforms about how impossible it was to do interoperability and instead of standing up and arguing, Matthew just said, I'm going to do a demonstration of making what, what, what did you have interoperating?

I think it was Facebook and Google or something, didn't you? Yeah, it was WhatsApp and Google chats. What? Yeah. So, so you'd heard the spokes, the spokes lawyers for those two companies saying how impossible this was and how it would take decades of research. Whatever. And then Matthew said, well, I just knocked up a quick demo of how it's possible.

And I'm going to send a message from, from Meta to Google and I'm going to do it live on live on the screen in the meeting room. I thought that was utterly memorable and I want you back doing the same thing because I think that makes a real difference. It breathes a. Breath of fresh air. It's totally alternative to all the boring people in suits saying, think of the children and you know, we need you back.

So don't give up, please, because we need you. And honestly, this is the role, not for you as a company, but it's for the matrix, the matrix foundation needs to get in there. So Josh, do you have any plans to go and engage in, in particularly in Brussels? To go and join in with those of us. You know, there's a little posse of us from a, from NL Net Labs and from Eclipse and from some other organizations.

And we are going out there every week and going and talking to these people and explaining which end is up and why the thing has to be plugged in. And we need, we, we really need people to come. Are, are, are, are you coming, are you gonna send someone.

Josh: So I am, I'm pleased to say that Denise Almeida, our data protection officer is already very engaged in these instances, responding to the comment periods of Ofcom and other regulators.

And the foundation we recently joined as a supporter of Open Forum Europe and are participating in a number of Eclipse working groups so that we are a party to these things and can do some advocacy. Transparently. The foundation is still an early stage nonprofit and and funding is a, is a thing that I lose sleep over regularly.

And so we do the best that we can on matters of advocacy with the resources we have we would like to be doing a whole lot more. And that is is ultimately going to be in contingent on on us being able to, to rally the support that we need to, to fund that.

Simon: I was also doing

Matthew: my stuff as Matrix by the way for this rather than Element.

So I was unashamedly wearing my Matrix t shirt and boxes and socks and everything else.

Simon: You know, I totally want you in the discussions about the Cyber Resilience Act and Open Source Software Security. Talking realistically about that. You know, your stuff on the Digital Markets Act was great. You know, come and join us in the Cyber Resilience Act meetings as well.

Digital Markets Act looked to me like it was going to be a gift to Matrix, of, you know, forcing the big platforms to open up and tolerate your existence. Is that how it's worked out, or has the reality been a bit different? I'm

Matthew: also a little bit jaded on this one , and that we hoped indeed that it would be a gift to us.

And we basically used Matrix as the reference to show how you could do end-to-end encrypted, interoperable communication between the big players. And we actually did a full integration with WhatsApp using their DMA APIs basically helping them develop the. Open APIs that they are obligated to expose to the world for DMA.

And as we speak, I'm meant to be writing a blog post to announce that and get feedback from the wider world on it. Although I'm about three weeks late, much to the irritation of WhatsApp, as well as everybody else who works on it. I promise I'll put it out soon. The catch though, the reason I'm jaded, is first of all, it only applies to people in the EU.

And WhatsApp are within their rights to say look, you're only allowed to use these APIs if you are also in the EU. At which point one would have to start IP geolocating matrix users to check that they're really in the EU. Which is not something that we want to do. But then Apple managed to wriggle out of it.

And then Microsoft and Google weren't really even in scope in the first place. So the only organization who currently is considered a DMA gatekeeper for messaging purposes is WhatsApp and Facebook. So Meta. So, whilst it's been great working with them, and honestly it has been. I saw that Will Gaffgut, the CEO at WhatsApp, was retweeting all of our chat control, propaganda yesterday and agreeing that, you know, you know, you know, it's a serious situation if WhatsApp and Matrix are doing kind of joint PR against legislation in the EU.

But on the other hand, they are the only people who are currently in scope for this. So either there's a way that Apple manages to un get, obligated to get involved, which seems quite hard because they've done a pretty interesting judo move where on one hand, They've added some really funky post quantum encryption to iMessage that makes it incredibly hard to interoperate with anybody.

And then at the same time, they've started speaking RCFs, the horrible Green bubble thing that Google have on Android. And so they can simultaneously say, Oh look, we interoperate anyway via RCS. Plus, obviously we would never be able to interoperate encryption that nobody understands. And that I'm not quite sure how we dig away from that.

So if nothing else, we might be able to at least do matrix to WhatsApp if you're physically in the EU.

Jonathan: There's about, there's about three different directions. I want to go with this all at once. On the, on the political side, We talked a little bit about left wing versus right wing. And of course, my perspective on that is entirely U.

S. centric. So I understand that it's a different conversation. I would just say this, regardless of which side someone is on of that political divide, I think if you actually explain to people what this means, like the idea of backdooring encryption, it means that Facebook or WhatsApp, whoever is doing it, they can read your private messages.

And as an extension, that means that the government can read your private messages. I think Basically, everyone understands that that's a problem. Okay. It's, it's just well, as we say, it's kind of a matter of how it's packaged by the different groups and the people that are in charge of them. Anyway, that's all I will say because I do, I do not want to dive any further into the politics here.

I, that is what I call a cup of coffee conversation. If I was with you in person over a cup of coffee, I would be glad to talk more about it, but not here on the show. In person over a cup of, I know I have my coffee too. But you've mentioned And this is something that intrigues me with encryption Apple is post quantum.

And so I, I guess the setup question here is, is Matrix considered a quantum? Post quantum secure or just as soon as someone gets an actually working quantum computer are all of the matrix conversations going to be a decryptable.

Matthew: So excellent question. We started working on it last year and in fact, one of our lead crypto and rust developers went on sabbatical for a couple of months, came back from it with a great big push request or pull request.

For the dosimates, which is, end to end encryption implementation for Matrix on the Matrix Foundation side, which adds PQXDH, which is the amazing five letter acronym from the SIGNL team that describes Diffie Hellman key exchange with post quantum protection. And it's what SIGNL launched, I think, about a year ago now, and they kindly documented it in the public domain, and it's what we have Dutifully gone and cloned and implemented in Rust.

Just like we did our original encryption as a port of the signal double ratchet over to matrix. Now we've done the post quantum double ratchet over to matrix too. Now it isn't live yet. We're hoping to finish it off in around September. We also potentially have a really exciting set of announcements around that, which I can't talk about yet, but watch this space for September, in terms of basically trying to position Matrix as a playground for post quantum.

Because it's not really clear what the best architecture is. We've got them in CloneSignal here because they're smart people and we trust them and we think they know what they're doing. But, for instance, there was a Chinese paper a few months ago proposing an attack on lattice based encryption, like in Kyber and NTRU.

Kyber being the, Primitives used for key encapsulation in PQXDH, and the only known solution to that, which is post quantum resilience, is this thing called Makilisi, or Makilise keys, with the catch that the keys are about two megabytes large. So, imagine putting that in your ssh. Authorized keys.

So, I think it would be really interesting to see what happens if you use Matrix's modularity to go and experiment with different post quantum or indeed normal encryption implementations. So today we already have double ratchet as well as MLS. The IETF RFC 9420 Implementation of Group Encryption.

Both of those can in turn be extended with post quantum primitives like PQXDH and Kyber. But then perhaps somebody might turn up and also experiment with Matalese on top of that. And perhaps it turns out, if the lattice based stuff never worked in the first place, everybody who was using the experimental 2MB keys gets the last laugh, even if it takes 5 minutes to set up a conversation, because it has to copy hundreds of megabytes of data around the place.

So it's looking kind of exciting and the apple stuff is weird because they just did their own thing. Yeah, and they did a Kind of a second order of post quantum key exchange on top of the normal stuff Which they call pq3 to make it a bit more different.

Jonathan: Shall we say? Yeah, well, I mean that's kind of apple's calling card these days.

They've got to do their own thing so obviously like i'm gonna put my You My amateur cryptographer hat on here for a second like the the threat of quantum computing to encryption We've got to take it seriously, but like what is what is your gut feeling? Is this really a thing or our quantum computer is going to be forever five or ten years off?

Matthew: I think it is definitely a real concern In terms the time frame who knows There is definitely some silicon snake oil going on where people are pushing this as an urgent thing where everybody has to invest and buy post quantum things now, now, now, now, now. But on the flip side, it's just a matter of time.

As in, any, the reality is that any competent intelligence agency is just going to be sitting there storing everybody's traffic onto disk. putting it in a big mountain somewhere and whether it's two years, 20 years, 200 years. At some point, somebody is going to find a way to brute force using whatever technology, the current encryption, and replay it all and see what everybody was saying.

So the sooner that you can get additional defense in depth against that, using wacky new encryption techniques, the better. And I honestly wish that we had the funding to have worked on this earlier, rather than relying on somebody going on sabbatical and doing it in their spare time, unpaid.

Jonathan: Yeah, yeah, the only the only real danger that comes to my mind when I think through this is that that concept that They're wacky and new encryption techniques And anytime you're talking about encryption and you're talking about new you have a a Potential problem, right?

Like it's there's something very useful very very valuable and using encryption that has been around for 10 or 20 or 30 years and people still not figured out how to break. But anyway, that's That is all the time that that really I have to dive into the the the cryptography thing, which utterly fascinates me I do have one very quick thing, which

Matthew: is that the trick is to do both You do hybrid.

You take the existing stuff, which you know works, and you put the wacky stuff on top, and then you hedge

Jonathan: your bets. That way if the wacky stuff turns out to be broken, you still have the existing stuff. Yeah, that's, no, that's clever. That's great. Okay, so I want to ask about the Matrix Conference. Is there actually going to be a proper Matrix Conference coming up sometime soon?

Josh: There sure is. We will be gathering at Mitosis Labs in Berlin September 19th through 22nd of this year. And for folks who are frequently on the conference circuit, that is Immediately following the Open Source Summit Europe. So that same week. So for anybody who's making the trip, it'll be a little easier to tech.

Matrix conferences on. So we are expecting to have a an unconference on the first day, two days of pop proper conferencing. We've got the call for papers is open presently. We're seeking sponsors, of course. And then on the Sunday, we will be having a contributor sprints and community meetings.

So the aim is to as we've been partnering with the matrix community summit. Team to create a space for the engineers, project managers all sorts of contributors in matrix, as well as policy makers and people who care about things like data sovereignty create a space for everybody together.

Jonathan: Very cool. I, I know there's a bunch of questions that you guys had listed in the rundown and we have not gotten to about half of them, I think. Let's say, pick one topic that we did not get to, that you would consider the most important, you want to let somebody know about. What, what did we not ask you about that we should have?

Matthew: Matrix 2. 0! Okay, what's up with 2. 0? So this is basically Matrix that outperforms the centralised normal apps. Like, historically, Matrix, first of all, was alpha, and it barely worked at all. Then it was beta, and it worked most of the time, but it was pretty slow, and the UX was terrible. Then we got to 1. 0, which was about five years ago, and it was starting to get properly usable.

The UX still wasn't great, but critically it was slow. Matrix 2. 0, we've just been doing nothing but optimization work, both for the usability, UX, as well as the performance. And it's things like instant login, instant sync, instant launch, invisible end to end encryption, so you don't have to mess around verifying devices and things, because when you logged in you did it automatically, like, Signal or whatsapp does and basically getting the ux polished to the point where we are out competing our dear friends at meta or indeed signal So to me, this is what the last 10 odd years have been building up to and I Should be going live at matrix conference on september the 20th, assuming that we Meet our deadline and I cannot wait to actually finally say guys It's here.

Please install your favorite matrix client, which supports matrix 2. 0, and you will see that it is as good as it was cracked up to be 10 years ago. And I'm so sorry that it took us 10 years to get here, at which point I assume the heavens will open and I'll ascend in a blaze of glory and head to Valhalla or something.

Yes. And that's all still interoperable, is it Matthew? Yep, it is. So it's backwards compatible with normal matrix, but it's basically new APIs. That are designed to not suck in terms of performance and usability.

Simon: Right, because I'm sitting here I've got a box full of very old computers on the floor down there.

That I'm busy, I'm busy reconditioning and, and dealing with. And one of the things you discover is that that you can't install anything new. And the, the, the, typically the reason is because the certificate, the, the, the crypto certificates have expired and there's no way of getting the new ones because of a version level problem.

And so, you know, one of the things I, I really look for in something like Matrix is to make sure that as you make progress you're not leaving all the, the old junk that people like me run. Behind so, so I'm very pleased to hear you saying it's interoperable.

Matthew: We haven't broken backwards compatibility yet.

It's been 11 years. I, it's a bit contentious. Some people on the team are pretty upset that we jumped through hoops to maintain compatibility with day one matrix, but I see it like the web. Like the Space Jam 1996 website that still runs fine on today's browser despite being 30 odd years old. And we want to do the same thing for Matrix, but on the other hand, encryption needs to move onwards, and so if we turn on post quantum, because we have it, And you're on some old client that doesn't support it.

I think we rely on the open source community to then patch your Windows 3. 1 client or whatever it is. So they can actually support it. If it's any consolation, one of my pet projects is to get my Dragon 32, a TRS 80 color computed free clone for our American viewers to talk matrix. And I actually had it up and running last weekend and I was very pleased that the very first computer I ever owned 40 odd years ago was able to talk through to Matrix.

So I'm gonna, that's my baseline compatibility to maintain. That's

Jonathan: very reassuring. Matt may have just answered the question I was about to ask. I like to ask people, especially with projects like this, What's the weirdest thing that someone has done with your project? What's the strangest, most surprising thing someone has done with Matrix?

And the answer may just be running it on a TRS 80 clone.

Matthew: Probably isn't. Some of the most creative things are definitely not appropriate for the show, unfortunately. But suffice it to say, there are some IoT use cases which surprised me, particularly as the German healthcare industry, I think, ended up hiring the people responsible for these IoT devices to do very serious work turning a slightly blind eye to their previous expertise.

But we'll see. So we did media for matrix, which is kind of fun. So you can jam over matrix, a musician. We've built an entire metaverse system called third room, thirdroom. io on top of matrix, which is still better than any other online virtual world. Infrastructure, frustratingly. What other weird ones have we seen?

There's a brand new project called Posca, which I saw yesterday that looks really fun. And it's written in Occamul, a well known language. And it is both message boards, and chat, and microblogging, and TikTok clone, all on top of Matrix. And it looks really exciting, and somebody going full crazy. Let's see how far we can push this thing.

Jonathan: Yep, excellent, excellent. Alright final two questions for each of you. We'll actually start with Josh. And I want to know, what is your favorite scripting language and text editor? Ha ha

Josh: ha, okay. Favorite scripting language? Alright, well, I got my start with with ASP and spent like a decade working in PHP.

So, PHP is, is my old standby but these days when I have to I have to do any coding, I tend to go to Python. Text editor, you know what? Nano has never let me down.

Jonathan: Ah, yes! I do, I do, I'm a Nano fan. And I think it's Well, for me, it's because I got my start in QBasic. In, in Microsoft's Basic. And the editor for QBasic is, is, Well, when I come to Nano, it just, it feels, it feels like home because that's where I got started, so.

Alright, Matthew, same two questions. Favorite text editor and scripting language?

Matthew: Oh god, it's tricky. So, on the text editor side, it's pretty easy. I have to say, I'm afraid I'm a Vi. Person, not a VIM person, but a VI person. And the reason is that even if my first computer was this TRS 80 clone, my second computer was the Silicon Graphics workstation.

In fact, the 149th machine that SGI ever made. Beta hardware back in 1984 at the expense of showing my age. And this thing was amazing. It's like half a height rack. It had 24 bit color. It had 32 megs of RAM. It had a 70 megabyte hard disk. And it had UNIX. System enhancements that turned into IREX eventually about three years later, and it had a 68, 000 processor running at like 8, 8 hertz, and that was my first proper computer, and that is why I'm such a geek, because I spent my entire childhood learning C and Iris GL and network programming and all sorts on this thing, and it was 100 percent done with Vi, because Vi and Edlin were the only editors that you had on this thing, and I, I've never looked back.

Now, on the scripting side, this is controversial, but I have to fess up, people who know me well know that this is my darkest secret, that I'm a massive Perl addict, and I used to run the world's biggest Lord of the Rings website, the one ring dot net, that was written entirely in mod Perl, and statically compiled down and run, it's served by Tux as a Linux program.

HTTP, Linux kernel HTTP server, and so I write huge amounts of Perl for that, and I still think and dream in Perl, unfortunately. Nowadays I would probably use Python or Node or Rust or something, but if you ask me to do a quick one liner to look through a ton of logs for something, I will produce the most unreadable Perl straight off the top of my head that you have ever imagined.

Jonathan: When, when Randall Schwartz checks in and listens to this episode, he will get a kick out of that. Alright. Guys, thank you so much for being here. We've had people in the chat room already tell us we need to come back and do a part two of this cause there are things that we did not get to. So we will be in touch about that.

Bring you back in probably a couple of months or six months. We'll see. We'll see when, when it works out. But both definitely have you guys back on and chat with you again about what's going on with elements and matrix and maybe a little EU and UK politics because that's just kind of all tied up into it.

But thank you so much for being here. Awesome, thanks for having us on.

Matthew: Pleasure to be here.

Jonathan: Yep, awesome. Alright, Mr.

Simon: Simon, what do you think? Well, you know, I have a lot of time and respect for what Matthew and Amandine have been able to achieve with Matrix. I think it's a great project that they have been good stewards of.

And the direction that it's headed, you know, the way that it's, you know, Become adopted so universally in the open and free software movement the way it's been adopted so widely in so many government contexts, I think is a testament to both the technical work and the also the stewardship of the brand and the, and the code.

So I'm, I'm a massive fan and there was nothing in the last hour that. Suggested to me that I've made an error in coming to that conclusion

Jonathan: about you.

Simon: What do you think?

Jonathan: No, I enjoy it. I, I very much like the, well, obviously it's, it's, it's federated. And I love that. That means that there's no, you know, there's no one server runner that can, that can say, you know, for whatever reason, I don't like this person and therefore they don't get to use any of the matrix services.

And then. The fact that it's decentralized, you can run your own server. I think it just, it ticks all of the boxes of what you want a communications protocol to be there's, there's no way there's no way to use the technology against anybody. And I think that's important. That that's that's important because let's see.

How do I want to put this? Even the times when you think you have a good reason to use a technology against somebody It's still a bad idea. And so the fact that you have that built into Your project is is super important there's no, you know There's no government that anywhere around the world can put their thumb on the scale because you could just fork it and run your own server So I, I love that aspect of it.

And then I think it's, it's really fascinating. And this is something I wish we could get, could have gotten into more. It's an open source project that they're actually making money with. And, you know, we didn't ask him a whole lot about the details about that. Maybe, maybe I should say we're using the term making money a little bit loosely because they're still on the path to profitability, but like they've got income and they're, they're, there's actual income from it.

And that in and of itself is a big hurdle for some open source projects Turn the corner to even having income associated with it without, you know, having to metaphorically speaking, sell their soul to be able to do it. And it seems like, it seems like matrix and element have avoided having to sell their soul to have any income.

Simon: Yeah. So, you know, I forgot to ask the question about who the backers were and that can, that can be in the next episode. And I think that, that question about, you know, looking for, you know, a way to sustainability models of projects that have become sustainable. I'm hoping matrix makes it, you know, Matthew makes me feel warm and contented that they're heading in the right direction.

They, they, they, you know, they're operating loss the last few years has been significant. But then so has the investment that's getting them to profitability. So I I'm very hopeful that we're going to see matrix under undergirded with a a profit making ecosystem that is going to make sure that, you know, just like Thunderbird has gone on forever.

Because there are so many people who want to contribute to making it succeed. I'd like to see matrix be in a position where whatever happens, there are people there to both fund it and to work on it. And, and I, you know, I think the separation of the matrix foundation out was, it was, it was a good move.

And I think that the the the funding for element is a good move. The big missing ingredient, as Josh said, is more than one party making a profit from matrix so that matrix can have a sustainable funding, whatever happens to the, the commercial counterparts. And I think that's the key missing. And I was going to say missing element.

That's, that's, that's. That's the key missing component at the moment is, is, is Oh no, we've lost

Jonathan: Simon. With such a, with such a serious look on his face. Making, making such an important point. Well, I was going to let him do his plugs. I know he is webmink. I believe you can follow him at webmink. com.

You can find him on GitHub. I believe also under webmink. And he's got a he's got a GitHub sponsorship there that you might want to check out as well. So hopefully, hopefully Simon will be satisfied with the things that I have plugged for him. If not, I'll get him to record a little, a little a little bit afterwards.

All right. I think that is it for the show. Thank you everyone for watching and listening. We, we did an experiment this week. We tried something. We shot out straight over YouTube. And that seems to have worked okay. And boy, that simplified things for the setup. Hopefully it will make things simple for the edit afterwards as well.

We will be back next week with a couple of developers from Armbian. That is the Debian port for arm, which. Gets used in a lot of places. You may not realize it, but it gets used in a lot of places. So we will be talking with them next week. If you want to follow me, of course, you've got my work over on hackaday.

You've got the security column goes live every Friday, and then, of course, there's the untitled linux show still over at twit. We appreciate twit over there, keeping that going. And we appreciate Hackaday stepping forward and keeping Floss Weekly going. We will see you next week at the same time, same channel here on Floss Weekly until then.

Episode 787 transcript

12 juni 2024 | min

FLOSS-787

Jonathan: This is Floss Weekly, episode 787, recorded Wednesday, June 12th. Video Ninja, it's a little bit hacky.

Hey, this week, Kathryn Druckmann joins me and we talk with Steve Seguin, the creator and maintainer of Video Ninja, Raspberry Ninja, Versus Cam, and the new Social Stream Ninja, it's all about WebRTC and comment aggregation for doing live streaming, it's a lot of fun, you don't want to miss it, so stay tuned.

Hey folks, it is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today as the co host, we have the lovely Catherine. Welcome Catherine.

Katherine: Hey, thank you. I'm glad you pinged me. Thanks for having me again. Yes,

Jonathan: yes, yes. I'm glad to have you back.

The last time we were going to have you on to co host, you sent me a last minute email and said, Oh, by the way. Yes,

Katherine: that is true. I live in Texas. I lost power two times within a two week period. One was about a little over 48 hours, and this last time was about 28 hours. Again, it could have been worse.

It was worse for quite a few other people, but It was kind of rough and difficult to get things done when you don't know if you have power or not, and then you don't.

Jonathan: 48 hours, that is, that is long enough to really sort of make you think about things. Like if power is just off for an hour or two, like it's fine, just try not to open the refrigerator and everything, everything goes, life goes on.

But you get over a 24 hour period and suddenly you start thinking about, What do I do if power's off for a week? Can I survive this? Yeah, yeah,

Katherine: you have to start, yeah, yeah.

Jonathan: What changes do I need to make in my life? What could I do to be more prepared for this? Like, at least for me, when I go through something like that, I immediately start thinking, okay, how can I be better prepared for this next time?

Yeah.

Katherine: Prepper mode, yeah, yeah, for sure. I need more solar panels. And then, I also had a hotel booked, like, and then my power came back on, so. There you go. So I was able to cancel it, but yeah.

Jonathan: Yep, yep, yep.

Katherine: Yeah, that was fun.

Jonathan: All right. Well, we have a, we have a guest today. We're going to talk with Steve, I believe it's Sanguine?

Sagan.

Steve: That sounds good.

Jonathan: Hey, there's Steve. Steve Sagan. And he is the guy behind Video Ninja, which you may remember as OBS Ninja. We'll talk about the name change. And then there's some other newer projects that he's got going on. I want to ask first, Catherine, how familiar are you with this? You said in the pre show that you you use some of these tools for some other things for, for day job?

Katherine: Yeah. Yeah. I, well, so I've, I've used things. I mean, yeah, yeah. I can totally talk about it. Yeah. I know it's something secret. Yeah. I, but I, I, you know, I like to use open source tools where public, where possible. But yeah, I, you know, I, I would, I'm not a I'm not a person who has a lot of experience with streaming video and that type of content, but I have done a little bit of it.

And when I have done it, I have used OBS or, Video or things like that and I you know I'm really excited about this because I would like to do more with these tools And so this is a really exciting opportunity for me to have a conversation about it.

Jonathan: Yes, absolutely So something something that we've been asked about a couple of times is how does the technical side of doing the show work?

And some some know we stream the live video to discord, which is a lot of fun That word fun gets a lot of mileage there because a lot of times it's a, it's pain. So one of the, one of the things that we're going to do today that I think is we're going to talk with Steve, I'm going to give kind of a walkthrough of the technical setup for the show.

And then I'm hoping Steve is going to point out some places where we could do it better. And but let's not, let's not faff about anymore. Let's not. Piddle wasting more time. We got started a little late. So just squeeze every minute possible out of the hour. Let's go ahead and bring him on Steve. Steve Sagan.

Sagan.

Steve: That's fine. Yeah.

Jonathan: All right. Steve Sagan. Welcome sir, to the show.

Steve: Thank you for having me.

Jonathan: And so your claim to fame, at least in this context is it's mainly video ninja, right? Like that's what, that's what most people know, know your work for. Right.

Steve: Yeah, at this point Video Ninja, a little over four years ago, at the start of the pandemic was a pretty popular tool.

It still is. And it is, yeah, it's just only grown since then.

Jonathan: Now, when you, when you first started it, it was called OBS Ninja, right?

Steve: Yep. It was OBS Ninja. That name was picked around 2017. But it kind of infringes on the OBS studio name and also, um, confuses some users thinking that it's affiliated with OBS.

Right. And so changing it to, to video made it more generic cause it doesn't work with just OBS. It works with, with video. virtually any tool for any purpose. So yeah, the name changed to Video, and some people ask, you know, it's, it's V D O as in, and it's a joke on video. If anyone's curious.

Jonathan: Yeah, so, it's, there, there is, there are two, there are two ways to pronounce that V D O, you can spell it out, like a noob, like I do, or apparently, the cool way to say it is you just say the word video, and it's a pun, and it's done intentionally. It is

Steve: a pun, I, I try to emphasize the D a little bit, so it's a little more to the name, but yeah, it's a, it's a pun.

Jonathan: There was something that happened with OBS where some other tool was using the OBS name, like without any permission at all. I think it was a proprietary tool. I don't remember all the details of this, but the OBS crew came out and said, please don't do that. And it was about that same time that OBS Ninja just sort of, yeah, just sort of quietly.

So by the way, guys, we're going to, we're going to rebrand and we're going to, we're going to become Video Ninja now.

Steve: And that was, I think Streamlabs OBS was probably the biggest offender at that time. That's

Jonathan: right. That's what it was.

Steve: So yeah it was time to dodge

Jonathan: change. So give us the, you know, the rundown, like what, why did you pick the OBS name to start with?

What is Video Ninja intended to let people do? What's the, what's sort of the, what, what problem does it

Steve: solve? Well, back around 2017 I was playing video. I'm doing a lot of WebRTC work, you know, VideoNinja uses WebRTC technology, which allows you to do peer to peer in the browser. And OBS has a browser source element, which lets you put webpages into your video stream.

And I noticed that WebRTC was working within that browser source. And it wasn't working flawlessly by any means, but it was working. And I saw that open up. massive potential because it's a lot easier just to put a URL in than configure servers with Nginx and other aspects. So the tool is really focused on being a production tool for OBS initially.

And so that, that's where the name came from. It was meant to be more of a a fun, flexible tool for OBS. I forgot the rest of the question.

Jonathan: What are some of the other problems that Video Ninja lets you solve?

Steve: Well, it's a highly flexible tool these days. So, some people are using it to do podcasts without OBS at all.

For example, you can record within the browser. You can record directly to your Google Drive. It lets you bring in streams from mobile devices without having to download anything. If you have an older device an older Android, it will still work generally with, let's say, Chrome on that Android. It also facilitates peer to peer video, let's say, over your local LAN.

A lot of people are used to using cloud services, but if you have a bunch of cameras in your house and you don't want them going to the Internet and back you can keep all that traffic locally securely private within your own personal LAN, yeah. It does transcription, translation video effects.

Really?

Jonathan: Yeah. I did not know. I did not know I did any transcription stuff. That's cool. I'll have to look into that.

Steve: So, when you're using something like Chrome or Edge, it has a built in API that you do transcription. So you can add closed captioning to all the guests and each guest can have their own label if you choose to.

Jonathan: Nice.

Steve: Yeah, there's quite a few other use cases in this actually, I feel.

Jonathan: So, speaking of other use cases I, One of the open source projects i'm a part of they have an apple, you know, like a an iphone app They wanted to get me as part of testing of that. And so they sent me the the cheapest ipad that you can buy on android but still It's apple hardware.

It's fairly nice and it's so now I suddenly have this big portable screen and It occurred to me that this would be a great teleprompter. So I, I bought a teleprompter kit that sits underneath my camera now. And so I can put the iPad up there and you know, it shows in the, the, the half mirrored glass and I have a nifty I have to nifty teleprompter, but then I, I, I had to solve this problem of, okay, what app do I want to use to be able to hook up my teleprompter to my iPad?

My laptop here to actually be able to record and, you know, scroll through my text as I record. And I found a website that works great as a teleprompter, but I couldn't get the, the remote aspect of it working. And it's like, I know how to fix this problem. Pull up video Ninja on the iPad, pull up video Ninja on the computer, share the tab and boom.

I've got a scrollable shared teleprompter using video Ninja.

Steve: You're not, you're not the only one that there's. Quite a few people are using it for teleprompting and there's actually a way to invert the screen for those who are using a mirror.

Jonathan: Ah, nice. I think I think the the website that I use has that built into it that I can invert it but that's that's useful if again I didn't know that video ninja had that as an option.

But that's super useful because then you can just use you can use your Google Drive to do it or not even a website at that point. It can be just a document of some sort. That's neat. I need it. I need to make a list. I need to start a new Google Doc. And it is things that Steve has told me that Video Ninja can do that I didn't know about that I need to look into.

That is, that is the document. That's going to be the name of it. Because there's a lot, I'm sure. You, you have all of these people that are part of the project. It's, it's open source. So like it's easy for somebody to come along and hack on it and add something. But everybody that uses it, they're like, Boy, wouldn't it be cool if And then somebody goes, yeah, I could probably do that.

And the next thing you know, there's a new feature. That's how it works. That's

Steve: the way it goes, yeah. It's getting a little harder to keep up with feature requests, but it's not often we refuse anything.

Jonathan: Yeah, indeed, the way that I see this working with a lot of open source projects is and it depends on the project.

So in some cases there are feature requests. There's just like there's no way to do that. It's not feasible or we've tried and it's not nearly the good idea. You think it is like that happens, especially more mature projects, but in a lot of projects feature requests. The response for the maintainers is yes, that does sound cool.

We would love to be able to do that. None of us have time to do it right now, but that does sound cool and we would love to have

Steve: that.

Jonathan: So one of the things I wanted to do, as I said, is I'm going to kind of quickly talk through the way we do the show here. And then once I get done with this and Steve has comments on it, we'll turn it over to Catherine and let her get some, get some words in, get some airtime in.

So I've got. Yeah, I don't want to, I don't want to do all the talking. I've got the DSLR with HDMI output and then a capture card goes into the laptop and that goes into OBS and the way it's working right now, it's only my camera that goes through OBS and that is to be able to, OBS has this really nice feature where it detects.

Hangups and it will basically just do a, a recycle on your your V V four L two device. And so if your frame rate drops to zero, it, it kicks it in the pants and gets it going again. And then from video Ninja, I'm using the virtual camera and I've got the, the, yeah, from OBS, I'm using the virtual camera.

And then I have in Chrome, a tab that is the video Ninja mixer. And. My, my feed goes into the mixer and then I can see you guys. And that's how I actually do the conversation. That's what happens on this laptop. I also have OBS recording me and then on your two streams, I have the local recording going.

So. You know, I, I now have in, in Chrome, I have three downloads currently running and that's grabbing your audio and your video which I had something weird happen the last couple of times I've done this, I'm not sure why, but the, the, on that download, the video. Resolution was really low, like 400 by 600 or 320 by 240, something, something really low.

And I'm not quite sure why yet, but anyway, I'll continue on. And so on the computer behind me, I've got a second copy of OBS running and it is ingesting the three URLs from VDO Ninja. So that's the, that's my feed and Catherine's feed and Steve's feed. And it's doing the side by side. And that is then doing a a windowed preview of that, which then is getting captured for a screen share in Google Chrome.

And that is what then is going to discord. And then I've got a pipe wire device. It's a tunnel. And so it's like a point to point tunnel where you can put audio in on one side and take it out on the other side. Well, so I am telling OBS to send audio to that tunnel. And then I tell discord. To pull the audio out of that tunnel, which works.

Discord is really picky about what audio device it wants to use. So I actually have to start it and then go into a tool like Carla or Helvam or. Q jack CTL to be able to control, let you control pipewire. And I've been remap where the audio streams are going, because for whatever reason, discord on Chrome just will not pick the right audio stream.

And then I also have our door running on the desktop behind me and it is spying on those three audio streams coming out of OBS because the way OBS does this, it sends them as individual streams, which is perfect for me. It spies on those and that's where I get my. Multitrack audio recording. So when I get a guest on, that's the, those are the steps.

And some of this is automated. Some of it's not, but those are the steps I have to go through to connect all the things together because we want to stream out to discord. We want to get the, the multitrack recording. I like getting some video recording and just all these different pieces fitting together.

And I've, I've, I've tried to work on how to make it better for a while, I was doing it all on the laptop. But something changed either in Pipewire or Ardor, and I started getting a lot of dropouts in the audio recording there. So that's when I split between the two machines. And in thinking about it, one of the, one of the problems, one of the things that really makes this difficult is the fact that we stream to We streamed to discord, which does not have a way that you could push a stream into it.

It's only in the browser. So like if, if I could go straight from OBS to discord, that would make things a lot easier. So maybe, maybe at some point we're just going to start streaming to YouTube instead and something to think about. But anyway, that is the way the show is produced. And now Steve, I'm sure it's nuts and there's a better way to do it, right?

Steve: I don't, well, it's, it's, it's a really complicated setup and you make it extra challenging for me because you're using Linux and Pipewire and such tools. Ah, okay. I say that because most of my experience and most of the community tends to focus with Windows, of course. So most of the solutions I work with tend to be there.

When we start talking about dropouts with Pipewire or Ardor my strengths aren't necessarily figuring out those problems. That can be something like a buffer underrun, a CPU overload issue. Those often cause dropouts with audio.

Jonathan: So one of the, one of the things, one of the other things I know that would make this easier is, and I believe it's being worked on, is tighter integration between OBS and VDO being able to do things like direct direct WebRTC rather than having to run things through the browser.

Because in, in, in OBS to be able to pull a video stream in from VDO right now, OBS is running an invisible Chrome window, Chromium. And it, it renders the video on that and then captures it off of that. And I, I know, in fact, we can do that. Back several months ago, we talked to one of the guys doing the doing the work on it, that there there are some patches to be able to do like direct direct over HTTP pull of video into, and then pushing it out of OBS.

And so that's that's one of those things that's going to make that a lot easier.

Steve: OBS has had some investment, especially around the WebRTC side over the last year, there's a new protocol. called WIP which lets you publish WebRTC video to a server. You can also publish it to Video Ninja, but it's not it's a little bit limited because you can only have one viewer at a time.

But potentially you could also use another technology in the future called WEP that lets you bring in a single WebRTC stream into OBS as like a source without A browser window.

Jonathan: That's, yeah, that's the one that'll be super helpful for what we're doing. Yeah.

Steve: So Video Ninja supports both those protocols.

And so you'd be able to use WIP, or in this case WEP, to pull into OBS from Video Ninja if you wanted to. You'd lose out things like overlays and more advanced controls when you do that. Yeah, it should, it should have a lot of opportunities.

Jonathan: So let's let Catherine get in a few questions here. And I'm, I'm sure she has probably not quite as convoluted as a use case, but some thoughts on where she would like to use Video Ninja and then here after a while, we'll talk about some of the, some of the other tools.

Katherine: No. Yeah, no, I do not have as complicated a use case. I, I have a question. So yeah, I was checking out, you have a list of helper apps and tools, and I was hoping you can kind of walk us through some of those because it looks like. There are so many things that I can do with video ninja video video ninja that I didn't, I had no clue about, right.

I'm, I'm kind of, I would say I'm a dabbler in this space. I, I tend to call in and let people like Jonathan manage all of this stuff. Right. So I'm, I'm really curious to like, what is, how do I use social stream ninja? Like does that compete with commercial projects or products rather? Yeah, I'm kind of curious to know, like, what could I be doing that I don't even know

Steve: about?

Social Stream Ninja is one of the newer growing projects that I'm working on. The concept there is when people were using Video Ninja over the last few years, one of the constant feedback I got was users might be using Another commercial application that has chat and video combined. They want to bring in chat from YouTube, from Twitch, and have that part of their show.

And when they used OBS, they missed out on all those features. The destinations they wanted weren't available as a chat source. So I took it on myself to make a tool to let people bring in chat from all the chat sites they want, consolidate it show it in OBS as an overlay if they want to and do other sorts of features like feature chat messages, do text to speech.

So SocialStream is started out as a Chrome extension. And what it does is it When you have a web page open that has chat, it pulls the chat out routes it to another page that you can use as an overlay or a control panel, and it consolidates all the chat messages into a single stream, including icons for where the image came from, the name of the person chatting, if they have a donation, it all kind of shows up.

At this point, there's probably a hundred or over a hundred sites now supported including some sites that generally aren't supported. For example, Instagram is a challenging site to support Facebook Slack, Discord, Microsoft Teams plus all the expected ones like Facebook, TikTok, YouTube.

Yeah, it pulls all those messages in and it makes it available. Now it is a free it's free to use. , and it has, again, a growing set of features that makes it quite powerful. People keep asking for more features and I just keep adding them. So it's a, it's a pretty bloated tool at this point. Yeah Yeah, that app has quite a growing number of users, for sure.

Katherine: Very cool. You mentioned something, you mentioned, you know, people are always asking for new features, right? I mean, such as, such as the life of a project maintainer. I wondered if you could tell us a little bit about how you manage these projects.

Steve: That's a good question. A good question. I'm, I feel like I'm not the most organized person.

So, really, I kind of camp out on Discord, listening to the community, and I try to stay on top of requests, stay on top of problems. When they come in, I try to just tackle them. Then in there, I feel like once I put something into a backlog, it really doesn't get touched again. So that's one aspect to developing the project.

When I think about developing the project, I don't get too caught up in, in the UI, UX and making things pretty and for better or worse. So it's really a little bit 1990s looking in terms of design. Documentation I try to keep on top of. The community does help there at times, and that, that's very helpful.

When it comes to, like, actual maintaining the code, I try not to make massive changes. I try to work incrementally and I try to always have some sort of staging. So if a bug gets put in more than the beta sort of users red flag it earlier on this way. I can reduce the amount of testing I have to do because that's quite a, quite over quite an overhead for development.

Otherwise.

Katherine: What kind of participation do you get in terms of, of Yeah, you know, people submitting pull requests and, and contribution to the project. I mean, how, how much of this is just you kind of steering the boat and how much do you have people, you know, kind of pitching in?

Steve: It's, it's not as much as I'd like considering the scope and size of the project, but I do have several people who are, are, are pretty active and pretty helpful.

One, one user has been really great at updating the documentation for Video Ninja. So whenever I do a feature update and I post information in, let's say, the Discord about that feature they're often ahead of me on updating the documentation. And that, that's a huge time saver. So it's not, It's not a whole lot of thought that goes into that, but it is time consuming and requires some organization.

I have a few users who help moderate the Discord. And one of them in particular helps with the the Discord bot maintenance. So when we have problems with spam, we update the Discord bot. We have an AI bot that helps to respond to questions. We there's translations, and so users will donate translations for Video Ninja or SocialStream.

And there's a few users over the last course of the last three years. I'd say there's been three users who've contributed code to certain extents. Usually pretty casually, and usually more on the superficial side of things. But it's still helpful, especially when it comes to the UI, because that's where often where I, I need some help.

And so working with them to kind of approach that in a way that I can support them, and then add that styling has been very helpful. But in terms of like core low level code, and especially in the core of the app, there's not There's not too much support there.

Jonathan: So I want to make sure that we've made clear what, what exactly social stream is about and like what it does.

It, it lets you, what, aggregate comments?

Steve: Live chat comments primarily. Yeah.

Jonathan: And that is it, does it, what does it do with it after it aggregates though?

Steve: So it, it can bring it into a dashboard, and so you can see all the messages streaming in from all the different sites consulted. As a show host like yourself, let's say, you could select a message, and it, it, you could add it as an overlay to OBS so that when you select it, it appears as a lower third overlay featuring the user's message.

So if it's a question, you could select it and then talk to it. You can then hide it, dismiss it, You could also have it maybe on the side showcasing all the chat from Discord, let's say. So other people may not be on Discord right now, or they may be watching this on YouTube. You could have the YouTube and Discord chat consolidated on the side of the video, so everyone sees what's, the questions being asked.

Jonathan: Okay. So for another show we do that's still over on Twit, we use the last couple of episodes, we used Restream, Restream. io. And it's got, it's got some of those, some of those same features. It, do you know which one came first? Like, was this, this is originally your creation or is it inspired by some other offerings?

Steve: Well, ReStream had their own consolidated chat before SocialStream. The request was not necessarily in response to ReStream, it was actually in response to StreamYard. Ah. Users were paying and using StreamYard as an end to end platform. They wanted more power and they wanted to move to OBS, but they felt like the chat functionality was missing in OBS.

Jonathan: Yeah, yeah. Yeah.

Steve: So, but that's kind of where the request came from. Of course, since then it's, it's grown. SocialStream now it does things like a lot of bot commands. People are now integrating their own apps into or integrating it with SocialStream. Not only is it being used for overlays, then you can pull the messages out via several APIs I have and use it to control, let's say Streamrbot, another open source project, let's say, that allows you to do actions and advanced scripting.

People, people want all those chat messages and I'm offering it.

Jonathan: So if, if you want to build something like Twitch plays Pokemon, boy, that's a, that's an old meme at this point, but you know, something like that, you, you have the tools there to be able to make, make craziness like that work.

Steve: Yeah. There's basic chat commands already added in, but people are doing far more complex things.

Features that currently have been added. There's like a draw system so people can enter a poll or a raffle and be selected that way. There's like hype meters, there's emote walls, so if people dump emojis into chat you can have them dance around the screen. Yeah, there's analytics, so some people like to store all the chat to disk and run a sentiment analysis on it.

Jonathan: Interesting. Okay. What, what's the process look like of connecting the the tool to your different chats? Like, how does it, how does it actually scrape, particularly those that don't have an API? I guess I'm really curious about, like, how does it scrape chat out of Discord and Twitch and YouTube and all of those places?

Steve: It's a little bit, a little bit hacky of course the, the current, the original version, let's say is a Chrome extension manifest version two. And What it does is you just have it open. Certain sites you have to opt in via a little toggle in the extension menu. You can, you can restrict to certain channels as well.

For other sites, the notion is you just pop out the chat. So let's say you're in YouTube chat, you can pop out the chat. In Discord, you'd open up the Discord in the web browser. You'd enable a toggle for Discord. You could select a specific channel ID you wanted scraped. And then it will just listen to your your webpage and scrape directly from that.

Chrome is dropping support for Manifest version 2, which is like a powerful extension. I, I have kind of internal builds of Manifest version 3, but it's really hindered, let's say. So I have also worked on a standalone app. It's been available for about a year now. And this is a, it's essentially an electron, like a Chromium app, and it has all the functionality built into that, so you don't need an extension, you just open up those chats, or you open up those sites within the standalone app, and it's able to do it there.

But as well as I, as I move forward with development, I'm adding in more official APIs where possible. So the hope is with sites like Twitch and YouTube and whatever else, you wouldn't have to go through those hacky steps. It's just work. As you'd expect.

Jonathan: Yeah. Have you, have you been able to reach out to any of these any of these sites?

Like let's say discord. I think I know the answer to this, but I'm going to ask anyway, does it work to reach out to discord and say, Hey, I'm building this great tool. It'd be neat if you guys actually had an API.

Steve: I'd like Twitch has an API tick tick talk has a community created API. Let's call it and that's available as a, as a node extension.

And so with the the standalone app, I could use that API there. When it comes to the things like Twitch, they, they only recently allowed. Creators to restream to other platforms. But in that case, they also want you to isolate, I think, their chat from the other endpoints. So there's a toggle specifically to let you show only Twitch and not other chat if you have multistream going.

So I, I don't know how open they would be to the idea of consolidated chat given that.

Jonathan: Sure. So one of the things you, you mentioned in your notes here is a raspberry. ninja and versus. cam. What are, what are those?

Steve: So like, I have a bunch of auxiliary projects related to Video Ninja. Raspberry Ninja was originally, as the name can imply, kind of targeted for the Raspberry Pi, but it lets you run Video Ninja without a browser using Python.

On a Raspberry Pi and where this is a little bit interesting is a Raspberry Pi has a hardware encoder for video So even of something like a Raspberry Pi zero Can encode 1080p video and publish to Video Ninja so if you wanted a very low cost way of publishing from a camera or webcam to Video Ninja without having something like a heavy browser open.

You can use the hardware encoder to publish and multi publish at that too to multiple users with a Raspberry Pi via Video Ninja. Very cool. It's grown since then. It now works really with anything from Linux, Mac, Windows Orange Pi, other devices. And users are now using it for both publishing and pulling in video.

Some people are using it for AI work. So if you want to pull low latency video or audio in run machine learning on it or do audio to text conversion, you can do that without having to have a browser in the cloud. You can just run this Python script.

Jonathan: And then what was the other one?

Steve: Versus Cam.

Jonathan: That's the one.

Steve: Right. So Video Ninja is a bloated application in some sense. Like it does a lot for a lot of hardcore users. Sometimes there's a simple task some users want. ESports is a popular sort of particular use case that some people want, but they don't want to go through the complications of configuring everything for that use case.

You need high bit rates, you need a different set of features. And so Versyscam demonstrates how you can customize VideoNinja for specific use cases. Essentially, you can treat Video Ninja as like an SDK. You can embed it into a web page and build a whole new web page on top of it. Oh, cool. So versus cam is a completely different looking version of Video Ninja, a little prettier.

You can sign in, have some sort of memory on your your, your session, has a whole different layout designed specifically for for eSports. And there's several of these applications I've created based on Video Ninja, kind of demonstrating different use cases that you can kind of build. Yeah versus GotKem is just one that I think is one of the coolest, cooler examples of that.

Katherine: I'm kind of curious to go back to the project itself again, because I am interested in talking to people who maintain these big projects, especially without, you know, a big company or a big organization behind it. And I'm wondering again, like, And I, I, and I, I feel like I'm speaking a bit out of turn because I don't know that much about the project and how, how it's governed and whatnot, but I get the impression it really is mainly just you, right?

Do you have a succession plan?

Steve: It, I'd say at the moment it is mainly me. Yeah. There are some light touch users, of course, so I appreciate. In terms of succession plan, it's one of those things that does haunt me a little bit. The notion in my head is if I was to ever step away from it, I would probably open up the license even more.

Right now it's an AGPL license, but I'd probably consider moving into MIT. In it Video Ninja is more than just a repo, as well. It's also an actively hosted service. And that's how most users approach it. So, it's a little bit concerning because there's a cost involved with maintaining it. There's a level of DevOps and support.

And I don't really see how anyone in the community would would necessarily jump on, on wanting to continue that themselves. It, it, there's quite a few people who are running their own version of Video Ninja, but they're often doing it for their own personal sort of community driven projects or their own production studio.

So I'm not quite sure on the succession plan. It's, it is a little bit concerning, but I'm going to continue to maintain the project regardless of going forward. I don't quite know how to transition away from that.

Katherine: Yeah, that's fair. I wondered if you could kind of tell us a little bit about your thinking behind the AGPL license.

Like what motivated you to pick that license in particular?

Steve: Yeah, when I started this project, I, I, I needed to pick a license. And I talked to some advisors, and they always said, start with something more restrictive, and you can always make it less restrictive as you need down the road. So, it was less It was more from a position of comfort.

Where can I start that I'm feeling comfortable? And how can I transition away from it? There hasn't really been a need to transition away from it. It's primarily all, almost all client side code. And it's hard to imagine there really being a strong need for more permissive licensing at this point.

I do like the notion of having companies that are using the code for the commercial purposes to feel obligated to open source their changes. There is, there is this sort of concern in the back of my head since I am doing this all myself. If the project gets hijacked by a larger company Okay.

How do I handle that? I, I probably feel like I wouldn't want to work on the project anymore. So it, it's one of those things where I feel like once I am done actively developing it, it's one of those times I'm feeling more comfortable where I can then make it more permissive.

Katherine: Interesting. I, so I, I talked to a lot of people who maintain open, open source software, and I'm, I'm really interested in, in kind of the, So many issues that come from being a solo maintainer effectively, and especially of a, of a project that's quite popular.

And I, I, I wonder like, what do you feel a heightened, like a burden, a sense of responsibility especially when you're, when you were talking about security issues. And, and I just wonder like, if you, if you could wave a magic wand and grant a wish, what, what sort of support would be helpful to someone in your project?

situation.

Steve: Yeah. So, I'm, I'm currently managing the full stack from DevOps, security user support, GitHub maintenance coding, virtually end to end and it is challenging. I would love, obviously a, a team that could be self Self continuing, let's call it. I'm not sure of a better way of saying it, but it'd be nice to have a project that can take care of itself and can be self maintaining as people come and go.

It can continue on. I struggle to see how that would be possible. At the moment, I'm, I'm, I'm trying to encourage more recently, more development from trying to make the code more accessible to more users. The Magic Wand, in my mind, it, it's a little bit hard, but it'd be more It's a little bit concerning, but if I could find maybe a company that could see value and a shared vision in what Video Ninja is, and they take it on as more of a, um, like a little bit of a community project, but supported by more of a, commercial company, that could be very interesting to me.

I think if a company, for example, like Logitech or Twitch or one of these other companies saw value in it, saw the community value in it, and they wanted to put resources into it and keep it as an open source project, but maybe add more premium functionality to it. I see that as a very interesting way to see long term value and see it grow beyond what I can offer.

Probably unrealistic, but that's kind of one way I could see it potentially. Surviving long term.

Jonathan: I'm curious, have you, have you reached out to any of those any of those companies about that kind of idea? Like, have you, have you rung Logitech or I don't know, maybe even somebody like Intel and said, Hey, I've got this project.

Are you, would you be interested in adopting an open source project?

Steve: Yeah. Probably not in, in such words exactly. I I probably not. I'm probably not the best at networking and business on that front. I have reached out to these companies. Most of them, let's say like Discord and Logitech, when I reached out to them and had conversations three, four years ago, let's say when the project was new, they all were using or fans of it, let's say But they've found ways to incorporate their own versions of it into their own software.

So Twitch has something called Guest Stars now. Logitech with, let's say, I think Streamlabs. They've had their own sort of guest system. So they, I don't get the sense they're too interested to adopt, but rather be inspired by. And I feel like I've helped inspire lots of companies I love that.

I love the innovation. But it leaves Video Ninja still kind of as an orphan.

Jonathan: Well, it may be an orphan, but it's a tool that's important to a lot of us. So, I'm definitely Rooting, rooting for you. I don't know what the future exactly is going to hold, but hope it's good things. We have a question from the chat room actually that is, is extremely on topic with what we're talking about now.

And that is, is this a full time job for you or is it just an intense hobby?

Steve: It's a, it's a full time passion project. And I say that in the sense that I spend pretty much every waking hour. Working on these projects, Social Stream, more now than you know, quite a bit. So it's almost split between Video Ninja and Social Stream, and things like Raspberry Ninja.

The only free time I'm really making is for my personal health, as it kind of deteriorates over the years. I'm trying to find more ways to go to the gym. So it is, it's full-time. I, I get some, I get a couple hours a week to do consulting for some other companies I was involved with prior to this, but it's pretty much full-time.

Jonathan: Yeah. Yeah. Do you, do you have like a Patreon or are you part of a GitHub's sponsor, I forget what they call it. It's GitHub sponsors. We have all of that set up.

Steve: I, I, I do, I don't really promote it heavily. I do have GitHub sponsors, I don't have Patreon, but I do have, like, Buy Me a Coffee. I'm not, I'm not necessarily doing it for money I think, I think if I go down that path I'm going to get misguided a little bit.

So I, I try to, I try to avoid being reliant or promoting that, but the option is available and I do strongly appreciate, obviously, any sort of gift users provide to me. The way I think about it is I, I do spend, you know, between 1, 000 to 2, 000 a month on server bills and other costs to maintain these services.

And so the sponsorships do help, you know, help subsidize those costs. And so that's that's kind of my goal for sponsorship, just kind of to break even on those server bills. But yeah, if I'm going to be looking to make money, um, I'll probably work on other projects unfortunately.

Jonathan: Yeah, yeah, understood.

What is a. Let's go ahead and let folks know what is your buy me a coffee and GitHub sponsorship. What is, how can we find those?

Steve: I'm not even sure. There might be a hot link sponsor. video. ninja, but generally I push users towards my GitHub and you can click on the GitHub sponsors page and I list more than GitHub sponsors.

I listed the buy me a coffee link as well there. My, and my GitHub is username is Steve Sagan, S E G U I N.

Jonathan: That is the place to go to drop, drop in the, drop in the tip jar.

Steve: Yeah. Thank you so kindly.

Jonathan: Yeah, I think I think that's definitely something that's worthwhile. One thing we've noted over the years of doing this show is that open source developers need to be able to eat and pay their rent too.

And it's really, it's really kind of a challenge for a lot of projects, you know, because there, there comes this point with a lot of projects where you want to kind of turn the corner and go from, okay, this is just a hobby too. This is big enough. Now we could use somebody doing it full time, but that's a, that's a big burden financially to try to do, at least for most of us.

Steve: I feel like if I try to turn Video Ninja into a paid project as you said, hiring developers would probably be required for me to grow it into the quality product in terms of what a paid product would probably demand. User accounts, enhanced security, more server based services, prettier UI, better mobile apps.

I'd probably need to hire, you know, two or three developers at the least. They would want to pay, given the number of users that are using the product, and the cost of overhead of then turning into commercial business, it would kind of kind of force it to be a paid project. And that's not what I want Bidian Engine to be.

Jonathan: You're, you're, you're stuck in a similar way to a lot of other projects. Like I said, it's, it's difficult to turn that corner and it's even harder to do it in a way that's not going to drive your existing user base away. It's a, it's a real challenge for a lot of projects. Some, some have navigated it.

Some have made it work.

Steve: Yeah.

Jonathan: I'm

Steve: sure there are. If, if, if I were to do it, I'd probably make a whole new project. with a different name and it would be a premium version. I would probably need to get venture capital to fund it. I feel like whenever I see people take on new projects, me particularly, I get obsessed with that and I neglect the whole project.

And so I would only imagine then The community version of Video Ninja, let's say, would then be neglected, and over time be put more and more out of sight. Again, I don't really want that. So I've avoided turning Video Ninja into a commercial project, but at some point making money and having a retirement of some sort is going to be important for me.

Jonathan: Yes.

Steve: I'm kicking that can down the road right now.

Jonathan: I understand. I know how that goes. I do. All right. Well, we are we are getting close to back around to the bottom of the hour from when we started. And I want to ask a few wrap up questions that the hardest one I'm going to ask you is if you think through the things that you wanted to talk about, and then you think through the questions that we've asked, is there anything that we Neglected to ask about that you want to let folks know about.

Steve: I can't think of anything. I'm happy to answer any questions. Of course.

Jonathan: All right. And then what is what's the weirdest thing that you know of that somebody has done with one of your projects, be a video Ninja or the, the social. Oh man.

Steve: Okay.

Katherine: It seems like a dangerous one.

Jonathan: I mean, it is a little dangerous.

Let's, let's, you know, let's, let's avoid anything NSFW, but in this context, because we're talking

Katherine: about video, you know, but

Jonathan: yes.

Steve: There there's neat things people are doing, of course. Some people are using it with drones and they're live streaming their drone to their website with video because you can embed video ninja into a website.

And then essentially use it as like a, a low cost CDN. It wouldn't handle many viewers, but you often don't need many viewers for personal website. And so people can stream very low latency video from their drone to their webpage. I thought that was kind of cool. There's been some art COVID, for example, you'd have, you'd have performing arts centers have people acting out large performances with green screens and really advanced effects using Video Ninja.

And it's, it's quite, there's tech issues always with, with live video. And so just imagining someone being on this complex half hour show with back green screens and people trying to have two way conversations from different areas of the world is fascinating. Yes. There's another project where people are doing collaborative music jamming sessions together.

So it's, Video Ninja has pretty low latency, usually around 50 to a hundred milliseconds, but it's not low enough for playing together.

Jonathan: Right.

Steve: So what people do is they delay the audio or video a couple bars out and they kind of play out of sync to each other, but in to the same beat regardless. And so Video Ninja has a capability to let you delay video fairly accurately upwards of a few minutes even.

And so there's groups of people who are across the world who are using Video Ninja in part with other tools to do jam sessions where they use Video Ninja to synchronize all the videos. So they arrive in, let's say OBS in sync and high quality And it's really kind of funny how they're able to have a musical performance out of sync to each other, but all end up play out in sync.

And I find that a very interesting project.

Jonathan: That is really cool. So I didn't ask about this, and I think we should. So when How does, how does the video transport work? And what I mean by that is, so do I send video to the video ninja server and it then sends out the streams to each of the watchers or is it doing is it doing handshakes and then point to point?

So am I sending, am I sending a stream from my laptop to my desktop and my laptop to you and my laptop to Catherine, or do I just send one stream up to the server and it broadcasts it out?

Steve: All those options are available.

Jonathan: What happens by default?

Steve: The default is point to point, peer to peer, and so it is really complicated.

The default is each viewer gets their own encoded stream custom just for them. So if someone's having a bad connection, that particular guest or viewer gets a lower quality feed. That's custom encoded. Everything's peer to peer and the system will automatically switch to keeping the video on your local network, for example.

If it detects that it can, send it over the local network, peer to peer. That's part of the handshake. It determines whether it needs to send through a server, whether it needs to send over the LAN or through the internet. I do host servers, so that if you can't create a point to point connection, let's say you have security on your browser cranked to 11 it's going to send the video to a server so your IP doesn't get revealed, and then it will bounce that off back to the guest.

There are other modes. You can encode once and broadcast to multiple users. That's more experimental. There's also servers that I host for broadcasting. One service I offer is called Meshcast. io. It's part of Video Ninja, but it's also a standalone free service that you can publish video to and you can broadcast to a small audience via a server.

You have some users who can't handle multiple. Streams, because their computer is too slow or their internet is too slow. And so that's an option that lets the users participate in Video Ninja even if they can't do the default.

Jonathan: It seems like, it seems like services like meshcast. io that are going to be very server heavy are good candidates for actually charging for them.

Steve: Yeah. At the moment, the way that's working is I support third party sites. I, I've had some interest in some demand to, to offer. Like a paid mesh cast. The reality at the end of the day though is it's just too much work for me to maintain that as a paid service and do everything else. It would have to be a full time job just doing mesh cast.

And I've tried to find a balance there, but I, I, I just, it's just too much.

Jonathan: Yeah. Yeah, understandable. Alright, I do want to ask a couple of final questions before we let you go and that is, what is your favorite text editor and scripting language?

Steve: Ooh I, I, I like Notepad for a text editor. I'm a simple person at heart.

On Linux I will default to Vim. Okay. So I'm pretty comfortable with Vim. In terms of scripting languages, it depends on what I'm trying to do. My two favorite are Python and JavaScript. Vanilla JavaScript.

Yeah, it really depends on what I'm trying to do. More server side services would be Python. I might use Node, however, if I'm doing some sort of Lite API. And I love developing in the browser because it's it's so accessible. It's universal as a, as a applicational layer.

Jonathan: Yeah, makes sense. All right, sir.

Well, I sure appreciate you coming. Kind of, kind of last minute. I think it was just yesterday or the day before I asked you. I appreciate you being willing to be here today. And glad to be able to talk to you. And looking forward to hopefully a bright future for Video Ninja and all of your other projects.

Steve: Hey, thank you so very much. I appreciate it.

Jonathan: Yeah, thanks. Thanks, thanks. Pleasure. Alright. Catherine! We had a, ironically, just a little bit of technical issues. I'm still here,

Katherine: sort of. You're still here.

Jonathan: Yes. Barely. Some slight technical issues. But barring that, what do you think?

Katherine: This is great. I look forward to listening back to it for all this stuff that I couldn't quite hear.

But yeah, I don't, I don't know what's going on over here, but yeah, I'm having a little bit of a connection problem, but yeah, this is cool. I haven't heard notepad plus plus in a while, by the way, that was, that was a, that was exciting. I used to, that was my favorite quite a while ago, but I haven't heard that name in a while, so that was kind of fun.

Jonathan: I also enjoy

Katherine: developing in the browser, so, which is another phrase I haven't heard in a while. I,

Jonathan: yes. Work has

Katherine: changed so much that I think about those things.

Jonathan: Two, two thoughts there. One I've never used Notepad but the one that surprises me, Has surprised me is that so many people started using VS code and then I started using VS code.

It's like, okay, now I see why people like it, but you're right. We haven't heard as much notepad plus plus for a while and almost wonder where they're having people move from notepad plus plus over to VS code. Anyway, then in the browser is great. One of the things I like about doing development in the browser is that it.

In Chrome and in Firefox, both the developer console is so good. It's like, it's really good debugging and it's so easy to get into. Like the, the thing that drives me crazy about debugging is sometimes, not always, but something like, especially if you're doing embedded development, embedded development for like embedded devices, trying to debug is just, it's hard, like getting your debug tool chain set up, whereas in the browser, you just, you know, you, you hit F 12 or F 11, whatever it is.

And. There it is. You've got it. You can run stuff right in the console and it's all JavaScript. So it's, yeah. One of the, one of the coolest things about doing web development is that the, the deep bug. experience is so good. So anyway, back to back to video ninja. I kind of took a rabbit trail there. Yeah, no, it's, it's it's great.

I, we, we use it a lot. We enjoy it. I, I have gotten some inspiration for how to improve the show's tool chain from the, the episode today, the interview. That's very cool. Yeah, hopefully, hopefully things will be even better in the future. And then I do want to let folks know, speaking of the show for the month of July, we are actually going to we're going to try moving to recording the live recording on Tuesdays and.

I'm not sure yet whether we're going to be publishing the episodes on Tuesdays or still waiting for Wednesdays to publish, but we're going to try it out for July and move into Tuesday. And if that works, then we will probably default to Tuesdays for the future. And one of the main reasons is that that will cut down my workload on Wednesday and Thursday, because right now my Wednesday is.

Get up, record the show, and then spend the rest of the day until I go to bed making sure that the show is edited and up and available, and then get up on Thursday morning and spend the entire day doing the security article, and sometimes not getting that finished until two or three in the morning, and that is wearing me out, so we're going to we're going to move, we're going to move, yeah, we're going to move the Floss Show to Tuesday for the recording, and we'll see if that works for us.

Catherine, is there anything that you want to plug before we let everybody go?

Katherine: Oh gosh, I don't know. Sure. Yeah, I, I still, I, I, I do a podcast over at Intel open dot Intel dot com. You can find it there. Yeah, I talked to a lot of open source people. I, I, I'm doing a lot of conversations about this, this very idea of Maintainer burden, right?

And burnout ultimately. And so I'm really interested in talking to more project maintainers about that issue. So maybe that's something we can take offline later, but yeah, I yeah, I, that's, that's where you can find most me mostly hopefully with fewer technical difficulties in the future.

Jonathan: I blame the Mac.

I blame the Mac.

Katherine: Who knows? Or it could be user error. I don't know. You never know. I wouldn't put it past myself.

Jonathan: I want to mention, of course, Hackaday. We appreciate them picking up Floss Weekly, being the sponsors of the show. Now, not really sponsors, the new home, the new home of the show, not the sponsors of the show.

And then of course you can follow my other work there. We've got the security column coming on Friday mornings. And then I've got one in the works doing a review of the Turing RK1, one of those little embedded embedded arm devices. That one is pretty interesting. Hopefully going to have that done soon.

And then of course you can find some interesting stuff over on my YouTube channel. Just Search for me Jonathan Bennett over on YouTube and a lot of Meshtastic related stuff. And I think I think we're going to have another video coming out on, on Meshtastic here pretty soon. So watch for that.

We sure appreciate everybody watching. We thank you for those watching live, the small handful, and then those that get us on the download. And we will see everybody next week on Floss Weekly.

Episode 786 transcript

5 juni 2024 | min

FLOSS-786

Jonathan: This is Floss Weekly, episode 786, recorded Wednesday, June 5th. What easy install script. Hey, this week Rob joins me and we talk with Brody Robertson, a YouTuber, a commentator about things Linux, things Wayland, desktop environments, and his odd decision to start his Linux journey with Arch. By the way, you don't want to miss it, so stay tuned.

Hey folks, it's time for Floss Weekly. It's a show about free, libre, and open source software. I'm your host, Jonathan Bennett. We've got a lot of fun today, but first off, our co host is Rob. Hey Rob, welcome. Hello, good to be back. Yeah, it is great to have you and we kind of put this together at like the 11th hour last night So I appreciate rob stepping in and being able to co host and we've got a we've got a really fun guest You may have heard of him brody.

Robertson has a youtube channel and a podcast. I think several youtube channels Some other things but he is kind of becoming a Well, something of a personality, maybe we could say or a talking head, a commentator around the kind of the Linux ecosystem. And one of the, one of the places that I first became aware of him was sort of the some of the kerfluffle, let's say around Wayland things and the, the kerfluffle around Hyperland and found it to be actually a pretty interesting voice something of a voice of reason.

Would be kind of the category I would put him into if we were going to categorize him like that. Rob, I know I've shared a couple of his clips with you. Are you familiar with Brody?

I don't recall offhand, but I probably watched those clips that that you shared with me, but.

Okay, so Brody's job today is to convert Rob into being a diehard fan.

And if he can do that, if Rob goes away and says, yeah, I've got to listen to more of his stuff, then Brody will have succeeded.

Rob: Yeah, or I may not have even looked at the links that you shared with me. I, I, I don't know.

Jonathan: So I must admit that one's, that one's on me, the links for this show. I sent just a few minutes ago.

Like I said, it was kind of put together at the 11th hour. But that's all right.

Rob: I'm definitely a little less prepared than normal. You know, that 11th hour kind of thing, but

Jonathan: that's, that's all right. Like I said, I think, I think our show is going to is going to basically turn into just the three of us having a great time geeking about all things Linux.

I'm not going to faff about any longer. I'm going to bring him on and I want to say, Brody, welcome to the show. How's it going? Hey,

Brodie: it is good, it is good, it's good to have you. Yeah, it's a pleasure to be here. The answer to your question before was about four channels. There's the main channel, Brody Robson, podcast channel, Tech of a T, the gaming channel, Brody on Games, and then the react channel, which I haven't uploaded to in like, four weeks.

But the main two are the main two, the podcast and the main channel.

Jonathan: Okay. And so what, what is your, what's your niche? What do you, what do you really cover? What's your bread and butter of keeping people up to date on?

Brodie: Well, with the main channel content right now, the main focus, it's sort of shifted a bit because early on, I was doing a lot of videos on.

Like how to use window managers configuring window managers things like that Then eventually shifted into there's a period where I was doing videos on vim plugins. I don't know don't question that period Then it was more like plugins. Yes vim plugins. Yes Yes, vim plugins. There's there is a whole playlist of them.

I can show you the playlist afterwards I think i have a playlist whatever doesn't matter then I shifted more into doing like software reviewy stuff There's been some other things in between but nowadays it's more a lot of You Linux news stuff, getting people up to date on what's going on. One thing I do quite often, which I don't know why people watch, is I'll go through like issue trackers, like GitHub, GitHub, things at GitHub, GitLab, things like that mailing lists and sort of, I guess condense the information there down into a more consumable form, because you'll see these discussions going on like I don't know, any sort of Wayland Protocol discussion, it's 3, 000 comments long, and there's no easy way to get into that in a consumable way.

And you'll see articles from certain outlets and they'll take like one quote from the entire thread, but I want to make the I guess the full context there a bit easy to understand to really know what's going on behind the scenes in these projects.

Jonathan: Yeah, so your, your more recent stuff, is it fair to say that you're trying to help people see what it looks like when the sausage of the, the open source sausage, as it were, gets made?

Brodie: I guess that's a good way to put it. Yeah. Yeah. I feel like I do an okay ish job. I, I don't get why people watch it. But for some reason they do. Maybe it's something to do with the Australian accent and I sound different from the rest of the Linux YouTubers out there. It's just like a, Ooh, look at this funny Australian man.

Let's listen to him.

Jonathan: I think, I think part of it has to be sort of the you're, you're taking. You're taking the, you know, all of the comp, all of the issues that are out there and you're kind of shining some light on it and say, Hey, this is the really interesting one. This is the really juicy one. And so I, I, I kind of suspect at least in, and I would say in my case, this is a true white, I like to tune in it.

It saves me time of having to go and read through the entire Wayland or what have you, you know, tracker. And somebody else has done some of the work, and you're finding the ones that are interesting, the ones that I should know about. I think that's a service.

Brodie: Yeah, no, I think that's a good way to put it.

I I do There's been issues where maybe I focus on certain aspects and rather than focusing on the technical goings on, focus more on like the the drama part of it, but I guess it depends on what I'm sort of focusing on in that case and what I feel like sort of deserves the most attention. Because look, in some cases the actual technical discussion isn't even the most exciting part.

Like there'll be an argument about, hey, how do we position a window? Should we have a a window have the ability to set an icon on itself? And you're like, I don't know why they're arguing about this. And. The argument part of it, I think, especially in cases like Wayland, because I described Wayland as a meta project.

It's a project where you have all of these different desktops coming together that all have to work on this one solution, but a lot of them can't agree what color the sky is. So it becomes kind of a mess.

Jonathan: Yes.

Rob: Yeah. I was just going to say, you're already drawing my interest in your channel and information.

I I'm kind of the, I'm the guy in the group who likes to bring the drama as much as possible. So if I can find some more drama to absorb out there.

Brodie: Well, that's not always what I do. Like I'll also talk about you know, I, I'd still like to do some videos on like software pieces sometimes I had to do these historical videos on old bugs, for example, like I did a video a couple of days ago about the can't print on Tuesday bug, which is a bug from the days of Ubuntu 12.

04. Where due to a bug in the file application, PostScript files generated by open office could not be printed. Because the file application was detecting them as an Erlang Jam file, not a Postscript file. But only specifically when Choo was in the the created by date, or whatever the specific line was.

So it was this weird bug where it was completely unnoticeable unless you used OpenOffice, because the tag was optional. And Only happened on this one day, but no sane person is going to think, Wow, my printer is broken because it's Tuesday. Eventually they worked it out though. And that's, I think things like that, it's always cool to go back and look at how someone actually came to that conclusion and sort of break down that story.

Jonathan: Yeah, that one in particular, I watched that one too, sort of in prep for the show. I found it really fascinating because it was one of the people that was working on it, it was like his wife. It's like, Oh yeah. It's, it's the printer bug. It never works on Tuesdays. And he goes, wait a second.

Rob: Tuesday?

Jonathan: It never works on Tuesdays!

Finding out that it was like, you know, the, the, the, the, the file itself, when it was set to Tuesday, it just lined up that it matched the Magic Bytes for another file format. Yeah, it was, it was great. It was a great story. I, I do yeah, I, I enjoyed those too. I was, when I, when I listened to that, I was trying to think through some of the weird bugs that I've seen, and It's like, I know some of them were really weird, but I can't bring to mind any of the really good ones.

One of the fun ones was way back in the day with the OpenWrt project. You couldn't do Your first compile of it could not be multi threaded. You had to do a single threaded compile, and then after that, you could do multi threaded compiles, and they would work. And so I, you know, I sat down with that to try to figure out why that was, and with that, when it comes to find out that they end up building GCC multiple times, which is typical for cross compiling.

That's a thing that you have to do. Well, they were, when they went to download GCC, they were overwriting. So they would start, they would download, start, compile, and then they would download a second time and overwrite that first compile in the midst of compiling it. And once I discovered that, it's like, Oh, well, that's why it's broken.

Of course I do. I enjoy bugs like that. They're obscure and weird and have way different. Kind of root causes than what you would think they would have. That's funny. So, I, I want to ask you about, about Weyland and X11. I'm just kind of curious how do you, how do you think we're doing? So, listening to, listening to some of your, your comments.

your videos about it, one could come to the conclusion of, oh my goodness, it's terrible, the whole process is broken, and maybe we should just stick with x. And I, I sort of suspected that's not actually where you would come down on it. I want to get your thoughts.

Brodie: So there's a couple of pieces of the equation here.

If we are talking about, you are a developer, you don't know what a video game is, you don't know what fun is, your entire life exists around a code editor and a web browser. Fedora swapped default to Wayland back in 2016. Yes, 2016. Back then you couldn't screen capture. Basic things didn't work back then.

But for a developer, it was enough to dog food. Come 2020, 2021 ish, like during like COVID era, a lot of things started getting addressed. One of the things being screen capture, and a lot of people suddenly became able to actually use it. There's still applications that don't implement the solution because Discord and whatever.

There's also solutions in place like Global hotkeys, which is a big part of Obviously discord having pushed to talk OBS having the ability to use like your hotkeys when you're not focused on the window But also it's a big accessibility thing. I think accessibility is the area where Everyone is aware that this is a big deal and everybody is aware.

It is broken But even though it's so important, it's such like a small amount of users who need it It just doesn't get the love and attention it needs but that's an area where I think we are really lagging behind When it comes to the data they use though, I think especially now with the recent NVIDIA Explicit Sync drivers and the Explicit Sync patches in the compositors, which has fixed a really bad flicker issue.

Where, for some people, it was literally unusable, like they could not look at their screen if NVIDIA was running under Weyland. For those people, it's pretty much good now on those drivers, but you've got to wait for the drivers to become available outside of beta and all that sort of stuff. As for like, as for other things, like a lot of the issues we have now, I would say for the most part, are more like these weird Rough edges, right?

Like there's a lot of solutions are in place But then there's issues like the whole not being able to do multi window applications There's issues like not being able to set an icon for a window. You have Actually, well, this is a pretty big one that I've talked to plenty about is the fact that VR headsets don't work under Gnome because they still don't have DRM leasing implemented and And All of these are important cases, but I would say they're more like Issues as opposed to hard blockers now for most people So

Jonathan: I my solution for this so I've been telling people it's it's simple you just need to run Fedora Use an AMD video card And run Wayland and everything pretty much just works now.

That's not true. There are, there are bugs still. And there are things right now that are driving me crazy about that, that exact setup,

Rob: I do often see, I'm, I'm, I'm a pusher of Wayland these days. And I often see people saying things like Wayland's not ready. And I'm like, well, why not? And like, blah, blah, blah.

I'm my Nvidia car. I'm like, I think Nvidia is just not ready. This is accurate.

Brodie: I would say it does do like they have been working to improve things, but nvidia is not a gpu company They're an ai shovel company That's fair. That's fair

Jonathan: The the other thing about this that really drives me crazy, particularly the the most recent shake up in fedora You know with fedora 40 the kde guys came along and I said, we're just not going to support x11 anymore And 90 of everybody was like, oh, okay and you had like two developer two maintainers You Inside of Fedora and then a few other people around the fringes that just went nuts about that.

And what drove me the most crazy about that is they're like this thing doesn't work in Weyland and it's a showstopper and everybody myself included was like Where's the bug report? Did you ever tell anybody that this is broken? It's like, if you guys would, rather than throwing a fit over this, would actually dive in and make bug reports and write code and try to actually fix things, Weyland would be ready by now.

Ah, it drove me so nuts.

Brodie: I think in some of those cases, there actually are open bug reports. A lot of the stuff with One of the big threads I did see was regarding, like, art usage. A lot of the, I'm not a big art guy, I do have like a drawing tablet here, so, like, I use it just for editing thumbnails, but my understanding is that Katie's, like, implementation and configuration tool for the drawing tablets is just not at the level that people want it to be at, and from my understanding that stuff is reported, but, again, it's like the accessibility stuff where it's important for a small group of people, so, yeah.

But it's very, very low down the list of priorities. And usually, unless you need it yourself, People aren't usually working on it, which is you know, it's just the problem you have we have limited resources Yeah, right and I was I was

Jonathan: let me let me let me rob. I'll rob. I'm gonna let you finish. Oh, that's like the second time in two days.

I've used that meme and it makes me seem really old. Anyway, the one of the things in in fedora that one of the developers was complaining about is he had like this completely cursed setup where He had a laptop that was you know 1200 by 600 and a 1080p display. And he's like in X 11, I can just scale these so that they look the same, but I can't do that in, in Wayland.

And you know, everybody that reads it is like, first off, why would you want to do that? That's cursed. And secondly, have you, have you told, and you know, there was no bug report. He opened the bug report and within a couple of weeks it was, it was fixed. So that's, that's the sort of thing that drives me nuts.

It's like this, this is your big blocker. Really? Oh, anyway, Rob, go ahead.

Rob: I was just going to say, I think my big blocker when I was anti whaling and all X and was really due to like the, the screen capture and when it comes to things like a remote desktop, you know, being in the IT support industry, being able to remote into someone's desktop and do what you got to do.

And that just didn't work for the longest time. And, and after that got around, I was like, well, it all kind of seems to work for me. And after I got to dumping it. Yeah, that's

Brodie: one of the cases where it is like a super hard blocker. And I think a lot of people mistake hard blockers for minor inconveniences.

Right, like a hard blocker is your, you literally cannot capture your screen. You're doing some work where you need screen capture. A hard blocker is your GPU inflicts a seizure because your screen is constantly flickering. A hard blocker isn't. Window icons being broken or other little things like that.

Like it's Discord push to talk doesn't work things like that. Like yeah, it's annoying and sure maybe you'd prefer it to like work but It's not at that point a reason to Halt the entire development of this process.

Rob: Yeah, one could argue I could have just got around that with SSH, but that's a little tricky in some situations.

Jonathan: Well, that was that was the other thing, right? For the longest time with Wayland. It's like, oh, with SSH you can do SSH dash X, and because X11 is transparent to all of this stuff. Yeah, we're transparency. The fun thing about that is it's like, it's not though. The, the amount of bubble gum and duct tape under the hood to make that SSH call work was painful and X11 no longer had network transparency.

It was, it was sort of simulated. And there's also now way pipe that just works and does it.

Brodie: If people don't know how broken a lot of modern applications are with network transparency, my understanding is Fedora Firefox completely dies if you try to do network transparency at this point. I haven't tried it in a while, but that's, that's the, my recent understanding.

And I would not be surprised if a lot of other things just do not behave at all like you would expect. Because yeah, you're right. Like it's, network transparency is a thing that made sense when Well, the reason why it was there is because X11 was developed in a time where you weren't running your ex server and your ex client on the same hardware.

You actually did run a client server model. Now, we have the hardware to run it on the same system and that's very much just a holdover from the way it was developed. Originally developed because there's no reason to redesign it when that's just the way it always basically always worked

Jonathan: Well, I would I would go a step further than that and say there actually was a reason to redesign it and Wayland is to Redesign.

That's true

Rob: Well, there's a lot.

Jonathan: Yeah, that's the other thing with Wayland It's like all of the x11 developers are now working on Wayland and and and this is this is interesting I think One of the Fedora guys, that's also sort of a RHEL guy, was talking about this. Apparently, here in another few months, when, when the, the last RHEL release that is X11 only, goes to long term support, RHEL is apparently going to move Their engineers off of doing bug fixes for X and so it's, it's actually getting ready to be kind of a dicey thing to run X anymore.

I mean, it's always been a dicey thing, but yeah, that would

Brodie: be the end of row seven.

Jonathan: Yes, yes, it's going to be fun. Okay, so if we have, not that we have exhausted the Weyland topic, but there are, there are some other things to get into. How about gaming? I think, I know Rob and I are aficionados of gaming on Linux.

Is this a thing that you two, you do as well, Brody?

Brodie: Oh, absolutely. Yeah, I've said before that Whilst I started using Linux during a time where I wasn't gaming If I couldn't game on Linux, I probably wouldn't be daily driving at this point like yeah for me like yeah That's it's important use case Well

Jonathan: it's true, and you do some, you do some let's plays, don't you?

Brodie: Yeah, yeah, I have a stream channel. I will be live there in about 8 hours from now for anyone who's watching this live. I'll take a nap first though. I'm doing a co op stream of It Takes Two with a Mate of Mine. And the other stream I'm doing right now is Sekiro. So

Jonathan: when was the last time that you, you, you bought a new game and went to fire it up on Linux and had anything other than just a stellar experience?

Brodie: a new game? There was a game I played recently, which is a very edge case game where on Arch Linux the, this is actually an issue that every distro is basically fixed now because of this game and a couple of others. This game had an issue with memory maps, where the default number set by the kernel was just too low.

And the game would use all of them because it's designed terribly and just ruins your entire system. But it's the way it works, and it's the way it was designed on Windows. But changing that setting? It works. Besides that, I've had issues with older games where they use weird cutscene formats and cutscenes just don't play.

But besides that no, no. The only other issue I had was, it was just a core issue with my system and I had a driver issue and nothing was working properly. But the games itself, no, it's pretty much been good. I'm not a big multiplayer guy for the most part, so I don't deal with a lot of the anti cheat stuff.

Yeah,

Jonathan: it works well. Have you been, have you been around Linux and Linux gaming long enough to remember the, the bad old days? Where, you know, you would have to wait a couple of years after a title released for it to finally start working in wine?

Brodie: I started using Linux daily in 2019, which is a year after Proton.

And as I said, I wasn't gaming at the time anyway. So by the time I started gaming again, it was probably like 2020, 2021. So things were a lot smoother by then. All the

Rob: good old days there.

Jonathan: Yeah. You missed out on all of that fun. You know, So, for example, I used to play Lord of the Rings Online, LOTRO, and the hoops we would have to Somebody, somebody built a launcher in Python just to be able to run LOTRO on Wine, on Linux.

And, you know, every time they would update, you would have to wait for the guy that wrote the script to update his script to make things work again. And that was just normal. And those of us that were real hardcore Linux fans, we just, oh yeah, this is just the way the world is. And then Steam came along with Proton, and, and You know, especially getting ready for the Steam Deck, really started putting some effort into it.

And it's just, it's, it's amazing. It's such a different experience.

Rob: Yeah, unfortunately I go back to the times where it just wasn't really a thing with most games, but yeah, so whenever I do hear people complaining these days about, well, you can't game on Linux, it's like, well, I think you're a little outdated with your information or sometimes they just don't know where you're going and tick that box, run with Proton.

It's like, oh, yeah. Well, that's all I had to do. Yeah.

Brodie: Yeah. I was watching a lot from the outside back during like the steam machine era, for example, but yeah, I, I know the things definitely have gotten, I was like aware of wine. Cause wine's also a thing over on the MacOS side. And during that period I was a MacOS user, but as for like doing the whole gaming back then, yeah, no, I, I, I was a, during high school, Console and a bit of Windows.

That's for the most part.

Rob: So you are a Mac OS user, so you know all about games not working well on your system.

Brodie: Well, look, here's the thing, right? Games work well when the only thing you play is Halo 1 and some old Unreal Tournament.

Rob: Yeah, I happen to have a Mac too, and I just recall running into spots where this just doesn't even support it on here.

Yeah, I suppose I could.

Brodie: Especially now between architecture changes and massive OS rewrites. I imagine all that stuff is completely busted anyway. Yeah, I haven't even tried

Jonathan: that. Not to mention the fact that Mac is basically dropping support for OpenGL. I still don't understand that for a long time though.

Yes, yes. Oh, it's hilarious. You know, you have better OpenGL support. On as Sahi now than you do on Mac Os. Mm-Hmm. . Mm-Hmm. Mac. The, the Max's? Like, no, no. We just, on our, on our chip, we can't support the new OpenGL stuff. And these Sahi guys come along and we, we made it work. ,

Brodie: it's

Jonathan: crazy.

Brodie: They just wanna push people over to metal and all that sort of fancy stuff.

Jonathan: Oh, yeah, yeah. No, that's, that's exactly what it is. It's working great. So you said something about Arch, and I'm wondering if that was your, I run Arch by the way, moment. Is that your distro of choice?

Brodie: Yes, yes, I've been daily driving Arch. Since the day I swapped to Linux, it's not a good choice to do that.

I've got a video coming out soon about, like, my my, my origin story, we'll say. I've talked about it in the past before, but it was, like, back when my channel was way smaller. Yeah, literally the day I swapped Arch Linux. Now, when I swapped was a bad idea, because I was in the middle of classes. I did it in the third week of the semester.

Rob: Well, did you at least use the easy install script, or were you just all manual, do it easy install script?

Brodie: There was no easy install script

back then. I could have installed Endeavor. But no, no, my, my plan, I was, I never planned to be daily driving Linux, this is the problem. So I had a laptop, it's somewhere I don't want to grab it.

I had this laptop, it has two drive slots. I had a Windows install. And I put a second drive in and was going to install Arch on that. Now, I must have made a mistake with fdisk or something, and I accidentally deleted the Windows disk. So, I was like, you know what? We're here anyway. Let's just, let's just do it.

Now, I did back up my files, luckily. I didn't lose any, like, classwork or anything, but Yeah, the, the whole swapping to Linux fully at the time was very much not planned.

Jonathan: had an experience like that back when I first went to Linux and what would that have been early 2000s, I think yeah, somewhere around there, like 2000, I don't even know. Some early 2000, somewhere around there and running Windows XP at the time. And. My laptop, because, well, I know now it's because it was a spinning hard drive, you know, conventional hard drive.

But I would have to reinstall Windows XP about once a year to keep it running reasonably well. And I discovered Linux, I think it was dual booting at the time, and Windows XP popped up and said, You gotta go to your C drive. Careful, modifying files in here could damage your, you know, your system. Are you sure you want to continue?

And I just sat and I looked at that for a while and I'm like, I don't really want an operating system that treats me like this. That was my point where I'm like, okay, I'm, I'm done with this. I'm going to go, I'm going to windows for all or for Linux for all of my stuff. That was, yeah, I was. Back in the day when we still called it Fedora core.

Rob: I don't think I've ever made a mistake like that exactly other than Accidentally overwriting the bootloader or something like that. But I have you know, I was dual booting Windows and Linux for off and on for years. And not too many years ago, my windows drive died. I'm like, ah, well, that's all right.

I bought, I, I ordered a new drive with the intention of reinstalling windows, just so it'd be there. And I still have a brand new unused drive that I've never even bothered reinstalling windows or anything on. So I have a spare drive sitting around.

Jonathan: What what desktop environment are you in these days, Brody?

Brodie: Right now I'm in KDE. Not the long term solution. I'd swapped over with Plasma 6 because I wanted to just, you know, I hadn't daily driven KD before, so I want to see what it was like. I want to see, is the grass actually greener? Look, it's, it's greener in some places, but the rest of the grass is dead.

Before that I was on Hyperland and I've used a lot of, like I've used Awesome, WM, i3, Plasma 6. BSPWM, things like that. But my next endeavor is going to be Cosmic when System76 gets the alpha ready for that. Yes. So that's actually the direction I was going. Cosmic, Cosmic looks really interesting, doesn't it?

Yes, it does. Yes, it does. I like to bully Carl about the light theme and his choice of accent colors. But besides that, I I, I think it is really good. Yes.

Rob: Have you tried it out yet?

Brodie: I've not had a chance My, my entire opinion on it is based on what I've seen from the demos and all of that. I've not had a chance to actually try it yet.

I, I, what I should do is just, Chuck, I know there's like a couple of Fedora based distros that ship Cosmic Like, there's like a couple of third party things. I'll probably chuck them in a virtual machine, give it a proper shot. But my, like, I was kind of waiting for the alpha to be ready, because I know a lot of just basic things, you know, pre alpha, a lot of basic things are just going to be broken.

So, I'm not going to run it on my system until at least the alpha, at least until they feel like it's ready. To be shipping to the Pop! OS users.

Rob: Yeah, it was probably February or March. I did try it out and demoed it. It's, it's pretty easy if you install Pop! OS and then install it on there. It's pretty simple, but it definitely wasn't ready.

I, I could have gotten by. I mean, if, if I absolutely had to, that was just what I had. You could run it. You just don't have some settings there and you could do some things in, in command line, or even, even I, I tested found you could still start up the GNOME settings and stuff like that and worked halfway.

But yeah, it was it was a nice looking environment. Looked, it definitely looked promising.

Jonathan: Yeah. What, what interests me so much about Cosmic is that it's so, in some ways it's so cutting edge. And it's intended to be like a desktop environment for the regular user So, you know, you've you've got some of these that are new desktop environments Well, like hyperland for instance, I mean that is very cutting edge But it is aimed at a very specific type of user, right?

Whereas you've got your your kind of general use case Desktop environments gnome and kde Well those have All of the baggage of having existed for however many years they've been around. I think it's really interesting that, that Pop! OS is coming along and saying, Alright, we're going to build a desktop environment from scratch.

We're going to do it in a modern language, Rust. We're going to do it with the modern backend, Wayland. And so it's not going to have It's not going to have any x11 baggage connected to it. And I think that's, that's really fascinating. That's why a lot of us are kind of keeping our eyes on it, trying to see, you know, what can they come up with?

What can they do without having the baggage? And what new ideas can they come up with for kind of the general populace? And that's, what's exciting about it.

Brodie: Yeah, I think the, what, what I really like about Cosmic is they aren't designing a desktop in a vacuum. They aren't saying, okay, we're going to make a desktop environment and just pretend like the last.

25 years of, maybe even longer, 27? Whatever. Probably, yeah, about 27 years of, like, genomic ADE have just never happened. Things that they've resolved, that they've resolved years ago, Cosmos is like, okay, yeah, good solution. Let's do that. Other things where maybe KDE is lagging behind on something, like, okay, well, people have had this, like, one of the things is the way virtual desktops work.

So on, on K KDE, it's lagging behind. If you swap your virtual desktop, it changes the virtual desktop on all of your monitors. On a window manager, on something like i3, it'll change it on the individual monitor. This has been open on KDE as like a bug for about, I think since KDE Yes, it's KDE 3, so it's a bit of a bit of an old one and the Cosmic guys like, okay Let's just do that.

Let's just make it work. And you're right like it's it's getting rid of a lot of that baggage The one of the things I do think is gonna be interesting with it is with them using Rust It's not them just using new toolkits. It's They are the reason there is GUI toolkits, really. Because there were GUI toolkits for Rust before, but they were like, fairly Fairly new, because Rust is a really new language, and GUI Rust is even newer than that, and if you look at ICED, They are the top contributors, like pretty much all the top contributors for ICED now are System76 people.

Like, they are really trying to make this happen, and Yeah. ICED is good, but it has a lot of, well, the documentation's rough,

Speaker 4: very,

Jonathan: very

Brodie: rough.

Jonathan: Well, that's, but, yeah. That's fairly, that's fairly common for when a tool like that is under heavy development. It seems like documentation tends to lag behind. And I guess one of the, one of the measures of how well System76 is doing is whether they spend the time and resources to catch the documentation up as they go along.

Rob: Oh.

Jonathan: Let's see if they do. Yeah, there is

Rob: a There was a lot of concern, I know, or a lot of, a lot of doubt when system 76 went into making their own distribution and nobody believed that that was going to be worth their time or was going to go anywhere. And I mean, Pop! OS, I'm not a user of it, but it did turn out to be a really good distribution.

So, you know, I'm not really doubting them with Cosmic as as I and many other people did when Pop! OS came out.

Brodie: System76 is a weird oddball in the Linux space because obviously Ubuntu is backed by Canonical and they put a lot of money into server development, but System76 is a company who, their entire business model revolves around Desktop Linux.

They sell desktop computers. They still sell, like, server hardware as well. Sure, there's that as well, but their main their main product skew is their laptops, and there's, desktops as well matter, but the main thing's the laptops, and that means you have a company that has a vested interest in making sure their desktop experience is as good as possible, and They just sort of exist as an outlier.

There's no other entities that do both of these things, both be a hardware seller and also make an operating system and make a desktop or make a distro, make a desktop as well.

Rob: Yeah. Unless you're Apple.

Brodie: Well, yeah,

Jonathan: I mean, in a Linux

Brodie: space.

Jonathan: Yeah. You could, you could almost make an argument that that's what Sun did on the server side of things, but yeah, that's, that is interesting.

All right, so there's one other thing I wanted to make sure and cover, and then we'll kind of go into more freeform. But I want to ask you about this. Are you, are you watching the mess around Windows Recall? And what are your thoughts on what that is going to mean for Linux?

Brodie: Well, I did put a video out yesterday.

I don't know what, what I, I tapped into with this video. Cause this thing is doing, it is doing the rounds. Right now it is at 75, 000 views. Which is, Over double my next top video. My views are like over the last 48 hour period are triple what they normally are. I tapped into something. Yeah. I think, I think recall is, it's, it's one of those things where it feels like the engineers at Microsoft sat in a circle and they're like, Hey, we have this co pilot thing.

Let's. Let's go, let's do something really cool with it. But they never spoke to anyone outside of the circle. They didn't ask, like, they didn't do any, any testing with like regular people. They didn't bring anyone in and be like, Hey, is this a good idea? They didn't speak to any of their like networking guys, their security guys.

It just seems like they wanted to do something really cool. And then it turns out there's a lot of really bad people. really bad decision made with it. Like the whole, it's a SQLite database and it's encrypted when you're not logged in, which is not a problem. We don't care about that. The problem is the fact that you can just exfiltrate the entire SQLite database

Jonathan: with basically no issues.

I saw a story Artist Technica did that apparently you can log in as another user and get into the database too. So like the whole, the whole thing is, is on a, there, there are, there are broken things about their security model, but I think most people don't care about that. Like most people don't know what a SQL lite database is, but the idea that their computer is intentionally taking screenshots of everything they do every five seconds, I think that kind of taps into like.

the creepy factor. Like that's a, it's a creepypasta that your computer is watching you and I've seen, I've seen people that are not Linux users that, that are just creeped out by that. And a lot of them are starting to look at Linux as an alternative, which I think is really interesting. I don't know, I don't know how much traction that's going to get, because it's easy to say, Oh, I'm not going to run Windows anymore.

And then, you know, you try to install Linux and for some people you, you hit the learning curve wall. And not everybody is willing to spend the time to get over that. Or some people just have weird bugs that they hit. So let me ask you this Do you think linux is ready for an influx of people coming from?

Windows

Brodie: It's funny you say linux influx of people so I know someone who is involved in Like device sales and device integration. They had a company just cancel a big order of windows devices and They're now swapping all over to apple. So it's not going to be entirely swapping to linux There's a lot of cases where people are like, hey, let's let's try out this whole apple thing Assuming they have enough money or want to take out a loan.

But for the regular people I it's weird right because You If you are seeing people discuss something online, those are already the, like, 1 percent technical people. Even if they're in, like, fairly non technical spaces, if you are discussing something on Reddit, you are already, like, a very small group of people that even know what an operating system is in the first place and know that you can change it.

Because most people, they don't change operating systems, they change computers. If you know someone who has a 10 year old computer, they're likely still running whatever operating system that device shipped with, because they That's just how it goes. Yes. I think, I think this is, this is a really tough one, right?

Because, yes, if you're, if you're a technical person, you can absolutely do it. Like, we've all done this. And if you're swapping to something like PopOS, Ubuntu, Fedora, you can do it. But even then, I'm still seeing people saying, like, I installed, And I, I, everything broke and they just don't know why because the whole like the whole idea of troubleshooting isn't really a necessary skill for using a computer at this point.

Everything's been so streamlined that You can kind of just, like, go your way through everything and without any sort of issues along the way. And then when something does pop up, like trying to use Linux, you just don't have that skill set or that, that knowledge base to know how to even go about trying to resolve the issue.

I, I think a lot of people are going to try it. At least those, especially in the gaming space, because a lot of people in the gaming space have Some knowledge of like, a lot of people have built computers, right? They've probably installed Windows once or twice before. I think they might try it. Whether people will stick around is another question.

And, I guess, keep an eye on the Steam Hardware Surveys because that's the only numbers we have on Linux users apparently.

Jonathan: Yeah. Well it's, it's it's, it's increasing. Let's see, we, what did the latest Veronix article say about that? We blew past the 2 or 3 percent mark, something like that. 2, I think we're like at 2.

Rob: 3 I think or something. Yeah, but it's

Jonathan: more than, it's more than Mac has, right?

Rob: Yes. Yeah, it's it's been it's been above Mac Mac on the steam survey since last summer.

Brodie: It's sad that it took that long though. Yes,

Jonathan: yes. So, I think I saw another one of your videos talking about the distros that we should recommend to new users.

And you kind of touched on it now, but this is something I see too and it drives me nuts. These niche distros, which I don't, okay, I don't have a problem with somebody doing a niche distro, like if there's an idea that they just want to go in for and, you know, repackage Ubuntu using this weird desktop environment that nobody's heard of, it's like, that's fine, if that's your toy project and you want to work on that, that's fine.

Everyone distro. In fact, there needs to be a very small number of people running your distro because there are going to be problems with some one person packaging all that stuff. Like, there's just going to be problems with it. And you particularly should not be telling new users to run this niche distro.

What, so, to turn this into a question, what do you think would be the best distro for a new Linux user to try?

Brodie: That's, that's, that's tough, because I think, Fedora I think we have to take off the table just because Fedora has such a focus on free software. And that's good for the people that know about it, but having to add this additional repo in just to be able to access, you know, proprietary things you want to access, it's, it's just this weird extra hurdle.

Hmm. Honestly, like, the tried and true answer is just Ubuntu. Like, I know a lot of people say Ubuntu, but realistically, it's fine. Yeah, I know, Snaps, whatever. But, for someone who's new, Ubuntu is perfectly reasonable. There are issues when it comes to Ubuntu having slightly older software, especially slightly older drivers.

And if you're on, you know, Ubuntu, the newest of newest GPUs, for example, that can be an issue for sure. But

it's probably the best option. Pop OS, you know, it's, it's also a good option if you are more in that, that gaming sort of focus. I think both of those are pretty reasonable choices. If you're going outside of that, like, you know, Debian, Debian's Debian, especially, you know, if you're anywhere near. The end of a Debian cycle, things are getting really, really out of date and it's become a problem.

Anything Arch based is just I think the only time something Arch based makes sense is when Valve has specifically, you know, they've made a thing. It's not really Arch. The Steam Deck's operating system is not Arch. It's a, it's a specific image based on a specific set of packages, treat it more like its own separate thing.

And I don't think there's really any Arch based distro that You can recommend, because Arch just has, it's rolling release, no matter what you try to do besides doing that, and it's just, that comes with too many issues, I just don't think it's worth trying to put someone on it. I think, just, go with a tried and true answer, go with Ubuntu, go with PopOS, look, go with Mint, it's, it's fine.

Like, look, Mint is a classic answer, but there's a reason it's a classic answer, it's a good answer.

Jonathan: Yeah.

Rob: I hear so many people recommend Mint all the time, and, It's not me, but I mean A lot of people

Brodie: also say Zoran, if I think the issue with going with something with like Zoran or like Mint is it's too like Windows and that might trick people into treating it like Windows and then getting confused when things are not in the place where they expect it to be in.

Rob: Half the time my answer is, if you're leaving Windows, don't you want something different? It's just so much like Windows. And, and then they're always like, well, you can configure it any way you want. I'm like, well, why not go to a good starting point? But there's, there's nothing wrong with Mint. I argue against it all the time because that's just the kind of person I am.

But in all honesty, I think Mint and Zorn are mostly fine distributions.

Jonathan: I put somebody on Mint for their first Linux computer back a few years ago. And then I came along a couple of years afterwards and it's like, okay, so let's update this. How do I, how do I update this thing? And of course, you know, it had gotten way out of date.

And I'm not a Mint user, and so I had to sit there on that machine for quite a while to try to figure out like how do I get them to the next release of Mint? I don't think I ever managed to do it. I think I reinstalled it instead. I come to find out, I don't think Mint has an upgrade path to go from one to the other.

So that, that kind of burned me out on putting new people on Mint, because you kind of want them to be able to upgrade. I'm, I'm sure there's a way to do it, but like there wasn't an, an easy, there wasn't an easy way to do it. And it wasn't my machine, so I didn't want to spend hours and hours and hours fighting

Rob: it.

I thought you could use the same Ubuntu, the disk release upgrade, or whatever it is, to Do whatever it is, do release upgrade. Or the fact that you have to have this

Brodie: discussion about how to do it already means there's some issues with sure.

Rob: I mean, to be, to be, maybe not to be fair, but I mean, Zorin literally did not have a upgrade path until, was it a year?

Maybe it's been two years already, but I remember, I remember the announcement when they finally put that in there, that. And I think it was just beta then, I don't even know where it's at now, but So, I know you literally couldn't for that.

Jonathan: Yep, yep. And before all the Mint people come and tell me, Here's how you do it, I'm pretty sure that was actually LMDE.

The Linux Mint Debian Edition. And so that's even yet another step removed from mainstream Mint. It was, it was, I would not put them on the same distro now. Looking back, I would probably go with Fedora. Probably Ubuntu. Honestly, because this was, this was back before Pop! OS was really good.

Rob: Yeah, I don't, I don't use Ubuntu on my desktop, but just like you Brody, that's, that's the one I recommend to people all the time.

Mainly I say, stick with the mainstream, you know, Ubuntu, Ubuntu mainly, but then, you know, if you want to go out there, Debian or Fedora, but you know, there's these, you know, things like you mentioned that that are good. I

Brodie: think once you're familiar with stuff, you can always branch out, but yeah, the first step should just be something where you know it's fine, like, it's not great, it's not perfect, it's just, it's just fine.

Not a ton. I do think the concept is really cool, and I have said if I was going to set up, like, I want to set up a capture PC at some point, I really should, because my capture setup is really not good, but if I was going to do a capture PC, I probably would just chuck something atomic on it. I think nowadays the idea of Atomic Distros has gotten a lot easier thanks to the existence of Distrobox.

One of the issues that Atomic Distros had early on is just software availability, and if you don't, if you weren't gonna have a distro that had all the software available, you'd have to have a bunch of layered packages, and layered packages slow down your update process massively. Nowadays a lot more things are available on Flathub, available as snaps, available as app images, and it is a lot easier, but At the end of the day, there's going to be some things that are missing.

And distro box has massively smoothed out that process.

Jonathan: So speaking of, do you, do you have, we're just going to ask about all of the holy war topics. Do you have an opinion on the snap flat pack and app image of war that we are currently in?

Brodie: I Refuse to run snaps on my system. I do not want them anywhere near my device.

Speaker 4: Okay

Brodie: My honestly the main reason it's it's people might argue. It's a dumb reason. It's snap just Constantly spamming your device your your drive with loopback devices. I don't know why That's the solution they've gone with. I don't, if I've run LSBLK, I want to see my drives and my petitions. I don't want to see all of this other nonsense.

App images, I, app images are a weird one because they're very much up to the developer. They had an issue a while back where they basically got deprecated on like major distros because they were running a, or they were relying on an old version of the, uh, user space file system thing that I'm forgetting the name of Like squash FS

Jonathan: or fuse

Brodie: or Fuse, yes.

They're relying on Fuse 2, and Ubuntu dropped that and moved up to Fuse 3. So it broke a bunch of app images, but as a, as a bit of technology, I think app images are fine. I think the issue with app images is the fact that they are treated as stand alone content. binaries. So unless it has an update process built in, there's no way to update it without just going and redownloading it.

There is a application I do have that does have a built in updater. It just, what it does is just up, it downloads a new version and then just deletes itself. It's like, here you go, here's the new version. I mean, but most things don't do that. Most it's you have to go and manually update it. Flatpaks, I think have I like Flatpaks.

I do. I think they have issues, especially when it comes to launch times. I've noticed a lot of, a lot of applications take a bit of time to deal with stuff. And when we're dealing with both Flatpaks and Snaps, there's obviously going to be issues with sandboxing and, you know, Yeah, that's especially if they're third party solutions and the developers haven't properly addressed those So yep, it can be very hit and miss.

Jonathan: I must admit I've not played with snaps at all yet Not on one of those desktops You got pop Well, yes, okay, that's true. I have pop. I'm probably running a snap somewhere in here and don't even realize it The thing that fascinates me about snaps though. Is it they can run? server side processes Whereas you know with with with flat packs, that's all all gooey stuff and Ubuntu claims it was snaps Well, no, you can do you can run your daemons as snaps too.

And it's like that's that's a fascinating idea I could see something coming of that. I could see that eventually being a killer feature maps

Brodie: Printing Cups is a snap, or it's going to be a snap.

Jonathan: That's interesting. That might work out really well for them.

Rob: And Pop! OS doesn't have snaps by default. But, you can run snaps anywhere.

You can run it on your Fedora or anything. You could. So that's the

Jonathan: thing about snaps, and we've, Rob, you know this, we've talked about this. There's actually a kernel patch to make snaps work right. So to have the, the sandboxing that snaps are supposed to have, there's a kernel patch that you need. Well, you get it automatically in Ubuntu, but if you run another distro, you do not have that kernel patch.

And so you can, yes, you can run snaps there, but they are running in a sort of crippled way where they can get to your system in ways that they're not supposed to be able to, and the, the. The thing about this that drives me nuts is apparently Ubuntu is no longer working on upstreaming that kernel patch.

It's like Why do you want this thing to work everywhere or not?

All right Brody, ARM. I'm curious Have you played around with ARM and RISC and all of those alternative architectures? MIPS even?

Brodie: I have a phone

Jonathan: running Linux. No, I'm,

Brodie: I'm pretty, I I'm pretty much just X 86 on everything I do. I, I really would like to go and mess around with some other stuff. And there's so many things I would like to do, right.

It's just a matter of sitting down, having time to do things. But then I'm like, Hey, I've got these other things I want to focus on. And You know, every project is just a matter of time. Yeah. Right? And there's only so much time in the day.

Jonathan: Raspberry Pi? You know, that's a For some of us, that's a big part of the Linux ecosystem now.

Is that a thing that you've messed with yet? Do you have a Raspberry Pi? No, again,

Brodie: that's another thing, like, I've really wanted to get into, like, doing basic, like, electronic programming stuff and just It's one of these things where I would love to sit down and do it. It's just Time! Yep. Yep. Nope. I'm

Rob: interested to see how the Snapdragon X Elite performs and, and getting Linux on that.

Brodie: Yeah, I'm curious to see what people end up doing with those devices and what I go how it's gonna really fit in there What what what sort of use cases people are gonna have or if it's just gonna be like hey Now you just have like dedicated arm chip on this device and I don't know run Android with it or something.

I don't know come with a solution

Jonathan: Well, so that's, that's the interesting thing. You look at the, the new Mac laptops, the M1s, the M2s, we've got with Asahi, we've got now a great version of Linux on them. The thing about it is you have to jump through the hoops to be able to do the install on it. And that's something I'm curious about how it's going to work with the, with the Snapdragon laptops when they come out, because ARM is Historically arm has been difficult to boot Linux on like it's been, it's been fiddly.

So either you have to go in and fiddle with U Boot, which I've spent the last 48 hours fighting with U Boot and I just You you, okay. U Boot is a great project. Thank you to the guy that wrote U Boot. It's great. It makes things work. I hate messing with you. Boot on X 86. You have, well it used to be, you know, your bio, so now you've got UEFI and like most of the time now Linux just works.

Pushed your flash drive in or your DVD or whatever and it said, Oh, hey, do you want to boot from this? And you say yes, and it just works. It's great. On an ARM, historically, you've had to fiddle with it for hours and hours to finally get it to work. And then you do an update and it breaks again. You gotta fi I'm a little jaded about running Linux on ARM just because of that.

Like the Raspberry Pi, they've got it down. They figured out how to do it. But so many of these other devices, it's so fiddly to get it to work. It's such a pain.

Rob: What I'm, what I'm reading here is that the Snap, Snapdragon X Elite supports UEFI based boots, so

Jonathan: And this, this is the solution that is hopefully going to rescue everyone, and that is that they have started doing UEFI on ARM.

And if, if If more companies would move to that, you know, more of these devices, they'd write the firmware to do it. Things would begin to just work on ARM like they do on x86. I have some hope but on the other hand, companies are very slow to upstream. Their bits into the kernel. And so, you know, you, you'll have something based on a rock chip device or a snapdragon device, and it's like, well, yeah, there's patches out there to make it work with the kernel.

You can either use the experimental patches that aren't good enough to be upstreamed yet, or you can use this five year old vendor kernel that has all of their patches on top of it. It's like, neither of these are good answers. Neither of these are the way to go. I don't like, I don't like A or B. Is there, is there a door number C?

Brodie: I know that there's some. Stuff that Intel's working on as well. And Intel is generally pretty good at getting things upstreamed. So I don't know, maybe, maybe something will happen there. Yeah. So Intel

Jonathan: is known when they want something upstream and someone else is working on it, they're known to just buy the company that's working on it to be able to get it upstreamed.

Like the, the, the real time patches. Apparently someone at Intel said, we really need to get more of this upstream. So they just bought the company and it's like, we'll do it this way. Okay, so, what if you had a, we're getting close to the end of the show, so, I'll ask you some wrap up questions. If you had a magic wand and you could just change something about Linux for the Linux desktop, what, what do you think, what would you change?

Brodie: Do I want people to get angry at me?

Jonathan: Yes. It's fine.

Brodie: I would like to instill a BDFL in the Weyland project. I think they need some leadership.

Jonathan: Benevolent dictator for life inside Weyland. I am, I am, I am in agreement with this. Weyland is too much designed by committee and they need somebody in charge of it. I agree

Brodie: with that. It is the dictionary definition of designed by committee. They literally have a committee of people who vote on changes.

I would, I would, I'm

Jonathan: in agreement with this. I'm, I'm for it. That would take a magic wand, though, to be able to pull off. It definitely would. Fork

Rob: it. Start over. Well, yeah, that's good. Well, not over, but.

Jonathan: That's really, that's really the way to, to, to fix that. And it, it's funny because other projects have had this experience.

Where someone forks it, a lot of people move over to the fork, which in some cases gets run differently, and then the fork is so successful that finally the guys running the original project are like, you know what, let's just let the fork be the project. So GCC had this happen way back in the day. OpenWRT had this happen, and both of those projects, big, big successful projects, both had these forks that the forks did so well that they kind of got folded back in with, with some of their changes becoming, you know, official in the way the project was run.

So yeah, maybe, maybe that's the way to do it. Maybe somebody, maybe we need to find a benevolent dictator and fork Weyland and give the fork of Weyland over to the, the BDFL.

Brodie: I don't know who would do it, cause like, there's a That's the problem, cause I A lot of the people I don't think are in that position.

There are a couple of people I think Could do it, it's just, I don't think they want to waste their entire life away doing that.

Jonathan: Yes, that would, that is indeed a life position. It, it works for the Colonel, because the Colonel has critical mass. And, you know, we, there's now enough money tied up in the Colonel, that you can give Linus Torvalds and and Greg You could pay them, like there's enough money in the ecosystem now that you can, you can give them a comfortable salary to be Essentially benevolent dictators for life.

I don't know that there's that much money in Weyland yet.

We'll see. That's an interesting answer though. I like that. I like that. Rob, anything you want to get in before I ask our final questions and we look to wrap?

Rob: I don't think so.

Jonathan: Okay. All right. I'll pass.

Rob: All

Jonathan: right. So Brody, let me ask you these final two questions before we let you go. What is your favorite text editor and scripting language?

Brodie: Text, terminal based text editor, Envim. But when I'm doing a lot of programming stuff recently, I've been using VS Codium. As for scripting language, I am Python. Thanks again.

Jonathan: Okay, yeah All right. Is there anything that we we didn't cover? We didn't ask you about that you want to let folks know?

Anything you want to plug?

Brodie: Anything I want to plug? Yeah, go check out the, I did, at least you got something, another point where I can save my links? Is this where I'm gonna save my, where people can find me?

Jonathan: Yeah,

Brodie: okay. The The main channel is Brody Robertson. The podcast channel, I upload there once a week with the actual episode and the rest of the days are clips.

That is tech over tea. The gaming channel is Brody on Games. I stream there twice a week, both on YouTube and Twitch. And I guess there is the react channel. I don't even remember what it's called. Brody Robertson reacts, I think. Where I don't upload stuff. So, whatever. But they'll, you guys have links, so whatever needs to be done with those.

Jonathan: Yes, we will make sure and get the links in the show notes. All right. It has been, it has been fun, man. I appreciate you coming. We'll have to have you back when you know, something really interesting happens in Wayland. We'll have you back and talk about it. Absolute pleasure to do this. Awesome. Thanks so much.

All right, Rob, have we turned you into a fan?

Rob: His opinions definitely seem to align with about everything I say and think. So I'm definitely got to check out his content.

Jonathan: So here, when you get into one of your famous discussion debates on some weird Facebook group, you just, you have a YouTuber. Like, look, I'm obviously right because Brody agrees with me.

Here's the link to his video saying the same thing. I mean, it's It's just great. It's a tool.

Rob: Right. And I don't really watch games on YouTube, but I'll have to check out if any of my kids are familiar with them at all.

Jonathan: Yeah. Because I watch a

Rob: lot.

Jonathan: I, I don't need any more Let's Players to follow. I have too many of those already from, you know, being on YouTube for years and years.

But I, I definitely appreciate the the, the Linux commentary and the Wayland stuff. And it was, it was fun to talk to him. I enjoyed it very much. All right. Rob, is there anything you want to plug before we let everybody go?

Rob: You guys can come find me, connect with me. That is at Robert P Campbell.

com and on there, you can find links to my LinkedIn mastodon, or as many of you may know, you can come catch me on the untitled Linux show with Jonathan, which I haven't been on for a couple of weeks, but I'll be back.

Jonathan: You need to be back. Goodness, we've had to do it with three people instead of four because you've been out and David's been out.

Yep, Okay, cool. And that is, of course, on the Twit network. You can find us over there at Twit, twit. tv. Things I want to plug, of course, we've got Hackaday, my work there. So we've got the security column goes live every Friday morning. And of course, Floss Weekly is now part of Hackaday. Records every Wednesday.

Contemplating moving to Tuesdays. We'll see if anything actually happens with that. But boy, that would, that would free my schedule up a lot to not be working 24 hours a day almost on Twit. to, on Wednesdays and Thursdays. So we will see if that happens. You can find my stuff. I do have a YouTube channel where I occasionally upload things, usually around either how to fix stuff on Linux or around the Meshtastic project.

But yeah, find, find our stuff there, follow us, and we will see you next week on Floss Weekly.

Episode 785 transcript

29 maj 2024 | min

FLOSS-785

Jonathan: This is Floss Weekly, episode 785, recorded Wednesday, May 29th. Designing GUIs and building instruments with EEZ. This week we chat with Dennis and Goran, two of the brains behind the EEZ ecosystem of projects. That started out with a DIY programmable power supply that quickly became a modular test infrastructure.

And more recently, they found success with the EZStudio. And that's a project for designing GUI and workflow for embedded applications. It's a lot of fun. You don't want to miss it. So stay tuned.

Hey, it is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today we're going to be talking about some software and some hardware. This is actually one of those interesting projects where the hardware came from. First and the need for some software sort of borne out the the, the software project that went with it.

So that's gonna be really interesting. We're gonna talk with EEZ, the folks behind EEZ, the EEZ studio. I'm gonna ask them if they ever get tired of saying EEZ like that all the time. Maybe they just call it EZ. Maybe that's the intent. We'll find out here in just a second. I am, I am flying solo today.

We had sort of a crossed wires with who was supposed to co host, but that's okay. We've done it this way before. So our guests today, let's see, we've got Dennis and Goran, and I believe that Dennis is the guy behind EEZ and then Goran is One of the one of the high profile power users, I was told, and so they're going to give me their perspective on what's going on here.

Now, just a note, we lost a tiny bit of audio at the beginning of Dennis answering this question. And the context that you missed is he's talking about designing and building the very first programmable power supply, the EEZ H24004. which is a mouthful of a name and he talks about that. So that is what he's talking about and the rest of the interview goes great.

I'm not going to waste any more time and going ahead, going to go ahead and bring them on. So Dennis and Gora and first off, welcome to the show. Let me start with, I think with Dennis because this is originally kind of your brainchild, right? Give us the, give us that first story. Like what problem were you trying to solve?

And you kind of put some hardware together. How did, how did all of that happen?

Dennis: And okay, I started with Mission Impossible and the company's name Anvox and that EZ comes from Anvox experimental zone. So actually I initiate a experimental zone within our company. Okay. Company. There is basically two of us my colleague Martin, who is a real guru in all type of software and product development.

Development software development stuff. And I actually tried to, from other side to, let's say, to, to see what I can do on the hardware part. Actually I did something with electronics many years ago before I started with that adventure. And okay. Step by step I start to, to put something together.

There is there was a lots of I don't know. Wondering around what could be what should be done in that about that project and from almost from the very beginning, I actually decide to To start to post my adventure on the couple of sites, one of them it's also Hackaday and another prominent it's that Eevee blog from a day from Australia and communities, community starts to gathering about the project They also helped me to somehow to redefine some ideas to just to redirect me from some dead end and stuff like that.

And again, step by step, I start to, it starts to materialize something what looks okay, usable or functional. Okay, anyway, I started to do that to, to really To really serve all my wishes and the requirements in that time. And even beyond that, even something what I, I didn't, I cannot imagine in the time.

So I really tried to do something. What is a. Flexible and programmable and when we talk about programmability in I think in first place about that connectivity with the PC and so some, some kind of remote control so that you can take control over the unit, the device and remotely program that from your From your PC.

So something what is what has happened, uh, during that journey is that in one time I decided to put a touch screen in something what in that time was really almost like a heresy. To to not to to not have a encoder and that all many of that knobs on yours so that to to to looks and feel much, let's say, professional or whatever, but OK.

Again, since I did it for, for, for my for myself, I say, okay, let's, let's try to do something with touchscreen. Anyway, in that time already touchscreen made a great premiere or entry in all that mobile phones. And so I didn't see a really a problem with why not to put a touchscreen on something like.

Power supply. Okay, and the community responded very well about that and they encouraged me to launch even the crowdfunding campaign. What we really did on the crowd supply. I don't know if you if you know about that, it's not like a Kickstarter or Indiegogo, but for let's say like for open source projects, it's probably the, the, the first place where one should go.

Yeah. Else to say? The, I was surprised with the, with the, with the result of the crowd funding campaign. It was, at least for me, it was a great success. We actually got almost three, three times more funds than what we initially asked for. And, we also successfully fulfill the campaign.

And that is the first part of the hardware story. Immediately after that, actually starts to think about what could be what could be done in a better way of what is. Done in that first project. The first project was okay. The, the, the name is okay. A little bit awkward, like age 24, 0 0 5.

So it's age for the hybrid 22. It's like two channel 40, it's 40 volts and zero five it's up to five amp. So, okay. Makes some sense, but it's it's not so easy to pronounce or to, to remember. Anyway, something what's interesting about that project is that it was built around the Arduino Due as a digital controller.

In that time it was the only 32 bit board with 32 bit MCU. All other was just 8 bit and we actually we started first with Leonardo and Mega and we actually went beyond Possibility of that's a small MCUs and end up with the, let's say, like powerhouse in that way in that time or from Arduino due and but even that we actually pushed to the extent.

So there was a. The memory usage and everything, it was really yeah, we push it to the to the limit. And that was one of the reason why actually I start to think about a new platform, a new, new unit. Okay first to have, let's say, a better digital controller and to put more modular, actually to put more modularity in the And in that in that way we end up with something which is not necessarily even a problem.

power supply, but it's more like I want to call it test and measurement modular chassis so that you, you can put a, up to three different modules, which could be like power source, like converter, DA converter some digital input, output tunnel or stuff like that. So idea is to have and to put all the time, a different type of functionality in the same modular and really nice and compact, uh, enclosure.

Okay. And, uh, yeah when it came to the, that part of the digital controller first I, I tried to do something with that X MOS MCU. It's, it's, it's pretty, I wouldn't say esoteric type of MCU. It's not like a general purpose from SD or from, I don't know, from Microchip or some other guys, but it's something actually in between MCU classical microcontroller and FPGA.

And we didn't try to go into the in the FPGA because it was really beyond our I don't know, capability. And I actually think that with that ExMOS we'll probably can find the middle ground, let's say and that we end up even with some working prototype like evaluation board for that microcontroller.

But after some time, we actually stuck on the, on the, on the software on the software part, because The software support for that that, that type of microcontroller, it's, it's very limited that it's, it was mostly that company was mostly focused to high end Audio like high fidelity streaming and like video conferencing real time with low jitter, low latency and stuff like that.

So yeah, I spend maybe, I don't know, For half a year doing that XMO stuff and eventually gave up. And continue with st actually St. Cortex M seven. In that time age seven wasn't developed, but like M seven was okay, again, the, the, the best in that cortex m series from, from SD.

And we also, because we needed some, let's say, universal interface to communicate between that hardware, that digital controller and that peripherals. That was that was ended up in something what I baptized like DIB, like DIY instrument bus. So it's like a simple bus to communicate over the SPI with your, with your peripheral models.

Yeah. So, and again we put that in a nice enclosure. We have now more powerful MCU microcontroller. We put even the now even bigger touch screen and again community ask for another crowd funding. And we did it once again. And yeah, this is a more or less. A short story of a harder part of the project.

Jonathan: So is that, is that hardware platform, is that still a thing? Like, can people go out and buy it somewhere? Did the development ever get finished on it?

Dennis: Unfortunately, no. What's happened

We have prepared everything to after that after fulfillment of okay. I forgot to mention what's the name of the second project? It's a BB tree like bench box tree. It could be even a blue box tree because it's painted in blue, but okay. It's a. It's acronym for bench Box three. Three means three three models.

So, okay. That's the, and it's probably, it's easier to remember that BB three than H 2 4 0 5. Yeah, yeah, yeah, yeah. So yes we actually prepare everything to to go with like cereal production and then we. And then that oh yeah, COVID stuff came and everything was just broken.

All the supply chains lack of many some of, of the components finally was starts to be available again to the, to the end of next previous year. So there was a really a long. Opposed in and it was really, yeah. Understood. Yeah. It, it was, it was impossible to, to, to do anything in, in, in, in that sense.

Yeah. We have some, we started to, to think about restore, mm-Hmm. production from our side. But in the meantime as we learned many people because. It's it's really hardcore open source. That means it's open to, to the, the, the, to, to the, the

Jonathan: hardware and everything is open, right?

Dennis: Yes.

Jonathan: Yeah.

Dennis: Yes, yes, yes. To to to to, to the extent to, to the biggest extent. So you, you can just go to GitHub and get all what you need included how to, to build mm-Hmm, that enclosure, and, all that wire harness harness for connecting everything and stuff like that. And some people in the meantime successfully assembled the unit.

And not just like just that. Some of them even start to to build their own which is a compatible with, with our harder. So that was actually one, yeah, that was one of idea to, to, to have something open and able to, to, to anyone to, to start to experiment and Stuff like that. So, yeah so, yeah, in that sense the project on the harder part, it's still in, let's say, officially in the limbo state but it's not completely let's say retired or stuff like that.

Jonathan: Actually, the hardware piece here sounds really interesting and I'm kind of excited to maybe eventually get my hands on that because I've, you know, I've got a little. Cheap Chinese power supply for doing stuff. And it's got a, you know, a voltage and amperage measurement built into it, but it's really bad.

It's really, it's a hardware is terrible. And so having something that would be better and open source, that'd

Dennis: be great. Imagine that. Yeah, yeah, yeah, yeah. It's it's really small powerhouse. There is it's packed with so many features, even many of that. It's really unnecessary. We just we just put it because we can.

We don't have any commercial marketing limitation to say, Okay, this is not, I don't know. feasible for we are not corporations. So we are really played with that to the, yeah, to the biggest extent. So

Jonathan: I I eagerly anticipate then these things once again, being available. I think that would be great. So I guess the other part of the story is the software side of the story.

And there was, there was just kind of, was it just a spinoff of trying to do the hardware? And that's the EEZ studio.

Dennis: Yeah. We started Yeah, it's like a spinoff or like, I don't know, sidekick or whatever, or something. What because, okay I started with develop that hardware and the, in one one moment of time when you realize that all that analog linear part and that power part, it works properly. And we have some, let's say, basic connection with that microcontroller.

You need something more than hello world in that sense. Hello world will be just set the voltage and and get or set empty current it's, it's, it's not something what will be satisfactory for me. And Martin, my colleague start to, to, to create a firmware for the. For the project. And since we pretty early starts to to To work with with touchscreen.

Uh, display as as as a main interface with the unit starts in that The, the, the. The thing starts to, to be starts to to be pretty complicated complicated because yeah, it's not so easy to to put something nicely on the screen and to have some yeah, you need some Go, let's say decent mm-Hmm.

Appearance. And some user experience should be like, let's say like yeah, something. What at the end will not por mm-Hmm. put in question the functionality during the operation and stuff like that. And also from other side because one of my, let's say a pretty high priority request was to have a remotely controlled unit.

We also need some software to control that unit. So, OK, on that on that side, there was a couple of still available solution, like probably the most famous is National Instruments LabView. But yeah, it's, uh, okay. It's if you ask me, it's really outdated in design. It's bulky, expensive, and of course it's not open source.

So it's not open source. And we started to, to to to work on something. What is Let's say which has which has two main, main parts. One, it's part to to provide control remote control of the unit, and another one which allows us to to, to, to, to fast in a, in a, in a easier way develop at all that graphical user interface on that, on which will be display run on that touch screen display and yeah, and we started to do everything from the scratch. So there was, there was also some tools available for embedded GUI development, but again, it was It was limited because no, none of them was was it was open source.

And so that means that in one moment of time, you can expect that you will need to put something, what is very specific to your project, your hardware, and you will be probably in that end, or you need to. To beg someone to, to put your feature in the next release or maybe within next one year or whatever.

Okay. People experience the same things with us also, but okay. At least we have something what is open source. We have that task list in the get GitHub and in some, if someone really wants to. To add something that is currently not available. Okay, he can hire some developer and put that in place.

So, yeah, it's the rallying cry of open

Jonathan: source projects everywhere. Pull requests welcome.

Dennis: Yeah, of course. Yeah, yeah, yeah, yeah, yeah, yeah.

Jonathan: All right. Hey, I want to bring, I want to bring Gorom into this conversation here. We haven't talked to him at all. And I think this might be an interesting place to bring him in and kind of get his his experience and his thoughts on using the studio himself.

So let's let's start with there. Kind of where do you, where do you fit into this project?

Goran: Yeah. So in this project, I work in the, like, In custom hardware designs. So from my side, as I am co owner of the company. So I am also designing the, because everything is like open source. So I, I take some parts of design.

So I do new designs on top of the existing designs. So there is like a full story in that part. So maybe I can just switch Here to a presentation. So as it all started when actually Dennis noticed Radeona Makerspace. So that is the space where, where I am like home. So he noticed our ULX3S project that is also open source.

And actually, he contacted us and he wanted to collaborate, so with us with actually the integration of the FPGA into the, into this BB3 project. So, so we started to work and After some time, we had we had actually the STM32ULX3S, also an encrypted name, for the project. And the main goal was actually To get the picture to combine so stm would be Doing everything that is does currently in the bb3.

So it is doing the like All the stuff but we wanted to integrate like the part that fpga could do So this is something like fast signaling like oscilloscope spectrum analyzer or something and we also As we also on the ULX, we had one project that is called Scopio from Miguel that is, he's from Argentina.

So we already had those two projects and we just wanted them to combine. So Miguel helped us actually to get the to get those things combined and we, and we actually got Got it to the place like it boots and it shows the picture of the scope you're in the background and we can control Control that scope with the actually bb3 or switch just to normal power supply or whatever Because bb3 now can do many things not just the the power supply but then also the so we had that board actually ready, but then also the The, that COVID and and shortages hit, so we decided like it, it would be not possible to like, and also the users could, would like, if they want oscilloscope they could, they needed to buy like the whole thing.

And if they want just the BB3, they need to buy like separate things. So, so we started to think like about Starting things even more modular. So BB3 is actually modular. It has modules But the main board is actually only one board So we wanted to be modular and at that time I actually started the project that is called ULX4M So with that project we We put the FPGA big, big bigger board, FPGA ULX3S to a modular form factor.

And that form factor was actually CM4 compatible. So because we decided to, yeah, yeah, we decided to go with compatibility because there's It's really a range of prod baseboards that we can actually use even in, with, even outside of this like BB3 project. So after I created this ULX4M board, we start, and this is currently in like in ending stage because we have two versions of this ULX4M board.

So, so it's almost available for the crowds, crowdsource campaign, but we are still waiting a bit. So when I had that, that finish, I started with the, with the baseboard for BB3. So recently we got it like we have the, the baseboard for the BB3 that is CM4 compatible, so we can put actually Raspberry Pi in it and boot it and show it, show it like, And also like Martin at that point was Was helping me so we, we figure it out that if we use like a BB3 firmware, so it is done in the studio, so complete firmware is done in the studio and you can actually run it on the BB3 itself, but it has the like one flag that you set and it can be built like a simulator so you can run it in the web browser or in the Linux.

Yeah, so yeah, and that was like starting point for like. Few fix few tweaks and we we got it complete almost completely running on the on the CM4 module so that is like really crazy. We only didn't didn't get didn't get the Peripherals so like SPI and things because this is different The first with the STM32, but it's also like a big possibility that when we are finished with some part of the project that we will start to do that we can have also the, like CM4, CM4 module inside of the BB3.

Also, the, the one thing that was actually missing, Dennis also, Dennis, Dennis mentioned it in in the start. So he started with the STM. Actually f4 and it's also when you have a baseboard. It's really like you need to have like lots of Custom boards with different different parts but now when we have like this cm4 compatible we actually could start the the production of the H7 module or any other like module.

So we can just switch modules and, and push push other models. So recently I also finished the STM H7 module that we, that we could use at, at some point. And currently this is just the, like running the simple firmware. And nothing is happening on screen because we need to like push push to the DSC DSI screen.

So we need to switch switch to the to the DSI screen. Now we have only like simple digital VGA, RGB screen. Yeah, that's the, and actually that is the. The end of my hardware involvement. There is probably more with smaller, smaller fixes and smaller modules, but this is like something around that. So, yeah.

Jonathan: Yeah. I am, I am intrigued by one thing. Where you're, you're looking at using the Raspberry Pi and talking to some of these hardware interfaces with your firmware. One of the projects I've actually been involved in and working on for the last few months is taking the, it's the Veshtastic firmware and actually making it run on the Raspberry Pi and talking to real SPI.

And I2C devices. And we've got a, we've got a little library. It's called Portduino that lets you, it sort of translates the Arduino IDE over into actual Linux system calls. There, there, there might be there might be some room there to, to kind of collaborate on that. That could be interesting.

Goran: Yeah, yeah, yeah, yeah. Yeah, because we stopped with that. So yeah, it's going to be called to integrate that part. Yeah,

Jonathan: I will, I'll shoot you some links to to what we're doing there. Cause that could be fascinating. Hopefully avoid some having to rewrite of code and get other people bashing on the, on the code that I've written that, yeah, that sounds like fun.

Okay, so let's talk, let's talk specifically about the studio then because that's kind of where I first discovered the entire EEZ sort of ecosystem and project, and we've talked about it a whole lot yet. So like, what does that, the EEZ studio, what does it let you do? What problem does it solve?

Whoever wants to, you guys decide, who knows it best. Dennis. Okay, Dennis, go for it.

Dennis: Okay, okay, okay, okay. First about naming you, you mentioned is it easy or easy? Yeah, it's basically it's, it's easy. It's easier to say easy. Easy Studio. But, okay, whatever, it's Yeah, that, as I said before, easy, it's like it's nVox Experimental Zones, and this studio, it's a softer part of that.

Adventure. Okay. So I just started to talk that we have that it's a, it's a two part in, in in easy studio. So one is to to have good control, remote control of BB three. But not just BB three you can communicate with any other like like oscilloscope, like function generator, programmable power supply, spectral analyzer for any other manufacturer like key site, like regal sigilent Keatle, just name it.

Because we are, we implement one, uh, de facto, it's the industry standard, it's a Skippy. It's a Skippy communication it's textual based and it can go over the, the, the, the, the. of interfaces like USB serial, like, uh, internet, like Modbus. And on top of that Skippy there is also something what is called Visa.

It's a, so it's like a software middleware, which reside between low level Skippy and Harder layer and your application on the PC side. And usually that application is called the Skippy controller. So like application who control your devices using Skippy Skippy command set and stuff like that.

So means that from the, from the center part using Studio, you can make connection with different, you can actually connect all your labs test and measurement units, whatever that means in, in, in your, Particular example. And you can make some con you can make some test and measurement automation so you can automate, sending and control some command and to, to get back some measurement data.

And you can put that in. In the searchable database, you can draw some nice graphs on, on, on the screen and stuff like that. On top of that, we also introduce something what we internally call easy flow. So that flow is a flow chart, a flow chart flow charting or flow chart type of programming.

So it's like a low code or no code type of programming so that you don't need to go down there in, I don't know, in Python or on the C, C and stuff like that. But we have a lots of that pre made widgets and components and you just drag and drop in your, In your project and make connections so and you can pretty easy build up a very complex, uh, automation and testing environment.

And we use that part of the easy studio even to to. To, to, to test in, in production when we when we were in production of all that part of, of the, of the BB3, all that modules and every, everything we just sent to our PCBA vendor to that easy studio project with a wiring and everything.

And they, they, they, there was in in they were in a position to automate everything and deliver us to us Assembled, tested and calibrated the modules so that normally stuff like that could cost thousands of dollars and that can also consume lots of time to To build something like that.

So this is a one. Let's say not so obvious or hidden part of the easy studio. So in in the recent recently we started to put in the first place that embedded GUI. Part what actually attracted you and and many people was attracted just because that dispute between LVGL guys and square square line studio guys.

Because yeah, in one moment of time, that was actually beginning of this year. There was a. A gap and some need to fulfill that gap. And yeah, people actually, I put some some post on the LVGL forum and yeah, the the acceptance was positive and. People starts now to, to to, to use easy studio more extensively than before.

But again many of the people don't still realize that even if they have some embedded GUI project that they don't, that they can use the, that same Very same easy studio to in the production to test the module or whatever it is, and even as a remote remote application to remotely control that that modular unit or device what is built using easy studio for that embedded GUI.

So this is a, let's say, a pretty unique as far as I know, or for example, that SquareLine Studio or some other, let's say, competitors they are, they are focused just on the building embedded GUI editor, visual editor, but don't have that. Another part, which is also, we found very, very important to if you want to have some really quick some completed project so that you have a hardware and excellent software support for that hardware.

And in that, in that way I think we, we got to that, uh, by implementing both, both type of functionality in the, in the studio.

Jonathan: For, for those that aren't kind of in the embedded world that are our listeners. Let's start what's, what is LVGL? Whichever whoever wants to take it.

Dennis: It's, it's a, it's. It's open source, it's open source, uh, library for, for which have a lots of widgets to, to, to build a embedded GUI.

So you need to so you can call that from your C, C or Python code and you can create a code. A very complex embedded GUI. The problem with that is that it's just it's just a library. So like many other, like for, I don't know, for communication, for with Ethernet, with USB and stuff like that.

You still need to be a hardcore programmer just to, to, to put All pieces together and with in case of easy studio you have, you have two possibility. One is, one is to use to call actually to, to use all that functionality of LVGL and and drag and drop and build all, all the stuff on the In easy studio visual editor and when you build a project easy studio will will generate all all code which you can embed in your let's say native project it could be C plus C plus plus it could be even in the arduino and could be in platform you and could be in python and stuff like that.

And but another possibility what we have in easy studio is that you create everything. Using that easy flow so that you don't need to to write a single code, a single line of low level code like C or stuff like that. So we have two possibility for. For working with with LVGL. Also the studio can be used for for building.

Obviously, I actually mentioned that before to create that dashboard. So which you can which you can use to control multiple devices using that Skippy and for test and measurement automation during development or in the production phase or even for the calibration and for serial production and stuff like that.

So that's something what we actually we added that LVGL in In the later stage, we first start with our native widgets, but that was a really overkill and Martin one in one moment to decide and to go with adding in parallel that LVGL because that, that project was in that time was almost very mature and Very visible in a open, open source community.

And yeah, that was a, a good let's say a good step what we did. And now we'll continue to to, to support all new, new stuff. What will be added in in A-L-L-V-G-L part something which is also important. I think or something what is good to mention that easy studio, it's Across platforms so you can run it on the windows Linux and Mac.

And something was also important when you create a runtime application. That the dashboard, which you can use for, to control your, uh. External instrument stuff. It is also a cross platform, so you can create some dashboard with lots of that nice graphing and I don't know, buttons and stuff like that.

And you can, you can run it as a, as a standalone application. So you don't need to have easy studio behind, below that like, uh, what is a normal required requirement to some other type of a solution, but yeah. Okay, very cool.

Jonathan: So I've, I've got a, I've got a note here to ask Goran about his use of EZStudio.

Is there a, is there a story there or something particularly interesting that you are using this for? And what's, what, there you, there we go. The mute button. I need to

Goran: unmute myself. Yeah, yeah, yeah, yeah. I wanted to talk a bit like, a little bit about actually our use cases for the studio. And the first first use case was, was actually that Dennis already told you about.

It is, it was like batch production testing. So at that point in time, so we had like the script for the batch testing that was done with by our good friend Dobrica Pavlinovic and the script was good enough when I was testing the boards because I know how it works. So I just plug the boards, see the text and remove, press the button.

It's really easy to use, but once we switched to PCBA company, they were all confused. First they need Linux, then many things to run it. So it was really

complex. And so I asked Dennis could we do something because I knew that they already had it for the for the BB3 project so at At that point actually martin, was already working on it.

So and he actually translated So it was like he used the dashboard, so easy studio dashboard project, and he actually transformed our script into the visual flow and with that flow, we actually got the, the graphics for the PCBI company. So they had like really fancy graphics with all the, all the parameters that they need to check all the all the, all the things like what they need to click if they need to press a button, it says like press the button or check the screen, connect the HDMI.

We even had like first screen is when you have stop on some, because it goes to two stages. So it has like 11 stages. Stages, so it goes through stages. And if it stopped on some stage, it will tell you what you need to check on the hardware side, like, like check ADC or check whatever button or, or something nice.

So it was really cool. And on the other side, because it was like it already had it studio has support for the saving into database. So it is really cool because on the other side of the in the other city, I can. Like lively monitor all those devices so and the status of those devices So I know when they like do something so you what is the status actually?

Yeah of the test so and like that was like mainly the first use case for us, but later I joined like in many, many other smaller projects and especially when LVGL came like this little, little graphic library. So they implemented the support for the LVGL and maybe Dennis didn't mention. So when you build the LVGL project, it will be platform independent.

So it doesn't it doesn't Fix you to a platform. So it will be built inside of one folder and it will be platform independent It depends on you. What platform will you connect to it? So you just push the calls to the LVGL to show the screen we have like this Flow actually has two calls. So one is like in it and second is like Tick or something.

So it's like it's doing it like in the background and it is doing the flow, but the LVGL itself, it's like platform independent. Also, the, the flow is completely platform independent, but you just need to set up to, to your to your platform. So we, I started to experiment because I had really a lot of like we all have like lots of gadgets inside of our like home.

So I had lots of gadgets with the screens and lots of gadgets with everything. So like with the ESP 32, RP 2040. So I started, yeah, with one, two, three. So we first ported things with so SDM 32 was already ported because they had it on the BB3. But we first from other platforms, I think the first thing was the ESP32 because it's so popular that everyone use it.

So, yeah, so you can use it with the screen or even without the screen. So you can use flow to, to do things like communicate with ESP, ESP if it is in the, like some. I don't know, measuring equipment, you can transfer it to to like Skippy. So it will respond to a Skippy commands and you can like change things.

And also like RP2040 is really popular, Raspberry Pi. Yeah. So we already, we have also supported that one. And both are supported with the like platform, your platform. And some are like in from Arduino and like ESP has even, I think like those ESP, ESP IDF support. And there is support for many screen, even screens, even the, the ink screens.

So there's really really, really a lot of examples that you can now use even from the studio. So now the studio has like a one. Part that you can go and check like examples and you can see all the examples we did and you can just start the project from there. So it will build you complete complete project for the desired environment.

So that is really cool. And like it is also running, like I already mentioned on the. on the Linux itself. So I have like a small open source like everything we talked in this, in this talk, it's, it's open source. So even my projects, all of harder projects are open source. So this is some open, open, open source tablets.

So this is also like, I wanted to run studio. So I have some widgets around to, to get it to work. And And maybe the, the last things for, for the last thing, what happened actually the thing that happened recently was I, I got some projects and I didn't mention many of them. At least mine projects, but I know the easy studio also is is finance true annual net grants.

So that is yes. So lots of our, almost all that I mentioned was, was actually go to the grant from the annual net. So they finance open source hardware and software stuff.

Speaker 3: Yeah.

Goran: And the, like the one project that I am working on is the I'm working on the I'm Creating some small EMC testing chambers.

So that is like the main, main thing in the project, but I am using, so I need to use like those tiny essays to check the signaling, so tiny spectrum analyzer. And I told Martin, but like, it's like, it's not so visible on the like spectrum analyzer when I use it, the signals are not like quite good. So he suggested me to go with the.

Go with the flow. So to create a dashboard and that we will work it out so we can like control that instrument at least get the data out of the instrument so I don't need to like put the SD card out and in to get the CSV and stuff. So I started to work on like this simple dashboard for tinies. And actually I got it really, really far, so I got it to, to have what I want, but, but after that, yeah, after that Martin Martin took over and he, he noticed that he can actually turn this turn this tiny SI into the instrument.

So he added the support for the studio and now you can actually. Use it as an instrument. So it is on the, like, you can just write, I want like a picture from it. I want whatever lines. So it is controlled like any other, like skip instrument. So it is serial, but it is like support is there. So it is like controlled, like any other instrument.

So that is like cool part when you like start something and there is lots of ideas of what to do and what, how, how to. How to actually handle things and like the things just building up. So it is really like cool.

Jonathan: Yeah. All right. So we are, goodness, we've just about filled the hour and I want to make sure and ask what is coming in the future.

And that that may be a question more for Dennis, although if Goran, if you have something you want to throw in there, you can, but what's, what's next for like the, the easy ecosystem.

Dennis: Okay. Yeah, something what I really I forgot to mention it's a great contribution from NLNet. So without them it will be hardly to imagine that, uh, EasyStudio will got this shape and functionality.

What, what is now? We are in we are currently in a phase four, let's say, and we have Agreed with NLNet a few more milestones. And so for everyone who will stay tuned in in, in this story can expect a lots of a lots of interesting features in a feature. And Possibility with with the studio on the harder part.

Okay. It's hard to say right now. Maybe we first need to somehow to resume production and probably try to to support and to, to collaborate even on the, on the production of New models. What some of members of community already did like we have, let's say there is a great guy.

Jan from the Netherlands, from the Holland, who made a BB 3 compatible electronic load Electronic load model. There is from Austria, there is precise time, time time standard. So it's it's a rubidium atomic clock model. There is another, uh, guy from from Germany who started to, to work on, high high resolution multimeter module for BB3 from outside we'll probably try to, to continue on work. What's Goran already mentioned so that to tightly integrate on the software side all that new type of, uh, modules like CM For Raspberry Pi or FPGA functionality or that age seven and stuff like that.

In that case, we can, someone can expect stuff like digital audio, digital video outputs so that you can play. On the, on the, on the bigger screen, like when you say, when, when you work on some measurement and probably we will try to, to, to work on at least one or two new models.

I don't want to mention it right now, but yeah. Idea it's anyway to to continue something on the. On the hardware part as well on the software, but on the software part currently one can expect much more progress and much more visible resulting in the near future. So, so the project it's it's, it's really live and kicking and we are looking forward Together with community and we are, I also want to, to, to thank to all members that there is some guys are really gave us a wonderful and fascinating feedbacks and, uh, ideas, how to, how to improve some, some, something what is already in in studio and yeah, stuff like that.

Jonathan: Yeah, it's always, with a project, with a project like the studio particularly it always grows a lot when outside people start using it. Because there are things that you would never have thought of to do that somebody from the outside goes, Hey, it would be really cool if this works. And that's really how projects sort of come into their own.

All right. Well, we are at the bottom of the hour. That means we've been going for an hour. And a couple of last questions that I want to get in right before, right before I let you guys go. And I'm going to start with Goran, actually. And I want to know your favorite, favorite scripting language and text editor.

And you guys are kind of hardware guys. This might not exactly apply, but we'll see what we get.

Goran: Yeah. Well, I am like, I do lots of languages, but yeah, Python is like. Simplest, so like I, I do C and JavaScript and like, but like,

I mean, I need something quickly. I will

use like Python and I do it in nano.

Jonathan: We are users. No, so it's, it's funny. Nano, Nano has been my text command line text editor choice for a long time. And I tell people, I think it's because I got started with my very earliest programming was in QBasic, the, you know, the Microsoft basic from years and years ago and the QBasic editor looks so much like Nano.

I think that's why. All right. And Dennis, same two questions. Text favorite scripting language and text editor.

Dennis: Scripting. I don't know when the last time I did any scripts, but it probably, there was something small in the micro Python, which is also related to BB3 because the BB3, you can create you can create apps.

Which can be downloaded and run on the BB3 and it can be yeah, it could be developed in the, in the micro Python editor. I don't know. It could be anything. Yeah, it could be even none or whatever. I really don't have any, any, any preference, something. What is it? Yeah.

Jonathan: Well, very cool. It has been fun to get to chat about the projects and I appreciate very much you guys being here.

Thank you so much.

So that was the Easy Studio and the Easy, I call it the Easy Ecosystem. The, the BB3, if you go looking for it, it's, it's literally just the letters and the numbers. So bravo, bravo. And then the letter, the number three. It looks really interesting. I, I definitely see sort of a I don't know, maybe, hopefully a future if they can get if they can get production going again.

That seems like it would be really neat to have as a, as a benchtop power supply and testing platform and, and all kinds of stuff. So I'm definitely on the lookout for that. for that one. And then, like I said, we, we, I, a project I'm in, we use the, the easy studio and found it to be a, a really, a really competent replacement for a square line.

And so that's neat to see. Usually I would hand it over to my co host at this point. I did hear from Catherine was scheduled to co host and she's of course down in the Dallas area and they are having power outages right now. So she has no power and we sort of decided that it wouldn't be very good to try it.

co host on the show without any electricity. That would just be a challenge. So anyway, so that is the show. You can find it, of course, on Hackaday. You can find my other work on Hackaday. It goes, we have the security column, goes live on Friday mornings. And yeah, that's it. We will see you next week on Floss Weekly.

Episode 784 transcript

22 maj 2024 | min

FLOSS-784

Jonathan: This is Floss Weekly, episode 784, recorded Wednesday, May 22nd. I'll buy you a poutine. Hey, this week Dan Lynch joins me and we talk with Francois Proulx about Poutine. That's an open source project from Boost Security. It is a code analysis tool looking for security problems in your GitHub and GitLab CI actions.

Things like leaking secrets or cash poisoning, or maybe even something worse than that, you don't want to miss it. So stay tuned.

Hey everybody. Welcome. It is time for floss weekly. That's a show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got, we've got Dan, the man, method, Dan, the, the one and the only with us today the original Linux outlaw, is that, is that right? Dan, I call you that sometimes that's accurate, isn't it?

Dan: I like to think so. I'm not entirely sure, to be honest. I'm sure there's a lot of other much older Linux outlaws than me who would, who would claim, would claim that. But yeah, I'm going to claim it. Why not? Nobody else did.

Now the real question, I suppose,

Jonathan: is have you ever actually been an outlaw?

Dan: Have I ever been an outlaw?

Only speeding offenses really in the car. I've, I've been, I've been on a few speed awareness courses for the for the local police, but only a couple. In 20 years. That's not bad. That's not bad. That's not too bad.

Jonathan: Well, we've got a we've got a fun show today. We're going to talk with Francis Pruhl about, about a project called Poutine, which, that is a, that is a Canadian it's kind of a Canadian junk food.

Is it a junk food? Maybe it's not a junk food. But it's, it's a particular type of food that is, apparently very messy to eat. And so that's sort of the play on words. Poutine is about figuring out a particularly messy security problem. And this is a project done by Boost Security, and they just recently announced it and made it open source.

And I thought it was neat. So I was, I was excited to be able to talk to him about this. Is, is this something you've gotten a chance to look into yet, Dan?

Dan: Only today. I've been doing my research today after you helpfully sent me some pointers. I've been on reading some reading some stuff about this.

I suppose the real question is, have you ever had poutine itself? Cause I have the the food. I mean, it's really good. It's very interesting. The unfortunate thing

Jonathan: is that it is now on my I cannot eat that list. I recently got diagnosed with Hashimoto's, which is a thyroid problem. And one of the most common solutions is going gluten free.

So I am going down that road to see if that's going to help. And yeah, so there's a lot of, there's a lot of junk food that I used to enjoy. That's like, I can't have that anymore because it's trying to kill me.

Dan: You need to find new ones. I'm sure you'll find new ones.

Jonathan: Oh, I'm sure I'm sure all right.

Well, let's let's not dally any longer We have the man himself rather than continuing to take guesses at what the project is about let's bring Francois on and welcome to the show, sir Thank you. Thanks for having me. Yeah, it's great to have you here. So, tell, first off, did I pronounce it right? Is it, is it poutine?

I am, I'm very ignorant when it comes to that particular bit of Canadian culture. Yeah,

Francois: that, that's how most, like, English speaking people pronounce it, poutine, but in French it's So a little softer on the

Jonathan: ending. Okay. And then give us kind of the overview of the, as we like to call it, the 30, 000 foot view, what, what is this, what, what's the problem that we're trying to solve with Putin?

Francois: Yes. So really it's in the category of security vulnerability scanner. So it's what some may call like a static code analysis scanner that is going to look typically at source code, but more specifically manifest, like typically they would be written in YAML. And it would be stored on GitHub or other get repos, not just get, but and.

Those yaml manifest would define build pipelines. The ones we currently support, it's GitHub actions. If you're familiar with that, or also on GitLab the GitLab pipelines. So they would define the the way to build a certain software and eventually package and publish to some kind of registry and we're, with poutine, we're looking at security vulnerabilities in those So we can get into the details of that.

Jonathan: Yeah. What is what, what does one of those security problems look like? Like what's, what's sort of the the category of issue that people run into with their GitHub and GitLab pipelines? So

Francois: first, like, let's kind of set the stage a little bit. So there are kind of two broad kind of types of issues.

If we look at open source projects where everything is visible, like including the build pipeline. So typically on GitHub, you know, you can discover not just the code, but the way it's built. That is definitely an area where the attacker can as, as is the case for the actual source code, find vulnerabilities, but potentially like unknown vulnerabilities, zero days.

Right. When it's like an enterprise setting, like a internal source code where. And an external attacker would not see the build pipeline or the code would be more like, you know, insider threat scenarios. But in the case of open source projects, you can see even the dependencies that this project is pulling in, how it's built, and really the scenario that the classic scenario is, if you're familiar with GitHub, like, you know, pull requests the, the, the term that we use typically to address like Most common type of problem.

It's called Pwn request. Like that, that is effectively a pull request from a fork that is specifically targeting a vulnerable workflow. And the way to do that would be either through the actual files that are modified in the pull request. So let's say the contributor, a malicious contributor is sending a patch.

Let's say, and that is a real example, let's just updating the readme. All right, readme. md, it's a markdown file, apparently just a text file. But if it so happens that the build pipeline is gonna do like a linting, you know, formatting, just to make sure everything is, is is, is by the book. That is running code, like you know, linter or whatever.

on untrusted data. In that case, just a markdown file. Most cases, it's pretty safe, but some tools have little known kind of dangerous features that can be exploited by the attacker to run some remote code execution. That is one example. The other way is also through all other kind of inputs that the attacker has control.

Let's say the title of the pull request. Or the body of the description of the pull request or the comments that common comments, threads you see or the basically there are a number of things that if you're an User on github you can influence and the build pipeline will consume it Also, even the git branch name the git branch name.

You'd be surprised That it can contain like arbitrary bash commands or JavaScript commands. It's long enough and it accepts enough characters. There are some restrictions, but it's more than enough. And we've seen in the wild just malicious git branch names.

Jonathan: Okay, let me, let me give you a softball question.

I, I sort of know the answer to this, but here, here's a, here's a softball for you to get further into this. I thought Git runs all of your workflow stuff in their hosted virtual machines. Doesn't that completely solve all of these security problems?

Francois: So, I think you're referring to GitHub actions. So GitHub actions.

Yes, they are indeed. By default, the GitHub actions run in ephemeral throwaway virtual machines that are created on demand right at the, the, when, when a workflow is being triggered, they actually, even in the workflow, you have what they call jobs and each job is going to be a completely Independent ephemeral virtual machine but then inside of that, typically you would be referring to secrets.

For instance, to push to NPM, PyPy, Docker Hub, or whatever. So as part of the, the, the execution in the end, you're injected. It gets injected some secrets that could allow an attacker to pivot to something. Like let's say secrets to connect to AWS, to, to run some testing. If you're developing some kind of tool that would run in a cloud.

In the runtime, in the memory of the runner, even if it's ephemeral, it would get high powered in some cases, like sensitive credentials.

Jonathan: So that's, that's really the kind of the problem space that Poutine is aimed at. Someone can get arbitrary code execution in some, in some projects, I will add in some projects, the way the build system is set up, you sort of have arbitrary build or arbitrary code execution by default, because you know, there may be a build script.

That just runs bash code and you can send in a pull request that changes the build scripts. I mean, in some cases that's trivial. It's what I assume what poutine does is it looks for these problems and tries to what give you a warning that says, Hey, the way this is set up, it's trivial for someone to steal your secrets.

Exactly.

Francois: So typically the, the way get, get, get an action, namely was designed, it was designed. With that scenario in mind first hand because GitHub from the beginning always had pull requests with accepting requests from forks Right, so that's just by by its definition accepting untrusted code from someone someone So they designed by default that this would run in a sandbox where no secrets are accessible But, in some cases, you need secrets.

For instance, imagine you're processing the incoming data, and as a maintainer, you, you want to, to lower your toil of just, like, accepting, like, the contributions maybe come at a specific format, and that they, they meet just the guideline of the project. And in that case, then you might want to reply automatically with a comment saying, Sorry, we are not accepting your contribution until you Address XYZ.

But for that, the, the script will need some credentials to push the comment back in the flow, and in some cases, even update, you know files or push artifacts. So it really comes down to limiting the amount of secrets visible to kind of risky workflows. Those that, that you need access to some secrets.

But if you really need that, then yeah, you need to not process arbitrary code effectively.

Jonathan: There's a, there's another kind of class of potential problem. And I'm, I'm real curious cause I've, I've been working on this in a project that I'm involved in. We wanted to be able to do arm and arm 64 builds.

Well, as you probably know, that means self hosted runners on raspberry pies or something similar. And. I have been working over the past couple of weeks on trying to figure out a way to actually do that safely. And the, the solution that I've come up with so far is with the self hosted runners, you can do a, you can set it up with an ephemeral tag.

And then what I'm doing is I'm running those inside of Docker images. But I know that like Docker images are not the same level of of safety as something like a virtual machine is. I'm, I'm curious, does, is Putin aware of like this side of the problem, doing self hosted runners?

Francois: Absolutely. So we have a, we have a rule that will flag the usage of.

Self hosted runners, which trigger which accept on pull requests. And what's interesting there is that because unlike the, the, the sort of hosted runners, which are designed to be fully ephemeral, like fully like cleaned up and there's no sharing and fully isolated between each run. In that case, you're responsible for making sure everything is fully isolated and there is no leakage between each job.

Because if you have two like jobs that run in parallel concurrently on something you host, there could be cross cross job kind of pollution or sort of exfiltration leakage of data. So and yeah, we do flag scenarios where you're combining accepting pull requests. With that and that adds even more like more risk because you're you're on your own.

Jonathan: Yeah No, it's it's a it's kind of a scary thing to set up and I've I've been I've been kind of slowly working my way Through trying to think through this Now I'm curious does poutine Does it actually send in pull requests to try to test some of this stuff or is it just looking at configurations?

Francois: No, at the moment it's fully static.

It's a static code analysis tool that works. It can work fully offline. In fact, like it doesn't emit any network request except for interacting with Git, so you could get clone everything. In a directory and disconnect from the internet and run poutine. So like at the moment, we're not pulling any vulnerability database or whatever.

It is in the road map to add that as a kind of opt in feature where you would be like, you know, dash dash. Allow synchronizing kind of vulnerability database at the moment. We have that kind of baked in offline. Many of the rules are purely static code analysis like to detect those problems. So they don't even need a vulnerability database.

But yeah, so to answer your question, no, we are not. Doing any kind of active exploitation validation purely based on inspecting the code, finding the code patterns that are risky.

Jonathan: It seems like there could be it's just, just looking at the problem from the outside and what little I know about poutine there, there could be some possibilities there where you might want it to be able to generate a I guess that's tricky.

If you want it to generate an active payload, then you kind of move into this area where Putin becomes a red hat tool. Or excuse me yeah, a red team. Wow, I just, I mixed my metaphors. Black hat, red team. But it becomes a red team tool. Maybe that's not what you as a security company want.

Francois: No, exactly.

We, we're really, we've really designed it in a way that is gonna help it. People who typically are responsible for writing those workflows to, to detect problems quickly, efficiently, and, and, and learn from the, like, we've also provide some kind of training, like guidance and like like best practice kind of bad, bad examples, good examples to, for, for people to, to improve and in that regard, yeah, it is not designed as a red team to To showcase that.

That being said, we have a different project that we call Messi Poutine, if you want to get into that, which is all about Allowing as kind of a sandbox for capture the flag where people can have fun exploiting things that we've designed to be Vulnerable and we, we built that really as a way to test poutine.

So we point put into that GitHub, literally a GitHub organization called messy poutine, and it's all about finding as many flags, vulnerable things in there. There are some that. At the moment, Putin is unable to find, I know them like, and you can play the game kind of to find those. And we want to improve the tool to detect as many as possible going forward.

Dan: So Francois, I am curious about why you decided to release this as open source. What's the advantage of that for you guys?

Francois: Yeah. So we're, we're, you know, a commercial or a vendor. We, we, we build a security product. For commercial product as part of that, we. Automate provisioning a number of open source tools already to name one, like Sam grep, for instance, or trivy or gripe, things like that or Chekhov.

So there was a number of open source tools that we help to orchestrate and aggregate the results, and then we add our, all of our kind of secret sauce on top of that, and that was on the one hand, kind of interesting. Give back to the community because we've been benefiting from a lot of open source tools ourselves.

And we wanted to bring something where we saw there was a gap, right? There was something that was not yet in the open source and even in commercial, there was like, there's kind of a gap in that area at the moment. And we thought that the, the, this type of problem, when we started to look at it at scale, because that's how it really started, it started more like as a hackathon project.

We, we knew this type of problem existed. We were finding it kind of purely manually, just by pure chance, like happening to see something like, Oh, that's exploitable. So we wanted to kind of automate that seeing just how big the problem is at scale. And we started this project that we call package supply, which is indexing millions of open source projects.

And finding this type of problem at scale. So also finding transitive kind of attack path that is not directly targeting one project, but like vulnerable workflows that are downstream of, of you know, say sensitive project. And at some point when we had that started to do a lot of responsible disclosures to many projects we wanted.

To bring that to the masses, like to help people fix their workflows and find those things easily. So that's, that's how we, we came up with, with, with poutine as a sort of a sister project, this bigger kind of package supply, which I spoke at the open source security foundations conference last month about that project.

And that was our way to give back to the community.

Dan: That's excellent. I was curious about how many people are interacting with it. Community wise, what's the response been like? Have you had a lot of bug reports or interaction?

Francois: Yeah. So it's been just about a month that it's open source now. And I think we're quite pleased, but with the, the, the number of interactions, like stars and like people like chatting about it, like, and you kind of getting interested in it.

So it's, yeah, I think we're getting. Very good interest by people that I think should, should know it exists. But we want to spread the word even more. And thanks to, to you, like hopefully we'll, we'll get up to more people that can benefit from learning, even learning. Like it's, I think it's first and foremost about awareness.

Like many people I talk to that have been doing a pen testing, like red team exercise for years. They're completely unaware of this class of vulnerability. Many people, you know, talk about top 10 and like very classic type of SQL injection, cross site scripting, et cetera, et cetera, that that's a very, very mature type of, you know, area.

Like there's tons of tools. The know how is, is, is very well, well advanced by now, but when it comes to vulnerabilities and build pipelines, that is still an area that is a bit Up for grabs.

Jonathan: And you got to think if, if red team professionals don't even know about it, then your run of the mill open source project on GitHub has no clue about it.

Francois: Yeah. I mean, and that, that goes back to this what we call package supply that was about in like doing, evaluating the sheer scale of that problem at scale, like millions of projects. And as I said, I've, we've reported a number of, of problems to open source projects in some cases. Some projects that were downloaded millions of times a day, not to name them in very critical infrastructure kind of pieces.

Like so it is a real problem.

Dan: Yeah. You mentioned in your I read your blog when you announcing the release of, of poutine and, you mentioned in there that it seems like maybe the things that you found so far feel a bit like they might be the tip of the iceberg. Do you think the industry has been kind of sleeping a bit on this as a potential problem or, or do you think it's just not been in people's minds or, or so?

Francois: Yeah, I, I think, I think most people have been focusing a lot on making sure. The code that they're producing, what, what they're delivering as an artifact, be it an open source or commercial is secure so they can, you know, they do their best, but like the build pipeline has always been seen more like as a, a means to an end, right?

Like you need to do it to automate certain things, to compile, to run tests, linting, and all that. But ultimately a lot of times, like even the, the build time components, like they, they're not seen in in the final artifact. So. Many people never thought this was much of a problem, but when you start to add you know, you've all seen this kind of the, the, the sheer complexity in terms of dependencies, you know, transitive dependencies of most current, like in a software nowadays, and you apply that to the build, the pipeline is the same thing.

So the, the complexity and the problems we're trying to tackle. When it comes to the code itself they apply equally to the build pipeline.

Dan: It's interesting because people really don't, as you said, there are people don't think about the pipeline. The only thing about the end product and the code that's going to come out of it, or the source code that people are looking at, we don't necessarily think so much about how that gets transformed through these pipelines and can end up in, you know, any kind of state, I suppose.

I was reading. Earlier about the chain guard vulnerability that you guys discovered. I read that little blog post about fascinating story. Can you tell us a little bit more about that? About the story? Yes.

Francois: Yes. Yes. So we discovered that like, as we were developing this, this tool We were pointing it at the stuff that we're using.

So it's, it's a lot about developing a level of, of of of assurance that the things that we're consuming to build our product we're picking the right, like, you know trustworthy components and chain guard is doing an amazing job, like great containers, like secure by design and all that.

Like with very low number of CVEs. But we're looking at their build pipelines and we found one such example that was exploitable to the type of problem that I described earlier. But in that case, what was a bit more interesting is that we stopped short of the kind of end goal, like the article I think you're referring to was,

A near miss incident.

In the sense that we got very close to the ultimate goal, which was to showcase to, to demonstrate that we could exploit and backdoor and like compromise and a component that itself themselves were using in the build pipeline of something that is like their mission critical, like, you know, Docker container machinery and we stopped short of just like the, the One thing where, in fact, it was almost like it was a feature, not a security feature of GitHub action that prevented us from, you know, closing the gap and that feature was made like by GitHub action to prevent recursive kind of infinite loop.

So it was not meant as a security. Like mitigation. But because like effectively a workflow using the default credentials cannot kind of trigger another workflow. Like that prevented us from kind of closing the gap with the way the attack was set up But it doesn't mean there are other scenarios where the credentials have the right permissions and you could have done that So that's like a more advanced use case, but in in fact like just A month ago a fellow competing research team, like like it's his name is Adnan.

And he basically, he, he, he managed to find an alternative way to, to achieve the same goal. So it was very creative in that case, he found that the caching mechanism and get of action. Could be could lead to some kind of cash poisoning where like you could basically add some vulnerable version of, in that case, the go tool chain.

That would be used in another workflow and like effectively the, you would have a, you'd be using a poisoned cached tool chain.

Dan: Wow. I was yeah, I was curious about whether, when we talked about the fact that it's open source and the benefits of that, and you're on GitHub yourself. So. Putin is on, he's on GitHub.

I was curious. I have a bit of mischievous question. Do you accept pull requests or have you had pull requests? Is that a danger? Because Putin itself, it seems to me would be a desirable, a desirable target for people possibly.

Francois: Yeah, I think it really comes down and I think that that is why we build this messy poutine project is to really educate people that there is a, there is a way to accept contributions from pull requests in a perfectly safe way.

It's just that when you start to kind of go beyond the default scenario that is supported by default, you want to do some things that are a little bit like more like and more interactive. That's when you start to pull some credentials and and then the whole like least privileged kind of principle applies where you don't want to put all your eggs in the same basket in that workflow that you know is a risky workflow that is.

You know, touching credentials, that one, you may want to separate it completely from the, the compile, enter, and like eventually release flow.

Jonathan: So you, you, you run poutine against the poutine repo from time to time?

Francois: Yeah, we've got like poutine on poutine pop,

Jonathan: yeah. So in thinking through this it seems like kind of the, the worst case scenario that a security problem on one of these one of these sort of builds throughout the pipeline, the worst case scenario would be something like the XZ backdoor, where someone could sneak something like that in using pull request.

And I'm just curious, what's the, like, what's the worst thing that we've seen that's happened with kind of this style of vulnerability? Has there been, you know, the, the, the XZ level of problem?

Francois: I mean, I would be, I would be shocked that it's not been already the case and no one knows about it because every since, since I've been focusing our research on that area every single day, I I'm, I'm shocked to see how easy, like how, how many low hanging fruits are still out there, even on.

Very reputable mission critical piece of open source software that are maintained by not to name them name, pick, pick a name of a big open source contributor and I have concrete examples of. Vulnerable workflows that either I've reported they've been fixed or that I'm in the middle of having, you know, having them fix.

So it's, it's, we, we would be fools to think that attackers are not leveraging that already. And to that point, I've built some proof of concepts like to demonstrate that to some of those in the case of responsible disclosure. Sure. Kind of a nightmare scenario example that was inspired by some of those things we've seen in real world.

And I showcase that like in the last conference talks, like I have kind of a video explaining all that and you can really see that it can be made. Extremely like fully scripted that the whole attack runs within just a few seconds and the attacker can clean up their tracks like the cover as many traces as possible so that what is left in the pull request appears completely.

Like innocuous but in the end, the attack has fully succeeded and it can be made in a very sneaky way. So just as much as, you know, XZ was more of a social engineering game. This could easily be combined with something fully scripted, automated, that would. be part of the whole operation.

Jonathan: To, to kind of drill down into that a little bit more one of the, one of the sneakiest things about XZ was that it wasn't an outside contributor.

Well, he started as an outside contributor, but then he kind of worked his way into the core team. And so this is, this is what we would call an insider threat. And I'm curious, what, what can we do? And is there anything that Putin can help with? To sort of protect from that insider threat.

Francois: Yeah, that's a much bigger problem.

And I, and I think Putin is the only way that I see that Putin could help in the scenario of like an insider threat would be. Imagine a scenario where the insider malicious co contributor is purposely making the workflow vulnerable in a subtle way, right? Like, like just like, Oh, look, I'm improving the GitHub action workflow.

Let me do this. Like it's going to be useful. And they know, knowingly add the vulnerability, which. Could potentially be detected by puts in, but puts in, you know, it's not perfect. And there are ways to write the vulnerability in a way that would not be detected. So in theory, someone using it to discover those kind of like vulnerable workflows that were added as a backdoor like just waiting to be used.

That could be useful. Because again, there's tons of low hanging fruits out there. So were they, were they maliciously added? Most likely not, but they're present and they could be leveraged by someone with malicious intent. For insider threats, I have another article for those who are interested that is all about addressing insider threats in open source projects.

And that is a much more. Complicated topic and then it, it comes down to, you know, some code review, like having like not just ideally, like not just one main maintainer and like having multiple, like more than one person approving code changes. But as we've seen in XZ, it can be done in such a sneaky way that it, it passes, you know, community scrutiny.

Jonathan: One of the, and this is sort of a tangent, but one of the things that really, really interested me about the XZ story is part of the problem was hand built tarballs. And so if that project had instead used GitHub built tarballs, The whole attack would not, it could have been pulled off, but not in the same way that it was pulled off.

And so it really fascinated me. You know, we talk about reproducible builds, but reproducible tarball builds is sort of important too. And that seems like that is at least a parallel thought to what Poutine is working on.

Francois: Yes. And, and to that note I called out one, something very similar to that, like literally two weeks before the whole XZ thing, I wrote this article about insider threats, an open source project.

And at the end, if you go to that article at the end, I specifically call out what I wouldn't say it's a vulnerability, but it's a weakness or like a poor choice of the, and designed by GitHub on GitHub's part when it comes to the release. Kind of and the GitHub, like the release part, which where, where Exe kind of put this tarball, right?

Anyone that has write access to repo, like you just have a contributor that has normal write access. You may use branch protection to prevent them from merging code in the main branch. Right.

Speaker 4: Right.

Francois: But simply having write access allows you to modify release artifacts. So, so there is no specific role called release.

Admin or release editor. Anyone with right access can literally, I like, just, just try it at home. Let's create, create a repo, create a release, create another user that you invite just as a right access person. Even if you kind of make it, it's not an admin and all that they will be able to go and edit.

And upload another artifact. In that case, the xxi, the Tarbell like literally can override and it leaves no trace. There is nothing in the audit log of GitHub to, to, to see, to see that it was tempered with.

Jonathan: Have, have you been able to get a hold of any of the security engineers at GitHub to talk through some of the various things?

Yeah.

Francois: Ma, ma, many, yes. Many times. Including, like two days ago, I, I met with someone in person at the conference last month, so. Yes, I've been in talk with several people at GitHub, you know, talking about a number of ideas that I have to improve at many, many, many levels. And they, they, they tell me that they're working on a number of initiatives, but yeah, there's, there's a lot of work to fix that.

Jonathan: Yeah, I do have a question from the chat room. I want to I want to get to because I think it's fascinating. It's from Ken McDonald. He says, I see your documentation represents references and open source vulnerability database that's hosted at osv. dev. Is that database maintained by GitHub or boost or you guys?

What's what's the what's the OSV?

Francois: Oh, OSV is the open source vulnerability database maintained by Google. So it's a osv. dev project. Like you can find it. I think it's kind of a sister project to the deps. dev. If you're familiar with that other Google project that is kind of tracking the dependencies, the dependency graph of open source projects.

So OSV is really attracting CVEs and there's a subset of it that is looking at GitHub actions. Namely and we effectively, because at the moment, the number of CVEs for GitHub actions is so small, literally less than 20. We just, we just pulled it offline. It's part of the Poutine binary itself. You can find it, let's just a JSON like file.

Cause it's so small at the moment. So we don't even need to kind of go fetch some kind of vulnerability database updated every day because there's so few. But we know for a fact that there's plenty more to find. We, we find, we find new ones maybe not every day, but like, especially GitHub actions that workflows depend on that are vulnerable.

I have many yet to be reported yet to be

Jonathan: fixed. One of the other things that interested me is you guys are not with With Poutine, it's not limited to GitHub. There's also GitLab support and is, is there Azure support as well? Or is that coming?

Francois: Yeah, at the moment we support GitLab pipelines and we do have plans to support CircleCI, Azure Pipelines and Jenkins and all that, maybe in different way, but at least like Azure Pipeline is a low hanging fruit because it's really the predecessor.

It's kind of the parent. Of GitHub action, like, you know, when Microsoft acquired GitHub, they, they brought this and that is kind of the underpinning of GitHub action GitLab pipeline. It's interesting because, you know, when we went and wanted to support it in poutine, we wanted to find the equivalent vulnerability class.

As the rules we were already supporting for GitHub, and there's a fundamental difference in the way GitLab pipeline are processed when accepting Pull requests, or in that case, they call it merge requests in GitLab, right? Accepting merge requests from forks by default, they always run in a different context, which does not have access to secrets.

So very similar to GitHub Action, what they call like pull requests in GitHub Action versus pull requests target. So it's as if GitLab by default does not have any way to expose the secrets when accepting contribution from a fork. But you're just one click away from saying, run it in the sensitive context.

So in that case, it's a bit more like a social engineering. Like you may want to hide it in a big talk. PR with like hundreds of files and just one small line, like the exit did, you know, this kind of space this extra white space at the beginning that changed the whole behavior of the, the bash script.

So it could be very sneaky and someone say, yeah, it looks fine. Let's run it in the guild pipeline with the sensitive context. So,

Jonathan: yeah. One of the, one of the other differences that comes to mind is with GitLab, particularly if you're hosting GitLab yourself. You're not running all of that inside virtual machines.

That's going to be pro I would assume what in a Docker image or something similar. And so there's, there's, you know, kind of another, um, another potential problem where you can, you can escape the, whatever jail it is, you know, it's, it's going to be something like a CH root jail or a Docker image using namespaces and depending upon how that's set up, those are, those are escapable,

Francois: absolutely, absolutely.

So, yeah. You mentioned earlier, like self hosted runners that brings that, that, that puts a lot of responsibility on whomever is maintaining those runners to make sure they're fully airtight and nothing is being leaked between each run, because if one run is poisoned and then you kind of sudo and backdoor the kind of environment around, then whatever follows.

Is not to be trusted so and that in that space, in fact, like we've seen, we've seen like big projects, namely those who kind of need the self hosted runners because like Gen AI, like LLM, like projects, open source projects, they need like big machines with GPUs. And to save costs, like they kind of manage their cluster of GPUs on say AWS.

So there, there's been a number of disclosures, not by my team, but by other teams doing the research where there were hosts. Self hosted runners on those open big LLM projects where you could basically poison the runner,

Jonathan: you know There's a I mentioned earlier, but there's a there's a scenario that's way simpler than that You know way way less budget involved and that is if you need arm or arm 64 builds You do not get arm runners on GitHub, at least not for free.

So for small projects, there's a, there's a real desire for some of us, at least to do our own GitHub runners to be able to do arm builds. I, I may, I may try to get ahold of you offline outside of this discussion because I'm, I'm trying to work through, I mentioned this earlier, I'm trying to work through how to set that up securely and maybe put a guide out on it and.

It would be nice to get a sanity check on that before I tell people how to fix it. So, like, I

Francois: can, I can quickly tell you, like, the way most people do it, like, in a securely, not, but not optimal way is with QMU. So they use QMU to emulate ARM on the GitHub free hosted runner. So it's, it's, it's a lot slower, but it works.

So that's a safe way to do it to say, like run a compile on ARM.

Jonathan: Yeah, yeah. I suppose that's true. Okay. So I, I assume that you're kind of constantly folding your discoveries back into poutine because it's, it's kind of one of those deals where you can only find the problems that you know are problems.

And there's this whole infinity stretches before you with the, with the number of different ways that get hub and get lab actions can be broken. And so is that just kind of a. Constant process of, oh, we found, we found another bad pattern. Let's add this into poutine.

Francois: Yeah, I've very much so, but it's always the, the back and forth between false positive and false negative, any kind of scanner looking for code patterns effectively that are vulnerable, like, If you add something, you may run the risk of, of finding things that are not a problem by like, like as an accident, like you, maybe a pattern that's a bit too broad or vice versa, you're a bit too conservative.

You're not finding the things that when you as a human just look at it, it's like, Oh my God, it's obviously exploitable, but I, I never thought of that code pattern and so, yeah, we, we do go back from time to time and improve that, but then like the more you add, there's performance consideration. So it's.

But we do learn quite a bit with that machinery that I, I call like package supply. That is like the big kind of large scale analysis machinery we have there. We have the ability to quickly run like kind of sniff test. Like sometimes I I see something that just by observing it, like manual inspection looks scary.

Like, I'm like, Hmm, that's almost, almost exploitable. And then I can generalize the pattern and apply like with the. Machiner to scan it and I crossed a million projects and within a few seconds, I can come back and see, yeah, in fact, that one was not exploitable, but then there are 5, 000 projects that appear to be very close to it, so it might require more like effort to manually validate it.

Jonathan: Yeah. Interesting. Dan, did you want to jump in with a couple?

Dan: Sure. Yeah. So I was reading up about another project that you guys released open source called living off the pipeline, which is a great name, by the way. What is that and how does it relate to Poutine?

Francois: Yeah. Living up the pipeline. It relates in the sense that as we were doing more and more of those responsible disclosures, we started to find.

Common patterns namely when you're accepting contributions from work and you want to run tests, for instance, right? What are tests? Like, you know, PyTest or whatever. Any kind of test, test tool, effectively, it's a binary that will take source code, execute it, as in the context that those are tests. But then, if the tests are modified by a contributor, they literally are PyTest.

You know, code execution by design, nothing, nothing more. So that, that is kind of an obvious scenario, like, and that is maybe too obvious, but there are many tools that are, that contain like little known features that most people don't even know about if they don't, you know, read the documentation in full that are dangerous, like mainly a number of.

Even security tools have kind of power user features to configure them with plugins, for instance. I'm not going to name some because you can go and find in the living of the pipeline project. And you'll find that some security tools, you can configure the behavior with simply adding a file like dot, you know, dot the name of the file that shall not be named, dot yaml.

And you can specify a plugin and the plugin can literally be arbitrary bash stripped or something. So then you run the tool thinking that it's just going to like, let's imagine puts in, have that feature, but that puts in like, we specifically design. With that scenario in mind, because we knew that other tools have made that mistake so that we can safely run puts in like even on untrusted code.

But like, in that case, if you can modify and add a malicious plugin on the fly, then yeah, it's a problem. So there are a number of examples you can find that. So it's really about. Creating an inventory of tools that CLIs, namely, that have features that are kind of foot guns, right? Like

Dan: Yeah. So I was curious, we've already mentioned messy poutine, which is your capture the flag kind of project, if you like. But I want to talk a little bit more about that. If anyone listening to this is interested in maybe trying that. How would they go about that? And can you just tell us a little bit more about how they would get involved in that sort?

Francois: Yeah. So they would simply need a GitHub user account. They can create a different one if they want. In fact, I would, I would maybe in that case, it's not so much important, but you can have different personas kind of but in that case, this is really a victim project. It's meant to be. an exploitable kind of goat project.

If you're, you know, in the web, web app kind of traditional thing, like you have vulnerable web apps just for practicing education. So yeah, you just go to github. com slash messy poutine in one word, and you will find a number of repos and GitHub action workflows, and they're kind of like by levels, like level zero, one, two, three, some of them have I can interactions, but maybe one, one workflow might trigger another workflow as a side effect.

So you start to play with it, like open pull requests. Like maybe put something fishy in the title, in the body of the pull request. I'm just giving hints. Imagine some kind of bash commands maybe it's all right. Like you can kind of play with that. And the goal being to, to get the flag, you need to exfiltrate those secrets, which are only visible if you get remote code execution in the context of the runner.

In some cases. They are as easy as they've been put as environment variables. So if you get remote code execution, you dump the environment variables, then you get the flag. In other cases, it's much more complex because you need to do a memory dump, like dump the memory of the process to get those secrets.

Other cases, you need like much more advanced Like multi stage attack. There is even one that's a self hosted runner. If you want to practice Jonathan there is one that is a self hosted runner thing, which I designed as a host, a self hosted runner on top of a hosted runner. So it's like a recursive thing.

Anyway,

Dan: excellent. So I was, I wanted to ask you being someone who's been involved in security and and in open source for a lot of years. What kind of changes have you seen over the years as the job changed a lot as the kind of, you know, new threats and new solutions arise. So that was one part of the question.

The other bit, which is a really difficult bit of the question, which I won't blame you for skipping over is, do you have any predictions for the future of where all this is kind of going with this? These threats and vulnerabilities and things.

Francois: It's not the end of the year.

Speaker 5: Like, yeah, we don't do a prediction.

Francois: No, but yeah, I think the industry, you know, since I started my career, I, I've been involved, my, my first open source project was 20 years ago. Like, before the podcast, I went on and found my first message on a mailing list in 2004 with an OpenWrt project that was like a, yeah, Wi Fi dog, a captive portal that we were doing back then.

Yeah. As if by

Dan: magic, I happen to have a, a very old ,

Francois: oh my God. Like I remember, remember those guys short, shortening, shortening the pins and all that to like reset it and like, yeah. Yeah. Mm-Hmm. , that's fun. But yeah, I guess, you know, I, again, I mentioned that ask top 10, there's like the average developer working at an average shop, like writing code.

Is gonna be forced to do some kind of OS top 10 training, even if it's like pre cam boring training That is not very useful. But at the very least I think you can safely assume that most developers have at least heard what's an SQL injection, cross site scripting, et cetera. So it becomes so, and there's just more like secure framework with secure by design, React is kind of, you know, preventing cross site scripting by design, things like that.

So and ORM is preventing SQL injection by design, et cetera. So it's just, the industry is getting more mature for those lowing fruits. But it leaves the door like the attackers just are moving on to the next weakest link in that case, the supply chain is, is the new weakest link in the chain, literally, because it's getting harder to even as an attacker to not get caught.

Like there's, you know, more and more seems like you know, dumping a lot, lots of logs with web application, firewalls, et cetera, et cetera. But meanwhile, You know, build pipelines are just not as monitored in terms of their behavior, what's happening in there is, is some kind of malicious, like abnormal behavior.

So I don't know the other part prediction. Well, I think the prediction is that the. We're just starting to scrap the surface. Like as I said, I'm amazed at how many lowing fruits there are, but sometimes I catch no, not lowing fruits, but they're, they still require me, let's say 15 minutes to get past the non obvious part, but like.

You know,

Jonathan: like, yeah, interesting. Okay. So I've got a project. This, this is not a, not a contrived scenario. This is being big, being pretty transparent here. I've got a project that I suspect has problems and I want to run Putin against it to kind of get a handle on what the problems are. What's the recommended way to do that?

Francois: Yeah. So basically Putin, you can just. Get it, find it on github, boost security io slash poutine. You can run it as a docker container or with homebrew on mac or or you can build it like build it from source yourself with the, with go, and then you just need a read only token to get hub or get lab.

I would advise like to create a, like an on purpose, like just read only token just for that, that that, that use case and you point it to either the GitHub organization and it will discover all the repos. And that are public in that case. But if it's a token that has a visibility into private repos, you can do just as much, so it will automatically discover the repos and iterate one by one.

It will do a shallow git clone in a temporary directory. So it's very much batteries included, like, like the, the, the level of effort to get from installing puts in, or like just running it as a Docker container and finding. Obvious lowing fruits is very, very low. Because you just pointed to like the GitHub repo itself or the entire organization to scan the whole thing and return results within minutes.

I designed it with testing on large organizations like Microsoft organization has like 5, 000 repos or and. It takes about 15 minutes on 5, 000 repos to return like the, the list of results. So it's like, cause we do like shallow clone with just pulling the YAML. So And that's very efficient.

Jonathan: That's great.

Along with that, if I find something that is a real vulnerability, would you guys like to hear about it? Do you have like a form somewhere where people can say, I found this thing and it saved our bacon, thanks.

Francois: To report. You mean like if you've found something in someone else's thing or like in your own and you kind of fix it and like in my own.

And I want to,

Jonathan: I want to tell you guys,

Francois: thank you. Like, is there a tip jar somewhere

Jonathan: or something like that?

Francois: No, not, not at the moment. It's but yeah, we don't have any kind of Patreon or kind of get up and sponsor thing yet.

Speaker 4: But

Speaker 5: at the moment, But like, if you, if you come to Montreal, I'll buy you a poutine or you can buy me a poutine, thank me.

And I'll, you know,

Jonathan: I turn a favor with

Speaker 5: a beer

Jonathan: or whatever. Sounds great. It does boost security offer this as part of their services. Like, is there a, Hey, I want you guys to run poutine once a week against all of my repos. Is that, is that sort of a service that's offered or maybe that you're thinking about?

Francois: Absolutely. It is. It is part of our commercial offering. As I said before, our commercial offering is all about automating the orchestration of those static code analysis tools and aggregating the results. And as we did with other open source projects, puts in, it's just one more scanner that we can easily have our customers provision, get the results in one dashboard prioritize the results.

Yeah.

Jonathan: Yeah. Awesome. I was going to ask you to give us a 15 second commercial on what boost security is about, but I think you just did that. So, I have a couple, I have, let's see, three closing questions I always like to ask. Now, you guys have only had this as open source software for about a month now.

So this one may, this one may be a category error. But I'd like to ask, I'd like to ask Projects, what is the weirdest or most surprising thing that you've seen somebody do with the project?

Francois: Well, it's, it's, it's a bit too early maybe to tell. But and I think, well, where it is, I don't, I think when, one area that for those who are interested to want to play with the tool, try it there, there's something we added recently that adds a lot more flexibility.

We made it into like a pluggable architecture in a safe way. Like we designed it in a way that like you can basically configure. Other rules, like you can kind of bring in your own custom rules or edit, modify existing ones. And that could be used in a more creative way by someone saying, you know, like your existing rules, they don't find what I know is a problem, or I want to find.

Like, dangerous code patterns that are just like specific to my project. And it's possible using kind of the rule engine we've designed, like kind of a DSL, sort of that is specific to GitHub Action or GitLab pipeline at the moment, so. Yeah.

Jonathan: Interesting. Alright. And the final two questions that we ask every guest, what is your favorite text editor and scripting language?

Speaker 4: Mm-Hmm? .

Jonathan: Yeah.

Francois: VIM. Yeah. Vi guy. More yeah, never, never understood the emax. Sorry. And yeah, just like, it just, it is, it, it, it, it's already there. When, whenever USFH somewhere, it's very lightweight. And I know how to exit them. It's fine. And, and scripting language. Python, like if we talk to scripting, but I really like go, go, go Lang.

Like a Putin is implemented and go, but more a script, like to like rough, kind of quick and dirty thing. Yeah. Python is kind of go to

Jonathan: All right, excellent. Thank you. Thank you so much for being here and presenting the project. Very, a very neat project. And like I said, I, I, I think I need to run this because I have a feeling that we have, we have some problems.

So I appreciate you coming and telling us all about it.

Speaker 5: Thank you.

Jonathan: All right dan, what do you think? Do you have a Do you have some git or github or gitlab projects that need this?

Dan: Well, I I didn't think I did but no, you know, I probably do I probably do the more I think about it. Yeah, really interesting project.

I think it's great what they're doing and it's an area that's That is definitely growing I think, in security and so on.

Jonathan: Yes. You know, in the last few years, we've seen kind of these supply chain problems really start to be big deals. And I, I imagine, you know, I, it kind of, it interests me.

He says, when we asked it, you know, it, do we see this being a huge problem with an XZ style? He goes, I would be surprised if it's not already, and we just haven't heard about it yet. It's the sort of thing that keeps me up at night. I'm glad, I'm glad there's people that are thinking about this and looking at it.

And I am just, I'm thrilled that they decided to release this as open source. Get, get all of the eyeballs on it. And, you know, I'm sure, you know, If it hasn't happened yet, one of the things that's going to start happening is they'll get additional rules sent in from the community and it'll just, it'll make the project better for everybody.

It's great.

Dan: Definitely, yeah. And well, many, it's, what do they say? It takes a, it takes a village to raise a child. Child or something. I can't get that right now. I've forgotten the forgotten the whole saying but yeah The more people involved the better definitely I would say I

Jonathan: prefer I prefer the the linus torvalds version of that rule Which is giving enough eyeballs.

All bugs are shallow. I like that one. Oh, yeah.

Dan: Yeah That's that's a that would have been a better way of putting it. Yeah, I agree yeah, but it's fascinating project. Definitely.

Jonathan: Yeah. Yeah, it's great. Okay, dan Do you have anything you want to plug?

Dan: Yeah, very quickly. I mentioned it last time I was here, I think, which is a little while ago, but Liverpool make fest, which I helped to organize is a festival of technology and and arts and all kinds of stuff.

Lots of people building robots and. There's a guy making hot air balloons. Who's going to fly hot air balloons inside the building. I don't know if he's got clearance for that, but we'll find out. Health and safety is, is interesting for that event, but if you, if you're in the UK or you can get to the UK in, in July, the sixth is coming up it's in the, it's completely free to come along and attend and, and get involved.

There's workshops, there's all kinds of stuff. It's at liverpoolmakefest. org is the website. And that's the place for people to go in and have a look. And I think that's about it. Yeah.

Jonathan: Dan if somebody wants to follow you on The social media service formerly known as Twitter or Mastodon. Is there a good place to do that?

Dan: Yeah. At Method, Dan is my username, but if you go to dan lynch.org, which is my website, everything's listed on there.

Jonathan: Alright. And

Dan: I even managed to work out how to get a, a Mastodon feed embedded into my website, which Cool. Took a little bit of JavaScript, which I was really proud of, which is probably really insecure

But there we go, , which I'll now have to go and test.

Jonathan: Yeah. Oh, that's fun. All right. Well, you can follow my work over at Hackaday, hackaday. com. We've got the security column that goes live every Friday, which I think this episode of Floss Weekly will probably get a plug in the security column. That'll be fun.

We do not yet have a guest for next week. If you have recommendations or you're from a project and want to be on the show, it's floss at hackaday. com. Just shoot us an email and we will get you scheduled. Yeah. Yeah, other than that, I just want to say thank you to everyone that watches and listens both live and on the download.

We sure appreciate everybody, and we will see you next week on Floss Weekly.

Episode 783 transcript

15 maj 2024 | min

FLOSS-783

Jonathan: This is Floss Weekly, episode 783, recorded Wednesday, May 15th. Teaching Embedded with the UnPhone.

Hey, this week Rob Campbell joins me and we talk with Gareth Coleman and Hamish Cunningham about the UnPhone. That's a piece of hardware designed specifically for teaching computer science students about the world of embedded firmware. It's a lot of fun, it's open source, you don't want to miss it, so stay tuned.

Hey folks, welcome to Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. It is good to have everybody here and we've got a, we've got a neat show for everyone today. Now we've got a co host and it's Rob. Hey Rob.

Rob: Hello. Always exciting to be here.

Jonathan: Yes. I appreciate you stepping in and this is, it's kind of right up your alley.

I don't know. Honestly, I don't know how much Rob does with Floss Weekly. electronics fiddling, but I know you've done some teaching and that's kind of an interesting tie into what we're talking about today.

Rob: Yeah, I've, I've done a little bit of university teaching and I'm not sure if this, how, how this will fit in exactly with that.

We'll find out. And as far as tinkering with hardware, I, I, I don't do as much as I used to. It's all about time, but I love to get into it more again.

Jonathan: Yeah, well, this is a, this is a project that might pull you back in. It might actually be useful when it comes to teaching too. So we're talking about the UnPhone, which is a it's a little electronic device that is, it's open and it's, I think it's based on the ESP 32.

We will ask, we'll find out. I don't know all of these details. It's got a Laura radio in it, and it is all about. Being able to get students, I think, I think university students kind of into the, the, the realm of writing firmware and working on these little embedded devices. So let's not waste any more time.

We've actually got, we've got Hamish Cunningham and Gareth Coleman to chat about it, to give us the details. And so let's go ahead and bring them on. Hey guys, welcome to the show. Hey. Hey, how are you doing

Hamish: us?

Jonathan: Yeah, so I have, I have worked, I've worked directly with Hamish just a little bit on bringing Meshtastic to this device.

But other than that, I don't, I don't actually know a whole lot about it. So let's, let's start, let's start with this. We'll actually start with Gareth and we'll go to Hamish. So Gareth. First off, tell me, like, what, how do you fit into the project? And then we'll kind of let Hamish answer the same question.

And I think in answering that we'll also get sort of a 30, 000 foot view of what the project is.

Gareth: Well I suppose I helped with the electronics design of the on phone and the software and the course was more Hamish's responsibility. Although we do occasionally help each other out to the best of our abilities, which in my case is a little bit limited, but yeah.

I I know a bit about firmware programming, but, um, the other more complicated things often elude me. So yeah I helped with the hardware design, and then, you know, revising it and, and testing it, and, and also helping support the students. using it as part of their course.

Hamish: Actually Gareth is far too modest and he did all the hard bits and I get all the credit, which is basically how you want life to work as an academic.

That if it's, if it's going really well, that's, and that's the sort of arrangement you, you end up with. So yeah, Gareth's the electronic genius and I was teaching we've talked now for almost a decade, a course on the internet of things which we define. Basically as microcontrollers with network connections, so tiny little computers, pretty much the smallest single die unit of computation that there is very little resources and a network connection And we ended up getting students to build all sorts of complicated Wiring looms and nests of cables and so on and so forth and waving one at the camera Which most people can't see so it's a bit pointless, but Some people take to that and some people find that quite a challenge just on a physical basis.

You know, you require a lot of dexterity some basic knowledge about wiring and so on.

Jonathan: And a lot of, a lot of patience, a lot of patience and care and not breaking the wires off once you solder them on. Yes. I've, I've played that game before. Yes.

Hamish: And you take it home in your bag and it doesn't work afterwards and so on.

Gareth: Exactly, particularly for our students who are coming into the lab and then taking it home in their bags and back and forth. Yeah with breadboards that doesn't end too well often.

Hamish: Especially because our students are computer science students, so they haven't got an electronics background or most of them haven't.

It's a bit of a shock, you know, they're used to using the latest and the greatest of the desktop computing. So we put together some boards that integrated some of the common sensors and actuators that we would be asking them to build circuits with. And that really cut down on the necessity for me to look at an oscilloscope and wish I knew how it worked.

Gareth's definitely the oscilloscope guy in the partnership. But we teamed up with a company called Pimeroni, who are a little electronics manufacturing company in Sheffield. And they now manufacture the boards for us. And in fact, it's gone on a retail sale just this year, only about five years late, but Hey, that's also typical for an academic five years late is on time.

I think.

Jonathan: Yes. So this is the, I've actually got, I was sent two of these as a mechatastic developer. I've got one of them here and the other is off in the other room. This is the first version of the unphoned. It's the first retail version.

Hamish: It's version 9, actually. It's been 9. Gareth's laughing.

Jonathan: Yeah. I had a full head of hair before

Hamish: that.

Have

Jonathan: the previous versions been used in teaching? How far are we into the process of actually using this for the course?

Hamish: It was version 4 that we first used for teaching, I think. So it's been used for about 4 years now, I think.

Jonathan: Okay. And I'm just because I'm, I'm curious. I, I am, I stand in, in a lot of the things I do for day job and hobbies, I stand kind of at that intersection between the programming, computer science stuff and hey, let's actually work with some hardware.

How, how did the computer science students take to the idea of here's a soldering iron and here's real hardware and oh, by the way, this, this device that you have only has, let's just say we measure the RAM in kilobytes and megabytes instead of the gigabytes that you're used to. Like, that's going to be a culture shock to computer science students.

Hamish: It is a culture shock. You know, I'd be lying if I said everybody talked to it, but quite a lot of people. just love to have something physical. They love to have, they've spent years and years and huge amounts of learning time figuring out how to get a screen to do stuff. You know, everything happens on screen.

And that's basically it. You're developing an application of one sort or another, just doing code all day, and all of a sudden they've got something in their hands. Which performs a specific purpose. I mean, the other thing about the internet of things is we're not talking general purpose computing anymore.

We're talking about embedding electronics in devices, which have very specific uses like washing machines or cars or mesh plastic communication devices or whatever. So you've got a whole different culture. You can't just throw more Ram at it. You don't have the perfect SDK. Quite often we're programming in languages like C or C plus plus.

The usual thing we say about those is that, you know shoot yourself in the foot. You're definitely going to take a leg off in the process. And yeah, there are challenges, but I think most people, they rise to the challenge, they do really well. And it's lovely at the end of the course that they take away something physical and they can show them to be showing a whole different thing to, to their mom when they go home, you know, I built this.

Rob: So so you talked a lot about soldering already here. When somebody gets one of these, is it in pieces and you have to put it all together? Is that just something you do in class? Do you, do you just dig into it and start putting on

Gareth: it? The, the board is fully assembled by Pimaroni. So they do all the hard sort of soldering and it's just needs its battery and case screwing together in the box.

But it's got the option of sort of connections on the back. So yeah, you might not be able to see this on the radio, but there's a sort of expansion Board that you can fit so it turns what looks basically like a phone form factor, and it gives it sockets on the back that Expose all the regular pins.

So it, it kind of crosses both worlds. You can just take it out of the box and, Okay, you might use, need to use a screwdriver for a couple of minutes, but, It's pretty much fully assembled. And then if you want to, you can make your own electronics. You could start off by just plugging in Adafruit. Headers, Adafruit Featherwing boards.

But you could graduate to making your own circuits. So it kind of covers the whole spectrum of pick it up and just start using it. But then hopefully leads you into putting a few bits of your own electronics on as well.

Jonathan: So I, I find it fascinating the, the, obviously we're here to talk about open source.

I think the, the, the, the board itself, the young phone, all the hardware of it. Is that actually open hardware? Is that, is that available? So if somebody looked at kind of the, the ultimate of open hardware is if somebody wanted to go and send it off to PCB way and fab their own, they can absolutely do.

So that's, that's part of the project here, right?

Gareth: Yes, yes, it's open in that sense. Absolutely. I mean, we, we're quite good friends with Pimaroni and you know I, I think it's just, you know, it's not good manners just to pick up a project and just clone it anyway because you're not adding anything of yourself to it.

And I, I think an etiquette of, of that should apply to anyone, you know, if you just want to make a copy for yourself, well that's one thing, but, you know don't just start selling it out your boot without adding something to it or, you know, giving us some kudos at least and stuff. Right. Very

Hamish: much cheaper if you happen to be, you know and have the option.

I mean, I think the way that Adafruit's ecosystem of open hardware products works is, is really great and something that we kind of, we both aspire to and profit from. Because they produce maybe 50 different little add on boards for microcontrollers. They've got a family of maybe 10 or 20 feathers.

That's the microcontroller boards. And we use those with our students and we have the same ESP32. In the on phone and then they have feather wings that Gareth talked about, which are all add ons, you know, I've got a Neo pixel array or you've got din connector for your MIDI synth or this or that, or the other thermal cameras, all sorts of stuff.

And they all use basically the same pinouts and the same form factor. So we put a couple of sockets on the back of the on phone that follow that form factor. And then we can just, you know, plug stuff in there and program from, from the end phone. If you get your prototype developed in this way. And you're not using any of the, you're not using the screen, for example, on the UnPhone, then you can very quickly take the circuit schematic and send it off to PCBWay or somebody else or any competent hardware manufacturer and they can put it together.

You've done all the hard prototyping stuff without really doing a great deal of wiring and a great deal of That difficult stuff.

Rob: So be being open hardware, does that mean these, like the schematics and stuff for the hardware is available? Is that I read on, on and Garth has shaken and said, yes, it is available.

So I've also read that there's a 300 page textbook. And I, I know I've paid more for. Educational textbooks than, than this whole project costs. But

Gareth: yeah, especially

Rob: recently. Yeah.

Hamish: I, I spent a lot of time on that in lockdown. I assume that you're doing lockdown. So I basically wrote down the whole course that we were giving.

Yeah, that's also open. If you go to unfone. net you'll find all the materials. You'll find links to our GitLab. repositories and the hardware schematics and you'll find a link to iot. unfoam. net which is the which is the book and that book describes it describes the unfoam but it also describes a 12 week That's the course that we give to our third level undergraduates based on ESP32 and the Adafruit ecosystem and the on phone and all that good stuff.

Jonathan: I'm curious what tool, what tool was used for designing this? My mind has gone blank. There's a popular open source tool that I, I know the name of that will not come out of my mouth for whatever reason.

Gareth: Well yeah, it's a, it's a great tool. It goes by two names. KiCad and KiCad. That's the one. Yes.

I can't, I keep forgetting which one is preferable or better, betterer, but yeah, that's the one that I used for the schematic, and, and then Pomeroni actually tweaked the board layout. And they did their usual magic with the silkscreen. Personally, I think they've got the best silkscreens around.

You know so, yeah, we're very happy that they took that part off, off my shoulders, so, so to speak.

Jonathan: Yeah, so it's always, it's always nice to work with like for, for some of what I do with writing, you work with a good editor and it makes your job a lot better. So you've got kind of a PCB editor there at Pimeroni.

It sounds like it was a great relationship. Absolutely.

Hamish: Paul, Paul Beach, actually one of the founders of Pimeroni, Pimeroni is responsible for the Raspberry Pi logo. Do you know the, the kind of raspberry on the raspberry pie? Paul won the competition to design the logo. It's actually a raspberry combined with a molecule called a buckyball which was discovered by a guy called Harry Crotto.

He won the Nobel prize for it. And he was a Sheffield university guy a long time ago. So it's a, it's a kind of Sheffield story. Paul's a Sheffield guy. One, the logo competition invented the first ever case for the raspberry pie, which was a bunch of Perspex layers cut on a laser cutter. Yup. I have, I have access space, which was the beautiful, aren't they?

Rainbow, rainbow colored cases. And that was made at access space, which was one of the first maker spaces in the UK. And the fact that that was open and available to somebody who just walked through the door one day with a crazy idea. Was how the, you know, how the first Raspberry Kai cases came into being and how the company Pimeroni came into being.

So it's kind of a nice success story for open source on, on all sorts of different levels I think. Yeah.

Rob: So. Bringing up the Raspberry Pi there, I know the purpose of that, a lot of that was for educational uses. And, you know, this, the on phone here, also you're using it for educational uses. But, you know, with the Pi, many people have found many practical uses today.

Are there any, any practical uses for the on phone? Any great examples that, that you know of? That the on phone has been used for

Hamish: We could be nice to john here, can't we?

Gareth: Well, yeah, I mean i'm just i'm just booting up. I don't know if it's going to show very well on camera and again from some apologies for the people at home on the radio but we're running the latest mesh tastic firmware on it.

Oh, there we are. And and that's I think In some ways the kind of killer app outside of education that some people might find the on phones a good sort of device. Unlike a lot of the mesh tastic devices, it's a sort of fully formed device with made by a sort of prop not proper company, but you know a company with a pedigree of of quality and they they're 100 percent tested and, you know assembled carefully and all of those sorts of things.

So There should be less problems for people than you get with some of the you know, amazingly cheap enabling hardware from AliExpress and places.

Jonathan: Yes, that, that, that hardware is notorious for being incredible and inexpensive and a crapshoot as to how long it's going to work and how well it's going to work.

Yeah. Yeah. I was,

Rob: I was, I was here for the Meshtastic interview too, a few weeks ago or whenever that was. And, and that also seemed like an interesting project that I'd want to get into. And, and this on phone right here, it sounds like a one way to get started.

Jonathan: Yeah. So let's, let's talk about the the, the actual hardware specs.

What what all is in the UnPhone? So obviously there's a display. Is that a touchscreen? I see we have some buttons. There must be a LoRa radio in there. There's a battery of some sort. What, what are, what, what all's in the, in the little box?

Gareth: Well, I suppose I'll carry on taking that one a little bit.

So one thing to say is we've, we are using the S3 the ESP32 S3. So we've used that for the last couple of on phones because it gives a boost in, in performance. And it's quite useful to give the students two cores potentially if they want to start exploring. Running two cores in, in parallel.

It's also got, as well as the screen and the touch, yeah, it's got a nice capacitive touch screen, and a decent sized battery, 1200 milliamp hour battery. It's got a vibration haptic motor feedback thing. It's got an RGB LED. And one of the coolest features is infrared blaster

Jonathan: LEDs.

Gareth: So if you put it surreptitiously pointing at a, an annoying sports bar television set, for example, then it might be possible to turn it off.

Although I wouldn't know anything about that, obviously. And accelerometer and a gyroscope. sensor built in. And I think that's pretty much all of the hardware. And then of course that's the expansion, of course, pack.

Jonathan: Right, right. That's a, that's a pretty, that's a pretty impressive list of hardware.

You talk about the, the ESP, the ESP32 S3 being dual core and how that, that, that enables students. Boy, the thing that comes to mind is, Writing code for single core is easy mode. And when you actually have to worry about concurrency in your programming, that is when things become challenging. So you've got, you've kind of got this, this added benefit from the educational side of things that, We get to talk about mutexes and we get to talk about concurrency and how you can easily crash your program by trying to access the same thing from two different cores at the same time.

That's, that's, that's sort of devious and I like it.

Rob: That is quite a list of hardware specs in there with the infrared, lower radio and all that. Are we missing anything in here? Seems like it has about every little Gadget spec, you can add to that.

Hamish: A few, few versions ago, we, we actually did have a some other stuff.

We had a microphone and a speaker. And the reason we took them out was because You know, fans are great, but they're also they're also massively susceptible to surveillance abuse, massively sustainable, susceptible to being hacked. And the thing I guess that I dislike about them most is that they just, they steal our attention, you know, and they steal the attention of our kids, you know, my, my kid will happily sit next to me watching telly and, you know, half the time the, the phone just never stops buzzing and she's looking down and, you know, I want to kind of, I wanted a device that would do some of the wonderful things that phones do, but without, without stealing all our attention and without, you know, without listening to me all the time.

So we took the. The speaker and the the microphone out. You can add them back if you wish. We have the I two s bus there that we do use for talking computer projects. But basically if what you are developing is something that just has a microcontroller and uses things like SPI or I two C or I two s or straightforward GPIO to talk to the local stuff, and then he uses wifi or Bluetooth or Laura.

To talk longer distance, and we think we've got a good representative set of what you're going to need.

Jonathan: You, Rob, you asked what's, what's missing and there is one thing that I know of that we thought was in there and then kind of got disappointed when we discovered it was not the, the accelerometer does not actually have a magnetic compass and we, we got a little excited because that's, that's sort of a thing that would be nice to have in the Meshtastic project.

In fact, somewhere around here, I've got, I've got a little tiny chip that has an actual magnetic compass in it. And we're going to try to add support for that.

Gareth: So that's the one thing. I don't know what your soldering skills are like Jonathan, but I think there was an issue during the chip shortage with we couldn't get hold of the MEMS gyroscope chip that we had been using previously.

And so we had to switch to a different chip. And this one that was available only had six degrees of freedom. So we had to drop the The gyrus sorry, the, the compass. But I think, as a consequence, the PCB still has footprints for two alternative sensors. So, if you've got a hot air station and, you know, a steady hand, then you're welcome to try luck.

Alternatively, wait for the possibly the 10th spin where we'll try and put it back, you know, but yeah.

Jonathan: Do you, do you, do you make the poor computer science students do surface mount soldering? No, no. I mean, no, I think that would be a step too far. Yes. Yes. I have I have attempted it a time or two and I don't know that it's ever gone well.

Gareth: If you need, you need some special equipment, I think, you know, you need a good hot air station and, and certainly I need microscope. So yes. Yeah,

Jonathan: it's a challenge. I'm curious, where did the, where did the name on phone come from? Like who's, who's brainchild is that? And what all are we trying to communicate by that?

Hamish: I think it just came up in conversation one day. The problem is that it looks a bit like a phone. And people expect to be able to do all the amazing things you can do on a modern phone with a microprocessor and gigabytes of ram and of course you can't so Calling it the on phone was a way of saying, well, you know, it may look like a phone, but it's not, it's something completely different.

Jonathan: We do have, we do have a, we do have a live chat. I just want to throw this in real quick. It's going to go back a conversation, but David Ruggles friend of the show. He says, does, does Tim you sell hot air stations asking for a friend? They probably do, and you probably don't want it.

Hamish: Buy the microphone first, and see if you can still find the parts, you know, and then think about it.

But they're too small for me to see with my eyes, but yeah, a younger person might. I don't

Rob: think my hands

Jonathan: are stable

Rob: enough

Jonathan: for that anymore. If you do it right, your hands don't have to be terribly stable. You know, you have your solder mask and your solder paste, and you scrape the paste on there, you pull the mask off, and then the only thing you have to be sort of stable about is actually getting it in the right My problem is I don't ever have the solder mask, and so I'm doing everything by hand, and that doesn't work very well.

Hamish: It's a really nice way to learn about how this stuff is made actually is what we show the students videos of Pimeroni's pick and place machine. You know, and it starts off with the solder mask, it puts the paste onto the board and then it goes into the next bit and the pick and place is so fast now, you know, placing thousands of tiny little components on these boards.

Trucks on into the oven and blah blah. So it's a great way to learn what's under the hood In fact, that's what we tell students when they start the course. This is you know, you've used this stuff From the outside in the past now you get to rip the hood off The engine and have a look at what's underneath.

I think people appreciate that and I think to be honest We need to learn that stuff, right? We need to know that stuff. If if everything that we use in everyday life Is a product has some kind of mysterious component underneath it that's manufactured somewhere a long way away and that, you know none of us actually know how to replicate or are capable of replicating and that makes us much more vulnerable and much less resilient.

So that's, that's another theme that runs through our technology development and so on. I actually started working with Gareth on a completely different electronic control system, which was for a sustainable agriculture project. Sustainable agriculture technology called aquaponics, which is basically fish and vegetables in the same water.

And that's the same stuff really. You know trying to think how we can make our communities more self sustaining and more resilient. The times aren't getting any, any easier or more stable unfortunately. That's probably why the hands are shaking a bit when you're trying to do the surface mount soldering.

It's only going to get worse.

Rob: You I mean, you've mentioned the specs and the battery and all that stuff. I probably could do the math, but being like a phone, but not a phone, the question people always want to know is what kind of battery life does it get?

Gareth: It depends what you do with it. Yeah. How deep of

Jonathan: sleep you're willing to put it into.

Gareth: Well, yeah, and the back, the back life is the real energy hog. So that chews up 70 or so milliamps. So that's probably one of the big considerations. If you can switch the screen off quite a lot, then you can go quite a few hours. I think I've, I've tended to get maybe four hours, five hours, and running Meshtastic.

So that's obviously using Both radios a bit quite well 10 percent of the time at least at most. So Yes i'd say yeah, four or five hours isn't a bad estimate for a sort of typical usage

Hamish: I think that's probably with wi fi on right? If you if you've got wi fi turned off it makes a big difference.

Gareth: I think it uses bluetooth but not wi fi So, yes if you were using Wi Fi that also really can sap the battery down a lot, yeah.

Jonathan: That's a, that's an interesting quirk of a lot of the ESP 32 devices. They use the same antenna for both wifi and Bluetooth. So in the, in the mesh tastic project, we get people coming in all the time.

Why can't we have Bluetooth and wifi turned on at the same time? And it's a hard limit of these devices. I, I, I kind of suspect that there's almost a bit of software to find radio happening under the hood and it uses some of the same circuitry to, to Talk to both of those things. It was just like, you just, you cannot do both at the same time.

Hamish: Yeah, it's one of the reasons why expressive got popular very quickly. Isn't it? Because they did have a really good wifi and Bluetooth stack. Some of the ESP 32 modules actually have external aerial connectors on them. Although with ours doesn't The big, the big pleasure for me in recent times has been seeing MeshTastic using the LoRa radio because we originally built that for connecting to the Things network, which was as I'm sure you know, a big international effort to produce a kind of neutral infrastructure for Internet of Things telecoms.

Got a bit stuck along the way, I think, when they did a version upgrade and everything stopped working but We're not now using LoRaWAN in Meshtastic, obviously, but the LoRa radio seems to work really well. And where I'm sitting, which is admittedly on a hill, I can get pings from people, I think 60 miles away was, was the furthest that I've got so far.

Yeah, that's impressive. And, you know, we can start to do what we've always wanted to do with this device, which is to have a telecommunications system, which is completely off grid. Completely without external infrastructure. And that's something that I think a lot of people are Well, the success, you know, Mesh tastic Discord, 50 channels, guys, I mean, you know.

50 channels?

Jonathan: There's a lot of

Hamish: people out there who are interested now. It's a real pleasure to see that.

Jonathan: Yeah, that's true. I mean, so the, the, speaking of Meshtastic, the thing that really got me interested in it is, around the time I first heard about it, we had a tornado. It was a small tornado, it wasn't very powerful, but a tornado in town.

And I found myself, you know, out helping cut down some trees and such. And I had the thought then that, man, if this was worse and the cell phone networks went down, I would need something to be able to coordinate this stuff. And Meshtastic really seems like it could kind of fill that void in an emergency.

And so that, that is very fascinating to me. I, I think there's an interesting connection there between, you know, what Meshtastic does. You know, the, the, the possibilities it presents and what you guys are doing with the academics course in coming along and teaching people how to work with these kind of lower level devices and this idea that you talked about, that that's important for resiliency because, you know, People need to know how to work under the hood.

I think that's real fascinating and maybe something we need to be thinking more about.

Hamish: It's never been a better time to do it. You know, it's, it's both more necessary and easier. You know, the, the 20 years, 30, oh dear, 40 years ago, I was studying electronics. And the idea that I could just go out and build something like this, you know, it was complete fantasy.

You know, if I'd been working for a huge company, I had a massive budget and years to spend. Could have done it now. It's actually very feasible

Gareth: Yeah, and yeah, we can't just accept these black boxes handed to us by you know wizards and we we have to be able to open them up and you know Otherwise we don't own them, you know, if you if you can't control these things at all if you've got no way of Changing what they do or taking you can't take ownership of it, you know in a way, you know, you're just a user

Jonathan: It's it's It speaks to me because my journey kind of into the hacker mindset, if you will, started with sort of two things.

One, it was my inability to see into, at the time I had a Windows XP laptop, and my inability to see into how it worked on the inside. In particular, I was having to do reinstalls of Windows every like six months to a year, just to keep the thing running properly. And, The thing that annoyed me so much is every time you do a fresh install and then try to go to the C drive It would throw up this big scareware warning that warning you may mess up your computer by modifying files in here and all that irks me so much and then the other thing is I Managed to I don't remember what the hardware was.

This is before the Raspberry Pi So it would have been maybe like the first generation of Arduino So I managed to use an Arduino with a relay connected all to the computer and And then control a light in my bedroom at the time and the ability to get a computer to interact with the real world, like you say, and something beyond just the screen like that was, that was kind of a light bulb.

Nope, no pun intended, but turning the light on from the computer was a real light bulb moment for me because then, you know, then the, the, the. The coolness of the computer and the computer code then has an impact out on the rest of the world. And that, that was, that was, that was the juice. That was the good stuff.

And so I guess I've, I've kind of always been an IOT nut ever since then. And I, I love seeing kind of this combination of these two things where we have IOT and we have it open sourced. It's just great.

Rob: Talking more about the software side of things. You mentioned Windows. I'm pretty sure this doesn't run Windows.

And before the show, you did mention this, this doesn't run Linux either. So what, what is the operating system? What, what, what is, what runs on this?

Hamish: Essentially, if you use the definition of an operating system, it's something that does time sharing and multiprocessing and stuff like that. We don't have one really.

What we have is is a library a C library, which implements what's called real time operating system which is a much lighter, provides a lot less facilities, but nevertheless is very useful. Precisely because it does have concepts like mutexes and semaphores and things like that, and tasks and timers that we referred to earlier on.

And we use one called FreeRTOS which has been around for quite a long time. It was actually taken over by AWS at some point, but I think the original developers still work on it and it still has an open source license. We are the board. Yes. Yeah. Yeah, Gareth and I were having a conversation about Open source licensing and this whole controversy about the way that big companies don't necessarily pay for What they get from the open source ecosystems Which I'm sure is in in your minds as well, but FreeRTOS is is very good and That's what we have instead of an operating system like Linux.

Jonathan: I think it's probably fair to call FreeRTOS an operating system It's just extremely minimal. It's down there as thin as it can be. But it does give you a lot of those same tools that something like the Linux kernel would. But you don't, you don't have to use that. You can do, you can do quite bare metal programming with these things.

Do you, do you guys ever or have anyone actually do assembly language code on it?

Hamish: We, yeah, we did assembler at some point. If you want to mess around with there's an extra There's actually two processors in ESP32. One of them is a ULP, Ultra Low Power Processor. And if you want to talk to that, you'd need to talk in Assembler, or you used to have to.

I'm not quite sure you do anymore. But actually we're, we're, we're basically going in the other direction because there's been, in the last few years, a lot of interest in CircuitPython. which is a microcontroller port of the Python language, obviously. It's a lot easier to program. Okay. You're, you're taking a step away from the hardware and you're getting less efficient and so on, but you know, not everybody wants to, to learn how to create a a composite binary from their C program and flash it and so on.

So we've got CircuitPython going kind of sideways step is that we've also, A student called Zalan has ported it to Rust. So we've also got Rust running on the, on the thing. I've forgotten what the question was now. We were talking operating systems, weren't we?

Jonathan: Well, I, I, I asked about assembly.

I think, I think Rust is really interesting though. That sort of gives you, well, what Rust does is it helps get rid of the foot guns. That C and C are so known for. And yet with Rust, you don't have the performance penalty that you would for something like CircuitPython.

Hamish: Yes, exactly. So that's, you know, that's quite possibly the future of this sort of stuff if it gets enough momentum.

I guess the issue that you always come across in this kind of context is that underneath all that the manufacturer of whatever chip you're using puts a huge amount of effort into developing their own SDK. And Expressive are no exception. In fact, I'd argue that's one of the main reasons for their success is that they've developed a very powerful and full featured and extremely optimized SDK for their chips called IDF.

And that's pretty much entirely C. And even when you're doing the C stuff that we teach our students, that people are familiar with from the Arduino world, you're using essentially those facilities underneath, and in fact the Rust port will use similar facilities. So whether, you know, for a language to actually completely replace C slash C It's quite a big ask because, you know, it's not just the chip and that really heavily optimized environment that runs on it.

It's also the fact that that interface is seamlessly with the Arduino ecosystem, and that has all your integration code for your sensors, for your actuators, for basically everything that somebody's done with a microcontroller in the past couple of decades. So, yeah, we're always talking layers on top of layers, aren't we?

But I think, I think Rust is a very interesting development that has the potential to go all the way down if it gets enough momentum.

Jonathan: I'm curious when you, when you teach the students to get started with this, what is, what does the stack look like? I mean, so in, in the Meshtastic project, we tend to use the Visual Studio Code, Platform IO, and of course, almost everything is in C Is it, is it the same stack?

Is it similar? Or, Lord help us, are you guys still working with the Arduino IDE?

Hamish: We, we tend to say, we say we give them 300 pages and we say, look, there's a thousand ways to do this. If you get one of them to work, congratulations stick, stick with it if you're happy. And the Arduino IDE even, we even, we don't even use version two.

We use version 1. 8. And that's kind of day one. This is absolutely the easiest. way to get something running. You know, you don't even get syntax highlighting. You certainly don't get autocomplete, whatever. But it is robust and it does work with all the different chips and so on. But yeah, Platformio and for myself, it's the VI editor.

But other people do use this thing called VS Code, which apparently does some things, but I think it's just Emacs version 2, probably.

Jonathan: Yeah, yeah, that's totally what it is.

Gareth: Oh, partly because we support all these students and they're all they've all got their own setups. So, we end up supporting four or five different stacks, which is a bit of a headache, you know.

So there's, you know, VSCode platform IO in, on Windows and Mac and Linux. And then there's a couple of versions of the Arduino IDE, and subtle differences if you install it from the web's app store, or if you download it. And, and then there's the online Arduino IDE that some students have used. And yeah.

So I think, yeah, there's There's a lot of possibilities there and we, we struggle to keep up with all the permutations to be quite honest.

Hamish: Sure. I like to, I like to use Docker. If you, if, if you can cope with the command line, then you fire up our Docker image. You do this, you run platform here from the command line and it's really repeatable and predictable.

Rob: I was going to say, working with this stuff C C and C and Rust are all compiled languages, at least to my recollection, and VS Code, I'm sure, doesn't run on this device. So, generally, are you doing development In a different environment and then moving the stuff over or how does that work if you want to make stuff for this?

Hamish: So you've got a serial connection to your device over USB and you're flashing from the computer to the device. Historically that's been done by the IDE. So the Arduino IDE would allow you to do this and VS Code slash Platformio allow you to do that. More recently, Web Serial has become quite a good way to do it.

So browsers now implement a kind of sandboxed version of the Serial protocol that allows you to flash your device. And if you're on Windows and Mac and you're using something like Docker or a virtual machine to, to do the compile, Which means that you don't actually have to do the complicated installation of the SDK, etc.

then you pretty much have to use web serial because Whereas linux will pass through the serial devices to your container and you can flash from in there On mac and windows that typically doesn't work. So then you have to do your compile come up with a bin file, which is your firmware, and then go to the webserial, select the bin, and flash it to your flash it to your device.

And all this is quite error prone Some of the microcontroller boards use additional chips. What are they called, those chips, Gareth? The CP, whatever. Oh,

Gareth: the serial UART chips. Some of them have a That's it, yeah. Yeah, a chip in the way of the USB and the microcontroller and And sometimes that helps you with the flashing, and sometimes it gets in the way, and yeah, I mean, I, I see on, on, for example, the, you know, Meshtastic Discord, there's a constant trickle of people saying how do I flash this thing?

I've been pressing the buttons and turning it off and on again and wiggling around, and I was just like, argh, you know. And I think usually they well, usually they stop, hopefully, because they got their problem fixed, but yeah, it is a, just a constant and it's not just us that struggle with this stuff.

I was chatting to someone who was does a lot of quadcopter sort of, hobby work, and he's, like, constantly struggling with getting firmware onto these things and getting them into DFU mode and I think, actually, a large percent of the population, probably, nowadays, have, Kind of struggled with updates of some sort that you click on something and it says the update process failed Try again later, and it's like oh what you know

Hamish: Welcome to the joy of embedded development.

Yeah

Jonathan: What year of university are students usually in when they when they take this course their third years in the UK? They're about to finish their degree. Usually. Okay, is it is it a single semester? Does it last all year? It's a single semester. How do you fit everything into a single semester? Isn't that, that's gotta be always the challenge.

Hamish: It is difficult. So we spend six weeks playing with microcontrollers and then we spend six weeks on a project. When we, when we started giving them the on phone and similar devices. So we also, we, we give different options. So we've got here a Lily go watch from. TTGO TTGO Watch, we've got a thing called the ESP Box from Espressif, which is kind of good for doing your Alexa like kind of things.

We, when we started giving out that kind of thing, like the on phone that's, that's pretty much a finished circuit extendable, but, but basically finished in its, in its basic form, then they, they, they start being able to do projects that they can get done in the space of the course. One of the ideas about the course originally, which has worked pretty well, is for them to be able to take the project hardware away with them.

And then they're sitting in an interview or whatever getting the, getting a job after college and they can, they can demonstrate some stuff, some physical stuff that they've done, instead of being reliant on the same old, Hey, I built a database backed web system you know, old. Yeah. So that's worked pretty well.

It's an advanced, it's an advanced course and not everybody takes to it, it's the truth, but yeah, everybody gets a bit of a flavor.

Jonathan: One of the things you guys mentioned in the notes you sent to me was a question about why you didn't get into microcontrollers in the 70s, the 80s, and the 90s. And thinking about microcontrollers way back then, there wasn't a whole lot.

I know there was things like the basic stamp in the 90s and some other things. But I'm curious, what's the story? What did it look like way back then and why wasn't it as appealing?

Gareth: Well, let me tell you a story, my boy. It was just so different. I mean I've always gravitated towards digital electronics rather than that tricky analog stuff.

But there were just so many barriers, you know, and, and like Hamish was saying, unless you were a, a big company with a big budget, you just couldn't even get started. The compilers were expensive the other bits of the tool chain development boards were expensive, and they were aimed at people working for big companies.

They cost a thousand dollars, you know, a thousand pounds easily, you know. And so I kind of bounced off a couple of times just unable to cross the, these sort of barriers and get into developing with microcomputers and then microcontrollers. And then a bit later on in the early 2000s, I had another kind of go at it, and I must have spent two, maybe three days trying to get, GCC embedded compiler configured and the linker configured and I think I gave up without ever being able to compile Hello World.

And I came along about three years later and I had an Arduino board, one of the first Arduino boards. I loaded up the IDE. And I wrote, like, Blink, and flashed it, and 30 seconds later, the LED was blinking, and my mind was blown. So that, the long answer is Arduino and open source basically were the enabling things.

Because the microcontrollers were cheap, as, as they always have been, and the little board was cheap. Cheap, but what was different was that the toolchain and the IDE and some quite good Educational resources were available even in the early days people were really keen to share blog about the projects They'd done there were lots of code examples and things So yeah, it was the Arduino that was the gateway drug to microcontrollers for me Yeah So yeah, and that's obviously built on the, the free GCC compiler and, you know, all the other good it's a, it's a great open source success story and it, it's turned hundreds of thousands of people onto electronics, I'm sure.

Hamish: I love the fact that it came from art students in Italy as well. Yeah. That's where the Arduino started.

Jonathan: Yup, yup. It's funny though, that's part of my story too about getting into all of this was again, working with the Arduino. I think it came, I came along at just the right time. I, I know some people get started, like I mentioned the, the basic stamp and that was available in the nineties.

But I think it, I think that, that must have had a, a harder on ramp, like the, the learning curve must have been steeper and it was more difficult to get started with it. Yeah,

Gareth: I missed that one, but yeah.

Jonathan: Yeah. Well, I did, I did too. I don't have any memory of using it. I just, I know, I know that it existed.

You know, it's kind of funny. We wonder sometimes, well, why, why is so much of embedded using like this Arduino framework? Cause you, you're, you're running on things now that have nothing to do with Arduino and yet you're using the Arduino headers, you know, some, somewhere there's Arduino that H getting included and you, well, why?

This is why it's because so many people got started on it. And Arduino was one of the first to really do a good job with it. So it's, it's stuck for better or worse.

Hamish: That's something we tell our students is that The speed with which you can develop is dependent. not just on the device you're using, not just on the SDK or the IDE that you are using, but crucially, it's dependent on a, a massive community of people and the e ecosystem, the code ecosystem that has grown, grown up around those enthusiasts and those people who have contributed their work to the to the open source code that's available.

And for Arduino, you, you still can't beat that, I don't think. Yeah,

Jonathan: yeah, absolutely All right. We are headed towards the bottom of the hour I do want to ask is there anything that you guys wanted to talk about or let folks know about that? We did not ask about

Hamish: We haven't talked about raspberry pi very much, but I think at least in the uk raspberry pi has made a huge contribution to Changing the way that we've been able to educate so when Hebdon Upton tells the story, he says, you know, back 10 15 years ago, undergraduates were turning up without any real hands on experience of computers, and that was because your home computer was an expensive machine, it ran an operating system which was easy to mess up.

Mentioned no names Redmond. And your parents, parents wouldn't allow the kids near the damn thing. So the Raspberry Pi has really changed that. You've got a fifty dollar computer, you can stand on it, break it. Um, hopefully it belongs to somebody who's, you know, lending it out to the school. We used to run We had a, we had a, what we called a pie bank some years ago.

And we'd go to schools with a whole bunch of raspberry pies. We'd give them out, let the kids play with them and so on. So that that's made a huge change. And I think we're profiting from that.

Jonathan: Yeah.

Hamish: That's another of the things that has got easier. One of the things that's really got harder and I really feel for the little companies doing electronics manufacturing is that I think the, you know, in the pandemic we had chip shortages initially because factories were closing.

But I think that's gone on and I think it's now actually, it's become a kind of not exactly standard practice, but I think people are hoarding, you know, the big companies are intentionally buying up every MMU of a particular type because they can see that, you know, then they'll be able to build their phones and their competitor won't be able to build theirs.

So, you know, that's, that's made it tougher and, you know, these, these things tend to go through more iterations now, smaller, smaller runs because you can't get the devices and you know, hopefully we're going to come out of that one of these days. Those are two things that I think we haven't covered yet.

Jonathan: Yeah, very interesting. All right. I do want to ask each of you. Okay, let me let me ask this. Let me ask this before we let you go What's the most surprising thing that a student or anyone else for that matter has done with the on phone? What what project what bit of software or hardware on there has surprised you the most?

Hamish: Rust made me fall off my chair. I must admit seeing that But um also we have a, we have a big, in our students union, we have this big nightclub with massive quantity of really high quality lighting in there and light effects and stuff. And this year two of the guys did a control system for that.

So you can, you can walk into the, into the, I'm, I'm, I'm really going to try this myself as soon as he submits his work. You know, I'm going to put the controller on. I'm going to go in there one night and just start running the lights myself, I think. That's,

Gareth: that's pretty cool. I think a student did a Etch a Sketch, which is such a simple idea that, you know and you know made Gareth on his screen and showed me.

I think that was the thing that tickled me the most, let's say. Yeah, that's fun.

Jonathan: All right. So two final questions for each of you before we let you go. And we can, we can start with Gareth and then go to Hamish. What's your favorite text editor and scripting language?

Gareth: Ooh. Well shamefully, I've got quite fond of the one in VS Code because I like, I like the pretty colours, and being able to click on a function and jump to it, so it'll have to be that one sorry Hamish and sorry, what was the second question?

Scripting, favourite scripting language. Oh it's gotta be basic. Basic, old school, you know, but yeah, I've got a soft spot for it. I don't use it anymore, but you know, definitely my favorite.

Jonathan: I, I got, I got my programming start with all the QBasic from Microsoft way back in the day. So the only way is up, you know, and then how much same two questions.

Hamish: Tip of the hat to Bram Molyneux, sadly died a year or so ago developed Vim. Porta, the VI editor, which is my tool of choice, very fine piece of software. Um, and scripting language has to be Bash, I think, the Bourne shell. I'm one of few people who's written a fairly functional web server in Bash, which I think, you know, probably go on my gravestone.

Jonathan: Probably lead me to my grave, actually. That is, that is definitely saying something. Is that source code available? That sounds extremely fascinating to look at. Or is that something you just dope show to everybody? A lot of my, a lot of my, a lot of the code

Hamish: that I write for the course Yeah, I need to hide it, it's so bad, but, yeah.

We can talk offline, as they say. That's fun.

Jonathan: That's fun. All right. Thank you gentlemen, both of you for being here and thank you for the UnPhone. It's a, it's a great project. Thank you so much.

Gareth: It's back on sale. Yes, hopefully some more

Jonathan: people get to share it. Yeah, there you go. All right. Rob, what do you think?

Have we talked you into it? It's back on sale. You can go get one now.

Rob: I did look it up on their shop and I think it was about 139 pounds. Convert that to US, I don't know, a few more, a few more dollars, but Just a

Jonathan: few dollars more?

Rob: Something like that. I don't know. I don't know the math. I'm not here for math.

But it's definitely something interesting to look into. Just because even if you think of all those extra things, if you were to get a pie, a raspberry pie or something like that and add, All those extra infrared and all those lower radio year and a screen year, you're right around the same, about the same price anyway if not more.

So I'm definitely, definitely interested in, maybe that's going to be my way to get into Meshtastic also, which I kind of mentioned early. Yeah, there you

Jonathan: go. You could next time they ask you to teach a course over at the local college, you could use it for that too.

Rob: Yeah, it doesn't fit directly with the classes I have taught, and I don't know that I have the pull there yet, just as an adjunct to get them to add a new course.

But, Someday, I'm kind of hoping maybe that can be my twilight year job or my retirement job. And, you know, maybe I'll have some more pull there someday.

Jonathan: Yeah, yeah, it's fun. All right. Yeah, I very much enjoyed talking to this project. Rob, do you have anything that you want to that you want to plug?

Rob: You know, just you guys can come connect with me. My website is robertpcampbell. com and there's links to my Mastodon social media there. Otherwise, also come and listen to me and Jonathan at the, on the Untitled Linux Show every

Jonathan: week. All right I do want to let folks know that the current plan is next week.

We're going to talk with the, talk with the guys behind Poutine, which is a security scanner from Boost Security. They recently released it. They released it as open source and it sounds pretty interesting. So we shot an email off to them and they, they said, sure, we'd love to come on the podcast and talk about it.

So that is currently the plan for next week. If you know a project or are a project that you think should be on the show, Shoot us an email. It's let's see, floss at hackaday. com. That's the one that'll get to me. And let us know about it and we will try to get it scheduled. We are always looking for guests, always looking for new projects to cover.

And then as for me, the only things I really want to talk about is. Of course, Hackaday, the home of Floss Weekly now, we appreciate them for that. It's also where my security column goes live on Friday mornings and a few other things happen from time to time. Yeah, make sure and follow us there. We, we sure appreciate everybody listening live and those that get us on the download and we will see you next week on Floss Weekly.

Episode 782 transcript

8 maj 2024 | min

FLOSS-782

Jonathan: This is Floss Weekly, episode 782, recorded Wednesday, May 8th. Nitric, in search of the right knob.

Hey, this week David Ruggles joins me. We talk with Rack and Steve, both from Nitric, a company that makes an infrastructure from code solution. It's all open source. It lets you write your applications in just about whatever language you want to, and then automatically provisions your cloud infrastructure.

It's really powerful. It's really neat. You're going to want to check it out. So stay tuned. Hey, it's time for floss weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And we are glad to have everybody here today. We've got an interesting show. Of course, it's not just me.

I've got David Ruggles in as the co pilot again. It's good to have you again, David. Thanks for stepping up and volunteering to co host. It's good to be here. So, We have today the company Nitric, and we've got Rak Siva, who is one of the founding team members and the vice president of engineering. And we've got Steve Demchuk, who is actually the CEO of Nitric.

And these guys do I want to say infrastructure as code, but I have been told that that's not exactly right. They do infrastructure from code, which makes all the difference. I'm told we will ask him about exactly what that means. Is this something that you're familiar with, David? Not yet.

David: But I'm excited to become familiar.

So I've. Some DevOps work. I've done some cloud hosting, nothing automated. I haven't even gotten into infrastructure from code or as code yet at all. So hopefully I have some questions that are above baseline, but they won't be very detailed.

Jonathan: Yeah, we were talking about this before the show, both David and I, we, we do, of course, sysops, we've done DevOps, but neither of us have done a whole lot of like cloud native work.

And as, as I, as I joked before we started, I built my own servers so that I didn't have to put things on the cloud. And so maybe part of this is going to be a rack and Steve trying to convince us that the cloud is the place to be. We'll see how that goes. All right. Let's not let's not waste any more time.

Let's bring them on. And Hey, rack and Steve, both of you, welcome to the show. Thank you.

Rak: Thanks, nice to meet you.

Jonathan: So, let's start with our patented and well known 30, 000 foot view. And maybe let's let Steve take this one as the CEO. Why, what's, what's the, what's the juice at Nitric? What's the what's the big tool there?

What's the reason that someone reaches for what you guys have?

Steve: Yeah, so thank you. Yeah, Nitric's open source multi language backend framework. And. It really truly is about letting teams scale DevOps without the complexity and maintenance burden of, of cloud development all by yourself. So kind of what you guys were talking about infrastructure as code and infrastructure from code, the, what we're doing is.

actually automated automatically provisioning your infrastructure directly from your app code. And then you can, the DevOps team can actually build customizable guardrails to actually help with that. So imagine a world where you're really focused on your app code. And you pick the cloud you want to to provision to, and it works.

Rak: Yeah, what's really cool about this is that Nitric lets developers actually focus on their application code. So it gives you a runtime that works locally, immediately. So from day one, you can start rapidly iterating your projects. But you're also ready to deploy that code directly to the cloud on day one as well.

So I'm not saying you're going to have your solution done in a day, but you're ready to run locally and in the cloud immediately.

Jonathan: So what kind of apps are we talking about? Are these like browser native applications? Are we talking mobile applications? You know, can you run an Android application and then use Nitric to set the backend up?

How deep, where do all the fingers of this stretch out to?

Rak: Yeah, it's typically the backend of any type of application. Could be a mobile application, could be a web application. As long as it's running on the backend side of things, that's where we play the most.

Jonathan: Okay. And, and then like what, what clouds, you know, what providers can we, can we plug into and how does that plug in work?

Rak: Yeah. So we have native support for AWS, GCP and Azure. That's, that's part of the framework today. But as you'll learn, like the, the idea behind nitric is that you can deploy to any cloud. It just, it's a, it's a pluggable sort of a framework. So you can additively extend to Oracle or to other clouds out there as well.

Jonathan: Is there anything built into it to to prevent the dreaded 10, 000 bill from your cloud company?

Rak: Yeah, definitely. So again, we'll talk about it a little bit more in detail later, but, but we do provision resources with the best practices and least privilege required to, to connect up resources to each other to avoid the dreaded 10, 000 bill.

So. Can't guarantee it. You can always, you can always find ways to work around it, but we do our very best to make sure that, that, you know, you can't accidentally get into that situation by deploying the nitric.

Steve: Sure. I think, I think one of the key things is as we work with, with users of the open source framework, and as we've worked with folks going into production, a lot of that, you know, it truly is architecting for the cloud, being knowledgeable about what infrastructure you are provisioning and why, and making sure.

You're only provisioning what you need versus maybe what you, what you wrote. So a couple of the teams we've worked with have told us they've saved up to 60 percent on their cloud build. And you know, a lot of that times it's, Hey, they, they, they decided to use the right compute based off of their, it's a long running service or it's a dynamic service.

But a lot of that too, is you're only spinning up what you need instead of leaving out there of what you've created from the past.

Jonathan: Yeah, that makes sense. So I'm curious about the kind of the open source angle on this, how much of the, how much of the tooling, how much of the, the offering is open source?

I'm like, what's the, what's the model? How do you guys pay your rent and buy your groceries? I

Steve: mean, as, as a, as a, Early startup. That's like every day, what you're thinking about obviously the open source piece came from, this is a new paradigm. It is something that's new. We ourselves are huge fans of open source.

We use open source in our projects with, with very large banks, with our own SAS companies that we've been entrepreneurs with. So number one, it was, it's the right thing to do to make this open source. And we knew if we're going to keep up with all the cloud providers, you know, big part of what we're adding here.

Is really a, but comfort as things change that we can actually move fast and take care of those changes for you. That's going to take a community. The other part of that is we want people to extend it. The fact that we've chosen it to be cloud agnostic, we've chosen for it to be multi language.

It's kind of a huge principle of ours to meet people where they are in their preferences and not just say, we're all in on language X, so you must be too. So it really came from. I would say it was never even it was everybody saying yes as the founding team. From the how do we make money part I would say really modeling as people are going to, to production there, they usually ask us for architecture consulting, for licensing and support.

So it truly has been licensing and support of the framework is the monetization model that, that we've seen start to work really well. Yeah. So that's how we're doing it today. Right now it's a hundred percent open source. We don't have. Any features behind, behind any gates.

David: Awesome. So when you talk about licensing specifically, I assume it's where you've got customers that are concerned about open source licenses poisoning their code, so they want a non open source licensed version of this?

Steve: That has not happened yet, David. Actually, I, I do anticipate that happening maybe someday, but it's been more. Hey, we've gotten really far and we love what's happening. Will you support what, what we have? Can, will you do some custom development with us? And if we ever write something with them then that we are going to ensure that and make sure, you know, support it with them.

So we have not had a fork that's been proprietary in any way, shape or form. Yeah,

Rak: yeah. I think I'd add that it's a trust building exercise as well. So. We're taking responsibility of a very important piece of the cloud deployment story. And if you abstract that away and you, you make it opaque it's very difficult for people to trust you.

So this way, you know, we're saying this is what we do and you can inspect the code and have a look at what's happening under the covers if you want to.

David: So I've been digging through your docs as we kind of talked about at the beginning well, first, I wasn't aware of you at all until yesterday. So this is exciting. But I don't have a whole lot of experience with the infrastructure from or as code either side before now, but your integrations rely on Pulumi, if I'm pronouncing that correctly.

Yep. And so maybe I'm getting a little bit ahead, but hey, this is one of those open ended questions. Why would I use Nitrux instead of just writing it directly in Pulumi? Because as I dug down into it, it looked like it was also cross programming language.

Rak: Yeah, yeah, I'll fill this one. Basically, you need two things to deploy to the cloud.

You either need, I mean, you need your application code, which, you know, you're always going to need that, and then you need a runtime configuration. And the runtime configuration is what tells AWS or GCP what componentry you're going to be leveraging when your application runs, and how to have it configured so that they can interconnect and talk to each other.

So day one, let's say you're starting from scratch. You may go into the console, the AWS console or the GCP console, and you'll manually configure all of these resources. So you'll say, oh, I need a storage bucket. So I'll enable that service. I'll configure it. I'll set up the IAM policies, the permissions required for my Lambda to access that storage bucket.

Right? So that would be one way of doing it. And that's very manual, but it's not repetitive. So each server you go to, you're going to have to apply those same configurations. So enter IAC, which is a way of either programmatically or via configuration, applying the configuration that's required for you to provision your infrastructure.

ISE tools are like Pulumi, Terraform, CDK, that sort of thing. Ansible, yep. Exactly, yep. And so what that brings to the equation is repeatability. So now you've got a script or, or a piece of code that you can repeat across different environments or different projects. But, but what you end up with is you've got your, your application code and you've got your IAC project.

Two separate siloed pieces pieces of code or projects. And the link between them is actually manual. So let's say you, you, you, your first version of your code and you've got a bucket in it. But then you decide you're gonna, you're gonna use a third party bucket instead. So you don't need to provision that bucket with your IAC.

You're going to have to go into your other project and remove the IFC as well. So you're manually keeping both projects in sync. And that's quite, like, complicated when you start to scale. Right, so So what IFC does is it's an infrastructure from code. It creates a bridge between the two. So instead of manually saying, Hey, I changed my application code.

Now I have to change my IAC. What we do is we infer the requirements, the runtime requirements from the application code itself. So when you declare a bucket in your application code, we create a resource specification that kind of maps all of the, the resources to, to each other. And then we can use providers to fulfill that request and the providers in the IFC world.

are actually IAC, so we're complementing where we're using modules of IAC to implement your bucket each time you make a request for a resource. Does that make sense? I think so.

David: I'm following. Okay. You did mention Ansible. Ansible is something I've used and I'm familiar with, so I can actually ask a potentially more intelligent question around that.

Is Ansible something that you can run on top of? Is Ansible a backend that

Rak: So in, in our current versions, we don't provision with Ansible, but we could there's definitely the, the potential to, it really depends on the community and how, how involved they want to get in that space. We've chosen Pulumi as our provider of choice, just because it is language agnostic, as you pointed out, does support, you know node Python and, and a variety of languages go but we're also working on providers in Terraform as well, and we can also extend that to other tooling as well. So it really just depends, I guess, on how far we want to take the community support.

Steve: And I will say sorry, just to jump in real quick. Pulumi has been great partners of ours. If they have an automation API that really made this first version of kind of the proof of concept a couple of years ago, when we was, this is a really cool idea, can we actually make it work?

So I would say a huge to the Pulumi team because then providing that gave us a great way to get started and think about it from a developer's point of view of. I actually want to do this in Typescript, or I actually want to do this in Python. Versus yeah, Terraform really is the de facto. I mean, I would say it has, obviously, the largest market share.

But that you are learning a totally different skill set of learning HCL. Which, obviously, for certain roles, that's preferred. Just like, maybe for you, David, it's preferred to use Ansible. So, you're kind of hitting, you know, Why we like what we've done with nitric is if there's a community around that and there's people want to do that We actually built nitric to support that style in the future We've just chosen that proving it out that it's super valuable to kind of go where where our passions were and now we're spreading out To where our compute community passions are

David: makes complete sense.

I was just wondering like if if you've got somebody that's a startup, whether they're starting a new application or they're a complete startup, then it's a lot easier. But if you're trying to integrate it with a company that already exists, and they've got a large Ansible infrastructure already in place, instead of reworking all that, if there would be a way to tie those together, and I think that's the question that basically got answered there.

Yep, exactly. So I think Jonathan had some further questions about licenses and such. I do.

Jonathan: Actually, before we get to licenses, I'm, I'm real curious. You mentioned that Nitric actually looks at your application and kind of fills in the blanks for you. Is that like, is it just static code analysis or is there like a, is there a Nitric library?

I mean, I'm just kind of curious how the, like the actual nuts and bolts of this works.

Rak: Sure. Yeah. Now you've, you've noted it's a, it's a Nitric framework, so it's an SDK that you use. So basically you're declaring your resources. If you wanted to declare a bucket or a topic or a queue, you'd import those resources from the Nitric SDK.

You'd configure them. What we require is for you to give an intention. So let's say, let's say, let's keep on the bucket topic. Let's say you were declaring a bucket. Typical actions against the bucket would be read, write, delete, to, to delete objects from the, from the storage. You could also associate a non event handler.

So that when someone wrote to the bucket or read from the bucket or deleted from the bucket that a separate service would be triggered. Those things are the flexibility of the SDK that you'd be, you'd be using with nitric. So so you could write out those handlers and you could write your application logic to interact with that bucket as well, using the nitric SDK.

Jonathan: So you say, Hey, Hey, nitric library, I need a bucket. I don't care about the details of it. I just, I need a bucket somewhere. This is what I'm going to do with it. And then you have the, you have this object that you run around with and you do this stuff in your code and you don't know if it's going to be on Amazon or Azure.

It just, you guys make it work.

Rak: That's exactly right. Yeah. You're delaying the decision on where it's going to actually end up. So what we do is we build a resource spec from that saying this is what they want to do with that bucket. This is the level of permissions that we need to grant for that bucket because they want to read, but they don't want to write from it.

For example. And we create that specification and the second part of it at provision time is fulfilling that contract

Jonathan: Yeah, that's that's cool. I can see I can begin to see how that would actually work. I

David: I was digging into the docs around that because one of the things I noticed early on Was secret management because that is a challenge.

You know people commit secrets into their source repositories so often. And that's something that Nartrix did, where you could have a bucket that you keep your secrets in and you're specifying right to the process that stores the secrets, but then everybody else is only allowed to read. And that's that's from 15 minutes of doc reading there.

So that's the extent of my understanding. Yeah

Rak: You've got it. Yeah, we we basically picked foundational components Which we've recognized from projects that we've done in the past and deployed for for big fintech organizations So, you know, you typically need storage buckets. You need queues you need topic subscriptions You need scheduled tasks or delayed events.

These are the foundational building blocks that a lot of APIs and applications require, so they're available to you immediately with Nitric.

Jonathan: So Maybe a question, a series of questions for Steve. Let's start out with the easy one. What what License did you go with for this? You know, what open source license?

Steve: Apache 2. 0 for this one.

Jonathan: Okay, and then have you gotten Outside contributions? Are people out in the community excited enough to start pushing code?

Steve: Yeah. Yeah, what's exciting for us? I would say We want a lot more. So whoever's listening, we would love more help, but it's, you know, it's, it's been more around language development.

So we have we had to request earlier this year to actually start working with Dart. And which is really exciting for us. And you were talking about the mobile component and being able to work with that community. So it was really fun to say, yeah, that's exactly why we built this. We just put out our V1 release and we'll talk, we can talk about some of the features there, but just focusing on the contribution side with that we're actually co developing now with go so that we, you know, we really want people to be Okay.

So the first thing is, we're really proud to be working with the Go community to help us with the Go SDK as well. And you can imagine, our team has a lot of expertise in languages, so we're really comfortable when it was with JavaScript and Python, but when we're working with folks that are all in on Azure and also still use C Sharp, our philosophy on these things is We'll get it started.

We'll create like the structure, but we, we really want the developer experience to be fantastic to the language that you're, that you love. So we do rely on community feedback and and welcome the contribution side there. So yeah, so early days for us on contribution, but we've loved the, we've loved the, the contributions we've had.

Jonathan: Yeah, that's cool. So the, kind of the direction I want to go with this as do you have a, do you have a CLA or a copyright grant for contributors?

Steve: It's funny. We literally just put that on the board yesterday. So we are with, I was just talking about this go project that we're going to start we we will put that in place for that one.

So for, you know, for everything, but to get that going, and I'm curious from your perspective. Can you, I would say that's kind of a requirement for us to get going forth. What, what, what are your thoughts on that?

Jonathan: It's tricky. So, one of the friends of the show, Jeff Geerling, just put out a video the other day about, oh, I forget which project it was.

It was something Red Hat was doing, though, I think. And they, they had a CLA in place, and they used that CLA to make a licensing change. And can't remember what company it was. I don't think, maybe it wasn't Red Hat. Anyway, so he, he makes, he makes the comment that, you know, if you're gonna do business with a company and they have a CLA, maybe you should think twice about that.

And it's like, well, I, I don't know. Because I could see Let me put it this way. I'm in, I'm involved in a project where we have a CLA, and when you ask why, it is, it is essentially so that if a problem comes up with licensing, it gives the project the flexibility to be able to fix it. And some people have handled this by having, you know, GPLv2 where, you know, if, if the, Free Software Foundation comes out with another version of GPL, then kind of all of the code falls through to that automatically.

And that's one way to handle that. But then you kind of have a lot of trust in the Free Software Foundation to not do something really crazy for GPLv4, right? So I, I'm mixed. My feelings are mixed about CLAs. And, and so I understand, like, on the business side of it, It's, it's sort of something that needs to be there, particularly if you want to ever do dual licensing, you just, you just have to have control over that copyright to be able to do it.

At the same time, I know a lot of people are beginning to get a little gun shy about it. So it's a complicated issue. I, I do not have the answer on it.

Steve: I'm actually glad you brought it up because that gives me a little more of maybe that's something we need to talk with our community about a little further and see, see where people are at on that.

Jonathan: Yeah, I think, I think maybe the one thing that I would say is if you're, if you do a CLA, be Be absolutely upfront and honest about why. And so, you know, you want to, you want to assure people that you're not going to pull a Terraform and completely relicense to a source available license. Cause nobody wants that.

Nobody ever wants that except, you know, the, the guy's in suits making the financial decisions. That is the

Steve: opposite of what we want. Right, right.

Jonathan: But if you know, if you're, I would say from my, from my perspective, if a business is just upfront and say, look. We want to be able to dual license this code so that we can one day sell it to a big company and they do not get, you know, this poisoning problem.

I think the community is going to be reasonably okay with that. And I, I don't know, maybe there, maybe there could be a CLA designed that would actually prevent a like a gross relicensing of everything. Maybe we need to do with what we, do what we did with open source licensing in the GPL and kind of apply that same sort of sleight of hand to CLA's to make them fair to everybody.

I don't know, I think there could be some work to be done on that. Maybe you, maybe you guys will be the ones to do it.

Steve: I mean, I think, I think, I think you just gave me something to do. So

Jonathan: yeah, that'd be fun when, when you figure it out, come back on the show and give us all the answers to it.

Steve: I mean,

Jonathan: all right.

Fun. Okay. So I'm curious how hard is it to get started with nitric?

Steve: Yeah. I mean, you can tell I hope David had this experience, but we love documentation. So the fact that it's open source, please try it free to use. The big thing that we want to do is provide you with example apps that look familiar to things that people have built before or maybe are interested in building.

The other thing I'll tell you We do use discord with, for nitric and our, you know, the folks working for us as a as a company, but also the contributors are all very, very kind there. And yeah, I would say too, like that take the time just to learn a little bit about the concepts before you dig in is the only thing I would say is.

I think just reading the context, concept sections is, is helpful. So that you kind of get that, like, Hey, what, what is this actually doing? So then you can trust by verification once it does it.

Jonathan: I'm curious, this is kind of an aside, but you guys, as we've mentioned, your documentation is great. What's, what's the secret to having good documentation?

How have you managed that?

Rak: We have a, we have a pretty pedantic CEO. Co CEO and he, so he, this is one of our founders. So Steve shares the role with him and he he is very adamant that that documentation has to be understandable, you know, by everyone and simple and, and not, not just padded with unnecessary, you know content.

So basically he's, he's He's gotten us to revise it several times and to the point where we think it's it's quite clean but it could always use improvements and And so I think I think the key is key there is simplicity clarity and really solid examples.

Jonathan: Yeah, good documentation is such an art do you make the developers write their own documentation or do you have people in the project and in the company that are?

Just for documentation.

Rak: Yeah, it's the development team. That's self documenting right now we take a lot of community contributions as well and and that's, that's the great part about a community, right? Is that they're, that's the easiest way for them to get involved. And so we've had a fair few clarifications and fixes from them as well.

Steve: I'm going to make fun of Jai just for a second, just so that I have the opportunity. So, so our founders are, are Tim Holm and Jai Kush. And Jai, we, we were all at so Rak Jai, Tim and I were all at a previous meeting. Startup that was working, it was a FinTech and we were working with large banks.

And so part of the key value that we were adding was writing a lot of integrations and difficult integrations with core banking systems. And think about the documentation there. And I, I could see him pulling his hair out every day of the week for, for multiple years. And so when, when, when the team would find great documentation They would really talk about it.

And we bluntly as a business, when we were, Hey, what do you guys use for payments? What do you guys use for identity verification? We recommended what our developers recommended, which is the ones that have the, you know, the same business benefits, but by far the better developer documentation. So that I honestly think, why is our documentation good?

It's because our team knows what it's like when it's not. And they recommend tools that have great documentation. So it's certainly an ethos of. I'm very scared to write documentation personally but I'm going to get better. So,

Jonathan: yeah, it can, it could be a challenge. It is difficult, but there's, there's nothing, there's nothing worse than looking at the documentation and it not having the answer.

And so then you either have to, well, you know, you, you're, you're. Your options are like to Google about this thing and try to find the stack overflow threat where somebody else had this problem. And of course, when you go to stack overflow and you find the answer, it's, it's always, it is always a stack overflow question that has been closed for whatever reason, because it's not the kind of question that stack overflow wants.

But then in the comments, somebody has the answer. And I. Repeatedly, I've had that, that experience. So you either you Google and find a closed stack overflow thread, or you have to go on somebody's discord, which discord is great, but it should not be your documentation, or you have to go read the source for yourself and figure it out, which is a useful exercise, but it's also not good documentation.

Steve: Yeah. And I will say just watching the team. I mean, if we're just watching our threads of work and what we prioritize, like. You have to continually update it all the time. And like you said, like if you let other systems be Bob, but that's in discord, like you said, or, or, or I documented that in get hub on the issue.

You're going to lose it. You're going to get really far behind really quick. And that's where having, having someone that cares deeply about it, we definitely, you know, it'll be brought up. Like we've gotten too far away from, from where we were. Let's get back to it.

Jonathan: So we've talked about this a little bit, but give me the rundown on the difference between infrastructure as code versus infrastructure from code.

Rak: Yeah. Yeah. So a slight rehash and it's, it's basically we're complimenting infrastructure as code with infrastructure from code. And it's, it's basically the principle of, of inferring the infrastructure from the application code and automating and streamlining that process rather than manually having to mitigate the differences between.

Your application and your infrastructure as code

Jonathan: So we're talking less less boilerplate to get started things just happen automatically.

Rak: That's that's right. Yeah. Yeah it happens automatically until you need to extend it and that you know That shouldn't be your number one concern when you're starting out your build

Jonathan: Yeah, so I think in the in the pre show I made a statement that it sounds like they have less boilerplate more sane defaults and Pretty much sounds like that's what it is.

Rak: Yeah, exactly.

Steve: I think it's interesting when when rack demos Nitric and you know just shows how it works from a console It is pretty cool to show like even a hello world example in AWS If you're going to use Terraform or Pulumi, you know, if you're going to build a production grade example that says hello world, that has all the things that you need, you're still going to see 80 to a hundred lines of, of infrastructure code there versus with this, you really are writing your seven lines of app code.

And it's going to infer that exact same infrastructure. And you can build guardrails around that. Use our, use our sane defaults, but then you could, if you have changes you want to make, you can make them.

Jonathan: Sure. Is there a, this brings to mind, this is kind of a nuts and bolts question, but is there a way that you can say, hey Nitric, I want you to do all these sane defaults, but I also want you to show me the code.

So can, can you ask Nitric to kind of build that builder plate for you?

Rak: Mm hmm. Yeah. Yeah. So because we export the resource specification from your application code, it's up to you actually how the providers spit out that information. So, you know, the, the default behavior is to use Pulumi to runtime deploy those resources, but we're also working on Terraform providers.

Which provides you with a Terraform script of of what you're going to be deploying to the cloud, which you can then init, plan, apply as necessary. But it doesn't even have to be those two. You could be, you know, It could just be a digest that you spit out for, you know, audit and tracking purposes. It's you're really in control of how you actually interpret the resource specification.

It's a completely flexible.

Jonathan: Yeah. Can. And so this, this is really what I'm getting at. And, and I've, I've not done the hello world example yet. So I'm, I'm speaking out of a little bit of ignorance here, but so nitric itself has a, A specification, essentially a config for your cloud, and it sounds like that could be extremely minimal or you can put a lot of detail in there.

Can you, can you give nitric that extremely minimal config and then get back out of it? Say, okay, I want you to fill the gaps in and give me. That, that the, the kind of config that nitric expects with all of these defaults filled in so that, you know, I don't have to go and figure out how to do, because one of the things that I've run into, it's like, okay, your program works.

Now you want to do this other thing that's a little bit obscure and there's so much documentation for, let's just say Android. I've done some Android development. It's like, okay, you want to do something different with Android. All right. They've got the knob to do that, but you have to go through pages and pages of documentation to find it.

find the right knob. And so if you could just get Android to tell you, okay, here's your manifest with all of the knobs that you're using and you don't realize. And so I'm just curious, is there a way to get nitric to kind of give you the nitric boiler plates that you can go in and make changes?

Rak: Yeah, I think you're talking about extension of, of our providers.

So, so basically we, we have a provider that's already prebuilt. But you can extend it. Right. So there's two ways of extension with Nitric and we've really focused on, on making this better by decoupling it as much as possible from the core Nitric framework. So you either take the Nitric provider and you add to it.

So let's say for example, Nitric does labeling one of your resources when you deploy to the cloud, you know, it applies some standard labels, but you may have a convention that you prefer, or you may want to add extra labels to it. You could take our provider and you could add to it. Without breaking the upgrade path.

And that's one way of extending our providers. But you could also just overhaul the provider and write your own provider instead. That's completely possible as well. Normally I'd recommend you kind of copy and paste from what we've started with. Because that's the easier path. But, you know, it really depends on how ambitious you want to be.

Jonathan: Yeah, yeah, makes sense.

Steve: And Jonathan, I just want to make it clear to the audience a provider is the combination of the cloud that you're provisioning to, deploying to plus the services that you've chosen to, so, hey, I've chosen this compute, this is how we're going to do buckets. This is how we're going to do messaging's queues.

Is that fair, Rack, that just a defined provider for people?

Rak: Yeah, that's exactly right.

Jonathan: Yeah, that's super useful. Alright, so we talked a little bit about a version 1 release. What did that look like? Yeah, so

Steve: Well, apparently

Jonathan: it was overwhelming.

Steve: Well, I think it's a, it's always a funny place for, I mean, I think for someone like us, of us basically saying we have, we've had a bunch of great design partners.

We have users now that are using it in production. What's all the feedback that we, that we want, we want to tell the world that we have a version. That we really want you to use and that's how we thought of the v1 advice And so the number one thing that we had to work on was when we show everybody what we're doing what nitric does They're like that looks powerful.

I love it what happened like the very first like worry is What exactly are you provisioning? What exactly are you creating? How, how can, if I need fine grain control, so in the V1 release, we, the thing we want to do is build the trust that you can see exactly what it's doing. So we built architecture for visualization.

So it's in real time as you're building your app code, you can actually see which resources will be created and the relationships between them. So that as soon as we started showing people that I literally saw attention. Tension out of the DevOps team, tension off of the security team. And. And honestly, I think it's pretty cool.

Like, you know, think about how many people want to build for the cloud, but that they're like, man, I don't know if I want to be an AWS expert or a Google expert, I kind of want to be a great developer expert. There's also this educational component of, Hey, I I'm writing this code. What would it create? So that was number one.

Number two was we did the CLI visualizations as part of this release. So as you're deploying to the cloud, you can see exactly what's being built for that cloud. So not just at the abstraction level, at that resource level, but now, This is the exact resource that's being created in just a nice CLI visualization table format.

Then yeah, our founders took that, that, that moment to say all those little DX things that, that, that was bothering them as we were helping people with projects, we fixed those and really improve, improve that. And then rack did a great job just talking about it, but this custom providers is kind of the key for us.

People love, like we, we believe that 90 percent of, you know. Applications that are going to be building for AWS, Google, or Azure could start with our default provider. But everybody asks us, how do I add this? How do I add that? And we get it because we like to do that too. So just making custom providers actually be easier to build was a big focus of this.

Brack, what did I forget?

Rak: You didn't forget anything actually, but, but I'll just add that the reason why the visualizations has really helped so far is that I'll give you an example. For you know, you. The same example as before, you've you built your application, you've got a bucket in it, you decide to use a third party library, but you forget to delete the bucket from your application code.

The visualizations actually lets people inspect and just kind of spot check their application and be, and see that there are rogue resources connected, or disconnected from the rest of their architecture. And it's kind of a nice way to spot check your application for little errors like that.

Additionally you know, you could be working with non to non non coders as part of your team, you know, project managers or the architecture team who don't really want to dig into the code, but want oversight of what's going on. And so the visualization becomes really useful there because they can spot check it and they can just be like, that's great.

You know, you're, you have an over, you have an overused a resource that's really expensive for our organization or your, you know, you are perfectly using resource as well. So. It's it's been quite useful and that's the feedback we've been getting so far

David: So i've got a couple of questions around that.

So first continuing down the version one discussion I was looking at the languages supported Node. js and python have full support Go c sharp jvm have v0 support. So that means that they're in version zero, but not in version one so far That's correct And then DART's experimental, which you mentioned that you're, you're actively developing that.

So, are you, so do you, well, let me think how to ask the question. Are you, working towards bringing Go, C Sharp and JVM into version one?

Rak: We are. Yeah, it's based on supply and demand. So we have a finite team of resources and we do our very best to keep things in sync. You know, the reason why Python and TypeScript have the support they have is because a lot of our early adopters were those were, were primarily targeting those languages.

So as we see more and more people get familiar with infrastructure from code and with nitric We do think that, that Go and and C Sharp will take a little bit more will get a little bit more attention.

David: A related language question, or maybe not related but what language is it actually written in itself?

Rak: Mm hmm. Yeah, the underlying, the underlying NYTCH framework, sometimes referred to as the membrane, is in Go for performance reasons. And, and maybe our CTO's preference as well.

Steve: Yeah. That's it. Enjoyment of coding. It probably had something to do with it too. Yeah.

David: I'm a Python developer myself, so I feel I have full support.

So I'm happy, but I just wanted to ask about the other ones as well. Yeah.

Rak: Makes sense. The, the, the, the providers and the extension. So the, the application code can all be written code agnostically. So where. You know, we're very happy for anyone to, to bring their language to the to the table.

Jonathan: I, I want to jump in actually and just say, I'll give it back to David here in a second, but it really fascinates me this, this approach.

I don't know if I've ever seen somebody do this before, that we have a partial V1 release based on which language you're using. And so, because the, the, the problem some projects run into is You, we have to bring everything along. And so there's this huge monolithic V1 release or, you know, your, your, your V dot next, whatever it is.

And sometimes that just drags out for months and months and years and years. And suddenly you've got five years of code built up to have to try to massage into a release. And this idea of it's a V1 release if you're using these three languages, and if not use the previous, that's, that's fascinating. Has that worked out pretty well?

Rak: Yeah, I'll be honest. The first time we've released nitric, we. So the, the V zero, I guess, if you could call it that, we, we did have every language in there and it's hard to maintain all of that without constant feedback. And you know, some languages got neglected because they weren't, they weren't used very much.

And so we just made a conscious decision. Let's, let's target, what people actually care about and want to use. And let's talk to our community members and use what they want to use. And a lot of our, you know in production use cases steer the direction of the language choices.

Jonathan: Is the, is the, the nitric code base kind of parallelized in such a way that you can do that easily?

Like there's, there's the module for this language, like maybe there's a shared backend, there's a module for this language, a module for this language, and so it makes it easy to make that?

Rak: Yeah, yeah, it's all, it's all based on gRPC contracts and protos, so we can generate a language SDK and then we fine tune it for the requirements of that language.

To make it the developer experience that the Python X the Python developer expects and the TypeScript developer expects.

David: So a lot of cloud deployments now you're seeing more companies start looking at hybrid cloud. Or even retracting from the cloud where they went and jumped 100 percent cloud.

Now they're like, Whoa, that's too expensive. And they start pulling back. And I know that we already talked about how we can code our own providers and, and do that sort of stuff. But is that on your roadmap of something that you're looking at more of a hybrid deployment functionality?

Rak: Yeah, definitely.

I think it's all within the realms of possibility that that's for sure. The way that we've architected Nitric is so that. Your, your provider doesn't have to be one of them. So your, your cloud provider doesn't have to be one of the major cloud providers. It's, it could be, it's within your control. So we'd be happy to co, co develop and partner with people to do those sorts of those engagements.

I think, I think that's going to be a space that we, we grow into as the requirements kind of flow in.

Steve: Yeah, it's interesting that, you know, when we talk to, you know, innovation centers of larger companies, like, given that we've, we've, we've really delivered for large financial institutions and insurance companies and regulated companies and previous, previous startups, right?

People reach out to people they know, so we've talked to them and it's, I've been fascinated by the interest of, you know, it always starts with multicloud of like, hey, is it possible that I could give my developers a similar experience with it with Google and Azure and AWS? And then the third that that next question is kind of where you're going up if we pulled some of the stuff in house.

And so it is yeah, again, what we've, we've put out with the default providers. No, they're, they're based off of the, the major three cloud providers, but that is where we see this going. A combo of those things of being able to meet people where they want to work.

David: And kind of a follow up question on that that sort of hybrid support, is that something that would make more sense to people?

For, for just like me, if I wanted to try to start contributing, is that makes, is that something that makes sense to do in nitric or would it be better to go like, look at Pulumi and do it there?

Rak: Yeah, I guess that's, that's an, that's an interesting thought. I'm not sure. I think you'd have to have an understanding of Pulumi as well.

So you might be using both together to solve that problem. Again, bringing us back to the infrastructure from code piece that we're bringing to the equation. That's gonna, that's gonna fast track and streamline some of your, your deployment processes. I would, I would say you'd probably end up using both to solve that problem.

Yeah.

David: And to clarify, I wasn't saying not use Nitrate, but I was saying implementing that layer. Would it make more sense to try to implement that later in Pulumi and then use that new API or

Jonathan: When you go to build this out in the source code, are you writing Pulumi source, or are you writing Nitric source, or are you writing both?

Yeah,

Rak: it's gonna be both, but yeah, you would use Pulumi to fulfill some of that contracting, so, yeah.

Jonathan: All right, I am, I'm actually fairly curious. We've talked about this term DX. And I have, I have a feeling that we're talking about the developer experience. You know, kind of going along with this, you know, UX and UI and DX. It seems that, but what, what exactly, like, what does this look like? What are we talking about?

We talk about DX and why does it matter?

Rak: Yeah. Developer experiences, you know, how you interact with the tooling that you're working with to, Bye. To build out your applications. And we've we're really tech, we've really taken a lot of inspiration from tools like super base or oversell. And it's kind of like you know, tools that, that help developers not, not restrict them.

So when I think about super base, you can get a database set up really quickly. It's almost like turnkey so that you're up and running and you're, you're developing as quickly as possible without really. You know, being, being blocked by anything. And that's kind of one of the philosophies that we've tried to take into Nitric.

It's that idea that you're running from day one without too many blockers in your way. Just everything available to help you build your application.

Jonathan: Yeah, that seems like that's important And this is just this is just because i'm not a very good developer But one of the most important things for me and my tooling is to be able to iterate quickly because I make dumb mistakes and if I have to spend an hour waiting for something to compile to discover that I made a dumb mistake It just makes everything so, so much terrible, so much worse.

And you know, if I can discover 15 seconds after I hit the compile button, then, oh, I made a dumb mistake there. Well, I can fix it a lot faster. Not have to spend all day on one single line of code. So it makes sense. It's important.

Rak: Yeah, exactly. That's the idea behind the Nitric CLI and dashboard. So our dashboard experience, aside from the visualizations, what you can also do is you can trigger your application offline.

So the buckets and the queues and the events and the scheduled jobs are all available to you locally. So even before you've hit the cloud, you can experiment with your APIs. And that's that rapid iteration that we're talking about where, you know, normally if you've got a delayed task or a scheduled event, you'd have to wait the duration to test it out.

But with our dashboard, you can trigger it immediately. You can fire off events and topics immediately. To get, to get testing.

Jonathan: Does Nitric, does Nitric have a simulated cloud then to be able to run local tests against?

Rak: Yeah, exactly.

Jonathan: Oh, that's, that's good stuff. I like that. That's, that is particularly clever.

Yeah.

Rak: Yeah. So you're actually running your local code. Let's say you wrote it in node, right? You're just running a node application locally, but it'll register against the Nitric server, which spins up all of the necessary resources locally. And allows you to talk to them and work with them. So, This is all done, you know, we're not containerizing to run this locally.

So you can still test your application with Jest or whatever you're using. Your every, every tool that you use locally, you still use it locally. You're just using the nitric SDK and the nitric server offline to. to help you with your, your DX and your, your app building experience.

Jonathan: Yeah, that's cool. So the, the nitric server, I assume it also has to run as part of the, the, the real world deployment.

Rak: We have a tiny little add on that. We, that we apply to your container that fulfills the runtime aspects of nitric. So. You know, when you're using the SDK and you want to write to a bucket, we have a runtime component that we deploy alongside of your your container, which takes care of those requests.

Jonathan: So there's not, there's not yet another virtual machine or yet another Docker container that has to run that just, that just gives you nitric. Correct. Not

Rak: required. Yep.

Jonathan: That's, that's also very nice. That's cool. You guys have really thought through this. Apparently, it's like you're, it's like you're developers that know what you're doing.

Rak: Well, we've gone through a few iterations to get here, but we're pretty happy with where we're at right now.

Jonathan: Yeah. Yeah. So we've talked about some of this, but what what's on the roadmap? What are you guys looking forward to? And particularly if there's anything that you know is coming that we haven't chatted about yet.

Rak: Yeah. We've briefly mentioned that we're working on Terraform providers. Yeah. And this is because, you know, a lot of ISE a lot of, a lot of people who work in the ISE space anyway, like Terraform and enjoy using it. So we want to, we want to bring IFC to Terraform developers as well. We're working on bringing database support into nitric.

So right now that's a separate piece of work that you'd have to take care of, but we're going to bring some functionality in house. We've mentioned that, you know, we've been working a lot on our documentation for customization. And, and so we're looking to get some community contributions for custom resources as well.

And co, co build support for, for other cloud foundational components that we haven't built so far. You know, like things like AWS recognition or something like that. We'd want to build support for that if, if the community requires it. Yeah, that's I think that's those are the big ticket items anyway.

Jonathan: Yeah, very neat All right. We are we are getting close to the end of the show and I want to ask you guys this and this is a Difficult question because you have to do some set math and that is of all the things we've talked about and that's one set And then the things that you wanted to talk about.

That's another set. Is there a place where they didn't overlap? So the question put a little bit, a little bit easier to understand. Is there anything that we didn't talk about that you wanted to make sure and let folks know about?

Rak: Oh, I think we covered, we covered most of what we wanted to talk about. Yeah, good questions and good session.

Jonathan: Yeah. Steve, anything, anything you want to let folks know about?

Steve: No, I mean really, it's just thanks for the time, please check us out, keep supporting these guys because they're awesome, and yeah, look, look forward to have another conversation about licensing next time, Jonathan.

Yeah, that'll, that'll be, that'll

Jonathan: be real interesting, I do look forward to that. Okay, couple, a couple of wrapping questions I want to ask you then What's the thing that somebody has done with nitric that surprised you the most? Like what's the most oddball or off the wall solution of product that somebody has built with nitric?

Steve: I'll tell you a favorite one. I, I just, I, I, we have a, a, a a user that we've love working with that wanted to build, try all, try out AI one code base, but multiple clouds ai, right? So imagine one code base, but now you can. See what Azure is doing. See what AWS is doing. See what Google's doing and try that same application code with all three.

So that was one.

Jonathan: Oh yeah, that's, that's actually a real interesting use. So that's something we didn't talk about and, and I, I don't know, is that something you guys are doing? Is there any, are, are you doing anything with cryptocurrency or artificial intelligence? You know, our two big buzzwords of, of the year.

I, I,

Steve: I would say we've made a lot of great demos with Copilot. Obviously the GitHub team's really interested in what we're doing. So it's more, it's more about Using nitric to help, you know, imagine using a copilot to help you with your application code using nitric than to be your guardrails for the infrastructure side.

Obviously, those two spaces drive a ton of interest, so we are interested in the type of provider support we can do there. But I wouldn't say there's something there yet, but it's certainly things that we are playing with in the background.

David: Related question then. So does GitHub copilot? Understand Ry?

Rak: Yeah, I mean that a lot of that depends on how much example code we put out into the into the ecosystem. And we're doing as much as we can to, to get code snippets out there that allow the mod, the the, the models to learn from us. So the more we put out there, the better. The more we get from the community, the better.

But. Initial trials with it have been really promising and we think it'll only get better.

Steve: I would just say like giving it a project structure like made a huge difference, right? This is like six months ago when we first started it. It was Hey, if we just give it a little bit of a little bit of a way to go, it does really well.

Jonathan: Yep. Yep. Makes sense All right. So final two questions and I will ask Rack first. What is your favorite scripting language and text editor?

Rak: I use VS code for the most part just, just, I just like it. I don't know if this is any other reason really. And scripting language? Getting more into Python again lately.

So but, but Node as well.

Jonathan: Okay. And Steve, same two questions, scripting language, favorite scripting language and the text editor you spend all day in.

Steve: I mean I'm not doing this as much as I used to VS code, but can I just tell you the one I miss and you're going to judge me? I miss CoffeeScript.

Yeah, that's fine. I was good in it and and I miss it.

Jonathan: Understandable. All right. It has been a blast. Thank you guys for being here and a really interesting project. I'm really excited to learn about it and hear about what happens in the future.

Rak: Awesome.

Jonathan: Thank you. Cheers. Yeah. All right, David what do you think?

Any thoughts on it?

David: That's very exciting so now I'm going to go download it and at least kick the wheels. There may be a role for that. I may not be able to leverage it immediately because a lot of the DevOps that I've been involved in is not cloud based. But it's exciting. It's something that's very interesting, and I really like the ability to run the nitric server locally and test it out and be able to see what's happening, you know, just so that you get that rapid development cycle.

Jonathan: Yeah, I think I don't know how many other frameworks and companies do this. Maybe it's common, and I just don't know. But being able to have a simulated cloud where you test all your code and you fix all your bugs without actually having to talk to Amazon. I think that's brilliant. If, if other companies aren't doing that, they really should.

And I also love that, you know, their, their hello world example is apparently only like seven lines. I, I need to actually go look that up because that, that is impressive. I remember my first Android application and it's like pages and pages and like 15 different files of boilerplate just to be able to get, you know, a side by side window so that you could put stuff in it.

And. So, I like, I like, I like getting rid of boilerplate. It's, it's definitely the way to go. Alright, David, do you have anything you want to plug before we before we go?

Steve: Well,

David: I always like to plug Twit, the Twit Club and ULS. I get to show up there. Actually, I've been there the last two weeks and I'll be there this Friday.

So I might have to remove my guest cohost label and just become a cohost. I think you

Jonathan: can do that. I think that would be fair. And in fact, you know, over, over on ULS, I've thought for a while about making it more of a rotating panel of cohosts rather than primary and secondary. So we'll see what happens with that.

All right. So next week we have a couple of guys talking about the UnPhone, which is a sort of a hobby slash educational device, which I've actually got one somewhere around here. I can't put my hands on it at the moment, but they sent me one to take a look at. It runs. It has been ported to run Nescaster.

Yeah, well, I know, I know where it's at. It's just not here within, within arm's reach. It will run Meshtastic, but I think they're also using it for teaching a a university level course on doing embedded programming. And so, some pretty, pretty cool stuff there. Get to talk to those guys next week. You already mentioned Twit.

The only other thing I will mention is Hackaday. We sure appreciate Hackaday being the new home for the for Floss Weekly. And, of course, I've got the security column that goes live on Friday mornings. Make sure to check that out and all of the other good stuff on Hackaday. And, hey, thank you to everyone for being here.

Those that caught it live and those that are on the download. We sure appreciate it. And we will see you next time on Floss Weekly.

Episode 781 transcript

1 maj 2024 | min

Jonathan: This is Floss Weekly, Episode 781, recorded Wednesday, May 1st. Resistant to the Wrath of God.

This week, Doc Searles joins me and we talk with Matthias and Paolo about HolePunch, the Pear Runtime, the Keet Messaging Platform, and how all of this works peer to peer without any servers needed. You don't want to miss it. So stay tuned.

Hey, it's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got the, the one, the only, the great Doc Searles with us. Hey Doc.

Doc: Better, better great than late. I was thinking about that. Cause I, I I'm having, it's just. Some, something done to my heart tomorrow.

So we'll find out, but I plan to still be here.

Jonathan: Yeah.

Doc: Yeah. Well, you've got to

Jonathan: stick around. We you're, you're a co host. You're one of my rotating co hosts. Can't check out yet.

Doc: Can't no, not for that. Do you know what Oh, George Burns said, when they asked him if he'd live to be 100, he said, of course, I'm booked.

Oh,

Jonathan: we got a fun project on today, and it's, it's Hole Punch, but it's also Pear, and so figuring out kind of the relation between those two is going to be interesting. Doc, you've done a little bit of homework on this, haven't you?

Doc: A little bit. There's also, there's also Keto, which is an identity thing.

I'm going through my, my tab. Oh, Keat, Keat, Keat under bar I O, embrace your digital sovereignty with Keat identities. I'm, I'm curious about that. And I hope these guys are involved with that because they just followed, followed threads. But anyway but I'm curious about that because I think we need.

Disposable identities, and that's just one of the things I'm curious about.

Jonathan: Yeah, to get into the identity side of this will be really interesting. I think there, I think there's places where it fits. But let's go ahead and I'll introduce the guys. So we've got, we've got Paolo Arduino. Not Arduino, but Ardoino.

And he is the co founder and chief strategy officer of Hole Punch. And we also have Matthias Busmasden, who is the CEO of Hole Punch. Have both of them with us together. Have some, some real high C level guys today. That's always fun. To both of you, welcome to the show. Thank you. Thank you. Thank you for having us.

Yeah. So where, where, where do we start? What is what's kind of the core here? Is it whole punch? Is it pair is whole punch? The company and pairs the software. What's kind of, how do these, these disparate pieces fit together?

Mathias: Well, we're very good at making up cool names. I think we'd like to use them all.

But basically, yeah, HolePunch is the company. It's, it's our it's a, it's a culmination of a lot of work, me and Paolo and and some other guys have been doing for the last 10 years in, in the, in the peer to peer space, making open source peer to peer software. And and scaling that and productionizing that and turning that into a real thing you can build applications on.

That's very powerful. I've been working on that for the last 15 years like basically my entire software career, because I'm very into, to to the things you just mentioned, like data sovereignty and, and and unstoppable apps and spreading data out. And, yeah, a couple of years ago me and Dr.

Paolo decided to get super serious about and start a company only focused on making peer to peer apps which we've been doing the last, um, couple of years, like I said, and, and, uh, Pairs, which is the Pair Runtime is, is, is a software stack made through this that's very good at running peer to peer apps.

And Keith, like you mentioned, is It's a chat application that's focused on data sovereignty and owning your own data and owning your own IDs and not having servers. That's built on top of that also. So it's all fits together. It's a lot of terms and a lot of technology, but at the end of the day, it's also just a very, very cool apps that we're we use every day and we run this technology in all kinds of places now.

And it's super exciting for us to work on.

Jonathan: Yeah. Yeah. So Pair is kind of that, that central runtime that makes everything work.

Mathias: Yeah, it's actually, it's pretty funny because it's called Pear because Paolo's Italian. He's a proud Italian and he kept, he we, I said peer and Paolo said peer, but he actually said pear.

And it turned into like a, it's a little bit, a little bit of a internal meme and it has a cool emoji. And then we kind of embraced it. I'm sure it'll go down in history as the best choice we've ever made. Cause it's really fun so I like pair to pair, yeah,

Doc: pair to pair is a, it's a homonym, you know, to say in English.

Yeah.

Mathias: It's also really easy to understand and it's a good fruit and and yeah, so, so we, we doubled down on that and so that we, our entire peer to peer stack is called Pears and just a bunch of small, very powerful software modules, all open source all on our, on our GitHub very battle tested that you can, you can get and build yeah, like consumer based applications that are very easy to use, but have to Power of peer to peer, which means that the data lives everywhere but nowhere and like no infrastructure cost and just very easy to use and build really powerful things.

Yeah.

Jonathan: So it is open source. I've got to I've got to tell you something. I spent some time on both holepunch. com and pairs. com and on neither of those sites did I find a link to your GitHub repository. I had to go Google for it to find it.

Mathias: Well, it's like one of those things when it was so obvious for us, we forgot, forgot to forget to mention it.

I guess we should get that fixed. Obviously it's just whole punched up too. I'm getting up. We had. I think we have 500, 600 repositories or something like that. We like making repositories all open source stuff. And on my GitHub, I have 1100 repos. I'm sure Paolo has 200 repos. So we definitely have a lot of stuff out there.

So you can start at one end and you can, you can take your time. Is that, is that because a lot of this is built on node. js where, you know, you famously have one line libraries.

Yeah, it's all, it's both me and Tyler are big believers of, of, of Node. js. But I do have, I think I have more than one line, more than one, one line of code.

So I'm a big fan of those. But I'm not sure everybody agrees on that one. But yeah.

Jonathan: So yeah. That's fun. Okay. So now what, how does this work when we talk peer to peer? Like what's the What's, what's the juice here? What is the juice that we're squeezing out of the pear? What, what do we do with peer to peer and what exactly do we mean by peer to peer?

Mathias: Well, it's like, you know, obviously there's many, many ways to look at this. I'm a nerd, big nerd, and I was very excited to talk to you guys because But one of the first times I can talk on a technical podcast so, you know, so for me, it's, it's obviously a very technical thing, but it's also future peers also about, you know, control.

It's about for some people about politics. It's also about, you know, just data sovereignty and things like that. I think it's kind of like reflects a huge part of society today, but on the technical level. It's one of those things that's really easy and also incredibly complicated. It's about connecting computers and having protocols where you can exchange data free of third parties with like, no trust in in each other, but like just trust in the data without having to use a bunch of servers that cost money or to have to sign up for things or pay subscriptions, because we, as long as we have computers, we have the computing power to do things.

It's a very, so like, it's a technical foundation. I think. You know, obviously you guys know BitTorrent, most people know BitTorrent, but I always wanted to, for me, like, the easiest way to sell understanding, it's something very technical, but at the end of the day, it just allows people to share content without central points and peer to peer is the generalization of that and one thing we've been very, I almost want to say obsessed by in the last many years is taking those ideas that are often just used for file sharing and then generalizing them a little bit more, making them a little bit more Modular so it can be used by, you know, developers who are used to just making using small modules like you do in the front and use react and whatever, just more things here and there, but with the same kind of power, so they can start building things without servers without having to be.

experts in anything. And just being, you know, normal developers without a PhD in decentralization. That's, that's basically what runs the, our technical influence. And that's, that's what it's all about for us.

Jonathan: I want to, I want to dig into the technical questions about this a bit more because it fascinates me.

I, you know, I spent, I spent years now fighting with networks and Nat and trying to make things work. But I want to, I want to bring Paolo into the conversation too. And I'm, I'm curious kind of what, what your angle is, which part of this problem did, are you working on?

Paolo: So I, I started coding at a really young age.

No, that's the only thing I could do back home. And, uh, I've been always a fan of you know, the work of Richard Stallman named Storwell's like Eric S. Raymond. And so to me I've been almost an activist when it comes to free software and and also free information. You know, I feel like when I was young in my, in my room, I was, you know, using the internet to access information and I was always, you know, over time become annoyed by the fact that the way internet was designed.

That was meant to be point to point and peer to peer was actually steered into a direction that today is, sees the vast majority of the information and the services being centralized across few powerful companies, I think, is losing its poetry, is losing its, you know, its power if we keep going towards that direction.

And the issues that, that, that direction is. is due to the fact that with centralization, and centralization is the only way for big companies to make money because they can retain the control over data, and that is actually not what humanity should try to go for. So also I've been a big enthusiast user of file sharing systems and developer on file sharing systems.

And I, you could follow the entire history of file sharing from Napster to, you know, Nutella, LimeWire, and all the others back to then ending with BitTorrent, as, as Mattias said, the, you know, all these iterations of file sharing systems had seen, you know, the first, you know, the, with Napster, everything was centralized, the index and the file sharing, the, you know, the connectivity across peers or across users was, you know, Was actually centralized and then over time, you know, they tried to make peer to peer the file sharing, but the indexes were centralized, like in LimeWire, for example, or Kazam, and then they realized that the only way to have really unstoppable hyperscalable infrastructure, file sharing infrastructure was BitTorrent with the DHT And with with a whole bunch of technique, all punching techniques to do the peer to peer.

And so basically the almost the simple idea that Matias, myself had was why, why we don't build on the similar foundations. We take, you know, this great technology that powered the most impressive file sharing network, but we adapted not only to file sharing, but for two real time data streams. So, you know, audio video.

You know, chat, but also services, right? So why we cannot reuse the same techniques? Also to build services that today you know they are built as client servers and but in the future they you know you can have a more peer to peer style service where you know you can almost every component of the infrastructure is equipotent and that is kind of exciting in a way because it makes the You know, the system much more resilient.

You know, I, I don't sleep at night thinking that entire governments work and use WhatsApp to coordinate themselves. Right. What if, you know, do you have all the connectivity to WhatsApp dropped? Mm-Hmm. , if you, you know, I'm Italian as, as Mattia said, if, you know, if I live in Rome and my family is in Rome, every, and I use WhatsApp, every single data packet that I send.

To my family's child goes through Frankfurt. So a few thousand kilometers up North, just to go back South for two and 2000 kilometers. So that is enormous waste of money of internet infrastructure. That is, it's quite stupid if I have to say so. So I think we, as humans are trying to go to Mars and we should know better that the way to, you know, the, the internet protocols are built to route and the best.

With a, you know, smaller number of hops and to achieve the best latency possible. And yet we keep on building and building, you know, network and internet infrastructure that is useless because we are trying to force data packets to do Longest distances when it's not necessary for most of the time. Yeah, that's interesting.

Jonathan: It's something I've thought quite a bit about. I too am watching the SpaceX and their quest to go to Mars. And it's interesting you say that, because a lot of the assumptions we make about networking and the Internet itself will just completely break down when you try to connect two planets together.

The lag times are too long. So that's, that's a, that's a fascinating thought to me. You mentioned BitTorrent and I saw in your bio that DHT distributed hash tables is something that you've worked on. I find it real fascinating that you've taken obviously inspiration from the way BitTorrent works.

Is there, is there actual like code sharing or, or technology sharing? Are you using some of the same actual techniques to be able to connect people?

Mathias: Well, I worked a fair bit on BitTorrent also in the past, just actually implementing BitTorrent and I find it to be a fascinating protocol because it's one of those protocols for me that's It's simple enough that somebody with enough passion can sit down and just do it by themself.

It's a very open community. I remember when I did it, I came in, like, with no background and I just emailed the spec offers because they're, like, these open specs. And they replied because they're just nice people. And I got very inspired and I just started hacking. And. It's complicated, but it's not that complicated.

And so, that was, that was, especially that last part was a huge inspiration for me as a developer. That you can make things that are inherently, you know, these teams are pretty complicated, but at the day, not that much once you start understanding them. Some very important foundational pieces and Pittorovich is very, very good at, at reading that.

So we obviously took a lot of inspiration from that. And The hard thing with a lot of these networks, especially , is that they're, they're made for a different time. It's very, it's made for very static things. It's made for a static file sharing. It's made for not sharing actually that much content.

In today's world, it's like if you share more than a bunch of gigabytes, the chunking and stuff, it's a little bit inefficient in BitTorrent due to the way it's structured. But that's why it gets its simplicity. So we took a lot of those ideas and we'd like, okay, what about we. We take some of the stuff that's come out since then, there's been a lot of breakthroughs and also patent releases and and elliptic, elliptic curve signatures and all this stuff apply those modern kind of things to the structures, make sure that it's like dynamic from the get go so you can build variable size structures that can grow with the same contrast, take the Same good ideas from the DHTs.

We, we use the same underlying routing protocol, Kadimlia. I'm getting very technical but that's, that's a very foundational piece that's been iterated on a ton for, to make it more secure, take those ideas and then And I think this is one of the things that I thought about the most, it's kind of like, and then make it fit to today's networks.

Like you said before, you're, you're, you're, you're finding your router and your net. I remember when I started using computers and I was a kid and we played I think it was command and conquer and one of those games, right? And you could actually connect to each other and play it online because we didn't have these crazy, crazy firewalls.

And now we don't just have firewalls and maps. We have like. Three or four layers between us and our ISPs because that's how the internet evolved due to IP exhaustion and things like that. So, like, those things never work anymore. But there's been a ton of breakthroughs in how you can actually connect computers still with modern hole punching techniques.

We started a lot of that and applied all those things. So our hole punching is very, very good. So we can actually connect people because we're using just a lot of modern things for that. Which BitTorrent could never do because it's just kind of stuck in that old way of thinking now it only kind of runs on servers, unfortunately, because of the nets.

So it's kind of like, I always think of it like, it's kind of like BitTorrent 2. 0 in that way, where we take the foundational pieces, we upgrade them to the modern world, and then we don't couple them to file sharing, we make, we make them work in the more essence of the ideas, and then we build. We build databases on top and we build file sharing systems still, obviously, because it's important, but like, especially databases, I think it's super important for somebody to start making things like chat applications.

Me and Paolo have done a ton of work on just, you know, sharing tons of data internally for organizations and things like that, which I think is super important. It's a far more interesting idea in the longer term than those files you've written. Boy,

Doc: there's so many questions in there. They seem to be overlapping and relating and first Paolo, I'm totally in sync with, with where you're coming from on, on, on fighting centralization of everything.

And, uh, and I know Salman and, and, and Raymond from way back and was very involved in promulgating their thinking and all that. I'm wondering, I mean, we, we live in a world now, I mean, even though the TCPIP under undergirds everything and, and at the same time, people think in terms of services, they think in terms of what giants can do, they think in terms of giant platforms.

And I, I, I, I could almost hear my, in my head, somebody saying, so what's the platform for P2P, you know, for, for peer to peer and to tell you, Oh, you're your own platform. Well, how can it be my own platform? And people don't even know they've got one of those. sitting on their desk. And it's because and I've said this often enough, pushback on it, but I don't think it's wrong, that for some reason we decided on client server as an architecture back in 25, 30 years ago, when Client server is a synonym, it's a euphemism for slave master.

So we're always the slaves and and that's I mean We've also called a calf cow because you're always the calf and you go to to the cow, you know for milk and cookies HTTP and cookies, but HTTP itself is also Essentially peer to peer. So I mean, I mean it looks at a server over there, but you don't have to be a client of that You're just exchanging files.

You're looking at a file and And And, and I'm wondering, so where that goes for me is toward who's using this now and what are they using it for you must have, I don't even know if there's a business in here or not, or if you're, I imagine you guys make money somehow, but I, I've known too many geeks for too long to know that there's, there's no one way that everybody does it other than writing code and getting paid for it.

So, so I'm wondering What does adoption look like now and where do you see it going? Because I'd love to, I'd love to sell that to people. I'd love to get, you know, people involved in it in addition to wanting to use it myself.

Paolo: So first of all, I think that just building on what you said, there is there, there should be, and it's kind of a worrying for me that in, in universities the only pattern that has been told, Is the client server model, right?

So it's, it's almost, they teach you, Oh, you should open an AWS account or Google cloud account install, you know, NGINX there and run some websites it's that, that you, you don't, you don't, you are not trained to think that what you have in your pocket that is, you know, a smartphone is probably much more powerful than the average server of 10 years ago.

And and same for your laptop or home computer. So that's something that hopefully it will change. When it comes to the business model, I mean Mattia said, you know, we have hundreds of open source repositories with really cool stuff with that, that starts from networking and solving all the networking problems of whole bunch to cryptography.

And and et cetera, data structures, you know, app hypercores and hyper bees are are really good data structures. Hypercores are up and only logs and hyper bees are basically binary trees built on hypercores and all this stuff is open source. So. We are quite lucky as a, as a company because we, we are backed by an important company from the cryptocurrency sector.

It's called Tether. So we honestly, our interest on making money at this stage is extremely high. Relatively low. We believe that is much more important for this technology to be adopted. I don't think we have much time before the centralization and the doors of centralization close up on on on everyone.

We are seeing maybe it's true or not. You might have seen what happened in Europe with that leak of, potential interest from European regulators in clamping down on peer to peer technology we use for communication and file sharing. It's unfortunately, it's something that we see, we see happening more and more.

We see, I feel like we are going to face challenges for companies that are helping people to retain control over their own data like it was a criminal offense. And so I think for that reason, we want to prioritize the adoption and the open sourceness of this technology compared to how we can make money out of it.

Because I feel like if you're trying to squeeze a way to make money on a protocol or, or on anything, you end up in creating control patterns for such protocol. So. The only way to have something that is truly unstoppable, that is resistant to the wrath of God is to let it be free. Then you as a company can think about how you can build services on top of it and the paid services.

That's fine, right? So creating a business is fine, but the underlying, the entirety of the underlying technology. Should be open should be free and that is what we are focusing on heat is the first biggest example of a product that we were able to build that is a peer to peer video chat text chat. You know it's great is getting adoption is getting users and is a showcase.

of the fact that peer to peer technology, that it can be user friendly as the client server model, just you have to start with the right infrastructure, the right, you know, technological infrastructure, and that's why Peer on Time was built, to give the, to create the foundations for anyone, any developer to, to build on, or to have almost like And Node.

js and NPM solution that would give you the primitives to connect to other peers in a peer to peer way, right? It's almost like a nice wrapping, nice scaffolding, a nice bootstrap system for you to build and start Node. js style to build. To build applications that are natively peer to peer and our main concept and also the choice of Javascript as the main language was that Javascript is the most used or seems to be the most used programming language in the world.

So what we wanted to do? With this feeling that there is not much time before the close, the doors of centralizations are closing on us. We wanted to use this platform and parent time fully open source and give it to the world and and and have many developers joining us because we feel like if we are, we can organize hordes of web developers that can build on parent time and build peer to peer applications.

We can kind of reverse. The push, the forces towards centralization, towards storing everything on centralized systems.

Doc: So, I have a couple of thoughts about all that. One is and this is actually something that came up in our, in our, at IAW, which is this Internet Identity Workshop that I help organize twice a year in Silicon Valley. And somebody just pointed this out, and it never really had occurred to me before, that On your computer, your desktop, the thing I'm talking to with now, I have, I have a unified data structure.

I've got a, I've got Unix directory paths all through that on my phone. I don't, I've got as many as there are apps and that the phone What the phone has done is helped centralize everything by telling people that they are in a different place with every one of those things and it's not theirs. It's not theirs.

This is Apple's. This is Google's. And at the same time, you know, I'm dealing with companies where they don't want to use say Word or just plain text of any kind. They want you to go on Google Docs and now you're, your, your document is living in, in, in a slave colony and inside of Google somewhere. And and I'm wondering how, so who, I haven't heard it yet exactly, who's using this so far?

What, what's your community where you're, obviously you guys are talking to each other using your own tools, right? But who else is doing this? And are they helping you develop it? I mean, are you, how big are the development teams that you've got going there and what are they doing?

Mathias: Well, like, first of all it's actually funny you mentioned that because having done all this open source for many years, we, so we're a company now, we are lucky that we can pay people to work on this, which is awesome.

That's that's obviously really important and all our recruitment came from our community, P2P community, because there's a lot of people very interested in fixing these things for the better. Very smart people. So it's actually been very easy for us to, to recruit through that because. The open source just makes it very easy.

And that's been part of it. And then you can also see that reflected in our apps. Like the, we have a pretty big decent sized early adapter community, I guess is the word it's a bunch of very, very people, which is, I think it's like very classic for decentralization she ever been in. And there's a lot of obviously a big Bitcoin crowd because there's a lot of people coming in from that.

There's a lot of people just very interested in decentralization for social reasons, people living off the grid. Also just a bunch of, I don't know, the acceptable term for it, but I'm gonna say normies, you know, like people who just are interested in, interested in technology. And I have worked on many open source things in the past, and there's a lot of interesting people in open source, especially.

But when I'm on the chat apps on this research session, the mix of people, it's very interesting to me because it's just very, very varied, but everybody comes in with because it's really adapted to a strong agenda for, I think, Not liking the status quo, I guess, in terms of like how these apps operate and thinking about data sovereignty a lot.

And I think what you're saying there is very interesting because your phone wants you to be a consumer. That's what, that's what they always wanted you to do. Like, it's very hard to produce on the phones. I think peer to peer obviously can't solve the fundamental limitations of the phone because that's the people who build it.

But it can put your data that you consume and produce on your phones in shareable formats that other apps can import and share with each other because that's what they do. P2P at a data level is all about. It's all about making sure protocols are, are, you know, clearly defined and open and consumable, like the data is the protocol, basically.

So when you make P2P databases, it's not like you're talking to a server, you're reading the data and interpreting the data. So it's very important that you have these things specked out. And I think that's super important for, for data sovereignty. We always talk about these things when we talk about this with our people about, you know, Do you actually own any of your profiles you have online?

If you're very popular on Twitter, for example, like who owns that profile, like Twitter can, can choose to mute you or shut you down if they want to. And it doesn't matter if you have a million followers, 2 million followers, you actually don't really own it because. You're just a guest in their system and they, um, can change the rules or whatever they want at any time.

But if it's in a peer to peer system, you own that data and you sign that data and you can take that data with you elsewhere and you can take your, cause the, the, the apps are more of a view of your data. And I think that's, that's super important for these kinds of things. And especially for this. Ad siloing that I think you're referring to also I want to jump back in we got a question from the from the back chat from the live audience a harebrain Asks, so what's wrong with signal or the half dozen other options that use that protocol?

Jonathan: I don't think there's anything wrong with any of them Signal is a great they always try to solve very specific problems I think you know, I actually often get asked this also in p2p because people just tend to think it's It's one thing, right? Signal is a great app. I use Signal all the time. But it suffers from some of the same things that centralization software suffers from, which is cost.

Mathias: Cost is just really hard to solve. There was an article out recently about just Signal having to raise money to run it it's a non profit, they cost them tens of millions of dollars every year and not profit, just like money they have to pay to run it. Which means that the more popular they get, the more they have to pay.

They're not making money because they don't want to because it's signal, that's awesome. But still, you know, the bill has to be paid somehow. You see, you see this on Wikipedia also, like, you know, the banner on Wikipedia gets bigger every year, right?

That's not because they're, that's not because they're greedy or because, Oh my God, Wikipedia is terrible. It's because things cost money. And peer to peer when, when, when done at a very advanced technical level, luckily we don't bubble it up, doesn't cost anything. It's inherently harder to build that foundational piece, which we call pair runtime, that requires, you know, like I said, we put 10 years into this.

Jonathan: Yeah.

Mathias: But that solves the cost problem, and that's what we're all about. So we want to, you know, like Paolo said earlier, we're not here to, To make a profit tomorrow, but if we roll up tomorrow and keyed our app had a billion users We wouldn't be sweating either because the cost is not on us It's just on the network growing and we don't have a token or anything like that It's just because of these protocols the way they do it.

And I think that's super important for anything That's going to last the test of time. And there's plenty of space for all of these approaches, but for us, that's super important. So we can actually make apps that, that, that can build some advanced features that can compete because there's no cost.

Jonathan: Yeah. So let's, oh, go ahead. Yeah.

Paolo: On signal. I mean, I really like signal as well. The, the, the solution it's an issue that they have is. That if tomorrow there is we live in a world that is getting crazier and crazier, and if they ban signal IP addresses of the servers, no one can talk on signal. So it's super attackable from that point of view.

Not everyone is lucky to live in a country that is democratic in a way. So I think the, The way I would describe it is the I feel most of the time we as developers design applications and services that are built for the best case scenario, rather than the way I see it is that whole bunch and parent time and it, for example, are built for the worst case scenario.

Right? So what if. You know what we have been used to start failing how we can communicating

Jonathan: yeah, so Let's just good follow up. I think to that. You have you have the keat Peer to peer chat app, but you also have Keet. io, which is the website that you go to. Does Keet the app live on if Keet. io goes down?

Mathias: Oh, yeah, yeah, the website is just a static page where you can get to the app. Simple as that.

Paolo: So how I think, Matthias, you should explain how actually Keet is being distributed. Keet updates are distributed through Keet. Yeah, exactly. Go ahead.

Jonathan: Yeah, I very much wanted to get into this.

Mathias: Yeah, so it's, it's, it's, it's, so we built Pair Runtime, which is the foundational piece on Pair Runtime.

So we distribute updates to the thing itself through, through itself. It's kind of mind bending. But that's really, really important for executive thing you're saying. So we don't, we don't run any servers of any design, other than like we need to keep the website up. The website is just a static page.

But all the updates. To the software. Once you get that first installer, but you could get that from a friend if you trust them. They're just. Bootstrap into a network and you start exchanging these updates between peers. That's super important, obviously, because that's how we can make this installable future.

It's also how, you know, if just for efficiency reasons, if we, if, if we release an update, then in the office, it spreads out for the office instead of having to go to like a bunch of servers. Right. So it's pretty fast and propagating.

Jonathan: Yeah.

Mathias: So the, the, the app itself is distributed through the app itself.

Updates from the app is on the network. It's all like I said, it's a multi sig, like multi signature structure that is signed by, by, by three out of five people in our company for, for security reasons. That's all, that's all public. So it's, it's, it's, it's, it's a V and it's through the same technology and you can download it and you can, you can audit that.

That the hashes are the same and the signatures match and stuff like that. But that's, that's super important for us. And it's one of those things. Once we started doing that, we do that for all apps. We build, it always seems very legacy.

Jonathan: I may have had the wrong idea then I expected Keap to be something that you would run in the browser, but it sounds more like it's a, it's a downloadable binary.

Mathias: It's a downloadable binary. And like I've been on many rants and many podcasts about browsers today, because I think browsers really lost, they really lost the ball. I mean, On on on breaking this trend. I mean it kind of makes sense not To get too into it, but like, you know browsers are built by big companies with lots of stakes and having people go to certain websites browsers You know, are what the name suggests.

They're good for browsing. They're not very good for making this kind of content. So, so we we made a completely different runtime, which obviously has a lot of harder adoptions and things because the browser is really good for adoption because it's easy. But we, we, we made this bet initially that we want to make something that's like, can stand the test of time and the wrath of God and browsers are not going to do that.

And so there's no servers. It's all just. Interdata networks that you can get from, you can get the bootstrap from our website.

Jonathan: How, when, when someone starts, and again, we'll just take keep, because this is, this is the one, the one example, that's easy to kind of talk about when someone installs it for the first time.

So let's say you, you got the binary on a flash drive from a friend, you install it. How does it discover the rest of the network without there being a centralized server to talk to first?

Mathias: It's the same way Spector does it and so first of all, the installer is really small. So if you go to the website, you'll notice that the installer is tiny.

It's like, it's like eight megabytes or something. Because all that is, is like a script, a JavaScript script that's can bootstrap itself to get it on the network. It contacts the DHT. Obviously to contact the DHT, these nodes, know some nodes in the DHT, but and we're always working on making that more resilient.

But right, that's obviously a list of nodes hard coded in there that is like, you know, known to be up since last time. But you just need to know one. That's all. When you join a key chat, you get this link. We can embed some more nodes in that one. That's what BitTorrent does also. The cool thing about these T's is that it's not like you need to talk to one server.

You just need to know, talk to one of them, and then the account unravels from there. So it's really hard to push down because there's no security involved in that. The security is elsewhere. So it's a pretty powerful technique, and we're always making that more resilient. But yeah, it's a bootstrap for the DST, finds some peers, pulls in the data, verifies it, All with a nice user experience, because that's really important for us that the user who, who who uses these apps shouldn't really know that they're using peer to peer apps that you just want to use apps, right?

So, so, yeah, super important.

Jonathan: Does, does IPv6 help with this? I, I saw a I saw a company that was apparently selling they're selling VPN access to static IPv6 addresses to be able to give people the ability to self host things. And I, I kind of, I kind of boggled at that because that's a really interesting idea.

But then we come and we talk about this and it seems like, well, IPv6 might might make a lot of these things at least easier to manage again.

Mathias: Well, it's definitely easier. I think it's, this is a complex subject. And I love that you bring it up because I love to talk about IPs. But First of all, IPv6 doesn't mean you don't have a firewall.

Like, you know, your ISP still will block it. You still need to do hole punching. It just means that the hole punching is easier,

Jonathan: which

Mathias: is obviously nice. There's this very interesting trend I've been following IPv6 for a while because as IPv6 adoption has increased, we started releasing more and more IPv4 addresses, which means that the IPv4 adoption has kind of gone up.

Jonathan: Yes.

Mathias: And I, and I think if you look at the trend now, the trend is very scary because the trend is like the IPv6 one is going a little bit like this because it's kind of like. Yeah, it's just

Jonathan: supply and demand.

Mathias: Yeah, exactly. So IPv4 is not going away anytime soon. I think the interesting thing that makes that easier is that it's very geographically skewed.

Like if you're in India, almost everything is IPv6. But if you're in Denmark, there's like six times as many IPv4 addresses that there are people. So, so it really depends on where you are. Go ahead. IPv6, so short answer IPv6 makes it easier, but not, not, not that much, honestly.

Jonathan: Yeah, I, I've, I've

Mathias: IPv4 and load function is not that hard.

Sorry. True.

Jonathan: I've, I've kind of found, I've just recently got a, a secondary ISP that gave me IPv6 addresses, and it kind of, it, it was kind of, it was very odd, let me put it that way, because To call up the ISP or even where my server is hosted, call up my, my ISP there, my server ISP, say, Hey, I would like to get some IPv6 addresses and to get by both of these places.

No, we're not doing IPv6 yet. Yeah. Really? Yeah. And 2024 and by what we doing. Yeah. And, and now with the ones that will gimme IPV six, they hand 'em out. But it's like there's no, there's no mechanism to try to get those set statically to be able to host something there. It's, it's odd. It's such a, the IPV six rollout is so weird.

It is very weird.

Mathias: But it's like, it's almost like they don't want you to host things. Right. I think that would, I would say that is like, it's not in their interest for having you host things. So, but yeah. Whole punching all the way.

Doc: Yeah, that's, that's really interesting. Do you think that might be part of it that they just don't want you to, to host things?

It makes sense. I hadn't thought of that before, but it does make sense. I

Mathias: mean, I think there's no upside for them to do that. I think the addressing makes it easier. And that's why they would all, you know, all the equipment you get from your ISP is always very locked down in this sense. They probably also scared of file sharing to a large degree.

But luckily, like, you know, you can get around all that stuff pretty easily with protocols these days. So that's awesome. I also suspect that your ISP is the ones that do, that are, that have somebody that knows what they're doing enough to be able to give you good IPv6 and static IP addresses. They're going to want to upsell you to a business account for you to be able to host things.

Doc: Doc, do you have some other questions you want to jump in with? It's well, these are just some things that occurred to me as I'm listening to you guys. What is the browsers? I, and I wrote this down, browsers are slave bracelets, they're slave bracelets you wear that kind of give you permission to go to lots of different castles, you know, I mean including ones you make for yourselves.

This is a little castle right here, it's Jonathan's castle, but but I think that's how people see it right now. I think they, you know, the browser is something that gets you to lots of other places that are not yours. you have this sort of illusion that the file that you have on your desk is yours for now.

Even, even this thing about personal data, what is personal data? I mean, if you, you, you buy a ticket to, you buy a ticket to the concert, it's your ticket, but it's their ticket too, right? It's permission to get in there. It's co owned. Who owns it? There's mean, right now I'm working on a on a standard for, for basically it's peer to peer contracts.

In other words, we don't, we don't agree to your terms on the browser. You know, you go to a new site and you want to, it tells you to agree to their terms. No, no, you agree to ours. And here's how we proffer those. It's called P7012. It's an IEEE standard if it happens. We were trying to finish it today after seven years, but it's you know, whether this gets implemented or not, I don't know.

I mean, I think you need a community of people that start using it, but we're pretty close with the people at WordPress and WordPress has like a third of the, of all the sites in the world. A lot of those are of no consequence, I suppose, but they're also not busy tracking it. They don't want to track you.

So. If you don't want to be tracked and you have a contract that says I don't want to be tracked, that's another thing. But I'm wondering how that works in a peer to peer way. It could be that maybe the way two peers begin a ceremony of, of acquaintance is, I agree to these terms, you know, I agree to your terms and I'm not going to share this with anybody else, for example.

Or that, I think it's, I think it's, I think, I guess it's a little key, you say embrace your digital identity with key identities. And so I'm wondering a bit about the identity side of this. Tell us about that identity thing you've got going and, and, and how that works. Cause I'm, I'm looking at that.

That's not key. It's I'm clicking on links here. I think it's, I think it's, I guess it's still key because you say embrace your digital identity with key identities. Yeah. What is a key identity and how do I get one or where do I use it?

Mathias: Well for us and I would probably add his five cents because you have to go soon but for us, It only starts at a cryptographic identifier, basically, right?

Because it has to be something that you can prove that you own and nobody else can prove. And then, so that's, that's the very, the very one liner answer for us is that it's a crypt, it's a key pair. And we have that's the, the thing you can, if you want to, in Keed, you can make a, make a key pair and that's, that's used to, Identify yourself across things as something you own.

And we have this whole duration protocol where we derive different ones for different things. But then also it's, it's about, you know, building a social stake in that identity. I think it's kind of like having a key pair is maybe not that interesting, but having a key pair that, you know, is me is super interesting for me in other cases, because that means that I can use that as my, Proof on other sides to log in as myself, as they follow the same standards as we do.

So I think it's very social also in that sense. It's like, it's a very technical problem, but it has to be something where you build up a bunch of social things. It's a social key pair, basically.

Doc: It's a social key

Mathias: pair. It also has to be opt in private. In that sense, what I mean is that, You shouldn't have to reveal your identity always you have to you have to have the right to Be in a chat and then be an anonymous person and then afterwards technically be able to say actually that was me Like kind of like the mean way to take off the hat And that's something we build in also.

So that's that's that's that's very Close to our heart for identities and I think just to touch on your thing on on on the terms I think peer to peer is actually really Close to what you're describing, because in peer to peer, there's no compute because compute happens everywhere. So you have to agree on like, what's the computer engine we're doing, which is like saying, what's the, the contract of which we're writing the rules of the chat with, and that's kind of like what you agree on normally in a peer to peer system upfront saying, well, I'm running this engine, I'm running this engine.

Okay, cool. So now we can chat because we're producing the same code. So it's kind of like, you're, you're kind of like putting that into a technology, technology. This gets me to

Doc: another question and maybe I could be really brief with it. Do we need a whole new compute engine? And I'm thinking actually of even a portable one, one that we carry around with us, like we carry a wallet or a phone.

Like basically the phones are lost to us. All right. The phones belong to Apple and Google and, and there, and every app that's on it and all of those are tied into Apple or Google and even Signal. They're all tied in. But what if we had one that's truly ours? It's just ours. It's my, I mean, like the wallet in my pocket is mine.

It doesn't matter whether Gucci made it or somebody else made it. It's mine once I bought it or once I've acquired it. And I don't have a slave bracelet. I just have this thing that I can select it. I can selectively disclose to other parties, whatever I want on terms that are agreeable to both of us.

And we sort of start with a new platform, but a platform that's actually a device. A compute, you know, a new compute box, but it's portable and maybe it's based on, maybe it runs Android, doesn't matter, you know, if Android is truly ours still in some level. I don't know if you thought about that.

Paolo: So I'm a big fan of of also pine 64.

I'm not sure if you tried their Linux phone. I also been following purism delivering phones as well. I have this hope that Linux phones will be possible in the future. Unfortunately, the amount of money invested in making Linux phones. Is negligible compared to the investments in well Android.

You can still claim it. That is you have a OSP, right? That is the Android platform. That's still based on on Linux. And you know, you can see that it can become really good. I'm a big fan of graphene OS. That you know, that's a Google eyes, you know, version of Android. So the definitely think is possible to, to have a proper version and a good phone that is vanilla Android or vanilla.

Well, vanilla Linux is trickier because, you know, it's all the user interfaces are not as as good as with Android. Honestly. I tried with Mobian and PostMarketOS and so many others, but that's, they will never, they, well, they will never is not a good answer, but it's, it's years behind OSP in my opinion.

But definitely our, the, the beauty of of pair runtime and kit is eventually you can, you can start building your web of trust. That is made by people and applications, right? The concept of better web of grass is super fascinating. It's basically, it's almost like the, the, the ability of, you know, what search engines use in order to rank content.

You can run people through connections. And that's something that we are looking into with Keith, right? When, as soon as you start building your social graph, you have different levels of undirection to contact people and that, and, you know, you have different weights for people based on your interactions and it's all local information.

You don't need to share. You don't need help to build your social graph. From a server or from meta, right? It's all local. It's all local data structures. And plus on the identity, I think the thing that really I like of what we achieve with the identity is that first of all, it's definitely opt in. But the way we design it is that you use the same Logic, the same concept of the bit 39 that that Bitcoin uses.

So now you can generate your identity starting from 24 words like in Bitcoin, so that with, you know, you can see with Bitcoin that you can travel across countries, you cross borders, and you can recover all your personal wealth that you have in Bitcoin just with 24 words. And we wanted to apply the same concept here with with key identities so that with 24 words, you can restore your identity, you can access to your entire digital life through 24 words.

I think is, is just a way to, or is a good way to give people a sense of freedom because we, 24 words, you can, you can access money and, and the freedom of speech altogether.

Jonathan: So I, I'm curious, you know, we talk about this idea of using it mobile. What, what does it look like using the pair runtime and Keat in particular on mobile?

Are there Android and iOS apps and are they fully featured or did we have to make compromises to be able to get it there?

Mathias: So no, it's actually, it's JavaScript because it's the lingua, lingua franca of, of modern development actually makes it easy. We run almost exactly the same code on the phone as we do on the desktop app for the peer to peer because peer to peer doesn't care where it runs.

That was really hard, but the runtime which is a JavaScript engine can pass everywhere. And so Qt itself is, it's actually. It's a fully featured app on the phone and has the same amount of features on the desktop because it's the same app. It's a peer to peer app at the non UI level.

Obviously, the UI is different and the only down, the only compromises you have to do on the phone is. Obviously the phone it's actually very Frustrating to me as a peer to peer developer because you have your phone with you always but you know The that always turns it off all the time So you're you're even though you have your device with you the apps are suspended more and you need to think about that a little More.

Yeah, that's something we we solved in the runtime. So it's It's not sitting there in the background using tons of battery and stuff like that. It's very, very friendly.

Jonathan: Can you do the, the update pushes through the, through the network? So I guess the question there is do Apple and Google let you self modify your own apps?

Or is that something that they will block you out

Mathias: for? Like, like, Like we talked about before, you're in a, you don't own your phone. You're borrowing from Apple, Google. So you have to follow the guidelines obviously, but within the guidelines, you can actually do a lot. That's like fine. It's kind of like, it's very equivalent.

If you know about, you know, people make apps with, with their views, where they load a remote thing, as long as you don't change the app, like you're not allowed to. That's like this is my pit camp and then turn it into a casino, for example without going through the app store. So you have to follow the rules and you have to follow, like, if this is a significant upgrade, you have to get them to approve it.

But anything else you can do over the air for the same P2P engine as you want to with, with, with a few constraints, but like, so it's actually pretty, pretty powerful. And I think it's one of those things where I think. I do think I'll relax this also more in the future because that's a little bit where some of the stuff is going where you can do more and more this way with the custom app stores in the future, but it's, it's it's really dang powerful and I'm surprised that not more apps are doing it.

Well, I'm not surprised because it was really, really, really hard to get it working. But but but once it's working, it was not that bad.

Jonathan: Yeah, all right. So we are getting close to the end of our hour before we let you go I do want to ask about the involvement of tether and I think you touched on this earlier Tether is a cryptocurrency company, aren't they?

How does that how does that piece fit in?

Paolo: Well Well, I'm the CEO of Tether. So

so for me, this is a life work and life passion. And so we have been extremely successful with Tether. And one of the good things about when you are successful, you should learn about giving back and invest in infrastructure that made What we did with tether possible, right? Without Bitcoin, there would not be tether.

Without many of the cryptographic primitives that were built over by the cypherpunk movement, Bitcoin would not be possible. So I think for us, we, when I think about tether we build products that disintermediate people, the problem with disintermediation, well, disintermediation is possible. It was,

Episode 780 transcript

23 april 2024 | min

FLOSS-780

Jonathan: This is Floss Weekly, Episode 780, recorded on Tuesday, April 23rd. Zone Minder! Better call Randall.

Hey, this week Aaron joins me and we sit down with Isaac Connor to talk about Zone Minder. We cover what's happened in that project in the last few years, why you might want to upgrade to Bleeding Edge, where AI actually makes sense, and more. You don't want to miss it, so stay tuned.

Hey, it's time for Floss Weekly. That's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we've got, we've got Aaron. We've got the, uh, the, the Retro Geek. I don't know, the Retro Hack Shack's what the YouTube channel is, but you're not the Hack Shack. What do you go by on the channel there?

Is it just Aaron?

Aaron: Yeah, just Aaron. It's fine. Yeah, I don't, I don't, I mean, there are different, in different forums, I am known as, uh, under different names, but Yeah on the channel. I just go by Aaron for sure.

Jonathan: Okay, you need some Retro styled moniker to go with that just just for the flair of it.

Aaron: Yeah I know I've also been trying to think of who how to describe my audience I'd like to give my audience a name on the channel, you know GPT to help with some some nice nicknames for my community and I think it came up with something like the The shack hackers like to reverse it, you know, the shack hackers and I kind of like that But my wife was like, I don't like that.

So I don't know. I'm still thinking about it I'm gonna ask the community for help.

Jonathan: Yeah, throw it out there and see if they came up with good ideas And that'll be fun. Yeah. Well, so today we've got uh, we've got a neat guest somebody I've So full disclosure, I've actually made a little bit of money off of this project.

I've put together a server and sent it out that is running Zoneminder, which is what we're talking about today. And we've got, we've got the man himself, Isaac Connor, or Icon, as I first, it's, it's funny when you meet someone online and you get to know their handle before you find out their name, there's always this weird dichotomy in your brain.

Like, he's, it's Icon. Oh, no, it's Isaac. That's actually the same person.

Aaron: Yep, that happens all the time for me when I meet people in person. It's like, yes. Oh, you're that person online. Oh Now I know

Jonathan: Never met you before. Yeah And then and then there's the fun of you don't look like how I imagined you looking based on the way you type

So Aaron, you, you kind of let, uh, you kind of let slip before the show. You, uh, you're an insider too. See, I, I was hoping Aaron would be the, the uninformed one that could ask the dumb questions, but you're not going to be able to today. You, you have a zone minder install yourself.

Aaron: Yeah, I do. And I really, really enjoy it.

So I mean, I'm not an expert in it, right? I just hook it up to get my recordings. Um, you know, my use case is, is pretty simple. Uh, where should I go into what the use cases and why I started with it real quick, or do you want to bring in Isaac first? Let's bring

Jonathan: in Isaac. And then I do want to get your thoughts because I think it would be fun to get kind of a, some live feedback, what works, what doesn't work and what the pain points are.

So let's go ahead and bring. Isaac onto the show. Welcome, sir. We sure appreciate you being here. Hi, thanks for having me. Yeah. Thanks for, thanks for stepping in kind of last minute. We didn't have a guest today. And so a couple of days ago, I got ahold of Isaac privately. I'm like, Hey, You're, you're on. You told me you'd be on anytime.

Well, it's, this is the time.

Isaac: Not a problem. Always a pleasure.

Jonathan: Yeah. So let's, let's start with kind of the 30, 000 foot view as we are want to do. And let's let Isaac take that and just kind of give us the quick background of what ZoneMinder is and what it's for. And then we will kick it over to Aaron and kind of get his, uh, his take on what he's using it for.

Isaac: Okay, well, uh, right off the bat, it is what we call a VMS or a CCTV VMS, which is a video management system. It's possibly one of the oldest ones in existence. Um, we actually just, I don't know, I didn't celebrate, but we made a point of mentioning that it's over 20 years old in terms of, uh, well, Git history and SVN history before that.

Um, it was, uh, originally started by a guy named Philip Coombs, who is still alive and well out there, but, you know, lost interest 10 years or more ago and was gracious enough to hand it over to a few guys, one of whom was me, to continue maintaining it. Um, and so that's about 10 years ago, and that's what I've been doing ever since.

Um, and so it's, uh, it's a, Being a free and open source, uh, package, it has all of the issues that they all have, right? Um, people come along to add the feature they want and don't document it. And that is left to someone else to maintain and all that sort of thing. And, um, but over the years, enough people have contributed and, um, You know, over the last 10 years, I've sort of taken a more commercial approach to it, um, making it fit my needs, but also trying to do some polish around it and make it, it's actually a serious contender for a proper enterprise grade commercial VMS.

Aaron: Yeah, for sure. Um, how, how much changed with a project that old? I always wonder like. How active is the, is the project right now? Are there any major releases coming out or is it pretty much all just minor tweaks?

Isaac: Um, It's been several years since the 1. 36 stable release series, and I've been doing a lot of work.

Not just me. When I say me, I'm going to call that the royal me. Um, you know, all the guys contributed to it, but really, most of the bulk of the work is me. Um, a lot has changed, and the world of video capture and the codecs, and even just keeping up with MPEG library API deprecations is a massive amount of work.

Um, so if you look in the sort of the nitty gritty, Yeah, not that much has changed, um, but then if you pull back a little bit, you look through the commits, you're like, oh, wow, um, you know, a recent example, uh, you know, we've got a contributor who, you know, I've having a look at, um, JPEG encoding, because for viewing, that's what we do a lot of, right, because, you know, the browser still doesn't support H.

265, so we end up re encoding the JPEG to watch in the browser. And like we're trying to switch over to the FFmpeg libraries to do that because we're at the moment where it's this 20 year old code. It's using libjpeg turbo. And, you know, we can't take advantage of any hardware acceleration for that. And it's, it's complicated code.

It's old because FFmpeg wasn't good 10 years ago. You know, um. So I'd say there's a lot of change. A lot of UI changes to make things nicer. Um, those are controversial. People always want simple. And then they want this button that does this thing. So we end up with a lot of buttons. But I think there's been a lot of polish.

And there's certainly been a lot of comments from people like, Oh, I just upgraded to the development series. It's amazing. You know, it's such a great thing. And you know, that makes me feel good. But, um, So there's a lot, I'd say. It's pretty active, um, if you sort of pull back and look at it.

Aaron: Yeah, that's something I didn't consider, uh, when I asked the question, actually, is the fact that the standards just keep changing all the time.

Or seemingly, in a way. I mean, and like you said, you don't have backwards compatibility, um, all the time that you can rely on. So you have to have workarounds, I'm sure. So that you can you can keep things as consistent as possible. I know for me I started just personally I started putting up So at the makerspace that I started about eight years ago or something We wanted to have some security cameras in there to make sure that You know people weren't sleeping in there overnight That you know, no in case like we have had a couple instances where somebody Was in the parking lot, like, trying to break into cars and things like that, and so we were able to use some of our cameras, uh, to, to provide some things to the police when they came and were asking us if we saw anything, uh, which is great, but when you try to set up your own network of, uh, monitoring cameras or security cameras, it gets quite tough.

And so when I graduated from using those little, you know, ESP 32 cameras and Raspberry Pi cameras to actual, you know, not professional stuff, but at least in my case, you know, decent cameras, right, that could do things like, uh, infrared and, and things like that, uh, automatically switch for night mode and things.

Um, I needed something a little bit better, and that's when I started looking into Zoneminder. Um, so I can definitely appreciate coming from a, from a DIY effort, right? Of we're just gonna do MJPEG capture and put that on a, on a, on a web page somewhere, right? Live. So that we can see what's going on when we want to, to, oh, I actually really need to record this stuff.

Uh, in case, you know, something does happen. You know, how do I, you know, I don't have the performance I need because, you know, the webpage gets bogged down after I hook up more than a couple of cameras. Right. Um, so going from that to zone minder for my personal use has been really, really amazing. Um, do you find that, uh, People generally have that experience, like the first time user of zone minder.

Like, I don't know how many people you actually talk to or get feedback from, but I mean, is that first time use for someone, the casual user, I guess, uh, kind of taking care of, or is it really meant more for the professional business user that has, you know, 20 or 30 cameras that they want to capture a video from?

Isaac: Um, I do talk to a ton of people and I mean, all the various forums, the Facebook chat bubble, I talk to a lot of people. There's different levels of people, right? The end user generally doesn't know what they're doing. And the first question is, if they can get it installed and running, the first question is, like, how do I add a camera?

How do I find out what the URL is? That's hard, right? And that's something that I've done a lot of work in, in 1. 38. Like, ONVIF takes us a long way there. Like, it's a great standard and stuff. Um, and our support is decent. But For all the cameras that aren't ONVIF compliant, uh, it's hard, right? You've got a, you've got a Google and nobody wants, nobody Googles.

Right. Doesn't happen. That's not

Aaron: something you typically look at. What is that? But can you define that for people that are listening that might not be familiar?

Isaac: Oh, ONVIF? Uh, What does it stand for, uh, Jonathan? I'm Googling now. Or even what is it? What is it? Yeah, it, it, it's a standard, um, pioneered by Axis, who is a, one of the major camera manufacturers and it's a XML SOAP based, uh, protocol for discovery and querying and commands and everything else.

Jonathan: It is, it is the Open Network Video Interface Forum.

Isaac: Yeah, and so that's great, but lots of cameras don't support that. So I've been doing a lot of work actually with, um, uh, based just ARP scanning, recognizing the manufacturer from the MAC address. Um, do a little bit of programming, and so now in 1. 37, we have a scan button, it'll scan your network and list all the devices and any that it can recognize, you just click add, and that's great.

Um, so I've been trying to remove a lot of the pain points for the end users, but really, the other side of it, as you say, the commercial, the large installers, the guys with 50 cameras, um, you know, they, most of the time, they, They know the URL, they're using a professional grade camera that we already support and so they get one and they copy and paste their 50 times and, and, uh, can onboard cameras really easily.

Yeah,

Aaron: so it does, because that's what I found that's been, you know, some of my feedback, which is not good or bad. It's just. It does require a little bit of know how, right, to get things set up. Um, you know, in my case, I bought some cameras, I can't remember what brand, I can look it up if anybody's curious.

Um, on Amazon, and then I had to use their web app to get the initial setup done, right, like the discovery and get the IP address. And I could have grabbed that, of course, off my, uh, DHCP server as well. But anyway, you know, that was the easiest way to find the IP address. And then once I had that. Then I could start plugging things in, but then you also have to look up, at least in my case, tell me if I did this wrong, I also had to look up the specs of the camera a little bit, right.

To see like what it supported. Um, and, uh, and then understand in my head, like, okay, how much video am I going to be capturing and how much storage is this going to take up and how much network bandwidth is this going to take up so that I could balance out my needs. here at the house with what my network would support and how much storage I had.

So a lot of that, I kind of had to become a, uh, um, uh, what do they call those? Uh, uh, if you're in the electrical work, you're, you become like a journeyman or something like that, or, uh, you know, so I had to become like a little bit of a CCTV journeyman in order to understand how to set it up correctly.

Cause at first I was kind of frustrated cause I wasn't getting, um, the results that I wanted. My storage was filling up too fast and I had to figure out how to expire recordings and, um, set up the, the policy that you do for that and everything. So, um, it took a little bit of know how and that's kind of where I see like the entry point for this.

This wouldn't be for someone that doesn't know anything about, um, this field and how it works, right? For that, you would just use the web app that came with the device that you bought, probably. It's like the next level up. Would you agree? I

Isaac: would totally agree, and, and yeah, and I guess that's, that's actually why, you know, all these cloud based or proprietary systems are, are popular is because your average person doesn't want to get into all this stuff, they just want their, their, to be able to call up the footage on their phone, or, you know, and, and so those solutions work for that, um, for some, those of us who don't want to send our video to some corporation in the cloud, too.

You got to dig in and this, and that's a lot to dig into. Like this is, it's, it's mind blowing how much there is to think about and tweak. And, you know, you get into motion settings. Okay. Well, that that's a whole thing. You know, we try to make it simpler and there's still so much work to do there, but, um, Yeah, anyone who, who starts to get into this, it's going to become a hobby, you know, quickly you can make a profession out of it, which is basically what I've done.

Aaron: For sure, Jonathan, I know you probably want, sorry, I don't mean to monopolize the discussion here, so jump in and ask some questions. I'm going to look up, uh, those cameras that I mentioned. I'll talk about my setup and why I chose this setup and how I use ZoneMinder too, but why don't you jump in with some questions and then I'll do that in the background.

Jonathan: Yeah, so we, we kind of just alluded to something and I'm, I'm curious about it, this idea of maybe you don't want your cameras around and maybe even inside your house connected to the internet to be able to send footage off to who knows where, you know, sometimes off to manufacturers that aren't even based in the U.

S. and therefore aren't Aren't, uh, aren't, aren't privy to, uh, you know, they're not party to U. S. law. Um, so I guess the first question is do we have any evidence? Like have we actually seen that being abused by these companies? And then how does ZoneMinder fix that problem?

Isaac: I don't know if we've seen abuse of it, but there was, for example, the Wisehack, where something went wrong and for a period of time people could see other people's footage instead of their own, right?

You know, it's not, it doesn't have to necessarily be malicious, um, it could be simply incompetence and we've seen lots of examples of that. Um, not to say that we are perfect, but when you. When your video doesn't leave your house, you don't have that problem, right? And so how we solve that is by giving the option of the video never leaving your house.

Um, you know, that's, that's how it's done. You have your server, it's in your, you know, Basement or office or whatever. And, you know, it doesn't even have to be a big server with loud fans, you know, for your home use, it could be a Raspberry Pi, it could be an Intel NUC, um, anything, any old PC, an old laptop, anything is, is probably powerful enough to handle most people's home needs.

Um, and so that video never leaves your house. Someone has to hack into your house before they can get it. So it's safe by default ish, you know, um, as safe as anything else that you have on your network.

Jonathan: Do you, do you recommend to folks to do like a, a separate network for the cameras if possible?

Isaac: I would always recommend it, but let's, let's be real.

Most people don't have the knowledge or the time or the effort. Um, and a lot of these cameras you get, you've got to put them on wifi. Bye. Well, how many people are going to segment their Wi Fi and, and all this sort of stuff. It's a level of knowledge that people just aren't going to do. Yeah, that's true.

And for your home, maybe that's okay. You know, because again, if someone's going to hack into your network, they're probably going to be able to hack into any other network you've got too. But on it, obviously the ideal setup is a wired, uh, um, uh, Segmented LAN that can only talk to the ZoneMinder server.

And then it's like a firewall, right, between it and, and, nothing can get those cameras, um, if proper firewall rules, they can't get out, they can't send anything. And that's, to me, the ideal setup. And I mentioned, uh, wired instead of Wi Fi because it's trivial to jam Wi Fi. You know, if someone wants to break into your house, and if you're like, not a casual criminal, but like a professional criminal, it's very easy.

Okay. Suddenly all your Wi Fi cameras aren't showing anything and you can go to town and there'll be no footage of it, right? It's gotta be wired. Um, So that's the ideal, but let's be real, people aren't gonna do that. So, whatever. There are

Jonathan: very

Aaron: few

Jonathan: of them anyway. There's a, there's an almost like 95 percent good enough solution there, where you just go into the camera settings and you give it a bogus router IP address.

That's And that, that'll take care of almost all of that.

Isaac: Yeah, exactly. I mean, in this case, in a, in a good zominder server is also providing DHCP to that segmented LAN and. Don't give it a gateway or, you know, the gateway is, we can provide time services. You know, that's another big thing that, you know, people don't address and they really need to.

Um, the time in the video has to be right or it's useless for law enforcement. You know, the first thing the lawyer will toss out, you know, the video clearly shows it was in 1971. You know, it's, it's got to be right. Yeah, that's true. That's true. All this stuff is way more complex as I'm going in and

Aaron: checking my timestamps, , as soon as you say that, are they all accurate?

Yeah. Yes, they are.

Isaac: Daylight savings time is another big one. Like every time they change those, well, the camera doesn't know about it. You've gotta, it's gotta go in and change. It's daylight. Savings times rules. No,

Aaron: they're not. Yeah, these timestamps are not accurate.

Jonathan: All you need to get 'em right. If you ever try to use the evidence.

What times is it?

Aaron: Okay, this one says 9 52. This one says 1152. This one says nine o'clock and I can't read that. I'd have to go into the larger view. Thank you. I'm going to go fix these. Yeah,

Jonathan: there you go. Now. So this is, this is an interesting question. It's something that comes up from time to time. Are those timestamps provided by the cameras or are they provided by zone minder?

Isaac: Is that a question for

Jonathan: Aaron or for me? Well, I guess, I guess for Aaron first.

Aaron: I think these are from the camera.

Jonathan: Okay.

Aaron: I, I think there's another timestamp, a littler timestamp from ZoneMinder on these. That is accurate. Yeah. What's, what's

Jonathan: real fun is when you have two timestamps on a, on a, on an image and, uh, you know, they may be showing something different.

Um, there's, there's kind of an issue with ZoneMinder. Yeah. And I don't know how we would go about fixing this, but, you know, folks should be aware of it. Um, and Isaac tell me if this is not the case anymore, but last I checked, uh, if you're doing pass through recording, which means we're not recording the JPEGs, we're just taking that H.

264 stream from the camera and dumping it straight to disk, ZoneMinder is not writing a timestamp on there. That's still the case, right?

Isaac: Absolutely correct. Yep. So, um, to do, to put it in there, requires re encoding the video, which, you know, now you need some, some, you know, GPU, you know, which, which can, you can do.

Lots of people have a GPU that's well capable of doing that. Um, and in fact, you know, most people, if you've got the Intel GPU, Intel chips have a, have a great, uh, media encoding support in there, and that will work just great. And then, you know, we can write anything you want into it, but, you know, so many systems It's running on systems that don't have a GPU, maybe because they're old, or, or perhaps it's on a server hardware that, you know, they give them these crippled little It's not even a real Matrox chip, it's, it's just Speaks that language and it's totally crippled in terms of video hardware, they don't have anything.

So it's going to be all CPU and it doesn't work. Um, I'm not entirely sure how we could even address it other than, you know, maybe you could occasionally like put in a timestamp, like only re encode. You know, every 10 frames or something, but ideally, you know, I think the best way is to have the camera do as much as possible.

Um, and that brings me to someday I really want to do a line of cameras that are running zone minder in the camera, because I just think that would be awesome and you wouldn't need big servers, uh, you know, the cameras are powerful these days. So that's the dream, but, um,

Jonathan: interesting. Yeah, that's a that's a really interesting idea and I know I know you've kicked that around before Uh, in in fact, i've got a camera behind me on the floor That's taken to pieces with me trying to look at the cpu like I wonder if we can get into that I wonder if I can ssh into that and there is a project out there for doing uh, custom camera firmware Um, it's just the problem there.

It's the same problem that like open wrt has and your custom TV firmware has. There's, there's literally 5, 000 between your brands and your models and your revisions, hardware revisions. There's literally 5, 000 different cameras out there to try to support and it's just, it's basically impossible to have one piece of firmware that would run on all of them.

Isaac: Yeah. Again, an open WRT like firmware would be fantastic. And I did once actually put open WRT on a D link, DCS 9 32 L camera that I had, that I, that's great. I, you know, I got pins on the jeg spots and, and they're really easy to get into, right? Mm-Hmm, . So I was able to do that and I got it working. And, um, I don't know, the cameras are so old, it kind of lost interest.

It's still in a box somewhere, but I've lost interest. I think if we were to go this route, we would simply contact a Chinese manufacturer and, you know, we'd have to order a thousand and just go through the requirements to, to put it on. Because they're all running Linux already. Um, compiling the, the subset of ZoneMinder for them should be pretty easy.

Not that much work. Um, anyway, someday we'll get to it. It's just, there aren't enough people contributing to zone minder to do it. Um, yet.

Jonathan: And, and nobody's written you that big seven figure check for setting up a zone minder server somewhere.

Isaac: No, no. I'm sure they're out there wondering how to contact me.

They're probably listening.

Jonathan: Sure. I'm

Isaac: here. I'm here,

Jonathan: guys. Seven figure checks are accepted.

Isaac: Yeah. State actors? I'm here. If you

Jonathan: Yeah. That's uh That's something that Do we want to talk about that? I think that's on your list of things you sent us. Uh, about the whole XZ, the LiveXZ vulnerability. Do we want to get into that?

Isaac: Um, I don't know how much we can get into it, but I, I, when I was reading that, it really hit home to me. Um, because I have felt that, that the whole process they talked about where um, it sort of broke him down, um, I thought you had blanking on his name now.

Jonathan: Lassie Cullen, I think?

Isaac: Yeah. And I, and I just like, oh man, I feel for the guy because I've seen it. Yeah. And, and I haven't succumbed yet. Um, every patch that comes in, I read every line. I'm looking for those vulnerabilities. Not with the assumption that they're malicious, but let's face it, not everyone's a very good coder.

And, uh, you know, we have to be secure. And so I'm looking for. You know, are you assuming that this input is proper and it's not, everything's got to be sanitized. And stuff gets by me, I miss stuff too, but the point is I do my best. And I've never really felt like I needed to give up maintainership. I would certainly welcome, and I do welcome more maintainers.

That's, you know, something that, uh, That's a problem I think Zominder has is, is that we don't have enough. No one's reviewing my code. I'd push for master all the time and no one is reviewing my code. They're all trusting me. Uh, I wish that wasn't the case. Didn't used to be the case. We, and we used to, we used to have a bunch of guys, um, Uh, Andy, Kyle and Steve, who, um, were very active and they're less active now, you know, their, their lives have taken them off in other directions.

They're still around from time to time, but they're, they're, they're Not reviewing my stuff. So, um, and that, that's a, that's a problem every package has, and it's probably susceptible to, is there should not be one maintainer. Um, it doesn't matter how much of a saint or, you know, extreme professionally, whatever superstar he is, um, or he is mm-Hmm, , uh, or he or she or they or whatever pronoun.

Um, it shouldn't be down to one person. Mm-Hmm. , it can't be, um. But I've definitely seen, um, behavior from people that is, is really breaks you down and makes you not want to be a maintainer. And, uh, so I just, I just thought that was really interesting and it's definitely happening.

Jonathan: Yeah. Do you, are you kind of, uh, suggesting that maybe some of that was intentional or are we just talking about, Human behavior can be disheartening.

Isaac: I think it's probably both. Um, and also culture is a big thing. You know, when you're dealing with people from around the world, Um, it can be a language issue, Um, it can be a cultural issue, Um, and, If you're not well traveled in the world, you may not understand that that is what's happening.

Aaron: Um,

Isaac: I've certainly had to learn, to learn that, and I've learned it kind of the hard way.

Um, different parts of the world, uh, operate a little differently.

Jonathan: I always try to keep that, that potential language barrier in mind when I get an email or I see a pull request or a bug report. You go, you read through it and your first thought is, man, this guy's an idiot. Like, just honestly, sometimes that's what comes to mind.

And then you realize, oh, no, wait a second. English is his third language. I'm the dumb one here. And he's actually making a really good point. It's just hard for me to parse it. Mm

Aaron: hmm.

Isaac: And it can be real hard to keep that perspective in the back of your mind, right? If you're in a, something else is stressing you and, you know, It happens, we're all humans, so.

But the XV hack makes us think, oh, maybe sometimes it is malicious. So you gotta keep that in the back of your mind too. It's scary, and especially, you know, in this time of wars and upheaval, uh, you know, you really got to keep your wits about you.

Jonathan: Yeah, true. Especially working on something like ZoneMinder.

Because it, you know, not, not the kind of target that SSHD is, but like, there are ZoneMinder installs in, uh, potentially sensitive places. I, I know some of the customers that you have had in the past, and I am sure that that would be, some of those would be considered, you know, sensitive manufacturing. Um, so, yeah, that's something to keep in mind.

Aaron, you were going to tell us a little bit more about your setup there, I think.

Aaron: Yeah, and I did have a question, um, but I'll, I'll tell you about my setup, and then I'll, then I'll, uh, get your, get your thoughts, and I'll, uh, I do have a question on cameras as well. But I'm using, I looked them up, I'm using, uh, Amcrest.

A M C R E S T. We like Amcrest. Cameras. Yep. And actually I'm using three different models. Um, because I needed a mix of some Wi Fi. Um, and some, well, I didn't have to go with the Wi I could have gone all Wi Fi, but there's places where my Wi Fi isn't as strong. And so I decided to go power over Ethernet, uh, wired cameras for those.

Um, so I've got those. They're like, uh, Ultra HD, 4K, 8 megapixel. outdoor cameras, uh, blah, blah, blah. They have the night vision and stuff. And they were pretty reasonable, you know, they're off brand. I'm assuming that they're probably, um, I don't know much about the Amcrest brand, but they seemed pretty good.

The reviews seemed pretty strong. So I got, uh, like I said, I got a mix of those. You've got the wifi ones for the ones that can do wifi, uh, with a pretty, pretty decent, um, rate over the network. Because I didn't want to have to run cables everywhere too, right? I mean, it's a trade off. How, how much, how much cables do you want to run, especially in your house?

How many holes do you want to drill in your walls and have to refill so the cold air doesn't get back in? Um, and, you know, so, so, you know, one of them is just completely Wi Fi. It just plugs in outdoor. Yes, I know someone could come along and, well, actually, no, that one I wired into my light socket. So, but you could come cut the cord, right?

Like if you were a nefarious person, you could very easily just come and cut the cord and turn that one off. But anyway, so I'm using Amcrest, um, and they're, they're working pretty well. They do motion detection pretty well. And the reason we got them was, was, was. Two fold, but really more for my wife than anything, because my wife is a 7th grade English teacher.

And, uh, what we found is that occasionally, uh, a young child will get upset that they got caught cheating on an exam and got a F on their exam for, for cheating on it. And, uh, sometimes kids like to carry eggs with them apparently. And, uh, you know, if you don't have any evidence that a particular kid, even though you're 99 percent sure which one it was, uh, did something to your house, then there's nothing really you can do about it.

So we decided, you know what, let's just get these, uh, get it, get them installed. And then if anything does happen, at least we've got some, some video evidence of it. And then also too, since I'm a YouTuber, you know, You just never know, right? My stuff is out there, it's pretty easy to figure out where I live, so, uh, I just figured, you know, better safe than sorry.

Uh, have something recording all the time and, uh, capture, capture that if something happens. So that's my, uh, physical install in my story. I'm running zone minder 1. 36. 33. Um, haven't upgraded it in a while, but I, so I don't know how far behind that is. I think you mentioned 1. 37 before, but I don't know how many minor releases are in between those.

Isaac: Yeah, it might be worth mentioning that. The way we have always done versioning, and I don't know how much longer we're going to keep this pattern up, but the even numbers are considered stable. So when I released 1. 36, everything since then, you know, through 3. 6. 1. 2, they're just bug fixes. Like really, no new features, um, nothing really changes unless we, it's like, really trivial.

Um, and then we go off and start breaking things. So the, if it's an odd number, 1. 37, uh, we make no promises about it, it working, you know, that being said, we fix things quickly. So I run 1. 37 in production for all of my, my clients. Um, I don't update them every day. Generally, I, you know, I know how stable things are.

Nobody's complaining for a few weeks. Okay. You know, update everybody. Um, and then I go break something new, you know? So it's kind of like a, it's like a sine wave pattern or something between stability and, and, and breakage. But very rarely do I ever break things really seriously. And like I said, if I do back it out real quick, get a new release out and, and, um, get things.

Moving on. So,

Aaron: yeah. So should I be looking at 1. 38 as my next upgrade target?

Isaac: That will be the next stable release. And we're starting to talk about like, Hey, maybe we should just release it instead of waiting for every single idea to be done, um, because it's, there's a significant, it's been several years of work and, uh, You know, I think there's actually been, uh, But there has been, there's been two full FFmpeg, if not three in that, in that time span.

You know, they've really ramped things up, so we've got to catch up. But, I kind of, I really feel like we need to be a little bit more like home assistant, you know? Like, maybe monthly, maybe quarterly, something. Because, um, honestly, I look at the 1. 36 interfaces and things, and I find them agonizing, and all these features that we've worked on, and I'm like, You know, that problem is solved if you'll just upgrade.

Um, so it's difficult. It's difficult, you know, figuring out versioning and stability versus new features. Um, and we have probably waited too long for the next release. But yeah, 1. 38 will be the next stable series. We'll pick a, um, A name based on a Metallica album, and, uh, off we'll go.

Aaron: Nice, nice. What about the, the, we talked about cameras and how you'd like to develop your own camera, uh, or somebody to develop a camera with ZoneMinder built in.

What about the ones that are on the market today? And do you have, like, recommendations to give to people that are more friendly to ZoneMinder or, or, or, you know, easier to set up on ZoneMinder? Thank you.

Isaac: Most proper cameras will be fine. Alright, like Amcrest, I just want to put a shout out to them.

They're the only company that's actually supported Zomato. They sent me a box of their entire home line. Unfortunately, they weren't the sort of more professional grade. They don't do ONVIF. They don't have a web UI to configure them. It's all with the app. But, Aside from that, they're, they're great cameras and the URLs are standard.

And, you know, they gave me those to make sure that they'd work with zoneminder and they do. Uh, so that's great. Um, so we like them, uh, and I also know, you know, their high end cameras are really nice. I know there's one guy who hangs out, uh, in their Slack channel and stuff, and you got some new ones. And, you know, proper optical zoom.

You know, he can zoom in miles away and see things crisply, clearly, and like, that's really nice.

Aaron: Um,

Isaac: and as we know, there's a, you know, whatever you can have on camera storage, you can, you know, their motion detection is good, it's all standards based, it works. Thanks. So love Amcrest. Um, VivoTech cameras, um, I've had experience with a lot of them, they work well.

One thing that's kind of interesting, and I've only sort of got into it, and discovered it, and figured it out, I find, now these are older cameras, uh, They're, I don't know about their new ones, but they really have unstable clocks. So back to like getting your time right, you've got to have NTP set up because in the middle of your stream, it'll jump back an hour or something and like, or it's too fast or kind of weird.

But in terms of, again, a solid camera, um, great reliability, easy to set up. As I've had experience with them, you know, Zominder will detect them and automatically configure it. Um, HikVision is probably the most popular, um, and, um, most of the cameras you're buying probably are HikVisions. They're one of the most OEM white labeled cameras there is.

Um, and today I find their hardware to be just fantastic. They have their own standard called, PSIA, which was a competitor to ONVIF, and I guess they lost. Except, can you really say they lost? Because probably most cameras out there are HikVision. Um, but we support both of those and, and again, they detect well.

The quality is fantastic. The reliability of the image. Um, China, like, Dahua are fine. Um,

Jonathan: all those Chinese, Hanwha. What if somebody wanted, for various reasons, an American made camera. Is there a company out there?

Isaac: That's a very good question. I think,

Jonathan: I think AXIS maybe?

Isaac: AXIS are actually, I think, based out of Montreal. They're actually Canadian, but that's good. I don't know. Of course, who, where's the corporate actual ownership? I don't know. But yeah, they're, they're North American. Um, where are they actually manufactured?

I couldn't tell you.

Jonathan: Um, I guess it depends who you're installing cameras for, how picky they're going to be about that.

Isaac: Yeah. And I mean, right now we have, you know, the governments are banning, for example, Hikvision cameras and cameras made in China. And so a lot of these other cameras, like I mentioned, Vivotech, they're Vietnamese.

Haiwa is Taiwanese. Um, you know, whatever, they're made outside of China, but still in the South Pacific. Um, Hard. You have to really do your research to find out where a particular brand of camera is made. And so, if you are a government agency, um, you know, obviously there, I'm sure there's a guideline somewhere of who you can buy.

Um, for the rest of us, do we care? Not really. Because, We either, like you've got to lock them down, don't trust a camera from anywhere, right? Lock it down, don't give it access to the internet, that sort of thing. Um, so that's what I have to say about cameras. Almost anything you buy today is good as long as it, it, if it says on VIF, that's good.

Um, But a lot of people that buy stuff off Amazon that it only works with the app. I mean, you mentioned earlier what's the first, you know, stumbling block people have. Well, their cameras cannot ever be supported by a zone winder. The video only goes to the cloud and through their proprietary app. Well, it can't do that.

You're going to have to go out and buy new cameras. Um, we've seen a disturbing thing. I wanted to mention another maker. Reolink has also traditionally been great. Uh, reasonable price, really good quality. But we're starting to get reports on some of their newer models don't work. They're going that cloud app only.

Um, and I don't have any, but I think I'm going to have to get some. Um, TP Link makes some nice cameras. Um, that. we can work with. Um, I guess they updated the code for doing the pan tilt zoom controls and our interface doesn't work with it anymore. So that's on the to do list to look into and fix. But it's, you know, cheap and good cameras.

Jonathan: There's been some fun things. So I'm sitting here thinking about what, uh, what Aaron is missing out on by still running the old version and, uh, Of course, I can only think of the things that I've worked on, right? So, like, the H. 264 passthrough right to the browser, and the ONVIF based motion detection events.

Those are a couple of things that are new that, uh, those are particularly useful if you have multiple cameras at high frame rate and you don't want to build a whole server for it. What else, what else is new in ZoneMinder that we can really tempt Aaron with to get him off into the developer releases?

Thanks.

Isaac: Oh, man, I, you know, in hindsight, I should have actually, uh, gone through and made a list. I started to make a video actually comparing the two one day and, you know, after two hours of recording, I realized I just had to quit and then I realized the audio was crap and I never got back to it. Um,

One thing that comes to mind immediately, um, we've got a great contributor who's just been going crazy on the user interface, you know, tweaking spacing, tweaking layout, um, and then he jumped in. And when you're zoomed into the video, if you weren't looking at where you want to look, you'd have to zoom out and then zoom back in somewhere else.

You may know, you can like shift. Click and like drag around and zoom around inside the zoomed in video, which if you're a person who uses zoom, that is so cool because it's painful, the zooming in and zooming out. Oh, I clicked the wrong spot. It's too far over here. You know, that's a big one. I might even have to backport that to 1.

36. Um, cause that one's really cool. Um, Oh my, um, there are features that. Um, don't show up. We're working on a floor plan thing so you could upload an image of your grounds and you could drop where the cameras are and show them and stuff like that. Um, geolocation stuff is part of that. You can, you know, do a map of the world where your stuff is.

If you're someone who is looking at a large graphical area, uh, there's that. You can where your cameras are in the world. Um, there's a new feature. It's on demand capture. Um, So this is for, you only want the system to be doing anything when you want to look at a camera, so instead of constantly streaming, constantly consuming resources, basically when you go log into Zominder and you click on that camera, it starts the capturing process then.

Um, So we have a customer and there are, you know, thousands and thousands of cameras out there around North America. But they're only interested, they just want to be able to view what the cameras are looking at from time to time. They don't, they're not recording, they're not doing anything like that.

They just want to be able to look into this place. So that is a great feature for them. Nice. Um, We've got this new feature called tags where you just you can tag an event and you give like a keyword, uh, which, um, should help in with the efficiency of motion detection and adding more data to, to events.

And, you know, so you, you, and just, I don't know, maybe your tag will be dogs, you know, you can now, as you're going through, you can flag every event that has a dog in it for later, whatever you want to do with that, you know, uh, really efficient, really nice. Is there,

Jonathan: is there anything around the idea of having AI doing tagging for you?

Isaac: Well, yes. Um, see currently the event server, um, it updates this notes field and it just adds text to it, right? And so I, I haven't pushed it, but I have a branch where we. Insert a tag instead. So instead of, you know, dog 36%, it'll tag it with dog. Um, and so then you can Like, currently Okay, so now the big thing UI wise, in the events list, I now automatically put, um, filter options at the top.

So you can, um, filter the events better. And one of them is, is, um, Is the notes field. And we had put default entries that, you know, come with YOLO, dog, cat, boat, car, potted plant, or any objects. So you can quickly jump to that and get all the events that actually have something in them as opposed to snowstorm.

Um, so that's a big one in terms of, of filtering down your events. Cause before you have to go and make a filter. You know, add all those rules and lines and, and, you know, that's not exactly user friendly, um, but you could do it, right? Um, which reminds me of another thing. Yeah, because I'm noticing

Aaron: as you're talking about this, I'm looking through my, um, my motion detect events, right?

And they're mostly cars going by. Like, I'm looking at the one that faces the street. Um, even the ones at night are, um, are just cars going by, right? So if I could just, like, filter all those out, um, because really all I'm concerned about, I think, is people. Yeah. So if there was, like, an AI that could just, like, filter out all the, all the ones that are just cars, like, unless there's a person in it, then get rid of it.

Well, you,

Isaac: you can configure your event server to ignore cars. Like, there's a matched pattern thing. Um, Again, this is not user friendly, uh, and that's one of the things on our list for 1. 38 is, you know, roughing in at least the basics of AI configuration in the WebZone Mitre UI because PlayablePixels, when he wrote all this stuff, um, you know, he used INI files and YAML to do all this configuration because it's really complicated.

And I don't think we can get totally away from that but, you know, at least defaults of, you know, Witch. Um, you know, whether it's YOLO 4 or 5, and I'm only interested in people and dogs, you know, we should be able to do that, at least in terms of the web configuration. But you can go in and you can filter out anything.

Aaron: Yeah, and we should say the filtering in here, the filters themselves are generic enough and powerful enough that you can do a lot with them. At least that's my opinion. Uh, So when we say filters, I mean, I've got three, three generic filters on, right? Which is mostly around my storage disk space. So like purge when full or delete old events.

And I've configured those to delete the recordings I no longer need automatically. So I don't have to, because the first thing I ran into was my storage device filled up on my server because I wasn't deleting anything, duh. Um, but, but you can do all sorts of things with filters. I mean, it's, it must be hard because I mean, You've got something here that's really written that can be used by anybody, right?

But it really has powerful features that can be used by security professionals too And so you have to be able to provide for both of those Audiences and i'm guessing the skew is more towards the security professional end Um, because if you only have one or two cameras You know, diminishing returns on, on zone miter.

Right. But if you've got like me, if you've got, you know, a bunch of them, or if you're in a, in a commercial setting where you've got a bunch of cameras, it really makes a lot of sense.

Isaac: Yeah. Um. You know, a common example, I mean, when you start getting into notifications, uh, you know, some people, they're like, every morning when I get into work, I want an email that lists if anything weird happened overnight, right?

So you can create a filter that looks for people. Uh, between the hours of this and that, um, you know, and, and that's really useful. It, it takes a little customization and, you know, it can email you that, or you can go so far as to have it SMS you when the event happens, right? That's why event server exists, because Playable Pixels wanted that.

He wanted to know when his son got home from school. He wanted to know if it was a UPS delivery, um, but it's complicated and then, you know, you start digging in and it takes a lot of effort to set up. And if you're You know, a business or a corporation, it's probably worth your time to give us a call because we have examples, we've done this before, you know, we can probably get you going real quick.

Um, and then we store those filters. And this reminds me of another difference between 1. is in 1. 36 there was only one email. Content body, whatever that you could do, you know, it was set up under options. Now it's part of the filter. Every filter can have its own different email, its own different, uh, from and to.

So you can now have different emails going to different people in the organization. Um, and we also, uh, created a, it's a summary email. Before it was, you know, one email per event. So you might have a thousand events in your inbox in the morning. Now you get one email. Um, a lot of work with, uh, inlining the attached images.

So on your phone, it, You know, the image just comes up, you know, it's just part of the email, uh, you know, instead of having to like save to desktop and view it, you know. Ton of work done in there to make that a little nicer.

Aaron: Are there, as you're thinking about it, are there features that like, uh, like Nest or Ring?

Um, you know, those kind of commercial devices, are there features that they have that you're like, oh, I wish we could do that. I wish we could integrate that. Like, what's on your wish list.

Isaac: I got a lot of my wish list, but that was a big one. Like, that's the two way audio thing, right? Is a big desire. Um, there are cameras out there with microphone or speakers and stuff, and we could do two way audio.

We need to integrate WebRTC better so that we can. It helps us with the viewing of the video, but also so we can do that two way audio thing. Um, and these are, these are proprietary devices.

Jonathan: Yeah. Well, the problem is WebRTC is, is such a, well, let's see how people described it to me. Um, if you're on the happy path, it's really, really good, but otherwise it's terribly broken and it's really hard to fix.

WebRTC is terrible and it does not just work.

Isaac: Yeah, well, they've done that thing where they pick a subset of everything that's out there to support. And so You know, it, it's, it, our gen, we did Janis didn't work so well for us because cameras frequently don't put out the right H 2 64. Yeah.

Jonathan: Mm.

Isaac: Right. Um, and so if you could get your cameras, I'm hoping that matures.

Jonathan: Yeah. If you could get your cameras to put out just the right H 2 64 baseline, that, that, and really what it is, it's the browsers because they have a separate code path for doing web RTC as opposed to all of their other video decoding. So if you can get just that right. H. 264 stream. It is beautiful and it works great.

Um, there is something coming. Uh, I have been haunting the, the, the Google Chrome development lists and, uh, there's active work on H. 265 in WebRTC and we kind of look to that as our savior, hopefully.

Isaac: Yeah, I feel like AV1 is going to come along before that actually comes to fruition, but um, AV1 is very exciting too.

Jonathan: But how many cameras will spit out AV1 streams? Well,

Isaac: there

Jonathan: aren't

Isaac: any right now,

Jonathan: so Yeah. Okay, so before we let you go, we've got to ask, I want to get this in, what's up with Xeom Ninja? And for those that don't know, that is the cross platform mobile application that kind of went away for a while. Is it back?

Isaac: Sort of. Um, So, for those who don't know, the author of Playable Pixels, uh, basically stepped away from all things. So, in minder, he went pure Google. Um, and he made it free, and, and, you know, the source was always open source, but, you know, basically gave it to us to do with what we want. And for, Two years, I sort of tried to get it to compile, but I, there's too much.

I'm not a JavaScript developer. I don't know any of these technologies. Um, you know, it's, it's a Cordova, Ionic, Angular app. And anyway, um, I was able to find someone who could help us out, and we got it compiling for, Android, uh, which is great. So there are new versions out. We've fixed a bunch of problems with it.

Um, I wouldn't say it's getting active development every now and then. I, I am supporting it. You know, I, I, I fix bugs if I can. Um, but the, at the end of the day, the technologies that it was built with have been abandoned. Like you can, it's like everyone who was using. These technologies just stopped two years ago.

And Android keeps changing its APIs and what it allows you to do, and so some features are broken. But in general it works. It's fine. People like it. They pay for it. And it's working fine on Android. We have not been able to get it to compile for Apple for iOS. But the old version still works. So that's okay for now.

Um But, you know, there are changes in the APIs for 1. 37 and it's going to break. Uh, so that, that's, it's kind of on life support, uh, for iOS devices. Um, what can be done? I don't know. Um.

Jonathan: We made it

Isaac: non free, you know, it costs five dollars. So the idea was to funnel that money to somebody who's willing to do that work.

Um, I don't know anyone who's willing to do that work yet, but if you're out there, if you're listening. Uh, we've also discussed rewriting it in, um, Flutter, is that

Aaron: the

Isaac: word? Um, I'm not going to have time. So it's really, it's going to be someone else who does that.

Aaron: Yeah.

Isaac: Um, but at the same time, we do put effort into making the normal Zoneminder web UI better on mobile.

There's been a little bit of work done lately. It's, it's improved a lot. There's a lot more to go.

Aaron: Maybe we can get, maybe we can get Randall to, to help with that. He's a big Flutter guy now, isn't he? That, that thought has

Jonathan: been kicked around. Yes. Well, like

Isaac: I said, it, you know, it's, our income from these apps is about 750 a month.

Which could. Even more than that could go to somebody if they got the time. The problem is I can't afford, you know, what do I, would I have to pay someone more like 5, 000 a month? I don't know. What is it? I can't pay a full time developer. Someone's got to do it for the fun and the glory.

Aaron: I'll agree. I've only used the uh, ZoneMinder.

I've never used the app, right? So I've only used the uh, the web. You know, pulled up the web page on my phone and it's been fine. It's not, not perfect. It's not pretty like an app might be, but it works totally fine for me anyway. And

Jonathan: it's getting better. One of the, one of the, one of the real problems, um, that is a problem with the app getting sunsetted is particularly on iOS because on iOS there's only one browser.

And, you know, you, you install Chrome on iOS and you just have, you have a Chrome theming on top of Safari. Um, and Safari doesn't necessarily always handle media correctly. So that's, that's one of the real pain points with trying to do some of the more advanced things like streaming H264 right to a mobile device that just tends to not work very well in, in iOS Safari.

Um, so yeah, it's a big problem. Solutions to be found. Uh, Aaron, do you want to get into anything? We are getting, we are basically at the bottom of the hour. Do you want to get any, any questions before I start to wrap?

Aaron: Uh, I mean, I don't think so. I think, you know, no, no, I don't have any other questions. Uh, you know, once I had this thing set up, I will tell people if they're listening and thinking about trying out Zominder, you know, once I had it set up It was great.

I mean, the initial setup took some know how, right? But now that it's running, it's been super reliable. I just go in, I look for stuff when I need to, when I don't need to, I know that it's running in the background, not in the background, but, uh, you know, at least in my mind, it's running in the background and it's just working.

And, uh, so, um, no, I don't have any final questions, but, you know, thanks to you and everybody that's, uh, that's worked on this project over so many years because, uh, you know, it really is a great, not only a great project, but a great service that you're providing so that people don't have to rely on their data being stored somewhere where they may not want it.

Um, and rely on those, the way that the camera developers want to, uh, position their products and work with their products. So we can all get a little bit more secure and, and uh, yeah, thanks. Yeah.

Jonathan: My pleasure. Um, I do want to ask, uh, there's about five different questions I was hoping to get to and we are out of time.

It's gone really fast. Um, are you seriously trying to write everything in Perl?

Isaac: Some, some days I want to PHP is, is an interesting language and I would say it's getting better, but at the same time it's getting closer and closer to C and I've always loved Perl. I've been coding primarily in Perl my entire career.

So some days. Uh, yeah, I, I make that joke, uh, but no, it's not going to happen.

Jonathan: I have, I have threatened to go on a tear and take out all of the Perl and all of the PHP and do everything in C and WebSockets and JavaScript. Uh, that is an equally monumental task.

Isaac: Yeah. Well, I mean, let's be real. There are lots of alternatives to Zoom that are in JavaScript.

Like, like Shinobi is all JavaScript, right? It's, it's doable. Um, I, my JavaScript. It comes from the 2000s, right? And I think it's fine for working in a browser. I don't want to use it anywhere else. And modern JavaScript just makes my, my heads explode, you know, the asynchronous stuff. And then it's just like, okay, I can deal with it, but I don't want to do it.

So. I'm very happy with my old school languages and stuff, and that's where I'm going to stay.

Jonathan: Yeah, yeah. Uh, what is the, what is the thing that has most surprised you about the way somebody has used ZoneMinder? What, what's the email that you've gotten where somebody said, hey, I took ZoneMinder and I did this with it, and, and you just couldn't believe it?

Isaac: Well, we've had some, some weird stuff, um, even, even recently, you know, we had a, someone who Because they, I think it's that Chinese camera problem, they want to use everything, uh, you know, homemade. They wanted to outfit their entire network with Raspberry Pis, with PoE hats, and, and, you know, distribute ZoneMinder across, like, 300 Raspberry Pis.

Like, a lot. Okay, that's a unique way to go. We're here to help you with that. That's cool. Um, but I know, you know, people monitor, uh, growing environments for fish or nature habitats or, you know, wonderful uses that are just really cool. I wish people would send me more interesting footage. Occasionally, someone does and, you know, maybe there's a bear in the yard or something.

I love that stuff. Um. You know, I've, I've used it to detect, um, mice. I think there's a mouse. Where is he coming from? I mean, you just put a camera on the floor and you've got the footage of it, you know? It's, it's, I did that. I

Aaron: thought I was having a raccoon get in my, well, I know there was a raccoon that was getting in my garage.

And I wanted to get him on camera. So I took one of my cameras, one of the, one of the wifi ones and put it out in the garage and, uh, yeah, never, never did get any good footage of him, but

Isaac: yeah. Actually, someone was, was posting, um, they had snakes in the wiring in their basement.

Jonathan: Oh, I remember that.

Isaac: Oh yeah.

I stared at that video for like minutes before I realized what I was seeing. Oh, wow.

Jonathan: Yikes. Oh, all right. So I've got to ask as we let you go, what is, uh, what's your favorite scripting language and text editor? I think we know the answer maybe to one of those.

Isaac: Yeah, I'm, I'm a Perl diehard and, uh, I'm a Vim user.

From way back. Um, I resisted the switch to colorized Vim, but for 24 years now, it's been colorized Vim. Yep.

Jonathan: Yep. Understood. All right. So if somebody wants to get questions answered or maybe even pay you money to help get their system set up, where's a good place to start to learn more about Zoneminder?

Isaac: Well, if you go to zoneminder. com, um, you know, all the information's there. There's a chat, one of those annoying chat bubbles will pop up. Um, I'm on the other end of it. Um, you can email me. Yeah, well, I mean, I might not be paying attention, but uh, you know, the response time might not be good. But I will.

And uh, otherwise email Isaac at zoneminer. com or really anything at zoneminer. com will get to me.

Jonathan: Alright. Awesome. Thank you, sir. So much for being here. And, uh, Aaron, what do you think? Have we, uh, have we convinced you to update to the bleeding edge version yet?

Aaron: Probably not. Like I said, I've got it, I've got it working, you know, I'll wait.

I'm happy to wait until things are tested unless there was like, you know, some sort of, you know, urgent security. Like we talked about, right. There are some things that will force me to, uh, to update, but I think, you know, I'm happy enough with the way that it's working. I look forward to using those features that you mentioned when I do upgrade.

Um, but for now, yeah, everything's working. It's, it's been working for a few years now, um, since I really. Really bit the bullet and installed zone minder and got everything working. So I'm totally happy with it. It's meeting our needs as a, as a family and as a, as a dad that wants to, you know, keep their, uh, keep a watch on what goes on around their house and stuff.

So. Yeah, it's working great. I'm quite happy with it.

Jonathan: I got, I got serious about having a zone minder installed when this house, we, we bought this house and gutted it and worked on it for actually for quite a while. And one Sunday we were off at church actually. And a couple of guys came and kicked in the back door of the house here and helped themselves to a very reasonably nice generator and so that just kind of walked off and thankfully that was the only thing that was taken because we had a neighbor that was Paying attention, but one of the next things I did it's like, all right It's time to start pulling cable and get internet there and we're getting cameras set up So I I wired everything about the house to be able to put cameras in now.

We're just Bristling security cameras pointing every direction.

Aaron: Yeah. Awesome. Well, that's too bad that that happened, but yeah, definitely something to keep in mind. I mean, I, I think that, you know, I try not to go. I know people that go a little overboard, right? Like they're thinking about it all the time and it's like, okay, that's my, might be a little too much, but yeah, like I said, I, I, I keep our, I know our stuff is recording and monitoring and doing motion detection.

And now I don't have to worry about it as much. So for me, it's a piece of mind. Yeah, now I know there's two timestamps and the one that comes from Zobeminder is the right one. So that's good.

Jonathan: There you go. Uh, fun times. All right. Is there anything that you want to plug, Aaron, before I let you go? Of

Aaron: course.

Of course. Yeah. Go to YouTube and look up RetroHackShack. If you like old computers, uh, vintage stuff and, and, uh, repair and, and history. I've got two channels now. So there's RetroHackShack, which is where most of the history and like, you know, Super interesting stuff ends up living. Well, I think it's going to be super interesting.

The last video didn't do too well for whatever reason. I thought, Oh, this'll take off like crazy. Right? I know it's like my worst video. Uh, so anyway, uh, but, but I try to put like more of the history bent kind of stuff on retro hack shack. And then if you go to retro hack shack. After hours, uh, you'll find my e waste Wednesday videos there where I go to e waste and I always find crazy stuff, uh, there and feature it on the channel.

And then also, you know, just random stuff, like more of the audio, if I do audio repair or. Just hey, here's this goofy thing I thought projects, you know things like that. I'll put those on that channel because there's a certain subset of my audience that's really interested in that and then there's a Rather large portion of my audience that isn't interested in that so I decided to carve that off Yeah onto its own channel and that's doing really well too.

So retro hack shack and retro hack shack after hours Go check it out.

Jonathan: Cool. All right. Well, thank you, sir, for being here. The things that I will mention is you can follow my work and occasionally Aaron's work. We cover his stuff from time to time over at Hackaday. That's hackaday. com. We've got the security column goes live there every Friday.

And of course, Hackaday is the sponsor, the home of Floss Weekly. Now we sure appreciate that. Um, Let's see, you can find more of my work on Twitter at JP underscore Bennett. I've got the YouTube channel. I've got to buy me a coffee, uh, all of those things and, uh, feel free to check them out. Uh, next week we are actually still looking for a guest for next Wednesday.

So if there is a, uh, if there is a project that you want to see either, uh, Let us know about it, or even better, get ahold of someone from the project and have them send us an email and that's floss at hackaday. com and that'll come straight to me and we will get them scheduled up. Uh, looking forward to whoever it is we get to talk to next week.

It'll be a lot of fun and we will see you then next time on Floss Weekly.

Episode 779 transcript

17 april 2024 | min

FLOSS-779

Jonathan: This is Floss Weekly, Episode 779, recorded April 17th. Errata Prevention Specialists.

Hey, this week Dan joins me and we talk with Andy Stewart, the creator of Andy's Ham Radio Linux. That is the Linux distro made by a ham radio enthusiast, for ham radio enthusiasts. We talk about all kinds of things, including why you might want to connect a computer to your ham radio, what you can do with it, and also what the current state is of the ham radio world, why the raspberry pi is so interesting, and my beef with the ham.

You don't want to miss it, so stay tuned. Hey, it is time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and I've got Dan, Dan the man, Method Dan. Welcome, sir.

Dan: Hey, good, good to be back, Jonathan. How are you?

Jonathan: I'm, I'm great. It is good to have you once again.

Always enjoy having having Dan in the co pilot seat is always fun. And we're talking, we're talking about something interesting. We're talking about Linux, which of course we are, we are both aficionados of, but we're also talking about ham radio and it's, it's Andy's, Andy's ham radio Linux distribution.

And It's going to be kind of an interesting mashup between the two of us because I'm a ham and I don't think, I don't think you are, are you Dan?

Dan: I'm afraid I'm not, no. I know the original plan was to have Doc here, who's a real, a bit of a ham expert, but unfortunately you're stuck with me today, so never mind, but we'll manage.

Yeah, so I'm here to learn, because I do know a little bit, I've done some research, I don't have a license, I've never passed any tests. Courses or anything, or any classes, unfortunately but I am keen to learn more about, I've been reading up about the distribution today and watching some videos and so on as well.

So hopefully fully researched.

Jonathan: Yeah, so we're, it's going to be interesting. I'm going to pretend to be the ham radio expert and you could be the Linux expert, which is sometimes sounds good, different from how things go, but that's all right. So let's go ahead and bring Andy on. Our guest today is Andy Stewart and he is the creator of a Linux distro.

Decided to roll his own. Hey Andy, welcome to the show.

Andy: Well, thanks very much. I'm glad that you invited me here and I'm looking forward to it. This will be a great fun.

Jonathan: Yeah, so there is, there is quite the overlap between Ham geeks and Linux geeks. In fact, we've got David in our, in our in our chat just says KN4AON representing.

So we've got quite a few hams that are into the show. That's great. So let's start there, maybe. Why, why is it that there is an overlap? What is the, what is the juice that is shared between Linux and HAM?

Andy: I'm not sure, really. What's, what it seems to be is, you know, people that participate in HAM radio have a lot of independence.

Same. Build a lot of their own projects. You know, they're talking on the air to each other without any other existing infrastructure. They help each other out. And the Linux community has a lot of those similarities. We all try to help each other out. We guide each other toward, you know, well, here's the documentation, here's some examples and so forth.

And, you know, we're, we're all in favor of you know, free software. And so there, there, there's a lot of synergy there, I think.

Jonathan: Yeah, it, it makes sense. So you've got a, you've got a Linux distro and I'm curious, like what What was the background that made you say, this set of problems that I have, it just, it just needs its own distro.

Andy: Well, it frustrated me that most of the software out there was for an operating system from Redmond and I, I don't use that operating system and I wanted to do things on Linux. So I started to look around it, Probably about the third year that I was in the ham radio hobby. And I saw that there was quite a bit of software that people had already created free software, GPL license, mostly.

And I said, well, let's, let's start running some of this stuff. And I did, and I built it onto my laptop. What was I using at the time? Was it Ubuntu or something earlier? I forget. But then I said, well, you know. I know how to compile these things and build them and install them and all that, but not everyone knows how to do that.

And really, do we want yet another computer to babysit or do we want a tool that assists our enjoyment of the ham radio hobby? And I'm aiming a little toward the latter. So I said, well, I wonder if I can package this up in a way that other people could use it. and benefit from it. And that's kind of how I started with Andy's ham radio.

Linux is taking all kinds of software that's out there already and putting it into one place and making it easy for people to access.

Jonathan: Yeah, it makes sense. So like, what are the things that we do with ham radio on a computer though? So, you know, we, we normally think of ham radio as being, you've got, you've got your hands set for For some of us, it's a Balfang handset, and you know, that's, that's become a bit of a dirty word in the ham radio community.

Although I think, I think people are getting over that these days. Yeah, they are. But, what what do you, what do you do, what's the point of connecting a computer up to, to the ham radio frequencies?

Andy: Well, there's quite a few things you can do. Some computer programs will directly control the radio.

So you might choose to click buttons in your GUI to change frequencies, to change from, you know, a Morse code mode to a sideband voice mode or something like that. Sure, you can touch the buttons on the radio too, but some folks will do this because the radio might be at a remote site. It might not be.

You know, in, in that person's room. Another application would be to log your contacts and there are many different programs for logging. So I'd say, you know, I, I talked to you know, KG five IAR at such and such date and time on this frequency. And I'd add it to my log. We might use that to we might upload that log to a server and say, Oh yeah, you guys made contact and check a box that says I, I got your state or something.

There are other applications that are on the web. So of course, bring up your web browser and you could go to a website where it will predict you know, what's the likelihood of me contacting somebody in South Africa on this date and time and on what frequency might I be able to do that. So that's a web based program.

Not strictly Linux, but you know, you'd use your computer and your browser to get there. And there's a there's also a whole host of what we call digital modes. So not all communications are voice and they don't have to be Morse code either. But if you think back to the days of the telephone modem where, you know, digital bits were converted to audio sounds and then transmitted over the telephone system, we can do things like that.

But Admittedly more sophisticated. And so digital stuff would come out of my radio. You'd receive it with your antenna and your software would decode it into either voice or possibly text that I typed on my laptop.

Jonathan: Yeah, some, some neat definitely some neat possibilities there. And so speaking specifically of, of your distro.

Is it, there's a lot of distros that are based on Ubuntu, and there's some that are based on Fedora, and there's some that are based on Gentoo and Slackware. Did you start with one of these as sort of the base?

Andy: I started with Ubuntu as the base because that's the one I was using at the time, and it's the one that I know the best.

So I started with that just a base install and then said, all right, let's install these programs one by one on top of that. And I created an add on to the existing menu system so people could get at these programs and things just kind of evolved from there.

Jonathan: And how many, how many of these programs are.

One of the, one of the things that humors me is when you have people that it's like, I'm going to roll my own distro and it, it just consists of installing packages that aren't installed, installed by default. Right. I kind of, I'm kind of getting the idea that you're doing a little bit more than that.

Andy: Well, so I started with Ubuntu because I don't, I've never used Gen 2, but I don't want to start from, you know, bare bones and build my, my way up. I want to stand on the shoulders of giants and build from there. So I picked Ubuntu, not for any religious reason, just because it's what I was using. And I installed a bunch of things from the Ubuntu repository.

Great. But there's probably, oh, 20 or 30 different programs that are not in the repository that have to be built from source code. And not everybody knows how to do that. And some of them are quite frankly, a pain in the neck to build from source code. So I said, well, rather than make people go through that pain, I'll do it for them.

And then I will package up the resulting image and send it out there for people to use. And right now that resulting image is around five gigabytes. It's getting kind of big. But that's, that's what I've been doing for how long have I been doing this now? 12, 13 years, I think. The first six or seven versions I created were not worthy of leaving my home.

Version eight was the first one that went and you can tell I'm an engineer, not a marketing guy because I number things, you know, incrementally. Yeah. That, that is awesome. It's, it's really interesting to see people making their own distributions and I completely understand why you would use like Ubuntu as a base.

Dan: Makes sense. One of the things I was interested in was some of the like Jonathan's already kind of touched on it there about the, why you would connect a computer to do your ham radio and so on. Now I would imagine that connecting the hardware, your radio hardware and stuff, you're going to need drivers.

You're going to need stuff like that. What's hardware support like on. On Linux for these kind of things. It's,

Andy: it's all built in. So modern radios simply need a USB cable from the computer to the radio, and you're done. Older radios have RS 2 32 interfaces and you might be able to get them to work with a u SB to RS 2 32 cable.

Mm-Hmm. I don't have any. older radios like that, but some people do and they have mixed success with getting them to work. But I have never had to install a driver or do anything like that to make anything work on Linux with ham radio. That's what we like to hear. It's all just there. Excellent. So you mentioned that the distro itself is now nearly five Gig the ISO image is nearly five gigabytes.

Dan: I was reading today that a lot of people who are into ham radio want something that will run on a relatively low powered machine. It's something that a lot of people are looking for. And I know that's partly the reason why you probably use Ubuntu because that will run on an older machine. Why do you think that is?

Do people want that? Well, what I, when I've talked to people many times, you know, they, they might not be familiar with Linux. They they're familiar with some other operating system and they get frustrated with it for one reason or another and want to try Linux for that reason. So they'll take a machine that's, you know, in the garage, maybe it's ready for electronic recycling or something.

Andy: And they'll, they'll dust it off and try out Linux. And sometimes they like it and they'll keep going with it. And this way I tell them you can have a laptop or a computer in your ham radio shack just for that purpose and you can keep your other computer for your checkbook or whatever you use it for and you can keep things separate that way.

And you know, a machine that's a few years old will certainly run Linux quite well. And I haven't found a ham radio program yet that doesn't run well on, you know, a four or five year old machine.

Dan: That makes sense. And you probably run it on a smaller form factor and all those sorts of things as well, because you could, you know where I'm going with this because I can see you.

Yeah,

Andy: I know where you're going. So in fact, this morning I just announced on SourceForge. So SourceForge is where Andy's Ham Radio Linux can be downloaded and there's a chat form there for help and all of that. Just this morning I announced the release of. Version 0. 1 alpha for the Raspberry Pi 5 of Andy's ham radio Linux.

And I got it down to 150 megabytes. And I took an entirely different approach to doing this. So rather than taking the Raspberry Pi OS image and installing stuff on it, and then trying to G zip that and release it, which was about six and a half gigabytes. I said, that's much, much too much. What I instructed people to do is get a clean copy of the Raspberry Pi OS.

5. 1 and 5. 2, I've tested put it on your SD card. If you've got about six and a half gigabytes free, you probably have enough space. And I wrote a bash script to do all of the installations of both the software on the repository and the source tarballs. So. I also supplied people with the source tarballs.

Now, the script could have just done wget, and maybe I'll do that in the future to make it even smaller to distribute. But right now, the source tarballs for about 20 or 30 applications are in there, so that way I know exactly what somebody's got. I know precisely how it works, because an upgrade to that might change a build procedure or something, and that would cause somebody some grief.

So I've constrained it a bit for the first release. I also have scripts in there that check to make sure you're on exactly a Raspberry Pi 5 and that you have enough memory and that you're running a 64 bit OS and it's 5. mumble. Most probably it will work on other flavors of Raspberry Pi, but since they're not tested, I didn't want to promise that yet.

So I'm hoping someone will comment two lines out of that script that make those checks and run it and tell me it works and then I'll, I'll relax the restriction.

Dan: Yeah, you, you want to over deliver, under, under promise over deliver or whatever you're saying. Yes. I

Andy: fully expect it should run on a Raspberry Pi 4, but I don't have one here to test it, so I can't promise that.

But if someone has that, or possibly even a Pi 3, let's try those. And if they work, then sure, I'm happy to release that restriction. Are you

Jonathan: accepting donations? If somebody wants to send you a Pi 4, 400, or a Pi 3, something like that?

Andy: I have a gentleman here who claims to have a Pi 4 that he's gonna loan to me, so I'd be happy to do that.

I have one friend who I think has a Pi 3 but that's a very generous offer. I appreciate that, thank you. And if I can't get those locally we'll talk again. Maybe somebody can send me one that I could borrow and do some testing. Donation's accepted. I just, I just don't want to get 10 of them on my front porch tomorrow.

Dan: I'm sure you find something to do with them.

Andy: I'm

Dan: sure I

Andy: would.

Dan: Yeah. So a lot of this stuff is compiled, isn't it? A lot of this, I mean, some of it will be in the, in the repositories and so on, but a lot of the specific ham radio software seems to be. Need to be compiled. So you're compiling it on the Pi, you're scripting that, is that what you were saying?

Yes,

Andy: it's all scripted and compiles on the Pi, and my Raspberry Pi has the SuperDuper fan on it, and it cranks full tilt with a load average of 5. 0. So if your cooling system isn't up to it, please be careful. Toward that end, in the script, I do a make j, and then the number of CPUs is a variable. Change it to something lower, and you won't beat in your machine quite as hard, and you'll probably be okay.

Okay.

Dan: Ah, cool. I was gonna that's why I was going to ask because i'm I've done a little bit of compiling stuff on the pie and to be fair. It was an older version of the pie but It can be it can be quite a process to try and compile stuff directly on the pie I found anyway Yeah, well,

Andy: the standard answer is it works for me, and when I released it today, now we'll find out if it works for anybody else.

I hope it does, and I don't see why it shouldn't. The toughest problem I have, actually, is I created a document called Getting Started, and I had this odd expectation that people would read it. And, and follow the directions. And sadly that doesn't always happen. And 99 percent of the time when somebody asks for tech support, it's because they didn't follow those directions.

So please take a look at that. I wouldn't, I wouldn't waste your time by asking you to read it if it wasn't important.

Dan: Yeah. And Have you considered hosting repository, you know, building the packages and then just like I've thought about it.

Andy: Like, you mean like creating a PPA or something like that.

Yeah, I've thought about it. I would have to look up how to do that. I've never done it before, but that could be a fun learning exercise.

Dan: Yeah, I was just thinking a PPA might be perfect for that kind of thing. That's a Personal Package Archive, I think it, I think. Yes, I think that's what that

Andy: stands for.

There was one out there, or maybe it's still there and not maintained. I believe it was called Ubuntu Ham or Ubuntu Hams, something like that. But I don't know if that one's maintained anymore. It was actively maintained for a while.

Dan: Yeah, that makes sense. So I noticed that all of the software on there is completely pre software.

There's nothing else you know, free and open source software.

Andy: Everything on there is free or open source or licensed compatibly. I, I do not want proprietary software on there and I don't think I have the legal right to distribute somebody else's proprietary software. I try like heck to stay away from proprietary software and just go straight, you know, free and open source.

Dan: Excellent. Well, I was going to ask if that was a philosophical thing as well as a practical thing, and it sounds like it definitely

Andy: is. It is for me, but I don't do anything to stop anybody from doing whatever they wish on their own computer. That's their choice. But I, I, I won't do it on mine if I can help it.

Dan: So, you've been going for almost, what was it, 12 years did you say now? I think it's, I think

Andy: 2011 was the first time I released a version. And so every four to six months, whenever I feel like it, and my workload allows it, I will update it and come up with another version. And I try to stick to Ubuntu long term support.

Versions, because I'm trying to get the latest ham radio software into people's hands. And you know, I'm still based on Ubuntu 22. 04, for example, and some people shudder in horror and say, Oh, that's two years old. I'm like, no guys, they keep putting updates out there for it. It's okay. And 20, 24. 04 should be coming out shortly, but who knows if it's quite solid enough.

I'll, I'll wait a couple of months before I update to that. That's what LTS means. Transcribed I was just going

Dan: to say, in Ubuntu LTS, long term support. So you've got five years anyway.

Andy: Right. And I'm, I'm opting for stability, number one, and number two, the latest ham radio programs that I can find to get out into people's hands.

So that, that's what I'm focused on. Now, this time around, of course, I took Ubuntu, installed things on it, zipped it up, made an ISO file out of it and distributed it, and it's pretty big. When I do the 24. 04 version, which won't be the next release, but the one after that, so probably six months from now, I think I'm going to shift to this scripted idea the way I've done it for the raspberry pi.

It may just be copy that script over and, and Tweak a couple of package names and go. It may not have to be that much different. And that could potentially save me an awful lot of time. I've, I've been into the, you know, new England, you know, handcrafted, you know, distro kind of thing. And it's, it's a lot of work and I tend to forget what I did to get each thing to work well now it's documented, it's in the script and anybody can do it.

Dan: Yeah, and is it just yourself that works on this, or do you take contributions from the community and do people patch things? It's

Andy: just me. I haven't received, I don't wish to get monetary donations. This is my contribution to the community, or one of them anyway. And you know, sometimes I get people with, you know, legitimate bug reports and such, and I'll fix them if I can.

And then I just put it out there. So. You know, that's how, you know, the ham radio and the Linux community share this idea of helping each other out and scratching each other's backs and so forth. And this is a way that I've, I've chosen to do that with, with the skills that I have.

Jonathan: How often do you have somebody come along and not only send you a bug report, but also send you the fix for whatever they found?

Andy: It's, it's uncommon. Most of the bug reports are please go reread this paragraph. And when you do it, you'll be okay. And then they are and it's occasionally somebody will say, well, this program has a bug and I will politely forward them to the the authors of said program. They can help them much better than I can.

And You know, usually it's not a unique bug. Either the authors know about it, or there's some work around they could implement. Yeah.

Jonathan: So it sounds like most of your users are ham radio enthusiasts as opposed to Linux enthusiasts.

Andy: Mostly, but they tend to be both also you know, occasionally I get somebody who's brand new to Linux that wants to try this out for whatever reason, but by and large they're people who have interests in, in both ham radio and in in Linux.

Jonathan: Yeah interesting. So I, I'm kind of, I'm kind of torn whether I want to do the pitch. But if you are, if you're doing, if you're doing cam radio on Raspberry Pis, then have you heard about Meshtastic?

Andy: Meshtastic. No, I have not heard of that one. Talk to me.

Jonathan: Okay. So full disclosure, this is a project that I'm involved in.

I do some of the code. In fact, running it on the Raspberry Pi has kind of been my baby for the past few months. Meshtastic takes LoRa radios. And so one of the other things I was going to ask you is whether you do anything with SPI. as opposed to USB. And SPI is one of the protocols that's built into the Raspberry Pi.

What MeshTastic does is it takes the lower radios, talks to them over SPI, and it builds a mesh. And so you can, you can go in and plug in your ham radio credentials. You know, this is my call sign and it changes the way the program works to, to, you know, to work with, with the losses that you have to do. Otherwise it does it encrypted, but the, the beauty is that, you know, one radio on the same channel, even if it doesn't know the password or whatever packet it sees, it pulls something out of the air.

It'll then retransmit it. And so you can build like a city wide mesh on less than a watt of a transmit power. And so there, there, there may be some, some juice here to find, to squeeze out the idea of making the mesh testic D binary work on Andy's ham radio. That that sounds like that could be fun.

Andy: That's interesting. I'm going to look that up. Cause I had not heard about that before.

Jonathan: Yeah, it's, it is reasonably new. We've, we've talked with the guys here on Floss Weekly a couple of times now. And after the first time we talked to them and I understood what they were doing, it's like, Oh, I've got to be part of this.

And so I've, I've been slinging code over there for quite a while now.

Andy: Nice. Yeah. Occasionally somebody will come along and tell me about a project like this, and they'll say, you know, can you include this in Andy Sam Radio Linux? And as long as it's free or open source software, and as long as it you know, looks like something that's reasonably useful to people, I'm happy to put it in there.

And so, but this is something I definitely want to look at. In fact, I did this recently for the M17 project. I don't know if you're familiar with this A relatively new project in ham radio, where people have taken a, a voice encoder decoder piece of software. Now that that's not easy to do, but there's a free software implementation out there that was invented by a ham in Australia.

And there are people that are taking this and doing digital voice. Over the air with high quality voice. And this is not an easy thing to do. And most of the algorithms are a proprietary and patented and, and, and secret and all of that sort of thing. This is wide open and it's called codec two. And it's been out there for about 10 years, but the M 17 project is taking it and doing all sorts of things with it.

On the two meter band, which is 144 megahertz ish. And they're, they're, Creating all sorts of different devices to allow people to use this. And it's, it's really cool. And if you search for M17, you'll have a hard time finding it because there's a military weapon of a similar name, but it is out there.

And if you, if you find it you might, you might find some interest there.

Jonathan: Yeah. That's, that's neat. Now, do you do much with say SDRs, the software defined radio dongles? And, and the, the background to this is somebody discovered a few years ago that TV tuners. So they were selling USB TV tuners and somebody discovered, wait a second, this, this is not locked to the TV bands.

This is not magic. This is just essentially it's a software defined radio plugged in over USB and the, the ham community in the Linux community kind of found it and went, Oh my goodness, this allows us to do so many things. Is, is there some of the SDR goodness in Andy's, Andy's ham radio linux

Andy: there is indeed i have not delved very far into that but i have one of those usb dongles that cost me a whopping 25 bucks i plugged it in and fired up a couple of pieces of software hooked my antenna up to it And the earlier devices wouldn't work below a hundred megahertz.

And so I built myself an upconverter to take ham radio frequencies around three megahertz and make them look like 103 megahertz. And then the dongle could, could decode them. And that was a fun and, and project to do. And so I've, I've, the more modern ones will go down much lower in frequency. And I have had some experience with that.

There is some software defined radio software available, and there's the GNU radio companion is also on there, which is not just software defined radio, but any signal processing application that you want to describe. GNU radio

Jonathan: is

Andy: sort of a Swiss

Jonathan: army chainsaw.

Andy: It is. It is. I, I built something with it once following a tutorial and got an FM radio as a result, which was kind of cool, but I, I haven't gone into it deeply enough to, to be able to invent things from scratch.

Jonathan: Yes. It's kind of like when you, it's kind of like when you buy the, my first radio kit, you know, they used to be Heath kits and then you could get them from Radio Shack and they had the little spring loaded. That's, that's how I feel going through one of those ham radio tutorials or excuse me, one of the GNU radio tutorials.

I know there's so much more to this. But following this tutorial, I'm just bending the spring back and putting the wire in. Alright, where does the next one go? And you put it there, and again, magic happens, and you get FM out of it.

Andy: Right, right. Now, it was kind of fun, but I figured just because I don't know how to use it, doesn't mean I shouldn't put it out there for other people who might be well versed in its use.

Jonathan: Yeah, absolutely. Alright so There is, there is something that I wanted to kind of pick your brain about while, while we had you here. And this is one of my sort of long standing beefs with HAM. And it's just because, you know, I come at this from a very Linux and even security centric background. And the HAM radio rules, there's some things that they don't allow you to do.

And, and one of those that really kind of ruffles my feathers the wrong way is you can't do encryption over HAM.

Andy: No, you're not supposed to do any sort of encryption over him. The idea is that anybody can listen to it you know, and, and know what that people are saying, and you just wouldn't say things that you, you don't want other people to hear.

Jonathan: And, and so the, well, the place that that really, that really, I think causes a problem is It can also prevent doing decent authentication. So, like, one of the things that always seemed interesting to me is if you had a remote device, let's say a repeater running on maybe a Raspberry Pi, something that would be really useful to be able to do is to use the SSH protocol to get in and make a change to the repeater.

But you can't, from what I understand, you cannot do SSH over over ham because again, it uses encryption.

Andy: That's also my understanding and I know people have talked about it and complained about it and what not, but yeah, my understanding is you're, you're not supposed to do that.

Jonathan: I, and I don't have a big enough voice to make a difference about this, but for the longest time I've wished that you know, the FCC and whoever else it is that governs the ham world would kind of go in and make an allowance for Encryption for the purpose of authentication.

And I think that, I think that would sort of revolutionize at least my corner of the ham world, it would sort of revolutionize it because it would allow suddenly more things that you can't do at the moment.

Andy: So along the lines of, of rules that some people might find silly is the limitation on certain frequencies of only up to 300 baud.

And that's based on, you know, things from the 1980s and say, well, guys, wait a minute. Why can't we have, if you want to put a restriction in there, why not have it be based on the bandwidth of the signal, not the data that it carries. And I, there may be something in the works now to, to modify that language because it's pretty dated.

But I mean, you know, some people would say Morse code is encryption because I don't understand it. Well, that's not really true. Somebody understands it and it's well documented just like all the digital modes that are out there. there. If you have the right software, you can decode it and see exactly what it is.

But you, you can't call it encryption just because you don't have the right software.

Jonathan: There's probably quite a few, because ham radio has been around for a long time, there's probably quite a few little weird wrinkles that maybe not everybody knows about. How quickly do things change? Is it a pretty slow process to get anything revised in the ham radio rules? It, it feels like to me, my own perspective the FCC seems to move at a glacial pace in, in my personal opinion.

Andy: It's, it's tough to get them to change and there's all sorts of there appears to be all sorts of bureaucracy and whatnot, you know, getting anything in government changed and that, that organization is no different. But you know, there are people who are working on it, there are people who make, you know, reasonable technical arguments for these things and, and little by little, you things do get changed, but there's also an awful lot of tradition in the hobby.

Things have just been done a particular way for a long time. And that's that's part of joining the club is is learning those protocols and traditions and kind of going with it.

Jonathan: Speaking of traditions, there are some other things that get done a lot in ham radio. And I'm thinking of like the civilian air patrol and then also and those guys are involved with this sometimes.

But things like disaster response. Just curious. Is there a tie in to to your work with either of those? Are there any, you know, cap specific applications that we can run on Linux? Or is there anything that's kind of geared towards disaster response?

Andy: Well, there, there is a branch of ham radio that refers to emergency management and in some parts of the country ham radio operators are well tied in with police fire emergency response and so forth and other parts of the country.

There's there's no such connection. And there is software that, that ham radio operators will often use in a so called, you know, emergency communications or, or MCOM setting. It tends to be based on that other operating system, it's called Winlink and it, it only, as far as I know. It only runs on a Windows system, which feels rather exclusionary to me, even if it were with a free or open source license, which I think it is not.

But there is a suite of free software that does a very similar task and it's called the narrow band emergency messaging system. And that. is a suite of software largely written by a gentleman down in, is he in Alabama or Georgia? I forget. It's a Dave Freese, W1HKJ. And it encompasses FL Digi, a couple of companion programs.

And an email tool, I believe it's Sylfeed. And those together allow ham radio operators to use the airwaves to send, you know, short text messages and, and, and other ham radio formatted messages to each other without any infrastructure in existence. I have a radio, you have a radio, we can talk, we can send things back and forth that way.

And in an MCOM situation, that's, that's important. Beyond that, I don't know, cause I haven't actually participated in such an exercise because they don't, they don't seem to be too, too big around here. But I have friends that have been doing that for quite some time and they, they use this this sort of software for that purpose.

Jonathan: Yeah, it's it's an interesting thing about ham is because it's for most people and most of the time, it's just a hobby. But there are, there are a few instances where like certain things fall into place and all of a sudden, you know, no, it's, it's a big deal and there are important things riding on it.

I, I had some friends that were in CAP, the Civilian Air Patrol, and one of the things that they would do occasionally is if there was like an airplane incident, I think. I think one time somebody was flying a small airplane and it went down and so their transponder started going, basically saying there's a problem, and the CAP guys got called and they went up and were part of the effort to, like, triangulate exactly where this plane had gone down.

And that was, that was like, that was official, but it was, it was ham radio. It was, it was kind of neat. Well, I participate in what we call fox hunting. Now, no, we're not shooting little animals. The fox is, is a transmitter, a very, very low power transmitter that somebody will hide typically in public conservation land.

Andy: And they'll send out an email and they'll say, Hey, Here's the frequency here's the piece of public conservation land where it is. It might be, you know, 50, a hundred acres of land go find it. And people have, you know, radio receivers and antennas that are made out of PVC pipe and properly cut lengths of of the, of the metal tape measure.

And they'll go out in the woods and they will track the thing down. And it's a great way to learn how to, well, not triangulate, cause you don't strictly have three, but to go out and track a signal find it. And now, of course, out in the woods, the signal might bounce off of trees or hills or other such things as, as radio waves are likely to do, and that adds to the challenge, but it's a great excuse to get out and go for a walk to

Jonathan: get some fresh air.

Absolutely. Okay, so you just said something really interesting. And this is, this is definitely more of a ham question than anything else. But public, public use lands, what, what is the Let's, let's just say that I wanted to put a little repeater, and, and so we're talking about something a little underpowered but ideally that's going to stay out for a while.

What is the, what is the rules and what are like the best practices about trying to do that on public use lands?

Andy: Well, when I put my Fox out there, I'll, I'll use that example. Cause it's one, I know the Fox identifies itself every 10 minutes, you know, when it's transmitting, I have it shut off at night.

And I always put a slash B after my call sign to indicate that, that typically means beacon or, you know, unoperated station, but, but I'm the one responsible for its transmissions. Now around here, our repeaters do slash R, you know, to let you know that it's a repeater. Now, as far as, you know, I put something out, it might stay there for 24 to 48 hours, and then I bring it in it's housed in a, one of those metal military.

Ammo cans that that used to have ammunition. It's, it's it's grown man's Tupperware really. And, and you can buy them dirt cheap at any ham flea market and they're durable as anything. And, and I've got a big sticker on it to let people know that there's nothing bad inside. Cause they see something like that.

Yeah. They see something like that and panic cause it's green. But. But it, it helps hide it in the, in the woods so people can find it. And you know, put a couple of those you know, gel cell batteries in there, let it run and bring it back, recharge it and deploy it the next weekend. And we've got a pretty good group of folks around here that like to go chase those in the various conservation areas.

But as far as something like what you were talking about, that, that feels a little bit more permanent? Or are you thinking like a temporary repeater for, you know, perhaps a week or so?

Jonathan: Well, I suppose either way, we would, we would love to be able to put something permanent up in the and, and where I'm at, it would actually be probably a, like a wildlife refuge.

Andy: Okay, so around here, the, the two meter repeaters that I'm familiar with have coordination of their frequency, so they don't all step on each other. And I don't know if, if you're talking two meters or some other frequency range, but if, if you're talking in the range of normal ham radio repeaters, you'd want to coordinate that frequency with others so that nobody steps on each other.

Sure. And some, some areas of high population density, there's an awful lot of competition for those frequencies.

Jonathan: Indeed. Well, that's something we find in the in the Laura bands as well, which is in, in the U S that's like the 915 megahertz band. Because well, so the, the reason that Laura is so popular is because so long as you're operating up to a watt it, it'll run on licensed.

And you can actually run whatever traffic over you, over it you want to, because that's, that's one of the other quirks about ham radio is you can't do commercial traffic, right? You cannot do business over ham radio. And the reason, the reason that is a thing is because if you could do business over ham radio, then some business would set up on it and sell it to everybody.

And suddenly the ham radio bands would just be entirely used for business instead of the experimentation.

Andy: There are frequencies for businesses already. So, okay, before cell phones were popular, you know, people might've had car radios and, and the pizza delivery guy might've communicated back to his employer that way.

So there's frequencies intended for that sort of use. Whereas ham radio is, is designed for, you know, people to, to talk to each other, either one on one or in a round table. But we, we don't we don't so called broadcast. We have, you know, AM, FM, you know, stations for, for one to many kind of, of transmissions.

We don't do that. It's, it's like I say, one, one and one or a round table.

Jonathan: Yeah. And, and the, the, the pizza guy example that would essentially be what we consider citizens band, right? CD.

Andy: That would be a way to do it. Sure. And then, yeah, so for example, sorry, for example, I can't I can't push any products of my employer, for example, on the air, because that, that would be considered business, but Andy's ham radio Linux is okay because I don't make a dime off of it and I don't wish to.

Jonathan: Yeah. Yeah. Oh, I was going to ask something and it's just gone. Dan, do you want to take it for a minute while I try to remember what I was going to go with that?

Dan: Yeah, no problem. Actually, Andy, you guessed one of my questions. I was going to ask you what fox hunting was in the ham. Yeah, it was when it came up on one of your slides.

I watched one of your talks earlier and it said Fox hunting was a big thing. And I was like, Whoa, that's interesting. Yeah. So I wanted to ask a bit about what it's like to run a Linux distribution. I mean, is it a big challenge to keep up with? Because one of the things that, you I was really impressed with when I had a look at your distro is the documentation, which is a lot of work.

I mean, keeping on top of all of that, but it's the actual compiling the documentation from the individual different little apps and, and, and things that you've done into, because what people may not know if they haven't tried Andy's your distribution yet, is you've got your own little interface there that you've added, where you've got the documentation and all that sort of stuff in there.

It's not a big challenge to keep on top of that as the it's, it's some work

Andy: you know, every time I update the versions of software, I go through and check all of those menu options every single one of them. There's, I don't know, 40, 50 of them to make sure that at least the program comes up. If the program dies after that, that's not likely.

An issue that I can help solve that might be something for those developers, but I wanted to put some of the documentation in there to, to give people a starting point and especially for the few programs that are command line driven, how would folks even know they were there? If I didn't expose them through the menu, so the menu, you know, brings up a terminal and then runs that program and also has the documentation typically it's pulling up a man page, which, you know, most of us Linux folks would know how to do that.

But somebody knew wouldn't necessarily have stumbled upon that. Now they don't have to. They can just go through the menu and, you know, don't find it have it, have it done for them. So I tried to help that way and I try not to give, you know, read the friendly manual type of answers to folks because that's, that's not really entirely helpful.

Which I say, okay, in the manual on this page and this section describes the solution to your problem. But I don't find. That there's an awful lot of tech support requests of me which is helpful because being just one person, if there were lots of them, that would be hard to manage. And it might suggest to me that I put out something of a quality of the level lower than what I would like to do.

Dan: Yeah, that makes sense. I mean, one of the things that I was going to ask you is how, if say I wanted to start a Linux distribution, is there something that you've learned in these years of running Linux? running your distro that you would, is there a nugget of wisdom you can give me that maybe I need to know?

Andy: Yeah, people don't always read docs. And so, yeah try to create ways to, to debug it when someone does have a problem. So for example, the script that I wrote, that does all the installations for the Raspberry Pi, every single function says, yes, I got here. Yes, I left. So in the blather of messages in between, at least I'll have a clue.

Okay. What was it trying to install? And that will give me some, some ideas as to where to go. Look is it something I can fix or not? All of the, the fine details of, of those menu files and making sure the menus come up. That's just more. tdm than it is difficult. But in each of those menu files, it says, list this program under, you know, ham radio under Andy's ham radio Linux under, you know, maybe two or three other things.

So some of those programs will show up in three or four different menus in a way that I don't necessarily think is the right way to organize it. But I don't want to get in there and start Rearranging everything that Ubuntu did. Cause as soon as you do an upgrade, you'll, you'll lose it. So I created a separate menu that says Andy Sam radio Linux, and then a sub sub menus of that are all the programs that that I put in organized in a way that I think is reasonable, but of course people can go in there and change that as they wish,

Dan: but at least it's

Andy: a starting point.

Dan: Yeah, and and of course you get to be the benevolent dictator there as well So you can decide what you know how you think they should be arranged and then people can customize it. That's the beauty of linux

Andy: Yeah, I, I don't do anything to stop anybody from doing whatever they want on their own computer.

I, people often ask me, why do I use Linux? Why do I do this, that, the other thing? And I, I used to be a little more heavy handed with that and say, well, you should do this because. And, and that doesn't work. People, people don't want to be preached to. I'll say, well, here's what I do and here's why I choose to do it.

And if you choose to go that route, let me know. I'd be happy to help you if you'd like. And that seems to be a much more positive thing. People don't feel threatened by that. They feel, Oh, here, here's a resource that I could use if I want to go this way.

Dan: Yeah, that makes sense. I was looking at some of the other Things that you're working on and some of the other things that you have worked on I should say and I looked at XLOG and you've become the maintainer of XLOG.

So it's an obvious question. Yeah. So what how did that come about?

Andy: Well many years ago. I forget how many now ten or more the author of XLOG put out an email that said effectively it said, I, I don't have time or, or, or perhaps desire to maintain this anymore. I don't want the bits to rot. If somebody out there has the requisite skillset to take this over send me an email, describe, you know, why you want to do this, what your credentials are, and I'll consider among the incoming emails and decide who the next maintainer will be.

So I said, well, what the heck? I use the program and. And you know, I know how to write code, so I could dive in and probably figure it out. And so I sent the gentleman an email and didn't hear anything for about a month and figured, well, maybe he's found somebody else to do it. Well, a month or so later, he sends me an email and says, Andy, you have the right qualifications and no one else sent me email.

So I wasn't quite sure how I felt about that, but he transferred, you know, ownership or maintenance rather to me. And mostly all I've had to do is update that program. To match the the ADIF specification, which is a, a file format by which we describe our contacts so that we can upload them to various servers and have the, the data properly interpreted.

That spec undergoes a change from time to time. And I just want to make sure that X log outputs a file with the with the correct format.

Jonathan: So we, we did actually have a question come in from the live audience. This is David Ruggles, one of, a friend of the show and also one of our, one of our fellow hams.

Wants to know, are you familiar, when it comes to fox hunting, are you familiar with anyone using drones for doing fox hunting or triangulation? And, he goes on to say obviously the default would be multiple points on the ground, but we are, we actually exist in 3D space. And, drones could be used to make a point vertically above whatever you're looking for.

Is someone out there doing that?

Andy: That's interesting. I had not heard of that before. We're not doing it here, but I, I took notes while you were saying that, and I want to look that up, because that sounds really quite interesting. We have a couple of people in the Ham Radio Club that are drone enthusiasts, and maybe this is a way to merge their, their drone interest with their Ham Radio interests.

Jonathan: One of the, one of the real fun things that people do with Meshtastic, which again is in kind of the same, the same space, is to take one of the smaller Meshtastic nodes and duct tape it onto a drone and send it up. And then you essentially have a repeater 500 feet in the air or whatever. And that's, that's fun.

It's, it's not going to be a permanent install up there, but for a little while. Nope. All right. So I saw in your bio that by day you are a digital logic verification engineer and I'm curious, does that background come into play at times with your ham radio work?

Andy: Once in a while it does. So my, my college degree says electrical engineering, but when I got my college degree, they didn't have a degree called computer engineering.

Today they do. And that's, that's what my, my focus really is. And so digital logic, of course, is the circuitry by which we, we build computer circuits and microprocessors and all of that. So. When I started my career, I was one of the guys that would design those circuits, and we actually drew them on paper and converted them to computer formats and ran simulations and, and got databases for etch boards and so forth.

And nowadays, everything that we do is all code that is. Is, is processed and, and, you know, various layers of software run and convert it to transistors and we we send it in and get fabricated. But what I do by day now is I, I run a computer simulations of very large digital logic circuits, and my job is to attempt to break them.

So, for my non computer friends, I say I get paid to break things before the customer ever sees it. In fact, before we ever fabricate it. It's a computerized representation of a big circuit. And I, you know, between the specifications and the circuit, I run my test and if something fails, we figure out what's wrong, fix it, and then and then move on to the next issue.

And so that's what I mean when I say I live in simulated reality. I'm, I'm playing with simulators all day. And I can change one line of code and change my reality.

Jonathan: You are an errata prevention specialist.

Andy: I, I try, you know, we want to put the best quality product out that, that we can. And ideally the first rev of whatever chip I'm working on would, would pass muster and be able to be shipped to customers.

Cause it's doing additional revs gets very expensive, very quickly. And worse, you might miss a marketing window.

Jonathan: Yeah. Or you ship it and don't realize there's a problem until somebody discovers it in the wild and then you have a security issue.

Andy: Right. So, so some of that knowledge helped me when I was hacking around with a a radio that was invented in India called micro bid X and it's the, the micro symbol and then B I T X a ham radio operator from India has been coming up with these kit based radios for quite some time aimed at the ham radio market.

In India, so his part selections and things would be parts that are obtainium in India and of relatively low cost for that population, but they've been hugely popular and they came here and there's an Arduino inside there and he released his software as free software. So I built that kit and you know, did some Arduino programming.

For probably six or eight months, I took his software and just totally refactored it and practice my, you know, C plus plus skills a little bit. And I put a, a voice synthesis chip inside there intending it to be used by visually impaired hams. And I got something that worked, but as a prototype, it was prohibitively expensive to build.

And then the voice synthesis chip which is the emic two went to EOL. So can't build that one anymore, but it was fun for a while. And I had gotten some interest from the Blind Ham Radio community because it might have been able to be cost effective for them. But unfortunately I wasn't able to take it any further, but the digital skills and the programming certainly helped me while I was hacking away on that project.

Jonathan: Yeah, interesting. So one other, one other question that I wanted to get in before I hand it back to Dan to ask about your future plans. We briefly mentioned the Belfang, and I happen to know that if you get the right cable to connect to them, there is actually a programmer. Where you can go in and set up your, your pre programmed frequencies and you can also, you can fiddle with it in some ways that you can't just by pushing the buttons.

And so is, is that sort of the, the thing, one of the things that ships with your Linux distro is these various programmers and specifically the one for the Baofeng?

Andy: The programmer that I'm aware of that talks to lots of handheld radios is called Chirp.

Jonathan: Yes, that's the one. I couldn't remember the name of it.

Andy: It, it is on there and lots of people use it and they really like it. Of course, you'd have to look on their website to see specifically which, which models are supported, but I've used it for my 10 and 15 year old handhelds and it works quite well.

Jonathan: Yeah. Chirp is kind of a gateway drug in a way to playing with your ham radio equipment connected to your, in this case, Linux computer, because it's one of the simple things to do.

But I feel for a lot of people, maybe they, they make that connection and go, Ooh, this is cool. I wonder what else I could do.

Andy: No, once, once I figured out how the program worked, it was pretty straightforward. Put all your data there once and loaded into the radio and go. And if something happens and the radio is damaged or forgets or whatever, you've, you've got it on your computer as a backup.

Dan: Yeah. Awesome. So, Andy, you talked a little bit there about about Arduino and I was listening to some of the stuff that you talked about today where you were talking about hardware hacking on the talk that I saw and building. Like kits almost for people to use to get into ham radio. Is that something that you do a lot?

Or is that something that is more of a side thing? So

Andy: when I say I build the kit, I purchase it from whomever provides it. I solder it together. I do all of that load the program if that's appropriate and then go off and use it. So one. Kit that's out there that seems to be pretty popular is called Morse Arino and its software is also freely available.

And it's for people who want to practice and do things with Morse code. So with that, I created on my Linux computer, I created some text files that I loaded into that for Morse code practice.

Dan: That's awesome. Because I was thinking about appliances, maybe, you know, with obviously you're now that you've You've got the alpha out for the Raspberry Pi, that will probably, obviously that will develop.

There might be a market for putting together The boards, you know the pies with the Into little appliances almost that people could use I suppose that i've just reinvented the radio there. No, that's a bit Now that I think about it You could put a screen on it so people could read what the frequency no,

Andy: sure Yeah, I don't I don't see myself starting a business in a garage or anything like that.

I'm going to stick with the things that I know how to do. And that's, you know, tickle the keyboard and, and hopefully make good things come out of it. But for example, people have said to me, well, how come Andy's ham radio Linux. Only works on Ubuntu. I have Linux Mint and I like that. Why can't I put your stuff here?

Mm-Hmm. . And the answer was, I'm, I'm sorry. I built it on Ubuntu and, and that's what I distribute. And I, I didn't mean that badly. I, but, so now that I think about this script that I wrote to install it on a raspberry pie, if I port that script over to, you know, you know, regular, you know, laptops and things, why couldn't somebody take that and run it on their favorite distribution?

Doesn't have to be a BU two. And. Would it work? Well, if you know, if it's based on on apt, then if the package names are the same, they could do that. If it's an RPM based distribution maybe the script could say if red hat do RPM, if Ubuntu do APT or whatever, I mean, some modification of the script could be done to support other.

flavors of Linux. And so I see some potential there for this to grow into a wider audience. The other thing that could happen is I believe there are one or two other ham radio Linux folks out there that have created things. And Maybe somebody takes this and make something out of it. That's much bigger and better than I ever imagined.

And I say, go for it. Yeah, I've, I've done the best that I know how to do, but maybe somebody knows how to do better and that would only benefit the community. So why not?

Dan: Yeah. Ironically, Linux Mint is itself built on Ubuntu, so they've done the same thing.

Andy: But that was one example that I often hear, but there's no, I have no philosophical reason for not running this on any flavor of Linux that's out there.

Just, you know, I didn't have the technology to do it, or the time to, you know, configure, you know, different flavors. But this this script that I will soon well soon, three to six months, modify for, you know, everybody else's laptops and other non Raspberry Pi hardware that has some potential to, to work on things other than Ubuntu.

Dan: So you've kind of answered it there, but I was going to say, what, what's the future, you know, if you've got a future, not what's the future, that's a bit generic, but you know, what's, what's the future roadmap? What's the, have you got plans in the pipeline? Obviously, the Raspberry Pi thing is going to be exciting.

Andy: don't have any plans yet. I want to see how this alpha release goes. I want to see how well the community accepts it. The community has been wildly accepting over the last dozen years of Andy's Ham Radio Linux. In fact, I was quite pleasantly surprised about two years ago when SourceForge sent me an email with a little PDF plaque and said, congratulations on your 100, 000th download.

And I was just blown away. Now there's probably five people out there with bad internet connections that are persistent. But to be that as it may, I want to thank the community for for their interest in this. You know, there are many, many different versions that I've released over the years and.

Some people have probably downloaded all of them. But to me, that's, that's a, that's an incredible response on the part of the community. And I'm very thankful for that.

Dan: Yeah, so you actually get a plaque? Did you get an actual physical plaque? No, it

Andy: was, it was a little, it looked like a plaque, but it was a PDF file.

But it was, it was nice of SourceForge to send that and I, I really had no idea that, you know, over 10 years there had been that many downloads. That's not the main motivation for doing it, that it's popular. The main motivation for me is I want to configure my laptop that way and I want to share it with folks.

And if, if 12 people were happy, I'd be happy. Well, it seems like maybe there's a lot more than that. And so that makes me even happier.

Dan: Yeah, that's awesome. I mean, I was going to ask how many, not exactly how many users you've got, but did you now have a rough idea? And obviously you've got, you know, a hundred thousand people downloading it over that time.

So somebody out there. Somebody out there is getting it. It must be really rewarding to know that what you're working on is being appreciated.

Andy: Yes. I really do like that aspect of it. And SourceForge has a section of their, their webpage that looks at the IP addresses and tries to figure out the countries of origin.

And there are people, if that information is correct, there are people from all over the world downloading this, you know, mostly USA and Europe, but many other countries as well. And Ubuntu of course supports many different languages. So that makes perfect sense.

Dan: So Jonathan's just asking me if I've got any more questions.

And to be honest, I'm starting to run out of questions other than I was going to ask other than I was, I was going to ask where I should start if I wanted to get into ham radio. But I think is, that might be too big an answer to give right now,

Andy: that that is too big of an answer. Yeah. Mm-Hmm. . Mm-Hmm. . There's a couple of services online that you can buy that will help you study for your ham radio exam.

Good. And I, I used those in the past and they were much more helpful than just reading a, a book that can be rather dry for some people. Mm-Hmm. . But the software that I know about online that runs through a web browser would. Ask questions, watch you answer. It had some AI built in to help you with weak areas and, and not worry so much about areas where, where you've got it.

And that's one of many possible ways to, to go about getting your license and then join a ham radio club.

Jonathan: And then run

Andy: ham radio Linux.

Jonathan: I was going to say, look for a local ham radio club. Absolutely. The ham radio equivalent of a Linux user group. And those guys are usually very welcoming to new people because they know that the new people is the lifeblood of their hobby.

Andy: Absolutely. And I've founded and run two Linux users groups in the last 20 years or so. One of them is still going quite strongly. It's been around since 1997. And the other group, unfortunately, folded when COVID started. But But again, this is joining those, those clubs and learning from people both Linux and ham radio is a great way to get going.

And I learned so much from those people in, in both the Linux and ham radio communities. And this is a way that I can give back and I like doing it.

Jonathan: Yep. So I've got a couple of questions that I like to ask folks when we, when we have them on. And one of them is. What is the, what is the most unusual or most surprising thing that someone has done with your, with your software?

So in this case, the distro, what, what has someone done with it that is, that has surprised you the most?

Andy: In terms of novel use, do you mean, or just. Something that I was perplexed why a user would even do that

Jonathan: either either way is acceptable.

Andy: Okay, so one time a gentleman sent me email Claiming that I somehow ruined his entire computer because he tried to install this and botched it quite quite a bit and His, the tone of his emails was rather hostile and, and this, that, the other thing.

And despite that, I did my best to help him bring his computer back into shape. And I think this gentleman was one of those people that should have just put the computer back in the box and mailed it to the, the manufacturer, but I did the best I could to help him with that. And, and sadly didn't even get a thank you, but I felt like I did the right thing, even though I have no idea how he got in that state.

Yeah, that was. By far the most ridiculous thing that I've heard. Other things are, are quite valid. You know, there's nothing that's perfect about any operating system and people will trip on things and ask. And I think that's valid. If, if I know how to help them, I will. Or if I don't know the answer, I will try to point them to a place where they can get the answer.

Jonathan: Yeah, that's fair. Is there anything that you wanted to talk about today that we neglected to ask about?

Andy: No, I think we covered an awful lot of ground here. I hope the folks listening enjoyed what we've done today. And if anybody would like to get ahold of me you can send me email. My call sign is a K B one O I Q or phonetically it's kilo Bravo one.

OscarIndiaQuebec at ARRL. net and you know, or you can find me on SourceForge and send email that way. But any questions anything like that, I'm happy to help people.

Jonathan: So you brought it up and I've been thinking this whole time, you're on SourceForge still. Any plans to move to GitHub or GitLab or any of the other solutions out there?

Andy: I have not thought about that yet. SourceForge is working for me, but I know people have reservations about the use of SourceForge. I don't see a reason to move yet. I mean, technologically it's doing what I need it to do.

Jonathan: There were, this is totally an aside, but there were a couple of years there where SourceForge was owned by someone else that was making some rather questionable decisions, but I think it's now, I think it's now actually owned by the same people that own Slashdot, and so seems to be moving in a healthy direction once again.

So maybe it's the place to be.

Andy: Yeah, I've, I've heard similar things. I didn't really track them down. I had heard allegations about things being inserted in people's software before it was distributed. I, I don't know if those allegations are true, but I tell people, check the MD5SUM. of the thing that I'm posting, because when I post it, I check it and make sure it's been not modified.

And if they download it and that's good, they should be okay.

Jonathan: Yeah, makes sense. All right, so last two questions I have for you, and then we'll let you go. So what is your favorite text editor and scripting language?

Andy: My favorite text editor is emacs, because I do a lot of programming. And I have bounced off the learning curve.

So half the time I don't know how I'm doing things. It just flows out my fingers and scripting languages. I've I've used so many different ones. It's it's hard to say. I, I briefly touched Python during my last project at work and knew just enough about it to get it going. Perl looks like modem line noise, but it can often make, you know, wonderful things happen.

Most of my programming lately has been in, in a C or or, you know, Arduino's flavor of C slash C plus plus. That's mostly what I'm doing. I, but then again, this the install script that I did is a bash script.

Jonathan: I guess

Andy: I don't, I guess I don't have a favorite. I picked the one that kind of suits me for the day.

Jonathan: Yep. I get that. I get that. I think, I think that's probably the answer I would give too. All right. Andy, thank you so much for being here. It was wonderful getting to chat with you for about an hour and sure. Appreciate your time and really enjoyed getting to meet you.

Andy: Jonathan. Thanks. This was fantastic.

I enjoyed it and I hope the the listenership did as well.

Jonathan: Yeah, I think so. All right, Dan, what do you think? Have we convinced you to go start working on your ham license?

Dan: to be a ham. Yeah, it's, well, it's something I've considered in the past, but I, I think it feels like it's one of those things that I've always told myself I would do like learning Spanish.

I've always told myself I'm going to learn Spanish one day, but so far it hasn't quite happened, but maybe I'll get there. The other thing is all this talk of ham with the time difference here. It's like, it's. dinner time here and you keep saying ham and i'm like oh ham in a homo simpson style way i'm kind of thinking ham which is good but it got worse before when you said you had a beef with ham and i was like oh beef with ham that's quite anyway we'll see The beef with ham.

Yeah, great stuff. Great to talk to Andy. And it's great to to hear somebody so enthusiastic about the project that he's working on. And, and it looks really great. I mean, I wasn't just blowing smoke there when I said I did look earlier on and short of actually firing up the live dish, you know, the live install thing.

I did go through all the, a lot of the stuff that he's done there and it looks great. And I hope it continues for a long time. So I will say it is refreshing to see someone that has been working on an open source project for over a decade that is still enjoying it and still enthusiastic about it.

Jonathan: True. We've had some distros or some maintainers on various things that it's like, yeah, I'm still working on it. It's like, oh man. So it was nice. It was nice to see that Andy was still in the saddle and excited about it. Definitely. All right. So. Let's see Dan, do you have anything that you want to plug before we wrap?

Dan: Yeah, very quickly. So every year I mention it, but Liverpool Makefest is coming up in the UK. It's the largest celebration of technology and arts and crafts and making of things in the UK. We, we usually get about four or 5, 000 people come through during the day. So it's quite big, not all at once because it's, it's in a library and they tend to kind of funnel through.

But if you want to find out more, if you've got a project and you think you're in the, you can get here and you'd like to exhibit your project, we're currently looking for, we're currently accepting applications. So if you just search Liverpool make fast you will find it there and go and have a look at that.

Jonathan: Very good. All right next week our show is going to be live on Tuesday instead of Wednesday. I've got some traveling I've got to do on Wednesday, so we will be back on Tuesday, I believe. That's the 23rd, if my math there is working. The only thing that I have that I want to plug, of course, is you can follow my work on Hackaday.

com, which Hackaday is now the home of Floss Weekly. I'm sure everyone knows that, but we do sure appreciate them doing that for us. And then also, there is still the Untitled Linux Show back over on Twit, and make sure to follow that. We have a lot of fun there. I think that is it. We will see you next week on Floss Weekly.

Episode 778 transcript

10 april 2024 | min

FLOSS-778

Jonathan: This is Floss Weekly, episode 778, recorded Wednesday, April 10th. Octoprint. People are amazing at breaking stuff.

Hey, this week Katherine Druckmann joins me and we talk with Gina Hoiska all about Octoprint. That's the open source software that you can put on a Raspberry Pi, or any computer really, to babysit a 3D printer, except it's got more tricks up its sleeve than that. It will slice, it will control, all kinds of stuff.

You don't want to miss it. So stay tuned.

Hey, everybody. It is time for Floss Weekly. That's the show about free, libre, and open source software. And sometimes we throw some hardware in. And today is going to be, I think, the intersection between software and hardware. And maybe talking about one of those magical things that takes software and turns it into hardware.

But before we get to that, let's bring Katherine on. Hey, Katherine, welcome back. Hey. As one of our favorite co hosts, it's good to have

Katherine: you. I'm glad to be back.

Jonathan: Yeah. It seems, you know, I've, I've pulled in, it has been, I've pulled in some other co hosts when people haven't been available. And so it seems like the rotation now is even longer.

But It's nice to have you back. Thanks. We're talking, we're talking 3D printing, and we were talking a little bit before the show, and I asked you, are you a 3D printing enthusiast? And you essentially said what I would say about some of my musical instruments. It's like, do you play the guitar? Well, I, I have a guitar.

Katherine: Yes, I aspire to be a 3D printing enthusiast, but I'm not yet.

Jonathan: Yeah. So you, you were saying you, you made a few prints, but you ran into like technical issues with the printer where it was slow and it was fiddly to work with. And I think everyone that spent much time doing 3d printing feels that deeply in their soul.

Katherine: It's a budget question at some point. It's less and less now, but several years ago when I, when I wanted to really get into it, it was very much a budget question. And the ones that were in my price range were, a bit painful.

Jonathan: I understand. I've got a I think technically mine is a mono select or mono price maker select three or something, which is a clone of a clone of a good 3d printer.

Okay. That's funny. Someone, someone described it as being a, it's an my first 3d printer. It's like, yeah, that's, that's what it is.

Katherine: Yeah. That's what mine was too. And it is now on a shelf because it's, it's. Yeah, someday I'll get another one.

Jonathan: There is this, there is this fun thing that happens though, where it's like, you can 3D print parts to put on your 3D printer to make it better.

And I did a little bit of that. That sort of got me going into the, the, the, it got that first, it was, it was that first serotonin cookie. I liked that term by the way, I

Katherine: liked that too.

Jonathan: I got that from, I got that from Eben Upton and we interviewed him, but that first serotonin cookie from 3d printing was, Making a part to make the 3D printer better to make better parts.

And it gets you on that same, that same ride that you get with software. When you're making your software tools better to be able to make your software better. So our guest today is all about that. All about, well in this case, software to make 3D printing better. better. Let's go ahead and bring her on.

It is Gina. Oh, I asked how to pronounce this, and I It's Hoyska. I got it. Hoyska. Thank you. Perfect. You're welcome. You're welcome, Gina. It is wonderful to have you here.

Katherine: I'm also an aspiring German speaker, sorry.

Gina: You pretty much nailed the name already. Like, that's half the, half the, half the job already done.

Awesome.

Jonathan: Yeah, it's

Gina: great to be on here.

Jonathan: Yeah, what, for our listeners, we know, but for our listeners, what do you bring, what has been your, your stellar and it is, it is stellar and it is shining contribution to 3D printing?

Gina: Yeah, I built this funny little tool called OctoPrint which is basically, yeah, you could call it a baby monitor for your 3D printer.

And yeah, I, I, I created that basically just to scratch my own itch back around 12 years ago when I got my first printer. Back then things were even worse than they are now with regards to reliability. And so I really did want to keep an eye on it doing its thing, but I did not necessarily want to have to do that in person.

And so I looked for ways to solve that so that I could just put it into my spare bathroom and be able to watch it from afar. And ideally do this with something that I could just throw A nifty little single board computer that had come out the very same year, the Raspberry Pi, back then in the first version, which still had no Wi Fi and was only single core and really, really slow.

So I had to make sure that stuff still worked on there. And I figured, well, someone will already probably have written something like this. And some solutions existed that allowed to, to, to basically throw stuff at your printer and hope it sticks, so to speak, but there was nothing that also had this back channel that I wanted in order to be able to watch things, to make sure everything is okay, to get access to the temperature data from the 3D printer, to see progress updates, and ideally also hook up a webcam to everything to be able to really watch it from afar.

Okay. And so over the course of my Christmas break that year, I sat down and wrote what is now known as Octoprint. Obviously only a very basic version of that at first. It did not have a plug in system as it has now. It only worked with my printer and a very limited set of other printers, but it worked and it did what I needed it to do.

And I threw it online and suddenly everyone said, Oh, wow I've been looking for something like this. And I started running and feature requests and then it took over my life and the rest is history.

Jonathan: It's a very, Torvalds sort of story. Actually, this is a little thing I did for fun. It'll probably only ever work on my machine.

And suddenly you throw it out there and everybody wants to use it and add to it.

Gina: Yeah, it definitely was not intentional, what happened there, but It's fun. But it's a good story. It is.

Jonathan: It's a great story.

Gina: Definitely. Like, even when people don't understand what open source software is, or what 3D printers are, I always have something to tell at the party, because then people go, what, you did what?

And then what? And you still do that? And you give this away for free? And people still give you money for that regardless? What? Eh?

Jonathan: I, I am very curious about that part of it. I think we're going to cover some basics first, but I'm extremely curious to hear the story of how you, you. Make money with this open source thing So let's but before we go there.

Let's talk about what all you can do with OctoPrint and then there's also OctoPi So I guess first off. What is OctoPi? Is that officially part of the project?

Gina: Officially that is still a third party project that was done by Guy Schaeffer from from I forgot the exact Town he lives in, but he's from Israel and we've been in contact since 2013, I think, when he came up with the first version of that and it basically is a box standard Raspberry Pi image that has been extended to include OctoPrint, to include a webcam server, to include some other scripting around all of that so that you can just Yeah, throw it into the Raspberry Pi configure, flash it to your, to a, to a disk and to a disk, to an SD card and put it into a Pi and it will boot up immediately, connect to your Wi Fi and then you can just use OctoPrint from the next browser.

Completely headless. You don't need a keyboard. You don't need a mouse. You don't need a monitor or anything like that. All it wants is. Network and a printer ideally because without a printer things will be a bit boring. I mean technically there is a virtual printer on board that I use for development But you will not have a lot of fun with that if you're not not a developer, I think And so a lot of people confuse OctoPi and OctoPrint.

OctoPrint is the thing that you actually interact with in your browser. That is the thing that actually talks to your printer. OctoPi is the image for the Raspberry Pi that ships with this and a bunch of other stuff so that you can get up and running really, really fast.

Jonathan: So OctoPrint is not necessarily limited to a Raspberry Pi, is it?

You could, you could throw it on anything that, I, I'm assuming it's probably Linux only, but you could throw it on edition. No, no.

Gina: You, it even works on Windows. It works on Mac, it works on yeah. It's as long as you can run a regular Python c Python, probably, I'm not, not entirely sure about pii, but as long as you can run C Python on something, be it a Linux machine, a Windows machine, anything like that.

It will work. You can use an old laptop. You can run it on the Steam Deck. You can run it on your NAS. You can run it on pretty much anything. It should have some RAM. It should have some CPU resources available for things. But Yeah, there's also a docker image you can

Jonathan: let's be honest if this started on the raspberry pi one It doesn't need a whole lot of ram or cpu

Gina: Well back then it did not have plug ins to think about so things have increased a bit with With regards to resource consumption these days.

I actually recommend getting an rpi3 at least because four four cpus are nicer than one and That makes a lot of things easier because you can suddenly know Do things like print a printable G code analysis in parallel without blocking the whole server with complicated calculations and such, but yeah.

Jonathan: Yeah. So that, that maybe is a good a good jumping off point to ask what, what all can OctoPrint do now? It's more than just a babysitter or, or at least my, My kid watching camera can't also change the diaper and also feed the kid. It feels like if we were to make that comparison, then Octoprint could do at least those things.

Gina: Maybe. So, I mean, what it could do right from the get go was things like you have a basic file manager on there so you can upload stuff upload things from your slicer there and most of the slicers actually now also either have an OctoPrint plug in available or come with OctoPrint support right built in so you can configure your OctoPrint instance in the slicer and then when you slice a file you can just say send to OctoPrint, maybe even send to OctoPrint and immediately start printing and then All of that will work.

This is thanks to the REST API that I have built in, which is also something that was in there right from the get go, because I wanted to enable third party clients to, yeah, utilize OctoPrint to make your workflow a bit easier. You can yeah, you can connect to it from, from a box standard web browser, watch a webcam stream, if you have a webcam connected.

If not, that will become a bit tricky. All this file management stuff, connect to your printer look at the, at the communication terminal control the printer, like move the head around, preheat it, things like this and it also has some basic time lapse functionality built in. So that is pretty much the standard package.

It also comes with a bunch of plug bundled plugins now that will also take care of some safety checks. For example, if your firmware is known to have thermal runaway protection disabled, OctoPrint will warn you about that. If you upload a file that still has placeholders in there that were not probably replaced by your slicer, OctoPrint will warn you about that.

If your Raspberry Pi is under voltage. is experiencing undervoltage. Octoprint will warn you about that because we have seen that this can lead to browning out and interesting symptoms that are like really hard to understand if you've never seen it before. Suddenly your browser only loads half the files or the print starts stuttering, but it still goes on.

Like it's not, it's not gone. It's just halfway gone. And if you update a system, in this case, things go wrong. really, really break. The newest version will also get achievements.

Katherine: Achievements. We've gamified it. That's cool.

Gina: Yeah. The idea actually was not to gamify it per se, but that is something that we might talk about later when we come to the whole, I'm working on this full time and I'm funded by the community thing.

I wanted to make it more visible that this is the case. And I figured that having some achievements in there that are something that people maybe might enjoy. And said achievements also having the little info, by the way, I'm working on this full time. And if you want to support my work, you can do so here would be a nice compromise between annoying neck screens and ooh, got to catch them all.

Katherine: Oh, that's awesome. I would really like to hear that story. But I also had another question. And that's we keep talking about the Raspberry Pi and OctoPi. Can I use OctoPrint without if I don't have a Raspberry Pi? Okay.

Gina: Yeah, we, we, we, yes, as I said, so that basically, you can, yeah, yeah, if you have an old laptop, that is also an option.

As I said, a Steam Deck is an option. I actually set up my full development environment on my Steam Deck as a test once and that worked well.

Katherine: And then, so so, yeah. What am I missing, though, when I don't have the Raspberry Pi?

Gina: basically just the setup will be a bit more involved. It will not be a case of put your SD card in your reader, open the RPi flash tool, select this image, enter your Wi Fi credentials, and you are done.

But instead you will, yeah, in most likely if you, if you're, if you're installing it on some Linux distribution, I would yeah, you would probably on OctoPrint's download page find OctoPrint deploy, which is a script that is maintained by someone from the community, which. That's all the installation for you from the command line through a little wizard.

So it asks you some questions. And then it does all of the package installing and, and hard work for you as well. So it's, it's basically a thing of convenience. It of course also makes it very easy for me, it being the major platform to test stuff, because I concentrate on everything needs to work on the Pi.

And then the likelihood is very high that it also works elsewhere, because that is definitely a more limited platform than your run of the mill. Core 7, Core i7 laptop with whatnot.

Katherine: So you might say Raspberry Pi is recommended. Assuming you can get your hands on one these days.

Gina: I would say it is, if you are, if you just want to get up and running really fast, it would be what I recommend, yeah.

Jonathan: So Catherine, the main advantage of the Raspberry Pi is that it's tiny, and you can strap it right to your 3D printer. Good luck strapping your old laptop to the side of your 3D printer. I mean, you could probably make it work.

Katherine: Depends on the printer, right? You could probably

Jonathan: 3D print some brackets to make that happen.

But the Raspberry Pi, you can just zip tie it on there and be done.

Gina: There's also a ton of cheap thin clients that you can buy these days on eBay and such. Yeah, or Knux and yeah, like a ton of options.

Jonathan: So I, I'm talking about the capabilities. Last question I want to, I want to get in right now on that.

I've seen that. But apparently it's possible to slice right in Octoprint.

Gina: Yeah, that was something that, like, I actually thought about throwing that out again, but for now I'll keep it in. So the thing is back in, I actually have to speak a bit about, about the stuff that we wanted to get later about the whole funding thing.

So. I started working full time on Octoprint when a Spanish tech company that also sold 3D printers, basically hired me full time to work on Octoprint, like no strings attached, just do your thing and continue to do your thing, make our awesome stuff and maybe help us a bit with our projects in that regard.

And then there's that. And they, Backband, really wanted to be able to slice in the web interface as well. And so that is something that back then I looked into and also made work in a way. By basically, yeah, remote controlling and a cura engine binary that I also put on there and all of this worked nicely.

You could upload a profile, a slicing profile. And then as long as your G code not, not, not, not your G code, as long as your STL files were from the rotation where okay ish it would work. The problem is that I started, Yeah, basically a cat and mouse game because the slices, the slicer engine evolved faster than I could keep up with keeping things in sync and the whole experience of using a slicer from the browser was also a bit meh because Yeah, it would always not be able to compete with the desktop version, right?

I made it part of the whole plugin system to enable people from the community to work on that instead. And look into it. And last I checked, I think four or five weeks ago or so, I saw someone had taken up work again on the full featured slicer plugin that actually gives you a 3d view in your, in your browser, in an, in an OctoPrint tab, allows you to rotate things and configure settings for the slicer and all of that and work through this.

But yeah, for me, I have to say it, it simply never was the right amount of work versus payout. Like, yeah,

Jonathan: I can, I can see why in some cases it's a nifty feature to have. Yeah,

Gina: totally.

Jonathan: I mean, if, if nothing else that it would then, you know, enable like a workflow where you could download an STL on a mobile device.

Yeah. And then kick that over to your printer and not have to have a slicer. Cause I don't, as far as I know, there's no slicers for mobile. But on the other hand, there's some magic that happens in slicing.

Gina: Yeah. And it's gotten only more complicated over the past. Don't let me lie. I think that was 10 years ago when I started looking into this stuff.

And there has happened a lot since then in slicer

Jonathan: technology. A lot of advancements, a lot of tricks people have come up with.

Gina: Yeah, so keeping up with that would have been a project on its own.

Jonathan: Yes, yes. So I already had one. Yeah, you already had one. Well, let's, let's talk about that. Yes. And so to move kind of into this, the bigger story about making money with it.

What first comes to mind is, where, where were you at that you had the 3D printer, and it was, I, I wish there was this piece of software, and you then came from a place where you said, I bet I could do that, and I'm just curious, like, where, what the, what the background was that, that led you to that conclusion, because most people, most people would go, man, I wish there was a piece of software that would do this, I wish somebody would write this, maybe put it out on Twitter or Facebook, and then forget about it.

Gina: Yeah. So back then I happened to still be in my, as I call it now, my corporate life. I was a consultant. I was doing customer solution development for really big customers with a, with a consulting company. And my day job basically consisted of coding Java in some enterprise environments for telecommunication companies and similar stuff here in Germany.

Jonathan: Thank you, by the way, for not writing OctoPrint in Java.

Gina: No problem. And and, and I was actually having a good career there. I had good chances to really move fast and high. The problem was that yeah, the more I moved, the more people, or rather, my boss told me I'm getting too expensive to code. I need to, like, produce more hot air than code, basically.

And the thing is that I fell in love with coding ever since I touched a computer for the first time at age seven. Like back then my, my uncle gave us a very, very old, already then very, very old Apple IIe. And I was like, yeah, computer. I want to play. And my father was no, no, this is a tool. I can show you how to use it.

And then he taught me, taught me some basic, basic, basic commands. And I was hooked because this was like playing with Lego, but without the the bricks running out. And so this is basically this was a defining moment for me because then a couple of years later, I learned that this was actually a profession like at age 11 or so.

And that was when I knew what I wanted to be in my life. I wanted to be someone who writes code for a living. And so, yeah, the whole, the whole school life was was, I was directed for that and I also, of course, studied computer science and then went into the industry. And then it suddenly happened that people were telling me, yeah, well, you are too good a coder.

We will move you away from code. And I was like, no, why? And that was when I started to really itch for personal projects. So, I bought myself a 3D printer in late 2012 and had this problem that I described and just figured, Hey, I got a vacation coming up and I haven't coded something fun in a while. I could just do that.

And I could also use another language that I haven't used so much like Java and I had taught myself Python a couple of years prior and decided, yeah, well, that could work, maybe. The other option was Node. js, but getting a serial connection to work under Node. js under Windows back then was an absolute nightmare, so I decided against that.

And, yeah, so, I just happened to be a nerd. Two weeks. Okay. Two weeks from, basically from start to finish, mostly spent coding on OctoPrint, and after that it was a minimal viable product. So, that was fun.

Jonathan: Yeah, that was going to be my guess. Somebody that already sort of knows what they're doing about two weeks.

Most of your Most of your fun projects get done in about that amount of time. That, you know, to get to that first, first working, because if it takes much longer than that, most of us are just like, Oh, well, that, that was fun to work on that, but I'll go do something else now.

Gina: The only, yeah, I mean, I should add, I had absolutely no idea how this whole printer communication factor worked.

So what I did was Going with Python. Anyhow, Cura was written in Python. Cura had a communication stack to talk with the printer and stream stuff to it. So I just took that from Cura. And used it and hooked into it. Originally OctoPrint was actually a plugin for Cura that would just offer the functionality of Cura for controlling your 3D printer in, on a, on a web on a web interface.

Then people complained, why does it have this Cura dependency? We don't need that. And that was when I refactored things, but yeah. So it's, it's all a bit. Weird how this happened.

Jonathan: Yeah, so Cura of course is an Ultimaker product which good on them for making an open source. Have you, have you had continual communications with like Ultimaker?

Have they ever offered to step in and sponsor your work?

Gina: No, I mean I think the One of the founders of ultimaker at some point at least supported me on patreon But other than that there was nothing official. I think back then cura also was not yet a full ultimaker thing, so the the the thing is that the the the creator of cura worked at ultimaker, but I think That was something he started in his free time actually we met at a conference ages ago once and We talked a bit about that, but then yeah Basically, i'm not even sure if he's still with that company or not but i'm really glad that ultimaker decided to keep cura open source and Floss and all because everything else would have been a bit sad.

Really. It's a good slicer

Jonathan: Well, I think we talked earlier about how that slicing has come along as much as it has. I think if Cura had not been open source, we would not be able to say that to quite the same extent. The fact that everybody can get in there and add features and, and try things is one of the big reasons why we have so much magic in slicing.

Katherine: Absolutely.

Jonathan: So. So. Go ahead Catherine.

Katherine: Oh, so I just I was actually I'm really curious because you know we've been talking about how it Yeah, the beginnings of the project and you've talked about you know you've recruited some other other developers for plugins and other things But I wondered if you could talk a little bit more about how your your development workflow has evolved over these many years Right the way that people make software is has changed quite a bit.

And I, I know that I understand there's something that you're kind of really excited to talk about there, but I, I wondered if you could talk about your tooling and your setup, and I would really like to hear about that, but I kind of would be curious to know what got you there too.

Gina: Yeah. So, I mean, in the beginning, it was just a side project that I did in the evenings and, and on the weekends.

And that is also how our approach distributing it. Like I just threw stuff, new stuff on Git. on GitHub and people updated from GitHub. There were no versions. There were no, there was no release log. There was nothing like that. And that of course had to change once things got a bit more serious. I started creating actual releases.

I started writing change logs which I, by the way, do and usually on every release takes me at least a day because I really do put everything that changed in there, even though I know rarely anyone reads them, but I know that is complete. And yeah, and then for a while I just had versioning and such.

Then at some point I started running into these issues where releases Something new on a Thursday or Friday and then you spent the whole weekend fixing it So I started to create frozen zones for myself. I do not release anymore anything after Wednesday That is a big rule for me And also not after December the 10th Because reasons then I found myself a couple of times in the situation that still things broke and really tremendously.

And even though they've now broke on a Wednesday instead of a Friday, that still sucked. So I introduced release candidates. And that has also been really really helpful because it turns out no matter how much I try to test I have a limited set of 3d printers here. I have a limited set of plug in Versus version combinations.

I cannot test everything under the sun And people out there are amazingly creative and breaking things. So, If I now give them a release candidate for four weeks to play with, I can be sure they find a lot of stuff that I would never have thought about trying out. And that way hopefully dig out all the bugs that made it into the release before they hit everyone out there who might not be as comfortable with yeah, basically rolling back manually or, or helping themselves out of a hole.

If push comes to shove, if I cannot fix stuff as fast as I would like to. And that has been, I guess, the secret behind having very, very stable and bug free releases in the past years. This whole change from, Oh, I just push it out to, okay, four weeks of release candidates, oops, a bug, okay. Four weeks of another release candidate, oops, a bug.

That of course means slow releases, but also less breakage. Makes sense.

Jonathan: There's a, somebody pointed me at a website a few days ago. Should I deploy dot today? And the answer's always, always no. If it's like a Thursday or Friday, no. If it's afternoon, no. So right now, if I pull it up, should I deploy it?

It's the idea of a magic eight ball, I think. And right now, if I pull it up, trust me, they will be much happier if it wasn't broken for a night.

Gina: I have to admit that I also depend a bit on my gut feeling. Like when I have. Something brewing, and I could push it out now, but my gut for some reason says, Eh, wait until tomorrow.

I listen to it. I cannot really say if it has prevented a lot of damage, but I just feel better and less stressed. And that is, that really does wonders for my work life balance as well. So that is just worth it. And I've actually had a bunch of times where overnight someone reported a big glaring bug that I then quickly still

Katherine: could fix.

So that was nice. This seems like a really great You just said it. A phrase that I'm curious about work life balance. So as a maintainer of a project like this, do you ever just get really burned out? Yeah. I feel like so, so burnout has been kind of a topic that I'm interested in lately. without going to going off on a tangent.

Burnout can have ramifications, right? If you can't sustain a project because you're too burned out or it can have security implications, for example, as we just saw. Yeah. Right. And I wondered, I wondered what, you know, what you could tell us about how you handle that. Like, how do you, I mean, I understand, you know, you have either your release cadence and you don't deploy on Fridays or anything like that, but, but what else do you kind of, have in place to maintain the balance so that you don't get so burned out?

What happens when you do get burned out? How do you back off and kind of recenter? I should say that

Gina: I pretty much feel like I'm constantly, actually am constantly circling the drain a bit less so these days than let's say three years ago or four years ago when it was really, really bad. So, Yeah, how much time do we have?

Because I can talk for ages about that but yeah, so one, one rule, one very strict rule for myself is that my free time is my free time. I don't care if things explode during this time. Like there are a bunch of trusted friends also in the Octoprint space who, when I am on vacation, do have my number in, like, my actual phone number in case something really, really, really horrible happens and I need to do something.

But I trust them that they will never call me unless something is bad. almost literally on fire and The rest of the time really my weekends are my weekends. My after hours are my after hours. I have split mailboxes This is also very very important like my Personal email doesn't touch on Octoprint stuff.

I even have a mail forwarding rule that if, if just the word Octoprint shows up in an email, it would immediately just go to the other one and stay out of my inbox because otherwise I constantly get dragged back into work. I have I have a postman. pinned on my Mastodon profile that says, Hi, this is my private account.

I love to be able to use it after hours and during the weekends and on my vacation. So if you have anything Octoprint related to say, please do so here to the Octoprint account. Thank you. And I also used to have the same thing on Twitter when I was still on there. Yeah, because otherwise people will just drag you.

Back into work, I actually had someone call my landline at once, like in Germany, you still have this legal situation that if you have a website that could be considered commercial. So if you, even if you're just putting ads on there, it is considered commercial. You have to have a legal notice that also includes a way to contact you by phone.

So I had a phone number on there, a phone number that directly went to a mailbox, but a phone number. Okay. And one day I got a message from this, this phone number provider. Hey, there's a new message on your mailbox. And I listened to that message. I was like, yeah, I had this problem with Octoprint and could you please, and that was like, I don't know, at, at 1am in the morning and no, just no.

And yeah, setting these boundaries and also enforcing them when. Half the internet is trying to get your attention is very, very important because they will try to drag you back into work. They will try to get their very personal problems solved right, right now. And I have learned that just being very clear when I'm not available helps a lot in setting boundaries.

And In fact, people will then tell you, Oh, Oh, Oh, sorry, sorry, sorry. And yeah, enjoy your vacation and such, but it's, it's really important to not let yourself get sweet talked into giving up on your free time all the time.

Jonathan: Yeah. I think, I think most people understand. They just don't think about it.

Like I developed an open source project. It's, it's almost like. People don't realize that that's a real person behind that. Exactly. I mean,

Gina: I also have the problem that I think most people don't know that it is pretty much me. Like, there is no team there. There are a bunch of Yeah, there's no off the print corporation.

There is no No, I, I'm not the, not the head of an invisible team of 13 to 15 worker bees that are working on it in the background. No, if you look into the commit log of Octoprint, you will see my name a lot in there, which is why I, I, I'm still writing most of the code myself. Of course, there are also external contributions, but if you look at like the constant maintenance stuff, that is still on me.

I just need a break from everything sometimes.

Katherine: So, do you have the desire or do you have a plan to, to recruit more, I wouldn't say successors obviously because I hope you'll be working on this a long time, but, but people to kind of carry some of the load? Yeah,

Gina: I've been looking into, that is actually something that is really Tricky to do when you are already at 100 percent workload, but I'm trying, I've been trying to work on better Documentation basically to do what I would say is pretty much a brain dump so because the problem with a lot of things that I get as feature requests and such is There is a lot of, of, I've learned a lot about how different printer firmwares work, how, what, what are the things to look out for all these differences that can, can really cause you to shoot yourself into the foot if you do a certain change somewhere, and that is something that Not very many people do have, I would say, on this whole planet, let alone people who are free and able and willing to work on, to help work on OctoPrint.

So that is always stuff that needs to be communicated then in some way. And of course, the same goes for OctoPrint's whole architecture and all the bits and pieces and such. So if I really want more core contributors, I think I need to enable them to teach themselves better. And for that, I need better documentation.

And for that, I need time that I don't have because yeah. Well, so it's a bit of a, of a cycle, but I'm working through it. I'm always just working out sometime here and sometime there and then making this and that. And so it's, it's a process. It's an involved and slow process, but it's a process. And I have to say that the plugin system already did a whole ton because, and I put this in, in 2014 with version 1.

2. 0. 2015. Sorry. Yeah. So because that enabled people to, instead of waiting for me to implement something that they wanted to just implement themselves and implement themselves without having to learn. pretty much everything else, but just concentrate on the stuff that they want to do on, on the, on the interfaces that they need and all of that, and do really, really nifty stuff that, and some of the, of the most active plugin developers are all now also helping with all of the issues and, and regular with regular pushes of code and such.

So that is really, really nice

Jonathan: talking about adding. core members and co maintainers. Obviously there, we, we, it's been so recent. We can't say that without thinking about X, X, Z and SSH. One of the things that I suggested that maybe projects need to do going forwards is follow the Debian model and that it essentially meaning do not bring someone onto your core team without having actually met them in person.

And I don't know if that

Gina: could get tricky though, if you do not have a big, huge fund of travel money that you can just use for that. It's

Jonathan: true. That is, and I think it's sort of onerous over on the, on the Debian side too, because that's, that's their policy on Debian. You do not get to be a Debian packager unless someone from the Debian packaging team has met you in person.

And then the way they handle that is they sign each other's GPG keys. But. it would make it a lot harder to pull off an op like what XZ did, what happened with XZ. So I don't know, but as you say, it is, it's, it's a lot of expense for a small project.

Gina: I think we will be talking a long time yet to come about how to prevent something from that.

Yes. Like that from happening again. And frankly, for me, when I read about this over the Easter weekend, I just, All the time, I just thought this could have been me. Like, obviously, if someone did the same to Octoprint, it would not have the implications here. But still, I have had some security issues and Yeah, every single time I feel sweat on my, on my head and I feel this cold run down my spine and

Katherine: Yes?

Get

Gina: some, get some what? Nauseous? And, and so, yeah, it's like It is an experience, something like this. But my personal thing that I'm taking from the whole Xe thing is really that if someone starts pushing me that I have to, that I should accept the change, then I'll just consider it a security issue. And like, I think, I think public pushback will get handled a bit more aggressively in the future from every maintainer under the sun now thanks to that.

Jonathan: Okay, so one of the things that I've kind of thought about with this, we're, we're going a little bit off track, but that's fine. There's been a bit of a push, and you can tell me if I'm wrong, correct me if I'm wrong, it's just my, my thinking on it. There's been a bit of a push that open source projects need to be nicer to people coming in from the outside.

And One of the things we've seen from XZ is no open source projects need to be able to tell people to go away.

Gina: Yeah,

Jonathan: and it almost seems to me that maybe we've backed ourselves into this corner by insisting that we're nice, when maybe that's not always the answer. And giving, giving maintainers back that permission to be a little rude, to channel their inner Linus is maybe an important thing.

Katherine: I think both things can be true, right? Yeah, I think we

Gina: need to find some way to have less entitlement on the one side and less rudeness on the other and just get back a bit more to collaboration in general because my experience as a maintainer has been a lot of getting screamed at from people when something doesn't work and getting demands that I should have should fix it right away getting told to kill myself over really like I actually have had gotten that and like things like this when you are giving away something for free versus I mean, I cannot, I cannot say that I do not get thanks enough, but because I, I do actually get people telling me that, that I have improved their life, that I'm a role model for their kids, which always makes me like, whoa, what?

I never set out to be that, but okay, thanks. And, and things like this, and this always makes my day, but it really only takes one or two of these things. Beepholes to ruin your whole week. And this, this is also what contributes and this is how we get back to burnout. This is really something that contributes to maintain a burnout because you constantly work and you constantly work your ass off.

And you are like, I would, I would actually say that a lot of my blood, sweat and tears is in Octoprint. And then people come in and basically just scream at me, often for an issue with the printer that is not even in my control. And that is just not working out well. And then someone comes in and is really, really nice and wants to help you.

And you just need to give them commit access. I mean, that sounds kind of sweet in that moment, right? I can totally understand how that happened.

Jonathan: Absolutely. I,

Gina: yeah, I, I am very, very careful about who gets write access. Yes, and I always have been, and if truth be told, if it ever comes to the point where I will stop working on Octoprint, I will not give over maintenance to someone else, but just archive things and say, hey, if you want to continue this, just to continue, then fork and rename it and do it that way.

But I can totally see why people are just desperate sometimes and take every help they can get.

Jonathan: flashing through my mind just then were potential names for the fork and it's like you could name it nano print for nine instead of

Gina: eight. That also gives you chances for future forks.

Jonathan: There you go. There you go. Okay. I'm curious talking about this issue of burnout and then connecting it to the other thing we want to talk about.

Does being able to do it as a day job, does getting paid for it help or does it make the burnout problem worse? Or does it not matter? I wouldn't

Gina: say I think it doesn't matter because the thing is Money or money allows me to have a roof over my head to Eat to sometimes also buy myself nice things or go on vacations And most of it actually goes towards a retirement fund because I am not getting younger.

So at some point I will not be able to do that anymore, probably. So I have to plan ahead for that as well. What it does not do is give me more time. It does not allow me to sleep less. It does not allow me to rest less in general. My day still has 24 hours and I still need at least 8 hours of sleep per night because otherwise, ugh.

So I would say with regards to the burnout situation, no, it really doesn't change things. And with regards to the burn out attributing entitlement and toxicity problem with some parts of the user base. It also doesn't help at all. But of course, it is really nice to be able to really fully concentrate on one thing, on one project instead of trying to squeeze this into your life as a pet project, which is what I did the first one and a half years of this journey.

And that was actually something that affected both my mental and physical health and is something that I would not recommend doing again. Like, that was Java during the day and Python at night, and nah. Just say no. At a certain project size you really need some way to, either you need way more people that work on it, or ideally you need to find someone who is able to do it full time, ideally you if you are the creator.

Because otherwise it just doesn't scale. We expect, I mean, the whole, the whole idea about open source for many people is not that the source is being shared, but hey, I don't have to pay anything. But all of this stuff does not happen in a vacuum and people should be aware that it has a cost attached.

And usually this cost is currently paid by maintainers in shape of their health. And maybe that should not be that way. And maybe we should rethink this as a society. But no, I'm Sounding like Star Trek again, so, yeah.

Jonathan: Oh, we could, we could dive into into the philosophy and economics of that. Which would be very interesting.

I don't think we have time to, though. Yeah, it's quite difficult. I do, I do want to ask, and maybe, maybe we'll take, we'll take a show sometime in the future and just talk about that. But, so I want to ask, you said you, you worked on it basically as a hobby for a year and a half, and then were able to go full time with it.

How did you, how did you make that transition? How did you, how did you find someone that was willing to pay, you know, your salary at a living wage to be able to do that?

Gina: They found me like this was an absolute accident. For some reason, precisely at the right time, because it happened when I found myself in the situation where I, I already 80 percent so that I had one day per week dedicated to Octoprint.

Mm hmm. And that proved to simply not be enough anymore either. So the weekends it was again and some, some, sometimes I didn't do anything for, for, for a month or more because I was just too burnt out because the day job was also very, very intense and a lot of, lots of responsibility and design decisions and whatnot.

And so, yeah. And right when I was sitting there actually with some horrible back problems, which. Looking back, probably were caused by stress. I got an email in my inbox going like, Hi, I'm from company such and such. And we were wondering if we could invite you to Madrid to just get to know you and talk to, talk with you about Octoprint because we would like to just hire you to work on it.

And I was like, okay, I've never been to Spain. Why not? And then I flew over for a weekend basically. And We talked and in the end they Yeah, we had we also talked about money obviously and we agreed on something that worked out for both of us which was not A step back thankfully for me, but also not like I I was not ripping our trees with that either.

It was just okay And what was very important back to me was that I had full control over OctoPrint, that everything would work out in that regard, that I was making the decisions, that they would not go, but we do not want that feature, we want this feature, or something like that, that it would still be something that focused on the community, the global community, and not just on whatever their printers needed.

And all of that was something I got in writing, and so that was fine. Back then they also said I would get a whole team of people. I never got a whole team of people but okay. And so that is how I worked basically from summer of 2014 to early 2016. And then they ran out of money. And so I found myself with a full time open source project that really Would not be something that I could still do as a side project because I already knew what it would cause me in in terms of health problems and I figured okay Either I now go back into my old corporate Role, which I mean I could have done easily but Lots of options here.

I was not worried about that, but it was also not something that I just wanted to do and give up on Octoprint because Octoprint felt like something that I was actually helping people with that it was actually making an impact in an industry with. And so I figured, okay, I will at least try to make this work with crowdfunding.

I'm a complete risk averse person. Going self employed was something completely against my nature, but I figured if I do not try this, I will kick myself in the butt for the rest of my life and asking myself what if so I did it. And roughly, actually now four, six, yeah, six years to the day almost, my Patreon campaign launched.

Where I also explained the situation and told people I want to keep this open source. I want to keep doing this. I need to do this full time because I simply cannot do this after hours as a side job anymore. It is not a side job. It is a full time job. So it should be able to pay the bills. Help me, please.

And to my utmost surprise people did just that and have been doing just that since then. So Octoprint is 100 percent crowdfunded. There is also some money coming in through ads on the forums and the website, not in Octoprint though. Octoprint is completely ad free. And also, yeah, I also have some, some shirts up and such that people can buy from which I, from which I get some, some commission.

But overall, it really is just Yeah, it really is just all the people who use Octoprint who are keeping Octoprint funded, and that is pretty pretty damn amazing.

Jonathan: It is. That is really impressive. Yeah Okay we are already getting towards out of time. Catherine, I'll kick it back over to you. I know, it's gone really fast.

I'll kick it back over to Catherine if she has any final questions she wants to ask, and then I will get into my final questions, and we can wrap.

Katherine: Yeah, you know, I, I, I don't think I do, although I think, I think I'm not sure we ever got to the tooling thing. I know you're excited about some tooling that you wanted to talk to and I don't think we got to that, but I don't know.

I'll let Jonathan tell us if we have time for you to share your. Okay. Awesome. Yeah. I would love to, I would love to hear about how, how you're making your life easier with the motivation.

Gina: Yeah. Okay. So I will try to give you basically the quick, the really quick version of that. So the thing is that remember when I said.

That I go through all these release candidates and the releases. And of course, everything needs to be tested as much as possible. So every single release that I push out needs to be tested against a multitude of starting points in software versioning. So bunch of software versions, bunch of Python versions.

And basically what I do here is a bunch of octopi versions, because I still have people running images from three years ago out there versus some even older. That's a really good question. that are really ancient by now in, in the anonymous usage tracking pop up recently, but yeah. And, and. A long time I did all of this manually, like I took an SD card, put it in my reader, flashed it, then manually edited some files on there so it would have the right starting point, then pushed it into the Raspberry Pi, booted that up manually, waited until it was there, yadda yadda yadda, it took a lot of time.

So on every single release I had to go through something like 5 to 10 scenarios like that, and doing all of these steps manually with one card reader meant Literally hours that were spent doing nothing but flashing cards, manually going through update steps, often getting confused, having to start again, all of, so it was absolutely horrible.

And I searched for a solution for that. for years when I finally came across a nifty little device called an USB SD Max from a German company here called I think Linux automation, GMD, GMBH or something. And that is a really nifty thing because it slots into the, into the micro SD card slot in, into any micro SD card slots, slot gets powered through USB and can then also act as both a mass storage device through the USB.

plug that you, for example, put into another Raspberry Pi, or as the SD card for the Raspberry Pi as the boot medium. And you can toggle the two between through the USB connection. So I now have three Raspberry Pi 3s, three of these USBs SD Maxes, a USB hub that I can control each individual USB.

be poured on, the power on, which I also modded a bit so it actually can serve, serve the power for three Raspberry Pi threes and one Raspberry Pi four that all of this is attached to and that acts as the flash host. And then I wrote a whole bunch of scripts and laser cut myself a nice little frame in which every point, everything is mounted.

And now what I can do is I fire off a little script. It will take care of. putting the Pi offline, mounting the SD card, flashing the SD card, provisioning the SD card, going to my start version and then boot everything up, wait until it is booted, wait until it is responding, fire up the website in my browser so it might pop and there is OctoPrint and I just have clicked the button and then everything works, I make a green checkbox and off the next command goes.

And I even can now do all of that through a tailscale, a tunnel through GitHub action. So when a new image gets built, it automatically fires off all of this chain in my own network, runs a bunch of end to end tests, fully automated, and then just sends me an email if this fails. So that thing, Saves me hours on every single release on every single release candidate, and I sing its praise every time.

Like, really, I remember how horrible was it was before. And now being just able to let myself generate a bunch of commands. I also wrote a script that generates myself a bunch of commands that I can go through, like, everything gets automated, everything that is repetitive gets automated. So just being able to do that is, is

Katherine: I love it when a plan comes together.

Yeah, it was precisely that feeling. Sorry, I had a little A team. I'm giving away my age again.

Gina: I understood it. I'm giving away mine as well, right?

Jonathan: Alright, so I did have, I did promise I would ask Ken McDonald's question. And this, it made sense when he asked it. Can most feature requests be added via the plugins?

Yes.

Gina: I would say the majority can, yeah, there are some things that require some core adjustments, but these days my approach usually is, if it is not something that really only makes sense to have in core, to have for everything, for everyone, then I rather build in a way to edit through a plugin. I might write the plugin myself.

I may, I might make the plugin bundled, which means it automatically gets installed with OctoPrint or even is included in OctoPrint. Things like the software update mechanism, the plugin manager itself, the achievements the checks that I mentioned before, the safety checks for the firmware, the, the sanity checks for the G code files, all of these are plugins.

They are bundled plugins, but they are plugins. My approach really is. Like if I can write it in a plugin, I put it in a plugin. And if I cannot write it in a plugin, then I look if it might not make sense to allow implementing it as a plugin, because that will also enable others to do fun new stuff with this.

Jonathan: Yeah. Very cool. Is there anything that we didn't cover that you wanted to make sure to let folks know about?

Gina: I don't think so. I think we went over a ton.

Jonathan: We did. We covered a lot of ground. Okay. So a fun final question that I like to ask folks is what is the thing, where have you found Octoprint? You know, what has someone done with it that has surprised you the most?

What has been the most interesting use that someone has had for it?

Gina: Someone actually used it to power their home build pick and place machine.

Jonathan: Oh, fascinating. Yeah, I did not expect

Gina: that angle at all. That was, yeah.

Jonathan: That's really cool though. I like that a lot. Yeah. Interesting. I wonder if we've covered that on Hackaday.

If not, that sounds like a thing that we should. It

Gina: might be. It's interesting. Another thing, I pretty much every year or sometimes a couple of months, I just get surprised how people use the plug in interfaces that I built. Like, people have done things where, from plug ins, where I, Looked at the plug in and had to go through the source code because I wanted to understand how to how they did that with What I gave them to and there were so creative solutions and sometimes sometimes they were a bit hacky But sometimes they were also quite beautiful and like made me go.

Oh wonderful amazing and those are really great moments when you look at how people use what you build to build even more awesome stuff and And you're just in awe at how they pulled that off.

Jonathan: Yeah, absolutely. All right. So final two questions then is what is your favorite scripting language and text editor?

Gina: Python obviously is a scripting language, so it would count. I think I also do a lot of, I also do a lot in bash to be honest. And these days I primarily use Visual Studio Code, but if we are talking command line, then it's VI.

Jonathan: All right, I think both are valid answers. Okay, thank you, Gina, so much for being here.

It was a lot of fun. You, you're primarily Patreon employed these days, so go ahead and plug, what is your what's your Patreon URL?

Gina: Yeah, patreon. com slash Fusel, but if you do not like Patreon, you can also just go to support. octoprint. org and there are a bunch of options. There's also github sponsors, donor books, Ko fi.

I'm trying to give you all the options to throw money in my way.

Jonathan: Make it as easy as possible. Yeah, makes sense. Alright, it has been a blast. Thank you so much.

Gina: Thank you for having me.

Jonathan: Alright, Ms. Catherine, what What do you think? Have we talked you into pulling your 3D printer back out?

Katherine: I don't know. I think you've talked me into, into maybe getting a new one.

I I, I do have some friends who are users of OctoPrint, incidentally, I did not mention. So I feel like, I feel like I might be asking for some advice on, on maybe a new printer. I keep saying this and then I go on and, and treat myself to something totally unrelated. So we'll see.

Jonathan: I felt personally called out when she talked about people running OctoPrint on ancient, ancient installs of OctoPi and I'm like, mine's right over there and it's not been updated for a very long time.

But it does what I needed to do and it's kind of on my, on the back of my to do list. It's like, you should really pull that Pi down and just rebuild it all together and maybe put something a little faster up there. So I don't even know if that's a Pi 3, it may be like a one or two. It's over there, not turned on at the moment, because boy, everything is noisy when it's over there doing something, so.

But, one of these days, we'll get it done. Between all the other projects. I love it. Yeah, fun times. All right, Kathy, you want to, you want to plug anything?

Katherine: Oh sure. This goes out pretty soon. So if anybody's going to be at the open source summit, I'm giving a few talks. You can. Find me on LinkedIn and I'm sure I've posted something about them, but I'm talking about security, which is the cool kid on the block again because of recent events, I think.

So yeah, open source software and security is always an important conversation. Oh, and I I have a podcast open at Intel. I also have another, yeah, that podcast and reality 2. 0, another podcast. So if you're a podcast listener, you can hear my voice in many, many places.

Jonathan: Yeah. Let's see, the, the conference you're going to speak at where, where and when is that?

Katherine: Ah, that's the Open Source Summit and also SOS Community Day, which is a co located event the day before. So that's all in Seattle and that's coming up next week. So SOS Community Day is an event by the Open SSF, the Open Source Software, the Open Source Security Foundation. Sorry. And yeah, it's going to be a good time.

A whole bunch of cool open source people getting together to solve the world's problems.

Jonathan: Cool. I love it. Hopefully. I'm sure they will be talking about the XZ problem. Oh, I'm

Katherine: sure. No pressure giving a security presentation.

Jonathan: Oh, fun. All right. Well, next week we've got Andy Stewart scheduled, which if you're in the ham world, if you're in that intersection between Linux and ham, then that's probably a name you know.

If you are not, he is. Obviously a ham that does stuff with Linux. And I believe he's actually got his own Linux distro that is specifically aimed at doing fun things with ham radio. We're going to talk to him and it's going to be a lot of fun. And then the only other things that I've got to plug are.

Catch my work at hackaday. com. We have this security column. I'm in this world too. That goes live every Friday morning. We have a lot of fun with that. And then we've still got the Untitled Linux Show over at Twit, twit. tv slash ULS. And that is now available to the public. The audio version of it is. And then there's Club Twit if you want to really get in there and be part of the live show and part of the discord.

Yeah, have a lot of fun with that. I want to say thank you to everyone. We had some folks listening live, had some live questions. And thank you to everyone on the download. We sure appreciate it from all of you. And we'll see you next time on FLOSS Weekly.

Episode 777 transcript

3 april 2024 | min

FLOSS-777

Jonathan: This is Floss Weekly, episode 777, recorded Wednesday, April 3rd. Asterisk, wait, faxes? This week, we sit down with Joshua Culp, the Asterisk Project lead. We talk about the Asterisk Project, its new corporate home at Sangoma, and then fax machines, and why we all still get spam calls. You don't want to miss it, so stay tuned.

Hey, it is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and I've got David Ruggles with me today. Hey, David, how is it? Going good. It's good to have you. First time on Floss Weekly as a co host. Not your first time to co host with one of my shows.

And we, we brought you over for a very particular reason, and that is today's guest was your request. In fact, you've bugged me several times about this. Get, get this guy. This would be really fun. We need to ask about this one particular thing. And I agreed. I thought it would be cool. So I reached out and we've got we've got Joshua Collins.

Culp from Asterisk as the guest. And David Asterisk is something that you're familiar with, isn't it?

David: It is. I've, I go way back with asterisks into the early 2000s. I happened to actually get to go to Digium down in Huntsville. For those of you on the video, I am wearing the shirt. So I, I went there, I got the shirt.

And I've actually, I don't do as much with asterisks now as I did probably before. Five to seven years ago or so, it was the last time I was really into Asterix. Now I'm into the other sides of the Sangoma company. I sell and support Switchvox and stuff to some of my clients. But Asterix is, I just, I love the project.

I love the story behind it and I'm excited to talk about it. So yes, I'm a, I'm a bit of a fangirling today.

Jonathan: I understand. Oh, I, well, I've been, I've been around Asterix for a long time too, which might make this an interesting interview because We're both, we're both kind of insiders on this. There might be a lot of inside baseball that happens.

I'm, I'm going to try to channel my inner new, but so that we can get some of those basic questions out. But I don't know, I've been doing Astros for a long time as well. And like you, not as much here recently. And part of that, I think is because everybody uses cell phones these days and some other things locally have changed.

Maybe I'll get into the story of, of my attempts to get one of our local telecom companies to give me raw sip. It did not end with a good end. They were not willing to do that, but we almost got there and I would have rolled out so many asterisk boxes if we'd done that, but anyway, let's not let's not waste any more time.

Let's go ahead and bring our guest on Joshua, thank you, sir, for being here.

Joshua: Hi, it's nice to be here. Thanks for

Jonathan: having me. Yes. So we're talking, we're talking asterisk, and I'm not sure. Which way I want to go first. Maybe you can answer both of these questions in one go The the big question. I think some people are going to have is what what is asterisk?

I've never heard of this and then the other side of that question is how are you involved with asterisk?

Joshua: Yeah, so asterisk is these days a communications toolkit We're about giving People the tools to build cool things in the communication space that might be a call center that might be an omni channel solution that might just be phone system for a business in the cloud or locally.

Previously in the beginning asterisk was really centered around being a phone system, but over time, we've really changed into that communications toolkit perspective and personally I'm. What I say is the project lead basically means I make sure that the ship is pointed in generally the right direction.

So that's taking feedback from the community, from internally at Sangoma, from other places, and just figuring out Where we need to go next and one needs to be a focus and also the normal day to day stuff, like deciding policies and that kind of stuff.

Jonathan: And how, how long have you been with the asterisk project?

Do you, do you go all the way back to when it was the, the pet project for a Linux support company?

Joshua: Almost. So I think I came in around 2001 or 2002. Okay. Asterisks asterisk itself. As it's known today, actually came about in 1999. So a few years. A fun fact that not many people know the version of Asterisk that exists now is actually the second iteration of it.

There was one previously. That was completely scrapped where it had no configuration. Your configuration was C code. And if you wanted to change your configuration, you would change the C code and recompile. It's,

Jonathan: it's just like, it's just like Unix.

Joshua: Yep. So the current one was actually Asterisk NG.

But then it just turned into

Jonathan: Asterisk. That's, that's fun. And so we, we kind of hinted at it, but when Asterisk first started, before you got on board, it was, it was a part of Digium, I believe, and they wanted to do, they wanted to do Linux support, and they needed a phone system. Isn't that the way that story goes?

Joshua: it, Digium didn't even exist yet. It was Linux support services headed up by Mark Spencer, and he needed a phone system, like you said but did not have much money, and Mark is the type of person that he will code what he needs, and so he coded a phone system. There you go. Over time, it took over.

Yeah,

Jonathan: it's We had, I think we had Mark on Floss Weekly years and years ago, one of the very early episodes. And I think he said something to the effect of, it turns out it's more fun to play with phone systems than it is to do Linux desktop support.

Joshua: Yep. Not, not wrong at all there.

Jonathan: Okay. So my experience with Asterisk was building, building phone systems.

And like I said, during the top of the hour, when it was actually the local cable company started rolling out. phone service through DOCSIS, they would, they would give us an analog port on the, on the cable modem. And I went to, cause I, I was, I was fairly well tied into the company at the time. And I went to him.

I'm like, Hey, I know that's running SIP on the backend. And in fact, It's probably talking to an asterisk server somewhere on the back end. Can you let me get to that SIP directly? They're like, oh, I don't know. We'll go and find out. And they, you know, they talked about it inside corporate or whatever. And they came up to me, they're like, no, we can't do that.

And funny thing, it wasn't too long after that, that they started selling actual business phone systems where they would do the thing essentially that I was asking them to do. Which struck me

Joshua: as funny. Yeah, so additional fun fact. I'm gonna, I'm gonna have a lot of fun facts today. Yeah. Yeah, cable systems, at least they didn't, and I don't know if they do these days or not.

They didn't use SIP. They used a completely different protocol called MGCP, which was much more SIP places much more intelligence in the endpoint. So like a phone or a ATA to convert from SIP to Analog MGCP was much more lower level, the remote side controlling the end point being like go off hook and that kind of thing.

Why the whole cable world went that route? I don't know. It's the cable world. Fiber on the other hand does generally use SIP, thankfully, or not thankfully. Depending upon your opinion of SIP.

Jonathan: Well, SIP tends to just work. I do remember another experiment where I tried to make IAX, the inter asterisk exchange format, work between two different boxes.

I found out the hard way that you have to have, I think it was really, really tight timing to make that happen, and I didn't have the hardware to do that. Is that one of those things that's kind of fallen by the wayside, or is IAX still around?

Joshua: So, from a model perspective, IX does still exist. Some people swear by it, but most of our effort and time goes into SIP because SIP has taken over the world, essentially.

Yes. Yes. People, every day, whether they realize it or not, are probably using

Jonathan: SIP. Probably using Asterisk. People every day, I'm sure, are using Asterisk and don't even realize it, at some point along the call path. Do you have an idea of like how many asterisk installs there are out there and and how many calls get handled on them?

Joshua: So the only so from a project perspective, we don't have we don't report metrics or anything back The most I have is download statistics of what's grabbing tarballs and even then if they grab it from git I don't have visibility into that So from a downloads perspective, it's about 1. 5 million downloads a year From a calls perspective I should also add that's only downloads that doesn't include products that use asterisks, such as switch box or free PBX or other stuff or custom solutions or any of that from a calls perspective.

Not really, but. Still a ton. A ton. There you go. Yeah, I have, I can derive certain stuff internally because we do hosted phone systems, so I can kind of see trends and stuff that way. Voice is still very much alive. This

David: is the Floss Show, all about open source. And as we already said, I've used Asterix for a long time, but I've kind of fallen out of it.

Back when I was active in Asterix I was using a lot of Sangoma hardware. Their T1 interface cards and stuff. So I've always had a positive view and relationship with Sangoma from I guess, end user perspective. And I think it's very interesting to, from a distance, follow the transition as they have become the open source phone system company.

And they're really expanding into a lot of different areas. I'm going to start a star. They just purchased recently, which. We're not even going there. But

Joshua: I think that was

Jonathan: two acquisitions ago. Yeah,

David: that, that would not surprise me. And they definitely an M and a company mergers and acquisitions.

But one of the things that I've seen some that I'd love to, for you to speak to, if it's not unreasonable is there's been some concern in the open source community that Sangoma. It may not be quite as open source friendly as Digium was. I haven't seen anything personally, but again, I'm kind of looking at it from the 30, 000 foot view, so.

Would you be able, willing to speak to that?

Joshua: Sure. So from an open source perspective, from an asterisk perspective, I'll speak first absolutely nothing changed between Digium and Sangoma. If anything, we were given more freedom at times to just kind of do our thing. From a FreePBX perspective I've actually been helping to try to get them to be more standardized on open source and more friendly on open source standardizing processes and stuff.

And then as a company, our latest acquisition, I think was a company called, Do you know who Funality is? That's

Jonathan: a SIP provider, isn't

Joshua: it? Probably. Everyone's a SIP provider these days. The

Jonathan: name sounds familiar. We'll put

Joshua: it that way. We'll go from do you know who Netfortress is? Not offhand, no.

Do you know who Trixbox is? Yes. Okay, so that's all the same company. Okay. So our latest acquisition was actually, through a roundabout way, kind of the Trixbox company. And, As part of that, they have their own asterisk that they forked and made changes to and such. And from an open source perspective, we took those generally applicable changes and just made them open source.

We took their giant change set and picked out the parts that were generic and just made it out there so other people could use it. Including competitors. I know there's some competitors that took some of the stuff and just Used it for their thing. I'm fine with that. So we're still the goal my personal goal is I don't want to maintain multiple versions of asterisk.

So if I can open source everything I can,

so I don't hold anything back if I can. And I'm pushing that internally at Astracon. I forgot what I called myself. I think the overlord of open source at Syngoma just overseeing all that stuff and pushing it. And the same goes, I'm really going off on a tangent now. Also getting teams and stuff to contribute patches upstream and stuff as well.

So trying to do open source as much as we can.

David: That's very encouraging to hear. And thank you for answering that because. Again, it was kind of second party. I hadn't seen any of the issues directly, but anytime there's a big change, especially with a company as large and as diversified as Syngoma, you get a little bit of panic.

I mean, IBM and Red Hat is a whole different open source story that we are not going

Joshua: on. Yeah. I mean, to give a little bit more of a glimpse, I talked to the CTO weekly about open source stuff. We have an open source. Team where we all get together and talk about stuff, so that's awesome.

David: So in to kind of circle back to the IAX conversation from a second ago, I actually have infrastructure that I sort of tangentially support where we bring sip into it.

And then we have a whole series of asterisk clusters behind it, and we're using IAX between all the clusters. So we, we bring SIP in from all our external connections, but then everything in house is IAX. And it's because of that tight timing and stuff, so. We might convert to SIP at some point. We're also running older versions of Asterix behind it because it works and it's not exposed to the internet, so.

Joshua: Yeah from a conversion perspective, I'm going slightly technical here. At a larger scale E not E SIP should scale Better with packets. Just because of the threading model of X2 where it has to stuff you can't, you can't distribute the load as much in X2, but in SIP it gets more distributed.

Jonathan: So I am curious then about hardware support. Because Digium used to make some hardware, and I believe Syngoma makes a lot of hardware too. And once upon a time, it used to be a massive pain to try to take one of these cards and actually add them to an asterisk machine, because so much of the hardware, the driver support, was out of tree.

It was not actually in the upstream Linux kernel. And so there was, there was compiling. And if you updated your kernel, you had to go and compile again. And if you forgot, then you would get the 8 a. m. phone call after the thunderstorm because the server turned itself off and turned itself back on again with a new kernel and nothing works anymore.

Have we gotten, have we gotten any better about this? Is asterisk hardware actually upstreamed into the Linux kernel these days? No. Ah,

Joshua: Hardware is a dying. Hardware is a dying thing.

what's It's, it's what it's turned into is just using. SIP gateways instead of appliances. I suppose that's what, that's what most people do these days. They just set up a box and generally just forget about it unless something breaks, which usually these days is upstream PRI problems.

Jonathan: Yeah, that, that sounds about right.

Joshua: Makes a lot of sense. Plus it means also that you can send it up. The cloud. I hate saying the cloud into hosted instances elsewhere. Someone else's

Jonathan: computers. Yes Did the idea of running your own hardware die because the driver's problem was so bad? Is that what caused that?

Joshua: From a hardware perspective

Jonathan: you mean just from an end user perspective like if somebody wants to build an asterisk box.

Why, why did people move away from, you know, buying a four port FXO card and slapping it on a PCI bus?

Joshua: Multiple reasons, I think. One, it's becoming harder to actually Get lines and stuff from upstream carriers from a phone's perspective, the cost of SIP phones has gone down and then the experience can generally be better than an analog phone.

Which is why ATAs kind of also came down in price. So it's like deal with the kernel driver, have it in there. Or just buy a however many dollar ATA physical appliance and just go from there. There's also cases in like hotel rooms and stuff where you need a lot of them. And so it's easier to do an appliance in that case instead of a physical card.

Or multiple physical cards. So I will

Jonathan: tell you the thing about this answer that drives me nuts. Those physical appliances are little computers running Linux and maybe running Asterisk. So we have the exact same problem. It's just, I guess it's just managed by somebody else instead of the end user now. I would still, I would still like to see drivers go up into the upstream kernel, but I guess it's not always practical.

David: To speak to that briefly. I mean, first off, I agree from an open source perspective, drivers upstream is always a good thing. But one of the benefits that I have seen personally by splitting it out is lightning and damage issues. We used to regularly lose. FXO, FXS ports beat from close lightning strikes and sometimes the hardware they were plugged in, whereas a relatively self contained, less expensive gateway, I can sit out there, isolate it, and if it gets fried, throw another one in.

Jonathan: This is true. All right, so we, we've talked about kind of Asterisk as a business phone system, but it sounds like that's not necessarily the the real focus anymore. When we talk about Asterisk as a toolkit, what, what things have we added besides just, you know, routing phone calls to, to really kind of make it a toolkit?

What's, what are the new toys?

Joshua: Ah, the new toys. ARI, ARI, ARI. ARI stands for Asterisk REST Interface. which is essentially a simplified. I don't need to know that much about telephony to write telephony applications. A goal is to make it as simple and self contained so that people don't have to worry about the internal workings of the asterisk or C code and stuff.

But still give the primitives to build cool things. So an example is I'll go back a bit. So it uses HTTP requests and a WebSocket WebSocket gives you JSON based events, events like someone pressed a key. So a DTM key a call went into your application, and then you have easy rest interface to control that call.

So you might say, you might say slash answer to answer the call. Or slash playback to start playing back an audio file, or you might have a bridge to connect multiple things together. One of the cool things about the bridges in ARI is if you put in more than two, it just becomes a conference bridge.

If you take the third one out, it goes back and optimizes itself all behind the scenes to be more efficient. And so it gives, it, it takes, Or it gives outside developers an easier way to extend asterisk without knowing C. Got it. One of the reasons we did this was actually kind of selfish. It was for Switchvox, one of our commercial products.

They wanted better call queuing and writing a call queue in C is not the best because call queuing at its core can be considered a lot of business logic. And so we made ARI to give them the primitives to. Right. A call queue in JavaScript, which they did. And that allowed them not as asterisk developers as basic telephony, understanding people to.

write a call center queuing application and they were able to easily add in skills based routing and all that kind of stuff. So that's where stuff has gone towards. There's also other cool things that that means, like you could do multiple asterisk, asterisk instances and have a, ARI application that spans across all of them.

So you can connect things together. It's just lots of cool stuff

Jonathan: making it moving that stuff out into JavaScript instead of C keeps asterisk out of the news for Getting getting owned by you know, insert hacker from foreign country here because you're Your business logic people that aren't great C programmers had to go in and write terrible C.

Whereas doing it in JavaScript sort of keeps you

Joshua: safe. Yeah, it's JavaScript is just one language. You could do it in Python if you wanted. Go Rust. Yeah,

Jonathan: okay. Yeah, so I suppose the fact that it's The fact that it's just REST and WebSockets, anything that supports HTTP calls and WebSockets, you can write it in.

So you could do it and see if you really wanted to. If you really, really wanted to.

Joshua: Yep. The other nice thing is because you're using, you're using outside languages, you can leverage outside. SDKs for doing things. Like the latest thing that we kind of foresaw or expected what happened, the whole AI boom like three or four years ago, we added the ability to send media in ARI out to the ARI application where you can then use an SDK, like.

Google or something else and just pipe the audio in and go from there So you text to speech it and then take that result and ship it off to chat gpt if you want All without having to touch asterisk or the c code

Jonathan: It's cool. And I hate it at the same time.

David: So speaking of ARI I've got a bit of a throwback question again, because it's been a while since I've been digging around at source level. But back when I was extremely active in Asterix we had a module called external IVR. And it was specifically designed because you could, if you had long IVRs, where people were keying ahead, they already knew what questions were coming and they were just punching in answers, you could lose touch tones in there.

Does ARI, does ARI solve that

Joshua: issue as well? Yeah. It shouldn't skip DTMFs or anything. What you do with those is up to the AR application, but there is a guarantee that it is serialized and you will get them all. So they do come in order. Awesome.

David: And then a second question completely unrelated to that, but we've been taught everything we've been talking about to this point is voice and DTMF.

Does Asterix talk SMS, MMS, any other traditional. Phone type communications.

Joshua: So there's an answer. SMS and MMS from a implementation and standards perspective is. Complicated and messy at times.

Jonathan: Some

Joshua: implementations are better than others. So there is, there is technically the ability to send and receive text messages over SIP, which some providers do use for SMS.

Generally these days It's delivered over webhooks and using a REST interface instead. So those don't have direct Ability within Asterisk to do that. From a company perspective that's what we do. And then we use SIP as a notification mechanism to say, Hey, something came in so that we don't have to maintain a persistent like web socket or anything.

We just send it over SIP. Makes

David: sense.

Jonathan: Ding. Check your messages.

Joshua: Basically. The way it works is yeah, it comes in, it goes into the dial plan. And then in the case of FreePBX, it executes An AGI that goes and queries the REST interface.

Jonathan: Back when I was actively trying to sell asterisk based phone systems, one of the fun things that I would tell customers is I would, I would, I would, I'd really play up the idea there.

It's based on asterisk. It's, it's super configurable. We can do it. We can do anything you want to. We can make it, call you up and sing happy birthday to you on your birthday if you really wanted to. And that's always been one of the things that I've done. I mean, still to this day, I think it's cool about the Astra system is basically anything that you can dream up, you could make the thing do because it gives you the ability.

I mean, back in those days, I was just building stuff right in the dial plans, but you could, you could make all of that stuff work. And I kind of assumed that as, as you go forward and you push into kind of these new arenas that's beyond just voice, beyond just phone calls. That's still sort of one of the guiding principles, isn't it?

That you want to, you want to make this as modular and configurable so that whatever crazy thing a business or an individual has, they can, they can make it happen with Asterisk.

Joshua: Yep. And the other thing is we're not afraid to just be a component in a larger system. We can't be everything. do everything to everyone and do it well.

It just doesn't end well. I value the stability we have and not having critical issues occur at night. So that, that's something there's a balance there, but we try our best. Yeah.

Jonathan: So we talk about, we talk about voice. I'm assuming Asterisk can play with video as well, can't it?

Joshua: Yes, yes, yes.

These days it can. What does that look like? I mean, it's video. It looks like pictures. Ha ha! Lots of pictures at once.

Jonathan: Yeah. So are we doing, are we, are we doing, do we do video over SIP? Is this RTSP? Like what, what, how, how does this work?

Joshua: So the WebRTC RTSP. I remember when, side tangent I remember when I was at a WebRTC conference, one of the first ones, people were like, WebRTC will replace everything in two years, including our, including our cell phones and desk phones.

Here we are. Context for people who may not know. WebRTC is a set of standards for the web browser to allow. Web pages to do real time communication. It uses underneath the hood, some existing voice over IP standards. They just kind of mudged them together and then made them more complicated at times.

And then made interesting choices such as. Yeah, there's no, they didn't define a way that you actually exchange the information that's left up to you. It gets messy real fast. So our implementation was based on that allowing multi party. Conference bridges to occur using WebRTC clients and within Asterisk that required adding support for having multiple video streams because we didn't, we didn't have stream support before we just assumed one audio, one video when you're in a multi party, that's not true anymore.

And then from a SIP perspective, it was adding all the WebRTC stuff. So it essentially allows you locally to hold a multi party video conference in your browser.

Jonathan: Yeah. One of the real fun things about WebRTC is you can use whatever codec you want to so long as it's H X, X264 baseline. That's

Joshua: the one.

There was a, there was a long argument about whether H. 264 was supposed to be mandatory or not in WebRTC. Yeah. For a period of time it was not going to be.

Jonathan: I know there is a there's a development effort over at Google right now to add x265 support. To WebRTC and I am I'm sort of looking forward to that and I'm hoping I'm hoping against all hope that they they do it in a way That's not quite as brain dead as the x264 support.

So and I'm sure I'm sure you know this Being one that's played with it. In Chrome and in Firefox, there is an entirely separate code path for handling WebRTC as opposed to all of the other video handling. And I know this because one of the other projects I'm involved in, we tried to take feeds from security cameras and use it with WebRTC and throw it to the browser for, you know, basically real time viewing.

And unless your security camera has a specific x264 baseline option, the browser is just like, no, we're not going to play with that.

Joshua: Yep. This is also slightly extending in over to, um, Twitch and stuff with whip, if I recall correctly, they require very specific H264 attributes to work properly.

If you don't, then nope. It just doesn't play. Yeah, I think it's a profile ID has to be, I think they're, I think they're literally doing a string compare.

Jonathan: Probably. In that project, one of the hacks that we added that makes it work is you can just override the profile ID with the one that works on your, on your camera feed.

And sometimes that makes it work.

Joshua: Yeah. Codec negotiation.

Jonathan: Yeah, it's always it's always been a pain though I know I assume you guys have you guys have fought with this too trying to make various things work in asterisk

Joshua: Oh, yeah, and along with varying Interpretations of standards and spec.

Jonathan: Oh, yes, that is that is always the that is always the fun part Is there anything else that's new?

We talked a little bit about Kind of this idea of call centers, like what had to change in Asterisk to be able to go from, you know, a hundred phones in a small business to a full blown call center, or maybe multiple call centers tied together? Like what, I'm sure there were some challenges there.

Joshua: Yeah, so all that ARI work and then profiling around that.

We essentially put in a, Message bus, an asynchronous message bus inside of asterisk. Previously, a lot of stuff would just synchronously do things in critical paths. Like, like it's sending voice and it's also writing it to a text file in the same thread, which is generally not great because you want voice to generally milliseconds.

What could possibly go wrong? This guy, Oh, is a little out there. So there's a lot of optimizations around that. And then just more flushing out of ARI to ensure that it has all the functions it needs, because like in a call center, you need to monitor the state of phones to know if they're available, they're on the phone, they're not on the phone.

All that kind of stuff. It's worked well. And community doesn't currently use ARI, but I'm pushing them too. But Yeah, they are a, so you're going off a tangent now, they're a multi tenant, multi Asterisk Kubernetes based VoIP platform. So it scales up and down. So they were leveraging their knowledge and the issues they ran into to, in the future, more refine and improve Asterisk in that regard too.

Yeah. It's different different problems for different areas.

Jonathan: Yeah. Different problems at different scales too. Like the, the, the issues that we would run into doing a, a business phone system are just completely different from trying to do that sort of a deployment. Let's talk about security for a minute.

And when, when David was asking about Sengoma as kind of the new corporate overlord, one of the things that I couldn't help but think about is, no matter how bad company is at managing open source, It's better than having a young developer named Giatan come along and, you know, help out until you finally make him a co maintainer and then he pushes a he pushes a malicious backdoor on one of your releases, like no matter how bad Sangoma messes it up, it's not going to be that bad.

Joshua: I mean, you'd hope. Never say never.

Jonathan: Well, I mean, I suppose, but I honestly, I can't imagine that. I can't imagine a serious company making a goof that, doing something that malicious. Or a mistake that would have quite the same repercussions. And for those that don't know, it's the open source library XZ.

A, a developer, Jia Tan, who is almost certainly not a real person. It's Probably a three letter agency from some country came along and volunteered to be a co maintainer and finally got the co maintainer position and then added a backdoor in XZ that adds a backdoor to SSH. And thankfully, a Microsoft engineer, this is hilarious, a Microsoft engineer caught it because SSH logins were taking an extra 500 milliseconds.

And It's hilarious. It's like, he must not have been on the teams developer group. If 500 milliseconds was enough to stand out to him, like all of, all of the jokes at Microsoft's expense are hilarious to me around this. But at the same time, the dude is an absolute hero for finding it. But anyway, so this, this was this thought that came to mind.

Sangoma is, Sangoma is better than that. Like, even if there are problems, it's better to have a corporation come and take over. Rather than asterisk fall down that sort of rabbit hole.

Joshua: Yeah, so I'll talk somewhat about how it works. So there's only technically three people who have direct commit access.

I think I'm one of them, but I've never used it, I don't think. And then I think George has it, and then I think that might actually be it. Just because one of my philosophies is we should never commit directly, everything go through code review. We're not immune from that. Sure. The same stipulations.

And we don't, there's no, we don't do code main, we don't do co maintainers or anything of a project as a whole from outside the company. That I have trust if I can, I have the ability to see who you are and stuff inside the company. I trust that more. So otherwise there is an elevated role which allows people to triage issues and stuff.

But otherwise everyone's treated equal.

Jonathan: You know, that, that idea that you're, you're only maintain maintainers are only allowed to be from within the company a month ago, I've made a found that a bit onerous. But suddenly that seems like a really good idea

Joshua: Yeah, it's just it's the way it's always been and I I personally don't see a reason to change it I also would not wish the responsibility of maintainership on anyone else

Jonathan: Yeah, I I get that. I assume there have been security security vulnerabilities over the years are there Are there any that really stick out as having been particularly noteworthy?

Joshua: None that immediately come to mind You I think there were maybe a handful of cases where a packet could crash asterisk. And I think one of those was in a really obscure protocol for Cisco specific phones called SCCP, which hilariously is not technically a Cisco protocol because it actually came from a company that Cisco acquired, which is amusing.

But yeah, we're completely open with our security vulnerability stuff. We publish. Security vulnerability reports. And this is something from going back to Syncoma and open source, this is something I'm also working on from a company perspective to standardizing our process, bug bounty program, all of that across the entire company, products, services, infrastructure, everything just to make it more open.

Jonathan: Yeah, and just to be clear, I was not, I was not taking a swing there at Sangoma and how their open source works. I was just, I was just making, making the point about how it is infinitely better than having, having the problem that the XC project did. You can take

Joshua: swings if you want, it's fine. Well,

Jonathan: I, I am not afraid to do so if I think the, the situation warrants it, but I have not seen anything at this point that warrants it.

I do know, kind of pulling on the security thread for a moment longer I do know that one of the problems people used to have is where they would, they would accept SIP calls from the outside and they would have, you know, essentially a weak password protecting that. It would get found and then suddenly you have a, essentially an open, or sometimes literally an open SIP relay.

And On the internet, open relays on the internet is a bad thing and you know, I, I imagined spam calls would get routed through that, but you also had calls going out to, um, to, to toll numbers where you would suddenly find several hundred dollars of, of toll calls on your bill. Is that still a thing

Joshua: that happens?

Oh, yeah. Oh, yeah. Still a thing. Yeah. Still a thing. Internally, I hang out in the trunking channels which are sip trunking channels. And yeah, it's, it's still a thing still showing up. People are either the more more common thing these days is finding phone provisioning files that are open over TFTP or some other mechanism and then grabbing the username and password from that.

And also weak usernames and passwords on web interfaces is also a common thing. Like trying to brute force these days, stuff is generally locked down enough with like fail to ban and other stuff that stuff gets caught fairly quickly that way it's the other mechanisms.

Jonathan: Yeah, that's a good point.

So is it generally these Either unrestricted or where, where a, via TFTP, one of these files has gotten leaked. Is that why we get calls about our vehicle's extended warranty? Is that how we get those calls?

Joshua: Oh no, I'm going to rant about this. Let's see, do we

Jonathan: support

Joshua: rants? Who wants to talk stir shakin Oh yes,

Jonathan: oh yes.

So give us the background first. I, I sort of know what you're

Joshua: talking about. No, I don't want to give a background. I don't want to give a background for someone to ask you. What do you think Stirshaken is and what is it for?

Jonathan: I, isn't that a, a law that got passed that basically said you're not allowed to, to send spam phone calls?

Am I thinking about the right thing?

Joshua: Not exactly. Okay. David, are you gonna take a stab or do you just know?

David: As I commented in Discord. I'm quickly Googling, so that's cheating.

Joshua: It is cheating. So I, I will say what stir shaken is stir shaken is a mechanism to assert your authority to use a phone number for your caller ID or basically a level of trust, so there's three different levels, a, B, and C.

A means. Yeah, this person is totally in their right to use this phone number. B is I know them as a customer, but I don't know that phone number. So maybe, and then C is nope. I don't know that much. You may have noticed I'm did not say anything about spam calls. And that's because. It doesn't really stop spam calls.

It just stops using random caller IDs. So it doesn't, it helps to a degree. The problem is that spammers. Are just getting phone numbers that give them a stir shaken rating of a or b And so it's still making through the whole stopping spam calls thing is a separate thing about the reputation of phone numbers Where stir shaken plays a factor but other information about the phone numbers is needed to make that judgment Essentially, so we've gone through all of this work Which is continuing to change and isn't really deployed across the world or enough.

And here we are.

Jonathan: It is very reminiscent of some of the schemes in email to stop spam emails like SPF. Actually, it sounds very much like SPF to me.

Joshua: I'll also say this. On a Starshaken call, you receive an HTTPS address that you then have to retrieve.

Jonathan: Oh, what could possibly go wrong with that?

Joshua: To get the certificate used for that call.

Because it is certificate all, it is all certificate based. So, yeah.

Jonathan: So, are we just stuck with spam phone calls then? For the foreseeable future.

I, I know, I know they've thrown a couple of people in jail for making millions of spam phone calls. And that seems to maybe have helped a little bit, but, I don't know, it'd be nice to be able to find an actual technical solution to

Joshua: it. Yeah, so one of the things about star shaken is it doesn't embed an identifier that you can submit to the authorities and they can trace it back and then go after the originator.

Which can help. Yeah,

Jonathan: David,

David: I was just going to ask. I've noticed on my personal cell phone over the last 66 months, a year or so, it started telling me likely spam call. Is that stir shaken in practice? Or is that just something that cell phone companies are doing or something?

Joshua: Both it can be stir shaken, but some cell phone companies are paying reputation companies to provide a reputation score for phone numbers so they look at like how They I believe they have honey pots and stuff that identify the frequency of the call use Where it's going and that kind of information and then you can derive kind of an intent behind that.

However, spammers are now catching on and rotating through phone numbers faster so that they don't reach whatever magical threshold to be considered Potentially spammy.

Jonathan: Yes, it is it is disheartening how many parallels there are between phone call spam and email spam Seems like the exact same sort of cat and mouse game Fun.

So something else that we still have to deal with are faxes Faxes are still a thing, aren't they?

Joshua: Faxes are still a thing.

Jonathan: Asterisk has to handle faxes and Sip does not like faxes Does it?

Joshua: How do I answer that? FAT SIP does handle faxes. However, the various implementations of doing so may or may not handle faxes.

Are we sensing a theme here when it comes to SIP? It's all about the implementation. I mean, that's just

Jonathan: a theme with technology in general, but yes.

Joshua: Yeah. So disclaimer, Sangoma has a faxing product. It works generally good, still sold. It's, it's still pains me every time. It's still. A big thing. Fax is still big.

I think it'll be big until the heat death of the universe. Yeah

Jonathan: All right. So here here's the question of all of the faxes that get sent What percentage of them do you think actually has a real fax machine on one end as opposed to? a digital service Making a fax to another digital service

Joshua: actually a lot.

Oh, you think so? Okay. Yeah. Yeah Doctors offices medical practices. They they use tons of fact like physical fax

Jonathan: machines. Yeah. Yeah, it's true It's true. So what's the what's the problem? Why are why are faxes hard when you digitize and then packet eyes them? Why doesn't it just work? People's voices just work.

Why don't faxes just

Joshua: work? Two reasons. One, if you're doing purely as audio, then they're not as tolerant to jitter and packet loss as we are as humans. And so that throws them completely off from a from a, so there is a spec called T38 loose spec

Jonathan: as such things are

Joshua: implementations reflect that which actually turns it into underlying they call them UDPTL packets to transport the raw fax information that generally works fine.

I would say 99%. So it's, it works as good as fax can. I will throw in an additional fun fact though. There is also a specification for doing modem over IP. In SIP. Oh, fun. We don't support that. My second, my question though, why do you think it exists?

Jonathan: Because, no, I can tell you, I can tell you probably why it exists. It's because you've got remote hardware, like network switches sitting in Network rooms around the world. One of the places I get to work on these is in hotels. So you'll have an MDF in a hotel and there's a network switch or some kind of a phone system even, and it's got a serial port on it and it's got an old, the whatever.

Robotics modem sitting there connected off to a phone line and there's a really good chance that that phone line goes over SIP And they want to be able to remote dial into it and use the modem to be able to get back into their phone system Or network switch or whatever when something happens to the IP address and they can't get into it that way

Joshua: That's one of the reasons the second might surprise you.

Okay, it came about during an age where credit card IP

Jonathan: That would be the other one that makes sense. Yeah,

Joshua: and they were like well Modem over IP anyone? Doesn't mean it's a good idea.

Jonathan: Yeah. Yeah. Does it ever just, do you ever just sit and stop and think like, take fax machines, for instance, like the, the level of abstraction we have to make this work.

So you start out with an analog piece of paper, you put it in a fax machine and it digitizes it. A modern, modern fax machine will digitize it and then converts it from that digital signal back to the analog. And then when it hits the next device, it gets converted from that analog fax signal back into digital via T38.

It goes out over all of the different, again, layers of abstraction to go from end point A to end point B. Goes back to analog. back to digital in a fax machine, and then back to analog at a printout. It's just, it's mind boggling sometimes. And this is not the only place in technology where this happens, but I think faxes are one of the great examples of it.

And just the, I don't know, is it, is it ludicrous? Is it crazy?

Joshua: I mean, I think it's amazing it

Jonathan: works. Also true. Also true.

Joshua: No, that, that stuff doesn't faze me. It's, because fax, fax while still being used is not as common as just calls. What perplexes me and I just have to like just stop sometimes just shake my head is SIP as a standard has been interpreted in different ways.

So I'm amazed sometimes that stuff can just talk to each other. And I will give an example without naming a provider. There is a provider with about four different implementations of SIP, some of which can't even talk to each other. And I'm just like. It shouldn't be that way. How did, how, how does this even

Jonathan: How did we, how did we get to this place?

Okay, so. Somebody wants to get started with Asterisk. And I, I will, I will say The, the barrier for entry for Asterisk is actually really low. You can run it, I'm sure, on a Raspberry Pi. You can run it on a virtual machine, on your desktop. But what, what are some pointers that you would give someone that Finds this fascinating and wants to start playing with it.

Joshua: So www. astros. org. There's some info there. Our main documentation site these days is docs. astros. org. And a project that we did a few years ago was called super awesome company which is like a pre created, pre formulated project. Set of configuration files for an office with fictional people, common functionality, that kind of stuff.

So if you're leaning towards more of a phone system perspective to get your feet wet that's a great opportunity. You can use a physical SIP phone or a actual clients on your desktop or WebRTC, but do not do WebRTC cause. We should talk about WebRTC. And then from like a developer perspective, there's some tutorials on the doc site as well showing ARI in some different ways, like interacting with calls, connecting them together, that kind of stuff.

It's basically pick what you're interested in and go from there.

Jonathan: Yeah, super interesting. I'm, I'm curious Well, I'll let David get a question in first, and then I want to ask about the direction that Asterisk is going in the future. So this

David: might be a decent precursor to that question. What are, if any, the big or small Asterisk competitors out there, and how do you compare?

Joshua: So there's FreeSwitch, FreeSwitch and technically Yates still Camaleo, you, I mean, Camaleo is iffy iffy. To be quite honest, I don't focus on any of them. I just listened to the community and stuff. So that's one of the times I said, I don't know these days. I know some people who have moved from free switch over to asterisk and said, we are actually ahead in the areas they care about.

Which is nice to hear. Yeah. Yates, who remembers Yates? Do I have their view? The name

Jonathan: sounds familiar, but nothing more than that.

Joshua: Yeah. So it was a, it was another Communications tool, kitty kind of thing that kind of went in the direction of, um, BTS what

Jonathan: radio it stands for, what radio ones it stands for, what?

Yet another telephone

Joshua: exchange, yet another telephone engine, I think. But they went in the direction of doing software defined radio for mobile. And then Camellia and OpenSIPs are not really communication toolkits or phone systems. They're SIP proxies, so they are more, vastly more efficient at moving SIP traffic around and that kind of thing.

David: So do you still see implementations where you put Camellia or OpenSIPs in front of Asterix to handle that, that SIP? Proxying and then Asterix actually handles the all the

Joshua: rest of it. Yeah. So from that perspective, a lot of people tend to treat Asterix as a component for doing media based stuff the application side, and then they offload more of the general SIP stuff to Camalio since it's just more efficient at doing that.

At a higher scale. It also allows you to load balance and stuff.

Jonathan: So that is actually a really great, great segue into what's, what's coming next for Asterisk. What are the things you guys are looking at and working on? What, what new features can you hype us up about? Can I

Joshua: hype

Jonathan: you up about? Yeah, come on, hype us up.

Let's get some energy in here.

Joshua: Hype us up? I'm not a hype guy. I just do stuff and things. Or tell people to do stuff and then yeah I'm really trying to leverage the knowledge and information we have from I tell you guys it was community. That's the whole tricks box thing. The platform is called community, which is a headache for me because you have community and then you have the asterisk community.

So now I'm like, I'm doomed on naming. Yes. So their scalable up and down thing. I'm trying to learn as much as I can about that. To more flush out asterisk in that area to make it scale more to add missing functionality. One of those would be a tenant identifier. Which is a real simple thing to just tie channels and calls to a tenant.

So you can have more information and events about what tenant a call relates to. We don't really have that kind of thing currently. But they leverage it heavily. I also want to more flush out our external media, which is the ability to send and receive media to ARI stuff. Right now that's very VoIP ish and not very web ish.

It's RTP packets, UDP RTP packets back and forth. I want to shove that over a WebSocket and just make it easier to send media back and forth. And then everyone ready to take another drink? AI, AI, AI! However, my goal isn't to shove AI into Asterisk. It's just to give the tools to make it easier to integrate outside of Asterisk.

So like, we did a demo at Astercon, which should be on YouTube. And was very annoying at the time we were doing live transcription as me and my colleague, Mike were presenting. And so we were just talking, it was just live transcoding or live translating, live, whatever, over on the other screen as we're talking.

And so I glanced over and saw myself. Speaking in tech and then promptly went, I can't look at this. Your brain just shuts down. Yes. Yes. Yes, it does. So more easily facilitating that kind of stuff. Cause I, it's no, it should be no secret. is something we're looking at, too. Sure.

Jonathan: That is actually a, a good question to ask.

What is, what is Astracon and when is it next?

Joshua: Astracon. Ooh, marketing spiel. Astracon is the Asterisk users conference generally happens once a year where we get together Have presentations about various things talk about stuff. We usually have a developer conference beforehand where we bring up our qualms, quibbles and talk about improvements and stuff.

I can't say when or where it is because I don't know yet. It's been Fort Lauderdale the past few years. Maybe it'll be there again. Maybe it won't stay tuned. I should also add Our videos, our presentations were recorded. And once this is done, I'll pop it into discord, the link to that playlist.

So even if you didn't go and you're curious, you can peruse. I did a two or three talks. I've already forgotten. I did a talk on external media for transcription purposes. And I did a asterisk over the past year. Like what we did some of the hints of what's coming up, that kind of stuff. Yeah.

People can take a gander there if they wish. All right.

Jonathan: We are getting close to the end. David, do you have any final questions that maybe one final question you want to get in?

David: Absolutely. I've got one final question that kind of goes back to Sangoma as a whole. Especially with all the mergers and acquisitions is Asterix Sangoma's core.

And as they're bringing in things because I'll pick on one thing that I know about personally you've got switchbox cloud and star to star and there's kind of a broad overlap of functionality there with switchbox cloud being switchbox. I assume is asterisk based Star to star being an acquisition.

I assume is not but I don't actually know are you migrating everything that you're merging? and acquiring To an asterisk core if

Jonathan: it's not already. I'm gonna, I'm gonna jump in and make a quick guess and I can be entirely wrong and that's fine. The name star2star makes me think that it is something to do with one asterisk talking to another because the asterisk is a star.

Go ahead.

Joshua: Star2star is an interesting case. I think they were asterisk and then they were something I don't know what they're currently on. I've lost track to be quite honest. But going forward I would expect our acquisitions to be, well, I hope this is a push at least to be asterisk in some way.

So, community is a purely asterisk they are currently on a forked version of asterisk, but I try to reduce that delta, as I've said before switchbox also on asterisk, as you know, they are quite literally don't have a fork of asterisk. They're not special. They are on certified asterisk. From a existing product perspective if it makes sense and there's some benefit, then it is always evaluated.

And then going forward for like new products that may, or new products and new services, our favor is always asterisk, if it makes sense.

Jonathan: Yeah, all right. So maybe the hardest question we've had, because you've got to do some set math in your head, you have to think about all the things you wanted to talk about and then compare that to what we have talked about.

And so the question is, is there anything we did not ask you that you wanted to make sure and cover?

Joshua: I wanted to rant about WebRTC.

Jonathan: I think we got a, at least a small rant about WebRTC in. I ranted a little bit about WebRTC,

Joshua: goodness. I was not a rant about WebRTC.

David: I don't know. Do we have enough time

Jonathan: for a rant?

We have, we have enough time for a very short rant. If you want to give us a couple of minutes

Joshua: worth. Okay. I don't know if it'll be a rant, but a caution. Okay. For anyone who watches slash listens slash is watching this right now. If you ever decide to delve into WebRTC, know this. It is relatively easy.

To do the demo stuff of making a call between two things that is vastly different than creating something that goes into production, because there are very many layers to WebRTC and many standards and specs, and it is not a question of. If it will fail, it is a question of when it will fail. Hotel Wi Fi being a very good example.

And then you need to know those specs in order to figure out what happened. Additionally, SIP and WebRTC embed IP addresses in the signaling, meaning if you opt for the cloud, such as AWS, which is a NATed environment, And you're actually, your asterisk or other WebRTC platform is on a local IP address.

You need to ensure that You configure things such that your public IP address goes in the signaling or else you will have no audio I'm i'm

Jonathan: my ptsd flashbacks from working on this are coming one more thing with that is the browsers Google will make some security change and not tell you about it and it'll break all the things for a while what mdns for a while our stuff was broken because of mdns.

Yeah, that was fun. Oh goodness Okay, so final questions I want to ask quickly What's the weirdest and most surprising thing you've seen somebody do with asterisk? Where's, where's the place that you've discovered it that surprised you the most?

Joshua: That I can talk about?

Jonathan: Well, yes, that you can talk

Joshua: about.

I can't give names, but at least one, one tax agency in the world uses it. Which I had mixed feelings

Jonathan: about. Yeah, yeah, I was just thinking that.

Joshua: But, that was nice.

Jonathan: Yeah, fun, fun. Okay, so, final two questions, and these I have to ask. I'm, I'm basically contractually obligated to ask, or people will, will send me mail about it.

What is, you personally, your favorite text editor and scripting language? Ha, ha,

Joshua: ha, ha. Text editor Sublime Text 3. Okay. I have a paid for license. Cool. Scripting language, define scripting language.

Jonathan: Not C, not a systems language, but something that you would, you would hack together a little, a little script to do something.

And whatever I mean I, I won't tell you that it's a wrong answer, so whatever one you want to, you want to pick.

Joshua: I mean Bash, however, I also do Quick Stuff and Go. Cool.

Jonathan: I think either of those are totally legitimate answers. I, for a second there, I expected you to say something like, well, the asterisk dial plan language, of course.

Joshua: No, that would be Lauren who I am trying to convince to, for you to, we'll see. Yeah,

Jonathan: we're, we're open to it. Fun. All right. Well, we appreciate the time, sir. Thank you so much for being here. It was a lot of fun to learn about Asterisk, what Asterisk is up to these days and get the get the story straight from you guys about the Sangoma acquisition and I've got to say, I feel pretty good about it.

All things considered. I, I'm happy with the new corporate overlords, as it were,

Joshua: so. Yes, much to the dismay of many, I'm sure, the world did not end. Nothing changed.

Jonathan: Imagine that. Alright, thank you so much for being here.

Joshua: It's been great. Thanks for having me again.

Jonathan: Yes, sir. Alright, David, what do you think?

I love it. What's your takeaway?

David: It's awesome to catch up on Asterix. It's, it's cool to just get the few questions I had answered directly. I mean, as I said at the beginning bit of Asterix fan boy and just in, in, enjoyed

Jonathan: it. Yeah. I'm, I'm real intrigued by the ARI, the Asterix REST interface.

I assume that's what that stands for. Absolutely. It sounds like you could do some really fun things with that. So I, I, I do some smart home stuff. And I, right now a lot of it actually works using Python Flask and a little REST interface I built inside Python Flask. And I'm, I'm now thinking, well, I could pull one of the desk phones back out and connect it to Asterisk and then write a little JavaScript script.

that would make those two things talk to each other. So then you could like dial in to your smart home. It's like the sky's the limit with this stuff. And that's one of the fun things about it is if you can, if you can dream it up, you can make it happen. And that, that, that was always the that was always the serotonin hit from working with asterisk.

Like somebody would go, we really, and it's happened to me. We, we have had. Problems with phone calls, we need to record all of our incoming phone calls. It's like, oh, I'm sure I can do that with asterisks. Spend a few minutes on Google. Yeah, here's, here's essentially how you do this. And you go and set up a little system to record all incoming phone calls.

And then, of course, you add the note at the beginning. Thank you for calling such and such. Just a note, your phone calls may be recorded for quality assurance purposes. But then, you know, you're off to the races. And. One of these days, I'm sure they will call me back with the problems that they are out of disk space, but it's not happened yet.

It's just, it's just fun. It's fun that you can do all this stuff with it. David, do you have anything that you want to plug?

David: Not specifically, but it never hurts to plug Twit and the Untitled Linux Show, which I also get the opportunity to co host on from time to time. So I would say go check that out.

Jonathan: I think the plan is for you to be one of the co hosts this Saturday?

David: That's what I was told.

Jonathan: Alright. So next week we have I believe Catherine is down to co host. And we're going to talk to Gina, oh my goodness, this is a German name. Hodge? Hobg? I have no idea how to pronounce that. She will tell us how to pronounce it when we have her on.

But she is the developer behind Octoprint. Octoprint. And that is the little Linux distro that you can put on a Raspberry Pi to control a 3D printer. And I've had one of those running for a long time, and Gina is probably going to tune me out because it's a really old version of it, but hey, it still works.

And so that is next week, April 10th. So make sure and come back for that one. Let's see, things that I have to plug. Well, of course there is the Untitled Linux Show over on Twit. We mentioned that at Hackaday. We sure appreciate Hackaday as the new home of Floss Weekly. And don't forget to check the site out and my security column goes live on Fridays.

Have a lot of fun with that. And I think that's it. That's, that's pretty much what we want to let you know about. Thank you to everyone in the chat room that caught us live. Thank you to everyone on the download that listens. We sure appreciate it. Sure to tell a friend about the show if you enjoyed it.

And hey, we will see you next time on Floss Weekly.

Episode 776 transcript

27 mars 2024 | min

FLOSS-776

Jonathan: This is Floss Weekly, episode 756, recorded Wednesday, March 27th. DNSmask, making the internet work since 1999.

This week, Simon Phipps joins me and we talk with Simon Kelly, the creator and sole maintainer of DNSmask. It's a program you may not have heard of, but it probably runs in your router. It probably runs in your cell phone and it definitely makes the internet work. You don't want to miss learning about it.

So stay tuned.

Hey, welcome. It is time for Floss Weekly. It's a show about free, libre, and open source software. I'm Jonathan Bennett, your host, and I've got Simon Phipps with me today. Hey, Simon, welcome.

Simon Phipps: Hello. It's good to be back.

Jonathan: It's always a pleasure to have you here. Now, today we might refer to Simon Phipps as WebMink because our guest is also Simon, Simon Kelly of DNS mask fame.

Now, Simon WebMink, this is going to get confusing. I don't know. Have you, are you familiar with DNS mask? I can, I can basically guarantee that you've used it before, because this is one of those projects that is in lots and lots of, well, routers is actually one of the big places it's at, but are you familiar with it?

Simon Phipps: I think for me it would be an overstatement but I'm running it because I, I, I, I'm hosting a bunch of services in a rack downstairs in the guest room that are running on, you know, host and, you know, host installs a DNS mask to manage it's to sit alongside it's NGINX installation so that the the name server provisioning is stable.

Down there. So I was updating the spot. I created a new domain this morning and so DNS must be configured And was thinking to myself, someone ought to explain to me exactly what's that, that's doing one day. So this is the golden opportunity to find out what it's actually doing when it says it's provisioning DNSmasq with my new domain name.

Jonathan: I am sure that we can get all of the details on that. I think I first used DNSmasq I think I first ran into it when I started working with OpenWrt, the the firmware replacement for little home routers. And DNS mask was the, the little tiny program that it ran to do, I think, DHCP and DNS which I always found odd that one of the big things that you do with DNS mask is DHCP and see, it always seemed to me like that was a weird name.

And so hopefully we will get the, the story behind the name today. That will be fun too. Well, rather than talking about him, let's talk to him. Mr. Mr. Simon Kelly. Thank you, sir, for being here. And Let's maybe maybe start with maybe let's start with the name. What what is the the story behind the name?

How did how did DNS mask come to be?

Simon Kelley: Okay, so so the how DNS mask came to be was was a classic story of scratching an itch But this was an itch that happened a very very long time ago if you could think back through the decades to the days when when We first got internet service providers and dial up internet and internet was something you could have in your home rather than in your in your workplace or in your office.

And I had a, I had a Pentium machine running, running Linux with a dial up modem plugged into the back of it. And I could dial through to a company called UK, who were one of the, the, the very first people who would do proper internet for proper geeks And that worked great, and I had Netscape Navigator, and I could surf the web as it existed at the time, and it worked fine.

And then, then things progressed, and I bought myself a, a, a laptop. 486 laptop, and I wanted to be able to connect that to the network as well, and that was, this was in the days where network cards were expensive, and I think I built you, you could run IP over parallel ports with a, something called a lap link cable, anybody remember that?

So I connected my laptop up to my, to my desktop machine, which had a connection to the, to the internet and, and configured what in those days, I never understood why it changed its name, but what's now called NAT was then in those days called IP masquerade. So, and this is, so this was at the time a very interesting new trick which allowed you to hang as many machines as you like on your, your, your sorry internet single, single address internet connection out to the world.

Some things never change. And I, and I, and I plugged, I plugged my laptop in and I started Netscape. And while my connection was up, it worked great. But when my connection was down, and you have to remember this is in the UK where we never had, you know, Unmetered even local phone calls. So every minute that your your internet connection was up cost you money and when the connection was down if I touched Netscape it would it would decide that it wanted to to Get some data from from the network.

And the first thing it would probably do is is a It's a DNS lookup, and in those days, Netscape didn't run threads or any sort of concurrency, so it would call, it would call the, the C libraries, DNS, get host by name, call, which would then block, because when it sent, The connect, it sent the UDP packet off of the DNS, the the, the DNS query.

It would make one hop as far as my main machine and then just be dropped because there was nothing going on. Whereas on my main machine, when it did that, the library would get no root to host connect error immediately. So that, so that. So that Netscape would come back with, I can't do that because I can't look this up on the DNS.

Whereas on my laptop, I just got a blank screen that it didn't even repaint the screen when I moved stuff. And that was kind of irritating. So that was the, that was the itch that I was trying to solve with this stuff. And all I wanted was, was something, was basically a proxy that would, that would, I could connect, I could point the DNS queries from my laptop to this proxy, and then it would do whatever.

It had to do to get that information from the DNS server in my ISP. And because that was running on my, on my desktop machine with the connection, it would get the no, no route to host error when the connection upstream was down and it would send a, there is no information straight back and I didn't have blank Netscape windows that didn't, that weren't repainting because they were sitting blocked in a, in a DNS request.

So that's how it started.

Jonathan: That's great. That's I love that kind of story.

Simon Kelley: So it was, it was called DNS mass partly because it was complemented using this clever new technique called IP masquerade to do connecting more than one machine to your ISP. And It's a long time ago, and I can't remember. I think this, this, this might have been applied to it afterwards, after the very first versions, but it's the, the algorithm that it uses is, is kind of similar to doing that or masquerade where you just rewrite packets, packets rather than parsing them and changing them and a DNS query that comes into DNS mask just gets sent on pretty much bit for bit.

The same as it arrived up to the upstream host, only changing the ID so that when the query comes back the, the relevant reply can get sent to the relevant host. And that means that the whole thing runs with very little memory and very little access, which in those days was very important.

Jonathan: That's a big deal.

Yeah. I'm, I'm curious. Why, why the decision to make this open source and, you know, being this many years ago? How, how are you even aware that open source was a thing? What, what did that kind of decision look like?

Simon Kelley: Well, this, this was around 1999 2000, so open source was Oh, okay. It's a This was this, you know, this was after Linux, I was running on Linux and this was after the first version of Linux or the first version, I think I was running Red Hat 4.

2 at the time. So open source was the regular thing. And I had a machine full of open source software and it seemed like a good thing to do at the time. And it wasn't that unusual.

Jonathan: Yeah. So somewhere along the line, DNS mask got picked up. It got discovered. Like how, how long did it take from you pushing source code out there somewhere till the next thing you know, it's, it's running on everybody's routers.

How did that happen?

Simon Kelley: Running on everybody's routers happened a little later. Yeah. I mean, I think that the, the the WRT54G was the big one that everybody that's, that's where open, open work gets, gets its name from. And that's, that was, that was a couple of years in, I think before that. And there was a big run on opens fully open distributions and software installs for, for all those little plastic, cheap plastic routers that came out.

But I, I. uploaded it to Fresh Meat. Remember Fresh Meat?

Simon Phipps: Those were the

Jonathan: days. Vaguely,

Simon Kelley: yeah. And, and, and, I, I can remember even, even the very first versions it was, it was publicized on Fresh Meat, but I was, I think, I think by the time it became, Even vaguely known, I'd gone from my, from the, from the dial up modem, which, which was the, the inspiration for the whole thing in the first place, to a cable modem.

So I had reasonably fast, 24 7 internet connection, and I was running a, a web server locally on, on, On a machine that was in the attic of my house. And I can remember sitting, sitting in front of a screen with my kids after making a release and watching the web server logs and being amazed at all the different places it was going to, it was quite, all of this stuff is a long time ago.

Sounds, it sounds very naive and innocent now, but it was, it was a different world. And that really meant, meant something we say, Oh, look, it's gone to somewhere in South America. And then somewhere in Europe, somewhere in Canada downloading this stuff. And it was, yeah, it was, it was good. And I think there were about three or four releases.

And then, you know, you know, it goes, it kind of did everything I needed to do. And I kind of lost interest in it. And there were a few bug reports that I, that I'd had that, that I'd not got around to doing, I think, with this. Why are you, why are you not doing this? You know, why, why are you ignoring your software?

And I think that was a, that was a kind of fork in the road. I could either say, okay, it's done for me. I'm ignoring it and going away. Or I can start to take this seriously and make it as something which, which, you know, I will support and make into serious software and the fork I took is

Jonathan: obvious. Yeah, you know, we still in, in some, in some software circles, we still have that that little tiny serotonin hit when you see someone from, and, and these days it's more like someone commenting on a YouTube video on, I am using this from South Africa.

I'm using this from wherever it's like that is on the other side of the world. I think that's still a neat thing to see that someone on the other side of the world is making use of. A piece of software that we've been involved in or they've they've gotten some benefit out of it so I think I think that's still pretty cool Do you do you have any feeling for like how many installs of dns mask there are around the world?

Simon Kelley: So there is a there's an install of dns mask in every android phone in the world. Oh my goodness. That must be hundreds of millions billions A lot that probably is that's probably the biggest one I, most, most cheap plastic home routers have it running in some form or another. I, I still occasionally send the, the, the relevant queries to Home routers in houses that I'm visiting just to see if it's running DNS mask or

Simon Phipps: not.

So I'm presumably you, you, you get a royalty through from each of each copy that's being run there, Simon.

Simon Kelley: What do you think?

Simon Phipps: Well, that does lead on to the question, you know, you sound like the like Randall Monroe should have made XKCD2347 actually say, Hey, A project some random person in County Wicklow has been thanklessly maintaining since

Simon Kelley: 1999.

Absolutely, that definitely applies to me.

Simon Phipps: So, you know, how, how are you making this, you know, much of the world's mobile internet is resting on your shoulders and you appear to be a sole maintainer for a critical component. How is that sustainable? What do you, what, how do you feel about that?

Simon Kelley: I guess there's two different questions in there.

One of which is how do I, how do I pay the grocery bills and the mortgage and, and, and those things? And the answer is that DNS mask doesn't pay all of those things, but it pays some of them. And essentially the way that normally works is if somebody turns up on the DNS mask mailing list and they have a request for a new feature.

And they clearly come from a commercial company. I normally email back to them off the list and say, it looks like you're using DNS mask to make money and you would like some work done on this stuff. Would you like to sponsor me to do this work? And two or three times a year, the answer comes back, yes.

And we make an agreement. And they, we agree. We work, work. new features need to be added that they would like for their system, and I agree that it's something that would be generally useful. In fact, it doesn't have to be. I've, I've always made it made an option that people can either pay to have new features added into the system, which go into the GPL version that everybody gets, or if they want to, keep their secret source private, then, then I will do stuff and they get it under whichever licensing terms they want.

And it gets maintained for so many months or years at their expense. Nobody has ever, ever done that. Everybody who's, who's paid me to work on DNSmask has always said, no, it's fine. Release it, you know, release it to the world. And which is partly because I charged them less to do that, but. But I think possibly mainly because everybody thinks it's a good idea and it gets They don't have to pay for it to be maintained in perpetuity if

Simon Phipps: that happens, right?

How many people are actually maintaining DNS mask? Is it just you? It you know, are you facing a torrent of of pull requests to a Git repo somewhere that you're having to wade through? Or is there a, are you part of a team of a hundred people who spread the workload out amongst you? I'm guessing I know the answer to that one as well.

Simon Kelley: I, there is a Git repo and I, I'm the only person who has committed access to that Git repo. But, but I, not all the code that gets committed to that Git repo has been written by me. Right. And there was an act, there was an active mailing list and there are people on that mailing list who fix bugs and pass me the patches or, or make changes.

Some of the Linux distribution that maintainers for DNS mask are active on that mailing list. Right. The guys at Piehole who use DNS mask actually use a fork of DNS mask have been very supportive and very useful and helped finding bugs and making patches.

Simon Phipps: I have that as well running on my network.

So that's that's two copies of

Jonathan: DNS mask. There may be more copies of DNS mask in the world than there are people.

Simon Phipps: Yes, well, there's certainly more than there are me. So, so and who owns the copyright to any of those contributions, Simon? Have you used a CLA or does DNSmasq have a distributed copyright?

Simon Kelley: has effectively distributed copyright. Right, so

Simon Phipps: I can't buy you? No.

Jonathan: It's kind of nice to know.

Simon Phipps: Yeah, yeah, yeah. I mean I wasn't going to offer to, because I shouldn't think that my credit card's got a limit that covers that, but Thank you. So and so how do you feel about this being, being the very slim brick under the great big tottering pile of, of mobile internet usage?

Simon Kelley: I, I, I feel about it, I feel differently about it on different days. There, there are days when I really like it and I think it's interesting and useful and it's, everybody wants to feel like they did something useful for the world in their life and this is one of the things I did that was useful for the world in my life.

That's good. But the problem is that, that, the the emails to the mailing list or the emails to me and the bug reports and the enhancements keep coming on the days when I'm interested in doing something about them or the months when I want to go and do something else. Mm hmm. That's the problem.

Simon Phipps: Right. And how is that, you know it, it, so, is it truly bus factor equals one here? Or can you afford, can you go on a cruise? and leave somebody else to look after things.

Simon Kelley: I don't have anybody else to look after things. So yes, I can go. I can. I can lose interest for a month, but then I come back to a massive backlog.

And the most demotivating thing for anything for anybody is a massive email backlog of reports or misunderstandings or whatever. So, so I have a choice. Keeping this stuff under control or walking away for a little while and then coming back to it. But then I have a huge mountain to climb in terms of not just getting over the over the backlog.

Okay, you know, back Solving, solving emails is quite easy. If you ignore them, they'll either, people will either send them again or go away. Right. But, but you have to you then have to motivate yourself to, to work on this stuff, even, even when you don't want to. And there are, there are times when stuff has to happen.

You know, this is, I, I think anybody who wants to get into, into Open source software that the one lesson that I've learned and I would tell anybody is never ever write open source software. That's going to be running as root on 100 million devices.

Simon Phipps: Yeah, don't do that. So on the subject of the

Simon Kelley: most sleep so well

Simon Phipps: on the subject of the motivational topics.

I've just spent my entire year looking at a piece of legislation called the Cyber Resilience Act. And I'm not even a, I'm not even a citizen of Europe. I, I live on airstrip one. Whereas you, you, you are a citizen of Europe. Did that have you worried, CRA, or were you leaving it, hoping that we would get it all sorted out?

Simon Kelley: I, to be honest, I don't know much about it. I'd be interested to hear. From you what I should be worrying about what I shouldn't be worrying about. In terms of being a citizen of europe, i'm not sure how that makes it Well, i'm only semi a citizen of europe. Anyway, i'm still a citizen of airstrip one, but I happen to be a brexit Refugee living on living in ireland.

So yeah,

Simon Phipps: I didn't have any irish relatives, unfortunately

Simon Kelley: Neither do I but the great advantage of of the Common travel area is you don't need any to live here. Yeah. Yeah.

Simon Phipps: I won't go into CRA now then but the, you know, the TLDR is you probably haven't got anything to worry about. But I'd be happy to talk to you about that over a coffee sometime.

Simon Kelley: Yeah. To the extent that I thought about it, I think my conclusion was along the lines of the EU seems to have done slightly insane things with tech. Yeah. Regulation before or normally the practical outcome of those things have been not, apart from everybody having to click to say they don't mind about cookies.

The practical outcome of those things seem to work better than the than, than the headlines to start with. Well,

Simon Phipps: the CRA was a pretty well, pretty smart headed move actually. It was about stopping people doing dumping cookies. smart tech on the market and then abandoning it and leaving everyone exposed to become part of a Russian botnet in the future And the the problem with it is that they overlook the fact that open source exists And initially at the beginning of 2023 they exposed people like you to strict liability over defects in the software that was embedded in routers So if the last time you looked was march 2023, you'd probably be very worried indeed You But the way that it ended up getting passed in the parliament in January 2024 was much better than that because we put a load of exceptions for open source developers in and persuaded the authors of the bill that that was a good thing to do.

So generally speaking you're in a pretty pretty good place at the moment, but any commercial downstream you have will probably be asking you for some attestations in the near future and I recommend you charge them for

Simon Kelley: them. Yeah, I mean, it has to be said that every so often somebody does a a recursive search over the all the addresses on the Internet that are running DNS servers and, you know, and finds out what they are and what version they're running.

And if you if you Plot the version numbers of DNSmask. There's an awful lot of old code out there, and not just DNSmask, everything, but it does irritate me that, you know, people are using, you know, I do my best to fix security bugs, especially all bugs, but especially security bugs in a responsible and timely manner.

And it's kind of irritating when you then find that all of the code without those fixes in is still running and

Jonathan: Yeah, so speaking of which This would be a good time to ask. I think there is one or two Pretty nasty security vulnerabilities that just got found in DNS mask that are maybe about to get fixed Is that is that accurate if I heard understood that correctly?

Simon Kelley: so know that that the fixes have been released now, so there's there's there was a set of security holes in DNSSEC. So this is, it's not always been the case that have in the past been security, security bugs in DNS mask buffer overflows, which were. Entirely down to me and I held my hand when it was the problem.

In this case, it wasn't, in this case it wasn't my problem, it was the design of the DNSSEC protocol. Ah, yes. Pretty much every single DNS server. I see bind unbound. All of them had all we've all followed the the specification for DNS, DNS sec like good boys and all ended up with software which behaved in pretty much the same way.

If you sent it. Particularly mangled and large requests, which is that it essentially locked up in a loop and took a very, very long time to do the crypto required to validate the DNS. Yeah, that makes sense. The, if you want more detail on exactly what the problem is, is that when you're validating Resource DNS, resource records in DNSSEC you can have, you can send with, with those resource records more than one signature require each and each signature requires a key to do the validation.

And you can. Essentially send if you're using TCP to send the connections you can send thousands of signatures and thousands of keys and the Conforming software has to to try every signature with every key. So it's a it's an n squared problem and there's those That work is about doing cryptographic validation.

So it uses a fair amount of CPU. So it's basically, you can, you can make a a domain somewhere on the internet. And if you can persuade somebody to resolve that domain name through a DNS server, which is doing DNSSEC validation, you can just spike the CPU usage on that DNS server.

Jonathan: Yeah, it makes sense.

So the, I assume the solution is basically to detect that there are way too many keys referenced here. We're just not going to treat this as a proper packet, a proper request. Exactly.

Simon Kelley: No, no, no, no, no sane domain would have that many keys. The real, the best solution to that stuff is to change the specifications for a DNSSEC to make a hard limit.

On the number of validations that a conforming domain will require. That is something that will probably happen in the future. The problem is with all of these things is if you if you've ever had anything to do with fixing security holes is that it all has this was basically had to be done in complete secret, right?

Because, it's, the problem is fairly obvious when you think about it. And anybody who'd got any wind of this, who knew what they were doing, could have exploited it to bring down Google's public DNS servers, and Cloudfire's public DNS servers, and pretty much every ISP's public recursive DNS server. So it had to be, the fix had to be discussed and deployed in secret.

And it's, it's the one thing that you can't do it's secret is to bring in the standards organizations and say, please, could you change your standards? So that, you know, the best we could do was all the various implementers between was to agree on a, on a rough number, which was the maximum number you could have.

And then, and then. It

Jonathan: seems a bit reminiscent of the Kevin Mitnick DNS problem from about 20 years ago, where all of the insiders knew what the problem was and had to fix it and were trying their best to be quiet about it so that the cat wasn't let out of the bag.

One of the other interesting security stories that I've followed in the past couple of weeks is what they call loop DOS or DOS loop, which is, it's not a new problem, but it basically takes advantage of the fact that with UDP, you can spoof the source packet, and so you can actually You can trick a couple of DNS servers into responding in, in quick succession to each other's DNS requests.

And, and essentially you pull off a denial of service attack because you get two of them looping, talking to each other. I don't remember, was DNS mask one of the ones that could have that problem? Has, has a vulnerability with that name come across your desk?

Simon Kelley: The vulnerability with that name has not come across my desk.

I can't see why it wouldn't be a problem with DNS mask.

Most implementations or most installations of DNS mask limit where they will take queries from to inside the local network, which makes it a lot more difficult to do that unless there's there are two DNS servers inside the local network. But yeah, in theory, that could be a problem. I'm not sure how you would fix that.

Jonathan: Yeah, it's it was it was an interesting it was an interesting problem And I think I think it was a similar fix like you would just you would detect that this the message that this is trying To get me to send back could potentially be interpreted as a request on the other side and just not respond to it but yeah, it was it was an interesting problem.

So I've been told that you are also a Debian developer and I think, I think WebMaker might want to ask some more about this too but I'm curious of the, the story of From what I understand, it is quite the process to become a Debian developer, to start with and I'm just, I'm just curious what that what that journey looked like for you How much, did you have to fly somewhere to show someone your ID to be able to get your certificate signed?

Simon Kelley: I know I didn't have to fly anywhere specifically to do that, but I did go to a few a couple of Debian conferences and, and go to the, they have parties at, for, for key signing at, at Debian conferences, which are quite the edgy thing to do. It's a good idea to get as many signatures on your key as possible before you start drinking.

Yeah, so I became a Debian developer a long time ago, essentially because I wanted to, to get DNS mask into Debian. I did put some other packages as well. And this was before DNS mask was a thing that it was blindingly obvious to everybody should be in Debian. So It seemed, rather than trying to find an existing de and developer to do it I would package it myself and become a de and developer and maintain that that package.

I have to confess, I'm not a very active de and developer these days, , and in fact, over the last few months, I'm in the process of, of handing over much of the day-to-Day responsibility, even for the, the DNS Mass package to, to somebody else in Debbie and who is much more active than I are. I am and will maintain it with the current standards rather than being basically looking like it hasn't changed in 15 years.

Oh,

Jonathan: that's, that's interesting. That, that seems to, to sort of, I'll, I'll hand it over to Simon in just a second, but this is really fascinating. That seems to sort of improve your bus factor. I think there's I think there's an interesting conversation to be had about what it looks like trying to transition a single person development process to where you have, you do have at least two maintainers.

And so I'm curious where you've got where you've got someone in Debian to help keep DNS mask packaged. Have you, have you considered and is there kind of a path to maybe that person or someone else, you know, getting right privileges to that Git repository so that you can step away for a week and somebody is still there to answer emails and all of that?

Simon Kelley: Certainly, as far as the Debian stuff is concerned, I'm delighted that, that somebody else does now have right privileges. Privileges to the debbie and packaging part of this stuff and can handle much of that. And I have thought about trying to, to find other people who would take, who would have right access to the, to the repo and do thing, be able to do things.

I guess the reason I'm not going to further with that now is because. I like the fact that I've looked at all of the code in DNS mask seems a slightly strange thing, but, but one of the things, one of the nice things about working on code that, okay, some of it, I wrote 20 years ago, but code, which basically I've, I've, I've, Almost all of it I've written is I understand even stuff that I hadn't touched for 10 years.

I understand much quicker than anybody who's a programmer who spends their life maintaining other people's code will know that there are great frustrations in having to find how other people's code works and how to and having to do fixes, which are, which touch as few things as possible, just to limit the amount of stuff that you have to understand.

And I one of the motivations for me for DNSmask is it's the one piece of code that I can I understand pretty much everything that happens in it, and I can work on it very simply from that point of view. So my disincentive for getting other people to take over. Bits of the code is just that It would be their code and not my code and then it it would be code that I had to work harder to understand And could touch less deeply without risking Damaging things that makes sense.

Simon Phipps: Do you think you you know, we all get old People who are listening to the radio, can't see the fact that i've got gray hair You have you of course haven't simon because you're I don't know, you're 35 or something. No,

Simon Kelley: mine is just falling out.

Simon Phipps: You know, we're all getting older and this is something we've noticed in a couple of communities where I've, where I've been participating.

And you eventually reach the point where you ask yourself, well this is obviously software people are going to want going forward. What are we going to do about the fact that I don't want to go forward? Do you have an exit strategy or are you going to leave that as a problem to be solved in a decade's

Simon Kelley: time?

I don't have an exit strategy. I have spent some time thinking about an exit strategy and not come up with a good one yet

Simon Phipps: Yeah, yes, I know that feeling.

Simon Kelley: I mean, there is, in failing anything else, there's obviously an exit strategy, which is, this code is GPL, you can just stop doing stuff and eventually somebody who cares enough will fork the code and get rid of it.

Make it work, but I would like not to do that if at all

Jonathan: possible. Yeah, I think I think the danger there would be that there would not be one fork, but you would have Google's fork and you would have The open wrt fork and you would have you would have five or six different forks then and it would it would kind of splinter Into different projects whereas it's sort of nice to have all of them in one code base that everybody agrees on

Simon Phipps: Yeah, I wonder whether an organization like nlnet labs would be able to Come alongside and help out because they're doing a bunch.

They're doing bind and a bunch of other Related services.

Simon Kelley: Yeah. Yeah, maybe

Simon Phipps: All right, let's ask a different question completely

Simon Kelley: I Think in a night in an ideal world I'm not sure this will ever happen. But but I DNSmask is now nearly a quarter of a century old. It's accumulated a lot of code and a lot of design decisions that were made 20 years ago I certainly wouldn't make if I was starting from scratch.

And I think my ideal solution to the problem of DNSmask is if somebody else would start again from scratch and write a replacement. , I don't want do it , but if somebody else did it and made di the existing DS Mass code redundant,

Jonathan: the, the next time somebody comes along and says, you should have written that in rust.

You can just invite them to go ahead and do it . Precisely.

Simon Kelley: Yeah. Yeah, yeah. But co code does have a, a, have a, a lifetime.

Simon Phipps: You know, I, I hear you say that. And yet I'm on the board of the document foundation where I can demonstrate that's not true. We still, we still have code in LibreOffice that dates back to longer than DNS mask.

And when people want to use it, it doesn't go away. It's true.

Jonathan: Sure. I mean, the goodness, the LibreOffice binaries are still s office. bin, aren't they? They are indeed.

It is that, that, that particular project is, it impresses me so much that it has, and I'm sure there are places where the show, the code is crufty. I get that, but as a whole, it is come into the modern age so well that it is a very usable office suite as old as some of it is. That just, that impresses me to no end.

Simon Phipps: Well, there were a bunch of people who a decade ago now. Spent a great deal of personal time updating it, you know, doing the, the the refactoring that have been due for at least a decade by the time they started. They removed all the German comments. They removed all the code that made it run on CPM.

They, you know, they did a whole load of, of things. Modernized it, built a test suite, made it run in continuous integration. They did a whole load of necessary and, and valuable things. And so that's the reason why LibreOffice is remarkably fresh and usable now. It's because of all that effort that people put in when the Document Foundation was established.

And you know, I don't think that useful software goes away. Sometimes it's soul transmigrates into a new, into a rewrite, but something like DNS mask, All the time. It's useful. And you know, the number of copies I have running in my own house suggests that it's still useful. All the time.

It's useful. It's not going to go away.

Simon Kelley: Yeah. Yeah. But your point of the soul is a good one. I think the soul of DNS mask will be useful pretty much forever. Implementing something which, which provides that soul re implementing something which provides that soul would not be a crazy thing to do.

Yeah.

Jonathan: And I, you know, I said it very tongue in cheek earlier, but you know, maybe there will be some young developer that comes along and re does it in Rust, and Android and OpenWrt and some other groups say, well, let's use the REST version instead. Maybe, maybe it'll happen. We'll see. So I saw in your bio for one, I believe you have a PhD in computer science, which is interesting.

And I'm curious about the timing. Maybe let me ask that first. What happened first? Were you, did you have the PhD first or did you start DNSmasq first?

Simon Kelley: PhD by quite a long way. Okay, nearly 10 years. Okay. Got

Jonathan: it. Eight years. I figured that's the direction that would go, but I wanted to check. And then one of the things I saw was DNS sequence, no, DNA sequencing.

Those are two different things. Yes. DNA sequence. Although maybe they're similar in some ways, but I'm curious the story about DNA sequencing. What, what, what happened there? What did you do there?

Simon Kelley: Okay. So this was in terms of timing again, this was immediately after my PhD. I mean, my, my PhD was on something called ATM, asynchronous transfer mode networking which is, was the response of the telecommunications industry.

To the success of the internet, but it was before the internet. Well, this was well before the internet was mainstream, but the telecommunications industry saw that packet switching was they'd been circuit switching forever. And they saw the packing packet switching was the way to go. And ATM was, was.

A properly engineered packet switching network that they could implement on a wide scale, not this crazy stuff being made by Long haired guys in California Didn't have the discipline to do it all. I know we all know what happened ATM did Disappeared more or less without trace and the ARPA Internet went to went on to take over the world.

Yeah But I was working on that and then partly because because ATM was disappearing without trace at the time, and partly because I was in Cambridge doing my PhD, and, and by the time I'd finished my PhD, I had a wife and a child, and my wife had a job that she didn't want to leave. I knew that I had to stay around Cambridge rather than go anywhere else.

And at that point the A big scientific research charity called the Welcome Trust put a lot of money into into a research institution, which was just down the road from where I was living. I could cycle to work, which is great place called the Sanger Center, which is subsequently become the Welcome Trust Sanger Institute.

And the idea of this, it was it was a spin out from. Some Cambridge some Cambridge researchers in another lab in Cambridge who, who were taking DNA sequencing from something which could be done on a very small, very laborious scale into something that was fast enough and efficient enough to possibly to possibly sequence the DNA of whole large organisms.

So nice. They started as a practice run by work, by doing the DNA sequencing for a tiny millimeter long worm that lives in soil called Caenobactis elegans. And then moved on to fruit flies and various other things. But the plan all the way through was this was going to be part of the Human Genome Project and we were going to sequence the human genome.

All of the DNA, all three billion bases in a human. So I was looking for a job, and this was just down the road, and they wanted people to do computing, and I enough molecular biology knowledge to be useful. As a computer person because, which is partly because Cambridge in those days, I did, I did a, my first degree was also in Cambridge in computer science, but in those days, in the 80s computer science wasn't a sufficiently What's the right word?

Anyway, you couldn't, you couldn't do a degree in computer science in Cambridge. You had to go and do a proper degree. You had to start doing a proper degree and then you could change to computer science. Something,

Simon Phipps: do something rigorous first.

Simon Kelley: That's, that's the one. So my, so, so my, my first year in Cambridge had been doing natural sciences.

So proper science rather than computer science. And I'd, I'd done a third of my time on, in biology of cells and molecular biology, so I knew enough about biology to know, to be able to know something about DNA sequencing and this, so this job came up. And I spent the first five years writing application software for DNA sequencing databases where you could keep DNA sequence and, and everything that was known about DNA sequencing.

that particular piece of DNA sequence and display it on screens with different, you know, sort of long scrolls of A's and C's and T's and G's and strange lines that showed where genes started or where genes finished and what the genes maybe did and how they were related to

Jonathan: each other. You ended up I, I think even doing some, some Linux work and some kernel work as a, as a part of that job?

Simon Kelley: Yeah, so, so the first, the first half was, the first part of the job was, was doing this application work and the, the Sanger Center to start with, there's DNA sequencing life scale, DNA sequencing needs an awful lot of compute. You generate it, it was one of the, the very first big data applications. You generate an awful lot of data.

You know, these DNA sequencing machines, which are running 24 hours a day and they're generating vast amounts of raw data and that. goes through processing pipelines that generate, use a lot of CPU and generate even more data. And we would the Sanger Center was installing petabytes of disk when nobody else quite knew.

Jonathan: What a petabyte was.

Simon Kelley: How many, how many, yeah, how many bytes were in a petabyte? Was that a thousand gigabytes or a million gigabytes or the first iteration of that? That, that infrastructure was digital equipment, alpha, 64 bit computers, Unix computers which became compact alpha when digital went compact, picked up the remains of it.

And then as Linux became more mainstream, a decision was made. Not to buy any more commercial Unix servers and to run everything on Linux on x86. So at that point, the fact that I'd been playing with Linux for a few years and I had some sort of background in, in open source software, and I was a Debian developer and I had Some code in the Linux kernel, there's some drivers for, for wireless network cards that I happened to buy a wireless network card, assuming that it would probably work in Linux and found it didn't.

It was one of those things that used to happen in those days.

Jonathan: It still happens from time to time. It still happens from time to time.

Simon Kelley: So that put, that put me in a good position to, to, to be in a, in a group in the, in the computer systems. area of the Sanger Center that was going to be replacing all of their infrastructure with Linux on x86 and helped, that turned out to be in the end, turned out to be Debian on x86.

And that was, we spent a long time building building Debian systems. And that was, that's one of the reasons that DNSmasq does DNS and DHCP. Because routers need both of those things. And it's, it's nice to link them together. Because if you have a D, a machine that gets its address, its IP address by DHCP and tells the DHCP server what its name is, it's nice to put that information and the IP address it actually got into the DNS.

So those two are linked automatically. But DNS has also got a TFTP server and some, some code that, that does pixie booting. So network booting. And the reason all of that ended up in DNSmasq is that we spent a long time at the Sanger Center setting up infrastructure where we could plug we could plug a 19 inch rack full of a hundred blade servers into our network and press one button and they would all netboot, do an automatic install of Debian, get their IP addresses, their configured IP addresses and, and their configuration and everything.

And two hours later, a hundred machines would, would be logged into our system and be available to run. Batch jobs. Yeah.

Jonathan: No, that's that's impressive. I'm chatting with Simon in the back chat and we find it funny that you know If someone were to do this now, they wouldn't do it this way anymore they would just use Amazon AWS or something like that for for all of this compute probably wouldn't build it out themselves and It's just interesting

Simon Kelley: That's, that's an interesting question.

I, it's, I, I left the Sanger Institute 10 years ago, so I don't know what they're doing now. When I left we had Amazon accounts and we used them for some things. But

the amount of, the amount of storage we were using and the amount of compute we were using would have cost us two orders of magnitude more to buy from Amazon than it was to run a data center and put the machines in it. And I suspect that's probably still true 10 years later for the core things that you're using all the time.

Amazon is great for bursty things. loads. You don't, you don't buy a computer to do one job and then it sits forever in your, in your data center, not doing anything afterwards. You can just go and hire that computer on Amazon. But certainly for, for what Sang was doing, it's also because it's so data heavy.

Our biggest problem with all our compute was, was giving it fast access to Vast amounts of disk. And, and you could, okay. You could keep all of that data in, in, in Amazon's cloud, I guess, but you would never, you wouldn't get anything like as fast access to it from Amazon's compute, and it would take an enormously long time just to upload that into Amazon's cloud or move it to somewhere else.

When Amazon decided that they had you by the short and curly, we're going to charge you too much money.

Simon Phipps: Yeah, it's a similar situation to what we were finding out. I went to visit CERN fairly recently and they you know, they have a lot of compute capacity for very similar reasons, that they're hammering it all the time.

They've got a lot of data. They've got to move a lot of data around if they try and move it. And so they, they've also got a lot of compute capacity in there.

Simon Kelley: Yeah, so CERN's CERN is very similar to the CIO. We used to, we used to talk to them and do similar things. And for the same sort of reasons, yeah.

Amazon doesn't, you know, cloud generally doesn't work well for that stuff.

Jonathan: Alright, we are, we are getting down towards the end of the show and there's some questions that I like to ask as we wrap. One of the big ones is, and this is going to be difficult because you have to do some set math in your head.

Is there anything that you wanted to talk about that we haven't asked you about? No,

Simon Kelley: I don't think so. As opposed to things

Simon Phipps: you didn't want to talk about that we have asked.

Simon Kelley: The main thing I wanted to get over and I think we have is my advice is never write open source software that'll run on a hundred million devices as root.

Jonathan: At least don't do it by yourself. So, and then one of the other, one of the other questions I like to ask is, what, what is the, what is the strangest or the most surprising way that you've seen someone use this piece of software?

Where, where's the weirdest place that you've found DNS mask running?

Simon Kelley: That's a very difficult question to answer. I found it in all sorts of places You would expect it to be . Yeah, but no. Why? No, I don't. I. I honestly can't answer that question. I can't think of anywhere unexpected that I found it in Esmask.

I found it in all sorts of expected places.

Jonathan: Or, or anybody doing something particularly surprising with it? Like maybe, maybe the first time, or I guess if it was intentionally done doing PXE boot with it, then that wasn't particularly surprising. Yeah, sure.

Simon Kelley: The, the, actually there is one thing that surprised me and did eventually work into a, into a rewrite of DNS mask, which is DNS mask has because of it's, it's a Swiss army knife for people to solve DNS problems on small networks, essentially.

And one of the things people want to do is Change what the DNS looks like locally to them that, you know, they want to be able to look up google. com, but they also want to be able to look up the IP address of the machine on their local network, which doesn't mean anything outside. So you can, you can overlay a set of, of, of DNS records over, over the top of the global DNS that just appear that DNS shows and does that stuff, which is a perfectly sensible thing to do.

It's always been able to do that. That's useful to do. What? People did, which I didn't expect and didn't surprise me, is, is I would get people having problems because they said, I've, I've downloaded a list of every domain that serves advertisements. And put those into my DNS configuration with the IP address of all zeros or something as ad blocking.

Which is kind of a sensible, in some ways a sensible thing to do, but I had never expected people to do that in DNSmasq and to load up this overlay of the DNS with a million domains. And there was, at one point there was quite a big rewrite that was made to make that work.

Jonathan: So, so PyHole, so PyHole is essentially the thing that surprised you the most.

Simon Kelley: Well, yes, except that PyHole does it, doesn't do it, that, PyHole does, did it, and Pyler's very good, did it in a sensible manner, which is they have a fork, they maintain a fork of DNS mask, which Looks up all of those domains in a database rather than having, rather, rather than abusing this facility that was provided in DNS, I said, PyHole does it prop, did it properly from day one, so that wasn't a surprise to me, but it was people, people, just loading these enormous files full of, full of, of ad serving domains into my configuration, which I'd expected to have, you know, sort of 20 or 30 lines at the most.

Yeah, exactly.

Jonathan: Oh, that's fun. Alright, so the last two questions I've got to ask are what is your favorite text editor and scripting language?

Simon Kelley: Emacs. I've been an Emacs person for forever. I script in

Jonathan: Bash. That's fair enough. We had, we had the creator of Bash on back a couple of years ago, and I asked him if, if Bash even counted as an appropriate answer for that, and he was very offended and said, yes, of course it did.

Yeah. Yeah. Well, it's fine. All right. Thank you, sir, for being here, Mr. Simon Kelly, and we sure appreciate it. It was a lot of fun talking to you. My pleasure.

Simon Kelley: All right. Yes, sir. Great. All right, Simon, what

Jonathan: do you think? Well, I, I

Simon Phipps: wonder what that piece of software was doing in my Raspberry Pis. Now I know yeah, and I I didn't know I could do pixie boot from it as well.

That sounds exciting I'm an odd to try booting all my phones from

Jonathan: it. Yeah, or or boot your Raspberry Pis from it. It'll do that Yeah,

Simon Phipps: yeah, there's a kind of a recursive problem there though, because it's the Raspberry Pis that are running DNS bus, so

Jonathan: But you, you get, you, you boot one of them off an SD card, and then you pixie boot the other one, and then you pixie boot the first one back from that, and then you could just get rid of the SD card.

Simon Phipps: Yeah, I needed a new hobby, that's right. I know, it's, it's, it's great, you know, and so I'm fairly frequently trying to explain to people in the European Commission why their approach to dealing with cyber security attestation has got defects. And now I have another case that I can give them of something that pretty much every European citizen is depending on somehow.

That could be simply wiped off the face of the earth if they got things wrong with their regulation. Yes because you know, I'm, I, I completely agree with Simon about a piece of code that size And that with that you're that familiar with you don't want other people messing on with honestly, it's It's what Git governance was created for, you know, people give you pull requests and that leaves you a complete liberty to completely ignore them unless they're good.

And this, this is a fascinating project for those reasons. It will become part of my lexicon explaining to the good people in DigiConnect. How their decisions will have consequences. What do you think?

Jonathan: I, I find it fascinating. I did not know it ran in Android. I, I, maybe I should have. I probably should have known that.

But I did not know that every Android phone has DNS mask. I, I hope that he, before too much longer anoints and air, I suppose, you know, in, in the Linux kernel, you've got obviously Torvalds on top, but you've got, you've got Greg crow Hartman, who is the heir apparent and over at Linux, they do have the one maintainer at the top who More or less looks at all of the code before it makes it in.

I believe he will tell you he doesn't necessarily understand all of it anymore, but he at least looks at it and approves it. But they figured out that bus factor problem. If Torvalds died today, God forbid, but if Torvalds died today, Greg Currow Hartman would just step right in. They've already got that worked out.

And from things I've heard from them, they apparently have some other people in the wings as well, if something happened to Greg at the same time. They've got it figured out, how the project would go on. And I kind of think that DNS mask is an important project sort of in the same way. I, you know, I joked, but it would not surprise me if there are more DNS mask installs than there are people in the world.

And that's, that's mind boggling and sort of difficult to wrap your mind around something being that just it's everywhere.

Simon Kelley: Yeah.

Simon Phipps: Having said that you know, Simon's correct. One of the great fixes for that sort of a problem is for somebody to write it again. That's true. You know, the DNS mask now gives you a manual for where all the problems are and and what all the issues are.

And you can study that and and re implement. And he's also right that it, it is GPL. It's forkable. It's got distributed copyright. You know, it can't be forked malignly. So solutions didn't you know, with the greatest respect, it isn't on the same scale and magnitude as, as Lennox. That's true. Where the co kernel is is of deep subtlety and you know, people like Greg Cage Will, will point out to you that, that, that it is of the deepest subtlety and making changes to it as of the greatest peril.

Yeah. Having said that, and having said all the other things, I'd be very happy to introduce Simon to the folks at NLNet Labs, where there are kindred spirits, without a doubt, and they're even in the EU, so and they do have funding, and they've got people who can work on things. But, you know, I wish them the very best with it, doing a great piece of software that we've become unexpectedly dependent on.

Jonathan: Yes, I didn't say it during the interview, I meant to, but Mr. Simon Kelly, thank you, sir, for your work over the years and essentially making the internet work for everybody. You're a scholar and a gentleman. Alright, Simon, is there anything that you want to plug? I want to make sure and let you get your plugs out.

Oh, I so

Simon Phipps: want to, I so want to, but we haven't actually announced. the date for OGCAMP this year. But I can, I can give you an early peak. OGCAMP will be happening this year because I sent the, I sent the deposit on the venue last week, so I know it's happening. But it is up to this year's organizer to announce it and he hasn't done so yet, so I can't on your show.

Jonathan: I see. You, you know it, you know it's happening and you even know when, you're just not allowed to say. I can tell you

Simon Phipps: where even, but I just can't tell you. But I would strongly recommend that anybody that thinks that odd camp is a good thing, which of course is everybody sets aside some time in the middle of October for a visit to a fine British city in the centre

Jonathan: of the country somewhere.

That it sounds like fun. Anything else that you want to plug? You have a a, a Patreon, maybe?

Simon Phipps: Oh, so I've got all my own stuff, you know, patreon. com slash web mink visit web, web M dot I N K for all the links about me buy me things on Amazon if you like, but honestly the most important thing anyone could do if they enjoyed this conversation is become a member of the open source initiative because OSI needs.

It's to have a growing base of members so that we can not just fund our growing work but also demonstrate that we have support for it from the community. As we go talk to the European Commission, as my counterpart Deb Brandt goes and talks to folk in Washington, D. C. about how open source needs to be protected as they attempt to legislate foreign tech companies out of existence.

Thank you. So please, please join os i opensource.org/join and and we'll be I dunno about eternally grateful, but a great grateful for many, many days after

Jonathan: you join. Absolutely. I, you know, I have not joined Ossi. I need to look into that.

Simon Phipps: It's a very reasonably priced Mm-Hmm, . Although, Jonathan, because you are a pro, you'd probably go for the pro level at $300, but you can become a member free of charge.

There is a very economic ordinary membership that you can have at 50. And you're doing it basically because you want us to exist and survive. Yeah. But there are some benefits as well, like being able to vote in the elections and make sure that the right people become directors.

Jonathan: Yes, yes, absolutely All right.

Next week we will be back on the third and we are talking with joshua culp About asterisk. I am excited about this one. I i've been using asterisk for years now I've got some fun war stories about asterisk. So it'll be a lot of fun to talk with joshua about that don't forget you can follow my work at Hackaday.

We've got the security column, goes live on Fridays. And we've also got the Untitled Linux Show, still over at Twit. That's twit. tv slash ULS. Check that out as well. Thank you everyone for watching. Those of you that watched us live and those on the download, we sure appreciate you all being here. And hey, we will see you next week on Floss Weekly.

Episode 775 transcript

20 mars 2024 | min

FLOSS-775

Jonathan: This is Floss Weekly, episode 775, recorded Wednesday, March 20th. Meshtastic Central.

Hey, this week we're talking with Ben Meadows and AJ McWilkin about Meshtastic. There's been a lot that's changed since the last time we've had them on. There's been a Rust developer that's added to the team. There's been some new privacy features. There's a web flasher that's really easy to use. Oh, and they convinced me to join as one of their developers.

You don't want to miss it. So stay tuned.

Hey everybody. It's time for Floss Weekly. That's a show about free Libre and open source software. I'm your host, Jonathan Bennett. And today we've got Rob Campbell with us. Hey Rob, where did you come from?

Oh,

Rob: well, just been hanging out with you at the Untitled Linux show, but.

Jonathan: Yeah, so today is today was sort of a scramble day we had We had a guest canceled not too very long ago, and all of my co hosts, regular co hosts are still traveling after scale, you know, that big open source conference I'd hoped to bring a couple of them on to talk about it.

And in the emails over the past couple of days, everybody kept saying, we're still on the road. I'm going to be in the air. I have a doctor's appointment that day. It's like, okay, fine. We'll put the call out. Anybody can come. That's how Rob got here. Anybody can come podcast with me. I'm so special. No, Rob, Rob just recently, Rob just recently joined the the cool group because he got the new the new microphone.

It sounds great. So that was a

Rob: requirement.

Jonathan: That's the requirement. That's really the requirement. We like to have good audio. So many people listen to the show. So today Rob, we've got a couple of devs from the Mesh tastic project. And this is not something that you knew a whole lot about and that's sort of on purpose because I We've talked to Mesh tastic on Floss Weekly before and ever since then I have been addicted to it.

and buying devices. And then I started writing code for the project and it eventually got to the point to where they gave me the developer badge and said, you're one of us now. And here, let us start sending you hardware to test with. So full disclosure, I'm sort of on the inside on this one and I have gotten hardware from these guys.

So just. Know that. And Rob is sort of here as the audience proxy. The one to ask, I have no idea what you're talking about, please explain that thing, that developer y thing that you just said.

Rob: Yeah, I've heard you talk about it for some time now, off and on, but I had not gotten around to looking into it until

Jonathan: yesterday.

So our goal for the show today is to convince Rob to go and order some hardware. There we go. All right. So our two guests today are Ben Meadows, Meadows, Meadows. I have to ask him how to pronounce that. And he is a full stack C sharp dot net and view JS web developer, but we love him anyway. And he is sort of the lead dev, the guy on the top of the mesh tastic stack.

And then we also have. Adam McQuilkin, I may call him AJ because his username in discord and GitHub is AJ McQuilkin. And he is a computer science and engineering degree holder from Dartmouth. He holds a patent in USB hardware, HID design. And he is, let's see, a certified wilderness first responder working with SAR port 10.

part time that's Search and Rescue, which is interesting. We may talk about that too. And he is one of our developers, and he's the Rust guy on the project, and that's interesting to dive into as well. We talked last week about Rust, and why everybody should use Rust, and maybe this week we'll talk about what it looks like when a, a young, talented, excited Rust developer wants to join your C project, because that can be interesting.

Let's go ahead and bring the guys on. Ben and AJ, welcome to the show. Great to be here. Thank you so much for having us. Always a pleasure. Yes. Thank you guys for agreeing to be here at the last moment. It was literally yesterday and I went into our shared chat channel and said, guys, we have less than 24 hours till the show.

I need somebody.

AJ: We need anyone bottom of the barrel. Just please come on

Ben: whenever you throw up the bat signal. I'm there. Yep.

Jonathan: Yep. And I appreciate it. So let's start with Ben and let's, because we're on sort of a different platform now being in Hackaday let's start real quick with that 30, 000 foot view, what is MeshTastic and why would somebody be interested in it?

Ben: Yeah. So the kind of the elevator elevator pitch of MeshTastic is. It runs on a largely like Arduino code base on little low power devices that have LoRa radios. And the idea is that you or at least the typical path for folks is that they go and download the Meshtastic app and they pair to these devices and send text messages and position, location, information, telemetry messages, any, any kind of small things that you can fit into small packets.

Meshtastic is capable of sending that over the LoRa protocol. And we have our own kind of spin on, on LoRa. We use LoRa peer to peer protocol to send those messages over the mesh network formed. By by these little devices. So that's kind of the elevator pitch. I like to say is off grid texting

Jonathan: so let's let's get out of the way real quick a couple of questions that we see all the time And first off is this lorawan?

And second is this a replacement for the internet? No,

Ben: and no You know the the lorawan question almost I would say comes up daily in our discord people are like Hey, how do I get this on to lorawan and and there's really You know, the, the, the first part of that, of those you know, Laura peer to peer and Laura Wann, they both share the same CHIRP protocol of Laura, but Laura Wann is very much like a more structured multi channel based communication, whereas Meshtastic using peer to peer is very ad hoc large packets.

They're fundamentally incompatible. There's, There's some shared hardware, which makes it really interesting. And I think that probably adds some confusion to the matter. But even the, the large nature of, of the lure of peer to peer packet size we're still talking about 256 byte packets.

So these are, these are very small payloads that we're sending. This is not a, not a replacement for the internet, unfortunately dashing some people's hopes and dreams.

Jonathan: You know, we barely have

AJ: enough space to send the IP header, not much less any video streaming, audio streaming, anything like that. So, unfortunately, yeah, keep having to

Jonathan: say no to that one.

There is a project where you can you can tunnel IP through Meshtastic, but it comes with a whole bunch of, there's a whole bunch of asterisks on that one. You know, like a bunch of caveats. Look, you've got to be doing this on the fastest settings. Please don't do this on the public channel. And You're on your own.

Good luck.

Ben: Yeah, I think, I think there's some interesting cases there for like SSH sessions over, over the highest bandwidth channel, but it's. It's not what I would call a great experience for the most part.

Jonathan: Yeah, so we do have a question and this this question actually came in even before we started the show.

It's David Ruggles, another, another friend of ours. He says, I'm excited to hear from Eshtastic today. I want to start playing around with this. My first question is how do we get started? He says he's played in both the Raspberry Pi and the Arduino ecosystems. He normally starts with a breadboard and wires.

And he's seen the Rack Meshtastic Starter Kit referenced in the docs, but he says it looks like it needs soldering to get started. Is that true, and is it the best place to start?

AJ: And do you want to take that one? I have some of the hardware, I can be like the showcase if you want.

Ben: Yeah, the Rack Meshtastic Starter Kit is a really great Great one to get started with.

Yeah, I just got one right there. So there shouldn't necessarily be any soldering required unless you want to add a screen, but you don't have to have a screen within Meshtastic. Plenty of people run, run nodes headless and just keep everything in the app. The screen is more for diagnostic usage unless you're trying to use the device as a, as a standalone.

A standalone MeshTastic device, which requires a lot more buttons and more, more involved stuff. But no, you shouldn't need, you just need a battery if you want to run it portably. And that's, that's a great one that I would recommend getting started with. And we have a lot of documents in our MeshTastic.

org that kind of have some instructional stuff in terms of getting started and recommended hardware. But that one, that one's the number one for me, the, the rack.

Jonathan: Yeah, and then what about Raspberry Pi? Is there any option for using that?

Ben: Yes, I'm glad you asked,

Jonathan: Jonathan. This is a very inside, very inside baseball question.

AJ: ahead. Yeah, I was going to say, I was going to divert to, usually I divert to you, actually, for that one. You are the one who knows

Ben: about it. One of our promising new developers, Jonathan Bennett, has has, has worked very hard on Linux native support for, for Meshtastic. So you know, we have Raspberry Pi, I think, is the most compelling target because it's, it kind of aligned with the whole.

low power portable operation. Not that the Raspberry Pi is, is equivalent to some of the Arduino stuff in terms of power usage. It's obviously because it's a full blown Linux SBC, it's going to be a little bit more power hungry, but it, you know, there's, there's a, there's a lot of opportunity there in the, in the future for like base stations and stuff, but the key, the key with the the Raspberry Pi and the, and the Linux native in general.

Is making sure that your Laura radio that you're adding on to that has SPI bus exposed. Yes. One of the things that we see almost all like every day is some folks coming in to the discord asking about

Jonathan: that one particular wave share

Ben: hat. Yeah. It's always, it's always that wave, the one wave share board that has you are, and, and we just can't support.

Those, those types of of hardware. We have to use SPI to talk directly to the LoRa radio because we have such a a tight communication with, with the LoRa radio using a library called RadioLib that's kind of underpins all of our transport level stuff. So that's, that's kind of the, the The basics of the Raspberry Pi is you need, you need header, your your Raspberry Pi with headers and your, your Laura radio hat.

And you can obviously breadboard your own if you can find a compatible bare Laura radio. But I always recommend that folks start with the. WaveShare SX 1262 hat that has SPI. Yes.

Jonathan: Yeah. There's, there's some fun stuff going on there. What's that, AJ? Yeah, I was

AJ: going to say, like, so I'm, I usually recommend, like, it depends on where you are.

Like, I think technically, like, it sounds like if you're doing Raspberry Pi and you're ready for soldering, then I would agree with Ben. I think you'd probably want to go the route of going maybe a Raspberry Pi or the rack starter kit. But I guess for non technical people, like I started this project and I wasn't.

Like super technical about all of this. I started with one of the TVMs, right? These are like pre made boards. You can get these, you stick a battery in and you can immediately get started and go. So I guess I'd caveat that with like, if your goal for this project is to just chat with people, then. The, the racks can sometimes be a bit of a pain depending on how much setup you want to do, but if you want something to just get started in going, we have Etsy shops.

We have, these are available on wherever Amazon, Alibaba, all of that. So I guess that's like a small caveat to what Bennett said.

Jonathan: Yeah. And it kind of depends upon what your budget is because you can, you can buy the rack for not very much. You can, you can buy one of the T beams for not very much, or you can, you go to the Etsy shop and buy something like the messenger.

Which that's you could barely see it. It's the little red thing back there with all the buttons on it. And that is It's a pretty impressive device but it costs like I forget exactly how much if you have him build it for you It's like 200 I think and you know when you when you first see that price compared to everything else you go Oh my goodness, that's ridiculous How much that is until you go to build one yourself and you realize One, how many parts go into it, and two, how fiddly it is to put it together.

And it's like, okay, that's, yeah, that's a reasonable price, isn't it?

Ben: Yeah, there's not a whole lot of turnkey devices out there. And so you, you know, right, right now, I think we're at a, we're at a point where there's a lot of, you know, kind of cottage industry makers like, like Tony Troffo that, that put out stuff to, to help people get started.

And then there's a few, a few manufacturers, like the TECO device from the Lego is, is a quote unquote turnkey device. It's got a screen, it's got a GPS it's got a injection molded case.

Jonathan: Yeah. So David asks again, can Meshtastic be used for IOT communication or is it overkill for something like that?

He says, he's looked at LoRan directly and it seems to have a high barrier to entry maybe Meshtastic is either better documented or easier to get into. I would say IoT is one of the things that is very interesting for, for Meshtastic. You want to expand on that?

Ben: I would say it depends on what kind of IOT stuff you're talking about.

Like telemetry is we have a lot of native support for that, particularly for we have a list of sensors on our, on our website that are, that are supported. We, when we launched 2. 0 of MeshTastic, we decided to cut a lot of the kind of fiddly analog based sensors. And. Kind of draw a line in the sand to say it's we've got to have I squared C sensors There's tons of stuff on the on the market that at a fruit and Rack wireless with their wisp block system for their for their starter kit support that can gather like temperature Relative humidity barometric pressure all those sorts of key key device met our environmental metrics and We really wanted to make it easy for folks to just plug and play.

So the, the thing about the I, I squared C device sensors is that they get auto detected on startup. So if you plug it in, if you plug in a supported sensor, it should just connect out of the box and you turn on the environmental telemetry module and it'll start sending those metrics out over the mesh and you can, you can do all kinds of stuff with them at that point.

AJ: Yeah, in comparison to LoRaWAN, I think they're, I mean, like Ben said earlier, they're the same technology underlying. So I guess it depends, like LoRaWAN is dedicated for this and there's a lot of professional support around it. So if you're gonna like deploy something that's like, that needs to be bulletproof and you want like a company to blame for it, if it goes wrong, then I don't know if we're the right way to go, but one of the cool things that I always like to accent when I hear this kind of question is that LoRaWAN is like, we are a mesh.

And so your range in theory can be as far as you want. And so if you need to deploy like kind of a variable, a variable network in like a very mountainous terrain, let's say, and or a very flexible situation, then oftentimes the mesh will serve you very well for that kind of thing. So I guess that's one of the advantages of our platform over lower WAN that I'd highlight too.

Sure.

Rob: So AJ, how did you get started in the project?

AJ: Oh, man. So how did you want to go? I guess. So I, I found MeshTastic. So, okay, so this goes into the start point. So I worked search and rescue in college part time. And so my, the team I was working on was having communication troubles. And so like, I went down a bunch of rabbit holes of like, how can I solve this?

And eventually I found Meshtastic through that. And then I did a Capstone project. I realized that there was not really a way to manage networks at scale. And so my Capstone project in college was to build what's now become the Meshtastic network management client, essentially to figure out a way to manage.

Networks at scale, or at least that's the, the North star of that project is to manage that network at scale. And eventually I, I bugged Garth and Ben enough and just kept poking them. And then here we are a couple, what is it? I think a year and a half later now I think I got started in like early 23 or early late 22, maybe.

Yeah, I don't know. We can, I guess, I don't know how deep you want to go into that side of it, but there's a lot of lore there.

Rob: So what's the status on the management project?

AJ: Oh man. So currently it's mostly works. No. So I think it's the management client is always in a state of churn because it's like there's, I think people, it's been cool to see how excited people are about the management clients.

And I think there's been a lot of. Different ways. I think people want to go with the client. And so right now like right now this week, like working on a, a refactor to make actually Jonathan, you, you know, this too, like to make the connections more reliable and to make sure that it doesn't randomly like timeout serial connections, that kind of thing.

I've been a

Jonathan: guinea pig for that.

AJ: You have, I've just been pinning, pinging him randomly. I was like, Hey, I just pushed a change. Can you fix this? Or can you test this? And then you're like, no, it's still times out. I'm like, okay, I'll go back. Go back and work on it, but yeah, like right now, I think my, my personal goal for it is to get it to a point at which it's completely reliable and it, it just works as a standard client and then to start accenting more of those like network based features, such as like graph analysis or.

Even remote administration so I guess I view those at this point more as like North Stars than actually practically implemented stuff.

Rob: How long do you expect it to take before you get it

Ben: to,

Rob: That kind of status or how

AJ: complete is it? How complete is it? Well, I guess, what do you, what do you want to do with it?

I guess is the, you know, the threshold. Yeah, I mean, I, I don't, I don't know. I think it's, I always hesitate to put like an actual timeline on it just because, you like the unfortunate reality for me is that I'm, I'm doing this as like as a part time thing and I like have a full time software job.

So it's kind of like at the mercy of that, that side of my life. I don't know, I guess it's, it depends on yeah, I guess what you're looking for. Like, I think in terms of it being bulletproof, like I'd love to get there by B3. Like one by the time, whenever best Chestick releases B3, I'd love to have the network management client ship as a fully.

Tested fully reliable client, but I guess I kind of hesitate to put more of like a strict timeline than that.

Ben: One of the things that I think is also worth highlighting about, about kind of the state of flux within the network management client is it's actually some of the features that relies upon or fairly new into the firmware so the, the neighbor info module that we, that we added to kind of support some of the.

Visualization of the, the mesh topology is still pretty new within the firmware. So that we've kind of had to go back and retrofit the firmware with some of these, these features that would even make the things that, that AJ is wanting to do with the, with the management client possible. So we, we're trying to remove some key features.

bottlenecks on our side to make sure that he's got what he needs on, on his end. And that's kind of a, a common theme I would say with Meshtastic is like, everything goes through the firmware. So you have to, You have to have a lot of collaboration and interaction between the client developers and the firmware guys.

Rob: I understand there are other clients. What is the, what's the status of those other clients? It's still a lot of work to do on those.

AJ: I guess, Ben, you want to take that? I can speak only probably to the web and Python

Ben: clients at this point. Yeah. I would say that the most complete experiences within MeshTastic are the, the Android and the iOS app, because those.

Those are kind of, I think, where most folks use MeshTastic is for like the text messaging and the mapping features. So those are pretty feature complete.

Jonathan: They also, they also each have dedicated developers, which I think is one of the things that makes a big difference. And,

Ben: and MeshTastic originally shipped with the Android app.

So that one's kind of, kind of been brought along the longest. And then Garth later on Came along and developed the iOS app. There's also some, some clients in terms of like the Python CLI client for doing some of the, a little bit more advanced features for, for configuring nodes, for instance And that one, we're actually, that's another one that got shipped with the original version of MeshTastic and has not really had a prod, a platform owner maintainer.

I've just kind of been helping it limp along lately. So we, we actually put patches to it, right? Yeah, we actually put out a I put out a GitHub issue the other day to just, Kind of call for like, Hey, somebody want this,

AJ: right? And that's the unfortunate like a lot of like I forget to mention this, but a lot of mechanics clients are like primarily single developer things where.

Like someone takes ownership of it. Like the management client has been kind of my like pet project for a while. And then like each, each client usually has like one person who's kind of that primary. And I think one thing one thing we've been chatting about is like because I'd love contributors to the management client, but I think like one problem that's having is it's, there's a huge barrier to entry on the code base, just cause like fundamentally the stuff we're doing is like somewhat complicated and we're working in like a weird desktop stack, that kind of thing.

But it's kind of like an open question on the project right now of like, how do we first of all, get people excited about actually working on the clients. Cause we were very lucky to have a huge contributor base to the actual firmware which is phenomenal. Like we always love, love that community.

But I guess one thing we're, we're considering now is like, how do we make our clients accessible? How do we get people to feel ownership over the clients? And specifically the. The, the, the, the very top of mind one, like Ben mentioned was the Python is the Python client where we haven't really had that person for a while or

Jonathan: for someone like I'll, I'll jump in real quick.

Sorry, I'll, I'll be one of the guests on this one for just a second. There's also the, the web client that is great. But it's, it's behind a little bit too. There's some things that just, you can't, you can't set there or that don't work there yet. And I don't know that there's anybody that really has made that their project either.

We, we have somebody that recently added, so a developer came and essentially said, well, why doesn't the web client work on the Linux native stuff? It's hard. And nobody has spent the time on it yet. And he's like, Let me see what I can do and came back, you know, a few weeks later with a big patch that made it work And it needed a little tuning up, but it works.

It works really well. I think there's there's one or two little little crash Issues that can occasionally cause a crash. I'm still trying to figure those out But though the wider web client like the actual JavaScript stuff of it needs some love And I'm not sure who's gonna jump in and and do that.

Maybe it's going to be me. I don't know. Hopefully not.

Ben: Maybe we need to put an issue out there on GitHub for that one too. Yeah, we probably should. There's also the there's also some unofficial clients kind of in the community too. I think there's some, some Go code that is capable of communicating with devices.

And then I've got a C sharp based CLI slash library and. It's multi platform, so.

Oh, there's a,

Jonathan: yeah, there's other stuff too. Like I've seen on over on on Reddit, somebody put together a a cross platform client like a really simple one. Well, no, I was going to talk about the LISP one first. Oh, I haven't seen the LISP one. I think he said it was, I'll see if I can find it.

Yeah. He basically, he said, there's no, there's no simplified client yet. So I wrote one and I contacted him. I'm like, dude, this looks really good. Would you be interested in coming on the discord and making this a little bit more official? And he's like, well, maybe, but it's in LISP. I doubt anybody would really want to do anything with it.

Okay, fine. And then there's also another one that's written in, in Flutter and Dart. Randall, when you go to listen to this later, you. Somebody, somebody's using your language. But and that one looks pretty interesting too, but those are, you know, strictly unofficial at this point, but I think, you know, it's, it, for, for one thing, it's pretty exciting to see somebody doing this, like an unofficial client, somebody cares enough to write it, but also it's pretty interesting for the project as a whole, because I think there probably is room to have a unified client that looks the same on Android and iOS and the desktop.

And that, There's some benefit to that too. So interesting stuff. Well,

Ben: and worth mentioning, you know, AJ's net management client started out as, as a unofficial client, but we brought it into the organization. So there is a path for that. Like if, if we see you know, promise there in terms of. of that being a useful thing to pull into the ecosystem.

It doesn't make sense for every project. Like, we probably wouldn't pull in that Flutter app, for instance, because we have our own official apps on iOS and Android, but there's definitely room for, if we identify other things in the community that look like they're kind of aligned with the project as a whole, we might pull those in in the future.

Yeah,

AJ: I guess something I'll, I guess I'll add to that, like from my perspective, like the fact that we're seeing a lot of interest in like a simplified client, I think is interesting to me just because that like from a design side kind of implies that we're too complicated. Right. And so there's this kind of like this issue of what persona retargeting in our actual client applications.

Like, we're kind of assuming with a lot of our configuration with a lot of our the, the, the, the number of knobs that we expose to people. Right. Seems like it's overwhelming. And I guess that that kind of begs the question of like, do, well, there's been discussions of like, do we need all these knobs? Do we, whatever, whatever, but also like, yeah, how do we design this for someone who's both new and someone who's pretty experienced with the project?

But I guess that's, that's kind of something for some visibility that we're discussing internally as well.

Rob: Speaking of someone who's new, someone like me who hasn't gotten into it yet. I could see all kinds of use cases like the sensors and stuff, but mentioning the iOS, Android, the ability

Ben: to text.

Off grid, how, how does the, for example, the iOS

Rob: app interface with the, the Laura?

AJ: Yeah. So essentially I'm speaking a little bit out of my depth on the firmware. I'm mostly client, but essentially the way that works is for example, like the TV, right. It has two, it has two radios in it. Technically one radio would be the lower radio that actually does the long range networking.

But it also has a Bluetooth radio in it. And so essentially the way the mobile clients primarily work, and I think the Android might also support serial. I don't quote me on that, but essentially they'll connect to Bluetooth and then that message will go into the radio, the radio will do whatever they need to do with the messages.

And then if it should, if it decides that's the right way to go, it will then send that out to the mesh, which is only over Laura. I don't know if you have anything to add to that then, but that's like, I think that's the highest level explanation.

Ben: Yeah. Yeah. Bluetooth is typically how we connect to devices from the mobile side of things.

The Android does. as AJ mentioned, it supports serial and Wi Fi connection for, for some of the like ESP 32 targets to have a Wi Fi radio, but Bluetooth is generally the Avenue.

Jonathan: One of the, one of the Outstanding to do issues that I'm aware of is adding Bluetooth support to the native client because you know, people will have their like a pi zero with a with a radio and a screen on it.

It's like this thing is great But how do I talk to it? So

AJ: What

Jonathan: do I do with this thing, you know, it's like, oh well the bluetooth it's not it's not there yet It doesn't work. It's it's on it's on the list. Eventually. We'll eventually get to it

AJ: Well, that's the fun thing is there's a lot on that list That's how these projects work.

Unfortunately, there's, there's always a list and whoever takes it off that list will be the person who, who sees it first, I guess.

Ben: I've

Rob: read on the site, it mentioned a range. I think it was 158 miles. And I don't remember the kilometers. And I believe those were more like. Theoretical maximums are the maximum that somebody has done.

What what's a realistic range? Well,

AJ: one of the cool things about Laura is it's it's essentially line of sight. Right. So the reason that our range is that distance is because that's the furthest people have been able to get line of sight.

Jonathan: There was a balloon test, wasn't it? Oh, I think there was a

AJ: balloon test.

There was also someone who did a shot from Calgary down into Montana. Which is that one blew my, that, that one blew me away when I saw that. It was like, you went, not only went cross border, but you went like significantly cross border. So I guess that's my understanding is that it's primarily just how far you can see.

Cause we don't, we don't diffract that much, but if you can see someone just because of the nature of how Laura works, you can get significantly signal specifically below the noise floor. And so yeah, that's, that's kind of one of the cool things about us is if you can see it, you can likely

Ben: talk to that person.

Yeah, and worth pointing out for that particular one that you mentioned from, from Calgary into Montana, that was just with there, there was no directional antennas or anything that was just little, yeah, just a little omni directional antennas. And that, that one, I believe was 254 kilometers. So really, really impressive.

I mean, it's actually hard past a certain point to overcome the curvature of the earth. So you really have to get like two spots with pretty significant elevation to be able to do something like that. But I always say, you know, a few miles is kind of the, the, the range that most folks would get. Getting one node up highs is really the, the game changer for, for extending range.

If you're I think if you're operating. In a mesh where you have a bunch of people with handheld nodes talking to each other you're, you're going to be fairly disappointed because it's going to be similar to like two way radio ranges. But if you, if you get one node up high, you know, on a drone or set something up on a hill or in a tree it makes a huge difference.

Jonathan: Yeah, so one thing to add real quick there, I don't think we've made extremely clear, is it meshes, which means, you know, if Radio 1 sends a signal out, Radio 2 up in the tree sees it, even if it doesn't, this is important, even if it doesn't have the encryption key to be able to decrypt it, because Meshtastic is all encrypted, It will still repeat that and send it back out.

And so radio three can be, you know, theoretically even over the curvature of the earth. But if you've got one in the middle, that's up high enough, you can bounce off of that and get even further. So, you know, if, if you want to spend the time to put your nodes exactly where they need to be, you can really have a lot of range on this.

So there, there's, there's options. It's flexible.

AJ: So with that, there's been a lot of, I'm sorry. Go for it.

Rob: I was just gonna gonna say, so with that meshing yeah. How many, how many notes can you mesh together? What's the scale we could really

AJ: look at? ? This is the eternal question. We don't, we don't have hard numbers.

I mean, we have like theoretical numbers and we have like, we have the levels at which networks start failing, but that's like, my understanding is that somewhat still somewhat of an open question of like exactly how many nodes we can support. Yeah. That depends a lot on like your hop. You know, you're so we have, we have a parameter like compliment, right?

Where it's the number of bounces, essentially a packet can make before the client just stops or a given node will just not repeat that packet anymore. And so it depends a lot on that, but I don't know, Ben, what's the number? Like a hundred something is I think the most we've had like

Ben: sustainably.

Yeah, there's, there's the theoretical numbers are, I think a little bit over 300. Yeah. Is, is the theoretical and we've definitely had some, some meshes that approach that size. We, a thing that we've actually been running into a lot more lately is larger meshes as we've introduced features like our MQTT.

So that's, that's a a message broker that you can enable a connection to to basically do internet backhaul to your, to your mesh. So you can kind of glue different, different mesh networks together. And that has created some absolutely massive regional meshes. The U. S. one, I don't know how many there are.

nodes are in that, but you, you pretty quickly, yeah, you pretty quickly run into limitations of what, what the little these little Arduino boards can, can handle in terms of, of RAM usage. So we've actually had to make some changes like rolling the oldest node off of, off of the internal database to make room for new nodes coming in.

And so as you get kind of closer to these theoretical limits, we have this. This sort of rolling ring buffer of nodes coming coming through the device so it's it's definitely presented a lot of technical challenges, but we don't have like a We always say that that maximum number of nodes in a mesh is kind of a fuzzy number because it depends a lot on on your configuration what hardware you're working with and How chatty everything is.

It's it's kind of a sliding scale.

Jonathan: Yeah. One, one thing to add in there with that, with my developer hat still on it depends upon how you have your nodes set up. So if you've got somebody that is sending a location packet every 10 seconds, you're not going to be able to get as many nodes on the network because you're saturating the airtimes.

Whereas if you have all of those nodes set up, you know, kind of carefully to behave well. On your mesh, you can get a lot more, you know, you have a lot more room, a lot more headroom to grow. So it kind of depends, as you said, it depends on a lot of things.

Rob: I was looking up when I was looking into this, I was seeing some different discussions about how long it takes, say you're just sending a regular, you know, Text message to somebody.

How long it takes to travel. I assume if you're just doing one hop, it's fairly quickly, but if you have multiple meshes, does that slow it down, or what are we looking at there?

Jonathan: Yeah, Ben, do you want to take that one?

Ben: Yeah, it's, you know, for, for point to point delivery, it's usually less than a second in, in most cases on the default, but then you obviously get some additional time to rebroadcast, and we actually have some some weighted rebroadcast delays that that make sure that there is not a as much of a contention window so that nodes don't sort of rebroadcast over each other and, and and cause packet loss.

So it could take a few seconds to, for, for that delivery to happen as you increase the number of, of hops involved to, to actually deliver that message if you've got, If you don't have direct line of sight to that particular node you're communicating with.

Jonathan: Because all of, all of your devices that can talk to each other, they're going to be on the same frequency.

And on that frequency, in a, in a given region, You can basically only have one of them talking on the airwaves at the second. It's a, it's a shared ether the way the old school ethernet used to work. And there's not a, there's not a clear path to go, to go from a shared ether to the way ethernet works now.

There's been a few ideas thrown around about how to do this in some very complicated ways. But for now, it's all, it's all shared airwaves. So, you know, only one can talk at a time. And that does limit how quickly things can propagate. Cause every, every, all the devices sort of have to take their turn.

All right. I'm going to take my developer hat off and I'm going to put my my web, my podcast hat back on. And I'm going to ask some questions again. We had another one from the chat room. And that is. Is the system store and forward or is it real time communication only and basically when an app connects could it see historical messages or only messages sent while it's connected?

And there's really, he didn't mean to, but he's asking two very different questions here. Let's talk first about what happens when you connect an app to a radio that's been running for a while.

Ben: Yeah, there's we have a concept of a we call it the two phone queue and it's basically a prioritized queue of messages that have not been delivered yet to the phone.

So those, those queue up as you receive text messages over the mesh. But there's not really a whole lot of storage there because again, we're talking small Arduino devices with not a whole lot of, of memory. So the, you know, we always say like there could be a few messages stored on the device and then once you connect, those will get delivered to, to the phone clients.

But it, I wouldn't count on, on guaranteed delivery for those if you're not connected. And, and the other question about the store and forward versus real time it's in, in most cases it's real time, but we do have. a new store and forward Module that you can enable on the firmware and that is specifically for router And router client role node.

So that's a configuration that you have to enable for a particular node and it's also limited right now to the ESP32 devices that have a external PSRAM module. And the reason for that is, We don't have a whole lot of memory to work with, so keeping those kind of a, a ledger of those messages that have been observed over the mesh takes up quite a bit of memory.

So, so we have to, we have to use external RAM basically to, to store those. For now, we've, we've talked, and I think there's an issue on GitHub to talk about the potential of maybe storing those on the file system instead. Even like SD card, flash memory. And I think Jonathan, you, you know, the, the Linux native, right.

That should be, that should absolutely be, be something we, we have plenty of resources there. Yeah. So but those are, those are, you know, features that you can unlock, but they're not necessarily native. To, to the, to the mesh if you're setting up things on the defaults.

Jonathan: Yeah, one of the, one of the tricky things about this is, you know, there's a, there's a bunch of schemes that you could dream up and add to it, but we're so bandwidth restricted.

You want everything to be simple and basically have no no control message overhead or as little as possible. So, you know, you could, you could imagine some scheme where. You know, you store a whole bunch of messages and then each client sends back and asks for them one at a time, and that way you would know that you hadn't missed any, but it just, it overwhelms the mesh because the bandwidth on this to be able to get that kind of range.

The bandwidth is so low. I've been, I've been noodling for a while on a sort of an inspired by git sort of scheme to do this. But again, you run into issues where you just don't have the bandwidth to send all of the control messages you need to, to make sure that you don't miss anything in the middle.

And it's just, it's a complicated problem. And the store and forward module, I think is really one of the nice solutions to try to figure that out. So, let's chat for a minute, and AJ, I'm asking this sort of tongue in cheek, I think you'll understand what I'm getting at here, but I'm going to ask Ben, and then I'll ask AJ, sort of the same question, maybe the opposite side of the coin.

Because we had, we had a Rust developer on last week we had Wolverson on last week, he's one of the guys that's written books about Rust, and AJ, He really kind of made the case that Rust is a really cool language and we should all try it out, which it is definitely on my to do list. But I'm curious, what does it look like when you've got an established C or C or any other language really, but C in this case, an established project and an excited young developer comes in and says, Hey, not only should you have written this in Rust, I want to rewrite some of this in Rust.

And your C developers. Kind of go, surely Rust can't be that hard, right? And then you had the experience I did where you go and you look at some Rust code and it's like, I have no idea what this is doing. So I'm going to ask Ben first, how do you successfully navigate this when you have the young, excited Rust developer come to your project?

Ben: I think with any, with anything like this, you know, you give people an opportunity to like, hey, what can we do with, with this new technology? Can you show us some, Some examples of how we, how we might integrate it. And I'll, I'll just go ahead and say, I'm not a big C, C, C plus plus guy. I, I you know, I hadn't really touched C or C plus plus since college.

But until I got back into kind of Arduino ecosystem and, and started playing with mesh tastic. So I'm not super opinionated about. about like idiomatic C So there was, there was never any, any any presumption on my part of, about whether we should do things with one or the other. So I'm probably the wrong person to ask on that front, but I know there, there's definitely some there's some opinionated you know ideas about how things should be structured with like idiomatic C versus, versus Rust.

I personally like what I've seen of Rust with the limited amount that I've played with it. I think I've done a tiny bit of, a tiny bit of code in the, in the

AJ: You built the flasher, right? You built like a prototype flasher. Yeah. Yeah, a little

Jonathan: bit. That was cool. Yeah. I want to talk about the flasher here in a minute, but I think it's an interesting topic to chase down.

So, AJ, let's, let's ask sort of the opposite side of the coin to you. How does a young, excited Rust programmer come to a project that is full of C code and Not get yelled at and told to get off my lawn You know, how do you how do you approach that in a in a way that's that's likely to succeed

AJ: Well, so I guess it depends on what you mean by succeed and I think this is kind of the journey i've had to go through right of like I think

Jonathan: No, right.

It's like I think like you have to change your

AJ: definition of success, right? because because fundamentally like what what makes meshtastic cool isn't the language it's written in it's the community that's around it, right, right And so I think one thing that I've been thinking a lot about is you want to kind of push the virtues of Rust in this case specifically without alienating these very important members of our community.

These people are very valuable to our community. So I think, I think the client has been a good way to go because the management client, the infrastructure is written in Rust. And I think. That's been a good way to kind of get a sense of what the community's kind of taste for this language is, is because there's, if there were to be a migration in however long.

Fundamentally, there's like two steps to that. One is people would have to learn the language and two people would then have to rewrite the firmware in that new language that they've learned. And so I guess I'm trying like kind of doing this incrementally kind of feels out that first step of like, is this even a language people are interested in learning?

And if not, then I guess I don't, I don't want to be the person who breaks the community by pushing rust at all costs, frankly, right? I don't know. The community is worth more than the specific language it's written. And I guess it's kind of my current my current stance on this. But I don't know, there definitely has been a bit of the, I know Such as another one of the people who's like very pro Rust does that have you worked with him at all?

Like he's, he's the, okay. Yeah. So he's the, he originally wrote the web client and so he's, he's also been pretty pro Rust. And we've been like echo chambering each other a fair bit, but at this point, I think it's I'd be partially waiting for the embedded Rust ecosystem to mature a little bit as well.

Is one of the things is. There's a cool framework called embassy, and I think it's like 95 percent of the way to the point at which we'd be able to push like a prototype, but I don't think it's, I think we'd want to wait until it's like closer to a hundred. So I guess, yeah, the ecosystem, you don't want to push people into an ecosystem one that's like not completely ready and two, yeah, that kind of like.

You almost have to like dip people's toes into it. I think before you actually like throw the bucket over their head of like, here's the new language, here's the new framework. Here's just write this now kind of thing. Yeah.

Jonathan: Yeah, that's perfect. That that's very much sort of the flavor I was, I was thinking of.

So let's, we can talk about that more, but we are, we are getting really close to out of time actually. And I know there's at least one more new thing. And I think this is, this is one of Ben's things. The, the web flasher is fairly new and actually really pretty cool. I, I used it for the first time just the other day to test something out and it was like, Oh, okay, this is, this is really neat.

So. Give us the give us the rundown on that, Ben.

Ben: Yeah, we had a we had kind of an unofficial community web flasher that that Thomas for, for those of you on, on discord and, and GitHub, he had actually hosted that I think like in his basement for the longest time. And it was really neat because it utilized ESP homes.

I think it's called ESP web tools, which actually uses the some Chrome chromium based browsers ability to use web serial. to actually flash ESP 32 devices. And it, it wasn't a permanent solution. It worked, it worked pretty well for what it was, but we wanted to bring a, a similar solution to to the official mesh tastic projects.

And also incorporate some of our other platforms like we have Pico, the RP 2040 support, and we also have NRF 52 from Nordic boards as well, which is what Rack Wireless runs on. And those, those flash in fundamentally different ways. So we wanted to kind of provide a little bit more of an interactive approach to to flashing those boards and kind of walk through walk people through the steps Because a lot of it has been confusing and unfortunately,

Jonathan: you have to you have to hold this button while you plug it in Okay, that's probably long enough now.

Let go of the button look for a drive that showed up. Yes

Ben: And telling people to go through the, and use the device install scripts in the, in the zip the firmwares is never fun. You know, people look at you like you have, you have five eyes, but but we we, so we brought the the new web flasher on online and it uses the official expressive port of ESP.

tool to JS and it works, works pretty well. We've also got like a serial monitor that I added recently. So folks can kind of debug stuff without having to download like the Python client or a, or a serial terminal application like putty or, or TO. So yeah, it's, it does a lot of things.

Jonathan: All this catering to windows peoples.

Come on.

Well, that's that's fun though. So what else is new? Is there anything that's new that we that we didn't ask about or didn't cover? I know we got to do some set theory, set math in our brains here. Yeah,

AJ: right. I'm like running through all the stuff we've been working on. I

Ben: would say just the growth of the MQTT stuff is really, that's been blowing me away.

The project in general has gotten quite a bit more popular since the last time we, we we did the podcast. But the MQTT has absolutely grown. And I think a lot of that is due to the feature that we added called MQTT client proxying, and so the elevator pitch for that feature is it uses your phone's internet connection to to actually connect to the an MQTT broker.

And most people just use the public one that we expose at MQTT. meshtastic. org and it allows, allows that internet gateway functionality, but not through a necessarily through a direct connection. So it, it can kind of proxy the traffic between the MQTT broker. And your device via Bluetooth or serial or any other any other way to

Jonathan: connect.

Yeah, and that has made possible some a couple of really cool online services projects, with with live maps That's one of the coolest things that's happened recently, I think so there used to be There used to be a map of meshtastic nodes, but the problem was that it was user reported, and it was all manually entered.

And so there was a lot of stale data in there, and there was a lot of things that, you know, nodes that would show either nodes that would show up that weren't actually there, or a whole bunch of nodes that were there that didn't show up on the map. And so a couple of people stepped forward and started scraping that MQTT feed and putting it, putting a live map together with that information.

And those are really cool. There's, there's a couple of them I know of. And I can't remember the name of both of them, so I won't mention both of them. Won't mention any exact names, but you can, you can Google for it and find it pretty quickly. And it looks like a semi official one. Do, am I supposed to mention this AJ?

Mostly for

AJ: your reference, but there's discussions internally about This is functionality that a lot of people want and how can we How can we kind of enable that in an easy way? I

Jonathan: had not seen this yet. So there may be, there may be an official MQTT based map in the works. We'll keep that one unofficial for the time being.

That's fun though. Yeah,

Ben: there's been a lot of community ones and I think it's kind of brought to light that we might need to have some official you know, curated mapping for, for the MQTT, just because it's, it's been so popular. And, and, you know, that it obviously exposes its own set of, of issues that we've had to kind of take, take a more discerning view of like, well, do we want, do we want this public server being used to, serve locations of folks, you know, so we're having those types of discussions.

Yeah. I would say a lot more just because it's such, such a big and popular project and we want to keep, keep everything Safe for folks and

Jonathan: yeah, I I I think everybody shouldn't everybody should know this If you have your positions broadcasted and you connect to the MQTT server You are sending out your positions over the internet And people can find them if they want to Keep that in mind.

Yes, keep that in mind. That's actually one of the, one of the main reasons why I did the position precision feature. And, and that's the idea. That's something relatively new in the firmware where you can set a value and essentially say, Hey, I want to shave this many bits. bits off the end of my latitude and longitude.

And so, you know, you can essentially, you set this and then it, it blurs your position essentially. And it, what it does is it intentionally sends out an incorrect position, but it also specifies, I am, you know, this is the size of the circle. And the position that I am reporting is right in the middle of this circle, but my actual position is just somewhere within it.

And so you can set that to be as precise as a couple of doors away, or you can set it to be as imprecise as this is probably the city that I'm in. That's new. And I think that, you know, once, once everybody knows about it, I think that's going to be kind of a big deal for these maps and things like that.

And

Ben: the iOS app has some really beautiful visualizations for that. You get some nice, Giant circles around your around your location.

Jonathan: Yeah, he had to talk me out of the way that I wanted to do it at first He had to talk me out of like that's a bad idea. That's a bad idea. That's okay. Fine. It's a bad idea We'll do it some other way

AJ: Love to hear what the idea was at some point.

I'm curious. I didn't follow this

Jonathan: it was a instead of a Value assigned to the channel. It was a set of eight values that was just assigned into the rest of the settings. And then as you, if channels moved around, whatever client moved, the channel would also have to move those values around. I was like, it should be easy.

Right. And he was like, This is not a thing that's going

AJ: to be easy. Well that means the client developers have to remember how to do it,

Jonathan: right? Exactly, exactly. He was like, this is going to get out of sync. It's just, it's going to be a disaster. Okay, fine, we'll come up with a different way. And actually, the way we came up with to do it, I really like, because it gives us now a place to put Per channel settings that don't, don't belong in the QR code.

We're, we're getting into inside baseball here. This is, this is where Rob was supposed to jump in and say, I have no idea what you guys are talking about. This is meaningless.

Ben: It's easy to do on this project. It's so

Jonathan: deep. I know why Rob wasn't jumping in. He's muted. Unmute yourself, Rob. I

Rob: was listening.

I was listening very trying

Jonathan: to figure out what are they talking about? Oh, All right, well, hey guys it is it is 12 30 central time at least we have filled an hour with this. It has been a lot of fun Thank you each for being here. Is there anything that you want to plug that we didn't mention? Ben

Ben: now just visit us at mashedastic.

org join the official discord Yeah, it's it's it's a huge community and there's a lot of helpful people there. Yes

Jonathan: The disc the discord is where things are happening. That is for sure. Yes Aj anything you want to get in right before we let you guys go? Discord

AJ: and come check out the management client.

Come tell me where it's broken

Jonathan: I think I think there's probably going to be a release of that like a Either a three run or a dot four release before too long and hopefully hopefully fewer things will be broken there

AJ: That's the dream. Always the dream.

Jonathan: All right. Thank you guys. Thank you so much for being here.

Thank you so much. All right, Rob, have we convinced, have we convinced you? Are you going to go order some hardware?

Rob: I mean, this talk really has got me excited about the stuff. I just haven't quite figured out what my personal practical use case would be, but I mean, that's never stopped me before on spending money.

Jonathan: Yes, yes, absolutely. Oh, you've talked me into spending money on a few things. I've got a pine watch somewhere around here. That seems like the coolest thing.

Rob: Yeah, me too. And I'd never touched mine after I took it out of

Jonathan: the box. Yup. So, there, actually, there is a there is a Meshtastic device, it's from Liligo, a couple of them, that are in the form of wristwatches.

So, I've got, I've got one here that I tried for the longest time to add Meshtastic support to, and it just doesn't have enough RAM to do it, not quite. Although, I might revisit that at some point. But, you know, one of the, one of the things that I find really interesting about Meshtastic is if you've got a GPS in your device, then you can set it to broadcast your position.

Well, you can, you could do that on a, you know, a private channel. And so then you can have like a home base where you can keep track of those devices. And so I've got one in my van. And so when somebody goes on a trip in the van, I can pull it up and say, Oh, okay. They're. at the post office. Okay. Now they're at the library and now they're on the way home.

And I find that really useful. And I also love that I can do it without having to send my data up to Google or to, to Apple or whoever. And then the other thing that I keep in mind is one of these days, cause I live in Oklahoma, you know, tornado alley, one of these days, the big one tornado wise is going to hit my town and It is not uncommon for when that happens that communications just totally go down.

And so the other thought that I have in the back of my mind is it would be nice to have this set up to where when that happens and the power is off and the phone lines are down and the internet is out, I can still get a hold of people and do some coordination. You know, who needs help, who needs a pickup, that sort of thing.

And so those are sort of my dual use cases that I keep in mind for Meshtastic. So when you get it

Rob: working on that watch, I could get my kids some new watches. And then when the power and everything goes out, I can find them wherever they're

Jonathan: at. Yeah. Yeah. And that is, that is exactly the one of the other things that I keep in mind.

Lily Go has been teasing. A, a watch, the, the more powerful version of this with a GPS built into it. And I think when that actually comes out, it's going to be a really interesting project, but it's, it's not quite done yet. It's not fully baked. I think they're having trouble squeezing the size of battery they want into it, as well as the full GPS that they want into it.

So fun times. We'll get there eventually. All right, Rob, thank you for being here, man. Anything you want to plug?

Ben: You

Rob: know, come check me out on the Untitled Linux Show with Jonathan. And also I always plug my website, robertpcampbell. com and then you can find links to connect with me. Yeah,

Jonathan: I love it. I love when folks plug their website, because if we ever get mad at Rob and kick him off the show, you'll still be able to find him.

Alright, next week we have, scheduled at least, Simon Kelly, who is one of the developers behind DNSmasq. DNSmasq is one of those projects that you, I am sure, use. Because it probably runs in your router, but you probably, a decent chance you've never heard of it. It's, it's one of those projects that makes the internet work, and it's going to be super fascinating to talk to Simon about it.

And maybe find out from him how many of these new cyber security laws are going to make his life difficult as an open source developer of this thing that is everywhere. So that is next week, and you don't want to miss it. Thank you to everyone that was live in the chat room, took some questions from there.

And thank you to everyone listening on the download. We sure appreciate it. Make sure to share the show. Let your folks know, let your friends know about it and we will see you next week on Floss Weekly.

Episode 774 transcript

13 mars 2024 | min

FLOSS-774

Jonathan: This is Floss Weekly, episode 774, recorded Wednesday, March 13th. Let's get Rusty.

Hey, this week we sit down with Herbert Wolverson and talk about Rust. He's sort of like one of the Randall Schwartzes of that language. He's written, he teaches, and he sat down with us for an hour and let us pick his brain about Rust. You don't want to miss it, so stay tuned.

Well, hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we have a bit of a different show. It's, it's just me. There's not a co host, and that is because there's this little thing going on this week called Scale.

Scale 21. All of the people that you normally see in the co host slot are doing something with scale, either on the way there, or they're at it, or something maybe unrelated, but everybody for the co host slot was tied up for today, and that means I get to do it, well, not by myself, I have a guest. In fact, I've got, I've got a really neat guest Herbert Wolverson is going to be the guest today.

And I told him in the, you know, in the, in the chat before the show that I was going to refer to him as the Randall Schwartz of the Rust World. For those of you that know, Randall, a wonderful friend of the show is, is known for being the guy that kind of wrote the books on Pearl, and that is one of the things that Herbert has done.

He has written some of the books on Pearl, and he said, well, I'm not sure that I'm quite the guy for that. And so we decided he is One of the Randall Schwartz of the Rust world, and that, that's fine, that, that works but written some books on Rust and doing some teaching on it and something of a Rust evangelist, which I am fairly excited about because Rust, Rust is one of those things that I've been excited about for a long time.

And sort of had it on my to do list for the longest time that I need to actually write some Rust code. And I've sort of always had this feeling, until recently, had this feeling that it's just another programming language. Surely it'll be easy. I, I know C, I know C I do PHP, I've done a little bit of Perl, I do Python, all of these languages.

And so it's, To the point to where most of the time I could just pick up a new language and spend a couple of minutes googling for you know exactly what the syntax is supposed to look like and be on my feet in a way. About a, about a month ago I had a conversation with someone and it was said, well it could be a problem that we were adding Rust to the Linux kernel because you have all these kernel developers that know C and don't know how to read Rust.

And the thing that I immediately thought was, it's just another language, surely you can't, it can't be that hard. And I decided before I would put my foot in my mouth that I really ought to go and check and take a look at some Rust code and make sure that it was as easy to read as I thought. And I pull up the the, the, the sysutils, kind of those, those basic Linux system utils that historically have been GNU projects.

And it's, it's the Rust implementation of them. And I go to look at the Rust code. And the first thing I see is if let sum, like, why is there a let statement inside an if statement and what does sum mean? And I just kind of, okay, it's going to take more time than I thought to learn Rust. I don't have that time right now.

All of that said, let's bring onto the show, the guy that can help. We've got, like I said, it's Herbert Wolverson, and we've got him today and excited to chat about Rust and maybe get some tips as a first time. Herbert, welcome,

Herbert: sir. Ah, thank you for having me.

Jonathan: So let's start with what you're up to. Let's, so what is, what is sort of, how do you fit into this, this brave new world of Rust?

What, what are you up to? And like I said, sort of the one of the Randall Schwartz, unpack that for us. What, what are you, what are you doing?

Herbert: Okay. So right now I have like five jobs because I'm an independent contractor, I'm the Rust series expert for Pragmatic Publishers. The people who.

Published the pragmatic programmer years ago made enough money doing that they became their own publishing house so that means I read other people's manuscripts and if anyone wants to send me one I'd love to and Say whether say yay or nay on publishing it or hey, this is great. You should maybe change this or things like that and That's, you know, that's the job I love.

If I could get paid to sit and read other people's manuscripts all day, I would be the happiest man on earth. My second job is, you know, 15 years ago, I co founded a wireless internet service provider in central Missouri. I'm less connected to the day to day running of the thing now. But we have a remarkably Rust based backend for managing everything.

The third job is I write books. Now it's a weird conflict of interest because technically speaking, I approve my own manuscripts. So we made sure somebody else got to do that. But my third book will actually be coming out into beta in the next couple of months, so advanced hands on rest.

So I'm hard at work on that. And that's another job I really love. My, that's the problem with all these jobs is I love all of them and So I'm going to keep doing them until I keel over. Yes, I understand that. Now, my books did well enough that a fellow named Naguel contacted me out of the blue and said, you know, Hey, would you like to teach Rust?

And so I signed up with Arden Labs who are mostly a Go shop. And they had been getting more and more companies asking them about using Rust as part of their ecosystem. And so they brought me on board to start teaching Rust and started out with a few webinars, turned into full week long classes where I teach you remotely or personally.

via recorded video from Hello World all the way up to building big multi threaded network servers. And you can get there in a week if you put every day into it. It's as long as you've already got to the point that you know, you know what a for loop is, know, know what an if statement does. So that's turned into this crazy job where I, I, I'm at REST Nations UK.

In two weeks time I was at RustConf. I was at GoConf of all things. They had me go to GoConference and give a class on Rust, which was weird, but also kind of exciting. They fly me around to teach people. I was in Silicon Valley a while back teaching a C company that's interested in Rust. So that keeps me Really busy and I also love most of that job.

So Keeping that one too. And finally, there's Libre QoS, which is the real passion project I know you had Dave Todd on here a while back. Dave is the chief science officer of Libre QoS I am the I think I'm technically the chief product officer because I wanted 3PO to be my to be my job title. In practice, I crank out Rasp that, and also some C that handles the task of reading you know, 40, 50, 000 packets a second in kernel space, parsing them, sending them off to the right queues, and then feeding them into this just enormous Rust infrastructure that gives you.

Vision visibility into what your network is doing. And labor, when we started a couple of years ago, labor QOS was the biggest rest project I'd ever tackled. And I'm kind of proud of it because we've not had one customer yet. Tell us that the core system has crashed and

Jonathan: that's that's saying something

Herbert: that's, you know, that I'm going to say that's more the language than it is me because, you know, I'm a, I'm, I'm a decent programmer.

I'm, but I'm certainly not. I'm certainly not amazing, you know, that's That

Jonathan: that sort of strikes me as being the the biggest difference between The Rust language and let's say C or C and I'll put some caveats on this in a moment, but with C the foot guns abound And with Rust, you don't have to be a wizard level programmer to not crash your program because you did something silly with memory management.

Herbert: Yeah, that's, that's about right. And, you know, the history of Rust is, um, Grayson Hoare and his team And a team at Mozilla had been deep, deep, deep in the C world, making Firefox work and other Mozilla products. And so they sort of had a laundry list of what is making my life suck. And it's also why the Rust syntax is a little strange.

They loved OCaml, so they wanted to borrow as much of OCaml and the bits of C that they liked all at once. Mix them together in a pot and out comes the syntax. Yes. So, you know, the core idea of Rust was that you never, ever, ever want to have a buffer ever run. It's where you read past the end or even the beginning of an array, and suddenly you're looking at something you shouldn't.

And it's far better to crash than to crash with an error message telling you what you did than it is to allow that. And then they Sort of went pretty crazy and just started listing off all the things that made their lives suck like race conditions So we'll stop those use after free Well, we won't do that.

And they also looked at other languages that were around at the time, you know, go was appearing java was very popular The whole dotnet infrastructure, but all of those did did do garbage collection And the problem with that is when you're trying to do something Like a browser, you don't want the browser to pause periodically to clean up your memory.

Yes. So you've got to do memory management and C and C are wonderful at that, but they give you all sorts of ways to accidentally shoot yourself. So Rust was basically a collection of let's try and have the compiler fix as much of this as we can and then have Runtime safety that stops us from, you know, accidentally sharing the corporate password database with the world.

Jonathan: So one of the things that made the news recently sort of in my orbit is the U S the executive branch, the white house put out a document, basically encouraging businesses and people to start using memory safe languages for writing important code. And rust was one of the ones I believe got a specific call out.

And there, there's, there's been sort of a, well, with anything that touches politics, as one might imagine, there is a multitude of responses to this. One, one, in fact, it was the fellow Hackaday writer I mentioned earlier made the point, well, hasn't, hasn't Ada had all of this for the longest time? And I think that's kind of a, a more interesting question.

Like Rust is not necessarily the first language to try to do some of these things, is it?

Herbert: No, it's not. Ada, particularly with the Spark extensions, gives you kind of a limited language in some ways, but it's deliberately limited in that, and you check preconditions everywhere. And a lot of bugs become compile time bugs, and that's what you want, but the DoD even adopted issued a mandate years ago that everything was going to be written in Ada, and it didn't, didn't go well because Ada Ada's a, An interesting language.

It's a great language in many ways. At the same time, it's really verbose and you get it, you tend to wind up with these incredible you know, 30 level deep nested statements that look, just look like a nice little diagonal line down your screen. And there wasn't a lot of Ada talent out there to use it.

And so this time around, you know, they listed a whole bunch of languages. They listed Go, they listed Java. Rust was the exception there that it didn't. That it doesn't give you a runtime that manages your memory. And you know, I thought, Hey, that that's encouraging. And I saw that rust recent through a company called oxide recently met the vehicle safety standards as well.

I just blanked on the name of them. I think there's going to be the same sort of headache that Ada had that now you've got to find a whole bunch of rust developers and it's a newish language. So there's not many of them out there. At the same time the thing that really made me happy when I So that announcement was going to reddit slash r slash CPP and looking at the reaction there You've got the and I'm not saying this in a schadenfreude kind of way because you've you've got the old school C coders Which was me five years ago Saying well, we don't have a problem and you know, they're absolutely right that C If you opt in, it gives you a huge amount of safety, like never use the index operator on a vector, always use at, always check the result.

You just got rid of a lot of the problems. And the problem is you have to remember to opt in to all of these. But then, you know Strassrup and Herb Sutter from Microsoft popped into the thread. And we're saying, well, you know, what we really need is C with a profile where we switch all the defaul all the defaults to the exact opposite, and now you've got 90 percent of the safety of Rust, and it's like, yes!

I've been asking for that for 20 years now! Yeah. Yeah, it's it's one thing to give people the option of wearing seatbelts. I prefer you wear a seatbelt by default and can take it off for the scary part where you have to do something dangerous. And that's one of the things I love about BuzzGo and Rust, is the the unsafe tag.

You're not saying I shouldn't do this, You're definitely not shouting, hold my beer, you're labeling. I've done something here that operates at a lower level than the static analyzer can verify. So please accept this as a big warning label that if this program crashes, look here.

Jonathan: Yes, and that that unsafe tag, that's particularly useful for doing things like writing drivers, right?

Is, and I guess I'm curious, is there another scenario that you have to use unsafe outside of we're working with real hardware, therefore we need to talk to this memory address that is uninitialized because it's not actually a memory address, there's a device

Herbert: there. Ed? That's a complicated one, but so unsafe means that for whatever reason I am opting out of most of the Rust checks.

There are still some that you can't avoid, like you can't, you can't have aliases where two references point to the same thing. Item in memory. The Rust compiler just doesn't let you do that. It's the same as strict, enabling strict aliasing on your C compiler. So if you're talking to hardware, you absolutely have to have unsafe at some level.

Because quite literally you're saying this blob, this blob of memory isn't even mine. If you're interacting through FFI with say a C library, your actual boundary calls are going to wind up being marked as unsafe because Rust Isn't going to make any promises that it's going to step into that C library and verify that it's correct.

Then you run into the unsafe that I discourage which I sometimes when I'm teaching call the YOLO unsafe. Because just like C has, you know, you can access a vector entry by index, or you can use dot at to see if it's there. Rust actually has get unchecked. Which is the same as the index operator on C Right down to it doesn't do a bounce check.

And if you call that with the wrong number, you will read out of your vector. And you will, you know, best case, crash your program. Worst case, do something you really didn't want to do. Now in Rust, that's unsafe. But, if you're writing something that's super performance critical, maybe you don't want to do a million bounce checks because you're blasting data into a vector.

from some hardware source. So that unsafe tag there is you saying that, you know, yes, this is on me, I've done the checks. And another programmer who comes along may look at it suspiciously. And if you the reason I discourage it is that please, you know, start with a profiler and find out if that, that is actually why your code is slow because the compiler LLVM is absolutely amazing at realizing I've got a million bounds checks here but I'm going to have one upper bound.

So let's check that first. It doesn't always pull it off, but compilers are way smarter than me. Yeah, to give you a real world example, some code I was working on last night. I've got a, I'm reading buffered data coming in from an eBPF driver running in kernel. And I've got a hard time limit on how, how long it can take to read this.

And I was missing my time limit, and I was all frustrated, and I'm like, Okay, well, here comes the unsafe. So I unsafe, start you know, get, accessing the data directly, and You know, I didn't blow anything up, because I did, actually did the math. But the speed improved by 0. 01%. And then I noticed that I had a mutex lock in the wrong place and hoisted the mutex lock out of the loop and instead of calling 18, 000 mutex locks per second, I'm now calling one and all of a sudden I'm at point back to point one seconds and everything's happy.

He's happy. And so it's like, you know, run the profiler. It's it it's gonna tell you it usually tells you something that you missed

Jonathan: Yeah, so i'm i'm trying to remember the exact quote, but somebody somebody said something like the root of all evil is Trying to optimize optimization before profiling or something like that premature optimization.

Herbert: Yes. Yes. I think that I think that's attributed to Donald Nutt But I'm not not a hundred percent

sure

Jonathan: there that is that is possible It was it was one of the one of the old timers one of the one of the greats So, let's see. Let's maybe let's talk about rust in the kernel because that's something that's definitely going on My problem with this is I've got so many different things that are rust adjacent, but I want to ask you about that So we may just kind of bounce around from topic to topic for a while and then slow down.

But let's do that. Let's talk about RESTing the kernel. This is obviously something you're watching, I'm sure.

Herbert: Oh, yes. And I'm actually happy to report that my first kernel module that prints hello world to a log works. It's not a great kernel module, but that's as far as I, that's as far as I got with, hey, how do I do this?

I was Pretty excited to see it though. Partly because the Linux kernel has been one language for so long. For years and years, I was following the efforts to get C into the Linux kernel. And I kind of agreed with last week that the problem with C is that it gives the, it tends to allocate behind your back.

It's so easy to use types that allocate memory and don't, and sometimes have side effects that you weren't quite expecting. And I honestly think that could be fixed, but it didn't seem like the C standards committee was agile enough to make that happen. And so quite a few years ago long before Rust was officially in the kernel, I ran into a few people who were.

Hacking away at rust until they could get a kernel module to load. Anyway, and they were they could do that because your rust outputs see ABI code that is effectively indistinguishable from C from the compiled binary point of view and so they hacked at it until it worked and finally started talking actually to the People who maintain Linux and eventually word got up to Linus and his comment was Hey, go for it.

Give it a try come back to me in a few months and We'll see what happened with your first try and everybody was like, wow, he didn't just say no and set the room on fire So there was a lot of effort for that and it's been a really positive experience Bidirectional process too with once you get into the Linux kernel, a lot, some of the Rust safety assumptions don't necessarily apply because you're dealing with a whole lot of things that happen in kernel land that you don't necessarily want to do in user land, especially with modern hardware being so incredibly complicated.

And so part of that was when, when they ran into some Rust does something that shouldn't be done in the kernel. The Rust. Core team have been really responsive in terms of changing Rust to meet, to match Linux. You know, a good example of that is that you've always been able to compile without the standard library.

You can now compile with no, with no memory allocator at all if you need to, which is what you want in a driver because you have to use the kernel allocator. And you can substitute. The memory allocator. So building, building those, just those bindings. And I think there was some compiler changes required that took months, but they did the work and now they've finally got to the point that I think it's 6.

8 is shipping with network driver written in Rust. And there's the full sort, they actually took one of the existing ones, ported it, it's the first time Linux has had duplicate drivers for something in there. It's one of the simple ones, and it's there partly as a proof that it can work, and then there's the crazy people over at I think it's Asahai Linux, I'm not sure if I just butchered the pronunciation.

I think they were pronounced that Asahi. That would make sense. I can say I've talked to them, but only ever by text. So they took the Mac, Mac and one chip output. And yeah, I, I really wish I had this much energy and enthusiasm, but they managed to feed that into. Tons and tons of Rust and get to the point that it boots.

You have X11 working OpenGL sound. I think they were having troubles with the webcam last I checked in. And they were making a lot of comments that they could have done this in C, but the Rust type system Stopped themselves, them from tying themselves in knots, and you know, I saw, I was snooping on a lot of the chat that went into how that was being done, and had a look at some of the code they've written, and some of it honestly looks like C.

The lower level in Rust you get, the more C like it starts to look. You start seeing pointers everywhere, pointer math everywhere, and the syntax is similar. Not quite the same. And you do always have to differentiate between whether a pointer is mutable or not. So is it read only or can you write to it, which honestly, I, I'm a big fan of the const tag in C in C anyway, to remind me that I shouldn't be changing this but it's, it's a remarkable.

Effort and it's turned into this big Rust on you know, Rust for Linux library that comes complete with instructions for how to bootstrap your own kernel module, compile it, it's becoming something that's pretty approachable. And I think that's can only be good long term because there's a growing pool of Rust programmers.

The People I know, the companies I've spoken to who hire C and C programmers, there's plenty of them around, but there's not very many of the really high level, good ones. And so I think allowing the wider pool is just going to be good for everyone. At the same time, I, I fully expect at some point.

One of the joking discussions you see on the kernel mailing lists where well, we just can't get this function to work and somebody says, well, we could write it in Rust and lots of people groan, and then they fix the problem. One of these days, somebody is going to rewrite something core and Rust, and then the real fight begins because right now Rust is in the place where you have to be able to build the kernel without it.

It's, it's at the edges. So it's at the edges? Yes. And if it ever makes it further into the core, then that's going to be a headache for the People who have been doing C for 30 years. But hopefully not because hopefully we can keep the boundaries well defined. And so at least you can interact with the Rust even if you're not changing it.

But we'll just, I think it's going to be an ongoing evolution where we have to just see where it goes.

Jonathan: Yeah, there's interesting things going on in the kernel right now. So as you were talking about that and you mentioned C I remembered back a couple of months ago now, there was a there was a a message sent in to the kernel, basically saying, let's do it in C Let's convert the kernel to C And the thing that was wild about it is C has come along to the point to where some of those problems have been either fixed or there's ways to work around them and the person that wrote this that wrote this message did not event did not immediately get set on fire so Personally, I think it might be kind of nice to be able to get some of kernel.

I don't know if it'll ever happen. But it is it's an interesting kind of new world that we're going into when the kernel is not just C code.

Herbert: Yeah, it's fascinating when you look in the kernel. Like to me the big C invention that everyone should Lord, forever is RAII. Resource acquisition is initialization.

Mm-Hmm. , which is a long-winded way of saying that you have a con you know that you have a destruct. So when this structure falls outta scope, something happens. It's not just gone. Mm-Hmm. . And that is amazing because then you can tie that distractor to cleaning up. Mm-Hmm. . And so if your bit of code is talking to a USB microphone and that USB microphone's disconnected, if the driver goes out of scope because it unloaded now it has a chance to run some cleanup call rust adopted that wholeheartedly. Much of you know, much of the easy memory management that you do in rust because honestly you can write thousands of lines of user user mode rust without ever touching ever touching an allocator Uses that wholesale It's right down to the, it's the same compiler mechanisms.

But if you dive into the Linux kernel, you'll find that there's a whole bunch of macros in there that basically added that into C to ensure that when something goes out of scope, it is destroyed because a lot of the time, you know, ownership is straightforward. I made this variable. It's my, my responsibility to clean it up.

And you'll see the lovely go to patent and see where something went wrong. So I'm going to jump to the end and clean up. And that works. RAII gives you the same effect because you free it up, the cleanup code runs, it's still deterministic. The problem comes when ownership starts to get fuzzy, like this This piece of data is now going to be used by five other things and it's unclear which of those five other things is going to finish first, finish last.

And I want to clean it up when none of them are using it anymore, but I don't want to clean it up before that because that's going to really mess up the day of one of those, of whichever ones are still going. And so tying, tying lifetime events like that at a language structure, you see C and Rust are both great at it.

I think Swift uses it. Garbage collected languages struggle with that because when the garbage collector will get around to deleting it is kind of indeterminate. But when you can do it, it's great. And, you know, it's lovely to see that there are structures in Linux for doing just that. And they've got this, you know, they've had the same problems we've had.

And they've converged on a kernel side approach that is basically the same thing. So I don't see there's anywhere near as much of a mismatch as there used to be. You just have to be really careful, like. You'll get some wonderful email if you start allocating in current regularly in kernel mode.

Jonathan: Yes, yes, that would not go over well. So one of the interesting things to me as well, Rust is, ah, there's several things here. So Rust uses what, generally speaking, the, the LLVM compiler.

Herbert: LLVM, although there is a GCC port that works. So that's

Jonathan: what I was going to ask about. What's the process or the progress on the GCC port?

And why do we need both?

Herbert: So, the progress is it just got published. Pulled into rust app, which is the tool that installs linux as an available backend the vast majority of stuff now works. There are still corner cases being ironed out but it wasn't too bad because they took the approach of when you compile rust rust c emits an intermediary That then gets read by llvm and so this one emits and then the intermediary that gcc likes and so You You just had to make sure that everything that was needed was there and I think gcc actually added a couple of facilities Also helped c because it uses the same thing.

So why do you need it ld? Llvm targets a lot of platforms, but not all of them whereas gcc Seriously targets everything. I mean, you can probably get GCC for your toaster. And while rust Is never going to work on 8 bit platforms according i'm told by the kernel team by sorry by the core team it'll work on 16 bit platforms you can Produce tiny binaries just like you can with C.

So it's a win there because once you've got GCC there's a well defined way to import your existing libc for a platform into rust and use that if you need to. And so most platforms out there ship with most of a libc and a GCC compiler. So now you've got the ability to bootstrap a whole bunch more platforms.

And,

Jonathan: and rust is low level enough that you can actually use it for embedded development, like on a little tiny device where the rust binary is. It is essentially the entire operating system, right?

Herbert: That's right. You can, there's operating systems written in Rust from the ground up, which on the embedded side you know, you, you enable what's called no STD to no standard library mode.

So all of a sudden you're missing some of the nice, the nice things. Okay. But you've got a very C like. Language with the Rust constructs that make people happy. Like some types, some types really strong typing and so on on top of that. And some, you know, some platforms also implement parts of the core.

So, you know, on some embedded devices, you can't, you don't even get to allocate on the heap because there isn't one on some embedded devices as a heap. So you can start using heap allocation, but Rust can go all the way down to I think this I, the smallest I've seen in person, if you remember those little pie badges that Adafruit used to sell, they're a tiny little liquid crystal display with a minuscule CPU on the back for an Arduino.

I've seen one of those in TinyGo. I've seen one of those in Rust. They all work. Yeah, it's cool. It's pretty cool. I personally have been going big more than small lately, but I'm, it's on my bucket list of things to try and fit in before all the jobs catch up with me. Yup.

Jonathan: Yup. So let's see, while we're on that topic let me ask you about, well, about cargo.

So this is, I guess this is a little bit different topic, but one of the, one of the things about Rust that sort of drives me crazy, and maybe I just don't understand the way it works is it, it reminds me of. It reminds me of JavaScript in that there are, there are all of these little tiny either JavaScript or maybe Java itself, all of these little tiny libraries that you grab at build time.

And, you know, coming from the C world, I'm, I'm much more used to, you've got, A handful of system libraries and it's it's very different and I'm not sure I'm not sure yet what I think about that and and one of the things I'm curious about is does rust have the problem that Java does that if one of those libraries has like a security vulnerability?

You can't update just the library on the system. You've got to rebuild the whole, the whole program.

Herbert: Okay. So first of all, I just want to say that I love cargo program because coming from C where I need I need either a make file or a C make to make a make file, right? I need a separate library for unit testing, another library of benchmarking.

All of that's in Cargo and then it's also got, you know, if you, if you like dependencies in C you need something like VC package, Conan, or all of those. That's built into, and Cargo is extensible. And so it's kind of got to the point that it's the Swiss army knife that does everything, it builds your program, it runs unit tests, it manages dependencies.

So to your question of lots of libraries. It is a difficult, it is a difficult one and when I first started using rust coming from c plus plus, I honestly looked at the build list and I was like, WTF how can I possibly trust that, you know, all of this other stuff. Is in some way good. Yeah, and then you I look at some of the games I like to write because I love writing games in c and I'm relying on glfw and everything else and You go in and look and they're actually depending on a lot of things too.

Yeah, just the c Approach often involves a directory where you've bundled some bundled the things you're depending on in so you at least know you've Got a set version like, you know you're using Zlib. So somebody's probably going to compile Zlib into your system at some point, or dynamically link to it.

Yeah. I was also a little shocked how big my executable was, but then I realized it was all statically linked by default. And you can, you can change that if you want to. So I went. Around as talking to people who do REST and working with it. And what I learned was that there's really a handful of crates that everybody depends upon.

And those are things like SERTI serialization and deserialization. I b I was CERD for years, and one of the authors came up to me at RESTconf and told me never to say that again, so now it's CERD y forever and other libraries like that, and also a lot of them make it, make their way into the standard library eventually.

But only once they're battle tested and in use. And so you can, so, you know, biggest thing I'm going to say against crates IO here is, you know, please, please, please give me some namespacing. Cause right now you can only have one crate. I named Herbert, you know, for example, so if somebody else wants to write Herbert, then that name is taken.

And so you end up with this big blob of not very searchable. You wind up searching by description. I'm told that's being fixed. I don't know a timeline for that. But in terms of figuring out what libraries to use I personally recommend going to. The rest playground it always has the 10 most used available and make sure you start with the ones that are popular and in use.

Then it make sure you install a program called cargo audit, which will download the current CVE database and check every one of your dependencies and transitive dependencies for current CVEs and can be built into your compile chain. So you can't, your CVE that gives you a lot of peace at night.

I usually put that in my code. You know, continuous integration pipeline. So I get nasty grams. And you can do, you can at any time type cargo update and all of your crates will jump up to the most recent version that you've allowed within, within your version number. You can also pin to specific versions.

And for shipping code, there's a built in command called vendor that actually downloads the source code for every single dependency you have. And then builds locally from those, never goes out to fetch them again. And so that's what you want to do if you're shipping something and you always want to have a repeatable build, because that way you've, you've guaranteed that it's, that you're not going to get the left pad problem of somebody yanking something and suddenly nothing works anymore because you've got your copy.

You're working on it. And part of it relies on Goodwill. The NPM problem is that. They didn't really think of the security problem until they had a million crate, a million packages and but there's also a philosophical thing. You know, I personally do not recommend getting down to the granularity where you download a crate for left pad.

Yeah, especially when it's built in try to, you know, just personally, I encourage people try to download. A crate that is big enough to justify being a crate. If it's three lines of code, just write three lines of code. Or go and cut and paste it if you have to If it's something like a massively optimized ace a star graft reversal Then yes, by all means go and use the one that's really good so I hope that kind of answers the question.

It is it is a culture shock. Yeah

Jonathan: It had have there been any problems with like typo squatting or dependency confusion?

Herbert: I haven't run into typo squatting. I have run into You I have no idea how it happened, but the maker of the mmap crate decided to make another one and called it mmap2. And the syntax is the same, but the number of times I've done cargo add mmap and I meant mmap2 is So it, if anything they've gone too far Down the it's hard to remove the old ones.

Jonathan: Hmm makes sense so I did when I when I threw the the fact that I was having this interview when I threw it out To kind of the people that I I interact with one of the questions that I got is what do you think about rust? This is from aj mcculkin. What do you think about rust as a high level language?

And aj says that his view is rust is particularly good when replacing c But he's not seeing consensus around whether it should try to replace languages like p Java and JS. I'll let Python into that.

Herbert: So you can actually write a fully functional web server that returns a JSON hello world in 21 lines of code with Rust.

So it's a very capable high level language. The question more become, isn't so much, should I, as or isn't, sorry, isn't so much, could I, as should I, because if you're, if you're So if you're writing you know, a blog and you just need to return JSON or whatever to the front end for your blog posts, you can do that in any language.

So pick the one you like. For structuring Rust programs, you know, there's usually going to be some nasty low level code somewhere that kind of justifies you using Rust, but you're very much encouraged to wrap that in a nice, safe, ergonomic interface so that the person who sits down to write a program doesn't actually have to worry about.

All of the really low level stuff I mean, one of the classes I just recorded was 18 hours of how to use how to use Rust to write REST, gRPC services, WebSocket servers, and similar, and it is really ergonomic. It's really nice on the other hand, you know, a lot of that, you know, If you don't have the low level latency requirements, you might as well use Go or Java or whatever currently fits your company.

So it's a matter of it's a matter of deciding I need Rust because I either want a really tiny binary. Like I did a webinar a few months back where I built up a little bookstore web service, Stuck it into Docker pound, pounded it with 5, 000 requests a second and watched it. It's still using 22 megs of RAM.

So sometimes you need that and Rust is great for that. And sometimes you really don't need that. Use, you know, maybe you want to use something that's more, I don't know, more comfortable to, to the domain problem. I'm a big believer in use the tool that fits, fits your problem.

Jonathan: Is there, is there a is there a mechanism to use Rust as like a scripting language?

Can you, can you write Rust scripts and then just have, you know, your your, your, your, your shebang at the top that says, hey, this is Rust, by the way? Thank you.

Herbert: Not a good one. There are plenty of toys that do it, but I've yet to see one that I would trust for anything important. Rust is very much compiled language.

And so at least one of them I saw quietly ran the compiler in the background and then ran the binary. And that's a little pointless.

Jonathan: Yeah, one of the, one of the things that I think it's the Dart language that, that Dart does that I thought was really interesting is they have a full fledged compiler.

And a full fledged, let's run it live as a scripting language. And the people that do Dart kind of had this guarantee that these two versions of the language are going to march on together and they're going to work exactly the same way. That was a little mind blowing when I, when I wrapped my head around that.

Herbert: And that is really cool. I think Rust is badly suited to that because so much of Rust's benefit comes from the static analysis that happens during compile. Yes. So the scripting versions just going to wind up being. Really slow because the it's gonna have to look at the whole thing to figure out how you start yourself in the

Jonathan: foot Yeah, yeah, and and the reason you know, the reason Dart did that is because they want well They're it's it's sort of built for doing mobile application Not just mobile applications But for user interface sort of applications and they want people to be able to write code real time and immediately see the update On on you know, they're they're gooey So it makes sense for that project Speaking of foot guns Something that I've kind of noticed over the years is a lot of languages have a, I don't know if I would call it a weakness, but A typical problem?

So, for example, when we talk about C, you get lots and lots of memory management errors, buffer overflows, that sort of thing. When you talk about Java, well, you're not getting memory management problems, but when it comes to security, one of the things you see often in Java are deserialization bugs.

Because it's just kind of endemic in the way java works, that unless you're extremely careful when you go to deserialize an object, it's easy to overwrite one of those important functions and suddenly arbitrary code. Does, does Rust have one of those endemic weaknesses somewhere? Or are we still waiting to find out if it does?

Herbert: Sort of. The first one is that the learning curve of Rust is much more vertical compared to most other languages. And so everybody who first comes to Rust, especially if they come from C, write C like Rust and then spend two weeks complaining about the borrow checker. Because a lot of, a lot of, Stuff that works in other languages, but is potentially not the safest is just going to refuse to compile.

But that digs all the way down that there are some, there are a tiny subset of programs that you can write and see that Rust wouldn't compile without dropping in some inline assembly, at which point all bets are off. The so okay, just occasionally you run into that and you can. Sorry. Big thumbs up, just appeared on my screen.

I think I, I think that was my Mac noticing my hand movement. . The, uh, so just occasionally you do run into that and, Mm-Hmm. you, because Rust puts so much emphasis on ownership, the idea that this is my pointer, it's my problem. It will live a certain amount of time, at some point you can kill it, and I will be the one to kill it.

Or I put it in a smart pointer and it will, you know, cease to exist at some time in the future. What you've, what you run into is newer Rust programmers realizing, Oh my goodness, there's this thing called arc that is the same as shared pointer in C it is a reference count. Stored atomically, which, what the R stands for, Atomic Reference Count.

And hey, this basically turns off a lot of the borrow checker. Because now the lifetime is indeterminate, it's going to go away on its own. I've got garbage collection. And the problem with that is once you start doing that everywhere, and now you're having atomic increments and decrements everywhere your program's a whole lot less understandable because you don't really know exactly when that's going to go away.

And so I've seen newer programmers to rest, tie themselves in knots because they want to not solve the problem. Not solve the bar checker issue. And just go for what looks like easy mode and C plus plus had the same problem back in 2013, I think it was when shared pointer appeared. I started seeing it absolutely everywhere, even in places where there was no sharing.

Because, because it feels like easy mode and neither language is that forgiving when it comes to, Hey, I want it, I want it. I want easy mode, you know, it's in both cases, it'll probably work in both cases, you're losing quite a bit of performance. And the other one I run into is. Just a hilarious side effect in a way, because Rust enumerations are a sum type, so only one of the possible variants actually exists at any given time.

So, for example, when you return an, when you return that example you were talking about where you said sum, That's what's called an option type, which is Rust's attempt to not have nulls. So it's a sum type that it's either equal to none or it's equal to sum and a value. And what you are doing with if let is a pattern match.

You're saying, if it matches the pattern of sum and something inside, please just give me what's inside. So the syntax for it's kind of messy, but it's super powered and then it collides headlong with desperate attempts to work with systems that aren't Rust. So I was building a demo program on error handling and the C standard library can give you literally thousands of potential error, error codes.

And somebody had gamely sat down and typed every one of them into this enormous enumeration of doom. And so. I, I call out to a C library, it goes through this translator, comes back, and I've got this type that's either going to be okay in my value, or an error. And it's like, okay, well, having an this is too important for me to just say, I've got an error, I'm going to bail.

Some of these errors I might want to handle. Right. So I type match you know, error type, sorry, error. kind, and allow my IDE to autocomplete and it entered 16, 000 possible line, possible errors that I might have to handle. And it's like, okay, well, that, that's not what I had in mind, but I'm going to spend a while because two of these were actually what I care about.

And so there, there's You know, Rust Rust prog the Rust core team and people who've been around it a long time tend to get very safety conscious. You know, you say safety way too much and take off everyone on Twitter by saying that wouldn't have happened in Rust. It might have done, you know, it's So you've got the weird tendency that PRs appear on GitHub in your repos saying that in some obscure circumstance, this function may not work.

And so Rust's biggest problem in some ways is that we really need to stop taking people off and be nice.

Jonathan: That's fair. That's fair. Let's see. One of the things we haven't talked much about is, is the Rust kind of concurrency guarantees. That's another one of those things that. tie C programmers up in knots. Concurrency can be difficult to work with. What's, what, what does that landscape look like in Rust?

Herbert: This was actually one of the things that, when I first started using Rust, I had a eureka moment, shouted, this is amazing, and realized I was going to keep using Rust. You can the compiler makes a very bold guarantee of you will not have data races. And there's actually now two languages, Swift, Swift has just offered the same guarantee.

And in both, both cases, it works by tracking whether or not ever any structure is safe to, A, send across threads. So you might have something that's pinned to a core, so it's not safe to move it to another core. And you might, but also whether or not it's what's called sync, which synchronized means I can safely read this value from another thread.

Right. And so what will happen is you use you create, so the old, you know, C demo where you create a variable called counter, spawn off three threads that count to a million and add one to counter, and then when the threads are all finished, you print out the result, you get a different number every time.

In Rust, that will not compile, and it will not compile with the error message that you're trying to mutably access a variable more than once, and that's the That's the underlying rule, is that only one thing at a time can ever have mutable access to a variable or memory location. And so, obviously, sometimes you need to do that.

So you implement atomics or mutexes. Now, if you're used to C or you're, you've probably used pthread create mutex, and now you've got a mutex variable, you have to remember to lock that. And you have to remember to unlock that. While C next level up, you make a mutex with a lock guard. As soon as it falls out of scope, RAII kicks in, the mutex is freed.

But you still have to remember to lock that. While Rust forces you to put the mutex around what you're protecting, it can either be on the outside, so a mutex protecting a whole structure, or a structure with mutexes, atomics, and safe structures inside, so you can access them independently. And that ties into the sync system.

You can now read that Mutex completely safely across threads because Mutex is a design for that. If you lock the Mutex, the Mutex then gives you safe access to what it contains and ensures that you don't get the concurrent rights to it. And so you can't forget to lock it because the way you Mutex protect an integer is Mutex and then the angle brackets and your integer type inside so that Mutex.

Contains the data, it's a wrapped template. And so, there is no sane way without dropping into really horrific, unsafe code to get to the contents of that mutex without locking it. So now you can't forget to lock it. You've got RAII, the lock goes away as soon as it falls out of scope. You can still deadlock.

So if you write mutex, if you write, you know, mutex. lock, mutex. lock twice next to each other, your program just stopped. Don't do that. But it means you can have a really big code base with lots of Atomic or mutex protected or read write lock and so on, you know, there's lots of lots of locks.

And the compiler will statically verify before it even finishes compiling that there is nowhere in your code that you have concurrently accessed this variable because you have these sync barriers are always in place and not having them as a compiler and you to give you a good example, you know, the Libre QoS monitor system is like 150, 000 lines of code.

Last I looked we have probably, well, we have one thread per core, plus about five more running at any given time. And there's all sorts, there's all manner of shared state. And we have in development, we had one deadlock because I did something dumb. But we've never had a race condition. And so I can attest from, you know, battle testing that you will not get race conditions in say for us.

Okay.

Jonathan: Yeah, that's, that's cool. Okay, so let's say that we have sold somebody on the idea of Rust, and actually, it sounds like various things are coming together that if someone wants a good programming job, one of the, one of the good languages to invest some time into learning would be Rust. Where, where do we start?

Where's a good place for someone to start learning Rust?

Herbert: Okay, there's lots of different learning styles. My personal one is, I pick a new language and I try and write a game in it. Okay. I like writing roguelike games, you know, the old NetHack style where the, where the little at moves around and kills the little g's.

That's actually how I got started was I took a seven day roguelike challenge. And decided to do it in Rust and made a really simple roguelike game that was some of the worst Rust I've ever written, but it worked. Then I decided to do it right, and that turned into an 80 chapter tutorial, the Rust roguelike tutorial that Is available free and that's a that is a great way if you already know some other languages If you're more of the study type go get the rust programming language by klapnick et al.

It's available free online or You can buy it if you want to support them if you like learning through games, I mean this Obviously, I'm going to plug Hands on Rust, my book, because you start at Hello World and that turns into Flappy, Flappy Dragon, which is Flappy Bird, and then a giant roguelike walks you through all of the basic, basics of the language and you have fun learning it.

If you're more into writing cool systems, Tim McNamara's Rust in Action is an excellent place to start. He, it's basically a book of cool little programs written in Rust that here's the program and here's how it works. So there's lots of different ways in, but I'd also encourage you, you know, find a, find a cool project and just pitch in.

And go on to the Rust discord. The official Rust discord is so ridiculously full of helpful people, it is almost intimidating. But I remember the first time I went in there with, with a really dumb newbie question. Nobody made me feel stupid. They were bending over backwards to help me. And I even woke up to PRs in my GitHub repo the next morning with suggestions.

And it is like that. It's, I mean, It'll be toxic if you come in saying, well, Rust sucks. But they're, they're really, really, really nice to people who genuinely want to learn. Also, you know, go out and look at some of the existing Rust stuff on GitHub. Often you'll stare at it. And think the syntax is alien and weird.

I've had that experience. Yes. Then you write some rust and then you come back to it. And once you get past the fact that if let is horrific syntax, honestly it should be called a one arm match. Cause It is a match statement, which is a pattern matching statement that can only match one thing.

And once you get, you know, once you get past a lot of that, you realize it is, hey, I know this, you know, because it is a lot, like a lot of other languages, the, but the syntax does make the learning curve a little steeper. Also chat GPT writes really, really bad Rust but it is fantastic for, here's a C function and here's a Rust function.

Are these equivalent? It does that amazingly well, right down, right down to telling you that on this line in the C and this line in the Rust, you're doing this non obvious thing different. Oh, interesting.

Jonathan: One of the, one of the few places where chat GPT is actually genuinely useful. I have opinions about that, we don't have to get into that.

Herbert: But what I don't recommend is, hey, chatGPT, I want to learn Rust.

Jonathan: Write me section one of chapter one of a book of how to learn Rust. Yeah, that might not go so

Herbert: well. Yeah, probably not. Although I think I've read that manuscript.

Jonathan: There you go. Oh, is that something, this is not really Rust related, but I'm just curious, is that something you guys have run into as a book publisher?

Have people tried sending you manuscripts written by LLM?

Herbert: So, Less Us, because Prague Prague is kind of a small boutique. Niche enough, yeah. I don't think any of them have made it past the first editor you talked to. But I know from talking to other publishers that the answer is, oh god, yes.

And you can, there's nothing wrong with having an LLM help you. Like, here's some text turn this into, You know, a 10th grade reading style or can you help me with this sentence? It's great for that but here's the outline. Write my introduction. Don't do that, please. Editors are getting really fed up with that.

Jonathan: Well, I mean, one of the other places we see it is people are starting to try it. People are coming after bug bounties with bugs, bugs discovered using LLMs. And I guess there might be something to be said for trying to find something that way, but only do that if you know enough to be able to actually confirm yourself whether it's a bug.

Because open source projects, I know the people that run them are getting sick of exactly the same thing. They're getting bug reports. Pay me a bug bounty for this bug that LLM found. No, it's not a bug. You don't understand what this C code

Herbert: does. You know, a funny unrelated one was while I was working on a chapter of my book, I forgot that I had turned on GitHub copilot to because I was curious to see what it did.

So I'm typing away and it suggests the next two paragraphs and I was like, no.

Jonathan: Yeah, thanks. Thanks Microsoft. That's yeah, that's real helpful.

Herbert: And there are, there are times it's good. It's good for I've written something that I now have to repeat in a similar vein 25 times. It's cheaper than asking, asking a junior programmer to fill that, fill out the blanks for you. On the other hand I've seen it suggest some really special code too.

So it's not a tool for, I don't know what I'm doing. It's a tool for, I do know what I'm doing, but I want a better autocomplete.

Jonathan: Yes. Yes. I think personally, I think LLM works well for me. for sort of like an autocomplete, maybe a glorified spell checker. It also seems to work okay if you're using it as a search engine.

And what I mean by that is you have to think of it the same way. If you type, you know, you type something into Google and you say, and how do you do this? And Google is going to give you several different websites where you can look for information. You can kind of do the same thing with, with chat GPT.

And rather than giving you a website, it'll give you kind of this condensed version of it. But as with a search engine, you can't trust, you can't trust it just because it's on the internet. You can't trust it just because chat GPT said it.

Herbert: I found it useful the other day. Cause I went a complete blank on I think they're pronounced chloropleth maps.

I, I knew I wanted a map where the country has changed color based on some data. And. So I'm trying to start type that into Google and I'm not getting very far because lots of things so I just asked cat GPT What's the what's the formal name of a map that changes color based on the value of some data associated with each country?

And it gave me the answer right away. Yeah, so it's great for that and I I was having a real senior moment, so chat GPT made me, made me feel better. So, yes,

Jonathan: I have those all the time and I've not started using chat GPT for that. Maybe I should tip of the tongue syndrome is I think one of the technical names for that.

All right. So we've covered a lot. We have a couple of minutes left. Is there anything that you really wanted to talk about that we didn't touch on?

Herbert: Obviously I'm going to say advanced hands on Rust is coming out in beta in the next couple of months. I am really excited about that. I'm going to be at rest nations UK.

So if any of you are in London I would love to say hello. I'm also going to be at rest conf in Montreal later in this year. Although exactly the exact details of that haven't been announced. So I, I don't know exactly when, but I know I'm going to be there. And so if anyone wants to say hi or talk about any of this, I'd love to hear from you.

And also the LibreQ OS guys wanted. me to thank you for having me, and the Arden Labs people wanted me to thank you for having me, and I want to thank you for having

Jonathan: me. It's been, it's been a lot of fun. So the advanced hands on Rust, is that also going to be kind of a let's build a game together?

It is

Herbert: indeed. The first section you take your basic Flappy Dragon, and you learn to make a library. And then fill it with generics and all sorts of cool stuff, while you add smooth animations, a really robust physics system that doesn't need to be that robust for Flappy Dragon. Parallax scrolling backgrounds, all sorts of cool stuff.

And then the second half of that book, you take your library, discover that you can now write a game and play it. 20 lines of code and start adding more stuff to allow you to learn about threading, async code, and some network things for high score tables. So it's, it's a continuation of learn the Rust language.

It picks up where the previous one finished.

Jonathan: All right. Where, where is the place to go to, to look for the hands on Rust book and when the second one comes out?

Herbert: All right, the publisher is pragprog. com. P R A G P R O G. com. They're also on the first two I wrote are on Amazon. Advanced Tens on Rust will be, but not until it finishes beta.

Because pragprog do a wonderful pre release system where you can buy, where you can buy the beta, you get every update as they come out, and a direct line to tell me what I did wrong.

Jonathan: I'm sure that's handy. That must be really

Herbert: helpful. It's actually a lot more helpful than you might think.

Jonathan: Well, I know, I've written a bit of technical stuff.

I understand how that would be nice. Maybe I should try to do that with my Hackaday

Herbert: articles. The first one, Hands on Rust, went out on beta. A couple of days later, I had a message from somebody, I'm not going to name, but they're on the Rust core team, telling me Your function on page 12 is a fantastic example of thinking around the problem, but here's the one line version.

Yes,

Jonathan: yes. That's great. Did you include both in the final work?

Herbert: I actually did because I used one to explain the other, so.

Jonathan: Yeah, I was going to say, that seems like a perfect example to include the more verbose version for the teaching tool, and then, oh, by the way, this is how you would actually do it.

Herbert: You know, and then you get, you get strange ones like I had a bug report at one point that I had a word in there that didn't make any sense in Ukrainian and, and I, I'm happy to change that, but I had not even considered that people were running this through a translator. Right.

Jonathan: Right. Yeah. That's always fun with bug reports when just generally speaking, when it's not, when English is not someone's first language, when you don't have a shared first language with someone, it could be, it could be a challenge in itself.

Yeah.

Herbert: Yeah. Actually, the funniest one was also that I, I was trying to demonstrate using Unicode to demonstrate and some of the pitfalls of that in Rust brain teasers. And I used an example with a word in Russian and got a message saying, you know, from, actually from another Ukrainian saying, you know, hey, the Russians really aren't very nice.

And. I was like, okay, I can understand that because, you know, they did, they did just invade, invade Crimea and Donetsk all that at the time, about the time that came out. And so I was like, okay, I'll change it. So I set out on a quest to find a language belonging to a country that doesn't have a hysterical beef with another country.

Good luck. I eventually settled on Iceland, just because they haven't had one in a few hundred years. Yeah.

Jonathan: Oh, I have no further comment on that. Not on the record at least. All right, so I've got to ask you two final questions before I let you go. I will get emails if I do not. And those are, what is your favorite scripting language? And what text editor do you spend all day in?

Herbert: Okay, I'm going to do this in reverse order because text editor if I'm teaching in front of a big audience, I'm in Visual Studio Code.

And the reason for that is that it is everywhere. And no matter what skill level you're at, you're comfortable. When I'm sitting at home working on something for fun, I'm usually in near them with The rust analyzer extension enabled. I like that because I have a big fat Linux machine at work and a tiny MacBook air, and I can sit in bed SSH into my fat machine, have it do the hard work.

Pause the TMX session, go work on it somewhere else. I also use rust rover from JetBrains. Partly because they handed it out free and I wanted to see how good it was. And it's getting better. Yeah. I don't use it all the time. Purely because my muscle memory hasn't caught up with the different keystrokes yet.

Yeah for a scripting language. I would say Python these days Even if it's overkill, I also write way too much stuff in just regular old bash But I

Jonathan: definitely counts nothing wrong with using bash We had the we had the creator of bash back several years ago, and I asked him like well What what's your favorite scripting language does bash count and he was almost offended.

He's like, yes, of course it counts That's just great for scripting Oh fun stuff Well, thank you sir for being here. I sure appreciate it. Thank you for letting me pick your brain about rust I I got a lot of my questions answered and honestly, I think I'm gonna have to go check out probably the Probably the free version first the, the, the, it's not Hands On Rust, what's the, the other tutorial, what's the title of it?

Rogue like! Rust, Rust Roguelike Tutorial. The Rust Roguelike Tutorial, yes, I've got it pulled up here, but I couldn't immediately see the title of it. I think I'm gonna have to start there, but I'm gonna have to look real close at Hands On Rust, because like I said, it's, it's kind of been on my to do list for the longest time to actually pick up and learn some Rust, and the idea of building a Roguelike with it, that, that sounds like fun.

I like it. Awesome!

Herbert: All right. Thank you for

Jonathan: having me. Yes, sir. I sure appreciate it. All right, folks. That was, that was Herbert Herbert Wolverson talking about Rust a lot of fun. I had a lot of fun with that. I appreciate that he was willing to talk, not necessarily just about his own work, but the, the wider Rust ecosystem.

And We get to, we get to plug his books. So good stuff. Next week we do not yet have a guest scheduled. So if you know of someone that is involved with an open source project give me a ring it's floss at hackaday. com or come into the discord and say, Hey, this is the person you need to talk to, or even better reach out to, you know, a programmer or the lead dev of your favorite project and have them get in touch with me directly.

But we don't have a guest yet for next week. We will scramble, we will make something happen, but we would love to have somebody to actually talk to. Alright, so let's see. The one thing that I want to plug for the end of the show here is, of course, the Untitled Linux Show. It is now, the audio version of it is available to everyone.

We have a lot of fun there just going over the week's news worth of Linux. And you know, funny thing? There's always enough news to cover. Whenever a week has gone by that we go, Man, there just wasn't anything to talk about. Nope. Pretty much always. Bunch of news going on in the, the Linux world that we all sort of inhabit together.

That is the, that is the big one for me. Thank you to everyone that caught us live in the Discord. We had a decent crew there today and those on the download as well. Make sure to share the show. We love letting folks know about it and we will see you next week on Floss Weekly.

Episode 773 transcript

6 mars 2024 | min

FLOSS-773

Jonathan: This is Floss Weekly, episode 773, recorded Wednesday, March 6th. NodeBB, don't do the math.

Hey, this week we talk with Julian Lamb about NodeBB. It's open source forum software that brings the idea of the forum into the modern age. There's some neat tricks it knows, like forking forum threads, talking on the Fediverse, and more. Hey, it's a great interview, you don't want to miss it, so stay tuned.

Hey, hello, it is time for Floss Weekly. Weekly, that's a show about free Libre and open source software. I am your host, Jonathan Bennett, and I've got Jeff Massey with me today. The one, the only welcome, sir. Oh, good to

Jeff: be here. Always, always love a crossover episode.

Jonathan: Yes. I, I love having one more co host that has some availability to help with the juggling.

So I appreciate that. Well, today we're talking about, we're talking about Node BB, which is Bulletin, not really bulletin board, but I believe it stands for bulletin board. Uh, software, so it's forum software that runs on top of node. js, which I kind of got started with the, the whole PHP side of things. So sometimes when I hear node.

js, I'm like, uh, we'll try not to hold that against the project. And hopefully we'll get into that. Maybe, maybe Julie and Lam, our guest can convert us over to being lovers of node. What do you know about the project, Jeff? Uh, just

Jeff: that it's forum software, you know, it's funny. You said BB bulletin board. I always think back in the Fido net days and the old dial in.

And, you know, when I think bulletin board, so. You know, I'm, I'm looking forward to see what, uh, what's in store for us to, to learn all about it.

Jonathan: Yeah, you know, there's, there's some of those, those old pieces of software that are kind of experiencing, experiencing a resurgence, um, between multi user dungeons and bulletin boards.

There are some that you can still dial into, either actually dial in or kind of do a virtual dial, dial in. So people are still interested in. And kind of that throwback. I don't know if, if we would consider forums to be sort of a throwback technology in the same way, definitely not as old. Um, but I mean everything's, everything's on Discord and Slack now, right?

Everybody's gone to Teams and Google Chat, and why do we need forums still? Well, no, I'm not being very serious, I still like forums. Although my, my experience with forums, really, I think of, uh, what is it? The simple machines forum from PHP. That's the one that I'm used to. Um, well, let's go ahead and not, uh, let's not dive down this, this particular well any longer.

Let's bring Julian on. Hey, sir, welcome to the show and let's chat about. About node BB. What did we get wrong? How terrible were we at our guess at an intro?

Julian: Well, first off, thank you for having me. I'm happy to join. Yeah So, forum software, and specifically bulletin board systems, uh, you mentioned BBSs, and I'm actually, uh, not from that era, I haven't actually used any of the old, and you mentioned throwback BBSs, those, um, older, old school bulletin board systems, but I sort of cut my teeth on, you know, those simple machines forum, PHP, BB, uh, VBulletin, and the like.

Right, right. And, um, You know, those things, those old forum stacks have aged fairly poorly, you know, the most of them are about 20 years old, and It occurs to me now that I used to say that they are 10 plus years old, but at this point we are 10 plus years old So I guess that makes us old too.

Jonathan: Oh say it ain't so Never never do the math.

Never. Yeah,

Julian: exactly Um, yes, we're based off of node. js um, which is Which at the time, 10 years ago or so, it was fairly new and modern and was very enticing for a lot of reasons. Not only because of the asynchronous, um, mess, uh, asynchronous flow of, um, and so you're not, you're not blocked by any sort of network calls and that sort of thing.

So in terms of scalability, we had, we saw some early gains there over PHP, but at the, the downside being it's not exactly Easy to install. It's not like you can just upload a script to a web host and you can get a forum going. Yes. Some trade offs

Jonathan: there. Yeah, that, honestly, that's probably the thing that makes it the most challenging.

Like that's, that's maybe the thing that annoys me the most about a Node. js project is it's not easy to install and it's not necessarily easy to integrate into something else, you know? So if you've got, if you've got a PHP based website, you can't just drop an Like you said, you can't just drop a script in there.

You can't just make a PHP file and go on. But at the same time, so I've, I've, I've, I've programmed in PHP. I've programmed in JavaScript. I've done Flask applications where you have Python as the back end and I can, I can definitely see the appeal to being able to just write JavaScript. Like, for all of its warts, JavaScript is kind of fun and easy to work with.

So I can, I could see the point of, let's just make, let's just do everything with JavaScript.

Julian: Yeah, I will admit the fact that we could sort of hit the ground running with JavaScript, you know, with all of its warts, uh, was very enticing as well. And so, You know, we decided to take the whole concept of forum software and bring it to the modern age.

So the modern web stack, latest web standards that bring the UI up to the expectations for modern web allocation. Something as simple as, uh, mobile responsiveness. You know, a lot of those old forums you can't use on your phone. Or you'll load it and then the text will be about two pixels high. And you have to, uh, pan and zoom.

Let's see

Jonathan: here. Yes, yes. Yeah,

Julian: so, you know, we've all had those experiences. We should have left those behind. And, you know, NodeBB likes to offer that sort of mobile, fully responsive, you can use it from, you can use it on a Nintendo DS if you like, uh, up to the largest, you know, big screens in the stadiums.

Not that you would, but you could. And, uh, you know, but then Forum software has got a lot of things right from back in the day, you know. The whole concept of extensibility. Plugins and themes that you don't get nowadays, you know. Um, You, you have, you can install plugins and there are APIs for things like Discord.

Which is fantastic. Uh, there was a time when we were talking about incumbent social media where you just couldn't. You know, you were stuck with. Facebook's interface, for example, right? And you couldn't really customize it. I'm not advocating that we go back to MySpace, where every individual person just has their own HTML laden profile page, but you know, somewhere and somewhere in between.

Yeah, you can find a

Jonathan: happy medium. Well, I mean there's there's a big area too where The communities that just don't fit well on Facebook for whatever reason, right? And it's kind of nice to be able to have control over your own community Um, to either, you know, kick someone out that's being a problem or not kick someone out that Facebook is thinking is a problem.

Um, and so I think there's a, there's a, there's a big niche still. And I know this is crazy to people, to some people, but there's a huge niche for running your own software, having your own either server or, you know, I'm sure there's ways to put this on AWS or what have you, so that you don't have to have a physical box in your garage.

Um, but hosting it yourself is still a viable solution for a lot of us.

Julian: Definitely. And for a lot of enterprise customers, just the whole idea of content ownership, being able to control the narrative is a big plus. Another one is having your content indexable by search engines. You just don't get that when your content is on Twitter or Facebook.

Facebook specifically. Uh, Slack is another one. I mean, you're not going to find any of those, um, any of that content on Google. And let's say you have a support team that uses social media, which is great for reaching your customers, but then you'll be answering the same questions over and over again. And because your customers can't help themselves because the only avenue to reach you is through social media.

Yeah. To which you cannot look at the history, not easily anyways. Unless you want to break your fingers.

Jonathan: Yeah, well I can't tell you how many times it's like, I saw this person post something on Facebook, let's see if I can find it in 30 minutes. Never found the post. Yeah,

Jeff: it's buried. Well, I've got a question for you.

So, kind of going back to square one, how did you just wake up one morning and say, you know what, I just need to, I need to write forum software. This is my driving goal right now.

Julian: Um, well, back when I first started in the industry, I was doing, I was doing Facebook games, Facebook apps. So that was all the rage back then.

And, uh, we decided to, I ended up taking two of my colleagues from that company and we started our own, uh, consulting company. Doing Drupal stuff and WordPress stuff. And as it turns out, Clients are, clients suck. So, we wanted to, we wanted to do something on our own. And at the end of the day, right now, we actually do work with lots of clients.

So maybe I shouldn't say that too loud.

Jonathan: Not you guys. If any of my clients are listening, not you guys. You

Julian: guys don't suck. You're the exception. Yeah, we had an opportunity there, you know, between clients, to think about How we're going to fill our time and so we experimented with a lot of things and one of those things was Looking back at forum software and thinking, you know, could we do something on our own?

At that time discourse had just launched to great fanfare and we thought that was fantastic and there was a lot that Jeff Atwood was saying that It made sense to us at the same time. We didn't know Ruby and we didn't really care to learn it. So we decided to do it on node. js. That's all makes

Jonathan: sense.

Julian: Nice. Does it? Oh, good. Okay. I mean,

Jonathan: I would be more willing to figure out at node. js than I would be Ruby. I've tried to install a Ruby application before, and I think I ended up walking away in frustration. Um,

Julian: so I think that also happened. Um, but I'm not going to go into that. Okay.

Jonathan: All right. So you, you mentioned something that I think would be interesting to explore, and that is this idea of enterprise customers.

Do you, do you have enterprises that are, that are using

Julian: NodeBB? We do. And the majority of them use it in an internal capacity, or either, just not public. Right. So perhaps it's a, um, a backend for their internal customers to use. Mm hmm. Which It happens a lot more than you can think. Just because it's not exposed to the public net, you don't really realize that it's there.

And a lot of it is, uh, gated behind, um, User authentication. So let's say we've, one of our biggest customers is the largest bank in Singapore. Okay, and You know, you need to make account with them to access the forum, the community, which makes sense. Interesting. But it's all behind there. Yeah, and so they're using, they're using NodeBB and it's customized for them and it doesn't look like NodeBB at all.

But yeah, happy to use it, happy to have them.

Jonathan: Yeah. I imagine you also probably have, uh, instances where node BB is, is almost the, uh, the internal knowledge base, right? So like if it's a, a sort of a technical field and so you have the various texts and engineers and they need some place to be able to talk about things and have, you know, their conversation searchable, then that might be a nice fit as opposed to, uh, you know, one of the other, uh, one of the other solutions that you don't have as much

Julian: control.

The really interesting thing about NodeBB is that you can do a lot with it. Um, part of it being you could make it your knowledge base. And we did have a customer at one point who, uh, who used Node, who actually paid for, with our hosting, an enterprise customer who paid for our service, and then just didn't use the frontend.

It was just the backend with our API that they used. And they just programmed their own frontend because they wanted their customer support agents to be able to search, search for, Answers. Oh, interesting. So they built their own search engine, tied it into our database, and then that was that, and sure they could have gone with a hundred thousand other different solutions.

Right. But they went with ours, which was really interesting because they pushed our software in the direction that we, you know, we wouldn't have thought to go that way. And it's really interesting when we have enterprise customers because they've got really, really talented engineers who are Um, who are paid to do really weird things with our software.

Yes. And they push the boundaries of what we can do, and then they come to us and say, Your software is falling over, fix this. And so we're happy to, because it wouldn't be good if we, if we could, you know, if our software could

Jonathan: support this. Yeah. Now, what's the, what's the uptake on some of those, we can talk about these talented engineers, actually pushing patches back into the project.

What is, what is your, uh, your base of contributors look like?

Julian: Depending on the customer, um, Oh, it's, it's a typical refrain with open source software that is that we don't see a lot of those patches land back upstream. Um, if they are one of our customers, we push very hard for everything to be open source.

So, sure, there's going to be some private repositories for their custom theme, for example. Or, uh, a private repo for their logic that deals with their API. Because there's no reason why we'd need that public. But let's say You want something supported in NodeBB core. Um, but that's open source, so we're going to insist that it be open source.

Because the last thing I want is for you to fork the open source code and customize it. Because then I have to maintain that fork. And that's twice the work. So no

Jonathan: thank you. Now, NodeBB is what, GPLv3? Yes. All right. So there's, there's kind of this, um, various people have called it either working as intended or a loophole, um, but because of the way the GPL works, so if someone just deploys this on their servers, they don't have to release patches back even if people use it.

So is, have you, have you, have you thought about the AGPL?

Julian: Um, not so much because I, I don't, well, we've had, we've had discussions back and forth and whether we're going to go for AGPL or whether we're going to relicense as MIT even is something that we talk about every once in a while. Um. It's not really It doesn't concern me too much that some of those patches don't land back upstream because we do, you know It would be nice.

It would be nice to have but If they keep it on their own, it's okay. If you could if you deploy your own nodeDB software, you're not you don't have to Release the source unless you're selling your software in which case yes you do. I think that's the distinction. I could be mistaken

Jonathan: So the, the actual text of the GPL v3, if, if the software is running on the end users machines, which, because we have JavaScript running here, some of this gets weird.

Um, if it's running on the end user machine, then you have to make the source available, but if it's just running on the server, then you don't have to. And so that's, that is that conflict between the GPL and why the AGPL was written. But the problem with the AGPL is You know, you've got, you've got places like Google, they've got built into their corporate structure.

We do not use the AGPL for anything because they're, they're afraid it'll, it'll get in and essentially infect, to use that term, all of their internal code base. So it's, it's, it is a, it is a challenging, it's a challenging problem. Yeah,

Julian: we've, we've gone through that, uh, that process before where, you know, an enterprise customer will want to go through.

How are licensing every single one of our dependencies and look for some sort of conflict? I Think we're okay We have a lot of dependencies and a lot of dependencies have dependencies. So I don't actually know But I Luckily, it's been okay. I and if you know from a business Standpoint if switching to AGPL does mean we lose some customers.

That's not necessarily a good thing. So sure. I don't know I don't actually have an opinion on that right now

Jeff: Well, you talk about you know self hosting so I'm I'm sitting here and I decide, you know I'm gonna run the premier forum on fidget spinners or whatever Sure, and I'm gonna host it myself Is there a way to incorporate advertising into the forum software so I don't have to carry my own server load totally myself?

Julian: Uh, well because everything is Extensible through plugins. You can do a lot with plugins. We have, um, a lot of internal hooks, uh, that plugins can take advantage of. One of those being an AdSense plugin, which you can plug right in, and then you can just put ads wherever you want. You can put them on the side, you can put them on the top, you can put them in between posts if you like.

That works. And, I think there are other ones? To be fair a lot of our plug in ecosystem is entirely third party driven So people will have a need and then they write their own plugins and they publish it to npm and I don't hear about it So it could be there could be a lot Of you know different themes different plugins that I just don't know about because I don't hear about them There are literally thousands of

Jonathan: them.

Oh, wow, that's impressive. So I heard you I just say node BB core And that, that sort of terminology can mean one of two things. It can either mean that simply, well, we've broken this up to be modular, so it's easier to work with. Or, our core is open source, and then everything else is closed source, and you have to pay for it.

Yeah. So, No, it's the former. Okay.

Julian: Yeah, we, uh, Purely from a simplistic, simplicity standpoint, I wanted to publish a piece of software that you could run with minimal dependencies. I mean, it's, it's, I'm not gonna say it's heavyweight, it's pretty minimal as far as things go, objectively. But, uh, you know, you could load NodeBB, and You, you know, it does it, it comes bundled with a markdown parser, but that is technically separate.

You could turn it off if you'd like. Sure. If you're coming from a software that uses BB code, for example, which I don't know when the last time you heard BB code was, but we have a BB code parer as well. I, I still, so you could

Jonathan: swap one out for the other. I still have some BB code that's muscle memory

Julian: if you'd like.

I think you can even, I think there's a parser for wiki code if you'd like. Yeah. To write in that sort of, uh, dialect. That's fun. So we wanted to keep the core, uh, simple in that way. So you could swap out different, different, uh, segments of NodeDB. Um, if for example, you don't, there's no need to include the kitchen sink.

There's no need for us to support, you know, login via some arbitrary, not everyone wants Twitter login or Facebook login, for example. So why would we put that in core? So those are all available as plugins. If you want to connect to your own. A, uh, your own OAuth 2 endpoint, the bespoke user authentication back end.

You can do that through a plugin. Everything is plugins. Themes are plugins, technically. Okay,

Jonathan: so I'm, I'm very much reminded of the way WordPress works with this. I, uh, were you inspired by sort of the WordPress plugin? Um, Scheme, you said you guys worked with WordPress before this.

Julian: We did. Yeah. Very much so.

Uh, WordPress did a lot of things right when it comes to their plugin ecosystem. The fact that, you know, you could just search for what you want, install a plugin, and then it just works. Now the downside of plugins with Node. js is that if you install a plugin and it doesn't work, then your forum goes down.

But, you know, fingers crossed, that's less

Jonathan: of an issue. Um, one of the, one of the interesting things that the way WordPress does it is some of those plugins are paid. They have paid functionality, or you have to pay for them. I'm extremely curious. Is there that sort of an ecosystem now in NodeBB? Do you have a plugin store where some of those are paid only?

Julian: That is a very good question, and it's something that we might have pursued, uh, if we had more manpower, but we don't. We're actually a very small company. We started off with three tech developers, three tech guys, and now we have two, because one moved on. Sure. And, um, I think at the end of the day, we're better at building software than building businesses.

So the whole idea of a plug in, like a plug in paid, um Sort of store was something that we talked about, but it's just who's going to execute it Yeah, and who's going to take on the maintenance burden because these are all technical considerations Yeah, who's gonna take on the maintenance burden of this code base?

That you don't really want to maintain because No, BB core is orders of magnitude more exciting than dealing with a payment platform and dealing with a stripe API in this net Yeah, and so when it comes down to it People who use your software, your software is sort of your public face. And so if you're putting out bad software, then it reflects poorly on you.

And that's something that I found really interesting with open source software is that, Um, you're, you have this sort of, Drive to produce good software or better software than normal. And you should see the stuff that doesn't get published to RootSource because it is just a nightmare. Because I'm not accountable to only myself.

And if I can read it, theoretically, then it's good enough. Or it's good enough to launch. But then I can't do that on NodeBB. I have to commit something because someone will call me out on it and say, this is garbage code, what are you doing?

Jonathan: Yes. Um, let's see. So you, you, you made this, you made this comment that I think is, is very apt and that is that a lot of us that are coders are not very good at running businesses.

And that's very true. And I've seen that in several different projects I've been involved in, but I kind of want to ask how, how is that going? Um, what, uh, what does the revenue stream for, for you guys? Cause obviously you do this as a business. What does that look like? How do you make money with it?

Julian: Yeah, that's a good question.

So we, uh, we ended up, uh, incorporating in Canada and we serve clients all over the globe. And, um, so that's, so we ended up, we ended up making money. We keep our core open source and most of our plugins are open source as well. And, um, we sell support and maintenance and custom development. And that's where our main revenue stream comes in.

Oh, and additionally, our. Self service SaaS offering so you could just pay us and we can host the forum for you as well And so that's where that's where our revenue stream is and that's worked well for us We haven't taken on any Seat and sorry any additional funding besides an initial seed round, which we're actually actively paying back And so it's it's working.

Well, I think yeah, that's impressive. Obviously. We're not making money hand over fist, but That's okay.

Well, a lot of that comes back, comes down to the question of people asking, Hey, aren't forums dead? So, yeah, I guess. Maybe in the past, however many years, forum, a lot of forums are shutting down and they're either moving to Reddit or they're moving to Discord. But at the end of the day, forums are the easiest way to build a social network, your own social network, while controlling the data.

You can't really rule your own Facebook. That's just to say. And, you know, we've got a lot of people using what I like to call incumbent social media that are actively Hurting the user experience. So to use the words of, um, Cory, Dr. O they've been and shitified essentially. So we've got a, so I've got a quote here from a former Google software engineer, Ian Hickson, who put it very succinctly that decisions went from being made for the benefit of users to the benefit of Google, uh, to the benefit of whoever was making the decision and transparency evaporated and you could sort of.

You know, think that this is going on for not just Google, but, you know, for Facebook, a lot of these large companies, because at the end of the day, um, we've got, and we've got an algorithm that, um, it values engagement above everything else. And, you know, all the second order effects are ignored, you know.

We've got, we've gotten to the point where some people are saying that the only thing you can do is to not play the game, um, the game of social on the web, which. It doesn't quite work because the whole idea of humanity is about interconnectedness. And if the tools that we have right now aren't working, then let's find some new tools.

Or, in the case of us, let's find some old tools and make them new again. Yeah.

Jonathan: You know, you talk about that idea of a business making decisions for its own good rather than its users. And there's this kind of magic Goldilocks zone. And I think a lot of it has to do with people at the top of the business doing this on purpose.

But You get to the point to where those two things are aligned. And if you do something that's good for the business, it's also good for the users. And you make decisions that's good for the users, and it's also good for the business. And, and when you can run a business that way, well that's, that's when magical things happen.

Um. And, and there's, there comes a point, I don't know if it's inevitable for every business, maybe, um, but there comes a point to where those two things diverge. And it seems like once they start to diverge, they just wildly go in opposite directions. And sure. And I think it takes. Probably take somebody, somebody at the top with an iron hand to say, no, no, we're going to align these things back again, or so help me.

Once you go

Julian: public, that's when it happens.

Jonathan: I hope not. Maybe. I, I, I hope not. Last week we talked to Evan Upton about Raspberry Pi and they're about to go public. I hope you're wrong. Oh,

Julian: that's true. That's true. But, yeah, you can, you can have opinions all over the spectrum on this. I mean, you mentioned the whole, like, staying small, um, And that really reminded me of Errol Balkin, who was on your podcast recently, or fairly recently, I think.

Um, about, you know, small technology, and how keeping things small. And I like, uh, I love the whole idea that, you know, small teams craft better code. I like to say that. Whether it's true or not is a different question, but

Jeff: I, I kind of believe it, because I think sometimes the projects get so large that things get watered down, so if A programmer has a really good idea.

It might have to go through many layers of management before something can be approved versus smaller, flexible teams to customize what you need. But I mean, saying that, okay, I want to start, you know, or I've got a, uh, open source project. What, what would you tell me to do if I'm starting out and, or maybe I've got something that's just kind of off the ground a bit and

Jonathan: I need advice.

Julian: Yeah, I think when it comes to open source projects, let's say you are, let's say you've got some traction, you've got some usage. I personally, I think one of the best things I did, uh, I think it was five or six years ago was to start a bug bounty program because you don't know what you don't know. And you don't want your software if it's suddenly picked up.

Then you get hit by something that is, that basically kneecaps you and you think, Oh, how could you have let this slide for, I don't know, 5, 10 years? You know, you don't have to commit to it with funds. You could just send somebody a t shirt if they submit. But the whole idea of putting something out there and saying, We have a policy.

Here's how you responsibly disclose something to us. Let's work together to figure this out. As opposed to Relying on security through obscurity, which doesn't work out when you get more popular. And so we started a bug bounty program, I think it was 2016 or so? Um, and, wow, that's a long time ago. Don't do the math.

Don't do the math. Yeah, don't do the math. Um, and, you know, we've paid, we actually, we attached funding to it. Our own funding, because, you know, You know, we, at that point, we were making money, and even, even though it wasn't, even though it hurt when critical vulnerabilities were found, it's orders of magnitude better that we deal with it, we patch it up, and we put it out, and then it gets published online, and then we have to scramble to figure it out.

Jonathan: Yes, very, very, very much so. One of the, one of the things that I do is cover the security beat at Hackaday. And there is, yes, it is always much, it is much better to be able to tell people, Hey, by the way, this company found this, or it got reported to this company. There's no proof of concept out yet. It's fixed in the code.

Even better is when you can say, Hey, it's been fixed in the code for three months, and we're just now letting you know about it. Right. That's right. As opposed to, yeah, this is being used out in the wild right now. And if you're running the software, zero days, I'm sorry. You probably just need to plan on formatting your server.

Julian: And it's very. Very, uh, prescient in the sense that, you know, Bassadon and PixelFed very recently had their own, uh, critical security vulnerabilities come to light. I mean, they were, they were responsibly disclosed, which was very, very lucky for them because neither of those projects have bug bounty programs.

So the fact that the community itself was strong enough to Discover it, responsibly disclose it, and fix it for no money whatsoever is frankly just a miracle, like that everything aligned and that worked out okay, which is fantastic because at the end of the day, yes, we just want to, you know, produce and ship secure software, but you can't rely, you can't rely on that, and that's why I personally feel that a bug bounty program is something very important.

Jeff: Well, and I think that's wonderful advice, and I don't think I've ever. Really heard anybody running a project or very few I should say actually come out and say that and I think that's a great way to look at it is, you know, maybe a short term loss for paying out the bounty or yeah, sending out shirts or whatever.

But the fact you're encouraging people to communicate these issues and you know, people feel free to go understand that there's no harm. Nobody, you know, you're not going to go after anybody for finding a bug or anything, you

Julian: know, Yeah, we're not gonna call the cops on you opening your dev tools on your browser.

Yes. Yeah.

Jonathan: So, uh, this, this You call it a bug bounty, but, you know, just having a responsible disclosure process, even if that's just, here's the email that you send it to if you find something. That's right. I think every project, open source or not, that has a user base needs to have that. And, in fact, it's interesting, you see the, uh, some of the laws that they're working on in the European Union right now.

One of the requirements that they are going to start making for every software project is to have a responsible disclosure policy. It is about to be law in the European Union, at least, that that is a thing that you must do. Yeah, it's pretty interesting. Look, so the, I forget the name of it. I'm sorry, Simon.

I forget the name of this thing that Simon Phipps is covering. It's got some stuff in it that's great. It's got some stuff in it that's terrifying, but it's got some great stuff in it, too. And that's one of them. That's one of the ones that I like. So, we looked at the project ahead of time and there were some, there were some things that really caught our attention that was interesting.

And, and I think Jeff has a couple of these that he wants to ask about, but the one that really, It kind of gets me excited, like I love that you're doing this, and I think there was some funding that was attached to this, and that is, you're going to be on the Fediverse with NodeBB. Let's talk about that story.

Julian: Yes. Yeah. So I guess you can think about this as a continuation of FOSS Weekly 759 with Kevin Podromo. Yes, yes. Yes, so, um, the context of it, the whole context of it being that we've got Twitter imploding, depending on who you're talking to, and Reddit moderators frustrated with the administration team.

And we thought to ourselves, something that we've been telling ourselves a lot over the years is that we want to build something that no other forum software has. And I had this idea years ago, actually, that we wanted, that I wanted to expose content from, from NodeBB to other NodeBBs. Um, essentially a micro network forums talking to one another.

Hardly a new idea. If you think about web rings, that's been around for a long time too. So, you know, I started thinking about this more in depth and I started designing this sort of pseudo protocol in my head. And then, uh, Twitter happened and Reddit happened, and then I discovered Mastodon, and I discovered ActivityPub, and then it sort of all took over

Jonathan: from there.

There was this meshing of gears in your mind, like, I've been trying to figure out how to do this.

Julian: ActivityPub! It's already done, essentially. You know, the stated goals of ActivityPub align perfectly with what I wanted to do with interconnecting forums. And then there were several key advantages. First of all, was that all the protocol work was already done.

So I don't have to make all the mistakes because someone already thought it through. Uh, they did all the heavy lifting on how the communication would work, how the, you know, all the social aspects of it would work. And I'm already joining an established network. So, the Fediverse, as they like to call it, or the social web.

It completely blew my small fish dreams out of the water. And now I'm in, you know, I'm implementing this in a sort of big pond world. Yeah. How, so how, yeah, the how was answered. And so all that's left is to do it, right? So, you know, that's the step back. How, yeah, how can we, how can we benefit forum software with ActivityPub?

Because if there's no point in doing it, right? Um, the, the biggest stumbling block with a lot of communities, small communities, and by extension building products in general is just getting your first users. Yeah. Um, that's, you know. stereotypical chicken and egg problem, right? You don't have users unless you have good content and you won't have good content without users.

And so lots of different forums of different communities have. Um, different strategies. So some of them will just tie an RSS feed into topics, or they'll just post prolifically and, and they'll just share all the links. They'll just, you know, spread all the links on social media and hope that they get some, um, forum users that way.

And then of course, some people just, uh, make their own, they just fake it until they make it, they make some fake users and talk to themselves, but the problem remains is that it's hard to build a community from scratch and. ActivityPub is a potential solution. Yeah. It's also a fairly novel one. The whole idea of sharing content between sites.

Um, you know, right now you just have to copy and paste the URL onto, you know, whatever forum or so on. Um, but then the discussions are separate, right? The origin site, let's say Twitter, for example, won't know about the new discussion that I'm having on my forum. Right. And so you don't have, you know, you don't have that interconnectedness.

Uh, evident there, but with ActivityPub, you can actually take this a step further so I can, in a sense, import an existing discussion onto the forum and then everything is in sync through this protocol. So then I have my comments and replies that get propagated out to the other site, for example, and vice versa.

And that is a radical shift in, in thinking about growth hacking. Because the focus is less on acquiring users and, you know, with all the usual tricks like gamification and all that, but, you know, including the users as they are in your content and contributing back to the wider conversation. That is a completely novel approach.

And it upends the entire assumption that you need users at all to succeed. You know, you could have a forum that only has one user, just you. And you can still be an active participant in the social web. And it's perfectly fine. You can have a thriving forum. With just one user. That's completely wild to me.

Yeah,

Jonathan: in the same way that you can have a perfectly usable Macedon experience by propping up your own Macedon server and you are the only user on it. You can still interact with everybody else. Yeah, that's intriguing. So, how does Very interesting. What are, what are the ways, I'm still kind of curious about the nuts and bolts of this, like what are the, what are the ways that these two things work together?

Can I, can I come to a NodeBB forum with my existing Mastodon account and log in with that? Or can I follow a NodeBB forum conversation from Mastodon? What, what are the, what are the ways that these two things fit together?

Julian: It would be the latter. Okay. So you'd be, so the content from a no VB forum federates out to, uh, to the Fed.

So master on being one participant in it. Mm-Hmm. . So you'd be able to find a no VB user and follow them, and then you'd be getting their content in real time. Okay. As it's published on, no. So less so that you'd be able to go to an OB and log in with your Mastodon account. Okay. Although that is.

It's more, um, thinking of the software you use as a window into a wider network of content, and it doesn't matter that you're not on my forum. You can still participate in the discussions. Yeah,

Jonathan: and vice versa Can a Again, I'm on I use Mastodon. So that's why all my questions are Mastodon centric Sure Can can I follow a like a topic or an individual forum thread for Mastodon and get and get everybody's?

Words on it or is it limited to following people?

Julian: That's a very good question. So we have Mastodon is very Uh, post oriented. Right. So you make a post, you get replies to it, and so on. And the whole idea of having a conversation or a context, uh, of sorting things into a topic, is just Plain not supported in Mastodon.

I mean, it is in a way, but it's not very, it's not meant to be done that way. And so, the fact that NoteBB takes that idea and says, you know, you have a bunch of notes, you can build a relationship between the replies, the responses, and you can organize it into a topic is fairly new. Uh, I think, activity pub wise.

At least in terms of the implementers. We're not the first one to do it. Uh, it's just everyone compares us, compares software to Mastodon. Mastodon does not have topics. Right. But NodeBB does. And so there's a lot of, uh, there's a lot of, uh, different considerations we have to take into account in terms of how we How we notify users.

So a Mastodon user, for example, will never be notified of a response to a topic because the Mastodon does not have the idea of the whole concept of a topic. You get notified if you are mentioned, and that's it. Whereas forum software specifically, if you reply to a topic, you get notified of responses because you contributed to the topic.

It stands to reason. Uh, and that is an option that we support, but it's just, it's, yeah, so a lot of the, a lot of the quirks between implementations, that's where half the fun is with ActivityPub.

Jonathan: Yeah, so, so right now you're trying to sort of map the NodeBB experience onto Uh, onto ActivityPub, essentially.

Uh, I'm reminded of what you said about your enterprise customers doing weird things and how sometimes that pushes NodeBB. Uh, do, do we see this happening here with ActivityPub? Is, is any of what NodeBB is doing sort of pushing ActivityPub along? Have you had conversations with the, uh, Podromo and those guys about, wouldn't it be nice if ActivityPub supported this and this and this?

Julian: I don't have it right now. I don't have any enterprise customers pushing for activity, but this is completely on me and contingent on me getting funding from an all net, which I'd love to spend a couple minutes talking about actually, because. When it comes down to it, okay, I have this idea of interconnecting forums.

And I think, you know, joining this whole Fediverse idea would be fantastic, except who's going to pay for it. Right. Because if I spent, you know, hours and hours and days and weeks on it, I'm not getting paid for it. And so this is where, this is where NLNet comes in. Because the NLNet Foundation, um, gave us, gave me the runway to pursue this opportunity.

They, uh, they partnered with the, or they're working with the European Commission to start what is called the NGI Zero Commons Fund. And then they have, I think they've earmarked 21. 6, uh, billion euros? Billion? Million? I don't know. A lot of euros. In small to medium sized grants for, for R& D. of which this qualifies.

I was frankly surprised that it qualified, because I thought, you know, shot in the dark, I'll apply to this, uh, this grant program based in the European Union from Canada, which is not in the European Union, and we'll see where it goes. And then as it turns out, yeah, they love the idea, and they said, let's do it.

So, you know, they, uh, they want to build out open, sustainable, digital commons software for things like virtual and augmented reality. I'm reading here. Generative AI and intelligent mediators. So that's, um, specific to the NGI Zero Commons Fund. So

Jeff: now, is this in the core? Node BB software or is this like a add on that you

Julian: it will be yes, so For a lot of reasons and this goes completely contrary to everything I said at the beginning about plugins.

I It's it's very difficult to make activity pub integration a plugin because it integrates so deeply into the software At the same time we want to be able to turn off because not everyone wants to To have this software, so this is this sort of interconnectedness if you want to have your own internal, you know, knowledge base Why are you federating everything?

Federating everything? Yeah, right. It doesn't make any sense So we want to give people the opportunity to you know, turn on and off the functionality But yes, because it integrates so deeply with the actual database structure. So we had to put it in core. I Think at the beginning. I think it's always going to be Off at the start, and then you turn it on.

That may change.

Jeff: Well, when you were reading there too, and I almost hate to mention it, but I heard AI in there. Oh, that's what it is. What's the role of AI in NodeBB?

Julian: Oh, AI, that's a good question. Um

So I believe AI is going to be an inevitable addition to our digital footprint content everywhere. I honestly, I, I think I searched something online last night about reviews for something. And I honestly couldn't tell you whether the article I was reading was written by a human or not. Probably not.

Yeah. But what it means, what it means right now is that it devalues content. That's, you know, that's a net negative, sure. But you could look at it the other way, is that, if you have general content devalued because of AI, it increases the value of user generated content. At the end of the day, humans want to talk to other humans, uh, without a machine in between abstracting away the diversity of the human experience.

So, you know, it's not all do and gloom. Because as with a lot of technology, it's best used as an aid. Um, if you think about I'm just going to go out on a limb here and assume that you guys have watched star trek So how many star trek episodes hinge around a character like geordie laforge talking through a concept with the computer I think there's a lot of the episode.

Yeah, a lot of them where there's an episode where uh Picard is talking to Data, and he goes to theorize Data, about the whole idea, about how it could be that you have a spacefaring group of colonists that have spinning wheels in their cargo hold, right? He couldn't, as a, you know, as a human, Captain Picard couldn't, couldn't mesh those two concepts together, but then he turned to, um, you know, Data.

An AI in a sense and had them theorize and it's a whole, it was a wild idea back then because everything at that point was, you know, computers are tools we have, we have calculators, we have computers, you have word processors, we've got tools that are, you know, you put something in, you get something out, uh, but it is what you put in, but it has, it has huge implications if you consider AI as yet another opinion To your thinking process But Studying back.

So within a forum context, I predict that we'll see a push for sites that have only user generated content How we get there? I don't know But we might have islands with firewalls against AI content because not that it's not that we don't want it It's just in certain contexts. Maybe it's unwelcome. Yeah

Jonathan: You know, when I think about social media and I think about machine learning, the only place that it makes sense, like an algorithm at all, the only place that it makes sense is for discovery.

And so, you know, on Twitter or Facebook, whatever, when occasionally I get the thing that says, Hey, you followed this person. Here are some other accounts that you might be interested in like that. That's okay. That's good. I am okay with that. Or, you know, when you go to YouTube, let's say and YouTube knows all the things that you've watched and you go to the homepage and then you then get suggestions.

You've watched content like this before. So here's a new video that you might be interested in. Like that, that sort of discovery makes sense in a lot of these contexts. I could see, I could see that having a place in Mastodon. You know, some servers would have some sort of algorithm turned on that would then say you have followed these Macedon handles or maybe even in a forum, maybe even in a forum.

You're interested in these topics. We just want to let you know here's this new thread about one of those topics like that makes sense. But I don't, I don't really ever see a time when I'm going to be terribly interested in machine generated forum posts. I just, I can't comprehend that being that interesting.

Jeff: Well, I had a little thing at work where somebody, somebody brand new, young, young kid, sent an email that basically said, I'm done with this project, and then I'm taking, I'm taking lunch. And it was like three paragraphs long, and just, it was like, oh my gosh. It's like, I was, he wrote it with ChatGPT and I'm like, yeah, I don't do

Jonathan: that.

Oh, so I saw something. It was hilarious. This idea of somebody doing that. And then on the other side, there's this idea that you can also use ChatGPT to condense several paragraphs down to the bullet points. So it's like. We are, we are soon approaching a time where we use, we type bullet points. We use chat GPT to turn it into two paragraphs of prose.

And then on the other side, they use chat GPT to turn it back into the bullet points that we started with. It's like, why, why are we, why is this, why is this where we're at as humanity? What, what, what mistakes were made that brought us to this point?

Julian: Hey, at some point you can just send us an entire podcast to chat GPT and they'll just summarize it for you in bullet points.

You won't have to listen either. We

Jonathan: actually, we actually use. Service that turns it into a text that, that transcripts it. And then I know we at least, well, and then I know we have at least one user that will then feed that into an AI and get the short version of it to be able to get it in just a few minutes, rather than listening to the whole thing.

Julian: All right. So within the, within the forum context, though, I think there's the expectations that. Recency is what you're going to get. So every, the most recent topics are on top, right? And it looks wrong if it's subverted, but something that we can experiment with. You know, the whole idea of resurfacing content, uh, because of high engagement.

Of course, if you start doing that, then it opens a can of worms as to what qualifies. And so, you know, maybe we're not quite there yet, but we'll see. Um, the main, the main thing is that. Stuff gets missed when you sort stuff by time, you know, uh, I can be using Mastodon and all the stuff I'll see is from people who tend to post when I post, you know, I won't see a lot of content surface from Europe or from Asia because I'm asleep when they're active, right?

So that's, uh, definitely an issue.

Jonathan: One of the

Jeff: features I liked when I was looking at, I looked at for a list of, uh, forums that I could get on the, so I could just see what node BB looked like. You know, kind of going back to the more traditional forums. I like that, like, for example, one of them had, uh, gaps between the, uh, replies where you'd say, Oh, two years.

And then the next one, or because I'm on some. Regular type, you know, say motorcycle forums and somebody will they call it zombie

Jonathan: a thread necro thread. Of course, yeah necro

Jeff: So I like that that was pretty easy to see it's like wait a minute this is this thing

Jonathan: is 10 years old, you know maybe I better repost or And another feature I saw I thought was interesting was you could I found one where that people

Jeff: were forking

Jonathan: threads

Jeff: So you had a topic And then somebody would, oh, they forked it because the conversation, you know, sometimes threads kind of wander around and yeah, and it was easy to go, Hey, let's just take this off to another thread kind of on this side topic while continuing on the main topic.

And I hadn't seen that before in forum software.

Julian: So, yeah, the whole idea of forking threads is. Well, I'm not going to say novel. It's just been around for a while now. But it wasn't, it just wasn't supported in those old softwares. And so you'd have, um, the whole forum user experiences. People yelling at each other to stay on topic.

Because if you, you know, if you, uh, what is it? If you go off the rails or whatever it is. Then, you know, you lose the whole context of the original discussion. And then again, there are some forums that completely do this on purpose. Um, One of our, oh, they're actually not one of our customers, but one of our users, the Daily WTF, loves to do that.

A bunch of trolls on the forum. And so you'll start a, you'll start a forum topic, it'll go on for like 900 pages. And then they'll go through 900 different topics in the interim and they don't care at all. It's just how, it's just how it is.

Jonathan: Well, that's one of the, that's one of the beautiful things. We talked about this a little bit earlier, but it's one of the beautiful things about being able to Host your own forum if you if if you have a if you have a group of users that enjoy that sort of discussion Enjoy being kind of trolly to each other.

That's fine. It's your server. It is your forum knock yourselves out I think it's one of the beautiful things about these sorts of solutions and I do know what you mean

Jeff: about trolls Yes, because I've been on forums where here's how you started. Here's how you fix a 69 Roadrunner carburetor And then you jump to post 300 and they're talking about, you know, home canning or something like, wow, it ran off the rails.

Yeah.

Julian: Automotive forums specifically are fairly, uh, they're fairly popular still. I think, um, any, any new car model comes out, there's going to be a forum for it within days. And it's just, I don't know, there's one for each of my cards.

Jonathan: Yeah, I'm curious, this idea of forking threads, I hadn't seen that, it's really clever.

Is that, is that part of core, or is that a plugin, an extension on top? That

Julian: is part of core, that is part of core. It was useful enough. that even if you don't use it, I think the, it's fairly simple enough because you just take, because the way we structure it is that you have categories that contain topics and you have topics that contain posts.

And so what is forking than just taking a bunch of posts and putting it in a new topic? That's all it is. So it's not necessarily a hard thing to do. So we put it in core.

Jonathan: Yeah, that makes sense. That makes sense. All right. So we talked about the bug bounty and I know there is at least one bug that was pretty notable that got found through that bug bounty.

Let's take a minute and talk about that.

Julian: Oh. Actually, I have two. You have two.

Jonathan: Well, I said at least. At least one. Two is at least one.

Julian: Yeah, we need one. Yeah, we, uh, I, okay. Yeah, I don't like to, to broadcast the fact that we had critical, critical vulnerabilities in NodeDB, but. It happens. You know, this was, this was disclosed to us through our bug bounty program and we gladly paid the top dollar amount, which is not a lot, but it was a lot for us.

Jonathan: you mind telling us how much the, the top, the top value is?

Julian: Top, I think it's 512. I think we doubled that, 1024 for this one. Okay. Of course we use, you know, in multiples of two because we're total nerds. Of course. I love that. So, uh, this specific vulnerability centered on our utility function for generating UUIDs.

Okay. And I'm going to say something that I'm embarrassed by in that. Our function for generating UUIDs was stolen from Stack Overflow in 2013. The question, the question was originally asked and answered in 2010. And I stole it in

Jonathan: 2013. I must admit, my code, my code has comments of Stack Overflow links.

Because that's what, that's what the license for Stack Overflow says. It's, it's licensed MIT or whatever, but there's this one modification that to be able to quote it, all you need to do is include a link to where you got it from Stack Overflow. And I have that. All over the place with my code.

Julian: And so, and so this, uh, I think it was like four lines, this little piece of code, completely black box.

I don't know what the heck it does. It was updated in 2015 because to address the fact that math dot random in JavaScript is not actually random. It doesn't have uniqueness guarantees specifically. And so funny thing about that, that happened in 2015. I stole the code in 2013. I was never updated that this changed.

I was notified in 2022 that this was an issue. So an analyst was able to use a specially crafted script to repeatedly call our password reset route, which uses that UUID generation. And he was able to programmatically determine the reset code of another user's account. Like an admin. And so, um, so that's account takeover, uh, no user intervention.

That's 10 out of 10 critical severity.

Jonathan: That's bad. Full bounty. That's real, real, real bad. That falls in the real bad category.

Julian: Real bad category, yeah. So when we, when we just, when we stepped through the specific logic, it was completely wild how they managed to do this. It was just chaining different vulnerabilities, uh, and making all the holes line up, until they could, you know, take over an admin account.

And that's why we have a bug bounty program for stuff like this.

Jonathan: Yes. So you said there was two. That's one of them. What was, what was the other wild one?

Julian: Nah. The other one was an object prototype vulnerability. So, JavaScript objects have a, uh, have a bunch of properties that come with it. Right? And so, in our code, we did a naive check, uh, to see if a property existed.

Just a truthy check. Because truthy is a word that only exists in Javascript, by the way. Ha ha ha ha ha. Well, actually, I think it exists elsewhere, but It's a few other languages, but yes. So a bad actor was able to use this oversight to set a value, because I think constructor is one of the values, and constructor.

assign is one of the methods that you can call. So, uh, they used this to Set one of the variables on the server side that we persisted between calls. One of those variables is the user ID. So, um, they were able to make this specially crafted call to overwrite the user ID to an admin. Privilege escalation, no end user involvement, 10 out of 10 severity, full bounty payout.

Yes,

Jonathan: yes, also in the real bad category. Yeah, fun

Julian: times. Yeah, those were fun ones. Again, just things that you don't normally get bug bounty reports about. And it's just, you know, you spend a day working on this, your mind is blown by how something, the end actual fix was one line. When we, you know, when we instantiated the object, instead of just using curly brackets, you use object.

create. One line. Yep. Thousand dollars. Yep, that's, yeah, that's

Jonathan: pretty typical. That's pretty typical. Alright, so let's see. Um, we are just about out of time, which is unfortunate because it's been a fun conversation. Um, I want to ask a few closing questions and let's go with this one first. Uh, what's the weirdest or most surprising thing that you've seen someone do with NodeBB?

I imagine there's a lot of these and you're now trying to figure out which one is the most interesting.

Julian: Yeah. No, there's been a lot of weird stuff. I think that the weirdest one is just when we just, like I mentioned earlier, when they, when they, um, came up to us and said, yeah, we're, we're going to just not use your front end.

We're just going to use you as an API. I think at that time we actually didn't have an API. A public API, it was just sort of, they reverse engineered our entire API and they built a front end on top using their own front end tools, which was frankly insane. Um, and we've got people talking on our community forum right now.

It comes up every now and again that people want to use their own. frameworks, um, instead of our own weird, you know, jQuery who, uh, frame on the front end. And so they want to build their own front end and they want to decouple the back end from the front end, which is possible, just fairly difficult to do.

Um, and so people are working on that independently of us, which is, which is always exciting. Are

Jonathan: there any websites to do something fun, like use NodeBB as the comment section for individual articles?

Julian: Oh, that's not weird. You can do that. Yeah. Yeah, we actually have a plugin for it called nodeDB plugin blog comments.

Nice. Yeah, I like it. Although with ActivityPub you won't actually need to do this anymore. You can just plug it into the Fediverse and then you have your comments. Cool. That's

Jonathan: cool. Alright, so is there anything that we neglected to ask you about? Is there anything you really wanted to cover that we didn't get to?

Julian: Oh, I can go on at length about how social media has failed us, but that's a conversation

Jonathan: for another podcast. We got into some of that. We got into some of that. All right. So last two questions I've got to ask you, Ben, or else people send me emails. What's your favorite scripting language and text editor?

Julian: Oh. I'm going to get hate mail if I say JavaScript, isn't it? Nah. Probably. You can say JavaScript. I expect you to say JavaScript. I love JavaScript. Yeah. I mean, with all its warts and, you know, eventually we might end up moving to TypeScript. We're having those conversations, long running conversations. The best conversations about moving to TypeScript.

Um, as for an IDE, I personally use VS Code, but it would be unwise for me to not mention Notepad because they're actually one of our customers. Ah. Um. Yeah, Dan, Dan Ho. What an interesting, interesting fellow. He, uh, I don't know if you remember a couple years back, he, um, He gets a lot of hate mail for his software, and then he, uh, He basically put out a form saying, If you want a refund for your software, just put your name on this form.

Of course, the software being free. I don't think the form went anywhere, it's just When you have a developer with a sense of humor, it's always very exciting. Yes, yes,

Jonathan: that's right. That's fun. Uh, we, I forget who it was, somebody made the point a long time ago on, on Lost Weekly that you, JavaScript was terrible when it started out, but because you have had, you know, the Microsoft implementation of JavaScript and the Firefox and the Google, and now the Node.

js, the overlap of what all of those different pieces of software support, that is what JavaScript has become, and that's actually okay. Like, that's become pretty decent over the years.

Julian: I think you can build pretty good software on Javascript. I, well, apparently. I mean, apparently. I mean, I have a copy of Javascript, the good parts.

It's, you know, a book this thick. But, uh, yeah. I still refer to it from time to time. Yeah, there you go. Or we're calloused

Jonathan: up. Okay. Alright, well, hey. Thank you, sir, very much for being here. We sure appreciate it. It was a fun conversation to have. Appreciate you being here.

Julian: Thank you very much. Happy to be here.

Jonathan: All right. What do you think? Jeff?

Jeff: I think, uh, that software is really cool. It's, it's, it does take so much the forum software that we've had for I'll say decades. For those listening, I've got a pretty white beard so I've been playing around with this stuff for a long time. As we said, don't do the math.

Yeah, don't do the math. And, you know, it really, like, the, the You know, the, the Fediverse, the, you know, the ability with all the different plugins, the, um, forking showing how long between replies. I mean, it really, I think it's really innovative and doing something that on all the forums I'm on, I don't, you don't see any of that.

It's kind of the standard. Somebody posts a topic and people put, you know, replies under that. And I mean, there's even stuff like I'll, I'll get to the bottom. And there's no button to go up or something, you know, I'm like, really? I mean, some basic type features that just aren't existing that you say. Why, who didn't think I'd want to scroll up, you know, I don't, I don't want to use the mouse wheel to go up or have to grab the border, you know, where's, where's my button

Jonathan: to go all over, you know, yeah, yeah, no, it's fun.

I'm, I'm glad there are, there are people in projects out there that are sort of bringing this idea of the forum, uh, into the future with us, um, because there's, there is something, there is something special. I think about the way forums work that none of the other solutions really, really quite capture.

Um, so that's cool. And then that they're, they're integrating with the Fediverse that just, um, That just makes my, uh, decentralized, uh, kind of almost anarcho capitalist, decentralized nut inside of me get really, really happy. I like that part. Well, and I think

Jeff: forums are going to be around because honestly, I think that's one of the best ways to catalog information by various threads.

And you can go into them because like we talked earlier, you look at Facebook and try to find something or, you know, Twitter, you scrolling around, it's like, oh my gosh, good

Jonathan: luck. Yeah, no,

Jeff: and especially when the algorithms are playing around and oh, you haven't seen this. Let me shove this in. It's like, no, I'm trying to look at what I had before.

And you can't. We're we're a forum. It's all right. They're easily, you know, searchable indexed.

Jonathan: Organized. Yep. Absolutely. Absolutely. All right. Well, very fun. We will have to, uh, maybe we'll have him back when the, uh, When the Fediverse stuff is finally fully cooked We'll have him back and talk about how it works in the end and come up with some new ideas for how to How to do new fun things Um, so that'll be great.

I want to let you know next week. We have Herbert Wolverson and we're talking about Rust Maybe I will get a crash course in coding Rust Maybe I'll do some Rust coding live on the stream that would That'll be, that'll be fun. That'll be interesting. Um, but yeah, that is, uh, that is next week. I'm looking forward to that.

Jeff, you have anything you want to plug before we go? Uh, if

Jeff: you enjoy me, just catch me over at the Untitled Linux Show. I'm

Jonathan: there almost every week. Yeah, and uh, ULS now has a free to the public audio stream. And of course, if you want the video, if you want to be part of the live chat, that is on the Twit Discord, which is part of Club Twit.

Which is, uh, a lot of fun. Um, as far as me, you can find my work also at Hackaday, with the security column, goes live every Friday. I'd love to see you there. Uh, and then I've also got a YouTube channel. Doesn't, there's not a whole lot going on there, but we just recently did a couple of Mesh tastic videos with some more to come.

And so, uh, that's a lot of fun. You can check that out. Find me over at, uh, on the YouTubes. Well, thank you everyone for being here. We had a few live in the chat. Appreciate that. And for everybody on the download, and we will see you next week on Floss Weekly.

Episode 772 Transcript

28 februari 2024 | min

FLOSS-772

Jonathan: This is Floss Weekly, episode 772, recorded Wednesday, February 28th. Raspberry Pi from the man himself.

This week, Elliot Williams joins me and we talk with Evan Upton about the Raspberry Pi and the Raspberry Pi Foundation, a potential IPO, and we get some feature requests in for some future products. You don't want to miss it, so stay tuned.

Welcome to Floss Weekly, the show about free Libre and open source software and hardware. Well, for today, it's about hardware too. It's not just me, of course. We've got Elliot Williams, the editor in chief here at Hackaday. And Elliot, what is this madness?

Elliot: What is this madness? Um Which madness are you speaking

Jonathan: of?

Well, it's, it's Floss Weekly and we have, we have Elliot here. We have a Hackaday person on Floss Weekly.

Elliot: Well, it's Inception because Floss Weekly is now on Hackaday.

Jonathan: It's, it's Hackaday all the way down. It is, it is. Well, today we've got a super special guest and this may explain why we've got Elliot here.

We've got Eben Upton of Raspberry Pi Foundation, the Raspberry Pi the business. Um, pretty much just All around the Raspberry Pi guy, and, uh, sort of one of our heroes here, I think. One of mine, at least. Um, I'm not sure if I want to bug him about looking like Jason Statham. I feel like that joke has sort of run its course.

Um, he gets it a lot, I feel like. Um, Elliot, you're obviously, I'm sure you have probably 15 Raspberry Pis within arm's reach, right?

Elliot: I think 10 is probably about the right number if I'm looking around here, but it might be edge and close to 15, especially So, yeah, if we get to play Good Cop, Bad Cop, or Software Guy, Hardware Guy, I get to play Hardware Guy, and If you count the microcontrollers, if you count the recent RP2040 microcontrollers, then 15 is a ridiculous underestimate Yeah But, uh You know, just Raspberry Pi, single board Linux computers, probably a dozen, good dozen.

Jonathan: Yep. Yep. Well, let's not, uh, let's not dally anymore. We've got the man himself. Let's bring Mr. Eben Upton onto the show. Welcome, sir.

Eben: Good to be here. Now, is it, is it Eben or is it Eben? How is that? It's Eben. It's Eben. It's like Ebenezer. Okay. It's I, I'm the most miserable guy at Christmas. I'm not Ebenezer, but it's pronounced that way.

Jonathan: You're, you're, you're, you're Ebenezer. After the

Eben: story, is it? Uh, yeah, so I'm Ebenezer after a bunch of old Welsh guys. Uh, who my dad met in Wales while my mom was pregnant. So I'm, I'm, I, I have a, uh, it's a good, it's a good solid, Wales is a, so Wales is a Christian but not a Catholic country. So they want biblical names that aren't Satan's names.

And there actually aren't a huge number of those available. I imagine. And, and Ebenezer is one of them.

Jonathan: Ah, I like it actually. Um. All right, so, we're here, we're here to talk about Raspberry Pi, and normally the first question that we ask people is to give us the 30, 000 foot view, like, what is your project about, for people that don't know about it, but at this point, I don't think we have any listeners that don't know what the Raspberry Pi is, because it's, it's everywhere, and it's, it's sort of in everything, and yes, we were not kidding when we said that we have just, Pis around, from, from the one to the four, um, I've even got a Raspberry Pi 5, the newest one over there.

And let's, let's talk about that for a minute. Um, I don't know, maybe give us the quick sales pitch. What's the, what's the upgrades in the, in the Pi 5? Like, why is, why do we want to reach for that one over the Pi 4?

Eben: Well, this is, this is kind of a fun day. Almost a fun day to be talking. So, so history, history fans will, will remember that we launched the product on the 29th of February.

Um, uh, the 29th of February, 2012. Um, so, so tomorrow's our third birthday, which is super exciting. We've come a long way in three years. Um, and I guess the pitch for Raspberry Pi 5 is, I think that's how it works. I think, I think, I'm pretty sure that's, I'm pretty sure that's how it works. We

Jonathan: did math behind the camera.

Yeah, yeah,

Eben: yeah, that's it. Um, so, um, I mean, the pitch really is, is more. I mean, the interesting thing about Raspberry Pi is over the, over the kind of, the course of the kind of, the history of the Raspberry Pi single board computer. Um, uh, I guess, qualitatively, the board hasn't really changed very much, right?

It kind of does the same thing, right? It's got some memory, it's got some processing power, it's got the ability to drive a display, it's got some USB, some GPIO, Ethernet. Um, the only real, um, qualitative change to the board was in 2016 with Raspberry Pi 3 where we added wireless, right? So we added, we added Wi Fi and Bluetooth.

Um, and pretty much everything else that's happened to Raspberry Pi on the hardware side over the years has just been a question of more. Uh, and I guess Raspberry Pi 5 kind of continues that tradition of more. It's about, um, it's somewhere between, it's on the three end of, we say two to three X, the performance of a, of a Pi.

four. Uh, I think it's kind of on the three end for most, for most use cases. It's kind of on the three end of that. Um, that makes it about 100 and depending how you count, it makes about 150 times the performance of the, uh, of the 2012 of the February 29th, uh, 2012 product. Um, so it's kind of a, it's kind of a further turning up at the dial.

Um, I guess in terms of, you know, we've always seen ourselves, you know, The majority of Raspberry Pis are used in, uh, in sort of, uh, in embedded, what you consider to be embedded applications. But they're still, because of our history of trying to get young, you know, young people excited about computing, um, there's still a real, uh, real history of us seeing ourselves as a client PC company.

We've been calling ourselves a client PC company since that very first device. I guess this is the one, if you think Raspberry Pi 4 was the first device which was really credible, particularly for web browsing, was really credible as a, um, I guess, uh, uh, some compromises, um, client PC, probably at least for me, Raspberry Pi 5 is the one that pushes you over into not noticing that it isn't a regular.

that isn't a regular PC. So those, although it's a kind of this kind of continuous increase in, um, in performance over the 12 years, um, there are kind of milestones. It sort of feels there are milestones that you go past. And that's probably for me as the one where when I sit down in front of one 10 minutes later, I'm like, Oh, I'm using a Raspberry Pi.

Um, that's probably the milestone that we went past with this, with this product. Yeah.

Jonathan: I, I ran an experiment when the Pi 4 came out and used it. As my main desktop for several days, if not a week. And there were a few things that would drive you nuts about it, but for the most part, yeah, it just worked. Um, you talk about milestones, though, and there's something, there's something in the Pi 5 that I think is maybe a game changer in the same way that adding Wi Fi was, or maybe even more than that.

And that's the, uh, the exposed PCI Express port. Yeah,

Eben: I mean, that's, that, that is, you know, you're right, that is a huge change, and it is a, I guess that is a, that is a genuinely A genuinely new feature, obviously in, in the PI four generation, you could have PCR Express if you were using the, uh, we used, we had a single layer of PC Express Mm-Hmm.

That was on the SPC product. Mm-hmm. Was connected to the, uh, via labs. Um, USB, uh, X-H-C-X-H-C-I, uh, uh, controller and hub. Uh, and if you're using the compute module for, you could, uh, you would have access to that, that chip wasn't on there. You had access to PCI express, I guess the thing that's happened in the PI five generation is we have enough PCI express that we can both use it on the board.

I guess we'll talk about RP1 in a moment. Um, so we can use it on the board to, to communicate with the I O processor, but we also have a lane left over that we brought to this connector. Yeah,

Jonathan: when I first saw that, I was a little disappointed that it wasn't broken out as an NVMe. On the, on the Pi 5 itself.

I think I've kind of warmed up to your approach, because it does give a little bit more flexibility.

Eben: But, what was the, what's the reason? Yeah, and the nice thing is, is, is, you know, you talk about flexibility, there is that sort of feeling that, you know, we haven't got our first party, there will be a first party adapter board that we trailed.

Um, uh, fairly, fairly soon. We haven't got ours in the market yet. But there's a bunch of other people, and what's interesting about the other people's is, yeah, all of these different, some people have made ones that go underneath. Some people have made ones that go on top. I saw some people who've made one that will take two, has two M2 sockets on, so you can, so you can raid, and it has a, um, a switch, has a PCI Express switch on there, and you can raid, raid a couple of devices together.

Um, so there's, so it's interesting that, that there is some Yeah, it happens all the time with the Bury power route. We, we try to do the, kind of the minimum. We try to do the kind of the core product, the thing where we think we, we add some value. Um, and then you see the, the community, the ecosystem, the business and enthusiast ecosystem around the product kind of go and then explore what can be done with Mm-Hmm, with the features we provided.

Yeah.

Jonathan: That is, that is. Super interesting stuff and I'm hoping we get to talk a little bit more about that the third party stuff But there's there's there's a pair of questions that I really want to get in and I'll hand it over to Elliot Let him ask some but let's talk about the the so I've got a cm4 here I understand the cm5 is coming which I'm excited about that.

Yeah And then what I've not heard about is, is there a Pi 500 coming? And please tell me that it's got an NVMe port in it.

Eben: Um, so I, um, let's see, we don't talk about unannounced products. Of course we do talk a little bit about unannounced products in this generation. Um, I think the, I think what we all say, I don't know if you've read, there's a book by Neil Stevenson called Cryptonomicon.

Um, uh, in which, you know, somebody challenges the hero about, I think they're laying fiber optic in the Philippines. Uh, uh, and, and she says to, she says to me, you're gonna do, you gotta more fiber optic in the Philippines. And he says, well, people in business seldom plan to do something once because it messes up the spreadsheets

Um, and I, I wouldn't wanna mess up the spreadsheets. So I think, you know, there are, you know, obviously, you know, we, uh, PI 400 has been a really. It's been a, it's been a really successful product for us, actually. I mean, it's a, it's a, it's probably the clearest, um, it's the clearest hardware instantiation.

If you think of all the investments we've made in Raspberry Pi OS as being the kind of software instantiation of our belief that we are a client. PC. Um, actually going and building something which is unambiguously a PC and a keyboard, unambiguously a client PC. That's the kind of hardware instantiation of that feeling that we have about ourselves.

Um, and, and so I think it would be surprising if we didn't, we still feel that way. So it'd be surprising if we didn't try and do something in that space at some point.

Jonathan: So I've got my Pi 400 here, which happens to have this little cyber deck attachment from Adafruit, which is great. Um, So two things about this, if I, if I, since I have you here, if I were to give you a wishlist for the Pi 500, and of course, I'm sure you have a lot of people that ask you.

Let me get a pen. Okay. It's only, it's only two things. It's only two things. First off, I would love to see a real NVMe port on it somewhere because running SD card is just a painful experience. And two,

Eben: I would love. Depends on the, depends on the SD card, but yes. They're not all created equal, but yes. I've bought the ones that are

Jonathan: supposed to be nice and still it can be a painful experience.

And then two, so many people do stuff like build cyberdecks out of them. I would love to see some built in mounting options, like maybe a VESA mount on the bottom, or some threaded inserts on the back of it, so that you can hang hardware off of it and be a little less janky. Um, so those are, those are my two wishlist items for the PI 500.

Eben: Well, I I've taken a, I've taken an, I've taken a virtual, I've taken, I can, I do still just about have a functioning memory, medium term memory. Um, so I don't know that. I mean, that's, you know, that's good input. And certainly the NVMe thing, you know, the, the. presence of that PCI express lane is a really, it's a really important part of the platform.

And it's something that obviously will be exposed on, um, uh, on CM five. We we've, we've released some forward guides. Well, we don't talk about future products for CM five. Obviously we've wanted to, you know, because of the lead times associated with particularly, when people are designing baseboards, one of the, the, the, um, the sort of direction of thinking was, well, people are designing CM4 based products all the time.

Uh, and if we know some stuff about what CM5 is going to be like, and it is going to be very much like CM4, but you know, there are always changes from generation to generation. We should totally write those things down because it could be that 10 minutes of work from somebody today will mean that they had, they designed a product that you can drop a.

Yeah. Um, that you can drop a, uh, a CM5 into, uh, rather than having to spin a PCB. So there's a, if you go to the, the product information portal PIP, um, there's a, there's what we call CM5 forward guidance, which is really kind of notes for CM4 baseboard designers to maximize compatibility. Sure.

Elliot: Cool. Yeah.

Speaking of CM4 and CM5 and the fours and the fives, one of the things I noticed When I started playing around with my five was that it kind of does need the fan And that's just I guess a consequence of well, it doesn't need it And this is something actually that I've always loved about Raspberry Pi You can

Eben: run them quite good at not melting and the

Elliot: and the old four the four and the cm4 I Played around overclocking them And, you know, you can do stupid things, like overclock it with extra voltage and no cooling whatsoever, and the thing just throttles itself down.

It doesn't burn up, nothing bad happens, but as long as it can run, it does run. And you throw a tiny bit of aluminum on it, and then it runs

Eben: beautifully. I mean, that's the, that's the interesting thing, because they're fully throttled. Yeah, Pi 5 is so much faster than Pi 4. They're fully throttled. Pi 5 is a lot faster than a heavily overclocked Pi 4, right?

Exactly, it still runs, yeah. And so, so my daughter, I have a six year old daughter, and her first PC is a Raspberry Pi 5 on her, on her desk in her bedroom. Um, and because I'm a lazy dad, I haven't actually got around to fitting a case or an active cooling. It's just been sat there on the sides for the last, like, 12 weeks.

Um, and, and she doesn't notice. And honestly, I mean, she doesn't put a finger on it. But, um, uh, but you know, she, she doesn't notice. Um, she did pile a bunch of clothes on top of it the other day. Um, uh, I'm looking at thinking, you know, I am glad that we made those investments in the, in the thermal throttling software.

Um, but, uh, but you know, it is a, it is a platform which is designed, when people say, uh, uh, you know, obviously the platform performs better if you, if you want to run it under heavy load, you want to cool it because it'll stop it from throttling. But there is this, I probably, I have this, When people say, there's a problem with Raspberry Pi because it gets hot.

That's not quite right, but it's designed to get hot because that's how thermodynamics works, where you get there's no way to pack so many

Elliot: gates into such a small space without

Eben: it getting warm. Yes, the way thermodynamics works is you can only shed heat to your environment if you're hotter than your environment.

Um, and so this is a, you know, people and people say, oh, the CPU has got up to 80 degrees. Well, 80 degrees is broadly nothing by silicon standard. So the silicon is closed to 125. Right. Um, the, the PMIC I think out, the, the PMIC probably out to 150 degrees, certainly the core logic to, to about 125. So you're not even beginning to explore the edges of the operating regime of the design at 80 odd degrees.

It's really about, it's really about finger safety. It's really about, about not getting so hot that you can't put your finger on it and then get your finger off it before you burn yourself. Right.

Elliot: And we were talking also about the Pi 400. I love your idea, Jonathan, of putting VESA mounts on it, because I actually use, I've, we have two Pi 400s in the house.

One is my son's first computer. So mission accomplished. That worked out very well. My wife actually bought it for him without even consulting me. I thought that was kind of sweet. The other one is in the basement driving my CNC machine. And. They're perfect for that too. It's again, it's kind of light duty work But then I can fire up a browser and download stuff and absolutely beautiful for that sort of application Except it's just sitting there on the desk and it would be so awesome if it had screws on the back but now that you mention it john then i'm just gonna Make a little base plate for it and glue something

Eben: to it.

That's where would you put the screws? Would you put the screws in the back or would you put the screws in the base? I guess it's honestly I'd

Jonathan: love to see both So my my

Eben: thought was that so I saw it so I know how much threaded inserts cost So I'm gonna ask you to pick one.

Elliot: I put the threaded inserts in the in the red underbelly in the raspberry colored underbelly I would 4.

Jonathan: I would say whichever is easier because people can make it work either way so one of the one of the things that I'm thinking of is like you can then put a you get by a by a little LCD from Waveshare and Put it on there make it hinged and make your own little tiny laptop out of a Raspberry Pi 500 And you can make that work from either place So I guess probably the bottom is where it would make most sense Um, to be more

Eben: mechanical, more mechanical strength to play with.

Right, right. Probably in terms of the layout. Interesting. Okay. Look, that's great input. And I will go and have a chat with, I'm going to have a chat with people and if and when we decide to do a PI 500, that's, that's, that's a useful, useful bit of data.

Jonathan: Non binding suggestion,

Eben: but I like I mean, I'll be disappointed.

I'm not sure I signed anything before this If I have signed up to implement anything you tell me then I will Board of directors will be a little bit disappointed with me I think the

Elliot: other thing that especially has been hitting hackaday has been the RP 2040 and You know, so, I see Raspberry Pi moving in these kind of two different directions at the same time.

There's the obvious evolution, like you say, of the, of the single board computer version. But then kind of out of left field comes this, uh, microprocessor, this microcontroller with just, from, from my perspective, absolutely stunning, unique, and interesting peripheral set. Um, and I actually used it on our badge for the Hackaday super conference and I made extensive use of the PIOs and the DMA stuff and made this crazy like six stage pipeline of one PIO feeding into the other and I actually did like decimation and bit flipping and stuff in it and it was tremendously useful for, you know, things you'd otherwise use programmable logic for.

Eben: I love that somebody, somebody built a pro. It's two favorites. One, somebody did a, I think a Commodore 64, um, external cartridge, um, uh, which, which had no CPU. Both WFI. Um, and, and it was doing all of the, you know, grabbing the, uh, you know, grabbing the two phases of the, grabbing the two phases of the address, getting, you know, going and doing the data fetch from memory, um, you know, effectively DMAing.

Aggregating them. DM aing them out into a DMA control block, triggering that DMA to go get the data from a 16 k line, bit of memory inside the device, then in presenting it on the bus and the whole thing was done using this, this kind of inter interlocking set of DMA of DMA engines and of PIO state machines.

Yeah. Um, which is kind of, it's kind of fun. And then somebody obviously inevitably did a processor with it. I think I, I remember that writeup of someone managed to do a kind of. AVR rate level of performance, uh, kind of a processor, which was just built out of, uh, of DMA training.

Elliot: Oh, fun. Yeah. I mean, for our application, it was superb because it did all the like high bandwidth stuff for us.

And then we had both cores free for people to play around with. And one of them was running kind of system stuff and the whole other core was just like here. You, you write your code on it, people at the conference, go have fun and that was a beautiful separation and

Eben: it was just lovely. Yeah, and if you, if you sort of think, I mean, you know where, so Raspberry Pi kind of comes from originally from this kind of, it does come from an AVR8 world, right?

So the first things I built in 2006 were kind of, um, can I build an 8 bit computer, an 8 bit equivalent level of performance with, um, Um, just with an AVR8 and a bit of SRAM, uh, and you know, can I, can I generate video addresses and can I, you know, it's kind of built this whole world on this piece of error board, right?

Um, and then this was sort of, I'd spent probably a year, six months maybe, uh, after the end of my PhD, just kind of, I was kind of burnt out, kind of just futzing about with, um, with AVR8. And it's a great architecture, right? Because it's very, it's, it's got high level of performance for the. for the price for the time and for the price it has.

Um, it has very determined. It has very low latency access to the real world from code. Um, and so there's always that sort of feeling that I could go and buy an and gate, uh, or I could just program an 80 tiny to read two pins and them together and then out them to another pin, you know, so not about and gate actually.

Um, so you've kind of got that kind of very. Low latency access, um, to the outside world. You also have very deterministic, um, access to the outside world. So you actually can cycle cam. So I, my toy was generating video signals and generating video signals out of an AVR is kind of a popular, a popular hobby.

Right. Um, and so really when you look at RP2040, what's the genesis of RP2040? It's like, well, Um, can we recreate some of that immediacy and determinism and high performance interfacing in a 32, in a modern 32 bit environment, in an ARM based 32 bit environment? And so you think, okay, well, well, okay, let's, let's put a M0, let's put M0 plus down and let's connect it at low latency, use the SIOB, um, to connect it at low latency to the GPIOs.

Okay, that's quite nice. It's quite a nice thing. Well, um, but one of the things that sucked when you were doing this AVR, well, it was very informed by the 2006 Raspberry Pi, actually. One of the things that sucked was that you would only do your, run your application code in H blank and V blank. Um, and you'd, you'd be there all the time in, in, uh, during the the display period, just generally video addresses.

So well, okay, perhaps I'll put another call down. That could be the application is back to your idea of having one call running. infrastructure maintenance stuff and one core running, uh, running application codes. Let's have an application core and let's have this go. Well, okay. But they're going to contend if they're both addressing the same memory inside the chip, they're going to contend.

So let's put a bunch of different banks in a bunch of let's split our SRAM on the chip into a bunch of different banks and let's put a crossbar between them so that you can, the two, as long as you're not addressing the same bit of SRAM, then you, the two. The two concurrent run concurrently. And so that's kind of like, that's the overall structure, two processes, um, uh, fab, um, multi ported, uh, multi ported sram, and then fully connected crossbar in the middle.

And, and that's an enormous amount of bandwidth actually. You, you add up all the bandwidth things like the chip, there's almost as much bandwidth inside an RP 2040 as there is an A 28 35 as there is in the original. Raspberry Pi application processor. And this is kind of a bunch of application processor guys came rocked up and designed a microcontroller, right?

Um, and then the PIO is the last bit where you say, Oh, it's a bit of a waste of an M zero plus to be bit banging a UART, isn't it? Um, I wonder if we could make a magic and of course Everybody, perennially, and this must have happened a thousand times in the history of, uh, of, of computing, computing hardware.

Everyone tries to design, oh, that machine that could be a SPI or a UART. Um, but the interesting thing about PIO is because we had so many people work on it over such a long period of time, and they were all bringing their own contributions to hardware people, software people. What you ended up with was something which is incredibly abstract and flexible, um, and had been used in house to do probably 10 or 15 protocols before it went out in the world.

And then, um, And it went out in the world and it had been used to do DVI. Um, it had been used to do DVI inside the organization. Um, but the, the, the, the moment I knew that we had something special with PIO is about a week after launch, someone took the DVI example where, you know, DVI is differential. Um, so there's a piece of software that occupied most of an M zero plus.

Um, that, um, uh, converted, basically just bit doubled, just converted 0 to 1, 0. Um, and, um, someone took the example and said, yeah, but you guys have got this thing, there's a feature called sideset in PIO which lets you set some pins. In addition to pushing data out, you can also just set some pins. Um, and what they said, well, you've got zero latency branching.

Um, and you've got sideset. So he said, you can just have a two line program, which does, does an out. Um, it does, uh, it does, uh, it does a branch. It does a computed branch. It consumes one bit from the input bit stream. Does a computer branch with your instruction zero instruction one. Um, and then the first instrument, they're both jumping, they're both computer branch instructions.

And the first one has a side set of zero one. And the second one has a side set of one zero. So the program is just lashing backwards and forwards between the two and all of the work is done by the side set. And that was the point where I'm like, this is not something we thought of doing. And yet someone has found a material improvement to our, what we thought was a pretty cool example, right?

Um, someone's found a material improvement to it. So, so PIO, PIO, we quite, we quite love PIO actually. I did

Elliot: this, I, I did this I2S thing with the clocks as well and, you know, there you need a really high speed clock and then you need two other clocks that are divisions of it and the sideset pins are absolutely perfect for that to shove data along through on these clock timings.

It was, yeah, it was a piece of cake. I mean, it took, it took me, Oh, it took me a few days to get my head around the assembly language, which is really neat and interesting and very compact and but also very powerful once you see a few examples and um, A lot of people have worked, got good, worked examples out on the internet, and I, I custom cobbled together an I two s thing that actually worked.

Eben: Yeah. And there were some, and you know, there, there are, there are some wonderful examples of, I mean, for me, 6 5 0 2 assembly language. I, yeah, I, I I entered your one K contest. I, I, I, it's, uh, I, I, I, I, you know, I love these, I love these, um, probably 6, 5, 2, 7 language, actually, the, the shader. Uh, processor instruction set for video core four, uh, which I designed, uh, and have written some substantial programs in, um, uh, and also the other one I really enjoy is Pentium.

One. Um, it's Pentium one UV, UV pipelining where you have the, you have the, or you have, it's super scalar, but in a very, it's kind of a very fixed super scalar where you have two pipes execute synchronously with each other. Um, and they're all examples that in PIO code, they're all examples of places where you kind of put in effort and get Rewards, cause it's a little serotonin, little serotonin cookies.

Um, 6502 is great for this. I mean, you literally will look at, um, I used to, I had the privilege of working with Sophie Wilson at Broadcom for a long time, who, who wrote BBC Basic and then designed the original arm, Instructions and Architecture. Um, and the story was with her. BBC basic ROM, there would be one 16k ROM and there'd be one byte left and someone would report a bug to her and she would work all day and at the end of it she'd fix the bug and there'd be one byte left.

Um, and, and, and, and it's just that anything that gives you those, you sit there for half an hour, ah, a byte, serotonin. Um, I just love it. And PIOs like that. It's wonderful to have made. A thing, or to have been around, actually, while such a thing was made, I think would be a fairer, a fairer attribution of credit.

Jonathan: So, it seems like this would be a good place to talk about the other sort of PIO solution, the RP 1. Kind of a, kind of a big milestone, I think, for Raspberry Pi. And, boy, an interesting little chip. Let's chat about the RP 1 for a minute.

Eben: It's, it's fun, right? So this is a, uh, so for the first time we have a big raspberry pi, so, uh, for the first time we have a big, um, raspberry pi, uh, with, uh, Some Raspberry Pi silicon on it.

Uh, this is RP1. The clue, in terms of you think about how long these programs have been going on for, the clue's in the name. When, when we brought RP2040 out, it says RP2 on the box, and people thought that meant we're short for RP2040. It's not, it's the second, it's the second chip. Uh, so this only took four, RP2040 only took two, only took four years.

Um, um, uh, RP1 took, took eight. Um, uh, and really, you know, what's the concept here? It's that you can Uh, you, you know, if you want to keep making Raspberry Pi faster, um, you have to go down Process Node. Um, and the problem with going down Process Node, of course, is that, is that your analog interfacing is basically big power transistors.

Um, and they don't scale very well. A, they don't scale very well. B, they're a pain in the neck. On any process node, it's a burden. So you have this big burden of I. O. Particularly like the MIPI I. O., for example. Um, that you, you, you, you drag down. You know, making 3v3, nice ESD tolerance, 3v3 GPIO patch, things like this.

You have, you have this burden that you drag after you as you go down process node. And so the kind of idea is, well, what if we took all of that interfacing, analog stuff, put it on a friendly old process node. And 40 is just the, I love 40 nanometers. It's, TSMC40LP is just the friendly, it's like a warm bath of a process.

And it yields so well. I remember the first chip I ever did on 40 nanometers yielded 1%. Test chips all day, you get one, two, two. If you, if you test them quickly, you get two chips, uh, two working chips at the end of the day. But like, right, right now that was 22, that's 2009. So we're 15 years later. This thing's like a warm bath.

So let's just do all of our analog on this lovely old process node with our friends at TSMC. Um, and then let's connect them by PCI express to the, um, uh, to, to, to a core logic chip. Uh, and then, then that core logic chip can run down the process. So in this case, you know, Broadcom did us a chip on 16, which is, which is lovely.

Um, and it's a. I mean, what this is is a chiplet architecture, right? We've been doing this for long enough that it predates the word, the word, the word chiplet. But what you call it today, usually with a fancier interconnect. I mean, the interesting thing about chiplets is usually, usually people out use some chiplet specific low energy pipette interconnect.

But this is basically a chiplet. And the advantage of using a standards based chiplet interface is that you can build the two chips in different organizations. That you, you know, we just agree. So we had uh, RP1s plugged into x86 PCs. So I have PCI Express card with, with a RP1 on. And so you could do all your development against 2711 or against a Xeon, a shiny, shiny Xeon.

Um, and then I know it'll work. And then at the other end, of course, you've got a program developing 2712. Um, and then you can just say, well, look, we're just going to make a standards compliant PCI Express interface. And we know that your standards compliant PCI Express device will plug into it and work.

And it did. Yes, so,

Jonathan: are the, can people buy RP1s by themselves? Like, is that a thing that's out there

Eben: anywhere? It's not a thing at the moment. I think we've not decided. There's quite a burden associated with selling a chip. I think if you look at the amount of documentation that went into RP2040. Um, there's a sort of a different level of collateral that needs to surround a device you're going to sell versus a device you're just going to use yourself.

So I think it's probably, in terms of broad market, do I think you'll end up in the broad market? I don't know. I suspect kind of narrow, there's a difference between broad market and sometimes in the chip industry, we call this white, white glove. Um, where you have, um, you have, if you have high volume, um, customers, a small number of high volume customers, uh, and you know, big, big mainline, most big like mainline semiconductor companies like this, right?

They have fairly small numbers of fairly large customers. And what you do there is, is you can not generate so much collateral, but apply engineering effort to white glove, give people a white glove service to kind of help them through the process of designing. It's possible RP1 might show up in a white glove.

Uh, kind of, kind of business model, whether we are resourced really to, to, to, to, to, to support it as a general product, I don't, I don't know. I've heard enough

Elliot: hackers thinking about hot airing them off of existing boards, so I don't, you know, you can either Cater to our market or not people know what they're

Eben: doing.

I mean, honestly, honestly, I am aware of you know we obviously we've had some inbound inquiries and I you know, I'm aware of somebody who's wanting to use it to replace a 300 FPGA in a design right and so at that point Yeah, maybe you should be hot, uh, maybe you should be hot air gunning, maybe you should be buying 60 Raspberry Pi 5s and hot air gunning the, uh, hot air gunning the RP 1 off them, right?

Jonathan: Oh, that's fun. I, you know, it's, it's, the thing that's most intriguing to me, actually, is that idea of a PCI Express car that you can slap in your x86 and have all of that juicy input and output. I mean, it's, it is non trivial to get. I2C and SPI exposed on an x86 machine. There's a couple of little USB dongles that will do it, but the driver support is terrible.

Eben: Yeah. And the latency is what's really interesting is, is how quickly you could bit bang. So we had somebody for, uh, we are working on, um, we had somebody, um, bit bang. Um, Something a bit iTunes, uh, no, just sort of like random, just random serial protocol, um, out of the PCI express, out of the, out of a GPIO over, PCI Express from the ARM core, and they were able to get, you know, sort of five megabits.

So it's interesting that you really can, you know, it's not quite the immediacy of the GPIOs that are on, sort of, think of a hierarchy. The GPIOs that are on RP2040 are literally one cycle away from the processor core. The, the, the GPIOs that are in 2708, 2711, the, the monolithic historical Um, Raspberry Pi parts are a few, you know, ten ish cycles away from the core.

Um, these are a lot further away from the core than that, but not as far as you might think, actually, and it would be kind of fun to see, you know, how, what could you bitbash from your Xeon. Processor for your PCI Express card plugged into the backplane. Yeah,

Jonathan: I'm actually working on another open source project, MeshTastic, and it's all about LoRa radios, and of course we talk to those over SPI, and one of the things that I've been doing is making because it runs generally on embedded hardware, and I've been working on making it work with mainstream Linux and trying to make SPI play with a desktop is just a pain.

And so, it's

Eben: actually very different worlds. They're very different worlds. The X86 Linux world and the most of the ARM Linux world are actually quite different, actually. Um, so, yeah, it would be a fun thing and it's a, it's a It's, they're objects that exist. And there's always that question, it's always about resource.

It's always about customer, the number of people you help. Because there's opportunity cost, right? We can't do everything. And the question is, How many people are going to be, how many people are going to be delighted by our PCI Express card? If it's thousands, hmm, if it's tenths of thousands, then, you know, then, then that's a, then that's a very different, that's a very different proposition.

So, so, it's probably, it's probably something for, You know, as we're now getting to a point where Raspberry Pi 5 is pretty well ramped, the production rate is pretty well ramped, we can have a little bit of a think about what else we could be doing.

Elliot: Cool. And mentioning documentation, like, the support that you would have to do to put the RP1 out there as a product.

I have to say that one of the things that, uh, you know, in the hacker community we most appreciate about Actually, all of the Raspberry Pi products is the outstanding documentation and educational resources and the operating system compatibility with the single board computers. All of this makes for a very pleasant, easy to get into kind of environment.

And that You know, you would want that to be there. I understand that you would want that to be there for the RTP.

Eben: People focus a lot on the hardware with Rails 3. 5, but you've got to remember the collateral. Yeah, we have more software engineers in this building than we have hardware, than we have certainly board level hardware engineers.

ASIC engineers, actually, we've got quite a lot of those. But, um, the, um, uh, the, the, you know, it is, you know, all hardware companies are really software companies. All ones that make hardware you might want. Uh, I'll, I'll, I'll release off our companies. Um, and certainly that. You know, I'm the guy who has the Raspberry Pi One on my desk and a new operating system release will come to me and I will be the person who boots it up, waits two minutes, uh, boots it up on my Raspberry Pi One, my, my, my 256 meg Raspberry Pi One and then, you know, just then, you know, types top because I want to know how much memory I've got left out of my 256 meg, um, and then waggles the, waggles the, waggles the window.

And You know, it picks a window up and waggles it. And it's like, mmm, waggling's got a bit, lads, waggling's got a bit slow. You'll see it anyway. That's what, what, what technically it used to be. Used to be six, five, or two assembly language. Now it's waggling a window. But we still make technical contributions, even though we are the chief executives.

Yeah. Well, who, you know,

Elliot: who is your end user experience at the end of the day? Right? It's, it's window wagglers. And that's super

Eben: important. Yeah. And it's, yeah, that's it. And, and, and who's our, I mean, it's really important about the end user experience. Who is the end user? People like us. Yep. Uh, one way or another.

And it could be, it could be adult. Enthusiasts, because we're all enthusiastic about the pupils, right? That's why we're here. Um, so it could be adult enthusiasts. It could be professional design engineers, because of course we're all professional design engineers. It could be a child who is like I was when I was 10 years old, right?

But it depends what stage of your life you're at. But, you know, there's, there's, there's a view of the other fruit company, um, in its glory days. Um, as, as being a company that made the products Steve wanted. Um, and it just, and that's all it did. Um, and it just turned out the most, not without AppleCube, not without failures, but, um, uh, you know, by and large, there were a lot of other people who wanted the product Steve wanted.

Um, and that's, uh, you contrast that with the kind of, um, uh, focus groupie kind of, like, could you have focus grouped the iPad into existence? Yeah, I don't think you could, right? Um, you just had to be con Could you really focus group the Raspberry Pi into existence? You just gotta, you, you, you You gotta have some conviction that this is a thing that at least one person The nice thing about, about conviction products like the iPad, is you know at least one person wants it.

Because he's the guy who commissioned it. Um, uh, and, and the problem with focus grouping things, with trying to kind of do market research based product design, is that you, you expose the possibility that zero people want your product. Um, uh, and that does happen, right? Um, So, so that's how we've always approached it.

We, and, and so that comes with then the, the obligation to have, well, what would I have found frustrating when I was 10 years old? Well, if I'd had to patch, if I'd had to cobble my operating system together by applying a hundred patches to something, um, and then the documentation was wrong, I would have found that frustrating.

Uh, if I was a professional design engineer and I was trying to design a product into something and I couldn't find out how much RF it was supposed to be radiating and I couldn't understand when I put it in the test chamber Why it radiated too much RF and I couldn't get my FCC Um certification I find that frustrating.

So it's all all of this stuff has come from Experiences, good and bad that not just me, but you know, on our hundred and we've got 60 odd engineers in the building, a hundred and some total employees that it all comes from experiences. Those people have had.

Elliot: And maybe then this is a good time to bring up the IPO because, because what, because what that does is that gives you guys a lot more money to play with.

And I think the question would be, and I think. A lot of the community's looking at this thing. Is this going to go into More hardware. Is this going to go into more software? Is this going to go into more accessibility? Where is it all going to

Eben: go? So so I should probably read for my piece of paper Virtual piece of paper things I things I say about the idea.

Um, yeah I mean, this is somewhere where you know, everyone knows that we had a look at this a couple years ago and We didn't like it for a couple of reasons. One of them was obviously the market the other was the business You know that the shortages made it very hard the business actually performed financially okay.

Um, but, but from the market's perspective, it became very unpredictable and kind of markets like predictability, almost like they'd rather have a predictable bad number. Sometimes you think than a, than a, than an unpredictable number that turned out to be good. Right. So, so there were reasons not to do it.

Um, we've turned our attention back to it. We don't have a schedule for this. We have appointed bankers to, to, to help us, uh, think about it. We don't have a schedule for it. I think what we said is when the market's ready. We want to be ready because you can't wait until the, if he's waiting until the market's ready and then start running.

If the wait till the windows open and you start running, it's going to shut and you're going to smack into the window again. So, so, so I think that's where we are at the moment in terms of, I mean, I guess a couple of things, um, think of it as a way for the foundation, which is our shareholder. So what are we, what are we talking?

What would we be doing? We have a foundation that owns. a regular commercial company. That's what I run, a regular commercial company. Um, the foundation would sell some of its stake in the company to raise the money. Um, why is that exciting? Because it lets the foundation continue to do. We've returned about 50 million to the foundation over the, over the first decade.

Um, uh, and it lets, it would let the foundation continue to do stuff for a long time, potentially do stuff at larger scale. than it does. So that's kind of the, that's the, that would be the rationale if we were going to do this. Um, I think the I think there's a, there's an understandable concern that we will become somehow the Borg.

Oh, wait, if we were to do this, then we would become, that we would become kind of some sort of, some sort of abstract schematic, um, uh, profit maximizing, uh, um, group of, uh, of, of, of, of, of, of people, biomechanical people, with little green lasers. Um, uh, I think the important thing to emphasize is that, um, well, that would be a really done thing to do.

I mean, it's such a stupid thing to do. Um, the, you know, it was, this is not, um, you know, this is an organization that has grown, uh, by giving people things that they like. Um, and it's given people, and as I said before, it's given people things they like by building the things we would like. Um, and there's no obvious reason for us to stop doing that.

Um, part of things that one likes, uh, is, say, the pricing. So people, uh, have used the word, uh, people, I've seen the word, Slightly rude word, I suppose, and shitification, um, uh, used, which I think probably either implies that people think that we're going to put the price up, or we're going to put the quality down.

Um, and of course Well, we've built a business by building things which are low cost and really good. So, why would we want to blow that business up by, by doing something different? So, I think the, I think the, there's no obvious incentive. We've always been incentivized to make money in Raspberry Pi because we fund a charity that goes out and teaches hundreds of thousands of children how to code.

Of course we're incentivized to make money. Then we could be incentivized to make money. Um, after an IPO, because you know, I would imagine the foundation probably continue on quite a lot of the business. Um, sure. You've got financial shareholders, financial shareholders now. Um, so, so I think the, I think people, I think the way I'd encourage people to think about it is, is in terms of incentives that nothing is going to happen that's going to change the incentives on us.

Um, and if nothing happens, that's going to change the incentives on us. I, I, I struggled to see how we would do something. Why we ended up doing different things in response to the same incentives, but the other ones wait and see, you know, um, wait five years You know if we've if we've made the product Plus five times as much and that's half the performance in five years time then, you know I'll take you out and buy you a beer and you were right I think my beer is fairly my beer is fairly safe.

Yeah,

Elliot: I don't think I don't think I was suggesting it and I do

Eben: I know you won't, but I'm replying to a, I'm replying to kind of like a stock, a stock internet comment. Um, you know, and by and large I've been super excited, I've been super pleased by how The vast majority of the community really understands what's going on here and understands what the motivations are and that they're good motivations Um, and it's just probably just that one little Strand of thought that I probably want to want to perhaps push back

Jonathan: on a little bit And even that you kind of have to keep in mind that that's just a sign that People love what you guys do and they love your products and they're passionate about it.

And there's a little bit of fear when you hear that somebody's going public. And you don't necessarily understand

Eben: all of that. Yeah. And of course, you know, it is a, it is an unusual organizational structure. Um, uh, it is an, it's an unusual organizational structure. So, so I think the, uh, so I think, you know, it's something we're, it's something we're looking at.

Um, I think if we did it, it would be a good thing. Um, and, um, You know, what the wonderful thing is, like, what was the narrow thing we were trying to accomplish with Raspberry Pi, right? We wanted to get people to apply to study computer science at the University of Cambridge. It was that parochial and that narrow.

And in 2008, when we started this, the easiest way to get your child into the University of Cambridge was to get them to apply to study computer science, because it was, it was roughly a 2 to 1 application ratio, which is nuts for a university. It's Turing's university, right? 2 to 1 application ratio. Um, Last year, I think it was an 18 to 1 application ratio.

It's become the hardest subject to get into. Computer science at Cambridge is the hardest subject to get into at either Oxford or Cambridge. And that's not all us, right? That's not, that's, we just, we did this thing. But we were part of a movement. That did that, you know, people founding club networks and people lobbying the government to change the curriculum and lobbying the government to spend money on teacher training and all of these things, right?

That's the legacy, that's what that 50 million dollars, our share of that success was bought by the 50 million dollars that people gave to us. For, you know, if you bought a raspberry pi in the last, in the last decade, you were part of that, right? So people gave us that money. We gave that money. We paid that money up to the foundation.

The foundation has done that work. And so that's why i'm excited about the prospect of an IPO because it secures it Right, it secures at least another decade of impact I would think um for an organization that probably With the best, you know the foundation look we do the engineering is great but when i'm When I'm old and grey, I mean, I'm not exactly young anymore.

I got an MIT 35 under 35 award when I went to MIT Tech Review 35 under 35 award when I started this thing. Um, so I'm not as young as I was, but when I'm old and grey, I, you know, this is the thing I will look back on with pride is most pride is the, that impact on young people's education. And that's the thing that we'd be talking about securing.

Elliot: That's fantastic. So my question was going to be hardware or software or, or documentation and your answer was foundation. So,

Eben: yeah, yeah. And, and because of course I'm, I'm recovering while I'm recovering engineer sadly, but I'm recovering software engineer. I'm a software engineer turned chip designer turned, uh, business dude.

Um, so, so, uh, I've done them, I've done them all. And. I do love the chips though, because it's gambling, right? You know, I don't gamble. Um, I was in Nevada until the weekend and I was staying in a casino and I don't gamble. I stay in a casino because the hotel rooms are cheap. Um, and I, but it is high stakes gambling, right?

You spend about a million dollars on a, on a, you know, you're not far off a million dollars out of pocket when you tape out. Even this 40 nanometers, right? 15 year old process node. Not far off a million dollars to tape out. Um, and then you wait. And you wait, and you wait. And you, sometimes you've, I've had I've had, I mean, sometimes you say, maybe get, you know, you've got however many million gates, and maybe what you get back is a diode.

Uh, I did have one that came back, and it was a zero ohm link. Um, so I had 27 10 when it came back. So that's 2837, the chip that's in, uh, the SymPy3. Uh, it came back as a zero ohm link and it was a zero ohm link because the, the, the die has alternating power and ground pads around the outside. Those are bonded down onto the substrate inside the BGA and then, and then out to balls.

And at some point in going around the package, someone had lost track, had, had had off by one error and had lost track of which ones were power and which ones were ground. And the, the, the consequence was that half of the power, half the Power balls were bonded to the ground plane. Half the ground balls were bonded to the power plane.

The thing was a zero ohm link. Um, so, so sometimes, and the nice thing about that is it's a packaging problem, not a, not a, not a wafer problem. But, but you know, it's, it's, it is a lot, it's a lot of, I commend it to anyone who gets a chance. And of course there are, with things like the Skyworks thing at Google, there are opportunities now to go do little chips.

Um, kind of if you, if you have an opportunity, you and your friends at college or something, have an opportunity to go. Tape out anything. Go do it, because these are the most, it's the bleeding edge of what, of what humankind knows how to do. Um, and it's, it's still wonderful to be involved with.

Jonathan: So I want to jump in real quick because I promised David Toth that I would, um, we had him last week or the week before and he brought up when we talked a little bit about the Raspberry Pi IPO that might happen and he brought up what happened with the Red Hat IPO and he made the point that when Red Hat went public, they gave shares away to some of their contributors and he said, He thought that was the neatest thing and as far as he was could tell that they were the only big company that had ever done that.

And uh, he wanted to make a plug that it would be, it would be nice if when Raspberry Pi goes public, that uh, there would be some remembrance for the open source folks that have helped them

Eben: get there. That's, I mean, that's it. So this is a, so this is an interesting, this is an interesting, and of course you see this with Reddit.

So you see, um, Not giving away, but giving, giving people the opportunity to, people with high karma, um, uh, and long moderation histories, uh, the, the chance to participate. Sure. Um, so, so I think we're some distance away from that. I think, I think giving away, and so what they are effectively saying is we're going, we are anticipating, I think, As I understand it, what they're saying is we're anticipating an IPO pop at Reddit, and therefore what we're giving away is your chance to participate in the IPO pop.

Um, so that's, which is an interesting, it's an interesting way, it's an intermediate way of thinking about it, I guess, in between giving, giving stuff away. Um, I think we're a way away from The deal structure. This is all comes into the heading deal structure. Um, and, and there are, and, and so I probably wouldn't speculate.

I wouldn't speculate even on whether it's feasible to do. What is what the, the, the, the Reddit thing is, I think. Broadly comes under the heading retail offer. It's a fancy retail offer, you know, the ability for ordinary human beings to participate at an IPO. Um, so I wouldn't speculate on whether a retail offer is, it would be feasible in an IPO.

It's a deal structure and I can't really talk about it. Sure, sure. Um, the, the, the giving away thing is interesting. It, it, it sort of, there is a, there's a question there about what. Who to? Um, because in the end you need a, you need a metric. Right. Um, and I will go and look at what Red Hat did because you need a metric.

Um, and it's um I'll go and look at what they did. It must've been what Colonel contributors was it? I, you know,

Jonathan: I, I went to look for the details of that and Google, my Google flu was failing me. So we may have to go back to, to David Todd, which the buffer bloat by Buffer bloat guy, by the way. Yes. David's great.

Um, I, I may have to go back to him and get more details from him.

Eben: Yes. Um, I mean, that's a, but look, it's an interesting concept because there is always this, this sort of feeling about like, you know, where does the, um. Yeah, where does value, where does, you know, where was value generated and where does value end up?

Um, and I will go and I will go and attempt to use my Google for you to find out more

Jonathan: about that. Yeah. So one of the, one of the things that's always intrigued me about Raspberry Pis is your, your relationship with Broadcom. And there's, there's some interesting things there. All of the chips, all the CPUs have been Broadcom chips.

Um, is that Let's see, how do I want to ask this? Are we ever going to see a Raspberry Pi with a ROC chip on it? Or a Raspberry Pi with maybe a RISC V chip on it? Or is it always going to be Broadcom?

Eben: Um, I look, it's always been Broadcom 'cause it's always been the right choice, right. And, you know, I was at Broadcom when we started, when, when, when, when this, when I, when the foundation was founded, I was at Broadcom.

Um, and it's always been the, the, the, the, the right thing. And one of the reasons, of course it's always been the right thing is that there's this continuity of, uh, you know, we, you mentioned the, the back risk compatibility, right? Mm-Hmm. . And one of the things that, that, that being single vendor, um, has enabled us to do is to effectively.

Not multiply the software to continue to support we can support 2835 so the the arm 11 design that's still it's still a flagship product because it's still it's still the chip that's in zero, which is the lowest cost, which is a which is a zero. Although 0. 2. w has taken the kind of 0. w's flagship status away from it, 0 is still a flagship product.

So we can use the same software team to support all the way back, um, to, so we have a single common code base that runs on the, um, the VPU, the boot processor. Well, it was originally the multimedia processor, but increasingly it's been relegated as, as we've, you know, big software evolution in Raspberry Pi is the pushing of, is, is the arm gradually, is the moving, moving from proprietary interfaces to all the multimedia components to, um, uh, to, to, to standards based.

So, you know, all this stuff in the blob, um, uh, that, that, um, Where you now have Mazer, uh, you have Mazer driving the, the, the, um, the, the graphics, uh, um, VFL2 driving the, the video, um, components, and, um, um, KMS, um, DRM driving the, uh, Um, driving the video scan out. Um, so anyway, so that's the kind of thing that started off being quite central to the thing where its role in the system has been whittled down and down and down to a point where it might as well actually be a microcontroller in there.

You know, it doesn't do much more than that. It's just a little scalar processor we happen to have. But anyway, having a single vendor has allowed us to keep a common code base there. Um, that would be probably the biggest thing that one would be giving up. Sure. Uh, there'll be a lot of treadmill, you know, um, we have, well, we're single, we don't sell anything with single, but we're functionally single vendor for wifi as well.

We do use, um, uh, in, uh, Infineon, Cypress and Synaptics, um, parts or across the two different products, but actually kind of the same, they're both Broadcom derived. They both have come from an original set of Broadcom designs. So we actually functionally single vendor for wifi similar. similar thing, right?

When we have, you know, WPA three support, for example, which is already kind of recent thing that we're, we're, that we're adding to the platforms. You only got to do it once because you only have one vendor. Um, you know, when you find a monitor that you can't talk to because the HDMI doesn't negotiate properly, you've only got to fix it in one place.

Um, uh, and so there are particularly lightweight engineering team, right? Um, you know, everything, including the ASIC development fits into 60 or 70 people. Um, it's hard to be multi vendor for something as central to the personality of your device as the Core SOC. Sure. One

Jonathan: of the neat things about the Raspberry Pi using Broadcom parts is the Broadcom support in the mainline kernel is getting better and better.

Eben: Indeed. I mean, the, the, the, the video, well, I mean, the video call, um, the video call, we'd be love work with, um, Igalia, um, on, um, and this was originally, so, um, uh, Emma Anhalt, um, is fairly famous, major contributor, um, uh, worked for me for a while at Broadcom and kind of, kind of, uh, stood up the, the, the, the video call, the original set of, um, uh, open source.

Um, video core graphics drivers, um, uh, on the back of the documentation release that we did back in 2014. Um, and, uh, and so we now do a lot of work with Igalia, uh, she's moved on. Um, we do a lot of work with Igalia on this and it has had the side effect that it's generated a really robust set of open source drivers for, uh, particularly for the HVS, the video scanner and the video core, um, video scan out engine, um, and, um.

VideoCore 3D, which was the team I used to be on, uh, here in Cambridge. So I, I'm kind of very, very GPU kind of guy back in the day.

Jonathan: Yeah, yeah, I, uh, it's, it's fun, it's fun that we can now, you know, you can boot the, the Pi 4 with the mainline kernel. You can throw, you can throw Fedora on it, and generally expect that everything's gonna work, or most everything's gonna work.

Eben: And it's another one of these things where, where people, we've got to go appeal to people's, appeal, you know, why is, why was Raspberry Pi Why is Raspbian non standard? Well, it's because we have a finite amount of engineering resource. Um, you've got to believe that the direction of travel, nobody wants to Unnecessarily maintain a large body of closed source code inside the blob.

Um, you know, if you can get that out in the wild, it gets us out of the way. It means that we're not the gate to innovation. So now, you know, there's a gentleman, I think at Red Hat, who's been working on, um, OpenCL support. And OpenCL hasn't been a priority for us, you know, we haven't seen doing compute on our GPU as a priority, but he wants OpenCL.

And he's been doing OpenCL, and the documentation is there, and the code base is there that he can submit patches to, and he's been working with Agalia. Um, and so that's the price you pay, you know, people need to understand there is no incentive on us to have Close source software, it's a boat anchor, um, that we drag behind us.

Yep, yep.

Jonathan: Um, I, I feel like we could sit around and talk for another hour without any problem, but it is an hour long show and I only have commitments for you guys for about 15 more minutes. So we need to wrap. And the first question I want to ask as we wrap is, is there anything that we didn't ask about that you really wanted to let folks know

Eben: about?

Um, I, I don't, I don't think so. I think we, we covered, we covered a lot of, we covered a lot of ground. We did. I, I think probably the, I think probably the thing, I think the, the, the thing that was worth emphasizing. It's worth just touching briefly on the shortage, on the shortage environment. We had a couple of years of really rough shortage, uh, as you may have noticed.

Yes. Um, what's really lovely now is we're back in. Supply and I think probably the thing that's pleased me most given that we're coming up to our twelfth. Let's just call it our twelfth. Um, uh, the thing that's really good. There's you're probably aware of our pay locator. Um, uh, um, the the um, Oh, yes. Uh, yeah,

Jonathan: the the website that'll ding at you when when somebody has pies that you can go jump on a drive by

Eben: how green our pilot cater is on our 12th birthday and how green it is for an enormous amount of efforts happened at sony over the last eight weeks um to get us from probably a production rate of 70 000 units a week for pi just for pi 5 alone um we saw about eight nine hundred thousand unit total but about Um, you've gone from about 250, 000, uh, say a month to 350, 000 a month of Pi5 over the last eight weeks.

And what that's done is that's got you from keeping up and treading water to a point where we're actually starting to get free stock. You go on Digikey, I think that's the best part of 3, 000, um, eight gig. Um, uh, uh, pi fives on there and that's just, it's, it's just a, Oh God, it hurts so much. Um, and, and, and it's just so good to be out the back of it and it's good to be out the back of it with, with all of our, with all of our product lines running.

Uh, now, um, zero two W needs to catch up, but apart from that.

Jonathan: All right. Um, I want to ask. What's the, what's the weirdest or most surprising thing that you've seen somebody do with one of the Raspberry Pi products?

Eben: Oh, well, look, the whole thing is weird, right? Well Everyone using Raspberry Pis to do everything is a bit strange.

Um, look, um, I I'm going to need to get a new example, but I, I'm always drawn back to the cucumber sorter. Um, the, the, the, uh, gentleman in Japan whose parents were cucumber farmers. There was a, uh, they would spend a lot of their day getting older. They would spend a lot of their day sorting Japanese spiny cucumbers into 23 different boxes, depending on how straight they were, how green they were, how many spines they had.

Um, and he built a TensorFlow train, a machine learning model to, to, to do an initial classification, a little conveyor belt, little flippers that knocked him into into buckets. Um, and I look, I like it cause it's whimsical, but I also like it because it's, it's actually a scale model. You know, we have people deploying tens, hundreds, high tens of thousands of individual customers to bring high tens of thousands of Raspberry Pi's in industrial environments.

And actually that's a toy model of what all of those things are, right? They're things that sit at the intersection of interfacing, particularly input and particularly vision. Um, output, actuation, compute, and network. Um, and that's the kind of, that's the bit, that's the Lego. You talk about, you couldn't have focus grouped it into existence.

Nobody really had perceived that there was this hole. Browse by shape, hole. This missing bit of Lego, this missing bit of glue that could stick all those things together and let people do projects. Um, and so that's probably, it remains. I love it because it's whimsical and it's got Cucumber, but, um, I, I, I also, I like Cucumber.

Jonathan: Alright, last two questions I've got to ask. It's sort of a contractual obligation. What is your favorite programming language and text editor?

Eben: Oh, uh, favorite programming language and text editor, uh, favorite programming language Um, it's, um, it's gonna be either 6502 or 68000 assembly language, they're both lovely.

And they're just the programming languages of my childhood, you know. BBC Basics is pretty good too, actually. Um, but it's, they're the programming languages of my childhood. I'd probably go with 6502 because it's, because I'm very time poor at the moment. It's the one that's still gives me has that haiku like, um, feeling that you can write 64 bytes of it and have some fun.

So I can go with 652 of some language. Um, in terms of a text editor, um, do still like entering it, the command line, uh, in, in, in BBC emulated BBC that actually the line it's just not bad on the BBC. Um, probably, uh, for, for production use, probably Visual Studio Code, um, actually is, is, is a nice, uh, it's, it's a nice, it's a nice platform.

It's kind of available everywhere.

Jonathan: Yes, and open source too. It's one of the neatest things that Microsoft has done.

Eben: Yeah, forgive all the JavaScript. You

Jonathan: will, yes. Alright, thank you so much for being here, sir. We do appreciate it. And, uh, boy, I think that was a great hour, hour and a few minutes of chatting with you.

Thank you so much.

Eben: Thank you very much indeed.

Jonathan: All right. Elliot, what do you think?

Elliot: I think it's awesome.

Jonathan: Thank you. Yeah. Um, I think it's, uh, some, some interesting observations about the IPO that, that may or may not happen, that everybody is.

Elliot: Sorry to press on that, but, you know, everybody's thinking

Jonathan: it. Yeah, well, no, I mean, it's, I would say it's better to, to let, to let Eben and, you know, the Raspberry Pi as a group speak for themselves to those concerns.

Because, obviously, they've, they've thought about it. Surely they've thought about it. You can tell by the answers that they've thought about it. Um, and, you know, giving them a venue to, to answer some of those questions. I think it's probably a good thing. Um, it's always fun as well. When you get one of these sort of hero products that you use so much to be able to talk to them and say, Hey.

It'd be really nice if X, Y, and Z and so maybe, maybe, maybe we abused our position just a little bit , but we got 'em in . Yeah. No,

Elliot: I, I, yeah, I never, never hesitate to talk to fellow enthusiasts about your enthusiasm, right? Yeah. Like when you've got something like that and you're like, you know, I , you know, I want this thing to have screw mount points.

Never ever hesitate to talk to people because you never know. Who's going to be listening and who is on your team with

Jonathan: that, right? Yeah. Yeah, I suppose that's true. So I think about the things that I'm involved in as a developer. And if somebody comes into our Discord and says, It would be great if we could do this.

You know, sometimes it's a thing that's been asked for a thousand times. And there's a really good reason. About half the time it's like, We never thought about that. That would be really cool. So, uh, alright. Um, thank you, Elliot, for being here. I do appreciate it. Uh, let's see, so next week on the schedule, we are talking with, we're talking with Julian Lamb about NodeBB, which is It's it's forum software.

It's it's modern up to date forum software and that's gonna be really interesting to talk with him Let's see. Do you have anything you want to plug Elliot?

Elliot: No, read more Hackaday.

Jonathan: Read more Hackaday. There you go. I will I will agree with that So the the two things that I normally plug is one on Hackaday.

We've got the security column goes live every Friday morning Be sure to check that out, follow me there. Who writes that? Uh, that would be yours truly. I keep up with the security happenings of the week, the things I find interesting. Um, and then the other thing is the Untitled Linux Show, and that's still over at Twit, twit.

tv. And the audio version of that is now available to the public. But if you want to be on the Twit Discord, if you want to get the video version, that's still in Club Twit. So go check that out. Thank you to everyone. We had a good, good audience in the Hackaday discord. Thank you to those there. And we sure appreciate everybody on the download that listens to, and we'll see you next week on Floss Weekly.

Episode 771 Transcript

21 februari 2024 | min

FLOSS-771

Jonathan: This is Floss Weekly, episode 771, recorded Wednesday, February 21st. Kalpa. Because nobody knows what hysteresis is. This week, Dan Lynch joins me, and we talk with Shawn Dunn about Kalpa. That's an immutable, or maybe we should say atomic, desktop. It's part of OpenSUSE, it's downstream from Tumbleweed, and it's carrying the KDE Torch.

Why would you want to run an atomic or immutable desktop? Well, stay tuned to find out.

Hey, welcome to Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. And of course, it is not just me. I've got I've got Method Dan, the man, the Linux outlaw. Hey, how are you today, sir? Hey, I'm good, Jonathan. Good to be back. How are you? I'm, I'm good. I'm thinking I would say I didn't get enough sleep last night because my brain has been on the fritz, but I know I got quite a bit.

Maybe I got too much sleep. Maybe I've not been awake for long enough yet. I don't know. This may be a day for tip of the tongue syndrome where, you know, my, my brain has a train wreck and I can't, I can't remember a word like concrete or abstract while I'm trying to make a point. So bear with us. It may be one of those days where it may not.

I may have worked it all out of my system. I don't know. We haven't hit the right run level quite yet. I guess. Yeah, I'm still, I'm still tricking along on run level two. Some, some weird run level. Ah, well, it is good to have you, sir. And we're going to talk, imagine this. We're gonna talk about Linux today.

We're going to talk about a Linux flavor open over at OpenSUSA. And from what I understand, it's, it's Kalpa Linux, which from what I understand is an immutable KDE distro. Although I don't think immutable is really the right term anymore. Maybe atomic is what, what people prefer. But we've got Sean Dunn to talk about it.

You've, you've been taking a look at, at Kalpa and some of what OpenSUSE is doing. Haven't you, Dan? Yeah, I

Dan: did some research today. I've been going through looking at the many. It's something we'll talk to Sean about the many flavors of immutable distros they've got over there. There's quite a few.

So yeah, I, I had a look at the the stuff that Richard Brown was doing with with he's now doing Aeon. Aeon, I think it is. I'm going to say Aeon. OS as well, which is a kind of sister project to Kelpa. So that should be interesting and I haven't yet really used an immutable Linux distribution.

Everybody I know is into this. It's the hot thing right now. So I think it's good time to maybe jump on board and learn more. Yeah,

Jonathan: I always have this thought in the back of my mind that for those of us that like are programmers and really like to fiddle and go deep sometimes and fiddle with the guts of our operating system is is one of these.

immutable Linux distros for us. And maybe let's ask, let's ask Sean about that. In fact, let's not fiddle around any longer. Let's bring him on. And, Sean, first off, welcome to the show.

Shawn: Thank you. It's nice to be here. So,

Jonathan: let's start with Well, let's start with what all you're involved in. I know, I know Calpa Linux, but like what, where, where in the solar system of Linux do you orbit?

Shawn: Primarily I'm involved just in the OpenSUSE project. I contribute here and there to various other things as they catch my interest. I've been poking at some things for the guys over at Universal Blue. Kind of helping them get ready for Plasma 6 coming with Fedora 40. And in the past I've been involved in a number of other distributions.

Feduntu, Solace I ran CrunchBang for a long time. So, I'm not actually a coder by trade. Most of my career has been in blue collar work. This has been a side project for me and I have more time to dedicate to it now, so. Sure. When When the micro OS desktop became a thing, which was Richard Brown's personal project There's a bunch of stuff in the background that went on, but basically there was nobody that was maintaining a Plasma version of the micro OS desktop.

Okay. So I had the time and I stepped up because I use KDE and I like KDE and the idea of, An immutable or atomic distribution just appealed to me, so. That's how I ended up where I am now. Alright, well

Jonathan: let's, let's start, sort of start there, micro OS, that was, was that the OpenSUSE original sort of atomic desktop approach?

Shawn: So, Micro OS still exists. That is our server offering. That basically acts as a container host. So, a very minimal base system and you do everything as a container workload. Very similar in concept anyway to like Fedora's core OS or I believe Ubuntu's IOT offering is very similar where you're expected to, you know, do everything you need to do in a container of some sort, whether that's Snap or Docker or Podman or whatever containerization you want to use.

Right.

Jonathan: So that's, that's more so what I would consider like a true immutable OS. Yeah.

Shawn: So the desktop came about actually and when I was said earlier that, you know, Aeon is sort of a sister project to Kalpa. It's actually a little different than that. They're actually more of an upstream for Kalpa.

Because I'm taking what Aeon is doing and what Richard and his team are doing and basically just adapting the Plasma desktop to all the work they're doing. I am involved in it. I am not nearly a class of coder as those guys are, so my contributions to making the underlying system work are limited at best.

Understood. So I'm primarily dealing with, with integration and configuration and trying to put together a desktop that we aim to be more, I hate using this as a comparison, but more Chrome OS like in usage where you just install it and use it.

Jonathan: I think for a lot of people though, Chrome OS is going to be the sort of this, the atomic desktop idea.

It's going to be the one that a lot of people have touched. Maybe it is a good example.

Shawn: It's a good example, I just, I It's, it's a browser as an operating system, which I have a Chromebook but you know, it's not an, it's not a bad comparison as far as if you're trying to explain to somebody what, what should you expect if I meet all my design goals, it should be a very Chrome OS like experience.

Yeah.

Jonathan: That makes sense. And so when we talk about Aon and Kpa, one of the things that I've kind of learned is when, when you're talking about the immutable desktop, it's not, it's not a binary or even a trinary. It's, it's a, it's a spectrum between, you know, on one side Mm-Hmm, , you've got your actual immutable installs.

And on the other side you've got, I guess we could call a full fat desktop where you could put whatever you want to on it. You can make whatever change you want to. Mm-Hmm, . And an atomic is somewhere in the middle. Where, where does the, the Aon and the Kalpa, where does it fall on that spectrum?

Shawn: So, part of the big difference between us and for instance Fedora Atomic we don't use RPMOS tree.

We are not image based like Fedora is. Oh, okay. We are pulling from all of the same repositories as OpenSUSE Tumbleweed. Mm hmm. None of our packages are special. Other than the config packages we need to make things happen. So we track Tumbleweed. It is a rolling setup. There's no point releases like you end up with in Infodor Atomic.

And we, rather than using RPM OS tree, we're leveraging the, the ButterFS snapshot system. So when we update, we're going to snapshot your running system. In the background, that snapshot is going to be updated, and then you reboot into the new snapshot. Ah. So, it's a fully atomic update process, because if anything goes wrong in that update process, it just discards the snapshot.

And, this tends to I believe Neil was talking about hysteresis in his, when he was on. Right. It. If you don't install anything into the base system it reduces hysteresis by a great deal. I'm not going to say it eliminates it, because tumbleweed is a moving target. Upstream is going to introduce a certain amount of that, just by the virtue of the way tumbleweed works.

So,

Jonathan: go ahead. Oh, so I'm, I'm, I'm kind of Wrapping my mind around this idea. I assume you have root and home split out so that when you do a rollback, you don't lose files out of home?

Shawn: So, in our setup anyway var user local temp, obviously, and then home are all subvolumes, and those are read write at all times.

So, if you need to do any sort of bind mounts or anything funky, that can be done in var. Your home remains untouched through the snapshotting process. Oh ETC is also read write. Okay. So, your, your configuration files are going to be handled just like it would on Tumbleweed or Leap. You just put them in ETC and, and process like you normally would.

System control doesn't, you know, running your system D services doesn't require any special anything. You, you treat it just like a Tumbleweed system. Makes sense. The one area where we do greatly differ from from Fedora, for instance we don't really encourage the idea of layering of RPMs into the system route.

We are all in on using Flatpak for your desktop applications primarily because that does reduce the chance that you're going to end up with instability in the core. So, our rule sort of is, the only RPMs you should be installing into the system base are going to be driver related. If you've got an NVIDIA card that you need the proprietary drivers for, obviously those need to go in the system root.

The install of shame. Network cards. Yeah, whatever it is you've got that, that, that I would qualify as a driver that needs to talk to the kernel like that, that obviously needs to be in the system root. But we really try to dissuade our users from trying to treat the system like you would Tumbleweed.

If you would like to be that user that tweaks every last square inch of your system. And Tumbleweed is still there. It's not going anywhere.

Jonathan: Yeah, I can, I can imagine there's, there's a pop up somewhere. You know, someone installs a bunch of RPMs. Okay. It's sort of in the, in the vein of Microsoft Libby.

Hey there, we see that you're installing a lot of RPMs. Yeah. Would you like to try Tumbleweed?

Shawn: It doesn't exist yet, but don't think I haven't thought about writing something like that. That's

Jonathan: fun. That would be a lot of fun. You could have a lot of fun with that.

Shawn: I mean, for instance, with CALPA your primary interaction for finding software is going to be through KDE's Discover app. Right. And that does not interact with RPMs at all on CALPA or GNOME software on Aon as a comparison.

All it handles is flatbacks.

Jonathan: And we're sort of at the point where just about everything you'd need is available as a Flatpak.

Shawn: Pretty much. There's a few, a little bit of stuff here and there, like for instance, Kate is still in the early testing phases upstream. They have not released it on FlatHub yet.

And I know a lot of people that use KDE like Kate, and I don't personally use it, but I understand that's a very important package to a lot of KDE users. Right. And actually at the moment the install image includes the K8 RPM as part of the base image because flat up can't provide it. There

Jonathan: you go.

So this is going to be a little bit of a trolly question. Not entirely though. There's, it's, it's a pass here. Have you thought about adding support for snap?

Shawn: No, the simple fact is I can't, Oh, really? Snaps are not in the official repositories for OpenSUSE. Oh, well, okay. I suppose. Yeah, that's true.

And to be a to be an official OpenSUSE product, everything you install has to be in. The OSS or non OSS repos. Right. And because of the kernel level patches that need to happen Snapd does not pass OpenSUSE Security Team's stamp of approval.

Jonathan: Aha. This is something that doesn't get talked about a whole lot, but there are some outstanding kernel patches for, for Snap to actually run.

Sandboxed the way it's supposed to there's there's some kernel stuff that still needs worked on and I guess the Ubuntu guys have just kind of They it's like they put the code out there and and then it works. Yeah, so

Shawn: You can install snapd on open SUSE You do have to enable a third. It's not a third party repo, but a development repo but you are basically running into unsecured sandboxes because The kernel patches aren't there.

Yeah. So, we absolutely do not encourage people on OpenSUSE right now to use snaps. You know, if snaps are important to you and part of your workflow, I highly encourage you to pick an Ubuntu, whichever desktop you like, there's nothing wrong with them, go for it.

Jonathan: That's almost a controversial statement these days, that there's nothing wrong with Ubuntu.

Depending on which forums and Facebook groups you're in and read.

Shawn: That's very true. I, I don't like to criticize other people's work. I, I know how much work can go into getting things going and they just have a different idea of how to do it and maybe it works, maybe it doesn't. I, I don't have personally have a real strong opinion about snap versus

Jonathan: flatback.

Calpa probably runs with SELinux enabled, is, is that, that's a thing that OpenSUSA does, right?

Shawn: Yes, all of the the micro OS based OpenSUSA offerings are SELinux based. Okay.

Jonathan: I've, I've done some reading that apparently Snap, to get, to actually work the way it's supposed to, it requires AppArmor. And SELinux and AppArmor sometimes don't play well together, maybe as, that, that may be as strong as you cannot turn both of them on at the same time.

Shawn: I believe that's true. I've not tried it or looked into it. Currently Tumbleweed as shipped still uses AppArmor. They have not enabled SELinux yet. Oh, really? You can turn it on yourself as a user if you wish to. I do not. But I believe that that is true that if you try and enable AppArmor and SELinux at the same time Things get very unhappy that

Jonathan: that sort of makes sense.

I mean, you know, that's been my experience in in my also years of trying to help people with their Windows boxes, you know, if you have Norton and Symantec at the same time that Windows box is gonna be real unhappy with life

Shawn: Yeah and that is one of the the development targets that we are still working on so Due to being based on Micro os.

Our SE Linux policies as currently written are sort of focused more on a a server workload. We do have a couple of guys that are working on writing a more desktop focused SE Linux policy that we will be able to track. Like right now for instance, if you want to use the Steam flat pack and run any proton games, you have to do, so a couple of manual SE Linux over or.

Yeah, SEMNX overrides. It's not ideal, but it's where we're at at the moment. It's part of the reason why the why the, neither Aeon nor Calpa have. Decided to declare themselves stable or ready to release

Jonathan: Yeah,

Dan: it does make sense so so Sean you're clearly a KDE guy which which is interesting I what it seems to me as though SUSE or OpenSUSE kind of really pushed the Gnome side of things Is that fair to say and what's the reason behind that and how do you get more

Shawn: people into your neck of the woods?

Here's part of the issue with it is

Sousa, the corporation, which produces Sousa Linux Enterprise doesn't actually have, there are no paid Sousa employees that their job is to work on open Sousa. That's a common misconception. I see people say it all the time. Why doesn't Sousa have open Sousa do this? Because that's not how the development model works.

But because SUSE Linux Enterprise does default to the GNOME desktop, I believe it's actually the only one they offer for installation if you want to run Enterprise SUSE. It gets more attention. There are more active developers. All of the SUSE employees that work on OpenSUSE, they're doing it as volunteers in their free time.

There's nobody up at the corporate level that is saying, you know, you, you work on gnome unless they work for SUSE and part of their job description is you work on gnome for the enterprise project or for the enterprise product. Part of that does come back to I don't know how familiar you guys are with some of the history of SUSE.

At one point it was owned, owned by Novell and Neil touched on this a little bit, I believe. And I believe he might have said the unholy marriage of Zimian and

Jonathan: Novell may have passed his lips.

Shawn: Yes. Yeah. Yeah. And at the time that was sort of Novell made the decision, you know, gnome is the Desktop that we're going to chase for the enterprise product product, which it's really funny depending on where you go on the internet Sousa is alternatively known as a KDE distro by some parts of the community.

It's known as a gnome distro by others. And I think part of that comes from when you got into the project and first installed it. Makes sense. Bye. I mean, it is hard for internet lore to die sometimes.

Dan: That's very true. It's strange because when I, I, Sousa was the first Linux distribution I ever used about 20 something years ago.

And it was KDE then. I'm sure it was KDE when I used it. It was 2002, something like that. I think around 2001 when I started with it. So that's very cool. So how many people are working on Culper? Is it, is it mainly yourself or is there a team

Shawn: behind it? At the moment, I've got myself as the primary developer and I have two other people that contribute pretty regularly as needed.

I would be extremely happy to have more help. But, it's open source, you kind of, you always end up with with more more users than you do have people doing the work.

I mean, I've, I've been around Linux since the late nineties and that's held true in every project I've been involved in. And to be fair, KDE is a big unwieldy beast with lots of switches to flip and lots of places. I mean, I, I think the current 6. 0 release if you installed everything from frameworks from gear.

You know, install the entire suite. I think it's something like 450 packages.

Jonathan: Yeah,

Shawn: sounds about right. One of the very nice parts about our setup is I'm not maintaining KDE by myself. Because I'm using the same package as the Tumbleweed does. The OpenSUSE KDE team, which I believe is Five to seven people are actively maintaining KDE.

So I don't necessarily have to take on that, that workload. So I'm inheriting development from other parts of the project, which does reduce the workload a great deal on the actual desktop development of

Dan: Calpa.

Shawn: Makes sense. Most of what I end up doing is. Playing with the configs. For instance, one of the issues right now is SDDM, which we're still using as the display manager.

Because of the read only nature of user, you can't change the theme in SDDM. Not easily. It can be done, but you've got to do some, I'll call them unblessed things in the system root to install different themes in SDDM.

Dan: You got it. Yeah. Sacrifice a goat or something.

Shawn: Yeah. So that's one of the things that's on my list of things to fix.

I may need upstream help. I may be able to do some bind mounts out of var in my own configs. It's just one of those peculiarities you run into where upstream expects USR user share to be read write at all times.

Jonathan: And SDDM is about to come, become an official KDE project, isn't it? It seems like there's some yeah, I

Shawn: believe I just saw something about that this morning over in the Fedora KDE chat on Matrix.

It looks like for 6. 1, they are planning on incubating SDDM as an official KDE project. I,

Jonathan: I'm hopeful that that will For one, make life easier for you, and also fix some of the weird things around the the login manager that the rest of us have. I, I've got, I've got a display that's, you know, rotated 90 degrees so that I have it portrait format.

And SD, I think it's SDDM that's running on the machine behind me. It is not like that. It does not know what to do with that. It has no concept of a rotated display. So I'm, I'm hoping that having this sort of as, as part of KDE we'll get some of these bugs fixed.

Shawn: So to be fair, the the machine I'm doing the interview from at the moment is actually running Aeon, which is GNOME.

GDM has no idea what to do with a rotated display either. Because I've got this display in portrait mode and GDM is sideways. Yeah. Yeah, that figures. And actually you know, like for instance, I've got my my, my main workstation. It's a laptop plugged into a USB C dock with two displays, one of them in a portrait configuration.

Not everything works. It does once I get into the desktop, but it took some fiddling because you've got so many different things in the, in the, in the graphics chain doing that. That, you know, it's, it's not as simple as, Oh, I have a machine and I just plug the display into the graphics card.

Dan: So, so Sean, I noticed that you don't ship a firewall.

With your distribution now, is that something to do with the nature of of the is that fair or is that something that you do is something to do with the nature of the containerized atomic desktop that you maybe don't

Shawn: need that level? So this has actually been brought up. It was it's been a great point of contention on the open source of forums in a couple of threads.

People insisting that we're doing something dangerous by not shipping a firewall. So if you are using Calpa as intended, the only service that is running and exposed to the outside world on by default is the SSH daemon. Port 22 is the only port. And everything else should be handled by your Podman container networking.

Everything's limited through there. There's no need to have Firewall D sitting in front of the Podman networking setup because you shouldn't be exposing any ports that you don't actually want to serve out to the world on. Additionally, part of the reason is we don't intend people to be doing the mixed use thing.

If you want to run a web server and serve it out to the world. Neither Aion or Calpa, that's not the use case that we're intending for them. You can do it. We won't stop you. It's your machine. Do whatever you want. But it's not an intended use case in our design goal. If you want to do that, we have the micro OS server product.

We Tumbleweed and or Leap if you prefer a stable release. Which do include a firewall, because that's what everybody's used to. But, if you don't fire up a container and expose the ports in the container, there's nothing, there's no attack vector. What are you going to connect

Jonathan: to? That makes sense. Yeah, and it's not like it's hard to go into your SSH config and tell SSH that you really don't want it to allow connections from the outside world.

No, it's not. I suppose that's fair, it's just, it, It makes everybody itchy to think that I'm running without a firewall.

Shawn: It does and I Don't know if I necessarily blame blame Microsoft for that one. I know they catch a lot of hate for a lot of things. Yeah but a Firewall is not a magic bullet that protects you from everything, especially if it's poorly configured.

Yeah one of the things that has continually been brought up is so Fedora Atomic does ship Firewall D in their default form. As far as I know, all of their Atomic offerings, the trick is if you go look at their firewall config, the firewall is not doing anything. It's there, right? So. Yes, we have a firewall, but the default config, it's not actually stopping anything.

Jonathan: In the same vein, I am sure that you ship a firewall in that the firewall modules are there in the kernel, just may not have an easy to grab program like firewalld to configure it. Yeah,

Shawn: our kernel is identical to the kernel that Tumbleweed ships, to the best of my knowledge. We are not, we don't patch the kernel in any special way for the the quote unquote immutable offerings in OpenSUSA.

Jonathan: Yeah, I think I would be more concerned about kernel patching than I am about the lack of a firewall. Like, when you actually get down to it, some weird kernel patch is probably more dangerous Running without a firewall, the way that system is set up.

Shawn: I, I mean, I'm far more concerned, honestly, from a security perspective, on a desktop machine.

People grabbing random pastes from webpages that invoke sudo and a shell script to install things. Yes, that's just insane to me that somebody would just, Oh, I'll just copy and paste this and go on my merry way.

Jonathan: Yeah, in the various forums, the places where I write or present, I've tried over and over again to tell people, no, no, if you find a place where you think this is what you need to do, copy it.

Paste it into KWRITE, paste it into Notepad, what have you. Because first off, there's no guarantee that what the website says you just copied is what you actually copied. I know enough JavaScript to know that. Secondly, you need to look at that closely before you run it on your machine. Yes. If you don't understand what something in there is doing, don't run it on your machine.

Shawn: Yes. And, you know, that's something that has come up repeatedly in various places, not just in OpenSUSE. The, I, I've heard the and seen many discussions. Well, how do I know that I can trust the Flatpak developers? I trust the distribution packagers. That's a fair

Jonathan: question, actually.

Shawn: It's a fair question.

So far I don't personally see any major issues with open source software on Flathub. Their GitHub is very transparent about how the things are built and where the sources are coming from. Anybody can go to their GitHub and look at their build documents and see what's being built. Proprietary software, you know, I, pick your poison.

Yeah, I, I have Discord installed on this machine from Flathub. It is built from a proprietary piece of software I can't see the source of. Do I trust Discord or not? I don't know. But, it, it, it, cause my, my question always comes back to, because I do a lot of packaging of various things how do you know you can trust me?

I'm just some guy on the internet. Just because I managed to get it accepted into the official OpenSUSE of REPOs doesn't necessarily mean I know what I'm doing.

Jonathan: Yeah, that, it's an interesting question. I, I think. Part of the reason it well, I would say honestly part of the reason is because you've you've got a name like you your name Is public you show up on a podcast like this people actually have an idea of who you are You're not jrandom developer.

You're not you're not mr. Anonymous from nowhere Yeah, and That kind of leads into something else I was thinking about Docker images There are, there are malicious docker images out there. In fact, there's a bunch of them. It's, it's kind of mind boggling how many docker images have, let's say, a Bitcoin miner built into them.

And there's not much that Kalpa can do about that, is there?

Shawn: No, there's not. Other than and I will fully apologize right now. I am not the most knowledgeable guy in the world about containers. Other than what security is in Podman itself to stop malicious actors from doing something weird with a container.

Right. There's nothing else we can do about it. Yeah.

Jonathan: Other than try to educate people. Yes. Be careful which, be careful which Docker image you download. Because anyone, anyone can and they do upload to Docker Hub.

Shawn: Yeah. And you know, we much like the the Fedora Atomic setups we do leverage containers for, you know, if you're a guy that likes to do everything in shell we ship distro box by default with the pod man.

And we have put together our own tumbleweed based images for that use. So if you just type distro box, enter, it's going to pull a tumbleweed container. from registry. opensusa. org. And then you can do whatever you need to do in there. I believe the Fedora atomics are all still using toolbox. Similar, but, you know, a little, a little different distro box tends to be a little more flexible.

In that it makes it very easy. I, if for whatever reason you feel you need a new boon to container to do something, a boon to specific or arch or whatever distro box makes that a little easier to pull down other people's containers than toolbox does. Has there, has there been any,

Jonathan: when it comes to, when it comes to Docker, has anybody thought about limiting the, what's available to just the Docker official images?

Like, is there a way to do that? Because I know Docker has, Docker has this project where it's like, these are the curated Docker images that, these are the ones that we actually put our stamp of approval on. I think it might be interested, interesting to, to have some way, I don't know about limiting a user to those, but strongly encourage making it much easier.

I guess that would be the way to go about it. Making it much easier to download one of those and more difficult to download just like random Docker images.

Shawn: I have actually not seen any discussion about that, but I'm, I, I just added it to my little notepad to, you know,

did just release an update to how they're doing verification which is a very similar sort of thing that, that, that Docker could be doing. Or anybody that's running an OCI container registry, really. It doesn't necessarily have to be Docker themselves. They just happen to be the biggest repository of OCI images out there, as far as I know.

It's an interesting question I hadn't considered, honestly. It seems to me that that would absolutely be a worthwhile pursuit to have some sort of verification that, you know, hey, this This container is not going to eat your hard drive and steal your lunch money. Yeah.

Jonathan: You know, that's, it's, it's an issue that's really become a thing on a lot of different sort of open source repositories.

Not as far as I know, not any of the distros like I don't think it's been a problem on a distro packages, at least not very often, but places like Docker hub, Places like the, the, the Python registry PyPy The, the Node. js registry, I can't remember the acronym at the moment, but they, they all have this problem of, Well, the, the things that really happen are either, You get a new maintainer that was not entirely honest When they took over maintainership of a project, Or you get typo squatted, where a project has the word color in it and someone goes and grabs a new project that's the same thing except the other spelling of color, and they'll get thousands and thousands of downloads, and they do, they do terrible things on people's computers.

It's a, it's a, like, it's, it's one of the problems that open source is trying to deal with right now, kind of as a whole.

Shawn: Yeah, you know, and sort of, You get back into that argument if you look at, for instance, the iOS versus Android ecosystem. Apple, for good or for ill, maintains their App Store with what I would consider a Stalin like level of control.

Whereas the Play Store on Android There's been some pretty ugly malware that has come through that thing over the years. I believe that Google has taken steps to help ameliorate that. But, it's sort of the nature of open repositories. I mean I can tell you if you get something from the OSS and or non OSS repos in the case of OpenSUSE we don't like patching unless we absolutely have to.

And if you've patched upstream sources, you need to have a reason for it, or it will get rejected, right? You know, I, I don't like the way upstream does it is not a good enough reason to be issuing patches. You need to be talking to upstream and getting them to fix it. Right. You know, most of our patches deal with RPAM deconfiguration is slightly different than say Fedora or Debian.

So yes, sometimes we have to patch for that, or, you know, it's, it's functional stuff and not design choices. Yeah, makes sense. And I do appreciate that, that Flathub in particular is. You know, focused on trying to get the upstream projects to be the ones that publish on FlatHub. I know there's a lot of third party stuff on there for various reasons, but A lot of it over the course of Flathub existing has gone back upstream where they've gotten together with the developers.

And now it is the developers that are publishing on Flathub, which is a good

Jonathan: thing. And, and ideally baked right into their continuous integration where they're, you know, they're on GitHub or GitLab, whatever. And they finally say, all right, you know, check in the change. It changes the version number that kicks off all of the, all of the, the build scripts.

And one of those build scripts is just, all right. Build a flat pack, push it out the Flathub. Like that's, that's the way to do it if you can. The,

Shawn: KDE Upstream has been working hard on that. And you know, if you'd asked me a year ago, the KDE presence on Flathub was not what it is now. But it's getting much better.

I believe all of the actively maintained stuff in gear is going to be there in the next year. Yeah, like, you know, I discussed a little earlier, I, Kate being one that is still in a testing process partially because of the the, the language server back end. It's a little problematic. Kmail and the comp, the, the, the PIM suite.

I've had issues with it in a flat pack because of Akinati. Other people have reported it works fine for them but that was true even if you were installing from RPM or DEB or whatever. That, that particular software suite I've always had various problems with it. Other people have reported it works fine.

Mm hmm. So, your mileage may vary. Yeah,

Jonathan: it's weird when that happens with applications, but it does. Yes. Let's see. So one of the other things we kind of touched on is Podman. And that actually surprised me a little bit that I know the Fedora stuff use Podman. I'm quite, quite well acquainted with that.

I've worked with it and fought with it a time or two. I did not know that the OpenSUSE sort of ecosystem was around Podman. Do you know, do you know why that is? Why they use Podman instead of Docker?

Shawn: First and foremost, because Podman does not require elevated privileges. Your standard, if you just want to install a container in Podman, you can do it as a user.

You don't need sudo, you don't need to set up anything, any escalated privileges to use it. I believe Docker is able to do that now to some extent, but that wasn't always the case. You were launching your containers as an elevated, with elevated privileges. Interesting. Whereas with with Podman, it was designed from the, the word go and I, I, I don't know any of the Podman developers directly but I believe that was one of their design goals was being able to run your containers as an unprivileged user, which from a security perspective, makes a lot of sense.

Unprivileged account firing the container off, there's only so much stuff that can get access to, right?

Jonathan: Which is probably what you want. Okay. Yes, probably what you want,

Shawn: And you know, I, I have been playing around with a little bit Podman now has Quadlets, which is sort of their take on what Docker Compose does.

I find it much less opaque than Docker, than Docker Compose. I should have to look into that. I highly encourage anybody that that, that is using a distro that ships Podman or is interested in Podman, have a look at Quadlets. They're pretty trick. They integrate with Systemd really well and simplify your containerized workloads, really, you know, especially in a server environment.

Yeah,

Jonathan: absolutely. I'm gonna let Dan jump in and ask about Project Greybeard. And then I'm gonna take it back. And we'll talk about maybe some Weyland stuff. But Dan, take it away.

Shawn: Yeah,

Dan: sure. Well the question's fairly obvious, actually. So I wanna know, what is Project Greybeard? And what does it do?

And what

Shawn: can you tell us about it? Is based on micro OS, just as I go on CalPAR. It is outside the open SUSE umbrella. It is not an official open SUSE project. Partially just due to it's meant to break things. It's meant to have some very highly opinionated ideas about how things work. And obviously it is the Sway desktop running on top of a micro OS base.

Ah, alright. So it is a, you know, obviously a, a, a tiling compositor. And it's meant to be much more bare bones. It installs, I, at the moment, I believe, the last time I checked the patterns, you basically, you get Firefox. As a default install, and that's it as far as graphical applications go. Hmm. So, It is outside of the OpenSUSE umbrella partially because, And, this sounds like a terrible way to put it, but I don't know another way to put it.

There's, No, community surrounding it so much to be able to say, we want it to do this. We want it to do that. This is purely developer driven, sort of based on what we want. Partially, you know, I, I, I would say, honestly, it's a little bit of a reaction to developer burnout where you do get. Everybody and their dog telling you how you should be doing it.

I, you know, yeah.

Dan: So who's the user base for this

Shawn: then? At the moment, I believe it's Richard and me. Okay. You know, we have not advertised this thing out. In fact, this podcast is probably the first time that it's really been talked about in public. Oh, wow. Big announcement. Yeah, but it is also not necessarily tied to

Because we're not trying to ship it as an official OpenSUSA product. We don't necessarily have to run things by the OpenSUSA security team. Or by legal or whoever else, cause you know, good, bad or otherwise OpenSUSA is in the same boat as Fedora. We are tied to a corporation which has its, its pluses and its minuses.

I'm sure you can, you can both recall the absolute uproar when when Fedora decided to pull the codex from Mesa. Because they were worried about potential luck. Yeah, and OpenSUSA did the same thing. So, we aren't necessarily bound to follow what OpenSUSA is doing. If we see Ubuntu, or Clear Linux, or Solace, or somebody doing something that we can't necessarily do in OpenSUSE officially, we can do it in Greybeard and see what happens.

I would compare Greybeard a little more towards something like Universal Blue, honestly. Different goals, but sort of working outside of The the official repositories to put something together. You know, the the Universal Blue guys, obviously they do work closely with Fedora, but they're not necessarily bound by what Fedora wants to do if they don't want to.

Which can be advantageous when you're, you know, throwing stuff at the wall and seeing what sticks. Yeah, makes sense.

Jonathan: Yeah, for

Dan: sure. So, talk to me a bit about Sway, because I've never used Sway. Am I right in thinking it's a Wayland tiling window manager? Is that anywhere near

Shawn: close? It is, and it is Wayland only.

There is no X11 version. So, are you familiar with i3? Ah, okay, yeah. A little bit. So, in usage, you can actually take your i3 configs Drop it right into sway and everything works. The, the, the configuration and the usage is almost identical to I three. It is just Wayland only instead of, instead of offering x Ah, cool.

So, and I believe, don't quote me on this. I believe the Sue developers are one of the primary drivers of the WL roots compositors. Which I believe everybody except for KDE and GNOME are primarily basing their compositors off of. KWin, Wayland is WLRoots compatible in most ways. Mutter on the other hand GNOME's doing what GNOME does, their own thing.

Jonathan: As they do.

Shawn: I can't speak for Gnome. So yeah, Graybeard is very amorphous at the moment and sort of intended more for a development playground. We it does not use a display manager. It's very rough. We aren't too worried about it being pretty. We're not trying to ship a finished desktop product. It's more, like I said, a playground to play with the the immutable design and the container workloads and sort of move fast and break things.

Jonathan: Yeah. I did want to ask though, you've got one Wayland playground that you're playing in, and I'm curious with Kalpa, and I guess with all of OpenSUSE at Tumbleweed, what's the story with KDE Wayland there? Is there still KDE X11? Okay,

Shawn: there is still KDX 11 due to the development model of having to support Open Susa Leap.

Mm-Hmm, X 11 is not going anywhere anytime soon. Ah it's a, I don't know if I really have the time to explain how the development model works and how that versioning breaks out, but anything that is in leap. Has to be in the official repositories, the OSS repositories. Okay. So, as long as Leap is still shipping Plasma 5, X11 is going to remain.

And the KDE team has We are shipping both Plasma I'm sorry, Waveland and X11 sessions for Plasma 6. I can tell you that when Plasma 6 comes to CALPA, there will be no X11 session by default. Okay. Users that wish to have X11 can install it. That will be a supported configuration, it's just going to default to Wayland.

Yeah, there you go. I, I don't personally see any reason to keep dragging X11 along. Oh. Oh. Oh. Oh. I understand NVIDIA users and some very special use cases still need X11 for things and that is still going to be available if you use Calpa. I don't believe the X11 session in Aeon is going anywhere, but they also default to Wayland.

But there has been no talk as far as I'm aware of within OpenSUSE as a project. To say, no, we're, we're, we're dropping X11.

Jonathan: I mean, there's coming a time, and I sort of think that it's going to be here pretty soon. That X11 is, is really, and maybe we're there already, is for all intents and purposes, unmaintained.

And I, I, I'm personally of the opinion that we need some, some people running distros to sort of wake up to that reality. It was not very long ago that someone discovered, and it got fixed, to X11's credit, it got fixed, but like a 37 year old critical security bug in X11. Mm hmm.

Shawn: X11 has been on live support for a decade, basically.

Yes. Everybody that used to work on X11, they're Wayland developers now. And the only work that I'm aware of that's actually happening on X11, other than secure, you know, security backpatches, is in XWayland. Right. The, the, the stand alone X server itself, nobody's touching that. Which you know, I suppose to, on some level, is a testament to the fact that it works.

They haven't needed to touch it. Yeah. Mm. Bye. It's, it's a, it's a display server model intended for a different age that was sort of tortured into being what it is. Yes. You know, it, it was the tool they had to work with at the time coming out of, I believe it was project Athena way back in the eighties.

It's just not it's a dead development tree, basically. Yeah. And the way the developers, the ex, the ex org developers realized this a decade ago, and were like, we're just, we're moving on, we're gonna start over. And that's not to say that Wayland is perfect. It's where we're going.

Jonathan: So I was just thinking about asking this question.

I've got, I've got sort of another thought. There, now you're back. I've got sort of, I've got sort of another thought that goes along with this. Let's, let's touch on that first. RHEL 7 is just about to end production support and enter extended support. And I've, I've heard some murmurings that when that transition happens, X11 is basically going to lose the last of its paid developers that actually cares about security bug fixes.

I don't know, maybe that's going to be a real important date when that happens.

Shawn: I, I don't have any visibility into that part of the world. But, that wouldn't surprise me. I mean, I can tell you from anecdotal personal experience, and sort of being around the SUSE Enterprise product just by osmosis, but from being an open SUSE member.

There are absolutely people companies still paying for long term support for like SUSE Linux 10 and SUSE Linux 11 and yeah, I don't believe either of those ever shipped Wayland to begin with, but I also don't believe that any of those clients paying for that long term support are desktop, right?

Interested in desktop in any way, shape, or form. They're running servers. They don't care about a display server. So, yeah, once the enterprise has moved on, and if RHEL is the one that drives that, yeah, I absolutely agree that your paid development on X11 is absolutely going to fall by the wayside.

Jonathan: Yeah, so you you kind of alluded to something just a second ago, and I'm curious your thoughts on it It sounds like you watch some Weyland development as it happens which Watching development as it happens is sometimes scary But it sounded like you have, you have thoughts and opinions on that, like, Oh, I don't know the the, the difficulty with which it is to get something as simple as setting a program icon in Weyland.

Shawn: I mean, one of them, and it's, it's part of the reason why I do prefer sort of what KDE has chosen to do with their Weyland support versus what GNOME has done with theirs. I, I, I don't mean to criticize the GNOME developers, but I absolutely do not understand their hardcore opposition to setting server side decorations for Windows.

That one baffles me, but they have their reasons and you know, I'll be completely honest, There are times when I look at GNOME's development model and sort of their stubbornness and kind of their clarity of vision to be like, No, this is how we're going to do it. You can come with us or not. Cause I think there is some value in software projects being able to do that.

Sure. It certainly makes them unloved in certain corners of the internet. I don't think anybody likes being told no but it does bring up challenges as somebody that is developing a KDE based desktop. Depending on how deep you want to get in the weeds, you can make the Plasma desktop be basically anything you want.

Which makes it incredibly difficult to support at times.

Jonathan: Yeah, I can see that. Makes sense.

Shawn: And that goes for Linux in general. You know, when you've got the freedom to do anything, it can lead you to some interesting places and some very strange bugs.

Jonathan: Yes. What's the What's the, the, the law that, you know, if there's two ways to do something, one of them's right and one of them's wrong, someone will inevitably come along and do it the wrong way.

It's not Poe's law, it's not Moore's law, but it's one, it's one of those laws that I think it, I think originally originated from rocketry, actually. Someone installed sensors the wrong way. We kind of, we kind of get that in Linux and in the Linux desktop, someone will inevitably come along and put this in wrong.

Shawn: You know, I, but. Sort of, you know, vis a vis coming rolling back around to Wayland and X11 good, bad or otherwise, it's a similar issue to, you know, when system D came around. I understand there are still distributions that For their various reasons, don't use SystemD. Slackware still doesn't and, you know, they, they, they're doing fine for, you know, for what Slackware is.

Things change from a case, progress happens. I wouldn't say that, for instance, SystemD slash PulseAudio slash PipeWire. Pick your controversial software choice. None of them were perfect, but neither were the ones, what came before. You know, it, it, I, when SystemD made its way into OpenSUSE at the time, I hated it.

Partially because, you know, I, I had just been involved in Linux and I was used to using the old sysv init. . Mm-Hmm. . And yeah. There, there's still the occasional thing that bugs me about the way system D does things. Sure. Depending on which part of system D you're using, but you know, you, you learn and you move on.

Yeah. Because do I want to take on all of the development overhead for an entire in IT system, just so I don't have to use system D? No. No, I don't. . Nobody's saying

Jonathan: does.

All right, let's we are, we are running out of time. It has been a great discussion. Let's get into some final questions. And one of the things we for sure wanted to ask about is, When is Kalpa gonna be, when is it gonna be done? No, when is it gonna be stable?

Shawn: Yeah, so if you go look at the, at the, at our, the Mastodon that I linked that's sort of the official news source for Calper related things. I, there's some pinned posts that line out a little bit of this. One of the issues we've got right now is OpenSUSE I don't know how much you guys are, have interacted with it in the past Yast and the Yast installer or whatever.

going away. They're not gone but that development has stagnated and the upstream project OpenSUSE is working on a new installer called Agama. We aren't actually going to be using that for Aon and Kalpa. Richard and his team are currently working on our installer. But one of the issues is, for instance, Aion uses Gnome initial setup, which on first boot, that's where you set up your username.

It sets up your home you sign into your online accounts, et cetera, et cetera. Plasma currently does not offer anything like that. It's something that's been discussed with the Plasma developers. It's something that I'm working on, on writing. But At the moment, that needs to happen before I'm willing to move it out of, closer to a release.

Sure. There's a few other minor issues there and quality of life things that I have to touch up. One of the things that I would like to get working is the ability to set SDDM themes before I say it's stable. Sure. So You're going to see at the moment, Calpa is, is officially an alpha release.

You will see it move to beta when our new installer comes out. I kind of have to wait because Aon is my upstream for this. Once they get it sorted out, and then I can do what I need to do to get it to work for Calpa. I'm going to move the project from alpha to beta. Release, I mean, I don't want to commit myself to a timeline because somebody will inevitably pop up when I said it was going to be ready and say, Hey, you said it was going to be ready, but I would expect that I could have it ready to go in the next 12 to 18 months as far as a stable release where I can say,

It's good. We're good to go. Yeah. Mm-Hmm. . Very cool. 'cause I, I do know that right now if you install Kpa with the current installation media, it is not guaranteed that we have an upgrade path from what currently gets installed to what will happen with the new installer. Richard and his development team have been working on that.

There is a non zero chance that moving to the new setup will require reinstall. I just, I can't answer that question yet. Sure. All

Jonathan: right. So, there's, there's So There's a couple of questions I, I love to, I love to ask folks before we let them go. And one of the ones that is the most fun is, what's the weirdest or most surprising thing that someone has done with this project?

What's the weirdest thing you've heard of that someone has done with Kalpa?

Shawn: The one that keeps popping up. And I'm not entirely certain, well, I know why they're doing it. But, is torturing the Nix package manager into playing nice with an immutable root. Cause, depending on how you want to define Nix it's immutable in it's own way. So, basically, you've got two different philosophies trying to argue with each other, and there have been some really interesting hacks to get this to work.

I will say it is not officially supported. If you try to do this, you're on your own. I'm not going to fix your bugs. But yeah, some of the, the, the If you go back and poke around the internet and look at some of the things they've had to do to get Nix to play nice with a, I'm like, wow, that's an awful lot of work.

Jonathan: You know is there, is there anything that you really wanted to cover real quick that we, we didn't manage to ask you about?

Shawn: So one of the things I would kind of like to, you know, since I have a a soapbox to stand on here isn't directly related to Calpa, but it is related to OpenSUSA as a project.

Sure. Various other I don't know what you I'll call them media outlets. The project has not been the greatest at communicating all the time with what they're doing. I don't know if either of you have heard of ALP. I read

Dan: about it today.

Shawn: Yeah. So it's bouncing around. Reddit is terrible for it.

Any number of other places. ALP came up and at some point, somebody decided that ALP means. Everything in OpenSUSE is going immutable, and we are all going to be forced to move to an immutable base distribution. This is not true. I just, I For anybody watching that is using OpenSUSE and is worried that you're not going to have a traditional distribution available when LEAP 15 ends development.

It's already been announced, there will still be a traditional distribution of LEAP, where you're still, you've got a full read write file system, you, it's just not going to happen. The other one that has come up repeatedly since the ALP announcement, which is primarily a corporate design decision, and doesn't, It doesn't, it doesn't affect OpenSUSA.

Tumbleweed is not going anywhere. I've seen repeated times, you know, is, is is, are Aon and Kalpa replacing Tumbleweed? No. We don't exist if Tumbleweed doesn't exist. Literally. I, because we're downstream of Tumbleweed. So You know, if I can get anything across, having appeared here, OpenSUSE, we're not perfect, we never have been, probably never will be, but your traditional distributions are not going away.

Nobody's talking about taking them away or sunsetting them. We're absolutely exploring immutability, atomic, whatever term you want to use. At some point, I think we probably need to come up with a better term for it. I believe, like I said, Neil brought up hysteresis, but that's, nobody knows what that hysteresis is.

But, you know, and we're not the only ones. Ubuntu is looking at immutable offerings. Vanilla OS is out there, which, I believe their model is similar to ours in that they're leveraging snapshotting. Like we do, I don't know, I don't recall if they're using ButterFS or not, but everybody is sort of looking at this, and I, regardless of which one you use, or don't use, an immutable system is not really taking away choice, it's just a different way of doing things.

You know, there, there's absolutely a way to remount the system route read write if you're insane and want to break things. Nobody's stopping you. It's

Jonathan: just not supported.

Shawn: But it does require a little bit of different thinking and I encourage anybody that's curious. Spinning up VMs on Linux now is so darn easy.

Download an immutable that you think is interesting. Toss it in a VM and play with it. See what it does. You know, it's a minimal investment to, you know, sort of investigate. Yeah.

Jonathan: Great. So I do want to ask you two final questions before we let you go. And that is, what is your favorite text editor and scripting language?

Where do you spend all

Shawn: day? Oh, well, that's easy. Vim, period. There's no other text

Jonathan: editors, what are you talking about? That is the text editor. You just used Vim because you fired it up one day and weren't able to get out and had to learn it, right?

Shawn: Yeah, I'm still stuck there. I never have figured out how to get out of it.

Jonathan: It's a

Shawn: scripting language. I fire up, I do a new install, fire up a new machine. It's still open to the same file I was in 15 years ago.

And I primarily do most of my scripting in Perl. Ah, there you go.

Jonathan: Oh, okay, cool. Randall Schwartz, I know you'll listen to this later. You got one. He's a Pearl fan. Alright.

Shawn: I have I've played a little bit with Python and some of the other ones. Pearl was the first one I learned and it does everything I need it to do, so I use that.

Yeah.

Jonathan: No, nothing wrong with that. All right, Sean, thank you so much for being here today, giving us a great overview of not just Kalpa, but the entire immutable desktop, particularly at OpenSUSE. Touched on a bunch of things. Thank you, sir. Appreciate it. Oh, thank you. Yeah, it's great. All right, Dan. Any thoughts?

Shawn: Yeah, I thought

Dan: it was great. Really interesting. Really interesting to hear from Sean about many different things. What he said towards the end there about you should try an immutable desktop, put it on a VM, see what it's like. I'm gonna, I'm gonna definitely do that and, and, and actually try and figure this thing out.

Because I haven't really used an immutable desktop. I don't know about yourself, but it is an interesting concept and it makes me want to try it

Jonathan: out. Yeah, I'm, I'm tempted to I'm tempted to grab Kalpa and put it on a VM just to, to play with it and see what it's like. I still kind of have this thought that it would drive me crazy, but I can, I can see it definitely being useful.

I don't know, maybe Maybe some of the things about it that would drive me crazy. We just need a little bit more time to bake for you know, your, your various docker and you know, your, your container support to get just a little bit better. You know, like he was, he was talking about the things like with Kate, there's, there's still these occasional problems people run into.

I know I have, I occasionally run into problems with things like trying to launch mumble which mumble, that's an entirely different. issue because that project just needs to get brought into the future. But anyway you know, some days, some days the containerized version of it will work. Some days the containerized version of it won't work and I have to go launch the RPM installed version of it to be able to get sound happening out of it.

There's still a few of these little problem points, but. It's getting better.

Dan: Yeah, it's an adventure. It's all an

Jonathan: adventure. Yes. Yeah but definitely something fun to play around with dan. You have anything you want to plug?

Dan: Not not specifically if you head to dan lynch. org, that's my website That's the place to look for all things i'm up to so keep keep an eye on

Jonathan: that dan lynch.

org awesome So, I will mention that next week we have, we have a really special guest, we also have a special co host, and we have a special time, so next week, February 28th, we are recording two hours earlier. Which means that I'm going to be even more brain scrambled, because I, I will have to be here earlier.

Our co host is going to be Elliot Williams, the managing h editor at Hackaday. And the topic is going to be the Raspberry Pi. We finally scored Eben Upton as a guest. So if Anyone out there that are rabid Raspberry Pi fans like I am, or if you're curious about the news about Raspberry Pi, maybe having an initial public offering, we'll certainly ask about that.

If you're interested in the Pi 500 that hopefully is coming, we're going to ask about that. All kinds of fun stuff. You don't want to miss that one coming up next week. And then the only other thing that I've got to plug is, well, two things I guess now. First off, The security column goes live on Hackaday every Friday morning.

Make sure and follow there for your weekly dose of security news. And then the Untitled Linux Show. There's been a change over at Twit. And the Untitled Linux Show, the audio is available to everybody. That's twit. tv slash uls is where you can find that. And if you want the video and if you want to be in the Discord, that of course is at clubtwit, which in my opinion, is worth the investment.

But anyway, make sure to follow those things as well. Thank you, Dan, so much for being here. Always a pleasure. Great to

Dan: be here. Thanks for having me.

Jonathan: Yeah. And Hey everybody. Thank you for listening. We will see you next week, two hours early on Floss Weekly.

Episode 770 Transcript

14 februari 2024 | min

FLOSS-770

Jonathan Bennett: This is Floss Weekly, episode 770. Recorded Wednesday, February 14th. 10 percent more internet.

Hey, this week, Doc Searles joins me, and we talk with David Tot about the state of the internet, IPv4 exhaustion, 10 percent more IP addresses that are just sitting there waiting to be used, and more, you don't want to miss it. So stay tuned.

Hey, welcome. It is Wednesday. It is time for Floss Weekly. That's free, libre, and open source software. It's not just me. I've got Doc Searls with me and we've got a we've got a pretty special guest. We've got, we've got David Taut with us. This is going to be a little different. It's more like a meeting of the minds than a full on interview.

And that's mainly because everybody in the audience, I think, knows David Taut by now. He's the, he's the buffer bloke guy. He's the big internet guy. But I think, I think we have something a little different we're going to talk about today. Both Doc and David, welcome to the show.

David Taht: Both of us are actually named David, Doc.

So Mind if I call you, yeah.

Doc Searls: I know, I know, that's why you go by David, I go by Doc. And, I mean, the year I was born, the two most common names were Michael and David. And and I got called Doc because I started a business with somebody else named David. I was very possessive of it. And I found out years later when I saw his driver's license that his first name was Paul.

So, you know, names are weird. Names are weird. And you went by another name when I met you, too. Which I would Yeah, the most common name. You went to the most, to the second most common name.

David Taht: Well, back then my name was, Mike Taht was number one on Google for like 10 pages in and I I tried to retire. I didn't want to fix computers anymore.

And I changed my name to Dave to become a surfer musician dude. And it didn't last. Here I am again. I hear Sting isn't using his name much. Maybe I'll try that one. It's also my hope we end up with a mystery guest today. I tried to get him on the show. He's late to arrive.

Jonathan Bennett: Yeah, and I know there's there's sort of a legal issue that maybe is why he decided not to be here.

Let's go ahead and dive into that. David, why don't you set the stage? What, what's the deal, what's the deal with the internet and IPv4 addresses, to be precise?

David Taht: That's a really big question.

Doc Searls: Like a drug or fire.

Jonathan Bennett: I kinda realized as I said that, that that's way too big of a question for one episode.

So we'll, we'll narrow it in a little bit and talk about IPv4.

David Taht: Okay. So it's really remarkable. Fairly recently, I went around and I polled normal human beings. And I asked them, Do you know what a packet is? And I went O for 30. So by some miracle, people click on things and it comes here, but they've never really thought about it.

Most of those people knew what an atom was. It was like over 50 percent knew what an atom, and I actually explained it in atom. But if people don't know what an atom is, what's a packet? And And then you get into the fact that all packets need to have a home address and a destination address. And when we first laid out the internet back in the good old days we thought 32 bits was enough.

Ah, 4 billion addresses. We had a couple hundred computers in the world in the first place, so what the heck, why would you need 4 billion addresses? And Vint Cerf is sometimes called IPv4, the experiment that escaped the lab. Yeah. And we realized that we were going to run out of addresses in about 1990, yeah, 1992, 93, and started drafting up a new standard called IPv6.

Which again, you explain to someone that doesn't know what a packet is, or an atom, and then you talk about IPv4 versus IPv6. You've already fallen off a cliff. You cannot have a rational discussion anymore. And this includes multiple people in government. I was talking to a certain three letter agency, not the NSA, with two intern lawyers that are supposed to be doing legisla They had no idea what a packet was either.

Anyway so we screwed up in not visualizing how big the internet would get. And after a lot of reorganization and struggle, we invented a new technology called NAT, which is how most of us are talking today, which allows you to multiplex your home address on top of one of those four billion addresses to get to the internet.

Jonathan Bennett:You hide multiple machines behind the one address.

Doc Searls: And you use DHCP to assign an address, right? So, this machine has an address and the printer has an address in your house, but it's not on the internet.

David Taht: Yeah. Well, it would be cool to find out if you guys have IPv6 yet. But, you know, you're nerds. You might actually have it, but you probably didn't know you had it.

Jonathan Bennett: I've tried, I've tried to get IPv6 and my ISP doesn't support it yet.

David Taht: How about you, are you, are you typing ifconfig?

Doc Searls: Is Comcast supported? Because I'm talking to Comcast right now. Yes. Fiber, they do. Can I, what can I look at to see? I mean, is it a --

David Taht: You bring up a terminal window and you type in ifconfig, and you look for a big number separated by colons.

Doc Searls: Command not found. So, so F config and IF config. Oh, ifconfig. That's why I've used this so many times. Well, I could, I thought there was something else that I didn't. Okay.

David Taht: The majority of our audience probably hasn't. Well, this audience may be Alright,

Doc Searls: I've put EN which are the --

David Taht: EN zero is probably your device name.

Okay. You should have something that begins with A two N 21. There's so many. It has hex.

Doc Searls: Very long list of them longer than said. I haven't looked at this in years. So, EN4 5, app 0

Jonathan Bennett: One of the, one of the challenges that you'll get to, though, is most machines, depending upon your router, will have IPv6 addresses.

They're just not routable IPv6 addresses.

David Taht: Yeah, they don't work. But anyway, Doc, you are most likely using IPv6 every day, and until this moment, you didn't know.

Doc Searls: No, I didn't know. Here's what I know about IPv6. It's coming. That's, that, that's the whole thing. That's the whole thing.

Jonathan Bennett: It's one of those technologies, it's one of those technologies like fusion, that's always been five years out.

Doc Searls: It approaches asymptotically. It's always halfway here.

David Taht: So, so for the audience the tool I use to see how well it's working is called IPV4 and that will test to see if you put that in your browser as a plug in, it'll tell you what kind of connections you're actually using. Oh, okay. So IPV fu. Anyway, we're a little off track doc.

You're using IPV six, but because you're using IPV six, you can't talk directly to this gentleman over here because he's only on IPV four. So in order for you two guys to talk, both of you have had to had an address that you both knew running over IPV four for this conversation to be taking place just between the two of you.

Okay, Jonathan, are you hosting the server here in the cloud?

Jonathan Bennett: It is, it is cloud hosted, yeah. Yeah,

David Taht: so there's a fourth party involved that doesn't necessarily need to be there, that's actually multiplexing your IPv6 to his IPv4 and my IPv6, mixing it together for video, and bang! You've got a working network connection.

And all that machinery is mostly invisible to you. But the original conception of the internet was, would be that for a call between me and you directly, I would connect between me and you directly, with no intermediaries involved at all. And it would be lower latency, better security, and so on. And we've gotten quite, quite far away from that in the last 30 years.

Jonathan Bennett: Yeah, I, I think if possible, the way this platform works is it will eventually connect the streams directly, but it's got to do all sorts of, it's got to hop through things like NAT traversal and, and it's, it's a lot because of the way things are put together now. It is a lot of work to be able to make those connect directly.

We're using VDO Ninja, and I'm pretty sure it does manage the trick most of the time. But it's, it's a bit of magic, on top of magic, to be able to make that work.

David Taht: Yeah, that's pretty good magic, but it could be less crazy. There's a couple really cool peer to peer VPNs. Did you guys ever have Avery on here for Tailscale?

Jonathan Bennett: he's great. I think we may have once.

David Taht: I think so. Yeah. So, Tailscale and Zerotier are peer to peer VPNs. Make all that magic, just so you can connect two machines together more or less directly. It wasn't, the design goal was to make that be easy. And It isn't. Anyway, so I'm going to try to get back to the history of the internet.

Doc Searls: So we had Avery Penner, we definitely had him on. We definitely had him on.

David Taht: Yeah. There you go. Avery's great. He's a funny guy. And I love it. Really good. And I really, it's a, you know, product pitch. We use Tailskill all the time around here. It makes you finding a device anywhere. You left it in your closet, it's still there.

You know, no matter where you go. So that was the internet that we were trying to build where you could talk to anything all the time. Anyway we got away from, we added all this extra machinery, but we saw it coming that we needed to have more addresses in 1992. And we got together and said, Ah ha, we've evaluated a bunch of possibilities.

Let's Make them be 128 bits. We should be able to cover the universe with the internet if we do that. And 1997 rolled around and people started deploying. There's huge investments into it. We managed to get into Linux by No, I don't know. 2000, there was a huge project in Japan, and we all thought that the next big thing that would happen was that we would just abandon ancient decrepit IPv4 and go to IPv6.

It's 27 years later. And IPv6 is at a little over 45 percent penetration to the Googles of the world. There are whole countries that have no IPv6. There are some countries that are almost entirely IPv6 now. But the idea of a common communication substrate has been in flux. This entire time. So, and we ran out.

We ran out of IPv4 addresses back in 2010 basically. And the price, and the commercial market started, the price keeps going up. So, if you as a business or as a person want your own IP address, where you can put your own servers, you gotta buy it. Or you gotta rent it. And that kind of gets us to where there's this big controversy happening today happening in the past few weeks.

So, historically, IP addresses were given away to anyone that asked. And then this secondary market started up due to scarcity. Which is, you know, by the way, I'm appraised of it. It's effectively reallocating scarce resources to the highest bidder. And it's not that much money, 30 bucks a piece. But, we're still out and we're still adding devices. So those prices seem inevitably to go up over time, unless IPv6 replaces it. And Recently Amazon announced in their cloud they were going to start charging for an IP address. So you as a business, have to pay them rent on these IP addresses. And the estimates are that they will be earning an extra billion dollars a year on the artificial scarcity of the IPv4 address space.

Jonathan Bennett: Now, let's ask something here, because I think this will tie into where you're going. What do you mean by artificial scarcity? Why is it artificial?

David Taht: Well, I'm, that's a little premature for my discussion here.

Jonathan Bennett: Okay, so we'll just take that idea and we'll put it in the corners of our head.

You're going somewhere here.

Doc Searls: So, okay.

David Taht: If we'd all gotten together and agreed in a room, you know, in 1997, that we were going to go full tilt IPv6, this would not be happening, you know and if we were still working together hard with a common agreement that we should. make IPv6 work everywhere, then this wouldn't be happening.

And pretty much everyone in the internet community agrees that IPv6 is the way forward. Everybody should upgrade. You first. No, no, you first. No, no, you. How about you know, and it's just it's just taking a really really long time and we have all these Legacy gear 2g 3g 4g. It's all gear that cannot be upgraded to ipv6 There's cold countries have made billion and trillion dollar investment in infrastructure that cannot be Upgraded IPv6.

Doc Searls: So for example, that's like the, the, the data on your phone. Okay, the, your data connection on your phone is IPv4 by,

David Taht: In, in our country and in many, because of the late phase of phones, Most phones IPv6 better than anything else. Okay. In part because the cellular carriers couldn't get any IPv4 for it.

Doc Searls: Oh, I see. Okay. So, but, but there's still lots and lots of sunk cost and legacy computers of all kinds that are, that, you know, haven't changed. You know, your, your, your Windows based ATM machine.

David Taht: Windows 95.

Doc Searls: Windows 95 based ATM machine at your serious bank. I wish you were joking. I know, I know. When you see the error screen there, I think, wait a minute.

That's Windows 95. Or 98 or something.

Jonathan Bennett: I will say that it's just as bad when it's a Linux machine running a 2. 6 kernel. Which you also will see out in the wild.

Doc Searls: I have seen this on airplanes. Most of the screens on your airplanes are on the back of a chair. are Linux, and when they fail, you see the little --

David Taht: tux.

Doc Searls: Like 2. 6 or something, you know. There we are.

David Taht: Here's a story you don't possibly know. A great deal of those airline seatbacks were my fault. We did that work. Running X windows on these little things for a couple airlines back in the early 2000s. So I had actually campaigned that we change the Tux logo to something else.

And they didn't. It would have been cool branding for the company at the time. And most of those are replaced now. I have another rant on this show. Isn't it great that you have Wi Fi on all the airplanes now? Yeah. When was the last time you had a good conversation with your immediate neighbor?

Doc Searls: Yeah, I know, that's another thing. Well, everybody's looking at their rectangle, because now, they've taken them, they no longer have them. The newer planes, they don't even have a thing. On some of the United planes, there's a clamp, and you pull it really wide for your pad, and it's narrow for your phone, that's on the back of the seat in front of you.

Yeah.

David Taht: Yeah. Hilarious. I spent a plane flight recently, sitting next to a young lady working on VR for Meta. And she was clicking and texting and typing and interacting with more people as fast as thumbs I've ever seen. She must have had three or four hundred simultaneous conversations go over the entire airplane flight.

Yep. And I finally got a chance to get a few words in edgewise asking her what she did. And she told me about that and then I asked her what a packet was. And she didn't knock.

Doc Searls: No, she, she had some in her purse. I mean, you know, you need a Splenda? I've got a couple of those. You know.

Jonathan Bennett: Goodness.

David Taht: Anyway, so I'm going to keep going.

Unwind my story for connectivity. The deconnectivity story. So, everybody producing a service in the cloud needs an IPv4 address. They're getting increasingly scarce. They're costing more money. And if you're an ISP, or a new ISP, or a new provider, and you can't get any IP addresses at all, you You're out of business.

You can't even start your business. So as the scarcity gets worse, innovation is going, is already in the decline. The ability to expand the internet to more people is in decline unless you accept only IPv6. And we all do have to work together. Somehow, to make sure it deploys. And, you know, I've been working for years you talk about Linux 2.

6, there are people still shipping Linux 2. 6 into devices, home routers in the home. Because it's good enough. And it isn't. And they're not able to enable the engineering resources to go do that. And there's no standards body or government or anyone saying, Stop doing that. Give me a modern Linux. There's, there's, there's no clue bat coming out through the internet saying, Don't do this anymore.

You're holding everything back. And it's been I've been kind of working a background. This is another segue. Oh, well have you heard of bead?

Jonathan Bennett: Not necessarily.

David Taht: No. No? Oh, the NTIA I've seen beads, but I The broadband program at the NTIA put together at the beginning of COVID. They threw 70 billion at the states and said, Hey, go bring, make broadband work for everybody.

Get back to us when you're done. And after a lot of legal rigmarole, it's, the money was distributed to the states and they're trying to go and subsidize and build out better internet for everyone, mostly fiber. And your city, by the way, should get that. And it's a good plan, especially because they gave it to all the states to figure out how to spend it best, because I had zero hope the federal government could figure out how to spend it best.

Anyways, this part of the BEAD program I was going up to all the people going out and getting billions of dollars and I'm saying, so, what's the packet? And how are you going to get enough addresses to add internet to all these new people for it? And, they don't know either. They didn't even know it was a question.

You just plug in the wire, right? You got internet. Yeah, okay. And I'm hoping that the educational level at the States does continue to get better as to what kind of demands they need, not just for addressing an IPv6, but security,

training and so on. So we'll see a lot of good stuff coming up with that. The other program is called Internet for All by the Biden administration. And, I don't know, go call, call up your state, all of you on this show, call up the state broadband office, looking for that, and explain to them that Linux and open source software can save them money, provide better security.

Doc Searls: Is there such a thing as a state broadband office?

David Taht: Yeah, they were all set up as part of this program. There is a place to call. Yeah. Interesting. Yeah, I would like very much for the open source community to be jumping in to say, Oh, you know, you can reflash your old routers to open WRT.

You know, this is how you build out a Wi Fi router correctly and do it in your home. And by the way, use wires! Wi Fi can really suck. But honestly, if we as a community could reach out to the government in these cases we should possibly get a better internet out of it. So, big pieces --

Doc Searls: Indiana has something.

There it is. Indiana

David Taht: Broadband. Yeah, so who's your local representative for this?

Doc Searls: I don't know, but we're getting Fiber anyway, so

David Taht: It may well be subsidized by this program. But there'll be added IP addresses. Yeah, I don't

Doc Searls: know. I mean you know, our section of town is planned, you know, so But there's a public private thing that's going on here.

In addition to the private thing that was already here, and told me I had Fiber until we got here, and they said, no, you don't have it in your neighborhood. And then they got Comcast, so Thanks, guys.

David Taht: Well, it's a real thing. That's a real thing too. It's also as part of this program that everyone got together and provided much better broadband maps.

Doc Searls: And it's called Beed. There's a thing called Beed. Look at that. Yeah.

David Taht: Look at that. Who you gonna call? Yeah. Beedbusters! Internet for all. Yeah. But rollout and provided some technical advice and stuff, it would be a lot better for everybody. And the reason why you don't have fiber is the maps sucked. And they have done a great job of improving the maps and developing a thing called the challenge process.

So if they are still claiming you had fiber and you aren't getting it, you have a, you've got a place to call. So go for that.

Jonathan Bennett: So we could, we could go down this alleyway, we could go down this road, we could talk about how Starlink interacts with this, but let's get back on the IPv4 topic, because that's the one we haven't talked about before.

David Taht: It's really difficult, obviously, to get there, because you have to first understand all these preconception things. Sure. So we ran out. They cost money now. There's a monopolistic sorry, there is a large corporation. Currently providing internet services to much of the world, and they are now adding this additional billion dollar a year fee to everybody.

And other, and hopefully this will help, people will migrate to IPv6, they will leave Amazon, but they're still going to make a small fortune off of renting what was previously free. Where I get really mad, and upset, and angry. Again, you've seen me lose my temper on your previous show, and I'm going to try to keep this.

I've

Doc Searls: also seen you sing, so it's okay. Ow! Ow! Ow! And play. I've actually heard you sing.

David Taht: Yeah, anyway, so Your guitar is not

Doc Searls: far from you right now, am I correct?

David Taht: It's I've got one in Nicaragua currently, and one behind me. All I did today I don't

Doc Searls: mean to totally digress.

David Taht: I brought my sticker today. I actually changed this because I was feeling mellow.

I changed it to this machine cures Vogons. I was feeling happy at the time. But still, IPv4! Okay, where were we going to do that? Anyway, so, years and years ago, when we first laid this out, John Cerf and so many people, they say, well, we have this enormous address space, let's use it, let's delegate portions of it.

Zero was for configuration, because zero slash eight, sixteen million addresses. And then we'll allocate all these other addresses to all these other big organizations, you know, Apple and IBM and big companies at the time. And the federal government has 11 of these Slash 8s. They have about 70, I think it's 70 million addresses.

They're mostly not using today. And then we carved up another space. This thing called multicast was going to be the next big thing. And so they carved up 260 million addresses for multicast, figuring that that's how we were going to distribute voice and, and, and video. We were completely wrong.

What we, all we had understood at the time was classic broadcast television. So we thought we would use this multicast technology and reserve space for it to. Do broadcast television over the internet. Didn't work out that way. Do you know how many? Well, I'll tell you later. Anyway, and then at the very top of that range, because who could ever use up a billion addresses to connect the 300 computers that were on the internet was the experimental address range.

And that's another 260 million addresses that were reserved for future use. Okay. We didn't know what we would use them for, but we feel just set 'em aside and when we need 'em, we would use them. And then IPD six started developing. I say, ah, well we'll just keep going. And a bunch of agencies got put in charge ICAN for names, you know VO gons for names and what's called the r IRSs for the numbers.

And then we had, iANA, which is a separate agency also in charge of the merger of these three. I sometimes think that this was designed just like we designed the Houses of Congress and the Presidency, to make sure that they deadlock and can't accomplish anything good, or bad, or anything much at all.

So, you know, those in the show look at what an RIR, Regional Internet Registry, is. We divvy up the world in five different regions. And, And then we started running out of IP addresses and people looked at this other new range saying, well, 260 million more addresses, that's 6 percent more internet.

Let's do that. Now, the amount of code required to enable this address scheme is less than the code that is mounted. To, to not do it. Having the check to exclude this cost more CPU than just running it. Yeah. So in 2008, it mostly started working for everything Apple, Lenox, et cetera. We said, ah, screw up.

We'll just, we'll just make that work eventually. But it got caught up in this bureaucratic in fighting as to, oh wow, who was going to sell dispute tribute 260 million addresses, and it has been stuck there since 2008. I got involved in this around 2016, 17 mm-Hmm, when a friend of mine came to me, John Gilmore, and he said, you know, hey, you know, it looks like we might need to have IPV four running for like another 150 years.

Yeah, , yeah. And we're running out. What can we do about that? And I said, well, we can make this work. So I made it work. I also made zero slash eight work, 16 million addresses. This is my all time favorite and the most controversial passion I've ever done. Okay, I made 0 slash 8, 16 million addresses just work in Linux.

It involved deleting five lines of code from the entire operating system. I scanned all the other software in the world to see what else checked for it and deleted it from there. And I was, I had, I was naive. I, I thought that the world would say, yes, thank you. 16 million, 16 million more people can be online.

And oh boy, the pitchforks and the torches came out for me. You can't do that. I was holding back the IPv6 distribution. And I'm like, I'm saving nanoseconds. Ah. So we did that and then we went through the formal process. We put in the IATF drafts to try to broaden it. And they were kicked down in flames by the IPV6 crowd.

In the meantime, there's all these people that really need IPV4 addresses that can't do anything, getting really frustrated.

Jonathan Bennett: Well, CGNAT, carrier grade NAT, which is a pain and limits what you can actually do with the internet.

David Taht: And the way we're going, we're going to have carrier grade NAT behind carrier grade NAT.

And it, it hurts the connectivity. It means you're, you know, if you have a house, and we see this all the time now, if you have your house here, and you're on a CG NAT, and you have another house next door, and you're on a CG NAT, you have to go all the way back to their hosting provider, and then come back to go 15 feet.

And that's, You know, I used to throw wires over to my neighbor, you know, so we would play video games and stuff. The direct connectivity goes even further and further away to the middleman if you do CGNet. So we're going to see a lot more CGNet in the future and it just seems inevitable that we will end up with more and more disconnectivity between things that should be close by.

I mean, my cell phone Which does a IPv6. Should be able to talk to a tower there, come off that tower, and head a cable right back here, and it'd be much faster. Or it should easily connect via Wi Fi to here, which is what it does today. Imagine if we didn't have Wi Fi.

Jonathan Bennett: All of our, all of our cell phones would still have ethernet ports on them though, right?

David Taht: And you'd have to wind them up by hand. Yeah, yeah.

Doc Searls: So. That little, that little tiny RJ45 sticking out

David Taht: the side. So I've now managed to cover 30 years of history to finally get us to what's going on today, and my contribution to the nightmare.

So I made it work, and I tried to get it through the bureaucratic processes. And we made it work so universally. that Google and Amazon and a couple other big companies started using up these numbers internally. Now, as the author, I can't really claim copyright, I don't think. That was not my intent.

You know, I, I, I, my intent was to, to, to make more addresses available for the world, 10 percent more internet. And instead, these companies, instead of doing an IPv6 transition found a way of crudging grabbing these numbers, squatting is another word we can use, to, to help leverage, to make their operations easier and simpler and without telling anybody, hey buddy, got an IP address.

We're documenting how it works. I mean, some of their techniques could be used by everybody. And I'm okay with that. It's the failure to recognize reality and say, look, we have a civilization wide problem here in trying to get our devices to communicate. And so I wanted very much to see these These numbers get taken over by the regulatory authorities that are supposed to be doing them, distribute them to the newer small businesses, et cetera, that needed to get on the internet and keep, keep that part going.

And maybe if enough people talk about it, there's a great article in the register. Are you able to put a link in the chat? That came out recently with a new gang of folk that have discovered that, Oh, damn, we're out of, out of IP addresses. Here's 260 million. Let's go do that. And I'd hope Carl would make it to the show because they have a different perspective on this stuff than I do.

I just, as the original author of the patches, I'm like, okay, guys, come on, please put, you know, work together. We were responsible for this internet together.

Jonathan Bennett: Right. So there's a, there's a process. I've been through the process to request IPv4 addresses. So like back in the past, there was just blocks and blocks of them that were unused.

And I thankfully got into this game early enough that I could just fill out some paperwork and say, Hey, I'd like 16 IP addresses. And, you know, it went through my well, the ISP that I use at the data center. And they're like, sure, here's your addresses. And That's the, that's the way this is supposed to work.

You've got these blocks of unused addresses. They've now started pushing businesses and organizations, hey, if you're not using addresses, give them back to us so that we can give them to the next people. And so there's, there's this, there's this process that's supposed to work. And the 240, 240.

through the end of the internet, it's just, What? It's never been handed over to that process. So are they as far as Amazon and Google is concerned, are they considered non routable addresses?

David Taht: They are presently using it in their route and their virtual routers system. So yes, they're routable, but not to the internet.

But

Jonathan Bennett: not, not. Publicly routable. So these guys are using this like a really really big 192. 168. 0. 0

David Taht: Yeah, the rfc 1918 is the actual specification, right? They ran out of 10 space They had the amazon used up 16 million addresses And they said, ah, let's go do something else. So it's not quite on the cia's numbers.

Let's use these other numbers Because they work And you know, again, they were in a bind. They were growing like weeds and I, I don't, I, I, I'm okay that, that they, but not all 260 million, we reserve these things for future use. Okay. Not for Jim Bezos's use. This constitutes an enormous taking from the public sphere of what we could actually do to make the internet a little bit better as we crutch along to the IPv6 transition.

Right. And it's all, again, it's blowing up again at the RIR level and for all I know that the responsible organizations will step up and say, Hey Jim, can we , Jeff, I should say. Hey, Jeff. C Could we have a couple of those? Just a, I I got a buddy in Africa. Really needs some. Yeah, just a little bit, you know.

And and at the same, this,

Doc Searls: this is written up where? Right now, I

David Taht: mean it, oh, it's in the register. Okay.

Jonathan Bennett: I, I dropped a, a link to the show note you can just search for. I see. Oh, I see it. I see it. I give you four block activism is the name of the, the link. So, now I'm curious, is, is Amazon actively talking to the RIRs saying, Hey, it would be really great if those IP addresses didn't ever get turned on.

Is that a conversation that you think is happening?

David Taht: Again we're living in a world where 99 out of a hundred people don't know what a packet is,

Jonathan Bennett: right? But there are people, there are people that Amazon that do though, like the, the, the engineers at Amazon know what's going on. They know that they're using these IP addresses and they know how much work like, okay.

I cannot, I'm not sure that I can comprehend in my mind the amount of engineering work that the guys at Amazon would have to do if the RIRs suddenly said, hey, we're going to start using these IP addresses. Amazon would have to renumber their entire cloud. That is mind boggling.

David Taht: But imagine asking the rest of the world to have to upgrade to IPv6 instead.

Well,

Jonathan Bennett: it is also a mind boggling amount of work. It is ridiculous. But this gets back to this question, though. Do you think the people at Amazon, in the know, are then saying to, you know, your regional operators and to the big, DI for, I forget the acronyms. I, I can't remember all the acronyms exactly, but, you know, Anick and, and ripe and all of these guys, please don't start using two 40 slash four please.

David Taht: please. I, I don't think that conversation's happen. It's more like we're Amazon, we're gonna do whatever the hell we want. , , you know, the R IRS are really small. I mean for example, so for those of us

Doc Searls: who have friends at Amazon, I only have, I only have one, but I think he's been on the show. I'll leave that one nameless.

I'm not even specified gender, though it's not hard to guess. What do we say? I mean, as somebody who works for AWS,

David Taht: what do we say? So, I'm not even sure there's, again, I'm trying to raise awareness of this stuff. For example, I know several people at Amazon that are deeply bitter that they didn't make the investment in IPv6.

They had a good architecture for it, and they are working towards making IPv6 work. They have to, eventually. And they feel some of the same guilt. That I'd hope more people feel about the initial shared spirit of the internet. So if I didn't have People on the inside that didn't feel bad about it. I would be truly angry But I'm sure Jeff business doesn't know or you know

Doc Searls: So my involved anymore, so I you know, yeah a guy whose name it we all forget or did never

Jonathan Bennett: knew Yeah, so there's a there's a point here I can't remember if we were talking during the show or during the pre show but this idea about people in government Don't understand what a packet is and like Almost nobody at the upper levels of government in the United States at least like you talk senators and congressmen.

There's Maybe maybe a dozen of them all together And that is probably being very generous that have a clue about like how the internet works What a packet is what things like net neutrality actually means and I think at businesses I know this because I work with small businesses. You have the same thing.

The people at the top that tend to make the decisions don't actually understand the technology. And so there's this constant tension we'll say between your engineering people who have very strong opinions about. Things and your management and upper management that also have very strong opinions about things, but they're coming at these, these issues from two totally different perspectives.

And it's, it's fascinating to see how that conflict plays out in the way the business gets run and how that impacts things like.

David Taht: Yeah, well said. I got an analogy though I hope will work on more governments, more politicians. If you don't have a physical address for the voter, you can't send them campaign literature.

Ah. Every device needs a physical, a virtual address in order for you to be able to send it spam and campaign literature. Oh, great. It's a little cynical. It's just a little. It's horrifying. Go ahead, Doc.

Doc Searls: You know, that I just said it's horrifying. There's no place to go with that. So

Jonathan Bennett: Hmm. I saw something, I think it was on Reddit.

Somebody asked, Why don't we just use MAC addresses instead of IP addresses? That's a

David Taht: really profound question, actually. It almost

Jonathan Bennett: happened. It is an interesting question. I think it would be a routing nightmare. But it is an interesting question.

David Taht: Believe it or not, it wouldn't have been. Honestly, there

Doc Searls: was an actual I mean, I always saw an IP address as the abstract layer on top of a MAC address.

Yeah. And then domain name, you know, or another abstraction through another different administrative system, but a different set of conventions.

David Taht: But, so,

Doc Searls: what's Amazon I mean, so, they're just, they're squatting on these. They could give some of them to The African nick or what? Afro Nick or

David Taht: whatever it is.

The five r IRSs? No, they could give them back. It weren there isn't the first place . Okay. Well, I I would not have a problem with the transitional phase.

Doc Searls: I read here some ham radio operators sold it to them.

David Taht: Oh, okay. That, that's a different thing that happened. This is a, that's is a good counter example.

Doc Searls: Okay. Different I, okay. I made the mistake of trying to listen to you and read the thing

David Taht: at the . Okay. This is a good counter example. So John Postel was approached in 1988. or so, by some ham radio geeks, and they asked them, Hey, it would be really cool if ham radio could do the internet thing. And he said, sure, here's, here's 44 slash eight.

Go ahead, have at it. Kind of foolish in retrospect. So whoever ended up controlling that address was suddenly sitting on 50 per IP. Sixteen million times fifty dollars is? A lot of money. Okay. Nine hundred million dollars. A ridiculous amount of money. I can't do the math. It was cheaper than Anyway that group of people that ended up sort of accidentally on a handshake agreement with John decided finally that they weren't using most of that and they sold off a block of it to Amazon.

I forget what the numbers are. They're published somewhere, but they, they, they pocketed well over a hundred million dollars. They formed a nonprofit and as a RDC, they are going out and funding wonderful ham radio projects. If there's anyone that shows that wants to be a ham, go to a RDC and apply for a grant.

And they've managed to use that accidental allocation for good. And, I'm very happy with what happened in that case. And this could happen all over the world. Apple could say, Hey Africa, we're not using part of our allocation, we'll let you have that. You know, we can manage to squeeze more space out of the IPv4 internet if we figure out a way of cooperating.

And it would be awesome if Amazon could work that way too. Here's another example. I'd like IPv6 to take off. And you know how Apple has a you know, walled garden? All right. In that walled garden, they mandate IPv6 supporters on all their devices. So, an Amazon could do that, a Tencent could do that, and we would manage to get rid of this immediate.

Crisis of artificial scarcity. Now, is anyone there as enlightened as I am?

Doc Searls: There's a lot of idealists. Not any queer Dave, that's sorry. So,

Jonathan Bennett: okay, I'm curious if you have a feel for this. There's a lot of these blocks of IPv4 addresses that are dark. They've been assigned or they're not used for various reasons.

Do you have an idea of what percentage of IPv4 of the entire space is actually being used?

David Taht: That isn't a good number, and I don't have it on me. I'll argue that probably 20 percent of it is not being used today, and that is exclusive of the U. S. government holdings. It may be lower than that, closer to 12.

Don't quote me, somebody had research in on the show. For example Eric Raymond has been holding on to two address, two two slash 24s, 512 addresses for the last, 30 years because someday he wanted to use them for something and it's really hard to get them routed and this is funny, I guess we, we did a deal years ago.

I agreed to pay him two bucks a year to rent them from him, 2 bills and under the condition I give them back in 2038. Actually, they've accrued more value. I'm not sure if Doc will get the joke. It looks like you did, Jonathan.

Jonathan Bennett: No, I didn't. That's, that's when, that's when Unix time breaks. Oh,

Doc Searls: of course.

Okay,

David Taht: gotcha. John, and so the bet I made with Eric is that in order to keep him incentivized he works on NTP. And so if the time If that works, and we survive the holocaust of the time rolling over, he gets his IP addresses back.

Doc Searls: Okay, that's good.

David Taht: Two bucks a year is a great little bet. I hope to use those addresses up someday in some worthy cause, and that might include using something to route, like, 240, or portions of it.

I don't know. But it's one of the funnier investments I've made in the IP address market. That's fun. Thank you, Eric, for being on another joke.

Jonathan Bennett: That's great. So I, okay, I, I, I want to get back to this, this question because you, you kind of blew my mind by suggesting it was possible and that is just using MAC addresses instead of IPv4 and the reason I think, the reason I think this is so crazy is because I know on a local network, the way that you handle MAC addresses, this MAC address to IP is You've got a table in each of your switches that just has memorized, okay, this Mac address is down this port and your network kind of loses its mind when you run out of spaces in that table.

And when you do the math about how many possible Mac addresses there are, let's just say you need a really, really, really big table.

David Taht: Yeah, it's true. So I was going back in history, you know, I, let me get my walker so I can take you a trip back. But there was DECnet, there was IPX, there was a bunch of other potential standards.

And IP predated Ethernet. The MAC address idea blew people's minds. It was a brilliant idea, but it wasn't hierarchical. And in order to be able to route, you need to be able to say, You 200 boxes are here, you And, and route. So yes, if you were to use MAC addresses to route over the internet, you're screwed.

However, if you prefix, CLNA tried to prefix a whole bunch of bytes in front of that to provide the routing functionality back end. It was a competitor to IPv6. And and it would have worked, and it might have worked better than IPv6 and instead we ended up with this crazy, somewhat crazy scheme that became IPv6, which again is a hierarchical routing system.

We, we take that enormous MAC address, we hash it together, but the prefix you get is stable and hierarchical and that, and that's the scaling factor. So you know, the 2. 0, whatever you all are there, that's a Pretty small number that only requires, I think the current IPv6 routing table is less than 100, 000 entries.

It's doable and it runs, and it runs the whole internet. The IPv4 routing table is almost a mega, a million entries now. And that, yeah, it has to be searched really fast on every single package. There's a thing called a CAM memory. It's required, you have this incredibly weird circuit. And Okay, it worked, it scaled, and here we are today talking over it, over all these different things, and up until now, no one listening cared about all this machinery managing to make our cat videos fly.

You know, there's big, we have big things in the future ahead of us, and we need to get out into orbit, you know, beyond low Earth orbit. This is a little, again, I don't mind getting off this topic now, but have you seen the designs for the, for the communication systems between here and the moon in the last decade?

Jonathan Bennett: I have not looked, but I'm very curious about them, particularly, particularly when we go further than the moon. Something I've been curious about for a long time is how do you make the internet work between here and Mars? Like, everything we've ever done on the internet breaks down when you have a round trip time that's that long.

David Taht: And Yeah, it does.

Doc Searls: Was it two? It's just two seconds to the moon, but it's like 17 minutes to Mars, I think. Something like that. Well, it depends. The, the, the eccentric orbit, you know, I mean,

David Taht: so we don't know. We do know how to make it work between here and the moon. Thanks to all the buffer BLT research, for.

Oh, good. That makes sense. Yeah, but not between here and Mars. There was a lot of books, there was a couple books that came out in 1993 that predicted all this stuff. And Snow Crash gets way too much credit. And it was a dystopia. More people should read about it. Read it in that light. There was a wonderful book by Werner Wenge.

that leverage the concept of Usenet and broadcast transmissions to describe a intergalactic civilization. So what works better for communications between here and Mars is to just basically broadcast everything all the time and keep it around and index it. So conversations will be difficult, but emails will get there in 17 minutes, and a copy of Wikipedia will always be present.

Sure. And so on. So that book's called A Fire Upon the Deep. It's very influential on me because it describes a race of alien beings that communicated through sound to each other. In order to be smarter, they would gather together in clusters of four or five individuals and communicate via sound with their friends.

Here's together. And it was, they're cool, really cool characters. I hope someday they make a movie. And then they discovered radio. Instead of having to cluster together in groups of four and five, they could put on their radio headphones and still communicate hundreds of miles away from each other.

Hugely influential on me. And So, Fire Upon the Deep is my book recommendation. Maybe I should turn it over to you. Have you guys heard anything good lately? Besides tweeters? Anything good?

Doc Searls: Yeah, I can say something good. Sure, go ahead. You wanna hear it? Okay, so the IEEE standards It's an IEEE standard called P7012.

It's not a standard yet. We're working on it. I'm the chair of the of the working group. It's for machine readable personal privacy terms. It's completely reverse the way contracts work online now, which is where you're always assenting to somebody else's thing. They agree to ours and we can scale it.

We can make it fun. Dave David Reed was our first chair. Oh, cool.

David Taht: David Reed of Boulder, or David Reed of, of, of?

Doc Searls: David Reed of the End to End Argument in Systems Design who with Seltzer and Clark wrote that, which informed TCPIP, who helped write UDP as well, a great guy, he ran out of patience with it, and two, and two chairs later, I am the chair.

But we, we're drafting this and it's we're, we're fairly close. We have a number of different approaches, but we could frame it up. Pretty well. I think it's gonna be good.

David Taht: I would so love to be offer, be able to offer my contract to humanity or my contract to the corporation.

Doc Searls: Yeah, well it is gonna be like creative comments.

It's gonna, you know, it's gonna be, there's a set of contracts that are, you know, I mean, business friendly enough so that you know they could easily be agreed to like. We have one already that just has, doesn't have a machine readable format called No Stalking, which says, go ahead and show me ads.

Just make sure they're not based on tracking me. A variety of that would be, okay, track me on your site, but nowhere else. But

David Taht: nobody, I think, is really, well, I would think nobody here is willing to say, okay, track me. At least I'm not. By the way, I recently had my Luddite moment. I got a new phone.

Heh heh heh. And instead of reloading my Signal, Telegram, Matrix, IRC Zulip, I forget Facebook, how many other chat programs, no, 10 different chat applications, I said, screw it! And I went back to MMS. First benefit, my battery life went to 7 days. I also haven't had a bill for my bandwidth since I did this.

So I managed to cut my bills to something reasonable, and and the silence, wow, I'm outside walking around and I can think again.

Doc Searls: You're not just a rectangle anymore. Escaping

David Taht: the rectangle. Escape the rectangle. Maybe we should have an escape the rectangle day. Yeah. You know, leave your cell phones behind, go out and promise you'll meet someone new, make some music.

Read a book, do the things your parents used to do.

Jonathan Bennett: So it's funny, you asked for a book recommendation. What immediately came to my mind is, I don't, I don't read books anymore. I listen to them. That's just what has started to

Doc Searls: work. This is what people do. It cuts into your podcast consumption.

Jonathan Bennett: Watch out. You know, honestly, I don't, I don't do hardly any podcasts.

I don't listen to hardly any podcasts. I know I'm terrible for saying that. But it's, it's just, it's just the way it is. But my book reading, I tend to do with Audible anymore. And it's, it's fascinating because I used to, you know, take so many physical books and read them. And goodness, I've got a full wall here in the office that is physical books.

I don't think I've picked one of them up to read one for months. And it's just fascinating the way that things change.

David Taht: It's a good weight loss program. I mean, you can just keep doing that. I was going to bring this one up. I'm staying at my mom's place. Which one's that? My father passed years ago. The cool thing about what my parents did to me is they didn't let me watch television.

And they stuck me in his office library. So there I am reading books like this at the age of eight and asking my dad's questions. So I didn't tell you the work there. I am like, anyway so what was cool about that, he had a lot of controversial books and I read them when I was a kid. I did not necessarily understand them.

So I've been here for a few months and I've been rereading stuff that I thought I understood then. And I think it really pays to reflect and have that memory of what happened before and some substance that doesn't. isn't on the internet. So, I won't do a live reading today, but perhaps I thought about, I like the Audible idea a lot.

I would love to hear dramatizations and stuff. I've never really done that,

Jonathan Bennett: so. Yeah, there's a, there's an interesting little quirk with that. You know, Audible uses DRM. And so legally, you can't download. Now, there's ways to do it. I'm not going to tell you how on the show, but legally, you can't download and keep your Audible purchases offline.

And so there's this, there's this whole other question, and this is why I still have paper books. What happens when either A, Audible, Amazon decides they don't want to do Audible anymore, or Audible decides that this author that you like is no longer acceptable. Persona Non Grata, yeah. Is a Persona Non Grata and starts deleting their works.

And so, in fact, I was smiling just a moment ago because my son listens to the show and he says, you know what that means? He doesn't need all those books anymore. It's like, no, no, no, no, no. There's a reason that we have the books. You cannot, you cannot Persona Non Grata any of these authors. Because. I still have a copy of their book.

Whether I agree with what it says or not, whether I think it's a good person or not, their, their words are recorded forever in the form of that book. And I think it's a, I think it's a, I think it's a really dangerous thing that we've come to the point to where, for any reason at all, we, we think that, you know, people's thoughts should be inaccessible.

And that, that, that worries me, that, that we're coming to that point as a society where, you know, you disagree with someone, you think someone's a bad person, that's fine. But to say that their thoughts should be inaccessible I don't know, I think that's, that's a bridge

David Taht: way too far. Don't say, I don't know, defend, yeah, defend that.

No, we have a right to, you know, look what was the name of the guy that discovered gravity?

Jonathan Bennett: I think Isaac Newton is the one normally credited with that.

David Taht: Let me use a bad word, Newton was a dick. He was a reprehensible person, he really was. And if we cancelled him, we wouldn't know about gravity!

Yeah, it's there's a lot of people

Doc Searls: that I Pretty much everybody in history is now unacceptable. I mean, if they're around now.

David Taht: Yeah. I want us to dare to be, I want us to continue as we move forward to dare to be unacceptable and dare to be crotchety and dare to say what we thought.

Doc Searls: There's another show title.

Jonathan Bennett: Well the interesting thing about Newton, you bring him up, they tried to cancel him during his lifetime. Right? The powers that be at the time tried to cancel Newton because he dared to suggest that the sun did not revolve around the earth. Now some of the, some of the nuanced details of this. I was sc Leo, that was Galileo

David Taht: too.

Sorry. Sc Okay. Yeah, sc you can read that and about that in the book. I,

Jonathan Bennett: I . Yeah. But the, the, the point remains though that you have some of these scientific discoveries that people at the time tried to squash. And so there's, there's kind of a, a lesson there of be real careful about the ideas that you try to squash because you don't know what those ideals will turn into in the future.

David Taht: yeah. Kids, kids

Jonathan Bennett: today! Yeah, kids today. Get off my lawn. Alright, well, I think we have I think we've filled up about an hour. It was great

David Taht: to see you

Jonathan Bennett: guys again. It was good to have you. I enjoy, I enjoy shows like this. It's a little, little less structured and more just you and the fat. Is there anything, David, that we didn't cover that you wanted to make sure and let folks know about?

Heh.

David Taht: I have a question. Oh, get a one minute project pitch, product pitch, go ahead though, go ahead doc.

Doc Searls: Okay, here's a question. When, when will we have too many satellites doing what Starlink already does?

David Taht: Ha! The day we have a Kessler event. A

Doc Searls: Kessler event being

David Taht: That's, that's the day we'll

Jonathan Bennett: have when one, that's when one satellite blows up, sprays debris into orbit, and you have a chain reaction.

Doc Searls: Oh, okay, and then we can never pick out of orbit, like MilSat 3 blew up that's another one that's huge, millions of pieces of debris.

David Taht: It's serious, you know, a meteoroid or a Kessler event or an atomic war will render the low Earth orbit unusable for about, at least five years.

Jonathan Bennett: Yeah, that's the, that's the thing people kind of leave out of the conversation about Kessler event.

If you're talking about low Earth orbit, all of that debris will decay fairly quickly. Satellites only stay in orbit because they've got ion thrusters to keep them there. Yeah. When you're talking about low Earth

David Taht: orbit. But if it happens and humanity is critically dependent on that kind of technology.

That's true. That'll be a tough

Jonathan Bennett: five

David Taht: years. Well, I mean, all the kids today, you know, they don't know how to get home without GPS. You know I have given a lot of, you know, in my Luddite mode. I, I'll give talks to explain people how to find your way home when your battery dies and your cell phone and how to find the Southern Cross and the North Star and it's news to people as to how to get around anymore without that kind of stuff.

And it'd be a really dark day if we get so critically dependent on that technology and it suddenly goes away. Yeah. Anyway, you asked me for a one minute project plug. One, look to the stars. Oh, two, I wanted to recommend a wonderful movie I did see recently Twilight Zone, which I saw as a kid.

Had one called Bookworm to Rejoice, with Burgess Meredith in it. And it's about a bookworm that survives nuclear war and gets a chance to read. Just read.

Doc Searls: Remember what happened at the end? I'm not going to say it. It's a mystery for the audience to explore. It's important, and a weird thing about, and that's how old we are, man, I remember, I remember the first time I saw Twilight Zone, it was I forget the name of the famous actor, but he's, he was a prisoner who was on a prison planet.

He was the only occupant there, and they gave him a female robot as a companion. And then when he could leave, he could only take five pounds with him or something like that. And they had to leave the robot there. And And he was in love with the robot, and so they shot her in the face, and the, and she just, his name was Corey, and he went, Corey.

These are amazing short

David Taht: stories, though. It's still almost as fresh and as interesting and as intriguing as now as it is today. They predict the various futures, some of which have happened. I've been catching that recently, that's been my current addiction so. Anyway, he asked me for a one minute plug.

My day job these days is a thing called Libre QoS. And I'm known for the fixing the buffer problem with algorithms like FQ Coddle and Cake. And they're pretty ubiquitous now, but they require that you update your hardware, your router hardware, to the latest and greatest stuff. And the right place to do that was really in your router hardware.

You should be updating and get this thing called IPv6, too. But we developed a box for open source software that the ISP can do, and they just plug it into the switch, and overnight, if you have 10, 000 subscribers, their internet gets massively, massively better. And so for many years I've been encouraging the geek community to go out and fix your own bloody routers and do that, and now I want the geek community to go yell at your ISP, to go install this free software to make it better for your, Neighbors that don't even know what a packet is.

And that would make a much better internet for everybody. We're in our fourth release. And we've got well over a million people using it today. I'm very proud of the team. Herbert and Frank and Oh man, I'm forgetting the CEO's name. Forgive me, man. Anyway and Robert. Forgive me, Robert. And together we are going to make a much better internet for everybody.

On the cheap. If more people go out and ask their ISP to do it. So, that's my plug on the ad for the day. Thanks for

Jonathan Bennett: having me on. I've got to ask you before we go, and I know we've asked you this before, but I will get emails if I don't. Scripting language and text editor. Have they changed? What do you use?

David Taht: I could start making jokes on this one again. Now, Rust has become my scripting language. Ah, cool. I'm kidding. I can't understand Rust at all.

Jonathan Bennett: I had a conversation with somebody, one of our other Hackaday writers, just the other day. We were talking about languages. It's going to be a problem. I think we were talking about Rust and Linux kernel.

And it's going to be a problem because you have so many of these Linux developers that don't know how to read Rust. And I started to type. I'm like, Oh, you understand how to read one language. You gotta understand how to read them all. It can't be that bad. I better go check on this before I say this. And I went to the Rust Replacement Core Utils, which is a cool project, but I went to their source code and like, Rust can't be that hard.

And it's like, I have no idea what any of this code does. This is the hardest thing I've ever seen to try to read. I don't even, like, it was It was crazy. Like if let some equals, I'm like, why do you have a let in an if statement? I don't get this at all. So yes, Rust is hard to read.

David Taht: Yeah. Well, my, my project is being done in Rust.

Herbert is a trainer on that too. So let me plug him, Herbert Wolverson for Art in Labs. And he's been doing a great job with it. There are three really as an old. There are three really amazing things about Rust. For starters, it's upside down and backwards from what C is. So that's part of the unnatural thing.

And if you thought Perl line noise was bad, you got that problem too. But the beauty of it, and it took me two months of intensive study to get it, is this thing called the borrow checker. And With that, you're able to write highly concurrent and parallel programs that scale across multiple cores with utterly first rate algorithms.

It is the first thing that will let you think in parallel and code in parallel without crashing all the time. Oh, cool. So I've seen I've become a believer by osmosis at the kinds of incredibly fast code you can do in user space anyway by doing Rust and hanging out with the master. That said, in my, in my job, I, I do a little sequel and I, Herbert, help.

I've met my match in Rust. It's for the, it's for the, yes, Rust is for the younger generation. And similarly he, he relies on all kinds of great things like ChatGPT and stuff to do templating. And and I still live in Emacs, and I tried to add ChatGPT, et cetera, to Emacs. And, you know, pencil and paper is probably my favorite text editor now.

Sooner or later I'll be carving stuff on stone tablets or clay tablets. Because it'll last longer and these stupid books will. Glad I could re re re answer that question for you. I'm very, very impressed with the borrow checker. It really is a breakthrough.

Jonathan Bennett: Yeah, that's cool. Russ is definitely on my to do list to go and learn a little bit more about it and actually write some code.

Yeah,

David Taht: honestly, set aside two weeks, throw away everything you already knew. The hard part for me was, I, I grew up on stacks, you know, and registers, and assembly language, and heap, and you, those abstractions, you have to stop caring entirely about that, and think about sharing. And how your data interrelates with each other.

Stop caring about how this architecture works and share. And I'm pretty sure that if I had, if I was a little younger and I had more time, I would be embracing the language as fully as so many are today. We're seeing 10, 20, 30 levels of performance improvement by people rewriting Shed and Rust. Yeah.

Finally screwed up your

Jonathan Bennett: I'll bleep it out, it's fine. Alright. David, thank you so much for being here. Thank you for coming on last minute. We had our, our previous guest that was scheduled had a bit of a medical emergency come up and is somewhere in the process of trying to get surgery done right now, so.

We'll have him, we will have him again, hopefully in a few weeks. We've got him rescheduled, but David, it was great to have you.

David Taht: Thank you. Okay, one last thing though. Happy Valentine's Day, everybody. Yes. Get off the internet and spend some time with someone you love.

Jonathan Bennett: Unfortunately, the rest of my day is going to be spent doing the production on the rest of this show, but maybe, maybe a little bit of time tonight.

We'll see. Okay.

David Taht: Great to see you guys too. Miss you. Yep. Thank you, man. I miss you, Doc. Keep on rockin it. All right.

Jonathan Bennett: So, Doc, what do you think? It's always a lot of fun to talk to David.

Doc Searls: It's always great to talk to David. I'm glad I brought him up so you could bring him in. Yes. Because, as I said on our back channel, Dave is an artesian well of good information and advice and commitment and all kinds of other good things we need from the people we depend on to make the internet work.

I, I actually, you know, that, that, that, that, that famous cartoon that the XKCD, Randall Munroe cartoon of, you know, the whole internet is maintained by one guy in Nebraska, right? You know, Dave's the kind of that guy. He's one of those, he's one of those. people, I think it's more like, you know, the internet is this complicated thing that can be improved in lots of ways.

And he's all about improving it all the time. And we all depend on it. And it's a largely thankless thing for, for them, but they, but he has to do it. So he just, he doesn't have much choice about it. He can't help it. Yeah.

Jonathan Bennett: Yes, you know, that, that comic is funny because I think there's actually about a dozen, maybe two dozen of those one guys, right?

Because you've got projects like NTP, which for the longest time was literally just one guy. But there's even more obscure things that we all rely on than NTP, because NTP you can kind of see from time to time. But there's things like the term info file, like how many people know what that is? But if that suddenly breaks we are all in trouble.

And there's there's some of these really obscure things that everybody uses There's just there's one guy or sometimes with the really scary one is where it's still there and it's still being used But there's not a guy that's maintaining it. It's just kind of a zombie project. Those are the ones that really worry me Yeah Alright let me get this in.

Next week, we are talking with S. Falcon, and I do not remember what his first name is, and I just have it in there as S. But he is, I met him through the Fedora Matrix, and he does a lot of things. things, but one of the interesting ones Cloverleaf Linux a Linux maintainer does a bunch of different Linux distros and Linux fun things talking with him next week.

And then the week after that on February 28th, we're going to be two hours early. And we're going to talk with even Upton of Raspberry Pi, and that one is going to be a lot of fun. There's been some. Pretty big news about the Raspberry Pi, not the foundation, but the corporation. They're looking at going public and we're going to talk with them about that.

We're going to talk with him about the Raspberry Pi 5. Hopefully we're going to talk about an upcoming Raspberry Pi 500. I don't have any inside information about this, but I'm going to beg him to make the Pi 500 and put an NVMe slot in it. Ah, it would be amazing. Anyway, that's what's coming up for the show.

Doc, do you have anything you want to plug?

Doc Searls: Yeah, actually. So some people I'm sure listening are watching may be at scale next month in Pasadena. I will actually be there for the first time. Cool. And I will be there as to speak at something that the choir people are putting on. That's kw ai.ai.

Mm-Hmm. . They're a collective of. Characters that want us all to have our own damned AI, and I'm all for that, I've been for that, that's why they want me out there. So, I'm gonna, I'm gonna fly out there As cheaply as I can, meaning it's partly on Spirit. The airline, not the,

David Taht: Not the Yes. Yes.

Doc Searls: Not, not, not, not the ghost.

Anyway, so wish me luck on that. Yeah. If they succeed. Anyway, so but I'll be there. So if anybody, you know, wants to see me or just feels like seeing another old man somewhere, I'll probably be the oldest guy there, as I always am. But anyway, that's a scale next month. Yeah. So

Jonathan Bennett: I'll be there. Snag somebody while you're there from Kauai, because that looks cool to have on the show.

Yeah.

Doc Searls: I know. Yeah, I was, I am going to pitch him. I'm going on the show. Oh, just, yeah. Yeah. It sounds great. Yeah. All right. So they're cool people. There you go. And you can join it. I mean, it's just an open collective.

Jonathan Bennett: All right. Well, thank you, sir, for coming in. Kind of last minute. Both of our co hosts and our guests were both kind of last minute.

I should appreciate it. All right. If you want to follow my work, the best place to do it is Hackaday. Hackaday. com. We've got the security column goes live every Friday. And then there's also the untitled Linux show. That's still over at Twit. And we sure have a lot of fun doing that. That is for now a ClubTwit exclusive, and we record that Saturday afternoons over in the Twit Discord.

So anybody that wants to, you can join ClubTwit and come check us out there. Thank you everyone that caught it live. Thank you everyone on the download. We appreciate all of our listeners. If there's a project that you want to see on the show, let us know. Either drop us a note in Discord or you can email us floss at hackaday.

com. And that'll come right to me, and we will if we can get a hold of somebody, we'll get it scheduled and talk to whatever project it is you guys want to hear about. Thank you so much, and we will see you next time on Floss Weekly.

Episode 769 Transcript

7 februari 2024 | min

Jonathan: This is Floss Weekly, episode 769, recorded Wednesday, February the 7th.

OpenCost. We spend how much?

Hey, this week Catherine joins me. We talk with Matt Ray, the Community Manager at OpenCost. That project is all about tracking where your dollars and cents go for your cloud compute costs. And it turns out it may just keep you out of trouble if part of your infrastructure gets compromised, too.

You don't want to miss it. So stay tuned.

Hey, welcome. It is time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. It's not just me. We have Katherine with us today. Hey, Katherine, welcome. Hey, Jonathan. Thanks. Yeah, it's good to have you again. For those that don't know, aren't aware, Catherine has quite the pedigree of Linux and open source geekiness from all the way from the Linux Journal to now a, is it open source evangelist at Intel?

Is that right?

Katherine: Yes, that is my, my, my official job that I do every day and some nights.

Matt: And some nights, yeah, that sounds about right. So

Jonathan: along this open source path, um, have

Matt: you ever done anything with the cloud?

Katherine: Gee, well, yes, funny you mentioned it. Haven't we all? I mean, you know, is there any cloud even aside?

I mean, today there are so many things that have become ubiquitous in the last 10 years, like Has anybody not had to touch Kubernetes, right? Like 10 years ago, I was like, what, what is that? But even five, it wasn't even as, as, you know, widely adopted. But anyway, yeah, yes, I have. I have touched some

Jonathan: clouds.

The saying that I love the best and it's a little bit cynical, but I still like it is the cloud is just a fancy way to talk about someone else's computers.

Katherine: Yes, and I can't remember which of the the open source people said that. I can't remember who said it Stallman? It might have been Stallman. I don't know.

Jonathan: It may have been. That seems like a sort of cynical Stallman thing to say. There's a bit

Matt: of truth. Other people's computers. Yeah, there's a

Jonathan: bit of truth to it, though. And one of the interesting things about that is when you use other people's computers, they charge you for it. Imagine

Matt: that. You have to pay money for using the

Jonathan: cloud.

Darn it. I know. Well, but that's what today's guest is all about. It's uh, Cube cost, open cost, I think, I think they go by both terms. We'll ask him here in just a second. Um, so we've got Matt Ray with us talking about this project. Is

Matt: this something that you're familiar with, Catherine?

Katherine: I am not, uh, I guess fortunately, I have not been the person in the position to be concerned with, uh, keeping costs, uh, reasonable, but I, you know, I get, I get the, I get the problem that they're solving though.

I'd like to hear more.

Jonathan: Yeah, so let's go ahead and bring him on. Matt, welcome to the show. And I've got to ask first, was there a multi million dollar cloud bill that led to all of this?

Matt: Um, definitely. There were, there were people around a lot of cloud bills, uh, in the early days. Um, so, so open cost is, uh, The name of the project, and it came from a company, uh, that my employer called KubeCost.

Uh, the, the, the two founders were on the, I guess they were on the board team, um, on the monitoring side of it, and they were watching the large volumes of compute and running internal metrics, uh, at Google. And so, uh, They were definitely around some very, very large numbers, and, um, You know, hopefully the chargeback wasn't too bad, but, uh, Yeah, uh, we've definitely seen some, some very, very large numbers.

Jonathan: There's a, uh, I was looking at this before the show started, there's a company, uh, they're the ones behind Basecamp. Uh, 37 signals and they came out, I think it was either 2022 or 2023. They came out and said, we spent 3. 2 million on the cloud this year. Uh, we're going to go back to, to, to real servers. And it was kind of a, I don't know.

I don't know if I would call it a wake up call, but it was sort of a sea change in the way everybody thought about it. So like, you can save money using the cloud. You don't have to pay for your own servers, and oh wait, that costs money, too.

Matt: Yeah. It's always gonna be money somewhere, uh, but you could definitely be more efficient with, with how you spend it.

So let's, let's talk about

Jonathan: that. What, uh, what's kind of the, the 30, 000 foot view of, of this project? project. Um, is it just a project? Is it a commercial offering? Uh, what? What's what are the pieces that go together here?

Matt: Sure. So, so open cost is a cloud native computing foundation sandbox project. So, and June of 2022, uh, KubeCost, um, worked with a bunch of other companies and volunteers and folks to write what they called the open cost specification, which was, uh, trying to standardize how, um, how to compute costs on Kubernetes.

So, you know, when you get your cloud bill, usually it says, Hey, you. Got a bunch of EC2 and you're like, well, how much does that? And then, you know, if you drill down into the numbers, you can say, well, I've got, you know, 40 MX larges. At, you know, 5 an hour. I'm just making up numbers, but, you know, it doesn't tell you, um, how much kubernetes cost.

You know, it might, there might be a management fee for kubernetes, but it doesn't say like the namespace within kubernetes cost, you know, 1 an hour. And, you know, this other one was 3 an hour and you're wasting 1 an hour. Um, and so, uh, what the specification did was it actually says, well, here's how we calculate how those things are split.

You know how you determine. Um, Shared CPU, shared memory, shared storage. How all that stuff is sorted out. And so, they, you know, hammered out a specification for that. And then, um, KubeCost open sourced, uh, OpenCost the implementation. And so, um, the OpenCost project is both the specification and the implementation.

So it's a monitoring engine, uh, that is now a CNCF project. And what you can do with that engine, Is, uh, deploy it within your Kubernetes cluster and it will pay attention to everything that Kubernetes is doing, you know, so it's recording, you know, pods and namespaces and deployments and, you know, all those primitives.

And then it looks, uh, at the on demand pricing from your cloud provider. It says, you know, Oh, you know, MX large, you know, uh, Arch 64 cost, you know, 3 an hour. Um, and it checks in periodically and it says, you know, okay, this is what the price was at that time. I checked spot instances. Um, you know, it works on Azure GCP and, and, uh, and others.

Uh, and then it just records out Prometheus. And so then later on, you can come and say, well, I want to see. Uh, how much this namespace cost, you know, from Tuesday to Thursday of last week. And so you can run those queries against Prometheus. And, you know, pretty much any Kubernetes primitive is stored in there.

Um, and, you know, we have that pricing at that time available to us. Uh, so that's what open cost does. It's, it's. It's also got a UI, uh, so we've got a pretty simple two page, uh, UI for viewing that stuff. You can run it as a Prometheus metric exporter, so, you know, you just run a headless and send those metrics to Grafana or, you know, your other, um, you know, uh, BI tool or visualization tool of choice.

Um, But, uh, yeah. And then recently we added, uh, what we call cloud cost support, which is, um, where you actually go and read the cost and usage billing reports, you know, so the, so if you're using AWS, um, or Azure GCP, they, they have a list price for, for how much everything costs, you know, but, people who spend a lot of money, um, are going to go in and make deals with their cloud providers.

You know, you have what they call reserved instances or, um, savings plans or, you know, some, some of the, the, the billing, as you hit certain thresholds, it gets cheaper. None of that's actually caught. Um, in, in, in, in open cost where we're, we're gathering the on demand prices, but we don't, what we, we don't do what they call reconciliation, um, which is go and fix those numbers based off your savings, but we recently added support for reading those bills, um, and, and adding an API and reporting over that.

So you can go in and, and dig into them, uh, in open, in open cost too. So, it's, it's a monitoring engine for cloud billing and, and Kubernetes

Jonathan: costs. Now, is there any support for, let's say, uh, S3 storage? Uh, you know, that's obviously, that's part of some people's solutions for their, all of their cloud stuff.

Is this just Kubernetes or are there other pieces like that

Matt: that plug in? Um, we are working on adding that. Um, so. So, uh, until like December, um, of last, last year, uh, it was just Kubernetes. And, um, we were working on, on adding, uh, what we were calling external asset costs. Uh, that's still under development.

Um, but with the new cloud costs, you can go and see your S3 costs. You just can't attribute them back to Kubernetes. Um, which, you know. It's actually really useful because there's not a lot of open source out there for reading the cost and usage reports. It's, it's, uh, I mean, I've seen customers with like gigabytes a day of billing data.

Um, because the, the Amazon ones, you know, it publishes out to an S3 bucket, you know, 24 hours, 48 hours, after you consume it. And then it's line item, Hey, here's how much. Compute, you know, here's how much of this m3x large you were using, you know, uh, every day each note And then, you know, how much S3 storage each bucket, how much, you know, network traffic, how much every single line item it's all in that, that cost and usage report, that thing is big.

And you know, what we recently added is you can read it and it's maintained. And it has an API so you can run queries against it, which is, is, is actually pretty new. And so, um, what we've been doing to support that is rather, you know, there are going to be people who that's exciting, you know, I mean, it doesn't sound exciting, maybe, but in the, in the world of open source spin offs, like there hasn't been anything like that.

And so we've, uh, you know, I recently, we recently added like Docker support. So you don't have to run inside of Kubernetes. You could run just the cloud cost part by itself. And so that's, that's kind of on, on the, the, the front. Yeah.

Jonathan: Now, obviously we're, we're, we're kind of focused in on Amazon because they're the, the big player here.

Not the only one, but they're the big one. Um, did you have to get buy in from Amazon to be able to make these tools work? Like, did they have to add APIs for you guys to be able to track this the way you are?

Matt: No, no, they have, I mean, probably, you know, I don't recall what year it was that, that, uh, EC2 and S3 launched, but probably within three months of them launching, people are like, I need better building.

I need to see what's inside this black box. And so, um, You know, they, they, they definitely led with how they published their billing. Uh, where, where they, they're like, fine, you can have a fire hose. 'cause your choices used to be, you get, you get a one page bill that's like, you know, you owe us, you know, 400,000 a month and you know, you just have this line item that's like 200,000 of that is EC2.

And you're like, what does this mean? Right. Or. Or you can eat the firehose. And so the firehose is an S3 bucket where they publish everything. And that, you know, people have written a lot of, you know, scripts. And, you know, fancy, you know, jQueries to read through that stuff. But, um, you know, it's, it's documented.

But it's just a lot of content. And, and, and maintaining that is hard. And so, um, that's, that's why like I, I tell people about this cloud cost because people who know, know that this is hard. You know, know that that part's tricky. And people who know Kubernetes know that tracking Kubernetes is hard. And then tying the billion all together is, is, is even, you know, it's, it's complicated.

Uh, but, and it's not just AWS, you know, so I, I would, we, we definitely support Azure and, uh, GCP. Um, Uh, Azure has, they've been very enthusiastic about open cost. Um, they have actually taken the open cost, uh, you know, the Kubernetes monitoring component and they've integrated it into their own billing. So, so now within Azure, um, it's, it's, uh, from my understanding, it's the first Azure service where you can drill down and see your own usage.

So it used to be, you know, if you went to the Azure billing dashboard, you could see, oh, I've got some Um, some compute. I've got some networks. I've got some storage, but it didn't tell you how it was being used. And now in Azure Kubernetes service, you can actually break it down into namespace. You can see which pods, which deployments, which containers are costing what inside your bill.

And so, um, you know, open cost is it's got end users. You know, people definitely use it. Just I need some. I need these metrics, but it also gets embedded into other solutions. And so, uh, Azure uses it, Grafana Cloud uses it, um, you know, KubeCost uses

Katherine: it. So, okay, a couple things. So, first, you mentioned Azure, uh, has really embraced the project.

Are they contributing to it?

Matt: Yes. Uh, so, one of the exciting milestones for us as a, uh, as a cloud native computing foundation is, um, we've just got our first non KubeCost maintainer. And, uh, it's a gentleman from, from, uh, Microsoft. So, um, you know, if you're familiar with the, the CNCF, there are different levels of projects, um, sandbox, incubation, and, and graduated.

Uh, You, you can move into incubation with all the maintainers working for one company, but you'll never, ever graduate if everybody works at one company. So, so, yeah, we're, we're, we're definitely excited to have, uh, you know, our first non, uh, Kubecost maintainer. And, you know, we're, we're working on, on getting more of those.

Um, we're, we're trying to move into incubation right now. So, uh, that's, that's our current status.

Katherine: Okay, yeah, worthy goal. Um, so, so speaking of the CNCF, that's actually something I wanted, I wanted to talk about a little bit. So, um, again, I think that the people who listen to me elsewhere, like my Intel podcast are probably sick of me harping on the massive scale of the cloud native landscape, but I do, but it's, it's massive and overwhelming and there's.

so many options at any step of the, of the development process. Right. Um, anyway, I, so I wonder, so when you come in, so I have several questions, actually, first is I I'd like to hear about the experience of entering, uh, and, you know, contributing your project to the organization and becoming. You know, in, uh, becoming a CNCF project and then, you know, and then finding your way as a project and your identity and figuring out how you fit into that landscape and how to promote yourself and how to attract contributors and all of that stuff.

How do you, how do you approach all of that?

Matt: Yeah, well, there's a lot to unpack there. Um, When, when the project, uh, wants to enter the sandbox, uh, you, you write up, you know, hey, we have this open source, we have this project that we would like that is open source, or we're going to be open sourcing. Here's what we're going to do with it.

Here's what, how we think it fits into your, uh, into your landscape. Um, the, uh, the CNCF has a, uh, a board, you know, the, the. technical oversight committee, the TOC, and they review sandbox applications. And, you know, they, they will look at it and say, like, you know, nobody, you know, nobody cares, or this looks compelling.

Um, how do you see yourself progressing? What, you know, what, you know, what, why should we accept this? What are you going to do with it? How does it fit? Um, And so, so OpenCost definitely fit a niche within the CNCF that didn't, wasn't filled. You know, there's no other FinOps projects within the CNCF yet. Um, and, you know, Kubernetes is near and dear to the CNCF, obviously.

It's, uh, the first graduated project. Um, And so, you know, we got in and immediately had to start doing a lot of cleanup. Um, OpenCost was called the KubeCost Cost Model. It was already open source. Their engine of their commercial product had been open source, but it hadn't been actively part of a community, or, you know, part of a foundation and a larger open source community.

It was, um, You know, kind of the, the, the classic, Oh yeah, we've got some open source. You're welcome to kick the tires and look around, but you know, nobody, very few people used it by itself. Um, and so. Kind of, you know, I, I joined a little bit of, I, I was at KubeCross when it was open sourced, but I wasn't the community manager yet.

And so when I, when I, you know, shortly after that, I became the community manager and started doing a lot of cleanup. Um, you know, telling people like. Changing the read me. So if something's wrong, you go to Slack, you don't go to KubeCast support. Sorry. You know, it's open source now you, you own both pieces.

Um, but yeah, so, so a lot of things like, uh, You know, internal variable renames or, you know, just cleanups in the readme and documentation to point people back to the community, um, to let people know, like, this is, this is open source. It's not a commercial offering. You know, this is, um, you know, we're as a, as a open source community, we're going to support each other.

Uh, but, you know, it's, it's divorced away from KubeCost a bit. Um, that's not to say KubeCost isn't involved. They're, you know, easily, you know, 80, 80 percent of the commits are, are from them. Um, but they're becoming They're getting used to being good open source citizens, right there. It used to be like, this is our thing.

We just commit to it. And now it's like, Oh, we have maintainers who don't work here. We have people, other people committing to this project. You know, we're, we're, you know, so now we have, you know, uh, you know, fortnightly, you know, uh, community meetings every two weeks. Uh, we have a calendar, we have our slack.

Um, you know, we have all the social media for, for open cost. And, um, yeah, we're, we're, Forming a, a, a kind of a fledgling community. Um, you know, we've got about a thousand people in our Slack. Um, it's, you know, I've, because we're moving into incubation, I've been, you know, running a lot of, uh, due diligence about, you know, hey, this is, this is how healthy we are as an open source project.

You know, because there are the kind of metrics you look for, you know, where are the contributions coming from, uh, who's actually using it. in production, who's using it as, as an integration, who's using it as an end user. Um, CNCF wants to know all that stuff before they, they move a project out of the sandbox.

Um, there are I think there are like 120 sandbox projects, which is why that landscape document is so crazily large. Um, and then maybe there are 60, uh, incubation projects and I think there are 25 graduated. So incubation means it's a healthy project. You know, this is kind of a, a green light to other folks that like, look, we think this thing has legs.

It's got a pretty healthy community. Um. We're seeing regular releases, you know, they're responsive to issues and, and PRS and it's, it's, you know, it's progressing and, you know, the, you know, tuning, tuning our own horn. I open cost is one of the better sandbox projects. Um, and then last year for the 2023 wrap up, CNCF said, you know, we were, uh, A top 40 Linux foundation project.

Uh, and you know, whatever metrics they're using, it's really, it's related to, you know, contributions, commits, releases, responsiveness. Um, so, you know, that's, that's high praise, uh, at least, you know. To me.

Jonathan: Yeah. No, one of the, one of the metrics that I like paying attention to that's just fascinating to me is your number of open issues.

You know, is it kind of a, a steady line or are you on the logarithmic curve of

Matt: open issues? It's that, that is a tough one because the, the funny thing about, um, about, you know, when KubeCast, uh, contributed it. to, um, CNCF, they kept the get history. And so the project already had a hundred issues that, you know, had been open.

They'd been using it as their commercial, um, you know, issue tracking. And so I kind of had to go through and say like, this is a commercial issue. This is not an open cost issue. And, you know, just kind of pruning the backlog and, and turning on the stale bot and, you know, Generally telling people like don't open your KubeCross issues here.

Um, you know, we, there are repositories for that and, you know, we've, we've updated, you know, some of the, the templates like, you know, where does this go? Um, but, you know, it's KubeCross is, is, is definitely a good open source citizen. You know, they're, they're trying to, you know, hey, this is great. You know, we're going to fix this issue in OpenCost because they are downstream of OpenCost.

And so, uh, it's, it's, it's pretty, pretty good relationship. And now we've got, you know, we've got Grafana cloud. We've got Azure as downstream of open cost too. Yeah. And others. Yeah. Y Yeah.

Jonathan: So there, there is a term that you used a minute ago that I wanna dig into because it's not one that I'm particularly familiar with and that's finops.

Yeah. I can take a guess at what that means, but let's talk about that. What, what are finops, what does all does that include? Uh,

Matt: so, so finops is, uh, the, you know, the Linux Foundation is a, is a very large tent and, uh, one of their, um, sub-projects, uh, is. or one of their sub foundations is called the FinOps Foundation.

It's um, it's the finance and operation, it's the intersection of finance and cloud operations. Uh, so understanding how to track what's going on in your cloud bill versus how you're consuming cloud. You know, so as, as you kind of mentioned in the pre show. You know, you're luckily, lucky enough not to have to worry about the bills, um, but as you start to get to scale, uh, the, the, you're going to run into like, wait, why are we spending, you know, half a million a month on, on this?

Could we be doing it better? And you know, it's not just turn everything off. It's how do we, you know, how do we tailor our consumption? Are there things that we could be doing? to improve this. And so the FinOps foundation, uh, has actually published a lot of guidance. Um, uh, uh, the FinOps framework explains how to kind of start tackling this problem, how to think about gathering those numbers, you know, gathering up your metrics, um, Testing your, uh, your assumptions about, you know, how can we fix this thing?

How can we make savings? You know, what, what are the things that we can do to optimize our spending other than, you know, just turn everything off, which is actually sometimes step one is like what's running that shouldn't be running because you're paying for it anyway. Um, and then, you know, repeating that process.

It's, it's not just, unfortunately, it's not a silver bullet. Uh, you're, you're going to. Uh, you know, iterate over the process, but they have this, uh, crawl, walk, run, you know, methodology and in different, you know, phases of, of, uh, operations and finance to kind of tackle that. And so. Yeah, there are a pair of, you know, there's an O'Reilly book that's now on its second edition, um, uh, called, it's on my desk somewhere.

Um, you know, called like, uh, right. And, um, you know, they now have a conference and they have a very active, uh, you know, seeing, uh, not CSF, but they have their own Slack. And, um, It's growing like wildfire because everybody's on the cloud and everybody's got this experience. And so, uh, part of what they do is, um, you know, they certify different solutions as part of, you know, Hey, this is part of the, how to solve these issues.

And so open cost is a, uh, FinOps certified solution, which means. You know, we're, we're a tool that you use to solve these problems. And so digging into your Kubernetes usage, um, digging into your cloud costs, that's what OpenCost does. And so, uh, we're, we're not, you know, we're not going to make recommendations, but that's the first step is having good metrics, knowing what's happening.

And sometimes it might be as simple as, Oh, check it out. You're paying for all this compute. That's not even being used. Um, no, and so there are a lot of tools built on top of open costs that are going to make, you know, recommendations. Um, sometimes it's, it's, you know, just, uh, you know, pattern matching where, oh, when you see that 50 percent of your compute is unused, maybe you should resize your cluster, you know, maybe, or, you know, maybe, uh, You're paying for, you know, a dozen medium instances when you'd get, you'd be able to get away with three extra larges and save money.

Um, you know, so there, there's a lot of optimizations you can do there. Um, OpenCrust is gathering those numbers. People have built, like, machine learning solutions, you know, AI on top of that. Uh, but, um, KubeCrust is, that's what they're doing. They're, they're taking all these metrics that OpenCrust gathers and they're building a whole lot of Optimizations on top of that.

They've got your recommendations reporting budgets all sorts of Great stuff, you know machine learning You know fancy dashboards. It's all in there to dashboard

Katherine: um, so you kind of I feel like you kind of hinted at this at a couple spots, but the type of optimization you're talking about that that that brings costs down also would necessarily maybe be related to, uh, sustainability efforts and the kind of sustainability efforts around the CNCF.

I wondered, um, I know there's a sustainability working group. Are you, are you involved in that? Are you plugged in?

Matt: Yeah, we, we, we definitely, um, Um, definitely are paying attention. Uh, so you know, the, the, the CNCF has a technical advisory group, uh, sustainability tag. And I've gone to, uh, you know, I lurk in their channels, I'm on their mailing list and, and, uh, I'm involved in, you know, I've gone to those and I said, Hey, I have this great CNCF project that is tracking all of our Kubernetes usage.

Um, what can you give me? So I could say like, you know, when I use Intel instances, they cost this much in carbon when I use it, uh, our arm, it costs, you know, this much less, um, can you give me those numbers? And they can't yet right now. Um, what you get for most of the, uh, most of the carbon costs is, oh, here is your total compute carbon footprint.

Uh, which, you know, you could look at that and maybe if you, if it was all kubernetes, we could split it up and, you know, eventually tell you, like, here's how much carbon that namespace costs. We can't do that yet, but we're working on it. So I'm working with, uh, there's another open source project, uh, called cloud carbon footprint, uh, that is working on getting finer grain numbers, you know, right now, um, Um, like I said, most of the numbers are at the compute level.

They don't tell you like by the individual machines. And so, uh, we want, we, we need, you know, we need the carbon numbers for more than just the service. We need it down to the, the machines or, you know, the, the, the S3 bucket or, you know, whatever it might be, because then we can actually correlate it back to your Kubernetes usage.

But we are working on that. Um, Yeah. And, you know, fingers crossed, we'll, we'll have some announcements around that soon. Uh, but then that'll unlock all sorts of optimization opportunities. You know, you'll be able to say, okay, um, I can, I, I, I see I could turn off some stuff that's, that's good for carbon costs.

Um, I could, you know, potentially I see that I have this, you know, this one workload that is super intensive. Uh, we could move that to some arm instances. You know, maybe that'll save us some money or, you know, maybe. You see, you know, something that is just burning through through, you know, burning through money and carbon.

Um, it might be like that's where we should be optimizing our performance. And so, uh, it's it's an investigative tool. You know, it gives you kind of where you should be looking and what you should be fixing. But, um, you know, We are going to have something in the carbon footprint in 2024.

Katherine: In time for a cube con coming up, maybe, I don't know.

Matt: don't want to jinx it, but we're working on it. I don't know. We're working on it. There's a lot of moving pieces, right?

Katherine: Paris is coming up pretty quickly, but maybe North America. I know,

Matt: I know the, the, the cloud providers. You know, they, they talk about providing those numbers, but, um, some of those numbers are only available through like their billing, um, visualizing, they're not exposed in their public APIs.

So they have them in internal APIs. But not public facing ones. And so, you know, I'm behind the scenes talking to different cloud providers saying, Hey, if you publish these numbers, we can ingest them and then turn around and show how much that costs. And, you know, they're, they're receptive to it, but it's, you know, it's something that they're slowly doing.

So I can't, I can't promise that it'll all be there, but we will, you know, there will be one who breaks, breaks open the dam for the others. And similarly, the Finops Foundation has a project, uh, an effort that they're calling the Finops. Um, uh, the fin ups, open cost and usage specification focus. And so what focus is trying to do is take all these different cloud bills and standardize and normalize them into one format.

So that way. You know, the, you know, reading your Azure bill, reading your GCP bill, you know, your Oracle bill, it's, it's each one of those is unique and different. And what focus is trying to do is normalize them and make them all use the same terminology. So then you'd be able to compare your apples and oranges.

You'd be able to say, you know, my compute on Oracle versus my compute on AWS. Um, it's, you know, this, this is cheaper for the same instance types. Um, And so, you know, we're, we're involved in that, uh, you know, that, um, standardization process too. So, you know, hopefully, uh, we'll be, we'll be bringing focus integration and, uh, carbon footprint integration into open cost, uh, in 2024.

Jonathan: So I'm, I'm thinking about this, and one of, one of the first things to throw out here is, you know, when you're talking about sustainability, there's more to it than just carbon, but that's kind of like the, the, the one easy thing to talk about that kind of refers to the rest of it. And, you know, we're not going to dive into all of those details here, but one of the things that comes to mind is there seems to be sort of a correlation, a one to one correlation, um, between the amount of money you're spending on your cloud compute and the amount of Carbon pollution and all of that that is the result of it and so it seems like this is one of those places where the the two, uh, the two goals of You know not having more expenses than you need to and not polluting more than you need to really go hand in hand and I just I have to say it's really nice when that works when when being good stewards of the environment Is the same thing as trying to run your business well.

I wish it worked out that way everywhere.

Matt: Yeah. Well, there's, there's another, I mean, that's definitely true and, and definitely love seeing that. Um, there's another level to the carbon footprint though is, is It actually, you can find out which data centers get their power from which sources. And so you might actually spend the same amount of money and you know, I'm just going to throw names out there like, you know, an Oregon data center versus, um, one in Texas, but one of them may be.

You know, powered by hydro, and one of them may be powered by coal, and they might charge you the same, but they have very different carbon footprints or depending on the time of day, you know, you may be having different carbon costs, uh, you know, associated with solar, um, you know, versus nuclear or whatever it might be.

And so. Uh, there's another level of optimization that happens. You know, yes, you can save money just by lowering that. But also you might start looking at like, Oh, does this workload actually have to be, you know, in this data center? What if we moved it to this one? We're not going to save money, but we'll still lower our carbon footprint.

And so that, that's kind of a secondary effect that we're going to get out of it. Oh yeah, that's

Jonathan: fascinating. I, I like that, that you, you just, you present that data and you let the company make the, make the decision, you know, how much do they care about this particular thing versus just simply saving costs.

Um, that is, uh, that's pretty nifty. I like that. So we've talked about this, this term cloud native, the cloud native computer foundation, how that's part of the Linux Foundation.

Matt: What

Jonathan: what exactly? I've had people ask me this before. When we say cloud native, what actually is the definition of that? What what boxes do we have to check for a, you know, an application or deployment to be actually cloud native?

Uh,

Matt: that that's that's a good question. I mean, to me, to me, I've, you know, I've been doing this for a while. I, um, my, my answer would be, uh, it's cloud native if you don't ever touch a box. Um, and so, uh, you can still behave in a, in a cloud native. Fashion within your own data center, right? If you can turn on a data center and start running your workloads without having to go and, you know, insert a thumb drive or, you know, log into a box and start typing, um, that feels more cloud native to me.

To me, it's, it's, uh, you know, workflows and processes that are run through automation, um, You know, yes, it might be as simple as, you know, SSH in a for loop, but, you know, we've come, we've come a long way since those days, which is surprising that, you know, probably, probably there are a lot of people out there who, you know, are still not cloud native, um, they, but they're still in the cloud.

And so we're, we're definitely trying to, you know, find those users to, you know, we want to give them the tooling that they need to like. Easily dig into these numbers, even if they're not, even if they're not on Kubernetes, even if they're not automating everything, uh, we want to provide metrics so they can start tracking these things.

But yeah, cloud native, um, you know, it's, you know, it's, it's a, uh, it's a curve for sure about how far along that you are. And of course you're going to have, you know, your, your Netflix and like at the far end of it. Uh, but I think the vast majority of people are still in the middle of that curve where, yeah, we're sure we're cloud native, but, you know, they're still kind of, uh, you know, they've named their servers there.

They're still logging into them and, uh, you know, checking on them. Yeah, well, I think

Jonathan: it probably depends upon how big of scale you want to talk about. And if you, you know, if you're just serving one website and you have maybe 100 hits on it a week, you don't need Kubernetes, right? It's only, it's only when I have.

Well, I mean, maybe, or maybe it makes more sense to run that website off of Raspberry Pi sitting on the bookshelf. Um, it's only, it's only when you think, okay, this is going to scale up to a million people hitting it. Well, at that point, we have to do something else other than the Raspberry Pi sitting on the bookshelf.

Um, and I'm, I'm curious, though, about open cost. Does it fit in to those less cloud native? Uh, applications. You know, if someone is self hosting, if they have their own hardware, um, if they're using Raspberry Pis, is there, is there a place where open cost still makes sense, even for those smaller deployments?

Matt: Absolutely. Absolutely. So open cost, um, does support on premises so you can provide your own pricing and, and billing. So, you know, even without a cloud provider, I could be tracking my internal costs. And so if I'm, you know, say I'm a managed service provider, I could have my own custom billing. And I say, like, look, We've got a bunch of racks of raspberry pies and, you know, your usage of them is you're consuming 60 percent of them and you'll just, you know, kind of make up some, some billing numbers, but you can use it to track usage across those clusters.

Um, I, you know, personally, uh, I've got two internal, uh, clusters. I've got. You know, an x86 and a, um, it's not all raspberry pi. There's some other devices in there, but I have an arm cluster, you know, running K3s that I run open cost on and, you know, I'm just tracking internal workloads just, you know, to exercise that part of the code, but, but also, uh, you know, it's open cost is not particularly, uh, heavyweight, you know, it's, it's, uh, it's not, uh, consuming a lot of resources anyway.

I mean, it's, uh. writing out every minute. Um, so it's not, you know, digging too much. And so there's definitely a usage, uh, for small deployments. If you need to track, um, splitting this up a lot of, you know, it's hard to, to track how much a, a namespace or a workload costs, um, and in a cloud native environment, because a lot of these are ephemeral.

They're coming and going, you have a job that might last, you know, 30 minutes. You might have others that last 30 days. And so what Opencast is going to do is track. Just all those numbers, you know, as things come and go, uh, we'll be able to tell you like, oh, you know, this namespace. Sure, the containers are only living for, you know, 10 minutes on average, but over the month, you know, you had 10, 000 deployments and, you know, it was, it cost this much.

So, um, there's definitely. On prem usage. There's definitely small deployment usage. Uh, we also have some, some really large appointments, you know, people running literally thousands.

Katherine: So we talked about all of those great data that you can gather,

Matt: what do people

Katherine: do with that data? And in particular, is there anything on your roadmap to make it easier to use that data?

Matt: Yeah, well, so out of the box, um, open cost depends on kind of the, the default Prometheus and, uh, the default Prometheus is tuned for 14 days of storage. Um, it's open source. You could change that to whatever you want. Uh, and then, uh, open cost has a relatively simple react UI. You know, it's, it's one page, two page that has calls to the API that renders.

Hey, here's your 14 days of data. Um, occasionally we get folks who are like, Hey, this thing doesn't scale. And they've got a hundred nodes and, you know, they're running some, you know, some queries over a week and it's literally going and saying like, give me every five minutes of data across, you know, a hundred nodes for me.

This doesn't like that that much. Um, usually, uh, what people are going to do is forward that to. Something bigger, you know, uh, Thanos, Mimir, Cortex, there are a lot of Prometheus compatible databases that are meant for, like, longer term storage. Uh, what, sorry, um, you know, what, what people do with that data, uh, you know, you can federate those, bring them all into one visualization, you know, start to compare all your different clusters.

Um, that's, that's kind of what they do with that. Right now, OpenCost is just Prometheus. Um, we've got a couple of issues open to, like, document how to do that forwarding. The support's there. People are definitely doing it. We just don't have it well documented. Um, KubeCost does that. You know, they forward to, uh, Thanos.

Um, Grafana forwards to Mimir. Um, you know, uh, it's a GPL licensed, um, uh, version of, of Thanos. Uh, you know, and so, you know, there's, that's one form of long term storage. Um, we're constantly getting Prometheus, like, query updates and fixes. You know, I mean, Azure's contributed a lot. Grafana Labs contributed a lot.

You know, a lot of people are just saying like, Oh, you know, this query could be optimized. And so the performance gets better all the time. Just because, um, you know, people run this thing in production. Um, we also have CSV export, you know, so if you want just daily dumps of your, your data, uh, you know, we can export them as CSV.

Uh, one of our community members, um, recently contributed a new repository around Parquet export. Parquet is, uh, uh, an open source standard for large, um, data. Exports. Uh, I don't remember which Apache project it came out of, but, uh, you know, so we recently added an exporter so you can get daily dumps of your data to be ingested into some other BI tool, you know, crystal reports or, you know, what, what have you.

They'll take parquet. So, um, I

Katherine: have to ask, this is both my favorite questions and one of Jonathan's favorite questions, but I'll go there. I'll go ahead and go there. I'm a steal it. I mean, it's a great question. I always ask this, but, um, are users surprising you? Like, in other words, um, has anybody used it in a way that was unexpected?

It's designed for a certain thing, but has anybody ever adapted it that you know of to do something a little different and that was surprising to you? Yeah.

Matt: Yeah, I, I mean, one of the, the, the great things is, you know, at KubeCon, uh, we've had a kiosk where, you know, they have a project Sam, a project pavilion where all the, the sandbox projects get to hang out and people are come up and they're always like, Oh, I'm doing this with it. Um, so definitely we get, I wonder that exact

Katherine: thing, the auto scalers and they can automate the solution

Matt: as you can say.

Hey, when this happens, do that. You know, when, when you see these Yeah. When you, when you, when you see these metrics, yeah, so people built autoscalers on top of open cost that say, you know, when my billing does this, start doing that, you know, turn things off. Yeah, please. Right. Don't don't allow. I mean, it's guardrails.

Um, or, you know, when, when the costs drop, you know, maybe make it bigger. Maybe, you know, if you see the pricing start to drop because it's after hours, um, start ramping up our usage, you know, or spot instances are available, you know, start using those instead. So, so definitely we are, we're part of a lot of, um, You know, auto scaling and machine learning and, you know, a I, um, where essentially you're looking at the patterns of usage and costs and then making decisions based on that.

And so, um, People build their own solutions on top of OpenCross, that's, that's always exciting. And so we have a lot of, OpenCross gets embedded in a lot of those, because it's, you know, we're just providing really useful metrics to, to gather that. Um, and, you know, that's, that's what KubeCross does on top of OpenCross.

That's what, you know, Grafana Labs is doing on top of OpenCross. You know, people who embed it, that, it's the engine for making those sorts of cool integrations.

Jonathan: So, I'm, I'm curious, does open cost, are you eating your own dog food? Like, how much, how much of this do you use internally to keep track of things?

Matt: Um, I, you know, the nice thing about being a, a CNCF project is you don't have to eat the dog food, or you don't have to pay for the dog food. There you go. Um, We are not, but, uh, interestingly, uh, the Linux Foundation is using, uh, OpenCost by way of KubeCost. So, um, the Linux Foundation is actually using KubeCost to watch their own internal usage of, of, um, compute.

So a lot of, uh, you know, a lot of the different cloud providers, uh, AWS, Azure, GCP, Oracle, Scaleway, Equinix, um, they're contributing. Compute hours to the Linux Foundation, who then turns around and gives them to the Kubernetes project, gives them to all these build forms, all, all of, uh, you know, all the different projects that need to run CICD and testing.

Um, they have an internal team, uh, the, the SIG Infra for, um, the Linux Foundation that, Gathers up, you know, serves all these projects and and all the incubation projects, all the graduated projects. They're all getting compute, you know, and they all get, you know, you just kind of get some access to this. Um, they're actually using kubecost to track their own usage internally because literally they have thousands of kubernetes clusters, um.

Which is hard to do with OpenCost. Yeah. I mean, it's, it's hard that you, you know, OpenCost is primarily single cluster based and then, you know, you could federate on top of that. Um, but you're starting to build a lot of your own tooling. Um, KubeCost is doing that for the Linux foundation.

Jonathan: Okay. Um, I just had a humorous thought that came to mind, and I don't know that you have any visibility into this, but so it goes like this, you know, there has to be some businesses out there that they're taking the open cost tools and, you know, they, they break things down into their, their different business units and they take those costs and bring them into their accounting.

And so then they have a spreadsheet where, you know, This is how much this particular business unit is making us. This is how much this particular business unit costs. And one of those costs is your cloud stuff. And, and the, the humorous thought that came to mind is how many things have been killed using the data from open cost?

Do you have any stories about businesses, and I don't know that you would, like I said, this is kind of internal data, but do you know of anything that open cost has led to the demise of?

Matt: Um, in a good way, yes. So, so what you've just described in, in, in FinOps and, and, you know, finance talk, um, is, is chargeback.

Chargeback is when. You have an internal bucket of money and you split it among your organizations and you're like look Our cloud bill is, you know, a million a month and, you know, 300, 000 goes to this team, 300, 000 goes to that team and 400, 000 goes to that team. And you know, you start tracking that, those numbers, the company pays that a million, but each team doesn't have their own bill, right?

So you have chargeback where you have internal billing. Um, the precursor to that is showback, right? You're just showing people, Oh, here's how much you're spending. Maybe you should do something about this, right? And, and then of course the first version of Showback is what we affectionately call Shameback.

Where you're like, look what you're doing, you know, um, change, stop what you're doing. And so, uh, Shameback is, is kind of the first stop when you turn this on and you start seeing where all the money's going. Um. Anecdotally, we catch a fair number of botnets. Uh, people have things that have been exploited in their, in their bill.

They're getting this bill and they're like, you know, this month it's, you know, 500, 000 next month, it's 550, 000. They're not seeing the individual things that are like popping it up. But when you start digging into the numbers, you're like, Whoa, it turns out like this. Namespace had been exploited or, you know, you know, maybe, you know, not at the Kubernetes level, but something inside the application had had gone wrong.

Someone had gotten credentials and, you know, all of a sudden, you know, there's a, you know, Bitcoin miner running nonstop. And so, unfortunately, we find a fair amount of that. Um, And so open cost, you know, when you start tying into dashboards and reporting, uh, becomes like just an early warning system. Um, one of the interesting things about the way we're gathering data is, you know, we're using on demand pricing because there's, there's a time delay of when your bill gets published.

You know, AWS and Azure, you know, all of the, all the cloud providers. They're not saying like, Oh, we know what all your discounts are. We can tell you. Within five minutes, how much everything costs? It's actually 24 or 48 hours later. They're like, Oh, you had some discounts. You hit those thresholds. Here's your real bill.

But people want to see like, you know, they want to see the cost immediately. And so we might not always have the exact cost. We might not match your final bill because we don't have those discounts built into Opacos, but we show you changes in velocity. We show you, you know, oh, you're spending a lot more money now.

Put that on a dashboard, you know, and use that as early warning. And so we get used in that function as well.

Jonathan: That, that humors me a lot, that OpenCost is, is almost accidentally, uh, an incident, uh, detection tool. That, that's, I mean, I've been there. I've been there. I've gotten the email from my provider.

It's like, hey, by the way, you know, this particular server is an open redirect for DNS now. Or this particular server is sending out a whole lot of spam. We'd love for you to look into that. And I, it would be great to have an automated tool. tool like this that finds that stuff earlier that you don't have to get the email of shame from your ISP.

Matt: Yeah, good stuff. Yeah. So that, that's why we're a monitoring tool. I mean, you know, we, in, in that, you know, CNCF landscape, we are in the observability realm. Yeah, that's, that's where we are.

Jonathan: Makes sense. It's, it's more than just the dollars and cents. I like it. Alright, so I'm gonna ask you a hard question, because you gotta do set math.

You have to think about all the things that you wanted to talk about, and all the things we asked you about. And that is, is there anything that we didn't cover that you really wanted to cover today?

Matt: I mean, I think, you know, we've hit all the major topics that, you know, we've, we've talked about the roadmap, you know, we've got, uh, you know, carbon footprint is coming.

Uh, we have some more cloud providers are going to be supported, uh, soon. Um, you know, we, uh, hope to be making some announcements around, around those. Um, you know, we are always, uh, always looking for more, you know, contributors, um, you know, definitely join our slack and get involved. Um, Um, you know, we, just like every other open source project, documentation is the hard part.

Um, cloud billing has a lot of knobs and so, you know, a lot of configuration options. Um, there's a lot of Prometheus versions out there as well. So we're this hard matrix of configuration. So that's, that's what actually open source is really good at is, is covering off all the edge cases. And so, um, You know, definitely appreciate everyone who's involved and look forward to seeing more folks.

Um, yeah, I think, I think, you know, we've, we've touched on, on, on most everything. Uh, you know, the, the project, like I said, the project is, is going really well. Um, you know, we've got the carbon footprints coming, more cloud providers are coming. Yeah. Yep. It's all good. Yeah, good stuff.

Jonathan: All right, I've got to ask you two final questions before we let you go.

And that is, what is your favorite text editor and scripting

Matt: language? My favorite text editor is Emacs. Um, I have been an Emacs user, whoo, a long time. And, um, you know, what's funny is, is I actually Uh, I, I started on VI and, um, proper VI, you know, back on, uh, Real VI. Yeah. And I worked at a company that, uh, literally was NetBSD on the desktop.

Um, because the, the. architect of the company was a NetBSD core maintainer and made us all go down that path. And he used Emacs, and so Emacs was the standard. So I started with Evil, uh, which is the VI bindings for Emacs. And eventually, you know, we ran everything through Emacs. And I can't escape its gravitational pull.

Um, I've actually presented at EmacsConf. And, um, I used to maintain the IntelliJ bindings for Emacs, and I've tried to switch to VS Code with Emacs bindings. So, I go, I, I, I'm an Emacs user through and through. Yeah,

Jonathan: and

Matt: then scripting language. Um, you know, I, I, I used to work in a Ruby shop, uh, so, uh, I, I love me some Ruby, but, um, you know, I, I still use a lot of bash too.

So I'm going to stick with bash.

Jonathan: All right. That is a, that is a fair answer. Well, it has been, it has been great to have you and, uh, we're at the bottom of the hour. We've covered a lot and hopefully after you guys make some announcements and we can have you back and talk about those. So thank you, sir, for being here.

Katherine: I guess. Yeah, I'm intrigued. I've learned something. I've, um, I'm going to go research synopsis on the side later.

Dig in.

Jonathan: Yes. I, I think of all the things, the one that fascinated me the most, of course I'm, I'm sort of a security guy, so it's No, it's no surprise. Yeah. But this idea, I like that too, by the way. When you track all of this stuff, it also lets you know when something has been compromised and someone is,

Katherine: money compromises expensive in more ways than one.

Jonathan: I, I kind of wish we had, we had touched on that sooner, so, so many ways because we could have talked about it for a bit more. I think that's, that's funny. That's pretty neat.

Katherine: Uh, sure. Yeah. So, uh, you know, I still do the other podcast. I've got open at Intel, which is fun. And then I, you know, Doc and I still have reality 2. 0. I also should mention, uh, since we're talking about cloud native stuff, I will be at KubeCon in Paris, uh, podcasting live from the expo and my little cute fish bowl.

Uh, so if anybody's going to be there, I hope you'll come by and wave, but

Jonathan: yeah.

Matt: Cool.

Jonathan: Yeah. Very cool. All right. Well, next week we have Kumar. Oh, I'm going to slaughter his last name, uh, Singrikanda. And he's going to talk about open source DevOps at Toyota. And he's got a book he's written about open source DevOps. I'm hoping we can talk about the DevOps stuff, and I'm hoping we can also talk about open source at Toyota.

I think that would be extremely fascinating. Uh, It does. It does sound really cool. And so that is what we are looking forward to next week. Uh, and then as far as plugs that I've got, well, of course, there's the Untitled Linux Show over at Twit. And, uh, for now that is a Club Twit native, or a native, oh my, a Club Twit exclusive.

I suppose native works. It's native to Club Twit. Um, Um, and then the other thing is on Friday mornings, you can go take a look at Hackaday. The security column goes live there every Friday morning and would love for you to check that out. Thank you so much to those of you that caught us live in the discord and thank you to everyone on the download too.

We sure appreciate it and we will see you next time on Floss Weekly.

Episode 768 Transcript

31 januari 2024 | min

FLOSS-768

Jonathan Bennett: This is Floss Weekly, episode 768, recorded January 31st. Open Source Radio. Hey, this week we're talking with Tony Zioli, the founder of NetMix, and the guy behind the Radio Station WordPress plugin. We talk with him about open source radio, the history of internet radio, and more. You do not want to miss it.

So stay tuned.

Well, good morning. It is time for Floss Weekly. It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got a real treat today because our co host is the, the venerable Doc Searles.

Doc Searls: Venerable, but not venerated, I hope. Yeah, yeah, I was the, the, the last host for the Old Floss Weekly, which is, I guess, like a 14, 15 year old podcast, you know, from back when there were dozens of podcasts.

Literally dozens of them. It's really taking off. There are dozens of them. Yes. Yes. Yeah. Yeah. So, yeah, I rode that one to the end. No, no, it's not the end. And now you've picked, you've jumped on that horse and the express continues.

Jonathan Bennett: Yes, and Doc, I'm, I'm happy to have you back. We were, we were talking about in email, but ahead of time I haven't really got to talk to Doc since the end of the show.

And I miss it. I miss it. I miss Doc. So I was, I was thrilled when he was I miss Jonathan too. Yeah, it's true. I was thrilled when Doc was willing to come back and I think that'll be great to have him from time to time as one of the rotating co hosts. So today, today we're talking about radio. Surprise, surprise.

I mean, that's why we have Doc on and we're going to talk with Tony Zioli about NetMix and essentially about internet radio. And Doc, you're sort of all over this, aren't you?

Doc Searls: Yeah, I'm an old radio guy. My nickname, Doc, is a fossil remnant of a character I had on the radio called Dr. Dave. There are too many Davids in the world, so I got this nickname.

And it's back, I mean this is old fashioned radio, so Duke University had a a commercial radio station, a student talked the president of the college to, into buying a local FM station because the owners thought that AM was the whole thing. And they had a little AM daytime station and they let this thing go.

The station is now by far number one in the Raleigh Durham market, but. Back then we were totally freeform college radio. We were also commercial and we had ads that we couldn't sell. So I made made up ads for things that didn't exist. I conducted inter, and that, that was a hit. And so I, I, I started, you know, you know, we had the mumbling pines apartment village and, you know, we have very long instructions on how to get there and end up only yards away from the interstate.

That's great. Anyway, so. And that was not for very long, but it, I, I, I became notorious that way. And then and then I started a business with another guy named David and and cause there are too many Davids, and, and he was possessive of it. So they started calling me just Dr. Dave and that turned into Doc.

But anyway, I've, I've been, I've been obsessed with radio since I was a kid. I, I lived with the radio under my pillow growing up. I listened to everything. I still try to, when internet radio took off, I was on top of it when the very first station, WXYC in Chapel Hill came on, and then when KPIG came on in California, you know, using you know, streaming, I guess MP3s at the time, I don't know what, but that was a thing.

And so I've been all over it, and I follow what Tony's been doing, because he's very active on the same list that I'm on, called Fo, or Fa. which is the correct pronunciation, but close enough anyway. Yeah,

Jonathan Bennett: so I, I do, I, I have been around, it feels like to me, I've been around internet radio for a long time.

It was early 2000s for me probably around 2002, 2003, somewhere in that era, maybe, maybe a couple years after. But I got hooked on WCPE, the classical station. And they were also one of the first, maybe the first, I don't know, classical stations to have an internet stream. But the thing that was so interesting about them is that they had an AugVorbis stream.

So I've been, I've been Linux only for a long time.

Doc Searls: And for those That dates you, because Aug was just The thing back then. Yeah.

Jonathan Bennett: But in the early days of Linux, there were still patents on mp3. And so you just couldn't play mp3s on Linux, particularly if you used a distro like Fedora. And so, you know, it was really, it was really useful back then to finally find, you know, somebody doing internet radio music that I actually liked, classical music, and they had an Augstream.

And I'm curious. And let's go ahead and bring Tony on. I'm curious, how many of these different things that we know about, did Tony actually have a hand in? Cause he's been doing internet radio since like 1996. Tony, welcome to the show and give us. What, what's, what have you been doing since 1996? How have you been plugged into this thing?

Give us

Doc Searls: the background. Among the too many things you've been doing.

Tony Zeoli: Yeah, yeah, yeah. So many things that I do. Right. I so thank you very much for having me on the show today. This is awesome. And I appreciate, you know, that, that I'm here with Doc who I share a listserv with and, and when Doc. I said pho or pha, it's because the list serve is really an ode to the actual bowl of Vietnamese noodle soup that some Phosters or the pha group like to get together in different cities and and talk about internet, copyright, streaming, music, royalties.

And all these kinds of things, and we get together and do that in different cities around the country, around the world, if somebody's in the city, say, let's meet for pho, let's meet for pho, and we might have four or five guys or, or, or women you know, meet at a local Vietnamese restaurant. But what happened for me is, I actually have been involved in the internet since 19, I'm gonna even go back further to 94, 93, 94.

My father had given me a compact computer desktop and I was playing around with it and I was on Prodigy and CompuServe early but then I saw AOL Beta come out and I got onto AOL Beta. And when I was on AOL Beta, I was also a DJ in Boston. I was a club DJ, a Billboard Dance Chart reporting DJ at some of the major clubs in the Boston area.

And, you know, I, I just said to myself, well, where are the, where are the DJs? Where's the streaming? Like, where's the music? Because there was no streaming back then. Right. You know, it was like, you could talk about it, but you couldn't hear it. And I think that Jim Griffin Had done a MP three distribution in 1991 of, or 1990 or 91 of a, an Aerosmith record.

And Jim Griffin is a, a long time music industry I'm gonna say legend. Basically, he, he was at Geffen, I think, and Aral works before that or something like that. But he did something with Aerosmith. But then when I was looking to the, you know, to the, to the worldwide web at the time, which was only a OL to me.

I didn't find any audio, so I had, I had to go out and look for it and I had to say, well, where is the audio component here? And I didn't really know anything about open source at the time. So forgive me. I had to look to a technology called real audio, which was coming out of Seattle, right? Real audio.

And I then said to myself, okay, well, if I want to put myself, I want, I want to put my mix, this is on the internet, or maybe mixes of my friend. In the industry. And I also, I also worked for a record label and I was working in the dance and electronic music industry at the time. And I really knew some of the world's most influential DJs because I had to work with them promote, market them, sign record deals and these kinds of things.

So you know, I just, it just got into my head. I'm a local DJ. How do I get international? And I found real audio and I went to internet, internet world. I remember internet world. It was a. Big giant conference where all the internet new internet companies would converge You know in new york city at the javits center or in boston at the world trade center And so I went to internet world, I think Maybe 94 95 And you know, I had gotten that computer.

I went, let me just step back, got that computer, went on AOL, started looking around, my girlfriend at the time in 92, 90, 92, 93, brought me to a friend's house at Harvard University. And he was on Telnet before before Netscape, you know, Navigator came out, right? Before Mosaic came out. And all that. So he introduced me to the internet, to the concept that it was there and you could share, you could share files over the web.

But I took that away and didn't do anything with it until I went to Internet World and saw the Real Audio booth. And Real Audio was simply just at a basic small table with a small banner. You know, now they're a billion dollar company, right? But they were just two guys and they were like, this is what we're doing.

And I was like, oh, that's super cool. You know, they showed me, oh, here's a baseball. Broadcast that we did with Major, to test with Major League Baseball. Oh, you can stream over the internet. Oh, this is how you can do it. So I went, I still didn't do anything with it yet. I went back to DJing. Was in a nightclub one night.

One of the guys I knew started one of the first internet web development companies in Boston. This guy named Jason Mayo. And he came up to me and said, Hey, do you have any ideas for the web? And I was like, yeah, I really want to put DJs online. And I saw this thing called Real Audio. But I guess it's a game.

Expensive. There's, you know, it wasn't open source at the time, and it was, it was like 10, 000 just to buy the server technology, right? So he said, well, if you have some ideas, you know, write it down on paper and, you know, come in to see me and we'll chit chat about it. So I did some mock ups on some just basic blank paper.

Now, now I use OmniGraffle, right? Or any, you know, any kind of, you know, information architecture software to, to, to present. You know, web layouts, but back then it was just paper and pen, right? So paper and pen, drew up my ideas, brought it to Jason. He was like, that's cool. Well, let's check out this real audio stuff.

So he ended up investing in the real audio platform. They had the money to do that. He ended up investing it. And then I paid a monthly fee and I built my website. And then I started putting the first DJs myself and other DJs like Paul Oakenfold, online Van Helden, Tony Humphries, Carl Cox. You know, from from the start of it 1996 through 2000, I moved it from Boston to New York in 1996 and partnered with a record promotion company, a music promotion company down in Soho, and I just kept kept it going and grew netmix dot com to a million unique visitors by by June of 2000.

When we exited with another company that acquired us, it was a youth aggregator. And so I really took it from 2000. You know, from my bedroom in Brookline, Massachusetts and Boston to, you know, to New York City. And then was part of that whole Web 1. 0 startup doing streaming music and, you know, then Napster.

I was even pre Napster. You know, I was, I was 1996 and Napster didn't come out until like in, I think 97 or 98. So you know, I was really the world, one of the world's leading websites, streaming DJ mixes on the internet. At the time. So that's how, that's how I really got started.

Jonathan Bennett: Yeah. So there, there's, boy, there's several things there that it's really fascinating.

One of the first ones that comes to mind is, you know, I was, I was kind of aware of I'm, I'm fairly young. But that was about the time when I was really coming to start playing with computers too. And thinking back to those days Trying to play music streaming was about all that a poor, a single poor computer could do.

And, you know, we compare it to nowadays. We have our, our, our beasts of desktops. You can have three or four different YouTube videos playing at once. All of them with 1920 by 1080 video as well as the audio. And

Tony Zeoli: We can do this. What are we doing

Jonathan Bennett: right now? We can do this. We can do, we can do a, you know, a live interview with video.

It, it's really, it's pretty interesting to me that pretty much as soon as computers got powerful enough to do live audio, there were people out there like, like you, Tony, and like the real audio guys that went, let's make this happen. And it, it's just, it, it, it, it kind of tickles my my geek bone, I guess.

That, that we've turned to using this tool for music as quickly as we did. It's, it's pretty fascinating

Tony Zeoli: to me. Yeah, it is amazing in Boston. Is really, you know, MIT, MITRE, is really where the internet started, so it was in my backyard where they started building, you know, there's that book, Where Wizards Stay Up Late, I don't know if you've read that, but it's a really great history of the internet, how it started at MIT Research and Stanford, and connecting the dots, and through ARPANET, and stuff, so yeah, somebody taking on me just being a DJ, I didn't go to call, I didn't really go back to school until I was in my late thirties, so I didn't come from a collegiate.

I was just a local guy, local dj, just trying to do something big and innovated on my own, given the impetus of Jason building a web development company. Mm-Hmm and given the real audio guy showing up in my backyard at at the World Trade Center. You know, and help me see the vision of the future. And you know what's funny is, I used to walk down the street in Boston, Yeah, yeah, I'm gonna put, I'm gonna put DJs on the internet.

And you know, all my other DJ friends were like, Yeah, yeah, yeah, that'll never, that'll never last. The internet, what's that? I was like, you gotta be kidding me. You know, and I have a really quick funny story about that, Is that in New York City in 2008, Travis Kalanick from Uber, was launching Uber. So I was at a tech meetup, and he hands me his card, I didn't know him.

I was at a tech meetup, there's like 200 people in the room, he hands me a card, and and I should know better, I should know better, but he hands me a card, and he says, here, you should try my new service, and I looked at the card and it said Uber. And I was like, nah, I live in New York City, there's taxis everywhere.

What happened? Why would I want to take They take this car service, and it'll probably be more expensive, and it'll never last, and now, you know, it's, I, I get in Ubers all the time, it's the funniest thing. So you can't sleep on new technology as the, as the, you know. It's the theory there, like you just can't sleep

Jonathan Bennett: on it.

And I think, I think the other half of that is it's so hard to tell which ideas are going to take root and take off and which, and which ideas are just not going to. Right. So many, so many times I've seen or even had some scheme and it's like, oh man, this is going to be huge. And it's not, nothing ever happens.

You know, it's so difficult to figure out how to. You know, get the lightning in the bottle. Well,

Tony Zeoli: let's go into that. Let's go into open source, right? Yes, that's right. Yeah, I was just waiting to get on that one. Yeah, so that's a perfect segue. Because in 1999, 2000, when I was trying to, you know, grow NetMix, I looked at my developer and said, there must be some kind of platform, some kind of software that we could use to automate.

Netmix and allow DJs to upload themselves and at that point I saw a typo three and then I saw Mambo in the open source world of CMS's and I kind of pushed him and he He may say that maybe because he didn't feel like doing it or whatever I didn't understand it all at the time, right, but I started hearing about Content management systems and I was like, well, how can that help netmix?

And we didn't get to that, and that was the death of us by not getting into content management, by not adopting open source. So now fast forward to 2004, I'm working at the Associated Press, and I'm looking at Movable Type, which was a proprietary CMS for blog publishing. And I'm looking at WordPress, which just launched.

In 2003, 2003, 2004, and I then, like I chose real audio, even though that wasn't open source, I chose WordPress in 2003, 2004 to latch onto because of the user experience and how they had laid that out and actually going through Mambo Jumla and Type 03 and seeing how difficult that was to utilize for someone like myself.

So that is where innovation happens. In the open source community, you know, how someone like myself takes that on and then brings it into my world. And that's what I've been using, you know, for the last 20, 20 plus years.

Jonathan Bennett: Yeah. And, and so there was There's kind of a phoenix from the ashes. I don't know where that, where that fits into this because you sold out of NetMix just before the dot com crash.

Right. I don't know if that was luck or if you saw, if you were able to read the tea leaves. But

Tony Zeoli: we were working on that for, we were working on that deal for about eight months. Okay. So it was, you know, they had raised about a million and a half. And they were acquiring companies with some of that money.

So we were working, so the deal closed. It was, you know, the deal closed on June 1st. So I got out because I needed support. I, you know, I knew I couldn't do it myself. I needed more people. I needed, I needed investment. They were gonna buy the company, employ us, invest in NetMix, and keep on growing it. And they did, but they couldn't survive the dot com 1.

0. Nobody could survive. I mean You know, I was at the last great party though, in New York City on , on, on Ellis Island. It was like two massive sound stages. We all took, it was bo I think it was box.com or box.net, I think back at the time. box.net. Mm-Hmm. . And they took everybody over on, on Harbor Cruise ships.

There was like 2000 people went over to Ellis Island. And we partied until like 2 a. m. and took boats back. And that was, those were some good times. Yeah,

Jonathan Bennett: I believe it. I believe it. Yeah. And so we're, we're in this net mix coming back. You were able, you were able to get the, the name at least going forwards.

Where does that fit into this story?

Tony Zeoli: Yeah. So in 2019 I was putzing around at a radio station here in Asheville, 103. 7. It's a local LPFM and I was working on their website and I bumped into this WordPress plugin called Radio Station that was developed by a woman named Nikki Blight out in Colorado.

It was modeled after Drupal, a Drupal plugin. Sort of the same type of opportunity, but no one had advanced that Drupal plug in and Nikki took over, took it, took over the concept, applied it to WordPress, built the plug in. I found the plug in, thought it wasn't doing what it was supposed to do, contacted her.

She said it wasn't something that she was interested in sustaining any longer. And so she asked me if I wanted to take it over. So I said sure I jumped at the chance because always an innovator always an entrepreneur I saw an opportunity, you know working with lpfms where this could benefit you know this this free and open source plugin could benefit these lpfms and I could potentially turn that into a freemium plugin which we did into radio station radio station pro and I say it's by netmix because I wanted to utilize the Netmix brand in some way since I still had control over it, and I own the IP, I own the patent, I mean, not the patent, I'm sorry, the trademark for Netmix and for streaming audio over the internet streaming audio and video over the internet, so I need to use The sur the the name to apply to the trademark, right?

That, that now you know, is the parent company of Radio Station Pro and, and radio station free. And that's how it all plays together. You know, I hadn't really been doing anything with Nimex since the early two thousands. I tried to do something here or there, but then I got busy working and, you know, trying to go back to school and all those kinds of things.

So now I saw the perfect opportunity. So now what Netm Mix represents is netm mix.com. Is the radio station aggregator. It's it's the, the directory. So it's a directory for anyone who's using radio station free or pro plugin can get listed in the directory. So it's anybody that's in the WordPress world.

It's not all stations. It's only WordPress stations using our software. So I thought that was unique and a nice way to use netmix to highlight those stations that might opt in to want to ping the directory and feature themselves. The feature themselves there.

Doc Searls: So, so, Tony, I have a question about copyright and handling, reporting, and paying for playing on, on internet radio.

And, in part because I was, I paid an awful lot of attention to this back, oh, as far as 1998 when the DMCA came along. Mm hmm. And, and basically said, okay, you guys have to work to, we're not going to set any rates or anything like that. We're going to work this out for ourselves. And the RIAA and the record industry had really seen the internet as a threat very early on, as early on as 95.

And they made sure that the DMCA, starting in October of 98, had this category called webcasting. That's what they call it. Very familiar. Internet radio webcasting. Right. Yeah. And, and, and, and actually came up with some. some rates which amounted to, you know, a few thousandths of a cent per, per play, per listener.

So there was a level of accountability that was not even contemplated by over the air radio. And over the air radio never had to pay any of this kind of stuff. They still don't want to, and it almost doesn't matter anymore because over the air radios is in trouble and and, but in the meantime, I'm, I'm wondering if, if the software you're, You're providing takes care of that.

I remember there was an open source thing called Riverside River something. It was done by Salem. Mm-Hmm. , the Salem Broadcasting group, religious broadcasters. But they had good open source. And, and that handles some of that for your, the more major stations. Right. But I don't know if is, does that, does it handle some of it, does it handle reporting and does it handle well?

Tony Zeoli: That's a good question. Yeah. That's a good question. So radio station free has an audio player, but WordPress has an audio player. It's just an audio player, right? Any embedded audio player that you can get, whether it's open source or whether it's proprietary, it's just a functional software. If the person chooses to use it to stream music, then that's on, that's their responsibility because I can.

I can embed any audio I want in a WordPress, using the WordPress player in a WordPress page. I'm responsible as the streamer for signing up for that, that licensing with ASCAP, BMI, CSAC, Exchange, right? So we just provide the, the software to allow you to stream, and we have it in our terms and, and conditions, and we don't, we don't host the streams either.

We hosted the streams, we'd be responsible, so the users of our software may use Live 365. As a matter of fact, we're talking about Right to Live 360. That's a distributor, essentially, right? A distributor essentially of of, of webcast, of webcasting of. And you know internet broadcasting right they charge a certain fee maybe it's three hundred dollars a month or five hundred dollars a month per Number of streams, you know that you incur over the over the monthly period so someone could use a live 365 Audio feed in our player and then live to 65 in that and that that's that customer of life to 65 would then be responsible because they're the ones who are activating the stream outside of radio station and radio station pro and stream player pro.

We're breaking off. The player itself has its own, you know, individual software that and I'll speak to that again in a minute, because there's another point there, but we don't host the stream. So it's when you're hosting the stream, like you know, Grooveshark or any one of these, Napster, right?

They they provided a mechanism to kind of like, you know, host the parts of these files or. Or whatever it was, right, that the industry looked at and said, no, you can't do that. So it's the hosting component that we don't do. Will we do it one day and, you know, and have that licensing component in place?

Well, I think about that, but I look at radio. co, I look at Live265. I look at MixAlert, I look at some of the other platforms, whether they do offer licensing or don't offer licensing, they're the stream hosts, and people take that feed URL, and they can just use it in our player just like they would do in WordPress with an MP3 file.

So, we don't see ourselves as responsible for someone's usage because they need to bring that licensing with them, and we're not, we're not activating their stream. We're only giving them a vehicle. To play the stream, just as they would on their own, on live 65 or in any other service that provides the technology to do that.

So,

Jonathan Bennett: so I want to jump in real quick. You guys are both. Radioheads and you understand all of this very well. I've got a question because I don't understand exactly how this works What is the the difference between? licensing and licensing fees for say an FM radio station and playing music on an internet radio station and Is there a difference if you're just?

Taking that conventional radio station and playing it on the internet. I remember back several years ago, some of the internet, some of the online radio stations I was listening to were talking about some changes to the law that were coming, potentially coming, that was going to shut them down because their licensing fees were going to change.

And that's what happened. I don't, I don't understand. I don't, I don't know much about what actually happened there. So I, I, I leave it to you guys. Give us the, for those of us that don't know, that haven't been following this, what, what is the history

Tony Zeoli: there? I'll defer to Doc to explain the licensing models and the jump between the past and today and the difference in cost.

And then I'll give you an example of a company I used to work with that was affected by that.

Doc Searls: Okay, at the risk of being wrong in the particulars, but right in the general thing the way broadcasting worked the, the artists, the recording artists themselves. And, and the people who produced the music didn't think to get the royalties for that.

It was the composers. So, so, all, everything played on radio, it, the, the, the royalties go to composers, they don't go to the performers. Right. And that's still that way. So when the internet was coming along, the record industry looked at it and said, wait a minute, this is digital, this is accountable, we can put accounting in this thing, and we can.

These are, I mean, because they, and this actually goes back to the early 1900s when And when the composers especially a guy named Victor Herbert, I think it was, heard his music being played in bars and said, wait a minute, I'm not getting paid for that. And, and then when, once records came along, he said, I want to get paid for that.

I want it as a composer, I want a piece of the action. And so the composers got in early on this thing, but the performers got in, didn't get in right away. But once, once the RIAA and the record industry in general. Saw the internet coming along, they said we could, we could make something with this. And so, the, the original, the original language said, in the absence of a, a market with a real buyer and a real seller, we're just gonna set some rates.

I happen to believe we could have real buyers and real sellers. I think, I think listeners would be willing to pay more for music than their, than the Distributors are charging and, and, but basically what happened was that it went from a few thousands of a cent per play per, per They used to call them needle drops, but, you know, from, from the, for, for the, for the play of a song from a few thousands to some larger number.

And by the way, Deals with Apple and Amazon and Spotify, they're all independent. Okay, they're, they're made directly with those giant companies. And one of the reasons you don't hear familiar music on podcasts and podcasts are almost all talk is because you have to clear rights individually for every damn thing you play.

So it's hard to do.

Tony Zeoli: So yeah for every song, but you can't there's no There's no overarching royalty that's only on streaming, but not interactive

Doc Searls: performance. Yeah, so it's, and there's a bunch of arcana in there, you know, mechanical rights, all kinds of stuff, but but what happened was that the, the, the register of copyrights with the, with the feds with the copyright office sort of arbitrarily said, Oh, it's going to be this now.

And it was, it went up and a lot of stations went out of business. Is that close enough, Tony, you

Tony Zeoli: think? Yeah. Especially yeah, that's close enough. The copyright royalty board, the CRB, right? The crb effectively is the board that sets the rates. Yeah. For broad, for, for webcast, you know, for broad internet broadcast.

Doc Searls: just a, a real quick interruptive thing. I'm sorry to interrupt, Tony. There was something called the copyright royalty arbitration panel that existed before they realized they, it should be called a copyright arbitration royalty panel. So it turned from crap into Carp Anyway, so. That's a real thing.

Tony Zeoli: Yeah. So I was that's a good one. I was I can't stop laughing. I was I was on the board of advisors for a company called A Tracks whose founder David Porter actually started at Live 365 and he went on to, he works at Amazon Music now in the playlisting division. He started this company called Atrax for all, all playlisting.

Then Spotify came about and the, and the change in the rates basically affected Atrax's ability to, to thrive and survive in that era. I think it was around, I'm gonna say like the end of 2009, like somewhere in the late 2000s, you know, early 2000s, you know, early 2010s, that he was hit by that and he had to take down.

He had to stop operating 8 tracks because they couldn't afford to anymore. So yeah, it definitely eliminated a lot of broadcasters. Which is why I don't host streams! Because, you know, right now you know, Radio Station is really a small company right now. You know, we have Thousands of users across the globe, but most of them are small streamers.

And whether they're Gospel, whether they're LPFMs, whether they're community radio stations in the UK, whether they're, you know, a Spanish radio station in the mountains in Spain and somewhere in Spain, you know, they're predominantly all small webcasts are using the free version, which is something that I'm proud of, of continuing to keep free for them.

And we obviously follow the open source concept of, you know, any feature that we bring into radio station pro, we're gonna obviously offer into radio station free. First, it may be at a limited basis, but it's still going to be in there. And that and that way we can grow radio station free and give people a free and open source tool.

And, you know, if somebody forks us great, that's what it's all about. Right? Yeah. You know, that's what open source is all about. So I'm happy to keep that for you. There's a git, you know, it's on git right now under netmix slash radio hyphen station github. com slash netmix slash radio hyphen station.

Anybody can commit if they want to come in and offer up an idea, they can, you know, issue, you know, do something with the code, issue a pull request. We'll look at it and adopt it. And then, you know, we'll utilize it for, you know, radio station pro, but we may improve upon it. And give it a different look or a different feel or different options that are more pro level.

Because, you know, we need to get paid to support this, so that's why we're doing it as a freemium platform. Because it's something that we love doing, but like some open source projects, and I'm thinking about one of them right now, I won't identify, you know, the guy's really struggling. You know, personally, he's really, really struggling.

And I've been reading reports about it and hearing about it. And, you know, I'm fortunate that I have Digital Strategy Works, which is my web development company. And then my wife has her, you know, small business, too. And we're fortunate to be able to survive and own a house. But, you know, building an open source product and marketing it and responding to support requests and bringing in new ideas, that takes time.

And time needs to be compensated for. So, So yeah, it's really cool what we're doing, I think. And then we're going to be supporting other, other open source platforms like Azuricast, like Libretime to make sure that we integrate with Station. What we, what great at Station really was, it wasn't just the player.

We didn't really have the player at first. We had an A player, but it was a fixed widget player. A widget player means you can embed it in a page, but you didn't have that Spotify or Mixcloud or Soundcloud sticky footer player that persists. We introduced that later on. It's really a show scheduling platform.

If you want to present your show schedule online and you want to say, Oh, here's the dates and times of the shows, here's a show page, here's a playlist that's attached to the show, you know, what we played. So that way it gives radio stations the ability to not have to go into WordPress and configure pages themselves and just slap text in.

It gives them the text and the date pickers and the timestamps and the. And the text field options and all that to, to create a show page with an avatar, with a featured image with the show description that's connected to playlist that's connected to in pro episodes. So we went from, from free, we have shows.

And then in pro we have shows plus episodes. And with episodes, you can then have episodes and you can use. Another open source WordPress plugin like Blue Blue Blurberry or simple podcasting to then podcast a custom post type, which means a custom post types are, I don't want to get too into the weeds here, but in WordPress, you have post pages, media, those are post, those are post types that come with WordPress, custom post types, you introduce Into WordPress that are data sets that are beyond your regular post or your page.

So a data set might be shows and shows might have data attached to them. Time, date. Language, you know place in a geographic region, time zone, that sort of thing, right? And so your show pages have all that data. Then we, we said, Hey, let's introduce this sticky player. So we introduced the sticky footer player, which people love.

They can only buy it in pro. They can't get it in free yet. It's coming to free as a, as a basic player may not have all the options of the pro player. But you'll still have that sticky footer player. And we're releasing stream player pro, which is now out, which is just the player itself and not the whole content management aspect of your radio station shows just getting the stream player.

We are in the process of going through. With wordpress plug in review team to get stream player free into the repo. So we're just fixing some some issues that they found that they're very specific about what you need to do And you have to follow the logic of their development rules So they have they had some feedback for us about two months ago.

We've been fixing that now. We're in new review With the plug in team and we're hoping to get stream player free So it's just that sticky footer player out to within the next few months, as soon as they approve us. So that's, that's kind of the process that we've been following.

Doc Searls: So, so I have a question about your, you mentioned some of your customers, or your users anyway, probably both, are LPFMs.

And for listeners who don't know LPFM stands for low power FMs. They can't be more than a hundred watts, which is a light bulb power or more than a hundred feet off the ground. Which means they go out like two, three, four miles. There's some that have more coverage because they're in fortunate locations but for the most part they're very local.

But if they're also on the internet, they're worldwide. You know, so they may have listeners anywhere. I'm also thinking, as you're going over your own process of iterating what you're doing and your offerings as you get the feedback and you get the development going what happened in the radio stations themselves?

I mean, when I started, there were turntables and you slip cued records that were on these big fat turntables with big flywheels underneath, and then those were replaced by cartridges, you know, and so you'd play the radio. the record once, it would go on a cartridge, and you'd punch buttons that played the cartridges.

For the longest time, you could go into a big station or a small station, and the studio looked the same. It was like, you had a board, and the board had knobs on it, and you had cartridge machines over here, and you hear the ads over in this cartridge pile, and there's songs on this one, and, and

Tony Zeoli: now Fortunately, cartridges are

Doc Searls: gone.

Oh, they're long gone, I know, I know, I, I, I, I document this stuff photographically by dropping it on old radio stations. It's a separate thing, but an interesting thing to me, it's a question about where things are going, because you're clearly on top of things. One is as there's so much more optionality now in what somebody can listen to.

Most of the listening devices are actually phones, phones can have an infinite number of apps. And there are some fabulous apps. There's one called radio. garden online and it's radio garden and it's done by, I don't know if it's open source or not, it's done by a guy I think in Holland or somewhere.

But it's basically a globe and you just zoom in on any place in the world and you can listen to all the radio there. And, But even the idea of a station is kind of, it assumes a stationary location, it can only be here. And the sense of geography gets lost when you're online. But the main thing is, there are millions of podcasts now, but none of them have music.

That's the interesting thing. And Well it's because of licensing, yeah. Yeah, because of licensing. So I'm, I'm wondering where you see this going. It's like, radio is like, it seems to be like, Internet radio is where radio is going, because I think over the air radio is, is probably on its way out. AM radio is ending hard, and I think FM's coming down in a more soft way, but the streams matter more.

If you look at the ratings, streams are down there, and the rating is pretty low, but they're coming up. Nielsen shows them coming up. Right. Do you see, I mean, and also I'm interested in, this is almost too much to talk about, what the symbiosis is between the streams that people listen to, whether it's over the air or on a, on a device and where music itself is going at the, on the performance side and how people, what people like and how they share and all that.

What's that symbiosis? I think,

Tony Zeoli: you know, it's interesting that you mentioned this because UMG is pulling their catalog from Tik Tok to force them into paying more money so they can pay artists more for those streams, right? So where you're really hearing music in streaming is on sound, on SoundCloud, right?

On Mixcloud, on Spotify, on Pandora, on iHeartz Apps on there's an old radio bar. Online radio box has a lot of radio stations using their service. You know, I do want to say that while FM, you know, it's interesting because I live in Asheville and I drive around, I don't have satellite radio.

I mean, it's obviously XM has a, you know, Sirius XM has a slice of the market too, as more people adopt and more automobiles adopt Sirius XM in the dashboard. You know, I really think that that the next generation, you know, what five G will really, it just hasn't really effectively transformed streaming radio yet, but I see that as being the catalyst for streaming radio and in my research for my business plan, you know, some of these research organizations that publish, you know, their.

Their market research right year after year suggests that in you know, any Radio itself is going to be is going to grow to a I think they said something like eight billion Or a $10 billion a year business by 2030. So, and right now it's like 4 billion or three, three to 4 billion or something like that.

Doc Searls: So it's like all of radio, all of broadcast. All of radio across the globe. Right? Globally. So streaming and streaming and broadcast over the

Tony Zeoli: year. Broadcast. And the funny thing is, is that we as Americans, you know, we have to think globally. I am married to a person who was in the study abroad industry and she.

And not only did she, but through my history of working in the record industry and working globally and selling records all over the world and traveling globally to to be a DJ and to be in that industry, you know, radio here may be a dying breed, you know, to some extent, most people may think it's a dying breed, but radio effectively in other nations that may not have The types of technologies that we have, or people don't have the incomes, you know, maybe in Latin America or something like that, you know, radio is still a very strong part of their daily life.

So, personally, I have to think globally. I can't just think what's happening in the United States. Obviously, you know, over the next hundred years, maybe that changes, but I won't be here, so I won't need to worry about it. I'll let my kids worry about it. But I do feel that, you know, Blue Ridge Public Radio here in Asheville is plays a really important part and they stream and they're on, you know, multiple stations across the mountains and I listen to them every day for that news and information they may bring the BBC and some of those, you know, the moth radio hour and all those kinds of shows that are very important to me.

So while radio itself may be dying a slow death, Blue If they were, I, I really believe if they were to allow LPFMs to open up to like a 30 to 50 mile radius, you know, that they would be able to be better able to and keep the current format and legal status and not allow them to be acquired by an iHeartMedia or somebody like that.

That we get back to local, because I think people are dying for local. And a lot of the, the LPFMs are quite very local about local, you know, local programming and local artists and local news and entertainment, you know, so I would hope, and I don't, you know, that, that somehow the LPFM industry can push for that, but even still just to get a new LPFM and just to, you know, I tried to look to see if I could get an LPFM here in Asheville.

And there's a process you go through to find a signal that you can broadcast over. And the only one is like up on Pisgah Mountain, which is like 6, 000 feet above sea level. And you have to drive up there, you know, every day, you know, 25 minutes up to the top of the mountain in the winter. That wouldn't be fun.

And

Doc Searls: you have to hike the rest of it because there's not a road that goes all the way up there. And there used to be a funicular, a tram, that ran up there that got overgrown and died. And so Channel 13 is moving off of Pisgah to to Brevard or some other place like that. Right,

Tony Zeoli: right, right. So, you know, I also think, see, you know, some of our customers and our free users are college radio stations, right, which you mentioned earlier, and community, you know, just community stations that maybe not be LPFM.

So, you know, I think there's a bubbling little market underneath that just kind of like persists there and can generate local ads and survive, but they don't have a lot of money to invest. And that's, that hasn't jumped the shark yet, right? They, they're still struggling. Even our 99 a year, I mean, it's 99 a year for RadioVision Pro.

You know, people go out to dinner with a family of four and they spend 300, right? You go out to Starbucks and you spend, you know, 12, you know, six times and you're already at 99. And it's funny that these people look at 99 for a plug in, for a freemium plug in, And they think it's so expensive in the WordPress world because in the WordPress world, and this kind of gets away from radio and into that world, WordPress paid premium tiered system.

There's been this 15 year. Control of pricing sub 100 right that nobody can seem only a few actors can seem to break in all of the larger plugins like all in one seo or gravity forms are all sub 99 dollars a year they're all in the 49 59 dollar range So when someone like me is building something more niche for radio stations that need these tools to survive and grow, for example, we're putting Alexa skills into, into both free and pro.

So if you don't have a developer, you know, you're going to get an Alexa skill builder, and you're going to be able to do that in radio station and radio station pro, right? I'm going to put that for free in both. I mean, I'm gonna put that for free in one and obviously be paid in the other, but it's so important for an LPFM to be able to get on Alexa or to get on Apple, you know, an Apple device or to get on Google where you can ask and have these, you know, and then also the, the AI part of radio.

So where are we going with that? So there's going to be a whole list and I know it, then I think at the broadcasters, you know, a convention in New York city, they had a whole AI panel. Right. So now they're all starting to think about AI and radio and what's that going to do? And we're already there with asking Alexa to field, you know, to field a question.

But in my mind, it's, can you tell me what the song was at four o'clock on this station in Los Angeles, you know, on June 25th? That's where I believe this is going to go, is that we're going to be looking to These you know, auto you know, speak, spoken word response devices to, you know, tell us things that the radio needs to tell us.

And I think, you know, even if it's a news piece or if it's the weather, you know, so if radio stations are broadcasting the weather, how are we going to ask Alexa what the weather, you know, we can ask what the weather is, but maybe you want to hear it from your local radio station. Maybe you want to hear some interesting local aside.

So I really hope that it goes back to local. I can't predict that just like I couldn't predict that uber was gonna be a thing

To my own detriment but But I feel it. I feel that people are constantly asking for more local programming. Yeah, and you know, I I do hear it The death of radio is is due to the conglomerates You know, playing you know, the same cover version of Fast Car 37 times a day, right? By Luke, whatever, Luke Combs.

Which, I tear up to that song, I love it. I really love, and I met Tracy Chapman once, and I absolutely love that song. But, I've heard it every day for the last two years straight. I heard it in a grocery

Doc Searls: store yesterday. Yeah, you heard it in a grocery store. In Kroger, I heard Tracy singing

Tony Zeoli: Fast Car. Well, you heard Tracy, but if you hear Lou Combs, Lou Combs is the later, is the newer cover version that's been on radio here in the South, in Asheville for, you know, for 94.

5 and 104. 3. I mean, you know, and the Nicki Minaj's and the You know, the, the Doja cats of the world. It's like, come on, you know, there's so much more out there. And I think people are really dying for that. And that's why they're tuning into the, to Asheville FM. And that's why they're donating to Asheville FM, and they're donating to Blue Ridge Public Radio, is because they want, you know, that open source philosophy of do what we feel, not by some programmer's decision in Washington DC for the whole East Coast.

of what they think, you know, radio should look like in this market. So, radio in some way is doing it to itself. And you can listen to Lloyd Ford's podcast. He has a great podcast that I was interviewed on. And he really talks about sales in radio, and the, and the growth, and the, and the shrinking of radio.

And you can get a lot of good insight and information from Lloyd on, in that regard. But, I'm hoping, I'm thinking globally. I'm not thinking U. S. I have to think globally. I get. I get support requests from all over the world and I have to go to Google Translate and I have to put Spanish into English and I have to respond to them in their native language.

And I'm okay with that because I just was in Mexico. You know, I lived in, in Cuenca Ecuador for two months. Like, I get, you know, that there's so many people out there around the world who want This audio broadcast because they're not able to look at something visual all day every day, and they need that audio, and I'm hoping that radio station and radio station pro and stream player provide those tools to those webcasters in these global markets that, you know, don't have the opportunity or the investment to grow.

They're not an iHeartStation. They're not, you know, they don't belong to some conglomerate. So there's the. There's the haves and the have nots, and I'm trying to provide to the have nots to help them grow. And I think that they can with our plug in, especially, you know, and I'll, I'll pitch for this now, especially in the open source community, and saying like, I need your help.

If you love radio, and you think it's important, and you want to facilitate that for a global, the global broadcasters who are trying to do this. Under their own volition with in their with their own pocketbooks then contribute to radio station free your ideas and I'll certainly love to hear what those ideas are and love to accept pull requests at any, you know, anytime they come in.

So that's why we're here, right? That's why I believe in WordPress and I believe in open source and I want. I'd rather see free go to 100, 000 users and pro. You know, be used by, you know, 10, 000, right? I'd rather see so many people using free because then that might then fuel the donation, funding, grant, grants, you know, and, you know, and things like that.

And that's another issue is how do I, as a founder of a, of, of, of radio station free, find the grants from those grant providers, which I hear. On the public radio, right? I hear the commercials for funded, funded by, you know, the Diana S. Knight Foundation or whatever, right? How do I, and I've gone to some of those people and they said, no, that you're not actually this, so we can't support you, right?

And then it's also goes back to one of the questions that Doc asked about in the, in the station. Stations are using LibraTime and AzuraCast and these open source station play out systems. So the play out systems exist inside to run your, your, your show schedule and your broadcast. Radio Station and Radio Station Pro are only there to present the show schedule on your website.

They're not a, it's not a play out system that you need an automation system that does all your scheduling for your commercials and your commercial breaks and all that stuff. That's in the open source LibraTime. That's in the open source Azurecast. And I'm sure there are other open source automation systems out there, does radio station become a software as a service automation system with a web component?

Wink, wink, maybe. Maybe. So. I prefer to potentially work with Libertime and Azuricast and other open source platforms to have that interplay, you know, between us. So

Jonathan Bennett: I want to, I want to ask about something real quick. We, we mentioned this idea of, you know, you've got music on the radio and music on these netcasts, but when you get the podcast, you get very, you get a lot less music because of the licensing problems.

And it, what comes to mind is. Creative Commons, and I'm sure you're familiar with Kevin MacLeod of Incomp Tech. One of the, one of the leading guys that makes essentially open source music, you could call it that really, it's Creative Commons music. There's, there's almost this kind of second bubbling ecosystem alongside, you know, your, your regular music labels.

There's this, this second system where people are making Creative Commons music. And we've kind of not seen that break out into the public consciousness yet. And I, I also had this thought, you know, some things, we talked about a, a, a mechanical license, which from what I understand, a mechanical license is just, there's an automatic cost set, you don't have to call anybody and ask for permission.

And I, I wonder whether we couldn't write a Creative Commons license that is somewhere between You know, you can use it commercially and non commercial use, maybe a pay me a dollar to put this in your podcast, you don't have to ask permission, here's the cost. And, and whether that might be a game changer for getting some of this music into people's hands more easily to use it.

I don't know, any thoughts?

Tony Zeoli: I believe that there are creative commons, there is creative commons around images and music. You know, people do give away their music for free. And the mashup DJs, those mashups are under creative commons licenses because they're mashing up stuff and they're not selling it.

They're giving it away for free. So that's already kind of happening in the mashup world of, I create a mashup of 37. You know, copyrighted works, I give it away for free and that's fair use because you're kind of commenting on the works themselves by mashing them up in an or in a, in a unique order and layering that does that didn't previously exist and it can be considered, it could be considered that you're, you're, you're you're commenting on the works by using them as art in an art project.

So, in terms of donating, you know, to a Creative Commons derived work, the artist would need to take the donations but not sell the work, right? So, so you would separate the two and not say, come and give me a dollar, you know, basically share a dollar with me to download this. You would just have to say, hey.

I'm sharing my music for free. If you want to donate to my lifestyle, you know, go over here and do that. So you can't, you can't connect those two dots. You can't say, here's the music for free. Now, can you donate something to me? I think you have to go outside of that and say, I'm giving you my music for free.

It's just a small distinction that, you know, that I've heard is what you have to say to, You know, or maybe, you know, maybe you don't even say, you know, people just donate to you because they love it. I don't know The exact i'm not a lawyer. So I don't know the exact answer to that question in the in the scope of You know of creative commons and giving that music away, but I just know after listening to some npr reports with matchup djs that that's what you can do and how you monetize that you know is different now, here's the thing it's very Interesting.

Radio Spiral is one of our radio station pro customers, and they broadcast all Creative Commons or license free works that people create and share with them and say, you don't need to pay me for this. And they do it in Second Life, the virtual world. So they broadcast on the web and in Second Life as a Second Life radio station, which is really cool.

And I just totally forgot about them until you So I just ran into my head, but they are asking, like, I can't do a mix show for them with major label music, right? They won't allow me as a DJ to do that. They say, if you find stuff in our free, you know, creative commons or free catalog, sure, you can use that stuff, but, and then we'll broadcast it because we're not paying royalties for streaming.

We're asking people to donate their music to us for them. We'll rebroadcast for the promotional benefit. So in that way. Your question is answered by that's the model that Spot Radio Spiral is following in Second Life. Yeah,

Jonathan Bennett: interesting. If we had more time, we could dive into that more because there's, you know, there's things about fair use that play into that and just all sorts of questions.

We are, we are getting down towards the end of the show though. I have some questions I love to ask folks before I let them go, and one of the first ones is, what's the weirdest thing, and you may have just, you may have just told us this, but what's the strangest thing or the most unexpected thing you've seen somebody do with the radio station plug in?

Tony Zeoli: The most unexpected thing I've seen somebody do with the radio station plug in that is such a good question. And I am so stumped. It makes you think. I'm trying to think I'm going around the world thinking, who has done something in this region?

Nothing that strikes me right now as to somebody who's not using it. And it's original intent. What I can say is some people use it and then pick it apart and only use certain features and not other features. They might have their own radio player, so they don't want ours, or they might have their own show pages system, but they just want our playlists or play listing tool or, you know, so I haven't yet seen a unique use case.

That somebody has done something more interesting with it. But I hope that person that asked the question is the first one to do it. So, that'd be cool.

Jonathan Bennett: Alright, is there anything that we, and this is another tough question, Is there anything that we didn't cover that you wanted to cover? Is there anything that we can, we can put in here right at the end?

That you wanted to make sure.

Tony Zeoli: No, I mean, I think I talked about the Alexis skills. One thing I was thinking about that I just saw come down the road, Was putting like matching lyrics because people love to see lyrics and there's a company that I just ran into That's a lyric, you know provided but we'd have to license those that lyric stuff And so yeah, would there be it would be but this it's hard to open source lyrics, right?

Like you can't You know, it's just like audio. It's just like music, right? So you, so you can't, you have to license that content. So I'd like to match up lyrics with some, you know, some, some of our streamers. So they get that as a benefit. And then there are advertising plugins for WordPress. That actually, you know, are open source and can display ads and track them and stuff like that.

So it's really providing advertising opportunities for local. LPFMs, you know but the, the one thing I guess the larger topic is all these LPFMs and college radio stations are so underfunded, right? And so they struggle with Web design, web development. They struggle with making decisions. They struggle with people that come in, the rotating cast of characters who are volunteering, you know.

And until some massive change happens, you're just going to continue to see that issue in local radio. Because people can only commit so much. Like, I had a child. We adopted a child nine years ago and I was on Asheville FM and unfortunately my volunteer schedule got disrupted and I couldn't, I couldn't volunteer as much as I wanted to.

And then they got all snippy, like, now you can't have your show because you can't do that. It's like, dude, I just had a baby, like, you know, come on, right? So, you know, and there's personalities in radio, you know. And I'll be honest, I have re approached Asheville FM because they use the free version! And I just emailed one of the DJs a couple weeks ago and said, Hey, you know, you're using the free version.

You know, I'd love to have, I'd love to give you a donated copy of the pro version. And the guy got back to me and said, no, we don't need your help. And I was like, but do you even know what I'm talking about? Cause you're not, we have a, we have a webmaster. We don't need your help. He didn't, he didn't understand.

He didn't even understand. And he didn't even ask me the question. Like you're offering us something for free that we already use, you know, that you can benefit us. And, you know, but that's the attitude. And sometimes. You know, I think beyond open source and beyond anything, we just got to learn to work together, you know, as people in this time of like, ridiculous, you know, split, you know, Democrats and Republicans and people just.

Whether it's losing job and you see the tech industry laying off 250, 000 jobs in the last so many months and people dying for work on LinkedIn every day, you know, posting, I'm looking for work, looking for work, we even groups happening, looking for, we, you know, we're working together at people, you know, I'm, I'm trying to build something and, and, and if you can't respond and, And, and take the, take the help, then how are you going to help yourselves?

Right? And that's just, that's just a philosophical question. That may have nothing to do with anything, but that's what I'm experiencing. So I'm here giving you something for free, you know, and I'm here to help you. So take the help, you know, cause you're open source cause you're nonprofit because you're a small organization in West Asheville looking for the help.

But when you say no. It just doesn't make sense. So, you know, sorry to out them, but, you know, that's just, it's just the truth, you know? Yep. Understood. I'm a, I'm a guy who tells it like it is. I'm from Boston.

Doc Searls: I had Boston, I had Boston questions too, but we can save them.

Jonathan Bennett: Alright, before we let you go, I've got to ask, what's your favorite text editor in scripting language?

Tony Zeoli: I use bbedit and then I'm only skilled in CSS. I don't have a JavaScript or I can read PHP and know what it's doing and you know, and then like, for example, I was working on a client site yesterday and there was a add function and add action script that's been deprecated in PHP eight. Okay.

I looked at it. I learned it. I saw that that's where it is on line 65. I went to Google, looked it up. Here's an issue. And then I told my developer to fix it, right? So, you know, hey, can you fix it? And that's on a client site, so that's a paid work. But, you know, I'll do that on Radio Station with my developer in Australia.

And I did. Oh, here's the one thing I forgot to mention. Tony Hayes. Tony Hayes is my right hand man. He is my lead developer. He's a partner in the company. He owns a certain percentage that I gifted to him for his participation in his work in, in, in Netmix LLC. And we've been working together for three years since I found him on a WordPress jobs board.

He had been working on radio station free for a friend of his and saw my post. Thought he wanted to get involved and then we got married in a legal agreement and it's contractual and we've never had an argument, I don't think. I think we've maybe raised our tone once or twice, but it's been really good because I understand where he's at, he understands where I'm at.

We work together well, but he's my, he's my Australian based developer for Radio Station, Radio Station Pro and Stream Player Pro. Until I raise more money or bring on somebody else. Where some other people will submit, you know, a pull request, you know, it's he, it's he and I, it's just us two, he's the guy and I really have to thank him from, I mean, I'm going to tear up now, like it's one of those things, like this wouldn't be possible if it wasn't for him and my idea, my impetus of, of product and program management, but his abilities to write this code and put it into play and us to work together to get it out to the world.

I mean, it's two guys, right, doing what I did in 1996 with NetMix, 1995, right, trying to grow something. And contribute and do something good and valuable while taking care of our lives and families. Yeah.

Jonathan Bennett: Awesome. Well, Tony, we have, we have a hit time. We've got to let you go, but I want to say first, thank you so much for being here.

It has been a pleasure. It's been great to get to talk to you and we'll have to do it again when, when things change when, you know, the next version, the big news comes out or, you know, whatever happens next over at Radio Player and Netmix we will have you back and we'll chat about it. Thank you, sir, for being here.

Tony Zeoli: Yeah, thank you for having me. I really appreciate it. Good to talk to you. All

Jonathan Bennett: right. Okay. Doc, what do you think?

Doc Searls: I think there's so many things we could talk about. Oh my gosh. Yeah I, I, I love Tony's Boston sensibility. I was going to ask. Next time about, about how, how that plays, cause and is he still a Red Sox fan?

I don't know. We can talk. See you for

Jonathan Bennett: next time. We'll talk about it

Doc Searls: next time. Yeah, we'll see you for next time. But it's, I mean, this is dear to my, the entire topic is dear to my mind and heart. I, I, I mean, I'm, I'm, I'll never stop being an old radio guy, you know, and but, but there's, I think, I think there's an awful lot to the local thing.

I think that's, it, part of it is going to shrink down to local and it's going to grow up from local. I'm seeing that here in Bloomington, Indiana. There's actually a very vibrant local, is it when people talk about the local station here, they're not talking about the commercial station, it's the last one there.

They're not talking about the big public radio station, they're talking about one called Firehouse Broadcasting, which. You know, their transmitter is far enough out of town that they need a translator in town, so they're, you know, to, to get it, but, but almost everybody listens to it, WFHB, and they're and they're online, they have a pretty, a pretty sizable audience outside the area too, but they're very local, they're very, they're very much here, they're very much about music they're very much about local music, but not just that it's all volunteer.

It's interesting to me that when I was growing up in the 50s and 60s, there were, I mean, there were, you know, people don't remember this, not many do, they're all old like me, but the first rock and roll stations were always secondary stations in markets, they were not the primary stations. Later, in a few markets like Chicago and New York with WLS in Chicago.

And WABC in New York and in Oklahoma City KOMA, which is now KKOC were, I mean, they were the rock and roll stations for large parts of the country, that the whole West was served by KOMA in those days. And, and, but they played 30 songs at a time and it was it, it was a whole list of 30 songs.

And if you wanted to succeed in radio, you played the same 30 songs as the other wise guys did. And now it's not like that at all. It's like. Everybody likes, the way tastes are changing and broadening out and, and mushing together and all the rest of it, I mean, the hip hop beat is in everything, as it's kind of a a primary beat that's out there.

Something else I want to ask Tony about, as he knows Brian Bellendorf, because Brian's very much in the house and electronic and he co invented Apache. So so again, next time we'll bring him back.

Jonathan Bennett: Yeah. You know, you talk about local. I just had this thought right before we go, I want to make it real quick.

Talk about things going local. I kind of feel like that is one of the inevitable responses to. Artificial intelligence taking over everything because really we don't have artificial intelligence We have large language models and they're impressive with what they do but they're also they kind of drain the humanity out of everything and Businesses are glomping onto it.

They love it. They don't have to have people working anymore. They just have LL Williams doing

Doc Searls: everything Well, it's all spoken by the dead. I mean that that's the problem he's like everything you've already written and everything's been played and everything that's been drawn is out there already.

Yeah. And so it's already like dead in the sense that living entities are not producing this right now. And then this, this automated thing comes in and takes an average of everything everybody's said and puts one word after another or, you know, creates cliches. For, for,

Jonathan Bennett: for, for some, for some maybe large section of the public.

They're going to look at that and reject it and the other side of that is going to be now You move towards kind of the the boutique content creators you move towards the smaller and the more local I think that's that's probably what we're going to start seeing his response to I think it's going to be really fascinating I see tony I can see

Doc Searls: We told him that in the beginning and he forgot it's okay, so But we'll talk about it next time. But don't go away, Tony. We still want to talk to you. Just not on the, on the, on the thing. Alright.

Jonathan Bennett: Well, let's, let's wrap, let's wrap the show itself up. Doc, do you have anything that you want to plug?

Doc Searls: Oh my gosh.

I just, I just go to my blog doc. searls. com. That'll be close enough. Doc. Searles. com, there we go. Yeah, that'll do it and you'll see what I'm working on, so.

Jonathan Bennett: Alright, well, the one thing that I do want to plug is Hackaday. com. We've got the security column, goes live every Friday morning. Make sure and check that out.

And then, let's see, next week, next week we're talking with Matt Ray of OpenCost. And that's all about track, in an open source way, tracking your cloud costs and saving money there. And that, that should be really interesting make sure and check that out for everyone that is, that has been here live and those listening on the download and to doc as well.

And I want to say, thank you. Thank you so much for being here and we will see you next time on Floss Weekly.

Episode 767 Transcript

24 januari 2024 | min

FLOSS_Weekly-Ep767

Jonathan Bennett: This is Floss Weekly, episode 767, recorded Wednesday, January 24th. OwnTracks, are we there yet?

Hey, this week we're talking with J. P. Menz about OwnTracks, a modular set of applications that lets you track, view, and even share your own location data in whatever way you want to. But the most important thing is that you stay in control of it. You don't want to miss it on today's Floss Weekly, so stay tuned.

Well, good morning. It is time for Floss Weekly. It's a show about free, libre, and open source software. I'm your guest, Jonathan. I'm your guest. It's gonna be one of those days, isn't it? I am your host. It's not just me, I have the wonderful Jeff Massey with me, which for some of you, this is a familiar face.

But for others, this may be the first time that you get to see him. Jeff, welcome!

Jeff Massie: I am excited to be here. I'm normally seen on the untitled Linux show with Jonathan. So this is my first floss appearance. So very excited yes.

Jonathan Bennett: Yes, I am excited to have you here and We're talking we're gonna be talking today with JP men's about own tracks and you know Maybe not about own tracks itself, but this idea you're a little familiar with aren't you alittle bit?

Jeff Massie: Yeah, I've You know, I, I like riding motorcycles and, you know, there's a lot of, uh, mapping software I've used in the past. I'm not an expert by any means, but, you know, plotting routes and figuring out where you've been and where you want to go. And so I've. I'm very excited to hear about this, this software.

Jonathan Bennett: Yeah. Now I have used own tracks in, in kind of a little different way. So one of the things that I do a lot with these days is the Meshtastic project, which that's Laura radio. and a, a mesh protocol for it, so you can get one radio here and one radio there and then one radio at your base station. And the far radio can send a message.

It'll get repeated by the middle one and then picked up by your base station. And one of the things that we do with that is send GPS locations around. And I went to looking for some solution that would let you keep track of those GPS locations. I kind of wanted a replacement for Google Maps and the Google Timeline feature.

And something that would work off grid, even if you don't have internet access. And so I went looking around, I found one solution, and went, Oh, this looks cool, and saw that it was written in Java, and went, Nope, look for something else. And, uh, eventually came across OwnTrax, and I really like it. And so I've got a, I've got a little OwnTrax installation set up here at the house.

And, uh, you know, it'll, it'll, When everything is working correctly, that's a whole different conversation, But it'll track me all the way across town. And then when we go to do bookkeeping at the end of the month, or the end of the week, We can figure out where I've been and then, you know, make notes about who we need to bill for it.

So some fun stuff there. Let's go ahead. Let's not delay any longer. Let's bring JP on the show. And, uh, we'll get the, we'll get the word from the man himself. JP welcome, sir.

JP Mens: Thank you very much. Nice to be here. Thank you for having me. Yes,

Jonathan Bennett: it is great to have you. So let's talk OwnTracks. Give us, give us first that 30, 000 foot view.

Did we get it right? What is this project about?

JP Mens: Well, first of all, you owe me 78 euro because I read yesterday that you were playing around with MeshTastic and immediately had to purchase two devices to verify that what you were talking about actually works. But that aside,

Jonathan Bennett: Sorry, not sorry. Sorry, not Sorry.

JP Mens: The 30, 000 foot view of OnTracks. OnTracks is a, is a location tracking software. We have two apps, one on Android, one on iOS, written by two different people. There are basically four of us in the team there's on the one hand Christoph Krei who does the iOS software, then originally it was Alexander Rust, but now it's Andrew Graus, who's responsible for our Android app.

Linus Groh, who's basically does our front end, and myself, I'm a bit of the janitor. I do a bit of documentation and I, I, I still. Believe in Father Christmas, and I think I am the team lead, but I think I'm the only person who actually believes that. Um, so OnTracks is, um, an application which we originally created, which originally invented.

Um. And I was thinking actually yesterday, first of all, OnTracks is now, this year, we're going to be 10 years old. And OnTracks, we originally created as a possibility, as a solution for a tracking app, which Does not deposit your data or my data, my tracks anywhere other than in a back end infrastructure that we control.

So that, that was the original intention and we are, um, still today a little bit proud that we mostly manage that. I say mostly, of course, because there are things that we cannot, um, That we cannot forbid, for instance, if we use iOS, then obviously Apple and the Apple Maps subsystem is going to know that we perform, for instance, a reverse geo lookup.

But other than that, supposingly, if Apple is trustworthy, other than that, the data that is collected by these apps goes to an end point of our own choice. That's the. That's that's the premise that was the original premise

Jonathan Bennett: and how does how does that data get sent around? What's the what's the back end protocol?

JP Mens: the protocol that we chose originally or that I originally selected was MQTT and MQTT is a TCP based protocol which was created for the Internet of Things It's a public subscribe protocol which permits Um, clients, in other words, programs, any application, any program, any scripting, uh, language, et cetera, to publish, uh, messages.

Messages are published on a particular topic name. Topic is a hierarchical, uh, 64 kilobyte, if I remember correctly, 64 kilobyte UTF 8 string, which is slash separated. And, um, A topic is created by publishing a message at that topic. On the other end, um, we have a broker. A broker is just a highfalutin name for a server.

And the server, um, uh, Knows about access control, for example, knows about security, knows about transport layer security, etc. And the server will also have, this broker will also have clients who are listening on particular or subscribing to particular topics. And if a client, um, If the broker gets a message for a particular topic and that broker sees there is a client interested in that particular topic, that message is immediately deposited at the client's doorstep.

So the client need not pull. And all this happens with a Payload binary payload of up to 200, 256 megabytes and a very, very small protocol overhead. And that was, uh, one of the reasons why we chose MQTT versus for instance, HTTP, which is a relatively verbose, um, protocol. So MQTT was, uh, the, the, the method that we use for color that we also still today use for communication to be fair.

Um, meanwhile it took a few years, but meanwhile we have also added the possibility that these apps at your choice speak either MQTT or HTTP, depending on Mm-Hmm, , um, the backend that you are able to provide. Right. But we have a preference for MQTT. It is faster, it is more lightweight. Um, we have, uh, yeah, certainly done more testing in the MQTT area.

Um. So the whole, the whole application stack works better with MQTT.

Jonathan Bennett: Sure. Now MQTT is unencrypted, right? And so I would imagine one of the big advantages you would get from supporting, I assume, HTTPS is that you can wrap it inside of a secure sockets layer. Well,

JP Mens: um, on the one hand, you are, of course, right.

On the other hand, MQTT does provide TLS, Transport Layered Security, what you just called, erroneously, I must say, called SSL, which doesn't exist anymore. It's called TLS. Yes, yes, yes. But it doesn't matter. I know what you mean. And MQTT speaks TLS. On top of that, so we have end to end encryption between the client and the broker.

On top of that, within OwnTracks itself, we can, or well, not we, you can at your choice, configure a secret key, which is used by the apps to, um, encrypt, um, the payload. So irrespective of whether you use TLS or you don't use TLS, the payload can additionally be encrypted and that is then decrypted on the receiving end.

Jonathan Bennett: So The way that I use it I didn't know that I had not looked deep enough to know that all of that encryption was set up. That's pretty impressive The way that I use this is just with a wire guard vpn, which on you know on an android phone is is great. Although I think on iOS that might be a little bit harder to pull off.

Um, so what, what is kind of the project stance on that? Do you, do you recommend that people use a VPN if they can? Or do you feel like the, the own track server is robust enough really to be able to expose it to the internet is the thing that makes me nervous the most about doing something like that.

JP Mens: Yeah, we don't, um, we don't have to. Expose the OnTrack server, the backend, what we call our recorder. We don't have to expose that to the internet. The only thing you expose to the internet would be your MQTT broker. And there you have a TLS connection between the client, between the app and the broker itself.

So you have full encryption there. VPN, by the way, you just mentioned WireGuard, which might be a bit more, uh, you said might be a bit more of a problem on iOS, is definitely not a problem, so it works perfectly well. The same way that you would use it on Android. In my personal opinion, that would be overkill.

It's just too much. If, uh, or assuming a certain amount of paranoia, which is probably quite healthy, um, you could do, uh, for instance, your wire guard or any other VPN on top of that MQTT over TLS, and just to be really sure that you will never be able to see your messages, uh, you could then encrypt the payload and forget the key.

Jeff Massie: So, you know, I. You know, I'll be honest. I'm not as network savvy as Jonathan, but I'd like to step back a little bit and just say what made you wake up one day and say, you know what? I think we need own tracks. I want to start this. I mean, was there an event or? What sparked this whole thing?

JP Mens: There actually was an event. My child was growing up and wanted a smartphone, and we decided, yes, you can have a smartphone, but only if we can see where you are. So this was consensual. I think that's a term that was a politically correct term. And the original The original solution was to, of course, configure a small Android phone, if I recall correctly.

The original solution was to configure Google Latitude. This must have been in the year 2012, I think, roughly, mid 2012. And, um, once I saw what Google Latitude or the data that Google Latitude was collecting, I thought, no, that's not a good idea. I really don't want to do this. And on top of that, that was the advent of Google Latitude, um, sunsetting itself.

I think roughly in the year 2012 or early 2013, they Disintegrated themselves, uh, or turned into some other product. I can't keep up with the naming schemes. Um, and, um, that was roughly around that time where I was toying with MQTT for all sorts of other things. And I thought it's surely, it must be relatively easy to have a small app on an, on a smartphone, which collects a position and publishes that over MQTT.

And. That was all fine and dandy, but I have no clue on the, on how to program these apps. So I asked around a bit, and Alexander at that time said, yeah, I'll do a prototype with Android. And my friend, uh, Christoph, whom I've known for very, very many years now, said, oh, I'm, I've been toying around with iOS.

Um, and we had the first version, which was sort of extremely minimal, um, but that was the, that was the beginning of what at the time, by the way, was called MQTTtude as a spoof on latitude. And of course, due to the fact that we use MQTT as protocol. That's great. Now that name, that name, I loved that name, but I think there were about five people on, in the world who could actually pronounce MQTT dude, there were about 17 T's in there.

And then we decided at one stage, now if we want to become rich and famous, we have to change that name. And that's when we rebranded to Um Trax.

Jeff Massie: Very nice. Well, I mean, it's, it sounds like, you know, a great story there. And do you have any idea how large it's grown now? Well, you know, where it. Where, you know, where, where do people use this?

Is this like global? Is this, you know, adopted more in certain regions?

JP Mens: Well, um, you can ask me any amount of questions you want and you can put thumbscrews on me, but the good news, the good news is we don't have a single clue. Because you see, it's not our data. We do not provide an end point. We do not provide a server.

We do not provide any portion of the infrastructure. So, uh, Jonathan was mentioning or mentioned earlier that he uses own tracks. I didn't know that. Yeah, we don't know. So to answer your question, Jeff, we really don't know. The only way, there are a few numbers that we have and a few, uh, a few bits and pieces of information.

So, um, I asked my, my colleagues, uh, Christophe and, um, Andrew the day before yesterday, how many, um, apps are currently, uh, currently installed. And the only thing we see is the installed, respectively the updated user base. So we know. That roughly two days ago, uh, on iOS, there were approximately 19, 000, almost 20, 000, um, active versions.

So these are versions of own tracks, which were recently updated. But this of course could be, for instance, um, Jonathan with his iPhone, uh, three years ago, five years ago, he installed. Or he downloaded OwnTracks, he forgot to remove it and it's been updated. Okay, that's, that's all we know. On Android we have roughly 13, 500 devices, so we have a total of approximately, probably, 34 35, 000 devices out there, which have OwnTracks installed.

Whether it's actually functioning, whether it's actually working, that we don't know. Sure.

Jonathan Bennett: Um, do, do you know, this is sort of an Android specific question, but is OwnTracks available like on the F Droid store or any of the alternative app stores?

JP Mens: Um, and, uh, Onetrax has been available on the F Droid store.

Andrew pulled it down, if I remember correctly. Uh, we are still in a bit of a redesign phase. Uh, it's been a bit, a bit of a difficulty having that. Uh, but it is planned. To be re reintroduced, if it's gone at all, to be reintroduced in the Android store. We know that there are people who, um, have Android phones and who do not wish to use Google play services, right?

And for these, it is important to have a completely Google decoupled, um, app. And yes, we have been there. I think it's. Disabled at the moment for a particular reason. I can't remember the reason. I'm sorry But it is it is there it is it is in planning

Jonathan Bennett: sure I can only imagine that you know the the set of people that want a Degoogled phone and the set of people that don't want to share their information with Google that says have a lot of overlap

JP Mens: Uh, yes, yes, definitely, definitely.

And, uh, so it's, it's important that, um, uh, the request for a, an Android version on F Droid is already many, many years old. I think something roughly in the order of six or seven years, maybe, already. But at the time, uh, or not only at the time, we just don't have the resources. We're a small team. One guy at the moment, one chap, uh, working on the Android, uh, version, and of course, I mean, this is open source.

This is, uh, these are the things that we do in our, in our free time, in our spare time. So it, it just takes a bit.

Jonathan Bennett: So, uh, a couple of different directions I want to go with that. And the first one is, so you mentioned this is a, a spare time project. It's a labor of love. Is there, is there any plan? Have you thought about trying to make any, you kind of off the cuff earlier, said something about becoming rich and famous from this.

Um, is there any plan to make any money from it?

JP Mens: Uh, yes. Uh, well, there have been plans. At the moment there are no plans. There have been plans. We actually, um Must already be something like six years ago. We spent an immense amount of time and effort Implementing something that we called hosted we wanted to offer The possibility that somebody could download the app and solve it on their phone and and get going.

Yeah and pay us a monthly Subscription sum and and then be done with it. So we have very many people who are very interested in own tracks In using the own tracks apps and also getting the data, etc But are not I just not Either willing or just not tech savvy sufficient to get the whole infrastructure set up because it is not easy.

Definitely, it's not simple. Um, so we spent months. Lots of months in testing and in store update or what are they called in app updates and in app purchases and things like that to get this, uh, running. And it actually was then running. And suddenly we realized what the hell are we doing here? We're offering, we're, we're, we're intending to offer a product, which goes 100 percent against our philosophy.

And that is, um, a tracking app, which we, which we do not control. So we actually tore out that software completely, um, within a few hours, and completely abandoned that, completely abandoned that idea. So While we would probably want to become rich and famous. Um, I mean, what did Facebook pay for Instagram?

I think it was five billion dollars or something like that. We'll take the one billion. It's okay. We don't, we don't have so many people. But I don't think that we'll actually get there.

Jonathan Bennett: Have you had anyone reach out and pay, like, bounties for, uh, particular features?

JP Mens: Um, it's, it's happened occasionally. Uh, we had one situation.

There was a chap who was, um, worked for the, I think, Australian Met Office, meteorological station. And he Or they wanted, uh, to deploy a whole bunch of iOS devices, interestingly, iOS devices, um, to obtain, um, the, the barometric pressure. And, um, so we thought about earning money doing that. But Christoph, it took about, it took Christoph about 15 minutes to get the software done.

And then it was just in there. So Christoph is somebody who, you give him an idea. And he just does it. He doesn't talk about it, he just does it. On the way to Mexico, Jonathan was mentioning earlier, that I was, that I showed him the track of my route through Mexico. On the way to, at the airport in Frankfurt, I had the idea, of being able to submit point of interest, which could then be visualized on the phone and also then later on in our backhands.

And I boarded the plane, local time here, something like 1330, and I deep, deep planed in Mexico City at 2100, uh, in the evening. And he had already done this after. Yeah, so it's, it's, uh, Yeah, so we, we, our software, our apps publish, uh, location information, um, as blobs of JSON. So in this JSON, we have, for instance, latitude, longitude, obviously, uh, velocity, altitude, battery level of the devices.

So that can, for instance, be used for monitoring the battery level of your spouse's iPhone. Um. We transmit meanwhile things like the SSID, the Wi Fi SSID. There are people who have been requesting that in order to determine which. Which location are they? Are they in the office or are they at home? We transfer a so called, uh, tracker ID, which is a two letter indicator, which is used on the map, on the app, to indicate, uh, Oh, it's Jonathan or it's Jeff or it's JP, yeah, who, um, at that position.

Um, then we have, uh, the possibility, for instance, of, um, producing, um, um, What we call transition events. So in the apps, you can define regions, uh, geo fences, we call them or regions, which are circular regions. And when the app detects that it has exited that region, it transmits, uh, over MQTT or HTTP, a transition event, which indicates I have now, I'm now leaving this location or, uh, conversely upon re arriving, I am now entering this location.

And people use that, for instance, for, uh, home automation.

Jonathan Bennett: I was just going to ask about that, because you're both using MQTT. That seems like something you could tie into Home Assistant pretty easily.

JP Mens: Absolutely. I think the first integration into a home automation software was done by our friend Ben, who lives in New Zealand.

And he created an integration for, excuse me, he created an integration for OpenHAB. Then somebody else, we don't know who it was, created an integration for Home Assistant.

Jonathan Bennett: You could do some fun things with that, like, uh, you know, automatically open your garage door when you get within a quarter mile of your house. So turn your lights on. All sorts of fun stuff. Or

JP Mens: it's, uh, it's evening, it's dark, we know it's dark, please turn the, please turn the lights on. Exactly. Things like that.

So that, uh, is used quite a bit to our knowledge. Um, when I say to our knowledge, it's because somebody sends us an email or because somebody raises an issue and says, I've been doing this and this and this, and maybe this doesn't work, etc. So, um, yeah, as I say, to our knowledge, these are things that are done quite a lot in home automation

Jonathan Bennett: Yeah, I imagine that, uh, the fact that it runs over, well, I mean, it's simple JSON over MQTT, that is a really accessible sort of data type. Imagine that makes it really easy to plug into all sorts of things. I mean look what what I did with Meshtastic It's because Meshtastic supports MQTT And so all you have to have is a little tiny translation layer right there in the middle and it'll just say okay Well, here's the Meshtastic stuff I'm gonna break this apart and then put it back together in the way that OwnTracks wants it Just sit there and take it off the server and feed it right back in everything's happy.

I mean, it's it's Basically trivial to work with that that must really empower a lot of things.

JP Mens: Yes for people who are tech savvy and who like toying around playing a bit Um, I think it is probably an excellent solution We know for a fact That there are people who for instance do not use our official If you can call it that official back end or recorder, but instead have created their own back ends because they pull out the, they pull out the Jason, parse out the elements that they want and maybe store it on a SQL database or whatever with it.

It's just, uh, really, really, um, accessible as you call it.

Jeff Massie: Awesome. So if I want to, if I want to start this, I load the app on my phone. Now, is it? At the most basic level, is that, am I ready to go? I mean, is that, is it self sufficient just in the app or?

JP Mens: No, uh, definitely not. Um, if you wanted to start with, uh, own tracks, the first thing that you would have to do, or that I would recommend you do is set up.

Some sort of backend. So you have two possibilities. Possibility number one is you have some HTTP server somewhere. It's maybe an Apache or an Nginx or whatever or PHP. It doesn't really matter, yeah? It could be Python. It could be Java. Whatever you want. Some HTTP endpoint which is able to get the data.

Uh, of the phone. Yeah, so the phone is going to via HTTP perform an HTTP post to your HTTP endpoint. And, uh, you can then get the data. You have the JSON blob. You can parse the JSON blob and do whatever you like with it. Alternatively, That would be my personal recommendation, is you would set up, uh, preferably, obviously, your own MQTT broker.

There are a number of different platforms. One excellent piece of software is, um, Mosquito with a double T. Written by Roger Light. It's written in C, extremely performant, extremely capable, has access control lists and TLS and, and you name it really, it's very, very versatile. But there are others. If you prefer something in Java, there's HiveMQ for instance.

So you need an MQTT server, which you set up and of course you lock it on as strongly as you wish. And then after you've done either the one or the other, then you download the app and configure the app to speak either MQTT or HTTP to the backend that you have set up. Now, this need not necessarily be yourself.

So, let us assume, um, that Jonathan and yourself wished to do this together. You both could set up one MQTT broker, and via access control list on that MQTT broker, you could, well, one of you, whoever is the manager of it, whoever is the system administrator of it, would define, um, Which of the two users can access my information and From there, we get to the topic of what we call friends.

So, um, the OnTracks apps permit me to see on my phone. I think I have 10, 10 people, uh, permit me to see friends. So these are, uh, acquaintances, friends of people. Mine who use own tracks, use own tracks with the same MQTT broker. They permit me to see their location. I permit them to see my location. So on our devices, we see each other where we are on the map and the address.

And when the, when the ping was last or when the post was the last published, et cetera.

Jonathan Bennett: So I I'm, I'm super curious when you use. The HTTP backend. Um, does that completely replace MQTT or do you still also have MQTT running? Uh, between the two. No,

JP Mens: it can completely replace MQTT. However, um, certain things are more difficult or more complex to set up.

So, for instance, this whole topic of friends is more difficult to, more difficult to set up. You need to do a lot more on the back end problem, yeah? Okay. But, um, No, it is a complete replacement. Okay. Now,

Jonathan Bennett: can you, can you do it both ways? Can you have, you know, an HTTP endpoint that then just forwards off to your MQTT server?

JP Mens: Yes, of course. Okay. Of course, that is, that is certainly possible. So you would have an endpoint, which you write in any programming language you want. And that can, that would obtain or that would get the HTTP post with the JSON blob in it. And, um, then your software would publish that somewhere else. For instance, over MQTT or AMQP or send it off by email or, uh, transmit it to your MeshTastic thing to have it transmitted by radio to some other endpoint.

Any number of, any number of, any number of crazy combinations are possible. Sure.

Jeff Massie: Well, just to kind of continue on with that. So, okay. Jonathan and I are friends. You know, we're set up so we can see each other's location and I go on a motorcycle ride that I think Jonathan should see now. How, how would I share that specific trip with

JP Mens: him that he should or should not see that he should,

Jeff Massie: that he should see?

I'll say, John, Jonathan, this is a great trip. You ought to see the sights. Follow this route.

JP Mens: Yeah, if Jonathan and you are friends, then basically you do not need to do anything, because Jonathan at any moment will see where you are. So, let us assume Jonathan kept his smartphone on, or his backend software on, he could follow you as you ride along.

Um, I think what you mean, or I'm going to interpret what you, what you said, slightly differently. Let us assume you both are friends on a single server, and you would like to show your family member, I don't know, I'm imagining a sister or a brother, anybody, aunt, uncle, doesn't matter. You would like to show your family member your trip.

Now, you wish to So, um, you wish to have them see your trip, however, you do not wish to give them access or unfettered access to wherever you are. So you can set up what we call a tour. And that is, you define that you're going on a trip. Leaving Monday morning, 8:00 AM and returning Friday at 1700. And you define that trip, you set it up and that's again, like a bit of adjacent, you could do that on the app itself.

Um, currently only in iOS, but you do that on, on the app itself. And we store that on the backend. And that trip gets, um, uh, U-U-I-D-G-U-U-I-D. You can then share that GUUID to, for instance, your sister, and she can at any moment, uh, see the track that you have created from Monday morning, 8am it was, till Friday afternoon, 17pm.

And, for instance, after your return, You go on another trip, but you don't want to see her. She will not be able to see that data. She only sees the trip within the time constraints that you have specified.

Jeff Massie: Okay. Yeah, that's what I, you, you interpreted that correctly because, you know, if somebody who's just watching me, there's going to be a heck of a lot of, you know, going back and forth to work and a lot of, you know, kind of.

Me to the grocery store kind of things and not, uh. Specifically being able to call that out, you know, here's the trip I want you to see now Can you with that person then if they wanted to follow my footsteps would they could they load that into like a? Google Maps or a base camp or a

JP Mens: In theory, yes. If they were knowledgeable enough to get the data, they would be able to access the data.

Um, we publish that as GeoJSON. And yes, you can load that into Google Maps or export it as KML or as, uh, what is the other called? GPX or as comma separated values. So we have a whole, um, a utility which is able to Uh, export those data in a number of different formats, but they would only see the data. And that's the important part.

No, they would only see the data for that specific timeframe. Right. Yep. Yep. Perfect. We also have, this was, I'm not sure how to politically correctly express this. So I'm going to be very, very careful and I'm going to start off by using the German term. We also have. Somebody who many years ago said, well, supposing I have own tracks, and my wife has own tracks.

And supposing I go somewhere, and I don't want my wife to see where I am. And then I said, okay, this is the beginning of what we will call the Puff Schalter. Schalter means switch. And, um, Well, can you imagine what the word poof means? Anyway, it doesn't matter what it means. Um, the place that you don't want your spouse to know that you've been, okay?

Um, and so you can, on the app, you can say we want now in what we call quiet mode. We want please, um, Jeff does not want his friends to see where he is at the moment because he's going to 7 Eleven or whatever to a liquor store. It doesn't matter where he's going. And upon leaving that location, Jeff could then switch off that quiet mode again.

And then, for instance, his friend Jonathan would see where he is. Yeah.

Jonathan Bennett: So I'm, I'm curious if there is a way to do that kind of the other way around. So, so we'll just, we'll just throw out our kind of a real life example. Jeff and I have talked about, because Jeff is way up in the, uh, in the north side of the country, and I'm down kind of south central.

So we've got hundreds and hundreds of miles between us. We've talked about trying to get together, and Jeff may be coming as a motorcycle ride. And it's like, well It would be nice for me to be able to keep track of him on that ride. Let's say something happens and he needs pickup or help, you know, so I can I can just imagine that for that particular trip That would be great to be able to keep tracking him on the way down, but I don't need to see Anything else about where Jeff goes?

I don't need to be able to watch him go to work or back. So is there, how would you do that? Would you make someone a friend just for the trip and then revoke the friendship? Or does that give access to all of your points? How does that part work? No, no, no.

JP Mens: Technically you could do that, but that's something that, um That is something that you probably wouldn't want to do.

Now, assuming Jeff has his own MQTT broker, and you have your own MQTT broker, then your IO, sorry, your OnTracks app is connected with your broker. Jeff's app is connected to his broker. You can From our point of view, you can no longer become friends. These two brokers would need to be bridged together in order to provide some sort of a friendship system.

Okay. So friends. Only work, um, friends today only work when we are connected to the same infrastructure. One of the disadvantages of not having a central

Jonathan Bennett: system. The Onetrax backend does not know about the Fediverse. Yes, exactly, exactly, exactly. I don't, oh goodness, I don't know if that would even be possible to make work, but that would be, that would be pretty cool if you could.

Make it sort of Fediverse enabled.

JP Mens: Sorry, somebody's drilling here somewhere. I'm holding my mouse on the mute button. Indeed, we can, uh, Well, technically we could actually federate, somewhat federate MQTT servers, MQTT brokers. But that will be today beyond the scope of what Umtrax

Jonathan Bennett: is. Sure. That that's that's the sort of thing where you would want some company that really really or some Individual that really really wants the feature to come and actually pay you money to develop it

JP Mens: Yes, um, although what you just said is not uninteresting um thinking fediverse and um

Jonathan Bennett: Well,

Jeff Massie: based on history, what you've been telling us, it sounds like that's, that might be a couple hour project.

JP Mens: I hope not because it's very painful. Um, that might be something, um, that, that might actually be something that we could think about. I, my imagination is a. Bit on the freeze

Jonathan Bennett: now. Yes, it's always fun doing one of these interviews when you ask a question or make a statement and you just, you see the wheels start turning in the guest's mind.

Exactly.

JP Mens: I've got a block here and I'm making notes. So, yeah, stay tuned.

Jonathan Bennett: So, one of the other things I wanted to ask about is the GitHub repo, like the project as a whole, How much interaction do you get? Like, how many people have come and starred it? How many people have come and forked it? And do you, do you regularly get, uh, pull requests from kind of the greater community?

JP Mens: Um, I do not keep track of the number of stars. We have a, we have a GitHub organization called OwnTracks. So GitHub. org slash OwnTracks. And then therein are the different repositories for the iOS app, for the Android app, for the backend, for the frontend Um, and yes, there are occasionally, uh, pull requests.

I noticed one recently for one of the Android components. Um, there are, of course, people who submit issues. Uh, today I think we got two or three issues, uh, over, over the, over the board. Um, sometimes for the iOS app, or for the Android app, or for the backend, or we have a repository called Talk where people can ask questions and sort of begin or initiate some sort of a discussion.

Um, Yes, there's, there's a bit of, there's a bit of interaction. Absolutely. Um, I wouldn't say a lot, but certainly it's there. So not, I'm not quite sure whether I've actually answered your question. No,

Jonathan Bennett: no. Uh, I think that's a great answer. I will, I will actually give a little bit more information because I'm sitting here looking at your GitHub repo.

So the actual recorder bit, which is kind of the core of own tracks, uh, 731 stars, 111 forks, and you have 37 different contributors, which. I've got to say, that moves you out of the little tiny small time project, uh, demographic. This, you actually, it's actually pretty impressive, the amount of community interest and the community, um, Well, the community you've built around OwnTracks, it's impressive.

JP Mens: Um, yes, to a certain extent, it is impressive. Um, I just see that Christoph is telling me 300 stars for iOS, for instance, 1200 stars for Android. Mm hmm. So, yes, there are people who interact with us. I would like to slightly correct, only very slightly correct what you said. You said the recorder is like a central component, like a central, sort of, yeah, central instance for OnTracks.

That is only partially true because the back end is eminently replaceable. So, once again, if you Ah, fair. Let us say you and Jeff set up OnTracks against your MQTT broker and you look at the record and say, yeah, well, that's all very fine and so on. But we don't trust all the C code. We don't want this. We want to do our own stuff.

Yeah, then you could here again, uh, write. probably trivial amount of code. Um, if you're so desired, you subscribe to your own MQTT broker, you get the messages, extract your latitude, longitude, address, whatever you want to extract, and dump that into the database and visualize that somehow. So our recorder is A possible back end, but it's not necessarily the back end.

Okay,

Jonathan Bennett: that's, that's fair. So I may, I misunderstood something when I first started using OwnTracks. I think we ought to cover it here because it might help people out. You've got, you've got a web interface, if I remember correctly, that's part of the recorder to where you can go in and look at data. But that is not the preferred, um, front end.

For own tracks. Is it, is there, there's a, there's another component there. You have

JP Mens: managed, you have managed to shame me that, uh, that, um, that web interface on the recorder is. Or might I say was, no, is my best effort at trying to get data onto the screen. I did, I love programming on C and in real languages, I detest anything that is web.

Um, although there's one, I don't know if you saw this, Table is a table of live devices. Now that is really, really sexy in my opinion, particularly if you have a whole bunch of friends, because then you see, you really, you see the move and going from one country to another, that's really quite sexy, but, um, indeed the recorder is basically our, our data storage system, uh, the recorder.

Also records, if you so desire, reverse geo positions, which are stored locally and also cached locally. And the recorder provides an API, a REST API, with which a client can consume location data. And, uh, our preferred Frontend is the project that we call OnTrack slash Frontend with a capital F. Um, that was written by Linus Groh.

And that is a really, really nice, um, track visualization thing. You can select users, you can select devices per user. You can say, I'd like to see where Did my friend Jeff travel from Monday at five at 8 a. m. till Friday at 1700 p. m. Um, and then you get the track drawn, uh, drawn at you. Yeah, so that is really, really sexy.

Jonathan Bennett: Yes, um, it, it is, it is very nice. Now, there are, as, as a user, there are some features that I would love to see in there. And in fact, there's already, there's an open request and somebody already has a fork where they're working on it. Um, and so I guess let me ask this question to kind of hone in on this thought.

Where, where does OwnTracks need help? Like if somebody said, I love this project, I'd love to jump in and do some programming. Where would you point them at that, uh, the project could use, say, another,

JP Mens: another programmer? Where were you when I said billion dollars?

Where do we need help? That's actually an interesting question. We, we probably need help all over the place, or maybe we don't need help, but we could certainly use help. So for instance, there are a number of ideas on the front end that, um, are awaiting implementation. There's work that Could certainly be done on the Android app together with, um, Andrew.

There's maybe also work that could be done on the iOS app, although I can't think of any feature that's not yet being implemented there. Um, so where we also need help. Well. Yes, we need help. More time, uh, would be for me to rewrite the documentation or something that I've been promising to do now for, I don't know, two years or something.

Our documentation used to be very good, but it's meanwhile, I think just too complex. People are wanting a more point and click documentation, something simpler, something getting started quickly. Yeah, and I've been, uh, I've been promising to do that for quite some time now. So. Um, that, those are basically the, the, the, the, uh, the, the topics that we could certainly use help for.

Now

Jeff Massie: I, I kind of want to expand on that a little bit, but first step back just a moment and say, so you're talking about interfacing with your database basically. Now can you, now that's for the PC or is this everything done on the phone for your, uh, like your web and your. Preferred, uh, back end.

JP Mens: Ummmmmmmmmmmmmmmmmmmmmmmmm If I correctly understand your question, no, no, the phone, our smartphone apps today have a map which visualizes, which shows where the friends are and where you are. They are in charge of publishing location data every so and so often. They are in charge of publishing transition events when you leave or enter a geofence.

But that is basically more or less it. Um, everything else occurs on the backend. So, the data is stored somewhere. The data is then pulled out via the, for instance, the recorder API and then visualized in the frontend. That all happens. I think you said PC side, I would call it server side. So on the, on the backend, yep.

Um, but I'm not quite sure Jeff, whether I've answered your question.

Jeff Massie: No, you, you did because I, you know, sometimes I, at least personally, I find. Phone apps, sometimes if you have a large set of data, it can be hard to navigate around versus working on a server or something like that, where I have a bigger screen and a mouse to

JP Mens: Definitely.

Definitely. Our apps have not been designed with that type of manipulation. in mind. Um, there is an idea for the Android app, uh, on maybe being able to display historic data on the, on the device. So in the application itself. Um, but I do not see that happening within very short time.

Jeff Massie: What, what features do you have on the roadmap for your Android and

JP Mens: iOS?

There are a number of things that we have. First of all, for the iOS, we are working at the moment with so called points of interest. So you can, when you are on the road, this is probably also going to happen on Android. We, we try, it's difficult of course, but we try to keep feature parity if possible. Um, so.

When you're on the road and you're sitting in a lovely cafe, you think hey, this is, I want to remember the name of this cafe and that I had, I don't know, a fantastic cup of chai here. You can add a so called point of interest. This, this text is transmitted along with our JSON payload and stored of course in the backend database.

And we will be visualizing this, we do that already on the iOS map. So, in, in app, um, and, um, I made a request yesterday that, that this feature is also added to the front end. And, um, today actually I committed, um, code which allows our recorder to automatically produce a, um, A GeoJSON dump of these, um, points of interest.

So those are things, that is something we're working on. We've been working with tours. We, uh, want to increase or, or enhance the possibility of doing. Remote configuration and also locked mode on both Android and iOS. So, for instance, um, Jonathan has set up an Android or an iPhone app with own tracks for maybe a family member.

And that family member has nothing better to do all day long than to muck about with some settings. And Jonathan doesn't want that. First of all, it's a pain in the ass. But, uh, second of all, it causes extra support requests, so he wants to lock down that device. So this will be possible. Um, we, uh, are adding logs at the moment, so more better and better logging on the iOS end.

This has already existed for a number of years on the Android end. Um, on Android we want to implement also tools that we have, like your motorbike tool, that we also have on iOS. Um, Andrew wants to implement a so called waypoint picker which will allow to create a waypoint or what we call a geofence directly from a map, which today is not possible.

You have to enter latitude and longitude. Um, Android is tedious keeping up with the, um, with the Google Play Store policies and all the work it generates. That's, that's quite tedious. And, um, What is also tedious and requires a lot of work is testing on real devices because all these Android devices are just different.

So certain things work on the Samsung A, but they don't work on the whatever, uh, B version. So this is, uh, this is a bit difficult. So these are things that are coming. Yeah. To a store near you. Sometime.

Jonathan Bennett: Awesome.

Jeff Massie: Now, so getting to the point you're at now, what are some of the, the harder pieces that you've had to work around or what would have been some challenges getting just to this point that You've had difficulty

JP Mens: with?

Yeah, that's a very good question. We've had a number of challenges. It started off with battery churn. I remember quite fondly, well, maybe not so fondly, that roughly nine years ago, Um, I was several hours late to a client because the evening before I'd installed the newest iOS version and the battery was drained within an hour.

So I didn't get my wake up call. Yeah. So, um, these are things that we've been working that have been tedious to, to test, um, Then what has been, yeah, a lot of work in the past now, not so much anymore, of course, but is testing. So I, uh, drove thousands, and I'm not joking, thousands of kilometers, uh, or miles if you prefer, it doesn't really matter, um, for testing, uh, these apps.

And, uh, I know that Christopher Henson's walked, uh, Probably hundreds of kilometers in and out of regions testing, um, transitioning events and things like that. So that has, that has been quite, um, quite challenging. Um, but I think my favorite challenge, although it wasn't my challenge, but my favorite issue was a chap who created an iOS issue.

Number 388, if you want to look at it, which if I recall correctly, and I'm paraphrasing slightly the title is significant mode not happening at 800 kilometers per hour. Now that's 500 miles per hour. It's a pilot. We were, this, this was a chap, I think he was called Neil. Um, this was a chap, I think he's Australian or New Zealand, I can't remember, somewhere down under.

And we were in contact with him anyway by mail because he had asked all sorts of questions. A really nice chap. And I wrote to him and I said, WTF, what the hell do you mean by 800 kilometers per hour? Are you crazy or something? He said, no, I'm a fighter pilot. And I take own tracks on iOS along with me in the cockpit.

And I noticed that at 800 kilometers per hour, this display is not updating correctly. And that's why I wanted to, that's why I submitted that bug report. So, um, yeah, that's really, to date, my absolute favorite ticket.

Jonathan Bennett: That's great. I like that a lot. So I, before we let you go, I want to make sure and ask, um, how difficult is this to actually deploy?

Uh, is, is the, uh, you know, the recorder in the front end, is all of that dockerized? Is there an easy MQTT docker? Um, for, for someone that knows their way a little bit around, uh, you know, a Unix command line, how challenging is this actually to get started?

JP Mens: It, I think for somebody who knows their way roughly through the Docker world, I think it is quite doable.

Uh, to answer your question, yes, um, our stuff is Dockerized, so we have the recorder as a Docker container, we have frontend as a Docker container. Um, What we don't have today, and that's my fault, there's no question about it, is an easy setup guide to get somebody started sort of with, I don't know, 27 commands, to set up the backend infrastructure to get themselves going.

That we don't have today. But if somebody is willing today to churn for an hour through documentation, they will, I think, relatively easily kept themselves going.

Jonathan Bennett: Yeah Honestly, I don't think it's even 27 commands. I've I've done the install a couple of times I've been I've written about a little bit here on hackaday I it for for someone that understands how docker works and is not afraid of the command line It's it's pretty easy to get set up.

Don't don't let Don't let jp's uh modesty there scare you away. It's it's pretty trivial to get going if you're familiar with with the linux command line

JP Mens: Yes I I do not disagree jonathan, of course, but um Let's also please not forget that there is quite a bit to know about Docker and how it works.

It's how to set up variables and things like that. So it is a little bit daunting and I do still want to, and it will happen. I just don't know in what century. Although if you look at my hair, I have to hurry up because I'm getting older. Um, I do really want to create an easy setup guide. Um, and. Yeah, it's, it's in the works.

Jonathan Bennett: I, I would, um, honestly, for getting people started very easily, I would honestly say, just write a little docker compose file, and boy, you could get that down to probably two or three commands and have, be able to have people up and running.

JP Mens: Yes, I think in our, uh, there's a repository called ontrackslash Docker dash recorder, I think it's called, um, there is indeed a docker compose file, which is exactly that.

All right. Um, but, but even so, I'm sorry, even so I have to say, um, just because I can launch a docker container or a bunch of containers does not mean that the whole thing works. Let's please not forget that this. App on the smartphone must be able to find that Docker container. So we're talking TNS, we're talking IP addresses, we're talking firewalls, we're talking TLS certificates and, and that it, it does all add

Jonathan Bennett: up.

This is, this is true. I, there, there were some of these things that I figured out so long ago and that I've set up on my own network so long ago I kind of forget that they still exist, I suppose. That is fair.

JP Mens: And, and I think that probably, that probably applies to most of us here. Uh, but for somebody who's really, um.

Non technical, I don't know if that's a word, but it, it can all be rather daunting.

Jonathan Bennett: Sure, sure. Okay, so in, in light of that, let's say somebody needs a little bit of help. Or on the other side of it, somebody knows what they're doing and wants to volunteer to help test the beta version of, let's say, the Android app.

Uh, where, where should people go to sort of get connected and

JP Mens: learn more? Oh, definitely to our respective repository. So for instance, when Christoph publishes a new version, uh, to which fixes some bugs that somebody has reported, um, he gives these people, um, access to test flight and they can then test a preliminary version.

Likewise, Andrew does the same thing. It's not called test flight, but Andrew does the same thing for Android. So, um, We, when somebody reports an issue and when we request them to test in, I think, most cases or all cases, they then actually go to the trouble to really then test the stuff for us and say, yeah, it's okay.

And then we deploy it.

Jonathan Bennett: All right. Is there, is there a discord or a Slack server somewhere where people can come and chat live?

JP Mens: No. Uh, no, we, I don't, no, no, and it's not going to happen, that is not going to happen. That, that will happen, that will happen when Facebook buys us. Um, no, that, no, it's really not going to happen because, uh, let's please not forget, some of us have, have real jobs.

Sure. And we want to at least pretend that we have real jobs. And no, we just can't offer that kind of interaction. Sure. That's why. We have, uh, issue trackers for the individual components and we try to answer very quickly. Uh, I think this morning I answered something in five minutes or six minutes. Um, and, uh, no, no, that is, uh, that is excitement enough for us.

Jonathan Bennett: That is, you know, that's, that's understandable. Um, it. That's very understandable. So getting towards the end of the show, in just a second I'm going to ask you if there's anything that we didn't cover that you wanted to, but I know that's some set math in your head and that takes a minute to figure out.

So I'll ask you a different question first, and that is, what's the weirdest thing that someone has done with own tracks? What's the thing that most surprised you?

JP Mens: Weirdest? That's an interesting question.

Oh, I don't think I can answer that question. And also, please, again, and as always, because we don't really know what people do, unless they tell us. Yeah, right, right, right. Trying to do this and this and we can't. What's the weirdest? Well, maybe, maybe our friend Neil or Nigel, I think. Who, uh, who took, um, own tracks up in his fighter, um, in his fighter plane.

Um, there are, uh, yeah, Christopher's, uh, souffling to me. There have been Well, it's not really weird, but there have been people who were doing, I think, a marathon or something similar who did 10 rounds around the Arc de Triomphe in Paris. And they were carrying, I think, Android or an iOS phone with them, and they wanted to record that.

So that was Yeah, quite, quite interesting. I wouldn't say weird, but it's quite interesting. Um, probably on the weird level, we'll probably have to talk to the people who do, um, who do home automation. Yeah,

Jonathan Bennett: that makes sense. Alright, so, and then I warned you this was coming. Was there anything that we didn't cover that you really wanted to let folks know about?

JP Mens: No, I don't think so. I think your questions were very good and thank you for that. Maybe one thing, um, and that is, or a couple, a couple of points, uh, interestingly, and you mentioned that earlier, uh, you said that there's an integral, um, integration with home assistance, for instance, or, uh, with Whoopi Tatt and so on.

Uh, we find it, I would say interesting for lack of a better word or a more politically correct word. Um, We find it interesting that some of these people have done OnTrack's integrations, but have never told us about it. So, we know, and we get requests from people who say, Uh, it doesn't work with my Home Assistant, or it doesn't work with my this, or it doesn't work with my Life360, or whatever it doesn't work with.

And we can basically only answer, Yeah, well, I'm sorry, you have to go somewhere else, because that's not us that will do this in these integrations. So that is, that is a bit sad. And what's also a bit sad is occasionally, it's happened three or four times in the past that we've had issues raised on one of our trackers for a particular feature, which, and I quote, my customers want to have, end quote.

Now, that's very interesting. If you have customers who are using Onetrax and by customer I associate payment and we're not getting any of that payment, then I feel quite

Jonathan Bennett: disappointed. Yeah, that, uh, that's, well, that's kind of the, the, the continual, the always ever present, um, challenge of open source.

Right, is how you actually get some of the payment to the developers doing the work. Yeah,

JP Mens: and it's not, it's not, it's not, I mean, I don't think we are greedy. I think we've demonstrated over almost 10 years now. I think June is 10 years. Um, we've demonstrated for 10 years that we have done all this free of charge and you're welcome to use it and we're glad that you use it.

And if you, if you, if you pat us on the back once in a while, then we think, Hey, that's cool. Um, so we, we, we don't, we don't really want the money. But if you say my customers, then please at least spend five minutes telling us what that is and what you're doing and why you're doing it and how we could maybe even help, even if it's possibly free of charge.

Sure.

Jeff Massie: Well, is there a donate button or anything or buy you a coffee kind of button?

JP Mens: Yes, not coffee because I already drink too much coffee. I've switched to a gin and tonic now this evening because

The proof is there's no lemon or lime in it, so it must be water. So, yes, we have a donate button. and I think my accumulated income at the moment is 1. 69 per month. Um, so, yes, thank you very much. Um, and that buys us One and a half coffees every two months.

Jonathan Bennett: Oh, come on, come on folks, jump on board.

JP Mens: No, no, I'm not, I'm not, but please don't get me wrong. I'm certainly not complaining, okay? And I'm far from begging. That's not the point of this whole thing. Sure,

Jonathan Bennett: sure. I imagine what you would be more interested in is when someone comes along and says, Hey, I'm using this, I've got paying customers, I would like this feature.

I imagine it would be nice to say, I will contract you to add this feature. Like that, that's always nice when that happens.

JP Mens: Yes, it has happened I think once or twice where people said, yeah, I would be willing to pay. Um, but then it sort of more or less petered

Jonathan Bennett: out. Yep, yep, understood. Alright, I've got to ask you two final questions.

I get emails about it if I forget to ask them. And that is, what's your favorite text editor and scripting language?

JP Mens: Um, well, my favorite text editor is unfortunately VI because I, I'm just, I'm an old fart and I can't do anything else in my next life. I'll do Emacs, but other than that, it's VI. It's in my muscles.

I know VI by heart. I set my command line to VI mode, of course, obviously. Um, so that's, that's what I use. And, um, I'm horrendously, um, I'm horrendously unhappy if you give me anything else, even though there are many wonderful things, uh, wonderful code editors in the world today, but I just can only do VI.

And scripting language, you said, um, Programming language, I would have to say C. Uh, scripting language, today I would probably say Python.

Jonathan Bennett: That is fair enough. All right, sir. Hey, it is, uh, it is past the bottom of the hour now, and we started a little bit early, so we are, we definitely got a full interview in.

I want to say thank you so much, JP, for being here. It has been a pleasure and a delight to talk to you and to learn more about OwnTracks. Thank you, sir.

JP Mens: Thank you very much, both of you, for your very good questions and it was a pleasure being with you today.

Jonathan Bennett: All right, hopefully we will bring you back again at some point in the future.

We can talk about what has changed. All right, Jeff, what do you think?

Jeff Massie: I think this is, this is pretty awesome software. I like the ability to, you know, record and stuff and, and keep it out of, you know, I guess big corporate hands, at least, at least somewhat, you know, they're going to have to work a little harder for it to figure it all out.

And, you know, being able to send. Uh, it's a good way to, to just send trips or something, or, you know, cause, cause like I said, where I ride motorcycles, there's a lot of times where, oh, we took this trip to wherever and you can see this waterfall or monument or whatever, and okay, here, here's the route to take.

And then you can share it with people and let them, uh, see what you've been doing and I, and I like that. It's not, you know, it's open source is always good, right? You know, it's. More control, less, Oh, Google killed

Jonathan Bennett: it. Well, that's, that's one of the things that I like about, about this one in particular. Um, so I, I think of, I think of kind of the, the location tracking for, for one thing I do, I use it for business to be able to keep track of where we've done work throughout the month.

But one of the other things where it makes sense for me is kind of a safety. Um, You know, being, being able to track someone where they're at, whether it's on a hike or on a long drive and being able to have, you know, okay, this is the last point they were at. I know they were traveling at this speed and suddenly they fell off the map and I haven't heard from them in a few hours.

Well, if I need to go check on somebody, I've got that data. And so that kind of makes sense to me from, from even like a, a, a safety standpoint and the fact that it works without having to rely on. Google systems being up or, you know, it's, it's not sending things to Amazon AWS. You can host it yourself that that's kind of a big win for me.

Um, and then, you know, you can, if you really want to do the work, you can do this totally off grid. It generally works over internet, but you can make it work over wifi. Uh, you can make it work over other radio systems. If you. Again, if you put the work in to do it. So, I just, I love the fact that it's so, um, it's modularized.

The way that they've built it. You can build out your own system just using some of the pieces that they have if you want. And so, I enjoy that. A lot of things are really neat.

Jeff Massie: I use right now Life360 for my kids. Just to keep track of them and you know they're selling that data. I'm sure they are. And this would be a totally good uh, replacement for that to where.

Okay, and, and I don't get all the, the spam built into that app, you know, it just, it does what I want and it's, you know, yeah, there's a little, little bit of work, set it up, but after that, then it's nice and clean and no, uh, no

Jonathan Bennett: advertisements. I think, I think we may have talked Jeff into setting up his own little own track server.

So that makes me happy.

Jeff Massie: I, I'm, I'm going to look in, I'm going to actually look and see. If I can get this done and see what it takes and, and I'm, I don't know Docker. I'm not a, I, for those that don't know me, I'm much more proficient on the hardware side. I work in the semiconductor industry, so I can, and I've been a Linux user for many years, so I know the command line, but I am not a.

networking administrator expert by any means.

Jonathan Bennett: Well, you, you know, some people you can come to and ask questions if you run into any problems. And that I do.

Jeff Massie: Yes. I've got Jonathan on speed dial.

Jonathan Bennett: There you go. All right. So next week we're going to have Tony Zioli and he's going to talk about WordPress plugins, open source plugins, and then free and open broadcasting.

That's going to be really fascinating to talk with him and, uh, our co host next week, the plan is for Doc Searles to be back as a co host. I'm pretty excited to have him back on the show and, uh, he's going to, he's going to slot in where it makes sense. Still definitely a friend of the show. Um, so looking forward to that.

Um, Jeff, do you have anything you want to plug before we close?

Jeff Massie: Uh, just catch me and Jonathan over on the Untitled Linux show on the Twit Network and, uh, always, always happy to have you over there as well. And I, and I really appreciate the, being able to. Step into Floss Weekly, been a long time listener, you know, first time co host and I really enjoyed it.

I had a lot of fun

Jonathan Bennett: and really appreciate it. Oh, yeah No, it was great to have you here. I appreciate it as well. All right, you can follow my work At, well, on Twit, the Untitled Linux show that's part of Club Twit, but also at Hackaday. Do the security column, goes live every Friday, as well as Floss Weekly, of course.

Uh, and then one more thing that I'll plug, hopefully I don't have to plug this for very long. I've got a, I've got a buddy that is a programmer. Uh, C, C pretty much anything you want to throw at him, he can manage. And, uh, one of his specialties is Android systems bring up and he is between jobs right now And he's starting to have to think real hard about what he's going to do to be able to keep the house So he is he is looking for something if that is a if that is a piece of your organization that you need Get a hold of me and I can put you in contact with him.

Great guy That is the main thing that I want to plug this week and we will see you next week on floss weekly

Episode 766 Transcript

17 januari 2024 | min

FLOSS 766

Jonathan: This is Floss Weekly, episode 766, recorded Wednesday, January 17th. WebRTC, the hack that connects everyone to everything.

Hey, this week Sean DuBois joins us. We talk all about WebRTC, including The Pie on Go implementation, how WebRTC helps people connect to Tor, we talk about how WebRTC is coming to OBS in some pretty exciting ways, and then there's also BroadcastBox, which really fills a niche need for live streaming. You don't want to miss it, so stay tuned.

Hey, welcome to Floss Weekly. It's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and I've got with me the man, Dan, the man, Method Dan, the, the original Linux outlaw. Hey, sir. How are you?

Dan: I'm good. Thank you very much, Jonathan.

Jonathan: How are you? I'm, I'm good. I'm good. I'm almost over the winter sickness that hit, I think, everybody in my town.

I still have a tiny cough, but other than that, I'm feeling pretty good. So, you know, hopefully springtime is here. Not literally, of course, but as far as health goes, springtime could be here.

Dan: Yeah, sounds good.

Jonathan: So today we have a really interesting guest and I'm excited about this because he kind of specializes in some things that I use, that I've had to fight with, that has driven me crazy, that can do some amazing things.

So we're talking with Sean DuBois and Sean is something of a WebRTC expert, which really puts him in a, in a class, almost all of his own. I'm not sure that anyone. Really fully understands WebRTC. So he's the creator of Pion WebRTC, which is a Go implementation. And then he's also the writer of WebRTC for the curious, which I need to go bookmark because that sounds really useful.

And then he's also been working with OBS and BroadcastBox to add some Really, really neat WebRTC capabilities to those two applications. And we're going to be sure to ask him about that. Without any further ado, let's go ahead and bring him on. Sean, welcome to the show.

Sean: Thank you so much for having me.

Jonathan: Excellent to have you here. So let's let's start at the beginning. Give us the I guess the quick version of WebRTC for the curious. What are we talking about with WebRTC? What's the big problem? It's trying to solve.

Sean: So with WebRTC, the reason I love it is you can connect two people on a Anywhere in the world, and through NAT traversal, even though me and you don't have, you know, like a public IP, we can establish these temporary holes in our routers and connect to each other.

So instead of, you know, think of it like almost like automatic port forwarding. So like, all of a sudden, me and you, we don't have to depend on any servers, we don't have to pay someone, like we could just connect directly and start exchanging audio and video. And then WebRTC is both an API and a protocol.

But the cool thing about WebRTC, the API in the browser, is Is in like 40 lines of JavaScript, me and you can whip up like a simple video conferencing, which If, if you remember like 15 years ago that it was just like Space 8, I mean you paid like expensive licenses or you used proprietary software.

Like I remember struggling, yeah. Or, or you had to

Jonathan: fight with the FFmpeg and FF, oh I forget,

Sean: FFserve or whatever it is, which

Jonathan: It was never great. Yeah, I, I implemented a web streaming service way back then and it never was great.

Sean: And like my, my memory is just fighting and fighting to get Skype to run on Linux because like I was new, I didn't know anything about the stuff and I was just like, I just love that we're free from proprietary software to be able to communicate with each other.

It just feels like kind of this like fundamental thing we should have.

Jonathan: Yeah, so I, I use WebRTC. One of the places I use it in a bunch of places. But I do some work on the zone minder project. And so, you know, that is all about being able to watch webcam or not webcam security cameras and you get to looking and security cameras.

Most of them these days will give you an H 2 64 feet or an H 2 65 feet. And if your security camera happens to implement the H 2 64 baseline, Well, that's something you can ingest into WebRTC. And so we had, we had this project where we use, we actually use the, the Janus WebRTC service or server, excuse me.

And so we're able to run Janus and have it go out and grab that H264 stream and then go directly to the browser. And the, the latency on that is incredibly low. And that's just part of how WebRTC works. But the thing that really drives me crazy about it is. So your browsers have they've got all of this really great, really robust video codec processing stuff.

And then they have the WebRTC stuff. And it's totally separate. So you've got to, unless you're coming from a browser to a browser, you've got to really cross your T's and dot your I's and give it exactly the video feed that it wants. Is that something you've run into? Are browsers fiddly? Does it drive you crazy?

Sean: I think it's a side effect of like WebRTC was made by Corporations because they were trying to ship a product and like when everyone got together and they saw money to be made by putting conferencing in the browsers like They weren't expecting people to, like, do webcams and stuff like that. So I think it's the same way that, like, we got Linux, you know, like, Unix was kind of thrown over the wall, and people used it for their own purposes, and that's where we're at.

And I think it's the same way with WebRTC, where you can, like, build all these interesting things with it, and we try to make it better for people building unique things and hacking up interesting things, but, like, totally. Like, if you go off that beaten path You will quickly find that like, yeah, your, your use case is going to be hard, but like you can build amazing things.

And that's, that's my passion is, I just want to empower people to build interesting things. Cause I feel like, yeah, this is like kind of, you know, maybe my Unix, you know, instead of, you know, like now that's all easy and accessible. And maybe that's my hope with WebRTC in the next 10 years. Sure.

Jonathan: Are you following the, kind of the slow plotting progress of making WebRTC work with H265?

Sean: Yeah, so I saw that they were like contributing patches for it and really like so WebRTC is built on RTP, which has been, you know, was in the IETF in the 90s. So it's like there's The only reason that issue 65 wasn't happening was because of like licensing and like politics between companies like it's not a technical reason, and then like the Web RTC implementation in the browsers is owned by Google's.

And so then you have this thing if you have to go convince them to merge it and they have other priorities and things move slow. So yeah, there's just there's a lot of moving parts to make it all work.

Dan: Yeah. It, it's, it's definitely there's a bit of p like all of these things. There's always politics involved always, which we love.

Yeah. We love politics. So Sean, I wanna ask you about, about Pion. Mm-Hmm. Tell us how you came to create pion and, and what it is and what it does.

Sean: Yeah, yeah. So it is a, so we RTC when it started, you just had like this. We had this API in the browser and two people could connect their browsers by kind of just sharing this bootstrap message.

So like on browser a, you would call create offer and then on browser B you would call create answer and you'd like send these each other's offers and answers back to me forth. But there was like there was like no official like implementation for like just doing it on the command line and doing and doing stuff like that.

But a lot of servers existed. Like the one you mentioned Janice, And so I, and I worked at a company that we were doing like things that were off the beaten path and like nothing was really working for us. And so I had left that job. I was at another one and like this, this bug never kind of left me that like, okay, there's something that could be done really great with WebRTC and that's how it started.

And then this feedback loop of like more people were coming on and building interesting things with Pyon and it's just kind of like. Fueled my love for it. So like Tor has this project called Snowflake where they use WebRTC to do NAT traversal and like get around firewalls. Because as you can imagine, like having to download Tor, that's, you can, you can block that pretty easily.

But you can't really block WebRTC conferencing, everything goes down. So like that, I think that's like the way to get around censorship. So yeah, that's kind of the tangent. It's like I started building this thing and I kind of fell in love with all the people using it.

Dan: Mm. And it's, it's a go implementation of WebRTC.

So what made you choose go? Was there a reason why you just thought you thought go was was the language to use?

Sean: So if I had done so a C plus plus R implementation already existed that was done by Google. And so that, that kind of already existed. And there was a couple of other things that made me fall in love with go.

It's like super easy to, to build and deploy and use. I just, there's no like, You don't have to worry about build systems, you know, people can just pull down the code and use it. I have a lot of concerns about like the memory safety aspect of it. So like, as I've been in this space, I see a lot of people that, I'm amazed at the exploits that come up with C and C and it felt kind of imperative to do something in a safe language for WebRTC.

And then, That, you know, honestly, that was just what I was familiar at the time. I've been writing, writing Go for years. I felt comfortable with it and it was kind of that sweet spot. So like, I think a lot of things I can look back and analyze and be like, I made this decision for all these good reasons.

Most things I just kind of fumble into and then I'm trying, you try to explain it after the fact.

Dan: Yeah, that's, that's the way to do it, don't worry. That's definitely the way to do it. So, you mentioned that people are using Pion to do really cool things. So, can you give us any examples of something that you've seen that you think, wow, that's really cool and you didn't expect maybe that people would do with it?

Yeah, yeah

Sean: so the one, the first one I The one that kind of got me started was with Tor, when I saw like, wow, like for me, this was just like video conferencing technology. But the fact that people could get around firewalls and exchange things the other one that I think is really cool is WebTorrent.

So this idea that instead of having to download a client, I can just pull up with my browser and I can start, you know, doing peer to peer file exchange with other people. Again, like very, very hard to block if you just have, if everyone has a browser. And so if you can go and connect to each other. And then The, the one that always amazed me is like, it was like one in the morning and someone shared in the pine slack a video when, when COVID happened, someone shared a video of them.

They were, they're working for a company that had all these arcade cabinets and people couldn't come into it. And so they wrote something so that you could remotely control a claw machine, a claw machine. Cause they were like, for the next two years, they had no business. They couldn't bring people in or whatever.

And so like someone like at the wee hours of the morning showed me like a claw machine being operated via pine. I'm like, that is super cool. Yeah, and then there are projects that made me feel, I would say, good about myself. Like I had built something that actually helped people. So, there's this company that they they have robots that are in hospitals, and when people have infectious diseases, they would bring in iPads with these robots that were able to call into people's families, like if, if they were, you know coming towards the end.

And like, I, yeah, I just, I always wanted to build something purposeful and like, that's useful to people. So yeah, those are kind of like off the wall things. And I, I sent a list of over of a couple of like weird projects that I always liked. And then, yeah, that's, that's why I do this.

Dan: Yeah, that's amazing.

So you mentioned that like the whole COVID thing, did that kind of accelerate it in a way? Because people needed ways to communicate and to interact remotely. Suddenly there was like the whole world needed it.

Sean: Yeah, yeah. And that. It dramatically changed my life as well, because, you know, I was working on this little piece of WebRTC technology just because I thought it was fun and interesting, and then COVID happened, and then you had like the gold rush.

So I had, you know, like Venture capitalists and people like messaging me and being like, you can do, you can turn Pion into a private company and become a billionaire and stuff like that. Like all of these like crazy things, I'm like, I, I'm just some guy that, you know, pushes code to GitHub. I have no idea what I'm doing.

So yeah, like the, the, it really changed things a lot and it's, it's definitely quieted down since then. But I remember at the time it was very exciting seeing, you know, you had all these people coming on with all these interesting things they wanted to build and do. But now we're kind of in that post everyone wants to go back to the office slump, but it's the same to me.

Like, I just enjoy working on this stuff. And like, I'm, and there's just always new things like the OBS stuff that there's always new things you can do. Hmm.

Dan: Yeah, definitely. I mean, so where does, we talked a bit about Pion, where does it fit into the kind of WebRTC stack? Does it replace a browser? Does it sit between the two browsers?

How does that kind of look?

Sean: Yeah, so I think, I would say it sits between the two browsers where, imagine if So if you're doing like a call that's one to one, it makes sense not to use Pion, but let's say you're doing a call where you have 500 people, you don't want to upload your video to those 500 people, like you only have enough upload to do maybe two or three.

And so Pion sits up there, and you upload to Pion, and then Pion kind of fans out the video. Or you have people that are building like unique clients, where you know, you're like, I need to, Do I need to grab X frames from X 11 via ion and then I put 'em in the browser so I can like remotely play a video game.

So like a lot of like remote desktop services or remote game streaming use ion. 'cause they run ion on those hosts and then that's how they get the frames up. And then via data channel, they send your key press events back in.

Jonathan: So, so the answer is it, it fits in, in either place. It can sit between the browsers, but it can also replace one of the browsers.

Yep. Yeah. What's kind of the list of video sources that Pion can take and turn into

Sean: WebRTC? I mean, it's just kind of like a generic pipe of give me H. 264, give me VPX, give me AV1, and I'll, and I'll packetize it and send it for you. Like, it doesn't make We have an implementation of, we call it Pion Media Devices, where we have like, Hardware support for random things like the Raspberry Pi and stuff like that to to give you easy capture stuff.

But Pion in it, Pion itself has no capture code.

Jonathan: Okay, and so what, what's kind of the connector? Is it, is it G streamer? Do you have pipe wire support? What, how do you interface with Pion?

Sean: Yeah. So some people, well, a lot of people don't want video at all. So, like, we have to keep video out of, yeah, out of that WebRTC repository because, like, they don't want that.

And then I've seen people use GStreamer for capture. A lot of people use FFmpeg. A lot of people are, like, restreaming or bridging stuff. So they'll, you know, like, they'll be uploading video via RTSP and then they'll put it into WebRTC. And so yeah. So I've, so I've seen all these different ones.

Jonathan: So if we're talking about using Pion, And I'm beginning to see that this is not at all the only use case.

But if we're talking about using Pyon for video do you see a future where Pipewire is part of that? I, I'm, I will just say now, I am a huge Pipewire fan. I have used, I've used Dalsa, I've used Jack, I've used PulseAudio, and almost every problem that I've ever had with audio on Linux, Pipewire has come along and solved it.

And now, Pipewire is about this close to solving all those same problems for video. And so I'm a huge fan of it. Yeah.

Sean: I don't, I don't see why not. Like it's, it would become like just a grab stuff from Pipewire and send it into Pion. But like if with Pipewire maybe it does, like there's other great WebRTC implementations.

Like there's a, there's a, so there's like this heavyweight one by Google called LibWebRTC. And then there's another lightweight one called the data channel that just uses like CMake, super easy to build. So maybe with Pipewire, it would make more sense. to do that like C and C interop with libdata channel.

I guess it just depends on like what the project looks like. And if you want to run all of your stuff in the same process yeah, there's like a lot of different things to consider.

Jonathan: Sure. But it, it sounds like, and I'm, I'm beginning to discover that the, the coolest thing about Pyon, while it works for video, is that you can put, you can put any data inside WebRTC.

I, I'm still trying to wrap my mind around exactly how that works. So does it, Does it look like WebRTC to, say, a firewall sitting in the middle that's doing deep packet inspection?

Sean: Yep, yep. It just thinks it's a video conference service. It doesn't know. Oh, wow. That's interesting. Because WebRTC, like, it has these data channels so, you know, people can send messages back and forth and metadata.

Like, this is my user icon and stuff like that. And now it's being abused, well, used, reused for all of these, like, interesting things. And that's, like, the cool part. I think that's the cool story here is, like, here's a technology that was made. To do, you know, all of these standard use cases and then people saw this and are just adopted and that's and that's why I want to come on and talk today like all these listeners, I'm sure they have interesting things they want to build, but their perception is WebRTC is just a video conference technology, but the more people that hear like this is like crazy powerful technology that can that can get around firewalls that can and for the purpose of like It gets around firewalls because it's just trying to establish for a video conferencing But now you can use that for all these other interesting things.

So you can pipe wireguard through it No problem. Oh, yeah. No people already do like there's a there's a company that already does that like they have a And then the other cool thing is since they're putting wire guard you can like They have like a shim that goes in your browser as well.

So you can like get a little shell and like access stuff from your browser and wire guard. And then yeah, and then like since it's like this generic protocol, it's way harder to block or like package shape.

Jonathan: Right. So is that a, is that an implementation of wire guard in JavaScript? Is that what I just heard?

Sean: Yeah, I think so. Like, I wish, I wish I, like if you go to the, there's a pion example. That's great. Or a pion use repo and I think it's in there that you can just see like the, that example. Oh, that,

Jonathan: that is outstanding. That is hilarious. So are, are, are people using this, I assume, to get connection in, in China, get around the, the great firewall and all of that?

Sean: Yeah, no, I get like so like for a short time Payam was blocked in Russia because it was being, and so like I, I noticed it because a lot of video people were reporting like, oh, like my, My video stuff has gone down in Russia, but it was blocked because of Tor Snowflake. Interesting. Because Pion's DTLS implementation was slightly different than OpenSSL's.

So I had to fix the diff. Because they could look at the handshake of like the TLS handshake and like find the differences. But I minimized that difference and now it's not blocked anymore.

Jonathan: Oh, excellent. That's, that's, that's fun. I've, I've done just, just a tiny bit of helping somebody get around that, that sort of thing.

And oh that's, that's really neat. It's also fun that you get that sort of immediate feedback because it's the cat and mouse game.

Sean: Yeah. No, I, I love it. I mean, I just love the, I also love the fact that someone had to struggle to figure out, to find that little difference between Pion and OpenSSL and the fact that I closed it after they were trying, they probably spent some time trying to get around it.

Like, it brings me joy to know that like, now more people can get on the internet that couldn't before.

Jonathan: Yeah. And so at, at the moment, as far as you can tell, it's, it's Indiscernible. So people are actively using this around the

Sean: world. Yeah, so like Tor has these nodes inside different ISPs around the world and they're always testing connectivity.

And I haven't heard from them recently, so I think it's good to go. No, no news is good news. Yeah, yeah, no news is good news as far as, yeah, so like yeah, no, I love this stuff. Yeah.

Jonathan: So let's talk for a minute about the, the book, the e book. It's WebRTC for the curious, and you mentioned I think before the show started that it's got a permissive license, so is that and, and someone from the, our audience, Bitten, actually asks, is it Creative Commons or something similar?

Sean: I think CC0, like it was like the least restrictive, like I don't, you don't even have to attribute me, I don't care.

Jonathan: Basically as close to public domain as you can get. Yeah, yeah, yeah. Alright,

Sean: tell us a little bit more about the book. So, with WebRTC for the curious I went on paternity leave and I wasn't able to do any program for programming for a couple of weeks.

And so I was just like sitting on my phone and writing it all in Markdown. And that's all it is. It's just a, you know, like a bunch of chapters of Markdown. And my goal was that a lot of people, they find WebRTC frustrating because they don't understand what it's trying to solve. You know, they're like, Oh, like WebRTC is so frustrating to use.

Like, why is it so brittle? But it's because it's trying to solve all of these like different problems and like you get an appreciation. You know, and like empathy for the technology when you understand that. And then it's also, I think, if you understand it at the protocol level, then it gets a lot easier to use the APIs.

So those are the things I'm trying to solve. It's not like a go to this. It's like, and it's also like an implementation agnostic thing. So you can use the Python or the Rust or the Go implementation or in the browser and like this book is applicable to them all. Like it's not like a here's how to use Pion kind of book.

Mm hmm.

Jonathan: How detailed does it get? Is, is this sort of a handbook that you can use to go write your own? Yep.

Sean: Yep. That's the goal.

Jonathan: How, how hairy, how hairy is that? I've kind of always gotten the, the impression that WebRTC, you know, under the hood is a real pain just because my experience has always been that live video is a pain.

Sean: Yeah, I think so WebRTC is like a gluing together of all these existing IETF protocols. And so it's like, if you, if you like, there is no way that. One person could learn the entirety of TLS and STP and ICE and RTP and like all have it in their head at once. But you go and you implement the minority of it you need and you could do super powerful things.

But yeah, I, I also get the sense that like people in the video and WebRTC community, they like to oversell how challenging things are because it makes them look and feel better about themselves. And like I just don't, I just don't think like as of anything like you learn it all of a sudden it's not that scary.

Like that's That's what I'm hoping to convince people. And at the same time, I'm watching all of these companies with these competing proprietary technologies to WebRTC. All their marketing material is like, WebRTC is challenging. Come use our proprietary thing that's in our set top boxes. Right. But there's no one doing that for WebRTC.

Like, there's no company that, like, evangelizes or is selling WebRTC. Like, it's just a internet protocol in the IETF and W3C. Yeah.

Jonathan: Let's talk about NAT for a little bit. Because that is we talk about peer to peer, and NAT is the thing that gives everybody headaches. And it, it seems like WebRTC handles this fairly well.

What's the, what's the story there?

Sean: Yep, so with WebRTC so I, I guess between the NAT, like for a quick explanation of NATs, it's like if You know, we imagine we've run out of public IP addresses. And so inside your router, you have, you know, I imagine we have, well, or there's the people that are hoarding massive amounts.

But I guess that for, no, I was actually pretty excited. My ISP finally gave me IPv6 and I magically, yeah, I magically had like a peer to peer connection with someone behind a host without any of that. It was cool. But anyway, so what WebRTC does is it uses You're inside. So you're inside your network.

You don't know what your public IP is. How do you figure this out? And so what you do is you send a single UDP packet to a stun server, which is like a public host. And that public host responds, Hey, thanks for sending me a packet. Your IP address is 103 blah, blah, blah. And the port that you sent me on is 4, 000 and now that you have that information, you can go and tell everyone, Hey, this is my public I.

P. And this is the I. P. And this is the port to send to me on. And your router will temporarily hold that port open so that anyone can send in to you. There are different rules about how that port allocation works. So most of the time, yeah, you can give out that port and anyone can send into it. But some routers say, okay, only the person you sent to consent back.

And so then you can't get peer to peer as well. But yeah, that's, that's kind of the short of it. And then there's other methods of opening up your your net. So you have like PCP and that PMP and those aren't implemented in most web RTC implementations, but there's also other ways you could like, Automate, making yourself available behind an app.

Sure.

Dan: So this might, you know, in the whole thing we say there's no stupid questions. I'm going to ask a possibly stupid question. So do I need to have UPnP enabled on my router for that to work? Or would it, you know, with the stun server and all that? Because I have an idea of how that works. Would I need UPnP enabled?

No. Or would I

Sean: not? No. So, like, UPnP is like You have that I'm actually not familiar, like deeply familiar with U-P-M-P-U PMP is like, you go to that and you, and it's like you can, and that's like a service that you talk to, to configure with stun. You just send a single packet in and, and like it opens that nap binding for you automatically.

And that was, I was this, this whole, war kind of happened before my time where it seemed like it was UPNP versus the stun people where some people wanted UPNP and some people wanted stun and they like went back and forth and talked about which one is best. But I've never, I, I, I was able to find some of the history on it on SSH.

org, like the company that like had that offers that like they had some of the history behind the two implementations or the two methods of naturaversal. But. I don't know the full history. And that's actually another part of Web RTC for the Curious, in the final chapter, I go out and I'm like interviewing a lot of people that were working on this technology back in the nineties and early two thousands.

And like, what was it like at the time? And like, what were people doing and like, what were we trying to solve? Yeah,

Dan: it's, I I just ask 'cause I, I don't have it enabled on my router in the end. Anybody I know who's. Got anything to do with security? Always says to me, for God's sake, don't use UPNB. Cause it's like, it's allowing, you know, you're allowing people to open all kinds of ports.

That's pretty cool. So I wanted to talk a bit about something that I know Jonathan's very kind of, is close to his heart, which is OBS, which is what we're using right now. So we're, we're talking, and Jonathan can explain this better than me, but we're talking via OBS and he's piping it all together with his pipewire magic and all the other stuff.

How does, how, you, how do we get, you know, how do we get Pyon into OBS and how's that looking?

Sean: So, A different WebRTC implementation, LibDailyChannel, is powering, so OBSv30 that just came out has WIP support, which is just WebRTC. And so, in the same way you use RTMP, you can just drop in a stream key and a URL, and then you can just use WebRTC.

So today, that means you get a couple new interesting things. If you use WebRTC output from OBS, you get more codecs. So WebRTC already supports AV1, it supports VP9, it supports Opus. The other interesting thing about doing WebRTC over RTMP is you get simulcast. So with with RTMP, you know, you just send up one stream of H.

264 and that's it. Like, but with WebRTC I can send up my high, medium, and low feeds. And then at the server I can forward those all out. So that, so that'll make it like super cheap for server operators to run and do things. So, you know, like my hope is that self hosting will then become way easier to do.

Because right now you need, you know, so much money to run all these transcodes. And now you can just have, you can just be forwarding out video packets. And use all of like, most GPUs, you know, can encode six, seven streams at a time. So why not let them do the work? So yeah, so I started with, The WebRTC support in OBS for doing the output, and I'm happy to answer any questions about that or talk about why that's cool.

I can kind of move on like to like what the next step is if if you're ready.

Jonathan: So I definitely want to talk about that. We missed one thing that came to came to my mind. One of the things that I've thought with with WebRTC is the different variations of, say, H264 is you've got H264 baseline. You've got Vanilla H.

264, you've got H. 264 and I kind of alluded to this earlier, your, your web browser implementation, some of them are, are extremely picky about what versions of that they will, they will run. Does, does Pion help us with this any?

Sean: No, so, so, yeah, so Pion isn't like a decoder for video, so it does understand like the bit stream itself, and it can like slice it up for you in the different now units, but like it, like really that comes down to like, In the browser, you're probably using open H.

264, which only supports like a subset of the full H. 264 spec. Right. My answer to that is always like, hopefully, you know, we can just get AV1, get VPX, and like just keep moving toward, and that, and that's why I did like this OBS stuff, because I'm like, I just want to see adoption of these things that are easier to use.

Mm hmm.

Jonathan: So let's, now that we've covered that, let's, let's pivot back to OBS. And I have, I have been real curious about the, the new WIP support. And so that is, that is mainly aimed at, you've got a server somewhere that's ready to ingest WIP. Is that probably going to be FFmpeg on the other side or something like

Sean: that?

Yeah, so right now GStreamer has WIP support. And FFmpeg has a patch out and then I have this reference server implementation called broadcast box Where it's just like a go server and You can just do Docker Compose up and you get this whole whole setup. And so then you can just like build your own, like, Conference, you can build your own broadcast thing in like 30 seconds and then just use the whip up to go out to it And then use all the cool WebRTC features Okay, well,

Jonathan: let's let's chat about that for just a second because it sounds interesting broadcast box you can build your own twitch.

You can do your own live streaming there. Now you've got to have the bandwidth i'm sure to be able to handle this But how many you concurrent listeners watchers can that handle? How does it scale? Yeah, so

Sean: like I, I expect it to scale RTMP since you're not doing any of these trans codes. You're just like accepting UDP packets and then you're forwarding them out.

Sure. I think it all depends on the box you run on. So like when I was, when I was doing someone asked me to do benchmarks for like, hey, if I could read, if I could send out RTSP, you know, we got up to like, I forget the exact host on AWS, but we got to like 30, 000 sessions. Because it's just, you know, it's just like forwarding UDP packets.

It's like we're not doing any work. Like we authenticate those packets to make sure they come from who we expect and then we forward them out. Sure.

Jonathan: Do you, do you have people actually using broadcast box? Has it gotten some some uptake? Yeah.

Sean: Yeah. So like on the, I have this discord called real time broadcast that has like a couple hundred people in it and we, I'm just like trying to help all of these like people that are trying to do interesting things with it and trying to self host and yeah, like I don't.

I can't, I don't really know who's using it. I'm getting PRs and I'm doing stuff with it. Like, my whole hope is just to evangelize what is possible with WebRTC. So you've got this simulcast that will let people do interesting things. So, you know, like right now, like you have to be, I forget the correct term for it, but like on Twitch, you have to be like chosen to get trans codes.

But with, but now with broadcast box, I can just do whip. Simulcast and now you can go and I can do 1080 P 720 P 40 P and I can do what you know, I can do high quality and all of those. I can do different viewpoints because WebRTC lets me do multi track. Yeah, so basically it's just like this laundry list of here's what the future could look like.

Jonathan: I assume you have some example HTML and JavaScript code of here's how you put this in a, you know, embedded in a website.

Sean: Oh yeah, so it's so you'll probably just I frame it. But like broadcast box itself. It's just like you do like a docker compose up or go run dot and it gives you everything.

That's the back end. The front end. It's it's themed. It's like it's all just ready to go. Like you just you start it and you give people the URL to it to watch and you push into it to publish and that's it. Okay,

Jonathan: very neat. Boy, that sounds that sounds really interesting. One of the things that blows my mind doing this show is we talked to open source folks.

And And so many times it's like, okay, I had this, I had this terrible problem 10 or 15 years ago. Why was this not around back then? Because it makes it so easy.

Sean: It's, it's just such a hard uphill battle to do these things because it's, a lot of it is convincing people that it's interesting. You know, a lot of people are just happy with RTMP.

They're like, this is good enough, you know. And it's, and it's a lot of people that are happy with it because they're in, you know, countries with amazing internet, but then people, you know, it's so yeah, people are just complacent. And like, I'm also incredibly idealistic. So, you know, I'm going to keep fighting this battle for, you know, Don Quixote.

I will keep fighting this battle no matter what, just because it's fun to me. Yeah. I

Jonathan: I can't help but think that Broadcast Box. Like, there's this, this niche use case where you want to be able to live stream something, but you either, you know, you really don't like Facebook, you don't like YouTube, or maybe you don't like the ads that YouTube or Facebook wants to put on your stuff.

I remember I helped a

Sean: church

Jonathan: out along, well, this is actually, this is the thing I was talking about. Back years and years ago. I helped a church out and you know, we had it put together where we could stream to YouTube That was easy but it's like you had no control over what ad was gonna get shown right before the church service started and sometimes that was just a It

Sean: was a problem.

Yeah, I can Yeah,

Jonathan: oh so this this all that to say Broadcast box itself. It seems like there if you could if you could get the word to the right people There's a really interesting niche use case there for Somebody that really, really needs to control their own streaming.

Sean: Which, in my opinion, is everyone needs to control their own thing.

Well, sure. Yeah, like if because if you accept free services, eventually they get taken away. Like, if you're not paying for something, then you are the product. You are the product, yeah. So, like, that's how I've been with this. It's like, with BroadcastBox, with Pion, like, I just want to give people things that they actually control, they can deploy, they can read the code and understand.

That

Jonathan: actually brings up an interesting question. Do you, do you make any money with this? What is the product?

Sean: So I had this opportunity, you know, like years ago, like when I'm, when I was like, Oh, do I start a company with it? And I felt that it would change my relationship with people. So like, I wouldn't be able to come on this podcast.

And talk about these things and just be honest about how I felt about things or that I would have to like I would have to filter myself in a way to make sure I make the most money possible, but now I don't like in this very freeing that I do. So I I pick up. I work at companies that use pion, but like I don't have to walk a company line, which so so no, I don't make any money off it directly, but I will work at companies that use it.

Jonathan: So you kind of have you've you've established some expertise in the field. Yeah, yeah. Yeah. So you're sort of making money with it. You just, your business relationship is not

Sean: Oh yeah, no, no, no. Some companies have gone down. Oh no, no, I actually make a lot of money because of it. Before I started I was kind of like this generalist programmer that would move between jobs, like becoming a WebRTC person.

I think like if I had to do the math, like my salary is like three, four X more than when I started. So yeah, no, like you should like, yeah. But like if anyone is wants to get into video, there is like so many people that have things they want to build and if you can help them build it, you know, you get a piece of the pie.

So like no, I'm very lucky and happy to be in the

Jonathan: spot. I am. Yeah. You're kind of the embodiment of that idea that you, you don't, let's see, how does it go? You don't make money with open source, you make money because of open source. There's a saying, something like that.

Sean: Yeah, interesting. And again, this is all accidental.

Like I, it's, I never like started out. I was like, I, I have this business plan. I'm going to figure out how to make money. Like I just, I, when I first started with WebRTC and Pine was small, I remember begging companies to hire me to work on WebRTC stuff. And people are like, we don't care about your podunk project that like and it's funny, like overnight, you know, things, things changed.

Well, I guess cause of COVID like that's what changed things for me.

Jonathan: Yeah. So it was a big part of it.

Dan: I, I heard we, we've kind of, we, we've talked about the history of web RTC and stuff, but kind of a little thing that just popped up in my head was I remember hearing that Amazon put a lot of money into web RTC development.

Is that true? And why is that something that they're pursuing, do you think? I mean, I dunno

Sean: if you can remember. Yeah, so I, I that was one of my jobs, so I worked at am so Amazon. They wanted to do a small WebRTC implementation for embedded devices. And so it was used in the delivery robots. It was used in the fulfillment centers, and it was used for a lot of the customers.

So, like, they had a lot, like most home security system cameras that you see, like they don't actually run their own video infrastructure. They pay AWS. And then they install this, like, small C SDK. So, like, the issue was that there wasn't a WebRTC implementation for embedded. And so I worked at Amazon for two years and wrote this, like, very small C implementation that could go on, like, embedded devices.

So that was Amazon's involvement in WebRTC was for that. And then I also worked at Amazon at Twitch doing WebRTC. So Twitch has this, like, built in guest star or watch together product that uses Pion. So yeah, so like I've, I've done stints at Amazon that do different WebRTC things. Ah, okay, yeah.

But yeah, there's like no top down leadership that says WebRTC. It's, you've got different, you know, like, very different, like, VPs, important people that take interest in WebRTC, and then I end up there and I do it for a little bit. That's cool.

Jonathan: That's very cool. Ah. So let's, let's talk about back to OBS.

We tried to get on this subject and got distracted. Now we've got, we've got WIP in OBS right now with version 30, but you, you've got some, you've got some things coming that sound real interesting. Let's, let's chat about that for a minute. What is, what is coming with OBS with WebRTC?

Sean: So we have this WebRTCC output, but now I'm going to do the inverse so you can pull video sources in.

So imagine if you wanted to have like a one to one call. I would whip my video to you, and then you would whip And then I would web your video back. And so, like, we, you know, you would send your video to the other person, and then you would pull that other person's video, and then you could just have a conference call in OBS, where you're just, like, sending WebRTC traffic back and forth, and it's in sub So, like, you could do this today with RTMP, but the latency just isn't good enough.

But I'm gonna make it so that you can do, you know Bidirectional communication with WebRTC. Oh, and the other downside is RTMP can't do the NAT traversal. So you have to run, like, a server up there. But now, like, you can just connect two OBS instances and you can start. So I want to make it super easy for people to co stream and, like, build things together.

But it doesn't

Jonathan: Okay. We take a very interesting view of that because we run an interview show and most people have OBS. OBS is the magic that makes the background of this work. So I am, I am extremely fascinated with that. Cause you know, right now we're using Video Ninja, which was originally called OBS Ninja.

It, it sort of was written to solve that problem. But it sounds like if we're doing it directly, there's, there's a couple of layers that you can just cut out of the middle of that. So that's really, that's really interesting. I, we will have to

Sean: give that a try once that is ready. I think it, it depends.

So with There will always be a video ninja because OBS, I think will, will lean towards more technical folks of like, you have to have like an understanding and gluing these, like understanding these protocols and stuff like that. Like, like there will, there will always be a world where people won't care about the actual underlying technology.

So I think for like this podcast, like we obviously love technology, we do stuff, but for the majority of people, they're like, I don't want to talk about whatever you see. I just want to pull up a website and, you know, like have things. So like my hope is that. Something like Video Ninja will exist where someone just opens it up in their web browser and then you can pull it into OBS still and then the, and then the other part is like the scaling aspect.

So like, let's say you're, you want to do a podcast with six people, you know, you're playing D and D and D together. You probably don't have enough upload to upload your video six times, right? And so you'd upload to Video Ninja once and then Video Ninja fans out to like those other five people. Sure. And then I am also very Personally attached to Video Ninja because the the creator hangs out in the real time broadcast discord and he, like, deeply, he, like, deeply cares about, like making WebRTC good, like, he, like So yeah, there's like, not a lot of like, products.

Well, I guess like, Video Ninja is one of those products where like, the creator like, deeply cares about doing right by people and like, building things that actually help for people, so like So I guess this is a shout out, like, if you aren't using Video Ninja today, like, it's, it is something that really respects and cares about its users, so like, I, I admire it.

Jonathan: No, it's, it's great. I, I really enjoy using it. The, the thing that irks me the most about using Video Ninja Is I know that OBS is running a web browser to be able to ingest it and that just sort of drives me nuts. Just knowing

Sean: that it's there. And we'll get, and the cool thing is we'll just like get all the way from that.

You'll still use Video Ninja, but then you'll just use like a web source. So like, it'll be like a pure C WebRTC implementation that's in OBS. And the other thing that's annoying about the browser is it's making all these decisions for you. It's like it's changing your bitrate and your resolution and it's like doing all these things in OBS.

And this pure C implementation, you'll set your exact bit rates, you'll set your exact codecs instead of having, because that's like the thing with I won't get, this, this is a fun tangent for me, but like, I, I don't want to derail if like, Oh no, no, no, no, we, we, we, we, we like the, we like the tangent. Yeah, yeah, yeah.

So like, you piqued their curiosity. Yeah, yeah, so like WebRTC and the browser is written in a way that it makes sure that It, it respects the amount of band, like the bandwidth you use, because imagine like an, if you could set your bitrate, an attacker could just like open a new tab and like spin up 10 HD videos and take down your internet.

And so WebRTC in itself is like designed that it limits your peer connections, it limits the bit, the bit rate, it sets the right codecs for you, it doesn't, and so it does all these things that are good for the user. But maybe not great for video for, like, getting the perfect video experience. But the nice thing about OBS is I can spin up a WebRTC session and I don't care what you say.

I want 10 megabits. I want 81 and like it just sends it out. And so and I'm gonna do other cool things. So WebRTC has this feature called FlexFec where with With Oh, with if you're sending video over RTMP and you hit packet loss, you'll just incur latency because it's going to like resend, resend.

But with WebRTC, you can do redundancy. And so I'll send, you know, 125 percent of video so I can keep that, you know, 100 milliseconds of latency, but I'll still have that sub second. Experience. So even if I'm, you know, on cellular or Wi Fi and I have a crappy network, I mean, you can still have the sub second thing.

And just in OBS, you can configure, Hey, I know I have a bad network. Send extra video data that is technically wasteful, but because of the network I'm on, I have to send it. So we're going to do a lot of cool things because now we like control it and we're not just pulling in a browser.

Jonathan: So, so let me, let me make sure I understand correctly.

I can, with, with the patches that you're talking about, I can give a guest a website. Video Ninja. They go there, WebRTC runs in their browser. I can then ingest from Video Ninja into OBS. Having to run the web browser underneath, it'll, it'll no longer be a URL source. It's now a, a, a web

Sean: source, a web source.

Yeah. Yep. And then you can tune all these things and like, you can be like, so like with UDP packets, you have to wait until they arrive. And so you can say like, Hey, I just want super low latency. So I'm okay. If I get, if I get some corruption, like. I want to play this video package as soon as they arrive, or I can be like, Hey, I'm okay with two seconds of latency, but I want absolute perfect video like you'll just have all of these controls that you didn't have before because the browser just decides what it's best

Jonathan: for you.

And then is there going to be an option to use whip to push a stream

Sean: back to video ninja? Yep, or that are video ninja already added that today. So you can you can push into this with OBS. And so the cool thing about that now. Is today we could have pushed into this and instead of, you know, like our videos, like doing H264 and maybe not being the best quality, we could have all pushed five megabits a second via OBS and had, you know, like, you know, you can control everything.

So, yeah, no, like Video Ninja today supports WebRTC output.

Jonathan: That is cool. When do you, when do you anticipate this landing in OBS? Can we have it now? Yes, can we have it now? Is it 31?

Sean: Do you think it's going to make it? So the, so the WebRTC output into VideoNinja, you can do that right now because you have the WIP stuff.

And then and the other tangent, like I'm super excited that we can kind of like chain operators. So like imagine you can do like WebRTC, you can do WIP into VideoNinja, and then you can do WIP into CloudFlare. And then you can use like CloudFlare to like do massive broadcasts and stuff like that. It's almost like a chain, a chaining WebRTC.

And then, okay, so back to the question about, so my hope is that Oh, so I'll share my repo, but I have. My, I have pull requests open against OBS with all of these things. And so you can go download the unsigned Flatpak and the unsigned Windows and Mac builds and just use them today. But then no estimates on when like it'll be merged.

Right, you're,

Jonathan: you're, you're not in control of the OBS repo. You've got to convince the people there that yeah, it's a good idea and that your code is sane. Yeah, which I've had, I've actually had good success with getting code into OBS. I've not done anything this big, but I've done some fixes for like Linux audio.

And they seem to be pretty reasonable to work with.

Sean: Yeah, so I think the, the, it's just, this is just a weird use case. Like, nine, you know, like a hundred percent of, a hundred percent of people are using RTMP to Twitch and YouTube. And to convince them to be like, hey, derail your release schedule so I can add my weird web stuff is a hard sell.

But, like, it's happening. Is there, is

Jonathan: there a place? That we can go and thumbs up your pull requests and be like, I would

Sean: totally use this. So I would encourage people, like you just show up in the OBS discord and like discuss with people and use it. Or like, I'll share a link to the real time broadcast.

I have like a Discord where people are talking. Like, you should join and just be like, Hey, I used this and it wasn't very good. Or like, Broadcast Box can be better there. So like, yeah, there are like communities to talk about this stuff.

Jonathan: Is, is that Discord the, the main place to, to work

Sean: with your code and your projects?

Yeah, yeah. So in the broadcast box, read me. There's like a link to this real time broadcast. And it's basically like, there's a couple like, Cloudfarer, Millicast, devs hang out there. Like, it's everyone that really cares about this problem of real time broadcast and WIPWEP and all that stuff. Alright. That is that is

Jonathan: extremely cool.

And I am, as you probably noticed, I am extremely excited at the idea of these additions to OBS. So I'm curious, what's the, what's the weirdest thing that somebody has done with your code? What's the thing that most surprised you?

Sean: The weirdest thing. So there was, there was sex toys that can be controlled via data channels.

I was, I was like, I was on that one, I was like, oh, that's an interesting one. I didn't expect that. Yeah. The one I really loved is like the immersive, someone like hooked up their like Nintendo Switch and the headset so you could play all of these games in like VR. I thought that was super cool.

Yeah. And then another one that I thought was super cool was like having a jump box. So like, You know, like every company I've ever worked at, you have like an SSH jump box or you have like a VPN kind of deal. But with WebRTC and the NAT traversal, you don't, you can like keep all your servers behind and firewall on NAT and just temporarily open up a hole to get SSH access, which I don't know, like that always was a fun one for me.

What else? I wish I should, I should have like scrolled through the the list before I came on, but those Yeah, I think those are the, those are the fun ones for me. Oh, and like the another one I really loved was the At the time it was cool, but I feel like now, so before, you know, like Stadia that shut down in xCloud, someone had built, like, game streaming and with a bunch of emulators, and so, like, I could go to this page and pull and play NES games and play Game Boy games, and I was like, that is super cool.

Doesn't seem as cool now that, like, all the companies have, like, tangled it on, but at the time it was, it was novel. Yeah, that

Dan: is very cool. I mean, that does sound awesome. And one of the things that you sent to us, Sean, in your, in your our secret back channel email that Cameron sorts out. It was you wanted to talk a bit about source available stuff versus free software.

Versus, you know, intentionally free software. So, what, what was it that you wanted to, to talk to us about?

Sean: Yeah, if we had gotten, like, I think and I had, this had come up recently who had talked about this. Oh, Bruce Perrins had, had like written something about this where there's open source. He told me, yeah, this post open source.

And so you have these two weird things where companies will join an open source project and they'll hire people out of it. And When they hire someone, then they get very busy and they don't contribute anymore. So it's been weird, like, you look through these generations of open source projects and like, they're very alive and exciting and then the people get hired out of it and then they get busy and then these projects die and it's not like, this isn't like the first generation for it to happen.

I feel like it's been happening since the beginning of open source where like, the creators get hired and they're busy and then, and so like, but maybe that's a cycle of life and then The other interesting thing is like the, the advent of source available. I think Chromium is kind of the poster child of this, of like, you can go look at the code, but then they roll out, you know, these like different ad.

I don't even know what that call I haven't been paying attention anymore, but like those the new like cookie cookie like ad identifiers that follow you around the web and like it's open source. Technically, I should be able to go and revert it and disable it, but you can't really because it's kind of the source available.

But at the same time, it's this chilling effect because it shuts down all the competitor browsers because everyone's like, Oh, I could just use this open source thing. Why would I ever do my own? So So, yeah, it makes me sad that, like, Well, it kind of goes back to the whole GNU thing. It's like, this wasn't about like, it's not just about the code being open, but it's like, does it respect users?

And is it best for the users? And like, how do you deal in this world where things aren't black and white? Where like, yes, the code is open source, but it actually is not respectful of its users. And like, it doesn't have its users best intentions.

Jonathan: Yeah. A hat tip to Simon Phipps, as more time goes by, I enjoy this statement more and more.

He makes a statement that source code licenses don't compile. They don't come down to precise meanings. And so really what you need is a community that actually gets it right. I, there's a, there's a story that I actually saw Steven J. von Nichols tweeted about it. And it's the idea of delayed open source publication as a rival to open source.

I, I assume you've seen this a time or two. The, the idea that, you know, we're an open source company, but we've got our proprietary stuff that we make money with. And so the promise that we make is, after this proprietary stuff makes us money for six months, then we'll release it as part of the open source product.

And I, I get, I get the idea that people need to be able to make money. Programmers need to eat and be able to pay rent. But some of these, some of these schemes that people come up with are just a little draconian. Are you familiar with the, the DOSP, the Delayed Open Source Publication

Sean: Model?

Yeah, and you have like these, All of these licenses with caveats, like you can use this software unless you make more than a billion dollars, and you're not, and it's, and it, for me, that's why I always end up MIT licensing things, because I'm like, as an, as a small fish, am I going to be able to go and sue, you know, Google if they use my software?

It doesn't, it doesn't matter. And they'll just use it behind closed doors anyway. So I'm like, all I can do is make my stuff as accessible and easy to use as possible because bad actors are going to be bad actors and there's nothing I can do to fight

Jonathan: them. I'm Yeah. Yeah, that's fair. One of the, one of the, whenever we talk about this, I just, I have to say kudos to OSI for sticking to their guns about the definition of open source.

And you mentioned, you mentioned Bruce Perens and, and his post open source. I haven't looked real closely at that yet. I've just looked at some of the things that he's talking about. And I have to say he is at least, he has the right idea. He's not trying to water down the open source definition. He knows that it's got to be something else.

I don't know if that's the, the, if exactly what he's laying out is the right way to go about it. I don't know if the ideas he has is what we need. But I, I definitely appreciate that it's not he's not trying to change what open source is. Because I think that is a, kind of a scary and dangerous road to go down.

Sean: And I think, I've become a little bit, like I always thought that, There was always this intention that we wanted to protect users that didn't want to protect themselves like they were never, they were, they were not, they were not wary of, you know, using non free software and they weren't wary of these services and I don't even know how do you protect people that don't care?

Like for myself, like I am always actively seeking out like right alternatives and things that respect me. But like if someone isn't, how can you, you can't force them to use things that care about them. I don't know. Yeah. Yeah.

Jonathan: Yeah. It's, it's an interesting, an interesting pickle. Well, you know, I think, I think we've covered a lot that was in the rundown.

I want to, is there anything that we didn't talk about that you wanted to cover that we can, we

Sean: can quickly cover? Oh yeah. I mean, like, do we, so we, do we have 10 more minutes is what, or are we out of time? Well, I don't know. It's

Jonathan: up to John. Not entirely out. We've, we've got about 10 minutes left on our recording

Sean: time.

Oh, okay. Yeah. No, I'm happy to talk about it. Like I, we can touch on like the open source career stuff. Cause like my career was made out of open source. So like I didn't, I started out in like the late 2010s like doing like lamp development and stuff like that. And I just kept jumping between jobs.

So like I was, I always want to encourage people to, to go down that route where you don't need a college degree. You don't need to like certifications. Like if you can open source gives you like. The tools are just laying out of the table. And if you start building things that are useful to people you can do Some really amazing things.

I'm sure I'm preaching to the choir. Everyone that's listening to this podcast already knows this But my hope is to get is that like, you know 18 year old me here's this again and feels a little less scared of the world because I remember being terrified You know, you're like put out in the world and what's next so

Jonathan: we we've talked to we've talked to some of the Linux kernel developers and One of the things, one of the statements that they make is that there's like, there's this number of patches that someone has to send in to the Linux kernel, you know, on average, and they will get a job offer.

And it's, it's, it's low. It's like you send four or five or six patches into the kernel and someone will be giving you a job offer. And that's, that's always been fascinating to me. If we're not talking about the kernel though, what does that path look like? Like, how does someone go from Sending, sending a source code patch to one of your projects, to actually getting hired somewhere and making money doing it.

It seems like, you know, it's, it's, it's one of those deals where it's like you know, there's the meme, you know, work on open source dot, dot, dot profits. What fills in the dot, dot, dot there.

Sean: Yeah. So what I see as most successful is you have to build something that is Useful that is just your name. So like if you go and send patches, small typo fixes to things, like you'll never get recognized.

But one of the great example, there was this developer Garrett Graves, and he built this thing called Project Lightspeed, which was there was this protocol called FTL that Microsoft had done. For their streaming site, Mixer, our Beam. pro, and he had made it so you can do FTL out of OBS and bridge it with WebRTC.

It's probably been five years. So he was like a sophomore in college. He went and built this project Lightspeed, and he got job offers, and you know, he got like a six figure job out of it. Like you just need to go, you need to go and build something that is just yours. Like you, you need to go and like start your own project.

You need to like, Respond to users. And I think like from there, you will see success. And even if you don't see immediate financial success, you will find a personal satisfaction that you have never felt before. Like for me, I never, I was never good at anything before programming, but you know, growing up, I was like mediocre at sports.

I was didn't get good grades, but like being like being able to write code and contribute patches and stuff like that, stuff like that. Yeah. It is so great for your self esteem and self worth and happiness. So, like, I would encourage people to do it just starting with that. And, like, you'll be amazed at how much it changes your life.

So that, that's my advice. It's like, make sure to go and, like, if you're doing something, make sure that you get the credit for it. Like, I see so many open source projects where people are working hmm.

Jonathan: Yeah, that's interesting. That's some, that's some great, that's actually some great advice. I like that a

Sean: lot.

I was

Dan: just going to say that's really good life advice, actually.

Jonathan: Oh, that's, that's true. That's true. Okay. So there are two questions that we are, we are required to ask every guest. And they're, they're pretty easy unless of course you give us a boneheaded answer, which we will call you out for.

Sean: I doubt

Jonathan: that that will be the, that it will be the case.

So it's, what's your favorite scripting language and text

Sean: editor? Okay. So, the, I'll start with the text editor. I am, you know, I am I absolutely love NeoVim, and it was because when I had started I had, you know, started my first job, and I showed up, and I think I was using the IDE Genie at the time, and I was absolutely bullied.

By all of the like Unix wizards around me and they would stand over me and laugh at me. And so I was like, I just want to fit in. And so I suffered and suffered through them. And then now i've kind of stuck to it and i've become that in you like And so yeah, so every so like everything, you know i've got like vimium installed and everywhere I go now like you're kind of trained to do the The key bindings and then I would say as far as scripting languages so I have a soft spot for PHP because I had contributed to PHP a little bit.

And that was where I got my career started. I can't use Python because. The package management absolutely drives me crazy. And every time I talk to anyone that uses Python, they'll come up with an excuse and tell me, well, there's actually this new thing that fixes everything. And I'm tired of hearing that for the last 20 years.

I don't care if this whole comment section gets mad at me about this, but I can't do it anymore. No, because I would say that is my If anything, I think my bin director is full of just a bunch of bash scripts. And then like every six months or so I get tempted to try out fish because people tell me that it's so great.

And then like I, and then I go to port my like bash RC over. And it's a mess and then I, but the cool thing, I don't even have a bash RC anymore because I switched over to Nick's OS like a year or two ago and it's all this like config driven stuff. So it's just like I don't even have a bash RC like it's just a config file where I say like here's all the different things I want in my bash RC and it does it for me.

That's YAML, isn't it, I think. Yeah, yeah, it's like, it's all YAML.

Dan: It's basically YAML all the way down.

Sean: That's what I realized. It's actually not the Linux kernel anymore, it's just a bunch of YAML config files and somehow they've got it to work. Yeah. So yeah, no, I would say that's So that would be the, and hopefully I've started a few flame wars in your chat.

Jonathan: Not too bad yet. We'll see what happens when we, when we, when we go live on Hackaday and the comment section there, that's when it, that's what it sometimes gets interesting.

Sean: All right. Yes. Oh, I was going to say, and then the, the I was at a company that Everything was written in common Lisp.

And so I was also forced to use Emacs for two years. And it was super cool because they would run these long running Lisp processes and you connect to them via Slime. So when like, the process would crash, you could actually go in and you could jump to that thread and you could walk through an interactive debugger.

And like debug the thing. It was absolutely magical. And I was like sold on it. And I was like writing scheme for all my scripting and stuff like that. But then like it was totally a cult. Like once I left that company, I realized I had been completely brainwashed. It's time to go back. Get off the Kool Aid.

Yeah, get off. Get off the parens.

Jonathan: Sean, it has been an absolute delight to have you with us today, and I anticipate that once things really start landing in OBS, we'll have to have you back and I don't know, maybe do some, some live demos, actually eat the dog food for, for the, the interview that time. Thank you so much

Sean: for being here.

Thank you so much. And we do prerecorded to make it as good as possible. No, I'm excited. Yep.

Jonathan: Sounds great. All right. So Dan, have we have we converted you into using WebRTC and OBS whip and web and all of that? Are you as excited as

Sean: I am? Yeah,

Dan: definitely. I mean, I already use OBS quite a lot for various things.

I don't actually tend, so I tend to be making stuff and then streaming it out rather than interacting with people. So I haven't done a lot of the WebRTC kind of stuff as far as interaction stuff goes, but it sounds amazing. And I think, yeah, definitely, I'm definitely sold on the idea. And the idea that we'll be running each running OBS and just connecting.

Directly almost in, I don't know, hopefully the near future, not to put any pressure on, on any. Yeah, that hopefully that will, that, that sounds amazing. I can't wait for that.

Jonathan: Yeah, I I, we will definitely have to play with that. I see a lot of potential there. Yeah, great, great conversation. Great guest.

Dan, do you have anything that you want to plug before we let everybody go? Yeah,

Dan: sure. I mean, people can head to my website, which is danlynch. org. I actually blogged last week about about Floss Weekly and the great job that you're doing in reviving it and not reviving it's the wrong word and continuing it.

And so I, I hopefully could drive a few people this way and I will keep trying to do the same. So if you go to danlynch. org, you can find various podcasts and blogs and music and silly things there. So go and enjoy those. Yeah.

Jonathan: There are two bits of credit that I want to give. We do appreciate, of course, Hackaday.

Continuing the show here. And I also appreciate Twit. They, they put one final episode on the RSS feeds. Letting folks know that, that Floss Weekly continues here. That was not something they had to do. That was just in my opinion, that was a classy move. And so, thank you so much to Leo and Lisa for doing that.

You can, of course, follow all of my stuff. Most of it is at Hackaday. We've got the security column that goes live every Friday. And then if you want more, there is the Untitled Linux Show, and that is a Twit Plus, the Club Twit exclusive, so you can follow me there. Go get on Club Twit and Yeah, be part of that conversation.

So, we have coming up next week a brand new co host, actually. We're gonna introduce Jeff as a co host and get him into the rotation from time to time. But we're also talking with Jan Piet Menz about OwnTracks, which is sort of a replacement for Google Maps, mainly the timeline feature of Google Maps.

Want to be able to track where you or other people are going without uploading all of your location data to Google or, or whatever service you're using, you might be interested in own tracks. So looking forward to that and just want to say thank you. We had a great live audience today. Took a question or two from there.

Thank you guys for for joining us. And then thank you everyone on the download that, that listens. We sure do appreciate it. And Hey, we will see you next week on Floss Weekly.

Episode 765 Transcript

9 januari 2024 | min

[00:00:00] Jonathan: This is Floss Weekly, episode 765, recorded Tuesday, January 9th. That ship sailed, and sank. Hey, I'm Jonathan Bennett. This week, Aaron Newcomb joins me, and we talk with the OG himself. Randal Schwartz joins us. We talk about Dart, we talk about Flutter, we talk about what Randal's been up to since leaving Floss Weekly years ago.

[00:00:22] Jonathan: It's a great show, you don't want to miss it. good morning. It is, it's Tuesday, not Wednesday, but that's okay. It's time for Floss Weekly. It's a show about free, libre, open source software. I'm your host, Jonathan Bennett, but hey, it is not just me. we've also got Aaron Newcomb,the wonderful, the great Aaron Newcomb of, boy, of a lot of things, of Floss Weekly for the longest time, one of our co hosts.

[00:00:46] Jonathan: He hasn't been here for a while, but we're going to ask him about that. Also, the Retro Hack Shack and some other things going on. Darren, it's good to have you

[00:00:52] Aaron: back. Yeah. Thanks for having me. Nice to be here in the new

[00:00:56] digs,

[00:00:56] Jonathan: so to speak. Yes. the new studio, the central studio. You haven't been here for a while.

[00:01:03] Jonathan: What's the deal?

[00:01:05] Aaron: Yeah, so I actually, for folks that don't know I work for a startup and so that keeps my life pretty busy and Wednesday was just not a good day for me. So I actually Took a break basically. I don't know what that was four months ago five months ago something like that I said look guys, I feel really bad keep saying no, I can't co host this week.

[00:01:26] Aaron: So Look, let's just Until we can find a different day or until my schedule frees up, let's just not have the stress of me feeling guilty about not being able to come on. so yeah, that worked out, but hey, today's Tuesday, and Ironically, this is the only time slot in my calendar that I had available, but it worked out perfectly, so that's

[00:01:46] Jonathan: great.

[00:01:46] Jonathan: Yes, Aaron is now our designated, when we don't record on Wednesdays. Guy, he's the one I'm going to call first, try to get him into the rotation. so we've got a, we've got a super special guest today. I'm super excited to have Randal Schwartz back. the man, the myth, the legend, goodness.

[00:02:03] Jonathan: He also, so Randal is all the things I know him of a Perl guy, the Perl guy, maybe wrote, literally wrote the book on Perl is now the dart flutter guy. he has been a computer security martyr. Literally throughout the years. Hopefully we can chat about that real quick. he was the floss weekly guy for the longest time.

[00:02:27] Jonathan: we'll talk about that too. Super glad to have him back. Randal Schwartz, welcome back to the show.

[00:02:35] Randal: Hey, thank you. Thank you for having me back on floss weekly. I think the only other time I've been on floss weekly as a guest was floss weekly number nine, which is my interview back. I think it's 20 years ago or something close to

[00:02:51] Jonathan: that.

[00:02:52] Jonathan: It's been a long time. I remember there was one other episode. I think a guest flaked out on us and we were trying to scramble. What are we going to do? What are we going to do? And we ended up interviewing you as the show that would have been that would have been somewhere around the corner. Four or five hundred in there.

[00:03:06] Jonathan: I didn't take time to go back and look it up.

[00:03:08] Randal: Yeah, that would be probably like three 50, 400, something like that. Yeah. Yeah. Somewhere back. I don't remember. And it, after so many shows, it just got to the point where people would ask me, have you ever had X on? And I said, I don't know. Let me Google and find out because it was just a long blur of.

[00:03:26] Randal: really wonderful people on this show, and I'm so happy, Jonathan, that you've taken on the task of continuing it on and maintaining the legacy, because that's actually, special to me that you've done that, because,this show has had such an amazing legacy, and I've got to say, just tooting my own horn a little bit, that it also had an impact on the industry in that there were projects that were fledgling projects that we brought onto the show over the years that became mainstream as a result of having publicity from this show.

[00:04:01] Randal: Oh sure. And so I'm glad you're continuing that tradition, by having the show continue on because there's always going to be, especially if you mention AI at all, there's going to be a huge number of new things going on continually in the open source arena that are going to need a platform to be able to get a little more exposure, perhaps, and I'm sure you're Doing all the appropriate research to try to find all those out.

[00:04:25] Randal: And, or hopefully you get the word out there enough. Some people can just come find you, which is how that's the longest sentence I've said in a while. So I'll just stop there.

[00:04:33] Aaron: Hi,

[00:04:38] Randal: thanks for having me on the show.

[00:04:40] Jonathan: Yes. Yeah. it has been, oh, it's been a long time since I've talked to you personally too and, so Randal, Randal reached out to me when the news broke that Twit was unable to, to keep hosting Floss Weekly, Randal reached out to me and said, Hey, what can we do to try to save this?

[00:04:55] Jonathan: I said, man, I'm way ahead of you. I'm already talking with Hackaday. Who is our new, our new sponsor, our new home. And,I think we had just made that deal happen when you reached out to me. but I thought this would be a perfect opportunity to have Randal back. Have some, get some OG cred with the OG Floss Weekly man.

[00:05:12] Jonathan: what have you been up to for the past few years? Most of our listeners, I imagine, haven't heard much from you. Not to say that you haven't been doing things. What's been keeping you busy?

[00:05:21] Randal: first off, I, leaving Floss was a good thing for me. I was starting to get a little burned out, but not willing to admit it.

[00:05:29] Randal: And some of the shows were suffering in quality as a result, and I appreciate that I got to leave on really great terms. Oh, absolutely. Because I got five hours a week back. Most people don't realize how much time you put into a show like this, even though we're only on the air for an hour. And it was extensive, and I'm glad I got that time back because I had just also started transitioning to a new career.

[00:05:55] Randal: as you said, I'm the Perl guy. I'm going to put myself just at number two. Obviously Larry Wall is number one, but, but I wrote more books and had more bylines and more publications about Perl than anybody else did. I had ran the number one training company during the dot com boom. Stonehenge was everywhere.

[00:06:14] Randal: and I miss those days in a lot of ways, but I also don't miss the hectic pacing of it. From, mid nineties to the, when COVID hit, I was on the road 40 weeks a year. Wow. it was quite a lot of time away from home, focusing entirely on my career, but getting to meet some of the most wonderful people all around the world.

[00:06:36] Randal: the 688 sea days on cruises as part of my job. I got to do some really wonderful things during that period, but I wanted a transition. I said, now that I'm bored with Perl and nothing's really happening new with Perl, I've mastered it. I can do it in my sleep. What next? And it just happened that Dart had come along three or four years earlier.

[00:06:58] Randal: I really enjoyed Dart because it reminded me of my Smalltalk days. And in fact, there's no secret to that, that was partially because some of the early design team for Dart were actually people that were working on Smalltalk back in the day. And so Dart felt like a successful small talk all of a sudden, and I wanted to start doing some more with that and maybe do the same with that I had done with Perl, essentially put Perl on the map and become one of its biggest cheerleaders.

[00:07:21] Randal: So I wanted to do the same for Dart and then, but then Flutter came along and I went, Oh, this is even better because I can write mobile apps. The day I discovered Flutter, I went home. And I installed Flutter. And, I had an icon on the front of my phone that wasn't there, earlier and didn't come from the store.

[00:07:39] Randal: And I thought, this is amazing. I never really considered doing mobile development, but all of a sudden now I did that. I then went from there to becoming a Google developer expert, which requires a careful vetting. there are, just to give you an idea, there's only 10 Dart and Flutter Google developer experts in the U.

[00:07:58] Randal: S., 13 in North America, about, 75 around the world, maybe 90 somewhere in there. Around the world. So I'm part of an elite team and we're essentially unpaid dev rel. So we sit between Google and the world and try to help people get things to happen. So it's, that's been a lot of fun. So I've been doing a lot of that and that involves.

[00:08:20] Randal: going to physical conferences when I can, although there's a story about that, that I'll get to in a minute. yeah, I go to physical conferences, even appearing locally in my Portland chapter of the Google Developer Group. which I'm going to be doing actually in March coming up, I'm going to be giving a Perl, or a dart and flutter lecture there, but I've mostly been doing virtual stuff.

[00:08:41] Randal: So I've been, I've been all over the world, literally at Google developer groups around the world doing virtual presentations. Like just a few weeks ago, I did one in Estonia. So it's this is really cool. They worked out the time of day so that I wouldn't be, have to get up in the middle of my night to present there.

[00:08:58] Randal: And that's what I've been doing as a, and as a Google developer expert, I spend a lot of time on the discord and the slack and reddit and stack overflow answering questions for people, which I really enjoy doing. It's what I was doing for Perl for decades. So I really enjoy doing that. And I'm fitting in the.

[00:09:17] Randal: The stretch of it, and I'm getting work from time to time. I'm not as much as I really want, but at least I'm paying some of the bills with my, dart and flutter income. I think people think I'm too expensive because I am highly credentialed. Here's a 45 year veteran of the industry and he's one of the.

[00:09:36] Randal: Top 10 people in the U. S. He's probably going to charge 300 bucks an hour. No, it's not that much. Oh, wow. I am affordable.

[00:09:45] Randal: just ask. You wouldn't believe how affordable I am right now.

[00:09:48] Aaron: I definitely want to ask, in a minute, I want to ask a few more questions about Floss Weekly, but I want to stick with this topic for now. So I'm just curious, Just generally, since I don't know as much about Flutter and specifically, you outlined it a little bit, but what do you find in these modern days?

[00:10:04] Aaron: what kind of problems are people solving? what, why would I go to Dart or Flutter? what kind of problem would I trying to be solved that I wouldn't be able to solve with another language? Because I've never quite understood why people are gravitating besides personal preference.

[00:10:18] Randal: Sure.

[00:10:19] Randal: So the key thing here is that Flutter is pretty much driving Dart. Flutter is the, a UI toolkit that originally was targeted for iOS and Android, but now has gone to every platform that has bits on it. So for example, the current Ubuntu installer is written in Flutter. So they're migrating from GTK and there are other things And all that stuff they're migrating from that to flutter now.

[00:10:48] Randal: So it's becoming it's so think of a good analogy in the, from the Linux world, yeah. GTK was like the universal way to put bits on screen. This, Flutter is a universal way to put bits on screen in orchestrated patterns. And so that's what it's at its core. And because of that it runs on iOS, Android, web,all the desktops, Linux, Mac, and Windows, it'll run on a Raspberry Pi, it'll run everywhere.

[00:11:16] Randal: And so you end up with this sort of universal solution for write once, run everywhere, similar to the GTK promise, but, but instead of using.

[00:11:27] Aaron: Promise.

[00:11:30] Randal: this one actually pretty much delivers, except for platform specific dependencies. Pretty much the core code at the graphics level stays exactly the same for all those.

[00:11:39] Randal: But it's written in a language that Google had also been developing in parallel to be a replacement for JavaScript. And it's called Dart. And Dart started out very simply just being a slightly smarter JavaScript. Now this was before TypeScript. So this has been in plan for about 11 years now.

[00:11:57] Randal: Been in process for 11 years. And as they grew up with it, and then Flutter came along and said, We want to use Dart because we want to be able to target and have that. They chose Dart because Google was developing both Flutter and Dart and that they could tell the Dart people what we need is both for it to be easy to update while we're debugging it, so have a JIT kind of compiler, but also when we push a button and want to drop it onto a mobile device, we want it to be highly efficient ARM code.

[00:12:34] Randal: And so there is the compiler is one of the interesting ones that has both a JIT mode and an AOT advanced ahead of time compiler for Dart. And it also targets JavaScript. So that meant that you from one. Dart program that was using the Flutter framework or not, you can target pretty much every platform and do it either in a JIT mode for doing debugging or just quick running of a script, for example, or you can do it AOT mode, where it's actually compiling to an XE.

[00:13:08] Randal: for example, I can, I can say, Dart compile foo. dart XE, and I get a binary that's standalone, similar to how you would do Go. Where go doesn't have any libraries. This Dart application does not have any libraries as everything baked into it. So you can drop it on anything. It's a similar architecture across.

[00:13:28] Randal: So there's a lot of advantages to Dart. There's a lot of advantages to Flutter. also Dart has been getting modern features added to it. Very strong typing generics, deferred, loading for web, just added. Pattern matching and records. So you've got tuples, you've got, the pattern matching from Scala and stuff.

[00:13:48] Randal: So it's been, it's been quite extensive as it's been moving forward, but slow enough that they're not just throwing in the kitchen sink. Every new feature is being very carefully specified, weighed out. By an incredible team of people and dart and flutter was, although it's originally a Google product, has now been seeing 40 to 50 percent of its commits in every release come from non Googlers.

[00:14:16] Randal: So this is a community effort already. That's

[00:14:19] Aaron: great. That's great. That's always a big challenge for any open source project is yeah, you know You start out a group of people getting together, you know Maybe one person and then they bring on a couple people but then you know really getting the community involved You know,is the struggle and that's what you want.

[00:14:36] Aaron: In fact, organizations like, Linux foundation and CNCF, which are open source project is a part of, there's requirements there. Like they're not even going to advance your project until they see that it's not just one company trying to. put together a project and run it on their own.

[00:14:51] Aaron: They want to see community involvement. And it's so hard. 50 percent is really

[00:14:56] Randal: good. really, it's huge. And it's funny because people still ask me, Google's abandoned things before, aren't you worried about this? And I go, when 50 percent of the people or commits are coming from outside the house, It's pretty sure that if Google decided to abandon it, it would be taken over by a foundation of some kind.

[00:15:15] Randal: And Google's fully up front about that. in fact, they already did it with a part of this. There was a project called Angular Dart, which is similar to Angular TS, which they also run, but they were trying to do the same thing with Dart. And it was. cool, and I actually started playing with it, started developing websites with it and stuff.

[00:15:34] Randal: And in fact, that's what got me into actually looking at Dart and looking at writing books about Dart was, in fact, the AngularDart project. But then Google announced, Yes, we're using it intensely internally, but we don't want to have external support anymore. We don't want to support it for issues that are coming in that aren't from internal.

[00:15:50] Randal: We want to prioritize internal. So what they did is they let the project be forked. So there now is a community version of AngularDart. That has, forked and is now straying from the internal version. But the fact that Google just said, yeah, go ahead. We'll let you own all the thing up to here and still own the name angular dart and move forward with it.

[00:16:12] Randal: So I believe the same thing would happen if anybody, if Google. ever decided to say, I don't, we don't want to do Dart and Flutter anymore. Now to do that, of course, we'll be cutting their own throat because there is a lot of people working on Dart and Flutter internally, there are huge projects inside Google that are using Dart and Flutter extensively.

[00:16:33] Randal: For example, the, AdSense and AdWords, that's where Google makes their money. That is all managed by. AngularDart, internal Google AngularDart apps now. And so there's no way this is going away internally. I know from talking too many team members and stuff. There it's incredibly invested.

[00:16:51] Yeah,

[00:16:52] Aaron: it's interesting because, I think I'm going to put out a supposition and you guys can agree or disagree. but I think, Google has canceled so much stuff over the years. But my feeling is that's more on the commercial side. My feeling is that. their ethos is, Hey, let's throw something out there on the commercial side, see if it makes money, right?

[00:17:11] Aaron: And if it doesn't, Google Plus, we're going to get rid of it, right? So we'll give it a few years. If it doesn't make money, we're going to get, we're going to ban it and get rid of it because that's part of the business model. But on the open source side, I feel like they have a really good track record looking back at things like,you could name a ton of them, right?

[00:17:26] Aaron: But Kubernetes is the one that comes to mind. Kubernetes was started at Google. Now you don't stand up a cloud ecosystem without Kubernetes. that would be ridiculous if you have more than one host or more than one application you got to run. I think they have actually a really good, I understand the trepidation, but I think they have a really good track record with, open source projects.

[00:17:45] Randal: yeah. And as we look at the, any of the trending things like, TIOBE and other places, and TIOBE is not a great place to look, but,stack overflow queries, things like that,Flutter. By, passed up React Native, about a year or two ago. And that was really interesting when you started seeing numbers tilt in that behavior, because prior to that, of course, everybody said, React Native's already got the entire iron roads, but, Flutter is I'm better product I've got to say

[00:18:14] Jonathan: so I'm curious and this is a little bit of a troll question But if somebody loved the idea of flutter, but held their nose at dart could you do something like write in c and make flutter calls?

[00:18:26] Randal: There is a Closure based Dart, if you want to go way wacky functional programming as your core. So yes, it's possible.

[00:18:37] Jonathan: Very cool. That's fun. Closure based Flutter, did I

[00:18:41] Randal: say that

[00:18:41] Jonathan: right? Closure Flutter. Closure Flutter, okay, I wondered. Yeah. Alright, so we could talk about Flutter and Dart, but I'm curious about, what else you've been up to, Randal.

[00:18:49] Jonathan: Now there's a, there's a little birdie that has told me that you're actually getting around with a walker. Right now and hoping to transition to a cane or crutches, and I'm just over here. It's there. We believe it doesn't go

[00:19:04] Randal: far because I can't go far without it. What's so about two?

[00:19:07] Randal: Yeah, how about two years ago? I had an apartment in Tijuana for eight years. Actually, I did many episodes for the Tijuana kitchen, right? Yes. Okay. just to show that yes, in fact, I lived there for eight years. And so I had been there on my 50th birthday and on my 55th birthday. And although I no longer have the apartment there, thanks to COVID, I basically killed my lease.

[00:19:29] Randal: And so that's it. I'm going to just visit and stay in hotels when I go there. for my 60th birthday, my friend still was in the other half of that apartment. And, I came down to see him and to celebrate with all my friends for my 60th birthday. And I'm old enough that when I slip and fall in the shower, I break my hip.

[00:19:51] Randal: So I was in Tijuana. I got rushed to a Tijuana emergency room. I spent the evening of my 60th birthday in a Tijuana hospital in pain. luckily with a fair amount of drugs, so I don't remember much of this, they put a device in a few days, I think the next day or maybe two days later, I think it was next day.

[00:20:12] Randal: And, the device, apparently it was worse than the, x rays proved it to be. And when they got in there, they said they took them a lot of extra work and, to skip forward two years after that. the, device they put in had four screws into one of the bones and three of the four screws had broken.

[00:20:34] Randal: Oof. So I was in chronic pain for the last six months walking with a cane just to be able to walk through the room and it would take me about 20 seconds to stand up and start walking every time. Thankfully, I finally got on the Oregon Health plan. And so they looked at it, they did cat scan. They said, that has to come out, that's got to come out.

[00:20:54] Randal: And I said, but I'm part Mexican now. They go,no. You're going to have to lose that part of your nationality at this point. And so they, they did a complete hit replacement. this happened, about, about three weeks ago, happened three weeks ago, just a little over three weeks ago. And, I'm, walking around with a walker. I'm probably going to do that for another couple months or maybe a walker for the next month or so and then transition to a cane. But hopefully this is a good enough repair that I'll be walking just normally in about six months or so. And I look forward to that because this is funny because just as we came off COVID.

[00:21:32] Randal: All of a sudden, I'm being invited to attend, Google events all over the world. South Africa, Europe, everywhere. But I can't stand longer than a minute or longer than five minutes or walk more than a block. And y'all know conferences can't be done if you can't do those. And, at least in the airport, I could get wheelchair to get directly to my plane and back.

[00:21:57] Randal: But that's not the whole conference. That's not the way the conference would work. And I've basically been only be able to do virtual conferences for the last amount of time and,in my peak of being at GDE. So it's I want to go, I want to go present, I want to go be places.

[00:22:13] Randal: And I'm going to give it about another six months, but if I can start just walking a block without any kind of aid, I'm definitely going to be hitting the road. Probably not 40 weeks a year, but maybe, five or six events a year, which would be nice. It'd be nice to get back out

[00:22:27] Jonathan: on the road again.

[00:22:28] Jonathan: Yeah, that'll be neat. Although, I have to say, it's amazing that we can do virtual conferences and virtual appearances. That is, Yeah. That's really a godsend for, especially folks that are in a situation like that where you can't get out to them. Yeah, I'm

[00:22:41] Randal: glad I, that's, I am glad this happened in the middle of COVID instead of being,like the last couple of years of me doing all the Perl work that I was doing.

[00:22:48] Randal: So yeah, or the cruising work. Yeah. Yeah. And I'm not, I haven't been on a cruise in five years. Cause I don't want to go back on the giant Petri dishes now. I'm going to avoid that for a while, even though I have 688 days and I'm a fifth star Mariner and I've had 112 cruises under my belt.

[00:23:04] Randal: I think I'm done with those. So that I may do one for my birthday coming up in the future. We'll see.

[00:23:08] Jonathan: There you go. I'm curious. Do you do any Perl work anymore?

[00:23:12] Randal: very little. And in fact, when I come in and I see some Perl program that's been used as a utility as part of my ongoing education, I've been rewriting those in Dart, so I'm turning all my Perl programs into compiled Dart scripts.

[00:23:28] Randal: And, so much nicer, actually, there's a Dart. until you have to process a string, Dart inherited Java's need to have a regex object that you spell out regex, and it's oh, going forward. from Perl to that. I can't, I just ate slash food slash. Come on. No, you can't do that. So there's this incredibly awkward.

[00:23:51] Randal: So I've occasionally typed deft R equals regions.

[00:23:58] Randal: This thing

[00:23:59] Jonathan: there, you were. Yeah, so I, real, real quick. There was this kind of, mythological post from back in, in yesteryear where someone was saying, oh, you, you can't just assume that Perl is gonna be on every system. And,sure you can. now, yeah, now you can. that was probably 30 years ago, or 20 years ago.

[00:24:17] Jonathan: Yeah. somebody wrote that. Now he, people look back on it and go, if only he had known,is, I think that was me that

[00:24:22] Aaron: wrote that . Wow. No, it wasn't. But seriously, I used to work. early in my career, 20 plus years ago, right? I worked for a very large company and it was, the enterprise applications were all Unix running on Unix systems.

[00:24:35] Aaron: And, I just assumed that pro would be there, but what I didn't assume is that some of these. systems would have different versions of Perl. so Perl 4 instead of Perl 5 at that time. And I was like, Oh, why isn't my script failing? And we were pushing the script out to thousands and thousands of hosts in the data center and it was failing.

[00:24:53] Aaron: Randal, when you were talking about the ease of being able to compile binaries for different types of systems, I used to have to use, I can't remember the application it was, but there was a Perl thing that would take your Perl script and encapsulate everything and turn it into a binary. I can't remember what the name of it

[00:25:08] Randal: was.

[00:25:08] Randal: Yeah, I know what you're talking about, and I don't remember it either. not bundling. What the hell was that

[00:25:14] Aaron: thing? I can't remember, but anyway, that's what I would do. So I resorted to that, and I would send out a binary to all of these hosts, just so that it would run correctly. And now here we are, 20 plus years later, and we finally have, a language and an ecosystem that makes it easy to do that.

[00:25:27] Aaron: And it's oh yeah, that makes total sense. Why didn't we think of that

[00:25:30] Jonathan: before? That's the direction I was going to go. is there a problem with Dart, with it being available on, older systems? Is somebody still running CentOS 7 or even, God help us, 6? can you still deploy Dart on it?

[00:25:41] Jonathan: is it there? Is it everywhere?

[00:25:43] Randal: I'm pretty sure it's pretty compatible with, I don't know what the minimum Linux version is at this point, but I'm pretty sure it's still available all the way back. there's still supporting. Android like four generations back. it's, and in fact, it's funny because when it first came out, material design, was not in Android quite yet, but you could build an app with Flutter a bit later, which understood material design and have it run looking like material design on a system that didn't have that in the library yet.

[00:26:17] Randal: Really amazing stuff, because again, once it went, one of the ways Flutter makes it easy to move from platform to platform is that it's essentially,it owns every pixel, so it's painting the entire screen. So when you see, an Apple widget or a material,drop button, those have been written in Dart so that they're portable.

[00:26:39] Randal: So you could have an app on the desktop look like it's running on Windows 95 because somebody came up with a Windows 95 design kit for it, Just for grins, I've made a phone app that was Windows 95 dropdowns

[00:26:55] Jonathan: and stuff. Oh, that's great.

[00:26:57] Randal: Or Ubuntu now, because there's an Ubuntu design kit, there's a Windows design, Windows, modern design kit, there's also the Windows 95 one that somebody did as a joke.

[00:27:05] Randal: But yeah, you can basically, since you're controlling every pixel, it's, it's easy to do. Yeah, just painting things.

[00:27:11] Jonathan: Yeah. So I do want to cover real quick. what is it that you've got on Wednesdays? Why are we here on a Tuesday and what's going on with that?

[00:27:19] Randal: I guess I missed doing a show on Wednesdays, for about six months before I got involved.

[00:27:27] Randal: a group of people,Simon, Lightfoot based out of the UK and, Scott Stoll based in, Midwest somewhere, had been doing a Wednesday hump day Q and a open session on their zoom channel. And so they would invite people to come over and just hang out with them and get questions answered and stuff.

[00:27:48] Randal: And they decided to turn that a bit and make it a formal YouTube channel. this net show has now been running for a couple of years now. I got involved. When they decided to take a brand new Flutter book called Flutter Apprentice and serialize it a bit and actually make it a show. So that was done in cooperation with the publisher of the book and with Google.

[00:28:11] Randal: Google was paying for this as well to help sponsor to get more people doing Flutter. This is two years ago. And I got involved being in every week's show as the backstage coordinator and answering questions as people are asking questions and stuff. And they just said, let's just keep going on Wednesdays in this public slot.

[00:28:31] Randal: And, do you want to hang out and, help manage the show? And I said, sure. So I've been doing that ongoing for. I think about two years now, and that show gets about a thousand downloads. So it's nice. Yeah. it gets me a little,a little bit of visibility. And because of our connections, because of my connection to GDE and Simon's just been inside the rim so long.

[00:28:51] Randal: Yeah. That he just has lots of people on the team that he knows and stuff. We've had often had Googlers on the show, to be special guests because they can answer questions directly, especially as things migrate quickly in the Dart and Flutter world. I like to say that, any book or,YouTube video on Dart or Flutter that is older than a year is already out of date.

[00:29:13] Randal: Sure. Because Dart and Flutter have been moving so rapidly forward that, and, Dart, I like to call it a 45 degree course change. So not really like a hard left or a hard right, but there's things like when they added null safety, when they added strong typing, when they added patterns and got rid of backwards compatibility, there are some key moments where the code you're writing now actually is not.

[00:29:39] Randal: In any moment, backward compatible at this point. And, Flutter has done the same things. There have been breaking changes over the years in good directions, of course. But every time you do a breaking change, you're going to upset some people. And it has happened. And and so it's the same thing.

[00:29:57] Randal: We get people on to talk about something. And then a year later we get them on to say, no, it's different. It's different now.

[00:30:04] Jonathan: Oh yeah. I just rewrote some code because one of the, one of the Linux core libraries, the GPIO handling, they went to a version two and I didn't hear anything about it. And I'm, looking at their new API going, Oh, it was so much easier the way that it was written.

[00:30:20] Aaron: No, it's, I had to update a bunch of Python scripts. I was still running Python 2 to

[00:30:24] Randal: 3.

[00:30:26] Aaron: Finally, something forced me. I'm like, ah, but I want to do this thing, but they retired that library. I can't make it work. So I went ahead and updated, spent an afternoon updating a bunch of Python 2 scripts to Python 3.

[00:30:37] Aaron: Yeah. a pain, but, and I know there's automatic things you can do, but it's just, it's interesting at the different, and you've seen this Randal, right? The different, the life cycle of a language, right? Because I'm sure you remember what a big deal it was with Perl,going from four to five and five to six and

[00:30:53] Randal: whatever it was.

[00:30:54] Randal: But you don't go from five to six. that's the thing. That ship sailed empty.

[00:30:59] Aaron: Yeah. But remember that was just a huge deal, right? and. No, now we're

[00:31:04] Randal: talking about ships sailed and sank.

[00:31:09] Jonathan: There is no Perl. Didn't even get out of the

[00:31:11] Randal: port, right? There is no Perl six. Yeah. Oh, Perl six is still being used by, but it's only at the academic level.

[00:31:19] Randal: I'm going to get yelled at. Dozens of

[00:31:22] Jonathan: people are working with Perl six,

[00:31:24] Randal: hundreds of people are playing with Perl six. Yes. It's

[00:31:28] Aaron: a different thing, right? back, considered what was going on back then. And then you're talking about, Oh yeah. Breaking changes every couple of months or so, it's it's just a different

[00:31:38] Randal: world.

[00:31:38] Randal: Here's the thing I like about dart. It is. Almost everything I would have wanted in Perl 6, in terms of streams and asynchronous and, generics and,just complex data structures. I'm doing functional programming by just adding a layer on top of A dart. . that's, this is all the stuff I would've wanted in Perl six, and I can build a binary today.

[00:32:02] Randal: Yeah. , it's done. Yeah, exactly. And it works and it's well specified. Specification can put you to sleep. It's about, 500 pages long, but there is a spec for Dart and it's there and it's complete. I really,

[00:32:14] Jonathan: I imagine that ability to build binaries takes out most of the pain of all of those breaking changes in the language, doesn't it?

[00:32:21] Randal: I imagine it would. Yeah. But,any project though, has the ability to lock down versions so you can pin. And in fact, the basic design is that in your pubspec. yaml, which is the file that's controlling, which versions and what, libraries you're using, recommends that you use, Upro syntax, like upro 1.

[00:32:42] Randal: 4, and that says 1. 4 or anything. It's not a breaking change forward from there. So all the way up to 1. 999, then 2. 0 would mean, I would yell at me if I tried to compile it again. Sure. theory that works, or you can lock it down a specific version too. and, as you build that, there's a pubspec.

[00:32:59] Randal: lock that you can check into to, get that says, this is exactly the ones I built it with. And you can use just that without doing an upgrade and see if it still gets the same, you should get the same exact behavior. Because the pub is also set up in such a way, the pub is like the CPAN for Perl. So the pub is set up in such a way that you cannot delete versions from there.

[00:33:20] Randal: So if you put up 1. 2. 3, it's there forever. So if somebody over here builds with 1. 2. 3, they can always build it two years from now, three years from now, with that exact sequence of bytes.

[00:33:31] Jonathan: Has the Flutter community seen any, any malicious packages getting added to that? if it's like CPAN, if it's some of these other Online library distros we've seen over the past few years, typo squatting and all sorts of stuff.

[00:33:48] Randal: Not to my direct knowledge. I have had discussions with sort of the administrators of the pub and the general housekeepers there. And, I remember with the CPAN, we did have, one, senior Perl person who uploaded for fun. A package that said, now you ran this install as root and it could have contained RM dash RF slash, and you didn't look.

[00:34:21] Randal: And but he did that sort of as part of the awareness. There was somebody else I, there was another one. Whose name again, I won't mention just because it's long passed and who cares. . But somebody who uploaded a, phone home in their install script because they wanted statistics of how many people were installing it, which is not something you could generally provide for CPAN in general because it's all distributed.

[00:34:45] Randal: And we didn't have to collect statistics and bring them to a central place. It would've been a real mess. Yeah. And but he put this phone home thing in. He's the only one that has ever received the CPAN ban hammer. We shut him off until he fixed his scripts. And he said, what's the problem with it?

[00:35:01] Randal: Oh, not only did he phone home, he phoned home to grab a text string from a get. And run that eval! He put it in our eval! And he did it because he wanted to have code in that eval that said if you're not running version 1. 2, please upgrade.

[00:35:22] Jonathan: yeah, the banhammer was appropriate for that.

[00:35:25] Randal: We had multiple banhammers coming from multiple directions. And he just, he could not see the worry of his ways. He could not see just how troublesome that was, but that's the only one I recall of the entire C PAN's history that was malicious in that way. and I don't think there's been any examples that at least were made public, or even to inner circles about anything that's similar in the pub.

[00:35:51] Randal: There's been some people that have. Uploaded packages that are pretty darn stupid, but, that's part of the game. if anybody can upload something stupid, people upload things too. it happens. Yeah.

[00:36:02] Jonathan: I boy, just looking at what's happening with the Python community and the node JS community, I can't help, but think that it's coming.

[00:36:09] Jonathan: keep that thought in the back of your mind. It's coming. It's gotta be. there will come a day when, you get that critical mass of the wrong people who realize, Oh, there's this repository of libraries out there. We can go upload crap to it.

[00:36:23] Randal: Ugh. the advantage though is that they,once you've claimed the namespace, other people can't upload to the same library name.

[00:36:32] Randal: So there is an ownership immediately for any given name. there's also the advantage that to download this and use it in your app, you're basically compiling as non root, always. Oh, now that is helpful. And at least you could get to the point where, you could stare at all the source code and go, Okay, this looks not malicious, I get it,

[00:36:54] Randal: and like I said, there are ways, there are administrators for the

[00:36:57] Jonathan: pub. do things added to the pub get reviewed before they're made public? Because what, particularly in,in the node JS repository, what people were doing was, there would be something that would have color in the name and that color would be spelled either with or without a U and they would go grab the same package name.

[00:37:15] Jonathan: But. the inverse. And somebody went to search for this popular package with color in the name and they would spell it the wrong way, here it is not notice that it's only got 100 downloads when your package you're looking for should have a million. So it were there.

[00:37:31] Randal: There, there is a voting system available.

[00:37:34] Randal: There's also a, a score based on, up to 140 point pub points based on, what it has. it does have documentation. Does it have tests? Does it pass the current release? does it have, read me files? Does it have a license file? Things like that. so you can sort by pub. points, you can sort by recency, you can sort by that.

[00:37:56] Randal: So there is some queryable things there that can drive you towards useful

[00:38:01] Jonathan: things. Yeah. Now, one of the, one of the other things that really fascinates me, getting back to your Wednesday podcast, you have Googlers on there. And I recently had this experience where there was a bug that I found was actually in the Raspberry Pi kernel and I was able to go and submit a bug report.

[00:38:18] Jonathan: to the Raspberry Pi guys. And within just a couple of hours, somebody that obviously knew their kernel very well was like, Oh, yeah, I see what the problem is. And you know how to fix out like 45 minutes later. It was amazing. Yes, you see sometimes these projects where the developers are so removed from the people that are using it that it's almost impossible to get that feedback.

[00:38:41] Jonathan: And it sounds like with Dart and with Flutter. Maybe, specifically through you, there's a really good feedback path, and, boy, that's

[00:38:48] Randal: important. No, it's actually much more direct and more obvious once you think about it. They have people that are specifically assigned to at least eight hour shifts of, screening all the incoming issues.

[00:39:03] Randal: And immediately start tagging those, every issue you post to either the dart or the flutter, repos, gets tagged within one business day. Oh, nice. And so that immediately puts it in people's queues and people's and stuff. it's really gets visibility quickly. I remember leading up to the flutter 3.

[00:39:22] Randal: 0 release. I happened to notice, and this is trivial, but I happened to notice. that there was a parameter that was misspelled. It was misspelled consistently. It was spelled wrong both in the parameter name and in its only use down inside the code. You can call both of those X and the whole thing would still work.

[00:39:40] Randal: So that was the point. But the point was it was misspelled. And I went, okay, OCD kicks in. I file an issue on that. And, because I figure, at least I'll record that it If somebody's in there, they might as well fix it. it got tagged good first commit as well as yes, this needs fixing. Somebody came along the next day, a brand new person who'd never committed before.

[00:40:03] Randal: He answered the 10 questions for, can I be a contributor to the dark project? And answer those correctly. And I said, no, this doesn't need any tests. It's just, the tests are already there. It's just, I'm changing the paper parameter. And within two more days, it was in master and it made the cut for dark or for flutter three.

[00:40:23] Randal: So within a week it went from me having the idea. to already being published by a brand new committer outside of Google. That's awesome. Just because the process worked so quickly all the way through the steps.

[00:40:36] Jonathan: Yeah, that's really neat. I, I don't know that I want to name names, but there is another bug report that I filed several weeks ago in an official API.

[00:40:45] Jonathan: And,the only response is, can you show me a code snippet where this is broken? Show the code snippet. Continue to wait for someone to fix it. Large, huge company bought a huge open source tool and we're broken waiting for things to get fixed. good on Google and FlutterDark for working really hard.

[00:41:07] Randal: And that's why I think it's important to understand how much Google has committed to Dart and Flutter internally. And that's part of why I'm happy to be on the external team for that. it does remind me so much of my Perl days. Because, Larry was the source of all this wonderful code coming out.

[00:41:23] Randal: But he,good or bad, he wasn't A, an extremely strong salesman. I had the sales touch to be able to go in and say, let's match this to your task. Oh, let's, oh, that task. Yeah. That would actually be good for Perl too. Let me see. Figure out how to make that work. . And so I was able to explain complex things simply and get, be in the right place at the right time to talk to it about, to, about it to people so that more people could use it.

[00:41:50] Randal: And people have given me that kind of compliment about. what I'm doing for Dart and Flutter. I'm basically being, a cheerleader for Flutter. And I'm happy to be doing that because it benefits me. It helps me to know that at the end of the day, I've solved, helped somebody solve one more problem with Dart and Flutter than we started the day with.

[00:42:08] Randal: it's really nice for that.

[00:42:09] Jonathan: Yeah. there is something, we're getting close to the end of the show, towards the end of the hour. There's something, last time I interviewed you on Floss Weekly years ago, I wanted to ask you about and didn't get a chance to. I, I alluded to it at the beginning of this show, you are a security martyr, aren't you?

[00:42:27] Jonathan: let's real quick touch on this because it's a fascinating story and I've never gotten to ask you about it. what was the deal? why were you convicted for doing a security audit? We don't have much time, that's a long story, I'm sure. First

[00:42:40] Randal: off, the short version is, I've been convicted of three felonies for doing my job with too much enthusiasm, and then ten years later it was expunged, which means legally I can say it's never happened.

[00:42:56] Randal: However, practically speaking, it's on my Wikipedia page. So how am I supposed to deny it? And I've been written up in 20 computer crime books and stuff. And so it's been interesting, over the years. I, how do, how will you do this in a few minutes? But basically I ran crack against a password file.

[00:43:16] Randal: And I did it, I was working for the company at the time, but just not that group. So part of it is group politics. And a bunch of other things, So let me,

[00:43:25] Jonathan: to put a point on it, let me ask you this question. I think maybe it'll give us some direction. what words of wisdom do you have for burgeoning security people to say, here's your, your guiding principle to stay out of trouble?

[00:43:39] Randal: There isn't any blanket one, and that's the problem. Anybody can after the fact go, but we didn't tell him he could do that. And as when you investigate a bug, sometimes you go off on a tangent because some other part's not quite working. But that wasn't on the original issue ticket. But you're over there because it's revealing something.

[00:44:00] Randal: That's where I was. I was seeing a problem and wondering how deep it was. And first off, that wasn't directly in my charter. And so that was already a problem. So the first thing is, yes, make sure your charter is clear when you're doing security, make sure your boss and you know exactly what you're doing all the time.

[00:44:21] Randal: and as much as attractive as it might be to go off on a corner, stop yourself and go, okay, I go, I need to go get some extra proofs over here to make the, do that. I don't know that it's going to solve it for you though, because we live in an age now where anybody can come back and make anything look like anything later.

[00:44:40] Randal: And that's part of the problem. And also there are some very bad laws. In most states in the U. S., that make it a felony for altering a computer without authorization with neither, with the computer being as broadly defined as it possibly can be, things with electronics, authorized vaguely defined, if at all, and, and altered also vaguely defined.

[00:45:09] Randal: One reading of this rule. is that if you visit a website in Oregon, but you didn't have explicit permission from the owner of that website. Yep. Then you have made a log entry in their weblog that they didn't authorize. That's altering a computer without authorization, class C felony, 100, 000, five years in jail.

[00:45:31] Randal: Yeah. And some,

[00:45:32] Jonathan: some laws go even further or even more onerous and they will say accessing.

[00:45:36] Randal: Yes. Yeah. That was, that's a misdemeanor in Oregon and I got nailed on one of those too. So yeah. Goodness. Yeah. All right. yeah. Be careful. The laws are not on our side. Yes. They are not written well because they're written by people who don't care.

[00:45:50] Jonathan: Don't care or don't understand. And not everyone that is in the position of authority. That's right. are particularly, on your side. So anyway, we could do a whole episode on that. Maybe we will, Maybe. I would love to get some security,Randal is a security person, obviously, but, Yeah.

[00:46:06] Jonathan: There's some other security folks that I would love to get on the show and probably dive deeper into that in the future.

[00:46:11] Aaron: and yet, Someone from the EFF

[00:46:12] Randal: or something. and yet my luggage is 1, 2, 3, 4, 5, 6. yeah. Keep that in mind.

[00:46:19] Jonathan: All right. I know Aaron has one more question he wants to get in.

[00:46:22] Jonathan: He put it in the back chat. So go ahead and ask that. Okay. Yeah.

[00:46:25] Aaron: I just want to, it's been fun talking about what's going on now, but I do want to go back since we're here and we have a minute, Randal with you. we've talked about this before, but there's so many floss weekly episodes, right?

[00:46:34] Aaron: Yeah. Any that stick out like top three. I know what mine would be, but I'm curious what your top three Floss Weekly episodes might be. I

[00:46:44] Randal: still chuckle, only because I came up with a joke right at the last second. When we interviewed the guy Who is solely responsible for NTP, which is one of the most critical services that run on the net that nobody talks about except sysadmins.

[00:47:01] Randal: And it's so important to everyday life, but just sysadmins are the ones that talk about it. I remember the whole point of NTP is basically my machine asks your machine what time it has and then puts timestamps on both ends so that it can do the round trip and figure out from the round trip where the offsets are and stuff like that.

[00:47:18] Randal: So it's basically I ask you, you tell me back. At the end of the show, I asked, oh, he's going to kill me for not remembering his name. I asked the guy, what time is it? And he told me, and I went, Oh, okay, I guess we're in sync,

[00:47:37] Randal: effectively the human version of NTP.

[00:47:42] Jonathan: Let's see, that was my

[00:47:43] Randal: favorite episode, I think just in terms of joke value. But of course we did some amazingly pivotal ones for salt. We put salt on the map, you, the only reason we still even talk about salt today is because they were on our show.

[00:47:55] Randal: and, we did a couple of other projects that were just like that. I actually want to go back over and if I ever had a ton of spare time, which I never will. I want to go back and actually like at least read the transcripts of all the old shows and see how many times we accurately predicted where they were going to go.

[00:48:15] Randal: And how many times we got that so terribly wrong or they themselves got that so terribly wrong because it would be a useful lesson to be able to tell fledgling, new floss projects. here's the advice of the sages from 20 years. This is what you really ought to pay attention to, and this is what you really don't need to care about.

[00:48:38] Randal: that would be, because there's a wealth of information there that's all locked up. Maybe I can get transcripts of all the shows and feed it into some AI, thump something, and it can just summarize that stuff for me. There you go. Possibly. I don't know. We're getting close to being able to do

[00:48:50] Jonathan: that. Yeah, I think, there may be transcripts already for a lot of them.

[00:48:53] Jonathan: yeah.

[00:48:54] Aaron: Somebody was archiving 'em, right? I think so.

[00:48:57] Jonathan: I think so. I've, when things happened and we went to Hackaday, one of the first things I did is I went to twit and, wrote a tiny bit of bash code and downloaded all of the, all of the audio from the show. So we at least have that. Oh, nice.

[00:49:08] Jonathan: Yeah.

[00:49:09] Aaron: yeah, the two that come to my mind immediately are, when we interviewed the Mars Rover team.

[00:49:14] Randal: Oh yeah. Mars Rover. Of course. How did I forget that?

[00:49:17] Aaron: that was, I just like, how am I here? participating in this conversation? This is so awesome. And it was like, it was crazy. They were telling us all sorts of stuff, telling us how it worked and Perl, mission

[00:49:27] Randal: critical Perl.

[00:49:28] Randal: I remember that. Every. Instructions sent to a Rover is passed through a giant Perl script. Every single instruction. That was so

[00:49:37] Aaron: cool. It was crazy. And then the other one I remember was Kubernetes. And I, at the time I was like, Kuber, what he's like, I don't know what it is. I didn't know what it was.

[00:49:45] Aaron: Cause it was so early on. It had been around for a few years, but they were talking about, changing the world with Kubernetes. what are you talking about? This isn't good. And then, now it's, I've worked with Kubernetes. professionally,as part of the companies I've worked for the past eight years or something.

[00:50:00] Aaron: And it's taken over, like I said, how we do cloud native and application infrastructure and all that kind of stuff. that definitely, I think about that one often is boy, if I only knew how big that one was going to become back in the day, when we talked to them,Maybe I would have paid more attention during the interview.

[00:50:14] Aaron: One of

[00:50:16] Randal: my favorites. It's still amazing, it's still amazing also that, that we did a show on VirtualBox and, Oracle still hadn't figured out they own it. Yes. So that's good. Yes. They still kept their mitts off it except put their name on it, so that's good. I'm glad VirtualBox still exists.

[00:50:30] Jonathan: Yeah. Yep. I remember interviewing the guy behind Bash and, at the end of the show, going to ask him what his favorite scripting language was and telling me, I'm not sure that Bash counts for that. And the horrified look of, yes it does.

[00:50:40] Jonathan: all right, Randal, so we are at the end of the show and you know what the two questions are. What, your favorite scripting language, has it changed? And what's your favorite text editor?

[00:50:50] Randal: Dart. Oh, it has changed. definitely changed. I still have Perl scripts that are running, but I'm not creating any new ones.

[00:50:57] Randal: and,my, favorite, my IDE now is VS code. Okay. And, I, I painfully shifted because VS code provided a full IDE environment for Dart and Flutter. And I. Start of, I actually stopped invoking Emacs probably about a year ago. but I would still have an Emacs running to answer my mail and VS code running in another window.

[00:51:26] Randal: And it was like, they're just, no, I just, I'm pretty much 100 percent on VS code. Now what I will fire Vmax for though, is that VS code can do remote file editing. but it only works on Linux systems because it downloads some package that only runs on Linux, whereas, Emacs can do remote file editing by actually tunneling an SSH connection and doing everything inside this SSH connection, it's really cool, Emacs is smart.

[00:51:54] Randal: Yes. yeah. Yeah. And there's still a little bit of me in every copy of Emacs, there's not a copy, a bit of me in every copy of VS Code yet,

[00:52:00] Jonathan: I'm working on that. Yet, yet, you gotta make it happen. It

[00:52:04] Randal: happens. It

[00:52:04] Jonathan: happens every time. Yeah. Alright, Randal, amazing to have you back. Thank you so much for being here.

[00:52:10] Jonathan: Let's, let's do it again. Let's not wait quite as many years as it's been. Let's,let's have you back in, a few months, a year. I don't know when it'll be, but let's make sure and have you back. Thank you, sir, for

[00:52:18] Randal: being here. Oh, thank you for having me on. It's just been a pleasurable, wonderful.

[00:52:22] Randal: Thank you. Absolutely. Good to see you, Randal.

[00:52:24] Jonathan: Yeah. Absolutely. Alright, Aaron, what do you think?

[00:52:28] Aaron: it's so great to talk to Randal. I mean, come on. we spent, I don't know how many years of our lives together. not every Wednesday for me, every Wednesday for him, but,it's just a, when I look back fondly at my days, Doing those shows and stuff like that.

[00:52:41] Aaron: it's a standout, right? It's Oh, yeah, we all knew what was how it was going to work and how it was going to go down. And Randal was a big part of making that happen. He did work. I know how much he worked behind the scenes for so many years on that show. Yes. it was really a credit to him that the show, Continued, right?

[00:52:59] Aaron: Especially in the early days after whoever was John O'Bacon came in, I think for a little bit or something. I can't remember who was before Randal. I think that was it. he came in for a couple of shows or something. And then there were some other people, Chris DiBona, maybe, if I remember correctly.

[00:53:11] Aaron: And, know, then Randal came in and it was like, whoosh. Okay, it was the Randal show. Now it's off and running, And so anyway, I, I, I owe a great deal of gratitude to Randal and just, it's just, nice to chat again. It's like an old friend that you don't, haven't talked to in a while.

[00:53:23] Aaron: So

[00:53:24] Jonathan: great. Absolutely. And,if he ever suddenly gets his Wednesdays free, we'll have to bring him in, maybe as one of the rotating co hosts. I don't know. We'll see if anything like that ever happens. but that would be a lot of fun.

[00:53:34] Aaron: Alright. that's two good reasons to move to Tuesday.

[00:53:36] Aaron: There you go. Cause you get Randal and me as co hosts, maybe. yeah. we,

[00:53:42] Jonathan: that is actually an intriguing thought. We'll, we will stick that on the, on the idea bin and think about that for a while, because Tuesdays would be easier for me too. So we will see what happens. Aaron, it was great to have you back.

[00:53:52] Jonathan: Hopefully it will not be another four months before we can have you back as well. Do you have anything you want to plug?

[00:53:57] Aaron: yeah, of course. I've got two YouTube channels now, that you can go check out. So there's retro hack shack and there's retro hack shack after hours. Yeah. Ooh. that's where I do some of my e waste Wednesday stuff and stuff that, isn't as popular with the larger YouTube community, but still really popular with my core audience.

[00:54:14] Aaron: I put that stuff there. In fact, I'm working on, I don't know, can I share my screen on this? Am I allowed to do

[00:54:20] Jonathan: that? it won't go out live as part of the podcast. So maybe let's do that after the podcast recording ends, then we can fiddle around with that. Yeah,

[00:54:27] Aaron: sounds good.

[00:54:28] Aaron: I will put up a, I'll show you a picture, of what I'm working on for this week, and people can guess, what old system this is, if they know. Oh, cool. so I'll definitely put up a picture. In fact, what I'll do is I'll put it on my YouTube channel as well, in the, community.

[00:54:41] Aaron: community section. I'll say, Hey, what system is this? We'll make it a little bit of a puzzle. Sure. And people can guess, but it's a really cool one. And it led to, one of the major PC architectures, that developed over the years. This was the first in, in the line, if you will, that really led to this particular PC architecture, that's still in use today.

[00:55:02] Aaron: So yeah, really cool board that I discovered on one of my,

[00:55:05] Jonathan: Neat. All right. we, we, so this is Tuesday. Next week we'll be back on Wednesday. And next week we are talking with Sean Dubois about WebRTC and all sorts of web media stuff. That's going to be a lot of fun. Make sure to be here for that.

[00:55:22] Jonathan: And if you want to listen live, we are live in the Hackaday Discord. Come join and, Wednesdays at showtime. That's nine to 30 Pacific time, 1130. My time here in central time zones, jump on board and join us live. if you want to see a project on the show, get ahold of the project lead or one of their engineers and have them email us, it's floss at hackaday.

[00:55:45] Jonathan: com. Have them send an email to us there and we will get them scheduled. and then the other thing, the one thing that. I want to plug is I also have another podcast. It's the untitled Linux show that's over on twit as part of club twit. We'd love to see you there. We do that live in the twit discord as well.

[00:56:03] Jonathan: I think that's it for this week, man. Thank you everybody. We had a few folks live in our live audience on discord. Thanks for joining us here and thank you to everyone that listens on the download and Hey, we will see you next time on floss weekly.

00:00 -00:00

FLOSS Weekly

It's the OG podcast about Free, Libre, and Open Source Software, FLOSS Weekly! Join us each Wednesday as Jonathan Bennett and the posse of Co-hosts interview big names of Free Software, cover utterly ...

Om podden

Avsnitt

Episode 832 transcript

Episode 831 transcript

Episode 831: Let's Have Lunch

Episode 830 transcript

Episode 830: Vibes

Episode 829 transcript

Episode 829: This Machine Kills Vogons

Episode 828 transcript

Episode 828: Incus Inception

Episode 827 transcript

Episode 827: Yt-dlp, Sometimes You Can't See the Tail

Episode 826 transcript

Episode 826: Fedora 42 and KDE

Episode 825 transcript

Episode 825: Open Source CI With Semaphore

Episode 824 transcript

Episode 824: Gratuitous Navel Gazing

Episode 823 transcript

Episode 823: TuxCare, 10 Years Without Rebooting!

Episode 822: Nand2Tetris

Episode 822 transcript

Episode 821 transcript

Episode 821: Rocky Linux

Episode 820 transcript

Episode 820: Please Don't add AI Clippy to Thunderbird

Episode 819 transcript

Episode 819: Session, It's All About the Metadata

Episode 818 transcript

Episode 818: I Don't Care About the Roman Empire

Episode 817 transcript

Episode 817: Incompatible With Reality

Episode 816 transcript

Episode 816: Open Source AI

Episode 815 transcript

Episode 815: You Win Some, You Lose Some

Episode 814 transcript

Episode 814: The Banksy Situation

Episode 813a: Happy Holidays

Episode 813 transcript

Episode 813: Turn Off the Internet

Episode 812 transcript

Episode 812: Firefox and the Future

Episode 811 transcript

Episode 811: Elixir & Nerves - Real Embedded Linux

Episode 810 transcript

Episode 810: A Rising Wallet Pays for All Boats

Episode 809 transcript

Episode 809: Pi4J - Stable and Boring on the Raspberry Pi

Episode 808 transcript

Episode 808: Curl - Gotta Download 'em All

Episode 807 transcript

Episode 807: Bitten By the Penguin

Episode 806: Manyfold -- The Dopamine of Open Source

Episode 806 transcript

Episode 805 transcript

Episode 805: Mastodon -- Bring Your Own Algorithm

Episode 804 transcript

Episode 804: The AI Alliance -- Asimov was Right

Episode 803 transcript

Episode 803: Unconferencing With OggCamp

Episode 802 transcript

Episode 802: Emba -- Layers Upon Layers of Bash

Episode 801 transcript

Episode 801: JBang -- Not Your Parents Java Anymore

Episode 800 transcript

Episode 800: Champagning the Ladybird Browser

Episode 799 transcript

Episode 799: Still Open Source at Percona

Episode 798 transcript

Episode 798: Building the Rust Desktop with COSMIC

Episode 797 transcript

Episode 797: Coreutils -- Don't rm -r Up the Tree

Episode 796: Homebrew, I'm More of a Whopper Guy

Episode 796 transcript

Episode 795 transcript

Episode 795: Liferay, Now We're Thinking With Portals