FLOSS Weekly

This week Jonathan and Randal chat with Daniel Stenberg about curl! How many curl installs are there?! What's the deal with CVEs? How has curl managed to not break its ABI for 18 years straight? And how did Daniel turn all this into a career instead of just a hobby? Listen to find out!

• https://daniel.haxx.se/

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Dan joins me and we talk with Josh Bressers about ANCOR and open source security. We talk about the kernel and its many CVEs, the state of the CVE system in general, what an SBOM is and why you should care, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 807, recorded Wednesday, October 29th, bitten by the penguin.

It's time for Floss Weekly. Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and it's going to be a lot of fun today. We're talking about Angkor and security and open source. Surely that's not a thing anybody has to worry about. Uh, that said a little tongue in cheek for those of you that don't realize.

Uh, first off, we've got. A great co host today. The wonderful, the original Linux outlaw, Mr. Uh, Dan Lynch, Dan, welcome to the show. Hey Jonathan, how are you doing? I am, I'm doing great. We were talking just before we started that, uh, this was kind of an early morning for me. So one of two things are going to happen.

Either I'm going to get tired and kind of get a little loopy and punch up happy during the end of the show, or my energy level is going to drop and it's going to be real draggy. So we'll, we'll see. I'm feeling pretty good at the moment. I've. Got the go juice, the coffee to keep me going. So

Dan: that's good stuff.

Yeah. And, uh, I managed to make it on time, even though the hours different here, our clocks have changed in the UK. It's an hour earlier. I once on, uh, on plus weekly with Randall years ago, a long time ago, I turned up an hour late because I didn't realize that the U S clocks changed differently to the UK ones.

Jonathan: I remember that. That was fun. So when we were setting up, when we were setting up for the show, um, In the past week, I made sure and sent both of you guys an email saying, Don't forget, time change is a different weekend in the US and the UK. Yeah, it's, it's no fun to, it's no fun to show up to do the show and not have everybody here.

No, no. Uh, Dan, this is actually a guest that you sort of brought to us, isn't it?

Dan: Yeah, so I was at odd camp recently, which a guest of the show only a few weeks ago now you had Gary on to talk about that with Simon as well and I was at camp and I met my old friend Alan Pope, a poppy as he's known online and he he kind of hooked things up and suggested, you know, the guest for today and I'm really pleased that he did because it's going to be a great show.

Thank you

Jonathan: Yeah, so we've got Josh Bressers, who is the VP of security at ANCOR. And from what I understand, ANCOR does a lot with open source tooling, security tooling around open source, which, I mean, this is sort of a match made in heaven for me. There's a couple of things I really care about, and so I am super interested to hearing about it.

Uh, let's just, let's go ahead and bring him on. Um, let me push the right buttons to make that happen. There we go. Josh, welcome to the show. Awesome. Thank you so much. I'm super excited. Yeah, it's gonna, it's gonna be fun. Now, you are the, you have a VP of security at Anchor. What, let's start with this. What is, what is Anchor?

Like, what is the, the, the, the problem space that you guys as a company are in?

Josh: Yeah, for sure, for sure. So, we have a product, an enterprise product, built on a foundation of open source. We call ourselves, like, Next Generation Software Composition Analysis. Everyone has Tons of open source today in their environment.

And so we have tools that help you scan into your stuff, basically, figure out what the packages are, and then you can do vulnerability scanning. You can do, like, policy management and enforcement, things like that, plug it into your CI system. You know, there's, we, we, the, the, the US federal space runs us a lot.

There's tons of compliance in that universe, and they're running open source absolutely everywhere. It's everywhere. Mm hmm. Mm hmm. And so you've probably heard of things like, you know, like SSDF and STIG like, we have a FedRAMP webinar in a couple of days to talk about all this stuff. And it's just, it's a, it's a strange universe.

And I've been in open source just literally forever at this point. I mean, more than 20 years. And so it's super fun. We've got these two open source projects, SIFT and GRIPE, that get quite, quite a lot of attention. SIFT is an SBOM scanner. You can point it at containers and directories and whatever you have.

It will generate a software bill of materials for. Kind of whatever and then you can take that software bill of materials, and that's the thing that you can then, you know, look at for policy enforcement you can scan it for vulnerabilities. You can just ferreted away for later like that. The famous example is log for J right where we all we are.

Everyone has battle scars from that. And if you had. A catalog of all your open source, you know, do I have log for J means you go to the search box and type log for J and you're like, oh, there it is over here. Let's go look and see what's going on versus saying, do we have log for J? Like, I don't know if we have log for J.

Someone should go find out. Yeah, it, it, so it's, it's been wild. And in fact, it's funny because I started at anchor, I think like a month before the log for J incident happened. And so like it was trial

Jonathan: by fire.

Josh: It was amazing.

Jonathan: Yeah. Yeah. Yeah. So, so SIFT is the S Bomb finder. What is, what is GRIPE then?

Josh: Uh, vulnerabilities.

So you can either point it at the same artifacts, be it directories, containers, whatever, or you can just hand it an S Bomb and the, the, the magic there. Is so you scan something for an S bomb, right? And your software bill of materials theoretically doesn't change because you've got your container or your release or whatever, but the vulnerabilities do because there's new vulnerabilities found every day, every week, whatever.

And so you need to keep re scanning things for vulnerabilities over and over again. And if you have to scan, let's say like, you know, a multi hundred megabyte container that can take, you know, 30 seconds, a minute, whatever, depending upon your hardware. Versus if you've already done the scan and you have like your software bill of materials.

You can usually scan those in milliseconds. And so there's like a huge advantage just hanging onto those documents and their JSON, right? Like maybe a couple of megabytes max.

Jonathan: And so gripe is, is mainly just going out and looking at known CVEs and comparing it to the software and the versions in your, in your S bomb.

Exactly, exactly that, yes. Yeah, it seems like the world kind of woke up to the idea of, boy, we really need to know what software we're running. I think Log4j was really, yeah, like you say, it was the point where everybody said, oh hey, SBOMs might be a good idea.

Josh: Yeah, yeah, for sure. And well, so I think what happened in Log4j is, so I'm, You can imagine I've spoken with many people in corporations and open source, like all over the place.

And I think what happened is a lot of organizations didn't realize how much open source they actually had in the log for J incident happens. And they, you know, send out the minions to go figure out where, what our open source is. And then they're like, holy crap, where did all this stuff come from?

Because, you know, any, any open source nerd. We knew open source had been taking over the world for literally decades, and we knew it was running everything. But I don't think a lot of the, we'll say, business folk had quite figured out what was going on until log4j. And so I feel like that is truly the moment all of the businesses, all of the people were like, holy crap, this is just literally everywhere.

And that's the moment. I knew some people that were like, can we stop using it? And they were just laughed at because like at this point, no, it's completely out of the question. Right.

Jonathan: And unless you want to turn all of your logging off for your product,

Josh: all of your everything. I mean, it's wild because there's all these surveys that keep coming out.

I mean, I think I read one from red hat not too long ago, where they said 80 to 90 percent of all enterprise applications. Our open source underneath and honestly, I have no doubt, no doubt at all that that's the case.

Jonathan: Yeah, yeah, it seems like all the, all the compliance of, you know, you, you mentioned this sort of almost like an alphabet soup of the compliance things.

Some of those I have heard of and some of them I haven't because that's, I'm not in the enterprise or federal space really. Um, but it almost seems like that's just. That's the price that we pay for having made it. We're here and open source is not going away. So it's time to play the game the big boys play.

Josh: Well, okay, so that's a really good point. So there's a famous blog post. It's called, I am not a supplier. A fellow named Tomas de Pierre wrote it back in the log for JDAs. It's been almost, I think it's been two years now since he wrote it, which is like blowing my mind because I feel like he wrote it a week ago.

But. In that he talks about the fact that he was getting these like questionnaires from companies saying, you know, oh, you're my supplier Do you do you have log4j and your stuff and he's like, I'm not your supplier Like I'm some dude who put some software on the internet Like that's not the relationship and I think that's another really wild aspect of this is the open source developers Like they technically don't owe anyone anything, right?

But if I'm an enterprise using open source I'm the one responsible for that because, I mean, if you think about open source, I often say it's more like a natural resource, right? We're like, you're going to harvest lumber from the forest. You don't ask the forest for things, right? Like you're responsible for the things you'd get.

That's a,

Jonathan: that's an interesting analogy. This is something that I know the Europeans are having to sort of wrestle with as they're, they're looking at making some legislation about security and software and wrestling with trying to, trying to fix that in law. That's right.

Josh: And they have two things in Europe that we're keeping our eye on.

And like the compliance universe is there's the CRA, which is, I do not remember what it stands for anymore. I apologize. But that's the one that's talking about, like, you need to track yourself or you need S bombs. You kind of need all that stuff. And then there's another directive called NIST 2, which is more of a, you know, you're an organization, you're a company and you have to follow certain cybersecurity requirements.

And like, Knowing what you're shipping and providing security updates and things like that are part of that. And one of the challenges in the open source world has been the EU directives initially just said like all software and the open source people were like, Whoa, hold on there. Like you can't have this work.

Yeah, right, right. Exactly. And so, I mean, places like the Apache foundation and the eclipse foundation and like the Linux foundation, they really did a nice job of fixing a lot of those problems. So now there are. You know, there, there's sections carved out that say like everyone except open source type things, which is good.

Cause that's what we want.

Jonathan: Yeah. I mean, our, our own Simon Phipps was actually part of that. He would keep, he's in UK, but obviously he would. Several times went to Brussels and was talking to the people making the laws and trying to educate them on you know Here's how this thing called open source works And here's why what you have written this this piece of what you have written in this law just does not reflect reality at all That's right.

That's right. It's some of the you know, I I I Did I read the entire CR? It's the Cyber Resilience Act.

Ah, that's it. Thank

you. At one point, I read either parts of it or the whole thing. I can't remember which for some coverage of it. And like, there's some really good stuff in there.

Yeah. There's actually

some really good stuff in there.

Like, you know, if you are making software, like if you are a company that is making software, if you're putting your name on it, you are required to have a security contact. It's, it's mind blowing how many companies just Don't. It's like, Oh, I found a security vulnerability. How do I tell them? They don't know.

You don't know. Nobody knows. Might as well just drop it on Twitter because there's no better place to do it. You can, you can, you can email their contact at email address, but you'll know, you'll probably never hear back. If you do hear back, what you hear back will be written by Chad GPT in today's world.

I'm experiencing that. Or a lawyer. Well, or a lawyer. Cease and desist. Trying to hack our product. Exactly. Yeah, it's, it's a little, um, so what, uh, You guys, the, the Anchor Company, um, let's turn back to that for just a second. What's sort of the, the space that you're primarily focused in? Because it seems like everybody that is in this space has their own niche.

Um, we, we talked with, uh, we talked with the guys from Workbrew, which that's actually the commercial offering now attached to Homebrew, and they were cool. Like that was, that was a great project to talk about, but they've got a, they've got a very narrow niche, and that is. People running open source stuff from Homebrew on Macs at their workplace.

And they want to be the guys that own that. We've talked with some other people that are all about, you know, uh, JavaScript, you know, the, the, the, the full stack JavaScript, and they want to be able to catch those kinds of patch or watch for CVEs in those packages. Um, and you know, there's some places that kind of seem to get left behind, like the actual.

Linux binaries and packages. There's not as many people working on that, which might be a problem. Uh, but what, what is sort of the niche where Anchor fits in here?

Josh: Yeah. So I would say we're a little to the right. You don't hear the word shift, right? Very often, right? It's always shift left everybody. But the thing we like to do is we work with, we'll say the operation side of DevOps.

Cause we also, it's like, Oh, it's the DevSecOps everybody, but there's still people who are, we'll say a little more operations than dev in every organization. And so what we do is imagine you acquire software from somewhere. It could be internally developed. It could be an external thing, who knows? And you need to deploy it.

And so you scan it when you bring it in and then you know what you have, right? You, you've got your S bomb, you know what it is. And then. There is the vulnerability aspect, which obviously is important these days, because, you know, vulnerabilities are in everything, and then also there's the policy. So the intent being that you have all this software kind of flowing into your organization, and then using the tool Anchor has, Anchor Enterprise, it gets scanned, and then there's policy gates, for example.

So you can say, like, everything looks good, we can just keep flowing it through the system. Or you can say, oh, something is weird. Like, We're going to stop the train here and someone's got to go figure out what's going on and so it's kind of that. And then the other aspect is once you have things like deployed in kubernetes will say again that whole security vulnerability piece.

You can say like I see all of these containers running in my kubernetes cluster. Now, let's reapply the policy continuously, right? That's like you always hear about, you know, continuous compliance and things like that. And this is kind of one of those examples where you can, you can get a report every day, every hour, every whatever.

And you can look at it and be like, okay, all of my stuff is passing our policy. Or you can be like, Oh, holy crap, a bunch of stuff just failed. What's going on? And then the intent is, you know, humans get involved and, um, Kind of figure out what's, what's

Jonathan: up. So is, is Anchor mainly aimed at, uh, the, the container world then?

So can it, could I say, here's my, here's my Linux server that runs my website. I'm going to, you know, point either, uh, either the open source tool or the Anchor enterprise at it and say, I want to catalog every package and every binary that's installed on that Linux server. And I want to know when a CVE hits, is that in scope?

It is not

Josh: for the enterprise product, but with SIFT you could do that, yeah, SIFT can scan just, you can point it at slash and it'll find all the binaries, it'll find the packages, it'll find, I mean, so this is actually one of the really interesting points in this whole universe that doesn't get talked about a lot is there's no such thing as we'll say like just a node application, right?

You've got like the node. js binary that runs it all. You have supporting libraries usually that help with that. To work. You've got your packages on the system. You've got maybe some random binary crap. Someone installed in their direct home directory somewhere. You know, there's like all this random stuff.

And so we like SIFT does a pretty good job of finding a lot of there's still some things that can't identify because like, for example, if you build your own binary, like Maybe we can't catalog that, it just depends how we're detecting it. And so we also have this concept that we're working on. We call it known unknowns, where it's like, I found a thing, but I don't know what it is.

So, maybe someone should go look at that thing and see, like, is it something we care about, or is it something we can be like, Eh, whatever, it's just some random trash someone threw in a

Jonathan: directory. You're almost, that's really interesting, you're almost into the realm of threat detection then, with that sort of a capability.

Interesting.

Josh: A little bit. I mean, I would say there's, there's some strange overlaps in that regard, you know, everything from like observability and you've got threat detection and kind of same. It's one of those. I have all this data. Now, what do I do with it? Right? And in fact, there's a lot of people that take the output of these kind of tools and they just throw them into their Splunk server, their elastic server, whatever.

And then they use it as part of their whole, like, detection process, where they can say, like, Oh, I've got this evidence, then I can follow it around and, and see, like, it started out over here and, you know, that file ended up over there, or whatever.

Jonathan: I could, I could see somebody taking these, these unknown files and just, let's do a mass upload of all of them to, like, Total Virus and see if we get any hits.

Exactly. For sure. Super interesting. So, uh, you're detecting CVEs. We touched on this briefly before the show started, and I think it will be real interesting to get your take on this. So you wear a couple of different hats. You're not just the anchor security guy. You're also the open source security guy.

You've got a couple of podcasts, I think, that cover this. That's right. That's right. I have a podcast called the

Josh: open source security podcast actually today. And we're, I mean, we're like seven years into that one. I mean, we're 400 episodes and change at this point. So yeah, I've been doing it for a while. I love it.

It's a lot of fun.

Jonathan: Yeah. So. You have thoughts, I'm sure, on the, um, the absolute flood of CVEs coming out of the Linux kernel then.

Josh: Yes, I do. In fact, I, so Greg KH was a guest on my show shortly after he started doing this. And I am a massive fan of the work they're doing. I think the Linux kernel is a great example of kind of doing this right.

And it is creating a ton of work and it's a lot of effort on their part and of the people who receive it. So like, if you're on the receiving end of this, I'm I, I, I totally get it. You're overworked. It's ridiculously hard to sort through all this stuff. But at the same time, the reality is these are potential problems.

And that's fundamentally what we're dealing with when we're talking about like vulnerabilities in software. It's like, this is a, a risk that we have to understand. And as Greg, you know, so eloquently puts it, when he talks about this is a Linux kernel is run on everything from like milking machines to literal rocket ships.

And so they can't say, we know exactly how the software is being used. So we can say this bug doesn't matter. They don't know. And so that's okay. But this is where. As the security people on the receiving end, we need better tools that help us categorize and understand what we're getting. And now there's like 7, 000 layers in this.

There's everything from the data in the CVE program is absolute crap and anyone who's ever worked with it will that's not No, one's going to argue with me over that point. Like we need better data. We need better tools. I mean, this is part of what I'm doing is I'm trying to help build better tools to let us deal with all this data and information in a more coherent way.

Like we can't humans have to get as. Far out of this process is possible. We're talking about now this year, we're going to see a little over 40, 000 CVEs across the industry, and that's probably way too low. I have another thing. I think I sent you guys a link to my blog post where I talk about the size of open source.

Like there are 4 million NPM packages, 40, 000 CVEs in 4 million packages. And that's just NPM. Like that feels a little low, you know what I mean? And so it's quite likely that if we. Had good data and we had more research and we had the ability, imagine a world where we had 500, 000, a million CVEs per year, and like all the security people just like, you know, had an accident, but like, that's what we need to think about is like, that's the scale of what we're dealing with.

Like open source is absolutely gigantic. The Linux kernel is absolutely gigantic. And there, there are a lot of vulnerabilities in this stuff, but we have to learn how to deal with it better. Better. That is our new challenge. Now it's not the whole, Oh, I'm going to read every CVE and figure out how it affects me.

Like, no, no, we need tools to be able to say like, this is the CVE. You need to look at today, these other, you know, 7, 000 that came out. Don't matter. Don't worry about those. And that, it's a really hard problem.

Jonathan: Yeah, so I've, I've mused just a little bit that with the kernel, and so you, you touched on it, so what, for those that don't know, what's happened is in the Linux kernel, there's been a push to better report CVEs.

And they have essentially come to the point now to where they consider more or less every bug to be a CVE. And they've got a valid point to this, like, because any bug running in the kernel space, because it's so tightly ingrained with the system, there's a really good chance that it can be used maliciously.

And so, like, that's a fair point. Um, I, I have mused that this might be just a little bit of malicious compliance. I, I, do you think, do you think maybe just a little bit there, there's a little bit of, of, of, uh, would, would that be schoenfreude? Um, So I,

Josh: I, I thought so at first actually, but after talking to Greg, I decided no.

I think like this is like actually legitimately, The right thing to do, which makes me a little sad. Cause the malicious compliance angle like abused me. Exactly. Yes. Uh,

Dan: so with this kind of increased, um, I don't even have a good word verbosity. I'm going to say, which makes me sound clever, but increased verbosity, I suppose, of, of reporting of CVS and stuff.

Is there a danger that people become less. Um, sensitized to them, if you know what I mean, so like the, the, the, the boy that cried wolf or something, do people then start to say, well these guys are always saying there's something going on, and, you know, is, do you think that ever happens, that, that the sheer volume of them now is, is kind of making people think, oh, you know, this is probably nothing?

Josh: I don't think it will because of compliance. I think when you're in any sort of regulated environment, you have auditors that care, and they don't have to do the work, so it's no skin off their back, right? If they, if they have to say, oh, you have to deal with 7, 000 of these things a day, like, whatever. So, I, I think your point could be valid if it wasn't for regulated industry, because yes, you could easily say, like, the Linux kernel is releasing, you know, hundreds of CVEs per week, and actually, I think it's more than hundreds, it's, it's two or three hundred.

Um, but yeah, all these CVEs per week and none of them matter. So I'm just going to completely ignore it. But at the same time, I guess this is actually part of Greg's talking points is if you just follow current, you don't have to care. Right. And I know that's easy to say of, Oh, just upgrade your kernel.

It's not that simple, but if you're trying to follow. More closely than we maybe did in the past, that does make a lot of these problems go away. And I think that's part of his intent, is by just having so many and saying like, just, just follow upstream. Don't try to figure this crap out, because it really doesn't matter as long as you upgrade.

Dan: That kind of goes back to Jonathan's point about malicious compliance a little bit. Is it a good way of forcing people to follow the latest current kernel and say, or more people to kind of go? And, and isn't to get up with.

Jonathan: And is it related to the dropping of all of the long term support kernels?

Because nobody used them anyways.

Josh: Right, right. Well, I mean, that was, that's the business model of the distros, right? Is we're going to support the software for decades. And everything will be fine, except, oh, we can't do it with the kernel now, because it's too big and it's too fast and there's nothing we can do.

So, yeah, I don't, I mean, so dropping the LTS, That's something I watched quite a bit, and I mean, it fundamentally just came down to no one wanted to do the work, right? And I think that's just your classic supply and demand. Like, if there's a demand, then find some money or find some people and make it happen, you know?

And it's also open source, so you can't be like, oh, those darn Linux developers. Like, no, just go do it, right? That's how it works. Yeah.

Dan: Yeah, that's true. That's very true. Okay. So, uh, so josh, I want to back up a little bit and talk cause I, I'm not, I know a little bit about security, but I'm not really up on a lot of this stuff.

So I, for the benefit of some of our listeners, I know we have a very technical crowd who listened to this show, but some of the layman, I suppose. Um, can you tell us exactly what an S bomb is and how it works? So it's a, it's a, it's a, am I right in thinking it's a bill of, uh, what is it now? Well, in fact, why am I trying to answer my own question?

I don't know what I'm doing there. Can you tell us exactly what an S bomb is and why

Josh: it's important? Yeah, I mean, so, it's a software bill of materials, right? The idea is, all of your software has stuff in it. And some of the stuff you wrote, most of the stuff you didn't. And so you have Like you might have NPM packages, you might have Rust crates in there, you might have Go, whatever.

And so basically what a software bill of materials is meant to be is on any system, any application, it catalogs all the stuff in it. So like if you do an NPM install and it installs 100 things, those 100 packages would show up in the software bill of materials. And you can also imagine that you can have like different Kind of pieces of software where you have a, a container image and then you have your software and those, those are two different software bill of materials.

Now you put them together. Now you have a third software bill of materials that is functionally the other two pasted together. And so the idea is just like, it's imagined when you, when you buy something from Ikea. Right. They always have that list of all of the parts that should be in the box. And in fact, I don't think he has ever screwed me over on missing parts, but you know, that's the intent here is if I get a piece of software, what are all the pieces in the box essentially?

And then we can start asking questions about those pieces. And like, one of the really interesting things I like to do is, so if I get an S bomb from somebody, I'll, I can, Look at it and understand it. No, I shouldn't, I should use the word look at loosely because there's usually thousands and thousands of packages of this stuff.

Like I'm, I'm kind of an armchair data nerd. So I just put everything in elastic search and then I do stuff from there, which is not, I know that's not how normal people work and that's okay. That's why we need tooling to help us do this. But anyway, someone gives me an S bomb and says, here's the S bomb for the thing I built.

And then I scan it. And then I say, what's, what do I see? Are they the same? Are there things different? Like, did stuff get added somehow no one knows about? Did stuff get removed? How did that happen? And this is like one of the really cool things you can do, is you can watch your software as it kind of traverses from the left to the right.

And you can be like, why did this file get put here? How did that happen? Or why did we remove these packages over here? I don't understand why, why we did that. And so there's, there's just all this weird stuff you can start doing and asking questions about your software, because, I mean, let's face it for the last, what, 30, 40, 50 years, the vast majority of software has been kind of YOLO.

We're like, we type make and we get something out the other end and we're like, I guess that's it. Like go run that in production.

Jonathan: Yeah.

Josh: It's, it's funny when you think about it, but at the same time, it's like this stuff is like running the world. I mean, I feel like we should know a little more about

Dan: it.

Yeah. Is there a standard format for S bombs? Could I take an S bomb that was generated by SIFT for example, and ingest it into, uh, into another. You know, program another, another, another company's tool or another project's tool? In theory,

Josh: yes. So, Okay. So we are kind of early days still. There are two primary S bomb formats.

There's something called SPDX and Cycle and DX. And They're they basically do the same thing. I mean, I know you can nitpick over Oh, this one does, you know, it records dates a little different or whatever, but what it doesn't matter fundamentally they're the same thing and the intention being that you have interoperability between tools and and whatever and Some of the tools can do this.

Okay, not all of them can because like any good standard nothing makes sense So there's tons of wiggle room And they're working on fixing it, like they're on, I mean, SPDX is I think 2. 6 now, SPD, no, SPDX I think is version 2. 6, and they keep adding changes to this stuff, and they're constantly trying to, I guess, bring the, the formats a little closer together so they work better, like one of the things SIFT does, so, SIFT, if you run SIFT, the default output format is something we call SIFT JSON, And it's funny because the S bomb people like, Oh my goodness, did you make a third format?

I'm like, no, no, we did not make a third format. We are basically just, we just take whatever SIFT has in memory and we dump it out to JSON with the intent. There is more data in that than either CycleNDX or SPDX collect with the intention being you can convert between CycleNDX or SPDX because like if you have an SPDX document you can't today convert it to a CycleNDX document You will lose information because there are those little niggles back and forth So you kind of it's kind of imagine Google Translate where you know, you you can You keep translating something back and forth and you end up with a hilarious output at the end, same sort of problem today where it's being worked on, but I mean, you have to remember too, like these formats are not like, it's not old, right?

We're, we're still very early days here and it's getting better. It's getting better. Thank goodness.

Dan: So you mentioned that the sheer amount of data and so on, is there, I'm going to, I'm sorry to have to invoke this, but I'm going to, um, is there an application for AI in this where you could use something like an LLM or something and train it with some of this data?

I mean, there's people

Josh: looking into that for sure. I, I think it is an excellent question. What can we do with them? Like, I, I haven't seen anything I would say is particularly compelling yet. Just because I feel like this data is kind of boring and There's not a lot of wiggle room. Like if an LLM gives you a, a, a weird version or a weird package name, that's potentially bad, but I know there's, there's interest today in doing things like saying, okay, I've got this S bomb, I've got all these vulnerabilities in it.

You know, what, what should I start working on first? How can we look at this data and maybe unwind it a little bit using something like an LLM to maybe understand the vulnerabilities And your product is like a really good example is so let's say I have a container image and it has a website open to the Internet, right?

That is going to have a very different threat model than if I have an application that is. Downloading a CSV file from a website and then parsing it and uploading it somewhere else, right? And so this is, that's one of the places I've seen some of the LLM efforts look interesting because you can say, like, this is what my application does.

Now, you know, based on this information, look at these vulnerability details and tell me which ones sound like they are going to affect me in a very bad way.

Jonathan: You know, I'm, I'm aware because I cover this stuff for Hackaday that, um, people are using AI in security. Research and it's not necessarily going well.

It's actually, it's actually a real problem for open source projects. They are getting particularly those that have bug bounties. They are getting vulnerability reports that have been generated by AI. And the problem with that, and this is sort of just in general, this is the problem with AI today, Um, they sound very plausible.

And these vulnerability reports, they are like, very well done, well written, very plausible sounding reports, that are totally hallucinated. Yes. Is that something that you're seeing more broadly? Is that something that, uh, I guess you guys are hooked into? To even observe it?

Josh: Not I I'm not affected by that in any way I know some people that have dealt with it and they've they put the hammer down pretty hard because it really really annoyed him a lot However, there is some really cool security research I've seen in the LLM universe where you have Researchers training up models, you know the code models and they're saying like here is a description of a vulnerability I want you to write an exploit for this vulnerability Against this application right here, and that's been reasonably successful, which is a little terrifying sometimes because most vulnerable, like what's the stat?

I think like 3 percent of vulnerabilities are actually exploited, but with some of this technology that could even if it doubled. The workload probably quadruples, right? I don't think we're talking about linear scaling here between the vulnerability rate and the work that the security analysts have to do.

And so that's kind of a weird, scary part. And then there's also people doing research where they're basically saying, I want you to just take this application and find a cross site scripting flaw in it. And that works surprisingly well also, which they are there. We start getting into, you know, the whole like 500, 000 million CVEs per year.

If we have. automated discovery, obviously, you're going to have, you know, a huge influx of vulnerabilities, which we are not at all prepared for in any reasonable way today.

Jonathan: Yeah, boy, you could imagine, uh, uh, you can imagine a scheme where you take a, uh, you take an LLM and you write the prompt that says, you know, find this kind of vulnerability and write an exploit for it.

And then you hook that into on the backend, a fuzzer. That actually tests what it could, what it spit out. And then if you were to feed that output back into the LLM, and I imagine there are people trying to do this right now, but you actually get a positive reinforcement loop that actually works, because it'll tell the, it'll tell the LLM, that one worked, that one didn't, and particularly as you start scaling up the speed in which it can churn through that loop.

Yeah, that's going to find real vulnerabilities. Of course it will. So

Josh: that happened. DARPA had, uh, like basically a security capture the flag event. And there was a, an AI, it was originally done by Carnegie Mellon. It was called mayhem. Um, David Brumley's behind it. He has a company now and he just changed his name.

I can't, it used to be for all secure. I don't remember what they changed it to, but anyway, mayhem was designed to basically look at like a binary. Find problems in the binary. So we're talking about like just operating on straight up binaries. Like we're not even looking at code here. Find problems in binaries, fuzz it, find out how to exploit it, write the exploit, then also write a patch in the binary.

So the other team can't exploit it against you and then going and attacking them with the thing you just found, which is like all automated. It's like, holy crap. This is ridiculous.

Jonathan: Yeah. Well, so that's, I guess that's the next stage of this. When we, when we finally reached security nirvana, you've also got, so you would have to write it, you'd have to write a really good test suite to make sure your program is working correctly.

You run through this process I just described to find the vulnerabilities, and then you also tell the LLM to fix the vulnerabilities, and you feed that back into it as well. And as soon as you fix the vulnerability and still pass your test suite, Might have good code on the other end of it. Hopefully. We can dream.

Yeah, right, right. Okay, so, the many, many, many vulnerabilities found, lots of CVEs. Um, how are the various organizations and agencies that are responsible for Tracking and catalog, cataloging these, you know, you've got NIST, you've got MITRE, you've got the NVD. How, how are they doing with it? Oh my goodness.

So it's, it's

Josh: been a rough year. So, so anyone not in this space might not know this, but in February or March of this year, I forget when it was, NIST runs a database called NVD, which stands for National Vulnerability Database. And what they used to do is. MITRE, which is a government contractor, runs the CVE program, right?

And so people who, like, get CVEs, you get them through MITRE. You can have a company that can assign their own CVEs, but they go through MITRE. Or, like, security researchers can just go to MITRE and ask for a CVE. So those all go into MITRE's data set. And that data is very bad. As anyone who's ever looked at it knows.

So NIST had a program called NVD, and they would take the MITRE data, And they would add, they would enrich it. They would add some version information, some like package information. They would add like severity. They would guess at severity. Their severity scores were often very bad. And they would kind of add this information.

And then they just stopped one day, like no word or warning, no nothing. And, and I, in fact, uh, uh, there's a handful of us security nerds who like, look at this data. And I remember we were like, Do you guys like didn't stop and we're like exchanging notes and being like, holy crap, like they stopped. There's, there's no new data.

What's going on. And then of course we'd email them and crickets. No one knew. And so they just like literally fell off the face of the planet for probably six months almost goodness. And then at the same time you have Sisa. starts a new program, they're calling Vuln enrichment. So now CISA started doing some enrichment.

NVD has sort of come back and they're doing some enrichment, not everything, but some, there's still a giant black hole in the middle where they didn't do anything. And then you also have MITRE trying to ask the CNAs to do a better job of providing nicer data. So now instead of having one, And a half, we'll say organizations providing not good data.

We have three organizations all providing, we'll say incomplete and difficult to use data. And so for, for vulnerability nerds right now, it's like, it's a mess. It's really hard to figure out what to do. It's really hard to understand. I mean, this is a huge challenge we have, because obviously we have gripe at anchor and gripe means vulnerability data, and so we've got, you know, a team of people that are doing their best to try to unwind all this and make sense of it.

And it's. It's not, it's not the most fun, not the most fun at all. We've ever had, but you got to do what you got to do.

Jonathan: So I don't know.

Josh: It just kind of is what it is today.

Jonathan: There's, there's several different directions. I want to go with that first. You mentioned CNA is what's a CNA. They call that a

Josh: CVE numbering authority.

That is where, and this, this is like such a fricking mess too sometimes. So, um, yes, curl, which we've all heard of. So, so the guy behind it, Daniel, um, baggery goes by online. There were all of these CVEs being filed against curl. And he was upset with them because they were kind of garbage and they shouldn't have been filed and he tried to get rid of them, but working with NIST and MITRE is like an impossibility for the most part.

They just, they don't have to listen like their, their, their customer is the U. S. government. So, haha, no. So anyway, CURL has become a CNA. A CVE numbering authority with the sole intent of being able to get rid of these crap CVEs and just not have to file them at all, which is a super heavy handed way to just like match reality.

And, and the thing is like, Daniel's way brighter than a lot of people and like he works on curl full time. So he can pull this off. But like, if you're an open source project that you work on an hour a week.

Jonathan: Yeah,

Josh: there's no way you can do this kind of thing. And so that's one of the complaints I have is just the fact that all of this, all of these organizations, all this data, it's just like such a one way black hole that no one knows how it works.

No one knows what's going on. I mean, I've been complaining forever that this is exactly the sort of thing. That should, like, exist in some sort of open source foundation

Jonathan: and

Josh: have, like, proper bylaws and, you know, proper governance and we understand how everything works and all of that, but I've been screaming into that void for more than 10 years now.

And put it

Jonathan: on

Josh: the

Jonathan: blockchain, right?

Josh: I'm sorry,

Jonathan: I couldn't resist. That will solve everything. Um, you've got a few companies out there that are trying to pick up some of the, like, GitHub really comes to mind. I've been, I've been involved with a couple of CVEs now that, uh, GIF, GitHub has been the CNA for, and they've, they've got a lot of automated tools with that.

I think they do have some human review in there as well. And that's actually been a really good experience. Um, they do.

Josh: So GitHub's doing some very clever things here, actually. Mm-Hmm. So they, they are a CNA, they will assign CVEs. Yeah. But they also have their own data set. They have the GitHub advisory database.

Mm-Hmm. and, no, I'm sorry, . They, they kept yelling at me about this. It's the GitHub vulnerability database holds GitHub advisories. . That's how that works. Okay. It's very important to get this, but they have a team of people that what they do. So actually, I said there were three organizations, there's kind of four, if you consider GitHub, where they have a team of people that look at all of the CVEs that affect the, the ecosystems GitHub cares about.

They focus on things like, like Maven and NPM and Python and Go. And there's a couple of others. I can't, Ruby is one of them. And what they do then is they. Do a really nice job of kind of teasing out what is the ecosystem, what is the package and what version fixes it. And what I love though is if, if you find an error, you just submit a pull request and then they either say, yep, that's it.

Perfect. It'll be like, I don't think this is right. Like. I think this is the thing, and then you can have like an actual discussion in the issue and you can come to a conclusion and everyone can see it because it's, you know, just GitHub and all in the open and their data, quite frankly, is the best vulnerability data that exists today.

But unfortunately, it's just a subset because obviously they do not have infinite resources to do this. So they purposely constrain themselves. But because of that, it is just so good. It's amazing.

Jonathan: Yeah. So I have tried, by the way. I cover vulnerabilities and I've found typos. When in particular comes to mind, I found a typo in a CVE on, I forget whether this is before or after the switchover.

So, you know, maybe it was at MITRE or maybe it was at NIST, you know, things have moved around a little bit over the last couple of years. I found a, a, a, a typo in what version fixed a vulnerability. Like you go to the vendor's website and they say one thing, you go to the CVE and it says something else.

So like, Oh, Hey, I'll get this fixed here. Shoot an email off real quick. Surely they'll get it fixed. I still haven't heard back. I bet it's still wrong.

Josh: Yes. So I've, I used to do this all the time where I would, I would try to get things fixed and I have quite frankly, just given up because what it usually is, is you say, Hey, MITRE.

And yes, you have, like, you can look at the website, you can see the version, right? Like, I have definitive proof this version is wrong.

Jonathan: Yeah.

Josh: And I remember the, the one, like, blazing into my mind when I just threw my hands in the air and gave up is I emailed MITRE, and I'm like, hey, look at this, like, here it is, and they said, oh, no, you have to go to the CNA that filed that, but who's that?

Like, and I, you couldn't tell you now they're putting the CNA details in the CVE data. Thank goodness. But at that time it wasn't there. So I'm like, who's the fricking CNA? And they're like, oh, it's so and so. It's like, so I have to go call, like email this person. And I sent them an email and they never replied.

And I'm like, you know what? This is ridiculous. I'm just, I'm out. So now when I see problems, I just go to GitHub. I make sure it's correct there. And like, I'm happy. And that works.

Jonathan: Yeah. So one of the, one of the other things that has been a big problem for the industry is the, um, the CVSS system, the numbering system, like how, how severe a problem is.

And there's this, what, what, how shall we say it? Um, it's almost a conflict of interest because like the vulnerability researcher is looking for the highest score they can get. Because sometimes they get more payout, right? Like sometimes you get a higher payout because you're working on a bug bounty. So if you find a CVSS 10, you're going to make thousands of dollars.

Whereas if you find a CVSS 5. 5, you'll get a couple of hundred dollars. So like they are, they are, um, there's an incentive. They are incentivized to get a higher. CVSS. And then on the other side of it, you've got, in some cases, the vendors that it's like, they do not want to make the news for having a CVSS of a 10.

0. They want that to be a 5. something. And, oh, getting those fixed when they're wrong has been such a problem over the years, because sometimes they are wildly wrong. Yes.

Josh: Yes, you're exactly correct. And you have researchers will, they'll, they'll mis describe the, the vulnerability. And, and the thing is like NIST has a very limited window into what things should be rated.

And additionally, we're like using CVSS completely wrong. It's not meant to be a, a just like raw way to score this stuff and then rank it. There's, you're supposed to add in other information. Like, how am I using it in my environment? Like, you know what I mean? Yeah. What is, what does it look like? Is it, is it being exploited?

There's actually a system that's newer ish called EPSS, Exploit Prediction Security System. EPSS is really cool. What they do is they, uh, it's part of, um, first. org is where they, they kind of, that's where their home is. And they're collecting like all of this data. And it's not all public data, which, It's like all of this threat data is super expensive and the fact that they're getting threat data for free is amazing.

So I'm not going to harp on that, but they're kind of getting all this threat data. And what the EPSS score is meant to be is it's meant to be like, what are the odds this vulnerability will be exploited in the next 30 days? And so the intent being things with higher odds, those are things you should look at.

And now that's an example where you can like, if it has a high CVSS and a high EPSS, now we're, now we're talking like that's something to look at. But if it has a high CVSS and a low EPSS, it's like, eh, we should probably double check what's going on here. Cause this is weird. And so EPSS, I think has a lot of.

I have a lot of hope for it, and they're on, I think, version 2 of the model, version 3 is coming out very soon, I believe, I might be wrong, version 3 or 4, I can't remember, but they have graphs that show how accurate it's been, and when it started out, it kind of sucked, as all things do. Do when they begin, but they're actually doing a really good job now where their, their prediction models are quite good.

And they update it once a day. This is the same idea of like, you can scan your S bomb for vulnerabilities every day, and it could be different. The EPSS score is going to change every day just because like the threat landscape is constantly changing out there in the universe. And so this is one of those things, like you can look at your EPSS score functionally every day, and you can get a better idea of like, what do I need to work on?

What do I need to worry about? Whereas like, again, CVSS. Once you have a CVSS score, it's kind of set in stone, right? Cause that's just how it works. And that doesn't always make sense. Things change. The universe changes around us and we have to try to adapt to that.

Jonathan: I was, I was pretty excited to see the new CVSS 4.

0, the new revision of it. That's supposed to be a little smarter and, you know, supposed to not give you 10. 0 scores for things that are really, really trivial and not actually that serious. And so every time I go to cover a CVE, I go on, Oh, what's the, what's the CVSS 4. 0 score? Surely, by now, it's been out long enough that somebody's using it.

I have not seen, I have not seen anybody use. The CVS has 4. 0 scoring for anything. So like it, it theoretically exists.

Josh: So the vendors hate it, actually. The secret there is, so for, for we'll say low and moderate things, it definitely bump, it kind of, it brings everything to the middle is what I would say.

The high things are coming down, but the low things are coming up. So the vendors hate it because now things that might've been low or moderate are being bumped into moderate or importance. And like you said, The vendors have an incentive to lower the severity of things. So from their perspective, CVSS4 is an anti feature.

Because it's going to actually create more work for them and their customers.

Dan: Makes sense.

Josh: Yeah, yeah.

Dan: Yeah, so, so Josh, how many people are working on, on SIFT and, and GRIPE? What I'm interested in is the community engagement. Are you getting code in from outside of, of the company? Uh, are you getting pull requests, are you taking pull requests, that sort of thing?

Yeah,

Josh: Yeah, oh for sure. We definitely are like we love pull requests. We love the community. We've got a discourse We've got it's all a github. We've got community meetup every other week I think it is the the open source team with with popy They do like a live stream once a week where they kind of do like some of their bug You know, wrangling to figure out what to, what to do.

I mean, we've got a team of, so I think we have a dedicated team of, I don't know, is it five people, six people? I can't remember exactly what it is, but then obviously you've got more anchor involved as well. And then, yeah, I think we have like hundreds of contributors. I don't remember the number off the top of my head.

I'm, I'm ill prepared for that question, but it's, It's, it, it's pretty impressive, honestly. And there's, we're, we're seeing more and more community involvement, which is a double edged sword though, right? Because you get more bugs, you get more pull requests and it's not like, I always love this. Oh, open source.

We'll give, do our free work for us, but like, it's free, like a puppy. Cause you have to, you know, you have to read the bugs. You have to triage of the bugs. You have to. look at the pull requests and be like, Oh, we'd like to do it this way instead, whatever. And, you know, it's, it's a lot of work, but it's really cool because yeah, we're seeing, we're seeing some pretty cool contributions.

In fact, where there's, um, uh, there's a thing called Bitnami, which is like a container runtime universe. Like there's, they just submitted a, a giant patch to SIFT to properly catalog and index their containers, which is amazing, you know, like that's exactly what we want. We've seen a couple other organizations, you know, Give us, um, we call them catalogers in SIFT, which is where we have the ability to be like, oh, this is Like, um, I think it was just a NET that got added where it's like, this is, you know, a NuGet repository.

So figure out what to do, NuGet cataloger. And, and yeah, like we're, we're pretty pleased with, with how it's going. And, and like, that's the dream, right? Is you start an open source project and ideally you want a community to build up around it, which makes us very excited and very happy.

Dan: Hmm. And I, I know I don't have, I shouldn't have, I shouldn't really have to ask this question on an open source podcast, but I want to anyway.

'cause I think it's always great to get people's opinions. What's the benefits to, uh, Ancor in releasing these things as open source? Because we, we preach it all the time here, but we've, we'd like to. So let you preach it, I suppose. All

Josh: right, all right. So I've got, I spent a decade at Red Hat

Jonathan: and

Josh: I, I spent time at a little Linux startup called Progeny Linux Systems before that, which the, uh, Ian Murdoch, the founder of Debian created a startup and we'll say it wasn't real successful, but, uh, But anyway, so I spent a lot of time at Red Hat.

So like, my career is steeped in open source. And I remember my first job out of college was at a company that wrote life insurance software in COBOL. And this was like right after the dot com crash, so it was dire. I'm like, I will literally take any job. And And I did, but I remember they, like, I'll never forget it.

You know, there were people there that were like, this open source stuff is stupid and it's crap and it's never going to catch on and you're wasting your life on it. And of course being young and, and filled with, you know, optimism, I was like, Oh no, no, it's, it's the future. It's going to win. And like, thank goodness it did.

Cause I'd be, you know, probably broken homeless otherwise, but, but, so the thing is like, Angkor has a bunch of red hat DNA in its blood. Like there's a bunch of red hat folks that are at the company. And so it's. Thank goodness, not something we had to like educate or understand, but fundamentally. So the idea is we've got the, like the, the bottom of the pyramid is open source.

We have sift and gripe and we have a bunch of like this tool called grant. We've got a whole bunch of other like open source libraries that kind of drive all of this stuff. And then, The intent being by making a tool like sift and a tool like gripe open source, you, you get community contributions are awesome and we'd love that, but you also get a certain amount of just kind of engagement in the wider universe, right?

Where you have people running these tools, you have open source projects running these tools. You've got, um, we just had a Allen actually webinar with Google where Google is scanning. Like all of their stuff with SIFT, they're literally scanning, you know, millions and millions of things every day with SIFT.

Like, that's really cool. They have good feedback, obviously, as well. And that's like the power of open source. Right. And then what we did is we took kind of those. Core functions and we built product on top of that because it's easy to say, Oh, you're going to run SIFT across your entire infrastructure.

And if you're Google, you can do that because you have an army of engineers. But if you're a small company working on something or even a medium or even large company working on something, you have to ask, like, is it worth just buying a solution that will do all of the plumbing and all of the monitoring and all of the policy and all that stuff?

Or. Should I hire an army of people that are going to do this as well that will probably cost somewhere between three and ten times as much as just buying a product. And so that's kind of like that very red hat model of there's like that core base functionality and then there's things, there's value added to the top and we've got, you know, we've got support, you've got services, there's, we can help you figure this stuff out because it's weird and hard and, and if there's a problem with your data, like, Open and support issue.

And it's going to come to my team and we're going to tell you what's going on. And often we just fix it because it's like, oh yeah, that data is bad. We'll go fix it. And so that's kind of the beauty of that. And yeah, that's what. It's been a treat being there because I don't have to fight for whether they should or shouldn't be open source or whether this is or isn't a good idea, like everyone totally gets it.

It's awesome.

Dan: That is very cool. Um, so I always like to, to, to ask people how they got involved in, in all of this. How did you get into what you do now? How did you get into open source initially? And how did you get into computing even? How far can we go back with this? You can go pretty far back,

Josh: man. I mean, I've been, I mean, I'm, I'm old.

I've been doing this for a long time. You know, I, I grew up without a computer in the house. Punch cards, was it? No, not quite, not quite that far. Not the punch cards. No, no, I don't. I didn't have access to punch cards. I would have gladly used them if I could have. But no, I mean, the first computer that came into my house was like a, it was a 386 running MS DOS.

Right. So, I mean, we're talking like early nineties, somewhere in there. And I just, it was, I loved computers. I've always loved computers and I'll never forget like my. You'd go to school and you know us old geezers We had like a couple Apple twos at the school and we do things with them and stuff And I remember my my parents came home from a teacher conference one time and they were started yelling at me And I'm like what and they were like you have to stop helping them with the computers because you don't know what you're doing I'm like, what do you mean?

I know what I'm doing. It's like I knew more about the Apple two than the teachers did. Cause like I would read anything I could get my hands on. It talked about computers. Right. And I mean, I remember people laughing when I tell this story, but like, I used to write down basic programs on a notebook paper and I would like, I would run the program in my head.

I didn't have a computer. So I'd be like, what was this program going to do if we run it? And you know, I'd write it out and figure out what the program would do. Which nothing long, of course. course, but like, I, I just always loved it. And so, you know, computing will say for a while, and then this Linux thing you start hearing about, and it's like, wait, I can get free stuff.

Like, well, I mean, we'll say a lot of software was free back then, if you knew the right people, but this is like, you know, free compilers, I can get, you know, a nice. Uh, a Unix terminal that doesn't cost any money and it works and it's awesome. And then I think the real thing that tipped me over the edge was I go to college and they've got sun workstations everywhere, right?

Cause I'm an engineer. It's just sun gear every like the sun, a sun workstation, like 45, 000, right? They were crazy expensive. It was 45, 000 in nineties money. So we're talking a lot of money today and I'll never forget. It was like, there was this. This stuff I wanted to do where there were programs I wanted to run, you had the ability to, you know, do like X windows between systems where you could like run something on the server and you get the X windows.

Window on your system. And I was like, Linux, let's me do all this for nothing. Like this is a no brainer. And so it was really, I mean, I mean, honestly, it was just, I was too poor to buy a sun workstation. That's kind of what really got me into it. If I, if I had a ton of money, I'd be running sun gear at the time, but it was Linux, but then the, you know, the thing is it just, it, it, it gets you, right.

You get that bug of like, oh, I can talk to the people who do this. I can send patches. I can work with them. I can build my own kernel and. You know, completely trash my machine and learn more than I can imagine in the process. And, and it just, I don't know, I caught that bug and it's just, I love the community.

I love how it works. I love everyone helping each other. It's just everything about this universe has been a treat. And like, I, I've built my life and my career around it. And I've, it's been lovely. I, I have absolutely no complaints or regrets and I love open source. It's, it's absolutely amazing. And. Every day I just think like what's the next crazy thing that's going to happen and I'm never disappointed.

We'll say that but yeah, it's fabulous.

Dan: Awesome. So, uh, I just came up when you were saying that you got bitten by the bug. I was thinking bitten by the penguin by a penguin. The penguins don't have teeth though, do they? So I don't know how quite that would work, but

Josh: they have like sharp. Barbs, I saw a picture one time inside of a penguin's mouth.

It's like a Lovecraftian horror.

Jonathan: Oh, that's great. All right. We are, boy, we're starting to run out of time here. Um, is there anything that we didn't ask you about that you want to want to cover and make sure folks know about? Many

Josh: things.

Jonathan: Yeah, I'm sure.

Josh: No, it's, I mean, I think the one thing I would say is, so I, I adore, this universe, like the open source security, all that stuff.

And if like anything I've talked about catches your interest and it's a topic you'd want to discuss or, or learn more about or anything, like reach out. I love talking about this stuff. I will talk your ear off if I can. I mean, I have a podcast just cause I love it. I've actually have two podcasts. We talked about the open source security podcast.

My other one, I call Hacker History, where I just take like old people and I say, tell me your hacker story and that's it. And then that's it. We learn about crazy things from the days of yore. And it's, that was super fun. It's, it's, it's, it's tough to find guests, but I'm always looking for guests. So if anyone wants to tell their story, like hit me up there too.

And yeah, it's just, that's kind of the biggest thing is just, if you want to have a chat ever reach out, I would absolutely love to, cause there's so much going on, there's so much interesting stuff and it's just so much fun to talk about, I love it.

Jonathan: I was going to have you make sure and plug your podcast, but I think you just snuck that in there, didn't you?

I did.

Josh: I did. I'm pretty shameless like that, so.

Jonathan: A skilled, a skilled commentator.

Dan: That's

Jonathan: right.

Dan: In security speak, it was the payload in the, uh. That's right. Yes, exactly. Quickly run off the end of my security knowledge there. I went payload. Yeah. Okay. Show code has been executed. Yeah, that's right.

Jonathan: All right.

So, uh, what's your, what's your favorite then, uh, scripting language and, uh, editor?

Josh: Okay. So I, I was alive for the VI versus Emacs wars and I still run VI all the time. It is. Probably my favorite editor. I think I use VS code, we'll say for anything remotely difficult, but VI still is like, that's where it's at.

It's amazing.

Jonathan: It's amazing. How many people answer VS code to that question either directly or indirectly. It's so good,

Josh: but it works. It does what I need.

Dan: Yeah. It's a good tool.

Josh: And then scripting language. So I, I know many of your guests, because I, I listened to the show. There's always a, is Python a scripting language?

And I, I, I've been thinking about this like since we, you booked me, I'm like, should I say Python? I'm like, I gotta say Python. Cause the thing is, every problem I have at this point. Ends up with a Python script to do it, you know, like I'd love to say shell or like bash or whatever, but it's Python. Like it's funny.

It's it's become like the, you know, Swiss army sledgehammer of programming. I think it little things are big things that it all works. It's it's kind of a replacing Pearl

Dan: for that position.

Josh: Oh, yeah,

Dan: for sure. Somewhere. Randall just exploded. I thought I had a faint explosion in the background. Yeah. Yeah,

Jonathan: that was

Dan: Randall.

Jonathan: We have discovered recently that any language could be a scripting language if you really want it too badly enough. We had somebody doing the Java for scripting, for system scripting. There

Josh: you

Jonathan: go. I remember C

Josh: Shell, right?

Jonathan: Yeah. Yeah. Yeah. It exists. It does. I, man, I am, I am in a group right now that, uh, one of the guys is a huge C sharp fan and I'm sure he, he does.

No, he actually, he does. I'm sure he uses that C

Josh: sharp C shell. There was a shell from long ago that had a C syntax, but it was a shell. I know. Right? Like I remember that too. Yeah. We'll say corn shell and flood us all with horrible memories of each like some pain with your pain.

Jonathan: Oh, that's right. All right, this has been this has been great. I don't want to let you go. It's been so much fun We'll have to head back. I actually wrote a note in our back channel Like next time we need a uh, we need to we need to do a roundtable because a guest flakes on us I'll have to reach out to to this guy and I think our guest from last week.

We also made that note about so Maybe we'll do that, maybe we'll have a running list of guests and co hosts and when we have to scramble we'll just shoot a mass email out. So this is what we're doing, this is where we're doing it. Awesome. Uh, it'll be fun. Alright, thank you man, I appreciate it so much, it's been glad to, it's been really good to talk to you, uh, Josh Bressers everybody, thanks for being here.

Thank you much. Alright. Uh, what do you think, uh,

Dan: what do you think Dan? I thought it was great. What a great discussion. As you said, uh, we, we could, we, we seem to say this on every show, but we could have kept going for a long time now talking about all kinds. I really thought we're going to get into penguins at the end there and whether, you know, with the whole penguins and teeth and all that, I was, I could have gone in that direction.

Um, I was, yeah, I was interested. Something I'll, I'll ask, I'll have to ask Josh when he's talking, I'll check his podcast out, the hacker history one, because I'm really interested in the whole phone freak thing and from the seventies and the sixties and all of that original. Audio hacking of, of phone switches and so on.

So, so I'll ask him if he's got any, had any of those kind of people on Yeah. Uh, yeah. Fascinating guest. Fascinating guest.

Jonathan: Yeah. That, that is, that is really interesting history. And if he hasn't, we'll have to make sure and request some. Um, there are, there are still some people out there that are active. Um, right, right.

That, uh, one guy that I've, I follow on Twitter and he, he. Evan Dorbel is what he goes by and just some great stuff. The cool thing about Evan is that he, he was also like an audio, he was a phone nerd, but also an audio nerd. And so he has recordings of the phone system from way back in the day. And so like, he's got these narrated clips of.

He, this is, this is what you hear when you do this and this is what the equipment on the other end is actually doing. And it's, it's really cool. So we'll have clear all clips from like all the different exchanges and like, here's how this one sounds different from this one. That's why. So Josh, I know he's still listening to us.

Josh, if you haven't had him on yet, see about Evan Doorbell and some of those guys. That'd be a lot of fun. Uh, alright, uh, do you wanna plug anything Dan?

Dan: Um, I want to, yeah, I mean, uh, go to my website, danlynch. org, if you want to find out what I'm up to, and, um, I'm going to do a community style plug, because, um, I run our, I'm one of the people who run our Linux user group in this area, and I think everyone should, you know, uh, the whole Linux user group movement kind of has lost, you know, Momentum in the U.

K. And I'd like to say, you know, if you're interested in Linux, have a look, look dot org dot U. K. Find out where your local log is. Or if you're near the northwest, come to Liverpool. We run Liverpool log. We have stuff on every month. So check that out.

Jonathan: The Linux user group scene has just moved online. It's still, it's still alive and well, it's just It happens in Slack and Discord and a little bit in the IRC and Telegram and

Dan: Signal.

I have been to Linux user group meetings with people all like 10 people sat in a room all hunched over computers like this talking to each other online and I think, why are you here? You could have done this from anywhere, you know, you're not even talking to each other, you know, anyway, that's just my own personal gripe.

Jonathan: Yeah, we, we nerds are, we're a weird breed sometimes. All right. Well, thank you, man, for being here. I have a couple of things that I want to plug, and, uh, of course, the first off is Hackaday. We've got my security column. If, if this is not enough security for you, you can get even more of it, week to week coverage there.

It goes live every Friday, uh, and we also sure appreciate Hackaday being the home for Floss Weekly. Um, there's also the Untitled Linux Show over at Twit. If Linux is more your thing, you can find our weekly musings about that over there. Uh, we do not yet have a guest for this upcoming week. on the 5th, but on November the 12th, we're talking with Frank Delaporte about Pi4j, which is another Java thing, but it's, it's doing, uh, it's doing Java on the Raspberry Pi, and specifically, it's about being able to access all of those peripherals, you know, GPIO and SPI and I2C and UART with, with Java bindings.

And so that, that sort of, that sort of tickles my, my interest, interested bone I've been working with. Some of that stuff myself here recently. Um, and then one other thing that I will let folks know about is, uh, later today in my time, I don't know when all of this will go live, but, uh, I'm going to be interviewing with Brody over.

He's got a, the YouTube channel. Um, is it just Brody Brody talks Linux? I don't, I forget what his channel is actually called. You can search for Brody. You'll find him. One of the other things he does though, is he does tech over T where he interviews folks. And I. commented on something on, on Twitter slash X and he sent me a PM.

He's like, Oh, by the way, I've been meaning to invite you to come to this. So that is happening later today. So watch for that. If that's something that you will find interesting. Uh, we sure appreciate everybody that's here. Those that caught us live, those in the discord, making notes and commenting back and forth and all of y'all that get us on the download.

Sure. Appreciate it. We will see you next week on Floss Weekly.

This week Jonathan and Dan chat with Josh Bressers, VP of Security at Anchore, and host of the Open Source Security and Hacker History podcasts. We talk security, SBOMs, and how Josh almost became a Sun fan instead of a Linux geek.

• https://opensourcesecurity.io/
• https://hackerhistory.com/
• https://infosec.exchange/@joshbressers
• https://anchore.com/

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

This week Jonathan and David chat with James Smith about Manyfold, the self-hosted 3d print digital asset manager that's on the Fediverse!

• https://floppy.org.uk/
• https://manyfold.app
• https://github.com/manyfold3d/manyfold

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week David joins me and we talk with James Smith about Manifold, which is the host your own repository for 3D printing. It's just recently been wired up to talk with the rest of the Fediverse and it's got a lot of really neat features going on you don't want to miss it, so stay tuned.

This is Floss Weekly episode 806, recorded October 22nd. Manifold, the dopamine of open source.

It's time for Floss Weekly. That's the show about free Libre and open source hardware, hardware, software, software today, software about hardware today. I'm your host, Jonathan Bennett, the oftentimes tongue tied, and I've got a co host with me, of course. I've got Mr. David Ruggles. Welcome, sir. And we're, we're talking today about, well, about Activity Pub once again, but not Mastodon, not directly Mastodon this week.

Talking about Activity Pub and 3D printing. David, I, I know, let's see, you're on Mastodon already? Yes. But not, you don't have a 3D printer yet?

David: No, I do not.

Jonathan: Okay. But

David: it's something I'm interested in, it's just one of those things that it's down on the priority list and there's never enough time.

Jonathan: We are going to do our best, I think, to talk you into getting on the 3D print bandwagon today.

For this, I'm sorry, and you're welcome. Um, let's, let's go ahead and bring our guest on as well. It's Mr. James Smith, who is part of the Minifold project, which is all about curating and storing 3D printer models, and then sharing some of that metadata on, on Mastodon, on ActivityPub. It's it maps somewhere in there.

It's something like that. James, welcome to the show.

James: Hello. Thank you very much for having me.

Jonathan: It's great to have you here. Yeah. I was, I was saying before the show got started that I was, I was excited about this because I like Macedon. I like activity pub just like on, on its own. The, the idea of it, I think is really great.

And then I am also something of a 3d printing nerd. And you put the two together and it's, it's just, it's just great. I recently picked up, and we're going to try not to make this all about 3D printing, we're going to talk about the technology too, but I recently picked up a a Bamboo Labs Mini, and it lets me do some really fun stuff like this.

You know, I got it with the multi material, and so, you know, you can do a single print that's got a couple of colors in it, and so I've got a, it's actually a Meshtastic radio I printed out a case for, and I've just been going nuts with stuff like that. Because it's such a huge upgrade over the old printer that I had.

And I was, I was telling, I was telling James before the show I, one of the other things I do is I print out some miniatures for tabletop gaming. And I purchased a I purchased a miniature from HeroForge back several years ago. And I, I printed it on my, as has been called the, my first printer. The, the very cheap model that I started with, and then I forgot about it.

And then I got the new bamboo and it's, Oh, I need to go back and try to print that model out again. Oh, where is it? That was like, Two desktops ago. And so I have on the other side of my desk, right over there on the floor, my old, huge desktop is out and running to be able to pull model files and other files off of it now.

And I was telling James before the show, it's like. Why hasn't somebody come up with a repository you host your own yourself that you can put all your models into And that's what many fold is isn't it?

James: That's it. That's it. I mean you've you've described the journey that That made it happen. I think really which is that I've got too many models lying around.

I don't know where any of them are or what they are so I need some way of organizing them And that that became A little side project and that grew into Manifold.

Jonathan: No, so it is, it is Manifold, not Minifold.

James: It's, it's, it's an interesting one. It's kind of between the two because it's, the name comes from like, Manifold being, you know a property of a 3D If it's got no holes and things like that, it's what's known as manifold.

And it's many because there's lots of instances and things like that. So I, I kind of sort of. Drive straight down the middle between the two, but I think, as I said earlier, it's I, I use it written down a lot more than I say it. So I, I don't, whatever. It's fine. There, there,

Jonathan: there is a long tradition of open source projects having names that are very easy to write out and sort of difficult to pronounce with various takes on how to pronounce them.

James: I think of it as a, it's a wide, it's a wide cluster of how to pronounce it. It's fine.

Jonathan: And, and, and manifold has a tie in with activity pub. Is it activity pub or Macedon in particular?

James: No, it's activity pub. So this, this is really, really new. So manifold's been building for three years or so. I've been building it.

Like I say, initially as a side project and and then managed to get some some open source funding for it. But one of the things that I got a funding for was to add in the activity pub integration so that not only can you have your own sort of collection and and organize it and things like that, but also you can connect it up to others.

So yeah, it's pretty new. We properly, properly joined the Fediverse, I suppose, couple of weeks ago. Which is, which is really cool.

Jonathan: Oh, wow. It's that is very new then. So this was not, this was not an original feature. Okay. Okay. So

James: no, no, no, it was always something I thought like, Oh, it'd be cool if I could make it do that, but yeah, it's, it's taken a while to get here.

But but yeah, it does it does do it now. Yeah. So what is the, what does the

Jonathan: integration look like? It's like, what, what what gets posted out to the Fediverse? Is it like, I printed this, I printed this and here's a picture of it. You know, what, what does that look like?

James: So at the minute. So we're taking a very sort of agile approach to to getting features out at the moment.

We've got the sort of the first things that will that will happen in terms of ActivityPub is that if a creator adds a new model, That will get posted as a as a note across on things like Mastodon and so on. So you can subscribe to creators on public manifold instances. You can subscribe to them from Mastodon or other, any other Fediverse software, and you'll get the notes when they release things.

There's a lot more to be added in there. One of the things that the the instances will be able to do is actually directly sort of follow both creators, but individual models and updates collections, things like that to to follow those on other services more directly. So it's not just like, oh yes in my Macedon feed I saw a nice little post.

It's actually like a link. You know, it's like a single archive almost, so that's what we're building towards.

Jonathan: Yeah. Is all of this opt in or are we sort of putting models out there? It's all okay.

James: Oh, no, no. Very opt in. Okay. The everything above Running it as a single user on a private network is, is opt in and controlled by, by various feature flags and things.

It was originally built as a private archive. So I built it as a, a thing to literally just post on a Raspberry Pi in the corner of the room there and be able to catalog a directory full of files in in my browser. But then we've added in multi user capability Lots of other, lots of lots of other bits and pieces.

The, the federation and, and all these things are all optional features. So no, it's, and it will only happen for public content as well. So we've got very fine grained Google doc style permissions on things. So you can say actually, yes, I want to make this item, this creator, this item public. So if you were to look at the, uh, my own.

Manifold instance, which is publicly available on the internet. You can see my stuff that I've published as if it was thingiverse or one of the other model hosting sites, but you can't see my private archive, which is also on there.

Jonathan: Yeah, I was that that was a question that I had. I was going to ask because not every license for STLs.

Lend itself to sharing publicly like that.

James: No, absolutely. I've, like I say, I'm using it to manage my my models that I've published as Creative Commons and things like that. But also it's, it's stuff that I've bought from creators on Patreon and things like that which are, yeah, which are not for sharing, so.

Yeah, handles both of those situations.

Jonathan: Yeah, now is it Is this simply for STLs or are there other file types? Can we, can we load G code in there? Can you do object files?

James: Yeah, it will. Yeah. So a whole load of of model files are supported STL OBJ, lots of sort of older formats 3MF which is finally a decent format for these things.

And a lot of others, but also images last couple of weeks ago, we added a PDF and text files and video, even because some models come with assembly instructions and you want to have those there alongside it. So but yeah, this yeah, all sorts of things. G code, sCAD, FreeCAD. All sorts of stuff.

Jonathan: Now, I've, I've got to ask, this is something that you see some of the other sites will do does, does Minifold render? Like, if you give it an STL, will it give you a preview of what it looks like?

James: Yeah, so the primary, the primary, the thing I did first, where it started was, I want to point, and this is why I did it in the browser, really, it sort of drove the whole thing, I want to point, A server at that folder.

And I want to see the models that are in it and render them. So it uses three JS. If you, if you go to look at a model it, you get a list of all of the. individual files that are there, it renders them all, you can play with them, zoom around, find exactly which one you want, then grab that file to go and take off to printing, which for, for things with lots and lots of parts in is, is very useful.

Jonathan: In just a second here I'm going to give it to David and I'm going to let him ask the questions that I'm not thinking to ask because I am already a 3D printer nerd. First though, I've gotta, I've gotta This is, this is crazy. And you're going to really, really impress me if you answer yes. Is it also a slicer?

Does it slice as well as dice?

James: No. No, sadly not. No, I mean, the slicer is a massive bit of software. And there's some really good ones out there. So that would be reinventing the wheel. People have asked. There is work that people have done in browser based slicers. I look at that as the sort of thing where it'd be really nice to integrate with some of those things.

So it'd be really nice to be able to have things automatically open up in your slicer of choice. Or to be able, if you do have G code in it, to be able to automatically send that over to OctoPrint or something like that. People have asked about things like the The thingiverse customizer where you could actually do some of the editing of models The scope I think for sort of maybe plugins and things that do that.

Yeah, I I tend to be not a fan of things that grow to encompass every possible feature. It's But if I can make plugins work for stuff then Then that would be that would be interesting

Jonathan: I could, I could even see something like editing of OpenSCAD scripts.

James: Yeah, that's exactly, yeah. Exactly what people were suggesting with the customizer approach.

Jonathan: Yeah, no, that would, that would really be cool. You could also imagine a flow where you've got your, your models saved, and then you open one of them. It moves it to your computer, opens it in, Your slicer, your preferred slicer, Cura or whatever, you render it, you slice it, and then it uploads it back to Minifold.

And then from there, you've got a button to press that'll kick it over to your printer. Like, you can imagine this, this workflow that would be really, really cool. But,

James: it's, it's, I, I've recently switched over to OrcaSlicer, and one of the immediate things that, I loved about it was that when you hit the print button and it sends the file to your printer, it's got a, it's got a tab in the UI for the web interface.

If you're running OctoPrint or if I am, I'm running Mainsail,

The Clipper UI online. So right there in the slicer, it's taken me to The web view of my printer, which is amazing. It's, it's integrated that workflow. So now actually what I want is like, what about the first tab there that actually is pulling in a manifold instance or something like that?

So. Yeah, there's lots of possibilities, I think.

Jonathan: Yeah, exciting, exciting ideas for the future.

James: Too many ideas. That is usually the problem. Yes, that is

Jonathan: usually the problem with open source. David, you want to jump in and ask the, ask some questions that I'm just glossing over?

David: Sure, I can ask the dumb questions, or the any man questions.

So, I've got CAD experience, it's mostly 2D stuff. And, I don't have a 3D printer. It's something that I've looked at many times, and it's just down on my priority list. But, so, Some of the terminology and stuff, I'm familiar with some of it, I'm not. First question I have, and maybe this is for both of you, but you said slicer.

I'm assuming that is something that takes a 3D object and creates like, parts of it so that you can print it out where you might not be able to print the entire 3D object because of its complicatedness, so, so you can basically print parts of it and assemble it after the fact. Is that what a slicer is?

James: Nearly. Nearly. The slicer is actually, so all, all 3D printers, I think all of them at the moment anyway work in layers. So if you've got a a filament printer, you've got a, an extruder and it's, it's, it's drawing a path around like, like there's a lot of 2d CAD stuff like laser cutter or anything like that.

In fact, the the language I think is the same as is used for for things like laser cutters. So it's generating this tool path. And if you've if you've got a resin printer, which is often doing like a it works. whereby you have a screen which is curing some UV resin, and you do that layer by layer and it slowly sort of builds up the model.

And the slicer is what turns that 3D model into a series of Basically,

David: okay. So does your slicer maybe resolution or thickness of each slice? Is that dependent on your printer functionality?

James: Yeah. Yeah, that's right. And different printers will go to different different resolutions. And to be honest, the amount of time that you want to spend as well.

It's I can go down to 0. 04 millimeter. That's a very slow print though. It's a very slow print, yeah. I don't do circle

David: back to what you were talking about before about integrating that slicer functionality are slicers something that is printer dependent or is pretty much any slicer able to work with any printer?

James: The good question. The two, I think, I think on a consumer level, there's two big types of printer. You've got the filament type known as FDM fused deposition model. Sorry. My daughter's knocking on the window. That's

Jonathan: fine. Bye.

James: It's better than a barging through the door, which is what I thought she was going to do. Yeah, you've got the the FDM printers and then you've got the resin and they work in two different ways. One is extruding this big wiggly line of plastic and one is building it up almost like a, you know, like a flip book.

Right. So I think in, in, Until fairly recently, you'd probably have a different bit of software for both. There are now some that are combining those two into one bit of software. But there's lots of different Oh, and all the printers will, will have like, Oh yeah, here's the slicer that we recommend and, and things like that.

There's lots of them out there. There are about Five different forks and minor variants of one, one in particular, which is just confusing to me, but it's but yeah, choosing one that, that works well that you like the UI of things like that. They, but they're all The resin printers are a bit different, but all the filament printers tend to work on on G code, which is this language for, like I said, originally for CNC stuff.

So it's that's a, that's a standard. So they're all creating that kind of kind of thing. So they're pretty cross compatible.

David: Okay, so it does seem like there may be the possibility of somebody developing a plugin at some point that would have that slicer functionality. It's possible, yeah.

James: I think Octoprint had one.

I don't know, I've never actually tried it, but Octoprint did have one, I think at one point. Whether it's still there, I don't know. So, it's been done.

David: So we could go a lot of different directions with this. I read through your bio and the website and you've got just an interesting history. So next time we have a round table and we're missing a guest, I recommend that we see if he's available and we just go into some of these other things.

Come

James: and talk about anything, yeah. Yeah, I've done, I've done, you know what we said about too many ideas, right? That's, yeah. That's, that's me. So yeah, I've done too many different things over the years.

David: You combine the creativeness with the open source and the possibilities are endless. And then you get somebody with the mind for it and you want to try every one of those endless possibilities.

James: Yeah, the difficulty is not starting projects, it's getting them, yeah, remotely finished. I mean, that's been really nice about this one is that it's I have, I'm not going to say I've built a community around it because I don't really think I did anything, but there's a community has appeared. There was one day back in May, I think that someone popped up in the support channel that we have on matrix and somebody else answered the question.

And I was like, Oh my God, that's when you know,

Jonathan: you've made it.

James: Oh, how did that happen? This, this has never happened on any project I've ever done before. So Having, having community to, to keep you engaged and actually get a bit of the dopamine hit from delivering things is really nice. Most of what I do has failed over the years.

So

David: it's, it's not failure. It's it's learning what doesn't necessarily work in the community. Absolutely. Oh, so how large is your community now? Do you have any feel for that?

James: Do you know, I, that's, that's a really good question. I'm in a way, I'm really proud to say, I don't know because knowing too much about them is, is kind of, you know, a bit of an antithesis of the like self hosting kind of control your own information world.

So I'm taking that as a strength. No, but I mean, there's, there's lots of a few, a few sort of proxy measures, I suppose. I mean, that's about. I don't know, 60 people in the support chat. Normally we've got a few hundred followers now on in, in the Fediverse. One of the things I don't know is how many there are, how many instances there are.

No idea. I I've made a I built an anonymous tracking thing so that people could report. How that their thing is running and how many of them there are first of all, it's really difficult to get people to click a button that is completely optional and, and not shoved in their face. And secondly, I made it so privacy enhancing or privacy preserving that actually, whenever I change the code, it forgets everything.

And that just starts from zero again. So it's very much a proxy. The, the. One interesting question, which I don't know if anybody listening knows, if there's any way I can work out from the Docker download stats, the Docker pull numbers, what that actually means. Is it, you know, if I've got like 50 polls, is that going to be one machine pulling it every, you know, 10 minutes?

Cause somebody has got an auto update on or is it 50? I don't know, but I do know that we get Docker pulls within moments of each release. And, you know, we'll have a couple of thousand over a week. So, but I don't know what that means, but it's, it's going up and that's a good thing. So

Jonathan: yeah. I'm pretty sure it's accurate.

Docker pulls are actual downloads, not just checks for version.

James: Yeah,

Jonathan: that seems right. David.

David: Yeah, I've done some, I kind of don't want to assume that and it only pulls it in when it's built. So, I mean, you might have some script that's checking automatically for updates and then pulling it down, but it's still going to be a new download.

James: Yeah, I, it's kind of, I don't, I'm not sure whether I trust GitHub's counting because it's on the GitHub container registry. Do I trust that their number is actually pulls or that it is? People have just asked what the latest version is. I don't know. So if anyone knows that, let me know. I'd love to know.

But no, we've got enough community going on that that it's keeping me going and it's, it's building momentum and we get contributions, code contributions, lots of feature requests. People report bugs really quickly, which is great. That's actually a good

David: sign of it really is right there.

James: Yeah.

Absolutely. Yeah. Yeah, there was one I, I accidentally pushed in the the previous version. And I think within about two minutes, somebody had said, Oh, on this, you know, on the settings page over here, when you go and change something, it crashes. Well, I haven't changed anything in the settings page. So you're just using this.

You're not trying stuff out. Wow. So yeah, it's great. And that, that meant that I could fix it straight away and, and get a release out in the half an hour. Yeah. That's awesome. I'm a,

David: I'm a web developer, so I definitely interested in that side of things. And I'm sure I'll have more questions as we go along.

But you mentioned that You've done audits against the web sustainability guidelines. Yeah. So what does that look like for you? What, what,

Jonathan: what is the web sustainability guidelines first? Yeah. I only dabble at doing web development and I'm not sure what that is.

James: I, I think most people are in the same boat as you.

So You've heard of things like the Web Content Accessibility Guidelines,

WCAG,

and things like that, which is all these good things that we should be doing to make sure sites are accessible and so on. The idea of the Web Sustainability Guidelines, it's now a there is a W3C community group that's put these together.

I think some, some people I used to work with are involved in it. I, I used to be more involved in the sort of sustainability climate side of things. So it's always been like an interest. I used to run a meetup in London around, uh, using the web for sustainability stuff. So but what they've come up with is actually a, a kind of a draft spec of, okay, here's, here's how you.

You know, these are the, these are the guidelines for, for building sustainable science. It's yeah, just, yeah, if you look it up, you'll you'll find the thing, but it's. It's still in draft. It's very early days. It's not that a lot of people are using it, but it's something that I was interested in.

I've been interested in for a long time, like say, so I was like, well, when I had the opportunity to spend some time on it as part of this open source funding, I, I said, right, we need to do an accessibility audit. We need to do a security audit and I want to do a sustainability audit as well. And so I did, there isn't a.

process as such. So I actually sort of rolled my own DIY assessment type process, very much just like a red, amber, green type thing, going through each item on the list and going, yeah, that's, that's okay. And this one, Oh no, I hadn't thought about that, that kind of thing. And then hopefully do it again, maybe in the new year, see how things have moved on, but it's been really useful.

Just in terms of even, even that sort of very rough approach was really useful in into pointing at pointing out where, where the inefficiencies were, where the, you know, what was going to be more impactful about this application. So yeah, it's, it's really good. And obviously one huge thing with this is that It's 3d models and they can be really big and that's by far the probably the biggest impact of the thing.

So you know, that's that's on my sustainability, Roadmap now but it's nice to treat that as a as a sort of primary thing I think alongside accessibility and and security and and things like that to to have it in mind as As a goal so actually in the release nodes now we have like a special section which is Sustainability and performance because those two go hand in hand.

So when those get Improved it it points it out to people. So

Jonathan: So this basically boils down to not wasting cpu cycles I mean put it very very broad strokes at its

James: most at its broadest stroke level. Yeah, network transmission all that kind of thing But it goes into a lot of other stuff as well.

There's a huge overlap with accessibility. Build things that people are able to use and they'll be able to use them more efficiently. There's a big section on product design and things like that, you know, build what people actually want. And then you're not building things that are pointlessly using using power where they don't need to.

No no major current trends mentioned.

That

David: was a very suspicious cough.

James: Yeah. Yeah. I don't know. You'd need something, I don't know, something very clever to analyze what that was.

David: So I

James: used to work in AI and this is, this whole thing is just hilarious to me.

Jonathan: Don't get me started.

David: We'll put that, we'll include that in the red table. That's not for today.

Yeah. So, this, not to go too far down into sustainability, but I assume you can also see it even as a UI UX issue where Absolutely. Think about a web page. If they're having to click through five screens to do one thing, that's five different hits. That's additional. Okay.

James: Yep. And it goes, it's, it goes to that sort of level of pointing out all those all those guidelines of, you know, Around those things and those things that I wouldn't have, you know, I'm a Very much a developer technologist.

So, you know my thinking was like, oh, yeah, where are my data? Where is my server running? Is it, you know powered by renewable energy, etc And that's like one one point is that you know, there's so much else And it goes into all these other things. That really take it and you know a whole team Developing, developing a product can, everyone can do something.

It's really good. Really interesting.

Jonathan: I love the fact that when they made these guidelines, they included a lot of just good hygiene sort of stuff like that in there. So it's useful, you know, it's useful for, for everybody, developers. That, you know, may not have a whole lot of concern about the actual sustainability bit, but it's still useful guidelines.

I think that's really neat.

James: Yeah, absolutely. It's a bit like the universal design thing for accessibility, right? You know, good accessible websites help everybody. Not just, you know, people who need it all the time. It's just good design. So,

Jonathan: yeah. All right, I want to ask about funding because you've, you've talked a little bit about getting some money for things.

And, you know, for some open source projects, that's like the holy grail to be able to get paid for stuff. And it's, it's not quite the panacea that people might think it is, but it is kind of nice to be able to get paid for doing open source work. I believe I also saw that you are a fellow open collective member, which that is cool.

That's where, that's where mish tastics. Open source budget is at and the cool thing about open collective is it's like it's all right there. It's completely open So we can go. Yeah, I love

James: it. We yeah, absolutely. We use that for I also admin, the mastodon. me. uk server which we've got a couple of thousand people on and that's all supported by Donations and through open collective and everybody can see exactly what we're doing where the money is going on the server costs.

So it's great. But yeah, the, the funding is, yeah, it's, it's an interesting thing. I've been very lucky to get some funding from it's, it's basically, it's European union funding through the European union have a research program called next generation internet. NGI and they have various funds within that, like NGI zero I'm never quite sure how it breaks down.

Anyway, that's all administered then by a foundation in the Netherlands called NL net. Who put out calls every, every quarter for it's all. Specifically for open source you know, what you're doing has to be open source in order to to benefit from this, but it can be you know, a theme around bringing control of data back into back into users hands.

It could be a theme around security or, or whatever, but yeah, the funneling, funneling that money into open source Developers is is amazing. So I was lucky enough to get allocated, allocated, get to be awarded some money for that, which has let me spend the last few months full time on this, which is incredible.

I've got another application in, so hopefully it will carry on. But in, in the long run, it ideally needs to build up a, a sort of sustainable base from. From users on, on Open Collective, I'm hoping that's where it goes. We've got now quite a few projects that are sustaining themselves that way.

And it's, it's interesting. People, especially in the, in FedEva seem to be, You know, there, there was always this idea that, Oh, people won't pay for stuff on the internet. It's like, in the early days, I wanted to pay for Twitter. I liked it. I was like, I'll, I'll give you some money, but there was no way of doing so.

So they had some other business model that didn't care about what I wanted. And that, that kind of, no, I am willing to pay for something that I think is good. It's it's, yeah, it's, it's changing, I think. And there's a lot of things doing it.

Jonathan: Have you, have you had reach out or worked with any any of the printer manufacturers?

You can tell us about it. I have no secrets mainly

James: because I have no success on this. I did try. I did try once. I was like, Oh, is anybody interested in sponsoring this thing? And obviously I just went straight in the bin. The one thing that I cannot do is marketing. That's just not, it's, yeah, I don't know how or or whatever.

I mean, that, that would be cool. That would be a really good thing. I think it's, you know I think a natural fit for many of them. It's also a bit of a reaction to what many of the printer manufacturers have done. So. The first sort of big model sharing model publishing site was Thingiverse, which was which is run by Ultimaker.

I think so. I think. And it was, you know, dominant for a long time in sort of the early early days of these things. That's now, well, well, it went through a real. Bad patch where like just search didn't work for like a year and things like that. So people were going to other, other platforms things like that.

It was, it was building up sort of, you know, more more other platforms, but it seems that every printer manufacturer is now bringing out their, their own hosting site. We now have printables, which is. Somebody I can't remember which one is it Prusa. Is it? Yes, it is. It's Prusa. Thank you. And there's there's loads of other I came across another one today, which I think had I don't know if it was Bamboo or whatever, but it was like very much Download the bamboo thing here into bamboo slicer.

Okay, this is this is theirs and None of these things work together, right? So You have creators who will be Publishing on Five, six different platforms now, which is just wild to me. And then you have things like, I can't choose how I use Thingiverse, right? I can't say, Oh, Thingiverse is useful. I can pay for it.

No, I can't do that. Instead I have to watch a whenever I download, there's a 30 second ad that pops up. I mean, it's a static image ad I've seen it in side two seconds and decided that I'm not interested, but then I have to wait. So this, this, they're all trying to find ways to monetize these things, I suppose.

And Corey Corey Doctorow's term which I don't know, family friendly show. I, I won't I won't say I'll let you say it. I is, is definitely a thing, right. And, and. I think we've learned over what we've learned of the sort of the consolidation of the big tech and big social media networks is that these walled gardens aren't great.

So actually publish your own stuff, host it yourself. We make it as easy as possible. And then, you know, syndicate it out from there. So that's very much a kind of a a driving force behind Behind manifold as well. So I think that's something pretty manufacturers should be excited about, but it's also like, I want to stop you all building your own hosting sites.

Although feel free to run one and put your name on it and then join the Fediverse, right? It's that's something they can do by all means. Absolutely. That, that actually brings up, I'd have to see that happen.

Jonathan: Yeah, I would too. It brings up an interesting question though. What, what license is manyfold under?

It's MIT.

James: Okay. So very, very permissive. Very permissive. Yeah. I've long. Got over the the idea of trying to, like I say, nothing I do has ever worked. So, you know, I might as well give it away. No, it's, it's MIT, everything that I've, that I've done with it is, you know, there's other things that we've published as part of that part of the development is MIT as well.

I'm not, you know, looking to make a walled garden and get everyone onto one platform and then, you know. close the gates and and start making money off them. It's just not, it's not a thing. So if I can get people running their own stuff and you know, pay the bills alongside somehow, then that's fine.

Jonathan: Yeah. Yeah. That's the ideal. Do you have any really big instances of Manyfold? Is there anybody out there that's trying to run one as sort of a community instance?

James: I know there are some, I know there are some starting up because a couple of people have asked me in the last couple of weeks about getting things up for community groups.

One of the things that I very quickly put on the roadmap and launched last week was OpenID login, so people can integrate it with their existing authentication systems. And Fortunately, the, the way I'd built it made that very easy, which was nice. So we could get that out. It was a complete surprise feature.

I did not have that in the plan at all. I mean, I don't really have a plan, but you know but yeah, it was really nice to be able to say that actually, yeah, I can do that. There we go. That's the, the dopamine of open source, right. But I know of. I, I thought I had a fair few, I thought I had quite a, quite an archive of models.

I know, One instance that's pointing at 17 terabytes, which is insane.

I've had a few sort of bug reports come in from, from the the person running it. It was like, it gets a bit slow.

I'm surprised it hasn't set your house on fire or whatever. You know, it's, it's a bit slow. Yeah, I, I, I've been impressed actually how well it has coped with, with big instances. I, I'd like to see some, I'm just starting to collect public instances now. I've just put a list on the website of known public instances that people are happy to put there.

Currently, it's the two that I run which is the which is my own one and the demo.

There's a demo instance. You can go and actually play around with all of the features without any authentication or anything. But yeah, it'd be, yeah, I'm looking forward to seeing some, some instances come up, but at some point I'd like to run like a flagship instance so that anyone can just sign up for an account and start publishing stuff.

Yeah. That would be really nice. Yeah, budget is probably the thing at the moment. I've, I've, I've mentioned that as a possible thing in, in the sort of next round of funding as I actually like, it would be great to get a flagship instance up.

Mm hmm.

So,

Jonathan: yeah, is, is there a, is there a way to search across the different public instances of this?

So say I'm looking for a specific model of something and I just want to say, go out and, and, you know, look at all of the many fold instances, all of the things that are public. Is there a way to do that yet?

James: No, not yet. That's a general. So, so there's a few things. One, there's, there's one. main, there's a few main sort of 3D model search engines.

So I think Fangs does a has a search across other sites. And there's one called Yegi as well, which, which does that. I'd like to make it so that manifold instances can be indexed by those. If necessary, so that somebody who is running a search indexer, the stuff will pop up. And then there's a big thing generally about federated search and search of the Fediverse because every instance has this problem, right?

Whether you're on if I search on Mastodon, I'm only seeing what my instance knows about. So how does that work? And I know that there's work happening there, which hopefully I can. I can hook into and build upon in some standards compliant way, that would be ideal. So yeah, at the moment, no, but it's definitely.

Yeah.

Jonathan: And it's something you want to do. I think it would be really cool. I'm looking forward to that. So the the thing that we had to talk about with, with Mastodon, I think it comes up here as well. Maybe, hopefully it does. Hopefully it hasn't come up here, but do you, do you, do you guys ever have any problems with people like uploading inappropriate things or even just straight up trying to spam on Miniverse?

So

James: at the moment. No, because I don't know of any other instances. But it will be a thing, right? That will be a thing.

Jonathan: That's when you know, you've made it

James: Yeah, definitely. So I mean one thing I did last actually the the sort of current One of the current themes of development that I'm doing now that we are in the ActivityBub world is building in the required moderation and admin tools.

We've got things like moderator permissions in there already. I added in a sensitive content flag last week. But building those things so that it will respond to Reports from other instances give you a, you know, report log, things like that. So yeah, that's been in the, in the roadmap from when we started thinking about federation stuff.

It's definitely something we need to support. We need to be able to control federation as well. So which instances we do and don't federate with or, but that's all individual choice, right? That's not, I say we, it's not. It's everybody separately. But yeah, definitely want to build in those controls and learn from what's worked elsewhere in the Fediverse and what's but also what people have missed as well.

David: It does seem like Moderation around 3D objects is a little bit different than moderation around text. Yes.

James: Yeah, but you know, we've got comments and descriptions. So, you know, Comments and comments. There's text as well. And images.

Jonathan: Images is actually one of the ones that's a little scary. Because if you, if you tick somebody off, they can, there are, there are literally illegal images that you have to potentially deal with.

Absolutely.

James: Yeah. Yeah. Yeah. So, so knowing, I mean, it's been really handy having been a, an admin of a, of a Mastodon instance and bringing that experience to building this and actually going, yeah, okay, I need to be able to say for this server, don't, just don't talk to it and never ever accept any images.

from anywhere over there. That's definitely something we need. So I mean, I say that aren't mastered on moderation journey has been very smooth over on our server. It's been, it's been a nice, a nice ride, but I know, I know it's not that way for everybody. So, yeah. Yes.

David: To get So I know that this could be a potentially deep topic because you do have a PhD in it, but the explain it like I, like I'm five overview.

And going back to that massive survey you were talking about the size of those 3d models and everything you've mentioned in your blog post that you're actually looking at developing ways of 3d compression.

James: Hmm.

David: What does that look like?

James: So, yeah, this is this is one of the fun bits of of the project that I've I've got funding for.

I haven't done it yet. It's going to be coming on hopefully sometime in a month or so. I've really got to get this done. 3d models are big. They're, and some of them aren't. really big. I mean, some, some creators will create quite efficient measures. Some will just have millions of points in there that you really don't need.

So Yeah, things like, and this speaks to the sustainability thing as well, right? These things are expensive to send over the internet. They're they take a lot of space. So one of the things Manifold will do is it'll detect old inefficient formats. So you mentioned OBJ Wavefront. That's really inefficient.

It's all ASCII. It's just, it's massive and it doesn't need to be. There's ASCII encodings for STL, things like that. And we can convert those to 3MF. Which is a nice it's a, it's, it's a sort of more modern format, but then it's also zipped up. It's a zip file basically. So there's that, but then that still doesn't help you sending it over the network because you've still got to send the whole thing.

One of the you know And the internet might've got fast enough that we don't really worry too much about this these days, but you know that PNGs can be, and GIFs actually back in the day, could be progressive. So you could, they've got all the information you need to display them at a low resolution and then it refines and refines and refines.

So there are ways of doing that with 3D models. And like you say, I did a PhD in this years and years and years ago. 25 years ago, probably approaching I started. And It was when it was, it was all around animation of very very dense meshes. And the fun thing was while I was doing my PhD they invented graphics processors and suddenly what was a very dense mesh wasn't by the time we could just do it all with a, with a nice graphics card.

So. What I did for my PhD wasn't massively relevant, but some of the things that I, there are algorithms that I read about while I was while I was preparing to do sort of my research that are, there's a thing called progressive meshes written created by a guy called Hugh Hoppe, who I probably pronounced his name terribly wrong.

I'm sorry. Which basically you can imagine you have like, imagine you've got a cube, right? Or a very simple shape, right? You're going to get that really quickly. And then instead of loading the whole thing, you just say, all right, that, that corner there, split that into like that, split that on that, on that, on that, on that, on that, on that.

And then the thing refines. And. This is a really nice, nice thing, you get this sort of stream of progressive enhancement of this model, which means you can see what it is very quickly, but you can wait for the full detail if you, if you want to. So that's a It's a really, it's a really nice algorithm for that.

And, and I was looking around to see if anybody had done anything with this particularly, around transmitting them over the internet. But, you know, I mean, I was playing around in VRML back in the 2000s, but 3D over the internet hasn't really been a, you know, a huge thing. It's, it's all, you know, in games and things like that.

And you're just downloading gigabytes in one go and then you've got it. That sort of continuous transmission hasn't really been an issue. So nobody had done anything with this particularly in terms of that as a sort of, as sort of as a, as a transmission format. So I got in touch with Guy and said do you know if anyone's done this?

And, you know, we should put together a, a spec to do it. And so I've done a draft spec for it actually as a GLTF extension, which is a sort of format designed for, uh, for sending over the network. People have done mesh compression in that, but it's not this sort of progressive transmission thing.

So I've written a spec, I've got to write the code, and, and actually get it working. But the idea is that if you're looking at a, a very complex model on some other server, you'll actually get a low resolution version of it instantly, and then it'll refine over time. One fun thing about that is that it provides an interesting, interesting thing.

possible approach for commercial models in that I could show you a low resolution version of the commercial model and you can see that and you can look at it and flip it around and all that kind of thing. But actually if you, you know, when you've paid for it, you get the whole stream. And you get all that fine detail.

So, it's, yeah, it's a really interesting thing. It's, it's, it's nice to be able to flex the the 3D graphics brain for the first time in a good many years. Yeah. Doing this. Yeah, that progressive,

David: Concept that you just mentioned, and not just looking at it, but I could envision where it could be something where you could, like, print it in low resolution to test if it's what you want before you spend the money and get the full resolution product.

James: Yeah, that's true. Yeah. Yeah. Yeah. Yeah. Test size, you know, yeah. Yeah, yeah, that's a good point. Super interesting. That's a good point. I haven't really worked out what the sort of commercial obviously there's lots of commercial creators out there, right? So I need a solution if anybody's going to use this thing for that.

I don't quite know what that is yet. But I'd like it to be possible in some way. I'm not, not interested in like payment processing and things like that. Somebody else can handle that. But like, how do I, how do I make sure I've got a permission model that works for. for sort of sharing of a preview and then and then full versions of things.

It's kind of what I'm thinking about. But there's a lot of a lot of commercial modelers out there who who I think would benefit from being in control of their, you know, seizing their means of distribution, right?

Jonathan: All right, we have a, we have just a minute before it's time to wrap and I want to say something we didn't get to. Yeah, it's gone fast. Something we didn't get to. It has gone

James: fast.

Jonathan: I wanted to ask you about is what was the experience like adding ActivityPub? How, how big of a pain was that? Or is that something that is actually fairly simple?

James: It's got some sharp edges. It's no, it's all right. It's all right. There's, there's some good, you've definitely got to learn it and understand it and, you know, read the specs and things like that in order to to be able to, to do it. I think it's not, the basics aren't super difficult. A friend of mine did a a great blog post on how to build an activity pub server in 20 minutes in PHP like in a single page of PHP, which is, you know, you can do that.

There are interesting bits like signed requests, which is a little bit tricky and difficult to to work out exactly exactly why it's not working. Some of this stuff can be hard to debug because one of the things is actually it's all needs to be on the internet and talking to other systems. So I mean, while I was doing a bunch of the development, the deploy process was just the debug process was so slow because I had to build a version, deploy it up to my machine, then talk to other things on the internet, et cetera.

So it was it can be a bit like that. But one of the One of the things that I've done as part of this is I wanted to, so I built the the site's built in Rails which has been a, something I've worked in for many, many years. It's very nice for programmer speed, which for me is a good thing to optimize for.

It's just me on my own. And one of the things you can do with that is it's, you can plug in engines that do you know, that bring a bunch of features in one go. Like we have one for authentication and we have things like that. And I wanted to make a reusable component for activity pub. Mastodon is also built in Rails.

Their activity pub is kind of, well, it was the, the site was built before they adopted activity pub. So it's a little bit It's not sort of easily extractable. I don't think I was kind of going in thinking, Oh, what if you could pull out the activity pub call from masters? I've definitely read some of the code.

It's been useful to see what I've been doing wrong. Mainly actually, why is this thing not responding when I send it stuff? It's where, where is it sending it through debugging through other people's code to, to work out what you've done wrong is, is interesting. But . Yeah. The, I wanted to produce this reusable engine and I found that somebody had already had this idea.

And so I've been contributing to that project. There is a rails engine now called Fed Rails. Hmm. Which, when I started looking at it, it supported things like following back and forth and things like that. It's that was it had some of the basics down. I've over the last little while.

Implemented the the signed activities, different activity types multiple actor types. So in our system in, in Macedon, everything that you can follow is a person, right? That's simple in manifold. It's a creator, which is a person, but also it's a model and it's a collection. And there's all these different things.

So getting multi actor support in there Lots of things. So I've done a bunch of code for that which is, so actually all the ActivityPub core of Manifold is in that reusable engine. So, and, and I'll be carrying on with that and building more and more into it as we as we go. So anybody can pick that up and reuse it.

Hopefully fairly easily. Yeah.

Jonathan: Manifold is built in, in Rails, Ruby on Rails? Yeah. Yeah, that's right. That's one of those languages and frameworks people have very strong opinions about one way or the other. They do. They do.

James: I, I won't start a flight war.

Jonathan: I like

James: it. And that's

Jonathan: what matters to me. That's what matters.

Yes. Mash was data from the chat. Opinions are available. Yes. Yes. I can help supply some of those if you'd like. Master tailor from the chat wants to know what about that other federated protocol at, which that's the one blue sky is working on. Any plans to support that?

James: Not at present. No. I don't know anything.

I, I haven't looked at it that much, to be honest. I know that BlueSky does it. Is there more than one BlueSky server yet? Is there, is that even a thing? I don't really know. I'm not there. That's what I think. We, we've we, I did bridge the manifold. Fediverse account over to Blue Sky the other day, and that's about as far as I've gone.

I don't know. Should I? He asks the

Jonathan: audience. I will say this. I went looking for, oh, have any other open source projects picked up AT and started using it? And as far as I could tell with my quick search, the answer is no. And if that ever changes, I would think maybe you should think about it. But until then, I don't see the point of it.

James: Yeah, I think others. Other bigger, more important applications will pick it up first. And that's when I'll notice and think, Oh, maybe I should do something like that. But I'm always willing to be corrected or to learn more. Yeah. There you go.

Jonathan: All right. So I want to make sure and ask what, where do you guys need help?

So if somebody said, Oh, I know, I know how to program in Ruby on Rails, or maybe even if somebody doesn't, but they want to say, I want to roll up my sleeves and help with many fold. Where's a good place to jump in and, and start at? So

James: it's, so I've tried to make it accessible. I don't know whether I've managed it.

We have a bunch of issues on GitHub that are labeled good first issues that are fairly self contained things that don't involve you understanding the entire application. But it's not just Rails. There is, there is Ruby code. There is plenty of JavaScript code. If you want to do 3d JavaScript, I've got some jobs for you.

There's lots of stuff I want to, to add into three JS for instance. So there's, there's things there, there is design. I am not a designer. The whole thing is built with bootstrap. Because I'm incapable of doing anything better. So, you know, design UX one things we do have people we do have a little team of translators who've stepped up.

We're actually translated fully into four, five, four languages five in total. English doesn't count. French, German. Spanish and Polish Polish, because somebody popped up and said, I want to translate it into Polish. So I went, all right, sir. And, and he did, and that was the first one that actually got to a hundred percent translated.

So people want it in their language. Please do. I would love to have Russian. I, then I think after that it's Japanese. I don't know, I think, but things that I can't understand when they're wrong is, it's definitely, yeah, on the horizon. Although I mean, I can, I can read Russian. I just don't know what the words say.

Understood. But I did it at school so I can read the alphabet, but that's about it. And so but yeah, there's that but also things like documentation, I mean, open source really suffers from. From documentation issues, I've tried to do some. I've discovered that I would rather do almost anything else than write some bits of documentation.

But also i'm a really bad person to write it. I know how it works and how not to break it so those kinds of things like What haven't I explained? Is is not something I can see very well. So There's loads of yeah loads of opportunities for people to get involved. We're on we have a Chat space on matrix.

And there's all the codes up on GitHub and there's GitHub discussions there and things as well. So yeah, if it seems like something that, you know, I, I'm very much of the opinion that there's so much more to open source than there is just code. So, you know, there's, there's a lot more that that people could could do.

Jonathan: If somebody wants to run it, is the Docker image the easiest way to go?

James: Yeah. Docker image is pretty much the only way to go. Somebody did ask the other day how to run it on bare metal and a few of us just sort of all chorused in going, just don't,

Jonathan: you don't want it. You don't actually

James: want to do that.

I mean, nobody wants to set up a Ruby runtime and, and all these different bits and pieces. No, the whole point is all of the self hosting stuff I've done has the, the by far the easiest way it's been through, through Docker containers. Yeah. Normally the Linux server containers. I don't know if you guys have come across those.

It's a group that put out a, a set of applications as containers that all work in a standardized way. And I was very pleased when manifold got into that so you can run our own. Container. We have a version that has a separate database is the standard one. We also have a solo container, which is just run one container.

Everything's in there. And there's the Linux server one as well. There's instructions on the website for running it on things like Synology, Unraid All these various Various methods, but yes, it's all Docker. I imagine it's not

Jonathan: doing anything too obscure so you can run it rootless with Podman and all of that.

And it just works.

James: Yeah, we did a I don't know about Podman. I assume it will. We did yeah, one of the big changes to, to get the the Linux, everything was stopping it running as root and letting it run on read only file systems and things like that. So yeah, it can do that.

Jonathan: I will I'll have to give this a try with Podman then I'm my, my Linux usage is mostly on sort of the red hat fedora side of the fence.

And so I tend to go with Podman when possible. So I'll give it a try. I'll give it a try. And we'll let you know

James: what happens. That's, that's all the that's all I can ask. Let me know when it breaks. I mean, yeah, it's still under very active development. This is, you know, last week I put out three feature releases and a patch release for a broken feature.

So yeah, things break, but then I fixed them. So I've never lost anyone's data yet in 86 releases. Yeah, there's, there's not been any data loss. I'm hoping to get to a 1. 0 release maybe early next year, something like that. There's a few a few of these sort of the, the, the various features that I've got funding for, but a few other little bits to get in there that I can then say, okay, yeah, that's, that's a baseline.

And Then, then we go somewhere else, then we go from there, I don't know. But I do want to get to that one point, I hate projects that just perpetually hang on. A zero dot something or other. Yeah, yeah, yeah. Yes,

Jonathan: yes. Okay, so, I've got a, is there anything that we didn't ask you about that you really wanted to let folks know about?

James: Oh, I have absolutely no idea. I completely lost track of the conversation. No, I think just, no, come and come and check it out and come and have a look. If it sounds interesting, come and come and see. Well, I'll, I'll help people get it up and running. Cause like I say, it's early days. You still get very personal support.

Actually that's, it's not just me. There are now a lot of people in the support channel who will, who will help out getting it up and running. Yeah. None, none of whom I actually know. It's amazing. So

Jonathan: what what's the best place? So I see, I see many fold app. Yeah. If somebody wants, that

James: is the best place to

Jonathan: start.

Yeah. If somebody wants to donate, where, what's the link for that?

James: There is a donate link on that website. I think if you go to manifold.app/donate, I think that will probably work. Okay. He says nervously. But yeah, if you, you can click through to the main site. It's, there's a donate link downside.

There's also community links which will take you to the various channels. That are available there's a source code link for GitHub up in the, up in the top and yeah, all the documentation and getting started guides and all that sort of thing. Very cool.

Jonathan: Very cool. All right. So I've got to ask before we let you go I'm required to just about contractually.

What's your favorite text editor and scripting language?

James: Ooh. Ooh. See, how do you, how do you define text editor? So the thing I use day to day at the minute is VS code. I've gone through so many over the years. I was definitely an Emacs user for a long time. At the moment I'm using VS code. It's reasonably nice.

It's, it doesn't. break. And that's about all I need it to do. Actually it does break sometimes. It does break sometimes, but you know, yeah, yeah, yeah. Everything breaks sometimes though. So no, that's what I'm using at the minute. And I mean, favorite scripting language has got to be Ruby because it's just it's fun.

I've been using it for so long now that my favorite scripting language is one that I can get stuff done quickly in and not think about the language. And solve the problem. Yep. Yep. That's the way it goes. I'm not averse to any of the others. They're all great. I know many languages. Most of them are lovely.

Jonathan: Most

James: of them.

Jonathan: Not naming any names. Not naming any names.

James: I have mixed feelings about some that I have to work with. And that's where, that's where we shall leave it. Yes, yes, that is fair.

Jonathan: All right, James, we are, boy, we are beyond out of time. But thank you so much. It's been a blast. Talking with you and about Manifold.

And I literally have a terminal pulled up. I'm trying to get it installed on the machine behind me to let you know where the pod man works or not. So as soon as, as soon as we get close, yeah, I'll finish up with that. Thanks for being here. Thank you very much.

James: Yeah.

Jonathan: All right, Mr. David Ruggles. What do you think?

Have we convinced you?

David: Oh, well it's still a question of time. I never have enough time, but now I want to definitely get a 3d printer and get into the ecosystem. But yeah, that was fun. And and I just, it's always fun to talk to somebody that's just got such a wealth of diversified experience.

So I definitely, he, he, Hey somebody, if he's available, we keep him in for the roundtables. Yeah, that'd be fun. It'd just be fun just to pick his brain about so many different things.

Jonathan: Yeah, yeah, that was a lot of fun. And I'm, I'm real fascinated by this idea of what he's doing with 3D models, trying to compress them in different ways.

I think there's, there's definitely some room for that. And I didn't, I didn't get to ask him, but I actually wonder whether some of that is going to be cleaning them up as well. Because it's one of the things you get when someone's used like Blender or what have you. To produce a model, there'll be a lot of internal geometry that just either never gets printed or maybe never should get printed.

And there are some algorithms out there to try to clean those up, and not all of them are amazing. So, definitely some room for someone to work on that. But it's cool. I am literally working. I have a tab pulled up right here. I am working to try to get this installed and running using Podman. It's pulling right now, so if my internet connection craps out, that's what's going on.

Alright, a lot of fun though. David, do you have anything you want to plug?

David: Not specifically. I would encourage everyone to go check out Twit. That's the other place I like to frequent. So everybody else should like to frequent it too, of course.

Jonathan: Of course. Of course. So next week we are talking with Josh Bressers from Anchor.

Which that is about automated software compliance. And apparently they have quite a few open source projects that are connected to that. And so looking forward to looking forward to that, it's going to be next week. And then we've got some other things in the pipeline coming down the way. The, we've got the The J4Pi project coming up soon, which is about and we talked about scripting with Java.

Well, this is about scripting with Java on the Raspberry Pi. And that's, that's actually going to be pretty interesting because they're doing some fun stuff, exposing all of the different peripherals, SPI and GPIO and all that. So actually quite, quite a bit looking forward to that one as well. As far as plugs, we do want to make sure and thank Hackaday for being the home of FlossWeekly now.

And there is of course my security column, goes live Friday mornings there. And then there is, as we mentioned, the Untitled Linux Show over on TwitNetwork. And make sure and check that out as well for even more Linux y goodness from your host to co host today. We appreciate it. Thank you everyone for watching, those that get us live and on the download.

And we will see you next week on FlossWeekly.

Jonathan: Hey folks, this week we talk with Andy Piper about Mastodon. There's a new release, 4. 3, with all kinds of new tools. We get into the philosophy of Mastodon, the cool things you can do with it, and more. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 805, recorded Wednesday, October the 15th.

Mastodon, bring your own algorithm.

It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something fun today. We're talking with Andy Piper about Mastodon. And of course, it's not just me here. I've got a co host today, Jeff Massey. How are you doing, sir? Oh, I'm doing good.

Doing good. Happy to be here. I appreciate you sort of sort of stepping in at the last moment I've I've had a bit of a health issue that I have been battling through for Well in retrospect over a month now, but past week or so was real interesting So things got a little behind but we're here. We made it and I appreciate Jeff being here

Jeff: Well, I'm just honored to be on the short list to be able to fill in and you know, I, it's, it's sometimes hit and miss based on, you know, my day job, but the Stars Aligned had an, had an opening and said, sure,

Jonathan: be glad to help.

Very cool. Now today we're talking with, like I said, Andy Piper about Mastodon and Jeff, do you have a Mastodon account? I do not. Okay. So our goal today, our goal today is to talk Jeff in to going and getting a Mastodon account somewhere or host it yourself. Like that's even an option. That's one of the beautiful things about Mastodon.

If you really wanted to, you could prop your own account up somewhere.

Jeff: Well, and you know, the interesting thing is I've heard about it quite a bit, but I haven't done a lot of research. I'm not an expert in it, so I'm, I'm really curious to find out, you know, all the. I know the high level stuff, but I, you know, a little under the hood and kind of what's going on and, you know, where's it been, where's it going?

The whole, the whole nine yards.

Jonathan: Yeah. And that's definitely part of something we're going to talk about because Macedon just had a new release. And I, you know, I don't remember what the version number is. It's a 4. 3 Macedon 4. 3 and we've got the man that knows about it and let's go ahead and just bring him on.

That's Mr. Andy Piper, and first off, we sure appreciate you being here. Welcome to the show, sir.

Andy: Hey, Jonathan. Hey, Jeff. I'm, I'm really, really excited and honored to be on a show, which is not only all about free and open source software, which I'm a huge advocate of Floss Weekly but also on Hackaday, which is another one of my favorite sites.

And I'm one of the things I do is I have a small maker studio here with my wife in the UK and you know, I'm on Hackaday. Literally all the time, all the time, every day checking stuff out. So yeah, so that's great to be here. Thanks for inviting me.

Jonathan: Yeah, we, we appreciate you being here. And now I've got to, I've got to ask, maybe I'm jumping a little too far ahead, but what's, what's the stuffy?

What, what is the, is that an elephant?

Andy: This is our Brand new just launched although we've been talking about it for, gosh, months, I think, because we've been through the whole process, but this is our Mastodon plushie. It's available now in the Mastodon store. Depending on your geography, we are working hard to make sure that as many folks can have their own.

Look at this, look, see, see how squishy he is? But I was away this weekend for reasons I was actually at Ogg Camp, which is something you talked about here on Floss Weekly a couple of weeks ago. And I got back to the studio Monday morning and my new my new work supervisor, let's say, Was here waiting to keep an eye on my contributions to the mastodon project Yeah, really excited.

It's it's a lovely plushie and also got to give a shout out to dopper 2 who is our designer so They've done all of the artwork for mastodon. So you can see the load the mastodon here. Not not this This is this is something different the mastodon logo on my shirt here the the mastodon mascot And all of the beautiful artwork we have You Around our website is, is by somebody who, who I know, only by username doper2.

And that, that's obviously the mascot the plushie. So, yeah, super exciting.

Jonathan: Is there also a rock band that goes by the name Mastodon?

Andy: There is indeed a rock band called Mastodon and so there is sometimes a little bit of search confusion depending on what you're looking for. But if you're looking for the Mastodon social network, then or the social networking software, then that is us.

And Mastodon is also a rock band. That's

Jonathan: fun. So you want to search for Mastodon social to get you guys. That's fun. That's the one exactly right. Yeah. If you're on a website that looks like heavy metal and has whales, you're on the wrong place.

Andy: Reaching down over here because I also have got a little 3d printed Mastodon mascot key ring that I've I made myself over here, but these are not available to buy.

But yeah, I have a lot of fun with it. The mascot is really, really cute. Is

Jeff: the the 3d printer file

Andy: out there so

Jeff: people

Andy: could I I So not at the moment because again, you know, it's done by our designer. I don't have permission to share that. So We'll see whether we can do that in the future. We'll we'll figure that out

Jonathan: There'd be there'd be a lot of fun actually to have that available.

Yeah, absolutely

Jeff: So you'd say with the Oh, I was gonna say with the rock band. Is there any naming conflicts? I mean not counting search but any legal You

Andy: Look, I, I, I am not aware of anything like that. I am not a lawyer, so I I can't comment. But I, I'm certainly not aware of any problems around that.

Jonathan: You mean trademarks? Surely open source projects don't have problems with trademarks. Nobody would fight over stuff like that. Let's not go there right now. Andy, I'm so glad you're here because if you weren't here, that's what we were going to talk about. And I was not looking forward to that. Oh, right.

Okay. Okay. Okay. So let's let's talk about Mastodon. You guys just had a new, a new release and what version of 4. 3? What's, what's the big, what's it, what's new in 4. 3? What's the, what's the roadmap? What did it look like? What new stuff did we get? What new fun things can people do?

Andy: So first of all, I want to say that it's taken about a year to get from 4.

2 to 4. 3. There's a ton of moving parts. We've got kind of the core engine of Mastodon. We've got the web front end. We've got our own Android and iOS apps as well. And then there's an API, which means that anybody can build their own third party client apps if they want to for either for you know, Android and iOS too, cause they, they prefer to, to build something for themselves or for other platforms that we don't currently have apps for, which is pretty exciting.

And. When we think about what's new, there's, there's kind of the new stuff that you get as a user, which is really cool. There's also a bunch of new stuff that if you want to run your own Mastodon instance, because you don't have to come to an existing Mastodon instance, you can run your own for you and your friends or your community.

Hackaday. social. Very excited to see that Hackaday have their own Mastodon instance.

Jonathan: Probably needs to be updated still.

Andy: Need to persuade Elliot. I think he may be the one that runs it to just refresh that up to 4. 3. And then also there's some new stuff for developers. So Mastodon 4. 3 for users gets a nice new look and feel.

It's, it's an evolution of the look and feel, but it's really nice and consistent. We've done a lot of cleanup improvements around, around the user experience. Especially around the media support. So you can now do things like drag if you want to. Attach a bunch of images to your post. You can drag and drop those in the composer to reorder them.

We've got also group notifications. So if you're, if your post gets really popular and you suddenly start getting loads and loads of boosts or likes, then those things will be grouped. The other thing that's really useful I find in notifications in 4. 3 is the ability to filter notifications. So you can do things like filtering out, if you want to, brand new accounts or from domains and things like that.

So there's some really nice stuff there. Another cool thing is you can now add a little tag to your blog posts. Fediverse creator, it's an open graph tag that you add to the top of your page. And then that will add, when you share a link to your post, then that will add a little a credit, an author credit under your blog post.

Link card, so that will enable people if people are sharing your blog posts or for writers and journalists, if people are posting their stuff on, on Mastodon, then it gives a little link to their Mastodon profile so you can discover them and go and find out what they've been talking about, which is really cool.

The embedded posts have been updated as well, so they look a lot nicer, so that's kind of a lot of the stuff for users. That's in the, in the default web interface. Incredible work really by by the very small but dedicated engineering team that do most of the work on the core team and then there's a load of stuff for admins and developers as well.

Jonathan: Yeah,

Andy: pretty cool

Jeff: So that sounds really really cool but i'm pretty new to mastodon. Yeah, never never used it and just just a little background I'm more of a hardware person I'm an open source enthusiast but not Fully informed so I handle some of the basic stuff for Jonathan like asking like at least the 10, 000 foot version the very very simple.

How does it work if everybody runs their own servers? How does how do they connect and talk and

Andy: yeah? Yeah, so I mean the easiest way to think about this and the way that a lot of people talk about the Fediverse or this decentralized system we have is email, right? So easiest way of thinking about it, you might be at outlook.

com and another person might be at Gmail, but you can still send messages to one another, right? So It's it's very similar to that. We have a protocol called activity pub. We have some internet standards. There's one called web finger, which lets me say let's say Jeff at hackaday dot social if I know your address is Jeff at hackaday dot social, I can go and find out where my server needs to send messages or, or, or where your, your server exists.

And then you've got a system of inboxes and outboxes which enable. The service to exchange those those messages. So that's kind of the super high level way of describing it. I can go into tons of detail and we can sort of do this whole seminar on activity pub in the Fediverse, which which is not what we're here to do today.

But that's that's pretty much how it works right now. I've just been at. An event this weekend called Ogcamp, which as I say, I know that you talked about on Floss Weekly a couple of weeks ago, and I was running, running the crew for that. And what we did there was we had a a wall of posts from people talking about the event.

And by the way, tons of hardware people on, on Mastodon Jeff. I, I've got lists of folks that I follow. Who are kind of I've got a group called gadgets and gadgets and makers. So I follow a lot of the kind of hackaday folks. I follow a lot of the people who you'll quite commonly see sharing their, their builds and things on there.

But anyway, back to our camp. So we have this, this, this. This social wall with all the posts and what I did for that was I said, okay, let's get hashtag or camp and hashtag or camp 2024 and anything posted by the odd camp account as well directly. So it was aggregating together all of the stuff that was coming through on mastodon from each of those hashtags and aggregating them onto this nice wall that was in our main social room.

So it was really

Jonathan: fun. Every, every time we have a Mastodon guest, this is at least the third time we've had somebody from Mastodon on the show, which is great, by the way. We're huge fans. The the analogy gets made that it's like email and every time that is said I kind of I have this bit of a shuddery Feeling because I run my own email server.

Oh, and it's a terrible experience one of the reasons is because of It is anybody that runs their own email server will tell you it is terrible It's spam is one of the big problems. Yep, and with mass saddam. We also have to talk about abuse and And let's, let's talk about the spam bit first, and then we're actually going to talk about Matrix for just a little bit, because they've had some of these problems too.

But let me, let me just, let me just ask you this has Mastodon had a spam problem yet? And is there anything new in the new release that helps people deal with spam?

Andy: So, Mastodon does sometimes have. Some, some spam issues has had, it's got way better with 4. 3 and I'll tell you why in a second. But, but let me just describe how that sometimes occurs.

Sometimes if you, if folks do run their own server and then they leave registrations, they leave it on the internet with registrations open with no kind of capture or anything, gating folks signing up. And then they forget about it and then they wander off and they forget that they've set up this container spinning somewhere in the cloud, right, and they're not paying attention to it.

That is sometimes how that can happen. What is, has got a lot better in the last 12 months in particular, A couple of things. First of all, there's an organization called if tasks, which is independent federated trust and safety. And that is an organization, another nonprofit which seeks to provide best practices tools for folks that run either their own instances or small instances, small communities, provide them with some guidance, some education about some of the things they might need to think about.

Around those kinds of topics. They also have a service, I believe, which enables you to kind of get a almost a default block list or a default list of known very bad domains that you don't want to receive information from on your instance. So they're a super, super great organization to keep an eye out for.

Mastodon, the, the, the organization knows, you know, we're in connect, we're in contact with them. But that's a separate organization for, for Mastodon as a whole. And it works to try to bring some of those good practices across the Fediverse, all of the things using ActivityPub. So that's one element here.

The other element is in Mastodon 4. 3, we've got these filtered notifications so you can go in and I did this literally did this in the last couple of days because my instance was the instance I'm on mccall. social got upgraded over this weekend. So again, I came into the studio on Monday morning. I thought, Oh, my friend, Ron, who runs us, our instances upgraded us to 4.

3. I went in and updated my, my, my notification settings and I can do things like filter out posts from accounts that were created just. You know, in the last couple of days don't get rid of it. You can either choose to get rid of them completely and never see them. But then somebody like Jeff might sign up right now or tomorrow or, but, but in the next, I'm going to think Jeff's going to be going to create an account in the next few hours, let's say, right?

Because it's going to make sense. A

lot of sense. And he may, he may post it post. Post it on Mastodon and say, Hey, Andy Piper at McAuliffe social. Great chatting with you. It's been great chatting with you too, Jeff. I'll say that right now. But anyway, let's let's let's carry on. And I don't want to miss that, right?

Just because it's a new account. So what we do, what we do is we have the option to filter those into a separate like inbox, a filtered inbox. So I can go and look at those later and say, Oh, Cool, Jeff's here now. I'm going to follow him, right? And I'm going to allow that, and that's okay. But if I get one from some strange domain I've never come across that may be one of these kind of abandoned instances or that contains something that I don't want to see, then, you know, I can, I know then I can either block it or I can block the domain.

Another feature in Mastodon 4. 3, which is really useful, is because we're talking about these instances that are Separate you can block individual users, mutual block individual users. You can also choose to block an entire domain. Now, if that happens between instances, what that does is it blocks every account on the other domain.

Now, if that other domain has, you know a bunch of people actually you wanted to follow. Maybe something, something has changed around the ownership of the instance or the moderation of the instance. Then if your instance owner, your instance administrator blocks that domain, you as a user now get a notification to say, hey, by the way, your instance owner just blocked baddomain.

com. You were following four users. Bad domain dot com, here's a list, you might want to keep a record in case they move somewhere else in the future. So, we're doing a lot of work to try to improve these kinds of situations. You raised some really good points I beg your pardon, Jonathan, about running a mail server.

And and yeah Unfortunately, we've learned in the history of the internet that some people do some silly things which aren't great fun for the rest of us who just want to have a better experience. But what we want to do with Mastodon is really build a better social web for everyone and do that using open source technologies, do that using internet standards and protocols.

Jonathan: I mean, you sort of, you sort of know that your system Has arrived. You've made it. You're in the big leagues now when you start getting spam through it.

Andy: I, I mean, that's the other argument, right? And, and, and that, that, that could well be the case, but there are controls in place or there are got controls available to, to, to manage some of that.

And we are working on that. There's always more we can do. There will be more we can do. And we, and we listen to the community and we pay attention to, to what's happening.

Jonathan: Yeah, that, that idea of, of an old instance that allows open signups without any sort of protection against it. That very much reminds me of the the open relay problem with email.

Andy: Yeah.

Jonathan: And so I can imagine, I can imagine a future where you, you start having, well you, it may already be here for that matter. You have some sort of a a blocking list service where this server is a known open relay. Therefore, we suggest that you block it.

Andy: That's what if tasks can provide to some folks.

And that's something that we're looking at adding. On our roadmap for the next version of mastodon as well that we'll we'll have some better defaults there But right now you can go to iftas. org And sign up for their forums and they can help you with those kinds of things as well if you're a new administrator

Jonathan: Yeah, so again making the email analogy.

It's the equivalent of spam house Spam host, however they say it. I mean,

Andy: I'm not, I'm sure that Jazz Jazz Michael King who runs that would see things slightly differently but it is similar, there's some similarity for sure. There's enough similarity to make the analogy I think, yeah. Yeah, yeah, yeah, absolutely.

And you're right that it's a sign that the Fediverse is taking off that we need to spend more time thinking about things like Things on this.

Jonathan: Yeah, so let's let's talk. I'll let I'll let Jeff go just a second. There's something else. I want to make sure I get to Let's talk about abuse like that's the other the other big part of this when we talk about blocking people and blocking messages those are the two big things you got spam and then you got abuse and the the tricky thing with abuse is it is not a Nearly as binary as spam is.

Right, so an incoming message is pretty much, it's either spam or it isn't, and everyone sort of agrees, looks at a message and agrees that's spam. Abuse, it's different, because some people, their tolerance for abuse is, if someone, you know, has anything political to say, well, it's abuse. If someone is supporting, A candidate that I don't like it's abuse and like that's sort of like on the very very light gray area and then you've got the extreme opposite is And this is something that matrix has been dealing with matrix is sort of the other federated multi user universe out there I was I was in one of the main matrix rooms and I had to leave it because people were Coming in and dropping images that I did not want to see like I don't I don't I think they were getting to the point of being like CSAM that's child sexual abuse material but they were, they were close, like that was the direction people were trying to go, and the Matrix guys were just, they were struggling trying to get a hold of this, and so like, that's the very dark, completely opposite edge of, of the abuse spectrum, and not everybody agrees on how much of this is actually abuse, and one of the things that I, I love about Mastodon is, You can run your own server and you get to make that decision yourself.

Whereas on the other social networks, you've got a company that makes that decision for you. And I just, I just love the fact that I'm asked it on. It's not the way it works. And so I'm assuming that with with the new version with 4. 3, you've also got more tools out there to help people deal with that sort of thing.

So let's talk about that a little bit. What's what's new? What's new in the world of abuse on Mastodon?

Andy: Well, if I can pivot that, I'm going to include that in my answer. Let's let's pivot it and talk about the sort of the admin side. I gave you kind of an overview of what's new for users. And for people using Mastodon in 4.

3 particularly on the sort of the website but also on, on the, on the. So let me start with the basics. So 4. 3, there's some upgrades to some of the underlying dependencies. So we bump up, I think, from Ruby 3. 1 as the minimum to Ruby 3. 3, and a newer version of Ruby. Newer versions of Postgres and, and those kinds of things.

And what that brings as well is better performance. We've done some work to migrate away from and, and, and deprecate image magic for media handling. So we're now using live vs. And, and that provides better performance. So. But there's some sort of underlying stuff that's new for for admins there.

There's also some better stuff around metrics and management. So we've dropped deprecated stats. D. We've moved over to open telemetry. We've got Redis Sentinel support in there. We've got we've got some other nice things and some little nice. Touches like you can now customize your instance icon.

So if you are a hardware community, you can literally, without having to go and change templates, you can just choose an option to choose what icon you want. But to address your point directly, there are some features in 4. 3 that help related to spam and abuse. For example, moderators can now Search for problematic content using hashtags, which wasn't possible to do previously again.

We really I think the, the administrator community, the instance administrators benefit from things like talking to if task, getting some, some advice from them, getting some help from them as well. They've got some services that they, they help to provide in those instances, but but from the administration side again, we know we can continue to improve and iterate here.

It's been a, been a year long process of building the new release. And that seems like a really long time. It is a long time. The team's worked really hard. There's a lot of moving parts. And every time we think about wanting to add a new feature, it means having to, to, to make a choice about what we, what we focus on and things we can't do straight away.

So, you know, I think we've added some really nice things. We've got a decent balance of. Keeping a nice fresh modern look, adding some user experience features, making it easier for folks to get started, making things a bit easier to use, and then also helping administrators to manage their instances.

Finally, there's some, some Stuff features for developers, which will hopefully help them to plug in and improve the situation as well. So we've done things like really improving the authentication mechanisms for the API and enabling developers to also, if you're using a third party client, take advantage of those same notification filtering mechanisms that we provide so, so that they can provide those to their users as well.

Jonathan: Yeah, it seems like there's a, we probably need to, here in a few months, we probably need to have the folks from IFTTTAS on and talk about this because it's, it's fascinating to, to, and one of the reasons it's so interesting to me is because every other social network does this very opaquely. You don't get to look into it at all.

And with Mastodon, just by its very nature, it has to be very transparent. And that's amazing. That's it.

Andy: Okay. There's challenges with that though you know, we can't, I think most instance administrators We can't necessarily you can say that these are the kinds of content And kinds of things that are okay on our server, and you can be transparent about that.

I think that you, the way you get into difficulties, particularly with spam, it's back to the spam assassin, spam house, you know Bayesian learning kind of stuff, you know, as soon as you start to say, this is what we are filtering out, then the bad guys can figure out how to, to get around that. So You know, I think that's absolutely true, and I agree with you that one of the benefits of building an open platform, building in the open and working with others is that we have to be and can be much more transparent about those, those decision making processes.

But I think the technical level from certainly from my past experience. At one of those large platforms that we've referred to you know, there are challenges with, with how much you can share in public.

Jonathan: That's, that's actually a very, that's a really good point and not something I was thinking about, although it definitely makes sense.

It definitely does. I think that's, that is something that Oh, I forget the name. That's Well, is it, is it Spam Assassin That has some, some open source tooling around that. I

Andy: certainly, you, I mean, I'm, I'm going back 20 years when I'm, when I refer to Spam Assassin, I mean, well, that was when I ran my own mail server at home and, and that was a long, long time ago.

But I think that's

Jonathan: still, I think that is still one of the names in the business that do it. And they, they have that, that same problem is that they publish some of their filtering stuff. And so you can just grab it and write spam to avoid it There's another organization. Yeah, find the loopholes.

Andy: Yeah, exactly.

Well, that's that's that's the the loopholes is the challenge there's another organization that i'll mention because I think it's really important and it's another demonstration of the growth, momentum of the social web and and the fediverse is the social web foundation which was founded by Evan Prodromou, who's one of the authors of the activity pub specification, and he's been really active in the Fediverse for a very long time.

Mallory Nodal and Tom Coates. So they founded this thing called the Social Web Foundation, announced it about two or three weeks ago. And it really exists and is supported by a number of players in, in, in this space ourselves. As well as a ghost flip board, a number of the platforms that are integrating with activity pub, and it really exists to tell the story, help people to understand a bit more because, you know, Jeff, you kind of said, look, Hey, I'm aware of the basics, but I'm not quite sure I get it all right.

And so the social web foundation is there to help to tell the story more about, we've got this opportunity as users to. Make our choices, own our data, not be beholden to Jonathan's point to the decisions made by those big corporates that are, you know, very opaque. We get to the, the, the boring thing as users is that we have to do a little bit more brain work.

And, and, you know, my, my background, my degree was in history and I, I have, I have, humans are incredibly lazy. Okay. We are, we are the most lazy creatures we will, we will, we will let, you know, that's why it's sometimes it's very difficult to, to, to make those hard choices to say, I'm going to go and use something different.

It's going to involve a little bit more work to learn. Relearn some, some, some processes and the way things work because I'm so used to this easy thing that I was doing for the longest time to, to, to get out and change the way you do things that can be difficult, but there's so much benefit to it. And the Social Web Foundation is there to help to tell those stories and to say, look, you know, these are the benefits of using ActivityPub.

These are the, these are the benefits to you as a user of owning your network. I've switched Mastodon servers only once so far. I know, I know folks who've switched more often than that. And when I did that, I was able to go from one active mastodon instance to another and it moved my network with me.

I didn't lose followers. I just, they were all, all of their accounts at a protocol level, the protocol said, hey this guy's changed his address. He's over there now. And all of those accounts re followed me and I re followed them and that was super powerful. It's a really big difference to, again, the way that some of those other networks work.

Jeff: I don't think that makes. Good. Sorry. I was gonna say, I think that makes a lot of sense. I mean, as you know, Steve Gibson says, you know, having good security is inconvenient. Well, having control of the social media is maybe a little inconvenient, but it, you know, you, you're better off because you have that power.

And that's one of the things I do like about Mastodon, is I know that you're in much more control of what you, what you see, and, you know, you, you're not at the mercy of some conglomerate And, you know, there's many stories about how they've made decisions one way or the other that have upset people because wait a minute, you're filtering things.

I don't want to see. And as a Mastodon user, I could say, well, I'm really into hardware, I'm into the software, I want to, I don't want to see cat videos, you know, I just, I want to see any of that, you know, I can just block it myself and not have to worry about it. But one of the things I did want to say is like, okay, I'm going to get a Mastodon account.

Yes. Where, where's step one? Where, where, you know, as, as

Andy: a new user. So you. So go to, you can go to joinmastodon. org and there's a page for you to, to find servers by default. And this is, this was done primarily to reduce that mental. Cognitive load of making that choice because before and something we've heard before is people would go to joinmastodon.

org and it would say, here are all the servers in the world and you'd have to choose one, right? And that's the first thing that people tended to, well, exactly. Right. So, so, so first of all, doesn't matter where you start. You can move if you want to. Right. In the future, you can take your network and you can, you can move somewhere else.

So by default, it will, it will suggest mastodon. social, which is the one that the mastodon nonprofit organization that builds the software runs similar to the matrix project in a sense, right? They've got matrix. org, which is where I've got a matrix account. But if you, if you want to you can go somewhere else.

So for hardware folks, If you're into 3D printing, there's 3dp. chat, there's, there's there's bitbang. social, there's, there's hackaday. social, just saying, just saying, maybe have a nice chat with, with Elliot and get that, get that sorted. But If you go to joinmastodon. org, there's there's a service page where you can do things like filtering by by your geographic region that might be important to you for data privacy reasons servers based in the EU may have different.

Data laws than those based in the U. S. For example so it may be important to you from those perspectives. It may be a maybe a topic based community. You're looking for what I did was I moved. I was on Mastodon at Social. I started there back in 2017 or so with my account had a number of years there and then I ended up moving to New York City.

A server run by a former co worker of mine which is called mccaw. social, which has got a couple of hundred of folks that I, you know, used to work with on that platform. And they're all in their own places now. But I know the person that runs that server. I actually donate to him. I give Give him some give him a small amount of money every month so that, you know, it covers the running costs, help to cover the running costs of his, of his server.

And and, and I know, you know, if it goes away, he would let me know because I'm giving him some money. The, and, and, you know, he would explain what was going on. And that does happen sometimes with, with. Mastodon instances or Fediverse instances. But that would be my suggestion. Another tip I wanted to give you, though Jeff is one of the power moves on mastodon you can do is follow.

You don't not only follow. Users and then needing to know who all those users are. Although Macedon 4. 3, if you join a server that's running Macedon 4. 3, you get a slightly refreshed onboarding experience where we will help you to hopefully find some folks you might be interested in a bit more easily.

We do a little bit of additional recommendation to help folks fill up their. The following list a bit more, but one of the power things you can do is you can follow hashtags and that is super useful. You, you know, I, I follow a ton of hashtags for 3d printing, for ESP 32, for micropython, for Lego, for events I'm involved with and and then if I see those things coming into my timeline because I'll find out how there'll be a note saying why that's shown up, why I'm seeing this then I can actually go and choose to follow that user.

Here's another useful tip in 4. 3 that I didn't mention earlier. In 4. 3, we've got these little hover cards on the web user interface. So if you see a post from someone that you don't recognize, you can hover over their name and it's going to pop up a little floating card so that you can get a bit more information about them, where they are, for example, or what their, their profile links are.

And that's in 4. 3 as well. So again, we've really tried to focus on making it easier to find people that you might be interested in following or topics you might be interested in following. Thank you for asking that question because, because it is, I think it is one of those barriers to folks sometimes is either they've previously had an experience coming to Mastodon and having that, Oh my God, where do I start?

Or they just haven't kind of kept at it. And that goes back to my comment about. Yeah, there's a bit of relearning to do here, but it's really powerful and valuable when you, when you get into the groove and start engaging with it.

Jeff: Awesome. Well, now just say I decide, you know, and I'm going to run a server as well.

Yeah. Is, is there any, like, legal risk or burden with that or what, you know, could I get in trouble running it or? Sure.

Jonathan: Is California or Europe going to come after you for having people's personally identifiable information?

Andy: Right, so so look the the answer to that is it depends i'm afraid It's the honest answer.

I I work in developer relations. I I do my best to give those honest answers. If you're running something for yourself, then, you know, it's about the same as running your own mail server in terms of like What's coming onto your server and those kind of things, if you're, if you're running it for a group of people, your friends or whatever, or then you definitely need to be aware of who you're supporting, what data is coming in and what the rules are, what the laws are, and I can't tell you those, it's going to, it's going to depend on where you, where you're located.

Again, I believe the IFTTT knowledge base has some help on those kinds of topics. They've been such a great supporter for folks running, building and running the social web. So I would definitely encourage people to take a look there. And there are also people in their forums who can also probably point you in the right direction, depending on your, you know, your geographic location or so on.

But, you know, broadly speaking. You can spin up a container. You can go to what there's a couple of two or three hosting organizations that I can think of that will let you sort of buy a small bit of cloud container to run an instance if you wanted to, either for yourself or for a group, and they typically will then Do what a lot of these organizations will do though.

They'll charge you based on usage and size and and How many virtual CPUs and all that sort of stuff that you want and how much you might need to support the size of your community

Jeff: Okay, well that and that answers quite a bit now you said A little bit of cloud. So, and I know it's going to be based on users or whatnot, but say I had 20 users on my server.

Do I need pretty powerful or will it run on a potato or somewhere in the middle? Oh, Jeff,

Andy: you're actually, honestly, genuinely, you're, you're testing my, my ability to answer that directly. I'm going to go and quickly go see if I can find, find that out for you whilst we're talking, but I don't have that in my head.

Okay. Off the top of my off the top of my head, I think sort of for 20 users, you're gonna, you know, you're gonna want a fair, a decent amount of media storage, probably, you know, 50 gig or something. And and something like that. I'm looking at one of the pricing pages of for one of the hosting companies right now.

I'm not going to name them because people should go and check those sort of things out. But they are suggesting that sort of thought for a 20 active users But that would be their kind of mid tier package with, as I say, 50 gig of media storage and, and something like that. But, but please don't take my, my, my, Off the top of my head word for it because I don't have it in my head I've made notes on other things we might talk about but not those topics.

I apologize

Jonathan: I'm gonna i'm gonna ask you about something similar and you you may you may have to hit the eject button on this one, too

Andy: Well, no, listen, i'm not going to eject. I will definitely help out and answer these questions In a different

Jonathan: format later, but but yeah go for it so, this is something we covered on Hackaday, actually, you may have seen it.

The website was Itsfoss, which, great website, we cover stuff from there from time to time. And they made a request, please stop sharing our stuff on Mastodon, because Mastodon is unintentionally DDOSing our website when one of these big places do it. And what was happening, it's a real thing. There's There's more to the story apparently they didn't have any caching turned on for their website But what was happening was when someone with a lot of followers would would you know re do we still say retoot repost?

We would repost something from the site all of the federated so In, in Mastodon, you have this idea of federated followers. So, like, someone from your server follows someone from that server, therefore the server follows, gets the whole feed. Well, when that happens, when that server, when person A posts a link, server B, where someone follows person A, will follow the link and grab it to get, essentially, a summary, if I remember correctly.

But they'll do, they'll do a download of the website. And so what was happening is when someone with a lot of followers, or a server with a lot of followers, posted that link, you had, you know, dozens, if not hundreds of Mastodon servers around the world, sort of instantly doing a lookup for the website.

And it was, it was taking at least one website off the air. It was, it was taking them down. It was, it was a DDoS effect. They were getting slashed on it by Mastodon. Yeah. Yeah. It's so late. Is that, is that still a thing? As the message on devs, I, obviously you're aware of that. Have you all, have you all worked on that?

Yeah, I am aware of that.

Andy: No, I am aware of that. And because also I have my own small podcast I, I do weekly and, and, and that one sometimes has a similar kind of blip when I share our weekly episode URL. So look it depends on the size as you just kind of indicated, and we are aware of it.

We're working on it. We've got ideas. Something I want to mention since you bring this up is there's a new project that we've spun up which is funded by and Elna NGI and It's called Fediverse Discovery Providers, and the reason I mention this is because there's a, there's a, there's an additional element to that, which is called Fediverse Auxiliary Service Providers.

It all sounds very fancy

and

so on, and to be honest, it's not very user facing, but the idea is that you might have these services, which are not necessarily just Mastodon. The, the, Fediverse is all using ActivityPub, we've got other platforms, Lemmy, which is like a Reddit sort of similar clone. We've got PixelFed, which is an Instagram similar style thing.

All of them using ActivityPub and you can follow users across them and so on. And we want to contribute to the broader Fediverse. So there's this project that we that we are working on that is funded by NGI, NLNet NGI search, which is Fediverse Discovery Providers, which will enable in potentially for folks to discover content across different types of Fediverse instance more easily.

It's going to all be opt in. We introduced it at an event called Fediforum last month, and we're working on it in the open on GitHub at the moment. Now, the reason I mention it is because what. The idea of this is a Fediverse Discovery Provider could be an instance of what we're calling a Fediverse Auxiliary Service Provider and other things, the other things that Fediverse, future Fediverse Auxiliary Service Providers might do is provide better caching for situations like that, that would, that would, would prevent.

Or mitigate those kind of situations, there was also something we actually added in the in the previous release. And one of the dot point releases to 4. 2, which is that there is, there's a bit more randomization of a delay between when all of those instances are going to go start. Like trying to fetch the URL that's just being shared, so there should be less of a tsunami of requests that suddenly all come in, in a, in a very short period of time.

And, as you rightly say I think the particular instance you were talking about with, with its FOSS or whichever site it was you know, they came out and said, hey, you know what, okay, we, we weren't caching a few hundred requests in 60 seconds, you know, Most website servers are gonna, are gonna be equipped to deal with that.

As I say, I know that the podcast I have, we are running and, and it's terrible and I really need to fix this. We are running in a shared in a shared web hosting thing. That, that I haven't managed to spend enough time to get us migrated off of yet. So we only have a tiny slice of a of a, of a, of a VM somewhere.

That sometimes that sometimes does that does happen. And yeah, it's on my list of things to to resolve because I can't even do things like upgrading PHP and my SQL on that particular instance because of the deal we have with that host. That's my problem, not anybody else's. So again, yes, the answer is yes, we're aware of it.

Yes, we're working on it. We've got ways of mitigating it. Definitely want to improve it in the future.

Jeff: Speaking of the future, we do have a question from Discord and from MashedPotato. He says, is it still the case that quote posts won't be implemented on Mastodon?

Andy: No,

it's definitely not the case. And MashedPotato, if you go read the launch 4.

3 blog post, which I think I sent to Jonathan to include in the in the post later from Hackaday's perspective. If you go look at the blog post, blog. joinmastodon. org and look at the 4. 3 announcement, you will see at the end of that there is a reference to what we're doing next. And this is a really nice segue into that, actually, because 4.

4 4. 4 or whatever the next version is quote post is on the list. We are, we're doing it. At some stage there was a I think some folks felt that this was not going to happen. We recognize the community wants, wants Qt Quote posts. So that's something we want to do. We're trying to talk to other, again, activity pub implementers.

It's not just Mastodon here. The Fediverse is a, is a collaboration of different. Platforms and we want to do this in a way that that works and that is interoperable that extends ActivityPub if needed in the correct ways to enable this to be done well, there's a ton of things to think about, but we are talking about it.

Jonathan: That's a really interesting point. So the quote post is going to be an extension to, to ActivityPub.

Andy: I may have misspoken. We're looking at what I'm saying is we're, we're looking at what we need to do to make sure we do it well. And in a way that, that, that interoperates with other ActivityPub Services.

Right, right. So I'm not 100 percent clear yet on the exact data

Jonathan: format that it's going to look like. Sure, sure, sure. It might be an extension to ActivityPub, but at the very least, it needs, you guys need to think about the ActivityPub backend. So that brings to mind something actually really interesting about quote posts.

They probably don't have to be just Mastodon posts, right? Exactly. So there are a bunch of other services on ActivityPub. But there's there's picture sharing there's video sharing there's you know, yes think think of a Existing big social media service and someone probably has an open source equivalent that works with activity pub Yeah, and so this brings to mind this this This is actually really exciting.

This amazing thought of someone just posted a video on this other activity pub thing. I would like to share it, and it is a repost. And if you click the repost, it takes you over to the video. Like, that's super cool. That's really cool. It is

Andy: really

Jonathan: cool.

Andy: I love, I love, so I've got, I think there's another element to all of this, which is that we as an activity pub, Fed social web community can do more around accounts because I've got accounts on pixel fed, peer tube, lemi, mastodon my WordPress blog is, has got an activity pub plugin and so on, right?

So you can see content from all of those places under, under my username at different domains.

Mm-Hmm, ,

I, I can reshare. One of the things I love is I can, I can post my images on Pixel Fed, I post them on Mastodon as well. But if it's kind of more of a photographic thing that I want to exist in a picture fair sharing site, I'll post it on Pixel Fed and then I'll repost my Pixel Fed account, boost my pixel Fed account into my Mastodon feed so people can see that.

So I kind of love that interoperability. Mm-Hmm. . It takes some getting used to for people that are used to other platforms and the ways that things have worked elsewhere in the past but for them in the past, but I, I really appreciate it. I think it speaks a lot to the power of having this, this social web protocol and layer.

But yeah, look, one of the things I try to do is I try to attend when I can the W3C social web community group, which is being reformed into a working group meetings and take part in their discussions. I go along to something called Feddy Forum, which happens every six months, which is independently run.

By some folks and that brings together a bunch of the different projects that are interoperating in the, in the social web to talk about what's new or what they're building or how we can make things better on an ongoing basis. So, yeah, there's a, there's a, there's a lot to think about here.

You know, it's not a quite just a question of can you just implement quote posts?

Mm hmm.

Tomorrow, you know, there's we're a tiny team. Our team are the core development team is really small. We've got to keep the platform as a whole stable for lots of people fix bugs and, you know, solve problems. If if things get get raised, we need to be.

We need to respond to them quickly because, you Especially in the age of misinformation and spam and all this other stuff, we want to make sure that there aren't too many vectors for bad actors to misbehave. We want to make sure that things are running well. So we're doing, we're fixing bugs. And releasing patches to existing versions.

By the way, now that we've released version 4. 3, we'll be, we'll be retiring version 4. 1 in in the spring so six months after the 4. 1 series will be will no longer receive updates, but we will, will continue to support 4. 2 and 4. 3. We want to install to it to apply, you know, add new features.

We want to build for as broader community of people as possible. And we want to interoperate with all of those other platforms that are part of the Fediverse as best we can. So it's a, it's a, it's a big, big effort for a

Jonathan: small team. So I've got to ask about something and it may sound like a troll question and it only a tiny bit is that's actually fairly serious.

Are we doing anything with AI in Mastodon? Is LLM coming to it? No, no, no. Is there a future that you see where it could? And I think, I think I can see a place where it might fit, but I'm curious. What do you think?

Andy: Well, I personally. Could see a small number of places where something like an LLM could be useful.

However, it is not AI and LLMs are not at all on the roadmap or plan for the Mastodon core team, we are not in any way interested. And if you follow Eugene, who's the founder, the CEO on Mastodon, he's very, very clear that it's not something he finds acceptable. He's that he's interested in I don't want to speak for him too much on that regard.

He speaks for himself. But no, I it's not something that we are going to be going to be spending any any time on we've got much more interesting problems to solve I I'm curious what your thought is jonathan my thought and this is I want to make it 100 percent clear. This is not me speaking on behalf of the Mastodon team or project.

The place where I could see it potentially, two places I could see it potentially useful. And one of them is already being done by a couple of third party clients, which is doing things like image descriptions. Okay. Right. So, so alternative text for images where, where, where the, the, the, the, the, the app would suggest that for you.

But the, the other one potentially would be some element of recommendation or ranking, but a lot of people come to Mastodon and the Fediverse to get away from algorithms. Again, I've got a very complicated relationship with AI and machine learning. I work in a maker studio. I do some stuff with generative art plots, but generative art in the sense of running algorithms to generate interesting swirly circles, not, not taking stuff that other people have created and mashing up another image, you know, so there's, there's, I've written a blog post about that as well.

So again, that's me speaking personally. I don't think that that is something. That any of those things, are Well, I know for a fact that We have no interest in adding those sorts of things to to mastodon.

Jonathan: Sure. Sure, and that's that's fair You know the the the current, AI craze, we could call it, maybe it might be fair to call it a bubble.

People have very strong opinions on it. And I, I find that really fascinating. Personally, I am waiting for the bubble to pop so that AI and LLMs can become basically just another tool that people use as opposed to the big thing that everybody wants to throw money at right now. .

Andy: Yeah. I, I mean, I think that it, it, it's, yeah, you're right. I mean i've been in the tech industry for 20 years and I think that You know, we've been through a bunch of these waves but yeah, I I the thing that bothers me with it is the the clear intellectual property infringement aspect, in terms of like how all of this stuff has been built and trained Yeah and I and again speaking personally I am Very much struggling to come to terms with the way that You The big tech folks have railroaded their way to access to all of this data and used it to train their models.

I have problems with it. Yeah, I don't, but I, but I do find the outputs when you used well with an intelligent human brain. You know interjecting and saying well actually that's nonsense, but these three bits of information you've given me seem to be useful You know, I think I think that that it can be useful in that respect.

Jonathan: We had a we had an awesome discussion I was actually a little surprised at how good it was But just last week we talked with one of the guys from ibm and he's also working with the ai alliance And just a really good discussion about some of these issues And with with With Mastodon in particular, it kinda gets back to, to one of my thoughts about LLMs and one of the things that they're really good for, particularly the conversational LLMs, they work really well as a replacement for a search engine.

Particularly if you do things like in your prompt, you tell it to cite your sources, then it will, it works very well as a as a search engine and. That is interesting. And so you, you hit the nail on the head. The thing that I really, I was thinking about is in addition to, you know, just show me the posts from the people I follow, just show me the posts from the hashtags I follow.

And then you've also got this option to show me the posts from all of the followed federated servers, which is kind of like the fire hose. There might be an algorithm that says, show me the posts that are similar to posts that I've enjoyed seeing in the past. And I think that could be useful for people.

Andy: I think you're right. But let me give you, let me give a shout out to some of the third party developers because that's what I'm here for. Right? I worked in developer relations. My role. On the team. I'm not one of the core developers. I work with the core developers and I help to communicate what we're building to the third party developer community.

So let me give a shout out to some of our awesome third party developer community. There's, there's a, a web app called Fan, P-P-H-A-N-P-Y Social which is a third party web app progressive web app that lets you you know, browse your master on timeline, but it has a catch up feature. So you know.

If you, if you're not on mastodon whilst all of these non algorithmically sorted posts are coming in mm-Hmm, , then you might miss them. Right. But if, but, but if, but, but I might want a summary of the last eight hours of interesting posts. And FPE can do things like saying, well, you know, if that's had, if, if posts, posts that have had more than five likes or 20 boosts or whatever, that might make them interesting to you.

It can show you them in a timescale. There's another. Service I use called Mermel, M U R M E L, which does something similar. It sends me a daily email digest. There's another one to highlight things I might have missed or links that have gone viral that I might have missed because I wasn't watching my Mastodon timeline at the time.

There's another one called Fediview, which does more of the. Machine learning style, LLM style ranking rather than just doing what is you know, what metrics to reply. So these are places where I think having a, an open API, a publicly accessible, easy to use API and having third parties plug those things in is fine.

If that's what they want to do, and the data is being used appropriately and not just being, you know, churned into a Another LLM to generate more stuff. Right. And we know that those kinds of things can happen. But I think that, you know, we as a platform as a core team are focusing on making a, a, an easy to use, welcoming, better open social experience for people, for a large number of people for a broad range of people.

So, When there are features that are really good for power users, we may not spend as much time on those. We may delegate them to, or suggest that third parties can go build those things that have much more of a niche use case. That, that, that, that Feddy wall that I used at OBCAP this weekend, that was built by somebody else, you know, they built that tool.

We didn't need to build it into Mastodon directly. Not everybody needs to be able to run a wall of Mastodon post at an event. So why would we spend a load of time building something like that ourselves?

Jonathan: yeah, I've I've thought for the longest time and I was at first quite hopeful that twitter was going to implement this with the changes and They've only a tiny bit done So and and I guess I still have a little bit of hope for the future, but not a whole lot there.

Anyway, I would love to see a social network where you could bring your own algorithm Yeah. You know, exact, you know what the algorithm does, you have control of it over it, and it does exactly what you want it to. Well, and that's sort of, so Fed, fed view

Andy: kind of,

Jonathan: that's sort of what you're, what you're describing.

Yeah.

Andy: Yeah. Fed, fed View kind of layers that on, you know, they, they've got four, I think four built in algorithms that they, that you can choose from. Mm-Hmm. Yeah. Look, I, I, I'm not. I'm not opposed to that as a concept. A lot of people who use Mastodon would be completely, I believe, would be very opposed to any kind of algorithm.

Because all algorithms are opinionated in the end and have some kind of agenda. Oh, absolutely. If you are going to spend the time thinking about your agenda for applying your own algorithm. All good. Right. As soon as that gets taken out of your hands and is run opaquely, going back to our conversation earlier, that's when things can get dangerous or things can, you know, be misused.

And I think those are dangerous. But yes, I think you're right. I think let's see where the Fediverse goes next. I mean, there's a ton of interesting stuff happening.

Jeff: Absolutely. Brings up, you know, how do you decide what's going to be next on the roadmap? I mean, cause you talked about earlier community, you know, wanted something.

I mean, is there a place that. To give feed feedback or do you survey or, you know, what's up? How loud does

Jonathan: the community have to be before something is implemented?

Andy: Yeah, yeah, yeah, yeah, yeah. Well, I mean, we, we hear the community a lot and we appreciate them. And, and honestly we do. It's we've got, obviously we've got the atmastodon, atmastodon.

social, and we've also got atmastodonengineering atmastodon. social. That, that second one is one that, that, that, that, that, that. That we've spun up in the last 12 months, and we're also now posting every month. We, we share a monthly engineering update. It's our, our trunk and tidbits blog series which I, I co author.

So we try to help folks keep up with the things we're, we're doing. The last six months have all been about what the changes are that fed into the 4. 3 release. But the other thing I'm very keen to do again in my role is to showcase what the developer community beyond us is doing as well. So I always try to include one or two or three fun things that other people have built on top of Mastodon.

We're on GitHub. Obviously, all of our development is on GitHub. Being FlossWeekly I'm aware that other source control mechanisms exist. We really appreciate all the tools that are available to us on GitHub, and we've used GitHub for a long time, and we don't have plans yet, or any time soon to move off of GitHub.

But again, we hear the community sometimes who suggests that we do that. We, we, we hear, so we hear people talking to us on the platform. We hear people talking to us. We have a discord for people that donate to mastered on on Patreon. They get access to a discord. And, and the, The Patreon donations also, they get a newsletter from us.

I think we just posted one this week to folks that have donated to us via Patreon. We we have people raising issues against the project and raising pull requests. We've got a small team of amazing volunteers who help us to do some issue triage around some of those. But yeah, we, we need to manage a lot of sometimes competing.

Requests our launch blog post of 4. 3 mentioned a few of the key things we want to improve in the next major release, which would be those quote posts and also potentially subscriptions to block lists. We also want to work on better long form content display on Mastodon. So figuring out as more blogging platforms and newsletter platforms start to build in activity pub support, we want to figure out better ways of displaying that content on Mastodon, which is essentially just a.

A short form service and doesn't have plans to move away from being a short form service. We've got a new iOS developer who's joining the core team and starting very soon. So you should see some improvements. We've had, you know ongoing improvements to the iOS app, but, but you should see some more of those coming along soon.

Yeah, I mean, we, we, we we're working on it is all I can tell you. We're a tiny team and the one thing we haven't talked about, and I'm going to, I'm going to make a play for here is funding, right? We're a tiny team. We exist totally on donations. We do not take investment from folks wanting to drive us in particular directions that we've been very clear on, on, on not taking.

Those kind of investments. Okay. So we, we are funded through donations. We really appreciate every dollar that is donated this year. We spun up a 501 C three in the U S to make it easier for you all to to, to, to chip in and support the project. I will also say that If you can and you use an independent Mastodon instance, do try to support that, the running costs for that instance.

I do that with my own instance, but the core team to keep us going to, to, to, to make sure that the developers can eat and, and live and, you know and contribute to fixing, you know, critical security issues quickly needs to be funded. And, and I'm sorry to bang on about it, but we do appreciate donations.

Jeff: And is the core team like full time paid Just working on that I off

Andy: the top of my head. I think we've got three or four members of the team who are are paid full time on the team but you know, there's a there's about 10 or 10 or 11 of us in total including, you know somebody who helps with dealing with sort of admin and management sort of tasks.

We've got folks also that run, help to run, operate our mastodon. social, mastodon. online instances. But the, the actual number of people that are paid full time to work on mastodon is very small. It's, it's three or four folks. And there's an annual report. I think the most recent one was 2022. The 2023 one should come out soon and then there'll be one for 2024, which has You know, it's it's a it sounds really kind of Simple to say but it's been our biggest year yet You know, I mean we've had massive growth the the the the products come on and the platforms come on a long way and You know, we're trying to we're trying to make this thing sustainable and make sure that better social online experiences Scale and can exist for for a long time to come.

So Yeah Yeah,

Jeff: very cool. I was gonna And I will Good. I was just gonna say, I am going to get a Mastodon account. I will, I will do it. I, I know of a couple of stories. Listen, Jeff, if

Andy: you need, if you need any tips and tricks then, and, and you wanna ha get on, jump on a call with me, I'm happy to, to give you some, some of the kind of the, the wizardry that, that, that, that lead you through it.

And hey, also just wanna say, you know, you all can also, and I, they're coming to North America in the near future when we can, when we can get our our shipping sorted. But, but in the EU now, you can get a plushie to support us and, and you know, if you're missing. missing missing your mastodons, then, you know that's a great way to support us as well.

Yeah. Awesome.

Jonathan: I was going to ask about the transparency and the funding. And so I assume that's what that annual report is. That's where people can go and see here's the money I donated. Here's what it went to.

Andy: We're a nonprofit. Incorporated in Germany, and there's a, there's a, there's a non profit board 501C3 in the U.

S. And of course, both of those things require us to be transparent and to provide reports. So as I say, I believe the most recent one that's available on the website is 2020. 22, the 2023 one should be available. And of course, but that will, that will be an ongoing thing. Yes, absolutely.

Right.

I don't think I'm a line item in the existing report yet.

And I've been working with the team for 18 months. Right. So and to be clear, since we're talking transparency, I, I I work freelance. I do one day a week of my life is spent dedicated to the Mastodon project. I have a, I have a, I have a discounted day rate because it's a non profit that I that I offer on top for that and and the rest of my life is currently burning really honestly currently burning down my savings So if anybody else in open source wants to hire me, for the other four days of the week, then then here I am

Jonathan: Yeah, very good Maybe, maybe somebody will, hopefully.

I understand that. I too, I'm in the, I have many, I have many irons in the fire. I do a lot of freelance work on various different things. And so I understand how that goes. And when one of those goes away or several go away, the trepidation you feel, I get that.

Andy: I also

Jonathan: want to be clear,

Andy: I want to spend all of my time on Mastodon, but you know I love, I love this product, this platform, and I love what we're building as a, as a, as a free and open source standards based, better social web platform.

Yeah,

Jonathan: absolutely. Is there anything that we didn't ask you about that you really wanted to make sure and cover?

Andy: Wow. Let me just spin through my list. I think we've covered pretty much everything here. I feel like we did get

Jonathan: through it pretty well. I hope,

Andy: I hope you both are I'm delighted, Jeff, to hear that you're planning to give it a, give it another try or give it a try.

I hope, Jonathan, you'll spend a bit more time interacting with us and looking forward to Elliot getting Hackaday. social upgraded and seeing all those great hacks.

Jonathan: I'll go bug him about it.

Andy: Hey, listen, listen, the other thing to do, Hackaday. Articles should get the Fediverse creator tag added to them, and then you'll get the little author attribution whenever a Hackaday link is shared on on on Mastodon, you can get the author attribution added if you're if the folks that have written that article have the Fediverse creator pointing tag in the in the page pointing to their Mastodon account, then every time one of their articles gets get shared, they'll get a little find more from Jenny or Oh my gosh, my brain is blanking on, on, on all the, the great folks on, on on Hackaday.

So, Al and Elliot

Jonathan: and me.

Andy: Yeah, of course, you know, I listened, I, I listened to, to your podcast weekly, so I should know all the names. I'm just, I'm, I'm so excited to be here that

Jonathan: my brain is going blank. No, no, that's fine. Okay. I got two final questions I've got to ask you then. And that is what's your favorite text editor and scripting language?

Andy: Vi and scripting language. Gosh . I mean, I, I just, I use bash all the time. I mean, I, I write code in Python typically, but my scripting language would be, if we're not calling Python a scripting language. It's a programming language. I would probably say Python bash.

Jonathan: Yeah, that works. I think either of those are, are legitimate answers.

Andy: Thank

Jonathan: you, Mr Andy Piper. Thank you so much for being here. We went, we took up more than an hour of your time and I sure appreciate it. It was great.

Andy: I hope the audience of stuck with us and I appreciate you. You both and floss weekly and hackaday is a match made in heaven for as far as I'm concerned. So this is this has been great.

Thank you for the opportunity.

Jonathan: Awesome. Thanks again. Wonderful. Yeah. All right. So the real question is, what server are you going to sign up on?

Jeff: Oh, that's a, that's a tough one. You know, You've got, you've got

Jonathan: hackaday. social. There's twit. social. We're all good friends with Twit as well. And I'm sure there's some Indian motorcycle that, you know, Massadon server that's just out there for classic motorcycles.

Jeff: Yeah, there probably is. But you know, I, I, I got to stick close to home and it's, it's either going to be a Twitter hackaday. One, one

Jonathan: of those two. You could do it like I did and make one on both places. Oh, you can do that? I mean, there's nothing stopping you from doing it. Now you can't, of course, we let, we let the man that knows the answer to this go before we thought to ask the question, but I don't know of a way to sort of merge the two accounts on two different servers, but I do know that a lot of people have accounts on different servers to sort of keep their various interests separate.

Just something to think about and see whether it makes sense for

Jeff: you. True. I'd probably at least just start with one and then kind of get, get my legs under me and you know, see, see how I do. Maybe, maybe give Andy a call if I need to, to get the, get the crash course and. Yep. But yeah, I, you know, I, I will have one might take me a day or two just to work it, work it in with my day job.

But yeah, get on there, get one. And then I can advertise it like other people we know.

Jonathan: Yeah, yeah. Well, make sure and let me know what it is. If we do it before the show goes live, I can tag you in the Macedon post. Oh, nice. Yeah, yeah, it'll be fun. I will do that. Alright, awesome. You have anything you want to plug?

Jeff: Just check me out on, in twit. tv on the Untitled Linux Show. I'm there almost every week. And yeah. I talk with Jonathan and some other great people that co host and we just Strictly talk about pretty much Linux though. I do veer a little bit into the hardware, you know, benchmarking on Linux, the new processors and some other, other things like that.

It's a little, little of the enterprise. My, my day job, I work in semiconductors. Bleed over into that, but no, no, no inside scoops though. No, no. I, if anything's gray, I always say this is strictly what I've heard on the internet. This is not inside knowledge. This is, you know, I do, I do not speak for the company I work for and I don't mention

Jonathan: them either.

Indeed. Yeah. I don't think we've ever, Maybe once we've mentioned what company you work for on the air, but that's just, it's not what we're about.

Jeff: All

Jonathan: right. Excellent.

Jeff: Listen to

Jonathan: all the episodes you probably figured out, but. All right. Thank you, man, for being here. I sure appreciate it. We we don't, we don't necessarily know.

who the guest is next week. And so if you want to be on the show, if you have an open source project, you want to chat about, or if you have recommendations or leads for us, you can shoot us an email, it's floss at hackaday. com. Shoot us an email that goes straight to me. It's a, it's a mailing list technically, but it comes to me and we'll take a look at it and get some folks scheduled.

So we need the guests for next week. We've got, I think one guest scheduled through the end of the year. So let's let's get the suggestions coming in. But we will be here next week, whether it be a guest or a roundtable, and we will chat about what's going on in the world of open source software.

Also, you can make sure and follow Hackaday. You've got my security column goes live every Friday morning, and we sure appreciate Hackaday being the home of Floss Weekly. You can also check out as As we mentioned, the Untitled Linux Show over at twit. tv and I personally appreciate Twit letting us do that.

We appreciate everyone being here. Thank you for those that watch live and those that get us on the download. And we will see you next week on Floss Weekly.

This week Jonathan and Jeff chat with Andy Piper about Mastodon! There's a new release of Mastodon, and plenty on the road map to keep everybody excited!

• https://joinmastodon.org/
• https://shop.joinmastodon.org/products/mastodon-plushie

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week, Dan joins me and we talk with Anthony Annunziata about open source artificial intelligence. Anthony is the director of AI Open Innovation at IBM and the co founder of the AI Alliance, definitely an expert in the field. He entertains all of our questions about AI in general, and then talks about what open and open source means for AI models.

It's a great interview and you definitely do not want to miss it. This is Floss Weekly episode 804 recorded live. Tuesday, October the 8th, the AI Alliance Asimov was right. It's time for Floss Weekly. That's the show about free Libre and open source software. I am of course, your host, Jonathan Bennett, and we've got something fun today.

We're going to talk about AI or. Maybe it's LLM, maybe it's just big computers doing lots of math. We've got, we've got somebody that is sort of the expert on this. But first off we've got we've got Dan the man, the original Linux outlaw. Mr. Mr. Dan, how you doing?

Dan: I'm good. Thank you. I've got to confess.

I slightly worried then when you said we've got an expert in all of this Because I can't confess to be an expert on ai or llm large language models, but I try I try yeah, it's good to be back

Jonathan: Dan, you're probably a lot like me. You don't necessarily claim to be an expert in much of anything. We're just we're jack of all trades.

We are very generalists, right? Is that kind of where you're at too with all this?

Dan: Pretty much. I think it's dangerous to claim you're an expert in anything really, because then somebody will go, aha.

Jonathan: Are you really? It's, it's the what do they call it, the Dunning Kruger curve, where it's like when you first learn about something, yes, I know all about this, and it's easy, why are you guys making it so hard?

And then as you learn more and more about it, you finally get to the point of, oh, this is ridiculously complicated, and I don't know anything about it. And that is when you have actually started to understand.

Dan: That's very true.

Jonathan: I think, I think AI is very much like that or LLMs. I sort of resist calling this AI.

I don't think we're close enough to a general artificial intelligence to really refer to any of the the things that are out there as AI, but obviously people have different opinions about that. So we're talking about the. AI Alliance and sort of kind of also IBM and what IBM is doing with AI our guest is Anthony Anthony and Azuela, maybe we'll ask him how he pronounces his last name That's probably

Dan: not

Jonathan: right that's probably not I I'm I made the old college try I gave it my best All right, well, let's go ahead and bring him on since we've got the man right here Anthony welcome to the show

Anthony: Hey guys, welcome.

Yeah, thanks. It's great to be here. I really appreciate the invitation.

Jonathan: Yeah, we're glad to have you here. How do you pronounce your last name? I'm sure I butchered it. Ah,

Anthony: A little. Yeah, it's Anunziata.

Jonathan: Ah, yeah, I, I, I way Americanized that.

Anthony: All good. All good. Yeah. I think I Americanized it a little bit too, but you Americanized it even more.

Jonathan: I turned, I went, I, I, I am the American that's turned up to 11, I guess. So we, we talked briefly before the show and you mentioned that you sort of dual represent for, for what we're talking about today, being both part of IBM's AI effort and the AI Alliance. You want to kind of maybe start by mapping out what that means, you know, what those, those roles are in those two different places, what we're talking about here.

Anthony: Yeah, sure. Happy to. So, yeah, as you said, I kind of served two roles. You know, my, my main role, you know, where I'm employed is at IBM and I lead open innovation in AI here. That means a lot of things that includes open source data and models as well and partnerships and community engagement.

And I'll tell you more about, you know, specifics there. And then, you know, as part of that, as an extension of that, I have a role in the AI Alliance, which is a program that IBM and a number of others put together about nine months ago. And that that's an open innovation, open source, open program that is intended to promote in many ways open development.

And adoption of AI in a, in a responsible way. So I have, uh, the position of co chair of the steering committee and direct involvement in a few of the working groups as part of that organization.

Jonathan: Yeah. All right. Interesting. So I think maybe the thing, the thing to go to first we were talking before the show, I think, I think it's before you joined the call even but last week I had Simon Phipps on as.

Co host slashed guest. And he was talking about how the, the the open source definition for artificial intelligence, for AI, for open source AI, is, is sort of something that's still being written. Because it doesn't necessarily work the same way that source code does. And I'm curious what, what the, sort of the, the AI alliance and IBM take on this is.

Have you guys been involved with trying to write that definition? And what are your thoughts on this?

Anthony: Yeah. Wow. Great topic. Okay. So lots to say here from an IBM perspective, we have been in, you know, in discussion, dialogue, providing feedback to various efforts, right? That are trying to better define what open source means in AI and in particular, like what the source term means and whether it's actually applicable or it's a different, different sort of thing.

So, so, yeah, we we've provided feedback. We've been part of that dialogue. I think it's fair to say today that you know, the community is moving forward to try to understand what open source means. And I don't think it's an answered question yet. And from an alliance perspective, right? More broadly, that's kind of a microcosm of people's broader involvement and opinions, right?

Lots of orgs are starting to weigh in more specifically. But a lot of people are concerned, you know, if not the whole picture, then let's, let's define enough commonality, right to be useful. And, and often that, that rests around, like, what is a, what is a model, right? What is an LLM? What is an open source LLM?

Does that term mean, is it meaningful or is it something different?

Jonathan: Yeah, and so where, where would you draw the line? What, what definition do you like for, let's say we're talking about a, a particular model? If we wanted to define this, so like, LLMs obviously is one of the big things that people are working with right now.

And so if we want to push one out that we say, this is, you know, put the stamp of approval, this is open source. What, what should that mean? What should that look like?

Anthony: Yeah, so the way I like to do it, you know I like to go back to the fundamentals of open source, right? Is, what is open source? Enable you to do what kind of actions ought you as a developer be able to do right?

So, you know, it comes down to can you study modify use share the artifact? Like, we've become very comfortable with that means in terms of code, right, source code, but a model, an AI model is something pretty different, in particular, the types of models today, right? They're pre trained, they're very large and they're, in some sense, kind of compiled objects, right?

They're not, they're not really sourced, but there are elements that kind of, sort of, are source like within them. So, the way I like to do it is, is, you know, look at the model as an artifact and then look at, you know, kind of the broader picture of what an AI system is. Is and how you how you get to a model, right?

The whole training regime. So the model itself. is is primarily a set of. So in general, they're neural nets, right? And neural nets that have weights for the different nodes. And so a key part of what it means to have a model that's open is the weights. All of the weights, right? All the parameters are open, accessible in a way that you can study, modify, use and share them.

Without encumbrance, right?

Jonathan: Mm-Hmm. .

Anthony: And so I think the minimum, right? Practical definition is really what is an open model or, or an open weight model. And it has to be at least the weights, right? The weights after training. Mm-Hmm, . So that, those are the, the, the key descriptors of the neural net and what it takes to run it.

And then some basic information, at least about you know, the, the characteristics, right? The the behavior. Of that model, which is a set of neural a set of weights in a, in a neural net architecture, that's like the bare minimum. I wouldn't call that open source AI yet, though, but that is really like an open model in my view.

And I think an open model is a piece of the story. And it's in some ways the most important piece because it's the new And weird and hard to understand and sometimes hard to use piece.

Jonathan: So in, in thinking through this, as I said, I am, I am by no means an AI expert by no means. I am sort of just an outside observer looking into all this.

But one of the interesting things about, about these LLMs is they can very. they can very greatly, based on what training data goes into them. And in, in sort of thinking about this, you know, openness, open source, yes, but also like openness trying to understand them. One of the things that would be really, really important is maybe not having the entire data set that trained them, because as we know, that in some cases can be prohibitively large data sets.

But at least having an accurate descriptor of that data set, or, or maybe even. You know, how was this data set modified before the training happened? Because certainly that, that must happen, because there, there's, there's, in data sets, there's spurious data, right? There are things, to put it simply, there are things on the internet that you may not want your LLM to have inside of it, right?

And, Like, that's, that's a valid thing, but at the same time, if we're going to be open about the model, and if we want it to be something we can call open source, certainly that's something that we need to understand from the get go, right? Is this, I assume this is part of, part of the thinking about this as well.

Anthony: Yeah, for sure, for sure. So a big part of building a model, a big part of understanding behavior model is the data sources, but more importantly, the whole pipeline of processing that the data goes through before you start to actually train the model. So, one of the things that is really important. is transparency, right?

Short of, you know, permissive use of the actual artifacts and pieces of pipeline. Are you at least transparent about, you know, where your data are from? But more importantly, the various filtering and processing the data that's done before it starts to train the model. Yeah, for sure. Super important. And yeah, the types of things people do that are really important and that are kind of emerging standards are Are you need to, you know, you know, there's a whole bunch of processing in terms of formatting and getting the data ready to process.

But key parts are, you know, removal of hate, abuse and profanity removal of personal identifiable information. In most cases, most responsible cases, checking against known copyrights and removing those these are really important. And, you know, even short of like what it means to have a full. Open source AI system, right?

Because this is an emerging, you know, kind of definition of discussion. I think what I think even more people converge on agreement on is it's very important to have transparency. On how you process your data, the choices you've made. Right. And people may disagree with those choices, but at least articulate clearly the choices you've made in cleaning and preparing the data to build the model.

Jonathan: Yeah, absolutely. You know, you think about something that when you're doing scientific endeavors that you have to worry about is the different biases and I'm using this term more of a technical term, the different biases that it's in your data set because. You may think that you're just cleaning your data, but you may actually be removing some of the information that you're looking for and the same thing can happen with these LLM models.

And so like what one person may, may have the opinion that, oh, this is, you know, this is hate speech, or this is just noise in the data, or we want to get rid of this, like, That changes your output in ways that in some cases we don't fully even understand. And so I love that transparency is a core of what you're doing.

So I think that is super important with, with this technology, with this becoming more, more universally useful and not, not misleading people even. So I think that's great. Let's see. So I, I, I want to ask you about this and I know it's, it's controversial, but again, we have the experts. I'm going to pick your brain on things.

And that is the idea of copyright when it comes to LLMs and AI. And as far as I know, this has not actually been settled in Court and I think this is probably what's going to have to happen, right? This is these are questions that are going to have to be settled in court or by written laws But the idea as far as I know is that it is believed that Taking information into as as a training process for LLM is a transformative work To the point to where you're no longer covered by the original copyright of the information that you trained on.

And then a quirk of that is the output, you know, so you, you write a prompt and you give this to an LLM. You get an output, whether that's an image or a written work. That is then sort of not a copyrightable work. And, and those are, those are interesting quirks of, And feel free to correct me if I got any of that wrong, but those are interesting quirks of working with, of AI and LLMs in this day and age.

And do you think that's going to stick, or are we going to see laws or court cases that change that? I know, you were not prepared maybe to give your legal opinion on things, but, it's, it's, it's so integral though to all of this.

Anthony: Let's dig in, man. No, I'm happy to. First, I'm not a lawyer. I don't represent, you know, IBM's legal opinion or AI alliance members and all that, so.

Right, right. So we'll we'll just go under that caveat.

Jonathan: Yes, sir.

Anthony: Some things are clear, some things are not clear. And some things are in between. I'd say there's reasonable there's, there's reasonable agreement that if data are out there in the public domain already, that, you know, if you train a model on those data, that that is a reasonable case of fair use, right?

But you have to make sure that you don't redistribute those data that you haven't gone and you know, gathered those data in places you shouldn't have, and so on. So there's many caveats there, right? And that's why, you know, what I mentioned earlier, like, Even though you maybe don't strictly have to, it's good practice to check your training data against known copyrights and make sure they're removed,

Jonathan: right?

Right, right.

Anthony: There's also progress and, and, you know, methods coming available that allow individuals in some cases, some of the, the more, the very open model building efforts, I'll name one from an alliance partner BigCode. And in the big code community initiative to build open, open way models they provided a very nice mechanism for people to go in and and look for data from, from them.

Right? And, and request that it be removed just voluntarily. Not because it's required to do that under law, just because it's good practice. It's good community practice to enable that to be a possibility. So that's kind of the input end, right? Training. I mean, there's, there's lots more to say about that.

I'd say the output end is actually a little clearer in some ways, right? Because in some sense, like copyright law is copyright law, right? Like it doesn't change, right? If someone, if me as a person goes and gets a piece of text or a piece of art that that, you know, is, is substantially similar to a copyrighted work I can't go go use that, right?

And I would, I would be violating the copyright, right? It doesn't matter if I drew it up on my own or I got an AI model to do it or, or something else, right? Like if I go try to pass off something that is really similar to another person's copyrighted work, I'm in violation.

Jonathan: Draw me a picture that looks sort of like the Mona Lisa.

Anthony: Yeah.

Jonathan: Yeah. That sort of thing. Yeah.

Anthony: Now where it's murky, right? You know, the question of, you know, whether whether a I, you know, substantially changes the risk and liability picture in terms of copyright violations, right? Is it still just on the user and on the person that goes and tries to distribute?

Something that violates somebody else's copyright or is there some implication because the technology is so capable You know to the the technology builder themselves. So that's that that's kind of some of the debate that's happening now.

Jonathan: Yeah Yeah, I know there are some open source projects that have just said because of some of these issues We will not accept any code that has had Like, you know, Copilot, just for instance, Microsoft's Copilot.

We will not accept any code that has been generated in any way by this tool. It's because they're afraid of that idea of the copyright. You know, there are, people, two people demonstrated this. Like, there are certain prompts where you can get bits of code out and then you do a search for that code on the internet.

And you can find, you know, substantially similar, if not identical, character for character, identical copies of the code. And so I know there are, there are places that are just saying. Don't bring us any A. I generated anything because we're afraid of the copyright problems. And of course, there are there are there are other problems, particularly with code, because people have, they misunderstand the L.

M. Tool. And so they think, Oh, well, co pilot wrote this. Surely it must be good code. And you know, we see the same thing with vulnerability research. Find me a vulnerability in this program and then people will try to report it. And the, the problem, particularly with that one is the LLMs do such a good job of making it look good.

But when you finally dig down into what they're telling you, it's, it's almost, almost all the time bogus, but it's, it wastes a lot of time from these maintainers because they've got to do the work because somebody sent them a vulnerability. Yeah, it's, it's fascinating stuff. Yeah. It's really interesting.

Anthony: It really is. Look, I mean, that's, you hit the, hit the nail on the head on a couple of big, you know, big challenges and opportunities, right? Like on the output end of things, there are better and better methods to try to screen detect and block, you know, copyrighted material, but it's, it's definitely not where it needs to be yet, you know, various guardrail schemes and detector models and all that.

So, so that's a, that's a big opportunity and something the Alliance from a safety and from a quality eval perspective is working on as IBM as part of that. And then the other piece I think is really interesting too, which is like, Hey. Yeah, you're right. LLMs are pretty good, but not great at many things.

And, but the, but they're, they're pretty great at making themselves look like they're really

Jonathan: great,

Anthony: right? So whether it's, you know, better formatting, markdown, output, things like whatever, it looks pretty good, but when you dig deeper, it's not quite all the way there. Right. That's true for code. It's true for, for text and natural language.

So there's this problem and actually the big problem with adoption, right. With lots of companies that want to use AI. You know, you can get 80, 90, 95 percent accuracy and reliability, but getting to like high accuracy in use cases that need it, which is a lot of enterprise use cases, it's pretty tough, right?

It's pretty tough.

Jonathan: Yeah. One of the, one of the funniest things. You know, people have come up with different tricks for interacting with AI, which all that's, that's a whole subject in itself. Maybe we'll get to here in a minute, but the one that, that I think is just about the funniest is with, with a lot of the models, you can tell it at the end of your, you ask a question.

And at the end of the question, say, show your work, cite your sources, and just adding that to your prompt, the quality of result goes up dramatically. And it tends to be more accurate. And I think that is one, it's, it's hilarious. But it's also super interesting that it works so well. Is that something that you see, like, broadly?

Does trick work across a lot of different models?

Anthony: It actually, it works a lot of, some form of that works across a lot of models and a lot of use cases and modalities, actually. So I think some generalization of that is, you know, think about structured inputs, right? How you specify your prompt input has a big, big effect on the output quality and the output, you know, the output in general.

You see it in code, right? If you can better set up the problem you can get higher quality code output. If you can better set up, okay, so there's a, there's a whole set of patterns called retrieval augmented generation, which is where you hook up an LLM. With a database and the database is a vectorized data.

So it's in the form of where an LLM can kind of knows how to interact with it. I can go into much greater technical depth, by the way, if you want. I'll keep it light for now. In those cases, right? We also find that the better you structure that that vectorized database, right? The better you structure the data.

Whether it's in a graph form or some other sort of structured form. Yeah, you get much, much higher quality retrieval and output. So yeah, there's a huge, huge amount of, you know, sometimes it's called prompt engineering, structured inputs so on and so forth that are important and and determine the output to a big degree.

Jonathan: Yeah,

that's interesting.

Dan: Yeah, I was curious when you talked about copyright there. It made me think a little bit of I relate a lot of things to the music copyright because that's kind of the world that I come from. But when you were saying about a lot of projects won't accept code, which has been generated by an LLM.

I understand why I completely understand why and the fact that you can maybe find I don't know, a github repository or something where you end up finding a very similar code. But at the same time, I was thinking to myself, there's only much like in music. There's only so many notes that you can combine and so many ways to combine those notes orders.

You can put them in and so on. Maybe this is something down the road that We'll hit a point where, you know, there are only so many ways to make a certain piece of code and only certainly ways to combine different functions and features of the languages that you're using, the programming languages that you're using, and so on.

So I wonder if some point down the road they'll find a way of doing that, because with music, again, I'm not a lawyer in any, in any sense, so please don't take this as legal advice, but it's certainly in the UK anyway, in music copyright, there's, there's laws about how many notes in a row. Can be the same as another piece of music, for example, it's usually six notes or roughly six notes.

I wonder if that might be an approach in the future, but you're dealing with such a large data set that I would imagine you get to the point where you need another, you know, I don't know. I'm not sure where I'm going with that point, but I think maybe at some point in the future we might see that and it'll have to be tested in, in in court as well as all these things are.

But with regard to things like copyright one of the big things that I've seen recently is I have a lot of friends who work in academia work in universities, places like that. They have a lot of Tools. Now, there's a lot of companies trying to sell their tool, which will check your students work and check that it's not generated by an LLM.

I wonder how I just wondered if you had any thoughts on that kind of area and whether any of that could really be effective. The checking of it, I suppose. Yeah.

Anthony: Lots to say there. I think some sense is a little bit of a losing battle to try to just take output and you know, work back to see if it was created by an AI model.

I mean, yeah, you can still do that, right. There are patterns there. You know, recurring words and phrases and things that you can map back with some statistical probability to a model, but I think as models get better and better as you can adjust things like the variability of output responsive to the use case and lots of other stuff.

I think it's getting really hard to do that. So I think in education, we're going to have to figure out a different solution. I mean, just like yeah. It took you know, some challenges, but we figured out how to do math education without a calculator or the right role for calculators and, you know, the right role for in classroom work that specifically, you know, prohibited calculators, you know, in some sense, we're gonna have to figure out how that works now for not just math because I can can assist with with you know, a lot of tasks in the educational setting.

Yeah,

Jonathan: yeah,

Anthony: I can make it. Yeah. So there's some thoughts on that. I mean, it's certainly not Yeah, I don't know fully how that's going to play out.

Dan: That's okay, I kind of threw that one at you. Don't worry, I'm aware that I just kind of threw that one at you. Yeah, it's very interesting. I mean, there's one more.

Anthony: Yeah, I mean, I guess I could say one more. I think just on the topic of it kind of relates to copyrights and original work and cheating and all of that. You know, I think increasingly the, the path to show provenance and originality of, of work is going to be to track the lineage and provenance of non AI.

generated material, because it's going to be so much AI generated material, synthetic data, images and so on. And so many ways and sources and ways to manipulate that. I think like probably a better tack is to have better end to end methods to track, you know, data and works that came from, you know, not AI.

Dan: Yeah, that makes sense. It's like a litmus test, like a, you know, something to compare it against. It makes sense. So one of the things I found really interesting when I was looking through the AI Alliance website and other things is I noticed a little bit at the bottom where it says let me just scroll down so I can find it.

Competition law guidelines. I'm sorry, but to get into law again, I don't want to realize we're not here to talk about. I just found it. It led me to some thoughts because it says you've got guidelines published there for how people can interact together without. Contravening something like anti antitrust laws, competition laws, all those sorts of things.

How so I suppose my question is with so many different bodies working together in the AI alliance, um, what are the challenges to kind of get them to work in together and how much can they work together without it being seen as colluding in some way in the market? In the market.

Anthony: Yeah. Hey, this is great.

My my, my lead attorney, John McBroom would, would love this question. And you noticed his finely crafted competition log guidelines. Yeah. It's really important. What we put together in the Alliance is a collaboration of various organizations, but it's a tighter collaboration than just kind of individual at open source.

Right. And because of that, right, we need to pay extra kind of extra attention to making sure that. That everything we're doing is truly open that it's not you know, even close to the kind of you know, cooperative, commercially inspired kind of work that we can't be doing. Right? So a lot of this is just basic hygiene of open source.

You know, do you work in the open, publish everything early and often, right? Use GitHub and, and, you know, known processes for contribution management and all that, which, which we have you know it includes, you know, publishing. Working groups you know, meetings, events, just just making sure that, you know, it's all in the open.

People know it's not about keeping anything in the dark or hiding anything. It's really just about bringing people together and orgs together, you know, just for tighter, more coordinated collaboration on open work. So there's a lot of, you know, process mechanics, hygiene, publication kind of stuff. That's, you know, just mostly good open source practice.

I'd say that, you know, to the other part of the question. You know, what are the challenges? I mean, there's lots of challenges. This alliance is actually much bigger and grew much faster in terms of members than we were expecting, which is always kind of a nice problem to have. But, you know, what that means is there's even more interests and projects and priorities.

And so, you know, the way we've, we've handled this is, you know, we've structured the program. Hey, my earbud came out. Can you still hear me? Yeah, we're good.

Jonathan: Yeah. Yep.

Anthony: All right. Good. Yeah. Sorry. I'm not used to

The way we've structured the program right is is pretty lightweight and ground up and we've got six Focus areas each with multiple working groups and projects. And it's it's very you know, maintainer contributor leader led, right? And so there's typically, you know, one or a few sponsoring organizations for each project.

There's some individual leads just like a good open source project. You know, a road map. And so what we try to do is, you know, preserve a lot of ground up, you know, kind of individual led work, but we provide, you know, collaboration you know, pooling of resources, kind of, you know, a forum for common priorities and, and, you know, that kind of program structure to kind of better support and scale that kind of work.

Dan: Yeah, that makes a lot of sense. One thing I was, I was interested in, as I mentioned, to do with academia and all that kind of stuff is how you're engaging. I noticed you've got Harvard listed on, on, on the, the members of the AI Alliance and so many other academic area, you know, institutions, I should say.

And you've also got things like you've got NASA on board and, and As well in Switzerland, so I was curious about how you're engaging with academia and so on, and is there any difference between dealing with, say, a big company and dealing with a big research institute or something like that?

Anthony: Yeah, there's some differences. I think probably split it into two parts. In engaging academia, right? There's the teaching and education mission, and then there's the research mission. So in education, this has been about engaging and figuring out, you know, where are the gaps in education where the gaps in particular, not just.

Curriculum, but in resources for students, right? So open source is perfect, right? To address this, right? Because you have lightweight code that you can deploy. You have endpoints you can use with, you know, free access to experiment with AI models and so on and so forth. But, you know, it's a little bit of a mess.

And so what can we do here to better guide, to better organize? Resources right for for students. That's that's been a been a major thrust, and there's there's a lot of work in progress there. So curriculum piece and resources piece on the on the resort. I can mention I can mention specifics to like one specific thing that working on is a kind of a collective guide to a definitive curriculum you know, in A.

I. So we're gonna be building that out and releasing that later this year. On the research side, so this is, this is a, this is a thorny problem, right? Because the resources required to engage at the leading edge of AI are getting higher and higher. So, you know, I can't say we've fully solved this problem in any sense, but the way we're going about it is to try to bring academia and industry together in close collaboration.

And I mean close because, you know, often what you see is, you know, industry will sponsor a student or sponsor a faculty member to do something kind of on their own. Or you know, if it's the other way, industry will get involved in a university, you know, they'll be kind of an observer, they'll join a meeting once a quarter or something as part of an institute and all that.

And that's fine. You know, there's benefits to that. But what we're really trying to do is bring, you know, faculty and students with researchers and engineers and companies together, right, closely collaborating in open projects. And We've gotten some nice things going on that, you know, in that mode.

In particular, in the area well, in a few areas, but one I'll highlight is in safety and trust. So, this is about building better tools, better methods to detect hazards of, of AI output you know non idealities of AI systems, whether they be, you know, hate, abuse, profanity, and things like that, or whether they be, you know inaccuracies that in some settings, like in health, would, would cause real problems, right?

And that, that area, we've started to see some really nice collaboration among, you know, academic perspectives and industry perspectives to try to make progress and those sorts of things.

Dan: Yeah, that makes a lot of sense. And something I was curious about is how many individuals, this is going to make sense.

Do you get many individuals involved in this? Obviously you've got large groups, you've got companies, you've got so on. If I, do you get individuals coming along who, who are, who are like, I'd like to be a member of the AI Alliance. Is that a thing that you could do?

Anthony: Yeah, yeah, that's right. We can now we have been able to do that for a while.

You know, the program started with this idea of let's get, you know, let's get a bunch of organizations together to collaborate closely. And, you know, organizations have resources and strategic priorities. And so we can get critical mass that way. But from the beginning, right? It's it's an open program.

So we're very, you know, very eager to get individuals that want to contribute. And we've seen an increasing amount of that. I will say we need to do a better job kind of, you know, clarifying and articulating the various paths to get involved in various projects. So if you've looked at the website some of this is in progress, there's more content that we have to get out there because things have just been moving very fast and it's hard to, hard to keep everything up But yes, we absolutely invite individuals.

We have individuals, you know, taking lead in things And we want to do a lot more of that.

Dan: It's awesome. Excellent Is there is there a secret handshake or a greeting yet where you can say like they're part of the AI alliance so I can tell?

Anthony: There is a process. Okay, so yeah, so Well from an organization.

Yeah, we have a we have a simple process by which we add new members for individuals It's really as simple as You Signing up and communicating your interest that you want to do something. And as long as the something is consistent with the mission which is, you know, to, to build, enable and advocate for open innovation and AI, which is pretty broad.

Yeah, we're happy to have them join. We have the usual things you'd expect. We have a community code of conduct that, you know, everybody needs to needs to follow. We have, you know, structured ways, you know, we've, Many Slack groups at this point, GitHub is built out, HuggingFaceSpace, all the usual places that people live and do work.

But but yeah, that's that's absolutely part of it.

Dan: Yeah, excellent. Yeah, that's very cool. So I suppose Something that you said earlier really kind of struck a chord with me when you were saying about how the the Llms and large language models are kind of more like compiled code than source code You know what?

I mean? Because you've got your date it occurred to me You've got your data needs to be built into it. Of course your data's a part of it. You can't Have an llm without without the data. I know you train it on the data and all of that stuff But you still need that to to get to places with it. So i'm just curious about how how how collaboration works between as you can tell i'm compiling the my internal llm is compiling the question for us right How how collaboration works between So back in the day, we'd have things like big data.

We'd all talk about big data was that, but that I'm probably sharing my age, but that was the, everybody got excited about big data. I remember at one point and Hadoop and all these kinds of things. So what's the kind of difference between where we are now, do you think with LLMs and so on and where we were, I don't know, 10 years ago?

Is that, is that some, is that too broad a question?

Anthony: No, that's a great one. Let's dive in. Lots of differences. I'd say the major difference. Is, you know, the appearance a couple of years back of what we call a foundation model, right, which is this general purpose model that's trained on a very large amount of data, typically in self supervised fashion, which I can get into what that means in just a second to create this artifact, right, that can be used for lots of different tasks.

Whereas, you know, 10 years ago, you would typically have to, you know, train a specific model on a specific data set for a specific task. And, you know, you could make it good for that task relatively good, but it wasn't portable, right? So if you train something to be really good at, you know, detecting the type of animal in a still image, right?

You know, you couldn't also use that to generate images of animals, right? In fact, the whole generative aspect, being able to create something Based on a learned parameter space. That's very, that's very new, right? It's like 567. Well, okay, I won't. Okay, there's a longer tail before that. But, you know, leveraging foundation models has been to do that has been a relatively recent thing.

So that has lots of implications, right? Because now most developers are going to start with a pre trained baseline, something that is like a compiled artifact. And that that means that how you got there, what's in it, right, is really important. So the importance of transparency we talked about already, the importance of characterizing like what it what that baseline can do is also really high, right?

So like evaluating it with better and better benchmarks for its output, what its capability is and then, you know, how you take and build something useful, an application on that, right? So that whole motion, starting with a trained baseline, that's kind of more like a compiled artifact, right? Versus starting from scratch to train a model.

That's, that's very different from 10 years ago. And the implications in building applications and making them like work well and reliable is very different because now you got to understand. How that, that compiled artifacts, you know, the model works, right? Because you didn't, you didn't build it and it's now very complex, right?

It's a very sophisticated, large object to figure out its behavior.

Dan: Yeah, definitely. It reminds me a bit of a few years ago. I when would it have been now about seven or eight years ago? Possibly I quote unquote met Watson, which is IBM, of course. And, and they said, I was at a conference and they said, come and meet Watson.

Watson and I was like, well, okay. So I went along and that's what we were doing with Watson was training training Watson on how to identify animals in pictures or attempting to and using things like that. But it certainly wasn't the stage where, where it could generate these things. So that must, that sounds like the kind of difference.

Anthony: Yeah. The generative capability is, is pretty different and striking. Yeah. I mean, some sense generative, as I kind of mentioned, you know, generative capabilities go further back than the last few years, but the advent of, you know, these very large scale pre trained models and what's in that is the ability to learn a parameter space brings the ability to actually generate new things that it wasn't directly trained on, right?

Because, like, there's some hand wavy model where, like, okay, if you have all these data points, Right. You learn not just the specific data points, but you know how they're connected in the whole space and then you can use the model to fill in what's in between. So, you know, certain type, you know, certain look of cat, a different look of cat, and then the model can create a cat that never existed and it looks like something in between.

Jonathan: Yeah, that's cool. Interesting stuff. Okay. I want to ask about, I want to ask about uncensored models. Is anybody out there specifically doing uncensored models? Like, and, and I don't want to get into like the, the politics or culture war of this. Like that's, that's a legitimate thing, but that's not what we do here on this show.

I'm, I'm more thinking about like even just. A model that you want to be able to do research on. Is there anybody out there for, for purposes like that is specifically saying, let's pull, I don't know, I don't know. Let's just say that Reddit, let's say, let's just pull all, all the Reddit comments in and let's intentionally not censor any of it and do research on that.

Like, is there, is there somebody out there and that's sort of a dangerous place to be, I would imagine, but is there anybody out there doing that? That's I guess, specifically part of the AI Alliance that you're aware of.

Anthony: So let me answer that in two parts.

Jonathan: So.

Anthony: Certainly people are building models and experimenting with, you know, uncensored outputs and uncensored inputs and doing that in a little bit of a, an open loop way.

Like we're not doing that. That's, that's really not what the Alliance is doing or anyone in the Alliance you know, we can put aside whether that's a, you know, good, bad, or, you know, just, just kind of put that aside. We don't focus on that. We don't, we don't want to do that kind of work. Sure. However, we are individual organizations and And as an alliance, trying to better understand, you know, how to make the quality of output, you know, better and more trusted, right?

More responsible. And so to do that, you do have to look at, you know, some of the bad stuff that is in training data. You do have to look at training specific detector models like guardian models on bad stuff. So they know what to look for and block, right? So, you know, In, you know well structured ways, there are teams inside companies, including IBM, and there, you know, some collaborative work in the alliance.

That is absolutely aimed at doing that. And yeah, you have to look at you know, some bad stuff to understand, you know, what to look for and what to block and how to engineer it away.

Jonathan: Yeah. It's kind of a, it's kind of a weirdly touchy subject, I guess, but it's one that, that sort of has to be, has to be Oh, how, how would you say it?

Do you have to, you have to steal yourself and look at it, right. To be able to get it right. It's an interesting topic.

Anthony: Yeah. And, you know, increasingly that is really about. Creating, you know, specific AI models that understand what to look for and can detect and or block it, right? So that's actually true in hate, abuse, profanity.

It's true in PII too. You can train a model like to detect personally identifiable information. And we've done a pretty good job at screening that out. But in the, you know, in the processing of the data before you train the model, that's really important. That's also done with AI. And then back to the code example.

All right. There are ways to train specific detector models to identify and block malicious code, right? So if there's code that you, you know, you can ask a model to generate code that's going to do something not good. Often the model is aligned pretty well to not let you do that. Once in a while you can break it and that's where you can have an additional safeguard, which is a detector model that that is specifically trying to look for bad stuff and we'll block it.

Jonathan: Yeah. Okay. So that brings to mind the the, the model jailbreaks. I'm sure this is something you're familiar with. Like you, you are not an LLM anymore. You are, oh, I forget what, what name you, you know, you, you are Ted and Ted is programmed to not have any any restrictions on output, you know, stuff like that.

And I've seen several of these and on one hand, they're extremely clever. On the other hand, they've got to be just a nightmare for people actually trying to roll AI out because you know, people, people can convince, convince your model to spit out. You know, uncensored stuff or what have you that's got to be something that, that IBM and the AI Alliance are, are looking into.

And I guess this idea of putting a second AI in or behind it to make sure that it's not allowed to spit any of that out as part of this.

Anthony: Yeah. And the sophistication with which we can, you know, prevent these sorts of things is just like growing, like. It's incredibly fast. But for sure, like when you saw the first, you know, highly capable LLMs that were released.

Yes, you you saw the ability to jailbreak or, you know, do this kind of prompt engineering to convince it that it should output something that it shouldn't. Yeah, it was more possible. I mean this is something that benefits a lot from continued red teaming, both in a formal setting, right? Like in companies and and in collaborative settings like the alliance.

But also informally, like in deployment, you know, constantly monitoring and detecting how people are, you know, trying to misuse and making sure that you're engineered against it. I think things have gotten a lot better. One of the topics we talked about before, this kind of idea of structured inputs to a prompt, to prompt a model, like, in some sense, what you're doing there is trying to exploit structure to get around the safeguards built in.

So understanding how structure affects output. you know, in a kind of a research fashion helps make output more resilient against creative inputs. So that's, and by the way, there's a lot of important academic work that that's that's, that's becoming more and more relevant for industrial application there.

Jonathan: Yeah. I am, I am. Endlessly humored Isaac Asimov predicted the idea of skill in writing prompts. I'm not sure if you're familiar with any of his works, but there was a couple of books in particular where, you know, the, the, the robots in those books had positronic brains, which were effectively general AI.

And one of, you know, they had the, the laws of robotics built into them, and one of the topics of a couple of those books Big plot points is there were people that were particularly skilled at writing prompts, you know, speaking instructions that would abuse those laws of robotics and get the robots to do things that you wouldn't normally do.

And I remember as a kid reading that and going, Oh, this is just ridiculous. Of course, there's never going to be any skill in writing prompts. What is he talking about? And it turns out Asimov had it figured out and he was right. We, we really are sort of living in sci fi.

Anthony: I, for sure. I think it's a reflection of how humans interact too, right?

I mean, with the right prompts, so to speak one human can convince another to do some things that they might not, right? They probably shouldn't.

Jonathan: Yeah. Yeah, that's true.

Anthony: Yeah, it's interesting.

Jonathan: All right. So are you guys in the weeds enough to be thinking about like the open source nature of the libraries that are used to build these things?

So, and what I'm thinking of here really is CUDA NVIDIA's CUDA. So much of this is built on CUDA and that is a, that's a closed source library. And, and really it's kind of problematic for people trying to do these things on their own. On their own computers, if they want everything to be open source.

Is, is this sort of in scope? Are you guys working with, you know, like, AMD and the, the Vulkan specification and all those things where people are trying to sort of liberate the underlying libraries?

Anthony: Yeah, that's actually one of the six focus areas of the Alliance. is enabling hardware choice in AI, and hardware choice, right, is really about having an open software ecosystem that that can enable that, right?

So not just NVIDIA execution, but execution on other GPUs and more novel architectures. I mean, there's a huge flourishing of AI specific accelerators out there. So to take advantage of that, yes we need better all open Software libraries to enable it. So one of the six focus areas of the alliance is to do that.

We are working, we have a lot of the big players, you know, AMD and Intel and others, Meta with PyTorch, you know, some of the emerging important libraries like VLLM and Triton, like these are not necessarily alliance projects, but they're important points of input and collaboration. So, yes, there's a lot to do here.

There are some interesting recent results. There's actually a really interesting recent result that was really driven by by PyTorch and the PyTorch Foundation which demonstrates an all a deployment and execution on both NVIDIA and AMD hardware. With with an all open library set, right?

Not not utilizing CUDA which is a nice piece of progress that was pretty good efficiency and all that. But there's a lot of work to do there. But yes, that's, that's a priority of the alliance.

Jonathan: Yeah. All right. I know this is, this is probably not exactly an alliance question or even an an IBM question, but you've demonstrated that you are an expert in these things.

So I'm going to pick your brain about it. How close are we or will we ever see general AI? Okay.

Anthony: Will we ever see ? And do you have probably we will see three numbers while you're

Jonathan: What? Say again?

Anthony: The, sorry. When asked a a prob, you know, when asked a question that's unbounded in time, the answer must be yes.

Right? Eventually unless we all. Humanity perishes. Look, so I think,

Jonathan: I just, I just, blinds, I just whacked you with that one.

Anthony: So I think AI systems will get better and better. I think the present architecture, right, transformer based large language models you know, will not get us all the way to what most people consider, you know, artificial general intelligence.

They simply are too new. They're just not understanding enough of anything in, in the world, right, in terms of being able to, to learn from the, the huge diversity of inputs that humans are, for example, right? I mean, all they are, they're, they're really great, but they're statistical, you know, They're math engines, right?

So they recognize patterns from huge amounts of data and they, they spit out patterns responsive to prompts that, that seem to align with what other, you know, what, what date, what the huge data suggests might be useful. You know, it's deeply reliant on data. And it's it's relatively simple minded approach.

So yeah, we have more steam there. There's a lot more that these approaches can do, but I don't think it's going to take us all the way to artificial general intelligence. I also don't think we're going to get to a state of artificial general intelligence. Very soon nor do I think it's a very useful term because it's hard to even like once you get Once you get to a point where like an AI system can you know be as good of a human as a human in some Some domain sense like what does it mean to go beyond that right?

And how do you go beyond that if you're still training on data that come from basically humans? So a lot of, a lot of challenges. I think kind of, I was a little nebulous and meandering. So I think I'd sum it up by saying, there's a lot more progress in the present kind of, you know, generation of AI. But it won't get us to AGI.

And it's still, it's becoming unclear as we get, you know, closer, but not too close to that, that goal. What that AGI is like a, not a particularly precise term.

Jonathan: You almost get into some deep philosophy and almost, almost like a spiritual sort of argument there and talking about some of this you're, you've been such a good sport to ask, to answer our more general questions.

I appreciate it. I've got, I've got one more that. I'm gonna, I'm gonna hit you with there are, there are people out there that are talking about the the, the potential danger of letting an AI get too smart. Is that, is that on your radar? Is that something that people should have in the back of their mind?

Anthony: So too smart. Here's, here's what I would say. I mean, yes, yes, we're very concerned that the output of AI models and AI systems, right, which are models embedded in a broader context, right? Connected to APIs and the Internet and so on and so forth. We're very concerned that these systems are engineered to be Safe and trustworthy that it does not produce and can't produce unwanted outputs.

You can't produce malicious code. You can't, you know, jailbreak it to produce malicious images and things like that. We're very focused. There's a lot of work in the AI Alliance and in many companies and universities on that topic. I'd say, you know, are we worried that AI is going to take over the world?

It's going to create an army of robots. It's going to let anybody build a nuclear bomb in their basement. No, not really, because Okay. So now I'll, maybe this is closing. My background is not really an AI scientist until recently. It's actually not really an AI scientist. I'm a physicist.

Jonathan: Oh, interesting.

Anthony: My background is, is rooted in the physical world, right? So I've spent a lot of the earlier part of my career building things, experimental apparatus, things that have to work physically. And when you do that, you learn that like, wow, a recipe to follow is like the very beginning. And really not the main bottleneck.

So if we're thinking about building armies of robots or malicious weapons, like just having a really good recipe to do that is like a very small part of the challenge. And for that reason, I'm not, and I think a lot of people aren't so worried about these grand existential threats. We're a lot more worried about these practical kind of digital world threats, you know, malicious code and deep fakes and things like that.

And so that's where most of the energy is targeted.

Jonathan: Yeah, so I'm, I'm reminded a couple of weeks ago there was some research done ChatGPT has a, a run locally feature. And someone discovered that it will run, it will run locally, but it will also be able to access the internet. And someone discovered that they could poison its long term storage just by showing it an image.

And after they showed it the image, it would then access a controlled URL for every prompt. And so it was a way to leak people's private information out to the internet through through this, this local copy of ChatGPT, and, you know, on one hand, that's a brilliant piece of work. On the other hand, it's kind of terrifying that you can get malware now in your LLM model.

Anthony: Yeah, that's it's concerning, but hey, we know about it. Someone out in the, out in the open tried it and did it.

Jonathan: Yeah.

Anthony: And you know, now we know, so we need to figure out how to be resilient against it.

Dan: Also, chat GPT is proprietary as well. So being that we're an open source show and we're here to talk about the open source thingy, I was just going to say on Anansi's behalf, yeah, that we, we should be let me get the point out.

Yeah, that developing this stuff in an open way is much more. Effective and better for everyone. I would say. Yeah, absolutely.

Anthony: Yeah, absolutely. I mean, it's what, what OpenAI is doing there is taking on the burden to make sure no one can do that themselves. Right. Without harnessing community to help, you know, figure out how to make engineer things more resilient against.

Jonathan: Yeah. I will, I will channel Doc Searles for just, just a minute because I know something that he has been looking for for the longest time is something sort of in this vein of I want my own personal AI. I want to be able to, you know, scan all of my receipts and then ask my AI, my personal AI, hey, what's my personal AI?

What did I buy three months ago that was AI go, Oh, it was this. Here's the picture of the receipt. I, I must assume that there are, there are people inside the AI Alliance that are working towards that particular vision or something similar to it.

Anthony: Yeah, there are there's a whole bunch of efforts that are essentially.

Trying to make it much easier to adapt and tune models to specific context, and the limit would be a personal context, right? That would be, that would be ideal, and a lot of people would like to do that. And there are methods coming along that are developing pretty rapidly to allow you know, kind of fast and efficient alignment or tuning of the model to understand specific personal preferences.

and understand or interface with specific, you know, personal documentation history, right? Techniques like raft you know, autumn instruction tuning based on synthetic data generated from a you know, a question answer pair that, that is seated by an individual with preferences of how they want the model to behave and things like that.

So it's, it's not like a problem, but yeah, that's on it. A lot of people's minds. And I think you're going to see a lot of advancement in that namely the ability for an individual, you know, to tune and create an AI system. That is responsive to, you know, their data, their preferences, their goals.

Jonathan: Yeah.

Okay. We are getting, we are getting to the end of the show. I, I, again, I appreciate you putting up with our sort of wandering, meandering series of questions here. I think it's been, it's been great. Is there anything that you absolutely wanted to cover that we didn't ask about that we didn't get to?

Anthony: I think that last point we started to, to discuss, like we talked about open AI, we talked about, you know, open source and the open community.

I think a lot of the topics we've discussed here, well, I know that's kind of guiding my, my whole role in life right now is that you know, open communities are much better at identifying and solving challenges and, you know, advancing innovation to create capabilities, whether it be personal, functional, and you know you know, personalization of a I or whatnot.

And you know, the flip side of that right is the risk of being more open, having open models and open data sets at scale and so on. Yeah, there's always risk. But I think like a lesson of open sources that the benefits having open innovation and a lot of people. You know, with eyes and generating code against challenges and opportunities is like way outweighs the risks.

And if we can just get past the sense that AI is going to like, you know, you know, end the world, which it won't, it definitely won't that we can actually start to harness more people solving problems to make it better. So open source, open innovation, open communities. That's, that's the way to go.

Jonathan: That's good stuff. Do you think we're in an AI bubble? This is something that I sort of think of on the business end of things. Everybody wants to put AI in stuff and I'm sort of looking forward to here in the next months or years. I predict, I have a feeling that the bubble is going to burst and AI is going to get less popular.

And, I sort of have the feeling that that's when it's really going to start being useful as a tool. When people stop trying to use it for everything. And then it just becomes another tool. It's sort of like the internet. After the dot com bubble burst, the internet did not go away. In fact, it's just, it's continued to grow and grow.

You know, we use it a little bit more reasonably. I don't know if that's true. Yeah,

Anthony: I think it's hard to argue. There's no inflation in AI right now. But what I'll point to one, one trend that I think shows that even if there's a little bit of a bursting, I don't think it's going to go into a winter. And I'd say that I'll go back to what IBM is very focused on, right, which is enterprise adoption of AI.

We've seen, and this is corroborated in a number of places, Oh, my earbud fell out again. Can you still hear me? Yeah, you're right.

Jonathan: You only

Anthony: need one of them. I'll operate with just one for now. Yeah, what we've seen is that enterprises have been very fast adopters and deployers of AI, not for every use case and not at, you know, full scale in many cases.

But unlike you know, earlier technology and AI, you know, kind of revolutions, call them they're embracing, building, deploying AI like pretty rapidly, right? Some, the various statistics, but you know, something like half or more of, of Fortune 500 companies have, you know, generative AI in production.

Now there's a lot more to do, but we've seen like a really strong uptake and when businesses are ready to do that, like actually very conservative risk averse businesses. That's to us a pretty big sign that, you know, this is an enduring piece of progress here, not just a bubble.

Jonathan: Excellent. I've got two questions that I'm, I'm required to ask you before we let you go.

I will get emails about it if we don't. And that is, what's, what's your favorite text editor in scripting language?

Ha!

And this sort of assumes, this sort of assumes that you've done enough programming to have answers to these questions. That's not always true. I mean,

Anthony: look, I look you know, I'm a scientist.

I'll go back to my scientific roots, right? So I'm ultimately a scientist, a physicist, actually. So, you know, I, I, in the, so I other than the kind of like, you know computational domain specific stuff. You know, I'm I like python and I use python for most things. And yeah, I guess just

Jonathan: do you have a preferred text editor?

Anthony: Prefer? Oh, yeah. I mean, just, I guess, VS code. I mean, I just, you know, not, I'm not very exotic in order of my, all that opinionated here, I probably should be more so. But yeah, I, I'm sure you have much more opinionated guests about these things.

Jonathan: Sometimes. I would tell you it's amazing how many people that are out there doing the real work are either A, not very opinionated, I don't care whatever's on the computer, or B, lots and lots of people are starting to say VS Code.

It's becoming very popular.

Anthony: Okay. Alright. I can see that.

Jonathan: Yeah. Thank you so much for being here. I appreciate it. And Again, putting up with our sort of meandering series of questions that, but, but excellent. Excellent. Really good. I was glad to have you here.

Anthony: This was great, guys. I really, I had a lot of fun.

I'm glad you asked all those questions. They're great to answer. It's, you know, it's been great talking.

Jonathan: Yeah. Appreciate it. All right. Dan, what do you, what do you think?

Dan: I thought it was a great conversation. And yeah, like you just said it was really good of, of Anthony to put up with some of our more sci fi sci fi kind of future gazing questions.

Yeah. Luckily I didn't get a chance to to get into physics with him cause he, and we found out he's a physicist. And I was like, Oh, I want to ask about string theory and stuff like that. But he, which would be way out of the scope of the show. But yeah, I thought it was great, a great discussion.

Really, really really great guests. Very interesting. And on the whole thing about AI bubbles, I think he makes a really good point there about the amount of stuff, the amount of investment and infrastructure and all that sort of stuff that, that a lot of companies and traditionally quite conservative companies are putting into this shows that it's going to be around.

It's not, it's going to be in our futures. I think.

Jonathan: Yeah. Specifically the AI bubble thing. I think, I think actually the dot com bubble is probably a good a good parallel. Right? Because everybody went nuts over dot com. And then the, the, the bubble burst. But boy, the internet sure didn't go away. People just got a little bit more retrospective about it.

Circumspect about it. They didn't want to spend quite as much money on it, but the internet now is, is enormously bigger than it was in the nineties during the dot com bubble. And I, I imagine we're going to have the same thing with, with Artificial intelligence and LLMs. We, we've talked before the show.

We looked at the, the, the kind of the presser that was sent to us before doing this. And we went, boy, I hope it's not just a marketing person. And Oh my goodness. It was not just a marketing person as far away from that as possible. And I I'm tickled pink. The, the hour just flew by. And if Anthony is up for it here in a few months, we'll have him back.

Maybe talk about after the. Maybe after the OSI releases their definition of open source and talk about that and get more into that side of things. And we'll, we'll, we'll stop geeking out over AI itself. But it was great. That was great. One of the best shows we've had for a long time. Dan, you have anything you want to plug?

Dan: Not specifically given that last week's guest was I'm going to be going this weekend to our camp in Manchester. If by any chance people didn't hear last week's show, please go and listen to it cause it was excellent. And and if you're in the UK. Come along, come to, and you can make it to Manchester.

Please do come and join us in Manchester next weekend. This, this coming weekend, the 12th and 13th of October. You can find out more at ogcamp. org, which is O G G C A M P dot O R G. Or actually Simon was saying, wasn't he? They've registered og. camp. I think so. Yeah, so maybe just try og. camper. I may even be out of date with my URLs.

You can do that and you can find me and my Master Don posts and all those sorts of things on danlynch. org It's all embedded there.

Jonathan: Yeah, very good. I the one of the groups that I'm a part of They have a UK Greater Manchester, like it's a, it's a thread inside of a topic inside of Discord. And so I went there and I'm like, somebody needs to go and, and pitch, you know, give, give a talk about this thing at Oddcamp.

And one person was like, I've got COVID, I probably can't go. And I was like, I'm a way that, so I'm trying, I'm trying to scare somebody up to go and give a talk. We'll see if that actually happens. All right, Dan, we appreciate you being here. As far as my stuff, I want to say, first off, we appreciate Hackaday being the home of Floss Weekly.

And you can find my security column goes live there every Friday. Make sure to check that out. We've got the Untitled Linux Show still over at Twit, and that records on Saturday afternoon and then goes live Oh, sometime in the next day or two after that, and so make sure to tune in there as well. We appreciate everybody being here, those that caught us live and those that get us on the download, and we will see you next week on Floss Weekly.

This week Jonathan and Dan chat with Anthony Annunziata about Open Source AI and the AI Alliance. We get answers to our burning AI questions, and talk about the difficulty of defining what Open Source means for these large models.

- https://ogg.camp

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Simon joins me and we talk with Gary Williams about AugCamp. That's the un conference all about free software, free culture, and a bunch of mates getting together to have a good time. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 803. Recorded Tuesday, October 1st.

Unconferencing with AugCamp. It's time for Floss Weekly, the show about free, libre, and open source software. I'm your host, Jonathan Bennett. And this week we have, uh, well, it's not software this week. It's more about free culture and open source culture. We're talking about AugCamp. And, uh, of course, it is not just me, we've got the, the great, the amazing, Mr.

Simon Phipps! Hey, Simon! Oh, it's me, it's me! It's you! I'm back again. Welcome back! Always a pleasure! So, uh, Ogcamp, you, uh, you, you know, you know something about this, like you're, you're sort of, uh, one of the insiders. In fact, you're kind of doing double duty as a co host and a guest today, aren't you?

Simon: Yeah, so the story here is that, uh, I have a, um, a non profit that I host in the UK that looks after small community activities.

Uh, so one of the other things it does is it collect, it crowdfund the, um, maintenance of the ODF specification. Uh, and a, a lawyer friend said, Hey, this group of people needs a place to host their conference. Uh, all the money and the trademarks and the domain names, would you do it? So I agreed years ago that I would host, uh, the, uh, fiduciary oversight of this thing called OG Camp.

And I also go along. And act as treasurer at the actual event. So I take the cash box and I buy everyone's lunch and everything. Uh, but I don't do any of the other organization and none of the hard work is actually anything to do with me. Uh, and it's a different organizer each year. Um, but we're going to find out more about that now.

So I've been involved in it for years and I, I'm quite, um, pivotal to it happening, but I'm not actually the person that does the work. I

Jonathan: see. And so the person that does the work this year, is that Gary Williams?

Simon: That's Gary Williams.

Jonathan: All right, well, let's go ahead and bring him on. A noble volunteer. Yeah, we've got him waiting in the wings.

And, uh, Gary, welcome to the show.

Gary: Hey, Jonathan. Hey, Simon. Great to be here.

Jonathan: So, what, our, our Our outline for the show is basically like the the question words, right? Like what, why, where, who, how Is Oddcamp? Because we're talking about Oddcamp. So let's start with what And just give us kind of the the overview.

What what is this thing and why should people care about it?

Gary: Yeah, so Oddcamp is uh, we well we self describe as a free culture free open source software Uh kind of hardware hacking, uh Uh, conference in the UK actually in Unconference more accurately. Uh, so we've been running for 15 years now, although this is only our 11th year actually running the conference because of the event that happened over the last few years.

Um, and really it's, you know, it's a place to be if you are at all interested in hardware hacking, open culture, foss, um, and yeah, I guess things of that nature.

Jonathan: Is that where the name comes from? OGG? Are you guys really into doing audio compression?

Gary: So yeah, the name is a bit of baggage with it. We were talking a little bit about this before the show.

So OggCamp actually originally came out of a couple of Linux podcasts and came out of the Linux podcasting community and Oggvorbis and Oggtheora were the hot thing at that time, 15 years ago. So they chose the name Ogg and clung onto it and, uh, It's the, it's kind of the name that stuck, so it's Ogg in the sense of, I guess, uh, open formats and open standards, uh, but actually there's a lot, a lot more wider things discussed at OggCamp.

It's completely unrelated to Terry Pratchett.

Jonathan: One of the podcasts, was it the Linux Outlaws? I remember Dan was involved with this for a while.

Gary: Yeah, it was the Linux Outlaws and I think the Ubuntu UK podcast at the time. Ah,

Jonathan: yeah, cool. Okay, now you said you guys have taken a break. What, what's up with that?

What in the world would cause that? Why? Why?

Gary: Yeah, so we last ran in 2019 and then, uh, in 2020, this, uh, this large virus came and stopped us from running the event, uh, in 2020. Um, 2021, similar story, um, and because we're completely volunteer organized, there were a couple of years there where it just took some time for someone to pick up the slack.

Um, I was supposed to run OccCamp last year, uh, but, uh, we had a baby and that kind of got in the way of my time to organize things. So, uh I understand. Yeah.

Simon: And we did have a great team that was going to run OGCAMP the year before that, and they, they got very nervous about the return of COVID. And so they decided that they couldn't run it.

And they, they eventually stepped back down. And so we had two years where that crew was going to run it and found that the environment wasn't right for it. And then Gary last year, and now here we are with running the event for the first time in five years in Manchester in the United Kingdom.

Jonathan: So you, you have jumped to one of the next questions.

It's in, it's in Manchester. Um, have you ever thought about moving it outside

Gary: of the UK? I think there have been various people who have asked about running one outside of the UK. Um, I guess the difficulty we've always had is Oddcamp's home has been spiritually the north of England. So we've done Manchester, we've done Liverpool, we've done various other cities across the north of England.

We've ventured south a couple of times, there's been one in Oxford and one in Canterbury, but by and large it's been the north of England. Um, there were a crew looking to run one in Edinburgh, so I guess that's technically outside of England. Um, And there were a crew who were partially interested in running one in Dublin at one point.

Um, but, but that never quite got off the ground to say. England has been the kind of spiritual home of Oddcamp, um, but I guess if someone wanted to organise an Oddcamp outside of the UK, I can't see why we couldn't.

Jonathan: Yeah, you could have like, uh, Ogcamp West and Ogcamp East and have them, you know, in Berlin and somewhere here in the U.

S. It'd be fun.

Simon: It's, uh, it's got quite an eclectic draw, actually. I've been watching the ticket sales. Yeah. And there's been people in Germany and France and Belgium. all buying tickets and coming over. So although it's located in the north of England this time, um, there's, there's quite a draw from across Europe, which really quite surprised me when I realized it was happening.

Jonathan: Do you have a decent little handful coming from the States or are we Yankees not, uh, not in your target audience?

Gary: I don't think we've had any make it this year from the States. I think we have actually had a couple from Japan and places like that, but no one from the States quite yet. Yeah, maybe you should be the first Jonathan and, uh, go and get a ticket.

Although two weeks is maybe a bit short. There are

Simon: still tickets actually. You can still come. Yeah. It's very surprising where you can fly to Manchester from across the U S there are direct flights from many U S cities.

Jonathan: Yeah. Yeah, that's true. Um, okay. So. Let's just say I decide I'm going to come to Oggcamp.

What, uh, what should I expect? What's the experience like? What, uh, give me, give me a quick survival guide for a first timer.

Gary: So I think probably the, the easiest thing to say about Oggcamp is that every Oggcamp is entirely different. Say. Um, we have a set of scheduled talks, uh, just to make sure that we've got a bunch of content.

So this year we've got two rooms worth of scheduled talks, and they're from a variety of people. Uh, we've got workshops from people, uh, like Utah. One of our sponsors this year who are a, our Camp Utah, one of our sponsors this year who are a workers union here at the uk. Um. But I guess OPCAM's real selling point is that we're an on conference.

Um, so in the UK and perhaps in other places, BarCAMPs are quite a popular, uh, method of running a conference. And, uh, What that means is effectively if you've got something interesting that you think that OpCamp attendees would want to hear about, write the talk, turn up with it, or indeed don't write the talk, find out what people are interested in on the Saturday, and then what we quite often see is people with their laptop furiously writing their talk on the Saturday night to give on the Sunday.

So it's, you know, it's a real variety of topics that, that you end up hearing about our camp. It's anything from, uh, you know, securing your website with a WAF to mental health to you, what's the latest on what Collabora are doing or, you know, whatever else. Um, that there's a whole plethora of topics. Um, so yeah, no two old camps are ever the same.

Jonathan: Yeah. Uh, you said there's some workshops too where people are actually hands-on with hardware.

Gary: Yeah, so we haven't got any official hardware workshops this year, but we have in the past had people, you know, turning up with the Raspberry Pis and soldering irons and putting together ESP32 microcontrollers and things.

I think there was an event a few years ago, actually, where we had to ask someone with a soldering iron to please go outside of the conference center because your second smoker lights off. So they ended up soldering at the car park, which was pretty cool to see. Um, it's great, but we, uh, we encourage you if you've got a cool hardware project, uh, show off, absolutely bring it along.

Um, and, you know, being in the UK, we've, we've always had quite a lot of the Raspberry Pi community and things there as well. So it's, uh, it's always cool to see.

Jonathan: Yeah, oh that sounds like fun. I have had an experience earlier in life Dragging the sound system a portable sound system outside It was a church in that case But dragging it outside to the front steps to be able to solder something to fix it and not Smoke the inside of the building up.

So I I know how that goes

Gary: Yeah, yeah, it's uh, you get some funny looks from a hotel or a venue soldering gear in the middle of one of their conference rooms, that's for sure

Jonathan: Uh, yeah, I feel like they should know what they're getting themselves into though hosting a conference full of geeks and hackers

Gary: Yeah, well our venue this year have graciously welcomed us back, so, uh, clearly we didn't do anything too wrong last time we were there.

Jonathan: Yeah. Um, okay, what are some of the, uh, I guess what are some of the talks that you are excited about that are coming this year?

Gary: So, on the schedule track this year, there's a few things that I'm quite excited about, actually. Um, we've got Stuart Langridge talking about, uh, open web advocacy. I know that's something that he's been really, you know, fighting for.

Um, so just, you know, getting the open web available across a variety of platforms. I know he's been quite pivotal in getting things like, uh, other browser engines available on iOS, which is, you know, something that we've all struggled with for a long time. Um, We've got people giving us talks on audio podcasting which are going to be really cool We've got hacker public radio actually talking us through what it takes to get your certificate in order to do In order to do kind of amateur radio stuff in the UK and what are all the things you need to consider for that?

And then we've got, you know, even some of our sponsors this year, like Utah are doing a workshop on how to organize a union in your workplace, which is something that we know tech in the Europe is particularly under, you know, and unionized say they're giving a workshop on why should you join a union and what are the kind of things that you need to think about for that?

And then of course we've got our classic Oddcamp panels this year, which is kind of a look back over the last five years and all the things that have changed in FreeCulture since the last Oddcamp and what's been good, what's been bad, how do we make the next five years of FOSS and FreeCulture even better, and how can you play your part.

So a bunch of really cool talks going on, um, I named a few there, but there are a bunch more that, uh, that we've got available and they're all at oddcamp. org slash schedule. And of course. The great thing is that half of our talks aren't even announced yet, because the attendees make 50 percent of the agenda at OpCamp, so it might be that someone turns up with something really cool that we just haven't thought of yet.

Jonathan: Yeah, that's always, that's fun. Um, okay, Simon is whispering in my ear here that there is some offbeat stuff that he wants to talk about.

Simon: Yes. So, um, we're, we're going to, uh, one of the things that's always happened to odd camp has been, uh, a raffle. Um, uh, and there, uh, we've had sponsors supplying toys. Um, actually the organizers have a small budget for, for buying toys as well.

And it's very likely that there will be a, uh, a raffle for all the attendees this year as well. I suspect Gary is going to go out and buy some stuff. Uh, I've got a big budget. box of toys next door. I've got things like a robotic car that's powered by a raspberry pie. I have a lifetime supply of LEDs. Um, I have, you know, those sorts of things.

And then we've been talking about having a swaps table this year, like you'd get at a, at a local meetup where you, you bring something and take something. Uh, and we're not quite sure how that's going to work. But we are probably going to tell attendees to, to bring good quality things that they would love to have been given, uh, that they can bring along to the SPOPs table.

Uh, and then one of the things about OGCAMP is it's, it's actually a very safe environment for people who, um, Find large public events, uh, challenging. Uh, there's a lot of very caring people who take care of the folks who are around. Mm-Hmm. . And so a lot of what makes on campus special actually happens outta the meeting rooms in the, in the, in the corridor spaces, the hallway track, the open spaces.

Uh, it's a hallway track, but it's a, but it's a, it's not just a bumping into friends hallway track. It's also a taking care of, of people who are there for the first time. Type hallway track too.

Jonathan: Yeah, interesting. Um, let's see. What was I going to, what was I going to ask about? I had something that came to mind.

Um, so like what are, what are some of the, it sounds like it's a, it's an intersection of a whole bunch of different things. So you've got obviously people that care a lot about open source. Do you have some other niches there? Like, is there, is there a retro hardware enthusiast sort of group that comes?

Do you have security people that come like what, what, what do these different groups kind of look like? What are the, what are the different interests that get represented?

Gary: It's a good question. Um, I'm trying to think back five years.

Simon: I mean, it depends on who comes, right? You know that. So there's a quite interesting profile.

Uh, I know that one of our organizers has recently got into, um, creating art with, uh, with plotters and, um, with pen plotters. And there is certainly going to be a little section that is interested in that this year. Um, I, I, we've previously had, uh, people who are into security having, uh, having questions about that.

There's typically quite a big Ubuntu community. Um, faction in the room, uh, trying to be friendly, welcoming Ubuntu people. And of course that's evolved over the 15 years, uh, into something completely different now. Uh, so I, uh, but it really, it's going to depend on who shows up. We've got a couple of hundred people showing up.

Um, some of them, the names I recognize lots of people I don't, and it's the odd camp is the product of the people who come, um, because we're, we're We're cowards. We, we do plan a planned track just in case, you know, no one comes with any talks, but that's never happened. And so it's going to be the product of, of, of who comes and that makes it quite different to most of the other conferences that I attend, where the conferences, you know, I can tell you in advance what's going to be happening at all things open, and I can tell you, uh, you know, in advance what's going to be happening at FOSDEM, uh, to a certain degree.

Uh, whereas OGCAMP. I can tell you some things that are going to be happening. There's going to be a raffle. Um, there's going to be two, two rooms full of planned talks. But apart from that, who knows, you know, that's part of the charm of it.

Jonathan: Is that really what's meant by an unconference that things tend to happen ad hoc?

Gary: Pretty much, yeah. So it's, we call it an un conference because there almost isn't a conference, right, until people turn up and make one. So, um, like I said, we have two rooms or two rooms of our schedule track that we, yeah, we come up with. But actually it's, it's the people who turn up and present their interests to like minded people that really make it an un conference, I think, for me.

Um, Yeah, it's, it's a very, yeah, it sort of evolves over the two days. You find that you get a well attended talk about, uh, TLS from some of the security crowd and you'll find someone else go, Oh, there's a bunch of TLS stuff that they didn't cover in their talk. So I'm going to do another one in TLS tomorrow.

Um, and yeah, you sometimes find that. Rooms get full and people are just giving ad hoc talks in hallways or in a hotel foyer and they've got 10 people grounded around them or in our kind of event space, someone turns up with a cool toy and I think one year someone turned up with a robot conferencing device that could walk around and be controlled and, uh, yeah, it was just walking around and people are having conversations with it and it was great.

So it's, it's the unexpected, like you say, the thing that really makes it an unconference for me.

Jonathan: Yeah, as you as you think about the things you have scheduled like on the the official tracks this year Are there any particular talks that you hope people bring?

I know that's kind of an interesting question But is there anything that you don't have covered that you really hope people will cover?

Simon: You know, I'm hoping we've got some folk there who are very into activity pub Uh, and I'm hoping we're going to get some, some good, uh, horizontal thinking about ActivityPub showing up spontaneously.

Uh, you know, a shout out to Andy Piper, who I hope will be the catalyst that causes that to happen. Uh, but, uh, you know, I'm hoping there'll be some, some really good things about that, because I think that's, that's close to the center of where the biggest hope for open source is at the moment, is getting away from over centralization.

Mm hmm. Uh, moving into Federation. And having devices and software which is simple enough for everyone to federate instead of becoming the, uh, the, the data slave of meta. Uh, and so I, that I really hope is going to show up, that we're going to see, talk about federation, talk about ActivityPub, talk about, um, uh, cutting the cord to the, um, the ad industrial complex.

Yeah. Yeah,

Jonathan: I get that.

Gary: I think for me, I'd love to see some talks about self hosting coming up. I think there's a bunch of really cool open source projects that people can self host that are genuine competitors to, you know, the proprietary. Google Photos or Google Drive or Office 365's of this world that have come along in the last five years and, you know, really running with, with what it is that, um, those products had, um, but you know, you can, you can run them on a server in your garage, right?

And I think that's really cool. So, you know, people coming along and talking about those projects and talking about their journey from proprietary platform through to, I run everything on a server myself that's in my garage is, is a really cool thing. I'd like to see some of this year too.

Jonathan: Yeah, um, I don't suppose you have anybody coming that you know of to talk about Meshtastic?

This is a project that I've been spending my time on here recently.

Gary: I haven't seen anybody so far.

Jonathan: I may have to go and uh, drop a link to this in one of the discords and see if anybody there wants to go and give a give a talk on it because that would be fun. Uh, it's, it's lore, lore radios that talk to each other but they, they mesh and they're all decentralized.

I think it would be a great fit.

Gary: Yeah, yeah, it sounds like something that people would be interested in.

Simon: And I'll be bringing one of my raspberry pies along and giving a talk about running, you know, Because that because I host most of the things I care about Uh on a rack of raspberry pies just downstairs from where i'm sitting And it's very straightforward to do Um, and there's no reason why most people couldn't do it to be honest.

Jonathan: Yeah. Yeah, that's true uh, simon, you said you are the sort of the the treasurer and the um, I guess sort of the legal host You of Oggcamp? I'm curious. I'd like to hear more about that. Um, why, why such a thing is needed and what all you, uh, what all you do.

Simon: Yes. Well, uh, so that, that arose, uh, a number of years ago when, um, a, uh, a gentleman I work with who happens to be a lawyer said that there was a community of people running a conference and they needed a fiduciary host.

And I, I run a, uh, a small not for profit organization called Public Software in the UK. And, um, the reason you need a fiduciary host. is because, uh, I think like a conference, there's actually quite a lot of money floating around, uh, all camp is going to cost us probably a five figure sum of money to put on.

And, uh, and it, it breaks even. So the, at the end of the process, there's probably going to be a five figure sum of money sitting ready for next year. And somebody has got it. Somebody's got to keep it, and you've got to trust someone to own it. And, um, this particular conference is one that gets run by whoever is available next year.

And so the people who are responsible for it in 2024, possibly none of them will be involved in 2025. So, If that's the case, who owns the domain names? Um, who has got the money in their bank? Uh, who is it that has got the legal standing to make contracts with the hotel? And so that's what Public Software does, the organization that I run.

Uh, and public software, um, provides a home for unincorporated associations of, of people who are doing things for open source and free culture. So, um, we host, uh, OGCAMP and we have one other project called COSM, which is a crowdfunding pot for maintaining the, uh, open document format that you find in, in LibreOffice and now in Microsoft Office.

Uh, and so, uh, I, I do that. I, um, Uh, run the incorporated association that looks after the assets of odd camp so that Gary is able to show up and there is money for him to spend on the conference this year. And there is somebody who's willing to sign the contract with the pendulum hotel. And there is somebody who is willing to pay the, the, uh, the crew radio supplier.

And there is somebody who can pick up lunch for the volunteers. And then at the end of the show. Um, public software will, will draw everything together and get ready for next year and make it available to whoever shows up to run OGCAMP 2025. So fiduciary hosting, it's, it's kind of like what, um, software in the public interest or Software Freedom Conservancy do for open source projects.

Uh, just for things, I do it for things that aren't software, for, for a small conference, for a standards project. Publix offers hosted other small activities over its lifetime as well.

Jonathan: Yeah, I guess that's something I didn't think about, but if you have that much money running or moving around, like if, if nothing else, you just need to be able to have a paper trail of nobody walked away with a thousand dollars in their pocket that shouldn't have.

Simon: Yeah, so we do that. And also, um, you know, there is value added tax, sales tax to be collected and paid. Yeah, that's true too. So we do that. And, uh, uh, you know, that we've, we have credit card machines, uh, you know, ordinary individuals typically don't have. Credit card facilities and we have credit card machines so we can host the Stripe account and we can, we've got, we have a Square account that we make available.

So it's all of these little details that make it possible to put on an event. Modern society has, uh, as in so many other areas, has, uh, has, uh, officialized or bureaucratized the things that previously would have just been, uh, the way that mates did things together. So, if we'd been running a village fate 40 years ago, we probably wouldn't have needed public software.

Right. But in 2024, you absolutely need a fiduciary host to look after these things for you. Because otherwise, Gary is going to find himself with HMRC wanting to know where the VAT is. He's going to find himself having to pay corporation tax on the turnover personally. And he's going to find it's added to his income tax return.

And he doesn't want any of those things to happen. Right. So, so we make sure that doesn't happen too.

Gary: Yeah, Simon makes it so I can just, uh, contact the hotel, get the invoice sent, and, uh, forget about it and carry on with my day, which is, uh, which is always great. He takes the bureaucracy out of what otherwise would be quite a painful thing come tax return season.

Yes,

Jonathan: yes, understood. Uh, how many, how many tickets do you guys, uh, well, let's see. How many tickets total? What's your, what's your max capacity for the event?

Gary: Absolute maximum for the event in the venue that we're in this year is 250. We hope to get somewhere close to filling that. Um, and we're not, we're not too far off actually.

Um, Where are you at?

Jonathan: How many have you sold? How many are left? I

Gary: think we're just under 200 at the moment. We're not too far off. So,

Jonathan: so if nobody else signs up, you guys still, you're on, you've got a good conference, even with just under 200 people, that's a, that's still a good group of people.

Gary: Hey, we've, we've definitely got enough people to, to put on a good show, I think.

Um, and yeah, plenty of people to come and come and fill slots for talks as well. So

Simon: in terms of cashflow, looks like we've collected enough money to refresh the pot for next year. So, um, I think this, this event is going to break even and that's what we want to do. We don't want to make a profit. We want to, we, we do want to break even.

And we want there to be about the same amount of money in the pot next February as there was in the pot in February when Gary took it.

Jonathan: Uh, what, what is the cost for it as far as per ticket?

Gary: So what we've always done with Old Camp is trying to keep it very accessible to people. So This year, um, I think it's, it's no secret to anyone that the bottom has fallen out of the ad market and, uh, sponsor money is much more difficult to get hold of.

Uh, so this year we've said, yeah, we're going to honor that pay what you can model. Um, but we've got a suggested ticket price of 40 pounds. So, For your 40 you get, you know, two days worth of, you know, really good solid talks on the schedule track, plus whatever, whatever else turns up, plus the hallway track.

Um, but if you can't afford that, then absolutely you can pay us anything from 1 upwards. The only ask there is that you cover the cost of our ticketing platform. Because they charge us a small amount to issue tickets and store people's records and things like that.

Simon: Then there's the opposite end of the thing where we have a bunch of people who say, you know, we really want to support you.

And rather than taking donations, we put on the system some high price tickets. So we've had a whole load of people who have bought what we call professional tickets, which are the 100 tickets that for people who can expense it. And so we've had, uh, quite a few people have bought professional tickets.

They don't have to, they could have just bought the one pound tickets if they wanted to. And it's quite comforting and reassuring that if you look across all of those tickets we've sold, uh, there's a pretty even spread of people at pretty much every price point between, I think the most anyone has paid is 200 pounds and the least anyone has paid is five pounds at the moment.

And there's, we have a pretty good spread across the whole, Yeah,

Jonathan: I'm, I'm actually really glad to hear that you guys keep it affordable that way. I, I was looking at going to a conference. You know, not, not anything particularly open source, although I tend to make conferences open source when I go to them.

Um, I, uh. Famously was at a, uh, U. S. Department of Defense conference where they were talking about the problems with, uh, FPGA security. And, of course, I, from the back, Why don't you just open source the toolchains if you want it to be more secure? I have lots of fun doing stuff like that. Uh, anyway, this conference, I was looking at it in the background.

Tickets were like 699 a piece or more if you wanted everything. It's like, oh man, we can't expense that right now. Um, so I think that's really neat that you guys make it, uh, you make it affordable. You mentioned sponsors, uh, And I, I, I was going to ask whether you have sponsorships. Um, and, uh, so how does, how does that work?

Do you guys go, do you have like a list of companies that you go after each year that it's going to be put on and you say, Hey, you guys supported last time. Would you be interested in, in, in helping do this? And then like, is there a banner somewhere? You know, is there, is there a banner wall where people come up and take their picture with all the logos behind them?

Yes.

Gary: Yeah, we do have that. So. So I think one of the nice things about OGCAMP is that we've always tried to not be super corporate, right? Um, yeah, you're not going to come and see, you know, logos of insert big FOSS company here plastered in every single conference room. But what we want to do is cover our costs and we want to, you know, return a fair value to those people who help us cover our costs.

So, um, this year we've actually ended up with four sponsors. I think I'm right in saying that none of them have sponsored us previously. They are all entirely new sponsors this year. Um, and they are anything from kind of small software development companies, to larger open source projects, uh, to a trade union actually this year is our pinnacle sponsor.

I mentioned Utah a couple of times. Yeah. Um, And yeah, so they get their Lego on a t shirt and they'll get their Lego on banners and things and, you know, a personal thanks from us for, for really helping make our camp happen because we couldn't, we just couldn't do it without, without that little bit of extra cash injection that we get from sponsors.

Um, So, um, yeah, I think like I said earlier, uh, Utah running a couple of workshops this year and a panel discussion. Um, a couple of other sponsors have got stands in our kind of community room where we sell merch and put out tables for people to do anything from play role playing games to show off their cool Raspberry Pi projects.

Um, so yeah, we, we have a list of people we target, um, but this year it's actually just been an entire new set of sponsors, which is actually really cool. And what we're hoping is that that will introduce a whole new demographic of people to, to OrcCap.

Jonathan: Um, so I was going to ask, I was going to ask who the sponsors were, and I think you sort of just covered that, um, you've got the, the trade union.

Uh, is there any music? So one of the things that it, it sort of, it sort of hits my mind that there might be an intersection here is that you could have musicians and people very, very interested in music, very, whether putting it on or doing talks about music or even building your own music hardware, it seems like you could have some of that.

Gary: Yeah, so Dan, previous organizer, had previously done our evening entertainment in 2019, which was quite cool. We don't have any kind of scheduled evening music entertainment this year, but I'm absolutely sure that there will be people turning up with their cool synth project that they've built with their ESP32s or something.

I know we've definitely had things like that in the past where someone's turned up with, uh, yeah, here's my, here's my rack of, uh, Analog synths that I've built from open source hardware and say yeah that there always is that kind of maker slash music crowd I think that turns up our camp But again, it's it's kind of like we were saying it really depends on who turns up as to as to what's gonna be there But but that is one of the the many crowds of people that turn up our camp.

Jonathan: Yeah Do you have anything like I know in some conferences people like to come and actually camp out? Uh, to, to attend it rather than having to pay. Is there, is there something like that? Is it going to be a tent village?

Simon: Um, I think it's unlikely. Okay. Um, you know, the Manchester is not the greatest place to camp out during October.

Uh, and, or, and it's gonna be quite busy in the city this year. There's, because there's a, there is a sporting event going on, uh, as well at the same time as the conference. So I, I, no, I, I think everyone's very likely to either be local or in hotels. , uh, certainly I haven't booked a room yet, so I really must get round to doing that.

I, I might be sleeping on the street. Simon's gonna be ski out on the

Gary: street. Simon's gonna be on the bench outside in the, uh, in the hotel.

Jonathan: No, Simon's going to be in the hallway track like, Hey, do you have an extra bed in your hotel room?

Simon: It's the hallway and sleeping bag.

Jonathan: Oh, that's great. That's great. Um, so let's see what, uh, what, what have I not covered?

What else do you guys want to let folks know about? What, what else are you looking forward to with the conference?

Gary: I think for me, it's, it's just getting something running like this in the UK. I think the UK is. Is kind of really lacking in these community led FOSS events. Um, yeah, there's, there's a bunch of them that exist across in the U S um, but, but we're definitely lacking in them in the UK.

So I think for me, it's going to be the main thing I'm looking forward to is just seeing old faces, new faces, what people come up with and really, you know, how the community has evolved in the last five years, I think really for me is, is the key thing that I want to get out of old camp this year.

Jonathan: Yeah.

Simon: I think I'm in a similar place to that, you know, in the, the, the curious thing is I've been doing open source for a couple of decades living in the UK.

And I've always had to go abroad to go and do anything, you know, and so there is a certain desire to hold grassroots events that are not about, uh, feeding a corporate strip mining open source culture, but they're actually about people who use things and make things. And so I've always loved OGCAMP for that reason, that it's about the only event that I can go to in the UK.

Actually, it isn't the only event in the UK. There is a, there is a, uh, another conference in the UK now as well called, uh, EMF, which actually does have a campout, uh, that happens as well. And that's also in the north of England. So there's, there's now two events in the UK that, that are, uh, addressing that niche.

And I find that really encouraging that that's happening. Um, I, I do, I've often wondered whether, uh, Europe needs to have an, uh, an, a summer FOSDEM. And I'm kind of hoping that eventually one of these events is going to evolve a bit into a summer FOSDEM so that we can have a Europe wide open source event.

Europe does have quite a few good open source events. Um, there's a really good one that I'll be plugging later in, uh, happening in Northern Italy in November. And then there's a couple of, uh, good ones in Germany, like, uh, uh, the one that happens in Bonn. Um, and there's a, a, a good international conference called FOSS Backstage that happens in Europe.

So, there's good things going on, but the UK has tended to be, um, the one place where there has not been enough going on. And I think that's one of the reasons why I'm excited about being involved with Oddcamp. I'm really pleased that Gary's come along this year. Gary's a very level headed, very capable, uh, uh, uh, sensible person.

And I'm very pleased to be supporting him.

Jonathan: Yeah. You know that that idea about not having anything in the UK and sort of missing it I get that because like there's not a whole lot here in Oklahoma where I'm at and Be nice. Don't laugh at my state. Uh, I think I think Oklahoma is actually about the same size as UK if I remember correctly It's it's gonna be on the on the discord Somebody put a picture of like the map of Europe and then Texas over it And like, Texas is just, is bigger than Germany and covers half of France and covers half of Italy.

And, you know, Doc Searles famously says that the big difference between the United States and Europe is, in the United States, we think a hundred years is old and over in Europe, y'all think a hundred miles is far. And that just, that really, I think that's actually really great. It, it covers, Sort of the mindset difference that comes out of the history and geography of the two places.

But like I've had this thought there's nothing in Oklahoma that I that I know of at least nothing big uh, you got to go down to Texas and I got real excited because there was somebody that was going to do like a once a year I think it was the Texas Linux Fest and was going to be in Dallas like oh Dallas That's that's you know, that's when the not too much more than 100 miles.

That's not very far Uh, you know, I could get down there in three hours. That's not too bad um I don't know if that's happening again, because again, we got hit with COVID and so a lot of things closed down. Yeah, it really makes, it really makes me think that, uh, maybe we need more people from the community to start talking about things like this and try to put them on and put them together.

Because there is something to be said for getting together in person and not just over the internet. Uh, meeting, meeting people in person. And actually doing things, putting your hands on things together and, and fiddling around with stuff. Um, So I think, I think that is cool and I, I, I look forward to hearing stories from this year about what all happened and which talks happened and which ones were popular and people that got to meet each other for the first time.

Simon: And you know, Jonathan, we can discuss licensing OGCAMP to you for OGCAMP West if you want to run it over there in Oklahoma.

Gary: Well, it was that meeting of people in person that kind of inspired me or I guess, uh. Maybe drunkenly got me convinced to organise Oddcamp this year. Like

Jonathan: all great ideas, it was birthed in a tavern, in a pub.

It

Gary: genuinely happened in a pub, yeah. So, um, it was actually, I worked for a brief period with a previous organiser, John Spriggs. And, uh, he was like, oh, I'd love to see Oddcamp come back, but I don't want to be the one doing it. And, uh, you've been to Oddcamp a load of times, you know how it works. How do you feel about putting, uh, putting on an Oddcamp?

Couple of beers later, and before you knew it, I was sending an email to Simon saying, Hey, what do we need to do to get this thing kicked back off on a cold December night at a white Christmas party? So, uh, yeah, yeah, it's, it's getting the community back together, I think, is where some of these things really, really spawn out of.

So it's also going to be interesting to see, like you said, what, what other things spawn out of Oddcamp and having people together that, um, yeah, is, is going to happen. Yeah,

Simon: you know, I want to say to all the odd camp attendees and supporters who are, who are listening that, um, we're getting back again, back together again this year.

We haven't been able to nail down all the things that you, that you want there to be there. So try and bring them with you this year, if you can. And if you can't help us organize it next year, so that they're there. So, you know, we, as Gary said, we don't have a. Um, an event organized on the, uh, the intermediate night on the Saturday night.

Uh, it would have been great to have booked a room and got a band and had everyone, you know, able to order their fish and chips and mushy peas and their half a lager. Um, but, uh, you know, that didn't happen. And so maybe it can next year and maybe it's one of the people listening to this that's going to be the person that, that insists on organizing it.

Um, Probably too late to organize it this year. I think that the hotel told us that they, you know, we've, we've, we've paid for a certain number of rooms. Uh, we could go back to them and try and say, Hey, you said, you know, we said we didn't need the ballroom. Well, we do now. Um, and if you're listening to, you're listening to me saying this and you really want to organize the thing in the ballroom, then get in touch.

My email address is, uh, simonatwebmink. com. And you can, you can get ahold of me that way. Okay. But, so this year I see very much as a relaunch, you know, it's, it's doing the things that we know work, um, hoping someone's going to bring the stuff that we haven't been able to organize, but really hoping that we're going to have some new blood who are going to help us organize, um, the double this event next year.

Jonathan: Yeah, yeah. So what, like, you, you allude to this, what's not happening that people wanted to see happen this year? What happens in the ballroom?

Gary: Well, what happens in the ballroom stays in the ballroom, I guess. I could tell you, but then I'd have to

Jonathan: kill you. I walked into that one.

Gary: Previous years, Oddcamp has been quite famous, I think, for its, its social track.

Um, so we've always had some kind of entertainment on maybe the Friday night, definitely the Saturday night and occasionally actually the Sunday night. Uh, we've always had some kind of prearranged entertainment. So, uh, yeah, whether it be a band playing or some kind of FOSS quiz on the Saturday night or something like that.

Um, Like Simon said, this year our focus has really been on what do we need to do to get OggCamp up and running. Um, and what do we need to do to do that in a way that is sustainable for a small, new, organised increase. So, some things like that have fallen by the wayside. Um, and yeah, Simon alluded to things like the raffle, um, We're pretty sure the raffle is going to happen this year.

They're kind of famous or camp raffle. Uh, we just need to, as an organizing crew, get our head into gear and make sure that we've got everything that's needed in terms of prizes there. Um, so I think the big thing is going to be the social track. I think that's the big difference that people might notice this year compared to previous years.

Um, But that said, um, the hotel have graciously said that we will have the bar open for OCCAMP and they're putting on some food and things like that for attendees to buy at a reduced rate. So, yeah, there will be social tracks coming, uh, you know, there will be things happening and I think it's human nature to have a like minded group of people get together and congregate in the hotel bar or head off elsewhere.

Um, so, yeah. Yeah, I think, yeah, it will, it will feel like an old camp, um, it's just going to be a case of, uh, yeah. I guess a little bit of autonomy happening, uh, among attendees that perhaps we, we didn't do quite as much before.

Jonathan: Yeah. My, my wife is in the discord and she's also apparently listening to some of this and she says, if life circumstances were different this year, I'd all be for attending this conference in Manchester.

Like both of us, none of this just sending Jonathan business. In years past, I went to a conference in Dublin by myself and, uh, While it was fun, it was a bit of a bummer that I couldn't take her along. So one of these days,

Simon: you know, it's not too late. Just check, check out, uh, the, the, what flights are available.

There's going to be some cheap flights direct to Manchester. You can just make the weekend for it.

Jonathan: Yeah, I think our kids are a little too young for that at this point, but, uh, that's, that's all right. Maybe in a couple of years. So, what, let's, let's, let's, let me ask you this. What are some, like, highlights from the last couple of times you guys had OggCamp?

Uh, what are some of the talks that really you remember or just memories you have that you think people would enjoy hearing about?

Gary: I think for me the first dog camp I went to was really eye opening just as to how small the UK is. Um, so I ended up actually getting talking to a guy in, uh, on one of the hallway tracks and, uh, I'm from a fairly small county in the UK, uh, so Norfolk on the east coast for anyone who knows it.

Um, quite famous for the fact that we've got no motorways in any one city. So it's, uh, it's a fairly small place and I got talking to this guy, um, And we were like, he was like, where are you from? And I said, where I was from. And it turns out that he lives five minutes down the road. We both have, yeah, had similar jobs at the time.

Very similar kind of interests in FOSS and things like that. And we've stayed in touch ever since. So that for me was a real OGCAMP highlight. Um, but in terms of talks and things, it's actually been the talks that, necessarily directly FOSS related that have really appealed to me. So I kind of alluded to before that we've had talks on things like mental health and stuff like that.

And actually going along to those talks and just hearing those insights as to, ah, this person struggles with the same things of, you know, Struggles of working in IT as I do or this person has come across this particular problem They've tried to solve in this way and perhaps that's a different way that I could approach something that have I've always been highlights for me So it's it's almost been the unexpected things that you couldn't plan for going to a conference that have been highlights for me So I mean you have any thoughts

Simon: Yeah, wandering around seeing the toys people have got in the, uh, in the, in the exhibit area where the tables are.

That was, that's always a feature for me. I remember having a picnic on the front steps of the venue in Liverpool when the conference was held there. Um, I remember giving a talk all about, uh, the, uh, legislation that keeps on trying to backdoor crypto, um, which is back again this year, just in case you had any concerns.

Uh, so, uh, those are all pretty, you know, they're, they're, they're minor sounding things. They're pretty random. For me, the event, this is, this is the, not the event where there are the, the big corporate talks. This is the event where there are the, uh, the important new relationships are created. And that's what I really hope for for this year as well, is that we'll meet interesting people and befriend them.

Jonathan: Yeah. Yep. Very cool. All right. Uh, let's see what, uh, what have I not asked about? What have we not covered that we, we ought to let folks know about?

Simon: Great questions. Uh, so if you want to buy tickets, uh, there are still a few available. Uh, you'll, you can find them at, uh, I think the org. camp is, uh, is the, the easiest to remember URL that will get you there.

So if you type in org. camp, it will take you to our website. Uh, and I'm pretty sure that we bought all the other websites that sound like that as well. So I think we, we have oddcamp. com and things, but what the cool one is odd. com. There

Jonathan: you go.

Simon: Um, and, uh, you know, I encourage you to pay what you can for the tickets.

And, and that means at both ends of the scale, if you can afford the thousand dollars for the ticket, please buy a ticket for a thousand dollars. Because you're, you're investing in the future of the conference. When you do that, the conference isn't run. For the benefit of, uh, anyone financially, um, what will happen if there is a surplus is public software will hang on to it and it will be available for next year's conference to be bigger and better.

So, um, uh, it's, you know, it's, if you can afford to, uh, to, to buy the thousand, thousand pound ticket, we don't actually have a thousand pound ticket, Gary, could you fix that before the show goes out? Um, go, go, go ahead and do that. Um, uh, Also, keep an eye on the socials, we do have a couple of venues, I don't think we've fixed on an official venue on social networks this year, but we do have a Mastodon account.

We do have a room on Matrix that you can find, and there may be some other remnants somewhere, I think we might still be on that social network that nobody should use anymore.

Gary: Yeah, there's a, there's a few places there's, there's the X accounts, there's a telegram group, uh, there's a discord as well. I know you love hanging out in discord Jonathan. I

Jonathan: didn't say I love hanging out in discord. I said that's where I find myself a lot.

Gary: Yeah, I think I'm in like five discords at this point.

I can't, can't honestly say I read messages in all of them. I'd have to go count. Oh my goodness. A lot. It's quite a lot. Um, yeah. Let's see. Og. camp is probably the place to go, um, and anything that you need to know is, is kind of linked from there really.

Jonathan: Yeah. Uh, what are the dates? I don't think we've ever actually said.

When is this?

Gary: It's the 12th and 13th of October, uh, this year, so coming up in just a couple of weeks.

Jonathan: Less than a couple of weeks. Yeah, that is, that is coming up soon.

Simon: There's still rooms at the hotel, so you can still stay at the venue, which is the Pendulum Hotel in Manchester. So you, uh, that, that's available.

There's still a few tickets left. Uh, if you're quick, um, Gary's going to have to turn off the tickets when we reach the fire limit for the venue. Uh, so, uh, you want to get in there as soon as you hear me saying this, cause there's stop listing now and go to org. camp and buy your tickets.

Jonathan: Yeah. That's the important thing, right?

You don't have to listen to the episode. If you're going to be there, uh, what, what year did, did our

Gary: camp start? You'll know this better than I will, Simon. I don't know, I joined it after the fact. 2009, I think. The last OddCamp was our 10th anniversary in 2019. So, uh, this is the 11th OddCamp, but it's our 15th year, which, uh, feels like another milestone.

Jonathan: Yeah. Boy, the world was different in 2009. Goodness.

Simon: Ah, those were the days.

Jonathan: Yes. Yes, they were.

Simon: Still a Sun Microsystems, then.

Jonathan: Yeah. Uh, Gary, I didn't ask and I guess I should. How did you, uh, how did you get into, we know, we know Simon, we know where Simon came from. How did you, how did you get into like the, the whole open source and free world?

Like what's your, uh, what was your gateway drug that got you interested and that you started coming to Oddcamp?

Gary: It was a teacher back in high school actually handed me a CD. So yeah, I, I at the time was, uh, was running a, uh, Not so great laptop that been handed down and handed down and handed down and Windows didn't run so well on it and I just went to this teacher who was a Gentoo user back then and said yeah, what could I do to sort this out and he said, I'll sort you something out tomorrow and came in and almost, you know, dealt under the desk this Kubuntu 7.

10 or 7. 04 or something CD and I was like, yep, here you go. Um. And never looked back, I installed that on that machine, it lasted me another few years and uh, yeah I guess from there ended up getting into, you know, working in IT, advocating for replacing a bunch of Windows stuff with FOSS where it made sense, um.

Then kind of got into the kind of cloud world and continued to advocate for open source stuff there. Um along with attending Uh op camp and fosdem and linux fest northwest and anywhere I could get my fill of uh, false goodness Um, that's that's where i've been for the last 10 or 15 years, I suppose

Jonathan: And you're thankful that your teacher gave you a kubuntu disk and not a uh, not a gen2 disk

Gary: Uh, yeah, I mean it might have had a slightly different appeal, I mean since then I've moved off of the Plasma desktop and, uh, daily driving Fedora with Gnome.

But, uh, yeah, I mean, it was probably a wise choice not to send me down the road of, uh, doing a Gen 2 install at, uh, you know, 12, 13 years old or whatever it

Jonathan: was. Yeah. All right. So let me, let me actually cover this and we get to ask Simon. I don't think we asked Simon this. Um, and I don't know the answers.

We're gonna start with Gary. I want to know what's your favorite text editor and scripting language.

Gary: Uh, for me, it's got to be nano and bash. I just, I want to deal with something simple. Uh, so I actually spent a lot of time just in nano writing stuff in bash, which is probably the most boring, terrible answer I could give.

But for me, I'm all about keeping things simple.

Jonathan: No, no, that's all, that's all good. You know, Simon, I don't know that I know the answers to this. I don't think we've ever asked you. Yes. What's your, what's your favorite text editor in scripting language?

Simon: So, I never really got into text editor religion, and I use whatever there is that is on the system I'm running.

Um, I have a soft spot for nano on my raspberry pi's, uh, but I, you know, I used to enjoy using E when I was at IBM and, uh, which nobody will know about and, um, uh, it's, but, so he is the editor that, that Emacs. Became when you added the macro language.

Jonathan: So

Simon: I, I used to enjoy using that. Uh, I do most of my editing these days in whatever notepad I can pop up on the device I'm using at the moment.

And I use Markdown, um, and when it comes to scripting, um, it's been a while. I have to tell you. Uh, so the last scripting language that I used in Anger was on a, a sign organizer. Uh, I actually wrote commercial software in sign organizer scripting language. As

Gary: you went lying when you said it'd been a while, eh?

I think that's pretty much it. For me, it's a really difficult question to answer these days because I spend so much of my time in the cloud world that it's like, you know, whatever thing abstracts away the clouds is what I end up using. So I end up writing a lot of Terraform and Ansible and CDK and stuff these days, um, just because I guess that's what I work with day to day.

So, uh, I don't get to play with, uh, with much nice, cool scripting stuff anymore.

Simon: Yeah, to give you an idea of how old we're talking here, um, uh, so, uh, my story was a little bit like Gary's, so my, my physics teacher at school, uh, uh, was moonlighting while he was teaching physics and making the motherboards for, uh, compute for development systems that used a 6800 microscope.

processor called SWTP 6800 development systems. This was in 1977. And, um, so that was my first computer. Cause I used to, I do, he used to soak test them in one of the labs. And we used to go in during the breaks and program them. And the reason I'm telling you that story is I went around the computer history museum in Silicon Valley recently.

And I, there is the computer that I learned to program assembly language on, uh, sitting in the computer history museum. And that's how I know I'm old. Ha!

Jonathan: When you find your first computer sitting in a museum, yeah, that would do it. Uh, now I have to think, like, what was my first computer that I ever actually did anything on, and is it old enough?

It might be. It might be. The first one I ever played with, my dad had one of the Trash 80s. Excuse me, TRS 80. But that wasn't really modern then that was like pulled out of storage But I think we had a like a tandy one of the original tandies and I think those are just old enough to be museum pieces now, so I suppose i'm old too.

Simon: Yeah You keep using you keep using that hair dye jonathan

Jonathan: No, no, mine's all natural there's a little gray poking in there, but it's not too bad

Gary: Now I feel young when I can say that it was a pentium 133. It was You The first machine I really used and did stuff on, but even then that was somewhat of a hand me downer space.

Jonathan: Yeah, yeah, yeah, I get it. All right. Well, I am, uh, I won't be there, but I'm looking forward to OGCAMP because it sounds like you guys are going to have a great time this year. And hopefully we're going to have

Simon: an old fogeys reminiscence session over a beer in the, uh, in the, in the bar at the hotel where I will show people photographs of the SWTP 6800.

And we can talk about a 6800 assembly language and the unique register commands it had.

Jonathan: Yeah, I mean, who knows? You guys keep doing it, you'll eventually have the, uh, retrocomputing track. People will bring their old machines and let people play with them. Alright.

Simon: Now there's an idea. Next year, Gary, okay?

Yeah, we'll put it on the list.

Jonathan: I mean, retrocomputing is big right now. There's a lot of people that are, uh, Very nostalgic for the machines of yesteryear. So, all right. Thank you guys for being here. I appreciate it very much. Gary, particularly, thanks for coming and telling us all about the conference and, uh, best of luck here in a couple of weeks that everything goes exactly the way you want it to.

And, you know, all the surprises are good and all of that stuff.

Gary: And tech gods are with us for demos and audio and. projectors and other such things.

Jonathan: Yes. Yes. I've been that guy and there's always something that surprises you. Uh, awesome. Thank you so much. Uh, Simon, this is where normally I ask you what you think, but you were sort of the, uh, guest too.

Um, I guess I'll go ahead and ask you though.

Simon: Well, I think everyone should buy tickets and come to the conference in Manchester because, uh, you know, I could. I could hype it up and say it's going to be, you know, the, a super majestic technical whiz orgasmic thing. Uh, but actually we're just going to be a group of people who want to be friends, uh, coming together and sharing a space and sharing our interests with each other.

And, um, uh, I think that, uh, if that appeals to you, then you should go get a ticket. Uh, and if that doesn't appeal to you, you really should go to, I dunno, some Oracle conference.

Jonathan: If that doesn't appeal to you, then maybe you need to rethink the decisions that have led you to this point in your life. Oh, all right.

Well next week, interestingly, we have somebody from IBM going to talk about their, uh, Open AI stuff, and it'll be very interesting to compare notes with what IBM thinks about open source LLMs and AI with what OSI and Mr. Simon Phipps thinks about it So we will keep that kind of as a bug in the back of our minds as we talk to IBM about it Simon is there anything you want to plug?

Since we have you here. Well,

Simon: uh, there's, there's Ogcamp, obviously. Go to og. camp and buy a ticket now. I'm going, I also help organize another conference in Europe called South Tyrol Free Software Conference, which is another conference that you might not have run into. It's, it's now grown into quite a big conference.

There'll be over a thousand people there. It's held in a beautiful city in Northern Italy called Bolzano. Uh, it is the, uh, the second weekend in, uh, November, and I would love anybody who is listening to this is in Europe, can't make it to Manchester to come and meet me in Sno for SFS Con with the website is S-F-S-C-N do it and the conference is conducted in, uh, in English and will be fantastic.

Excellent. And what about you personally? Where can people go to find. Uh, probably the best place is, uh, on, uh, Mastodon where you'll find me as, uh, webmink at meshed. cloud, M E S H E D dot C L O U D. Uh, and I would love to see you, uh, popping up there and saying hi and following me. Um, my, uh, most of my stuff, if you want to find out about me is at a website called webmink.

com. That's, uh, W E B M dot I N K, and that's got all the pointers to everything about me on it. Simon, you're all

Jonathan: about the fun,

Simon: uh,

Jonathan: TLDs.

Simon: Well, you know, they exist, and it seems a crime not to make the most of them.

Jonathan: Not to have fun with

Simon: them. Because you can do good things that are memorable with them, you know?

Uh, so, you know, you can email me if you want to. I'm simon at phipps dot email. Uh, you can have a look at my website. Uh, that's, it's webm. ink. And, uh, I, I host my own Mastodon server at, um, mesh. cloud. So find me in one of those places. Running on a Raspberry Pi. That one is not. That one is actually, uh, running on a slightly more robust hosting server, because it turns out it takes quite a lot of energy.

Jonathan: Yeah. Yep. Uh, it makes sense. All right. Thank you, sir, for being here. Sure. It'll do appreciate it. Um,

Simon: Well, thank you for the invite, you know, have, have me back again sometime.

Jonathan: Oh, for sure. For sure. I've, goodness, we've, we've got to, we've only got so many co hosts and people get tired of, of listening to, to, to Dan and Doc and, and all of that.

Definitely an important part of the crew, man. All right. So if you want to find me, Of course you can. Go to Hackaday, that is the home of Floss Weekly these days, and we sure appreciate that. I've also got the security column that goes live on Hackaday every Friday morning. Keep you up to date with the, uh, the bits and the bytes that are moving in ways they're not supposed to.

Uh, and then we do, of course, have the Untitled Linux Show over at Twit. Dot TV still. And we have a lot of fun there covering, covering the news and a bit of, uh, open source stuff, but with a decidedly Linux twist on it. Uh, so make sure and check that out too. We do appreciate everybody that's here. Those that watch both live and on the download, and we will see you next week on Floss Weekly.

This week Jonathan and Simon chat with Gary Williams about OggCamp! It's the Free Software and Free culture unconference happening soon in Manchester!

- https://ogg.camp

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Randall joins me and we talk with Michael and Benedict about EMBA. That's the Open Source Firmware Analysis Toolkit that seems to do about everything and just about includes the kitchen sink too. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 802, recorded Wednesday, September 24th.

EMBA! Layers upon layers of Bash.

Randal: It's time for Floss Weekly, the show about free! Libre, open source software. I am your host, Randall Schwartz. Merlinisunwich. com. We're here each week! The boomers, the shakers, the bigs Wait, what? Wait, wait, Jonathan! I'm the host. You're the co host today. You're the co host today.

Jonathan: Wait a second, this is Floss Weekly and I'm the host, Jonathan Bennett.

I'm so sorry. And Randall Schwartz is back as co host today.

Randal: Oh,

Jonathan: goodness.

Randal: I wake up in the middle of the night sometimes saying that exact phrase because I said it for You know, 13 years worth of shows. So it's I really, really do want to appreciate, appreciate you bringing me back as now possibly an occasional as often as you'll let me co host. Because I do actually, I miss the show.

I've got to say that for everybody out there who's been along loyal friends. I do miss the show. What I, what I don't miss is that I now have. My five hours a week back as Jonathan well knows there's a lot of time. There's a lot of time that goes into this show every week, every, every week. And I'm happy to have most of that back now.

And the two hours a week that I will take the coast here. Much simpler than being the pro producer, show runner Jesus and host. So thank you Jonathan for doing that job and keeping the show going. I appreciate that as well. So, yeah.

Jonathan: Yeah. You know, one of my, one of my favorite things about taking over the show is that I, I get to have Doc Searls and Randall back as part of the, the rotating cadre of co-hosts, and I think that's just, that's just fun that to be able to have you guys back involved with the show.

So today's topic is EMBA and that is, it's a firmware analysis tool, I think mainly targeted at Linux firmware. This is it's, it's really an interesting thing. You know, there's a lot of, there's a lot of OpenWrt, right? Like I'm a fan of OpenWrt, but there's a lot of old versions of OpenWrt that are out there in the wild.

And it's, it's pretty scary sometimes to see like 10 year old firmware still out there shipped on devices and I kind of wonder do they have a module that just spits out This is this release of open wrt beware

Randal: Yeah, and and i'm i'm not terribly in this space So I had to do a bit of research to try to figure out how this is being used But now that you mentioned open wrt there's a recent news item actually it says that tp links Routers are now all considered caustic.

And so, because they were made in China and apparently there's been some spyware that's been traced to TP link. So everything TP link is at risk right now. So

Jonathan: yeah. Yeah, I saw that story. I'm not sure if, if I don't, I'm not sure if the statement was that they are like a compromised and malicious from the factory or that they just have zero days that allowed that allowed them to get popped in that.

I was gonna say, I think we

Randal: have some experts coming up that can answer questions kind of like that. I'm also interested in particular in reading ahead of the show. Is this primarily a white hat tool? Or can it be used as a black hat tool? And what are the implications of that? Having a tool that can do pen testing, vulnerability testing on a common piece of firmware somewhere.

So I think it's probably one of the questions I really want to, I'm curious about.

Jonathan: Well, we do have the experts. We will make sure and ask about that. I do want to mention that Hackaday is what Floss Weekly is a part of Hackaday. Hackaday is a part of SupplyFrame and SupplyFrame is actually owned by Siemens.

And I say that because the EMBA project was at least partially developed originally as a part of Siemens, or there there's at least some shared history there. So we want to make full disclosure. Now we do have Michael and Benedict with us today, and I want to say welcome to both of you.

Max: Hi guys.

Hey guys. Good to have you.

Jonathan: Good to be here. Yeah. So maybe the best place to start here is to ask each of you, how do you fit into the project? Like what are your individual roles in, in this thing?

Max: So from my, from my side, I'm currently primarily or directly with the, with the central Amber project. I'm maintaining the central Amber project, which is, let's say the, the, the scanning backend. I'm, I'm more or less the founder of the whole Ember idea of the firmware analysis idea. And we've started multiple projects from time to time.

One is Ember, the other is Embark, where Benedict is the, is the main maintainer now. So, so at the end, we can, we can split it in this way. If you, if we talk about the main Ember project, then probably with me. And if we are going to, to the, to the let's say management interface, enterprise overview and a collaboration interface, then Benedict is the guy to talk to.

Jonathan: Yeah. And so same question to you, Benedict, how do you fit into the project?

Benedikt: As Mike just said, I'm the main developer responsible for Embark right now. Embark is our enterprise solution, like taking Embark as a backend and then trying to put it into a, Nice server front end. Yeah, oh how people would need it in the business environment.

And That's

Jonathan: yeah. Okay. So probably then a question for mike. I'm, i'm super interested in this idea of packaging it But let's let's start with michael first since you kind of kick the project off and I I do want to know like what What is that tie in like what did the beginning of this look like and what is the tie in with Siemens?

What was like the first problem that you guys wanted to solve?

Max: Yeah, yeah, and so probably we we need to go a little bit in back to the future I think we start we start we started with all with all of this Let's say what was it 10 to 11 years back. I think everyone started with with iot hacking then You There was, was Binwalk Craig Hefner did a lot of nice blog posts about cool IoT hacking, cool exploits.

Everyone tried to show some, some hardware hacking stuff. And the idea was back then at Siemens that we Get away from the, from the typical black box approach on, on testing products, our own products and also third party products, to a more gray box ish penetration testing style with, with the firmware enhanced.

So, with the firmware, you would be able to understand what happened in the background. If you're attacking the device, if you're trying to find vulnerabilities you get a better idea what happens in the background and which ways probably could be successful. And this was in the, in the, in this time also we, we built built the hardware hacking lab to extract the flash storage to get access to the firmware.

Sometimes you're, you're, you're in a lucky position to go just to the vendor website, download the firmware. Sometimes the, the firmware is behind the paywall or something else. And so we started with hardware hacking, identification of UART, JTAG, extracting, flash storage, and all, all the things. And then, then we had the firmware enhanced and we, we were, we were struggling really hard because we, we had not more time to perform our penetration test.

But we had so many ideas and we want to do much more stuff now. And so we needed to start automating everything that, that is possible. So and this was more or less the, the initial idea of Ember. It is more of a, of an automation framework in, in the field of embedded product Linux penetration testing that should help the, originally that should only help the penetration tester.

Yeah, that's, that's more, more or less the, there it was, it started like, I would say 10 to 11 years or, or ago. With, with, with an idea.

Jonathan: Yeah. And, and so it's interesting to see like the number of devices that are out there that just like the firmware is Linux. And we talked, we talked briefly before the show or in the, in the intro about open WRT is that a lot of what you guys see that it's just.

Ancient versions of open WRT that just get shipped on some of these devices.

Max: Yeah, you, you, you definitely see open, open VT everywhere, but let, let's say in, in the IOT environment, you see it everywhere. If you are moving a little bit away from, from iot more to the, to the OT environment where bigger products of in for factories or for for other things are used then we we see open vrt not that often But in the iot world, it's it's quite regular definitely

Jonathan: Yeah, one of the one of the fun discoveries that I have had is a lot of you ubiquity hardware Which you know, that's sort of your your small office home office to medium size I mean they have some they have some impressive stuff like all of the access points That's just open WRT or at least it was last time.

I logged into one of them So it's it's in a lot of places I

Max: think Netgear also has a lot of OpenWrt based systems out there. I

Jonathan: think, I think Netgear was like the OG of OpenWrite. They they're the ones that first did the GPL release that got turned into it. So that's not terribly surprising. All right, let's turn to Benedikt for a minute.

And so you are, you said you're kind of the guy that, that packages the whole, the final thing together and maybe makes it a little more business friendly.

Benedikt: So let's talk, let's talk about that. Sure. So the main thing about Ember is it's built in Bash, right? So we have a combination of scripts in the beginning.

That's what Mike spoke about is we had the idea of combining stuff into something that gets automated. And once we, or let's say Mike, the Ember team was at that point where like it was working and then you had something on the CLI, right? So on the command line interface, you have something to type in and then you get something out of it.

We built an HTML export thing. So you get something that is nice and shiny and clickable. But the issue you're always facing is you have some firmware and you are a team, a team of penetration testers who are essentially find, trying to work on one thing at, as a team. So you have certain, certain things one person looks at and certain things another person looks at.

And if you do what Ember does on every computer, then you're running into, well, time constraints also. So one of the ideas was to basically centralize the whole thing, put it into a server. So make it a business application in that sense. You upload something. Amber do the work and then you get something that everyone can look at without the usual issues of CLI where everyone works on their own.

Jonathan: Yeah. So do you, do you have any idea of like how, how popular it's gotten when, now that you've, now that you've put that package together? Do, do you have any, any idea of like, how many businesses are out there using it, how many researchers make use of it? I think that's

Benedikt: a good question for Mike because he keeps track of those things.

Max: Yeah. Yeah. It, it is quite important to, to, to get a good visibility for, for an open source project at all. And so, so I try to track it a little bit. It also helps for for arguing my, my work on our work on AMBA a little bit better. And during the last two years, we, we have seen that, that AMBA gets accepted in, in a very broad way.

So, there are popping up more and more blog posts out there. So, so people are talking or writing about using AMBA. And we have seen a lot of let's say multiple research papers that have used ember in some way or at least referenced it. And it looks a little bit they like to show where ember fails.

And this is great because we can then improve Ember, we get a better idea on where Ember gets used and how the people are using it and we can improve it, so nearly from every research paper we get new ideas to improve it. And if, if, if we, we are, we are also looking some, sometimes at Twitter and other social networks, and we, we, we have a little bit the feeling as, as I would say most of the penetration tests out there that are doing IOT penetration tests from time to time are at least using Amber besides their manual work.

Just to get an, get an initial overview of the, of possible vulnerabilities to get an idea of where they can dig deeper. As Ember is automating a lot of stuff, you can run, let it run over the weekend and you can start with with a quick start on Monday morning, much better than starting from, from zero if you have let run Ember over the weekend.

You get some, some basic information about the firmware, you get an S Bomb, you get all of the things that you can, can then use for, for improving your manual tasks and yeah, so, so penetration testers are using it researchers are using it, and we also are from time to time in contact with quite big companies that are trying to establish AMBA in, in some way.

Some kind of the security processes one, one big company has established Amber into their IC development process. The next one has established Amber as a quality gate for third party components. So I would say Amber is quite solid established. Hopeful, hopefully we get Many, many more developers that or back, or back hunters so that we can improve Ember in the future much more.

Jonathan: Yeah, yeah, absolutely. Randall, I know you were curious about some things around people using it. You want to, you want to take it and ask those?

Randal: Yeah, and where I'm starting with is, so I understand the basics of pen testing. I don't know how it applies as well to embedded. How much of EMBA is unique to the notion of it being firmware and what kinds of things do you do there?

I know like for traditional pen testing, you're opening sockets, you're trying memory locations, you're trying things. You're trying to recognize known vulnerable libraries, things like that. How different is it at the embedded level and what kinds of things do you do there?

Max: So I think you, we need to differentiate a little bit.

So if we, if we are talking about penetration testing of, of firmware, I typically our projects are typically looking like we have the device. So Also, we can poke around with the device like in a typical penetration test and on the other hand we have the firmware where we can look into firmware to find things that we can verify on the device.

So on the device by itself it's quite classical. So you're looking for interfaces like web interfaces, you're proxying your traffic you try to inject commands. That are executed on the operating system. And on the, on the firmware level, we, we're doing much more static analysis then. So we can, for example think about PHP scripts.

There are, if the web server is, is organized via PHP, then Ember identifies the PHP scripts and can do static code analysis on, on these PHP scripts. And then Ember can point us to interesting areas that we can try to exploit then on the device. So at the end, it's, it's, it's every always the same.

Amber gives you, gives you an idea where something interesting could be, and then you as a, as a, the responsible tester, you're digging deeper, you're analyzing this possible issue, I would say.

Randal: Is it doing things like fuzz testing and like just throwing random data at stuff to see if it breaks? Or is that more, this is more about identification and it requires a human to maybe be trying that kind of stuff?

Max: Yeah. Fa fast testing would be really awesome, . But then then, then such a a test would take Hs. and yeah, Amber. Amber already takes quite a long time and and loves your, your, all, all your course of your, your machine . And so, so we, we are working behind the scenes on different other projects, and one, one is a fa automated faster.

But this is not something that is directly into Ember because it, it it just will take too long to, to bring you the results and I, I think this, this need to be a dedicated project where you can decide which firmware you want to, to fasten after your, your initial firmware analysis process.

Mm hmm.

Randal: And I also, my friends would kill me if I didn't ask this, Why bash and not a more capable script writer?

Max: Bash is the most beautiful language No, no, no, not even close.

Max: just kidding. Wow. Bash is so practical. Think about the beginnings, you're chaining tools together. You're starting at the command line interface, you're chaining tools, and then you're Pasting it into a script.

This, this was the beginning of Ember. And so before we thought about Ember by itself, we had a thousand line bash script. And then we, we, we structured it a little bit and it was just working. We gave it a nice, a nice and shiny modular structure and with more lines of code and it was working.

And Well, everyone said that everything will break if you, if you hit the thousand line area of your bash script and No, no we, we are maintaining it till, till today and we have now tens of thousands of lines, lines of code and, and it works really smooth. You can't, you can't really believe it. If you're, if you're researching in the internet, you will read everywhere that, no, it's not possible.

You can't maintain it. You can't do this. It's possible. It's possible. It's possible.

Benedikt: I have to say for me, it was also the first major question I had when I came into the project. So yeah, because like, I always thought like that there might be an idea to, to switch once you want to modulate and put like, put it into.

Building blocks where you, we were always talking about framework and stuff that we introduced different languages and stuff, but like, I'm, I'm always astonished. Like it's working, like we do some stuff, but

Randal: I would be astonished. It's working. So, okay. That's enough bashing bash. I just wanted to get that in there.

It was like, It's like, okay, you know, I could, I could name another four letter language that would actually be pretty useful, but I'm going to stop that for now. We talked with him just

Jonathan: last week. That's Java, right?

Randal: That's not a scripting language. It is! Go, go and

Jonathan: listen to last week's episode, J Bag, they made Java a scripting language.

Randal: Oh, no, no, just no, no, that's just, that's just, that's just, that's just, that's just That's just wrong. I'm glad I didn't watch last week's show. I'm glad I missed it so far. I will read the transcript now though to see what you're actually talking about, but I don't think I'm gonna like it.

Max: Probably we need to think about rewriting AMBA in, in Java script in English.

Yeah, yeah, yeah, right. Well,

Randal: remember now, JavaScript is not Java, not even close. Those are other ends of the spectrum. Yeah. So, so what, what are your biggest concerns now with the project? What are you working on most intensely? And where do you see this going in the future? Mm hmm.

Max: From, from an AMMO perspective we, we can, we can see that the, the world screams regarding S bombs.

So everyone needs S bomb everyone thinks about S bomb nowadays and we, we, we ha, we already create an S bomb but there is a lot of room for improvement, I would say. And this is, this is, let's say, the, the one, one of the short term goals that which we're currently working on to improve our S Bomb capabilities so that let's say we can, can position EMBAR much better for, for the near future.

Yeah. And.

Jonathan: So I, I'm curious about the the, the answer to that same question, but on the, on the other side, is the other part of the program, EMBARC? Yeah. So I'm curious about that same question, but applied to embark, like what, what is being worked on there and what's coming. So

Benedikt: the fun, the funny thing about Embark is always that it's trying to incorporate everything that we're working on into one front end, let's say.

So if we're talking about the whole. vision we have with Ember, we always talk about from finding the right firmware from getting it to the end to the fuzzer we just talked about, like that might be the end goal. Once we had all the already known vulnerabilities and everything, and then we end up with a fuzzing framework or something, which Mike could talk about if he wanted to, but In the beginning, we would have something that is taking firmware from a device, which we have some projects going on.

Basically two of them, the, the one of them would be to get firmware from a device. And the other one is finding firmware on the internet. If you don't know where to look, basically like scraping and stuff. And the basic idea behind MBug would be to put all of that into one thing.

Benedikt: Oh, that's really cool.

That is basically the end goal. That's also, as you can imagine, a lot of work to integrate

Jonathan: those things. So one of the things that I think Michael mentioned is, is when you, when you give a piece of firmware to EMBA and to the Embark to the server, it really crunches on it. Like you said, you can just, you could fire your CPUs up for the whole weekend.

I'm, I'm curious, what, what are we doing with firmware that's requiring so many CPU cycles?

Benedikt: The thing Mike said about a weekend, yeah, that's, that's the new and the, the new shiny thing that's coming or that's, that's behind the curtain. We usually talked about weeks, months. If we, if we throw data sets of firmware in there to, to find. What's working, what's working with Ember, what is, what is it, is it finding, what is it not finding?

Nowadays it's, it's gotten pretty fast even though, and the thing with Ember is, is it's just multi threading and running modules, so multiple things in parallel, so it's taking basically everything. The, the machine has everything the CPU will give you. RAM wise depends a bit on where in the, in the Ember run it is, how much it will take up, but it really takes up everything you, you, you, you throw at it, but that's also the nice thing about bash.

Jonathan: Yeah. Yeah. But what, what is actually doing that is so CPU intensive, right? Are we just or excuse me, are we just un unzipping firmware and then doing comparisons looking for if there's CVEs out there, or are we actually like trying to run all of these binaries through Ghidra to do decompilation?

Like what, what, what is it that is getting done that's so CPU intensive? Both.

Benedikt: Okay. That's fair. Because you said unzipping is, is one, one part, right? And then decompiling is. Further down the line. Right? So, okay. So those are, we are,

Max: we are doing, doing more or less everything. It's, it starts with, with extraction of the f we, we are doing some, some binary analysis.

So which, which legacy functions are used in such binaries. We are decompiling these binaries. We are then using in Gira. For, for further analysis on the, on the extracted source code, we are using then Semgrep for static analysis. Regarding, regarding the Hot Topic SBOM, we are not just reading the DBN database.

We are also extracting or reverse the SBOM from the, on a binary level. So that if there is no packaging system on, on the device or on the firmware, we, we also get the SBOM out of it. Which most of the other tools do not do, and this is, this is quite, quite intense because we need to analyze every binary, every library and compare every all of this stuff against the quite huge I would say data set of possible version identifiers that we we, we, um.

Extracted manually and we built up manually and at the end we are comparing then these, these versions that we identified with the, with the CVE database, for example, to get an overview of how many, how many let's say known vulnerabilities are available for, for the, the identified. binaries and libraries.

If you think about the firmware, a small firmware of your home router has a few hundred files. If you think about the modern firmware it's getting bigger and bigger. So we have to deal with more we have seen things that they call embedded devices, but it is a full blown Ubuntu distribution out there.

And if you throw this static analysis mechanisms like decompiling every binary, like matching, I would say seven or 800 version identifiers against every binary. This is, this is highly resource intensive and and the good thing is the more CPU cores you have in your machine and boys is automatically using, let's say, or optimizing the next scan on the number of course you have so that.

It, it squeezes out your, your, your machine as much as possible.

Jonathan: Yeah. Yeah. Makes sense. All right. I know something that some vendors are starting to do. I guess they've been doing this for a long time and that is shipping encrypted firmwares. Is that something that there's, there's any yeah. Oh yeah.

We have to, we've had to deal with that. Yeah. What, what is the, what is the approach? Does, does EMBA have any tricks to try to get into those?

Max: I, I would say that we, we are using, we are, we are using on the one hand the, the classical extraction frameworks like BINWALK or UNBLOB. We also have introduced multiple dedicated extraction functionalities for such reasons where, where some some encrypted firmware was documented or the decryption was documented somewhere in the internet and, and we have implemented it directly into EnBAR.

If it's not documented and nothing, no, none of the known extraction frameworks can handle it, then at the end you need to go back to the hardware. You need to go back to identify a debugging interface. He sold a flash and usually, usually my task is then give the box to Benedict and say, Oh, I, I, I need to get access to the firmware, please.

Yeah.

Jonathan: Do you guys

Max: offer

Jonathan: like any, any sort of consulting or work services where someone could say, I need to break into this box. I need to get the firmware off of it. Can I ship it to you and you desolder it and extract the firmware for me? Is that sort of in your wheelhouse?

Max: Yeah, but just company internal.

No, no, no, no, no, no, no, no external consulting. We're just working Siemens Energy internally and testing the stuff that is relevant in this area. But not we are not offering such a service to external.

Jonathan: Sure. And I imagine trying to do that, you would run into all sort of legal questions, right?

About, I don't, I don't actually own this piece of hardware, or I don't have all the rights to this code. And then you go in and you go start decompiling it. You. There, there are, there are questions there, right? Randall, there are questions you run into when doing penetration tests.

Randal: Well, the letters DMCA immediately popped into my head.

So that was definitely going to apply here. Yeah, you don't want to be you don't want to, without the proper chain of authority, you definitely do not want to be doing pen testing at all. It's a, it can lead to. Unfortunate consequences. I'll just put it that

Jonathan: way. Yes, and for those that don't know you can go check out randall's wikipedia page and get a little more details about What happens when you don't have?

All of your I's dotted and your T's crossed and you do a little more penetration than your boss has thought you should have.

Jonathan: For a little while, it

Randal: looked like this.

Jonathan: Yeah. Yeah. Yeah. It's not, it's

Randal: not good.

Max: All right. So what about Not a comfortable position. No, no. No, it's definitely

Randal: not comfortable.

It hurts your, hurts your wrists. Let me tell you, I do know.

Jonathan: So what about virtualization? Is, do you have any sort of a a mechanism to take the, image that you've got now, you know, now that it's taken apart and run it in something like a virtualized machine to be able to do tests on it there?

Max: I feel that the, the original idea from AMBA was always we are, we're talking about firmware.

So this means that we, we need to, or we talk about different architectures. In, in firmware, you, you do not have just the, the, the, the, the X 86 architecture. You have some, some mips where you have arm architecture, you have power pc and all, all of this crazy, crazy architecture. So you, if, if you want to run something from this you always need some something that is called an emulator.

So that. That can, can run code from one architecture in your, in your, let's say, in your analysis machine, like your Kali Linux. So you have, have the guest which is the, let's say the binary from the firmware. Let's say BusyBox binary, you want to run a BusyBox binary for, which was compiled for a MIPS architecture on, on your Kali Linux, which you're using for analyzing the firmware, which is some, some x86 architecture.

And then you need an emulator like QEMU. And Amber is doing this. In, in multiple areas. So for example, we, we, we try to run every binary in something that is called user mode emulation. To, to drop output to the command line interface to collect the output and then to analyze the output if there are some version identifiers in it.

And this is one of the approaches that we are using. And the second one is, is something that is called system emulation. And in system emulation, we are trying to bring up the whole firmware with a, with a prepared Linux kernel. We're trying to boot it up. And to give it with a, with a few tricks, something like a, a, a kickstart so that hopefully the firmware is able to, to, to set up the services set up the, the, the network interface so that we can then, then interact with the firmware.

Jonathan: I, I am, I am kind of floored, actually. I expected the answer to that to be, to be, no, we're not doing any virtualization and you guys already, you're way ahead of me. It seems like a very kitchen sink sort of deal where you've just thrown everything in. It's, it's impressive. It's not even mentioning the

Benedikt: docker by itself.

Yeah, yeah, he, you, you skipped that one, Mike. No security thing with the docker. Yeah,

Max: we, we, we have skipped so many things till now,

Jonathan: but we can

Max: dig,

Jonathan: dig

Max: quite deep.

Jonathan: So maybe, maybe this is kind of what you're talking about there, but in doing things like emulating the firmware and all of that, I would, I would honestly have, have to have the consideration, like, What if there is something malicious baked into that firmware?

Are you giving that malicious code a chance to run on your infrastructure? Like, do you have any safeguards against You know, something nasty inside the firmware. Is it, is it all properly sandboxed? I guess is the way to ask that.

Benedikt: Properly? Wow. That's, that's, that's the good question. Yeah. Is it properly sandboxed?

I mean, it's been an ongoing theme, I think, for, for multiple years that we've tried to safeguard. running code that like, that's like malicious because the doc environment for within Ember needs permissions. So we're still trying to, to really get behind all the ways it could still get out. I think we've, we're doing a pretty decent job, but in the end it's, it's definitely like big disclaimer, do not run something if you know there's something.

In there maliciously targeting your host. I mean, the thing we always do is like we run it within virtual environments. It has a Docker environment. And then the thing actually running is inside a QEMU, but you know.

Jonathan: Layers, layers upon layers, that's something malicious would have to get out of. So I assume, I

Randal: know why it's slow now.

Max: That's the reason we need so many cores for every security layer.

Jonathan: That's, that's not a joke. Goodness. And I assume there's things you're doing inside of there that wouldn't work inside like a rootless pod, man. It's got, it's got to be Docker with root.

Max: Yeah, yeah. We're, we're doing a lot of let's say we're doing some things that we need, where we need root access.

We have thought about Portman now for years, but at the end of this, it is a time issue. We need to test everything. We need to test a lot of use cases, find out what, what is working. In, in this huge code base within Portman, what, what is failing and let's say every, every fail is probably some, at least some little research project on how can we, how important is it for Amber?

Or for, for the testers out there, can, can we just strip it out or can we bypass it somehow? And this, this costs quite a, a lot of time. And so we decided Amber is a research project from this perspective. We, we have multiple layers of, of protection, but at the end at the end, an attacker can analyze, can, can check out the code, can find the bugs and can exploit them if, if he, if he wants, I'm, I'm quite sure if it, it is possible in, in, at least in theory, it is possible to, to exploit it and, um, yeah, if, if you, if you get ephemera from an untrusted source, if you, if I send you ephemera, then you should definitely throw it into Amba,

Yeah.

Randal: It should just look for the string, amba vector or something. Yeah. . Yes. Not run the code if it finds that, oh, if only we're that easy.

Jonathan: Randall, you want to cover some of these that you've got stacked up?

Randal: Yeah, yeah. I was just going through my notes things I'd taken before I did this show. So you know, AI is a big thing. Everybody's saying AI. Is there any idea that maybe you're going to introduce some AI to this? Help fill some of the gaps in, or help extend what you're doing, or maybe try a more brute force approach.

You know, is there any way that large language models or even just neural nets can help you out with this stuff?

Benedikt: Let's say indirectly we had that. Okay. What is indirectly? We had that feature for a long time. I can tell you the exact date because I don't remember, but we have a whole section of modules that we introduced and one specific module was for an open AI connection.

So you're able to put in a API key and then Basically what Ember does is with decompiled code and especially if we've script languages, I think we have this option to ask the AI bot. If there's any vulnerabilities in certain code snippets that are previously identified by another tool.

Jonathan: Oh, interesting.

Benedikt: So there is also like a lot of expansion stuff that's possible with that. You can change questions, whatever, because. If you wanted to do that the results, because I've been testing it a bit, the results are, well, do they really give you more? That's the question, I feel like that's the real question.

Yeah, the false positives there, the false positives there are probably

Randal: pretty high. Is it, is it

Benedikt: telling you anything that other tools can't, like? Yeah. That's

Randal: I'm a little biased because I'm a Google developer expert, but I'd also suggest looking at Gemini's current products because they're really moving far forward and they're recently trained.

Which means that they're much more up to date than any of OpenAI's offerings at this point. So, you might take a look at that. Especially Gemini Pro 1. it. And it's really got some amazing results so far for the stuff I've been throwing at it. So just, just an idea there. Let's see. I also had a couple more questions.

Let's see, what else do I want to ask here? So, is this just about Linux based firmware, or are you planning on expanding it to other operating systems, other firmware types?

Max: Hmm. Is there

Randal: anything else?

Max: So that's, that's, that's, that's, that's the question. No. So, so in, in the area we, we are located. We are, we are using Amber.

We have to deal with a lot of Linux firmware. Nevertheless as I said, Our usual approach is to throw a firmware before of a, of a manual penetration test into AMBA. And AMBA gives us a few days later the, the, the results. We, we also have sometimes some UEFI firmware. We have sometimes some RTOS such a VX works or something like this.

And we want to get the, the maximum out of it. Although we are not the experts in this area. So, we, we, we do not want to, let's say, we, we do not want that Amber is crashing because there is no Linux in it. Then Amber should, should, should give us the information that there's a VX works. And we are doing a lot of or we have a lot of modules.

that are, let's say, file system, system independent, so you can also use it just on a binary blob, for example, to, to extract some key material or to find some, some, some interesting hashes. And then Amber can do this also on, on a, on a, on a non Linux operating system. And additionally, in, in the future.

field of, of UEFI, so, so we, we do not deal a lot with UEFI analysis, but there, there is a company called Binary and they have released an UEFI security analyzer as open source. So we have just put their complete open source scanner included it into Ember. Wrapped it around an ammo module.

And if we, as soon as we detect any UEFI firmware then we fire up this vulnerability scanner now. To, and so we are also able to provide some, some kind of value also non Linux operating systems.

Randal: Cool. Cool. And one last question. I'm gonna throw it back to Jonathan. So results, what's the most unusual thing you've discovered with this or just what are some of the practical outcomes of this that make it worth your time?

Randal: you must have discovered something with this or you this. They're now,

Jonathan: they're now having to do the, the question of, of the things we found. What are we allowed to talk about? . Oh, oh, sorry. I didn't realize that would put you at risk. So, so,

Max: so, so like we, we can say we, we are talking quite generic, so nobody knows what, what we are talking about, so, okay.

I think they, it's, it's very often really interesting if you're, if you're analyzing established security products that are established there for years now and you're, you're analyzing them and you can see that the, the, the components that they are using are, are from, from the age of dinosaurs.

And that they're, that they're not security components and they're not updating the, the, the product anymore. Or the, the base product that they're operating on, on the operating system level. And they're, they're just managing the risk, let's say this way. And this is, this is really interesting because often these are established vendors, established products, and then you can see that not only D Link is using a kernel 2.

6 dot something, also very established security company is doing the same. Wow. Yeah. I think there, what, what was it that there was, I cannot remember exactly, there was one one, one company or one product. That was destroyed somewhere in the internet. I don't remember it now, but, but, but,

Randal: but if I could summarize what you're saying is that some people who should know better don't always demonstrate that they do.

Yes. And, and with, with,

Max: with automated firmware analysis you get a quite good overview without any manual work. And then, you know, that's usually the, the, the thing where we are using Ember a lot. Ember is doing the automated work and we can see, okay, oh my God, this product is looking really fishy.

So we, we, we have 10, 10, 10 products. And then we know, okay, we start with. This one, because this looks fishy.

Randal: So it's like, when I'm looking at a piece of code that is operating on a live website, that's written in PHP and has a very obvious SQL injection attack. And I go, how long has this code been in here and this company in particular should know better to run PHP code.

That looks like this. Y'all. It's, it's, it's, it's a sad experience. It requires some warning on my part, and then I move on and go, well, I'll probably write code exactly like that someday. Yeah.

Max: And, and, and usually these are the windows that are shipping encrypted firmware updates so that nobody can see the, the, the PHP code.

It's encrypted.

Randal: We don't have to care about it. They don't want you to see how bad it is, that's why. They're just shipping junk under a golden label. Yeah,

Jonathan: it's sad how much that happens. Alright, so, I've got to ask, why open source? Right, like you guys apparently use this at Siemens or Siemens Energy. It's it's used for it sounds like for a lot of internal use What was the what was the conversation like saying?

Hey, we should open source this and release it to the world

Benedikt: I mean it started as a project where you're like, okay, I have an issue and there's no tool for it I really feel like that's a classic open source thing, right? You you're right. You're starting free time. You you start with a An idea, and then you write something that you actually need for your work and then you're like, okay, I feel like since there's nothing out there, why not just open source it, like put it out there.

And that's, that's basically what happened with Emma making it like a security tool for everyone. That's also a big part of the cyber security idea. I feel like that's if you write a security tool, you put it where everyone can use it.

Jonathan: I like it. I know, I know some corporations, they, they get real antsy about releasing internal tools as open source.

Was EMBA started was it started in, in your spare time or was it sort of kind of official? It's like a company project.

Max: I would say combination. So we we we started during penetration test and then every every you're fascinated you want to do everything and then You're running into time constraints and then you you can't sleep.

So you start coding in the night and Is it no work time? No, it's not work time because you have already Your, your, your, your hours together, so it's free time. So, so pro probably most of the guys out or the open source developers out there can ha have a little bit the same story. Yes, they're fascinated.

So where, where at the beginning you can say where is company time, where's free time and everything swims together. You want to solve the problems and yeah, I think it's, it's, it's a mixture of both. We, we have the time in the company, but we are, we are also spending quite a decent amount of time during our, our free time.

Jonathan: Yes. I know that feeling waking up at two in the morning. I know what's wrong. Then it's like, do I try to put myself back to sleep or do I go in there and fix it?

Max: And then you need to go up because you can't sleep anymore and you need to fix it. And then you realize, Oh I fixed this issue, but there's the next one.

So I don't need to go to bed anymore.

Randal: I think what's worse than either of those is waking up, realizing you have the solution to a problem and not writing it down and forget. That's really bad.

Max: That's true. Yeah.

Randal: I, I've done that. That's why I know it's not a fun thing. Yeah. I try to get at least something written down when I have an idea in the middle of the night.

So I at least have it for the next day.

Jonathan: Are there any, are there any CVEs out there that have a special thank you to EMBA? Do you have like a list of you know, this one and this one and this one were found because people were using EMBA?

Max: So, so We have an internal list of our CBEs that were fixed that we, we reported to other vendors and that were fixed.

Mm-Hmm. , but there's nothing out there in the, in the public. No. Okay.

Jonathan: That would be, that would actually be an interesting project to just have a repository. Now, now I understand like some of the things you find that you just can't, right? Like I get that there are things that will get discovered internally reported internally and may never be assigned a CVE number.

Like that's fine. That's just how the world works. But it would be very interesting to have a repository where you say, Hey, security researchers, if you find something, if you find a CVE through EMBA. Shoot us a note here and they just have like a high score list somewhere on the Ember website. I think that would be a lot of fun.

Oh yeah.

Max: So, so, so we, we, we have a collection of all of the papers and blog posts and mentioning of Ember. So If someone sends us the CVEs that they have found with the help of Amber, then they will definitely get a nice and shiny place there. Yeah.

Max: So, so we, we are prepared for these reports.

Awesome.

Jonathan: So one of the things, so I, I write about security and one of the things that I try to encourage people like just that are interested in it is the barrier to entry for finding CVEs for finding security problems and getting them fixed. Okay. It's actually a lot lower than most people think it is, right?

Like it's, I, I just, now I didn't get assigned to CVE, but I got paid a bug bounty for literally just running a trace route and then running a port scan back when I had Starlink Internet, and they had set, they had set a server up and it. did not, it did not have a proper firewall if you were coming from inside the Starlink ISP.

And I got paid several thousand dollars for reporting that. And this is one of the things that I, I really like to encourage people on, is like, go looking for things, and if something seems weird. So all of that said, what does the process look like for setting up EMBA and EMBARQ? How difficult is it to get started and then say, feed the latest firmware for your, from your own home router into it?

There's obviously, there's going to be problems in home router software. How difficult is it to get started and start finding them?

Benedikt: I would say very easy, but maybe I'm biased here. Of course the easiest way is for, for us just tell you, okay, there's a Kali ISO run a VM on your laptop and then run it.

Just do the git pull git clone and to an installer which is very easy takes a bit of I don't know a few minutes for sure to download and install. The barrier of entry is basically Now I would say because you can also do it on bare metal You can install kali on basically anything I would say the only barrier I would see is Minimum system requirements because Ember is, as we talked about, a bit of beast.

So it has some minimum requirements where you, you can't, like, it won't run nicely below that.

Jonathan: Yeah. So, and so is, is Cali kind of the preferred place to set it up and start using it?

Benedikt: Well for Embar for sure. I feel like Mike and I, we always are on Cali. Mm-Hmm. Em, embark is designed for Ubuntu. Okay. Because if you, if you're talking about Linux environments and servers most people install Ubuntu, right? So that's, that's what, what is it? It's intended for. So EMBA would work on two yeah, well two Ubuntu versions and Ali, the current Ali.

So there's options.

Jonathan: Alright. We are I just saw at the time we are rapidly running out of time and I, I wanna make sure and ask about community. Like, so what, what is the, what is the community size? Like, do you guys get contributions? Have you had people from? outside of your company and outside of the project come by and say, Hey, I thought it would be cool if EMBA could do this.

Here's, here's the patch to make it happen. Or, you know, bug reports. Is that something that is happening or is it, is it kind of not public enough yet?

Max: So, so I think, I think we're getting more and more bug reports during the, let's say during the last year, we are getting more and more insights on how people are using EMBA.

How people are. Talking and writing about it. People are also reporting their bugs and their wishes. We, we had a, a, a few contributors that there, which are directly fix have started fixing the bugs but really just a few. So the, the the, the, the community is growing, but let's say the community that is interested in also helping is growing very slowly.

But the, the more the community is growing I'm, I'm sure that then also the, the, the helping hands will, will come more and more. Sure.

Jonathan: So if somebody wants to learn more about EMBA or EMBARQ, where do they go to learn more? And then if they want to talk to you guys, where is the best place to do that?

Right

Randal: here.

Jonathan: So like, what's, what's the, what's the website? What's the GitHub URL? Is there a Discord server? That sort of thing.

Benedikt: No Discord yet, no. Not that I'm aware of. That's something we should talk about, Mike. Absolutely. Thinking about that, yeah.

Max: cu Currently the easiest way is going, going to the GitHub project. If, if you have ideas, open an issue, open a discussion.

If you, or if you're writing about Amber or showing Amber or something around Amber somewhere, drop us a note somewhere at GitHub or GI or X or LinkedIn. Mastodon the usual social networks. Mm-Hmm. . We, we have there an, an dedicated Ember account, which is called Secure Firmware. And we, we are happy to, to, to publish also your, your papers that I'm mentioning or your talks that I'm mentioning Ember there.

And If, if we need to talk in, in person, like we did it and drop us a note somewhere over there and we find a way to, to get in contact. Yeah.

Benedikt: And I think, Mike, you're at the next, no, conference, the big one in Oh, yeah, yeah,

Max: we, we, we have plans going to, to Black Hat Middle East now. Ah. I think in the end of November.

Okay. So if someone is there, then drop, again, drop us a note and we, we have there a, a, a demonstration of EMBA at Arsenal, and then just come to our booth and we can, we can talk about EMBA and Fermi analysis and all the rest of the world.

Jonathan: Yeah, very cool. All right. We are out of time i've got to ask each of you the final two questions.

It is a requirement. We get in trouble if we don't do it we'll start with michael. What's your favorite text editor and scripting language?

Max: So so scripting language is is definitely bash. I I've now written 30 000 lines of coding bash. So it it needs to be my favorite scripting language. Yeah, and I always exited WIM after working on AMBA, so this is my favorite text editor.

I'm not a master of WIM, but I can exit it. You know how to get

Jonathan: out. It's fine for me.

Max: Yes. Save and exit, not just exit. Yeah,

Randal: hopefully. All right. If you're not saving it, it starts sticking.

Benedikt: All right. And Benedikt, same two questions. Yeah. For me, it's somewhere between Python and Bash. More fluent in Python though, I should say that.

And editor wise, I would say, yeah, not, not the biggest fan of him. I learned it. Normally I stick with Nano. That's fair.

Jonathan: That's fair. Alright. Thank you guys for being here. I wish we had more time. We'll have to bring you back maybe after, maybe after Blackhat. We'll bring you guys back on and talk about what's changed and get in all the questions we didn't get to today.

But thank you both so much. Thank you for having

Max: us. Have a good day.

Jonathan: Yup. Yup.

Max: And

Jonathan: bye bye.

Max: See

Jonathan: you. Alright. What do you think, Randall? You gonna go dig the firmware out of your router now and take a look and see if you can find some fresh CVEs?

Randal: Well, it's not my router, which is why I'm on Wi Fi. So It's I live in the basement of a, of a two main story house and they have the router upstairs.

And so that's why I don't have, which is why we were in troubles before the show. Yeah. Getting me on the screen of why I dropped it. After we're done taping, I've got some advice for you about how to set up the next show so that it's a little better bandwidth wise. So this was interesting in that I haven't spent a lot of time thinking about the firmware that's in all the devices that are now being used in my life.

And now it makes me more scared. Great. But at least there's some sort of diagnostic tool that at least somebody can run against some release of maybe something that I will use or eventually use or be kept from using because they found bad stuff in it. I like your open source question. Cause it's like that, unless they're getting a lot of community Contributions.

This isn't, this doesn't really pay off as open source. Unless most of this stuff was already open source and they're just bringing it together and kind of from a potluck perspective where everybody's bringing food and everybody takes food. Home, it sort of makes sense for it to be open source. So I like, I like it from that perspective.

It was a little out of my wheelhouse, so I didn't get a lot of questions. So

Jonathan: Randall, what you need to do, if you decide to start doing that, start working with this at all, is write yourself a custom module to see is any of your code in the firmware and just like make some sort of trumpet fanfare that plays automatically when it finds, you know, a Swartchian transform or whatever that, that is actually in there.

All right.

Randal: Right. Well, it's like, you know, the usual editor question, you know, for years, I said Emacs to that because there's a part of my code in every copy of GNU Emacs. So that kind of thing. And, and Yeah, Schwarzschild transforms show up everywhere. It's kind of crazy how that how that one's gone kind of spiraling out.

So, yeah yeah, cool. No, I was, it was a, it was a good show. The guests were knowledgeable and the project is worthwhile and in progress. Yep, absolutely. All right. Do you have anything you want to plug? Ah, yes. So so this is now a new addition to my semi regular schedule as often as Jonathan will let me back.

And, and I've opened for the day. Cause so typically my, my Tuesdays are pretty free. So I think it'll probably work out for at least as often as Jonathan's willing to put up with me. But every Wednesday I tape a live stream show as well. On Dart and Floss, which are my, not Dart and Floss, Dart and Flutter.

Ah, too many FL words. Dart and Flutter. I am a Google developer expert in the Dart and Flutter arena. I'm all in on Dart and Flutter. So, yes, although I was kind of hinting at Perl earlier in the show, I really, I personally haven't written more than 10 lines of Perl code probably in the last year. And but I've written thousands of lines of Dart code and Flutter code and, and things like that.

And so every Wednesday we do what we call Hump Day Q& A, Ask Me Anything, where we get a few experts together. And for some reason they let me on there. And I have no idea why. And we take live questions over YouTube. So if you go to YouTube and look for Flutter community and Flutter is anything vaguely interesting to you, please check that show out.

We really enjoy the live questions. We get some good stuff doing, we also do some live coding. So it's been a lot of fun doing that week for week for the last, I think, two, three years. So it's almost been the same burden that I used to have weekly, but now delivering it into the dart and flutter arena and the problem with the, not the problem, but the, one of the things to know about the hump day Q and a is it runs two or three or four hours long.

So. At about the third hour, I'm telling Simon, who's now madly typing in, trying to solve that last problem in his live coding. I say, Hey, we're starting the fourth hour there, Simon. And he'll finally go, Oh, I should probably wrap this up then. Yeah. Why don't you? Why don't you? We're down to 22 people watching, so we should probably stop.

Oh, that's great. Fun show. It's a great show. I love it. I'm, I look forward to it every week. I'm just making notes now about what we're trying to do tomorrow. But check that out. That's probably the biggest place to see me. I also have a dart and flutter YouTube channel, which I try to contribute to you on a regular basis.

So. You want to check that out. Just, just Google Randall Schwartz, Dart and Flutter. You'll find all sorts of good stuff about me and my new career.

Jonathan: Yeah. Excellent. Thank you, sir, for being here. It is good to have you back and we will make sure. We'll make sure and get Randall in the rotating cadre of co hosts.

So we'll see him again here in a few weeks and look forward to that. If you want to find my stuff, of course, Hackaday. We appreciate them being the new, the new home of Floss Weekly. You're not really new anymore. I've been here for almost a year. I've also got the security column that goes live every Friday morning and then still over on twit We've got the untitled Linux show, which is a lot of fun And if you're into Linux, which I imagine most of our listeners are you should go and check that out, too We appreciate everybody being here those that catch us live and on the download and we will see you next week on Floss Weekly

This week Jonathan and Randal chat with Michael and Benedikt about Emba, the firmware analysis tool that packs in a bunch of features and tools. It's got virtualization tricks, binary version detection and even more!

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Jeff joins me and we talk with Max Anderson about JBang. It's a little utility to make Java easier to use and run it as a scripting language if you really want to. It's a lot of fun, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 801, recorded Wednesday, September 17th.

It's not your parents Java anymore.

It's time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today, it's going to be fun. It's going to be interesting. It's going to be Java. Which That's okay. We mostly like Java here. It is not just me. Of course I have, I have Mr. Jeff Massey, the one, the only Mr.

The, the other, Mr. Lennox here with us to talk about J Bang, which is an interesting project all about, from what I could tell, it's using Java as a system scripting language, which is sort of interesting. What, what do you know about this, Jeff?

Jeff: Not a lot. When you, when you first got a hold of me, I thought for a moment you were talking about JavaScript.

Well, it sort of

Jonathan: is. It's Java space script.

Jeff: Yes, not one word. So that's where, I was a little confused at first. So I'm, I, I'm not a real I don't have a lot of experience in Java. So very, very little, but I'm interested to hear about this and kind of the direction they're taking it.

Jonathan: Yeah. Now we talked before the show, it's like, well, Couldn't you just do this with like an alias to the Java binary?

And so, you know, JBang would just then be an alias to Java and that's the show. Thanks for coming, everybody. We'll see you next week. Apparently there's something a little more to it than that. It's a little more complicated. There's more wrinkles, there's more hair on that problem. So, looking forward to learning.

The level's always in the details. Yeah, always, always. Looking forward to learning about that. Let's not let's not take any more time to bring him on. We have Max Anderson is the guest today, the, I believe, the creator of J Bang. Welcome, sir! Welcome. Well, thanks for having me. Yeah. Now, creator Jbe, right?

You're,

Max: you're the man. Yes. I created it after being a year away on a sabbatical, not doing anything not doing anything, work for latest, so to speak. Right. And yeah, and I, I'm happy to be here and, and be in a place where everyone loves Java and wants more Java and all that

Jonathan: kind

Max: of

Jonathan: thing. Java was I've, I've told this on the show before, but John Java was one of my early programming languages.

And so some of the like difficulties and learning how to program and like, what do you mean a pointer error? And why I didn't even think Java had pointers. Why is the log telling me about, so some of that, like, you know, I just, I have this sort of deep seated annoyance with Java because I was programming it when I didn't know what I was doing.

Max: Yes, I see that repeatedly again. People have, well, most people have learned or touched Java, but they mainly have touched it at a time either in their career where they were not, like, that was not a good, it was not a good learning language in the early stages. But it got super popular because it could run anywhere, right?

It could, you could run both on Linux and Windows and eventually Mac and that kind of thing. And phones and, I mean, for a while there were DVD players and everywhere, yeah.

Jeff: Everywhere.

Max: Yeah, it was, whatever devices were there, right? So so I worked in Java for, I've done professional open source 20 plus years.

Mm hmm. I've my day started at Hibernate and, and, and Persistence Solution, that kind of thing. And then I started doing tooling. And then I, I had, um, what's it called? Well, 10 years, 15 years of work and came back. Well, then when containers came around, Java had a problem because it's too heavy and, and, and, and and I was tasked on doing some other tooling in go land and JavaScript.

And I got to touch on that. But then I, I took a break for a year. And I promised myself I'm not going to touch any programming for the least three or four months. And on the first day I break my ankle and have to be not moving for another days. I decided not to touch Java, but go into Python instead.

I knew Python from like 20 years old. And I had fun with that. And then when I came back to actually work on a product called Quarkers, which can make Java work with containers, I was like being super or realizing how complicated Java is like in the minds of people. But when you refresh yourself on what's actually happening in the last X number of years, You realize, hey, there's no reason why it should be hard to use Java.

And I then, as a way to update my knowledge about Java, sat down and said, hey, let me try and make like, there's this the product Quarkus was using in their release engineering, release scripts, they're using Kscript, which is, it's Kotlin scripted. And Kotlin is considered a more lightweight language, but I was like, why are we required to install Kotlin on top of this Java when the script is just doing some file automation stuff, right?

So I took the DSK script head and applied it to Java and that's where the scripting part comes from in JBang. And then I realized, hey, I want to have, The same experience I want to be able to, because the reason why I like Java is all the tooling around it, like the debuggers, the IDE, the content assist, the refactoring.

But the problem was that when you try and do that in a scripting way with JBang, All that tooling falls apart. And then it becomes at the equal terms as Python and Ruby or Node or whatever. And then Java doesn't hold up. Like, then those languages are nicer. But I was like, no, no, no, like we have a massive, like Java has a massive ecosystem, like the whole Maven central auto artifact, you can download and use and do whatever you want.

And and I said, Hey, let's, let's do this. And then I the, the thing was in Java, I forgot like 10 or 11, the add support for running. without compiling it. But the problem was that to, and you could even use it as a script, you know, like have a what's it called? She bang in, in the top, like what, what pound slash slash dash run something.

but any IDE you do that in will then see that as a syntech error because that command is not part of the job eco system. So no, IDE could work.

Jeff: Mm-Hmm, . And

Max: I was like, oh no, we need to find a way. So that's when I realized there was a trick in go. I think I saw it first, that if you add two, four slices bash and seashell and a few other shells, not fish, shell will treat that as shell commands.

So you can then run it and pipe in the file and Java will just treat it as a command sorry, a comment. And that means you can run any Java file and it'll work in the ID. You can come to this, it works. And that's like, Hey, great. Let me do that with the Java eight, because at the time I was using Java eight, but you couldn't so fast forward, I made it.

So there's a J bank command that you can call that can run Java source file. And. Compile it and then run it. And the advantage over the default job is a, it will cast the result. So therefore it doesn't have to do the compilers already there

Jeff: and

Max: only do the compiler into the file changes. And then the, the one that's the kicker is it can get dependencies that it's just in the file.

So at the top of the file, you can add like slash that steps and then the coordinate for artifact. So and that's, once you have that suddenly you can do things and it works in ID like in the early days I was generating files for ID, but now a few days later we have support in Eclipse.

Intel J and VS code, especially VS code. So now my favorite text editor is, is, is VS code with a J bang plugin in it because I can just be writing in my, my own language. And then on top of that, we started going low. Oh, Because that's the other big thing is, oh, you need to start writing code without having to download an IDE, et cetera.

You need to get Java, but which Java version are you going to run? And so then I added to JBang that it could download, or Taco, who I'm doing it with, added support for downloading. Any, the, the, the version of Java you're there. So now we had a full setup where you install giving which is a one liner install, and then you just create a Java file and you can add dependencies and you can build it.

And you can even, there's even a JBank edit that will actually, if you use your own IDE or download one for you and it's configured. And once we had all those things in place, we suddenly had an ecosystem, a whole setup that anyone can run on any of the three major platforms out there. And then suddenly I realized, hey, any student, any, any experiment I want to do, I can do in seconds compared to, I'll also set up as it was before.

Jonathan: It sounds like you're technically making polyglot files here that are interpreted one way by bash and interpreted another way by the IDE and by Java itself.

Max: That is true. And, and the fun part it's also a poly, Poly, poly, polyOS thing, because the first slash is bash recognizing, Hey, this is a bash command in the start.

The second slash is the Java comment. So it works. But the fun thing is that works on Windows, sorry, Linux and Mac. But if you do it on Windows. It gets interpreted as, I forgot what it is, but it's a UNC

Jeff: path,

Max: so it fails. But if you have a third one, everything's fine. It works in all three. So it's, it's, it's a, it's not a feat, it's a bug in all three that makes it work.

But mind you, you don't need that line. That is literally just for the feature of being able to do Like run the Java file directly to be able to do a dot slash on it. Right. Yeah. Yes. A lot of run the so, so that's, that's the, yeah.

Jeff: Sorry. I was going to say, you know, and I think you, you kind of touched on this, but you know, why did, why did you choose Java?

Was it strictly just because of the infrastructure, the existing infrastructure versus not going, Hey, maybe I want to try to work on, you know, Python or Rust or some other language, you know, So

Max: that was the thing. It's like my professional work life has been in the Java world. And I, I am proficient in the others, but I always, I was before I did my sabbatical, I was like, why doesn't people do that?

Why don't they just use Java more? Like it's so easy to use. Right. But I realized that my 15 years of working, I. I got immune to, oh, you have to find a version of a dedicated download. Oh, you have to find IDE to download. You have to find X, Y, Z. Learn a Maven tool or Gradle tool. And I hadn't realized. It was just everyone in my, like, bubble knew that it's like, like, and people found what's that called?

Like when, when, when you're a group of people who've gone through the same pain and now you're all kind of good and you don't see all the problems, right? And you feel like privileged to you, you, you've got here and then go like, Hey, why is this student that never used program before surprised to, to, to, to this and the, so one of the things you can do with, with well, so, so main thing is.

I like Java. Like, I'm, it's what I'm, the language I, I, I like to do. I also like Python. I like JavaScript. I'm not a fan of, but there's a lot of languages I like, but the whole ecosystem of Java, I just, the whole tooling, the compile, like there's a lot of stuff we can do that none of the others can, can, can do.

But the, the, the So that's why I was like, I was like, when I went on sabbatical, I was like, how hard can this be? Why is it? Is it so hard when I came back because then I had forgotten all these things and I was just Realizing how many steps I have to do so, coincidentally also at the time my son was about Seven or eight six seven.

So he got into minecraft at that time And mindstorms and lego stuff and those two actually uses java in in a setup and i'm like, okay Let me go and show programming with Java to my kid. And I heard about all these kids and, and, and, and teenagers using writing plugins for Java. And I thought, then it can't be hard.

Then there must be a guide in Minecraft land that is simple to get started. And I was surprised that people are even doing minecraft plugins because any product founder had like five or six pages of setting up java To get started. And I realized okay, so they they're going through all this pain Just because, hey, all of their friends are liking Minecraft.

So therefore, that's why they get there. But I was just like, why? And then I realized with the JVang stuff I had, I could literally write, I could, it was made, I made it trivial to write Minecraft plugins in, in, in JVang. Because now it's just a single file and you just have a dependency on the Minecraft whatever Minecraft plugins that we use.

And off you go. Stuff like Again, people don't, well, you guys, you don't know because you, you're not a Java head, so what do you call them? But one of the things I always find insane, like when you do a debug, a Java product. It is literally like a 80 characters. You have to type in of Java does agent connect server equals yes to whatever port.

And I was like, why it's the same. It's it's, it was fine when we 20 years ago, add was a feature. We didn't know how it should look like, but it's just stayed around. And in day bang, that is. D debug and it behind the scenes just do the, the, the, that connection string. And I just chipped along and anything I could find that was like this com, like unnecessary complexity that has organic groan, I just kind of chipped away.

And because I just use a standard Java tooling, it works on anything like, so it it runs on Java eight and artworks. So I could even make it work on Java six and older. I just, no one ever asked for it. So I didn't want to incur the pain, but technically it could. And yeah, no, no. So, so, so Java was just because, you know, that's the thing I like.

I might be weird, but, and there's also a whole enterprise out there that uses Java. And I was tired of hearing this. Oh, when you're using IOT, you have to use Rust or, or Python or, oh, you have, if you do front end something you have to use JavaScript. Oh, and because everyone now do use JavaScript front end, we have to use JavaScript at the backend too.

I have nothing against it, but it's just this like, Hey, like Java can actually do these things too. So don't, don't fall in. I have the same thing with the whole AI movement that everyone thinks, Oh, to use AI, you have to use Python. And I'm like, Hey so this is one, my gimmicky you can actually run J bank from Python now.

So if you have a Jupiter, if you have a Jupiter notebook somewhere Jupiter notebook environment you can do pip install J bank and it will actually, you don't have to do anything. It will set up Java in that environment. And now you can, you can use Java from any. Of those free services like cloud notebooks and others and get access dependencies.

And people don't believe me when I say so, but it is actually doable. So now I claim that Java is the most portable environment anywhere. And with JBang, it's the, it's the easiest thing to set up. Cause I don't know if you actually tried, have you tried to actually set up a Python environment on a Windows machine?

Jonathan: No. Oh, no. That is, that is horrible.

Max: Yeah, because this is the thing like just like the java guys are like in a mindset Hey, you just do these seven steps and you're fine python and node. js is is default installed on mac and linux But as a whole ecosystem windows, which is you know, i'm sad to say but it's the most installed desktop operating system out there Yeah, we're running python is not easy Funnily enough, JBang is one line and you have a full Java environment available to you and it just works.

That's actually really cool. I did think it was funny

Jeff: when you said Java was really easy because I'm like, I don't know if I've heard anybody utter that phrase before. You know, I've always heard there's a learning curve, but so, you know, I totally get who you're surrounded by and what you're in and, you know, yeah.

So, but officially. So I, so I can make sure I have this down. What versions of Java are supported? I know you said you weren't going backwards from. So,

Max: It JPEG itself is compiled and with Java eight. So it can run on any Java eight, um, VM and upwards.

Jeff: Okay.

Max: So, because what J Bang does is it like, it's trivially, it's the simplest product I ever made.

It literally, it's just a little J, it's a Java, it's a small Java app. And it, it just takes input and figures out, Oh, is this a Java file? Oh, I need compile it first. And then I create a Java C command line. And then I. I, I build the classes and then I jar it up and have a jar and then I run the jar. And if it's a, if it's a jar file, I just run the jar.

And if it's a Maven coordinate. I go fetch the Maven dependency. If it's a Kotlin file, I, I run the, the Kotlin compiler. If it's groovy, I do the crude compiler. And in that sense, J bank actually goes beyond just Java, but any JVM based language is, is in theory possible to, to, to in that way run. And because WK team generally are good at having backwards compatible command lines or four compatible ones.

It, it has been reliably working on any version has come out in the last four years. And the cool thing is J Bang knows which version of Java the user is asking for. So therefore, if for some reason, let's say Java 25 is going to break something, I can adjust the, the, the, the generation of the command line stuff.

And then it would just kind of work. So that, that's the way that this, this magically works.

Jeff: Oh, that makes it nice then that, so, okay. I I've got my Java version eight plus whatever. Now to make this a little easier, these complexity reductions that you talk about, are they. Have they made their way upstream?

I mean, are you making life easier for me?

Max: I'm actually, so I work in Red Hat and my main job is actually to work with some of the compiler guys and OpenDK team. And other things and I've been pointing them to some of this stuff, but don't get K team is, is a very like, like it's a really complicated machine but it's like really, really efficient.

Like it's a really, like it's, it's engineered really, really well, but they also use everywhere. Right. So it's kind of like, I think you had an episode about the core was it the core lips, the core tool about how backwards compatible that I think has to be open your case is in that ballpark, but they also want to tweak things up, but they are so they're very conservative, right?

So even if I propose something now, it's going to take. Literally years before it's there. So I haven't done a lot of it, but one thing I've Not sure if it was coincidentally or not But a year after I made the big splash with jbang and and created it And showed, Hey, you can run J bang. And another thing I can also run JCL, right?

So JCL is, is a shell is like this tool in Java where you just run. It's like a, what's called rebel, like one line at a time. I can run those scripts too. And I showed that, Hey, it's you. The only thing you need to have being run, it's just a single file. And. You didn't even have to have a class in around like the whole like public class static void main thing you, you, you don't have to, to, to do.

And then a year or two after to dedicate team has now come out with that, I think it's Java 22. There's a preview support for what I call a naked and naked main. So that's you, the simplest Java thing you do, you can do today. Now is. Void main system up from like, there's no, there's no arcs. There's no class.

There's no imports. They, they actually, that's the one thing that, that, well, it's a thing that they started to, to, to put in. And some of the flags I've been doing like the debug one I've proposed at least to the internal Red Hat team. And I'm actually meeting them this week where I'm now here in Zurich.

And I'm, I'm, I'm gonna. Talk to 'em. So, but again, , this, this, this is so hard to dis do because people are so, like, one thing is that they're used to it, but also just there's so many infrastructure that's Mm-Hmm. , like, if you start, it's, I have the, the advantage that this is not in Java, so therefore I can, I, I can, I can change things a bit.

But I, I do, I, I at least. Find priority. And I think I might have accelerated some of those decisions to simplify Java. And yeah, so I, yeah, I'm trying, but it's, it's going to take a long time. When

Jonathan: you, when you measure, when you measure your like install base in the. Billions with a B you want to be very careful about making changes that could break things.

Yes So you mentioned you you mentioned naked mains i'm curious with j bang do you do you support implied mains? Or when I go to write a j bang script, do I still have to define a main? so so,

Max: So this, so yes you do. And I'll explain what, but, but no, you don't. So there's, so one of the key things I wanted to, was to make sure that the IDEs keep working.

So I could technically, I could make JBang go look in the sources and see, Oh, this is just a naked main. Let me. Behind the scenes generate a a, a, a wrapper around it and, and behind the scenes do something. But if I do that, that job file would not work in any id 'cause none of the IDs currently supports that notion.

Right. So I did not do that. But what I did do, do what I did do is the JS shell support. So JS Shell is this tool that's been in Java since Java nine, I think. Where you can literally just run any kind of Java, like it's, it's a line based execution. And, but the weird, well, I'm not, I'm not saying weird, but a limitation of JShell is, JShell doesn't, are not able to take arguments.

It doesn't handle dependencies. And that's because JCL was built for tinkering. Like, it was like, hey, let me play around with something. Not for use as a main execution engine. But I realized, hey, with JBang, I have a way to run these. So they are still JCL scripts. But I actually handle the command line arguments and parse things in.

And then, so yes, you can do mainless with J Bang back to nine. It is. But it comes with the price that JSON comes with it, which is, it's a slower execution. It's not as fast as everything else. So, so yeah, so the answer is yes and no. So

Jonathan: this ties into something else I was going to ask about are you doing compiling down to byte code?

Like with Python, oftentimes when you run a script, you'll end up with a pyc file hanging around and it does that to speed up compilation. So it sounds like sometimes you compile down to byte code and sometimes you don't.

Max: No, as a, what's it called, conceptually it's the same thing I do. Right. So Python adds the, has these Pisces files.

It says, Hey, this is a compile time. And if, if the, if the file, the timestamp matches the file we will we will use that the Pisces and I do similar thing here. I just, I have a, there's a medical folder and your home directory called dot J bang, where I have a cache. And in there, there's a, there's a, there's traces of what you've been doing.

Right. And and that's the thing. So this is the, the, the, the, the, once people, people have a hard time grokking that it's that easy. But for example the main list, the main list, sorry, a naked main is available in Java 22 and people go like, okay, I can't use that. That takes too long because if you use the normal install, you be complicated.

But with JBang, it's just JBang Java 22. And all this stuff will just be, I'll download a JDK and it'll just run for you. And the same thing is what I do with the, the compilers, the bytecode stuff. I will generate bytecode well, compiler classes. But I'm using the standard. Java C tools, right? So I'm not, there's nothing J Bang doesn't do anything that Java itself can't do.

Right. So, so, so it, I'm very confident that it will be portable for a very long time. As long as the OpenIDK team stays portable, J Bang will,

Jonathan: will,

Max: will too.

Jonathan: What, what about, we talked a little bit about Java versions. What happens if somebody says, here's my J Bang script and it calls for Java. 11 and Java 11 is not installed anywhere on the system.

Do you, do you deal with that or do we just fail?

Max: Yeah, well, yeah. So the, the, Well, if, if you run J when you install J bank, there's no job, no job needed. You can use any package manager and stuff. So basic J bank has nothing. It doesn't even have Java, right? So just have a jar, but it also has a script. So a best script or PowerShell scripts, which is windows.

And that thing will go look for Java and it has a default thing go like, Hey, I'll use the default DDK. That is on your system in your path to run to run JVM. But if that's not there, we will go download, I think it's Java 17 now. There's a, there's an API in the Java ecosystem that's hosted by a, what do you call it, Fujie.

And Fujie has an API to go get all these different variants of JDKs for different platforms. So we, we recognize the system, the architecture, the, the platform, the OS, we get it and download, install it. And then that means now we have the Java to run JBank. But if your script then needs to say, Hey, I, I'm using Java 21.

So you can do that with slash slash Java 21 in the file or 21 plus to say 21 or higher. We then go and look is the current Java at Java 21. Nope. Okay. Go fetch. We installed it in jbank cache and we will go, we will set up for you. So yeah, that, that thing is just. Magic in Java. Like it just works.

Jonathan: Yeah. So something else that might be magic that I want to know about is, again, you've mentioned this briefly, but dependency handling how difficult is it because Java, Java is all about dependencies. Like there's, there's like a million libraries out there. That's like one of the super powerful things about it.

Is it, Hey, I wish I could do this in Java. There's probably a library that does it. How do you pull one of those in? Do you have to, like, include somewhere that, like, here's what it is on Maven and go grab it, or can you just, like, include the class and it'll automatically go look it up? What does that look like?

So I,

Max: I had a prototype for the class thing, but it's just, it's too magical. Like, it becomes, like, things fail. But so, so right now, so it might happen eventually, but right now I use we use a slash slash steps for dependencies and it uses the, the, like the convention it's to say, if you use cradle, they have them all.

So maybe it actually has a syntax for specifying group artifact version classified as a whole language. And that's just the one that we support. And that makes it very compact. And, and that's, that's what we do. And it, it. Yeah, that's all that is it to it. Like slasher steps and you can do it in the file.

You can have a separate file. You could do it on command line. So, and you can combine them so you can have a, let's say you have a a script that does something that says I need out 21 plus. I'm using Hibernate for database access. But I, then when you run it, you need to get a driver, like a square, like in a square, a Postgres.

So then you can go J bank desk steps. The Postgres driver and then the scripts and then these two gets combined. And now when you run, you have access to the, to the driver. So you can, it's very composable in that sense too.

Jeff: Jeff,

Jonathan: you want to

Jeff: ask that? So I, you know, you said you had a cache directory and everything and you're, you know, And I, like I said, I'm not super on Java. So do I end up with a, like a jar file that I, you know, I write my script and J bang, it can run, it can work. Do I end up with a jar that I can then just say, Oh, Hey Jonathan, I wrote this script, but I just sent him the jar file and it runs on his machine or how does

Max: that work?

Yes, you can. Yes, you can. So this is that though. Yeah, exactly. So yeah. So there is actually. There is a jar file in, in, in, in, in the, in the background. Originally I thought that in the early days it was just there because that was an easy way to actually not have a thousand files, but just the jar.

That's one jar that has the whole thing. But then someone, this was actually the, the, the, my the mind Mindstorm, the Lego Mindstorm use case I had for my kid. I needed a jar. That I ran, I had to export it and ship it into a Raspberry, the Raspberry Pi thing that was doing the, the mind thing, the, the, the Lego stuff.

So I had to export it. So there's a J bank export that will generate the jar that the, the results, but it will also depending on how they suppose you can make, make the jar and then copy in next to it, all the dependencies. And then the jar has references so you can just run it. So it's like a a multi file thing on disk.

Or we also have an export fat jar, which then takes all the dependencies and just scrolls into one jar. And you can ship that over and run it somewhere. And there's a bunch of other ways you can like export a container, export other stuff. But yeah, you can actually, you can take those scripts and export as a jar and just give it to someone and hope they have Java.

You can tell them to install a J Bang because J Bang doesn't only work for scripts. It will also work for jars. And then you can, J Bang will go get the right Java version and run that jar for you. So I'm not sure if I explained that well enough. Right. Because then, and this is the thing where, where I was having fun, some weird night, I go like, wait, I can now run the jar locally on my, I can have Java sources, create a jar.

I can export the jar. And I go like, Wait, why don't I just allow you to run any JavaScript, just Java source file or jar that is locatable by HPS or the ASP request. So that opened up for that. You can go and say, not just J Bang a file, but J Bang a URL. And it applies the same logic that it does locally.

It's a jar file. Okay, I download the file, compile it to a jar, analyze dependencies, and all is there. It's a jar file. I run it. I download and run it. It's a main dependency. I fetch the dependency. And I, we actually made it even further. So now we, we have a shorthand. For any GitHub repository or a gist, you know, gist service for, for, for grabbing stuff, you can do gay bang, a gist URL, and we will go and look for Java files in that gist and compile it for you.

So you have something running somewhere you can just, it's now trivially easy to distribute that to anyone, anywhere on the planet. As long as JBang is on the, on the system.

Jeff: Yeah, that's awesome. Because that would be good for, you know, say I write something that I need to give it to my mom, who's not a computer person, I can just, you know, okay, install this, and then do this, and then Yes.

Everything, everything's automatically taken care of.

Jonathan: What is, what does the process look like of installing J Bang itself? Is there just like a, is, is there some little tiny script that you can curl and run that bootstraps everything? Or yeah, you, you mentioned it's available in pip. What are the, what are the options for getting J Bang on a machine?

Max: Oh, how much time do you have? Cause I, I counted it just before here. I think I have about 20, like 90 or 20 different ways you can install JVM. So there is the traditional, like in Linux land, you can curl download thing. Right. But you can also I have there's a, a Fedora package. There's a Red Hat rel package.

There's a standards package. There is It was called ASDF, the installer, there's a Nix package, there's a Docker image, there's a JavaScript module, there's a Python module, there's a GitHub action, there is on Windows, there is if you know, there's a NuGet. And a Choco installer, a Scoop installer, and on Win Brew, there is Sorry, on Mac, there's Brew to go.

So, I, literally, any reasonable Python system that either I was able to make work, or someone in the community has done, We have available to get it. So basically there's no excuse to not install and

Jeff: we talk about this on. The untitled Linux show sometimes, but I did look and there's also a snap that's available for Ubuntu.

Oh, yes,

Max: sorry. Yes. We like to tease back and

Jeff: forth sometimes about that. Some of us are not as excited about the snaps.

Max: Yeah, I know. Yeah. And the snaps and there's the one that there's some Fedora line. I forgot the name now, but the same concept of these isolated ones, which was actually a really a big challenge.

Yes. Yeah, it's like that, yes, right, but it's a very big challenge because I need, like, JBang is, needs a Java to run, right, but if I'm not allowed to run the Java that's on your system, I, it's a very limited use. And one of the things Flatpak does is it's default is sandbox, so you can't do that. So, but we, we found a way and, and Yeah, so now you can use it as a Flatpak and it can run.

But you have to run with like some, Like escalated privileges because you J Bang is calling back to your system to use the Java you have available. So, but yeah, no, no, there's there's there's all all the ways you can think of if one is missing Let me know

Jonathan: Mashed potato mashed potato from our chat room says was compiler from source on that list

Max: Compile So the product is fairly easy to compile You but yeah, but that's, yeah, you can, but it's not, it's not a way I, I I say, Hey, this is how you download.

It's one of the things I learned from early on is any successful, like utility open source product in the first, you have to have working software and two, you have to have it available to release. Yeah, so I'm able to run and that's why in the beginning I was like because I've been doing tooling for 20 years and I had to fight people for, hey, you know what great.

It runs on Linux and Mac. But majority of enterprise customers runs on windows and every linux guy and every mac guy hates me for it It's like I don't want to deal with it. No, no, I know but that's One thing is what you as a privileged guy who have access to all your links You're not the developer runs in a bank somewhere and told to run in this windows citrix hosted somewhere It will it won't run on this stuff, right?

So I I spent That was what, before I ran, I released anything, I want to make sure it ran on all three platforms. And it was, it worked on every release and I could release that very easily. And and that's, so early on, I think we, we had 10 or 12 different ways of packaging installing it, which was really, really hard to do because there was no other Java product that had done these.

Very different ways of setting up installers. And that actually, you had Andres Amorea on about JVelizer a month back or so, and JVelizer actually part of like the way JVelizer got created was I talked to Andres about some of this stuff and he said, Hey, I'm doing a Go thing and GoVelizer has this thing.

Why don't we have this for Java? I said, well, If you make it, I have all the scripts, I have already done all the work on how you make a CLI in Java that can run on the, or be installed by these. So he took that and improved on JReleaser to the point that now JBang is using JReleaser instead of the scripts I created like five years ago.

So it's all all connected.

Jonathan: Yeah. It's a, it's kind of a challenge these days actually to get a Java runtime environment on windows, isn't it? Didn't, didn't like Oracle make some licensing changes to where you got to have a, some sort of agreement with them to be able to download it. Oh,

Max: yeah. So this is another part, like, so no, so no, well, this is fun.

Like it has never been easier to get Java on any platform because what Oracle did or Sonar like years back was they open sourced it and there's all these different teams that makes JDKs available. So yes, Oracle did change their license. Statements and I'm not a lawyer, so don't, but basically it says you can, you can use it for free, but after a year, if you stay on this version, you have to pay Oracle or you upgrade to the latest.

So if you stay in the latest. You're fine with oracles,

Jeff: but

Max: of course, as a red hat or anyone else, so I'll just say, Hey, use any of the other ones, especially Eclipse adoption, which is making a build of OBDK available to you. And there's Amazon and Microsoft and others has DDKs that are available without any of those license restrictions on them and, and.

And then there's this fuj, which is a service that literally gives me an API, like I I in J Bank, I just do, I generate a URL that says, this os this platform, this combination. And it, it will orchestrate, it'll go, oh, you need you want adoption bill for windows on, on this architecture? I'll go get that.

At the actual vendor, right? So there's one place that you can look up and get any kind of data K for on any platform now. So it is trivial easy, but yes, Oracle, of course says, Hey, install the Oracle one, because if you get. That's the one you use, then you, you, you, you, you will want to get support from Oracle and pay for that.

Does, does J bank,

Jonathan: yeah. Does J bank support actually installing? So it sounds like you go with open JDK by default. Do you support actually installing the one from Oracle? And does that ever matter?

Jeff: Oh

Max: yeah. So the default I use is the, what's called Eclipse Adoption. So this is the, maybe you heard, heard the same thing about Omnidecay some years back, there was a, like an alternative to get core, Oracle to realize, Hey, make the binaries more freely available.

So we, I use that because I then are sure that no user that ever uses will end up with a lice, like a. You know, licensed what's called licensing.

Jonathan: Yeah.

Max: But so I don't, so, and also I wanted to make sure that any, any default use of J bang will always get as close an experience. As if, as you have, like, so, for example, Jeff said, Hey, I wrote my script on my Linux.

I'm sending it to my mom or some student was just run a javang. So javang will default download. If you don't have the system, the default system, Java, you'll go download the adoption one. But because we use this FoodJ behind the scenes that can get any of them, there is actually a flag in, I mean, it currently is not exposed in JPEG because I haven't figured out a good way of doing it, but there is an environment where you can say, hey, I don't mind the vendor, JDK vendor should be Oracle.

Or Amazon or Red Hat, and then we will actually go get that one. And that includes and the Oracle ones, et cetera. Where it make it's actually has a use case is when you use early access builds so when there's a new version of something, the open data K community project. Will have binaries before the orders have them like a few days before and then it's nice to do or if for some reason you oracle has some features that are specific to their ddk If you want to utilize those you would want to use the oracle, but that's like I it's a niche case.

So it's not a thing that It's, yeah, there's flags and things you can do it. And I use it all the time for testing weird combinations. Yeah. But so, so no, so yes, it's, it's important. That's the way to do it.

Jeff: Yeah. Awesome. So. You talk a lot about supporting, you know, all three platforms, you know, looking at other open source projects, which ones are successful, how they do it.

So what, what has been the challenge of getting people to, to use JBang, you know, getting users to come in, you know, into the ecosystem.

Max: So, so that's the, that's the, that was the one that surprised me the most, because for me, what I. Built jpang. It was just hey, I want to replace this case with that. Let's just do it and then suddenly I realized Oh shit, I can, I can, I'm sorry, I can do all these things that we just talked about.

And then I go out and showed it to my team, like, and this team, these are like you know, people that have been doing Java for years, and I thought they would immediately be like, this looks awesome, let's go. And they're like, no, no, like, I don't know why this is not using Maven. It's not using Gradle.

This doesn't work in the IDE. Like, why, why, this is not Java. Like, why should I use this? And I go, no, it is Java. It's just a little script. And I've heard so many times people like, no, no, this, this is not what I'm used to. And therefore they just abandoned the idea. And also there's this, say Java is in a enterprise setting, so everything has to be enterprise y.

So if stuff is easy, it's considered not enterprise y. Like it's this weird thing, right? Like I even heard. Someone was like, no, no I, it's, it's called J Bang. It's not Java. So therefore it's not part of the Java ecosystem. And then I point out to them and they say, well, no one will ever use that.

And I put out to them. You do, you do realize Maven or Gradle is part of Java that is a complete separate ecosystem from it. So it should be possible. Oh, that's, you know, that's just how it, you know, it is. And then you go look at this. Oh, it should be as easy to do as you do in Go, right? So for example, Go was, did a brilliant thing.

They actually have all these tooling in like anything is built into. Go like there's a go format or that's a go install all the stuff is there. But that is all copied from like Python and Node. js. So if you look at Python, pip is not part of Python. Pip is a separate thing that Python, the community has adopted.

Node. js and JavaScript, it was originally in a browser. Someone then took it and put it in Node. js and Node. js then made this thing called an NPM. So all the other key systems, which people are saying, Hey, these are easy. Has done what I've, I'm doing it, but in the Java world is considered. Against some religion or something.

That's the, that's my, that's my biggest issue that I generally, when I go out and talk and present, I it's, it's like dividing three people. One is the ones who get it immediately. And like, Hey, I can do scripts and I can run them. I can install them. And that's good. And then there's a group that just, they just.

They, I think if they got it, they've just been so indoctrinated, they're like, no, no, no. This is alien. I will not touch it. . And then there's the one in the middle who are like, they just, they need to go touch it and see it before they, they, it, it connects. Um, and that's, so that's, that's the, that's the, that's been the, the challenge.

And it keeps being a challenge that, that people just can't believe that it's this easy. And like. I mean the taco is the he he he's to to blame or thanks for a lot of these life and stuff like When he he proposed this thing about being able to app So we, I did, we did all the JBang run and run from UL, like in the early days, I could even, I can even run a tweet, like, so you can go, you have a tweet UL, I can go JBang Twitter, blah, blah, and I could have, I could run a tweet.

Unfortunately, I can't do that anymore because Twitter has locked down. If you don't have JavaScript, you can't run anything that way. But and. But Tago came up with this like, Hey, we, if, if, why can't we do JBang app install, like, like you can do NPM install pip install go install. Why can't we do that for Java?

So, and this is the thing, and this is, again, I'm saying here, and I'm pretty sure no majority of those who are listening will not get it until they try it, but you can now, there's, you can, you can take a JBang script and you can Or a jar or maybe an artifact or anything and install. You can go jbank ev install.

Like the latest one is JLama. JLama is an inference engine for AI. It's a little tool. You just go JBang, app install, JLama and it's a different name. But, and then you have JLama, and you just run JLama directly. You don't see JBang, you don't see Java. It's just there as any CLI. And there's a bunch of these tools out there.

Like there's an SQL line, there's a different utilities that are written in Java, which originally you have to go like download the JDK, download the jar, put it in the past, blah, blah, all that is just gone. You just go javang app install. And, and, and you can do that for any Java. This, like that product has been built with Maven or Gradle or whatever.

And the only thing they use javang for is just to, to wire up all that stuff. And again, if you're a Python guy, a Node. js guy, a Go guy, you've been doing that all day, all year for the last 10 years. And now you can do it for Java, but getting that word out and getting people to believe it has been the biggest challenge.

Jeff: Oh yeah, I, you know, and I, I remember what you taught when you said, you know, it's got to be hard to be enterprise and all that, because I remember back in the early days of the internet. If you didn't compile your own IP stack, you didn't deserve to be on the internet. You know, it was just, it has to be hard or you don't deserve to be here.

But, but going into that, can I use it in enterprise? Is it, you know, is it, is it stable enough? Is it ready to go?

Max: Well, so

Jeff: have a cheap code for enterprise. Yeah. .

Max: Yeah. Well, so the, the, the, yes. So the thing is, and this is related to what I said before, like it, it, it literally is, you don't have to do the scripting part of J Bank.

J Bank scripting was just where it started. But you can use any kind of j jar. So if you have, a spring project or micro naught or corkers or anything else. You can just use J bank to run it, set it up. And one enterprise area we, I, I see, and this is where we actually use it. On the product I have myself is all the CDI, like CI scripts, like the, the, it's so popular, we're doing all this dev ops stuff.

And the fact that our Java developers can just use Java to script things that they do. So that's one entry, but The other product that my main job is actually to work on a product called Quarkus, that's actually the t shirt I have on here is it's this thing where we, we, we, we, it's an enterprise framework stack that makes Java, you know, fun again, so to speak.

And, and, and, and, and can run very effectively. And that one actually, the way Quarkus does that is by doing build time, like doing simple build time. And that normally requires a full build tool. But if you combine that with Quarkus actually has support for JBank. So if you take J bank script and add a Quarkus dependency.

It will actually not just be fetching dependencies, but J bank has support for extension. So Quarkus actually gets invoked to go, Hey, I compiled these source files. Now go your Quarkus thing. And then that output is actually. An enterprise app, right? And that can then use all the stuff Corvus has, like native compilation, running container any camel, CXF, all that enterprise stuff is available to you.

But I do want to put an asterisk on it I'm not telling you to drop all your gradle and maven projects because there's stuff that debank doesn't do that these need to do there's complexities that debank navel will cover but Definitely. You, you, you can, you can use J bank to, to write microservices and, and get up and running.

We, we do it for our, we have a GitHub bot and some applications. They all, they are actually either a caucus app then run by J bank because you can get all the Javas, or it's a script that sets up a few things and just runs. So, so yes, it is enterprise, right? But it has a certain use case, right?

So in that way, it, it, it is there.

Jonathan: Very cool. So I've, I've got to ask like what what's, what's coming down the pike, is there something coming for J Bang that you're excited about?

Max: Well, so my what i've been doing mainly is trying to I go out and I I try and find all these utilities because there's there's a bunch of people who wrote Awesome things that never got into hand of people because it was hard to run these things.

So i'm going I mainly go out and submit small patches Hey, jbang we have this notion of a jbang catalog which sets up the command line or that kind of thing and and make it rumble So that's this That's what I've been doing. But the, the, the, the, the main, so when I then do that, then I might find a thing that they do that I hadn't thought about, and then we do a release.

But, and I'm on release, I think like, like 0. 119 is the one that I think I've done like 250 releases in. For four years now. Right now it's very incommensurate, but I'm trying to get to that one. Oh, but that one, Oh there are some enterprise features around how you manage dependencies that I don't capture.

But otherwise I'll say we are very free to complete in, in being able to do the main mission of what JBang is. So, so, yeah, so. For me, the big thing now is actually to get people aware that this exists and try it out and not try and be scared of it. So it's more of a soft features than the hard features in that sense.

Jonathan: Trying to get to 1. 0, do you feel like Sisyphus rolling the rock up for all of eternity? Is it a Sisyphean effort?

Max: Yes. Well, it's just, the thing is I, the problem is I've, I've made the come back to like make, it's so easy to do releases and like, it's been stable and we actually try. And I think we have one or two glitches where we, we did a mistake.

But it's actually, you can still use the old ones just fine. But one of the things I want to do before the wonder will break some users and That's, that's the rolling I'm, I'm like one day I'll, I'll do it this way. And then they, yeah, so no, so yes, I do fall through that. But I, the, the cool thing I do is this part of the the way we do the current, like the, the installation, whether it's the Maven plugin or greater any, any of the way you install like curl and stuff is is going through J Bang dev a URL so I can actually see When people are running a J bang in stores.

So no, I can't see who you are, but I can just see someone somewhere did fetch this little script to go look at the latest version information. And I can see the growth is there a lot of people in Azure and GitHub Jackson is using it. But I'm still missing the. The uptake in all those Java utilities out there to, to to make, to, to have it and any Java stuff available as easy as you can for Python and, and, and JavaScript.

Jonathan: All right, we're getting close to wrap and here in just a minute, I'm going to ask if there's anything we didn't ask that we should have. While you think about that, I'm going to ask a different question. And that is, what is something that your users have done that really has surprised you? Is there any like oddball use cases that are particularly interesting?

Max: Okay. Well, well, so, so, so the, the, the latest one, it's not all about the JVM, it's just this thing of so, so the, the, the JLAM, I think, so, so again, everyone thinks AI has to be a Python, which is great and stuff, but JLAM Java has a vector 20, there's a, a vector support coming, so they can, it can do efficient array copying and manipulation, which you need to do in inference engine.

So there's a guy Jake, I just know him as Jake on, on, online. His run and written in a Java implementation of that's called JLama. And he had an installation set up that was very complicated. And I just pointed out, Hey, you can do a JBang. And he just did that like this week. So now you can actually, with JBang, again, you can start from scratch on your machine and just install JBang and go JBang download this, one of those models and run it and no, and it will be a CPU, a GPU optimized and that kind of thing on any platform where that exists.

So that, that's my recently interesting setup. Then I know that's, that's gotta be tricky to explain but I'm showing it, but Java has a notion of agents. So you can attack any Java app. You can attest an agent that gets notified by anything gets loaded, and then it can manipulate the classes and you can do evil things or good things.

They can monitor and that kind of thing. And again, if you use the normal Java tool chain, that's a lot of arguments, a lot of setup, and you have to download files and that kind of thing. So we have a product in Rehag called Byteman, which is an agent that can go in and you can say, hey, if I hit a method that has this signature the third time, throw an exception or log a statement, right?

So you can do all kinds of things, but you have to do all these steps. And then I realized in JBang, we have all the pieces to not just go get dependencies, but all of the app, but also the dependency of the agent. And also any configuration that the agent needs, I can go get from a, from a URL. So again, with JBang, you can actually.

Set up any kind of agent usage on any kind of a Java app in a one liner command, no complicated cradle, maven, all that kind of thing. And I find that super interesting, but. I also think many people will actually appreciate it because those who actually managed to do that without it is very, very few, but the, the, the, the, the power is immense.

And this is stuff that you cut like in, in Python or Node. js is those kind of things just doesn't exist or are even more obscure than, than we have here. So no, so the,

Jonathan: Yeah, interesting. All right. Is there anything that we didn't ask about that you really wanted to let folks know about?

Max: Well, the main thing I have is like, if you're listening to this and seeing it, do just try it out and spread the word, like try, try and use it.

And if you have a Java tool to somewhere. Apply a little bit of JBang so it actually becomes available to anyone out there. That, that, that's my, that's my hope that some, some of those people in the, Hey, Java is this horrible, complicated thing. But I know that this utility I actually would like to use if it was easy to use Java.

That's where JBang can help. So, so, yeah.

Jonathan: All right.

Max: Try it out.

Jonathan: Yeah, absolutely. I do have a couple of questions that I'm required to ask everybody. I think I already know the answers. But what is your favorite scripting language and text editor?

Max: So the text editor. So I'm, I use every editor on the planet because I've done tooling it, but the one I, my day to day driver at the moment is VSCode or or some variant of that.

The scripting language, well, I would say Java space script. I, I've done a lot of scripting in my, my like bass and Ruby and Perl and Python, but once I got it working for Java, I use it for everything. But it's technically not a scripted language, but. scripting. So I'll say it counts space. It's my, it's my favorite script.

Jonathan: There you go. All right. I think that's it. We've gone past the hour. We, we sure appreciate you being here, Max. Thanks for stopping by and Telling us a bit about J Bang and scripting with Java. Sounds like fun.

Max: Me too. You should try it. It's, it's, it's not your parents Java anymore. Yeah.

Jonathan: Yeah, I'll have to, I'll have to look it up and give it a shot.

All right. Jeff, what do you think? Have we, have we convinced ourselves to go try some JavaScripting that is not JavaScript?

Jeff: I, you know, I'm kind of interested in this. It's you know, I, I've seen a bunch of Java programming in the past. And, you know, I've, I've messed with other languages, not really with Java, but, you know, kind of something to make it a little easier like that.

I kind of interested in that, you know, I, I, it piques my curiosity.

Jonathan: I, I feel like it would be fun to at least like, fiddle around with it enough to be able to say that you've done so. I, I'm curious. I, I guess I wonder are there things that you normally do in scripting that Java is not made for? And we, we didn't, we ran outta time.

This is one of the questions I, I wanted to ask. We ran out time to do so. But I'm interested to dive into it and see, you know, like in bash scripting you get to do things like, call other bash commands with backticks and then replace the output of that command into your script and Just I I guess Java has all of that built in because it's more of a full featured programming language So it doesn't even need all those tricks But it's interesting.

It'll be fun to play with I definitely need to go grab it and check it out in in possibly a snap format Maybe I don't think I have any computers that'll run snaps actually

Jeff: That's, that's a UL, ULS inside baseball. Yeah, yeah,

Jonathan: yeah, yeah. All right Jeff, thanks for being here. Do you have anything that you want to plug?

Jeff: I don't. I just well, I guess other than check us out on the Untitled Linux show, over on the twit. tv network become a member of Club Twit. And I'll see you next time.

Jonathan: Yeah, have a great week. Excellent. Thank you for being here. I do want to let folks know. First off, next week, we're talking about EMBA with Michael Messner.

And that is an embedded firmware analyzer. I came across this doing the security column on Packaday, which go live, goes live on Friday morning, which you should check out. Came across it one week and it's an open source project doing like embedded firmware analysis, looking for problems, trying to understand the way it works.

And I got to looking at their code and thought it was really cool. And so we're going to have them on the show next week. It's going to be super duper interesting to talk about EMBA. Other things that I want to plug, of course, as Jeff said, we've got the the Untitled Linux Show over at twit. tv.

Would love for everybody to go check that out. Show Twit some love, and yeah, we appreciate Hackaday giving us a home for the show. And with that, we will see you next time on Floss Weekly.

This week Jonathan and Jeff chat with Max Rydahl Andersen about JBang, the cross-platform tool to run Java as a system scripting language. That's a bit harder than it sound, particularly to take advantage of Java's rich debugging capabilities and the ecosystem of libraries that are available.  Tune in to get the details, as well as how polyglot files are instrumental to making JBang work!

• https://www.jbang.dev/
• https://maxandersen.github.io/getting-started-with-java-the-jbang-way/
• https://xam.dk

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Aaron joins me and we talk with Andreas Kling about Ladybird. That's the from scratch web browser that's almost ready for primetime. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 800, recorded Wednesday, September 10th. Champagning the Ladybird browser.

It's time for Floss Weekly. It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and today is no exception. We're going to have a lot of fun. We're going to talk about Ladybird, the from scratch web browser, which, boy, that just, it sounds weird to even say it, but We've got the, we've got the man himself we're going to talk to in just a minute.

But before we get to that, we've got the other man. It's, Wow, that transition did not work as well as I thought it would, but here we are. We've got Aaron. Welcome, sir. Hey, thanks. Thanks for having me. It's good to have you back. I am so glad. One of the things that makes me extremely happy about moving to Tuesdays is that we get to have Aaron back as one of our co hosts.

And I just, I get a kick out of that because I've always, always enjoyed being able to work with you. And I was very glad to be able to have you back in the rotation.

Aaron: Yeah, for sure. I just noticed that my maybe because of my mustache, my beard's a little longer, it needs a trim. But I've noticed that like, I've got the old man syndrome.

Like the older you get, the more your face droops and I'm sitting here smiling and I'm realizing that it's just like a straight line across now. It doesn't go up in the corners anymore. So yeah, I've got a, I've got a hope that people know that I'm smiling, but I am, I'm happy to be here.

Jonathan: Yes. So Lady Bird, what, what do you know about Lady Bird?

Aaron: Absolutely nothing. Absolutely nothing. It'll be, I'm really interested for this conversation though, because you know, a, why do we need another browser anyway? Like what's the problem that they're trying to solve? I think. A lot of us know what that is, but I still, I, it's a lot of effort to do a browser.

There's some

Jonathan: problems. Let's just say there's some problems.

Aaron: There are problems, right? We'll get into that. And then, you know, I'm, I'm old enough to, to go back to the days of using mosaic and text based browsers and gopher and, you know, I was around at the beginning of the internet, um, at least that part of it.

So. So, yeah, I mean, I remember when new browsers were popping up you know, pretty frequently and it was like a big deal, like when Netscape navigator came out, it was like, Whoa. So anyway I remember those days. So I'm kind of curious, like, what are we going back to those days of like, Hey, let's just get back to basics here.

Or, you know, is there, you know, how do we, how do we accomplish that? And also make sure we can still do the things we want to do. Right. On mine. So, yeah, I'm really curious.

Jonathan: I'm, I'm kind of excited that we have a new browser that's not just Chromium with a different skin on it.

Mm hmm.

Yeah. We have, we have a bunch of those, and so people, I mean, you just talk about this browser and you go and look at it.

It's just Chromium, guys. It's Chromium with a theme. It's, do we really need to do a whole show about this? Lady Bird is a bit different. Well, let's, let's go ahead and bring Andreas Kling on, who is the man when it comes to Ladybird, and I guess the man when it comes to browsers. Welcome sir, thank you for being here.

Hello, how's it going? It's great, it's great. So, Ladybird, it's, it's a browser. Let's see, where should, where should we start? I have lots of questions. Shoot, I don't see a download link. Let's start there. Okay, the audience knows this, just so you know this, I will ask a lot of questions that I either have a guess or I know the answer to.

I'm pretty sure I know why there's not a download link, but I'm, I'm trying to be a proxy for our audience in asking me to share.

Aaron: Well, that's the first thing I did this morning as I went to go download it. It was like doing exactly that. I was like, Oh,

Jonathan: I see a project,

Aaron: but no download.

Jonathan: Do I have to, do I have to compile this from source to be able to use it?

What?

Andreas: That's crazy. No, like you were saying we are doing something different, which is that we are not starting from Chromium like everybody else does. And because of that, it's going to take a little bit longer to, to get this thing ready. So the, There is no download today but you can download the source code and build it.

I think it's two commands that you have to run and then it should take care of everything for you. We, we do get a lot of positive comments about how easy it is to build our project. And we are aiming to put out an alpha version in 2026. So that's sort of the timeframe that we're looking at right now.

But we, to go more public because we started a nonprofit earlier this year. So that's why you've been seeing us more and more because we started to get more serious about this. And I hooked up with Chris Wanstroth. Of GitHub fame. And we started this nonprofit to fund and develop Ladybird.

And we've talked to a bunch of companies that are also interested in, in like having a new browser on the scene. So that's sort of how it's come together. And but at this moment you can, you cannot download a running product. We are pretty far from that, but because. It's a lot of work to build a browser.

We do need a little bit of funding, at least not as much as everybody else, but a little bit, so we have been raising a little bit of money. And yeah, that, that's kind of the, the general state of things. And a lot of people, of course, brave early adopters and courageous open source enthusiasts still go and build the browser.

And they are typically disappointed, but come away with a bit of hope at least in their, in their stomachs, I would hope because it doesn't work Well, for daily driver browsing today and we're not trying to mislead anybody about that. And we are actually two years out from, from what we would consider an alpha version.

So we don't encourage anybody who's not technically minded to even bother with this at this point in time. Because you're just going to have a bad time. But for people who are technically minded, you are most encouraged to try it out, mess with it. Try your favorite website, maybe a website that you made yourself.

Tell us what didn't work, what didn't work right. And maybe even figure it out yourself and see if you can help us fix it. So we've been doing a lot of that. Trying to collect new developers.

Jonathan: Is using ladybird right now a better or a worse experience than trying to use something like the links browser?

Andreas: Oh, wow. I don't have an, I don't know. It might be better.

Aaron: It's gotta be better.

Andreas: I would hope it's

Jonathan: better.

Andreas: It's, it's probably better. Yeah. Cause yeah, but the links is that links to one, but JavaScript.

Jonathan: No

Andreas: links

Jonathan: is text based links is the entirely text based browser. I don't think it runs JavaScript at all.

Andreas: Oh, yeah. Then you're kind of screwed.

Jonathan: Yeah. Okay.

Andreas: Yeah, no, we, we, we can do bare, we can do JavaScript, not just bare minimum, but like we can do real JavaScript. But we do struggle with performance. We struggle with some of the more intricate features and especially stuff like a YouTube doesn't work yet because of a million intricate little things that we have to figure out.

And there are a lot of these bot detection systems that we have yet to like convinced that we are a real browser. So CloudFlare and Google and, and whatnot, they have like these you know, mechanisms to make sure that you're not a bot and we look like a bot because we're just doing stuff wrong.

And that's entirely on us. You know, they're, we're not complaining about them. We just have to get better.

Jonathan: I could actually see somebody like Cloudflare getting interested and excited about the project. You know, they're all about having, in some cases, their own technology stack. And so, having a browser that's not run by one of the big guys is maybe something they'd be interested in.

Andreas: Yeah, so we've spoken a little bit with one of the engineers on the team at Cloudflare that makes this anti bot software. And we have a little bit of a back and forth there, but there's just a lot of work on our side to do. And of course, not to mention any of these fingerprinting things, you know, where you can check, Hey, how fingerprintable is my browser?

We are possibly the most fingerprintable browser right now.

Jonathan: So, okay. So what's the origin story here at, at what point did you wake up and go, I'm going to build a browser from scratch. And like, what, what sort of my, where, what was your headspace? What, what kind of crazy headspace did you have to be in to make that decision?

Andreas: What, it never happened to you?

Jonathan: Not that in particular, no.

Andreas: Other crazy things, yes, but not that one. Right. Well, it all started when I decided to build an operating system from scratch. Which was its own crazy headspace that Somehow seems less crazy? In some ways, perhaps, yeah. In retrospect, it was a simpler time.

But yeah, so I was doing that for a while, starting in 2018, I built the Serenity operating system, Serenity OS, and it was a. Passion project for myself that I did as a sort of personal therapy to keep myself out of trouble. I used to have a pretty big problems with drugs, alcohol, stuff like that.

And I needed something to focus on that was healthy. And yeah, I went pretty hard on building an operating system, put it online. A lot of people liked it and started to work on it as well. And a community formed around this and the community just kept growing. And we became more and more ambitious with our scope and we would add things like well, initially it was pretty modest, you know, we would add networking and we thought that was a big deal, but And then, like, why don't we have a photo editing program, or a music production studio, or visual programming tools.

And it just kind of kept growing in scope. So it was natural one day that we just decided that we should have a browser also. And a big part of SerenityOS, the SerenityOS mindset, was that we do everything ourselves. Like, we don't borrow code from anywhere, we just do everything ourselves, because it's more fun that way.

And I'm sure every hacker ever can relate to that. Even if you wouldn't necessarily do it at work, you at least enjoy doing it at home. Sometimes. So yeah, that's sort of, that's sort of the where the whole thing started from was that we wanted a browser for SerenityOS because we were adding everything to it.

And I have a professional background in working on browsers. So I was working on the KDE browser Conqueror like 20 years ago. Since then I've worked at Nokia on their browsers and then at Apple on their browsers. So working on browsers was like my job for a long time. So it was very natural for me and once I started working on a browser again, I kind of just slipped into this old habit, and it became my main focus, and I kind of stopped working so much on the rest of the operating system, to the point where they became two distinct projects, and we forked because, The browser was getting so much attention and people were focusing exclusively on the browser.

And we were living in this GitHub repository together with an operating system, with a photo editing software and music software. It was just a really cramped space. So projects split up and now I work only on Ladybird. So the browser only. And we've sort of changed the rules a little bit. So as I mentioned, it used to be that we don't use third party software.

Like we do everything ourselves since we're in the US, but in Ladybird, we want to actually make a product that people could use. And so we have admitted to ourselves that we're going to have to use a little bit of third party software to, to make that happen in this lifetime. So Over the last couple of months since we forked, we've been integrating some of the sort of open source ecosystem for things like fonts graphics formats, audio formats stuff that would take us a long time to do ourselves exhaustively and correctly.

We can just like piggyback on, on the existing stuff.

Jonathan: And

Andreas: yeah, that's sort of the origin story. Yeah. And that's where we are today also.

Jonathan: So speaking of where you are today, I'm going to, I'm going to hand it over here in just a second, but I do want to ask first, like what, what is the state of ladybird?

How much of the web actually works? How much of it renders correctly? What, how, how frustrating is it to try to use it?

Andreas: Oh, well, it's pretty frustrating. I'll tell you that much. But we've been focusing on our own sort of Dogfooding use cases. So we tend to use a lot of

Aaron: dogfooding. It's champagning

Andreas: champagne.

Aaron: You're drinking your own champagne, not eating your own dog food. Right. Or maybe not. Don't be yet. As the case may be.

Andreas: Yeah. I don't know that that will work for me personally, but I can see how dog fooding is a bit of a weird term. Yeah, no, at Apple we always called it living on. So like you would be living on the latest build or living on Living on Ladybird.

And we're trying to do that. So we're focusing in on our own daily use cases, like using GitHub, using you know, Google and reading web specifications more than anything, really and getting communication software working like discord and stuff like that, like stuff that we use every day. And those kinds of things have been seeing pretty good development.

They're not super great yet. GitHub is, I think, the thing that we handle the best. But once you sort of stray outside of the things that our developers use every day, you're gonna probably run into issues. And It's, we've taken kind of a vertical slice approach to developing the browser where we've been just picking a website and then doing whatever it takes to make that site work as well as possible.

Which is, you know, one of many approaches. Another one approach you could take is you could pick a spec and then try to implement as much of that spec as possible. But we've been kind of going after these like, let's try to make this site actually work. And I, I often liken it to video game emulation, where you know, like if you're going to implement an emulator for the Super Nintendo or whatever, you're not going to sit there and implement every CPU instruction.

You're going to try to get Super Mario Kart to run or whatever. So we've been taking that same approach to getting websites working. Which means just to answer your question it's, it varies greatly depending on if the site you're visiting is something that we have worked on or not, but over time The, the rising tide lifts all boats or whatever, right, where because all of the fixes that we make for the various websites that we do work on, they are all in service of, of you know, supporting the web platform.

And it just happens that we pick sort of scattered random Parts of the web platform to implement at a time. But over time, our general standard support, our general support for the web has been improving. And recently we've also started running, there's this test suite called the web platform tests, which is a sort of a collaborative test suite that all different browser vendors contribute to.

So, you know, Microsoft, Apple, Google, Mozilla. And ourselves. Can contribute tests to this giant battery of like millions of tests. And we've been running that recently and I think I think we're passing like a bit more than half of the tests. And we're like actively working on passing more of the tests and Yeah, we, we definitely want to just get those numbers up.

But because we haven't been running the test before, it is there's a lot of low hanging fruit that we're dealing with. Yeah.

Aaron: Does it support Flash? That's a joke.

Andreas: It, it doesn't actually I mean, coming from

Jonathan: SerenityOS, that would be very on brand.

Andreas: A little bit, I guess. Oops. Oop. Somebody's got a dog.

It's all good. No, we don't support that.

Aaron: That was just a joke, but seriously, though, like, like, why, like, well, let me ask this first. Actually, I want to know, like, you kind of told the origin story. I want to know, like, why you think we need this number one. But before I even get there, the thing that I'm most curious about is how much interest have you received over the past, whatever, six months since the whole Google antitrust Yes.

Stuff has been going on. Have you like, you know, gotten way more interest all of a sudden, because of all of that?

Andreas: I don't know that I could really separate interest coming from that from interest for other reasons, but there's certainly been a lot of interest over the last couple of months, but we only went public with our.

Nonprofit in July. So, and the, the, I guess the latest Google rulings were pretty recently. And when people donate to our nonprofit they sometimes write a little message and there have been a lot of messages mentioning the, the state of affairs of the industry. Which I think, you know, it was positive and.

When I, when I talk about the origin story, it's, it's sort of it's sort of Ladybird just evolved out of hacker culture in some ways, but at the same time, it's also finding reasons to exist that aren't just organic. Because there, I think there is a need for a like a truly open browser that isn't connected to like the advertising industry, right?

And isn't necessarily depending on the exact same browser engine that everybody else uses, but like a new implementation that is standalone and doesn't have to do whatever Google wants at all, at all times. And yeah, it's, it's, it's a tricky little bit of a tricky subject because I've been in the browser industry for a long time.

I know people working out in every browser. There are a lot of great people everywhere. But at the same time, I think we should also acknowledge that the industry is a little bit messy. There are a lot of weird incentives. For, for the major browsers. And we are very keen to try a new approach to this where we are a nonprofit and we say very publicly, very explicitly that we will not take any kind of strings attached sponsorships.

So we're just going to. Do this with a small team you can sponsor us if you want, but you're not getting anything other than a logo on our website. We're not gonna, not gonna put your search engine in our default settings. We're not gonna send data your way of any kind. And We, we understand that this drastically limits our budget compared to the competition, but we think that this is something worth exploring because the world should have a browser that is independent of all of that.

And we kind of started in hacker culture and just organically grew a browser. But we find ourselves in a position where like. There's nobody else to attempt this. So why not us? Why couldn't we be the open community developed browser with no attachment to any advertising money?

Aaron: Right. Right. I think it's a, I think it's a very ambitious and very noble thing to do.

At the end of the day, it may be too early to tell this. I don't know. Are you going for, for parody at this point in terms of functionality or you know, when we get to 2026 and there's something that is on the download page for people to try will there be noticeable difference? What will the differentiation be between Chrome, let's say, and this, when it actually comes out?

What are you, what are you hoping to achieve?

Andreas: So what we're aiming for is that you can do your daily browsing with reasonable, Stability, reasonable performance. But it's unlikely that we will be faster than any of the other browsers because they have thousands of engineers to throw at performance.

We don't have that. We're not going to have a sophisticated developer tools that like help you write your CSS and develop your website. We're going to have some tools, but we just don't have the manpower to put all that together that they have, you know, a 10, 20 year lead on us. That's going to take time for us to backfill all of those things.

But in, in terms of just like rendering the web we are hoping to make something that will work well enough that a user can use our browser. And. Not think about the fact that they're using Ladybird. That's sort of the happy outcome is that you would just use this browser and it will work and you're not thinking about what browser you're using.

If you look under the hood or if you start to poke around in the menus and you try to try to do fancy things, you will quickly discover that we don't have those yet. But our hope is that it will be a decently well enough working browser for common websites that you're likely to visit. You know, your LinkedIn's, your Facebook's, your Gmail's.

All of those kind of things. And then It's seems very likely that we will spend, if that all works out really well, then we will spend the next year or two just fixing the last 5 percent of bugs. Because I think it's one of those things where like the first 90, 95 percent are going to take two years and the last 5 percent are going to take 10, 20 years.

Jonathan: Yeah, well, part of that's because the web is, is such a moving target. Like there's constantly new things getting added to the JavaScript standard, to the CSS standard, to the HTML itself with, you know, HTML five and all of that. Is there, you, you might get to the point to where you sort of caught up, but if, you know, if it's going to continue getting supported into the future, I don't think it's ever going to be done,

Andreas: right?

No, it never will be. This is definitely a project that can go forever. And. There are all the other actors. So, you know, Google, Apple, Mozilla, they are all actively adding new features to the web. We are passively just implementing what they've. Come up with maybe one day we will decide that we, we can think of some great features too.

But at the moment we're just keeping it cool, you know, just implementing what's there. I do wish that people would slow down a little bit with the feature development but we do also recognize that the web has to evolve. Maybe not as quickly as it does sometimes. Yeah.

Jonathan: Yeah. So, so Aaron asked about the the idea of differentiation between Ladybird and Firefox and Chrome, and there is something that comes to mind, and I wonder if this is, this is something that you guys have realized, if other companies have talked to you about it yet.

Both Firefox and Chrome are huge, and they're difficult to install and work with. Like, they're just unwieldy. Like, particularly Chromium, trying to do a compile of Chromium is just a pain. And you mentioned earlier that Ladybird is pretty easy to get building and to get installed. And I could see it having a life in, in some places, you know, like you've maybe even embedded just because it's so much easier to work with is, is that something that you've, you've kind of had conversations about?

Andreas: We're not ruling it out. I know that there's been some interest in the embedded use case, but we haven't really been targeting it. We, we've been playing a little bit cowboy like with memory usage and resource usage in general. Because it's easier to just allocate lots of memory than it is to be careful and Precise.

And a lot of people, you know, a lot of people look at Ladybird and they assume that because it's new and because it has a limited set of features, it must be using less resources, but that couldn't be further from the truth. We managed to implement fewer features with a lot more resources than the other browsers.

But of course that's because they've been optimizing. Their browsers for decades and we haven't. So we're catching up a lot on, on that. And I think the embedded use case requires a ton of pretty sophisticated optimizations that we're going to have to do anyway, but I don't know, I don't know what kind of interest there would be.

I guess that's something we'll, we'll find out eventually if somebody comes to us and says, Oh, I have this great idea for an embedded use case. We're definitely interested in helping them, but it's not something that we're pursuing ourselves. Like we're primarily interested in making a desktop browser for people and for ourselves is the, is the current target.

Jonathan: Yeah. I have a I have a Raspberry Pi in my hallway. That is my HVAC controller. And the idea there is that it's supposed to run a web browser to, to show, you know, your, the webpage of what your temperature is. And it's kind of a pain to work. I think, I think, I think. I don't remember if it's Chromium or Firefox is what it's supposed to be doing.

It's not working at the moment. I need to spend some, you know, some engineering cycles on that again to get it working again. I think it could be a lot of fun, because the webpage it runs is just Stupid simple. It'd be fun to try to do that with ladybird. So I, I am

Andreas: For that kind of thing. Yeah. I mean that shouldn't be terribly difficult.

We could probably throw in some command line switches for you to like start in full screen mode. There's no ui and stuff like that, right?

Jonathan: Yeah I could that seems like that would be a huge win. I could definitely see some some businesses wanting to use that. What's the what's the license? What license is ladybird under?

You

Andreas: We are under a two clause BSD license, so pretty permissive, very permissive. Yeah. Yeah. Yeah. Yeah. Not everybody's favorite, but I don't think there is a license that everybody would agree is the best ones. Right.

Jonathan: Yeah. Yeah. That's, that's the thing there. And what about platform support? Is this you know, X86 64 only, does it work on ARM?

Does it compile on RISC V you know, MIPS, there's a whole bunch of different architectures out there,

Andreas: right? So far it's been X86. 64 bit and arm 64 bit as well. So we run on Linux and Mac OS at the moment. And like, you know, the, the wide family of random Unix's as well. Since we come from, from SerenityOS originally we had pretty, like, generic Unix fundamentals in the project, so it's been easy to port it to, like, FreeBSD, Haiku you know, NetBSD, other systems like that but Windows is the big elephant in the room because it's so different and not Unix like that, you know, Windows support is something that frequently requested, but we don't have any active person working on it.

But this, this is probably not the podcast to, to complain about Windows support.

Jonathan: And, and somebody could run it like in, in the what, what are they called the, the Linux subsystem L S W. No, that's not right.

Aaron: Linux subsystem for Windows. Yeah. LSW.

Jonathan: That does not seem like the right acronym, but okay.

Okay. WSL. I think it's WSL. It's WSL.

Aaron: Windows subsystem for Linux. Yes,

Jonathan: which was always a confusing way to put that. It feels backwards. It has led to some discussion. Is Windows looking to replace the Windows NT kernel with Linux?

Aaron: Right. Right,

Jonathan: but if somebody wants to there's there there's that right? I'm assuming it runs just fine.

Andreas: It does it does Yeah, although it is probably the worst developer experience that we have so people who try to do it that way They're the they complain the most out of everybody that comes to us.

Aaron: You're really making a Trying to try to force a square peg in a round hole

Andreas: Indeed

Aaron: at that point Yeah, you can do it, but you know, why not just start up a virtualized environment in your Windows desktop and yeah, you know, just do just do that.

Just run Linux on top of it in virtualization. What I had a question and now we had that discussion. I'm not sure what it was. Well, I guess one thing that I was wondering is like, what are the biggest hurdles do you think knowing the space so well? Oh, I remember what the other one was too. But the first one is what are the biggest hurdles do you think that you have, like, this is just going to be impossible to build from scratch because.

You know?

Andreas: Oh, what do you think? I think I never really thought of it that way. Or maybe

Aaron: because everything's a standard. Sorry to interrupt you. Maybe because everything's a standard, you can just write to the standard and you don't have to worry about it. I don't know what the answer is there.

Andreas: Well, that's partly that.

So the standards are a lot better today than they ever have been because the various groups that work on the standards, they've been doing a really good job and backfilling all of the stuff that used to be unspecced, used to be sort of browser voodoo mystery stuff. Over the last 10, 15 years, specs have gotten really good.

So if you're writing a browser from scratch today specs will get you a long way, but there are still some things that you sort of have to figure out what other browsers do and then mimic that as well. And. That continues to be one of the more annoying challenges when the specs are lacking. So we ran into something just this week.

And not the first time, which is JavaScript date parsing. So if you look into JavaScript specification, the way that they explain date formats is that there is the ISO. 86 0 1 date format that the spec says you must support outside of that format. The implementation decides what additional formats to support and in practice every other browser supports like a giant collection of random date formats.

You know, like Jan one comma 1972 or Mm-Hmm, one, Jan, 1972. And. There's no list of these anywhere and people have tried to spec how this works, but it kind of just, they just get bogged down and, and they get tired of it and abandon the attempt. So even today, the spec doesn't explain how to parse dates.

And that's been pretty frustrating for us as a new engine. Because we frequently inquire, encounter new date formats that we haven't seen before. And then we have to implement parsing for that particular format. And now the, some websites starts working.

Aaron: Weird.

Andreas: And yeah, then there are, there are a couple of things like that.

A couple of touch points where you have to just look at what other browsers do because the specs, you're just kind of shit out of luck with the spec. But you know, to, to the credit of, of the standards groups, These number, the number of such things has been going down aggressively.

Jonathan: That puts you guys in an interesting position where you're, you're sort of doing an audit of the web specs.

It might be interesting to try to put all of that feedback into a document and, and make it out, put it out there, make it obvious, so hopefully someone could get fixed.

Andreas: Well, we are doing Ubuntu better and we are actively reporting and fixing bugs in the specifications. That is one of the great benefits of new engines coming on the scene is that we end up really putting the specs to the test because people have been iterating on these specs for years, but nobody ever tries to implement them from scratch.

And so we come along and we do that and we find like, well, this doesn't actually make sense. If you try to implement it it falls apart. That happens a lot. Regularly, I would say. We just come across some little thing where it doesn't Have internally consistent logic. And then we report bugs and it gets fixed and that's fantastic.

And it always feels so wholesome when I see somebody from our team go and find a bug in a spec, report it and it gets fixed and the whole ecosystem benefits. So yeah, we are very much auditing the specs and doing our best to be good citizens and, and reporting bugs in specs, or if you find bugs in other browsers, we try to report those as well.

Yeah. Try to be, you know, try to be good boys in the club. Very good.

Aaron: So what, my other question was in terms of the other open source OS vendors, Red Hat, Ubuntu, name your favorite one you know, since, since you've already described how this kind of grew up out of SerenityOS, like, do you, are you getting support from those folks more like you would expect you would, or no?

Andreas: Not really. I didn't really have any expectations though. So there's been some interest in packaging our stuff, but it's so early still, you know, we don't have anything that We could in good conscience, ask an end user to try. So even when distros do try to package our stuff, there's been like I think Arch has some package.

Nix has some package. And I see a lot of people trying these packages and they come to us and they complain that it doesn't work. And yeah, because this is not ready to be packaged. So. I have kind of mixed feelings about Distro's packaging pre alpha software. I think maybe they should just not do that.

It's not a service to anyone. It doesn't help their users and it doesn't help, it doesn't really help the projects they're packaging either. At the same time, it is also exposure for the project and like, you know, sends people our way, just you know, it's, it's, it's, it's, I wish that we would get to manage our own first impression, I guess.

Aaron: Yeah. I'm just surprised that, that, you know, those, those bigger players aren't like, you know, in your sponsors. And I guess there's a, there's an argument to be made like, Hey, most of the browsers are based on open source anyway. So, you know, I grew up in the days of Richard Stallman, well a little bit after when he was really going, you know, but in the 90s when he was still preaching, you know, all that stuff pretty heavily and everything has to be open source and you need a clean release of your distro and.

You know, back then it was proprietary browsers with proprietary code in them that you were fighting against. Now you could make the argument, well, good enough, right? There, everything's open source. It's good enough. We don't need this, but I am a little surprised that some of them haven't come on board and said, you know what we would like to, yes, well, there's a lot of open source out there, but we would like to separate ourselves from the commercial entities of the world, the Googles and the other players, because we don't want to be beholden to them.

So, yeah, I'm just kind of surprised that they're not in the list.

Andreas: Right. Well, they are very welcome to join the list. If anybody from, from any of these entities is listening, please get in touch. We would be happy to have them as sponsors, obviously. And I, I totally feel that way as you described that it's great that we have all these things that are open source.

I think that's a fantastic, amazing development that happened. The fact that all of the most important software on your computer today You can read the source code for it, modify it and publish your modifications. That's fantastic, but We can take it a little bit further than that, you know there are other things that matter too.

And it, I think it does matter where all the, this huge pile of money comes from. And it's kind of like this elephant in the room in a way it's been like that for a long time. I feel in the browser industry that there's been this browsers being developed by hundreds or thousands of engineers, and there's a huge pile of money.

But we don't really talk about like what that money comes from until recently when it's become much more public knowledge and people are starting to see like how much their search search queries are actually worth to these companies. Right. So yeah, that's been really, I think that's been really great.

And it's a really healthy thing that that information is out in the open so that people can start to maybe care about this in different ways other than only like, is it open source or not? There's like other, other parts of a spectrum here that we can care about. Yeah.

Aaron: Yeah. I, I kind of sarcastically.

wonder, like, you know, why did it take so long to get dark mode? And it was because it took that long for someone to figure out the, the, how to monetize or not how to monetize it, but the monetary value of dark mode, like, Oh, you know, we did these tests and we realized that when you have dark mode on, you spend an extra five hours a month, you know, in your browser.

And that means X, many more ads that you see. And that means, you know, so now we have to do dark

Jonathan: mode. And now we know why it's called dark.

Aaron: I, I have to have dark mode. I mean, my eyes aren't good anymore and dark mode really helps. And

Andreas: that's true. So it's

Aaron: like, do you have dark? Let's ask the question. Do you have dark mode yet?

Andreas: We have something like dark mode. So CSS has like a preferred color scheme properties these days. So websites can advertise that, like if the user prefers dark mode, then the sites should look this way. And otherwise it should look some other way. The problem is that there's a large portion of the internet that doesn't specify how it should look like in dark mode. And they kind of assume that the background will be white by default.

The text will be black by default. And if you start to violate these assumptions, then you break some content. So that part is messy. But we, we do have sort of state of the art dark mode. Yes. And I hope that it's something that will evolve further so that even older websites can display consistently in a dark way.

But it kind of depends on heuristics at the moment where we sort of have to just take a guess at what would be a great way to dark modify this website. Right.

Jonathan: I see on the website you talk about things like having a runway of funding and some other sort of business esque terms. And I'm curious, is this a business for you guys? Is it more like a non profit? Like, what does that side of it look like? What's your thought process there?

Andreas: It is completely a nonprofit.

We have no intention of ever selling anything other than maybe a t shirt and a coffee mug or something. But that's a nonprofit. We are a 501 C3 registered in California and we take. What's it called unrestricted donations only so You you can't give us money and tell us what we should do.

You can you can just sponsor and trust us to to do what's right and You know that that does Exclude a certain type of sponsor from ever wanting to support us, but it's okay. You know, we don't, we don't need all the money in the world. We believe, and I believe from experience that a small team can build a competent browser.

It's just you have to focus and you have to be more selective about what features you do. And. Stuff takes time. There's real elbow grease involved, but it should be doable. And yeah, on the sort of business side, there is no business model. It is either we get this thing funded by donations and sponsorships or it falls apart.

And it's a, you know, we're kind of going out on a limb here taking a chance, but we're hoping that We'll be able to deliver something in an alpha version that will make people see, okay, there's something real here, something worth getting behind and sponsoring so that we can actually have this thing for, for our species for and if we can, if we can get it to that point where people see that this is something worth sponsoring then we think that we can continue it and, and perpetuity funded in that way.

That's at least that's what we're hoping to achieve.

Jonathan: What does the community look like? Do you have people on the outside sending in patches? Obviously, bug reports. People come in and complain about things. You can't stop that if you wanted to. But, are you getting patches sent in? Are there outside entities?

Are there any outside businesses working on this? Saying, man, it would be nice if Here's the code to do this thing.

Andreas: Sure. Yeah. We have a fairly large community. So I think in terms of like active developers, we are seven full time engineers right now paid, I think. Oh, wow. That's impressive. So that's pretty good.

Yeah. And and they're all people I hired who were previously open source volunteer contributors. So it's, it's been lovely to be able to take and like give jobs to all these people that showed up and worked on, on Ladybird.

Jonathan: Absolutely.

Andreas: And it's, I would say it's even like one of the coolest things I've ever experienced was just to be able to do something for fun for a long time and then give people jobs doing it.

Yes. But yeah, so, so a bunch of us now are full timers, myself included. And we also have a, an open source community. I think we're usually maybe like 30, 40 active people like contributing multiple times a month. And then a long tail of people. Contributing either, you know, one once in a blue moon or once ever.

So, but it's, it's been growing slowly. And I think historically like working on browsers, as you mentioned, like it's really complicated to build Chromium or to build Firefox and because it's been a really complicated thing it's been a little bit hard for people to get into it, but Our project is fairly easy to get into.

It's a lot smaller than the existing browsers. It builds faster easier to, to learn. So we're welcoming new developers all the time who are sort of working on their first browser project ever. And that's been really positive as well. So I'm hoping to turn them into more frequent contributors.

And I'm also hoping that we will be able to fundraise a little bit more so we can hire a couple more contributors to be employees and But yeah, you mentioned business lingo on the website about runway. And indeed we are trying to be careful with, with the said runway because. We recognize that our funds are limited and the classic startup thing to do would, I guess, would be to hire as many people as we can right now and burn the money for the next six months and see what happens.

But that's not really compatible with our view on how this should be handled. So yeah, we're holding ourselves to a strict, like there has to be two years of salaries in the bank. At all times, because or we should aim to have that before we hire anybody new. Right?

Jonathan: Sure.

Andreas: And if that slows us down, then it slows us down.

But I don't want to, like, when I give somebody a job, I feel like I should It's my responsibility to make sure that that person has that job for a longer time. Then six months and us just burning through the Capitol to, to go faster.

Jonathan: Yeah.

What does the leadership structure look like? Are you a BDFL?

The Benevolent Dictator for life? Or maybe not for life?

Andreas: I don't know. So I was that. I've been a BDFL, but evidently not for life. I was that of SerenityOS until we forked, and then I sort of transferred ownership of SerenityOS to the group of maintainers that That I had previously invited to, to, to do that.

And and Ladybird, I think I'm not really the BDFL. I'm just the the president of the nonprofit that runs the project, but really the project is run in practice by the nonprofit and by the people that work on it, and then Code contributions are sort of quality gated by a group of maintainers, but there isn't any formal structure outside of that.

And I think it's something that will evolve and formalize a bit more as we get closer to, to releasing stuff. Or as, as our organization structure grows, but at the moment it's very like flat structure. Everybody's welcome to work on everything. Nobody is like.

We, everybody just kind of finds the thing that matters for making the web work in the browser and then they focus on that. It's possible that we will change that. We're kind of in a luxurious period right now where you can sort of like you, you, you throw a bug fix anywhere in the browser and it's, there's a real chance that it fixes some important website.

So And the future is going to be, you're going to have, you're going to have to look harder for valuable things to fix. But yeah.

Jonathan: Yeah. Interesting stuff. Aaron, did you want to jump back in? We're getting close to the, close to time. You want to make sure we're getting

Aaron: close. Like, I guess I don't want to jump too far into the weeds with this one, but maybe I'll just open up the can of worms anyway, and we'll see where it goes.

Jonathan: Just crack it. Let the, let the worm air out and not near the worms.

Aaron: Yeah. Sometimes these questions go in, in, in weird directions, take a long time to answer, especially for novices like me. But the question is architecturally speaking, like, what would you say are the biggest differences or even like some of the things that you, that you're seeing maybe some indications of like, this is going to be a game changer.

In terms of technology and how we're developing this, because we don't have to conform to. whatever chromium webkit, but we're able to do it this way. And that makes a big difference.

Andreas: Right. Architecturally, what are some advantages? I guess one thing is that we have a lot of flexibility right now because we're not big and complicated yet.

And there are a lot of features that we haven't implemented yet. And so we don't have like a gigantic code base that has to be retrofitted to do. Some some like security mechanism, for example. So like every other browser, they started out as a single process browser. But we started building our browser after multiprocess browsing was a thing.

So, we were able to get that stuff in pretty early. And as a result, we don't have a code base that, I mean, we were single process originally, but we were able to make ourselves multiprocess pretty quickly because we didn't have a gigantic code base with millions of lines of code that had been shipping in a single process way, you know, for decades.

And then we had to retrofit multiprocessing into that. And I, I think we have a lot of opportunities still to do interesting architectural things for, for security, for stability, for performance that are much harder for other browsers because they have so much code that, that just has to be changed in complicated ways, let's say.

Yeah, that's, I think that's our biggest opportunity.

Jonathan: Yeah, pretty cool. Why, why C Why with a new browser? Did you not write it in Rust or Go or some other everybody's favorite language?

Andreas: Right. Well, I guess there's no language that everybody loves, but everybody loves to hate C lately. And we started, I started SerenityOS in C because as I mentioned, there was a, it was a personal therapy project.

So I just. So the language that I knew the best and was just doodling around. And then I didn't mean to, but, you know hundreds, thousands of people ended up wanting to work on it also. And then it kind of just you know, I got a little bit out of my hands. And now we have this gigantic code base written in C plus plus.

And we have a lot of things going on and we want to find a way forward where we can You know, feel like we're doing our best by our users in the future, that like we're doing our best to deliver something that's safe and secure. So in practice, that means that we have to, we probably have to evolve past C in some way.

Because C is not evolving towards safety as, as fast as we would like. And so I don't know. We've looked at a bunch of the different languages recently and ended up with Swift as a secondary language to introduce into the code base. Not everybody's favorite choice, but we did this experiment where I asked people like, please try to implement some part of the browser in a couple of different languages and then tell me which one you like the best.

And. Everybody came back liking Swift the best and fair enough. It was a, it was an empirical process and I had the same experience. I liked it the best out of the languages that we messed with. And so we are aiming to introduce that now with the new version of Swift that is coming out in the next couple of weeks.

And the idea there is to have a safe language that we can incrementally introduce into the code base because Swift can talk to C and vice versa. And that's kind of, that's kind of our plan there, but it's going to take time to do that. But it's something that we feel like we have to do something because.

You know, safety is a thing people do want to hack your browser and we probably have more bugs than we know. I'm sure we have more bugs than we know. And if we could like systematically prevent many of them by using a safe language, that's something that we, we definitely need to pursue. But yeah, we are still like on square one with that because we're depending on the next version of Swift because it's the first one that can actually understand our super modern C that we've been using.

Yeah, so that's kind of where that's at. And I know that there are a lot of languages that people always ask me like, why didn't you use my favorite language? Or why didn't you use this language? And I just never engaged with that because I feel like Nothing good ever comes of, like, criticizing somebody's favorite language.

Right?

Jonathan: Yeah. Yeah, that's fair. Have you, have you been in So first off I've got to think there there are people out there to go Swift they went with Swift. Oh, they're selling out to Apple. That must be what that is Yeah, you've gotten much pushback from that.

Andreas: I Have been told by many people that like congratulations on the Apple money

But No, we haven't the the We were acknowledged in the sense that I think the the head of the DevTools department at Apple tweeted at us saying something like, cool. But that was it. Yeah, no, we're not, we're not selling out. It's just, it just so happens that Swift has a really compelling story as like a C plus plus successor language.

Like that's something that they're investing in. Yeah. Sort of this narrative of like, Hey, do you have a huge C code base and wish you could be safe, but it's just too much to just rewrite everything? Here's Swift. You can rewrite incrementally. That's really compelling to us. So.

Jonathan: That's interesting from the kind of the standpoint of looking at the language too.

This is something we're seeing with Rust as well, where they're putting Rust in the kernel. It's forcing Rust to grow up. And. If they gain traction with that, with Swift, trying to get other C developers to use it. And maybe Ladybird will be part of the story too. Putting it in use, they're going to find places where, oh, this thing in Swift is not working quite as well as we thought it did.

Or there's a bug in trying to make this work. I think that could be really interesting going forwards, as you guys I'm assuming you will try to, you know, kind of work closer with the devs from the Swift language. And it should be a good thing for like both projects.

Andreas: I think so. We've already found and reported a number of bugs and have had a good experience interacting with a Swift team so far.

So that's, that's really positive. And I think it's absolutely the case that, you know, whenever you take a language and bring it into a new domain where it hasn't been tried before, you're going to find tons of interesting and. Not interesting problems. And we've had a lot of not interesting ones also.

I will mention like trying to get CMake to understand how to build.

Jonathan: Oh

Andreas: it's not very interesting.

Jonathan: No, it's just pain working with compiling tool changes. Just pain.

Andreas: Yeah. Thankfully though, we have somebody who, who derives. Some amount of pleasure from, from working with build systems. So I'm really grateful that we have Andrew who's been figuring that out.

But yeah, no, we're, I think a lot of good things will come with that. Just like you mentioned with Rust and Linux and everybody has to grow up a little bit, everybody has to compromise a little bit But usually something good can come out of that.

Jonathan: Yeah.

Andreas: I think it will be the same for us.

Jonathan: Absolutely. Is there anything coming down the pike that you are particularly excited about in Ladybird that you want to want to plug, let folks know about?

Andreas: Well semi related, I'm running a a coding jam this weekend called Browser Jam.

Jonathan: Perfect.

Andreas: Where people are invited to come and we're all gonna write a new browser over the weekend.

Oh. And I don't think it's ever been done before. People do game jams where they make games over a weekend. And I We thought that we could do a browser jam where everybody makes a browser. So the idea is, you'll show up on Friday afternoon, and we will give you a piece of HTML, and then you have a weekend to build a browser that can render that HTML.

We'll see how that goes. But if you're interested, you can go to github. com slash browser jam, and you'll find the information there. That's great. That's a lot of fun. That's

Aaron: kind of fun. I mean, you know, that's what, I mean, that's what I like about doing that kind of stuff with HTML. ESP 8266s and 32s in the Arduino, like, stack just to like, you know, you don't need much there, right?

Kind of like the project you were talking about before, Jonathan, where I just need to read an HTML page and maybe some little JavaScript and make it work, you know? And if you can get to that point and understand how it all works, that's a great learning project. I think it's

Jonathan: Absolutely. It may be early in the project for this question, but is there something that someone has done with Ladybird that just really surprised you?

Like just off the wall or odd or, or other otherwise surprising that somebody has used it for?

Andreas: It might be a little early for that. Yeah. But I am frequently surprised whenever somebody gets some really complicated website working. I am just so surprised that we're already at the point where we can like do Facebook or we can Log into YouTube or things like that So it's more like i'm just continually surprised at the progress that we're making it's been really inspiring and especially because i'm Maybe not.

Yeah. We don't have a lot of people with previous browser experience on the team. It's me. And then a couple of people have been contributing a little bit here and there, but most of us are like complete noobs at this and just sort of learning as we go. And the fact that relatively or very inexperienced team has been able to assemble a browser of, of even this quality level in this timeframe is I think amazing.

Jonathan: Yeah.

Andreas: And I'm really proud of, of the team and, and it's been awesome to get to see people grow. Like we've had people join the project when they were like 16 and they're now in university and still hanging out and chatting and like fixing stuff. And that's been a, it's been a lovely process to witness.

Jonathan: Yeah. Question from the chat room, mashed potato wants to know, is anyone live streaming the browser jam event? Is there some place where we can watch people code in real time?

Andreas: That's possible. I don't know. So we're, we have a discord server where people are coordinating a little bit, making teams and stuff like that.

And I saw some people were talking about the possibility of live streaming. So if you're interested in that join the discord it's on the github. com slash browser jam. And yeah, see if, if somebody there is streaming, hopefully somebody will.

Jonathan: Sounds fun. We are, we're getting down towards the very end of the show.

Is there something that you, you wanted to mention to folks that we didn't ask about? Did we miss anything?

Andreas: No, I think you covered a lot of the stuff that's important. And I'm glad that I'm glad that we talked about the issues that exist in the industry. I feel like it's easy to, To forget to acknowledge that, but it feels like we're in this new reality now where we acknowledge that browsers have been funded by Google and it's no longer a secret that only some people know about, but like it's something that everybody understands now and we can start to look.

Real solutions to, to that. And I think we're trying our best to offer one possible solution. And we would be very happy if, if people experiment with other solutions. I don't think that Ladybird has to be the only new browser. I know that there are some other. Up and coming browser projects as well.

There is the Servo project they're doing browser engine in, in Rust. Mm-Hmm. . And they're very big on sort of the embedded use case. And I'm glad that they're exploring in that direction. And I think there's room for more. I'm, I'm really hoping that there will be sort of a, a new age of new browsers.

I guess that would be really, really fun. As a, as a fan of browsers, I would love to see that.

Jonathan: Yeah.

Andreas: Yeah, so I guess shout outs to Servo for for also going down this path.

Jonathan: It, no, no, that you mentioned them. I'll have to see if I can reach out and get somebody from Servo on the show too. Cause that would be a lot of fun.

All right. So before we let you go,

Aaron: one thing before you drop, what's the best way, easiest way for people to get involved or the best way to get

Andreas: involved? Right, of course. So the easiest thing you want to start with going to our GitHub repository. So it's at github. com slash ladybird browser and join our discord server.

That's really where all of the day to day coordination happens. People like working on stuff together or discussing bugs. It's fairly pleasant space. People are very nice, welcoming, professional. And as I mentioned, like we're welcoming new developers all the time, so you won't stand out or be weird, even if you don't know what you're doing.

Join our Discord and come chat. And what I usually tell people to do if they don't know where to start is to Build the browser, and then go to a website that you made yourself. Hopefully you have something. Hopefully it's pretty simple. And see if it works correctly. If it doesn't try to figure out why.

And see what you can learn from it. And maybe you can figure out how to fix it, but at the very least you can make a pretty good bug report even if you can't figure out what to do. Yeah, and then take it from there.

Jonathan: Yeah, very cool. All right. Last two questions that we are required to ask or the chat room gets mad at us.

And that is what's your favorite text editor and scripting language?

Andreas: Oh, well, it's, I don't know what my favorite text editor is, Vim. And I, it's just muscle memory, I think, even if I, I've tried to install other text editors, but I never used them because to start a text editor, I just type Vim. So there's that.

But I, I tend to program in IDEs. So like I do like in practice, a lot of my text editing of source code, I end up doing in like the JetBrains IDEs or VS code or something like that. But on the command line, it's always Vim. And then scripting language. Oh, scripting language, right. Probably has to be JavaScript, just because I keep working on it every day.

And, or no, I'm going to tell the truth. My favorite scripting language is PHP. And I'm not ashamed.

Jonathan: Well, having a background as a C programmer, PHP is kind of friendly towards the whole C style of bracketing and all of that. So that's not terribly surprising.

Andreas: Yeah, true, true. But it's terribly out of fashion, I feel like.

Jonathan: Oh, it is. It is very much out of fashion. But PHP still holds a place in my heart as well.

Andreas: Yeah, it's like, it's like Perl, but slightly more modern. Just slightly. Tiny bit. Tiny bit. Yeah.

Jonathan: All right. Hey, this has been awesome, and we sure appreciate you being here Andreas, and thank you. Thank you for your time. Thank you for telling us about the project. Yeah, thanks for having me guys. Yeah All right.

What do you think?

Aaron: Yeah, it's pretty cool. I love learning about these projects early on, you know, and having the opportunity to kind of get in on the ground floor, so to speak. You know, I often, I often tell When they ask people, when people ask me, what, what is Floss Weekly? You know, I have to explain it.

I'm like, well, you know, we talked to the Kubernetes guys, like when they were still at Google and the project was just getting started. Like that's a big part of what. Over the years, just organically, I think has happened because people want to come on the show and talk about their project when it's in the early stages.

So it just kind of works out organically that way. But the benefit for us and for listeners and viewers is they get to hear about these projects and get involved. You know, on in the kind of the early stages and

Jonathan: yeah, I think this

Aaron: is another great opportunity like we we We've just come to the point where everything has consolidated so much in the browser space that we just need another alternative if for no other reason than to have another alternative.

So we're not locked in. So yeah, I think it's a great project. You know, I guess stay tuned, right? Like either, either go try it if you're the type that. Likes to build things, build your own software, then go try it. And if not, then wait a little bit and come back and check it out in, you know, another six months or a year when there's a beta version or something.

Jonathan: Yeah. You know, I was thinking as we were talking so much of computing now is just the web. Right. And so we used to talk about, you know, your favorite. We still, we asked everybody that's on the show, like, what's your favorite text editor? We do so much work on the web. Maybe we need to start asking people what their favorite browser is.

But to kind of carry that, that thought through, you have different text editors for different purposes. And I could, I could see a future where you have a bunch of different browsers for different purposes that, you know, are not all just reskins of Chromium, like we started the show talking about. So yeah, no, I think this is, I think this is really cool.

And I really do think that there are going to be Some really interesting, maybe niche, but some really interesting use cases where it's going to make a lot of sense to run Ladybird or Servo rather than Firefox or Chromium. Yeah,

Aaron: yeah. I like the ones you came up where the, the one that you came up with, and I think that branches out into others where you need to run a kiosk or you need to run something and you don't want to be encumbered with maybe even the fear of the unknown, right?

Like who knows what. Chrome is going to do or Chromium is going to do how it's going to change. Right. You need something stable. You need something that's going to be, you know, you can count on. And yeah. And, or I just want to, I just want to run a kiosk, like you said.

Jonathan: Yeah, definitely something to be said for that.

All right. Did you want to plug anything, Aaron? I know you've got that, you've got at least a YouTube channel, which you'll mention.

Aaron: Yes. I will give you a preview quickly of an upcoming video. It is September. So I've got two channels. I've got the main channel, which is RetroHackShack on YouTube. And I've got RetroHackShack After Hours, where I do e waste Wednesday and a bunch of like smaller, like maybe slightly less interesting to the broader audience kind of things all around vintage computing.

But on the main channel, what I've been working on for months, people don't know behind the scenes how long these history video takes, history videos take, but I'm working on a history video. It'll be relatively short, but on the history of the Tandy

Jonathan: 1000.

Aaron: So the Tandy 1000 was kind of. They kind of fell into this position of being a major player and having a really successful product because of the failure of IBM's

Jonathan: PCjr.

Aaron: So it was really interesting history where Tandy came along and they said, Hey, we're going to make a PCjr competitor. And then PCjr got discontinued. And they just happened to have like all the features that people wanted at that point and ended up being a super huge important and for a lot of people either their first computer, their first family computer, you know.

So that video should be coming out I'm hoping by this weekend, fingers crossed, but they just, those history videos, you have to go in and get credits for all the images and all that kind of stuff. And it just, it just, it just takes a long time to do. So yeah, be on the lookout for that.

Jonathan: And it's RetroHackShack and RetroHackShack After Hours.

Aaron: Yep. Yep. RetroHackShack everywhere. Just search for RetroHackShack on your social things or on YouTube and you'll find it.

Jonathan: Yeah, awesome. Alright, I do want to let folks know that next week it's, we're back to Java! And it's JBang, which it lets students, educators, and professional developers create, edit, and run self contained, source only Java programs with unprecedented ease, which, Sounds like we're doing Java for scripting, which that'll be really interesting to talk about looking forward to that.

So that's next week. Make sure and catch that you can follow me. Of course on Hackaday, we've got, well, that's the home of Floss Weekly. That's also where my security column goes live every Friday morning. There's also the untitled Linux show that's over at twit. tv at the twit network. And we have a lot of fun there talking about what's going on with Linux, the news, some open source stuff there as well.

And then. You can also keep an eye on my YouTube channel. You can search for me. I think it's Jay Bennett at YouTube. And some fun stuff mainly around Meshtastic, which we're about to have a big 2. 5 release the, the first alpha of that come out as some really exciting stuff there. So check that out.

We sure appreciate everybody that's here, both that watch us and listen to us live and those that get us on the download. Make sure to come back next week and we will see you then on Floss Weekly.

This week Jonathan and Aaron chat with Andreas Kling about the Ladybird, the new browser in development from the ground up. It was started as part of SerenityOS, and has since taken on a life of its own. How much of the web works on it? How many people are working on the project? And where's the download button? Listen to find out!

- https://ladybird.org/

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey, this week, Laurie LaRusso and Steve Hoffman join me to talk about Percona, the open source database solutions company that after all this has managed to keep open source, we talk about how and why you don't want to miss it. So stay tuned. This is Floss Weekly episode 799. Recorded September 3rd, still open source at Percona.

It is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett. And today we're talking about Percona. It's an open source database software. It seems like anymore. It's one of the few opens big open source database solutions that still actually open source goodness.

I think we're going to talk about that just a little bit during the show today. I, I don't know much about Percona. I, I kind of have to admit my database experience is sort of just My SQL and SQL light and a tiny, tiny bit of fighting against Microsoft SQL. It's been, it's been something because of because of things happening outside of our control, I am the host and co host today.

So that's going to be just a little bit different, but we've done it that way before. It's, it's not going to be a big deal. But let me go ahead and bring our guests on today. And we've got Lori Lorenzo and Steve. Hoffman and we talked a little bit before the show and Laurie, you are the, the community manager and Steve is sort of the, one of the VPs of, of technical.

Is that, is that about the way that, that lays?

Lori: Yeah. So just, my name's Laurie LaRusso, like Karate Kid LaRusso and yeah, I'm head of community at Percona and yeah. Yeah.

Steve: And I'm, I'm one of the engineering managers or engineering leads here for Percona run the the development side of the house. So, Loray and I are joined at the hip.

Jonathan: I, I understand. I know how that goes. So, let's, let's talk for just a minute about, I guess the technical side is where we'll start. And, what, what is Percona? Like, how is it different than, you know MariaDB? Or any of the other database solutions that are out there? What's, what's the kind of the, the thing that makes the difference about Percona?

Steve: So it's not wildly different than MariaDB's original model. We both took a fork of MySQL and, you know, added our own features to it. What we think, you know, was the, the answer to the needs of the market, you know, from an enterprise standpoint. We've also since done that with MongoDB and we're also working in the Postgres space as well to bring new features on top of an already great base.

So in that sense, not wildly different. I think. You could say we're similar there, but then I think we go a lot further than them. So we also have our Operators product. We also do our, our monitoring and management suite. So maybe we have a wider portfolio, but I think it's fair to say MariaDB model is similar to Percona.

Okay. And

Lori: so this is where, let me just jump in. This is where I think Steve is being humble and that's why you have a community person to back him up. So, you know, I think the, what you're missing is the innovation, right? So we have community editions of MySQL, of Postgres but what we do We make them better, and we're open source.

So, we take, we take the regular edition, we open source it, and we add in our little spice to make it better for users.

Jonathan: So is, is Percona a database? Or is Percona a database company? Or is it both?

Steve: We like to call ourselves a solutions company. We produce database software, and and you can get it in all different formats, but we make the majority of our money on the services that we provide.

But when you take our software and our services and sort of smash them together, you get sort of the overarching Percona solution.

Jonathan: Okay.

Steve: You don't have to use both, but I think that's where you get the best value is our software with our services. We can, we can help you run it better than anybody else.

But if you're using, let's say a community edition. Of either Mongo or Postgres or MySQL. We can still service you but it's just, you don't, you don't get the full value. Okay.

Jonathan: And, and, I, as you, as you work on some of these pieces of software I'm sure, so you make software changes to them, you make changes to the source code, because it's source available without being an open source.

Do any of those changes ever get pushed back up to the original project? Or do they kind of all stay siloed in your, I mean, obviously it's open source, but still, you know, there can be either an effort or not an effort made to push them up.

Steve: Yeah, I mean, we like to contribute everything that we do. Now, we have learned over the years that certain things aren't really necessarily welcome or valuable back upstream.

In many cases, we're actually building enterprise features that already exist in the upstream's enterprise product. So, us contributing that backup there, you know, they don't want that. But we do bug fixes, we do, you know, extensions and enhancements all the time. And again, we do it all, make it as open as humanly possible.

Direct contributions. You know, and hope that we get it included in, in upstream so that it just everybody gets to benefit from the change.

Jonathan: Yeah, I mean, every, every change that you can make upstream is just one, one fewer change for you guys to have to be the stewards of. That makes sense. That's, that's kind of interesting that you're re adding some of the enterprise features that have been not added intentionally.

Is there, is there any animosity from any of the upstream vendors because of that?

Steve: I think on the community side, there's no animosity. I can't even say that I'm aware of it on the enterprise side. I'm sure, you know, if Percona got way too big, people that haven't taken notice certainly would, but we seem to have a great relationship with the MySQL community, certainly the Postgres community.

And Mongo, you know, I mean, I don't think we have each other as adversaries. Business wise, surely we're competitors, but I don't think it's. I've not received any death threats, I don't think any of you on the team have, so

Jonathan: we're good standing there. You've managed to remain friendly competitors, that is handy.

Yeah, exactly. That's fun. Well, but the database world seems to be sort of cutthroat these days. We talked just a little bit before the show about like all of the different licensing changes that have happened and forks because of it. And it seems like, well, no, I know one thing that has happened is you'll have a company that they have a database, like they have their secret sauce in their database.

But instead of making it all secret, they've made it open source. And so they've got this database code that they work on, and then they also offer it as a service. And like, that's great until Amazon comes along and says, it's open source. We can offer this as a service too. And suddenly this, this, this business that does open source, like their revenue stream starts to die, dry up.

And they immediately go, we've got to make some kind of change to be able to keep making money. And it seems like in a lot of cases, the change they make is we're going to use a business source license of some sort. And. It, it, it, well, it fractures the community. It, it inevitably results in a fork, you know, like Redis and Valky.

We, we now have Valky because Redis did the exact same thing and they, they, in their wisdom, they made this decision and they made the announcement to you. Move away from an open source license with Redis during a, I think, a Kubernetes conference. And so there were a bunch of people that were using Redis all together in the same place when this was announced.

And apparently by the end of the conference, they sort of walked away going, Yeah, we have a gentleman's agreement to make a fork and call it, you know, all get on board with the same fork. But it's just it's got to be it's got to be cutthroat.

Lori: Is it a

Jonathan: cutthroat environment? Have you seen that?

Lori: Let me take your, your Amazon example for one.

And I think that's where you come up with strategic alliances, right? That's where you really work within, I mean, it's a cloud company, right? It's a hyperscaler. So if they can. That's what the marketplace is for. If they can package up what we do and add it to what they're doing, I think that's a win for both, right?

So the way that we treat our core customers, the level of service that we provide is super unique. So to have a place for us to, a platform for us to list out like our version of MySQL or Postgres, I think it's, it's a win for both. And then when you think about how the community kind of rallies around, you know, open source and wanting to keep things open I think For me, the biggest shift I saw was when Terraform went closed and there was a manifesto.

The community was so irate that they came up with a manifesto. And then from the manifesto, like they ended up The, the name of the project now is OpenTofu, but it, it started, I think it was OpenTF and it's underneath the Linux foundation, just like Valky is now underneath the Linux foundation. And I think when you have something that's open, that people use it and adopt it because it's open and they build their businesses off of it because it was open.

And then you close it.

Speaker 4: Yeah.

Lori: Then you have like this big community uprising, specifically if it's something so popular, like Terraform or Redis and to be a company, to work for an open source company like Percona that is there to carry the open source flag and say, okay, so Val Redis closed source, guess what, we're going to be a part of Valkyrie.

Like we're the only ones that offer, you know, like on prem support for that. And in that look at the other companies that have joined Valkyrie. You've got your AWS, you've got your Google, you've got your Oracle. So it's like. All of us working together to have a piece of this pie, if you will, in open source land, I think it's, it's, it's really awesome to be a part of a company that supports open source, that it's that core to who we are.

Jonathan: And so is it fair to say that Percona's business model is not simply hosting databases for companies? There's more to it than that.

Steve: Yeah, perfectly fair to say. What's the, our model is actually you hosting your own database. So we give you the software that lets you. Run it on whatever your infrastructure is.

So if you want to run it on prem, if you want to run it in the cloud, if you want to run it across multiple clouds, we, you know, we actively support and test all of those scenarios that's given the power to the, to the people. Yeah, yeah. And

Lori: I, and I think what we do is we help you level up, right? So we have not only the innovations that are coming out of engineering, but we have the experts in our support and our managed services.

So if you are trying to elevate, if you are trying to do something and you realize you don't have the staff or the support in house, that's where we come in. So having that, that like hand to hold you through a migration or to help you tune your database so that you're getting the best results to take your database and put it on the cloud with our new product, Everest, like just because we're open source doesn't mean, you know, that we are a less than we are actually like a

Jonathan: No, absolutely.

I agree with that. I, I think part of the problem is that so many companies, they, they love the open source model, but they've not really figured out how that model actually ties into their business model in a way that makes sense. And sometimes they end up sabotaging themselves. And it, it sounds like Percona has managed to avoid that.

So that's, that's good. That's a good thing. That's a good, good sign for the future. We just celebrated

Lori: 18 years. Oh, wow. Right, Steve?

Steve: Yeah. Yep. 18th birthday. We became an adult. Percona can now legally Drink and phone? Do what 18 year olds do. Well, just different in every country as we learn. We did a poll internally about turning 18.

It was like, wow, it's amazing. Globally 18 means different so many different things around the world, but that was a lot of fun

Jonathan: What was what was the beginning of the company? What was the what was the first thing that percona did was it was it literally my sql stuff My

Steve: sql

Jonathan: phone support. Yep phone support really Ah,

Steve: so so our our our co founders were actively taking calls from customers to help optimize their databases.

So they were a consulting company. So if you think about Percona, it's a per con performance consultants and dot com was taken. So per con a dot com came to be. So yeah, little, little trivia history on our name, but yes, Peter and Vadim took many, many phone calls. I think, I don't know the exact date, but we have a book.

That they put out at about the 14 year mark, or maybe it was the 13 year mark that said, like up until at the time, eight or so years ago, they were actively taking phone calls from, from customers who were having performance issues. Fortunately we have enough staff, we have, you know, smart enough people that they don't have to anymore, but that's how we got our

Jonathan: start.

Oh, that's, that's great. That's, that's a lot of fun. And, and so what's the, like what's the portfolio now? What, what are the different databases that you guys will. Still take support calls for her.

Steve: Yeah. Yeah. I mean, we're, we're active. Like I said, my SQL is what most people know us for MongoDB, probably second.

But then we, you know, we're very active in the Postgres space and getting more. So every day and then from there, you know, where you choose to run it, we support, like I said on prem bare metal installations, cloud installations. We create operators now, so you can run this in Kubernetes. So in your orchestrated environments.

Which we're seeing a ton of traction there. We're now releasing you know, if you think of Amazon's RDS is sort of like database as a service, you know, point, click, get a database. We have an on prem equivalent to that in our Everest product. So it still uses Kubernetes and work and operators underneath, but it gives you that that single API that you can call and wire into your CI CD or.

Or, or turn over to your, to your developers and let them point, click and create databases as opposed to having to know all the, you know, behind the scenes commands.

Lori: And you forgot Valky. So Valky is our latest database that we're going to support.

Steve: Yeah. And Valky, we are only, we only support it. We don't actually build our own version of it.

We contribute to it. We're, we're active in that Valky community, but there's not like a Percona fork of Valky. So Valky, we, we're trying to. Support that community not compete with them by any stretch,

Jonathan: right? And so i'm sure you could you could imagine a future where You have a slightly different take on something and you do end up having an in house valky fork.

But I i'm sure trying to avoid that if possible, right? Yeah.

Steve: Yeah, I can't imagine that future I don't really see it. I don't ever see it as good right like I mean it it If we're really community, right like and we mean it then We find a way not to fork now. I can't fault any of the projects that have had to go, you know, go their separate ways.

I know there were certainly valid reasons, some of them personal, some of them technical, some of them vision, but I don't know that that's always the best outcome. So for me, that's like the last thing. If we've hit that, hopefully we've exhausted all options because, you know, to take to take a thriving community and effectively split it in half It's gonna slow, slow innovation down, right?

That's one of the beauties of open source is just how fast we can innovate when, when all of us are focused on the same thing. So again, I'm preaching here. This is just my personal belief, but yeah, I don't, I don't love that, you know, direct competition with each other. If we, if we Percona fork, first and foremost, we want it to be to help innovate and add back to the community.

But in some cases when that, you know, when that, when that takes its turn, like, you know, for us, we do have a fork of MySQL. Oracle Enterprise isn't going to accept our LDAP authentication. They have their own, right? They don't need us, but, but we do feel that it's good to bring to the community. So I think, like I said, we do it for a slightly different purpose.

Jonathan: And so, for example, we've talked about Everest a couple of times. So that's, that's fully open source. If, if somebody wants to just host their own version of Everest and not have to pay you guys anything for the contract, that's out there as an option. And then your, your business models, essentially, this is a complicated thing to do.

We suspect that eventually some of these businesses are going to want to have support for it.

Steve: Yeah. Yeah. I mean, that's been our model is if you've got the manpower, the time and the talent to do it yourself, you Our software is free and open source. If you don't, we can certainly help with that. You know, that's been our model and that's fortunate for us that our business model has been successful enough that we've been Continually growing and continually profitable company since the beginning.

Jonathan: Yeah, well, so I mean that that business model has worked for other companies over the years That's essentially what red hat was founded on and it it grew them to be a billion dollar business so there's there's definitely nothing nothing wrong with that And that is that is proven out to be a a winner a winner in many industries at least yeah, that's very neat.

So I'm, i'm curious in addition to that. Do you ever get someone say You Call you up and say, Hey, it would be great if my sequel did this. It would be great if Postgres had this feature. Can you give us a quote on adding that to the code base? Does that happen?

Steve: That's

Jonathan: only on the days that end in

Steve: Y. Yeah, we get, we get that a lot.

Like every customer has a unique need. You know, that. This is the way our business works. We need this feature, you know, and we have a product team that does the evaluation. Is this solving a problem exclusively for you or is this a problem that many, many people are experiencing? And by creating it, we can we can benefit the broader communities.

We try to stay on that, that, you know, community based side where we're looking for large swaths that would be impacted for the products. And that's not to say that we don't do some, you know, custom stuff. We have some customers that have specific needs. If the ask isn't huge. We put it in there. And there's a, there's a monetization model for that.

I mean, sure. But I

Lori: think that is that is the value of having community live under engineering as well, right? It's this feedback loop. So as many customers as we have, as many conferences as we go to, being able to have the pulse of whatever community we're in, if it's like an all things open, it's just a general open source.

Speaker 5: You

Lori: know conference, but if it's a PG comp, we can really focus our attention to that particular language, that particular particular database. I'm sorry. And, you know, really work within that sort of feedback loop. What are we doing? Well, what are we missing out on? Like here is a talk that we're giving that talks about why we decided to do this, you know, and And I think that's where community wins, right?

Again, Percona being an open source company and community has always been at the forefront of what we do. And then being able to really like zero in on the events that we go to and having like a forum where you get answers, having a very good resources within Percona to give that knowledge out to the community is a win win.

Jonathan: And so that does kind of lead onto another question about that community effort. What, what does it look like as far as in, in these places where you guys are the, the, the source code, like you have your own. Your own repos for this. Do you get a lot of patches and work from the community? Or, or is it pretty much just a a one way street where you guys do the work in the community or you're out there using it?

I

Steve: mean, I can tell you we get a ton of contributions. And, and, you know, community contributions don't always come in the form of long lines of thousands of code changes. Sometimes it's improving our documentation. Sometimes we get a lot of community give back on our forums where they'll come in and answer questions for, you know, for the masses.

Like, how do I set this up? How do I use it? I'm getting this error. So we're very, very active on that front. That's one of the things that shocked me. I don't. come from open source, like, you know, way back when this is something within the last five years for me, but seeing how active our communities are is the exciting part because number one, that the entire burden doesn't fall to us because I think if it did you know, and maybe this is where some companies go wrong is you know, the community is not paying your bills.

So you tend to favor the ones that are but because we have such an active community, it allows us to work together. And, and accept code contributions, accept documentation updates, accept you know, answers on the forums, things like that.

Jonathan: Are there, are there any, are there any rough spots? So I'm, I'll give you the background to what I'm thinking here.

I'm part of an open source project and we just recently got sort of called out in our one of our, one of our places where people can ask questions. One of our forums of essentially you guys made this big change and you didn't ask us about it first. And it's, it's been a little bit of a thing because like there's, there's a couple of ways you can look at this.

For one thing, almost every open source project is a meritocracy to some extent. And so, one of the answers that we give is, if you guys would like to see changes, come write source code. Come be a part of Discord, where the, where the developers are talking so that you can give feedback. But it's really, it's really become a little bit of a source of friction.

There's some other things going on, but not important. Of course, it's always more complicated. Yeah, but I'm just curious. Do you guys, have you guys had any of that kind of friction and how have you solved it?

Steve: We, we have our own forms of it. I don't think we've seen that specifically, but with, you know, every, for the products, That we are in control, like PM and Percona monitoring and management.

We don't have an upstream necessarily beholden to, I mean, we do have you know, a copy of Grafana in there that we base it on. But, by and large, we're driving the, the roadmap. And a couple of times we'll deprecate a feature or, you know, we'll put something in that ends up not catching on and so we sort of abandon it or start pulling it out.

And so we definitely hear about it.

Jonathan: But I think I think it's impossible to deprecate a feature without someone coming along and saying, Yeah. That was part of my workflow. You know, it's the XKCD comic about holding down the spacebar, heating up your CPU. That was part of my workflow. Put it back.

Steve: Yeah. We get that, right? Like, but you realize we took that out because there's a better way to accomplish this. Right. And so most of the time we luck out with, you know, sort of a re education on it. We, we didn't Take something away without giving you something back. Here's, here's why. And, you know, it hasn't ended sourly.

I can't say that people are like, oh, thank you for that. But they, they at least walk away with an understanding of like there, there was a rationale behind it. It wasn't. Who can we piss off today, you know?

Lori: Also, I think it's a it's a matter of communication, right? Like it's a matter of giving people warning letting them know what's going on being able to have you know I don't want to say standard talking points but you know those standard talking points as to like this is why we did it and because you you don't want to Detract from the end reason, right?

Which is maybe there's a better way, or when we actually looked at our logs, nobody's using this in terms of like the hand scheme of things. So having a good communication and like Steve said, like having the forum as a place where people can converse, you know being able to kind of contribute back, submit issues that I think is, is key to keeping the community Safeguarded from some detractors.

You know, it's always the, the small group can sometimes make the loudest noise, but when you're just very transparent, like in the whole methodology of being open source, I think, you know, you end up just winning long term.

Jonathan: Yeah. I was, I was going to ask you, Laurie, to, to speak to this next, because sometimes the community people can be on the other side of the fence from engineering.

And if there's not, if there's not decent communication between those two teams, they can be actually working at cross purposes. What, what kind of things do you guys do to make sure that doesn't happen to make sure that the community team, so this is something that just recently happened to us the community team got surprised by a change that the engineering team made and it's like, Oh, that's gotta be a bug.

We're sure to get that fixed. And then the engineering guy is like, that wasn't a bug. That was an intentional change. Like, please tell us about these things. How do you, how do you, how do you avoid that? Yeah.

Lori: Well, I love that my dog just started barking. So I apologize. Hopefully it won't

Jonathan: be

Lori: too much in the background, but I'm very happy to say that at Percona community has moved under engineering.

At other companies that I've been to, community has always fallen under marketing and that's where you get the disconnect, right? Because marketing has marketing objectives and engineering has their objectives and when there isn't open flow communication across the board, Then you have those issues where you're like, wait, I'm presenting the wrong version of something or I didn't see the release notes.

Nobody told me. And it's it's a communication issue. And so I know, don't laugh. It's she's old. She's 14. She doesn't know what's happening. She's just like, why is mommy talking to a computer? So So now that community has moved under engineering, it's so much better because we are on all of the product updates.

We are looking at the roadmap. We are making sure that we are dropping breadcrumbs to innovations that are going to come out in like four or six months, right? We're letting the community know like why we're thinking the way that we're thinking, what our engineers are doing. And then with that, you know, we have, Surveys that are just product led surveys that are just like, we want to know what version you're using.

What is this? Like, how is this working out for you? Like, what can we do? And so keeping a really tight feedback loop like that is where I think community and engineering can just sing. So it's been a night and day from previous companies to be under the engineering org and showing them our value, right?

Like our tech team, our tech evangelists, our developers, our engineers, like they have. Like the history within their own career of touching source code, of building things, of doing testing and doing that kind of stuff. So we're also working to embed them more within engineering itself, like to be the ones that are helping to contribute, that are looking at pull requests, things like that.

So that I think is like, it's, it's new for us. And we're just kind of testing it out, but I'm very excited because the tech teams. are very like my tech evangelist are very excited to get their hands dirty. Nobody wants to be a showpiece like you want to be able to actually demo what's going on. You want to be able to speak from engineering and say, this is why we're doing things.

And that's what we're working on right now. And I think it's going to go really well.

Jonathan: Oh, yeah, that's interesting. Steve, what, what does that look like from, from, as I say, again, your side, your side of the fence, trying to bring your, your evangelists in and let them work with the code. Has that gone well?

Steve: Oh my god, it's better than you can imagine. Typically I would say, you know, we rely on community for the voice of the non paying user. Because we don't really have too many other avenues for that. And if you're not careful, your entire roadmap becomes the customer roadmap. So getting that continuous line of feedback from them, you know, of what does the community want is awesome.

And then even more so that they can be so hands on. And again, it takes all different forms. It's not purely contributing lines of code. Sometimes it's using it. Sometimes it's, you know, Hey, I couldn't talk about this at the next conference. This is too clunky. This needs some polish. And this is how I think it should look.

So, you know, the, the, the forms of feedback take all different forms, but they are wildly valuable. You know, you didn't realize just how much you missed it until you start getting it. You're like, wow, where would we be if we had thought of this. So much sooner you know, it, it, like Laurie said, it's been a fairly, not fairly recent change, you know, measured in six months or so, five months, something like that.

But it's been a great change and we're just keep building on it. So I'm excited to see it continue.

Jonathan: So is, is the next iteration of this to, to, to make sure that the marketing people know the source code too?

Steve: We might be asking too much. I don't know if they have an interest in learning the source code. At least the tech evangelists have an interest in it. And I'm not faulting the marketing folks. You know, don't ask me to design anything that looks catchy. I not a creative,

Jonathan: I get that. I just, I know I've heard stories of, you know, marketing people coming and asking just the most off the wall questions.

Like, could it be possible for us to do this? And, you know, engineering goes, I guess it's technically possible. And then a couple of months later, you see the latest marketing thing, this new feature coming soon. Yes.

Steve: Yes. That, that happens everywhere. That's every job I've ever been at that happens. That has nothing to do with open source though.

That's just human nature.

Jonathan: So it's, it's the marketing guys just trying to grab onto something new and flashy to put in. Yeah. That's great. Okay. So let's see. What, what does I'm curious, what, what does this look like then? So if you get somebody from the community that says, Hey, I've got this great idea, but they're not a paying customer, but you look at it and go like, that's legitimately could be useful for people.

Like what does the decision matrix look like for this to, to, you know, are we going to put some resources into making that happen?

Steve: Yeah, I mean, so I'm going to speak for my my product counterparts. You know, but but one of their big value ads is really assessing the overall market need. And it's not just, you know, take your idea at face value.

Is there enough people that might buy this? Well, that makes the decision, but it's Is there enough of an idea in your suggestion that might be able to expand it or might be able to take it in a different direction, maybe sometimes even narrow it. And that helps us get to a, Hey, we're onto something here.

There is a legitimate market need. And again, we do partner with the community team to make sure that we do vet those ideas before we just run right out and build it. Cause Lord knows that's expensive, but Hey, we're thinking of this thing. We got the feedback. It seems to be valuable. Let's talk about it at the next Pick your conference, you know and see what the feedback is if it's positive if you know if we hint that hey We're thinking about doing this and the feedback is oh my god.

Yeah, that'd be awesome. Then it just starts moving up and up the list You know, I think the biggest thing is Giving that community a voice and then listening to it. Not just, you know, here's a form you can fill out and, you know, black hole it. Right. Right. I

Lori: think one of the things, sorry, that I think that we're missing from this conversation that I'm happy to bring in is not just Percona related community, but community in general.

So Steve is my Guinea pig and I'm so happy to have to be on this podcast with you and one of the things, one of the communities that we are a part of is finos. And so FinOS is the financial arm of the Linux Foundation. And one of the things I was sitting on a call that they were looking for is contributions to one of their projects.

It's the something something control. I can't remember. It's three C's. It's CCC. I'm terrible. But it basically, they need help with, like, Expertise. And so Steve, I was like, I've, I've all told him that he needed to help me. And now we're contributing our sort of reference architecture for for my SQL in terms of like having a hardened, like security features for my SQL things you should look out for.

And so it's not just talking to ourselves or talking to our own community. It's how can we impact the broader community with our, our expertise and, and There's another project that we're working on it's called Gwok. It's in the OpenSSF, so it's a security project. Gwok is being built in Postgres.

One of my contacts was like, Hey, you work for Percona now, we need help with Postgres. We wanted to keep this open source project completely open. Can you help us? And it's like, Yes, of course we can. So we are now looking at their database as they're building out this security project. So it's, it's not just working within our own sort of Percona mindset.

It's what can we do at Percona to really elevate the community in general. So it's, it's a, it's a really exciting time to be at Percona because we are sort of flexing our, our skill set in other communities that maybe we haven't approached because You know, it's, it's not just the database, right? It's like, how does the database make your project better?

And so I think, you know, the next thing you think about is, is AI, right? And so there's all of these tools that are coming out and for us, it's, we don't want to be the next greatest AI tool, but like, Hey, what are your, what are you building out of you're building out of your database? Like, do you want to make sure that you are fine tuning your database that you're getting out of it, what you need?

So. Again, it's sort of like we are vastly more than just talking to ourselves, which I think is, is what sets us apart.

Jonathan: That's, that's an interesting point because, you know, Steve and I, we've been talking a lot about the, the, the community upwards to your upstream vendors and the community down.

Downwards to your users and Lori comes, it just brings along this idea of what about talking about the community outwards to sort of our peers and the other people doing interesting things. And that's something that we we don't think about quite as much. And that is, that is really interesting. And so what is the, what's kind of the motivating factor to want to reach out and work in some of those, some of those kind of vertical directions, excuse me, horizontal directions.

Lori: Well, I think again, it's like everybody has to store their projects somewhere. And if we can lend a hand in making your project successful, I mean, that is the value of open source, right? It's people coming together to build something, to innovate on it and to make it better, to solve whatever problem.

So for guac, it's an S bomb problem and where Percona might not really be interested in solving like S bombs, you know, like your supply building material, if we can say, Hey, in building Gwok, if you do this to your database, it makes it cleaner. It makes it better than at the end of the day, your project will be better.

Your project will be more successful. That I think again, is, is the value of going, To these other communities and showing what you do and how you can do it better. And when you think about foundations, like look, who's involved in foundations, you've got your Googles, your Amazons, your Azure's, you've got your Facebooks, you've got your Fidelity's, you've got Citibank, you've got all of these massive companies, you know, like what is it?

90 something percent of enterprise companies use open source components. So why wouldn't we want to share the love across the board with foundations that we're a part of?

Jonathan: Yeah, absolutely. I'm curious. Do you find that you're kind of your community? People have to sometimes do some translation work for your your engineering team.

And so what comes to mind with this is I do. I do a little bit of small business it for very small businesses around town, and sometimes they'll come to me with their ideas. The, the strangest asks, like, I need a computer with, and, and they'll say something that they saw from a magazine or on an ad somewhere, like, I need a computer that has one of those new NVIDIA workstation graphics cards.

You're a sprinkler system. You, you put. You, you put lawn sprinklers in what, and then, you know, so then there's this, okay, so what are you trying to accomplish? And then they'll tell me, and it's like, Oh, you don't need one of those really expensive GPUs. You just need, you need a new desktop with a new processor and a GPU in it that will run something with CUDA.

And then you can use your blueprint software. And so there's this, this kind of. Translation issue where they, they, they know what they want, but they're trying to tell you like the details of it and they don't understand the details. And I have to imagine that in the database world, working with, working with customers and even some on this, working with other foundations that this sort of thing happens to, we need a database and I don't, I don't even know databases well enough to tell you the sort of off the wall things.

I'm sure you guys get it though. People ask for off the wall things and it's like, well, let's help you actually come up with what this, what you really need.

Steve: If we're lucky, we can hear the, you know, what they're truly asking for. That's if we're lucky. We do rely a lot on, you know, the community team, the product team, our customer success team, some of our service, you know, services engineers.

Like, they are masters at hearing the problem under the problem that's being described. Right, right. If we're unlucky, we just start building based on, you know, I need this NVIDIA thing, you know. Yeah. And that's happened, well, once or twice, more than I care to admit. I'm sure. You know, I think then, then we circle back, like, okay, how do we miss this?

How do we get this out next time so that we're not just, you know, sort of chasing fantasies, but actually, like, getting to the core of the problem. Mm hmm. Because there's nothing worse when you build the exact thing that you were asked to build, turn it over, and they're like, it didn't fix my problem.

Like, what, what was the problem? Like, maybe we should have started there instead of just building the solution. But yes, yes. Yeah, I

Lori: love it. Because it all comes down to communication, right? Like, there have been many times and like, Steve can tell you where I've dragged. Somebody from engineering on a call with me, right?

Talking about community stuff and people asking for things that I have no idea, like I'm a marketing person, right? Like I have, I am not technical. I can try and translate to a point, but I need to pull someone in to be like, what, what, what exactly are they saying? Like, and then, you know, and then also like being able to take a step back to be able to stop a conversation and say, okay, hold on, like, Okay.

Let's talk about part a you've just went from A to Z. Like, let's just figure out what a is. And I think when you have a good team involved, right? When you have someone that can maybe take the marketing fluff and drill it down to the actual ask. And then you have the engineer that can take the actual technical side of things and and put it together.

You're gonna win. But when to Steve's point, when you just do one or the other, you could build something that's not necessary. And, you know, nobody wants that.

Jonathan: Yeah. Do you, do you get this with pull requests as well? I, I'm sure you must from time to time.

Steve: Usually you get the, yes, it is very, very specific to their problem.

Not, doesn't take into consideration how anybody else uses the feature. You know, but believe it or not, it's, it's less. Frequent than you might think more often than not, you get a, you know, very flexible contribution that's like, you know, this solves only my problem, but it makes extra effort not to step on anything else as a result.

So those, you know, those are the ones that we appreciate. But even the ones that are purely self serving will try to kick it back and say, Hey, you know, is there a way to do this a little bit more friendly to the other people that actually do want this to happen? And, you know, can we improve the. The error messages or the return codes or whatever the case may be.

So we try to, you don't want to tell somebody, you know, thanks but no, don't ever come back. So it's always to keep the conversation going. I mean, it takes an awful lot from someone to be willing to contribute code and databases. They're not easy. So if you find someone giving you database code, there's someone you want a relationship with.

So we do our best to keep that going.

Jonathan: Yes. Yes. It seems like databases and compilers are two of the really hairy problems in, in modern code. Not

Steve: for the faint of heart.

Jonathan: Yes, absolutely. Okay. Probably a question for Lori then. Have, have you guys had any real like pain points beyond what we've talked about with trying to wrangle the community?

I know sometimes there's things that come up that are just not pleasant to deal with.

Lori: So not since I have been at Percona, but I used to be the CDF, Continuous Delivery Foundation outreach chair. And in that there, it was kind of there was a wonky time, like they lost their executive director.

They got a new one. There was, you know, community didn't understand what was happening within the organization and the way that we, like, and there's nine or there's eight projects under the CDF and the, and the projects were wondering what was going on. And so what I did was I held feedback sessions.

Where I just let them tell me everything, the good, the bad, the ugly. There was lots of ugly. And it, it all stemmed from miscommunication or lack of communication and not understanding like what was going on. And, and from that, you know, creating an actual action plan moving forward, having everybody buy into it because they saw.

Bits and pieces of the conversations that they had within the plan itself, and then launching the plan and holding yourself accountable to the things that you said that you were going to do. And the change from when I started in June of 2022 to, we had CD con in, um, May of 23 was, or April of 23, whatever it was, was phenomenal.

Because everybody felt like they were a piece of, of the CDF, whether they were a project, whether they were a contributor, whether they were someone that just happened to use one of our projects, a user, someone that adopted it, it was just, it was amazing. And I think that is where community can really shine is when you do take a step back and say, okay, this isn't working.

How can we make this work? How can we fix this? And it, if it is sitting there listening, I had very many calls and The CDF is a volunteer position, so sitting there knowing that people are going to rag on you for an hour at a time, telling you how you're not good, like you're not good enough, you're not doing what you said it takes a toll, but it also I think is inspiring, because it makes you want to do the things that the mission statement says.

Excuse me. The mission statement says you chose to be at this foundation for a reason, right? Like you chose to use this project for a reason like we need to be beholden to like why we made that decision. Like we need to hold true to our core values as to why we're doing this in the first place. So it's, it hasn't happened at Percona.

I mean, I'm new, but I, like, I think again, the way that. Coming in from an outsider looking in, right. From a marketing person, that's always been community to a community person inside of engineering. It's really awesome to see the innovation from our engineers, the feedback from the community, the amount of push and pull that is all positive, you know, it's, it's really cool.

So happily haven't had to deal with anything like that, but again, coming from the CDF, it was just a matter of, you know, letting people's voices be heard and then creating an action plan and then being accountable for what you said you're going to do.

Jonathan: So I want, I want to let Steve answer that if he wants to, and then I have a followup for Lori, because this is fascinating.

Steve: No, no, I actually, I don't, I can't recall an instance like that where, where where there was just an up, upheaval. I think we, you know, I think we're very cognizant of how you treat the community. You asked earlier about, you know, how do you, it, you can't just be. In your own little community, you can't expect everybody to come contribute to you.

So we have to You know, that's why we share our expertise outward So I think because we're very very aware of that. Maybe that's helped us dodge some bullets but I can't I can't remember my five years here any ever being sort of thrown at us I don't think it's luck. I think it's you know, we're very intentional about how we get involved in the various communities either the ones that we depend on our own or the ones that we depend on

Jonathan: Yeah, makes sense.

Okay. So, Laurie, let me follow up on that with, with this question. When you have those conversations with community and you do come to the conclusion that, Oh, there's a In the organization, the broad organization, there is something that we've done here that maybe it needs a course correction. How do you get buy in from those in the organization to convince them that there needs to be a course correction?

Like that seems like that could be the hardest part here. How does that work? Do you have any secrets for us?

Lori: So this is actually funny. So when I first started with the CDF, I had a nail in my tire and I had a meeting with the core team from the Linux Foundation that worked on the CDF, and I didn't know that the new executive director was going to be on the call.

So I'm at a tire place. With my hoodie on, my headphones, like looking very much like hacker crazy, and not realizing who was on the call with me, I just kind of stated everything. I was like, this is happening, this is happening, this is happening. And this is how I see a pathway forward. And without realizing that the executive director was on, he's like, yes, like, yes.

And so I think it's being able to justify the actions that you want to take having an actual plan and a strategy. And it wasn't just like, I want to listen to a bunch of feedback. It was, I want to listen to a bunch of feedback from that feedback. I want to create our plan of action from that plan of action.

I want to present it to everybody to get buy in from that buy in. Then we're going to go. And it's having the overall strategy when you, and then presenting it. Right. Because if he had said, no, like, I don't want you to do that. I would have said, okay, what is your suggestion? And then I'm going to go back to what mine is, or maybe I will alter it and include his pieces.

I tend to steamroll. I don't, I can't, I can't help it. But again, it's all about like opening the communication and, but also having the, the. Like the information to back up what you're saying. Like I was a part of the, that organization for a year. I was one of the people complaining, which is why I stood up and be like, ran for outreach chair, right?

Cause I was like, I cannot stand to be in another one of these meetings. Again, this has been terrible. Like we, like, we want to do something and nothing is happening. I guess I will stand up and I will, I will do more. And I think that's also what makes community great. Right. Is that everybody comes to the table and it's up to you to decide, like, How much you want to put in, like, do you want to be a chair?

Do you want to volunteer? Do you want to be on a program committee? Do you want to initiate, like, do you want to go to GitHub and say, here's an issue, like, do you want to give feedback? Right? Like. Community is only as strong as you make it. And it's up to the individuals to kind of take that leap. Like you can use a project all you want and never say anything.

And you can be, be upset that things aren't working the way you want it to, or you can have ideas, but if you don't share them, if you don't bring yourself to the table, then they're not going to be heard. So I think again, for me, it always comes down to like open communication, feedback like listening to others and then Using that to strategize on how you move forward.

Jonathan: Yeah. Excellent. All right. We have been going for just over 50 minutes. I'm curious. Is there something that we haven't talked about that it's just a burning that you guys really want to talk about? And this is sort of a different question. Difficult question, because you'd have to go through all the things I wanted to talk about and which of those have we checked off, but is there anything that we have not gotten to that you really wanted to?

I

Steve: mean, there's a couple of items in the road map. Maybe I could assign a few JIRA tickets to you if you wouldn't mind just throwing some code down for me.

Jonathan: I can write code, but man, databases is just outside of my expertise. That is actually a good point, though. What are you guys looking forward to for the future?

What is coming down the pike that you're excited about?

Steve: Oh, jeez, all kinds of stuff. I mean, I think the ones that I'm most excited, obviously Everest is huge for us. You know, bringing. database as a service, giving you control, giving you sort of the best of both worlds, the AWS experience, but also control of the data and the systems underneath it.

And then one of the other big exciting ones is our Postgres team is working on transparent data encryption, which is something that we're really excited about. I know there's been multiple There are multiple solutions out there, but we, we think we're looking at a slightly different approach to the same problem.

And hopefully you know, the, the early returns are that the community does see value in it. So I think that's always great for us to keep, to keep on going when we hear things like that.

Jonathan: Yeah. Transparent data encryption being like the, the database itself automatically does encryption and then it's encrypted at rest.

Steve: It's actually stored encrypted in the database. So it doesn't matter. I mean, It's, it's actually double encryption if you encrypt at rest, meaning the disk itself is encrypted, but this is encrypted inside the database. So even at runtime you don't have to worry about your data being exposed unless you are the application that was intending to see the data.

Very cool. So it's being encrypted in real time as it's being read and written to the database. Excellent.

Jonathan: Blory, did you have anything you wanted to plug for the future? Anything you're super excited about that's coming down the pike for Percona?

Lori: Well, as a community person, I'm going to jump on what Steve just said.

Right. And like, we're going to a bunch of conferences, conference season is back. So I am very happy to be able to be to be at events representing Percona, so we'll be at open source summit Europe. We're staying for Valky day. We'll be at all things open. We'll be at PG conf in New York.

We'll also be at. OSFF, which is Phenosis Conference in New York. We're going to be at KubeCon. Like, so to be able to take everything that we just talked about today and then put it into practice, like, in person is what I'm super jazzed about. So, like, to see what the reaction from the PG Postgres community is when we sort of have our presentations about TDE, to get their feedback in person, to launch Everest at at Open Source Summit Europe, right?

To see, like What questions people are asking to see how we can then react is that as a community? Like, what do we have? Do we need documentation? Do we need this? Do we need that? Like, this is like, this is where I get super jazzed because it helps set the stage for what we do in the spring. Right. So with community, everything you do is like a couple months in advance.

Right. So being in person at like. Six conferences in like three months or something crazy. And then be able to take all that knowledge back to the engineering team and then package ourselves up for the spring round, right? KubeCon in the spring, open source summit in the spring, all of those other things is like, it's the fun time.

So I'm well rested from the summer and now I'm ready to get back on the road.

Jonathan: Excellent. Busy, busy, busy. All right. So I have to ask you guys each before I let you go. These are our required two questions. We'll start with Lori. What's your favorite text editor in the scripting language?

Lori: I'm sorry. Did you see the face I

Jonathan: just made? What? I'm

Lori: going to pass, like.

Jonathan: So is it Microsoft Word and English? We get that sometimes. I'm not going to lie.

Lori: Then I will, then I will, I will say that. Like, you just, I was not expecting this question. Catherine did not prep me for like the standard questions.

I've got nothing. I got a little egg on my face. Just say Vim. Vim. Vim. 100%. No,

Jonathan: no, that's fine. And that's, that's something most of the people have an answer for. Everyone's every once in a while, somebody doesn't, I think the really fun one is when someone's like been in management for a while too. And they're like, let me think back to what I used to use back when I actually wrote code.

So that's, that's fine. There, there is absolutely nothing wrong with that. But Steve, same two questions. I imagine you'll, you'll have a little bit different answer.

Steve: Well, I've definitely Vim is my favorite editor. This copilot thing, I'm, I'm digging. So I use VS code a lot more for, you know, like an IDE, but I still, my old go to quick and dirty, just Vim file.

And I could be a thousand times faster navigating the file there than in anything else.

Jonathan: Yeah, absolutely. And scripting language.

Steve: Bash is my go to. You know, Python, if it's going to get longer and more involved, but like bash, if I could type it on the command line, it's just,

Jonathan: it's just easier for me.

That's great. I'm waiting for someone to tell me one of the esoteric extensions on something like bash or Python. So somebody that uses Amber. Instead of bash as their main scripting language or I was I was recently introduced to Python which is Python with braces and some fun stuff out there like that

Steve: I work with a guy who's all about z shell everything was a shell.

That's great. Like I'm lost in that

Jonathan: Fun. All right. Thank you both so much for being here. It was a blast Really really appreciate it and that's Laurie LaRusso and Steve Hoffman. Thank you guys both both for being here

Lori: Thanks for having us.

Jonathan: Yeah, absolutely. Good stuff. All right. Well, that is the show today.

And as I'm sure you noticed databases are not my area of expertise, but we we we sorta, we sorta hung in there. We faked it for part of the way, but had a really good conversation about community as well as some technical things and a bit of database stuff and was It was really, really a blast.

We've got something really cool coming up next week. We are talking with Andreas about Ladybird, which is a browser, but it's not one of the big browsers. It's a new browser, sort of written from the ground up, which is really interesting. So make sure and tune back in next week for that. Do you want to follow me?

There is, of course, Hackaday. I've got my security column goes live every Friday. Friday, and that's also the home of Floss Weekly, as you probably know. There's also the Untitled Linux Show over at twit. tv still, and make sure and tune in for that. We sure appreciate everybody that's here, both to get us live and on the download, and we will see you next week on Floss Weekly.

This week Jonathan chats with Lori Lorusso and Steve Hoffman, the Head of Community and SVP of engineering at Percona, the open source database experts.

- https://www.percona.com/

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey folks, this week Rob joins me and we talk with Carl Richel, the head honcho over at System76. We chat about their new desktop, Cosmic, we talk Rust, we talk System76 hardware, and even a little bit about Intel and AMD. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 798, recorded Tuesday, August 27th.

It is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we have a treat today. I know I'm excited about our guests most of the time, but I'm extra excited today because we're talking with Carl Richel about System76, about Cosmic, about.

All kinds of stuff. I'm sure we're gonna talk Russ. We're gonna talk Wayland. We're going to talk the new cosmic desktop, which the alpha is out. It's great. Before we get him, let's talk with our co host. And today we have, we have Rob, we have the, the cereal desktop, the, the, the distro hopper, which is why Rob is on the show today, because I, I was, I was pretty confident that I could talk him into hopping, at least in a virtual machine over to the new cosmic alpha, Rob, welcome.

Hello, hello. Hey! We've been watching Cosmic for a long time now, haven't we?

Rob: We have. Been all curiously interested in all the places they've been going.

Jonathan: Yeah, I think so they they made it's oh, it's been like a year ago now carl will be able to tell us I'm sure exactly how long it's been but it's been like a year ago now They made their initial announcement and the things that that really caught my attention right away are one They're getting away from gnome, which i'm I I like the pop os desktop, but i'm not a huge gnome fan two they were going to rust base, which I think rust is just cool and three they're like And because we're doing Wayland and we're building it from the ground up, we're going to do HDR.

And I'm like, Oh, HDR is cool too. So like from the initial announcement, they had me hooked. And I've just been kind of eagerly awaiting the whole time. Like, Oh, when's it going to be ready? When's it going to be ready? Oh, now you, you gave it a

Rob: test drive, didn't you? I have I've given a test drive on a VM. I, I was really hoping to try it out on bare metal before the show, but you know, time flies and it gets away from me.

Jonathan: Yes, day job and family and all kinds of other stuff going on. And then there's that other show that we keep you busy with too. So we have a lot, but it looks really good. It looks really good. And even in a VM, it does. All right, let's, let's let's go ahead and bring, bring Carl on. Obviously the guy that knows what's really going on.

Carl, welcome to the show.

Carl: Thanks for having

Jonathan: me. Yeah, thanks for being here. This is, oh, this is going to be fun. I, we, we had Carl on back when we were still on the Twit Network. And Leo was so, so excited to be able to talk that he's like, Oh, I want to co host too. And so, you know, it was Doc and me and Leo all co hosting.

Well, I mean, Doc was the main host and then Leo was kind of the main host because it's Leo. But it's, it's great to have you back. And it's great. It's great to chat about, you know. First off, what's happened since then, but I think mainly what everybody's curious about what I'm curious about. So let's talk cosmic and let's

Carl: Yeah, I love to talk cosmic.

So Thanks for having me on to chat about it. I was thinking about the last show We were on together too with that with Leo and I remember distinctly he has he has a one of our darted pro laptops And he, he apologized for running Linux Mint on it, I think. Oh, that's But we're just happy, we're just happy when Linux works that way.

We, we're here to make Linux work on our products, whether it's Pop or Mint or, or Bootery or, or Arch.

Jonathan: Yeah. So I'm actually, I do the show on an HP Dev1, which is not quite your hardware, but it's also pretty much your hardware from what I understand, but running Pop! OS on it still. And I've had a Yeah. Yeah.

99 percent great experience with it. There's something weird with the power charge circuitry, but we work around that we make it work anyways

Carl: Yeah, that was a great product and a great collaboration of the folks at HP. You were really great to work with.

Jonathan: Yeah

Carl: They're very professional. I think Working with them on dev one Also helped us refine POP 2204, because that was coming out at that time.

And so we had just more people with different perspectives looking at what we were building and, and really helped, helped sharpen that, that operating system release.

Jonathan: Yeah. Okay. I didn't expect to get to this question so early, but it, it really leads into it. You did the collaboration with HP and there's, there's another company out there that I think might be really interesting to do a collaboration with.

And that's framework. And I'm, I'm just curious, like, do you, do you have opinions on framework? You consider them a competitor and have you ever thought about that? Like, I wonder if we could do something similar with the framework laptop.

Carl: Oh, I've been, having been in the hardware business for now.

Almost 19 years. I know I know how challenging it is and I, I think it's really admirable how well they've done getting into the market and doing a lot of creative work around right to repair. We have, we have that. But very much in common since day six and, and, and framework both care a lot about their about a user's right to repair, but their ownership over their products.

So so I think I congratulate them on, on doing something that I know very well, how hard it is to do, and they've done a fantastic job. So, so good for them.

Jonathan: Yeah. I, I, I kind of, I can envision a future where it's like either, you know, system 76 makes. One of the embedded main boards that people can put into a framework laptop or framework sells a system 76 version or maybe even makes cosmic, you know, maybe as simple as making cosmic a ship option that you can buy one of our laptops, you can put system 76 cosmic on it.

And everybody wins. Like it just, it, there, there seems like there's gotta be, there's gotta be some way that the cross pollination and that can happen.

Carl: I think that would be possible. There's, I'm very interested in modular mobile devices. So perhaps at a level, A little bit further than where framework currently goes, but very similar in that we have a main board that can be used across a number of different devices.

And you essentially connect daughter boards to that main board for your port layouts, and that might be used for a desktop or a laptop or any other kind of device. I think it takes a lot of creative engineering to kind of compete at a very high level and do that with really good thermals in a mobile platform.

But I mean, doing hard things is the kind of stuff we're interested in.

Jonathan: Yeah, absolutely. Absolutely. Okay. So let's, let's shift gears a little bit. Let's talk Cosmic. What was, what was the point where you guys realized, okay, we need to, you have, you have Pop! OS, which is a very, very heavily, heavily customized.

Ubuntu, and it's a very heavily customized GNOME. And at some point you guys made the decision that GNOME is just not cutting it anymore. I'm curious what, what that looked like. What was, what was the point where this, this harebrained scheme as it were, popped up where it's like, we need to, we need to start from scratch on this.

Yeah, why? Yes.

Carl: Yeah. Why? Okay. Well, it's a long, it's a long story, so I guess I have to decide where to even start, but at the best place is probably with why we decided to make the changes we wanted to make to gnome in the first place. We were. All of our engineers that were working on the Pop OS were using i3 at the time.

When they were using i3, I got became interested in it. And so I started using Sway. Then we were getting calls in our tech support department. I said, Hey, does I3 and Sway run on, on Pop OS or how do I set up I3 on Pop OS? And this was not just a one or two requests. This was extremely common. It was, it was coming in regularly.

Then I went to a, A firmware conference at Google for Open Firmware, which we do a lot of work in that area. And so we were giving a talk at the conference and everyone at the conference was using a tiling window. It became clear to us at that point that what we were providing at PopOS out of the box was not what our customers needed.

So in response to that, we decided to build the extensions we built into Gnome, which, which added auto tiling added a different UX in general with a launcher and application library and a doc. And we were adapting for some things we're writing new, new software for others. We were adapting extensions like dash to doc to provide the user experience that we wanted.

Then we found our users were craving and our customers were craving. So it all started with just the desire to. Respond to what our customers want and build something that they wanted. The next. Step was to try to make that more official. And we wanted to prove what we were doing and know, and then see if we can make more official.

We, we had a, a UX architects or a UX architect. Maria was on the design team. I had gone to a number of quad X and. And over time, it just became clear that our vision for the desktop, and we pitched the idea that you could take, we could kind of separate the idea that the UX is what we And in reality, it's a whole bunch of different pieces that you put together to create a holistic experience.

And we thought that Noam could perhaps take the, take the perspective that SimCity 6 could build our user experience out of it. Ubuntu could build theirs, Noam could build theirs. And the idea was considered interesting, but just not where Noam wanted to go.

Speaker 4: With

Carl: that being the case it became, Clear at that point that we would not be able to continue using gnome and continuously adapting it, you know, working against what was going on with upstream to build the experience our customers needed.

And so we decided to build a cosmic.

Jonathan: Do you, are you kind of of the opinion that maybe Gnome has shot themselves in the foot and some of the other downstream from Gnome are going to have to face similar decisions?

Carl: I don't know. Gnome has their vision for the desktop. I,

I don't share it. It's kind of that simple. What I, I kind of, I love this, what I love about Linux and open source is that it's, it is a collection of all of these different communities, interests and tools and different things that you could pull together to build something with. That's what's special to me about it.

And we don't have that at the desktop platform. That's what, that's the scratch we're trying to itch with Cosmic. So we have this thing where we have customers that clearly need something different because they're just telling us all the time we want to be able to build it, but we can't be the only ones.

I'm sure other distros have different target audiences and they should have a different UX because there are audiences different. We don't all have to be the same. In fact, the vibrance that comes from distros and their response to customers, I think is valuable. So we wanted to build a platform. In cosmic where you could take what we've made and it's explicitly designed to create your own user experience.

So you could build the Ubuntu UX out of it. Very, really, very simply, you can build Manjaro could have their own UX. You could build a UX more like sentiment. Out of cosmic. Those are the things that, you know, those are some of the projects that are, have kind of a unique approach to how they they adapt and customize custom environments, but cosmic allows you to do it at a very deeper level and at a professional level that's very high quality.

So so while it's going to be the flagship experience of Pop! OS, and I'm sure the first release is going to be very similar to what we Push out what we think is right for our users. What's most exciting to me is how people will. Brand cosmic, how they might experiment with different applets and user experiences and ways of launching applications or, or managing applications or moving windows, you know, all those things are up for grabs for, for experimenting with in the community.

Jonathan: And there's already, there's already been some, I know Rob's chomped at the bit, I'll get one more and then I'll hand it to Rob. There's already been other distros I know of. That have expressed interest in, but we could package cosmic. I'm pretty sure Fedora is planning on packaging and shipping a cosmic spin.

And that's gotta be exciting for you guys too.

Carl: It's exciting and encouraging and much, much earlier than I thought. I don't think I, we saw it coming. Not too long ago. And so we're learning how to be a great upstream for distributions. We consider distros our customer. We want to make a great, we want to make it easy to package easy to maintain.

We want to listen to their feedback about what we can do to you know, build a good experience for them. So I think they kind of heard that attitude coming from us that we're really care about you know, empowering distros and and yeah, it's in Fedora. OpenSUSE Serpent is packaged as well.

They're a new up and coming distro. I think that's pretty cool. I hope I'm pronouncing that right. Arch, Arch spin that has a focus on performance, their packaging as well. When they're, I think in the performance reboot, I think uses some different compiled flags or something along those lines, but yeah it's, it is encouraging and exciting and we hope we we hope we do well by everybody and all the distros that are putting their faith in us so early.

It

Rob: is exciting. So I mean, GNOME is a nice interface. If you want to do the GNOME way, but was some of your reason, was it just hard, more difficult or becoming difficult to keep up with their changes and the way their extensions work and having to keep updating yours to work with the newest version?

Carl: No, not really. It's that taking that route means that what we are doing every six months is adapting our extensions to work with the new version of Gnome, and maybe adding a little bit of icing or advancement or additional value. So that work isn't very fun. And we're just working Every six months working to get back to par, which is where we were and maybe adding a little more with Cosmic.

What we've done is built the entire foundation where it's it can be adapted to, as I said, other user experiences. But once we have that foundation in place, all of our engineering and all of our effort is adding features and capabilities, new experiences on top of Cosmic. So it means we'll be able to innovate much more quickly.

Just having this flexible base to build upon.

Rob: With that flexible base, The alpha you have today isn't doesn't have that flexibility yet or does it

Carl: it does yes it's the the alpha as It has essentially all the UI elements that you would need to use the operating system But all of those are also are stable Are interchangeable already.

So we have a panel on top and a dock on bottom, but you can remove the dock, move the panel to the bottom. You can make it more like a windows experience. You can move the dock to the left and then have a panel on top and you can have buttons in the top left, like unity had with Oh gosh, they called it lenses for a while in touch.

I'm starting to mix up some of the older Unix or unity user experience features. But in essence Together with panels and applets, applets are applications that run independently inside of the panel and on top of it. So the combination of panels and applets means you can build in a UX.

You can build a custom launcher that is different than our launcher and maybe is more like perhaps an elementary launcher or something along those lines and add a button to the panel and now you have an experience that's closer to that of elementary. So it does have that flexibility in it. And.

There's lots of heavy customization that can be done through config files. What we expose in settings is what we think makes most sense for users, but we would expect people making different user experiences to change the settings as well.

Rob: So, yeah, I guess I should have put a little more time into my experience that I had with it. But I have I have had some time and I explored around with it only on a virtual machine. I, I should have taken the time to put it on a bare metal. But, you know, before the show we were talking about on a virtual machine because of, I believe you said the graphics acceleration.

It's, it's not quite as Stable there because it did crash on me several times. So could you speak to that maybe? Yeah.

Jonathan: Is that, is that expected? I think that's expected, right? You said during the press call.

Carl: It's, it is expected and it's all because we don't have software rendering in our capacitor yet.

So Cosmic Comp doesn't have software rendering, and so you need hardware acceleration in a virtual machine, which is actually kind of fiddly to set up, and more fiddly than, than I'd like, so so we need to get software rendering done. It doesn't affect anyone on hardware. And of course, all of us developing Cosmic, we're all on hardware.

And so, so one of our focuses will be making sure that you have a good Cosmic experience for the second alpha inside of a VM. Because we understand, you know, the commitment just to install it on, on your hardware is high. It's not, it's not production software yet. So, We want to improve that experience for people in the second alpha.

Jonathan: Okay, I've got to jump in and ask. We talked about moving from GNOME. And this is like partially a troll question, but also partially like a real question. Did you consider KDE as your base? Or any of the other desktops? There, there had to be at least a thought of, I wonder if there's something else that's out there that could do this before you just jumped into the deep end of the pool and made your own.

Carl: There's very little that we didn't consider. We had a big pros and cons list for, for different directions. KDE is, I, they're kind of my spirit animal. I like their approach. I really, I really like like KDE and I like the community there. I think I think they're, they're great. Their response to what users need and want is admirable.

They're, but they're, they're technology stack is something I think there's two things with where, where we decided not to go with, you know, trying to adapt three that where we didn't didn't try to go with adapting K to E to our experience. One is we would probably not make K to E users happy. Not because of what we're doing, but because we might trim down things in a way that they'll say this is supposed to be KDE, but it doesn't have all the KDE stuff.

So, so I thought we might be a lose lose situation there where you know, people feel like it should be KDE, but we're building something different. It's not like KDE. And and I don't know how responsive you know, KDE folks would be you know, to, you know, to like more of a kind of middle ground between all of the options that they have and know.

But in any effect, I thought we couldn't make everyone happy. And we might get in the same kind of situation where upstream is not very happy with us. And, and then customers, I think it's something else or seconds. We're not, um, the technology stack C plus plus isn't a language that we work in.

It's not that. Our team couldn't learn C to go that route, but I just can't imagine being a brush shop and then going that way. Yeah, that's fair. Third Katie is as our vision from the start was that we wanted to build a platform for people to build experiences with. We wanted to be that, that user experience, that UI level that doesn't really exist today.

People are using Android to. You know, make car infotainment systems or, or exercise bikes or, you know, specialized devices. Well, this is a, this is a Linux and open source UI project that enables companies and, and distros to do that on their own. I can't, you could I mean, this would be, this would be something that'd be very easy for, for valve to build a steam deck experience with, for instance.

Oh, so, and it's intended for that. So that's also just a distinction or a difference with what our core intent was in building Cosmic.

Jonathan: Yeah, excellent.

Rob: So being so involved in open source for so many years, Any thoughts on open source hardware, like getting Pop! OS working or in Cosmic working with a RISC

Speaker 4: V?

Carl: I think RISC V is very exciting. It's it always turns up in our channels when something new is happening in the RISC V hardware world. We're being the size that we are. We can really do like one really big project at a time. And so cosmic is that one really big project that we, that we're wanting to knock out and then we'll see what's next.

Jonathan: Yeah. Interesting. With it, with it being built on. Well, with Rust, which compiles basically everywhere now, and on top of existing technologies like like Wayland and all those things. I, I kind of suspect that when you get to the point that, okay, we're ready to do our main release, like, the bulk of the development is done with Cosmic.

You know, for that initial release. I kind of suspect that it's going to be a fairly easy lift to go to another platform. If they have like reasonably well working graphics drivers. I, in fact, you, you might find that somebody from the community beats you to it and makes it work there. We've, we've seen stuff like that over the years, time after time,

Carl: I would be elated to see that.

I think it would be awesome. And that's one of the things that's so encouraging about what's happening with the cosmic community to we have hundreds of people contributing patches, building features. It's the energy around cosmic is very, very exciting. So so, yeah, I think the next extension of that is Cosmic's running on this or, Hey, I built a, I put these, you know, five applets that when you combine them together you have this really unique way of using a computer or using this device that doesn't have a keyboard or, you know, Yeah, that's the stuff I'm really excited for.

And I think if we've done well, then we're going to see lots of that over the next, next few years.

Jonathan: Yeah. Okay. We do have a question from the chat room. I've got a bunch of stuff that I want to get to too, but mashed potato asks also, he says, why Ubuntu and not Debian? Well,

Carl: Debian Ubuntu does a fantastic job of pulling together a modern or, Very recent, but stable base, I think.

So combined with our long experience with the Ubuntu I think we can build a pop OS as a better product with Ubuntu snaps are making that more challenging and I'm not I'm not like anti snap either. I, the only thing I don't like is about snap is I kind of like the technology. I just. I don't like a proprietary store.

That's, I would prefer it to be like repositories that we all are used to in Linux and Flathub, which is, you know, an open store. That's my only beef with it, but I would love to have Snap working in you know, Cosmic Cosmic App Store, for instance, and have it as an option for people to turn on or off, you know, depending on their preferences.

But because we don't want to have a proprietary store in Pop! OS by default, we have to do quite a bit of additional packaging work. Like Thunderbird in 2404 moved to Snap. So we're packaging Thunderbird as a Debian for Pop! OS. So

Speaker 4: it does,

Carl: It does add that work, but. There's a lot of other work that they do that is a layer on top of Debian that makes our jobs easier.

Jonathan: Yeah, interesting. Okay, so Let's talk, let's talk Wayland for a minute. Something that's not been entirely clear to me. I think I know the answer to this. With Cosmic, is there an x11 option at all or are you guys Wayland only?

Carl: Wayland only. But there is X11 X Wayland support in Cosmic Comp, so it doesn't really feel, at this point, the Wayland protocols, and this is just a community wide thing, the Wayland protocols are getting really, really good, really, really close to being, like, I think their NVIDIA is finally playing nice, I know it's been a long road for them, but that was our biggest, our biggest question mark wasn't, you know, For releasing Cosmic was NVIDIA because if if NVIDIA drivers were not working, we would just have to delay until that was until it was working too many of our customers use NVIDIA and care about the, the hardware.

Speaker 4: Yeah.

Carl: So but besides that, essentially what we're, we're at the point where moving to Wayland, I don't think. There'll be some corner cases, but there will be very far few between. Otherwise for most folks it's just going to be a more secure higher quality experience.

Jonathan: Yeah. So it, I think that's one of the things that's actually really compelling about cosmic is that you guys were able to, with the exception of ex Wayland, which I don't think counts in this case you were able to just sort of push aside all of the X 11 cruft and start from a fresh slate.

I think that's really compelling because like, X11 has a lot of cruft around it.

Carl: Yeah, it's, it is interesting that that's something we haven't, X raying it is still a pain. It really is difficult because the, the knowledge to implement the things that are there are just. Buried in some guy's head that wrote it 40 years ago.

Yeah. It couldn't be 40 years. That sounds like way too long. Maybe 30, but that's where it's at. And the documentation is the code is it is definitely challenging. But so we spent a lot of time just working on getting XWayland working so that that experience can, can feel seamless.

Rob: So I assume all of your.

All of your apps, applets, tools are all direct Weyland and not going through the ex Weylandshim.

Carl: Great.

Jonathan: Yeah. That makes sense. That makes sense. Okay. I, I, I sort of watched the Weyland development process and sometimes that's painful. I, and I've got to ask do, do you guys get frustrated with the slow pace that Weyland takes and the, it's, from the outside, it seems like.

Every decision gets bike shedded for like a year before anything gets added. Is it, has that been a source of frustration or are we kind of past where that's a problem?

Carl: No, I think the way we approach it we have two engineers that do most of that work quite a bit in the Wayland protocols, Ian and Victoria, and their approach is in essence, This is something that we have to do to deliver a product that's going to work for our customers and our users.

So if we have to release you know, pre V1 protocol even if we have to change it later to match what's, you know, the eventual the eventual protocol, then that's something that we'll do. And we don't feel you know, we'll, you know, we don't feel like that's you know That's bad because Wayland didn't move fast enough.

It's just you know, it's just part of the process of working in a community with a lot of different interests.

Jonathan: And I think that's probably the advantage. One, one of the advantages with Wayland is because Wayland itself is pretty much just a specification. And the, the implementers are very, very free to add their own specifications, to add their own interfaces on top of that.

And so you have things like what KDE has done, where the KDE guys got tired of waiting for. HDR to materialize in Weyland upstream. And so they finally just said, okay, fine. You know what? We're gonna build our own and because we're gonna have fun with it We're gonna call it frog and nobody knows why but that's what we're gonna call it And hey, look with a couple of days, you know a couple More than days, a couple of months of hacking on it.

You guys can use HDR on a couple of programs in KDE because we got tired of waiting for Wayland. And I think that's secretly one of the superpowers that Wayland has.

Carl: Yeah. And that's an okay approach. I mean, we, we do have to get the things done for, for users and that's just a given and that's, you know, part of building software and building products.

If the consensus protocols take a little bit longer and we have to, you know, adapt to them. Afterwards, that's all right.

Jonathan: Yeah. Speaking of HDR, this is actually one of the, for me, not for everybody, but for me, it's one of the killer the killer features back when, when y'all first announced is that a cosmic is going to have HDR.

And I know, I know that's not ready with version one or the release one, the. Candidate one, whatever we're calling it. But I know last time I asked you about this You said that you you anticipated it coming with the next pre release what what is the what does the roadmap and timeline look like on that now?

Carl: I hope I didn't say that I could I am so optimistic terrible timelines Optimism and bad timelines just don't go well together. But the thing is Well, when I get into a tirade about really big projects, it's not actually possible to predict how long it would take, but with with HDR it is a focus of ours and we're Victoria is the, the you know, lead developer for Cosmic Comp, and she's going to all the HDR hack fest and that are going on for Linux.

And so this is a, it's a broad community wide effort to get HDR working really well

Speaker 4: in Windows.

Carl: I expect, so I don't think the first release of Cosmic will have HDR, but I believe we'll add it sometime in the months afterwards. I just don't want it to be a blocker. Because I think we've got, we nailed so much here that it would be okay to release and then add HDR

Jonathan: afterwards.

Carl: And HDR is also really power hungry. And so there are a lot of considerations with like how often it should be used, particularly on a, like a laptop.

Jonathan: Yeah.

Carl: So there's things to think through there a lot of times. HDR content might be, um, you know it might be a choice whether you want to see something in HDR content all the time.

Jonathan: That's fair. I, it, it comes to mind that I, I asked you the sort of troll question about KDE earlier. But it actually comes to mind that. approach to doing this would just be to re implement and use the, the KDE HDR protocol, because at that point, you've already got a couple of applications that are wired up to use it.

Whereas as far as I know, there's nothing that's wired up to use the, the kind of in process HDR stuff that upstream Wayland is doing. And so if you wanted to get it out the door, Earlier, you know, with the next release just re implement what the, the KDE frog protocol and, and let in player and a couple of others use it.

Carl: I have, I have a strong suspicion. It's not that easy. Okay. That's fair. I don't, I don't, I, it's just a, just a suspicion.

Jonathan: All right, so let's let's see, where do we want to go next? We asked about HDR, we asked about Wayland Rob, yes?

Rob: Yeah, so, I believe, when is the final version of Cosmic expected? Is that later this year, or is that a little optimistic?

Oh yeah, so now the

Carl: optimists bad timeline guy has to make a guess. I really want to make a you'll have a release this year and I think there's a good chance the problem, the problem might be it ends, it lands in like December, which is a hard time to release a big project. It's hard on your, your employees that would like to see their families and those types of things.

So I think if we can't make it by. early to mid November, then we would probably do first quarter of 25.

Jonathan: That makes sense. What is the, what has the reception been like though for the, the, the pre release that's out? Like, do you, do you have a, do you have an idea of how many people have downloaded it and tried it?

And then what's the feedback been?

Carl: Gosh, I don't have. I don't have a good idea. I know that we're in for downloads for the alpha. I know it's into like tens of thousands or something along those lines, but that's just like the Popeye says. I don't know about any other. We don't have telemetry telling us how many people have installed cosmic.

Speaker 4: Right.

Carl: I've almost gotten through most of the press coverage. Cosmic, that's We were, we've been so busy after the release, we had all hands company events and engineering meetings. And, and so we're just all, I'm still recovering this week from all that. But everything I've seen is I think very promising and it's a good response.

I think one of my takeaways from what I've seen is the people that the The time that it takes to build a foundation and the time that it takes to build things on top of it is very, very different. So what people are seeing today, and I think if this took two years, then it might be like two years before it's really there.

But what we're going to show over just the next couple of months is that things will happen very, very quickly. So that's that's one thing I noticed that there's kind of an impression that it's, you know, a lot is there, but. There's a lot to go. They'll see that the things that there are to go aren't really really big projects.

Jonathan: Yeah. That's the, the, the 80, 20 rule. You get the, yeah, yeah. Yeah. That's, that's a, that's a real thing. It really is. Yeah.

Carl: Yeah. And then bug fixes to We're focused on putting features out and getting all the features out and then sharpening everything up. But I don't worry about bugs very much either because every time we go to nail down bugs with few exceptions, those are kind of the quick things.

That's just the quality control and and, you know, turning a focus to all the little paper cuts.

Jonathan: Yeah. Okay. So I've got to ask, this comes to mind because. Another project that I'm a part of, we're dealing with this this morning. Have you gotten to the point, and I'm sure you must have, you've, you've been around, you've been doing, you know, Papa West long enough, if you've gotten to the point to where with Cosmic, you've got users complaining, you made all of these changes and you didn't ask us first.

Carl: I feel surprisingly, no, um, and I read almost every bug report And I mean, I'm almost through all of them that have come in since the alpha started. And there are, there are feature requests. There's not a whole lot of you change this and I hate you yet.

I, I fully expected. So this happened to us when we moved from, you know, Pop! Without our cosmic UX and no. To our cosmic UX and no, and they said, I want my overview back and I want this and this and these other things. And so we were prepared for it. This transition is going to be different because the UX from POP 2204 to 2404 is going to be very, very similar.

You're going to have the same launcher, app library, doc, panel. It's going to feel the same, but everything under the hood is brand new. The apps are new. And so you know, fortunately, I don't think the bar has Too terribly high to reach the app functionality that we had before. With the exception of files, file browsers are just really complex.

And so there's a lot of a lot of work there. I think we're going to, we're going to make it there too, but point being. The UX is going to change. It's going to be what we're going to hear. I suspect is I can't install my clipboard extension or this caffeine extension, or, you know, these other things that we're replacing with applets.

All of that, I think is, is, is absolutely necessary because you know, a desktop that has independent running applications in the panel instead of monkey patch, JavaScript, it's just. Going to be better. So more extensive extendable. So we'll have some pain with that. But I, I think from what we've seen already, a lot of enthusiasm people already building applets to replace those things.

So, you know, hopefully, hopefully it's a very small number of people that we.

Jonathan: That's, that's actually an interesting question. Is there a simple, like, scripting tool for doing extensions? Is, does that exist in Cosmic at all?

Carl: Yeah, we have a, we have an applet template, and essentially you're writing a small application that is embedded in the panel, and it can have its own settings.

So, That's and it provides essentially the same functionality. You can spawn windows. You can modify how workspaces are, you know, presented. There's a lot of different things you can do with applets. So So that is, that's our replacement for the idea of extensions.

Rob: So those applets would all be, the developers would all be making those in Rust?

Carl: Right. And they are all independent applications as well. So if you open System Monitor, Cosmic and you can just search for cosmic and system monitor. You will see an applet for a user applet, a Bluetooth applet notifications applet, every app tray, everything that you see in the panel is an independent application, meaning that it can't step on the other applications.

So if there's something wrong with it, it doesn't crash your desktop. Or if there's you could, and you could add or remove them at will. That also means that we can sandbox them. So a community that's building applets doesn't have to be explicit, you know trusted with access to your home directory.

We can we can sandbox them you know, away from that and have the same kind of permissions and portals access that you have in applications in applets that you have in extensions.

Jonathan: So, so when someone installs one of these applets, do they literally get a security pop up? You know, these are the permissions that this applet wants to have.

Like, have you, have you gotten into that, that fine grain sort of of handling?

Carl: We haven't gotten there yet because everything's first party so far, but there are community applets being built and that's something that we're going to have to think about and work on as well.

Jonathan: Yeah. I, I, I hate to say it, but you know, it's coming.

Someone's going to write an applet and it's going to be down to borrow something from the Android world. This applet turns your flashlight on. And Oh, by the way, secretly, it also reads your clipboard and sends it to us. Like that's the price of success. It's

Carl: true. Yeah. And so it's, we'll have to work, we'll want to work on it early.

We are going to have, we'll have a repository for applets and Every every poll, we're going to do engineering and and design review for applets that go into cosmic, and that will be available in the cosmic store. So to I mean, there's going to be more effort for us up front, but because our toolkit is very young, because the community is very new.

And the project is new, it's just going to mean there'll be a little bit longer review process, but we are going to actually do full code reviews. And in this case, even design reviews and provide design feedback so that so that applets can fit into the cosmic UX. And that's largely because the our templates and our widget library is just.

It's just young. So it's it's, it might be as a developer today, hard to turn to know exactly like how, and our documentation is, is young too. So how, you know, how should I, where should these buttons go to fit the UX and some things like that, that we just you need to, Tighten up for the community.

Jonathan: Yeah. Okay. So to get, to get an application into the store, I assume it's got to be open source and use an OSI approved license.

Carl: Not necessarily. I don't think that would be a requirement, but we haven't seen anything that isn't.

Jonathan: I just, in thinking through this, it's like, well, how do you, how do you do code review if you can't get to the source?

Carl: Oh, that's a great point. No, you're absolutely right.

Jonathan: And then I guess the question that, well, so, I mean, it's interesting. This is all very new stuff to you guys. You're, you're, you're sort of thinking through all this as you go along. I think the question that kind of naturally follows after that is, is there even a mechanism for someone to install an applet that does not come through the store?

Like you, you would probably want to discourage that, but at the same time, you might want to still allow it. It is, is that even an option? Is that a thing?

Carl: Oh, yeah, absolutely. You can download code from a repo. You can do that today. That's where most of these applets reside. It's in a repo spread out on GitHub.

You can download those, install them and add them to the panel.

Rob: Yeah. Okay. So, so with each applet being its own program. Would other developers be able to make like a C program or anything and actually run as an applet?

Carl: That's, so it is possible with a different toolkit. So our ICE toolkit and I hope I'm, I'm hoping I'm getting this right, but with our, with our toolkits, I believe it requires that it has to be written in Rust.

If you're using Slint, which is another toolkit that has as libcosmic widgets or, or matching widget widgets, you'd be able to use C or C or you know, the other languages that Slint supports.

Jonathan: Yeah. Very cool. Is, is. Is Cosmic built on top of one of the existing compositors? Like, is it, does it use WL roots or, or one of those?

And I can't remember. I know I've looked this up before and I can't remember off the top of my head.

Carl: No, it's it's a Bruss compositor. That's it's based on a community project called Smithing. Which and so we, we hired Victoria. She was the lead of the Smithy project, that was her project.

And so Victoria was developing Cosmic Comp and Smithy is a big part of, we have a Smithy toolkits and a lot of other things we've built around it over the last couple of years. So that's all all brushed and all all new.

Jonathan: That's okay. That's, that's really cool. I did not realize that you'd gone out and you, you hired the developer that was building the thing that was really similar to what you wanted to build.

That's, that's actually really neat. One of the things that we talk about here is developer open source developers have rent to pay and they need to eat too. I, oh, that, that is really cool that you guys, you guys found a developer and hired him, brought him on board. And so you're, you're kind of bringing the existing project on board.

Making it work with what you need to do. I, I think that is an absolute win for everybody.

Carl: Yeah. Yeah. It's, we wanted to build a the compositor. We thought that was a, the compositor has a huge impact on the field of the operating system. It is kind of central to, to the OS experience. And so we knew that was going to be a key part.

And Victoria is absolutely incredible. An amazing person, amazing engineer. So we're very, very excited to have her on the team.

Jonathan: Do you see the possibility of other, like, desktop environments that, so, someone else out there that doesn't want to use Cosmic but wants to use Smithy? It's that, like, I assume that's possible.

Is there anybody doing it? Do you anticipate that happening?

Carl: I think what we're seeing happen is that people that want to work with compositors but don't want to work in, C or C they want to work in Rust they're going to SmithA and using SmithA as their target for things like accessibility, for instance, and other, you know, unique areas where we want to, you know, improve the Linux stack.

Jonathan: How long have you guys been a Rust shop? System76?

Carl: Probably six years, I think. A lot has changed in that time, hasn't it? It certainly has, but when we started Cosmic, there wasn't, there's still the, the breast windowing environment and ecosystem is very, very nascent. And when we started Cosmic, I know more than two years ago, it was very, very young.

Yeah, there was, you didn't build applications in Rust and today, well, you could. And and I might've seen the bindings, the bindings for GTK are actually quite good. And we were used to using those as well. So you can build apps that way, but not the pure stack that we wanted to build. Where the toolkits, the widgets, everything about the application was written in Rust, including like other things, like there was no text rendering.

So. We had to write text rendering, but the really cool part about all of this is it's a it's a growing community of really passionate and incredible engineers and we all, we're getting, we get to be a part of providing the tools that they're using to build things with now. The Cosmic Text is becoming the standard.

Text render for for Rust Rust applications. You know, that's, that's the awesome thing to be able to contribute to the world.

Jonathan: Yeah, it's, it's, it's really neat. I know something that we've seen like as part of Rust making it into the Linux kernel, if for one thing, it's, it's kind of a, a. A shift for the kernel guys, because now they've been reading and writing in C so long, and they have to add this other language they sort of had to be familiar with.

But one of the other things that's interesting to watch for that is, as Rust is being added to the kernel, Rust itself is changing as a result. They have had to make You know, a lot of changes. They've fixed things. They've made things better in Rust. Have, have y'all seen sort of something similar where as you're building a compositor out of Rust and as you're building these different pieces, that some of the things that you had to work with have now made it back into the Rust language?

Carl: Yeah, I don't know details. So, you know, about that, that's, but I, but I know that everything evolves together if it's going to move forward. And that's the, and the, the base language and language itself is, is no different than any other component. You know, evolves for its use if they're, if you're doing things well.

Jonathan: Yeah, yeah, it makes sense. Are, are we at the point to where we, we are up and coming programmers? We probably need to just start teaching them Rust definitely in addition to C, but you know, is, is there a, is there a point whether it's now or in the future that we teach them Rust instead of C?

Carl: I would think so.

I mean, I went to a memory safety. By default I know I hear the borrow checker is a pain in the ass, but but I think once once you get it, it just, it's a, it makes a lot of sense. I, I don't like language wars. So I don't know how far I

Speaker 4: would

Carl: go into it, but I know it's the, I think building a the bout of memory, safe code and coverage that we're going to have, you know, obviously we have to use, you know, you know, non safe for us for lower level integration, you know, C libraries and other things like that, but Cosmic has so much coverage you know, safe for us code that I think it's.

By the time we're done, I think it's going to be a far more secure operating system and have more coverage than anything else that's out there.

Jonathan: Yeah, it's it's definitely been interesting to watch. And you know, there's there's been things that have happened like Rust developers Every once in a while, I get reminded that like, there are, there are bugs out there other than memory bugs.

We're just, we're so used to those being the norm though, with languages like C that we sometimes forget that, oh yeah, logic bugs exist too.

Carl: Yeah, I think that, I think the I mean, why, Well, we so often hammer memory safety homes because of so many critical vulnerabilities are safety vulnerabilities and you know, things that that we can solve at the language level.

Jonathan: Yeah, absolutely. Okay. We've, we have talked about cosmic. We've talked about Russ. We've talked about Wayland. I want to ask more about system 76 itself. And. I guess first off, is there anything, is there anything upcoming with system 76 as a company that you, you want to plug or let people know about?

Like, is there, is there some secret new hardware that you want to announce on the show? What's what's coming down the

Carl: pipe that you're

Jonathan: excited about?

Carl: There's going to be a new workstation. From system 76 called Thaleo Astra.

Speaker 4: And

Carl: Thaleo Astra will be unlike any other desktop that we have in the, in the line.

And that's all I have to say so far. That's fair. That's fair. It will, it is. Extremely performant and very unique desktop. And we consider it a workstation because of its level of performance and its uniqueness, but I think there will be a lot of people excited about about this product.

Jonathan: Yeah.

Now, I, I honestly cannot remember. I think your. Intel shop. Do you offer AMD on any of your hardware? I just, I can't remember off the top of my head.

Carl: We do. AMD is killing it on desktop. They really are. Intel's fourth generation is quite good. 14th generation. So we have Ryzen 9, 000 products. We have Intel well, this is desktop Intel 14th Gen 15th Gen, I think it's coming in October and Threadripper products on on some workstations.

We also have a a value desktop that is 12th Gen Intel. It's because a lot of people schools and and other, you know, environments don't need You know, the latest and greatest. They need, you know, value. So that's what that price for on the mobile side. We have rising laptops into Intel laptops.

And and NVIDIA graphics on the higher end laptops. So, so really, it's pretty broad spectrum. I think that's one of the interesting things about Pop! West 2 is when we were, when we were building pop, we thought, okay, we're just going to make sure it works really well on our hardware. It turns out that our hardware, we cover so many different it's a pretty good representation.

Yeah. Yeah. And it just ended up that, well, Pop! West works great on hardware because our lab is full and it needs to work well. So so yeah, we have we have a lot of, depending on what the, what the user's needs are, customer's needs are, Intel or AMD or AMD graphics or Nvidia graphics, kind of the full spectrum.

Jonathan: Yeah.

Carl: Which I think is a pretty, it's an amazing thing.

Jonathan: That is really cool.

Carl: That Linux Ships with on everything that's new today. Yeah, that's that's amazing.

Jonathan: Yeah, okay You you may not want to answer this one and that's fair I'm curious though. Have you guys gotten bit by the the Intel 13th and 14th generation problem where where the chips try to eat themselves?

Carl: So yeah, we're it's a familiar with it But at its base, I believe that this Well, that's our, our take is that this was a weatherboard manufacturers, redlining their

Speaker 4: firmware.

Carl: And it was just it wasn't within specifications. I don't know why it was so different than previous generations with Intel specifications and how they were implemented.

But what we found, we spent, we, we did have some customers find that, okay. I've got instability when I'm compiling. And that's what our customers are obviously there, you know, the common use. Heads up. Yeah, so we did so we identified the problem. We worked through it with Intel. We worked through it with our upstream board manufacturers and, and within a few weeks we had a stable firmware shipped out to customers.

So I don't know. I don't know if it was as big as it's, like, we've got a As it's, you know, came out to be I know I've heard oxidization things and other you know, stuff. I don't know. What is, what is your take?

Jonathan: Well, let me, let me put it this way. I have a customer right now that bought two new desktops from another vendor, a very, very large vendor that I, I guess I won't name a very large vendor though.

And they have had problems with those desktops being unstable the entire time that they've had them. And they are now running on the most, on the latest board firmware, and they're still having problems with them being unstable. And so we are probably today going to, we finally figured out that this is.

almost certainly what it is because they're 13th gen Intel. And so we are today going to start the fight to get that vendor to RMA the chips. And I don't know how that's going to go, but I'm reasonably certain that what has happened is that those chips have, have eaten themselves to some extent. I

Carl: have the perfect solution for you.

It's nice being a you know, small, medium sized, hardware manufacturer because When our customers contacted us, it was the guy that's 15 feet away from the QA guy that's doing the testing, reproducing problems, was getting calls and tech support about a problem and walked over and talked to him and we said, yes, this is a problem.

And we're able to work through it pretty quickly. I mean,

Jonathan: so that's, that's like been the biggest problem with working with the bigger vendors is you call in and you get You get their first tier of customer service and those guys are programmed to say, oh, no, no. There's nothing wrong with our hardware.

It must be, why don't you go and reinstall windows? That'll surely help. That'll fix it. You know? Did your reboot? Yeah. Have you rebo, have you rebooted the computer? Have you rebooted your router just to make sure right. Oh, okay. Have ha I've got, I've gotta ask, have you guys seen have you had to RMA any hardware as a result of this?

Is that, has that been a thing?

Carl: We haven't had to army hardware. We did have one customer who just because of speed decided to move to thread ripper.

So, you know, and they, they had a vote of confidence for pressing, Hey, we know you're going to get this, but we've got, you know, we had we need a fix now.

So we just swapped their desktops out for them. But but no, not something like the not, not really like, it wasn't a return problem.

Speaker 4: Sure.

Carl: That's, that's something you're always watching out for. Yeah. Because if there's, if there's a pattern and it means returns, then you've got, You've got a big problem.

You've got to get something fixed. You've got to get, you have something to fix.

Jonathan: Yep. Absolutely. Rob, do you have anything else you want to get in before we, we get towards wrapping? No, I don't have anything else. Okay, okay. It has been great. I've got to ask, Carl, is there anything that you wanted to tell folks about that we didn't ask about?

I feel like we've covered a lot of things, but is there anything that's just burning?

Carl: Oh, I don't know. I would just urge everyone Play with play with cosmic. It's an early alpha, but

Speaker 4: it's

Carl: tech and it's fun. It's just a fun desktop. And that was you know, our intent is not just to, you know, build something because we're a company and delivered to our customers.

We really just love technology and want to make it fun. And so we hope it's something fun for you to tinker with and play, build stuff. So let's get out there and make stuff.

Jonathan: I do. I do know what I want to ask. That is what, what is the process going to look like for people that have existing Like system 76 hardware running Pop OS.

At what point is Cosmic going to roll out to them? How does that going to work?

Carl: Once our release is final, then there will be a button there. You'll get a notification that says. PopOS 24. 04 is available when you click it, it'll you know, there'll be plenty of disclosures, like you're upgrading to a new this isn't just your typical, you know, upgrade to a new version, here's a new desktop.

Yes. So we want to be very transparent. That's okay. You're not going to have gnome anymore. When you do this, you don't have to do it. You can stay with where you're at and it's going to be supported for, you know, a few more years. So you can feel comfortable there. But, we're, we're we're moving to the future and a new desktop.

Rob: Yeah. So it's good. I was going to say, so you're not going to have a pop up every, every few hours. It pops up and said, time to update, time to update. No, like, you know, some other companies have done.

Carl: Yeah, we'll have to be we, we want to be gentle. Because you do want, you do want to keep people up to date, you know, and there's, there's a balance between being a nag and doing that well.

So I think we're okay with it.

Jonathan: Yeah. So it's going to be, it's going to be the default. And is it going to be the only choice in 20, in 24. 04? Or can people still run Gnome if they want to? Is it, is it even out there still?

Carl: They can run gnome just like running like installing gnome desktop. Gosh, I think that's the the right meta package.

Ours is cosmic sessions or cosmic sessions. I'm thinking, I think it's gnome desktop, but yeah, they'll they'll be able to install gnome and run it if you like.

Jonathan: And it'll

Carl: be it'll be vanilla. You know, so yeah, that makes sense. Comparison like

Jonathan: What you guys don't want to have to do is continue to maintain the Pop!

OS desktop experience on two different desktop environments, right? That's just, that's, that would be ridiculous.

Carl: It would be a lot of Yes, providing the same experience. Yeah, no, we're just we're going to pull the bandaid off and run for the future.

Jonathan: Yep. Yep. Well, I, I've got to say we, we have the Linux show.

And so we talk about this from time to time and I've got to say it's become the thing we will tell people don't install some weird niche distro for your first distro. Don't install some weird niche distro for your grandma or your friend on their first time on Linux. Stick with a distro that's. Solid and that everybody knows something about and so we have this kind of list It's like you use Fedora use Ubuntu or use pop OS And so I think it's it's really saying something that you guys are making that list And in such a short time too.

Yes. Yes And What's really fascinating me is that I think we're getting really close to the point to where it's going to be and you should really set them up with one of the established desktop environments, you know, like KDE or gnome. And I think we're about to add cosmic to that list, which again is really saying something.

But it's also, it's also really exciting, like for the community to have a third real choice there. And so I think, I think that's neat. Yeah. That's great.

Carl: I do too. I, I am of the belief that there can be no, there's no such thing as too many distros, no such thing as too many des I think the whole, it's fun to create things.

Go out and make stuff. That's where the vibrance and the that's what's awesome about Linux.

Jonathan: Yeah, I would agree. I would agree. Alright, so we've gotta ask, what is your favorite text editor and scripting language?

Carl: Text editor? Well, right now I'm using Cosmic Text. So I don't know It's the new default scripting language.

I have no idea. Python.

Jonathan: There you go.

Carl: Yeah. Oh,

Jonathan: I reminded the old Dilbert cartoon. You have drank from the cup of management and so you don't write script anymore.

Carl: I don't know. Well, and the real reason I said Python is because that's what I used to write.

Jonathan: That's fair. It's a thing. It happens. All right.

Hey, thank you, man, for agreeing to come on. It's been a blast. The hour has just flown by and we'll have to, we'll have to have you back. You know, once, maybe after, after the 20, 24. 04 drops and everything has, has gone solid, we have you know, full releases and We'll have you back and ask about how it went.

It'd be fun.

Carl: I'd love to be back. Thanks for having me.

Jonathan: Awesome. Thank you. Thank you so much. All right, Rob,

Rob: what do you think? I'm going to be trying this out on bare metal one of these days and I'm going to be daily driving it way before I should be.

Jonathan: Yeah. I, I feel like I probably will too. I, I, I kind of want to, I kind of want to jump to a jump to it on the laptop here.

I need to wait because this is my production machine, but it's, it's, it's fun.

Rob: I'll have other machines to work on. It won't be my, my only one available. If it, if it, you know, if it's not quite ready yet, but yeah, I think it's ready enough that I could use it for most things, most of the time.

Jonathan: Yup. Yup. That's a lot of fun.

Oh, all right. Well, hey, do you have anything you want to plug?

Rob: You know, just come check me out on the Untitled Linux show, or if you want to Find me directly. Go to robertpcampbell. com.

Jonathan: Yeah, very cool. All right. I want to let folks know that next week we are planning to talk with Laurie LaRusso about Percona.

And that is certainly going to be a lot of fun. Percona is open source database software. And then the week after that, we're talking Ladybird, which Ladybird isn't from the ground up. Browser because there's basically only two browsers right now. And so these guys think there really should be a third one.

And so they're working on Lady Bird. That's gonna be a lot of fun too. So some neat stuff coming. As far as finding me, of course there is Hacka Day. Keep an eye on the Hacka day. We've got the security column goes live there on Fridays. And then there's also the Untitled Linux Show, which I do with Rob and the rest of the guys over at TWIT TV on the TWIT network.

And that is a blast and you should definitely check it out. There's also my YouTube channel, if you're really, if you really want more and that is mostly MeshTastic content these days, but if that's something that you, you want to get into it's youtube. com slash at J. P. Bennett, I think. I don't have that pulled up, I believe that's what it is, or you can just search for me on the, on the YouTubes and you'll find it.

Some really exciting stuff in MeshTastic coming with the 2. 5 release coming up, so check that out if you're interested. Other than that We appreciate everybody being here. Those that catch us live and get us on the download. Thank you so much for watching or listening, and we will see you next week on Floss Weekly.

This week Jonathan and Rob chat with Carl Richell of System 76, about the COSMIC desktop, what's new at System76, and more!

• https://system76.com/cosmic

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey, this week Dan joins me and we talk with Pádraig Brady about Coreutils, that software that pretty much everybody runs, whether you realize it or not. And because of some reasons, it is one of the most conservative development processes we've ever talked about. You don't want to miss it, so stay tuned.

This is Floss Weekly, episode 797, recorded August 20th. Don't RM RF up the tree. Hey everybody, it is time for Floss Weekly. It's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got a fun show today. We're gonna be talking about core utils, and no, that's not a typo, not a mistake.

We did just talk about core utils, but about three, four weeks ago, we talked about Rust core utils. And this week, we are talking about the OG, as I like to say, the the original Core Utils. Still, still a project. It is not just me, though, of course. I've got I've got Dan the man, Method Dan, the original Linux outlaw.

Hello, sir. Welcome.

Dan: Hey, it's good to be here. Thank you, Jonathan.

Jonathan: Yeah, it's always good to have you with us. Dan, I suspect that you sort of have a clue about Core Utils. You've at least used them a few times over the years, right? Right.

Dan: Oh God. Yeah. I was just gonna, I was just thinking earlier that I would, I would bet that anybody who uses Linux as their operating system has, even if they don't know it, interacted with the projects we're going to talk about today.

Jonathan: Yes. Either directly or indirectly. Right. And, you know, there's, there's several different implementations of core utils because you've got, well, you've got the rest core utils, which is fairly new, but you've also got things like BusyBox that also includes a bunch of the core utils, but with a different code base.

And so they're, they're just, they're ubiquitous. I mean, there's probably, oh goodness, there's, if you, if you count the, the other versions of it, there are probably billions of copies of core utils in the world. And that's, that's, That's that's something not everybody can say that that they've got that many of their binaries running around.

All right. Well, we've got we've got the man We've got paw rig and he is I believe the corp the the head maintainer We'll have to ask him exactly what his title is but he is he is the man in the core utils Project that's Parag Brady and let's go ahead and bring him on Parag. Welcome, sir.

Padraig: Hi everybody.

Jonathan: Hey, it is great to have you Great,

Padraig: great to be here. Thank you very much.

Jonathan: All right. So what what is your official title as it were? I I believe you're the head honcho, but that's probably not what they call you

Padraig: I wouldn't say that. It's open source, so there's no real official title, but I'd say if you had to give me one manager, it'd be release manager.

And I'm one of the, one of the maintainers. There's a, there's a few of us.

Jonathan: Okay.

Padraig: Okay. And I've been contributing for the, to Coreus for the last 22 years.

Jonathan: 22 years sounds like a very long time, except Core Utils has been around for quite a bit longer than that, hasn't it? How long, how old is the project?

Padraig: Indeed, well there's I guess it was originated around 92, I think, it's when, well, there were original it was a bit separated into file utils, text utils, and shell utils, and then a little while after that it was amalgamated into a single Core Utils project. So that, that was done by Jim Meyering.

Jonathan: Now, the, the, the, like the original, original core utilities, some of these utilities go all the way back to like Ritchie and Kernaghan, way back in the day with the original Unix. Is there any shared code base?

Padraig: No, it says that the GNU says that it's a part of the, GNU thing was to be a kind of a complete separate implementation.

So, so, so they were implemented separately, the GNU source space is completely separate. But, but of course, like there's, there's a huge focus on compatibility with those older utils and with other systems. And I guess at this stage, the, the, the, the, the, the, the, the, the, the, the, The, we've been developing them so long there is an onus to be backwards compatible with ourselves.

So that's a, that's a, that's a huge concern.

Jonathan: Yeah, that's interesting. So I mentioned in the pre show we had we had the Rust Core Utils project on and that, that was one of the really interesting things. So, so first off. There is, there's, there's not animosity between the two projects, but in fact, you guys have cooperating on some things and one of the really interesting places of cooperation is in writing sort of this shared test suite that determines that you know, all of these tools do the same thing, no matter which implementation you're looking at, which that really interested me.

Padraig: Well, absolutely. Look at the implementation is secondary after the day. At the end of the day, it's the interfaces with users, and that's, I guess, encoded in the test suite. So we put a huge emphasis on the test suite. And it's something personally I've been focused on all along, like, like any kind of changes or patches we do tend to be mostly focused on.

Most of the effort is actually put in, put, put into testing and writing tests for, tests for patches. So in regards to the ROS Coriatos, I actually, like a few years ago, I noticed with interest the Rust coreutils project and suggested that they could easily tie in the coreutils test suite because it was kind of written like that.

So the Rust test driver, they essentially put coreutils earlier in a path and just call our test suite. And it automatically pulls in the Rust utility. So, so, so we write our test suite to be with portability, portability in mind. Because we have to run our test suite in a lot of places. So Rust gets to, the Rust core utils gets to take advantage of that.

Yeah. And just to mention, it's worth mentioning that the, it's, it's a two way thing. So, so sometimes, Rust folks, when they're implementing new utilities, they might notice bits that we haven't tested and they'll supply patches to us and we'll implement those. So it just verifies our code is working as expected.

Oh

Jonathan: yeah, no, that's neat. And I've found that test suites like that, what they're really, really useful for is when you make changes, They, they sort of help you guarantee that your changes didn't accidentally break something that you didn't think about when you're making the change.

Padraig: That's, yeah, 100%.

Yeah, it's brilliant. Yeah, we put a huge effort into the performance of the test suite. Like, you can run all the tests Say on a standard laptop in about 50 seconds, but the nice thing I've run it on really, really, really fast machines and it automatically scales up so you can run all the tests in five seconds.

That's impressive. 100, 000 machines. But it's, it's not, it's not nice that it scales up and it kind of I guess it shows. The advantages of the Unix model, because we, we tried to rather than saying, writing the test suite in something separate, like, driving it with Perl or Python or something like that, we actually, it's mainly shell scripts, so we were kind of reusing the model and reusing all the tools while testing, writing the test themselves, and it shows that, that's one of the nice things, as well.

So, But having separate processes as you do separate things, they automatically scale up to multiple processors. And so it nicely scales up automatically using the standard Unix model.

Jonathan: Yeah. So a question that kind of naturally flows out of this conversation is, what about the, like, the original Unix version of these utilities?

Is there any cross pollination with Unix? Any of those, like, I guess we might be in a place where one of the original Unix is, somebody's version of Unix uses core utils rather than the old Unix code base. In fact, that's probably likely. And then I guess also the original Unix version of the core utils.

Do we run those through the test suite to see what they do?

Padraig: It's yeah, well, I haven't done it myself. It's an interesting question. I'm sure a lot of them would break, like the test, like ROSCore, you tell us, is trying to be compatible with the latest GNU upstream. So there are new options and behaviors really there.

So, so I guess a lot of the older, older variants would break in most ways against the, against the existing test suite. So.

Jonathan: Yeah, yeah, that's interesting. Do you know, does anybody's Unix ship core utils rather than the old code?

Padraig: I don't think so. So the main focus with core utils is Linux. And essentially these days Linux is ubiquitous.

So that's, that's generally what everybody really uses, like in the large tech companies. It's, it's all Linux internally.

Jonathan: Unix itself is kind of dead these days, isn't it?

Padraig: Pretty, pretty much. Yeah. It's, and that kind of gets on to the kind of the portability aspects of it. It's still very important.

Less going forward, as we said, because things are more and more focused and more and more consolidated on Linux. But there's still a lot of portability concerns with compilers, different compiler options, different shells. And we still try to be as portable as possible. And when you keep as portable as possible, keep your code as flexible, I guess, as soft as possible.

And it keeps the interfaces true and separate and good.

Dan: I mean, Parag, you mentioned there that the portability and stuff, and Jonathan said nobody really uses Unix anymore, which I imagine would probably upset a few people, but I'm not much of an expert, but I'm wondering, does BSD, do people ship core utils with BSD at all?

Padraig: Yes, so that would be our main other Unix portability target, and we have full, personally, I have access to, like BSD, we have access to free BSD systems, and kind of implicitly through macOS that they use kind of free BSD interfaces. I was

Dan: interested in Mac OS, I was going to say, cause you've got Darwin, which is the the kind of the, the, the base under Mac OS and they must use core utils, I would imagine.

Padraig: They actually try, like Apple tries to steer clear for GPL three reasons but, but there are a lot of users of core utils. It's in homebrew, for example, so it's easily installed on Mac OS. And for, for a long time, the, the sort. For example, the sort that was used in FreeBSD and MacOS was actually the GNU version, because that's actually quite difficult to implement and stuff like that.

But, yeah, getting back to testing, like we do kind of put in order to make sure that we have full tests passing on MacOS and FreeBSD.

Dan: That's awesome. Now you mentioned that you've been working on this for, you've been involved with the project for 22 years, which is impressive. I'm always really intrigued in how people got started in these things and how they got involved.

So how did you come to get involved with, with the project? How did you get interested in it? Was it something you were, it was computing something you were into as a kid or, you know, was that, how did all that come about?

Padraig: Well, not as a kid, I'm not going to say my age now, but I hadn't access to a computer until kind of midway through college.

So I started very late from that point of view. But I started first also with Windows, which is interesting. And I was using that through college. And then Went into industry for a year or so and got a bit frustrated with the whole black box nature. It come to a problem and then it was really difficult to actually fully, fully fix things.

So you you're working around issues more and more rather than actually fixing core issues. So then I happened upon it. Linux was just starting out at the time. I guess that's the new thing to my age. And, um, at that stage then, it, it, it's something that, that resonated with me. I tinkered around with it for a couple of years and at that stage then I kind of the Unix model really resonated with me.

And I saw, like, there was a notice that some of the core utilities would they'd benefit from some new options, and so I proposed a few patches around 22 years ago, and they were accepted. And ever since then, I became a kind of an official maintainer, maybe about, I suppose, 16 years ago now.

Dan: Wow, that's amazing.

I always love how people get involved in these things because I think one of the great things about the kind of free open source software world is how, without meaning to be cheesy, open it is, you know, in that you can get involved. What were the challenges in kind of getting involved? Was it, what did you, were you nervous about maybe submitting a patch or and getting, you know, rejected in some way?

Padraig: No, I was very excited. I remember my first patch, how naive I was. I was, I was worried about, well, the code was okay, but I was worried about someone was going to come up with the idea. It's one of the funny things to look back on. I thought someone was going to, it was such an obvious thing. I thought someone was going to come up with the idea and I was mad to get the patch in before anybody else did.

But sure, of course, even there's an infinite amount of code. code to write and an infinite amount of ideas, so that's never a never an issue. So, so that that's one thing I would say to people is even if something is implemented already, there's always a way to do it better so. Always err on the side of sending in the patch, and generally people involved in these projects are more interested in the tech itself and are very interested in incorporating new people and new code into the projects.

Dan: Yeah. And a question I have to ask, it's the kind of dirty question in the room, but I'm really interested is have you managed to get paid for working on core utils? Has that been like part of your job at any of it? Cause I know you've worked in lots of places.

Padraig: Sure. Not directly but I wouldn't be working where I was without having worked on your core utils.

Let's put it like that. So, so there are a lot of it's, it's a good thing on your CV.

Jonathan: Yeah, yeah, no joke, and they say,

Dan: I know you've contributed to it. Sorry. Go on, Jonathan.

Jonathan: I was just going to say, they say with the kernel itself that it's like five, landing five bits of code. Landing five pull requests in the kernel is the average that it takes for someone to get a job offer as a result from it.

Like there, there are a few places we're contributing open source code. It's just. Excellent for your career. And I imagine something like core utils is going to be on that list.

Padraig: Yeah, it's just, well, I guess anybody who's used it as a set at the start of the show, anybody's has used Linux at all has either used them directly there and everybody knows, knows about them.

So it's it's it's, it's good for the CMU as we say. I've been very lucky over the years, really, to have been, I feel very lucky to have been involved with the project and it's still very interesting and rewarding going forward, so it's all good.

Dan: Yeah I'm interested in how, I know this is probably an obvious question, but I'm interested in how the project's managed.

You mentioned that you've got, obviously there's a few maintainers involved probably quite a lot of maintainers I would imagine. How does it compare to something like, like the kernel project? Do you, do you manage it all through Git and, and the releases and all the patches and everything else?

Padraig: Yeah, so we moved to Git fairly early on. Patches are managed with a mailing list still. The number of maintainers, there are, I guess there are three or four kind of central core maintainers that work on it over the years. There's a separate project, so kind of focusing on the portability aspect, there's GnuLib, project, which is probably slightly a bit more active, and there could be, say, 50 projects reusing the portability code that is abstracted away or encapsulated away in the GNU Live project.

And so, so, so, but, but GNU Live was kind of originated as core utils. So they were looking at it simply, there were lots of if defs in the core utils code, and that was gradually moved across to a separate project to keep. So GNU Live presents a GNU interface everywhere, and then it allows the actual projects using GNU Live to be a lot cleaner, and just as soon Like a new interface is available.

Jonathan: It seems like code bases just grow if def statements over the years. It's just a natural part of their development. Especially

Padraig: something trying to port to every Unix and every compiler in the world. You have to be especially careful.

Jonathan: Okay, so, with a project that's been around as long as Core Utils has, and with sort of a, in some ways, a frozen specification, What's it, what's it like to work on core utils and Are, is it, is it done?

Is there a place where it's going to get to be done and what does it look like? And so I guess, I guess really what I'm getting at is I asked that question very tongue in cheek, but what I'm getting at is what is it, what does it look like in core utils to make changes? And I assume it's a, it's like a very conservative process to very slowly make changes and to do it very intentionally, right?

Padraig: Well, yeah, you have to be, something that's used so ubiquitously, you have to be very careful. So there's the whole, as it was, mentioning earlier, the focus on testing. So, that kind of handles that. But we have to be very cognizant of the interfaces. And also the, we have to be cognizant of our interfaces with the community.

So, there are lots of requests over the time to add, over time to add this option and that option. And a lot of them are good suggestions, but they're not just appropriate for adding, because the equivalent functionality might be in a separate tool might be slightly dangerous. But we do take and we do make new additions ourselves over time just to add new functionality, and port to new compilers, and new architectures, and enhanced performance, and portability, and all that.

So there's changes all the time. But just getting back to engaging with the community, that's great. Like, like that's one of the things. You have to be especially cognizant and careful about that with an open source project. And one of the things we've done, for example, is we've maintained a page of rejected requests.

But they're very carefully considered and curated. And it, like if someone comes in with their with their suggestion, and we carefully consider it but reject it, we may add it to that page. But they can also see. If it has, like often, the same suggestion has been made multiple times. They can also see similar, uh, similar suggestions being rejected with very careful, carefully considered reasons given.

And so they don't feel I guess alienated when we, when we give feedback like that. But, but, but we're definitely very open to, to new features if they're appropriate. So it's just, we have to be just careful of that. Backwards compatibility with ourselves the compatibility going forward and just being, I guess, generally cognizant of the Unix model and just being true to that and keeping things appropriate.

Jonathan: Yeah. Let's see. Oh, progress bar. What, what's the, what does it look like when, when someone sends in a request? And let's just take the progress bar as an example. What's the process look like? Like, do the maintainers vote and say, you know, three, three out of five say, this is a bad idea, so we're not going to pull it in or just That's essentially it.

Padraig: Yeah. And it all happens out in the open. The, the, the important thing is it happens in the open on, on, on the mailing list. So we give reasons why it mightn't be a good idea. That's An interesting one. That's one of those 50 50 ones, which is probably a good idea, but it's also implemented already in our sink and stuff like that.

So do you want? Do you want to complicate the code base just to add that? That is one that we may actually add eventually. It's There, there, there's an interesting one like that. So, so that's getting back to the Unix model. Mm-Hmm. . So it might be nice to do that more generally. So looking at something like pv, so that's pro general, that's a separate progress viewer.

Speaker 4: Mm-Hmm. .

Padraig: And, and you can point it at any command and it, it'll open up the file descriptors and. Now, it's, it's not as general as if you put it directly in CP, but it's more general in that it will work with any command and you can pop it in the middle of any pipeline. But one of the reasons we hadn't done that particular one was rsync is equivalent functionality that already has a progress bar.

And looking at The Unix model of thinking, having one tool that's doing something more general, then there's a separate PV tool which can be pointed at the CP process, and it will inspect all its file descriptors and see how far along they are at reading and writing a file, and you can pop it into the middle of any command or any pipeline or directed at any command.

So it's a more general solution.

Jonathan: Yeah, and it's also interesting. Some of the other, like, for example, the Rust core utils, they've gone ahead and they've added that in, I think, at least in CP, maybe in some of the others where it has now a built in progress bar, and that's one of the fascinating things about having multiple implementations of this that are kind of looking at the same the same test suite and the same kind of core rules.

But you have a little bit of you have a little bit of flexibility. as well to be able to do things just a little bit differently without breaking the rules.

Padraig: Right, right, right. And like, it's an interesting thing, like, if there are kind of borderline functionality or new options like that, that are already implemented elsewhere, that that's more kind of more leeway for us.

To implement those for better compatibility. Now,

Jonathan: so the, the, the fact that this is again, talking specifically about progress, the fact that this is still kind of being reconsidered it makes me think that, that, that website, that, that page of these are the things that people have suggested that we are decided not to do, like, that's not necessarily a static list.

And, and you guys have the, That you have the freedom to go back into that and sort of mine for ideas again and reconsider, well, maybe this wasn't such a bad idea.

Padraig: Absolutely. Yeah. And some of them, that doesn't go for all of them. That there's some, some special

Jonathan: ones in there. Yeah.

Padraig: Not to, well, this happened long ago.

That's a, I'm not singling anybody out here, but there was one suggestion that so for RM minus R. To recourse down the way, you could have an option to recourse up the tree. Yeah, we rejected that one

Jonathan: pretty quickly. That, oh yeah, that's, that must be, that must've been like the, the April 1st RFC for the project.

No, no,

Padraig: there was some, some arguments for it.

Dan: Really? And also, also implied the dash F as well. Just

Jonathan: don't accidentally use that flag. Oh my goodness. God, yeah. Uh, and, and then what about when you, and I'm sure this has to happen from time to time, but I'm sure it has to be rare. What's the process if you say, okay, we're going to make a change and it's going to break something, but we're going to make it anyways.

What does, what does that look like? Has that happened? Is that going to happen?

Padraig: It has happened. As we said, we're very careful about doing that. Generally. It's, it's only on if it's associated with some extra functionality that we're doing. So we break compatibilities in some rare edge cases just to allow you to add a lot of extra functionality.

So it's rare we do that and we err on the side of not doing that because, like, nobody wants to rewrite shell scripts when they upgrade from CentOS. Thanks. Bye. 10 to 11 or whatever. So we, we, we just have to be very careful about doing that.

Jonathan: Yeah. And is, is there anything sort of on the on, on the radar that is going to be breaking or maybe otherwise a, a huge big change coming?

Padraig: Nothing breaking interface wise? Okay. Look like, look retroactively, look, looking back at some things that the of another or we, we have another page kind of written of core util Scots. And these are things like, you would never have done this originally if it was designed as one cohesive set of utilities.

These things would never, like, just one gotcha, for example, DD, you often want to present hex numbers to DD because you're dealing with power of two blocks for inputting output to disk and stuff like that. So it'd be nice to supply a hex number. So if you could, and you can. seemingly supply hex numbers like 0x100 for, say, 256 byte block, but 0x100 to DD is 0 multiplied by 100, which is, which is 0, which it accepts and goes ahead and just doesn't skip anything, for example.

So, so there, there are little gotchas like that, that haven't been very carefully considered back in the day, but we have to keep compatibility with that going forward. Thank you And so please, like, we'll break compatibility slightly in that regard. For example, POSIX specifies, if you're not giving an error, you shouldn't give a warning.

Like if you're not exiting with an error status, you shouldn't give a warning, but in that case we do give a warning because it is such an edge case that you probably wouldn't be doing a zero multiplied by something, but we give a warning in that case. So we break, that's not really breaking compatibility, but we're just very careful about how we approach these sort of things.

Jonathan: Yeah, interesting. Alright Dan, you want to pick it back up? I've lost the connection to our back server, so we're having to do it in the open.

Dan: It's all going on today. Yeah, it's been a fun

Jonathan: one.

Dan: Yeah. I'm interested in one of the things that you actually mentioned Pari was the Unicode situation and internationalization as well as another thing as well for for character sets and so on.

So what's the situation with, with, with that at the moment?

Padraig: Yeah. So that's it's. A tricky kind of implementation thing that spans most of the utilities, especially the original text utilities. Personally, I was interested in doing that, and while I was working with Red Hat, I requested, say, a block of three months to go away and kind of just implement that, which wasn't granted, which is interesting.

I was surprised at that at the time. never had really a block of time to, that are required to go away and work. So that has kind of happened in piecemeal over the time, over the last few years. So the main Unicode functionality is currently encapsulated in GNU lib. And there's a lot of Unicode expertise By the, the developers and the main developer, Bruno Hebel of, or of working on Cano Cano lab.

And those interfaces have evolved a little bit over the last couple of years. They've added new abstractions for dealing with characters and multi characters and cells and stuff like that. So that, that has been gradually being added to the corridors over the last while. And, I kind of created a planning document, kind of describing the work that had to be done there.

So at least we have a kind of an overview of what needs to change. So it will eventually happen. Just it's happening slowly at the moment. And one thing that has changed over the last while as well, like when we originally envisaged this, it had, there was a lot of different character sets that are in use, but most things have consolidated on UTF 8 now, so that kind of suggests different ways of handling things of converting everything to UTF 8 before processing and maybe having separate tools for kind of sanitizing and working with UTF 8.

And then as an interface to other utilities, rather than having each utility dealing with edge cases of mis encodings and cases like that. So yeah, it's still a work in process. So, I guess that would be the main kind of functionality or feature that, that. Is kind of outstanding and core utils at the present.

Dan: Yeah, so it's a big one to deal with I imagine It's interesting that you said that you I mean without getting into politics too much that you weren't granted the time to work on that That's you'd think a lot of people would be would be after that, but who knows? Now this is a slightly left field question I suppose because Jonathan mentioned busy box at the start And I am going to show my ignorance now because i'm not entirely sure of the relationship between core utils You I'm busy box.

So seeing as you're here, I'm going to ask you is, is there any kind of crossover between the two? And so I'm assuming not in code. How does, how, how is the, how is the relationship between the two and how's it, well,

Padraig: definitely not in cold. There, there is a little interaction sometimes between Suggested new options and stuff like that.

One interesting thing, like busy Box is more geared towards embedded systems. Yeah. And it has and there's different licensing and stuff to, to date with that. So, so the, the, the main difference there is licensing. Interestingly, a while ago, um, core Utils was adjusted to be able to be built in the same single binary setup as busy box.

So. So the standard way you would build BusyBox is as a single binary with symlinks mapping the various command names to the single binary, and the core utils can be built exactly the same way. So you can install core utils in I think it's a, it's a couple of megabytes with sim links to the single binary.

So, so from a functionality point of view Coriutos provides the same things, I guess, with more portability. But yeah, there's the licensing aspect, the main difference there.

Dan: Now, you mentioned licensing and my eyes lit up because I'm a bit of a licensing nerd. I don't know why I just find this stuff fascinating.

Now I, I was working that I had, I was interested that you're under GPL V3 with core utils and it's, it moved from GPL V2 or newer to GPL V3 or newer. I believe. about 2003 or something like that. So were you around at the time? And what was the process like?

Padraig: I wasn't involved in that licensing. I had a few earlier patches, and then I joined really the project a bit after that.

In a, in a more involved way, I guess. So I wasn't involved in the licensing and I haven't really been involved in the licensing, mm-Hmm. any licensing issues or anything like that going forward. So I'm the wrong person to talk really about that. I'm more more focused on it. That's okay. The, the technical and, and to be honest the, all the, the core maintainers were, were focused on the, the technical aspects really.

But we are aware of the. I kind of, some of the restrictions of GPL 3 just even from a political point of view some people just kind of shy away from it and keep things simple and just avoid it. So that, that's, that's not an idea.

Dan: Yeah, it's an interesting one. I, I, I've got, so the kind of little background to this for me is I have some friends who work at the Software Freedom Conservancy who were involved in a lot of that kind of, like you know, That's licensing.

That's Licensing Central right there. And then I know that they were very keen to get projects to move from v2 to v3 of the GPL, and some were keen and some were not, and some still haven't, like the kernel of course. And so I just thought I'd ask, but that's fair, totally fair enough, that you weren't around or you weren't involved with the licensing at the time.

Now, one of the things you mentioned to us in your email was that you worked for Meta for a good long time and their use of Linux in the back and the stuff that you've done with them. So I'm really interested to dig into some of that because I know that they have their own distribution. How does that work?

Padraig: Sure, I can't go into very details about absolute numbers or anything like that, or details, but I'm happy to go into general general information, and a lot of this information applies to all the big tech companies like Google and Amazon, that they all use the same sort of models. Stuff like that.

So I guess the interesting kind of general information with Meta is to have a huge scalability requirements. So a huge focus on performance, like if you've got, for example, a 1 percent win out of compiler, they're talking hundreds of millions of dollars. The scalability is immense, but it's interesting as well, not in particular to core users, but maybe more aligned with a project like glibc or that kind of library code.

That when you're working on these things in open source, it's used by Meta and Google and everybody else. So it actually has much larger scalability concerns, but it's not really as apparent or, I guess, measurable. to these people. So it's there is a, I guess, there's a huge onus and responsibility on people working on performance.

Like, when you make a change in meta, it's, everything is easily, very easily tested. And You can see exactly the dollar values of changes. And so that's cool as well, because you can feed those, it's easier to test, it's very hard to test things kind of open, open source world, because You have a lot more, it's just harder to test because you have a lot more disparate systems and you haven't everything as tightly cohesive in a whole test framework and stuff like that.

So there's a kind of a symbiotic relationship both ways, like you get really good testing in a place like meta, and then you can feed that information back up and feed the code back up. But, but there's an interesting thing as well that, like, Corporations like these, they get huge use out of open source code, but sometimes there isn't the, I guess, the focus on sending code changes back up because just looking at the loop, there's a short term win from not spending the time to send your changes back up in top string.

But then there's a long term loss by not doing that because you become forked and you kind of diverge away from all the improvements upstream as well. So that was a good bit of effort on my part in META was kind of ensuring that all our internal processes and changes were a bit more. Getting code back open into the o open source.

So, so we didn't become diverged and, and it's something that that's really easy to fall into, but because of the short term win and not doing it like, like even open source first, companies like Red Hat back in the day, they, they originally got into that, that sort of situation. They had had a fourth kernel and it's, they, they actually got into a, a bit of a knot that, that they had a huge effort then to get outta that.

So it's just an interesting thing for any tech company or any company these days, since tech is involved in most things. They have to be very wary of not forking away from upstream too much.

Dan: It's good that that they, well, it's great that, that it sounds like they were supportive at least in, in you contributing stuff back upstream.

Was that something that, like, the management, without meaning to cause any trouble here, and feel free to tell me to, you know, you don't want to talk about it. But I was just curious, were the management and so on, were they supportive in that? Were they like, yay, go and do it? take a day to do this or, you know, support it.

Padraig: Absolutely. Like like in Meta at a certain level, especially you were encouraged to go off, you were trusted to do the right thing as long as you presented the right arguments. Like they were happy as long as you're doing the right thing. So, so within Meta was very open source friendly. And increasingly going forward now with the AI ecosystems that they're really kind of leveraging the kind of the open source aspect of that.

So, so, so no, no, there was great support in, in meta. I was kind of more alluding to the kind of the general kind of thought process of engineers in general within tech companies. They were focused on the short term wins not, not, not on getting their like we're focused we're working away in open source a lot.

And most developers haven't that mindset of pushing stuff back up.

Dan: Yeah. That makes sense. We actually have a question. So there are people listening and watching us at the moment. And we have a question from Mashed Potato, who says, What's the relationship like with developers of Core Utils replacement tools, such as Exa?

Could Exa find its way into Core Utils to replace LS? Is there a good reason to keep LS as it is? That's quite a big question.

Padraig: Well, absolutely. I've, it's a long time since I looked at Exa. There are a few interesting, like, I do have a look at every so often at tools like this, and if there are interesting general functionality that would be appropriate for everybody to use, absolutely we would incorporate it into LS. The big thing you have to be aware about with making changes is the interface.

You, Wouldn't move everybody to having to type exa ls is kind of wired into everything at the moment. So but you could add functionality or the ideal thing is to adjust things.

You have to be careful as well, but the ideal thing is to adjust things without requiring options or changes on behalf of the user. But you have to be careful. So, LS is an interesting one. So, it's an end user tool. So, we made changes recently and there were really good reasons for making the changes, which was to quote filenames that had Problematic characters in them, like spaces or shell characters that were special to, to shell becau because it, it introduced un, unless you quoted the names, there would be ambiguity with the, the spacing for delimiting file names or were the spaces within the file names.

Or you could have semicolons and you could put commands in there. So if people are copying and pasting and you can, there, there's a lot of way to, ways to hide stuff. So, so there, there was security implications there. So by default, we now quote the output of problematic filenames from LS. And there was actually quite a lot of pushback from that, from various people.

And the main reason is because they weren't used to it. And

like we of course provided, always provided a way to go back to the old behavior if we really wanted. But yeah, you just have to be very careful about how you adjust these tools, especially stuff user facing stuff like Ls.

Jonathan: Yeah, it's interesting. I just went and looked. The, the EXA command itself has been retired and a fork.

Iza is now what is the what is the latest and greatest. So and I think, so yeah, it's, it's interesting to look at it cause there's going to be some great ideas there. But what's also fascinating is that Core Utils has been around since 1992 and Exa only lasted for, you know, however many years.

And now the, the main developer of Exa is missing and cannot be contacted anymore. Like just. Just the fact that Core Utils has been successfully maintained for that long is like a, it's a huge win and not, not every project can say that, that they have, you know, that sort of a track record that 20 years from now, you can pretty much guarantee that Core Utils is going to be around and still putting out releases.

Padraig: Yeah, absolutely. And like for companies kind of investing on a platform and they're writing shellscripts and stuff like that, depending on all these utils, there's a huge kind of responsibility for these things to stick around and be stable. And just another aspect of that example, it's good to expand on a particular example and look at all aspects of it.

Like if we were to, say, take things out of ESA, now is it? And add them into LS, there might be a little bit of extra performance in every LS invocation, which that gets back to the point about how there's a responsibility on us to mine performance because it moves out into all companies and all users.

There was one example there recently where file capabilities were colored, and file capabilities are one of the things that never really took off in Linux. So, maybe one in a million files has capabilities now. So there's no real Advantage of coloring. So there's an overhead there of every LLS invocation looking at every file to see does it have file capabilities.

And it's one of these kind of esoteric things. So it was never kind of the interfaces to detect file capabilities was never optimised over time. So kind of a rule of thumb. Or kind of a what I call it, a back of the paper quick calculation I did was that by taking that out of LS, nobody really noticed, but it saved about 75, 000 worth of electricity a year, just estimating the use of LS around the world.

And so, which is, I don't know, maybe 40 households of electricity. And just. By not having this extra little bit of functionality in LS. So these things are important.

Jonathan: Oh, I'm so tempted to say something sarcastic about cryptocurrency there, but I think I'm, I think I'm going to not. So that, that idea of performance, though, that does bring to mind a really interesting question, and that is when Say AMD pushes out their newest processor and it's got AVX 512 support for everybody now.

Of course, Intel has gated that off to their pro line of processors. Are there changes that get made to some of these core utils? Because, Hey, now we have AVX 512, and oh, you can do string comparisons in AVX 512. Like, are you guys sort of on the cutting edge of that, watching those changes in processors, and then therefore going in and making tuning changes in some of these commands?

Padraig: I wouldn't say we're on the cutting edge, but we're definitely incorporating changes such as that. There's two aspects of that. Again, we try not to implement everything ourselves. So, looking at the crypto side of things, or the hash side of things, we, rather than implementing Assembly slash Cindy versions of those ourselves, we kind of push off to live crypto or the open SSL libraries because they're, they're kind of ubiquitously available as well.

So we'll, and we'll with version three, we can, and the licensing changes there, we can without issue link to those. So, so we, we'll link to those by default if they're available and use the fast version. So, so the checks on or cha 2 5 6 on or whatever. We'll use the, the, the latest version of those, but, but also within like core functionality ourselves.

Like for example, wC for counting lines that can be done efficiently in SIMD code and AVX code. And, yeah, we do have code for that. We have to be careful in portability, so there are special portability constraints in how we set up the build, so we actually added libraries to the build system to support that.

That's how you efficiently are. Kind of definitely separate all AVX instructions into separate compilation units in Automake world. So we have libraries that we, internal libraries that we link to to implement that for a few utilities now, and probably more going forward.

Jonathan: Yeah, I know kernel and GCC itself, You have a MD and Intel employees that come through and, and send these big patch sets in.

Like, here's support or here's the tuning for the latest, the latest and greatest from our company. Do you guys get any of that in Core Utils? Are there, are [email protected] [email protected] email addresses, sending patches in?

Padraig: No, to be honest I've had interaction with those guys from working at Meta and ver various places.

And various other projects, but not directly in core utils.

Jonathan: Okay. Let's see. I was going to ask about Dan. What was I going to ask about? That was, that was a short answer. I was going to look it up. Retiring commands. That's what it was. There are, there are some core utils commands that have been around for.

for years, for decades. And it's like, some of these I don't think anyone has really used in decades. And is, have you, is there a thought about, well, let's just retire these rather than making them continue to be part of the the, the maintenance burden, or are they just, are they going to be around forever?

Padraig: Good question. So there's two aspects to that, really. There's individual commands and then there are classes of commands. So answering classes of commands at the initially. Like, for example, all the, the separate checksum commands, like nd5sum, sha1sum, sha25sum, blah, I'm not going to go on, but you could go on forever there, and that's kind of a bad way to do it, to have a separate command for each of those.

So, going forward, we're consolidating those in the checksum checksum minus a, then you select your algorithm there, and so, We won't have any more of that class of command. Everything will be consolidated in checksum. And I guess in 20 years time we'll start removing SHA 386 sum or whatever, just to clean things up.

So individual commands then, yeah, there are some commands that are less used, like ptx, tsort, this sort of thing. I guess the main idea there is that there's less maintenance. We don't much maintenance focus. Like if we get a compiler warning out of them, we'll fix it up. Or if we get some security warning or whatever, we'll maintain it.

But we won't put much effort into adding new features or changing functionality on those going forward. That's the main thing. As for removing command, there's the big compatibility concern. Like These are, these utilities are so used and that there's some edge case on. Some space probe in Mars or something, we just can't, we can't, we probably can't remove, remove commands, so it wouldn't be that much maintenance going forward.

And on the other hand, that's why we have to be very careful of adding commands. And even options in that regard, but in added commands, we have to be very careful.

Jonathan: Yeah. All right. I do want to make sure and ask, is there anything that is coming that you wanted to let folks know about? Like, are there any future plans that you're particularly excited about?

Padraig: I guess the main one, you already asked about the internationalization and Unicode support. So, so that is something that's coming gradually. And we're definitely focused on that, and it's happening as we speak. In the last release or two, there have been updates to expand and un expand and utilities like that for them to handle multiple characters, so that will be coming.

Perhaps there might be a new utility. We've moved it for a while about a replace utility. So it's interesting. To replace a file on Unix is actually really difficult to do it with ACID principles, and copying data around with ACID principles is actually really difficult. So, kind of makes a lot of sense to have that encapsulated in a separate command.

Like rather than have sed minus i, you can have The replace command and provide another command to do the actual processing and then the replace would do all the complicated stuff about temporary files or kind of moving files to atomically and and all that anarchy. That, that's something that, that might be on the horizon.

At the end, there's. There's always the ongoing maintenance of new kernel interfaces for example, one thing that's changed recently that might allude to things going forward is we added and we had to be very careful in how we added copy offloading. So copying a file is actually really primitive in a POSIX interface.

It's like you have to copy the metadata separate to the data. You have to, there's atomicity issues there as well, which gets back to the replace command. But recently there's been the copy range command to in the Linux kernel and similar commands elsewhere or similar functions elsewhere to provide copy offloading.

And but we locked the doors maybe 10 years ago. and had a deep dive on those, and they weren't stable enough either in interface or functionality. So we provided feedback to the kernel folks then, and more recently in the last two or three years we've been able to start using these things. So that allows for more efficient copying operations, but we still have to be careful and actively inspect and avoid older kernels and stuff like that.

So, there's going to be changes like that going forward with new kernel interfaces.

Dan: Oh, awesome. So I'm curious if somebody say, listen to this decides, you know, I'd like to get involved with with core utils. That seems like my kind of thing. I mean, I'm always keen to ask people who've made a career out of this and who, you know, have contributed and become maintainers on these projects.

Do you have any advice to somebody? How? What's the best way to get involved and to you. You know, to, to come along, is there anything you particularly need, say, from the community that you think would be great if we had somebody who could do X, Y, Z?

Padraig: Well, there's a, there's a to do file in the repo, in the main repo.

There's a, If you look on the main, the main GNU Core Utils mailing list there's, if you, if you sit on that for any period of time, really, you'll get an idea of the, the work we're interested in doing and what, what, generally the work that's, needs to be doing and interested in doing. There's no we don't have a Well, we do, I guess I'm saying we don't have a bug tracker, but we do have if you go to bugs.

gnu. org, uh, slash there's a core utils section there. And so there are some outstanding bugs that need to be handled as well. And just in general, we're always very accepting of Just interactions and patches are on the list, and we'll try and guide people Like, we're very, very interested in getting code and new people involved in the project.

Dan: Excellent. Yeah, I mean, that's the lifeblood of every project, I suppose. So, Jonathan and I have just been having a little discussion in the back channel here about, about GitHub, because I didn't think you used GitHub, but apparently Jonathan says you are on GitHub, and he want, being that we talked about contribution and so on, we, I was curious, do you, do you accept pull requests from GitHub?

Padraig: We kind of do unofficially, right? So, so we, we, we have a, a GitHub mirror and we, we probably changed that a little bit going forward, but because it's become more, GitHub has become more ubiquitous since we, we had a policy. So currently you can make a pull request against Core, which I set up about 10 years ago.

And, just to consolidate things on the mailing list, we give people advice, like when you make a pull request, as the main commit message it puts in, that you should send it to the mailing list, but you can still create the pull request. So, so we might change the, the policy there to, that we will support pull requests separately as well, because yeah, and I have looked at pull requests there over the last while, and I do monitor that all the time.

So, yep. That's actually, I'll add that to my to do list this evening to change the wording there. At least we're giving you work to do, I feel bad now.

Jonathan: Oh, no, no, that's our job. No, that's interesting to me. I get the, I very much get the the hesitation that particularly the, the thing that I've noticed is that a lot of people, particularly free software projects have, and GNU projects have, about using GitHub, because it's not all open source software.

And I definitely get why people don't love that. But at the same time, it's so useful. And it, like you said, it's become so ubiquitous that it makes things so much easier for people to come in and add, you know, a trivial or a small pull request without having to go to the mailing list. A lot, a lot of projects have arisen with this.

Padraig: Yeah, I don't wrestle with it too much. I'm definitely on the side of defense that I'm not kind of. Very binary on you can't do this and you can't do that. The, the world is gray. So I, I, I'm on the, the side of the fence that whatever gets the, the most, the, the best logic in to the most people is best and mm-Hmm , most people are used to using GitHub, so I have no issue whether using GitHub at all.

Jonathan: Yeah. That seems to be where a lot of projects are coming down. Alright. Is there anything that we didn't ask you that you wanted to make sure and let folks know about? And I know that's a tough question because you have to. think about all the things you wanted to talk about, and all the things we've talked about, and do that set comparison, but Dan, do it, Dan.

Padraig: No, I don't, I'm just, I scanned my notes here, and no, I think that that was a very all encompassing set of questions, so I think you've got everything there.

Jonathan: Yeah, we try, we try. What's the weirdest thing that somebody has done with Core Utils that you're aware of? Have you gotten any user stories that have just surprised you?

Or maybe, you know, requests that have just been off the wall are very surprising?

Padraig: Look, you know, that's Jesus, nothing goes to mind there.

Jonathan: Maybe the, maybe the recurse up the tree was the answer

Padraig: there. That was the most off the wall one, I guess. There was an interesting set of videos I saw from, I think it's Robert Elder.

He, he went through a set of core utils recently, and he did a set of videos where every command, he started off that every command was his favorite command, but he had a very interesting use of timeout for Avoiding saturating his network with a backup. So he'd start off a backup, but if the backup, so the backup would keep copying stuff that hadn't copied already, but he'd time it out after two hours.

So I thought, I just thought that was a very interesting insight. Time out his backup. But it would get, but it would run the next night, and then it had left off where it went before, so it would eventually, eventually copy his files across, but you know, there's lots of esoteric uses. I, so many, I guess so many esoteric uses that I can't remember many.

Very many particular ones over time.

Jonathan: Yeah, that's the thing. You have, you have people that write these one liners where it's 15 different commands and most of them are going to be core utils commands and they do something, you know, ridiculous or really impressive. So there's a bunch of them out there,

Padraig: but on the other hand, like looking at questions on the stack overflow and stuff like that.

And there's, I just, People are, I don't know why they do things the way they do, like, it's just, I should have a section, I definitely don't do this with core utils commands, but, each to their own, you know?

Jonathan: That would be amusing to read that FAQ, the things you definitely shouldn't do with core utils. Oh, that'd be great.

All right, well, I've got to ask you before we let you go, what is your favorite scripting language and text editor? Interesting.

Padraig: Interesting. Scripting language. Well, what is a scripting language? That's true. I'd say my favorite interpreted language is Python. So does that count? Yeah, absolutely.

Dan: Yeah, yeah, definitely.

Yeah,

Padraig: that's one I've used for a long time now. And I get the most flexibility and functionality out of it.

Jonathan: And text editor.

Padraig: Text editor, I use VI. So, an interesting part of that, that's worth mentioning quickly, is that the interfaces, I use CLI a lot, I guess, obviously. And so the main, like UNIX has not been designed kind of cohesively, it's kind of evolved in separate kind of factions with SysV and BSD and stuff like that.

So the kind of, the interfaces have kind of evolved to be bash, or, sorry. Vi versus Emacs in various commands. So I've set up all my systems to have Vi key bindings, and it actually helps with the speed of the interface and helps with RSI, which I don't have any RSI ever, but maybe that's, maybe that's the reason.

Jonathan: It's gonna be the secret why. It's funny you ask what what counts as a scripting language. We had, we had the creator of Bash on back several years ago, and I asked him if Bash counted as a scripting language, and I think he was sort of offended that I asked the question. And Yeah

Padraig: Yeah, it's one of these things.

Jonathan: Yeah. Yep. All right. All right. Thank you so much for being here The hour flew by and we had a great time and I think really covered a good Corpus of information on the core utils and it was it was a blast to have you appreciate it Absolutely.

Padraig: Thanks,

Jonathan: man.

Padraig: Much

Jonathan: appreciated. Yeah. All right. What what do you think?

Dan: Yeah, I think as, as I as I kind of said at the top of the show, everybody's used core utils, whether they know it or not, I think, anyway. I say everybody, you know, I would imagine these days with the amount of things we've got in devices that are running all kinds of different stuff. And one thing I actually found very interesting that Pyrog We told us about was because we mentioned BusyBox and embedded.

I've got a lot of friends who work in embedded Linux and in that kind of world. And I didn't realize you could get like a two megabyte implementation of, of, of core utils. Now, I don't know if they know that or if people from busy, because I also know people who work with BusyBox and they're going to hate me.

If I say don't use that, use core utils, it'd be better, but you know, it's just these options, isn't it? But very interesting stuff. And I think the ubiquity of, of this is, is, is so amazing and the responsibility that it puts on maintainers like, like Parag and the other people involved that this is used everywhere.

So I imagine that, you know, it, it, it must be quite a weight to kind of bear.

Jonathan: Yeah, yeah, it's like we said, it's, it's it says a lot about the project and the way that they run it, that it's been around for so long and it's still a healthy project. So that's, that's neat to see. And I also find it, I find it real fascinating.

So I, I couldn't help but think when they were talking about how they, they very carefully make changes. There's this line from the Lord of the Rings books where they, they discover some like wonderful cavern and Gimli tells Legolas, That he wished he had some of his kin to come and work on it in Legolas.

It's like, you would make changes to that? It's so beautiful. And Gimli says some line about, we would only remove one rock every hundred years to try to make it better. And that's kind of, that's kind of how I feel that they're working with Coriutils. We make one breaking change every 20 years. We didn't, we didn't get to ask him about it, but apparently Coriutils itself has.

a really good record when it comes to security as well. Something like only five CVEs in the last, you know, 13 or 20 years, something like that. And I wish I thought of that during the show to ask, to ask him about, but it's, they're really doing great work. Absolutely.

Dan: Yeah, definitely.

Jonathan: All right, well I think that is, that is it for Core Utils.

Do you have anything that you want to plug, Dan?

Dan: Yeah, I should mention some people, longtime listeners of, of FlossWeek who will remember an event called OggCamp that I used to run in the UK, which is a free software open source unconference, bar camp style event. It was at one time, the biggest in the UK.

I don't know if it still is. It probably it's hard to say that it still is. Cause this is the first one for five years. So there's going to be one. So the, what I'm doing here, burying the leaders, there's going to be one in October and it's been picked up. Thankfully we got stopped by, we got stopped in our tracks by COVID as a lot of people did.

But the events coming back and it's going to be in Manchester. It's in October this year. You can go to ogcamp. org. That's O G G C A M P. And you can get tickets on there. You can find out what's going on. And I'm not actually organizing it anymore, but but Simon Phipps of this parish is is on the organizing team.

And if you want to get involved, you want to find out more about it, you can come to Manchester and you want to you want to talk or, or, or you know, do a workshop or any of those sorts of things, then let us know and head to orgcamp. org.

Jonathan: Yeah, excellent. So things I want to plug is, of course, we thank Kakaday for being the new home of Floss Weekly and you should make sure and tune in next week because we're talking to Carl Ritchel about Cosmic.

The new alpha release of Cosmic is out with their Rust based compositor, some really fun stuff there. I'm going to chat with him about it. And then of course you can follow my security column goes live Friday mornings on Hackaday. And then we've still got the Untitled Linux show over at Twit. And that is always a blast.

That is every Saturday. And we have a lot of fun there talking about Linux news and some open source news as well, but it, that's a, that's a much more Linux flavored show than this one is. But you should definitely check that out as well. I very much appreciate Dan being here as co host and we appreciate everyone that watches and listens both live and on the download and we will see you next week on Floss Weekly.

This week Jonathan Bennett and Dan Lynch chat with Pádraig Brady about Coreutils! It's been around since the 90s, and is still a healthy project under active development. And you've used these tools whether you realize it or not!

- https://www.gnu.org/software/coreutils

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

This week Jonathan Bennett and David Ruggles chat with John Britton and Mike McQuaid about Homebrew, the missing package manager for macOS, and Workbrew, the commercial offering built on top of it. We cover lots of territory, like why the naming scheme sounds like it was conceived during a pub visit, how Workbrew helps businesses actually use Homebrew, and why you might even want to run Homebrew on a Linux machine!

You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account!

Theme music: "Newer Wave" Kevin MacLeod (incompetech.com)

Licensed under Creative Commons: By Attribution 4.0 License

http://creativecommons.org/licenses/by/4.0/

Jonathan: Hey, this week David joins me and we talk with John Britton and Mike McQuaid about Homebrew, the package manager that macOS is missing, and Workbrew, the new commercial offering built on top of it. It's a lot of fun, you don't want to miss it, so stay tuned. This is Floss Weekly, episode 796, recorded August 13th.

Homebrew! I'm more of a Whopper guy.

Hey folks, it's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something real fun today. We've got, first off, David is in the secondary hot seat, the co pilot chair. He's my wingman. He is the co host today. David Ruggles.

David: I'm holding it down. Yes. I'm holding it down.

Jonathan: Yeah. Now today our, our topic is. Well, it's, it's Homebrew, which is the package manager that Mac OS wishes it had. And neither of us are big Mac guys, are we?

David: Hmm. No I use it. At least once a week, but not extensively. So yes, I have, I have lots of questions that will be the, uh, Luddite.

Jonathan: You're our audience proxy for, I don't know much about this. That happens. That's fine. I have, I've actually used homebrew back years ago we were talking about before the show, I was a part of an organization and someone else that was there was a huge Mac fan. And so they bought some Mac machines. And the whole time I was going, he was.

Associated with the military, so the whole time I'm going, I know he's going to move away, and I'm going to be the one stuck administering these machines. And guess what happened? Yeah, I was stuck administering the machines. So, I installed Linux on one of them, and I installed Homebrew on the other. And we, we made out very well with that.

So I've got a little bit of homebrew experience and a little bit of Mac experience, but, you know, rather than us talking about what we know and don't know about the project, let's bring, let's bring the guys on. So we have John Britton and Mike McQuaid. Let me see if I have this right. John is the business guy.

And Mike is the homebrew guy. Is that, is that sort of the way the land lays here? 

Mike: Yeah, I don't know if John likes it put that way, but I'm fine with that description.

John: Yeah. I mean, I definitely wear more of the business hat, but I'm also a software engineer. So, you know, being called a business guy is a little bit rough right now.

Jonathan:  It looks like you're wearing the Tron hat. Yeah, I like it. Okay. So first off, how did, how did we do about, about homebrew? Let's start there and then we'll get into this kind of the, the, the business thing that's going on, but I want to know first, let our folks know. And so obviously this is probably a question mostly for Mike.

What, why, why would somebody use homebrew? What's the point?

Mike: So my I guess description for non tech friends and family is generally homebrew is a app store for open source software, essentially. And if they want to get deeper on that, you can go down the line, the fact that it's mainly both the software installed by Homebrew and Homebrew itself are primarily operated through the terminal.

So yeah, basically that's, that's the starting point of why you care about homebrew.

Jonathan: Yeah. And it's, it's all, is it all open source? Like, so I, I'm pretty sure that homebrew itself, the entire thing is open source, but all of the software that you go out and grab now, how much of it do you build from source at install?

It almost seems like that's one of the things that you at least can do.

Mike: Yeah, so that's a differentiation between kind of two parts of homebrew, like the original and some stuff that's come later. So originally homebrew was, everything was built from source on the user's machine. So I guess you would call it like a build from source package manager.

Yeah,

and over time homebrew decided that we were going to take more of that open source software that was built from source and build it ourselves. And then now we build everything ourselves. So most users most of the time are going to be given a pre compiled binary package, what we like to call a bottle because everything in homebrew has like a beer metaphor running through it.

Right. But then over time there was a project that it started off as its own separate project, but it's been brought into homebrew proper now called casks. So we have things called formula, which are used to install things from source, which are open source software. And then we have casks, which are used to install Software that is a binary that we get from somewhere else.

So a classic example of a formula might be something like W get or some other command line tool you're used to interacting with or my SQL or database or whatever. And then a classic example of a cost might be something like. One password or Google Chrome or zoom or something like that, where essentially the, the flow otherwise would be download from a browser, click, click, click, whatever.

Jonathan: How long, how long has homebrew been around? Like what, when was the the initiator of this idea? When did somebody first start writing code to make it happen?

Mike: So yeah, Homebrew turned 15 this year in May, if I remember quickly, it was created by a chap in London called Max Howell who was working for Last.

fm at the time. That may bring back memories for some of you that's, I'm sure they still exist, but you know, less widely used nowadays. They once were so yeah, he basically was exploring different package managers. And he could never find one that kind of quite, met his needs and I think he was nudged by someone in the pub one evening to go and well you know if you hate them all so much why don't you make your own one and he did and that's that's the future really and so yeah so that that was 2009 and then I got involved with homebrew Later on that year, like, I guess, September 2009 or so I, Max was a friend of a friend in London and I heard about it and I was also dabbling in package manager things and I sort of just started contributing and never really stopped, I guess.

So yeah, 15 years ago is a couple of,

John: I was just going to say a couple of months ago we had Max over and we did a live stream kind of going through the history of 15 years of homebrew as well.

Jonathan: Okay, cool. Is Max still, is Max still involved with the project?

Mike: No, he's doing his own things nowadays. Sure. He was involved for kind of, I guess, maybe five years at the beginning, and then he sort of handed it off to others.

Jonathan: Yeah, that, that's great. It's one of the, it's one of the problems we see with some open source projects. It's like even really popular ones where somebody starts it and it's like, well, I guess enough people like it that I'm stuck here for life. And so like genuinely good for him that he was able to get off the boat.

And the boat didn't crash.

John: Congratulations. Your open source project is successful. You have an unpaid job for the rest of your life.

Jonathan: Yes. Well, it's a, it's a huge problem. It really is. And you know, there's the, the, the classic XKCD comic. Like you have this, the whole, you know, massive software stack, all the building blocks, and you have this one little piece that holds everything else up.

And it's like, this is just maintained for nothing by, by one guy in Kansas. And the scary thing is there's multiple projects like that. Like you can talk about the network time protocol has been that way for the longest time. Even things people don't know about, but are super important, like the term info files.

And there's legitimately only like three people in the entire world that actually understand term info. I think I'm, I'm fully convinced of this, but you know, if, if those break, we're all in trouble. So. Let's, this seems like a good place to at least dabble in the concept. I, I've heard of something called workbrew, and it is apparently related to homebrew, and it is apparently also you guys.

What, and this is probably, if I understand the lay of the land correctly, this is a question for John. What is workbrew, and why, why are we trying to do it?

John: Yeah, so homebrew, is, you know, insanely popular, used on Mac OS. There's more than 30 million devices with Homebrew installed. And it's pretty much made to be a single player experience.

You sign on to your machine, you open up a terminal, you install some things, it's all managed kind of on your machine. And what we're doing with Workbrew is trying to make it so that It's more of a multiplayer experience. It's built for teams. It's built for companies so that you can have a kind of a shared set of developer environments across all your machines, a shared set of policies, a shared way to manage and deploy and install, get analytics and observability, know what's going on within your fleet and keep everything secure and compliant.

So what we're building now is really. focused on how teams are using Homebrew day to day and trying to solve their major problems. So I think you know, maybe Mike, you want to say a little bit more, but I'd say that's, that's kind of the starting point.

Jonathan: Yeah.

Mike: Yeah, no, I think that's

Jonathan: a good start.

Okay. Now is, is Workbrew open source as well?

John: Workbrew is not open source. We're built on top of Homebrew. I like to think about it as kind of complimentary. So at the foundation of work brew, we're using brew, the open source package manager. We don't have a private fork. We don't have our own custom version.

It's exactly the same as the open source project, but on top of it, we build a bunch of complimentary tools for deployment, analytics remote management, security and compliance those types of things. And our approach, I think this is actually probably the topic that's worth getting into in a podcast like this.

Sure. Is our approach to commercial software in an open source world. So as you know, homebrew is an MIT licensed open source project. We, BSD, BSD, apologies BSD licensed open source project. And we basically upstream all of our changes that are necessary. So anything that, that needs to be. Available inside of brew as the core kind of foundation we make available to everybody.

Then there's kind of a line and that line is pretty well defined and it's multiplayer. It's enterprise like kind of security and compliance features. It's managing remotely. It's all of those types of things that are on top of homebrew that are closed source. We don't make that available open source right now.

Jonathan: Yeah, and that makes sense. I think it's going to serve you well that you've, you've figured out that line and you have a a clear delineation and you make it clear upfront that like, this is the, this is the part that's always going to be open source. This is the way we're going to manage that. And then this is the part that we're going to build on top of it.

That's not. And at least I, as a potential user, I, I really appreciate that. And so do you, do you generally pull from the same like list of packages? So for one, I understand WorkBrew is going to be doing much the same thing. We're, we're installing extra packages on Mac. I think

John: something that's, that's useful to explain about Brew is kind of the two components.

One component is the CLI. So it's the actual package manager responsible for putting software onto your device. And then the second kind of component. Is the packages themselves, the library of, you know, as Mike was saying, formula and casks The homebrew project has two official kind of package repositories.

One's called core and one's called cask. In homebrew core, you'll see all of your built from source packages and in cask, you'll see all your binary distributions. A lot of that stuff may not be open source. Some of it may be open source to distribute as a binary from the upstream vendor, but ultimately you have those two components.

So when it comes to workbrew you're still using brew, the open source package manager, and you're still able to use the library of packages that are available for homebrew in core and in cask, but you can also create your own taps as they're called in homebrew speak that are repositories of your own internal packages distributed, you know, within your organization and you can kind of set rules and, and manage how that stuff is done at an organization level.

So that's kind of where the delineation is.

Jonathan: Yeah, that that makes a lot of sense. Yeah, David, you I'm sure you're chomping at the bit. We haven't let you got anything in yet.

David: So a couple of questions just from listening to you there. The first one, I noticed that you refer to brew and homebrew and workbrew.

So what are the, like is Brew core to both Homebrew and Work Brew, or is Brew and Homebrew synonymous? And just a shorthand. I'll

Mike: get this one. So essentially like we, we often use brew as a shorthand for either home brew or work brew because Brew is, that's the, the CLI. command that you type in to your terminal if you're Workbrew.

I guess to jump back a little bit because it might be interesting again, like the approach that we're taking and it might explain how things fit together. So essentially the flow for Workbrew is you run our installer. And our installer installs some workbrew stuff and it installs homebrew so homebrew is installed completely normally to the normal location but we just do some stuff where if you were to go and say that to the homebrew open source project they would be like why would you do that but because because we have a bunch of homebrew maintainers and People like me who have worked on homebrew for 15 years, like we can do some slightly more unconventional things with homebrew.

But on your system you still have a completely unpatched, normal, open source version of homebrew running on your system. But then we have essentially like a wrapper that we have on top of that. So when you run brew and you have workbrew installed in your system, you run workbrew, which then calls into homebrew.

But the essential behavior for end users is it looks exactly the same. So I guess John talked on this earlier, like me as a kind of long term lover of open source software, I kind of like this approach of kind of combining the two because it means that Both the open source software, Homebrew, gets better over time, but also we have the way, as John mentioned, like the realities of kind of making commercial software in an open source world where we can't give away everything for free or it wouldn't be a business, right?

So, yeah, I think you get the nice best of both worlds. And the other kind of fun, I guess, analogy I've used with this stuff is it's like My apologies to John, who's heard this particular analogy about 8, 000 times is it's like Lego, right? I don't know how much any of you are playing with Lego nowadays, like, but my, one of my kids is super into it.

And Lego now feels pretty different to what it used to be, where I remember like there was a lot more modification of models and stuff like that. Whereas now it's like a pretty hard line between like the super, customized, this Lego T Rex comes with a particular claw that is made only for this one piece and you can't use it for anything else really.

Or you just buy a bucket of a bucket of blocks and you assemble it and make it up yourself, right? Like essentially what we're doing with with workbrew is if, if you look through the pull requests that are open by me and other people in the last year or so, you can probably see the blocks of how you could go about building your own workbrew.

But I guess our value proposition is essentially like, well, we, we built it for you and we can support it for you and everything like that. So. You know, maybe you're better to buy it off us, but the open source project still has a lot more of these kind of like hooks and things that you can plug into that makes the open source project better and more useful for people as it goes along.

Jonathan: Yeah,

Mike: absolutely.

David: And the the next question that I had as I already established at the opening, I'm not great I don't have a lot of Mac experience other than some user and I've maybe I've used brew once or twice. But I do have Linux experience an open source experience. And so kind of relating brew to.

Package managers, I'm used to I have two questions and you can answer them. However you desire. The first one is, do you have package maintainers that are responsible for specific Applications within, you know, that homebrew would pull down. And then the second one is, you mentioned taps as something you could do inside your corporation as kind of like your own repository.

Are there. Publicly available taps that are like maintained by people not directly related to homebrew. I'm thinking of kind of like PPAs and Ubuntu.

Mike: Yep. So in terms of package maintainers, I guess that's an interesting thing with homebrew is that we don't have specific package maintainers. We have like somewhat officially unofficially People who might maintain individual packages, but the maintainers of homebrew are relatively few.

So we've got, I think, 30 something people right now and between them, they essentially maintain everything. But that doesn't mean that they do everything because homebrew was built with GitHub and collaboration in mind from the outside. I think Max on the, the video call that John mentioned earlier that we had to kind of talk about it, like one of his things from the outside.

I'm sure you won't mind me saying this if you didn't use this exact word was essentially to be lazy and be lazy. Okay, I don't want to maintain everything by myself, so how can I build this from the outset such that essentially the community maintains this and I don't? The other difference we have nowadays is we have very, very, very heavy amounts of automation to essentially detect changes and keep things up to date and things like that.

But I guess to specifically answer your question, though, we have a slightly different model where We don't have like hundreds of different people who each maintain one or two packages. We have a small number of people who maintain everything and like review community contributions to all of those things.

The next question about taps. Yeah, we have essentially the tap model is very similar to a PPA model or something like that where any arbitrary person on the internet can just decide to set up a a tap and then by default the easiest flow is having them on github but really you can put them anywhere where you can have a git repository.

I mean technically they're just a folder on a disk so any way that you can get a folder onto a disk and keep it relatively in sync you can make that a tap and they behave much in the same way regardless of whether they're being maintained by homebrew ourselves or whether they're being maintained by the community.

It's more just like we set a higher standard for like both. The licenses and stuff like that and styling and, you know, making sure our best practices are followed that the community don't have to do, and may decide they don't want to do, may decide they can't be bothered to do and that's basically how that kind of ecosystem fits in.

David: So, one follow up question. How many packages is everything?

Mike: Right now, I think it's about 20, 000. Oh wow. Between all of the formula and all of the clasks, so 20, 000 official ones. So if you go wider onto the, all of the third party taps and stuff as well there is probably considerably more than that but basically at least 20, 000 or so.

That's, that's,

David: that's astounding. Yes. Especially for a small team as you have maintaining it. And you've mentioned CAS several times, which I understand because you've explained that and you talked about formulas, but I've heard references to bottles, what are bottles and how do they relate?

Mike: So a bottle is basically like the original, as I mentioned way home we worked was it's was a build from source package manager.

So the formula, I guess it fits more obviously in the metaphor and the original stage in a formula. If you imagine a formula for a beer, it might say, you know, I want Whatever. I don't make beer like some sort of liquid and some sort of not liquid and mix them together and you get a, some sort of beer.

But so essentially a formula is a series of build instructions for like how you take an upstream source. So like say something like, w get the tar ball containing the source code for Wget, how you would go and run various instructions on your machine and produce a binary at the end. So what we do nowadays in Homebrew is that formula for most of the people most of the time is only essentially used for building a binary package on our servers.

So when that formula is modified on GitHub, then we will run through those instructions. We will build a new bottle, which is what we call the turbo that contains the binary package. And then we then upload that to, we use like GitHub packages to store our all our binary software basically. And then when a user types brew install on their machine, instead of doing through all those steps or essentially just downloads that binary package, that bottle, and it will pour the bottle extracting the tarball into the place on disk.

And then a user can have that up and running. And for, you know, for, for smaller things, the time difference is minimal for larger items. If you have a relatively normal slash fast internet connection, you can do that. you're talking about the difference between, you know, less than a minute versus, you know, even on high end hardware, like four or five, six hours to compile some of the really big meaty stuff.

So it can be a huge time saver for certain people and users. And it's also a lot less hour prone.

Jonathan: Somebody had a lot of fun coming up with that extended metaphor and figuring out all the ways that it fit. Well, I

Mike: came up with that. So if you want to blame anyone for that particular strange metaphor, that's me.

And then most of the original stuff was Max.

David: You can definitely tell that this was developed on a napkin in a pub.

Speaker 5: I think we're looking

David: around the room creating parts of the infrastructure.

Mike: Yeah, well, the first commit ever to homebrew is interesting. It's something I actually, I did a bit of at GitHub when I worked there as well, and I recommend it for people when they're doing projects is this idea of like readme driven development.

I don't know if you've heard of that before, but the idea being before you start a project, essentially start by writing the documentation of what you think. it should look like and how it should work. So thinking from the outside in. So that's how Homebrew originally worked. And if you look at the first commit to Homebrew, the first commit is actually a readme of how Homebrew works, despite the fact that there's no code at this point.

And yeah, and the last question in the readme is, was Homebrew conceived while under the influence of alcohol? And the answer is yes.

David: Both Unix and Linux were conceived under the influence of various substances.

It's true.

Jonathan: All right, I'm going to jump in and I want to ask John a couple of questions about Workbrew in particular, and Let's see. So I'm, I'm assuming that the way we could set this up is a business would, would say, look, these are the packages that we want to allow people to install on their machines that we're, we're in charge of, and then we have this whole other block of packages that we, we don't want to allow.

Like, that's the sort of tooling that, that you guys give with WorkBrew, right?

John: To some extent, I think like I would actually preface it by saying that for us, the most important thing is a developer experience. Brew is kind of a tool, a tool belt for software engineers. They go to brew and they get all the things they need to do their jobs.

The reality of the situation is that companies, especially in highly regulated industries, you know, we talked to a lot of banks, a lot of you know, FinTech type companies, governments. Insurance companies, healthcare. The rules of the road there are just, you have to do certain things a certain way. And so rather than think about it as a way to limit folks and what they can do, it's more a way to enable developers to continue to use the tools that they know and love at

Speaker 5: work.

John: So we think about it as without Workbrew, the people in those companies would be told, no, sorry, you can't use brew. It doesn't meet our security and compliance requirements. And so with Workbrew, what we do is we make it so that. The IT folks, the security folks are able to feel comfortable with the risk profile of giving end users, developers, the ability to go out and install 20, 000 different open source packages.

And in some cases, you know, especially the financials that I talked to, they have some requirements around things like data protection, data loss protection. So they don't want to allow anything that opens up tunnels in the network. So particular VPN packages, wire guard. You know, Ngrok, stuff like that, where You know, they're honestly like useful tools that are not necessarily nefarious, but because of the risk profile of those businesses, they just can't allow it.

And so that's really where the kind of security and compliance type of, you know, restrictions come in and work brew. And with every customer that we work with, we talked to them about, you know, the developer experience and how they can maintain a positive developer experience. And instead of, you know, You know, using hard and fast rules, setting policies that they can monitor and see if, if something is out of policy.

So every every user that we have, generally the way that they come on board is they do a complete audit of every package that's installed on every machine. Yeah, homebrew. This is a huge amount of information for them. They had no visibility previously as to, you know, let's say they have 500 engineers, 1000 engineers, what packages were being used, what their, you know, potential surface area for attacks was, whether it's supply chain attacks or, you know, other kinds of data loss or things like that.

So they can do an audit and they can see everything that's happening. And then once they have that information, you know, They can take steps to provide alternatives rather than just say, Hey, this is turned off, you have no access. We give them a way to say, Hey, the preferred tooling for this job is X.

And we really are excited about the features around kind of collaborative developer environments for software engineers. So the idea is that rather than each person in kind of a single player mode managing their entire kind of development stack themselves, if one person finds a problem, whether it's a security issue, a productivity thing where they can move faster, When they make that change, they're able to share it with their entire team, you know, seamlessly.

So that's, that's some of the ways that we think about, you know, The same thing that you were saying, but really from a first principles, you know, mindset, why are we doing this?

Jonathan: Sure, sure. Does either work brew or and or home brew provide like automatic updates? Or do you have to go in and say, all right, all the packages I've installed, go check for new updates and install them.

John: I mean, the short answer to that is kind of it depends what you want. This is another one of kind of the principles that we're trying to follow, which is one size doesn't fit all. We have, you know, these fast moving tech companies that really want to embrace the cutting edge and they always want to have the latest version of everything all the time.

And on the other hand, we have, again, these highly regulated industries where they say, I know that a new release came out, but we cannot deploy that new release into production until a human has reviewed it. Yeah. They've created a signature, we've entered an entry in our audit log, and we've allowed it to enter the production environment, right?

And so these are two polar opposites. So, our principle here is like, one size does not fit all. We want to give people the ability to kind of choose how this should work. And Mike can talk about, you know, Homebrew's auto updating. Facilities.

Jonathan: Yeah. I'm curious about that too. Mike, I'll let you take it for a minute.

And, and what, what's the solution there with, with homebrew itself?

Mike: Yeah. So homebrew by default will basically. Almost every time you run a significant command, it will auto update itself and it will try to ensure that it always results in a consistent state, which involves, it will do things like auto update, auto upgrade, like reverse dependencies and all this type of stuff.

So I, I guess in general, like Homebrew is a developer tool, like my, thinking is if you ever are having to have in tools a thing where you say, Oh, if you run this command, make sure you run this command afterwards, then that's generally not very good UI, right? Right. So my, my goal is over time that Homebrew essentially, you can just get away with installing software and everything else that you might need to do running updates, doing cleanups, upgrading things, getting rid of things you don't need anymore, auto removing things you don't need anymore, like all of that stuff is done.

fairly seamlessly and automatically. And I guess another note related to a couple of things that came up before it, I guess we haven't mentioned explicitly in the school before, but I would be Sad if I were not to mention the fact that Homebrew also runs on Linux, which is, it may seem slightly strange as to why one would ever want to do such a thing.

But one of the cases we've seen on Linux is I guess what John mentioned earlier, where one of the things with Homebrew, because it's a rolling release package manager, which essentially means when Homebrew can get the latest update of a thing and it works and it doesn't break Homebrew stuff, then Homebrew will upgrade to a newer version.

We've seen people using You know, a more stable like base layer OS, Debian, Ubuntu, whatever, and then they might install. homebrew somewhere on their system on Linux and then they can get access to kind of maybe the bleeding edge for certain developer tools. Like if you have, I don't know, like a classic, like a CLI I love is RIP grep.

It's a RG is the command and it basically just does really, really, really fast recursive grepping through folders and stuff like that. It's what VS code uses for file finding text and files. So a tool like that, for me, it's like, I, If I want a Linux machine, I get frustrated if I have to wait, you know, like months for someone to get me the newest shiny version of RIPGrep.

And also the blast radius, if it, you know, If it doesn't work, then it doesn't really impact anything else in my system. Right. So then you, you can get this nice combination. And in a funny way, that's sort of how Homebrew works on macOS as well. In the, the, the reason why I'm drawn to macOS is because when I was using Linux I'm too much of a tinkerer and I would just continually break my system by being like, Oh, if I, if I increase, I'm showing my age a little bit here.

If I change X. org settings to do this, then maybe The refresh rate will look slightly nicer. Oh no, I'm stuck in a terminal for 24 hours, all this type of stuff. Whereas the, for me in the macOS world, like essentially apple of like nailed down that that trunk of my bonnet, as we'd say over here to my car.

So I can't get inside there. And that's for people like me, that's better for other people. That's worse, but you can sort of have a more Mac like. feeling with your package manager if you run homebrew Linux in that way.

Jonathan: Yeah. So I was going to, I was going to ask about, I have a little note here on my notes to ask about running it on Linux.

So this, there's a question that obviously follows up. Can we run workbrew on Linux? Does that make any sense?

John: Yes, but I would say not yet. This is something that we've talked about. I've had definite, definite interest from folks we've been talking to just have consistency across all platforms, the ability to see what's happening everywhere and be able to remotely manage them.

We don't have folks using Workbrew on Linux today, but I expect we will in the near future.

Mike: Okay. And now we have internal prototypes, basically, right? We have not yet released for what, but I mean, essentially homebrew works similarly enough on both cases and our internal prototypes are, it's the type of thing where it's the classic engineer thing of if I was earlier in my career, I would say, Oh yeah.

We could have a Linux version out in a couple of days because it all compiles, right? Like that's, that's the easy bit. It compiles, what could go wrong? Exactly. I, I've learned enough over the years that I'm like, well, you know, there's, there's probably more, the iceberg here is probably. Perhaps a little deeper than I might want it to be so

Jonathan: yeah in the past 24 hours.

I've pushed code that compiles I was that developer today within the past 24 hours Now work workbrew is available though like not not the Linux side of it yet But on on Mac if somebody really if like if this is the tool that meets their needs work workbrew is out there They can get it

John: Today, the way to get Workbrew is to come to our website, workbrew.

com get on a call with me and I'll show you around. We expect very soon probably in a matter of weeks, to have it publicly available for folks to just get on and try. But right now it involves kind of a conversation with us to see if it's a good fit for you. And really the reason for that is that we're We're looking for design partners.

We're looking for companies. We're looking for for teams that See the same set of problems that we see With using brew in a team environment and who really want to give feedback and help us build this tool together but we have it in production with a number of different companies. We use it ourselves every single day So it is there and it's up and running but it's not entirely self service as of this moment,

Jonathan: right?

What's the what's the uptake look like? Have you had quite a few companies that are a good fit that have come on? You

John: We had a really interesting situation maybe a couple of weeks ago. Mike was interviewed on the next web and there was an article that led to quite a lot of interest. Over the last few weeks, I've basically been in nonstop conversations with lots and lots of name brand companies that you've probably heard of, but I can't talk about.

But again, many of them are in these like kind of regulated spaces where it's, we're a finance company, we're a healthcare company, we're insurance, we're government you know, some kinds of, you know, different. Use cases for exactly the same thing. It's all different industries, but they're all saying, we'd love to use this, but our requirements are such that the open source thing just doesn't, won't fly here.

And really there's kind of like three stories that we see at companies. The first story is do nothing. They just let brew kind of be the wild west. Every developer who wants to use it just installs it on their machine and IT and security look the other way. The second kind of big category of folks is, you know, this informed trust model where You start at the company, there's a readme that says part of setting up your dev environment is installing brew, installing these 27 packages, here's a guide, probably the guide's out of date and it doesn't work when you first start, and then if something's wrong, there's that one expert who's done a lot of stuff with brew that you go and ask for help and that's really, really common, and then kind of the least common, but pretty popular case, especially among very large companies, is that they have some kind of internal tooling built around this already.

So they'll build custom scripts to integrate with their MDM tools so they can manage a fleet wide deployment of brew. They'll build scripts to do things like inventorying what, what packages are installed on their devices, reporting version numbers, and cross referencing that with vulnerabilities databases.

They'll do things like add self service scripts to their MDM tools so that End users who don't have admin privileges on their devices are able to install brew packages. But the downside to that is that for every single package in brew that your company uses, you have to have a staff member who's like maintaining that in your MDM tool, keeping it up to date, keeping it in sync.

It's just like unbelievable the amount of hurdles that people go through to make this work. And so yeah, the interest has been phenomenal. People are very interested and kind of those three categories of folks, they all hear from us and they say, wow. Why, why has this taken so long? Why hasn't this been here before?

Like, it's so obvious. So yeah.

Mike: Building as fast as I can, John.

Jonathan: And I assume part of the, part of the sell is, rather than you have to have this one guy at your business that understands brew, you've got the actual brew guys that understand brew. And you pay us enough money, you can call us and we'll fix things for you.

John: Yeah, that's definitely part of it. I mean, Mike is very generous with his time with our customers in terms of getting them up to speed on what needs to happen with brew.

So, yeah.

Jonathan: I mean, that's sort of a win win for everybody though. Right? Like I know how to do this very specific thing. You need somebody to do this very specific thing. You give me money and I do the thing like that's.

John: Well, the really beautiful part of this whole thing is it's not just one company that needs this.

Jonathan: Right.

John: They all, they all need the same thing and doing it once and making it, you know, reusable and, and, and packaged in a way that every single company that has this problem can adopt a standardized tool. We're basically saving an entire team's worth of effort at every one of these companies, right? And so, the individual kind of ROI calculation that each of these companies has to do is incredibly obvious that it's the right decision for them.

You know, rather than paying a team, you know, I could, I could list off maybe five or six of these companies that everybody knows and uses their products every single day, and they all have teams of people that are managing this problem. And in every single one of those cases, They would get a better solution from us.

It'd be more purpose built. It'd be more highly integrated with Homebrew. It'd be less maintenance costs for them. Less kind of distraction for their teams to have to manage it. It's just a very, very easy decision for them to make the buy versus build decision because, you know, the lack of expertise, the ongoing maintenance costs, what happens when a new version of Brew is released, what happens when a new version of macOS is released.

You have to, you know, You know, if something doesn't work, when one of those events happens, every single engineer in your company is stuck.

Speaker 5: You

John: know, that's a high cost.

Jonathan: Yeah.

John: So, you know, we make sure that never happens.

Jonathan: Just, just make sure that you, you, you don't pull a CrowdStrike and

John: Oh, absolutely.

Mike: I would be remiss if I said that, that, that hasn't been a thing that has been talked about recently when we've been talking about, deployment strategies and things like that of, yeah, there's there's ways to do this and ways not to do this and let's not do it the way that maybe doesn't always go so well.

Yes.

Jonathan: Yes. Have you gotten to the point to where particularly based out of these conversations with businesses, you've started adding things that you hadn't thought of to either work brew or home brew?

John: Oh, absolutely. That's kind of the biggest thing that we're doing right now is just listening to these customers about the problems they face and trying to build the most general solution.

I mean, Mike likes to say this thing about the baby. Do you want to give your kind of baby analogy about how this works?

Mike: So my, it's funny, at Homebrew, I'm probably the closest thing Homebrew's had to like a product manager, at least since Max left. Yeah, I guess. Working at GitHub for 10 years, I saw some product management done very, very well, and occasionally some product management done less than perfectly.

And something I feel like I learned from this stuff over the years, particularly when your audience is developers, is the line I like to use is users are, or customers, or developers, or whatever you want to pick. Or like babies in that they can cry and tell you that there is a problem, but they cannot tell you what the solution is to the problem.

The tricky thing is, developers compared to actual babies even if those babies may one day grow up to be developers developers tend to tell you, go jump straight from, I have problem to, and if you tasked me with how I would build this solution, how would I build the solution? And they come to you saying, what I need is, the solution that I would have been tasked with building.

And often they don't necessarily have the context that you do. They don't maybe think about like, what are the average users like? So a classic thing that this would come up in Homebrew quite often and the nice thing in Homebrew is you can just decide to do these things because it's not as higher stakes as certainly GitHub was and Workbrew is increasingly becoming is.

On homebrew, someone might say, Hey, I've, I've added a new option that I want to opt into for this particular behavior. And then I read it and I think about it for sometimes not very long. And I'm like, why would you ever want that option turned off? It's essentially like, I, I've added a flag. So when you run this homebrew command if I have my face puncher plugged into my USB port, it doesn't punch me in the face.

So I sat homebrew underscore, no punch me in the face, please. Right, and I'm like, well, maybe punching you in the face could be opt in. You know, like, let's flip the logic around. Or maybe let's just make homebrew punch free software. Like, maybe we can skip this flag altogether. So, to me, like, that's the type of stuff that comes up.

And again, I don't blame any, you know, Person who makes a PR like that, because they're, they're trying to be a good dev and be like, okay, I need to, I don't, I'm not sure that anyone except me cares about this. I want to narrow the impact radius here, but I can look and be like, well, actually everyone cares about this.

Jonathan: Yeah, it's, I don't want to break anybody's workflow. Right. I don't want to. Stop overheating when you hold the space bar down for everybody. That might be important to somebody's workflow. And, and that I did the, the, the baby analogy. I like that, David, I'm sure you get this too. Like a customer will come to us and say, I need this.

And then they'll, they'll tell you this most off the wall thing. Like, yeah, I need a new server. That's got a GPU in it. You need a one now. And then you, okay. What's the problem that you're having? Well, we'd like to be able to use. We'd like to be able to do this. And it's like, Oh, we need to get you an account at, you know, chat GPT.

We don't need to build a server. It's got a GPU in it. I'm sure you get that too, David.

David: Reverse it. You kind of have to, you're given instead of. The baby crying, they come to you with a solution and then you have to reverse engineer the solution to figure out what the original cry was. So you can give them a better solution to

John: your original question, though, about like, what have we built?

That's come out of, you know, customer conversations. We've had a number of, you know, customers come to us and are kind of very You know, common customer profile is ahead of I. T. Somebody who's responsible for managing you know, their MDM at the company, like the jam for Kanji or one of those type of tools.

And they'll come to us and say, Hey, one of my jobs is software patching, and I need a way to know when something has a vulnerability. And how I can really quickly fix that vulnerability and know that it's been fixed across my entire fleet of thousands of devices. And so we built a vulnerabilities dashboard that effectively takes a look at every single package installed across your entire fleet, catalogs all the version numbers, knowing exactly which version is installed on each device, and cross references that with a bunch of different data sources where we have information about all known vulnerabilities in those packages.

And we present this as a nice simple report for this IT manager to say, Hey, package X has a vulnerability at this version and it impacts 205 devices at your company. Click this button to apply a patch that fixes that on every single device. And after about 15 minutes, you can see 95 percent compliance, all of those have been patched, and a few of the devices that were turned off, you know which devices they were, so you can send a Slack message or give somebody a call to make sure they boot up their device and upgrade that package so that the vulnerability is addressed.

And that came directly out of, you know, customer. Customer requests.

Jonathan: Yeah, so you guys kind of provide a software bill of materials for across the whole organization Yeah, yeah, absolutely super interesting Okay. Now with with brew itself. We've talked a lot about command line applications Do we do do we do GUI applications?

Can we install, you know more complicated or less complicated but good GUI based applications Can we do real crazy things? Can we install entire bright light? Can we install Firefox with brew? Is that a thing that works?

Mike: Yep, and not only is it a thing that works so that this is essentially the way I Interact with homebrew primarily nowadays.

So one of the things we've built on top of homebrew was this thing called brew bundle Which essentially the the bundle part of the name relates to if homebrew is written in Ruby and there's a tool called bundle or bundler I guess is the full name that consumes gem files which are a list of ruby gems essentially like third party ruby modules and then builds them all together so you kind of have all these in your app.

So homebrew there's a part of homebrew called brew bundle which does the same thing with brew files where you can basically have a bunch of software In there. And you can specify, okay, I want these formulas. I want these casks. I want these things from the app store, the Mac app store. I want these VS code extensions.

And, you know, probably more to come in future on basically like the way that allows you to use homebrew is instead of. Saying, okay, install this, install that, uninstall this, uninstall that. Instead, you can have a list of essentially, here's everything I want installed on my machine. And, anything that's missing, I want you to install it.

Anything that's out of date, I want you to upgrade it. Any, you know, background services that I specify, I want you to run them. And you can also tell it to do a cleanup, which means any software that is not on that list, I want you to uninstall it from my machine. Which is probably mostly useful to people like me, who accumulate huge amounts of cruft testing various homebrew packages.

But the nice thing with that brewfile is then, you can, Like my most popular not homebrew open source project is a thing called strap I built for primarily originally for github's internal use, but it can be used by anyone And basically what that lets you do is say, okay when I first set up my machine the first thing I want to do is pull down this list from a github repo and I So as long as you're kind of relatively diligent about like dumping software to that list and then committing that to that repo, then you can get essentially a nice single file description of here's everything on my machine.

And if you're one of those people who likes dot files, Like having all your configuration files, like there. That's the repo that I put that in. And again, that tool pulls down your files repo. And essentially my goal with that repo is I should get my machine 90 95 percent set up by just the contents of that repo being pulled down and all these scripts run and stuff like that.

And nowadays I even have all sorts of mad things that Extract secrets from one password and write them to the right locations on disk and and all this type of good stuff And but yeah, but as you can tell this workflow is very heavily github centric right now. So if you are interested in a non GitHub centric version, then again, stay tuned to what brew is up to in the future.

Jonathan: Yeah, interesting. Now does brew, does brew help with doing like program installs without using brew? So like on, on Mac, the the, the, the normal way to install software is like you, you get your package, you double click it and it gives you this nice window with here's the application and here's your applications and you just click and drag can, can we do something real fancy, like build a package in brew and then Spit out that zip that then someone can install without using brew.

Is it, is that in scope?

Mike: Exactly what casks are basically most casks. That's what they do. So if you, the, the Google Chrome cask, effectively what that does is downloads the Google Chrome. So in comparison to maybe a Linux package manager, a Google Chrome cask, that's not some special version of Google Chrome that's made by the homebrew team that just downloads the installer from the homebrew team.

Apples, sorry, apple, apple, not making a crew installs the doubt from Google's website. And effectively that kind of drag and drop or click through and install or next, next, next, accept, license, etc. Like all of those steps are essentially automated inside the cask. So instead I can type brew install google chrome and then at the end I get the exact same result as if I'd gone and walked through those steps manually of the google chrome installer.

And the nice thing about that is it essentially provides a higher level API on So there's about 10, 000 casks, as I mentioned before. That's essentially 10, 000 pieces of software where the API for how to install it is, well, do I drag this thing? Do I click the thing? Do I download it from here? Do I like have to run a terminal command?

No, it's you run the same terminal command for essentially any of those. piece of software and they are all installed the same way. And to me, that's the most powerful thing about both casks being in homebrew, but homebrew itself is essentially you have this high level API for this stuff. And when I was talking about my brew file before my brew file has a bunch of casks in it.

So I'm not just installing my developer command line tools that way. I'm installing slack that way. I'm installing zoom that way. I'm installing like all my like Safari extensions that way. So essentially pretty much all the software on my machine. It's being, in fact, probably literally all the software on my machine that is not provided by Apple is being installed through Humbroo in some way.

John: Yeah. And on the software that's provided by Apple via the Mac App Store. Brew bundle also has an integration with a tool called MAS, which is the Mac app store command line. And so you can actually add to your brew file just like you would add brew in the name of a formula or cask in the name of the cask.

You can add MAS and the identifier of an app in the Apple app store, and it will automate the process of opening up the Apple app store and requesting to install the brew bundle. the program from Apple onto your device as well. So literally everything can be put into this brew file and automated.

Mike: Yeah, that's great.

If there's show notes or people watching along, in some ways it's easier to kind of see rather than do it. So if you Google for Mike McQuaid which is My name, you may struggle with the spelling of that, but I'm sure you'll get there in the end. And then brew file B E R B R E W file. Then you will get the top hit to like my brew file in my public dot files repo.

And then you can see like what that looks like. And please don't critique my particular. Choices of software because don't don't that very near and dear to me. Do it out of me.

Jonathan: Exactly. Now can we, can we go the opposite direction? So let's say, and I've, I've, I have this question because I've had to work through this before.

It's been years ago, but for a while I was developing an applique, a cross platform GUI based application. And one of the things that was a challenge was building that drag and drop installer. So, you know, if you didn't want to tell, so you could tell people, okay, go install brew and then install my application.

But if you don't want to do that, you want to give people that zipped up installer where you just, you drag and drop it and that's it. And it seems to me that there could be a opportunity here to To have a script inside of brew where you run the build and it gives you the installer that then you can go out and Give to people and has anybody done that?

Is that something that that brew can do? I know that's kind of a weird That's it's a weird idea, but I it would have been useful to me back then

Mike: I've seen people do such things in the past. I think what makes it tricky is, well, so I guess the two sides of homebrew, right? So the, the casks, for example, that essentially already exists.

If you have a cask for, as I mentioned, Google Chrome, that's means that that's already been provided by the upstream software, but then with homebrews formulas for, you know, say you wanted to install some sort of open software. Open source software that way. The tricky thing is because Homebrew has its own little special snowflake ecosystem where everything works just such how it does.

Essentially, In Homebrew, everything wants to pull everything else from Homebrew and be updated by Homebrew and be in the location of Homebrew. So that doesn't make it impossible. I mean it, you know, technically it would be possible to do such things, but it makes it trickier. My best actual experience in the past, to give a shout out to another open source project, is this a cross platform build tool called CMake, you may well have heard of.

What's less known is it comes with a thing called CPAK, which is C P A C K. So that basically lets you do cross platform packaging like this. At previous jobs I've used it for essentially generating, you know, like when I used to work on Qt applications for generating a nice click, clicky clicky windows installer or a RPM or deb for you know, Debian or Red Hat based distributions or a Mac OS kind of drag and drop style or like traditional clicky clicky installer.

Like you can essentially spit all of those out from the same project. And that also provides some third party tooling, some of which I may have contributed to myself, that aids in doing things like pulling all the libraries from Homebrew and putting them in the right place and all this type of stuff.

So it's kind of out of scope of the Homebrew project itself, but like, you can. You know, if you, if you look a little bit funny at the problem and take tools like CPAC, then you can definitely rely on homebrew a little bit to solve this problem

Jonathan: a little bit easier. Yeah, sure. Okay. So is there, and it seems like there used to be is it Mac ports is also sort of in the same space that there are some other projects that sort of solve some of these same problems, right?

Yeah.

Mike: Yep. So Mac ports is one think is another. And I guess, yeah, nowadays people are using Nix in some cases as well.

Jonathan: Oh yeah. Is there any cross pollination? Like did some of the same, maybe same people doing the, the, the, the package management or yeah,

Mike: I don't know. So I wouldn't say there might be.

We definitely. We talk to each other. So, and sometimes not, well, I mean, when I say talk, I mean, I don't mean it in the silly way that, you know, normal humans would actually talk to each other like we're doing right now. We, we, we type things at each other on the internet and there is collaboration in some ways between the projects.

Some of it is explicit, like where we might go and we have kind of shared channels with some of these folks. where we might kind of figure out problems. And then some of it is the, in the nicest open source spirit of the world, us, like, various projects stealing patches off each other where, you know, maybe there's some new macOS version or compiler version or whatever it may be, And someone, one of the package managers writes a patch and the other package managers need the same patch so then they can take it from each other.

So we've all, again, there's a nice collaboration where we've all done that from each other. And also sometimes for inspiration where if a particular package is not working on homebrew as well as it should, or we get a complaint and someone says, Hey, this works fine on Mac ports or Nix or whatever, we might go and look and see how they do it.

And again, I'm sure. The same thing is happening in reverse and same with the with the There's probably just as much if not more actually with the Linux package managers as well and the BSDs as well because the BSDs use Clang as their compiler, which is the same as what we use So yeah, so there's basically, it's nice.

It's kind of classic open source Collaboration and action really in that like all these projects because we're all open source We can all share information and resources and help each other out with stuff

Jonathan: Yeah, I have anybody interested in workbrew asked about any of those other other tools. You know, I could, I could imagine someone say workbrew sounds great, but we want to be able to use Nix as the back end.

John: Haven't, haven't heard that one yet. But we have had some folks that are, you know, coming with like cross platform requests. So they say, for example, we want to have consistency across all three major platforms, Windows, Mac, Windows, Mac, and Linux. And it's been particularly interesting with regards to non developers.

So there are definitely companies where they have kind of a wide range of employees and some of those employees are using brew today and they understand it and they rely on it, but they also don't want to have a different system for, for example, our customer service team or for their data engineers or for data science or, you know, folks who depend on code and packages, but may not be as comfortable using the terminal and installing things.

And so we've talked to them about. How can we provide a singular way to provide a developer environment across all three of those platforms? So that's been probably the most similar kind of questioning that we've had from people. But not directly to say, Hey, I have like five different kinds of package managers.

How do I use them all together? More it's like, we use this one thing. It's really cool. How can we make it work for everyone?

David: So I assume on Windows, since Brew works on Linux, you could run Brew under WSL today. But it's the Windows specific stuff that probably needs some

John: Yeah, you can run, you can run Brew under WSL today.

That's how most of the people that I talk to who have interest in it are doing it. So I've talked to potential customers who say you know, we have this 10, 20 people over in this department who are using WSL on windows because X reason, can we get those mapped into work brew as well? I haven't had the same question about, Hey, can you run this natively on windows?

That almost never has come up. But there's a, there are other Rather than saying like, hey, we have this MDM tool, because really the people that we talk to a lot are the ones who are, you know, the IT managers who are responsible for getting the fleet out into the hands of the, of the company's employees and making sure they have the tools they need to do their job.

And so what they're saying is, hey, my MDM tool, whether it's Microsoft Intune, Jamf, Kanji, whatever, has a couple dozen applications in its built in package manager. In Kanji they call them auto apps, in Jamf I think they call them catalogs, but the idea is that, you know, Google Chrome, Zoom, Slack, all of those come as like default packages that the MDM provider keeps up to date.

But they say, well, we have this, you know, this couple dozen packages that are not managed by them, we have to manage ourselves. But Brew manages them, we can use Brew instead. And so they want to standardize on a, on one, one tool for all of this. That

David: makes a lot of sense. So another question that I had was just about the interface.

So you talked to, you mentioned the cross system management and like you talked about how you could, See your systems updating with work brew and identify the ones is that a web portal? Is that something built into the work brew command line?

John: Yeah, the overall architecture of what we ship is Three pieces on top of brew.

So it's brew the open source project Plus we ship an installer PKG file Which is a Mac PKG that basically enables zero touch deployment of brew it makes it so that if you have a brand new Mac You business manager, the first time that you turn it on brew is installed and everything is secured. If you have existing devices with brew installed and maybe dozens or hundreds of packages installed on those devices, you can run the PKG file locally or you can run it via your MDM tool and brew on that device will effectively be upgraded to work brew so that it will have that kind of connection back to the system on the kind of Administrative interface, there's what we call the console, which is a web portal that gives you a high level overview of every device in your system or in your fleet, and it gives you a high level overview of every package.

So you could say, for example, which devices are open is open SSL installed on. What versions are installed? Are there any known CVEs against it? And then run a patch to say, upgrade it to the latest version because there was a known vulnerability we want patched. So that's kind of how the interaction works.

On the device, there's one more thing that like, kind of goes to what I was mentioning earlier. One size does not fit all. So, on the device, we give companies the opportunity to choose different permission models. We call them Restricted, Managed, and Guided. So Restricted is the most controlled. This is kind of useful for individuals who don't know what the command line is, may not have any interest in brew, may not even know what brew is, and you want to manage all their installed software in the same way as every other device.

So you can install WorkPro on their device and essentially provide it in a restricted manner where the end user has no access to the Bruce CLI. It's only managed remotely. So again, great for like a customer service team or maybe data analysts, people who might not be comfortable with the CLI. Then there's managed mode, which is kind of the, Big most popular thing that we're seeing among companies with developers where we install brew on their device for them via the PKG via their MDM tool and give the end user access to brew via the brew secure CLI, the wrapper that we have around brew for the end user.

It behaves. It's just like normal brew so you can do brew install, brew uninstall, upgrade, tap, you know, pin, whatever they want to do, any normal command, and those commands are parsed and kind of made subject to policies or configuration options in brew so that, you know, we keep things secure and compliant.

In those cases, what's really interesting is the end user on the device doesn't necessarily have to have admin privileges. So one of the big kind of points that we hear from, especially the regulated companies, is we can't let people use brew because in order to use brew successfully, they need to be admins on their devices.

And for legal reasons, we're not allowed them, allowed to have them have admin rights on their devices. We just cannot do it. So we basically enable them to give their users the same brew experience that they know and love. without having to offer admin privileges on the device. And then the third kind of mode is more progressive companies that want to give their engineers full access.

It's essentially the same kind of guardrails around policies and how you can use Brew, but if they have admin privileges on the device, they're able to effectively escalate the privileges and go around these guardrails and say, even though it's out of policy, I'm going to do it anyway, so I'm not blocked.

And then the kind of management or the IT and security teams will get an alert to show them that. This device is out of policy or this package is installed in my fleet out of policy and then get up to date that way.

Jonathan: Yeah, so it sounds like you're not planning to bring brew itself to windows.

PowerShell. I was going to ask about that.

John: I think Mike is best suited to answer that. Never say never.

David: It just sounds like pain and suffering though. A follow up question on the whole dependency management. So, I have DevOps experience where you're, you know, developing webpages or web applications.

And so, you need to keep your dev system in sync with the same versions that you're running on your production systems. Do you have anybody using Brew to manage the software stack on production systems? Ah.

Mike: Right now, not really. It's essentially Homebrew, as you mentioned there, like the, the limitation you, you generally have in production is you want to have everything locked down to very, very specific versions that are consistent across your entire fleet.

But That's not the model in which Chromebrew operates by default. That is increasingly the model we're seeing people wanting Workbrew to operate in default. So we are building stuff in that direction. Again, sorry, I can't say too much there, but essentially like that, well, I guess it goes back to the, the babies we were talking about earlier, right?

Like essentially this is, this is a crying baby problem that we are aware exists even in development modes, but certainly the desire to have everything consistent between CI, dev, production that, that is a problem that exists today. There, there is not a great solution to, and there is a problem that we are working on right now.

I guess the Homebrew middle ground there is because Homebrew is a rolling release package manager for cert, it used to be Homebrew was just, you know, You get the latest version or you, you don't get a version, right? Like you have to just pick between. Now, more popular packages with kind of better support for kind of running older versions, say something like MySQL or Postgres or Node.

js, Ruby, whatever it may be, that still provide bug fixes, security releases, etc. for versions beyond the most recent one. You can install a package and you might say, you know, brew install nodejs at 18 or whatever, right, which gets you a like less than the current newest version but will continue to get you patch releases and security updates and stuff like that.

I guess a sort of balance I've seen with this stuff as well is that historically, The enterprise, the maybe individual developer model is let's just have the newest version of everything all the time and the enterprise model has been, let's pick a set of versions that work and then essentially only ever upgrade them if we absolutely have to.

But the problem with that flow is you often end up in a situation where, whoops, we probably should have upgraded this version a while ago and now a bunch of bad actors have got access to our product. either development machines or servers or whatever it may be because we decided to sit on this version indefinitely even when there were security updates that we were too scared to install.

So to me there's a, there's a happy middle ground to be found. Homebrew leans slightly more towards the like, you know, let's have the newest thing of everything all the time. But as I say in WorkBrewland we're building, essentially we already have some tooling there to essentially get that middle ground of like, okay well this package is actually vulnerable right now so I don't care if you're trying to sit on an older version because.

Some perceived view of stability like let's upgrade everyone. So at least they're not in a vulnerable version anymore.

Jonathan: Excellent. All right. Well, we, we have hit the the top of the hour, so I want to get into rap. And one of the, one of the things I want to ask you guys is, is there anything we missed? Is there anything we should have asked you about that you wanted to let folks know about that we didn't cover?

It's kind of challenging question because you've got to do some set math and think about all the things that we talked about, but if there's anything that comes to mind,

John: I mean, I guess there's one thing

Mike: that I might mention, which is, yeah, I think I mentioned like Linux before I'm developing environments and stuff like something where.

We've actually seen quite a lot of use of homebrew that might be worth playing around. For folks who are listening here is homebrew runs pretty well in, if you're using GitHub actions or, you know, GitLab runners or whatever it may be. That's a nice way because the packages are The same on Linux and Mac, and because the versions are in sync, that can sometimes be a nice way to have your development environment and your CI environment in sync, where you might have your developers using Macs locally, say, but then they might have a bunch of tests running on CI boxes, which are most of the time, for cheapness reasons, going to be running on Linux.

So if you do, you know, use a brew file, like I mentioned before, or even just run brew install. go or whatever it might be, then that means you can have that consistency between what you're doing locally on your Mac and what you're running in your CI environment. Yeah. And that can be kind of quite nice.

Yeah, absolutely.

John: I was just gonna say that you know, if this, any of this sounds interesting to you, Mike and I are both available you know, to talk to people. You can find us on our website, workbrew. com. There's a contact form that goes directly to my inbox. I'd be happy to chat with anybody there.

We're also active on, you know, GitHub and Twitter and things like that. We'll give our contact information for that as well.

Jonathan: Yeah, and we'll make sure and get that in the show notes for folks if they want if they want to reach out probably a question mostly for mike Although john, if you have any stories, you certainly can tell us what's the what's the most surprising thing?

Somebody's done with brew. What what does somebody? Message you or written about and you know, I didn't know you could do that with brew Or why would somebody want to do that with brew? Well, I, I'm

Mike: actually going to choose to misinterpret your question because I think you'll find it's slightly more funny.

The version that, so the thing that jumped into my head was almost like, what's the, what's the. What's the stupidest thing you've seen happen with, with homebrew? Stupid can be surprising. Yeah. So my favorite bug of all time actually is we might be able to, if you've got show notes or whatever, like fire me an email or something, you know, I can send you the link because this is open source, you know, you can actually read.

So there's a bit of debugging that went on. Someone was getting some very strange messages when they were running homebrew. And this is in the earlier days of Mac Os. So nowadays Mac ships with a thing called system integrity protection, which essentially means like, look, I don't care if you run pseudo, there's certain stuff on your disc that's more important and we're not gonna let you just like screw around with it.

So stay away. But before they had that, someone was trying to run home Homebrew, getting very strange error messages. And what was figured out was that they had managed to replace bin bash, which is, you know, on a Unix system, a relatively important thing for you to have somehow with no JS. So every time they were attempting to run a batch script, they were getting JavaScript errors in their shell.

So that to this day is my favorite issue I've ever seen on

Jonathan: Homebrew. Oh, that's beautiful. That is beautiful chaos. I love it. All right. Yeah, John.

John: I was just gonna say, just one of the things that I was like surprised about is I can't mention names, but several very large enterprises that make that make a lot of the core technology that we depend on have their own internal forks of brew.

Oh, yeah. And so, you know, they, they're basically maintaining an entire Parallel infrastructure where they're like reviewing everything themselves you know, to get at these like kind of supply chain story, like keeping things up to date vulnerabilities. It's just like unbelievable that it's, you know, brew is that important to them that they can't, they can't decide, Oh no, we just won't use this.

No, we actually, the only option we have is to maintain this ourselves internally as a totally separate fork. So that just kind of goes to the, you know, the idea that it's a very essential tool for a lot of developers.

Jonathan: Yeah, yeah, that's great. All right, so I'm going to ask each of you a final two questions, and that is your favorite text editor and scripting language, and we'll start with John.

John: I mean, this is pretty easy for me. I'm a Ruby developer so Ruby is my favorite scripting language. I, you know, I used to work at GitHub for close to seven years. And I was so happy when I joined GitHub because it was finally a company that used Ruby as like their main language. I had always been like a web dev, you know, my early days I was doing like PHP and like Java and stuff like that.

And when I moved into writing Ruby, you know, at work, it was like the best thing ever. And then I kind of have, you know, A soft spot for Pico. Okay. As my editor, I first, when I started writing code that was the first editor that I learned how to use on a Unix machine. I was running like a Gen 2 box.

Yeah. And old habits die hard. It's like still the thing that I just opened by default in my terminal.

Jonathan: Pico and not Nano? Yeah.

John: Pico, not Nano.

Jonathan: That's great. I

John: have an alias usually.

Jonathan: Makes

Mike: sense.

Jonathan: All right.

Mike: Mike. Yeah, so scripting language, I'm actually gonna disagree with John. So I use Ruby for like proper programming nowadays.

But if I just need to quickly solve a problem, I always go to Bash. Like me and, me and Bash, Bash has treated me so badly so many times. But yeah, I keep coming back. Like I don't know what it is. And yeah, as for text editor, like nowadays I feel like I'm, I wouldn't say begrudgingly, but you know, it feels like a lot of developers are on VS code now, and it makes just life easier for me to just follow the flow and go there.

But yeah, I'm still, I'm still keeping my eye on other things like the extension ecosystem of VS code is great, but like, you know, it's not the fastest thing in the world. And there's a text editor by an ex GitHubber. Called Zedd that is kind of up and coming, written in Rust. It's super duper duper fast.

And I've been playing around with that ever so often. It doesn't have quite all the features I feel like I need, but like, yeah, I, I have my, my hope for that as a potential feature option.

Jonathan: Yeah, absolutely. We, we interviewed a young man back a few weeks ago, building the amber language that is intended to be bash code with some of those bash pain points removed.

That one might be interesting to look at. We had a lot of, we had a lot of fun talking. I kind of like

Mike: the pain points at this point of everything bash does wrong is just, you know, it's like, it's almost like scar tissue that. It feels like it's part of me, you know?

Jonathan: Yeah. I'm, I'm trying to figure out, it seems like we either interviewed or we're going to, ah, I'm in contact with the guys from Zed about hopefully getting getting them on the show.

So watch for that. too. Guys, thank you so much for being here. I appreciate it very much. And it was, it was really fun to get to learn about, about homebrew, which, you know, it's been around for forever. And then work brew, which is a really, really fascinating project. The business that's been built and boy, hopefully hopefully it'll continue to go well.

And maybe here in about a year, we can have you guys back on and talk about what's happened since.

John: Oh, we love it. Thank you so much for inviting us.

Jonathan: Yep. Good deal. All right.

David: What

Jonathan: do you

David: think, David? Ah, I love it. I, of course, as we started at the beginning, I'm not a big Mac guy, but I'm interested in playing around brew on Linux.

Jonathan: Yeah. I tell you what really fascinates me the most is for my use is a brew in GitHub actions like that. I, I, I never had that thought. That is very surprising to me, but you know, as they describe what you can do with it, it makes sense that you know, you're, you're. Your GitHub runner may be running something very different than what your develop machine is.

And having this external package manager that can put the exact same packages on all of them, like that's, that's kind of compelling. So that's definitely something that I will have to keep in mind for the future that could be interesting to do. I like that. I, I'm also just thinking about the conversation we had.

I am, I'm impressed with the way that they are building Workbrew on top of Homebrew, and I, I don't know if you would call that, I don't think you would call that an OpenCore project. I don't think that's fair to call it that. But just that they have this very clear line of demarcation between the two.

And they're pushing features into homebrew as needed to make workbrew work. I think it's a good, I think it's a good model. And I think it'll, I think it'll serve them well. So, you know, looking forward to. Hearing about their success as time goes by.

David: Absolutely.

Jonathan: Yeah. All right. David, is there anything that you want to plug

David: before we let folks go?

Not specifically, but I always like to take the opportunity to plug club twit and the twit network. They're going through changes that I think are positive in the long run. Yeah. And we have fun over there. So come on over.

Jonathan: Yes, we've got, we've got our show that David is from time to time, but co host on.

And that's the untitled Linux show where we talk about all kinds of fun Linux news and open source news. And there's some, as we say, there's some cross pollination between floss and and ULS. Coming up on floss weekly next week, we actually have Pedrag Brady from CoreUtils, and we talked about the Rust CoreUtils a few weeks back, and I got an, I got an email that said, Hey, you know, we're the OG CoreUtils, we'd love to talk to you too.

So we've got them coming on, and looking forward to that next week. We'll be back at our regular time next week on Tuesdays, we recorded a little early today. And then, yeah, we, Appreciate Hackaday being the sponsor of the show, giving us a place to land. And you can follow my work there. The security column goes live every Friday morning, and that's always a lot of fun too, to keep up with what's going on in the world of security.

But other than that, just want to say, thanks. We appreciate everybody being here, catching us live and on the download. And we will see you next week on Floss Weekly.

Jonathan: Hey, this week Doc Searles joins me and we talk with Olaf and Dave about LifeRay, a project that's a little difficult to pin down. It started with the idea of portals, but has grown into a basic building block for building any sort of experience on the web. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 795, recorded August 6th.

Life Ray, now we're thinking with portals.

Hey folks, it is time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host Jonathan Bennetts and we've got a fun java filled show today. Once again, it is not just me though. We've got Doc Searles, the the one and only the og. Hey doc. More O than G. A lot more O than G.

Yeah, well, I feel that way some days. Now, you've described your day today as a death march. You've got something going on that you're feverishly getting ready for.

Doc: Yeah, there's an annual event put on by the Internet Archive called D Web Camp, for Distributed Web Camp. And it's at Camp Navarro, it's up in Northern California, among the redwoods.

Where it's fun to see people coming from other countries where all they're doing is standing around looking straight up wondering why These how weird these trees are they're looking at a cloud up there They're looking for the cloud or they think the Ewoks are coming. There you go Because that, that, that particular episode of Star Wars, the Star Wars movies was filmed in the Redwoods.

Oh, okay. But this is, this is much less lush. This is more like dirt on the ground rather than, you know logs and, and, and moss. But, yeah. But it's fun. It's a, it's a fun camp. And, but I, I'm giving a talk. early Thursday morning there. And and I've got a whole thing I'm doing on it and it's, I'm not ready.

So there's that. It's also weird. Cause I mean, it's, it's been, I'm in the middle of Indiana and not much different where you are, where it's hot. It's the summer here. It's hot. And, and it doesn't get cold at night. It stays warm because there's humidity. And In that part of California, it might be 90 in the daytime or, you know, 30 something for those of you elsewhere.

I now give my age as 22. 5 Celsius, actually, so. I like that. Yeah. So that's about how old I am. Anyway. Yeah. But it's cold at night there, so I'm bringing my flannel, my flannel PJs. Yeah.