Episode 817 transcript

FLOSS-817

Jonathan: Hey folks, this week Dan joins me and we talk with Stefano Zaccaroli about Debian, but also Software Heritage. That's the source code archiving project that you probably haven't heard of, but really should know about. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 817, recorded Tuesday, January the 21st.

Incompatible with reality. It's time for Floss Weekly. That's a show about free, libre, and open source software. I'm your host, Jonathan Bennett, and we've got something very fun today. First off, we've got Dan the man, Method Dan, the original Linux outlaw. I like that intro, Dan. It's, it's fun to introduce you that way.

Dan: Thank you very much. I like that. It's fun to be introduced that way. I have to say. Yeah, it's great to be back. Good to see you, Jonathan. Hello everyone.

Jonathan: Yeah, you know one of the things I enjoy we talked briefly in the in the pre show But one of the things I enjoy about doing this show is like the group of people that we get to hang out with You know, we've got we've got dan.

We've got simon aaron I've brought some guys on in the form of you know, rob and jeff and those guys But we also will have randall and even we're having the The plan is for next week doc to come back as co host, right? And so just the fact that we get to hang out with all of, all of these guys as part of the hosting team, I enjoy.

And then, you know, there's the fact that we have all of these guests that are just, you know, every one of them are great, have, have super interesting things to talk about. We've got another one of those today. And I understand that you've got you've got some connection with Stefano Zach Zaccaroli.

What's, what's your history with, with Zach, with Stefano? I interviewed I interviewed Stefano a few times for Ledex Outlaws back in those days. Once we were just discussing earlier before we got started once when we were both pressed against the window of a bus, quite literally, physically in Brussels on, at Fosdem because Fosdem is very busy.

Dan: Busy and those buses are not built to take that many people. So it was quite funny. We were just next to each other. And then I re I didn't actually realize who he was at first. And then we got chatting and it was like, Oh, can I interview you? So we, we did it right there. And then

Jonathan: Foster must be great for that because like almost every, basically everybody that is there is potential interview material.

Dan: Yeah, yeah, you need to take a lot of spare SD cards or whatever it is You use to record on when you go to somewhere like Fosdome.

Jonathan: Yeah All right. Well, let's go ahead and bring him on So Stefano, I know that he has been involved in Debian over the years Probably still involved to some extent and the other real no Not the only other.

Been involved in OSI but the thing that really interests me is software archive, because I, I am a, I am a fan of the idea of archiving software, and I'm also aware of some of the challenges, and so I want to, I want to pick Stefano's brain about all of that but first, just Stefano's Welcome. We are absolutely delighted to have you here.

Stefano: Hello. Welcome. Well, thanks a lot. Thanks Jonathan. Thanks Dan. And thanks also for the memory, Dan. Well, that was a long time ago. It was a really fun interview to actually do.

Jonathan: So what, give us your background. Like, so if somebody asks you, what, what, what do you do? What open source stuff do you do? What's, what's your answer?

Stefano: Yeah, so I've been around for a while, so I'm myself in let's say in my real life, I'm a computer science professor at the Polytechnic Institute of Paris, but I've been a geek for my, a computer geek for my entire life. So I started getting into free software when I was a computer science undergraduate at the university.

And we were, we had the lab at the university that was managed by the faculty itself, and it was some proprietary Unix at the time. And we have a second lab, which was managed by volunteer students. And of course it was Linux. And so that. Things intrigued me from the very beginning. And I had one of my first objective was becoming one of those, you know, volunteers is that mean that was helping out students and that's, that's how I started.

Yeah. It was that yet, of course. And then the rest of history, as you say,

Jonathan: there's an interesting crossover. And I think there always has been between that idea of political science and open source. And, you know, we, On the show, one of our rules is that we don't dive into partisan politics because nobody, nobody's happy about that.

But with that set aside, there is an interesting crossover and it is a thing that it is important to have people that sort of are aware of both of those realms, particularly when you start talking about legislation that's going to deal with cybersecurity and copyright law and things like that. Is that something that you, you focused on where those two overlap?

Stefano: Oh, no, I was actually doing computer science. Oh, computer science! But, since then, especially in my days at Debian, I've been involved mostly in policy work, and so, yeah, what you say, I completely agree. So software and technology is really political, and free software, if possible, is even more political.

So, yeah.

Jonathan: Well, yes, but actually no, but actually yes. Yeah.

Stefano: It's complicated.

Jonathan: Yes. Yes. So how, how did the how did the Debian piece come along? I've, I've always thought of Debian as difficult to get started with. Like you, from what I understand, to even be able to publish packages on Debian, you've got to go into like physically beat one of the other packagers to get your keys signed.

And it's just, you know, there's some, there, there's some challenges.

Stefano: Yeah, that was part of the thrill and of the challenge at the time. So the technically, the reason why I started looking into that was that, so I was passionate about for now, nowadays, somewhat obscure programming language, which is OCaml, Objective, Objective Camel, which is a functional programming language.

And I was really passionate about functional programming. And we were using Debian, and we were using GoCamo. And as you start coding, you, you discover that, well, some library is missing, in the sense that you cannot just, you know, apt get install that library and say, Hmm, let's look at this. How could I make this package, this library, which exists in this free software available to myself, my, you know, students, colleagues, and everyone else in the world?

And that's why, that's why I started looking into this Debian thing. thing, which was indeed quite intimidating at the time. But then, you know, I find friends that were already collaborating. A friend of mine, a student as well, was already a Debian developer. So it basically got me started into the process.

That must have been 98 something. So yeah, at some point you need to meet people and then the chance to meet a lot of Debian developers that were, you know. Traveling around and, you know, getting to my city at a time that was Bologna and finally get my keys signed.

Jonathan: It was pretty

Stefano: amazing.

Jonathan: Now you, you did, you were not just content as being a Debian developer.

You actually became a, you became the head of Debian for a bit, right? The Debian project leader for a while?

Stefano: Yeah, I, I, I've been dev project leaders for three terms in between 2010 and 2013, I think. Mm-hmm . And that was, so at the time I was a postdoc researcher. I already moved to, to Paris at the time, and I was really, really involved in the project and I, I was realizing that there is a technical part, but the, I think the most interesting innovation, innovation of Dion is actually the development model.

It was the first distribution of being actually a community driven distribution. And it's also a community that essentially self determined itself. So it's a project with a constitution that developers themselves created that was before my time, but it was not really common at the time to think of self determination of online communities.

Later, it was a notion that started to become more discussed at least, but at the time it was really, really an innovation. And so I was really passionate about that. Thing. And I say, okay, what can I do to help? And beca I did a bunch of things before , you know nominating myself for for the DBL election.

But I was really passionate about improving the processes and, you know, try to see how we can make the community give. The best of themselves.

Jonathan: And you're still involved with Debbie and I assume

Stefano: so. I'm still formally a, a developer, technically it's called a non uploader developer because I, I'm not doing technical work anymore in Debbie and this day, so I, I announced the, you know, the, the, the right, the upload right associated to my GPG key to avoid, you know, reducing the tax surface.

But i'm still a voting member of the project and i'm still following The yearly election the policy decision the political discussion on the project and some legal work work That's the kind of stuff i'm still involved with

Jonathan: well while we're on the topic of debian If you if you have your sort of finger on the pulse of the project still something i'm curious about is What's what does the health of debian look like these days and i'm thinking, you know, primarily in terms of Are there still developers coming on board?

Are there enough developers that are there to handle the workload? What direction is that trending? And like, is there a, is there a funding model there? Does there need to be? Are people getting paid to work on Debian? I honestly don't know how all that works inside Debian land.

Stefano: Okay. So many questions there?

Yes. So Debian as up. So starting from the, the funding project, Uhhuh Debian as a project is actually a nonprofit project in the sense that the Deion project doesn't pay developers or maintainers or other technical roles or even non-technical roles to, to, to actually do stuff. But there are a number of companies that are.

present in the broader let's say Debian ecosystem. A big example is of course Canonical that the, that the company behind Ubuntu, which is a commercial distribution based on Debian. And so all these sorts of realities, not only Canonical pays people for making something happen either downstream distribution or in Debian itself.

So that is part of the, of the Debian, let's say sustainability model. But what's beautiful about the Debian model is that a lot of people are just. You know, volunteers to the project there might be paid by research labs or or other institutions, but they might also be just working in their spare time.

And that's correct. And it poses some challenges because when you start mixing the work of volunteers with the work of people pay to do something, you might have some sort of tensions. But it's I think it's really working well for that. And regarding the turnover question, I think it's still pretty healthy.

So probably I don't have the exact number numbers in mind, but it might be a bit lower than it used to be in the in the early days. But it's still quite healthy. People come and go and and actually. You know, making packages is becoming more efficient. So with team maintenance and with the number of automation tools, the workload is actually cheaper in terms of the number of packages you can maintain and you can act on than what it was like 10 years or 15 years ago.

Dan: Yeah. Pretty cool. Wow. So did they still have to Debbie in kilts by any chance? Cause that's something that you used to be able to identify people at conferences. I certainly do.

Stefano: I still have my Debbie in kilts and I even wore it at my wedding. Not for the entire time of the ceremony. My wife wasn't really okay with that, but at some point at my wedding party, I did wear the Debian kilt.

And that was created for the DEBCOV in Edinburgh and by an actual Scottish Debian developer who registered a Tartan. So we actually have a Tartan registered in the name of Debian in. Whatever official Tartan registry there is out there.

Dan: It's probably a bit like software heritage. There's a Tartan heritage somewhere where they've got it all archived.

And, and they're very serious about that kind of stuff. Don't get

Stefano: me started about the trademarks of Tartans because I have no idea. I'm sure there is.

Dan: Yeah. Yeah. And so sticking to the Debian kind of subject, a lot of people listening to this I know we have a lot of very technical listeners, but we have some who may be less technical might be thinking I've never used Debian.

Why does that matter to me? But you mentioned it just there about the fact that the upstream thing. So Debian to me feels like the upstream of so many distributions that I suppose I'm interested in like the relationships that Debian has with say Ubuntu, Canonical is a big one that we all know, but also things like Linux Mint, and I'm not going to go into listing how many Debian derivatives there are so many.

So how does that relationship work? Do they give back to the Debian project?

Stefano: Yeah, so that's a very interesting question, because one of the things that I I. I sit down and thought about a lot when I, when I first become DPL, Debian Project Leader, was actually understanding the role of Debian in the ecosystem of distribution at the time.

And I realized that a lot of the impact that Debian had was not directly in the sense that people were directly or primarily using Debian. There were a lot of people that were benefiting from Debian. Indirectly, possibly not even knowing that Debian exists. And so I started measuring at the time, essentially the tree of derivative distribution.

And it was hundreds of downstream distributions, meaning that people have installed those distribution. They know the name of that distribution. They don't necessarily know it comes from Debian, but they're still benefiting from all the work that Debian people were doing and are still doing today. So, It's free software, right?

So there is no strict obligation to give back changes. The license says that you can just use the code for whatever you want, as long as you respect the license. But there is no obligation to contribute back. So you have some distribution that do contribute back and that benefits Debian a lot. And then from there, it also goes down again to other distribution.

And there are others that Contribute less, maybe because they don't have the means, maybe because they are not interested, and that's fine. So it starts to get, you know, tense and complicated when people in Debian expect contributions that are not coming. And so that was part of the time of the debates between Debian, Ubuntu, and Canonical.

Dan: Yeah, it seems to me that there's a lot of scope for discussion. I hesitate to quote argument, but discussion about things like system D, for example, when Debian brought system D in Jonathan's a Fedora guy. So he's looking at us a little both like everybody knows about that. Oh, I'm aware. I am

Jonathan: aware.

Dan: So there was arguments about system D or no system D. Is it difficult to wrangle? Is it difficult to get like a clear purpose? Is it it's a lot of debate going on all the time?

Stefano: So it's I mean, I think it's the the reverse of the coin of you know, having a large community of maintainers in the sense that in theory the Debian model and the Debian Constitution actually it's it's designed so that everyone can work on their own packages and essentially they can work independently from others but you can only work independently from others As long as there are no strict technical interactions between packages.

And when you start touching like low level packages that are pre installed and they deal with the operating system and they manage services like systemd, you can have very complicated technical decisions and you can have maintainers that are on opposite side of the decision you need to take. So this is the example was a big one and the way that Debian has to govern all that is that it has a body which is called the technical committee, which is some sort of judge it's not a single person, it's multiple people, but it's a committee that can actually make decision when they are not in the scope of a single package and they are like they have broad implication on the on the technical setup of the distribution and the system, the one at the time was, was a big one.

Dan: Yeah, that was a big fallout at the time. So while I'm kind of on this tack, I need to ask a bit, but as well about the relationship with FSF and all of that kind of stuff, because am I right in thinking that Debian isn't on the approved list of, of FSF distributions?

Stefano: That, that is correct. So still to this day.

So the, I guess we, the, the relationship within Debian and FSF went through many different periods. So in the very beginning, the FSF actually funded. The work that then led to the creation of Debian. So Ian Murdoch, the founder of Debian, for a time was funded by the FSF to essentially create a distribution on top of the free software that the FSF was providing.

Either developing themselves or funding to achieve the GNU system and the GNU operating system. So that was something at the time. Then I would say that culturally and even before my time, Debian went through a big, to put it simple, open source moment. So let's say we are here for the tech part. We are here for the collaboration.

We are not here for the political part of the software and for specifically for user freedoms. Then, and that was Around my time as a Debian project leader, the two projects got closed together again. So I, I insisted a lot on the discourse that Debian is primarily about free software and one of the main goal of Debian is creating a completely free operating system, but Debian also has some non free parts that needs to be explicitly enabled by user to be used.

And the FSF didn't like that part. So this was one of the reasons. And then there were another big reason. For the FSF for not listing Debian as a free distribution is firmware. So Debian historically took a decision that essentially the core firmware that you need to make some of your devices run was okay in some part of the archive and the FSF was not okay with that.

We got closer at the time, and at some point, FSF sent out a statement saying, well, Debian is not a community free distribution yet, but they made a lot of progress toward that goal, and that was a big achievement at the time.

Jonathan: I am personally not a fan of the line in the sand in the place that FSF has drawn it there.

Years ago, we had somebody from FSF on the show, and I tried to make the point It's actually rather ironic because all of these devices have firmware on them And it's actually a little bit more user friendly if you give the user the ability to replace that firmware as opposed to just baking It in forever and the FSF has the exact opposite take on that that no No, if the If the user has to touch the firmware, that's when it's a problem.

But if the firmware is baked in forever, that's, that respects your freedom. And I just think that's the weirdest take.

Stefano: Yeah, I think I personally agree, but I, so one point that I made to the FSF specifically, so I attended LibrePlanet, the main FSF conference as Debian the time. And one argument I use is that knowing that there is a problem, Part of software, which is completely free software, and then, you know, a red line separating that from a part of non free software is actually an opportunity to teach users about the importance of their user freedoms.

So saying, okay, there is software that is not enabled by default, and it's firmware that you need. To have your specific device run. You can enable it, but we take that moment in which you need to enable it as a teaching moment. We teach users that, well, you can enable it, but then here you don't know what the software is doing or what the firmware is doing.

You cannot change it or that kind of stuff. It can be something useful for the free software movement to, you know, to use rather than just pretending that non free software does not exist. Because essentially the results of some of the takes of the FSF is just pretending that. Yes,

Jonathan: sad but true. Have things changed very recently in Debian?

I, I, I don't remember the details, but I've, I've had at least one person tell me that there are some things that are now a little bit easier in Debian, like getting, I think it was getting NVIDIA. Drivers working that that recently changed in it.

Stefano: So what they changed recently is that essentially before there was a Essentially a big part of the archive which contains either known free software or software depending on non free software And essentially to get for instance just a non free firmware you had to enable all that software So that means that you enable that to get your firmware, but then suddenly you are exposed to all the rest of the software Non free software that's there like I don't know, Acrobat Reader or you name it.

And what has changed recently is that there is a specific section of the archive to just enable the non free firmware part.

Jonathan: Which I

Stefano: think is a good improvement.

Jonathan: Yeah, yeah.

Stefano: I mean, I also have here, unfortunately, a laptop with an Nvidia chipset. So I could say that technically installing that kind of proprietary model, it's technically easy.

But I don't know if it has changed recently or not.

Jonathan: Yeah, does that change and some of the other things that Debian has done recently to make it easier to use? Does that steal some of the thunder you think from? Distros like Ubuntu and Pop! OS?

Stefano: I think it's different targets. So I, so I, I haven't been using Ubuntu for, for many, many years.

So I don't know what's the state of, you know, usability and and of those, let's say desktop distributions, but I think Debian really is a different target. So it's, it's a perfectly fine distribution for the desktop. I'm using it myself here and it works just fine. But I think the main target is being like a foundational distribution for, well, servers, of course, everything that is in the cloud.

A while ago, I was checking and in the major public, so called public clouds, Debian was the most installed VM.

Dan: It

Stefano: was a few years back, but, and that's really, you know, that's a key thing. Because to get your. Autonomy, for instance, if you want to self post stuff, being very easy to use there is really, really important.

Jonathan: Makes sense. All right. I know I am itching to, and Dan is as well, that is to get into the idea of software heritage. And so what, give us the, give us the overview to start with. What, what is this project? What is it looking to accomplish?

Stefano: Okay, so it's a project that started 10 years ago now. I co founded it with my colleague Roberto Di Cosmo and it was initially supported by INRIA, which is a national research center here in France for computer science and UNESCO for, you know, preserving important stuff for humanity.

And the key idea for this software is to retrieve Preserve in the very long term and share with everyone the entire body of software that is available in source code form that of course is a super set. So it's a larger set of all of free software. And the starting idea was that essentially. asking ourselves, well, is anyone archiving free software so that it does not get lost for future generations?

And when we first asked ourselves that question, we, we thought, well, of course, someone should be doing it. It would be crazy if nobody is right. And actually nobody was doing that. And that was a huge surprise for us. So we, we've, there were a number of other initiatives, including by UNESCO for preserving the executable form of software, like for doing retro gaming.

Or also for making sure that images stored in some proprietary formats, which are only implemented by some software 20 years ago, will still be viewable 20 years from now. So that kind of initiative existed already, but the preservation of source code was not a thing, at least not at the scale that we were imagining.

And so we set to actually create this this project. And as it happens, while we were thinking about that, Turned out that it can be useful for many different use cases. You have preservation for just, you know, not losing all the important knowledge that is embedded in source code. Do you have a bunch of research use cases?

Like I'm a researcher myself, and I've always been envious of physicians that create these? Yeah. amazing infrastructures like, you know the CERN or the very large telescope in the, in the Atacama desert. And I wanted to create something similar for computer scientists tuning software. A place where scientists with an hypothesis about how software is developed can go, run their experiment.

And then go away and do something else. And then finally, you have also a bunch of important industrial applications like create, helping with creating software bill of materials, of indexing all the software components that are in your smartphone or in your IoT devices. It's something that was not easy before and that's something that software editors can help with.

Jonathan: So, I'm going to ask a bit of a troll question, but it might be one that people are actually wondering. And that is, why do we need this? We have GitHub.

Stefano: That's such a serious question. You couldn't imagine. So another big field of use for software is open science. So, open science is the idea that all artifacts related to science should be open.

The papers themselves. Software created for scientific experiments, data, should all be open. And it's amazing how many scientific papers out there, when they want to tell you where the software used for that experiment is, they just add a GitHub link. Which is, which is a great development platform, aside from the fact that it's not free software itself, of course.

But The repositories on GitHub disappear

Dan: with

Stefano: very, very quickly. There are a number of scientific papers that that studied that that try to retrieve software associated to papers in any scientific fields. And you literally get the answer like, I'm sorry, the dog ate. The the homework that kind of answer and say, well, that was a good time owned by a student that is no longer it's been longer with us for the past five years.

We don't know where the code is. So that's why you need that sort of archive.

Jonathan: So when, when software, when you guys archive a piece of software, do you get like the entire get history? Or is it just a snapshot of where something is at?

Stefano: Yeah, so the short pitch I mentioned before is that we archive the source code, but in reality, the form of code that we archive is indeed development history.

And that is really, really important. So it means that essentially, if a Git repository disappears, we can recreate it from the archive, obtaining a hash that matches the one of the repositories that disappeared. And it's actually not only Git, it's multiple version control systems, and it's not only GitHub, for sure, All the different software platforms out there, and there are a number of very interesting use cases.

We are seeing these days maintainers of our repository hosted on public get rewriting the history to retroactively change the license and to make sure that the code that today is free software will no longer be advertised as free software in the future. So when this kind of stuff starts to happen, the importance of an archive is even greater.

Because you need to, you know, to empower users to show that that code was under the GPL one month ago or two years ago. So it means it's still under the GPL today.

Dan: So

Stefano: that are a number of use cases we are seeing. So people are really using the archive for defending themselves. Against, against these strategies of, you know, stripping rights that were there in the past.

Jonathan: So there is a, this is something that, that really concerns me with all of these data archiving things. And that is that there are laws out there, especially in the European Union, that give a right to be, I, I'll, I'll, I'll use the term right to be forgotten. That's not technically the right term, but that's one that's easy to be understood.

How does that work? When you have, well, for one thing, when you're trying to archive all of these things, but also, like, just mechanically speaking, when we're talking about Git, how does it work if someone says, I want to be removed from your database, but it's like, but your name is on half of these Git commits on this particular project.

Like, that's gotta be a nightmare.

Stefano: I mean, that is a tricky issue for all archives out there. And of course, all the applicable laws in this case in Europe, because the data, the data owner is in the world, the data, the DPO, the Data Protection Office is the one from India. And of course, if there are takedown requests, that request to remove code from the archive, that should not remain public anymore.

Oh,

Jonathan: so you, you could go in and you know, this person is now John Doe as opposed to whatever, you know, a real name. If someone wants their real name to be forgotten.

Stefano: people for reason for removing code from the archive that are legitimate under the law, they are implemented and people can remove their code.

And then ideally, if it is free software, the ideal way would be people republishing the code under some other format. And we are kind of like, yeah,

Jonathan: it's, it's such a, it's such a tricky thing. I, I guess the, the place that I come down on this is if something was made intentionally made public, I don't think there should be a right to be forgotten.

I think that is incompatible with reality.

Stefano: I mean, I don't have a specific opinion on that topic. What I say is that in some cases there is, there is to consider also the right of, you know, free software users to not lose access to some piece of free software. Yes. So, and how to balance those two rights in some cases can be very complicated.

Jonathan: Yeah. Dan, did you want to jump in here? I was about to send you a message on the back chat and he beat me to it.

Dan: That's okay. We're, we're revealing the inner workings of the show. Much like maybe we can archive them in future. Maybe Stefano can archive the inner workings of the show. Yeah. So I'm curious about how some of this works technically, really.

I suppose I've been reading through your documentation, which is excellent, by the way. And I've been reading about some really cool acronyms like sword got me. I was like, Ooh, they've got a tool called sword. That sounds really cool. So can you tell us a little bit about the kind of actual physical process of doing this?

So how does it work?

Stefano: So essentially what Git does, and I'm taking the example of Git because essentially models of most other version control system out there can be mapped to Git, is essentially creating a big graph. So your Git repository is itself a graph and each object that you store in Git as an identifier can be a file, can be a directory, can be a commit, can be a release.

And essentially what we're doing is that We are either crawling, if they're available from some public development platform, or receiving requests for archiving specific repositories, as you would do for a web page on the Internet Archive, for instance. And essentially, we are storing all the objects we retrieve from the version control system into a unified model.

This way, if a file that is in your repository is also present in a gazillion different repositories out there, we store it only once. Same thing with a single commit, and same thing if you have one million forks. Of the exact same version of the Linux kernel git repository while you store it on the watch.

You don't need to store it 1, 000, 000, 000 times. And so essentially it's a fully duplicated draft data model keeping information about where stuff you have archived. Come from in the in the public Internet and something else. So it's a protocol that we did not create ourselves. It's a protocol used by open access and platforms for store papers.

It's an interchange format so that if you deposit a paper on a specific open access archive, it can be pushed. To a different archive, and we support that for integration with open access platforms, for instance, scientists, depositing paper on those platforms can say, hey, by the way, I have an associated software in source code form for this paper.

And when it is recognized as being. source code, it's also pushed to the Software Heritage Archive via this work protocol. That's super useful because you, you can know that some specific software you have archived is associated to some scientific paper and then keep the link between the two.

Dan: Oh wow, that's excellent.

Is there a mass import of stuff from GitHub? How do projects get, do they have to request to be included in the archive? Or do you proactively kind of go out and say, these are all public? In some platforms, we archive all publicly available Packages from pipe high or indeed repository from the public guitar.

Stefano: In other cases, users request that we explicitly archive some specific forges like a GitLab instance, also operated by a public administration or operated by a research institution. People from those institutions or third party users of those repositories can request that we archive all those GitLab instances, that GitLab instance in full.

Dan: Wow, excellent. And where, I imagine physically there's quite a lot of data here, so do you have masses of cloud servers and stuff out there that you put the stuff on? Are you duplicating it all over the place so it can never be lost?

Stefano: So in terms of data size, it's about two petabytes of data. So it's big, but it's not, you know, a video of video archive big.

And we have, of course, multiple copies because the only way you can keep something safe is to have multiple copies of it. So we, we make a distinction between copies that we operate ourselves, like to protect against the virus. Discs failing or that kind of stuff and independent mirrors that even if we wanted to, we couldn't destroy ourselves.

So we have three copies of ourselves that we operate. One is on bare metal, also here in France. The Inria data center is a couple of racks full of, completely full, I think, to this day. And Two other copies on public cloud that offer in kind storage space. One is Azure from Microsoft and one is Amazon S3.

And then we have independent mirrors. So one full mirror is already operating, operating from Italy. And we're in the process of setting up another mirror in Germany and another one in Spain.

Dan: Oh, wow. Excellent. And talking about the legal side of this as well, you've mentioned a bit about licensing and stuff.

That really interests me. What what set of licenses do you accept? Is it the stuff that relates to, say, the OSI approved licenses and things like that?

Stefano: So right now is really is much larger than that. So everything that is public is archived. And in some cases you can archive stuff for Public interest, even if the license is not speaking for software, but in addition to that, we do some, let's say, mining of the stuff.

We have archives that we recognize the license of the software. We have archives. So you can actually query that guy. Not in batch mode, but for a specific project, you can see what the license is of that software. And if you only care about, you know, OSI compliant licenses or FSF compliant licenses. We have the metadata to actually support that kind of searches.

Dan: Oh, wow. That sounds, that sounds brilliant. So how many people are involved in this? Because obviously, I know you started off with just two people, I believe. So is there, is there now a growing team?

Stefano: Yeah, that is correct. So in, in terms of the different sort of roles, but so we are about 20 people operating on this, and I should mention also that is a completely nonprofit initiative.

So there is no company associated with that. It's really for the public good. And you have all sorts of different roles. So you have software developers. So we have operators, system administrators. We have a bunch of researchers that are also doing research on this kind of stuff. And that is the part I'm mostly focusing on these days.

Okay.

Dan: And how do you keep it going financially then? Is it through donations and so on?

Stefano: So the the source of funding is diversified because it's the best strategy for financing any non profit initiative So we do accept donations, but the bulk of the funding comes from sponsors that are in part public sponsors like I've already mentioned inria, but there are also a bunch of other institutions that can be specific universities or public bodies around europe And we also have a few of the tech sponsors that finance a part of the project, but it's quite diversified.

So we are not dominated by any single sponsor and that's a very healthy position to be.

Dan: And on the subject of diversity, sorry, I'm jumping in again, but on the subject of diversity you've got some stuff to do with DEI. Dave, do you want Diversity, equality, and inclusion and trends and so on that you've been looking at through the research.

Can you tell us any more about that?

Stefano: Yeah. So as part of my, so here changing hat a little bit. So I'm a researcher myself and I'm a lot of the stuff I'm researching is related to free software. And of course I'm using the a lot of the public data that software heritage is producing for, for my kind of research, which tend to be very large scale.

So some of the studies I've done in the past are related to diversity in at least a couple of sense, gender diversity, and. Geographic origin, diversity, and essentially you can mine the software heritage archive to see who is committing code and seeing what is the ratio of male contributors, female contributors, and doubt that is evolving over time.

And you have a very long window of observation because we have not, well, we started archiving in 2015, but of course, version control system preserve history. So, and you have the timestamps. And so you can observe essentially. Something like 50 years of development history as recorded by version control systems.

So in one of the study I did a few years ago, we observed the ratio of yearly contribution coming from women. And as you can imagine, in the entire body of public code out there, it's pretty abysmal. So it essentially, overall, overall, 5%. But the trend. was actually growing yearly, and it reached for the first time, just before the pandemic, to 10 percent of the yearly contribution coming from women.

And unfortunately, the COVID pandemic changed the trend. And we have actually proved in a recent paper that it was not just a coincidence. And actually, the COVID pandemic caused the inability of women to contribute to free software, or at least to public code in general. This is not surprising, because essentially, The pandemic has heightened some of the interesting discrimination that exists in households and whatnot.

But we've actually proven that it has been the case also for public local contributions.

Jonathan: So in, in that particular realm, one of the things that's always interested me is, and I kind of see this as a strength of open source and the way things are done, but like a lot of commits just come with an email address and that is sort of inherently non gendered, right?

I, I'm just. Sort of curious, how do you go from all I have is an email address or even all I have is a name? and And it's it's sort of dangerous these days to even make an assumption about gender from a name how do you how do you go from that to determining whether a commit comes from a man or a woman?

Stefano: Yeah, absolutely. So there is essentially a spectrum of the kind of methods you can apply to this kind of studies Which depends on the size of your sample. So if your sample is small enough The right way to do is just go and interview people and you ask themselves which gender, what gender you identify with.

But here we're talking about tens of millions of authors, so you cannot possibly go to ask tens of millions of authors and you shouldn't do that either because they've not opted in to be interviewed or that kind of stuff. And so the approach that you use, you actually recognize not from the email, because it's usually not very informative, but from the full name, you recognize their there's a number of characteristics can be the gender, or it can be geographic origin, and you validate that with external data sets.

For instance, people can self declare. The gender on the top metadata or their own pages, but you also have some ground truth data set. For instance, we obtain a data set from the Olympic committee, essentially, with all the statistic of all the athletes that have ever participated into the Olympics with their name, their origin, their gender, the country that we're coming from, and you can use that to essentially.

Test if your automatic tools for detecting gender or geographic origin is correct or not. Of course, it's not as good as interviewing people, but for analyzing such huge data set in aggregate form is, is good.

Jonathan: Yeah. Interesting. What, what are some of the other fascinating sort of scientific queries that people are doing?

What, what, so we kind of dug into gender and you've mentioned a geographic area, which is super interesting to think about. So if you want to speak on that briefly, you can, but then I'm curious, like, what, what else is there that people are looking for? In, in the status set.

Stefano: A quick thing about geographic origin, because it was really fascinating that we were being able to observe essentially the history of technology through the lens of geographic origin, like in the early sixties, you see a lot of contributions coming from North America.

That was the early year, next day. And then you see later in Europe coming up and in more recent years China and other countries coming up again. So this is really, really fascinating, but so more than that, there are a bunch of other things. So one thing is software evolution. Like, you would be sort of surprising, or maybe not, to know that the ratio at which new code is published has been exponentially growing for more than 20 years now.

And is stable in both the number of new commits that you see coming up, or of new files that get published into public code. So there is exponential growth of code that is, at the very least, Publicly available and a big part of that is also called is actually free software. So that's great. So the humanity is producing more and more Digital commons and releasing it to the public for others to see that's another thing and you also have a bunch of Studies related to security for example You can observe how vulnerabilities propagate from one repository to another.

You can observe vulnerabilities that are fixed in an upstream repository, but not integrated into downstream forks. That's another big example. And then you have a bunch of technical advancements that you need to create, essentially, to be able to analyze data at this scale. And so we've been working a lot on compression techniques.

So, can we make the entire body of software that we have archived fit on your laptop one day using specific compression techniques? Maybe yes, maybe no. Can we make the entire development graph fit in the memory of your single server that you have at home? The answer to this is yes, for instance. We have developed techniques to do that.

Based on compression techniques. So graph compression techniques. Sorry. So you see, it's really a huge ramification of thing that you can do. You can study using this data and you have to develop yourself to make studying these data actually possible.

Jonathan: Yeah. Super interesting. So what, what about.

Documentations like I'm curious about the things that are that are included in this kind of repository and is documentation in scope. So obviously, in some cases, documentation is just part of the source code, you know, it's part of the get repository. And I would imagine that that sort of gets hoovered up automatically, but brought more broadly than that is is documentation.

And I'll get to a more specific question about this in a second. But like website content, is that is that in scope? Is that something you're thinking about?

Stefano: Yeah, so the essentially the choice in software heritage was to at least initially archive what is stored in version control system repositories that are publicly available and we select development platforms that are usually used to develop software, but you're absolutely right that incidentally archive a lot more than that per capita documentation, archive essentially taking the example of guitar, Everything that people could put on GitHub.

And there is a lot of trash in there. Okay, you have, of course, all your random website. You have people using GitHub just to create their portfolio of contributions to look good on, you know, on the market. Yeah, that sort of thing. So the solution there that we well, our approach is being we archive everything.

And then we develop techniques to recognize what's what. For instance, we develop classifiers to decide whether automatically whether a repository is a website, is a data set, is a is some source code. And you get from there. So downstream users of this data can search what they care about. And only consider that in your analysis, for instance.

Jonathan: So you could, someone could put together a query and say, show me all of the websites that are you know, hosted on GitHub pages, you know, or show me all of the places where people use GitHub to manage their website as a, you know, as you just do a Git pull to be able to do updates to their website.

Stefano: So that is correct.

That said, we are not operating ourselves a cloud service that can, you know, at the CPU capacity or memory capacity or storage capacity to execute queries. So we're essentially the storage and computing resources of the project are usually mostly for their cutting part, but we create data sets, for instance, that.

Researchers can actually use and do these kind of queries on their own hardware. Sure.

Jonathan: Sure makes sense So this this leads into sort of an obvious next question and that is Are you do you anticipate are you planning on changes for what exactly? The the archive holds, you know, are you going to expand into we want to archive more web pages?

Stefano: Web page is not really but we are getting into the territory of for instance Issues, bugs discussions in pull request, because essentially if you take like an archivist or a museum curator, curator approach to rebuild the history of a specific software project, especially a community one, all that kind of things are really, really relevant.

We decided not to archive that kind of stuff in the beginning because a common data model for that stuff looked way more complicated than the common data model just for the code. And but we are starting to look into that right now. So archiving issues are kind of a pull request. Discussions is something we're looking into right now.

Jonathan: So something something along the lines of archiving the Linux kernel mailing list.

Stefano: Well, yes, I guess so. Not really. We're not really specifically looking into mailing list. We're first focusing on what is Part of the common development platforms out there, and if you look at most of the Gitforges out there the GitHub one, but also all the GitLab or GTI instances usually support at least issues, discussion in issues and discussion associated to pull requests.

So that's what we're starting to look into.

Jonathan: Yeah, I imagine at some point though that's going to have to include emails because there are still Quite a few projects actually that their discussion and their issues and their pull requests all happen over email and their mailing list

Stefano: Yeah, they are correct.

Jonathan: Yeah,

Stefano: but to be honest, that's easier to handle than you know All the different pull request discussion and whatnot, but they are more difficult to link To be interesting objects in the archive. More difficult to link to the commits and what not.

Jonathan: Right, we already have tools to be able to to collect emails all in one place.

Stefano: Yeah, and so actually Jemaine, that was a, I don't know, has it been restarted? Because Jemaine was a precious resource, but at some point it was discontinued. I don't know if it has been restarted since. Okay, anyway

Jonathan: so in the kind of in the pre show, maybe before Stefano joined even I, I was talking to Dan about the internet archive and, you know, that is a super useful resource, but they are also sort of seeing some trouble and, and it, it kind of occurs to me that you guys are in, in a very small slice of it are a backup to what the internet archive does.

Stefano: Yeah, so the archival model is pretty different in the sense that the Internet Archive takes a traditional archivist model in which you archive stuff and you put the stuff you archive in a box, and you can retrieve that box to obtain this website, this webpage, or this zip file that you have archived, while we take a more unified model in which everything is intertwined.

So that's the big difference in terms of archival model. But other than that, yes, it's a precious resource. I've seen some of their str technical struggle due to attacks and whatnot in recent years, and that was really scary

Jonathan: even for us. Well, yeah, so I, I genuinely, I, I hope so. Like, I hope you guys are looking at that and taking notes of there are trolls out there, there are these problems out there, and it's probably going to come to your door at some point.

Stefano: Yeah, I mean, it's, it's, it's only an uphill battle, right? Because we're, we're, we're fairly size. Team, like 20 people is not nothing, but it's not the fire capacity of the huge companies out there that have, you know, if they have an attack, they can just deploy 50 people to defend against that attack. So, but, so we do what we can and we take, of course, security very seriously.

Jonathan: Yeah. What do you think the history, excuse me, the, the future of archiving looks like? Like, so this is obviously something that's very important, I would say to humanity. Both the internet archive and software heritage, both of what you're doing. And there are, there are other archives out there. I'm, I'm not sure of the details of all of them, but I know there are some other ones out there.

What is, what does this look like in the future? You know, are we, are we, are we getting to the point to where the things that are being archived are just going to become so overwhelmingly huge that it's not, it's not practical for a small organization like yours to be able to do it are we looking towards a future where maybe governments need to step in and.

Budget towards this.

Stefano: The digital preservation community is something really fascinating. So there is a huge community of both practitioners, companies working on that researchers doing the groundwork for how to do that properly. And it's fascinating. Everyone focuses on different, let's say, kind of artifacts.

And the question of size is really interesting. And I was surprised that one of the first. questions we got from the digital preservation community when we present a software heritage was indeed about sustainability. And sustainability is not only funding that we have already discussed, but there's also sustainability in terms of sizing.

So is it doable to archive what you're archiving? I think with software source code, we are in a kind of a sweet spot in the sense that very precious human knowledge, Takes a lot of time to produce, usually brain time, and then is, let's say, serialized into a fairly compact representation. Like, every developer knows that they can spend a day finding a bug, then result in a single line change in a piece of code.

So in comparison to, like, archiving videos, public videos, where you can do a 10 second HD video of a cat and get a gigabyte of data, I don't know, maybe not a gigabyte, but a lot of data. We are in a safer spot than, well, let's say archiving the other kind of format. But still there is a question of how you will find something interesting into the trove of the 300 million projects that we have already archived today.

And we're working with archivists, curators, and to actually enable this kind of this kind of research. But we see ourselves as a necessary building block for doing that kind of work in the future, rather than being the ones that should, you know, answer that right now before, instead of letting the expert of that part work on it later.

Dan: So I'm interested in security as well. You talked about. The security of the archive itself, but I'm thinking of the useful, there'd be really good use case for the archive would be tracing security vulnerabilities through different pieces of software and how they've affected it over time. Is anyone doing that kind of thing with it?

Stefano: Yeah, as a matter of fact, we're working on that right now. And so it's something that is already done via other techniques. So for instance, you already have a bunch of. even, you know industry products that do that based on package metadata. The way we work on that is looking at the commit graph. And we're, we're trying to see if the increased granularity, which you can do can help finding vulnerabilities in projects that might be unaware of that.

But there are a bunch of other secure, relevant security use cases for the archive. Like there are people that are trying to reproduce vulnerabilities. So there was a vulnerability 10 years ago in a specific piece of open source software. Can we reproduce it today to see if the resultant time were sound?

Or even just for historical reasons, like the previous talk, right? And that are kind of activities in which a software archive, a comprehensive software archive like Software Heritage can help. Because you need the vulnerability, you need all its dependencies. And, and, and only with a comprehensive archive, you can know finally all you need for that.

Dan: And I heard you say 300 million projects there. I mean, I'm sure that's a ballpark figure, but I was going to ask, do you know how many how many projects are actually being archived right now?

Stefano: So we do know. And so project is kind of a blurry term that we use in, in for software a lot without Defining it properly because it's very hard to define.

And so the figure I've mentioned is if you want the number of URLs that we archive, where a URL can be the URL of a specific Git repository, or the identifier of a specific package in some repository. Package manager repository like pipe or NPM or or or whatnot. And we have 300, I think 30 millions such software origins that, the way we call them archived right now.

Dan: Wow, that's amazing. Is there a a target that you'd like to get to at some point? Like is, you know, we want to get to a billion or we, no,

Stefano: not really. There is no target question. We, we get a lot, and it's really hard to answer is that, can you estimate how much you're missing? And the answer is that noise.

That's very hard to do.

Jonathan: So I am, I am listening to this and I'm, I'm, there's a thought that's going through my head. And that is, you guys saw this need to archive source code. And you jumped in and it's there. And so it's, it's taken care of now. What is not taken care of, right? And so I'm thinking of things like you know, the old GeoCities.

Lots of people had little websites put on GeoCities, and there were some efforts to archive those. And I think Perhaps all of them were archived, or at least the majority of them. You've got the internet archive archiving things that they can get to from the internet, but then you have places like YouTube where people are archiving things to YouTube.

You can imagine a future where. Google pulls the plug on YouTube because they, they do that sort of thing. And then I'm thinking of even things like, you know, there are, there are TV shows and movies from years and years and years ago that never got digitized and there were fires in the you know, in the place where the studio was storing their master copy of it.

And there are some of those things that, you know, as far as anybody knows, they're just gone forever. And it brings to mind this question of like, what's the next frontier or, or what, what are we missing? Is there some place out there that. There needs to be an archiving effort that's, that's not there. You know, do, do you have any, any feeling of something that, boy, it would be nice if the, if this was archived in a, in a safer way than it is now.

Stefano: Yeah. So let me stick to my comfort zone or focusing on computer stuff. And what we discussed thus far is essentially. Public code that we archive, but there is a lot of important code that is not public, not because it's proprietary, but just because maybe it's not even digitized, it can be on a punch card.

Okay. And we, in computer science, we're in computing in general, we are still in a pretty lucky let's say period in which a lot of the funding fathers and funding mothers of computer science are still around and it's the time to go see them and see if they have that floppy disk or that punch card or that.

magnetic tape still in a drawer and work with them to digitize that stuff and archive it. We have done this for a number, for a very small number of historical software, but it's really time we do that for essentially all the important historic software that have made the history of the world before it gets lost because People just are no longer with us anymore, and nobody else knows that there is something important in that doorway.

Jonathan: Yeah, so you talk about not proprietary software. I think there's a, an argument to be made that proprietary software, really, it would be ideal to have archives of that source code as well. And just, you mentioned earlier on retro computing and retro gaming, like, I just, I kind of shudder to think, but I wonder, like, how many of the 8 bit and 16 bit era games is the source code just irrevocably lost?

Stefano: Yeah, and I've been talking with a lot of people developing retro gaming emulators, so free software retro gaming emulators, but they have to deal with, okay, how do we support this specific game that is, you know, From this specific realm, and they get a lot of issues in which in some cases they cannot do that.

And it's even hard in most cases to identify who are the right owners of that piece of code. So maybe you have the bytes, but you don't know legally who has the rights on that piece of software. And I mean, don't get me started on proprietary software. So for me, we could even as well expropriate all the proprietary software, make it free software and archive it all.

But it doesn't look like we're in a world where this is going to happen anytime soon. But the day it happens I want to archive it for sure.

Jonathan: Yeah. I don't know.

Stefano: On the other hand, it's more complicated for companies, but for code written by individuals, if it is publicly available at some point, the copyright on it will expire.

It will take decades, but at some point it will expire. So it is indeed important to archive that as well. You usually cannot do that with a public archive like Software Heritage, but you can imagine an escrow schemes in which it is deposited somewhere and the day it becomes the day copyright expires is automatically published and automatically archived.

Jonathan: Yeah, you know, I think, I think a lot of people are with the resurgence of retrocomputing and retro gaming. I think a lot of people are sort of starting to think about that. And every time there's another story about, you know, how something is just lost to time. I think more and more people are sort of becoming aware of the problem.

And I don't know that we're, I don't know that we're ever going to completely solve it, but just having more people paying attention to it is, is a win at least.

Stefano: Absolutely. I mean, people think about that. I mean, people think that, you know, GitHub is not an archive or YouTube is not an archive, you know, sending out that message, at least thinking about where I should put this stuff to maximize my chances that it will be available in the long run.

This is a very important message to send out.

Jonathan: And even, even the fact that you know, in the United States, when things started entering public domain again, that the, the, the companies, the, the, the usual suspects of companies that don't like that, they did not try to pass another, you know, get another law passed that would extend copyright, you know, another 20 years.

And I, I did some reading on, I've done some reading on that several times, and the statement that they made, that those companies made was, There is enough awareness of the issue now that they didn't feel like that they could make it happen. That there would be so much pushback from the, so much outcry from the public that it wasn't worth even attempting it.

And that's, it's, it's a huge win.

Stefano: Okay, that's interesting. So I, I think we all have in mind that there's a diagram on, on Wikipedia on the evolution, the periodic evolution of copyright terms over time. And so you're saying the, pulling this out this day would be more difficult than in the past? Well, I hope you're right.

I'm skeptical, but I hope you're right.

Jonathan: Well, so in the United States, at least things are entering public domain. And so we have public domain and it was not that very many years ago that nothing was entering public domain. Right. And, and so I, I saw an interview where one of the CEOs from one of the big companies that was behind a lot of those laws, in fact, The, the law was, was named after them, not, you know, not on the, on the actual law, but people were referred to it as the, the nickname that it was named after this company.

And there was an interview made where they, they made the statement that there would be enough public outcry against trying to do this again, that they opted not to. They just didn't think it could happen. Very interesting. So, yeah. All right. We are. At the bottom of the hour. And there's a couple of questions that I am required to ask you or else I get in trouble.

And you, as a computer science major, I'm sure you've got answers for these and this, what's your favorite text editor and scripting language. So I made the news when at the time I was the Debian Vim maintainer and they switched to Emacs. So I'm still on Emacs these days, but with evil mode. So with the VI key bindings into Emacs.

Stefano: And

Jonathan: scripting language

Stefano: scripting language I would say Python.

Jonathan: Yes. Yeah. That's, that is very, very typical of these days. Do you do anything with OCaml anymore?

Stefano: No, but so that place in my heart has been replaced by Rust. So I'm a big Rust fan. So for everything that needs to really scale, I ditched Python and switched to Rust.

Both of them with bindings and whatnot.

Jonathan: Yeah, yeah, I I finally sat down with the this year's Advent of last year's Advent of code. I finally sat down and started doing some rust work. I made it about 3 days in before everything else started happening and I had to shelve that again, but I will eventually get back to it.

Stefano: I got started with Rust. I also decided to do Advent of Code. It was a few years back.

Jonathan: Yeah. It's a, it's a good way to get started on it. It kind of gives you an excuse. Yeah, exactly. Exactly. All right. Stefano, thank you so much for being here. It was an absolute pleasure. And we will definitely have to have you back in a year or two maybe six months.

I don't know when talk about when things have changed because things are obviously always changing in this particular realm, but we sure appreciate you being here.

Stefano: Thank

Jonathan: you both for the invitation. Yes. Bye. All right. What do you think, Dan?

Dan: A great discussion and such an important project as well, because as, as Stefano was saying there, you know, so much of this stuff could be lost otherwise.

And it's really important that we have, you know, a good archive and they're doing a great job of it, so it seems.

Jonathan: Yeah, you know, and we, we mentioned this a couple of times, but there are, there are things that you read about that have been lost to time. And it's just, it's, it kills me every time I come across one of those, you know, like you, you have a memory from childhood.

Oh, I remember this, this TV show we used to watch when we were little kids. And sometimes you, you go and you search for it and here's all the, here's all the episodes. They've done a re release on DVD, or here's a bunch of the episodes on YouTube. But sometimes you go and you look one of those up and it's, yeah.

Those were never digitized and they were lost in a fire in 1987 and it's like, oh man, just forever gone.

Dan: In the case of the BBC and in the UK, where I am, they actually ran out of tape and started to record over old episodes of things. So they recorded over lots of episodes of Dr. Who. So if you're a Whovian, you know, you'd be upset by that.

They, yeah, they recorded. new episodes of soap operas and things like that over these classic episodes of sci fi and all kinds of stuff just because they were like, we're not going to buy more tape. We've got lots of tape here. We're just going to reuse it, which is such a shame. Yeah, it is. It is.

Jonathan: Yeah. We could, we could dive into that more.

I, I have questions. I've heard that before, but there are, there are just questions that come to mind, but we don't have time for that. So there, is there anything that you want to plug before we let the folks go?

Dan: Yeah, so I'm, I'm heavily involved with Liverpool Makefest, which we've talked about in the past.

It's the biggest maker kind of celebration in the UK. We have about three, 4, 000 people come along. Sometimes we've had up to 6, 000 in the past. Not all at once because they tend to pass through during the day, but I'm just in the early process of sorting stuff out for that this year with my colleagues who run that.

So if you go to liverpoolmakefest. org, you can find all the information on there. And I'd encourage people to to give that a look. And we're coming up on the, it's going to be in July. It's the first Saturday in July. So you've got quite a bit of time. So if anybody fancies coming over from other countries, we have people coming from Canada, we have people coming from all over.

So a distance is not an excuse, Jonathan.

Jonathan: I am, I am making it out to one convention this year and assuming that goes well, maybe next year we'll expand a little bit more, but I don't, I don't know what the future holds for me. So we will just have to see. But yeah, I, I, I've got several friends now that are going to be at FOSDEM.

So that'll, that'll be fun. I'll get reports from everybody on how it went. Alright, I got a couple things that I would like to plug as well. Of course, we have Hackaday, my work there. You got the security column goes live every Friday morning. Make sure and check that out. We appreciate Hackaday as the home of Floss Weekly.

Also have the Untitled Linux Show, which is over at Twit. Dot TV, make sure and check that out as well. If you want more of my ramblings, as we said today. But yeah, we sure appreciate everybody that watches the show and listens, both those that get us live and on the download next week, as we said, we're going to have Doc Searls back as the co host and we're going to talk with someone from CIQ and that's actually the company behind Rocky Linux.

I am very much looking forward to that. We've had Alma lakes on the show. It's only fair that we have rocky links as well. So make sure and be back for that next week. And again, thank you to everybody. And we will see you next week on Floss Weekly.