Jonathan: Hey folks, it's time for Floss Weekly and this week, Rob joins me. We talk with Alistair Woodman about FR routing, but also the future of open source, open source business, software liability, and all kinds of other stuff. It's a lot of fun. You don't want to miss it. So stay tuned. This is Floss Weekly, episode 814, recorded December 31st.
The Banksy situation.
It's time for Floss Weekly. That's it. It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we're back after a couple of weeks break for the holidays, which is kind of ironic considering that today is December 31st, so we're not quite through all of the holidays yet, but that's okay.
Uh, we've got a guest, we've got a co host speaking of which, uh, let's go ahead and bring Rob on our other resident, uh, networking engineer, networking experts. And, uh, Rob, I think we're going to have to take our networking expert hats off for today because we've got, we've got a, a real expert, or, uh, he was telling us before the show that he doesn't do a whole lot of coding these days.
So, so maybe he's the business manager for the real experts. We'll, uh, we'll ask him
Rob: about that. Yeah. I'm interested to see what he has to say about, uh, about the business side of things.
Jonathan: Yeah, yeah, that's something we're very interested in around here. Um, so our our guest today is Alistair Woodman and he he has an interesting background Uh, he says by education.
He's a physicist And by inclination he is into Product management and business development. He's done apparently some coding in the past, but he said he retired from that. He hung his head up on that one. Um, but he's involved with some really neat things here, like, uh, FR routing, which I've been telling people this week is going to be about like sort of the, the deep internet protocols because they do, uh, they, they do some of the, the routing things that you've never heard of, or maybe you've heard of once or twice, but you don't have like, BGP, the Border Gateway Protocol.
I know, I've never messed with BGP. Uh, I've written about it because every once in a while you'll find a story where, uh, somebody advertised a bad BGP route and we accidentally routed the entire internet through Brazil for a few hours. It's like, I'm sure nothing malicious happened as associated with that, right?
It'll be fun. All right, let's go ahead and bring him on. Mr. Alistair Woodstock. Welcome to the show, sir.
Alistair: Hello, good morning, good evening, whatever good next year for those of you, uh further afield in the east So,
Jonathan: oh,
Alistair: yeah.
Jonathan: Yes. I've i've already gotten some well wishes for 2025 in my various discord channels and that's always fun Yeah All right.
So What, what, what all do you have your fingers in? What all are you involved in? Um, are you a wizard of the deep internet magic? What, uh, what are you up to these days, sir?
Alistair: Well, so, well, I, I left, uh, formal, you know, working business stuff in 2011, uh, after putting in a solid 25 years in high tech. Um, for companies being well paid.
Um, so I took early retirement and then I pretty much figured out that I didn't enjoy golfing. Um, so I think the last time I did play golf was as an 11 year old. So I pretty much figured that I wasn't going to enjoy it. So I decided what else am I going to do? And I actually figured out that I really like hanging out with nerds anyway so, um, I still hang out with nerds and I work on open source projects and And hanging out with nerds is either a terrible thing for many people and uh, but I actually enjoy dealing with people who are sort of uh, uh, very diverse very focused on what they're interested in and um, I mean It's sort of a popular meme here in silicon valley that we have a lot of you know on spectrum Uh type individuals and I I like that Working with them and I like getting, you know, good results out of them.
And I like helping them, um, interface with, uh, business issues because people in the tech community tend to be very focused on what they're doing. And of course the thing that they've done. Is super important and super useful. And the question is how are you, you know, keeping a roof over your head and paying your bills?
So a lot of the time, um, I spend my time running around, making sure that that part of the process is working for open source projects, unfortunately, I can only do a couple. Um, but I'm heavily engaged in the two that I'm active in at the moment.
Jonathan: Okay, so there is, there's a lot, there's a lot in just that opening statement and uh, I didn't realize we were gonna go down this route but it is super interesting to talk about.
Um, and that is that it's quite a, it's quite a skill set to be able to talk to nerds. And, and I, I very much appreciate that thought of, Some of these people are on the spectrum and I think that's true because I've known some of them over the years and It sort of is a Useful, but also important skill set to be able to I I call it translating.
I refer to it I translate for nerds and I think that is that is super useful. Um, I'm i'm thinking of a there's a there's a youtuber, uh, david plumber, I think um that I follow he worked at microsoft for years and he's like at some point people told me enough times that I ought to go get tested to see whether it's on the spectrum and I'm like You know after laughing them off enough times I finally went and did it and then I started reading about being on the autism spectrum And he goes so many things about my life started making sense.
I think it's that's super fascinating I think it is a real thing. I think it definitely is.
Alistair: Okay. Well what I think it attracts people with um, special Gifts and skills, right? So some of it can be just pure IQ related stuff, but a lot of the requirements for being a very good coder are sort of very good memory, understanding of structural things across space and time, being able to sort of figure out what that really is.
means in terms of where to find something in code. Um, and these are very interesting skill sets. Um, you know, people who, you know, study the Hebrew Bible, um, also have those types of skill sets, right? It's, it's very focused a lot on the written word and, and what that means, right. And it attracts a very gifted set of people.
People and sometimes they also have, you know, um, other proclivities, right? And as we all do, and Uh, they sometimes want things to happen in a particular way and get frustrated when you know, it doesn't happen Um, or the rest of the universe doesn't appreciate that. They aren't really the center of the universe So
Rob: I feel like I feel like a key strength may be the ability to hyper focus on something Yeah, whereas me when I got into it and technology I feel like I went the opposite way and I just so many things are flying at me from so many directions that I'm all over the place and and if I had maybe more of a hyper focus, I could really focus and go down one of those paths, but I'm more of a jack of a lot of things
Alistair: and that is an interesting point because when you building larger scale.
Communities and interrelationship you can't afford to be hyper focused on You need to be able to Frequency shift from being very focused on a particular thing at the time being and then try to build connectivity or causal relationships to other things and and that's um, That's an interesting challenge that that one faces when you're doing business development, right trying to find the Yeah The way of, of connecting something that a project's doing with a need that somebody else has somewhere else.
Jonathan: Yeah, so one of the, one of the interesting things that I've been faced with a few times, honestly, trying to make money writing code. Um, when, when you, when we're writing code, things are very, um, binary, you could say. You know, either this code is written correctly and it works, or this code is not written correctly, and either the compiler tells me, or I get, you know, runtime errors and things break.
Uh, the business world is not quite that, uh, cut and dried. And oftentimes when we are convinced, you know, our internal compiler is telling us that we've got just the awesome, the best business idea. This is going to make us millions. And then you go to launch it and nobody cares. Uh, it's, it's quite a different world.
Um, I have found in trying to take my coding acumen and move it into the business world.
Alistair: Uh, that's very true. Um, and both in the business world and even just people using hobby projects from one another, right, amongst open source stuff, right, the new mousetrap or somebody's done something that already exists somewhere else and everybody goes, I don't care, right, the marginal improvement isn't good enough.
So. You know, have a nice day, right?
Jonathan: Or, or the idea is so niche, there's, you know, all of five people around the world that are ever going to care about it. Yes. Yes. Been there. Um, All right. So let's, let's get into, um, a little bit of maybe, maybe background. So you're involved from what I understand heavily with, uh, the FR routing.
Project is that is that the only one you said maybe there was two projects that you you kind of have a hand in
Alistair: Well, so the other thing that I'm involved in is the Erlang ecosystem foundation, which is a group Of projects so, um, the the two are different in scope and scale um, but if we talk about the free range routing project or frr as most people refer to it as um, you probably Um, well certainly people listening to this will be getting their traffic over, um, routers that are running FRR.
Um, we're usually, typically in OpenWRT, um, releases. So anybody who's running on a lot of the embedded hardware for home routing stuff, it's got OpenWRT in some flavor underneath and with a routing stack on that. Most people don't turn on all the different daemons that we have and support. But, uh, some people do run, um, decent routing topologies in their own home.
Um, so some people turn OSPF and ISIS and other type of stuff on and have relatively large, um, home networks already, especially if they've got a lot of IoT devices. So, um, that can be. Um, n not the typical home. I've just got a wifi and I need need an uplink type of thing. Um, which is still what open WRT will serve, but many people turn naturally a lot more knobs on.
Um, the FR R is also used at the other end of the scale on Sonic, um, which is a white box nos, and there were a couple of white box noss out there. So via, um, BISD analytics. Um, that use FRR for general routing purposes. So the forwarding plane and control plane stuff is not part of our project, but as many things in the open source software go, we're, they're downstream of us in terms of integrating FRR into their projects.
And they, Then talk to the appropriate hardware, silicon board support packages and whatever it is to build the embedded system, um, installations. So we've got a lot of code, um, on different code bases, uh, usually. As far as the users are concerned, there's a binary, although many home hobbyists will compile their own and put it on their hardware, but most people will be just downloading something and running it there.
Um, so that's what FRR does. It's also used internally, um, By telcos and some of the hyperscalers for um, sometimes purely just operational backup But other people are using it for more interesting projects so it's um, it's been around for over 20 odd years now because it used to start out. It was quagga and then it got forked Well zebra quagga and then free range routing.
So we stuck with the animals, but we went from Uh, we went from the individual animal to the the paddock basically when we changed the name to free range routing Yeah,
Jonathan: yeah, that's that's funny Um, one of the other protocols that I saw that is part of frr is uh, BGP, the board of gateway protocol. Um, I'm curious.
Are there, are there any of the, like the big level one ISPs that use any of the FR routing stack in their sort of big iron routers?
Alistair: The answer to that is yes, and then you'll have to go figure out the details yourself. So, so yes.
Jonathan: That's fair. That's fair. So it is, it is very, very likely, if not certain, that our, our bits that are being broadcast right now are going through the FRR stack. At least
Alistair: in one of the nodes, yes. At one of the nodes, yeah.
So somewhere along the path, yeah.
Rob: Is that because it's built into hardware they're buying, or are they building stuff upon this, are you suggesting?
Alistair: Um, I would suggest that from purely a percentage likelihood, it's mostly people have got a binary that they've got that came bundled with the hardware, and this will typically be the home router of most cases.
But if there are more. FRR on that link and possibly further up towards the data center or hyperscaler or some service that you're accessing, um, then they probably built it themselves on, on white boxes and possibly be tinkering with that. Um, but it is possible to just download the NOS, uh, you know, from, from folks and then just turn a binary on as well.
Jonathan: What's that people
Rob: have.
Jonathan: What's the, what's the license? What's the license FRR is under? Then we'll give it back to Rob.
Alistair: GPL. Okay.
Rob: So that sort
Alistair: of makes a, okay. Sorry, Rob. Go ahead. Go ahead.
Rob: Go ahead on the license.
Alistair: Right. So, so the interesting thing about licenses, I mean, I think it makes a hell of a lot of sense that routing is under a GPL licensing model because if people make modifications to it, they should be making modifications to a standard routing protocol that you really don't want routing protocols that can't interoperate with one another.
So, um, you sort of really, from a community perspective, it's a better solution from a licensing perspective than to allow um, People to just fork their own and then do what they want. Um, there are some hyperscalers who actually do use it internally and have built their own flavors of routing and because it's, uh, they're running it as an internal, uh, software, they do not have to give those, uh, modifications back, but many of them do.
Um, and it, but it makes sense if you want to interoperate with a router on the outside that you better be, you know, capable of doing that. So, forking and, and not reconverging is a bad thing in networking.
Rob: So, if people are using these, uh, higher end routing protocols on, on their Linux system, something they're building or something they bought, Um, Is, is this, you know, this being around for 20 years, is this what they are using or are there other, um, alternatives out there that people use?
Alistair: Oh, there are several other open source alternatives. Uh, BIRD's a BGP, um, code base that, that is quite popular amongst, um, service providers and inter exchange carriers. Um, the, um, The, if you want to be doing multi protocol routing stuff, uh, you're basically in the FRR domain, or you're going to be using, you know, Juniper or Cisco because you, you know, you, nobody got fired for buying Cisco or Juniper products, right?
That problem, right? But if you are willing to, you know, be fired and want to run multi protocol routing, then you'll typically be running FRR.
Jonathan: Yeah, so if, uh, if, if I do actually finally pony up the money and, uh, buy, buy into and get an ASN, an autonomous system number to be able to truly own my own IP addresses, uh, it sounds like FR routing is the, uh, is the project that I would want to go with because I don't have Cisco or Juniper money.
Alistair: Well, so you're certainly being a bit of a piker if you haven't bought your own AS, right? I mean, that's relatively cheap. It's like 500 a year. Exactly, right? That's chump change, right? So you should at least go get the AS. Now, whether you do anything with it is a separate topic.
Jonathan: I look forward to being successful enough that I can say the 500 a year is chump change.
Okay.
Rob: Question from Discord. Does FRR support MPLS?
Alistair: Um, yeah, that's an interesting question. Um, we do. We're on boxes that support MPLS. Um, but the, the routing it, so it, so the answer is partially we don't get in the way, but, uh, we don't typically, um, It usually requires people to have done something else or want to do something else.
The Linux stack does actually support MPLS and you can turn it on. And it will work under those, uh, circumstances. If you're running on dedicated hardware, you'll need the drivers and other types of things to do MPLS on the hardware. So, the answer is maybe, uh, probably, if you just want to run it on Linux by yourself.
It doesn't work on FreeBSD.
Rob: I imagine MPLS being, being that path, um, FRR would support the routing over the MPLS.
Jonathan: Correct. Makes sense. Um, so it sounds like this is a, a fairly complicated project, FRR, and it supports some fairly complicated protocols. Uh, How did, how did we get, how did we get to this point?
Like how, first off, how did, how did FRR get started? And then at what point did it kind of turn into, cause it sounds like it's been commercialized now. It's been used in a lot of places. Hopefully there is some money being paid to help keep it supported. And so how did we get to the point we're at now?
Alistair: Um, so as I said, it was started over, uh, 20 years ago as, um, By in Japan to actually try building routing and as a self study, uh, type of project it as, as it was. Under the Quagga flagship thing, it was already being used and was a part of, so Quagga still was and is used in some of the embedded systems.
So it's probably still out there in, in old routers that somebody hasn't updated for a long time. Um, which is one of the other topics that we might get into is, you know, how to end of life and, and maintain software over time. Um, it's. So when we, when we forked it to FRR, there were, um, business interests.
So there was, um, couple of startups that at the time, if you remember software defined networking and a whole white boxes were a thing. Um, so it became, there was enough startup money being put into the space to actually make it from a very good. Um, code base, but we're still not really competitive when you looked at, you know, Cisco and Juniper and, you know, people would use it, but many people would scoff at it, um, as not being sort of industry ready.
And certainly if you wanted to run networks, the old adage about, you know, You're sleeping at night and your page are not going off. Most people will buy Cisco and Juniper gear. But with the advent of white box, uh, switching, um, there was enough money flowing into that that startups, um, worked on making the code a lot better.
So Cumulus Networks was, um, around at that particular period of time. They no longer, Uh, exist as a separate entity. They were acquired by Mellanox and Mellanox was acquired by NVIDIA at about the same time. So there are folks over at NVIDIA now working on the project and they're a, um, major contributor to FRR.
Jonathan: And just for, for anyone listening that might not know what, what exactly do we mean when we say something is a white box, like a white box switch?
Alistair: Um, it basically a, uh, uh, Vendors have built something to spec, um, so this happened as well with, uh, what happened in the hyperscalers, where they, where they basically said, we don't need to be buying all these pieces of equipment, we can have servers that we will specify, please don't put these extra chips in there, so there's a lot of, you know, specifications for white box servers, and then there were also specifications associated with um, switching infrastructure, we can just have layer two on layer three switching, which doesn't need all the bells and whistles, um, or rather, they were so focused on chip count and hardware costs that they would specify this is all you need to put in this, you know, don't, don't over Egg the hardware.
This is all we want. Please put some open source software on this and we're off to the races. So um white box white box anything is a Statement about the fact that it wants to be relatively generic um and not particularly complex, although Now they are actually sometimes quite complex, right? Uh, but, but you know, they're, they're trying to create a lot of the interest in the white box switching thing was to remove, um, to, to restructure the business in the sense of having a vendor specific, um, my things better than your thing to actually, this is just a utility thing.
It needs to do these basic things. Please just build them at scale cheaply.
Jonathan: Yeah, definitely. Um, okay, so back to FR routing, and I think, I think back into the kind of the business side of things. Maybe the first thing I want to ask is how did, how, how did you go about, so I don't know if you were there at the time, how did they, whoever was, was doing FR routing, um, How did they get in touch with and start getting that startup money to flow into the open source project?
Uh, we see sometimes that open source projects get used for startups, and sometimes it's difficult to get that money to flow back to the actual people working on the code. So
Alistair: that is actually a very interesting question, and I do think we're a relatively rare bird, and if we segue later in the conversation to other examples, I think we are, we are Accidentally rare and that it ended up with this good symbiosis and so, I mean you you Yeah, you were talking about the whole concept of curl a couple of weeks ago, right and just how well supported curl is correct Uh, how, how well used it is and now that it is almost has a business model, uh, associated with it.
Right. And, um, that's just astounding to me, right? I mean that, that we've got something that's, that well used and that's still struggling to get, uh, a viable business model. Yeah. And so when I look at that and some of the other. Uh, projects which obviously do struggle, um, the, the reason that I think we got lucky is that routing is a complicated problem and back to this other issue, it needs a lot of interworking support.
So, when, uh, Where the F are an active project, there are probably 30 very active committers to the project and there are probably another hundred that commit, um, more than 10% of their work time to adding features. And then there's a long tail of. People who just do some feature or bug fix, you know, that show, show up, but because this all has to interoperate with other people's equipment, specifically Juniper, Cisco, and the installed base out there, um, you have to be able to prove that you've not in, you've not just added code, but you haven't put any entropy into the system.
Uh, And so there's a lot of testing necessary with those things to, to ensure that. That you've not broken the code and to your observation earlier, that you think it's a bit of a miracle that networking sort of works and you've got all these things that can connect to things, but it's not quite clear and some of them are in the process of going down or being rebooted and the network, you know, continues to work.
That's because there's all this complicated state machine machinery in there, which can easily get broken by neophytes. And even the experienced folks can break it too. So we have a very, very large CICD. Pipeline and we throw a huge amount of resources at this And so the our budget for doing testing on this type of thing is, you know over You know half a million dollars Uh, a year in terms of testing, both in terms of people and CPU utilization to keep the project around.
And I think it's obvious to anybody who works in the networking space that that's the cost of doing that business. So, The the pool of companies that are involved using the software understand that it's that hard because Everybody has worked in those companies Basically used to work at cisco or juniper in the you know sometime in their career Um, or they might have worked at you know, some of the other guys like alcatel lucent or whatever it is, but people know What the cost is?
And and how much work you need to do to avoid regressions creeping into the system So It it's not too much of us of a hard sell. I still need to go talk to people about this um, but it's it's it's a communicable problem amongst a very small number of Consumers that allow the business development process to occur Um, and we sort of all know one another that helps.
Um, so this is not very we're not an anonymous project So if you if you thought of stacking um some relationship relationship diagram or sort of two by two matrix of how how well people so Daniel, for instance as he Set on the program, right? Doesn't know most of knows nobody who uses his code.
Everybody uses it, but he doesn't know what they're using it for and how they're using it. Right. And we're the opposite end of that spectrum. We have a reasonably small number of. People who use the code seriously, and they also know why they're using it and they know who we are. So we can, we can forge business relationships in a much tighter manner.
So that's how we manage to run an, an, a system which is, um, which is working from a business perspective.
Jonathan: Yeah. What, what is the, what is the actual, um, like funding model? There's a couple of different ways I can imagine going about this. So do you, do you go to all of those white boxers that include your code and say, Hey, why don't you pay us 2 percent of your MSRP on everything you sell?
Because we're in there and you know, we need support. Or do you go to, you know, one of the companies and say the, the amount of, of support we're giving you, we think is worth a million dollars a year. Why don't you write us that check each year? How does that part of it work? I'm very curious about the nuts and bolts.
So
Alistair: it's it's certainly not on a per copy basis um So it's much more along the lines of the this is our support load this is how much it costs to run this infrastructure and um, and then just saying what a fair fair cost payment would be To those people and and some of it scales by the size of the entity in the organization um, and It's it's just a usual sales negotiation thing, uh, because the the quantum size the number of Number of parties that i'm having conversations with is as I said reasonably small.
It could be larger Uh, there is a group of people who still fall into the free rider You know user category that I can't get to or I don't know how to get to them in organizations um, but it's very much You You know, I basically tell them how much it's costing to do the job and then discussing, you know Whether they're a bigger or a smaller user than somebody else and and what does the pecking order look like?
Yeah, and again, this is a tractable problem because it's small right So those
Rob: aren't those, uh, fees or whatever you discuss with them, that's not backed by say a support contract. Is it just complete voluntary?
Alistair: Well, we do also provide support to, to companies who want support. We do that too, but we do get a, a set of donations, which are pure donations to run the CICD.
So we do both.
Jonathan: Yeah, that's interesting. Um, How, I guess the first thing that comes to mind is not every, not, that doesn't make sense for every project,
Alistair: right? Well, that's, that's my observation, right? We are, I think we are quite peculiar or special in that particular regard. The, the other class, similar class that looks like this is, is projects which were started by some very large, Entity that then get put into the public domain, uh, and still have large support from those entities, right?
So if you think about something like Kafka or any of those types of projects, um, they typically, um, Have a similar model where people understand how that's going to work and people continue to contribute. Um, as I said, I think we're relatively is that we organically grew around a type of thing. The code itself did not come out of a large entity as a, an open sourced project.
It, it organically grew but has been used because of networking's focus by larger entities. Um, so we certainly, Are at an opposite end of other some spectrum than say daniel right as he's definitely at the other end um, and and I think we're not typical so if I go and look at if I put my Erlang ecosystem hat on for a while, right?
That, that, that area encompasses multiple different, uh, projects. And so the, because there's a virtual machine underneath that, that is the Erlang virtual machine, which is run by People who want to program in Erlang, or Elixir, or Gleam, or any of the other languages that run on, um, the virtual machine.
There's a much larger tail of people doing stuff, from the very, from hobbyists that are just doing stuff, uh, in their own area, to the other end of the scale. Um, Ericsson and, uh, Facebook with WhatsApp. that use this at, at industrial scale, um, in networks, but we have a much longer tail of people and it's much more difficult to, there are a multiple projects that run on top of the VM.
So for instance, Phoenix, uh, you entered, you had a conversation with Lars recently about the nerves project. Um, so there's a whole bunch of different projects and they all have their own scaling. Systems and they're much broader from a scaling system. So it's much harder for us as a community to reach out to everybody to encourage them to support because it's diverse, right?
So we we have in that space, I think over 1000 commercial users of the technology, probably more. And most of them are dark Uh, and many of them are
Jonathan: free riders There's there's something that's sort of changing that I I am hopeful is going to help with the open source funding problem Um, and and that is there's been some laws passed and there's laws being passed Um, I don't remember the name.
I don't remember what project it was But I saw this statement where someone from a project was was saying We had a business reach out to us and say we need this documentation And for us to be, you know, to, to comply with such and such law. I think it was, was something related to a, you know, software bill of materials.
And it's like, so this, this company reached out to us and said, we need you to do this work for us. And the person running the project was like, I couldn't believe that they had the nerve to do that. And my take on that is no, no, no, no, no, no, no, no. You don't get it. That's a good thing. And your response needs to be.
I will do that for you. Here's my fee.
Alistair: Right. So, so you're entirely articulating the level of excitement that I had when I first saw the Cyber Resiliency Act over two years ago. So, um, most of the people When I first started engaging with the European Union Cyber Resiliency Act wanted to put their libertarian hats on and run away screaming and say, you know, keep, keep these bureaucrats out of my, you know, code base foo, leave me alone.
And I have,
Jonathan: I have a lot of sympathy for that point of view. I sometimes fall into that myself. I have one of those hats and I occasionally wear it.
Alistair: And, and I did it for like all of about three years. Three seconds before the penny dropped that's like, oh Oh This law so the interesting thing is not the law What it says you must do that's not the interesting bit.
There's a whole there's Hundreds of pages telling you what you must do and there will be much more Specifications written about you follow this standard and do these types of things from an implementation standpoint the interesting thing is what happens if you don't do these things and the The law sees that companies will be accountable for Not following business best Practices so you can be held, you know culpable as a company and there's even um One of the other european directives the product liability directive will see criminal Um law associated with some of these types of things if companies and their senior executives Do not follow the law So the interesting question is, oh, that allows me to talk to people about what they need to do to follow the law and the Um, I had a very useful conversation with the Apache Foundation, uh, again about 18, well actually almost 2 years ago on this very topic, is that their observation was that if you, if you just look at the law the way it's written, As you're a company, right?
You'd say to yourself, okay, fine. I've got 10 software devs in my company and I'm shipping something at the moment. And if I look at all the requirements, I'm now going to have to put in place to manage this software, then, um, I probably need 30 percent more resources to do the testing at the backend and the documentation and all this other type of stuff.
So how hard can that be? Right. Um, I just need to hire three people. But. Then your VP of engineering turns around and says, uh, actually 90 percent of our code is open sourced. Yes. And. The, the executive team sits around and goes, okay, so we have to hang on. We have 10 people, but they're only writing 10 percent of the code.
So that really means we've got a virtual engineering team of a hundred. So we now need to hire 30 people. To do the testing and verification That's where we don't want to do that, right? So that's why Everybody thinks that the government's interfering with their you know, their business plans But the obvious other solution to this problem is well Turn around and talk to the folks who are actually maintaining this stuff, right?
Um,
Jonathan: yeah, it it seems like a a real golden opportunity for a lot of open source projects, and I'm, I'm, I'm very thankful that when the law was written and then amended and they worked on it, um, they, they did go in and acknowledge that not everyone publishing source code. Is making a product right because from what I understand in one of the early, uh drafts of the law Um individual open source developers were going to be liable and that was terrifying and a terrible idea Um, so thankfully that got that got yeah,
Alistair: I don't think it was ever I think they clarified their position.
I don't think it was ever their intent to do that, right? I I do think that um that they So i've had the I I discussed I describe this as the Banksy, um, situation, right? So, if somebody just goes and posts something on a GitHub repo of the, you know, I've done the best bubble sort algorithm, you know, look at this and, you know, weep, right?
In terms of, you know, how efficient this is, right? And that's it, right? Everybody looks at it. It's a work of art, right? This is not a product. Product. This is a a political statement. This certainly should fall under the category of freedom of speech and um Fundamentally somebody does pure work like that.
It's just matter of algorithmic genius, so people will be able to do that and you know, I tend to I did talk to many people and they were they were throwing their toys out of the pram to start with and I said look They're not going after the banksy Situation, right? The interesting question is are you actually then maintaining it and do people use it?
Jonathan: And
Alistair: this gets to uh, toma de de pierre's Conversation about well, i'm not your supplier, right? And at some point Teams need to figure out whether they are a supplier And they just don't know their consumers or that they are not a supplier the law will create a bifurcation in thinking and um, you know in the In the conversations that you had recently, right?
I mean it would look like curl is already You know, in the situation of, of accepting and looking for maintenance, um, money, right? So, that has become, that's moved that side of the boundary line in terms of that. And I think a lot of projects are going to have to ask themselves, which side of the boundary line do we want to be on?
Yes. Yes. And the other problem is, is if you want to be on the You know the support In some way shape or form side of the maintenance line, right? How do people this gets to your other question is like, how do you how do you figure out how to pay? That right. So I think the ethos of the communities are most people Don't want to have a per unit License fee for maintenance that I mean that would go back to the you know, the the times before we had You know open source stuff, but people need to cover the costs of the maintenance and That's We need mechanisms of expressing what the costs of the project will be and what a reasonable apportionment is.
Uh, and I, I don't like the idea of just one benefactor turning up and saying, Oh, we'll take care of this cost, right? I think that's also bad from a relationship perspective. You actually want Um, major users who are using it commercially to actually be able to put their hand up and say, yes, I need to have a, I need to know something about these people, especially if they're going to have to fill in S bombs, figure out when CVS are going to get fixed and all these things.
And this is, this is all new territory at the moment. So, um, to Thomas point. The regulators want to see a tighter supply chain, um, by definition, open source and Open sources doesn't have that connection built into it. And we sort of, we need to introduce that reverse path into the system, which is an interesting problem.
Jonathan: Yeah. And you have other people that are working on this, right? So like, um, uh, post open source, Bruce Perens is working on the, the idea of post open source. And it's, it's intended to be sort of an addendum to all of the open source licenses that says you get to pay for your software. And his, his idea is let's make a central clearing house that handles this.
And I think it's really fascinating. I'm hoping to have him on the show to talk about it. Um, but I, I think there's been just sort of a awakening in a lot of different places to this idea that there's gotta be some money that flow back into these projects because developers need to eat. And we expect open source projects to have, um, quality standards.
Of commercial offerings, but and honestly a lot of times the quality standards open source projects are higher than commercial offerings But you find out, you know after a while, that's not necessarily sustainable unless somebody's paying the bills
Alistair: Well, well, it's it's that and more right? Um, the So so let's take the I I you know, I I'm not in it.
So I I spoke To daniel recently in a couple of months ago in stockholm on this topic, right? Um, it's not it's good that he's now in a situation where he has Uh, he has a sustainable business model and he can actually be paid to work on the thing that he enjoys working on Oh, yeah, right, but what happens if he gets hit by a bus, right?
We need In principle some a project like that should have You Couple of three apprentices working on, you know, a sustaining plan for when he decides he wants to, you know, hang up his clogs and do something else. Yeah, right. And there is, if you think about this, this is sort of in the Middle Ages, they had better plans for these types of things.
Yeah. You know, the, the, the famous artists would have people in their studios who were helping them, you know, and learning to do stuff. And when they died, right, the, the guys took over and it took us a long time to even be able to figure out that some of the works of art were done by the apprentice, right?
And, and we need a system like that for, uh, You know, artisanal software, which is of the highest quality, right? I mean, it really is, right? Daniel's Daniel stuff is the highest quality. Yes. And so there are systemically. At least a thousand projects, possibly 10, 000 that should have that same level of coverage, right?
I mean, the NTP keeps coming up as the topic, right? Um, I'm pretty sure that that's the one that's the, you know, the one maintainer in, you know, um, that's exactly right. So, so you've got all these things that if they were not managed and, and what's the succession plan? Nobody's thinking about this stuff and we need to figure out how to make this happen Because otherwise something's terrible is going to happen and then we end up with these Oh, let's just throw a large amount of money at it from the linux foundation.
It's like We could Plan better than this, people, right? We could have, we don't need to have, you know, Y2K bug problems just randomly turn up on a Friday because somebody got hit by a bug. Bus, right? I mean, we can work better than this.
Rob: Yeah, and that's a problem. That's a problem just all over the place in commercial endeavors where you have one person who knows or does everything and you don't know what tomorrow's gonna bring with that one person.
Yep. Yep. Right. Well, which
Alistair: is one of the reasons that industry has said, well, this is why we don't want to use open source. But then, unbeknownst to them, from the executive perspective, their engineering teams have been, you know, borrowing the code because it got them away, got them to their finish line faster and better by doing that, right?
So they've inherited this problem, but they don't want to face the fact that they've had all this benefit and what do you do about it? Yeah.
Rob: And that really could be quite the contrary, where in a small if you have one or two people doing the development and something happens to them.
Jonathan: Yeah.
Rob: You know, you've got to start from scratch.
Whereas open source, you have people at least looking at someone who can probably step in.
Alistair: Well, yeah probably step in but let's be let's be real about this right what we would really like Is somebody people who were actually wing wing people? On a regular basis and that they were actually capable and knew how to step in right?
And and then you can avoid The sort of xz exploit problem, right? Because the the we're talking about The same thing that whoever came up with the cunning plan Of doing xz looked at the way open source works and go there's a poor maintainer over there who needs help So guess what? We'll provide him with the help and this time it was with malicious hats on right?
But why don't we look at this and go well? That's a systemic problem. We should be providing the help so that there's, there's a general, I mean, doctors, for instance, when they go on holiday, they have backup. Other doctors that you can call. My dentist says, you know, call this other person, and they have access to my records, right?
Other industries have figured these types of things out, and we haven't.
Rob: You know, in your notes, you, uh, you mentioned abandon, abandon wear and, you know, even myself, I, I, I don't do a lot of programming these days, but I used to do quite a bit, a little bit of, uh, open source and, and what I found personally, I mean, I never grew to the heights of really anybody, but I'd start to build something.
I'd put it out there in the hopes that, okay, this is going to get some attraction. Some people will come along and help me. Um, maybe I'll get some, some other contributors to it, and then I just Get tired and burned out because I'm doing it myself and it's like well Nobody's interested and I am banning it myself So I mean
Jonathan: so it's interesting to me you have some of these projects where people just that Sounds like fr routing was one of these to go back to that as an example It starts as a single person sort of a labor of love And you started throwing code together.
And, you know, Rob says he's tried this a couple of times and never, never gotten anywhere with it. But what happens when you do get to like critical mass and there are some people out there to depend upon your code? I guess the real question that so many of us have is how do you go from that single person project where nobody really cares and it's okay if you get hit by a bus to there are people out there depending upon it and we suddenly have a budget, how does a How does a project get from, from that point to the point of having a budget and having sponsors and is there, is there a blueprint out there for this?
Alistair: Well, at the moment, it's entirely luck, right? And a lot of that luck is based on, as I mentioned earlier, the circumstance of the connectivity between the producers and the consumers of the code, right? And whether they know one another. If you end up in these situations where there is Coupling between things like people are using.
So in the, the Erlang space, right? We have a web web server on, on Elixir called Phoenix, which if you just want to do live view, Phoenix stuff. It's it's excessively popular with people who want to use web services stuff, right? But if you're just doing web content, you don't really need to talk to the guys who are keeping the html Um machinery working underneath you use a bunch of features you do turn up right?
but in principle you're You're writing, you know, your web service stuff on top. So there are a lot of web servers that have been run by, uh, very large companies that are on top of the Phoenix framework stuff, but their connectivity, I would guess less than 10 percent of those users are actually reaching back into the community other than, you know, to hang around a mailing lists and figure out how to get stuff done and, you know, Bitch about stuff, but they're not actually there is no formal way of setting up that relationship um, you know, so it's it's a difficult problem and and I think This gets back to the the topic of if you do now need to verify your supply chain It's it creates a bit of a magnetic field on the businesses that they want to start to have that conversation
Jonathan: So the gpl specifically I think most licenses have this but I know the gpl does i'm familiar with its text It has a big section in there.
This code includes no warranty, you know express or written Or is express or implied it's the term it uses And so it's, it's very expressly part of the GPL that you do not get a warranty. The, I am not your supplier is basically built into the GPL. I do not take any liability for this code. Um, do we just need an addendum that some open source projects can add that sits sort of on top of the GPL that says, if you need a warranty, if you need me to be your supplier, there is going to be a cost involved and here's how we handle that.
Alistair: So that's a way of solving the problem right sort of the essentially the dual license Model and that might be the way that folks decide to go. I don't think it has to be the only way That that you could do that, right? I mean you can think about things in a very different manner of of Um, showing support and sponsorship or any of those other types of things.
So, it doesn't have to be that way, right? It could be that way, and I'm relatively neutral on that topic. Um, so, What what is obvious is that we need a better way than the one we have at the moment, right? That's clearly not working, right? And I think you know getting people to sit down and talk about What the appropriate ways of making things work would be better, right?
Um You know, I, I, the, it, it amazes me that, that we are at this particular stage that, you know, there's so much dependency on this technology from a societal perspective, and that we as communities haven't quite figured out what the appropriate, uh, ways of, of, Running this are now. I'm not saying that the conversations are not happening, right?
Because I've been on many of these calls You know with the folks in European Union spaces as well as folks in the u. s Domain, and I think things are move that the the pressure is there To reach solutions. What's not clear is that I think we as Community activists are not actually sitting down and saying okay Well, there are four or five different ways of doing this.
Well, how do we think this could work? Right. Sometimes we need to actually talk to industry itself and ask them how they want to play. Yeah
Jonathan: Oh, yeah. So I've heard, I've heard stories of like a big business wants to sponsor an open source project and they're like, but we need you to send us an invoice and the open source project goes, we don't have any kind of business things set up.
We can't even send you an invoice and the business walks away and it's just like, well, we wanted to support them, but I guess we can't. Yes.
Alistair: Yeah. So, so there is, um, there's certainly that, and this leads to these perverse situations that either a project gets over. Supported by some company that thinks it's being like beneficial and in fact, we could talk about the you know So it's it's great when those things happen, right?
but the We need something that's going to work generically across scales and works for small and medium business customer companies that are consuming software and works for larger business and and and I think tomodipierre was very free He made a very cogent point about one of the reasons that open source has become So successful is that it's a one way graph of People downstream of you do not know What's upstream from a commercial perspective?
Yeah, right So the business, you know a the vp of engineering, you know doesn't even tell The CFO that they're bringing in a, you know, a project and nobody even knows all the branches and twigs that are associated with that project when they bring it in, right? Which, uh, led to the log4j surprise for everybody, which is like, oh, but we don't have that in our, oh shit we do.
Right? So, so, so that. We need and and this is what I think is certainly happening about the supply chain software bill of materials Technology. So all the the work that's going in there about generating s bombs will allow people to identify what that back path Looks like right and then the interesting question is like, okay fine.
Now, we know what you've got in there How what's an appropriate way? You Of providing funding down that path, right? So for instance, if i'm a consumer of a project should I also be somehow trying to make a micropayment to the log4j guys or Should the project that's using the log4j thing be making a larger donation and said Where is it turtles all the way down?
Yeah, right or and and Those are the types of questions Questions that I don't think we're we're sensibly asking ourselves because the solution to many of these things is oh Let's just throw a lot of money at the log 4j guys and that problem went away and it's like well No, you just you avoided Making the connectivity work and you've just fixed.
That's a point. It's a diving catch Solution which is not very good
Jonathan: Yeah, and I know there are groups out trying to trying to work on these problems like we years ago We interviewed the guys from Tidelift And so they were kind of, they were kind of way ahead of this. Um, we need to have them back on, because I'm very curious to hear how that has gone for them, whether they've had continued success or not.
Um, so I've got to ask, I've got to ask, this is kind of a, maybe a weird question, but I think you'll understand where I'm coming from. As, as we try, open source, historically, obviously, has been lightning in a bottle. Right? It has just been wildly successful. It has made things possible that were impossible before.
It has literally changed the world. As we try to sort of solve these problems and come up with blueprints for open source projects to solve them, do you think we have a danger that we, uh, we let the lightning out of the bottle? That we change open source in such a way that it becomes less impactful than it was?
Alistair: No, I th I
Jonathan: th So,
Alistair: I think if you
There are ways that you could Over engineer it and break the system, right? But on balance, it's working. It's working incredibly well in the downstream mechanism. The thing that isn't working. Is we're not getting the nutrients back up in the upstream mechanism and we don't have, you know, these longer to, you know, who's the, who's the apprentice for the master, right?
Who, what's the replacement, the backup strategy, those types of things are not happening and they, there needs to be a structural mechanism and making that happening. And to your point, this requires. organization on behalf of the, the open source projects. So good example is that in the EF, um, we've, we're in the process at the moment of applying for a CNA.
Um, and you, I think talk to Daniel at length about the CVE problem associated with this. So, um, I had the same conversation with him as well. And, um, it was pretty, it'd been pretty obvious to me for the last 12 months that we, as a, uh, an ecosystem wanted to have. Uh, you know, uh, autonomy over CVE, um, registration stuff just to avoid some of this frivolous stuff and be able to understand what was going on.
Right? So, um, I've articulated this as running your, you know, local fire station. Right. I mean, you need to have a volunteer fire station and you, and we don't want to avoid, we want to be able to manage that process and you need to set up your resources. That means we're actually hiring somebody to, to do this full time.
It means that we also need people to support this and we need to be able to talk to our community about why a fire station is necessary. So it's an interesting problem about how. Open source projects are going to be able to manage that back office infrastructure, right? And to your point, if somebody doesn't know how to write an invoice to somebody, right?
They probably need to bulk up. So, maybe we need group collectives. Maybe we need things that look like the mini Apache Foundations. Because, I mean, the Apache Foundation doesn't want to take on, um, lots of small stuff, right? But, you know, Um, that they take on big substantive projects, and I think they're doing a great job, but we need something that manages and looks like that for smaller projects.
Fortunately, we can do that for our language ecosystem, and I think you're seeing that happening in the other language ecosystems, like the Ruby project, the Rust project, they're all focused on doing things in their particular realm. Um, you know, there's, there's a whole bunch of stuff that, that needs to understand that it needs to be even thinking like this, right?
Jonathan: Yeah. Uh, is there, is there a resource somewhere that, uh, if someone is part of an open source project and they're kind of stuck in the middle of this, they're, they feel like the project has gotten too big for its own good and they've got to make a change. Is, is there someplace that you would recommend people go to, to kind of start learning about the options or to get help?
Alistair: Ooh, that's a very good question. Um. So actually that, uh, so coming up in my future is I'm off to FOSDEM, um, at the beginning of, um, the first weekend in February. Um, there are a lot of folks out there. I mean, one of the reasons talking to you guys, I think from this perspective is, uh, Is um, a lot of my knowledge about what people think and feel at the moment comes around about listening to podcasts So we are communicating over radio um Related to you know, what what people think and care about so I don't know I mean, maybe you want to put your hand up and and say i'm certainly happy to take redirections, uh Anytime you want to send them my way, but um, yeah Not clear who's trying to work on this at a total global scale.
I think it's probably too big for one person to work on. Yes.
Jonathan: Yes. Um, I, I do know of a couple of places that are, that are sort of thinking about this and trying to help. Uh, the Open Source Collective is one of them. I believe they're based in the United States. Uh, our friend Simon Phipps helps with a couple of companies, or a couple of projects, excuse me, and trying to, to Solve some of these same problems.
Um, and there, there are people out there, but I, I, I can't help but think that it would be useful. And, uh, I'm sure people are working on this, but it would be useful to have a blueprint where. You know, maybe, maybe we publish it as its own open source project. And it just, here, here's the documentation.
Here's what we suggest you put on your GitHub repository. Um, here's the way that you, you form a company to be able to take donations or, or what have you. It just seems like there, it would be useful to have a, a blueprint document out there for people to try to bootstrap from. I'm a single person. Open source project, but people depend upon me too.
Okay. Now I'm a healthy open source project with a succession plan, all of that.
Alistair: Uh, I couldn't agree with you more. Um, I've been on a couple of calls with Simon, um, possibly earlier on this year related to the EU cyber resiliency act and, um, um, and that there are obviously folks in some of the larger foundations that are, that are worried about.
This class of problem everywhere from the Linux Foundation to the Apache Foundation to the Eclipse Foundation. Um, what's, what is missing at the moment is something that scales. Across all the domains and, and, and to a great extent, even thinks about the really hard problems of the apprentice ship and survival.
So that's my usual test question. When somebody said, Oh yes, we're thinking about these things, right. Related to the, you know, cyber resiliency act. Then I say, okay, so if you've thought about them, what's the plan for that? And everybody goes, Oh, we hadn't thought about that bit. So, I mean, the, the, the, We sort of need to have a sort of discussion about how ambitious We want this to be and how you would think about stuff, right?
I mean at one level um You know other fields of endeavor have fixed this right? So they have things like the fields medal and they have whatever it is, right? I could imagine Somewhere where you actually decided and said, okay, we view these things as systemically interesting. We're not going to give um You Say a boatload of cash to um the curl project because They might just all then goof off and you know buy a boat and you know, hang out in the stockholm harbor Um, but we will we will pay You know the apprenticeship salary for three or four years or something like that, right?
There is there needs to be something that's going to go work on that level of sustaining. Yeah um So Yeah, don't know yeah interesting stuff
Jonathan: All right, we could go on for I think another hour chatting about all of this it is super fascinating Rob Is there anything you want to ask before we before we let Alistair go?
Rob: Um, no, I don't have any final questions. I
Jonathan: want to make sure and let you get, let you get the last word in if you want to do. Uh, Alistair, thank you so much for being here. The hour is just absolutely flown by and we will have to have you back here before too long because again, there's so much more to cover here and more thoughts to be had.
Um, but for now, I just want to say thanks. Thank you for coming and talking about FR routing. And then also this, obviously your passion is about having healthy open source projects. Yep, my pleasure. Before I let you go, I do have to ask two final questions. Uh, and then I guess I'll get a third one in too.
Um, so the, the, the third one that we usually ask, is there anything that you wanted to cover that we didn't ask you about?
Alistair: I think that, that, well actually, that, that you, you, that the one shout out that I would put out there is for the, you know, the first robotics, uh, programs. So, um, uh, I'm, Whole of january is going to be lined up doing uh, both Judging as well as inspector management for the local competitions in the bay area And for those people who haven't looked into first robotics and what they're doing.
It is absolutely wonderful um you You hear so many people especially my generation, right? But maybe even your generation like to complain about the kids are today And whatever it is, and i'm sure that there are problems With some of the kids are today, but the self selecting subgroup that actually turn up to these robotic competitions are absolutely wonderful.
And so it's a pleasure to be engaged in that activity.
Jonathan: Yeah, this is things like battle bots and line following robots and just all of that?
Alistair: Yeah, they don't. So the interesting thing about first is that they don't battle it's um, There is a competition. There is a scoring. It's a bit like doubles tennis that you play with other team members and it really Uh, it's not a zero sum game there.
There there is a Point system and you're trying to beat the other team, but it's not at their At the disadvantage of the others and there's you know, highly refereed and curated type of stuff and it it's a very interesting gaming theory uh process that whoever thought that up i'd like to go and you know have dinner with them to figure out whether it Came out as whole cloth or whether it took them several years to figure out how to make this process work.
So well, but um, But yeah, it they they're always striving to do some particular thing right as opposed to beat the other robot so so it's uh That
Rob: sounds similar to uh, we have a vex robotics in in our area
Alistair: Yes, it, it, yes, same thing, yes. Very cool. Yep.
Jonathan: Alright, and then Alistair, I've got to ask, what is, and this, this may be a historical question for you, but your favorite text editor and scripting language?
Alistair: Um, so certainly, uh, text editor and stuff, I would use nano, um, and I think the argument's been made before that it's just so ubiquitous. And I think that's, that's a good answer. Certainly if I want to get something done and I'll use bash if I have to do scripting stuff. Sure. Sure.
Jonathan: All right. Well, again, thank you so much for being here and, uh, we, we sure appreciate it.
Rob: Okay. Talk to you.
Jonathan: All right. Rob, what do you think?
Rob: Well, I definitely approve of his, uh, last two, uh, answers, nano and batch. But, uh, as for, you know, the, the open source projects he's involved in and FRR, um, routing, I was really surprised how ubiquitous that was. And definitely, uh, you know, in agreement with all the, the The needs we have to fund and, and have apprenticeships for open source and all that.
Jonathan: Yeah. So he, he threw out the numbers, uh, a thousand mission critical open source projects, maybe as many as 10, 000. I think 10, 000 is a low ball estimate. Of the number of open source projects that are mission critical to various things Um, they're probably there may be 10 times that many that if any one of them were to go down it would be It would well it'd be the left pad situation right when when the guy behind the left pad Uh did javascript like library, you know all one line of it Deleted it in protest and half of the internet broke.
I mean there are so many of those even In
Rob: one large piece of software they're Yeah, maybe I'm stretching it, but I was going to say there could be a thousand different little open source projects. It's possible, depending upon
Jonathan: what language it is. If you, if you expand that and say in a single server, there's going to be at least a thousand open source projects.
You probably have at least that many installed binaries. Um, yeah, there's a, there's a lot of them out there and, uh, it is, uh, it's quite a, Problem, but also an opportunity to really think through some of these things about how do you What does it look like for a project to be healthy? And then how do you get to that point?
Definitely interesting. We're going to be talking about more of this sort of thing next week as we have uh, Matija Silke Of FOSS, well, it's about FOSS legal. We're going to be talking about free and open source software, legal issues. Um, and he is involved with a couple of different organizations where I believe open source projects can reach out to and have a legal representation, get opinions and such as needed.
And I am super interested in that because one of the projects that I'm involved with, we have legal concerns because we occasionally deal with people's personal, uh, inter. Personal, identifiable information. And, uh, that is, that is sort of a hairy issue these days. So I'm gonna try to pick his brain next week.
Not sure who's gonna be with me as a co host. I need to figure that out. Um, but we will be back next week on the 7th to talk about that. Legal issues and open source. Be a lot of fun. Uh, Rob, is there anything that you want to plug before we let everybody go?
Rob: Well, you guys can catch me on the Untitled Lenox Show with Jonathan Bennett.
Yeah, and if you want to connect with me directly, you could find my website, Robert p camp bell.com, and there'll be links there to, uh, various social medias and things and information about me.
Jonathan: Yep. Great. I appreciate you being here. All right. As Rob said, you've got my work over at twit tv, the Untitled Linux Show, and then there's also Hack a Day.
Where you can find the security column goes live every Friday morning when it's not a holiday or I'm not sick Which was a very interesting past week or two for me Um, but we're back. We're gonna be back at the grind for a while at least Um, and yeah, make sure to check that out I've also got a youtube channel where you find a smorgasbord of things talk about meshtastic talk about modular synthesis That video did well, so we're gonna have to do more of that.
Uh, that's uh, that's you can just You search for me, Jonathan Bennett, on YouTube and find that. Um, but anyway, we will, uh, we'll be back next week with more Foss and Floss goodness. We'll see you then. We appreciate everybody that listens, that watches both live and on the download, and we'll see you next week on Floss Weekly.