Episode 776 transcript

FLOSS-776

Jonathan: This is Floss Weekly, episode 756, recorded Wednesday, March 27th. DNSmask, making the internet work since 1999.

This week, Simon Phipps joins me and we talk with Simon Kelly, the creator and sole maintainer of DNSmask. It's a program you may not have heard of, but it probably runs in your router. It probably runs in your cell phone and it definitely makes the internet work. You don't want to miss learning about it.

So stay tuned.

Hey, welcome. It is time for Floss Weekly. It's a show about free, libre, and open source software. I'm Jonathan Bennett, your host, and I've got Simon Phipps with me today. Hey, Simon, welcome.

Simon Phipps: Hello. It's good to be back.

Jonathan:  It's always a pleasure to have you here. Now, today we might refer to Simon Phipps as WebMink because our guest is also Simon, Simon Kelly of DNS mask fame.

Now, Simon WebMink, this is going to get confusing. I don't know. Have you, are you familiar with DNS mask? I can, I can basically guarantee that you've used it before, because this is one of those projects that is in lots and lots of, well, routers is actually one of the big places it's at, but are you familiar with it?

Simon Phipps: I think for me it would be an overstatement but I'm running it because I, I, I, I'm hosting a bunch of services in a rack downstairs in the guest room that are running on, you know, host and, you know, host installs a DNS mask to manage it's to sit alongside it's NGINX installation so that the the name server provisioning is stable.

Down there. So I was updating the spot. I created a new domain this morning and so DNS must be configured And was thinking to myself, someone ought to explain to me exactly what's that, that's doing one day. So this is the golden opportunity to find out what it's actually doing when it says it's provisioning DNSmasq with my new domain name.

Jonathan: I am sure that we can get all of the details on that. I think I first used DNSmasq I think I first ran into it when I started working with OpenWrt, the the firmware replacement for little home routers. And DNS mask was the, the little tiny program that it ran to do, I think, DHCP and DNS which I always found odd that one of the big things that you do with DNS mask is DHCP and see, it always seemed to me like that was a weird name.

And so hopefully we will get the, the story behind the name today. That will be fun too. Well, rather than talking about him, let's talk to him. Mr. Mr. Simon Kelly. Thank you, sir, for being here. And Let's maybe maybe start with maybe let's start with the name. What what is the the story behind the name?

How did how did DNS mask come to be?

Simon Kelley: Okay, so so the how DNS mask came to be was was a classic story of scratching an itch But this was an itch that happened a very very long time ago if you could think back through the decades to the days when when We first got internet service providers and dial up internet and internet was something you could have in your home rather than in your in your workplace or in your office.

And I had a, I had a Pentium machine running, running Linux with a dial up modem plugged into the back of it. And I could dial through to a company called UK, who were one of the, the, the very first people who would do proper internet for proper geeks And that worked great, and I had Netscape Navigator, and I could surf the web as it existed at the time, and it worked fine.

And then, then things progressed, and I bought myself a, a, a laptop. 486 laptop, and I wanted to be able to connect that to the network as well, and that was, this was in the days where network cards were expensive, and I think I built you, you could run IP over parallel ports with a, something called a lap link cable, anybody remember that?

So I connected my laptop up to my, to my desktop machine, which had a connection to the, to the internet and, and configured what in those days, I never understood why it changed its name, but what's now called NAT was then in those days called IP masquerade. So, and this is, so this was at the time a very interesting new trick which allowed you to hang as many machines as you like on your, your, your sorry internet single, single address internet connection out to the world.

Some things never change. And I, and I, and I plugged, I plugged my laptop in and I started Netscape. And while my connection was up, it worked great. But when my connection was down, and you have to remember this is in the UK where we never had, you know, Unmetered even local phone calls. So every minute that your your internet connection was up cost you money and when the connection was down if I touched Netscape it would it would decide that it wanted to to Get some data from from the network.

And the first thing it would probably do is is a It's a DNS lookup, and in those days, Netscape didn't run threads or any sort of concurrency, so it would call, it would call the, the C libraries, DNS, get host by name, call, which would then block, because when it sent, The connect, it sent the UDP packet off of the DNS, the the, the DNS query.

It would make one hop as far as my main machine and then just be dropped because there was nothing going on. Whereas on my main machine, when it did that, the library would get no root to host connect error immediately. So that, so that. So that Netscape would come back with, I can't do that because I can't look this up on the DNS.

Whereas on my laptop, I just got a blank screen that it didn't even repaint the screen when I moved stuff. And that was kind of irritating. So that was the, that was the itch that I was trying to solve with this stuff. And all I wanted was, was something, was basically a proxy that would, that would, I could connect, I could point the DNS queries from my laptop to this proxy, and then it would do whatever.

It had to do to get that information from the DNS server in my ISP. And because that was running on my, on my desktop machine with the connection, it would get the no, no route to host error when the connection upstream was down and it would send a, there is no information straight back and I didn't have blank Netscape windows that didn't, that weren't repainting because they were sitting blocked in a, in a DNS request.

So that's how it started.

Jonathan: That's great. That's I love that kind of story.

Simon Kelley: So it was, it was called DNS mass partly because it was complemented using this clever new technique called IP masquerade to do connecting more than one machine to your ISP. And It's a long time ago, and I can't remember. I think this, this, this might have been applied to it afterwards, after the very first versions, but it's the, the algorithm that it uses is, is kind of similar to doing that or masquerade where you just rewrite packets, packets rather than parsing them and changing them and a DNS query that comes into DNS mask just gets sent on pretty much bit for bit.

The same as it arrived up to the upstream host, only changing the ID so that when the query comes back the, the relevant reply can get sent to the relevant host. And that means that the whole thing runs with very little memory and very little access, which in those days was very important.

Jonathan: That's a big deal.

Yeah. I'm, I'm curious. Why, why the decision to make this open source and, you know, being this many years ago? How, how are you even aware that open source was a thing? What, what did that kind of decision look like?

Simon Kelley: Well, this, this was around 1999 2000, so open source was Oh, okay. It's a This was this, you know, this was after Linux, I was running on Linux and this was after the first version of Linux or the first version, I think I was running Red Hat 4.

2 at the time. So open source was the regular thing. And I had a machine full of open source software and it seemed like a good thing to do at the time. And it wasn't that unusual.

Jonathan: Yeah. So somewhere along the line, DNS mask got picked up. It got discovered. Like how, how long did it take from you pushing source code out there somewhere till the next thing you know, it's, it's running on everybody's routers.

How did that happen?

Simon Kelley: Running on everybody's routers happened a little later. Yeah. I mean, I think that the, the the WRT54G was the big one that everybody that's, that's where open, open work gets, gets its name from. And that's, that was, that was a couple of years in, I think before that. And there was a big run on opens fully open distributions and software installs for, for all those little plastic, cheap plastic routers that came out.

But I, I. uploaded it to Fresh Meat. Remember Fresh Meat?

Simon Phipps: Those were the

Jonathan: days. Vaguely,

Simon Kelley: yeah. And, and, and, I, I can remember even, even the very first versions it was, it was publicized on Fresh Meat, but I was, I think, I think by the time it became, Even vaguely known, I'd gone from my, from the, from the dial up modem, which, which was the, the inspiration for the whole thing in the first place, to a cable modem.

So I had reasonably fast, 24 7 internet connection, and I was running a, a web server locally on, on, On a machine that was in the attic of my house. And I can remember sitting, sitting in front of a screen with my kids after making a release and watching the web server logs and being amazed at all the different places it was going to, it was quite, all of this stuff is a long time ago.

Sounds, it sounds very naive and innocent now, but it was, it was a different world. And that really meant, meant something we say, Oh, look, it's gone to somewhere in South America. And then somewhere in Europe, somewhere in Canada downloading this stuff. And it was, yeah, it was, it was good. And I think there were about three or four releases.

And then, you know, you know, it goes, it kind of did everything I needed to do. And I kind of lost interest in it. And there were a few bug reports that I, that I'd had that, that I'd not got around to doing, I think, with this. Why are you, why are you not doing this? You know, why, why are you ignoring your software?

And I think that was a, that was a kind of fork in the road. I could either say, okay, it's done for me. I'm ignoring it and going away. Or I can start to take this seriously and make it as something which, which, you know, I will support and make into serious software and the fork I took is

Jonathan: obvious. Yeah, you know, we still in, in some, in some software circles, we still have that that little tiny serotonin hit when you see someone from, and, and these days it's more like someone commenting on a YouTube video on, I am using this from South Africa.

I'm using this from wherever it's like that is on the other side of the world. I think that's still a neat thing to see that someone on the other side of the world is making use of. A piece of software that we've been involved in or they've they've gotten some benefit out of it so I think I think that's still pretty cool Do you do you have any feeling for like how many installs of dns mask there are around the world?

Simon Kelley: So there is a there's an install of dns mask in every android phone in the world. Oh my goodness. That must be hundreds of millions billions A lot that probably is that's probably the biggest one I, most, most cheap plastic home routers have it running in some form or another. I, I still occasionally send the, the, the relevant queries to Home routers in houses that I'm visiting just to see if it's running DNS mask or

Simon Phipps: not.

So I'm presumably you, you, you get a royalty through from each of each copy that's being run there, Simon.

Simon Kelley: What do you think?

Simon Phipps: Well, that does lead on to the question, you know, you sound like the like Randall Monroe should have made XKCD2347 actually say, Hey, A project some random person in County Wicklow has been thanklessly maintaining since

Simon Kelley: 1999.

Absolutely, that definitely applies to me.

Simon Phipps: So, you know, how, how are you making this, you know, much of the world's mobile internet is resting on your shoulders and you appear to be a sole maintainer for a critical component. How is that sustainable? What do you, what, how do you feel about that?

Simon Kelley: I guess there's two different questions in there.

One of which is how do I, how do I pay the grocery bills and the mortgage and, and, and those things? And the answer is that DNS mask doesn't pay all of those things, but it pays some of them. And essentially the way that normally works is if somebody turns up on the DNS mask mailing list and they have a request for a new feature.

And they clearly come from a commercial company. I normally email back to them off the list and say, it looks like you're using DNS mask to make money and you would like some work done on this stuff. Would you like to sponsor me to do this work? And two or three times a year, the answer comes back, yes.

And we make an agreement. And they, we agree. We work, work. new features need to be added that they would like for their system, and I agree that it's something that would be generally useful. In fact, it doesn't have to be. I've, I've always made it made an option that people can either pay to have new features added into the system, which go into the GPL version that everybody gets, or if they want to, keep their secret source private, then, then I will do stuff and they get it under whichever licensing terms they want.

And it gets maintained for so many months or years at their expense. Nobody has ever, ever done that. Everybody who's, who's paid me to work on DNSmask has always said, no, it's fine. Release it, you know, release it to the world. And which is partly because I charged them less to do that, but. But I think possibly mainly because everybody thinks it's a good idea and it gets They don't have to pay for it to be maintained in perpetuity if

Simon Phipps: that happens, right?

How many people are actually maintaining DNS mask? Is it just you? It you know, are you facing a torrent of of pull requests to a Git repo somewhere that you're having to wade through? Or is there a, are you part of a team of a hundred people who spread the workload out amongst you? I'm guessing I know the answer to that one as well.

Simon Kelley: I, there is a Git repo and I, I'm the only person who has committed access to that Git repo. But, but I, not all the code that gets committed to that Git repo has been written by me. Right. And there was an act, there was an active mailing list and there are people on that mailing list who fix bugs and pass me the patches or, or make changes.

Some of the Linux distribution that maintainers for DNS mask are active on that mailing list. Right. The guys at Piehole who use DNS mask actually use a fork of DNS mask have been very supportive and very useful and helped finding bugs and making patches.

Simon Phipps: I have that as well running on my network.

So that's that's two copies of

Jonathan: DNS mask. There may be more copies of DNS mask in the world than there are people.

Simon Phipps: Yes, well, there's certainly more than there are me. So, so and who owns the copyright to any of those contributions, Simon? Have you used a CLA or does DNSmasq have a distributed copyright?

It

Simon Kelley: has effectively distributed copyright. Right, so

Simon Phipps: I can't buy you? No.

Jonathan: It's kind of nice to know.

Simon Phipps: Yeah, yeah, yeah. I mean I wasn't going to offer to, because I shouldn't think that my credit card's got a limit that covers that, but Thank you. So and so how do you feel about this being, being the very slim brick under the great big tottering pile of, of mobile internet usage?

Simon Kelley: I, I, I feel about it, I feel differently about it on different days. There, there are days when I really like it and I think it's interesting and useful and it's, everybody wants to feel like they did something useful for the world in their life and this is one of the things I did that was useful for the world in my life.

That's good. But the problem is that, that, the the emails to the mailing list or the emails to me and the bug reports and the enhancements keep coming on the days when I'm interested in doing something about them or the months when I want to go and do something else. Mm hmm. That's the problem.

Simon Phipps: Right. And how is that, you know it, it, so, is it truly bus factor equals one here? Or can you afford, can you go on a cruise? and leave somebody else to look after things.

Simon Kelley: I don't have anybody else to look after things. So yes, I can go. I can. I can lose interest for a month, but then I come back to a massive backlog.

And the most demotivating thing for anything for anybody is a massive email backlog of reports or misunderstandings or whatever. So, so I have a choice. Keeping this stuff under control or walking away for a little while and then coming back to it. But then I have a huge mountain to climb in terms of not just getting over the over the backlog.

Okay, you know, back Solving, solving emails is quite easy. If you ignore them, they'll either, people will either send them again or go away. Right. But, but you have to you then have to motivate yourself to, to work on this stuff, even, even when you don't want to. And there are, there are times when stuff has to happen.

You know, this is, I, I think anybody who wants to get into, into Open source software that the one lesson that I've learned and I would tell anybody is never ever write open source software. That's going to be running as root on 100 million devices.

Simon Phipps: Yeah, don't do that. So on the subject of the

Simon Kelley: most sleep so well

Simon Phipps: on the subject of the motivational topics.

I've just spent my entire year looking at a piece of legislation called the Cyber Resilience Act. And I'm not even a, I'm not even a citizen of Europe. I, I live on airstrip one. Whereas you, you, you are a citizen of Europe. Did that have you worried, CRA, or were you leaving it, hoping that we would get it all sorted out?

Simon Kelley: I, to be honest, I don't know much about it. I'd be interested to hear. From you what I should be worrying about what I shouldn't be worrying about. In terms of being a citizen of europe, i'm not sure how that makes it Well, i'm only semi a citizen of europe. Anyway, i'm still a citizen of airstrip one, but I happen to be a brexit Refugee living on living in ireland.

So yeah,

Simon Phipps: I didn't have any irish relatives, unfortunately

Simon Kelley: Neither do I but the great advantage of of the Common travel area is you don't need any to live here. Yeah. Yeah.

Simon Phipps: I won't go into CRA now then but the, you know, the TLDR is you probably haven't got anything to worry about. But I'd be happy to talk to you about that over a coffee sometime.

Simon Kelley: Yeah. To the extent that I thought about it, I think my conclusion was along the lines of the EU seems to have done slightly insane things with tech. Yeah. Regulation before or normally the practical outcome of those things have been not, apart from everybody having to click to say they don't mind about cookies.

The practical outcome of those things seem to work better than the than, than the headlines to start with. Well,

Simon Phipps: the CRA was a pretty well, pretty smart headed move actually. It was about stopping people doing dumping cookies. smart tech on the market and then abandoning it and leaving everyone exposed to become part of a Russian botnet in the future And the the problem with it is that they overlook the fact that open source exists And initially at the beginning of 2023 they exposed people like you to strict liability over defects in the software that was embedded in routers So if the last time you looked was march 2023, you'd probably be very worried indeed You But the way that it ended up getting passed in the parliament in January 2024 was much better than that because we put a load of exceptions for open source developers in and persuaded the authors of the bill that that was a good thing to do.

So generally speaking you're in a pretty pretty good place at the moment, but any commercial downstream you have will probably be asking you for some attestations in the near future and I recommend you charge them for

Simon Kelley: them. Yeah, I mean, it has to be said that every so often somebody does a a recursive search over the all the addresses on the Internet that are running DNS servers and, you know, and finds out what they are and what version they're running.

And if you if you Plot the version numbers of DNSmask. There's an awful lot of old code out there, and not just DNSmask, everything, but it does irritate me that, you know, people are using, you know, I do my best to fix security bugs, especially all bugs, but especially security bugs in a responsible and timely manner.

And it's kind of irritating when you then find that all of the code without those fixes in is still running and

Jonathan: Yeah, so speaking of which This would be a good time to ask. I think there is one or two Pretty nasty security vulnerabilities that just got found in DNS mask that are maybe about to get fixed Is that is that accurate if I heard understood that correctly?

Simon Kelley: so know that that the fixes have been released now, so there's there's there was a set of security holes in DNSSEC. So this is, it's not always been the case that have in the past been security, security bugs in DNS mask buffer overflows, which were. Entirely down to me and I held my hand when it was the problem.

In this case, it wasn't, in this case it wasn't my problem, it was the design of the DNSSEC protocol. Ah, yes. Pretty much every single DNS server. I see bind unbound. All of them had all we've all followed the the specification for DNS, DNS sec like good boys and all ended up with software which behaved in pretty much the same way.

If you sent it. Particularly mangled and large requests, which is that it essentially locked up in a loop and took a very, very long time to do the crypto required to validate the DNS. Yeah, that makes sense. The, if you want more detail on exactly what the problem is, is that when you're validating Resource DNS, resource records in DNSSEC you can have, you can send with, with those resource records more than one signature require each and each signature requires a key to do the validation.

And you can. Essentially send if you're using TCP to send the connections you can send thousands of signatures and thousands of keys and the Conforming software has to to try every signature with every key. So it's a it's an n squared problem and there's those That work is about doing cryptographic validation.

So it uses a fair amount of CPU. So it's basically, you can, you can make a a domain somewhere on the internet. And if you can persuade somebody to resolve that domain name through a DNS server, which is doing DNSSEC validation, you can just spike the CPU usage on that DNS server.

Jonathan: Yeah, it makes sense.

So the, I assume the solution is basically to detect that there are way too many keys referenced here. We're just not going to treat this as a proper packet, a proper request. Exactly.

Simon Kelley: No, no, no, no, no sane domain would have that many keys. The real, the best solution to that stuff is to change the specifications for a DNSSEC to make a hard limit.

On the number of validations that a conforming domain will require. That is something that will probably happen in the future. The problem is with all of these things is if you if you've ever had anything to do with fixing security holes is that it all has this was basically had to be done in complete secret, right?

Because, it's, the problem is fairly obvious when you think about it. And anybody who'd got any wind of this, who knew what they were doing, could have exploited it to bring down Google's public DNS servers, and Cloudfire's public DNS servers, and pretty much every ISP's public recursive DNS server. So it had to be, the fix had to be discussed and deployed in secret.

And it's, it's the one thing that you can't do it's secret is to bring in the standards organizations and say, please, could you change your standards? So that, you know, the best we could do was all the various implementers between was to agree on a, on a rough number, which was the maximum number you could have.

And then, and then. It

Jonathan: seems a bit reminiscent of the Kevin Mitnick DNS problem from about 20 years ago, where all of the insiders knew what the problem was and had to fix it and were trying their best to be quiet about it so that the cat wasn't let out of the bag.

One of the other interesting security stories that I've followed in the past couple of weeks is what they call loop DOS or DOS loop, which is, it's not a new problem, but it basically takes advantage of the fact that with UDP, you can spoof the source packet, and so you can actually You can trick a couple of DNS servers into responding in, in quick succession to each other's DNS requests.

And, and essentially you pull off a denial of service attack because you get two of them looping, talking to each other. I don't remember, was DNS mask one of the ones that could have that problem? Has, has a vulnerability with that name come across your desk?

Simon Kelley: The vulnerability with that name has not come across my desk.

I can't see why it wouldn't be a problem with DNS mask.

Most implementations or most installations of DNS mask limit where they will take queries from to inside the local network, which makes it a lot more difficult to do that unless there's there are two DNS servers inside the local network. But yeah, in theory, that could be a problem. I'm not sure how you would fix that.

Jonathan: Yeah, it's it was it was an interesting it was an interesting problem And I think I think it was a similar fix like you would just you would detect that this the message that this is trying To get me to send back could potentially be interpreted as a request on the other side and just not respond to it but yeah, it was it was an interesting problem.

So I've been told that you are also a Debian developer and I think, I think WebMaker might want to ask some more about this too but I'm curious of the, the story of From what I understand, it is quite the process to become a Debian developer, to start with and I'm just, I'm just curious what that what that journey looked like for you How much, did you have to fly somewhere to show someone your ID to be able to get your certificate signed?

Simon Kelley: I know I didn't have to fly anywhere specifically to do that, but I did go to a few a couple of Debian conferences and, and go to the, they have parties at, for, for key signing at, at Debian conferences, which are quite the edgy thing to do. It's a good idea to get as many signatures on your key as possible before you start drinking.

Yeah, so I became a Debian developer a long time ago, essentially because I wanted to, to get DNS mask into Debian. I did put some other packages as well. And this was before DNS mask was a thing that it was blindingly obvious to everybody should be in Debian. So It seemed, rather than trying to find an existing de and developer to do it I would package it myself and become a de and developer and maintain that that package.

I have to confess, I'm not a very active de and developer these days, , and in fact, over the last few months, I'm in the process of, of handing over much of the day-to-Day responsibility, even for the, the DNS Mass package to, to somebody else in Debbie and who is much more active than I are. I am and will maintain it with the current standards rather than being basically looking like it hasn't changed in 15 years.

Oh,

Jonathan: that's, that's interesting. That, that seems to, to sort of, I'll, I'll hand it over to Simon in just a second, but this is really fascinating. That seems to sort of improve your bus factor. I think there's I think there's an interesting conversation to be had about what it looks like trying to transition a single person development process to where you have, you do have at least two maintainers.

And so I'm curious where you've got where you've got someone in Debian to help keep DNS mask packaged. Have you, have you considered and is there kind of a path to maybe that person or someone else, you know, getting right privileges to that Git repository so that you can step away for a week and somebody is still there to answer emails and all of that?

Simon Kelley: Certainly, as far as the Debian stuff is concerned, I'm delighted that, that somebody else does now have right privileges. Privileges to the debbie and packaging part of this stuff and can handle much of that. And I have thought about trying to, to find other people who would take, who would have right access to the, to the repo and do thing, be able to do things.

I guess the reason I'm not going to further with that now is because. I like the fact that I've looked at all of the code in DNS mask seems a slightly strange thing, but, but one of the things, one of the nice things about working on code that, okay, some of it, I wrote 20 years ago, but code, which basically I've, I've, I've, Almost all of it I've written is I understand even stuff that I hadn't touched for 10 years.

I understand much quicker than anybody who's a programmer who spends their life maintaining other people's code will know that there are great frustrations in having to find how other people's code works and how to and having to do fixes, which are, which touch as few things as possible, just to limit the amount of stuff that you have to understand.

And I one of the motivations for me for DNSmask is it's the one piece of code that I can I understand pretty much everything that happens in it, and I can work on it very simply from that point of view. So my disincentive for getting other people to take over. Bits of the code is just that It would be their code and not my code and then it it would be code that I had to work harder to understand And could touch less deeply without risking Damaging things that makes sense.

Simon Phipps: Do you think you you know, we all get old People who are listening to the radio, can't see the fact that i've got gray hair You have you of course haven't simon because you're I don't know, you're 35 or something. No,

Simon Kelley: mine is just falling out.

Simon Phipps: You know, we're all getting older and this is something we've noticed in a couple of communities where I've, where I've been participating.

And you eventually reach the point where you ask yourself, well this is obviously software people are going to want going forward. What are we going to do about the fact that I don't want to go forward? Do you have an exit strategy or are you going to leave that as a problem to be solved in a decade's

Simon Kelley: time?

I don't have an exit strategy. I have spent some time thinking about an exit strategy and not come up with a good one yet

Simon Phipps: Yeah, yes, I know that feeling.

Simon Kelley: I mean, there is, in failing anything else, there's obviously an exit strategy, which is, this code is GPL, you can just stop doing stuff and eventually somebody who cares enough will fork the code and get rid of it.

Make it work, but I would like not to do that if at all

Jonathan: possible. Yeah, I think I think the danger there would be that there would not be one fork, but you would have Google's fork and you would have The open wrt fork and you would have you would have five or six different forks then and it would it would kind of splinter Into different projects whereas it's sort of nice to have all of them in one code base that everybody agrees on

Simon Phipps: Yeah, I wonder whether an organization like nlnet labs would be able to Come alongside and help out because they're doing a bunch.

They're doing bind and a bunch of other Related services.

Simon Kelley: Yeah. Yeah, maybe

Simon Phipps: All right, let's ask a different question completely

Simon Kelley: I Think in a night in an ideal world I'm not sure this will ever happen. But but I DNSmask is now nearly a quarter of a century old. It's accumulated a lot of code and a lot of design decisions that were made 20 years ago I certainly wouldn't make if I was starting from scratch.

And I think my ideal solution to the problem of DNSmask is if somebody else would start again from scratch and write a replacement. , I don't want do it , but if somebody else did it and made di the existing DS Mass code redundant,

Jonathan: the, the next time somebody comes along and says, you should have written that in rust.

You can just invite them to go ahead and do it . Precisely.

Simon Kelley: Yeah. Yeah, yeah. But co code does have a, a, have a, a lifetime.

Simon Phipps: You know, I, I hear you say that. And yet I'm on the board of the document foundation where I can demonstrate that's not true. We still, we still have code in LibreOffice that dates back to longer than DNS mask.

And when people want to use it, it doesn't go away. It's true.

Jonathan: Sure. I mean, the goodness, the LibreOffice binaries are still s office. bin, aren't they? They are indeed.

It is that, that, that particular project is, it impresses me so much that it has, and I'm sure there are places where the show, the code is crufty. I get that, but as a whole, it is come into the modern age so well that it is a very usable office suite as old as some of it is. That just, that impresses me to no end.

Simon Phipps: Well, there were a bunch of people who a decade ago now. Spent a great deal of personal time updating it, you know, doing the, the the refactoring that have been due for at least a decade by the time they started. They removed all the German comments. They removed all the code that made it run on CPM.

They, you know, they did a whole load of, of things. Modernized it, built a test suite, made it run in continuous integration. They did a whole load of necessary and, and valuable things. And so that's the reason why LibreOffice is remarkably fresh and usable now. It's because of all that effort that people put in when the Document Foundation was established.

And you know, I don't think that useful software goes away. Sometimes it's soul transmigrates into a new, into a rewrite, but something like DNS mask, All the time. It's useful. And you know, the number of copies I have running in my own house suggests that it's still useful. All the time.

It's useful. It's not going to go away.

Simon Kelley: Yeah. Yeah. But your point of the soul is a good one. I think the soul of DNS mask will be useful pretty much forever. Implementing something which, which provides that soul re implementing something which provides that soul would not be a crazy thing to do.

Yeah.

Jonathan: And I, you know, I said it very tongue in cheek earlier, but you know, maybe there will be some young developer that comes along and re does it in Rust, and Android and OpenWrt and some other groups say, well, let's use the REST version instead. Maybe, maybe it'll happen. We'll see. So I saw in your bio for one, I believe you have a PhD in computer science, which is interesting.

And I'm curious about the timing. Maybe let me ask that first. What happened first? Were you, did you have the PhD first or did you start DNSmasq first?

Simon Kelley: PhD by quite a long way. Okay, nearly 10 years. Okay. Got

Jonathan: it. Eight years. I figured that's the direction that would go, but I wanted to check. And then one of the things I saw was DNS sequence, no, DNA sequencing.

Those are two different things. Yes. DNA sequence. Although maybe they're similar in some ways, but I'm curious the story about DNA sequencing. What, what, what happened there? What did you do there?

Simon Kelley: Okay. So this was in terms of timing again, this was immediately after my PhD. I mean, my, my PhD was on something called ATM, asynchronous transfer mode networking which is, was the response of the telecommunications industry.

To the success of the internet, but it was before the internet. Well, this was well before the internet was mainstream, but the telecommunications industry saw that packet switching was they'd been circuit switching forever. And they saw the packing packet switching was the way to go. And ATM was, was.

A properly engineered packet switching network that they could implement on a wide scale, not this crazy stuff being made by Long haired guys in California Didn't have the discipline to do it all. I know we all know what happened ATM did Disappeared more or less without trace and the ARPA Internet went to went on to take over the world.

Yeah But I was working on that and then partly because because ATM was disappearing without trace at the time, and partly because I was in Cambridge doing my PhD, and, and by the time I'd finished my PhD, I had a wife and a child, and my wife had a job that she didn't want to leave. I knew that I had to stay around Cambridge rather than go anywhere else.

And at that point the A big scientific research charity called the Welcome Trust put a lot of money into into a research institution, which was just down the road from where I was living. I could cycle to work, which is great place called the Sanger Center, which is subsequently become the Welcome Trust Sanger Institute.

And the idea of this, it was it was a spin out from. Some Cambridge some Cambridge researchers in another lab in Cambridge who, who were taking DNA sequencing from something which could be done on a very small, very laborious scale into something that was fast enough and efficient enough to possibly to possibly sequence the DNA of whole large organisms.

So nice. They started as a practice run by work, by doing the DNA sequencing for a tiny millimeter long worm that lives in soil called Caenobactis elegans. And then moved on to fruit flies and various other things. But the plan all the way through was this was going to be part of the Human Genome Project and we were going to sequence the human genome.

All of the DNA, all three billion bases in a human. So I was looking for a job, and this was just down the road, and they wanted people to do computing, and I enough molecular biology knowledge to be useful. As a computer person because, which is partly because Cambridge in those days, I did, I did a, my first degree was also in Cambridge in computer science, but in those days, in the 80s computer science wasn't a sufficiently What's the right word?

Anyway, you couldn't, you couldn't do a degree in computer science in Cambridge. You had to go and do a proper degree. You had to start doing a proper degree and then you could change to computer science. Something,

Simon Phipps: do something rigorous first.

Simon Kelley: That's, that's the one. So my, so, so my, my first year in Cambridge had been doing natural sciences.

So proper science rather than computer science. And I'd, I'd done a third of my time on, in biology of cells and molecular biology, so I knew enough about biology to know, to be able to know something about DNA sequencing and this, so this job came up. And I spent the first five years writing application software for DNA sequencing databases where you could keep DNA sequence and, and everything that was known about DNA sequencing.

that particular piece of DNA sequence and display it on screens with different, you know, sort of long scrolls of A's and C's and T's and G's and strange lines that showed where genes started or where genes finished and what the genes maybe did and how they were related to

Jonathan: each other. You ended up I, I think even doing some, some Linux work and some kernel work as a, as a part of that job?

Simon Kelley: Yeah, so, so the first, the first half was, the first part of the job was, was doing this application work and the, the Sanger Center to start with, there's DNA sequencing life scale, DNA sequencing needs an awful lot of compute. You generate it, it was one of the, the very first big data applications. You generate an awful lot of data.

You know, these DNA sequencing machines, which are running 24 hours a day and they're generating vast amounts of raw data and that. goes through processing pipelines that generate, use a lot of CPU and generate even more data. And we would the Sanger Center was installing petabytes of disk when nobody else quite knew.

Jonathan: What a petabyte was.

Simon Kelley: How many, how many, yeah, how many bytes were in a petabyte? Was that a thousand gigabytes or a million gigabytes or the first iteration of that? That, that infrastructure was digital equipment, alpha, 64 bit computers, Unix computers which became compact alpha when digital went compact, picked up the remains of it.

And then as Linux became more mainstream, a decision was made. Not to buy any more commercial Unix servers and to run everything on Linux on x86. So at that point, the fact that I'd been playing with Linux for a few years and I had some sort of background in, in open source software, and I was a Debian developer and I had Some code in the Linux kernel, there's some drivers for, for wireless network cards that I happened to buy a wireless network card, assuming that it would probably work in Linux and found it didn't.

It was one of those things that used to happen in those days.

Jonathan: It still happens from time to time. It still happens from time to time.

Simon Kelley: So that put, that put me in a good position to, to, to be in a, in a group in the, in the computer systems. area of the Sanger Center that was going to be replacing all of their infrastructure with Linux on x86 and helped, that turned out to be in the end, turned out to be Debian on x86.

And that was, we spent a long time building building Debian systems. And that was, that's one of the reasons that DNSmasq does DNS and DHCP. Because routers need both of those things. And it's, it's nice to link them together. Because if you have a D, a machine that gets its address, its IP address by DHCP and tells the DHCP server what its name is, it's nice to put that information and the IP address it actually got into the DNS.

So those two are linked automatically. But DNS has also got a TFTP server and some, some code that, that does pixie booting. So network booting. And the reason all of that ended up in DNSmasq is that we spent a long time at the Sanger Center setting up infrastructure where we could plug we could plug a 19 inch rack full of a hundred blade servers into our network and press one button and they would all netboot, do an automatic install of Debian, get their IP addresses, their configured IP addresses and, and their configuration and everything.

And two hours later, a hundred machines would, would be logged into our system and be available to run. Batch jobs. Yeah.

Jonathan: No, that's that's impressive. I'm chatting with Simon in the back chat and we find it funny that you know If someone were to do this now, they wouldn't do it this way anymore they would just use Amazon AWS or something like that for for all of this compute probably wouldn't build it out themselves and It's just interesting

Simon Kelley: That's, that's an interesting question.

I, it's, I, I left the Sanger Institute 10 years ago, so I don't know what they're doing now. When I left we had Amazon accounts and we used them for some things. But

the amount of, the amount of storage we were using and the amount of compute we were using would have cost us two orders of magnitude more to buy from Amazon than it was to run a data center and put the machines in it. And I suspect that's probably still true 10 years later for the core things that you're using all the time.

Amazon is great for bursty things. loads. You don't, you don't buy a computer to do one job and then it sits forever in your, in your data center, not doing anything afterwards. You can just go and hire that computer on Amazon. But certainly for, for what Sang was doing, it's also because it's so data heavy.

Our biggest problem with all our compute was, was giving it fast access to Vast amounts of disk. And, and you could, okay. You could keep all of that data in, in, in Amazon's cloud, I guess, but you would never, you wouldn't get anything like as fast access to it from Amazon's compute, and it would take an enormously long time just to upload that into Amazon's cloud or move it to somewhere else.

When Amazon decided that they had you by the short and curly, we're going to charge you too much money.

Simon Phipps: Yeah, it's a similar situation to what we were finding out. I went to visit CERN fairly recently and they you know, they have a lot of compute capacity for very similar reasons, that they're hammering it all the time.

They've got a lot of data. They've got to move a lot of data around if they try and move it. And so they, they've also got a lot of compute capacity in there.

Simon Kelley: Yeah, so CERN's CERN is very similar to the CIO. We used to, we used to talk to them and do similar things. And for the same sort of reasons, yeah.

Amazon doesn't, you know, cloud generally doesn't work well for that stuff.

Jonathan: Alright, we are, we are getting down towards the end of the show and there's some questions that I like to ask as we wrap. One of the big ones is, and this is going to be difficult because you have to do some set math in your head.

Is there anything that you wanted to talk about that we haven't asked you about? No,

Simon Kelley: I don't think so. As opposed to things

Simon Phipps: you didn't want to talk about that we have asked.

Simon Kelley: The main thing I wanted to get over and I think we have is my advice is never write open source software that'll run on a hundred million devices as root.

Jonathan: At least don't do it by yourself. So, and then one of the other, one of the other questions I like to ask is, what, what is the, what is the strangest or the most surprising way that you've seen someone use this piece of software?

Where, where's the weirdest place that you've found DNS mask running?

Simon Kelley: That's a very difficult question to answer. I found it in all sorts of places You would expect it to be . Yeah, but no. Why? No, I don't. I. I honestly can't answer that question. I can't think of anywhere unexpected that I found it in Esmask.

I found it in all sorts of expected places.

Jonathan: Or, or anybody doing something particularly surprising with it? Like maybe, maybe the first time, or I guess if it was intentionally done doing PXE boot with it, then that wasn't particularly surprising. Yeah, sure.

Simon Kelley: The, the, actually there is one thing that surprised me and did eventually work into a, into a rewrite of DNS mask, which is DNS mask has because of it's, it's a Swiss army knife for people to solve DNS problems on small networks, essentially.

And one of the things people want to do is Change what the DNS looks like locally to them that, you know, they want to be able to look up google. com, but they also want to be able to look up the IP address of the machine on their local network, which doesn't mean anything outside. So you can, you can overlay a set of, of, of DNS records over, over the top of the global DNS that just appear that DNS shows and does that stuff, which is a perfectly sensible thing to do.

It's always been able to do that. That's useful to do. What? People did, which I didn't expect and didn't surprise me, is, is I would get people having problems because they said, I've, I've downloaded a list of every domain that serves advertisements. And put those into my DNS configuration with the IP address of all zeros or something as ad blocking.

Which is kind of a sensible, in some ways a sensible thing to do, but I had never expected people to do that in DNSmasq and to load up this overlay of the DNS with a million domains. And there was, at one point there was quite a big rewrite that was made to make that work.

Jonathan: So, so PyHole, so PyHole is essentially the thing that surprised you the most.

Simon Kelley: Well, yes, except that PyHole does it, doesn't do it, that, PyHole does, did it, and Pyler's very good, did it in a sensible manner, which is they have a fork, they maintain a fork of DNS mask, which Looks up all of those domains in a database rather than having, rather, rather than abusing this facility that was provided in DNS, I said, PyHole does it prop, did it properly from day one, so that wasn't a surprise to me, but it was people, people, just loading these enormous files full of, full of, of ad serving domains into my configuration, which I'd expected to have, you know, sort of 20 or 30 lines at the most.

Yeah, exactly.

Jonathan: Oh, that's fun. Alright, so the last two questions I've got to ask are what is your favorite text editor and scripting language?

Simon Kelley: Emacs. I've been an Emacs person for forever. I script in

Jonathan: Bash. That's fair enough. We had, we had the creator of Bash on back a couple of years ago, and I asked him if, if Bash even counted as an appropriate answer for that, and he was very offended and said, yes, of course it did.

Yeah. Yeah. Well, it's fine. All right. Thank you, sir, for being here, Mr. Simon Kelly, and we sure appreciate it. It was a lot of fun talking to you. My pleasure.

Simon Kelley: All right. Yes, sir. Great. All right, Simon, what

Jonathan: do you think? Well, I, I

Simon Phipps: wonder what that piece of software was doing in my Raspberry Pis. Now I know yeah, and I I didn't know I could do pixie boot from it as well.

That sounds exciting I'm an odd to try booting all my phones from

Jonathan: it. Yeah, or or boot your Raspberry Pis from it. It'll do that Yeah,

Simon Phipps: yeah, there's a kind of a recursive problem there though, because it's the Raspberry Pis that are running DNS bus, so

Jonathan: But you, you get, you, you boot one of them off an SD card, and then you pixie boot the other one, and then you pixie boot the first one back from that, and then you could just get rid of the SD card.

Simon Phipps: Yeah, I needed a new hobby, that's right. I know, it's, it's, it's great, you know, and so I'm fairly frequently trying to explain to people in the European Commission why their approach to dealing with cyber security attestation has got defects. And now I have another case that I can give them of something that pretty much every European citizen is depending on somehow.

That could be simply wiped off the face of the earth if they got things wrong with their regulation. Yes because you know, I'm, I, I completely agree with Simon about a piece of code that size And that with that you're that familiar with you don't want other people messing on with honestly, it's It's what Git governance was created for, you know, people give you pull requests and that leaves you a complete liberty to completely ignore them unless they're good.

And this, this is a fascinating project for those reasons. It will become part of my lexicon explaining to the good people in DigiConnect. How their decisions will have consequences. What do you think?

Jonathan: I, I find it fascinating. I did not know it ran in Android. I, I, maybe I should have. I probably should have known that.

But I did not know that every Android phone has DNS mask. I, I hope that he, before too much longer anoints and air, I suppose, you know, in, in the Linux kernel, you've got obviously Torvalds on top, but you've got, you've got Greg crow Hartman, who is the heir apparent and over at Linux, they do have the one maintainer at the top who More or less looks at all of the code before it makes it in.

I believe he will tell you he doesn't necessarily understand all of it anymore, but he at least looks at it and approves it. But they figured out that bus factor problem. If Torvalds died today, God forbid, but if Torvalds died today, Greg Currow Hartman would just step right in. They've already got that worked out.

And from things I've heard from them, they apparently have some other people in the wings as well, if something happened to Greg at the same time. They've got it figured out, how the project would go on. And I kind of think that DNS mask is an important project sort of in the same way. I, you know, I joked, but it would not surprise me if there are more DNS mask installs than there are people in the world.

And that's, that's mind boggling and sort of difficult to wrap your mind around something being that just it's everywhere.

Simon Kelley: Yeah.

Simon Phipps: Having said that you know, Simon's correct. One of the great fixes for that sort of a problem is for somebody to write it again. That's true. You know, the DNS mask now gives you a manual for where all the problems are and and what all the issues are.

And you can study that and and re implement. And he's also right that it, it is GPL. It's forkable. It's got distributed copyright. You know, it can't be forked malignly. So solutions didn't you know, with the greatest respect, it isn't on the same scale and magnitude as, as Lennox. That's true. Where the co kernel is is of deep subtlety and you know, people like Greg Cage Will, will point out to you that, that, that it is of the deepest subtlety and making changes to it as of the greatest peril.

Yeah. Having said that, and having said all the other things, I'd be very happy to introduce Simon to the folks at NLNet Labs, where there are kindred spirits, without a doubt, and they're even in the EU, so and they do have funding, and they've got people who can work on things. But, you know, I wish them the very best with it, doing a great piece of software that we've become unexpectedly dependent on.

Jonathan: Yes, I didn't say it during the interview, I meant to, but Mr. Simon Kelly, thank you, sir, for your work over the years and essentially making the internet work for everybody. You're a scholar and a gentleman. Alright, Simon, is there anything that you want to plug? I want to make sure and let you get your plugs out.

Oh, I so

Simon Phipps: want to, I so want to, but we haven't actually announced. the date for OGCAMP this year. But I can, I can give you an early peak. OGCAMP will be happening this year because I sent the, I sent the deposit on the venue last week, so I know it's happening. But it is up to this year's organizer to announce it and he hasn't done so yet, so I can't on your show.

Jonathan: I see. You, you know it, you know it's happening and you even know when, you're just not allowed to say. I can tell you

Simon Phipps: where even, but I just can't tell you. But I would strongly recommend that anybody that thinks that odd camp is a good thing, which of course is everybody sets aside some time in the middle of October for a visit to a fine British city in the centre

Jonathan: of the country somewhere.

That it sounds like fun. Anything else that you want to plug? You have a a, a Patreon, maybe?

Simon Phipps: Oh, so I've got all my own stuff, you know, patreon. com slash web mink visit web, web M dot I N K for all the links about me buy me things on Amazon if you like, but honestly the most important thing anyone could do if they enjoyed this conversation is become a member of the open source initiative because OSI needs.

It's to have a growing base of members so that we can not just fund our growing work but also demonstrate that we have support for it from the community. As we go talk to the European Commission, as my counterpart Deb Brandt goes and talks to folk in Washington, D. C. about how open source needs to be protected as they attempt to legislate foreign tech companies out of existence.

Thank you. So please, please join os i opensource.org/join and and we'll be I dunno about eternally grateful, but a great grateful for many, many days after

Jonathan: you join. Absolutely. I, you know, I have not joined Ossi. I need to look into that.

Simon Phipps: It's a very reasonably priced Mm-Hmm, . Although, Jonathan, because you are a pro, you'd probably go for the pro level at $300, but you can become a member free of charge.

There is a very economic ordinary membership that you can have at 50. And you're doing it basically because you want us to exist and survive. Yeah. But there are some benefits as well, like being able to vote in the elections and make sure that the right people become directors.

Jonathan: Yes, yes, absolutely All right.

Next week we will be back on the third and we are talking with joshua culp About asterisk. I am excited about this one. I i've been using asterisk for years now I've got some fun war stories about asterisk. So it'll be a lot of fun to talk with joshua about that don't forget you can follow my work at Hackaday.

We've got the security column, goes live on Fridays. And we've also got the Untitled Linux Show, still over at Twit. That's twit. tv slash ULS. Check that out as well. Thank you everyone for watching. Those of you that watched us live and those on the download, we sure appreciate you all being here. And hey, we will see you next week on Floss Weekly.