Episode 811 transcript

FLOSS-811

Jonathan: Hey folks, this week we talked with Lars Wickman about Elixir, a modern take on Erlang, and NURVs, a really clever way to run it with Linux on embedded devices. You don't want to miss it, so stay tuned. This is Floss Weekly, episode 811, recorded Tuesday, November 26th. NURVs, real embedded Linux. It's time for Floss Weekly.

It's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and we've got something real fun this week. I've got Lars Wickman with me, and we're going to talk about Elixir and nerves and running things on embedded devices and the Raspberry Pi. It's going to be a lot of fun.

It is just me alone in the hot seat this week, and so I'm, I'm just going to go ahead and bring Lars right on. And first off, welcome to the show, sir. Thank you very much. Yes, I appreciate you kind of stepping in, not exactly last minute, but a little late and we didn't have anybody. And so we, we avoided doing yet another round table discussion or, Oh, God forbid, even worse me doing a monologue for an entire hour.

That probably wouldn't go well. So thank you for being here. Oh, my pleasure. So let's see, where do we, where do we start? You, you are, you're responsible for something called NERVS, which I don't know what that is, I know a little bit, but I don't know what that is. And I

Lars: refuse to take responsibility, but I can't tell you who is responsible.

Okay, well, you're,

Jonathan: you're involved with something called NERVS, which is based on something called Elixir, which is maybe based on something called Erlang. And I, I sort of know what Erlang is. And so maybe let's, let's start with what, what is this elixir thing and why should people care about it?

Lars: Yeah.

So starting with Erlang is not a bad idea. So 40 years ago, the Swedish forests of Stockholm, no, I'm not going to do full story time, but Erlang was created. in Sweden by the telecom company Ericsson, who people might know from like Sony Ericsson and a few Ericsson phones. But they're a massive company that does a lot of telecom equipment, 3G, 4G, 5G, all the Gs.

No longer 2G, I think, I don't think anyone does that anymore. But back in the 80s, they were developing telecom switches. And they wanted to experiment with whether they could develop a language for Developing resilient, reliable, performant telecom equipment because it's, it was a really hard problem at the time the computers were not that fast and they needed to have sort of massive concurrency.

So many phone calls at a time. Yeah. One phone call having an oopsie should not bring down a thousand other phone calls. So a sort of error error handling and resilience. They also ship these sort of resilient redundant boxes that were physically connected. It's not like, Oh, they were network.

No, no. They, they shared a backplane essentially. So they needed to actually run it as clustered devices.

Jonathan: Yeah, so back in the 80s, telecom equipment was basically big iron, right? That's kind of the era still that we're talking about now.

Lars: It's the era of, okay, we're going to put this under, we're going to put this in the ground before we put a parking garage on it.

Jonathan: Yes, I've been in those rooms a time or two, yes. Yeah.

Lars: They also did not want to have to remove the parking garage or even go to the parking garage to update the software. Yeah. So Erlang has supported concurrency distributed computing like fault isolation or reliability, consistently low latency is kind of important for, for telecom and hot code updates since somewhere around the 80s.

It's almost 40 years old now. Wow. I just. recently interviewed on Beam Radio one of the developers that have been maintaining the virtual machine for Erlang. And he's just been doing it for like 27 years now. He didn't make it. He's just the guy who maintained

Jonathan: it. He's a real noob in the field at that point.

Lars: Yeah. So, so Erlang has been around for a long time. Yeah. It's what Like RabbitMQ was built on Erlang. WhatsApp was built on Erlang famously. So they had like 10 years of uptime before they actually had to reboot all their servers for the first time. They did hot code updates. They did all the fancy things and they had very few developers, but a very performant, massive system.

And then they sold for a lot of money apparently, but Erlang has always been sort of a bit of an open secret. It's like only deep nerd wizards. known to use it or, and sort of fought through the documentation to use it. It has not been, it has not been focused on adoption. Not in, in the way that modern languages are.

And at some point Ruby showed up like Ruby, Ruby on Rails. And in that community, Was a guy called Joseph Lim. Well called it's his name. And he, he built up a decent following in Ruby. He did a lot of cool libraries, a lot of useful stuff for the community, very active. And then he got a little bit sick of Ruby being very slow and sort of not great operationally and started exploring other things.

options. And he settled on Erlang, and then he built Elixir on top of the Erlang virtual machine. So it is a separate language, compiles to the same thing.

Jonathan: Ah, okay. And Elixir

Lars: is a modern ecosystem with modern tooling, good build tools, good dependency management, a nice web framework, database layers, all the niceties built out and built up over the last 10 years.

So it's new in the sense that Just 10 years old.

Jonathan: Yeah. And so the original Erlang is, it sort of reminds me of Java and in that it's got, it's based on a virtual machine and it's got some of these, you know, really impressive features under the hood. And so would that make a Elixir kind of like the equivalent to I know there's, I guess Kotlin is one of them, right?

Languages that are compiled to the same kind of bytecode that runs on that existing VM.

Lars: I would say it's kind of the a closure or similar? Well, Erlang and Elixir are very equivalent languages. Okay. With Java and the JVM, if you look at like, I guess Kotlin is a fair comparison in that Kotlin is not a very different language from language from Java, aside from trying to modernize and make things nicer.

Erlang is a functional programming language, so it's functional programming, but it's the functional programming because they wanted the massive concurrency and like, so they, they do, they spawn lightweight processes and communicate via message passing, a bunch of that stuff. And then it really helps if your data is immutable.

So. They chose, they ended up doing functional programming as a consequence of what they were trying to solve for. So Elixir is also a functional programming language. Where I would say the JVM is a fairly general purpose virtual machine as far as I know. I don't spend a lot of time in, with it. I mean, object oriented programming is a very general, Right.

Paradigm. Right. It's not specific in any real sense, but the Erlang virtual machine has, has these primitives of like spawning processes and sending messages. Those are things inside of the machine code, well, the virtual machine's machine code. So it's like, it's a fairly high level of abstraction for, for a system language like that.

And Erlang was built. To build systems. It's like, yeah, you have an OS, but you want to find out if you have disk space. Well, you can just ask the airline system. You, you can stop and start parts of the system. You can inspect the running system at runtime. You can of course, update the system at runtime.

That's the whole hot code updates part. And it had preemptive multitasking from very early on. So it's, it's very much a language and a tool for building systems.

Jonathan: Yeah. And so is Elixir kind of focused on that same thing? And then I guess along with that Elixir Erlang, do they have object oriented stuff added onto them or are they strictly functional?

Lars: They, they are. The least strict, strictly functional language you can run into. So they are high level dynamic languages. So it's not static typing and it's not like Haskell or Elm. And some people are very upset about that. Some people are very happy about that. I come from like PHP and on into Python and then I went to Elixir.

So high level and dynamic has been my tool for the longest time. Sure. So didn't bother me. The FP part was kind of weird to get used to, but it also clarifies a lot of stuff. You don't get this very messy thing where you have classes and objects that interact in weird ways. It's like, no, no, the data you have in front of you is all the data you can deal with and have to deal with.

They're, they're not pure in the sense of the purest functional programming languages, because you have, you're allowed to do a bunch of side effects. You can do logging, you can do IO, you can. Do a bunch of things and you don't have to involve any monads at all. So,

Jonathan: okay. So what, what are people primarily using Elixir for these days?

What's the, what's kind of the problem set somebody today would have that you would say, Hey, Elixir would be great for that.

Lars: A lot of people pick it up to do something like messaging related, like chat or real timey stuff, because it is very good at. sort of soft real time. But it is also just generally good.

So Erlang was built to run services. Services turned out to be like the entire world. Yeah. It's like, okay. Yeah. Define services. But Elixir comparing Elixir to Erlang. Elixir has much more clearly built out like the web toolkit. So the Phoenix web framework, very nice to work with for building APIs and WebSockety stuff and all of that.

And then they built Phoenix live view, which is like, honestly, the best pitch for it is you probably don't have to write very much JavaScript.

Jonathan: I can see that being a winning pitch for, for some people at least.

Lars: Yeah. So it is a server side driven Interactive you, it's a interactive UI framework, but you drive your components and your templates and all that, and keep the state on the server. And you just ship optimized diffs over to, to update the browser side.

So there's, there's a whole thing for like. Very small teams can build fairly ambitious web applications using, using that tool set, but it also plugs together with all the other cool stuff like Erlang comes with clustering out of the box. So if you shove like Elixir onto fly. io that has this. Good private networking across geographic regions.

Like all your machines can just talk to each other and you can do RPC calls across the, across the cluster transparently, you can pass messages around the cluster transparently. It really brings a whole paradigm with it and often cuts out a lot of tools that you end up needing. So if you've done Python web development, for example, that there's like a counter That is just like, what's your time to Redis?

Like, how soon will you introduce Redis to cache something or do a janky queue of some sort, or will you bring in Celery for the queue? Celery has always scared me. It's so strange. But. But I don't need it in Elixir generally. And there are good like queuing libraries and all that tooling if you need it.

But for some very simple queuing needs, it's like, it's almost like you have a small microservice architecture inside of your application. Because you can do multiple things at once and they generally don't impact each other very negatively.

Jonathan: How, how big is the sort of the community, the user base for Elixir these days?

Is it still fairly niche?

Lars: It's very hard to say because, well, ElixirConf EU polled like, I think 800 people, total 650 or 700 in person this year in the EU. So it's not tiny but it's not massive. It's not KubeCon, like, but like, I've worked with Elixir professionally full time for six years now, I think self employed and I keep finding work.

I'm a fairly loud person in the Elixir ecosystem, which helps. Yeah, but yes that

Jonathan: does help a lot. What what is the this was this was not on the the list of topics that We were going to talk about ahead of time, but i'm very curious what's like the security status of it? Like is there are there bug bounties?

Are there people that are? Sort of looking specifically at the, the security of the system, finding vulnerabilities and fixing them. What, what does that look like?

Lars: So for Erlang I don't think there are any bounty programs, not that I've heard, but Erlang is maintained by Ericsson. It still is. Okay. Yes, and it is heavily used by WhatsApp.

Okay. And it is heavily used by Cisco. And also backs a lot of big game company tools. So you'll find Blizzard uses Erlang. League of Legends has used it. It's a bunch of Elixir discord runs on Elixir. So there are large companies that are using it and they take their security reasonably seriously.

Then there's the Erlang ecosystem foundation, which is kind of the umbrella foundation for, for the whole ecosystem. And they do have a security working group. So when things come up, like right now, there's a bunch of conversation around supply chain. Because EU legislation is going to force, force a lot of people's hands in that regard.

So there's a bit of that. And I think the EF recently turned into its own. So if you file a CVE for Elixir for, for Erlang, it's Ericsson but for, for Elixir and a few other things, it should now end up in, in the EF. So. The, the sort of underpinnings for the security work is there. I think it's still fairly early if you compare to the big ones, but Somebody is thinking about it,

Jonathan: at least.

Yeah, I wouldn't say

Lars: I know what the security story of Node. js is either frankly.

Jonathan: Oh yeah, they're, they're, they're working on it.

Lars: That doesn't sound good. Not reassuring at all. So, okay.

Jonathan: That's actually an interesting question because let's see. So does, does Elixir kind of have its own library repository?

So like when you, when you get it, when you start talking about the, and this is what I mean by that. When you start talking about the things, the, the the dependency tree for something like node js, you get into this, this. problem space where just anybody can throw a library up. And so one of the security problems that they have is people put malicious libraries on the you know, on that repository.

Is that a problem that exists in the Elixir space?

Lars: I have not seen or heard people note that there are like malicious libraries coming on to hex but hex. pm very nice repository has a lot of good things in it. Of course, like we can have the same problem there. There's nothing stopping people from publishing.

We don't have the culture of. Sort of proliferating packages in the way the node ecosystem is i've i've run into a package and gone. Huh. This is seven years without updates. I wonder if it still works Cause it seems like it might solve my problem and it just works. Like the, the Elixir language was announced to be generally done.

What was that? Like Elixir 1. 7 or something? It's a ton of versions back. It was probably like three or four years ago and it was, it was, Just met with a standing ovations because it's not a cultural or ecosystem that wants a ton of churn. There are libraries that are sort of leading edge, for example, live you changes fairly frequently because that's still heavily under development, but the language itself.

Changes very little and they have no plans for a 2. 0 because they don't see the need interesting.

Jonathan: So there's no You guys don't have the problem of left pad where you have a whole bunch of you know Libraries, that's just one line of javascript No, that's

Lars: fairly unusual. I think the the general cultural consensus is you can vendor that It's like come on.

Yeah, you just you just add that that's a snippet not a library But of course like there are small libraries I'm sure, but I don't know that they get a lot of use.

Jonathan: Yeah, yeah, that's fair. Okay, so that seems like a reasonable overview of Elixir. We could spend more time on it, I'm sure. But we have other things to talk about.

What, what is what's Nerves?

Lars: Yeah. So when I got into Elixir, I got in, like, I poked around with the web framework and thought, this is cool. But then I saw a talk by Frank Hunleth, who created NERVS. Okay. And he showed putting together some Elixir application and. Compiling it and then burning the firmware to an SD card in record time, because apparently it was like 16 megs instead of two gigs.

And then starting up the device and poking away at like the IAX prompt and running web apps or whatever he was doing, I don't recall the details. And then when he wanted to change something, he just recompiled it and then did. Mix upload and mixes the general tool runner for Elixir mix upload. And it would just shove the new firmware onto the device over that network and do an AB partition switch.

And that compare that to like working with a Raspberry Pi on Raspberry Pi OS, and you'll. either be installing a bunch of stuff inside of Raspberry Pi OS that you will never ever reproduce or you keep needing to re flash Raspberry Pi OS and that takes time. Yes. Yeah, I think the, the base image for nerves has been bloated to 30 megs now, but it's still seconds to burn to an SD card.

Jonathan: And so that's, that's what, the Linux kernel, enough of a minimal system to get you booted, and then the Elixir virtual machine?

Lars: Yeah, so that's fundamentally it. The Erlang runtime system and virtual machine gets started by a tool called Erlinit. Early net runs as process one, and then there, it churns through a few pits before it hits before it hits the Erlang virtual machine, but you don't have anything like system D under nerves because the Erlang the Erlang ecosystem was built to run services and orchestrate workloads.

Interesting. And. This, this is a hearsay, or I'm taking this as second hand information, but I hear that there are challenges running systemd on a read only root file system.

Jonathan: There, I know it's possible but I can imagine that it might not have been designed for that specifically. Sure. Yeah,

Lars: so NERVs by default ships a read only root file system and then a data partition, which means like you can have the really crappy SD cards and they still won't bail on you, or and you won't churn through your eMMC.

So a lot of people take nerves for being like a DIY hobbyist thing. And it's like, Oh, it's a cool thing. You can put nerves on your, and do elixir on your raspberry pie. It was never designed for that. It was designed for industrial and professional use right out the gate. Frank has worked. in embedded in various ways for a long, long time.

And he just found that like the Erlang philosophy of like, Oh, something is failing. Let's find a known good state to re to recover from.

Jonathan: Right.

Lars: Resonated very well with his experiences of trying to make embedded devices behave themselves.

Jonathan: Can, can you run Erlang and Elixir under systemd and, and, you know, more traditional Linux stack?

Yeah, yeah, yeah, yeah.

Lars: I mean, you can, you can put it on anything, essentially. People run it on FreeBSD, people run it on, MacOS, Windows, Linux, all the things. Windows too. That's impressive.

Jonathan: Okay, so it's just when you're using it for sort of this, this embedded or the system is only doing Erlang. That's when you have the, the Er and it start as a, as process one.

Lars: Yeah, yeah, yeah. So that's an ERVs thing. And there's a lot of niceties to that as well. I'm pretty sure, yeah. You can have the, There's a mechanism called the Erlang heart that can monitor your Erlang runtime and restart it if it fails. So it's essentially a watchdog process. And then you can integrate it.

Nervs integrates that with the hardware watchdog. So if either the hard watchdog fails or the Erlang runtime fails for any reason, it knows to restart the device. And this is like stand, kind of standard embedded stuff. If you've done embedded long enough to find out that it's standard. Cause a lot of people just Like, pick up Debian and ship it.

And go, Ha! Embedded device! And that will work until it doesn't.

Jonathan: Right. So, I have to take a moment and say, Isn't it just a crazy wild world where you can put a full Linux Debian install on something and call it an embedded device? Like, we're at the point where that's a thing. That's just, that's nuts.

That is the way it works, but that's just nuts at the same time. Yeah.

Lars: Yeah. Yeah. I mean, two gigs of embedded goodness. Honestly there, there are definitely embedded devices that end up going with like two or four gigs of payload, but that's because they pulled in CUDA. If you, if you buy one of the small Nvidia devices, I would still consider them embedded devices, but they need so much supporting code that it's ludicrous.

Jonathan: Yeah,

Lars: no, but nerves. is the entire concept. Like it's the, it is the system that helps you build your underlying Linux actually I had an interesting conversation with Frank at one point that clarified something for me. He doesn't particularly call NERVs a Linux. It's not like a Linux distribution.

It's not a tool for embedded Linux. He could, he considers. Linux and implementation detail in that regard. And I think if he had his had his way, he would have probably preferred FreeBSD, but Linux gives a lot more sort of support and tooling and like, you get a lot of stuff if you pick Linux. So I think he picked Linux for that reason.

But essentially, NURBS tries to paper over a lot of that. We use the network stack because the network stack is kind of proven. We use a lot of the kernel. facilities because they're useful and good, but You can run a lot of your device from Elixir land, which means you get this message passing. You get like various parts of your system like, Oh, we have this sensor and that needs to inform the backlight.

If you have an ambient light sensor, for example, and it needs to inform some part of the system to tell the backlight to actually bump it up a little bit, because we're in broad daylight, that type of internal communication needs to happen over something.

Jonathan: Yeah.

Lars: And Erlang already has message passing and pub sub and like broadcasting and all that stuff.

So it's very easy to build out this stuff with within an Erlang system. So everything that you can do within The Erlang system is very nice to deal with, but you can pop off like, Oh, we need to start a Python process here. We need to pull in all of like Podman and run a Docker container. Like it's doable.

But the more you can run in Elixir, the more niceties you get, essentially.

Jonathan: Yeah. So, well, this is interesting. Is this, what is, what is nerves primarily targeted at then? Is it for the raspberry pi or did that just kind of come about by accident?

Lars: If you look at the supported systems, it's going to look like it only supports the Raspberry Pi.

Okay. But that's because there are so many Raspberry Pis and nerves, like the base systems we use in nerves are very specific. That's why they can be small. And they, once you put one into production, you can get to know it and know what's in it and make it yours. So it's prefers explicit systems. So that's why we end up having the Raspberry Pi 0, 1, 2 0, 2, W.

Jonathan: I'm very familiar with this. 3a, 4

Lars: and 5. Yeah. And also I think we have kiosk systems that pack in like a whole browser shower thingy. Essentially small compositor and cog, which is a nice embeddable browser. And those are also for the Raspberry Pi, mostly as sort of getting started systems.

But then there's also the BeagleBone Black. We support a bunch of different systems. OSD32, MP1. I, there's some IMX board in there that I don't know off the top of my head. There are also non sort of non officially supported ones. I have a couple of devices from TI's like Sitara processors. AM62 series boards.

And there are contributors that have developed those for, for their clients and made them publicly available. So that's kind of base systems and the fun part is of course If you have an application and you want to put it on a bunch of various Raspberry Pis, if your application doesn't rely on anything too weird, you can just pop it on a variety of them.

Like the, the system part is pre built and separate from your, from your application. You just plop the application down on top.

Jonathan: It's probably the application doesn't necessarily care about even what, what architecture. So, like, one of the things I've run into with the Pi is that most of them are Arm V.

Eight, I think it is. The recent ones are, and some of the old, there's a V

Lars: seven in there somewhere. I think it's the zero in the old ones.

Jonathan: The, no, the zero is actually a V six. So like the, the, oh, yeah, yeah. So that, that tripped me up for a while. I was, I was shipping binaries that weren't working for anybody because we were compiling for armed V seven and those are actually armed V six and that is not the same thing.

Yeah. Yeah. I, I guess with, with the elixir and nerves, you don't, you don't have to care when you're writing an application because all of that gets handled on the, on the OS side.

Lars: Yeah, so like, I know in the embedded space, a lot of people are like, yeah, I'll, I'll write this and see why would I write it in anything else?

But actually I would beg people to consider the fact that there's a very good high level framework for writing C it's called Erlang. So Erlang interrupts very well with C and as a consequence Elixir interrupts very well with C and with SIG there's also good interrupt with Rust and so on as well.

Jonathan: By nature of Rust working so hard to have good interoperability with C I'm sure.

Lars: Yeah. You're catching me at a weird time when I'm very upset at a Rust based library that will not precomp cross compile cleanly. But aside from from the ONNX runtime that I'm having trouble with, generally the Rust libraries have been well behaved. Right, right.

Jonathan: So I, I, I want to know a couple of different things and it's related.

And so I'm curious what, what, like what real work people are doing with nerves. And then I'm also curious, is there a, is there kind of a place where it makes sense to run nerves on x86? And it, it kind of seems like there that you might even do like. VM based microservices, right? Where do you have a virtual machine?

That's just nerves with, you know, whatever elixir application on top of it. I'm curious if people are using that out in the real world.

Lars: I know some people have taken the so we have a generic x86 system in the nurse project that you usually Like you can put it on almost any computer and it will probably boot and work.

But you, if you want a bunch of hardware support, you'll have to adapt it. Sure. But someone took that and hacked on it to get it to run on Vultr, which is a virtual machine cloud provider. So they definitely run nerves based cloud cloud instances. I've been. Curious to experiment with it, but I haven't.

Also like there's a bunch of industrial PC type stuff where you could definitely want to put nerves as a single. So the point of nerves is essentially like it should not really be running your desktop or anything, right? It might, it might work out, but it's not what it's for. It's Single purpose devices is kind of the best explanation, but essentially it's like IOT, IOT gateways, kiosks, that type of thing, where it's like this device has one job.

It might may need to do a lot of different things to do that one job, but it has roughly speaking one job.

Jonathan: Yeah. So, okay. On the other end of that spectrum, then how, how embedded can it get? You know, does this thing run on ESP32s? Does it run on any of the STM devices?

Lars: So, hypothetically, you could get it to run on one or two of those devices just because people have made Linux run on them.

Jonathan: Okay.

Lars: But you shouldn't. Fundamentally. Generally, I would say they have they're too resource constrained. Like the Erlang VM is not terribly, heavy, but like Linux prefers an MMU and a microprocessor, not a microcontroller. And we, NURFS is a thing that runs on top of Linux as it stands.

Jonathan: Has anybody, has anybody tried, and this is nuts, but has anybody tried something crazy like running, running the Elixir, the, so the Erlang VM on like free RTOS?

Lars: So there is a project called Grisp that runs that runs the Erlang VM on Artemis. And then they coupled in the FreeBSD networking stack. It's a cool project, but I haven't done anything with it. It's, it's a wild wild German. I think it's a professor. Professor. But yeah wild German engineering.

Interesting stuff. Per Stritzinger. But then there is also actually a project called AtomVM, which is a re implementation of the The Erlang VM for tiny, tiny embedded devices. So they, they managed to fit it. If the device has 400 K of Ram and they need like two megs of space, I think, for, for a small image, and then they can do a lot more if you have more than that, it's a very impressive project.

They claim it's not production ready. I think they should should experiment more in that regard, but it's very cool. I saw a talk about it in Berlin this year and it's a very cool project. Yeah. Yeah. That's neat. Not. As battle tested and mature as the virtual machine, which has been tuned for like 30 years.

So that's the trade off. I, but I would say like nerves is not particularly heavy, right? So smart is a company I, I work fair bit with him, Frank Oleth, who made nerves. Mm-Hmm. works at Smart Rent and they make like smart thermostats and ho smart home stuff for rental. rental properties. And they are probably the big, as far as we know, they're the biggest user of Nervs.

So it's around a hundred thousand devices is what I've heard. And their most used device is a smart thermostat, LEDs and stuff. It doesn't have a touchscreen and it's a 32 bit single core. All the winter processor clocked down to 325 megahertz. I think it shipped 256 megs of Ram and the U they have like a hundred makes despair.

And I think the overall image is like 30 megs on disk. Yeah.

Jonathan: I am reminded of some work that I've done with OpenWrt, which is very similarly, you know, it's an embedded Linux system. They can get down to very, very small targets. You know, back at one time, they fully supported running on boards with 4 megs of flash.

And, you know, ideally, was it megs of RAM, something like that, but just insanely small targets. And so this, this. Brings to mind an obvious question. Is anybody making routers with NURBS? Is that is that even possible? I know

Lars: some people have definitely explored it. I know a guy who does wifi somewhere somewhere in Europe, Austria, I would hazard a guess, but I don't think he has switched over.

He's still do doing open WRT, I think. And, but we heard. Like, randomly in the Elixir forum, this is, you never find out who's using an embedded framework because, like, it's just secretive industries and not quite as open generally as, as like the web world or whatever. But we found out Southwest Airlines uses NERVs for their in the flight Wi Fi.

Jonathan: Oh, cool.

Lars: On a bunch of their flights. It's like okay had no idea but of course, it's perfectly suited for networking like cisco does a bunch of networking using airline Yeah, that makes sense. It makes sense.

Jonathan: I could I could even imagine honestly I could imagine a mashup project that takes like the open wrt kernel and puts the the nerves runtime So the the whole the whole user space would be nerves and then you'd use all the kernel work The open wrt guys have done and you could probably make a really impressive little embedded device with something like that You

Lars: Yeah, and the way we build our Linux is just Buildroot.

Okay, very similar then. So, very straightforward. I don't,

Jonathan: does OpenWrt use Buildroot? I, it's, it's similar. I think it's sort of their own take on Buildroot, but I'm sure there's some cross pollination that goes on there. So what what, what license is all of this under? So if obviously if people are doing secretive IOT stuff, it's not going to be a viral GPL sort of license, at least.

No, hopefully not. That would be problematic.

Lars: I think most of NURVs is under Apache, but it's like Apache or MIT or some BSD class it's all fairly permissive licenses. So no, no real concerns for doing commercial work. Right,

Jonathan: right, right. Yeah although boy, it's interesting, you know, I'm sure you know this and like a lot of our listeners do too, but the fact that that, that Linksys used Linux and a bunch of GPL stuff is the reason that we have OpenWrt and a lot of this, right, because Linksys built the, built a router and they didn't.

Put Linux on it. And some, some smart user figured it out and said, Hey, by the way, how about you release all the sources that you're supposed to under the GPL. And to their credit, Linksys did it. And then a group of hackers hacked on it and said, well, hey, why don't we just compile this ourselves? And that's where open wrt came from and a lot of this stuff came from that

Lars: and that was how you fixed your blue and black little bug like Linksys router.

I used dd wrt at the time. Yep Yeah, it definitely made it work. Yes. Yes compared to the links firmware

Jonathan: and all people did some crazy stuff with that too Like what what if we desoldered the leds and put a usb port there instead and just bit bang the usb? And oh, it was crazy people did some crazy stuff.

Very impressive. Okay. So now what about what about nerves hub? What is is that sort of a repository for for packages or what's up with nerves hub? You

Lars: Yeah. So nerves hub and here's like disclosure. I'm trying to start a business called nerves cloud hosting nerves hub. Okay. But but I'm also one of the maintainers of nerves hub and I'm in the nerves core team since a little while back.

But yeah, nerves hub is an over the air update service, completely open source project for firmware updates. So you essentially set up your device. We have a, like the lightweight developer offering of like, yeah, you can just set a shared key on your device and can connect, but the good way to do it is of course, with some, some little device certificate chip.

So. There's a good library for the attack 6 0 8, which is a nice little cheap chip that everyone has seen at this point. But you can also do it with a TP except it

Jonathan: doesn't support the 2 5, 5 1 9 curves. Sorry, I, I have been, I have been fighting. There are so many upsetting

Lars: things about that. I have fighting many upsetting exact

Jonathan: problem for days now, but anyway, continue.

I'm sorry.

Lars: Yeah, . But some, something it does do is that it will, do a TLS handshakes with a reasonably private device certificate, which means we can use that for authentication and letting, and then letting the device connect and we can know which device gets which firmware that type of thing.

So fundamentally it's like Nervs Hub is a whole suite of things. It's like, there's some CLI tools for when you built your firmware, if you want to sign it and upload it. You can only use signed firmware because like this, this is not hobbyist grade. It's production grade. It's being used with real devices.

And you actually want to be sure that someone that the right, someone is sending that firmware. Yes. And. All of this is built on top of a small tool that Frank built not in Elixir, and it can be used for, for many other things, but it's called fwup. And one of the cool things it can do is that it can stream writes from like one of these firmware files that it packages.

It can stream the writes essentially like DD would. But straight to the partition you want to update. So NERVs by default ships A and B partitions for switching back and forth and being able to revert a bad firmware. Yeah. Resiliency. It keeps coming

Jonathan: back to resiliency IoT, right? You, when you're doing IoT devices, you never want to have to tell somebody, All right, take the device apart.

Connect to the serial port and then, yeah, just, it's just like, yeah,

Lars: we're going to have to send a technician out to 10, 000 devices across the U S like, no, thanks. I also know a company that does nerves that have power meters across Africa. Getting to one of these devices might take a week. Yeah. Yeah.

Yeah. So important to be able to revert and also important to be able to operate under. Non ideal network conditions. And it's also kind of important to not use too much memory or too much disk space when you're dealing with IoT devices. So what FW up will do by default is it will stream the update.

and only use a fixed amount of memory and no disk while it is fetching, fetching the update. And it will just write them at the pace it can get from the network and like whatever throughput bottlenecks it might have. It will write that to the new partition. When it's done, it will do the switch over, but that means it can sip through a straw if that's what it has, or it can do.

Do it fairly quickly in many cases.

Jonathan: And I assume with, with checksumming and all that built in to make sure that you don't Yeah, yeah, yeah. Flash bad code.

Lars: Yeah. There's a bunch of Blake checksums in there and all that stuff. Yeah, so FW up is kind of the underpinning there. But then of course, Nerf's hub is built with Elixir because Elixir is very good at building scalable, fault tolerant web services.

So You want to run a couple of hundred thousand web sockets. That's not really a problem. And actually there was a blog post very early in the Lexer's history. Excuse me. Where Joseph Lim and Chris McCord, so Elixir's creator and Phoenix creator, dear, my throat is just seizing up. That's fine.

They wanted to see how far they could push the web socket implementation, Phoenix channels. They ran out of big beefy boxes to try it on. But essentially they, I think they scaled it on one machine to 2 million. Oh wow. Active chattering web socket connections. Wow. Without too much issue.

Jonathan: That's impressive.

Lars: And that's kind of the, the scale, like the scalability we're building on top of, and so there's web ui, there's ACL I, there's APIs, and then there's a lot of shipping firmware.

Jonathan: So Nerves hub, is it, is it available to hobbyists if I've only got a couple of devices. Yeah. Does it make sense for me to, to, to get on board too?

Lars: So you could set it up. It's a Docker container. It's very straightforward to set up. Okay. Or you could sign up for Nerves Cloud and use like our hobbyist plan, which is essentially 10 devices. Do whatever you. Because like the real customers in this case is not a hobbyist, it's, it's someone who has a couple of thousand devices.

At least, yeah.

Jonathan: I, I imagine that hobbyist here is also nice to be able to give those potential customers a way to do a test run. Yeah, for sure. Works out pretty well. Cool.

Lars: Yeah, so it's a, it's a clear differentiator. Like, There are many over the air update tools. There are also a lot of assorted, sort of, Embedded frameworks, but most of them don't come with any opinions.

It's like, if you lock, look at Yocto Linux, like what is the opinions of Yocto? It doesn't have any, it's supposed to be general. Buildroot is slightly more opinionated about how to design the whole thing, but it's fundamentally also very open ended. It's just like, here, we will help you get a Linux and some stuff on it.

But nerves is very much like a whole package. It's, yeah, it's actually a good starting point for embedded. I didn't come from embedded, but I'm learning a lot about how to do embedded devices based off of the fact that there's a lot of experience written into the framework and written into the tooling.

Jonathan: Right. So what does it look like if somebody wants to bring something from outside of the Elixir world onto a NERVs device? So the example that you give in our notes here is Python, but you know, there's a, there's even a world outside of Python that, you know, people have stuff that they want to do. So like, can I run a Java virtual machine alongside the Erlang virtual machine on a NERVs device?

You know, can I do Python? Can I do Perl, Flutter, you know, whatever, is there is it possible to run that stuff alongside NURBS?

Lars: Yeah, so you have a bunch of different And some things are easier to run than others. Like, I don't know how much work it would be to get a decent JVM into a Buildroot project.

I've never tried. I don't particularly want to try.

Jonathan: If somebody pays you enough money, I'm sure you'd be glad to, right? I would do it. Yeah. That's the way that works.

Lars: But there are also some blessed paths. Like if you want to do. If you want to integrate Rust, there's a few nice ways to do it. Sure. As I mentioned, C and C interop, pretty straightforward.

Then there's cursed things, like there's a community member called named Coco, and she has implemented an embedded Python, like you run Python as a NIF under Erlang. So essentially you have Python in your Erlang virtual machine. And that's cursed, but it also seems to work pretty well. It's very early days for that particular experiment, but you have a lot of options.

You can also just I mean, if you have Go, Go code, Go makes very nice little binaries. You plop that in just as a file and you can start it easy. But then of course, if you actually want a proper Python. In your system, Buildroot has that, like you can just, you'll have to modify your system usually, is the answer to running more complex things.

Jonathan: It sounds kind of like anything that you can do in Buildroot, you can actually make nerves do as well. Yeah, pretty much. And honestly, again, I have experience with OpenWrt's flavor of Buildroot, but I'm pretty sure it is similar. It's not that difficult to add stuff to Buildroot. You just, you know, you write, you go find somebody else that did something similar, you take their makefile, you modify it a bit, and hey, Bob's your uncle, you got it working.

Yeah, defconfig,

Lars: defconfig, defconfig. Yeah. It's been a wild ride for me since I, like, I've run Linux since I was a teenager, but I've Not really. So for one thing, like actually engaging actively and contributing to open source was something that I started like seven years ago when I got into Elixir because the community was very good and since I've started poking around with nerves, I've had to pick up like kernel configs and def configs that I Just like, I've probably poked some of that when I was doing Slackware and my audio wasn't working, like, but not, not really, I didn't know what I was doing at all.

And now I, I do have a patch for a build root thing where like the WPA supplicant couldn't be configured the way I needed it to. Like that's, that's an interesting journey. It's like get email patches and all of that.

Jonathan: Yeah. I, it's been, it's been years ago, but I when I was messing around with OpenWrt a lot I, I found this edge case where if you were trying to do a compile from scratch, but a multi core compile, the OpenWrt builder would just fall on its face during compile.

And so I've one day, I cleared off some time and I, you know, I troubleshoot it and really dove into it turned out that because, you know, it, it, it builds GCC to be able to build GCC to be able to build the final GCC. Right? It's, it's like cross compiling. That's the way that tends to work. And it would start building GCC and then download the next copy of it and then unzip it over the copy.

It was already downloading. And that was, that was the problem. That seems rude. Well, it, it was broken. It was causing problems. Yeah, the compiler thought it was very rude. But it was one of those deals where it's like, well, once you understood the problem, oh, that was trivial to fix. You could just go in and say, well, don't actually compile GCC with with multiple threads.

And so I've got, I've got that fixed. It's been an open WRT for like a decade now. And it's, it's really, it's really fascinating to me. Like this world of open source that we're in. How easy it is for someone to, you know, if they find a problem like that, to jump in, figure it out, and then, you know, you, you email the patch, and here, you know, here's the fix.

Or, you know, these days, you can just make a PR on GitHub for a lot of these projects. And something that's always really fascinating to me is that it's so it's, it is so accessible for people to jump in and, and make a fix like that, and, and Make the world just a little bit better place, right? I love it.

Yeah, it's beautiful. It is.

Lars: It is. Now you talked about multiple cores. That's kind of one of the sweet parts of running on top of Erlang. So a lot of Embedded devices these days have a few cores to play with. And because Erlang was built to do concurrency and to be able to, like, share nothing, immutability, and also being able to run workloads in a distributed fashion, when multi core came, they were like, if we start more schedulers.

Maybe we can just use all the cores. Of course, it was more complicated than that in truth, but essentially that so by default, when you start Erlang, it will start one scheduler on a thread per core. So like your Erlang program will run your number of cores, number of threads, and then it will distribute the, these like, well, green thread type of, they're called Erlang processes.

across these and like in our most Erlang systems can manage like millions of these if you ask it to but the max parallelism is of course the number of the course you have right but But yeah, so adding more cores is kind of straightforward. So scaling up your, your cloud like, Oh, let's do 32 cores.

And it will actually just use them all. This was not my experience with Python.

Yeah.

Jonathan: Most languages, you've got to work fairly hard to be able to do that. Yeah.

Lars: Yeah, no, it's it's kind of ridiculous in many ways, but the, the probably nicest thing for, for example, in IoT for this is like, Oh, we are downloading a firmware update, but at the same time we're serving the user what they need.

And because it's like preempted multi, multitasking doesn't even show up to the user. Yeah. Because. Nothing can block the the scheduler for very long before it gets kicked to the back of the

Jonathan: line. Cool. All right. So let's say somebody listens and they go, man, Elixir and Nerve sound really good. Where's a good place to go and learn, let's just say Elixir first, so like if I want to dive into the world of Elixir, where's a good starting point?

Lars: So I think the Elixirlang website has a good starting documentation. There's also Elixir School, there's also oh, well, I don't know if exorcism. io is still around. I think they might have folded. They had a very good Elixir track. I have a, I have a good overview, I would say, of Elixir. It's not like, this is how you learn the language.

This is more, it's more, this is how the language works. And this is how the runtime works. But I have a blog series called Unpacking Elixir. And that might give a good idea. I've also given a talk that can, that's kind of a good, slightly longer overview of Elixir, LiveView and all the assorted cool things you can do with it.

But there's a lot of good resources. So Elixir, the language itself, I would start from their website. Then there's the Phoenix web framework. Of course, if you're in the web space that's probably what you want to pick up next. Also good documentation like Elixir and Phoenix. came out the gate with just really good documentation and the documentation tooling for Elixir is very good.

So the whole culture is essentially documented well because we have very nice docs and you like it when the docs are nice. So when you write a library, you better make the docs nice or you will be ashamed. You will feel shame. Yes. Yeah. So I would Just essentially dive in there. And if you want to try nerves, there's a few nice ways to do it.

There's nerves live book, which is a cool sort of interactive thing. You can put on a recipe pie and then open in the web browser and tell it to do things like you can. flip the GPIOs from, from essentially a code notebook or tell it to find your Wi Fi from a code notebook.

Jonathan: So you can, you can write, you can write Elixir code right in the browser then and get it to run kind of live.

Yep. Very cool. Yeah.

Lars: Livebook is also good while you're learning Elixir and you can run it as a desktop app. That's what it's usually Livebook is a bit of a special one. Then there's the Nervs Quickstart project, which is also like just Nervs packages. Bundled up with most of what you would need. And there's some, some decent documentation around for, for all the different parts of Nerve.

So it's just again, and like. There's a good forum, the Elixir forum. We're not typically on Stack Overflow as a community. So there's the Elixir forum and the Erlang forum and they've, they've served us very well. And then there's a Slack and a Discord of course, so. If you start poking around with nerves, dive into any of those and you'll find me right there.

Jonathan: There you go. Goodness, I don't know if I can handle programming in a language that doesn't have all the answers on Stack Overflow.

Lars: But you have all the answers on a real forum.

Jonathan: There you go. So I would be, I would be remiss if I didn't ask on the, on the Raspberry Pi does, do, does nerves have support?

You've mentioned GPIO, but there are a bunch of other sort of interfaces on the Pi. Things like SPI and I2C. You mentioned the ATEC device, right? The little crypto chip, you know, that's over I2C.

Lars: Yeah, so there are libraries for I2C, GPIO, UART, SPI. My first library that I The way I got into nerves, I think we sidetracked from that was that I was trying to get an E Ink display, the Inky from Adafruit, I think running.

from Elixir instead of from Python. And since I'd done Python, I could read the Python code and try to bash my way there. I got pretty far. Then I got a lot of help. The tricky part was like the, the spy stuff and the LUTs and like, oh, a display is not the best part to start, especially not E Ink.

They're kind of complex. Yes, they are. But it worked out. The library works. Cool. And that's it. Like that was GPIO and I2C and SPI, I think, at the same time. Yes. And that was six years ago. And like I have a couple of these like more serious boards. Someone recently ported. Well made a system for this cute little thing.

So this is the M five stack. Mm-Hmm. , which is an adorable little yeah, little device. M five. I don't have an SD card in it right now. Otherwise it would show the lo nerves logo when it booted. But there's a German company that has been working with the Seed Studio re terminal dm. Mm-Hmm. , which is a raspberry pie fundamentally, but it has a lot of nice things on top of it.

So. I've been playing around with that. So this is like Phoenix and live view for listeners. It's an industrial kiosk that you can buy. It's kind of pricey, but it's also IP certified. If you do a proper enclosure, it's like meant for industrial. I also got it working on the clockwork pie you console, which is this adorable little data pad looking thing.

And yeah, people get it running on all sorts of things. So. If you have some hardware that could run Linux some people show up with a, like an ESP 32 and get very disappointed. Yeah, if it can run Linux, generally, you could put nerves on it. It might be a challenge if you're not sort of used to the Linux bits, but generally generally, it's very doable.

Jonathan: Yeah. Yeah. Interesting. It's, it's fun too. You're working with some of the same brands and companies that we are. One of my, one of my side projects is mesh tastic, which is Oh, putting Laura radios on. Some of those things. And I've

Lars: heard of mesh tastic. You're involved in developing it?

Jonathan: Yes. I, I'm, I'm the Linux guy over there, so we now, we now run the firm.

We can run the firmware as a Linux binary, and I, I made, I made that work with real hardware, so that's my claim to fame over in that system. You now have me thinking that, hmm, we ought to see if we can get our stuff put into Buildroot, and I think we may already have a user trying to do that. So, you know, at some point in the future, maybe there'll be an Elixir front end to be able to talk to the Meshtastic binary running on your, you know, little embedded device.

I mean,

Lars: is it just a binary that does most of the radio?

Jonathan: Yes. There's, there's a, there's a couple of there's a couple of libraries that have to be there. Like lib, lib yaml C is one of them. And then there's some kernel stuff that's got to be there, but I can't imagine that would be too difficult.

And it's, it's pretty much all contained in a binary after that.

Lars: Yeah, and like, yeah, the way you do things when they're not in Buildroot is you just add your patches and you add, you make your own Buildroot packages like there's endless ways to get code into your nerf system, because that's necessary if you're doing professional work, it's not like, Oh, no, I'll have to wait.

Eight months for this to be accepted by Buildroot. I refuse to run patches.

Jonathan: Oh yeah, indeed. Indeed. Okay. So is there, is there anything that we did not get to? We, we've been going for, I think over an hour now, which is fine. Is there anything we didn't get to that you wanted to make sure and let folks know about?

Lars: Oh, there's, there's a lot there. I feel like we covered

Jonathan: a lot too, though.

So,

Lars: yeah, I mean, I need to do an honorary mention of like the membrane media framework. This is something I don't see in a lot of ecosystems. There's a company in Poland, like a consultancy that specialized in media streaming, and this, this means that. Instead of trying to make GStreamer do what you want it to do, you actually build your pipeline with Elixir data structures and like the the actor model of Erlang and do a bunch of message passing.

They're of course calling ffmpeg and libmad and whatever you need because That stuff's horribly complex, but, but you can operate all of that from Elixir. And if you've ever done like media streaming and also wanting to tell people what's going on live in a web UI, that's kind of complicated. Yes, it is.

But I very recently, well, a year ago or so, I gave a talk where I was controlling the entire slide. Show with my voice. So that's the elixir machine learning effort is also very interesting, but we, I don't think we have time for it, but membrane was my media pipeline for that. Getting my voice from the browser in processing it, sending out like.

Audio measurements that I could make a waveform from, and then shoving it into a machine learning model and getting everything live into a web view. And this is like, you just stitch these things together and everything's just message passing. Like everything follows the same paradigm. This is. This is kind of the wild thing, whether you're doing a web UI, whether you're doing a sort of a, the internals of an embedded device, whether you're talking to your database, if a thing can tell you that something is happening, like interrupts are lovely in Elixir because.

events that can trigger messages can do such nice things across your system. It's not like you have to poll anything. And that's, that's kind of the ideal. Sometimes the hardware won't let you get away from polling. Thankfully it's not that hard to do polling yet, but yeah, it's a, it's a big ecosystem with very, very particular things going on in it.

I think it's always worth a look. There's a lot of good talks about it as well. There is one particular talk that I think has flipped a lot of people to trying it, which is the soul of Erlang and Elixir by Sasha Yurich. His book Elixir in Action is also a very good starting point for people that are already experienced developers that just want to pick up the language and the concepts.

Sure.

Jonathan: Where can, where should people go to find your work in particular? And the Nervous Hub as well. Plug the URL for that.

Lars: Yeah, so go to underjord. io. So I, hopefully this will be in the show notes because Cause I have no idea what you just said. Underjord you don't, you don't speak Swedish? No.

See,

Jonathan: I read that as underjord.

Lars: Yeah, yeah, you would. The Americans do. It's fine. But yeah, I would search for, if you want to find Nerves, I would search for Nerves and Nerves Elixir, not just Nerves. Yes. It's a common word. You get a lot of neurological stuff. But you can find it. NervesHub, Nerves, it's all out

Jonathan: there.

Yeah, we'll make sure to get it in the show notes. Alright, before I let you go, I've got two final questions that I am required to ask. And that is, What is your favorite scripting language? And I suppose you can, you can talk for a moment about whether Elixir counts as a scripting language, scripting language and text editor.

Lars: Yes, Elixir is my favorite scripting language. It used to be Python, but there was actually a very important, small, small, but important facility introduced in Elixir where you can just write mix install at the top of your script and state which dependencies you wanted to pull.

Jonathan: Mm

Lars: hmm. And having a script that can fetch its own dependencies.

It's pretty cool. It's like, okay, if I have Elixir installed, the script can do the rest. And there's a good example repo of like full of sort of one, one file, do something weird with Elixir. Like start a web server or whatever that, that you can look up, but yeah, I, I generally do my scripting in either bash or elixir depending on sort of how evolved it gets.

Can you do

Jonathan: a, can you do a splat bang elixir to actually run a script?

Lars: Yeah, yeah, yeah. Cool. Userbin and elixir is the usual thing. I was like splat bang. I was like shebang is what I've heard. But I thought you were going to, into Python. And my mind was going to Python because that used to be my scripting language.

It was like splat and double splat and all that. And Python is still a mighty fine scripting language. But actually the dependency management is hell.

Jonathan: Yeah, it has problems. And then text editor.

Lars: So I like Vim or Neo Vim the most but I use Visual Studio Code the most.

Jonathan: That is a very common answer, that exact combination of things.

And yeah, I find myself in a very similar, very similar case. So,

Lars: yeah, so usually Vim mode in VS code is, is how I get the bulk of the work done.

Jonathan: Yep, that makes sense. Alright Lars Wickman. thank you so much for being here. It was absolutely a blast and I got to learn about Elixir and about Nerves, two things I knew very, very little about, and now I feel like I at least have an idea of what they're about.

So, thank you.

Lars: Oh, thank you very much for having me. It's it was a fun time and happy to come back and clarify anything. If you get the weird questions about it.

Jonathan: Yeah. Yeah. All right. And so, yeah, a lot of fun. I do want to plug a couple of things before we let everybody go. Of course, first off, we've got Hackaday.

You can find my security column. Goes live every Friday morning there. And of course, we appreciate Hackaday being the home of Floss Weekly. Over at twit. tv, there's still the untitled Linux show. We record those on Saturday afternoon. And they go live Sunday or Monday, just depends upon when things get finished up.

But you could follow my stuff there and we sure appreciate it. Thank you everyone that watched us both live and on the download and we will see you next week on Floss Weekly.