Episode 802 transcript

FLOSS-802

Jonathan: Hey folks, this week Randall joins me and we talk with Michael and Benedict about EMBA. That's the Open Source Firmware Analysis Toolkit that seems to do about everything and just about includes the kitchen sink too. You don't want to miss it, so stay tuned. This is Floss Weekly, Episode 802, recorded Wednesday, September 24th.

EMBA! Layers upon layers of Bash.

Randal: It's time for Floss Weekly, the show about free! Libre, open source software. I am your host, Randall Schwartz. Merlinisunwich. com. We're here each week! The boomers, the shakers, the bigs Wait, what? Wait, wait, Jonathan! I'm the host. You're the co host today. You're the co host today.

Jonathan: Wait a second, this is Floss Weekly and I'm the host, Jonathan Bennett.

I'm so sorry. And Randall Schwartz is back as co host today.

Randal: Oh,

Jonathan: goodness.

Randal: I wake up in the middle of the night sometimes saying that exact phrase because I said it for You know, 13 years worth of shows. So it's I really, really do want to appreciate, appreciate you bringing me back as now possibly an occasional as often as you'll let me co host. Because I do actually, I miss the show.

I've got to say that for everybody out there who's been along loyal friends. I do miss the show. What I, what I don't miss is that I now have. My five hours a week back as Jonathan well knows there's a lot of time. There's a lot of time that goes into this show every week, every, every week. And I'm happy to have most of that back now.

And the two hours a week that I will take the coast here. Much simpler than being the pro producer, show runner Jesus and host. So thank you Jonathan for doing that job and keeping the show going. I appreciate that as well. So, yeah.

Jonathan: Yeah. You know, one of my, one of my favorite things about taking over the show is that I, I get to have Doc Searls and Randall back as part of the, the rotating cadre of co-hosts, and I think that's just, that's just fun that to be able to have you guys back involved with the show.

So today's topic is EMBA and that is, it's a firmware analysis tool, I think mainly targeted at Linux firmware. This is it's, it's really an interesting thing. You know, there's a lot of, there's a lot of OpenWrt, right? Like I'm a fan of OpenWrt, but there's a lot of old versions of OpenWrt that are out there in the wild.

And it's, it's pretty scary sometimes to see like 10 year old firmware still out there shipped on devices and I kind of wonder do they have a module that just spits out This is this release of open wrt beware

Randal: Yeah, and and i'm i'm not terribly in this space So I had to do a bit of research to try to figure out how this is being used But now that you mentioned open wrt there's a recent news item actually it says that tp links Routers are now all considered caustic.

And so, because they were made in China and apparently there's been some spyware that's been traced to TP link. So everything TP link is at risk right now. So

Jonathan: yeah. Yeah, I saw that story. I'm not sure if, if I don't, I'm not sure if the statement was that they are like a compromised and malicious from the factory or that they just have zero days that allowed that allowed them to get popped in that.

I was gonna say, I think we

Randal: have some experts coming up that can answer questions kind of like that. I'm also interested in particular in reading ahead of the show. Is this primarily a white hat tool? Or can it be used as a black hat tool? And what are the implications of that? Having a tool that can do pen testing, vulnerability testing on a common piece of firmware somewhere.

So I think it's probably one of the questions I really want to, I'm curious about.

Jonathan: Well, we do have the experts. We will make sure and ask about that. I do want to mention that Hackaday is what Floss Weekly is a part of Hackaday. Hackaday is a part of SupplyFrame and SupplyFrame is actually owned by Siemens.

And I say that because the EMBA project was at least partially developed originally as a part of Siemens, or there there's at least some shared history there. So we want to make full disclosure. Now we do have Michael and Benedict with us today, and I want to say welcome to both of you.

Max: Hi guys.

Hey guys. Good to have you.

Jonathan: Good to be here. Yeah. So maybe the best place to start here is to ask each of you, how do you fit into the project? Like what are your individual roles in, in this thing?

Max: So from my, from my side, I'm currently primarily or directly with the, with the central Amber project. I'm maintaining the central Amber project, which is, let's say the, the, the scanning backend. I'm, I'm more or less the founder of the whole Ember idea of the firmware analysis idea. And we've started multiple projects from time to time.

One is Ember, the other is Embark, where Benedict is the, is the main maintainer now. So, so at the end, we can, we can split it in this way. If you, if we talk about the main Ember project, then probably with me. And if we are going to, to the, to the let's say management interface, enterprise overview and a collaboration interface, then Benedict is the guy to talk to.

Jonathan: Yeah. And so same question to you, Benedict, how do you fit into the project?

Benedikt: As Mike just said, I'm the main developer responsible for Embark right now. Embark is our enterprise solution, like taking Embark as a backend and then trying to put it into a, Nice server front end. Yeah, oh how people would need it in the business environment.

And That's

Jonathan: yeah. Okay. So probably then a question for mike. I'm, i'm super interested in this idea of packaging it But let's let's start with michael first since you kind of kick the project off and I I do want to know like what What is that tie in like what did the beginning of this look like and what is the tie in with Siemens?

What was like the first problem that you guys wanted to solve?

Max: Yeah, yeah, and so probably we we need to go a little bit in back to the future I think we start we start we started with all with all of this Let's say what was it 10 to 11 years back. I think everyone started with with iot hacking then You There was, was Binwalk Craig Hefner did a lot of nice blog posts about cool IoT hacking, cool exploits.

Everyone tried to show some, some hardware hacking stuff. And the idea was back then at Siemens that we Get away from the, from the typical black box approach on, on testing products, our own products and also third party products, to a more gray box ish penetration testing style with, with the firmware enhanced.

So, with the firmware, you would be able to understand what happened in the background. If you're attacking the device, if you're trying to find vulnerabilities you get a better idea what happens in the background and which ways probably could be successful. And this was in the, in the, in this time also we, we built built the hardware hacking lab to extract the flash storage to get access to the firmware.

Sometimes you're, you're, you're in a lucky position to go just to the vendor website, download the firmware. Sometimes the, the firmware is behind the paywall or something else. And so we started with hardware hacking, identification of UART, JTAG, extracting, flash storage, and all, all the things. And then, then we had the firmware enhanced and we, we were, we were struggling really hard because we, we had not more time to perform our penetration test.

But we had so many ideas and we want to do much more stuff now. And so we needed to start automating everything that, that is possible. So and this was more or less the, the initial idea of Ember. It is more of a, of an automation framework in, in the field of embedded product Linux penetration testing that should help the, originally that should only help the penetration tester.

Yeah, that's, that's more, more or less the, there it was, it started like, I would say 10 to 11 years or, or ago. With, with, with an idea.

Jonathan: Yeah. And, and so it's interesting to see like the number of devices that are out there that just like the firmware is Linux. And we talked, we talked briefly before the show or in the, in the intro about open WRT is that a lot of what you guys see that it's just.

Ancient versions of open WRT that just get shipped on some of these devices.

Max: Yeah, you, you, you definitely see open, open VT everywhere, but let, let's say in, in the IOT environment, you see it everywhere. If you are moving a little bit away from, from iot more to the, to the OT environment where bigger products of in for factories or for for other things are used then we we see open vrt not that often But in the iot world, it's it's quite regular definitely

Jonathan: Yeah, one of the one of the fun discoveries that I have had is a lot of you ubiquity hardware Which you know, that's sort of your your small office home office to medium size I mean they have some they have some impressive stuff like all of the access points That's just open WRT or at least it was last time.

I logged into one of them So it's it's in a lot of places I

Max: think Netgear also has a lot of OpenWrt based systems out there. I

Jonathan: think, I think Netgear was like the OG of OpenWrite. They they're the ones that first did the GPL release that got turned into it. So that's not terribly surprising. All right, let's turn to Benedikt for a minute.

And so you are, you said you're kind of the guy that, that packages the whole, the final thing together and maybe makes it a little more business friendly.

Benedikt: So let's talk, let's talk about that. Sure. So the main thing about Ember is it's built in Bash, right? So we have a combination of scripts in the beginning.

That's what Mike spoke about is we had the idea of combining stuff into something that gets automated. And once we, or let's say Mike, the Ember team was at that point where like it was working and then you had something on the CLI, right? So on the command line interface, you have something to type in and then you get something out of it.

We built an HTML export thing. So you get something that is nice and shiny and clickable. But the issue you're always facing is you have some firmware and you are a team, a team of penetration testers who are essentially find, trying to work on one thing at, as a team. So you have certain, certain things one person looks at and certain things another person looks at.

And if you do what Ember does on every computer, then you're running into, well, time constraints also. So one of the ideas was to basically centralize the whole thing, put it into a server. So make it a business application in that sense. You upload something. Amber do the work and then you get something that everyone can look at without the usual issues of CLI where everyone works on their own.

Jonathan: Yeah. So do you, do you have any idea of like how, how popular it's gotten when, now that you've, now that you've put that package together? Do, do you have any, any idea of like, how many businesses are out there using it, how many researchers make use of it? I think that's

Benedikt: a good question for Mike because he keeps track of those things.

Max: Yeah. Yeah. It, it is quite important to, to, to get a good visibility for, for an open source project at all. And so, so I try to track it a little bit. It also helps for for arguing my, my work on our work on AMBA a little bit better. And during the last two years, we, we have seen that, that AMBA gets accepted in, in a very broad way.

So, there are popping up more and more blog posts out there. So, so people are talking or writing about using AMBA. And we have seen a lot of let's say multiple research papers that have used ember in some way or at least referenced it. And it looks a little bit they like to show where ember fails.

And this is great because we can then improve Ember, we get a better idea on where Ember gets used and how the people are using it and we can improve it, so nearly from every research paper we get new ideas to improve it. And if, if, if we, we are, we are also looking some, sometimes at Twitter and other social networks, and we, we, we have a little bit the feeling as, as I would say most of the penetration tests out there that are doing IOT penetration tests from time to time are at least using Amber besides their manual work.

Just to get an, get an initial overview of the, of possible vulnerabilities to get an idea of where they can dig deeper. As Ember is automating a lot of stuff, you can run, let it run over the weekend and you can start with with a quick start on Monday morning, much better than starting from, from zero if you have let run Ember over the weekend.

You get some, some basic information about the firmware, you get an S Bomb, you get all of the things that you can, can then use for, for improving your manual tasks and yeah, so, so penetration testers are using it researchers are using it, and we also are from time to time in contact with quite big companies that are trying to establish AMBA in, in some way.

Some kind of the security processes one, one big company has established Amber into their IC development process. The next one has established Amber as a quality gate for third party components. So I would say Amber is quite solid established. Hopeful, hopefully we get Many, many more developers that or back, or back hunters so that we can improve Ember in the future much more.

Jonathan: Yeah, yeah, absolutely. Randall, I know you were curious about some things around people using it. You want to, you want to take it and ask those?

Randal: Yeah, and where I'm starting with is, so I understand the basics of pen testing. I don't know how it applies as well to embedded. How much of EMBA is unique to the notion of it being firmware and what kinds of things do you do there?

I know like for traditional pen testing, you're opening sockets, you're trying memory locations, you're trying things. You're trying to recognize known vulnerable libraries, things like that. How different is it at the embedded level and what kinds of things do you do there?

Max: So I think you, we need to differentiate a little bit.

So if we, if we are talking about penetration testing of, of firmware, I typically our projects are typically looking like we have the device. So Also, we can poke around with the device like in a typical penetration test and on the other hand we have the firmware where we can look into firmware to find things that we can verify on the device.

So on the device by itself it's quite classical. So you're looking for interfaces like web interfaces, you're proxying your traffic you try to inject commands. That are executed on the operating system. And on the, on the firmware level, we, we're doing much more static analysis then. So we can, for example think about PHP scripts.

There are, if the web server is, is organized via PHP, then Ember identifies the PHP scripts and can do static code analysis on, on these PHP scripts. And then Ember can point us to interesting areas that we can try to exploit then on the device. So at the end, it's, it's, it's every always the same.

Amber gives you, gives you an idea where something interesting could be, and then you as a, as a, the responsible tester, you're digging deeper, you're analyzing this possible issue, I would say.

Randal: Is it doing things like fuzz testing and like just throwing random data at stuff to see if it breaks? Or is that more, this is more about identification and it requires a human to maybe be trying that kind of stuff?

Max: Yeah. Fa fast testing would be really awesome, . But then then, then such a a test would take Hs. and yeah, Amber. Amber already takes quite a long time and and loves your, your, all, all your course of your, your machine . And so, so we, we are working behind the scenes on different other projects, and one, one is a fa automated faster.

But this is not something that is directly into Ember because it, it it just will take too long to, to bring you the results and I, I think this, this need to be a dedicated project where you can decide which firmware you want to, to fasten after your, your initial firmware analysis process.

Mm hmm.

Randal: And I also, my friends would kill me if I didn't ask this, Why bash and not a more capable script writer?

Max: Bash is the most beautiful language No, no, no, not even close.

Max: just kidding. Wow. Bash is so practical. Think about the beginnings, you're chaining tools together. You're starting at the command line interface, you're chaining tools, and then you're Pasting it into a script.

This, this was the beginning of Ember. And so before we thought about Ember by itself, we had a thousand line bash script. And then we, we, we structured it a little bit and it was just working. We gave it a nice, a nice and shiny modular structure and with more lines of code and it was working.

And Well, everyone said that everything will break if you, if you hit the thousand line area of your bash script and No, no we, we are maintaining it till, till today and we have now tens of thousands of lines, lines of code and, and it works really smooth. You can't, you can't really believe it. If you're, if you're researching in the internet, you will read everywhere that, no, it's not possible.

You can't maintain it. You can't do this. It's possible. It's possible. It's possible.

Benedikt: I have to say for me, it was also the first major question I had when I came into the project. So yeah, because like, I always thought like that there might be an idea to, to switch once you want to modulate and put like, put it into.

Building blocks where you, we were always talking about framework and stuff that we introduced different languages and stuff, but like, I'm, I'm always astonished. Like it's working, like we do some stuff, but

Randal: I would be astonished. It's working. So, okay. That's enough bashing bash. I just wanted to get that in there.

It was like, It's like, okay, you know, I could, I could name another four letter language that would actually be pretty useful, but I'm going to stop that for now. We talked with him just

Jonathan: last week. That's Java, right?

Randal: That's not a scripting language. It is! Go, go and

Jonathan: listen to last week's episode, J Bag, they made Java a scripting language.

Randal: Oh, no, no, just no, no, that's just, that's just, that's just, that's just, that's just That's just wrong. I'm glad I didn't watch last week's show. I'm glad I missed it so far. I will read the transcript now though to see what you're actually talking about, but I don't think I'm gonna like it.

Max: Probably we need to think about rewriting AMBA in, in Java script in English.

Yeah, yeah, yeah, right. Well,

Randal: remember now, JavaScript is not Java, not even close. Those are other ends of the spectrum. Yeah. So, so what, what are your biggest concerns now with the project? What are you working on most intensely? And where do you see this going in the future? Mm hmm.

Max: From, from an AMMO perspective we, we can, we can see that the, the world screams regarding S bombs.

So everyone needs S bomb everyone thinks about S bomb nowadays and we, we, we ha, we already create an S bomb but there is a lot of room for improvement, I would say. And this is, this is, let's say, the, the one, one of the short term goals that which we're currently working on to improve our S Bomb capabilities so that let's say we can, can position EMBAR much better for, for the near future.

Yeah. And.

Jonathan: So I, I'm curious about the the, the answer to that same question, but on the, on the other side, is the other part of the program, EMBARC? Yeah. So I'm curious about that same question, but applied to embark, like what, what is being worked on there and what's coming. So

Benedikt: the fun, the funny thing about Embark is always that it's trying to incorporate everything that we're working on into one front end, let's say.

So if we're talking about the whole. vision we have with Ember, we always talk about from finding the right firmware from getting it to the end to the fuzzer we just talked about, like that might be the end goal. Once we had all the already known vulnerabilities and everything, and then we end up with a fuzzing framework or something, which Mike could talk about if he wanted to, but In the beginning, we would have something that is taking firmware from a device, which we have some projects going on.

Basically two of them, the, the one of them would be to get firmware from a device. And the other one is finding firmware on the internet. If you don't know where to look, basically like scraping and stuff. And the basic idea behind MBug would be to put all of that into one thing.

Benedikt: Oh, that's really cool.

That is basically the end goal. That's also, as you can imagine, a lot of work to integrate

Jonathan: those things. So one of the things that I think Michael mentioned is, is when you, when you give a piece of firmware to EMBA and to the Embark to the server, it really crunches on it. Like you said, you can just, you could fire your CPUs up for the whole weekend.

I'm, I'm curious, what, what are we doing with firmware that's requiring so many CPU cycles?

Benedikt: The thing Mike said about a weekend, yeah, that's, that's the new and the, the new shiny thing that's coming or that's, that's behind the curtain. We usually talked about weeks, months. If we, if we throw data sets of firmware in there to, to find. What's working, what's working with Ember, what is, what is it, is it finding, what is it not finding?

Nowadays it's, it's gotten pretty fast even though, and the thing with Ember is, is it's just multi threading and running modules, so multiple things in parallel, so it's taking basically everything. The, the machine has everything the CPU will give you. RAM wise depends a bit on where in the, in the Ember run it is, how much it will take up, but it really takes up everything you, you, you, you throw at it, but that's also the nice thing about bash.

Jonathan: Yeah. Yeah. But what, what is actually doing that is so CPU intensive, right? Are we just or excuse me, are we just un unzipping firmware and then doing comparisons looking for if there's CVEs out there, or are we actually like trying to run all of these binaries through Ghidra to do decompilation?

Like what, what, what is it that is getting done that's so CPU intensive? Both.

Benedikt: Okay. That's fair. Because you said unzipping is, is one, one part, right? And then decompiling is. Further down the line. Right? So, okay. So those are, we are,

Max: we are doing, doing more or less everything. It's, it starts with, with extraction of the f we, we are doing some, some binary analysis.

So which, which legacy functions are used in such binaries. We are decompiling these binaries. We are then using in Gira. For, for further analysis on the, on the extracted source code, we are using then Semgrep for static analysis. Regarding, regarding the Hot Topic SBOM, we are not just reading the DBN database.

We are also extracting or reverse the SBOM from the, on a binary level. So that if there is no packaging system on, on the device or on the firmware, we, we also get the SBOM out of it. Which most of the other tools do not do, and this is, this is quite, quite intense because we need to analyze every binary, every library and compare every all of this stuff against the quite huge I would say data set of possible version identifiers that we we, we, um.

Extracted manually and we built up manually and at the end we are comparing then these, these versions that we identified with the, with the CVE database, for example, to get an overview of how many, how many let's say known vulnerabilities are available for, for the, the identified. binaries and libraries.

If you think about the firmware, a small firmware of your home router has a few hundred files. If you think about the modern firmware it's getting bigger and bigger. So we have to deal with more we have seen things that they call embedded devices, but it is a full blown Ubuntu distribution out there.

And if you throw this static analysis mechanisms like decompiling every binary, like matching, I would say seven or 800 version identifiers against every binary. This is, this is highly resource intensive and and the good thing is the more CPU cores you have in your machine and boys is automatically using, let's say, or optimizing the next scan on the number of course you have so that.

It, it squeezes out your, your, your machine as much as possible.

Jonathan: Yeah. Yeah. Makes sense. All right. I know something that some vendors are starting to do. I guess they've been doing this for a long time and that is shipping encrypted firmwares. Is that something that there's, there's any yeah. Oh yeah.

We have to, we've had to deal with that. Yeah. What, what is the, what is the approach? Does, does EMBA have any tricks to try to get into those?

Max: I, I would say that we, we are using, we are, we are using on the one hand the, the classical extraction frameworks like BINWALK or UNBLOB. We also have introduced multiple dedicated extraction functionalities for such reasons where, where some some encrypted firmware was documented or the decryption was documented somewhere in the internet and, and we have implemented it directly into EnBAR.

If it's not documented and nothing, no, none of the known extraction frameworks can handle it, then at the end you need to go back to the hardware. You need to go back to identify a debugging interface. He sold a flash and usually, usually my task is then give the box to Benedict and say, Oh, I, I, I need to get access to the firmware, please.

Yeah.

Jonathan: Do you guys

Max: offer

Jonathan: like any, any sort of consulting or work services where someone could say, I need to break into this box. I need to get the firmware off of it. Can I ship it to you and you desolder it and extract the firmware for me? Is that sort of in your wheelhouse?

Max: Yeah, but just company internal.

No, no, no, no, no, no, no, no external consulting. We're just working Siemens Energy internally and testing the stuff that is relevant in this area. But not we are not offering such a service to external.

Jonathan: Sure. And I imagine trying to do that, you would run into all sort of legal questions, right?

About, I don't, I don't actually own this piece of hardware, or I don't have all the rights to this code. And then you go in and you go start decompiling it. You. There, there are, there are questions there, right? Randall, there are questions you run into when doing penetration tests.

Randal: Well, the letters DMCA immediately popped into my head.

So that was definitely going to apply here. Yeah, you don't want to be you don't want to, without the proper chain of authority, you definitely do not want to be doing pen testing at all. It's a, it can lead to. Unfortunate consequences. I'll just put it that

Jonathan: way. Yes, and for those that don't know you can go check out randall's wikipedia page and get a little more details about What happens when you don't have?

All of your I's dotted and your T's crossed and you do a little more penetration than your boss has thought you should have.

Jonathan: For a little while, it

Randal: looked like this.

Jonathan: Yeah. Yeah. Yeah. It's not, it's

Randal: not good.

Max: All right. So what about Not a comfortable position. No, no. No, it's definitely

Randal: not comfortable.

It hurts your, hurts your wrists. Let me tell you, I do know.

Jonathan: So what about virtualization? Is, do you have any sort of a a mechanism to take the, image that you've got now, you know, now that it's taken apart and run it in something like a virtualized machine to be able to do tests on it there?

Max: I feel that the, the original idea from AMBA was always we are, we're talking about firmware.

So this means that we, we need to, or we talk about different architectures. In, in firmware, you, you do not have just the, the, the, the, the X 86 architecture. You have some, some mips where you have arm architecture, you have power pc and all, all of this crazy, crazy architecture. So you, if, if you want to run something from this you always need some something that is called an emulator.

So that. That can, can run code from one architecture in your, in your, let's say, in your analysis machine, like your Kali Linux. So you have, have the guest which is the, let's say the binary from the firmware. Let's say BusyBox binary, you want to run a BusyBox binary for, which was compiled for a MIPS architecture on, on your Kali Linux, which you're using for analyzing the firmware, which is some, some x86 architecture.

And then you need an emulator like QEMU. And Amber is doing this. In, in multiple areas. So for example, we, we, we try to run every binary in something that is called user mode emulation. To, to drop output to the command line interface to collect the output and then to analyze the output if there are some version identifiers in it.

And this is one of the approaches that we are using. And the second one is, is something that is called system emulation. And in system emulation, we are trying to bring up the whole firmware with a, with a prepared Linux kernel. We're trying to boot it up. And to give it with a, with a few tricks, something like a, a, a kickstart so that hopefully the firmware is able to, to, to set up the services set up the, the, the network interface so that we can then, then interact with the firmware.

Jonathan: I, I am, I am kind of floored, actually. I expected the answer to that to be, to be, no, we're not doing any virtualization and you guys already, you're way ahead of me. It seems like a very kitchen sink sort of deal where you've just thrown everything in. It's, it's impressive. It's not even mentioning the

Benedikt: docker by itself.

Yeah, yeah, he, you, you skipped that one, Mike. No security thing with the docker. Yeah,

Max: we, we, we have skipped so many things till now,

Jonathan: but we can

Max: dig,

Jonathan: dig

Max: quite deep.

Jonathan: So maybe, maybe this is kind of what you're talking about there, but in doing things like emulating the firmware and all of that, I would, I would honestly have, have to have the consideration, like, What if there is something malicious baked into that firmware?

Are you giving that malicious code a chance to run on your infrastructure? Like, do you have any safeguards against You know, something nasty inside the firmware. Is it, is it all properly sandboxed? I guess is the way to ask that.

Benedikt: Properly? Wow. That's, that's, that's the good question. Yeah. Is it properly sandboxed?

I mean, it's been an ongoing theme, I think, for, for multiple years that we've tried to safeguard. running code that like, that's like malicious because the doc environment for within Ember needs permissions. So we're still trying to, to really get behind all the ways it could still get out. I think we've, we're doing a pretty decent job, but in the end it's, it's definitely like big disclaimer, do not run something if you know there's something.

In there maliciously targeting your host. I mean, the thing we always do is like we run it within virtual environments. It has a Docker environment. And then the thing actually running is inside a QEMU, but you know.

Jonathan: Layers, layers upon layers, that's something malicious would have to get out of. So I assume, I

Randal: know why it's slow now.

Max: That's the reason we need so many cores for every security layer.

Jonathan: That's, that's not a joke. Goodness. And I assume there's things you're doing inside of there that wouldn't work inside like a rootless pod, man. It's got, it's got to be Docker with root.

Max: Yeah, yeah. We're, we're doing a lot of let's say we're doing some things that we need, where we need root access.

We have thought about Portman now for years, but at the end of this, it is a time issue. We need to test everything. We need to test a lot of use cases, find out what, what is working. In, in this huge code base within Portman, what, what is failing and let's say every, every fail is probably some, at least some little research project on how can we, how important is it for Amber?

Or for, for the testers out there, can, can we just strip it out or can we bypass it somehow? And this, this costs quite a, a lot of time. And so we decided Amber is a research project from this perspective. We, we have multiple layers of, of protection, but at the end at the end, an attacker can analyze, can, can check out the code, can find the bugs and can exploit them if, if he, if he wants, I'm, I'm quite sure if it, it is possible in, in, at least in theory, it is possible to, to exploit it and, um, yeah, if, if you, if you get ephemera from an untrusted source, if you, if I send you ephemera, then you should definitely throw it into Amba,

Yeah.

Randal: It should just look for the string, amba vector or something. Yeah. . Yes. Not run the code if it finds that, oh, if only we're that easy.

Jonathan: Randall, you want to cover some of these that you've got stacked up?

Randal: Yeah, yeah. I was just going through my notes things I'd taken before I did this show. So you know, AI is a big thing. Everybody's saying AI. Is there any idea that maybe you're going to introduce some AI to this? Help fill some of the gaps in, or help extend what you're doing, or maybe try a more brute force approach.

You know, is there any way that large language models or even just neural nets can help you out with this stuff?

Benedikt: Let's say indirectly we had that. Okay. What is indirectly? We had that feature for a long time. I can tell you the exact date because I don't remember, but we have a whole section of modules that we introduced and one specific module was for an open AI connection.

So you're able to put in a API key and then Basically what Ember does is with decompiled code and especially if we've script languages, I think we have this option to ask the AI bot. If there's any vulnerabilities in certain code snippets that are previously identified by another tool.

Jonathan: Oh, interesting.

Benedikt: So there is also like a lot of expansion stuff that's possible with that. You can change questions, whatever, because. If you wanted to do that the results, because I've been testing it a bit, the results are, well, do they really give you more? That's the question, I feel like that's the real question.

Yeah, the false positives there, the false positives there are probably

Randal: pretty high. Is it, is it

Benedikt: telling you anything that other tools can't, like? Yeah. That's

Randal: I'm a little biased because I'm a Google developer expert, but I'd also suggest looking at Gemini's current products because they're really moving far forward and they're recently trained.

Which means that they're much more up to date than any of OpenAI's offerings at this point. So, you might take a look at that. Especially Gemini Pro 1. it. And it's really got some amazing results so far for the stuff I've been throwing at it. So just, just an idea there. Let's see. I also had a couple more questions.

Let's see, what else do I want to ask here? So, is this just about Linux based firmware, or are you planning on expanding it to other operating systems, other firmware types?

Max: Hmm. Is there

Randal: anything else?

Max: So that's, that's, that's, that's, that's the question. No. So, so in, in the area we, we are located. We are, we are using Amber.

We have to deal with a lot of Linux firmware. Nevertheless as I said, Our usual approach is to throw a firmware before of a, of a manual penetration test into AMBA. And AMBA gives us a few days later the, the, the results. We, we also have sometimes some UEFI firmware. We have sometimes some RTOS such a VX works or something like this.

And we want to get the, the maximum out of it. Although we are not the experts in this area. So, we, we, we do not want to, let's say, we, we do not want that Amber is crashing because there is no Linux in it. Then Amber should, should, should give us the information that there's a VX works. And we are doing a lot of or we have a lot of modules.

that are, let's say, file system, system independent, so you can also use it just on a binary blob, for example, to, to extract some key material or to find some, some, some interesting hashes. And then Amber can do this also on, on a, on a, on a non Linux operating system. And additionally, in, in the future.

field of, of UEFI, so, so we, we do not deal a lot with UEFI analysis, but there, there is a company called Binary and they have released an UEFI security analyzer as open source. So we have just put their complete open source scanner included it into Ember. Wrapped it around an ammo module.

And if we, as soon as we detect any UEFI firmware then we fire up this vulnerability scanner now. To, and so we are also able to provide some, some kind of value also non Linux operating systems.

Randal: Cool. Cool. And one last question. I'm gonna throw it back to Jonathan. So results, what's the most unusual thing you've discovered with this or just what are some of the practical outcomes of this that make it worth your time?

Randal: you must have discovered something with this or you this. They're now,

Jonathan: they're now having to do the, the question of, of the things we found. What are we allowed to talk about? . Oh, oh, sorry. I didn't realize that would put you at risk. So, so,

Max: so, so like we, we can say we, we are talking quite generic, so nobody knows what, what we are talking about, so, okay.

I think they, it's, it's very often really interesting if you're, if you're analyzing established security products that are established there for years now and you're, you're analyzing them and you can see that the, the, the components that they are using are, are from, from the age of dinosaurs.

And that they're, that they're not security components and they're not updating the, the, the product anymore. Or the, the base product that they're operating on, on the operating system level. And they're, they're just managing the risk, let's say this way. And this is, this is really interesting because often these are established vendors, established products, and then you can see that not only D Link is using a kernel 2.

6 dot something, also very established security company is doing the same. Wow. Yeah. I think there, what, what was it that there was, I cannot remember exactly, there was one one, one company or one product. That was destroyed somewhere in the internet. I don't remember it now, but, but, but,

Randal: but if I could summarize what you're saying is that some people who should know better don't always demonstrate that they do.

Yes. And, and with, with,

Max: with automated firmware analysis you get a quite good overview without any manual work. And then, you know, that's usually the, the, the thing where we are using Ember a lot. Ember is doing the automated work and we can see, okay, oh my God, this product is looking really fishy.

So we, we, we have 10, 10, 10 products. And then we know, okay, we start with. This one, because this looks fishy.

Randal: So it's like, when I'm looking at a piece of code that is operating on a live website, that's written in PHP and has a very obvious SQL injection attack. And I go, how long has this code been in here and this company in particular should know better to run PHP code.

That looks like this. Y'all. It's, it's, it's, it's a sad experience. It requires some warning on my part, and then I move on and go, well, I'll probably write code exactly like that someday. Yeah.

Max: And, and, and usually these are the windows that are shipping encrypted firmware updates so that nobody can see the, the, the PHP code.

It's encrypted.

Randal: We don't have to care about it. They don't want you to see how bad it is, that's why. They're just shipping junk under a golden label. Yeah,

Jonathan: it's sad how much that happens. Alright, so, I've got to ask, why open source? Right, like you guys apparently use this at Siemens or Siemens Energy. It's it's used for it sounds like for a lot of internal use What was the what was the conversation like saying?

Hey, we should open source this and release it to the world

Benedikt: I mean it started as a project where you're like, okay, I have an issue and there's no tool for it I really feel like that's a classic open source thing, right? You you're right. You're starting free time. You you start with a An idea, and then you write something that you actually need for your work and then you're like, okay, I feel like since there's nothing out there, why not just open source it, like put it out there.

And that's, that's basically what happened with Emma making it like a security tool for everyone. That's also a big part of the cyber security idea. I feel like that's if you write a security tool, you put it where everyone can use it.

Jonathan: I like it. I know, I know some corporations, they, they get real antsy about releasing internal tools as open source.

Was EMBA started was it started in, in your spare time or was it sort of kind of official? It's like a company project.

Max: I would say combination. So we we we started during penetration test and then every every you're fascinated you want to do everything and then You're running into time constraints and then you you can't sleep.

So you start coding in the night and Is it no work time? No, it's not work time because you have already Your, your, your, your hours together, so it's free time. So, so pro probably most of the guys out or the open source developers out there can ha have a little bit the same story. Yes, they're fascinated.

So where, where at the beginning you can say where is company time, where's free time and everything swims together. You want to solve the problems and yeah, I think it's, it's, it's a mixture of both. We, we have the time in the company, but we are, we are also spending quite a decent amount of time during our, our free time.

Jonathan: Yes. I know that feeling waking up at two in the morning. I know what's wrong. Then it's like, do I try to put myself back to sleep or do I go in there and fix it?

Max: And then you need to go up because you can't sleep anymore and you need to fix it. And then you realize, Oh I fixed this issue, but there's the next one.

So I don't need to go to bed anymore.

Randal: I think what's worse than either of those is waking up, realizing you have the solution to a problem and not writing it down and forget. That's really bad.

Max: That's true. Yeah.

Randal: I, I've done that. That's why I know it's not a fun thing. Yeah. I try to get at least something written down when I have an idea in the middle of the night.

So I at least have it for the next day.

Jonathan: Are there any, are there any CVEs out there that have a special thank you to EMBA? Do you have like a list of you know, this one and this one and this one were found because people were using EMBA?

Max: So, so We have an internal list of our CBEs that were fixed that we, we reported to other vendors and that were fixed.

Mm-Hmm. , but there's nothing out there in the, in the public. No. Okay.

Jonathan: That would be, that would actually be an interesting project to just have a repository. Now, now I understand like some of the things you find that you just can't, right? Like I get that there are things that will get discovered internally reported internally and may never be assigned a CVE number.

Like that's fine. That's just how the world works. But it would be very interesting to have a repository where you say, Hey, security researchers, if you find something, if you find a CVE through EMBA. Shoot us a note here and they just have like a high score list somewhere on the Ember website. I think that would be a lot of fun.

Oh yeah.

Max: So, so, so we, we, we have a collection of all of the papers and blog posts and mentioning of Ember. So If someone sends us the CVEs that they have found with the help of Amber, then they will definitely get a nice and shiny place there. Yeah.

Max: So, so we, we are prepared for these reports.

Awesome.

Jonathan: So one of the things, so I, I write about security and one of the things that I try to encourage people like just that are interested in it is the barrier to entry for finding CVEs for finding security problems and getting them fixed. Okay. It's actually a lot lower than most people think it is, right?

Like it's, I, I just, now I didn't get assigned to CVE, but I got paid a bug bounty for literally just running a trace route and then running a port scan back when I had Starlink Internet, and they had set, they had set a server up and it. did not, it did not have a proper firewall if you were coming from inside the Starlink ISP.

And I got paid several thousand dollars for reporting that. And this is one of the things that I, I really like to encourage people on, is like, go looking for things, and if something seems weird. So all of that said, what does the process look like for setting up EMBA and EMBARQ? How difficult is it to get started and then say, feed the latest firmware for your, from your own home router into it?

There's obviously, there's going to be problems in home router software. How difficult is it to get started and start finding them?

Benedikt: I would say very easy, but maybe I'm biased here. Of course the easiest way is for, for us just tell you, okay, there's a Kali ISO run a VM on your laptop and then run it.

Just do the git pull git clone and to an installer which is very easy takes a bit of I don't know a few minutes for sure to download and install. The barrier of entry is basically Now I would say because you can also do it on bare metal You can install kali on basically anything I would say the only barrier I would see is Minimum system requirements because Ember is, as we talked about, a bit of beast.

So it has some minimum requirements where you, you can't, like, it won't run nicely below that.

Jonathan: Yeah. So, and so is, is Cali kind of the preferred place to set it up and start using it?

Benedikt: Well for Embar for sure. I feel like Mike and I, we always are on Cali. Mm-Hmm. Em, embark is designed for Ubuntu. Okay. Because if you, if you're talking about Linux environments and servers most people install Ubuntu, right? So that's, that's what, what is it? It's intended for. So EMBA would work on two yeah, well two Ubuntu versions and Ali, the current Ali.

So there's options.

Jonathan: Alright. We are I just saw at the time we are rapidly running out of time and I, I wanna make sure and ask about community. Like, so what, what is the, what is the community size? Like, do you guys get contributions? Have you had people from? outside of your company and outside of the project come by and say, Hey, I thought it would be cool if EMBA could do this.

Here's, here's the patch to make it happen. Or, you know, bug reports. Is that something that is happening or is it, is it kind of not public enough yet?

Max: So, so I think, I think we're getting more and more bug reports during the, let's say during the last year, we are getting more and more insights on how people are using EMBA.

How people are. Talking and writing about it. People are also reporting their bugs and their wishes. We, we had a, a, a few contributors that there, which are directly fix have started fixing the bugs but really just a few. So the, the the, the, the community is growing, but let's say the community that is interested in also helping is growing very slowly.

But the, the more the community is growing I'm, I'm sure that then also the, the, the helping hands will, will come more and more. Sure.

Jonathan: So if somebody wants to learn more about EMBA or EMBARQ, where do they go to learn more? And then if they want to talk to you guys, where is the best place to do that?

Right

Randal: here.

Jonathan: So like, what's, what's the, what's the website? What's the GitHub URL? Is there a Discord server? That sort of thing.

Benedikt: No Discord yet, no. Not that I'm aware of. That's something we should talk about, Mike. Absolutely. Thinking about that, yeah.

Max: cu Currently the easiest way is going, going to the GitHub project. If, if you have ideas, open an issue, open a discussion.

If you, or if you're writing about Amber or showing Amber or something around Amber somewhere, drop us a note somewhere at GitHub or GI or X or LinkedIn. Mastodon the usual social networks. Mm-Hmm. . We, we have there an, an dedicated Ember account, which is called Secure Firmware. And we, we are happy to, to, to publish also your, your papers that I'm mentioning or your talks that I'm mentioning Ember there.

And If, if we need to talk in, in person, like we did it and drop us a note somewhere over there and we find a way to, to get in contact. Yeah.

Benedikt: And I think, Mike, you're at the next, no, conference, the big one in Oh, yeah, yeah,

Max: we, we, we have plans going to, to Black Hat Middle East now. Ah. I think in the end of November.

Okay. So if someone is there, then drop, again, drop us a note and we, we have there a, a, a demonstration of EMBA at Arsenal, and then just come to our booth and we can, we can talk about EMBA and Fermi analysis and all the rest of the world.

Jonathan: Yeah, very cool. All right. We are out of time i've got to ask each of you the final two questions.

It is a requirement. We get in trouble if we don't do it we'll start with michael. What's your favorite text editor and scripting language?

Max: So so scripting language is is definitely bash. I I've now written 30 000 lines of coding bash. So it it needs to be my favorite scripting language. Yeah, and I always exited WIM after working on AMBA, so this is my favorite text editor.

I'm not a master of WIM, but I can exit it. You know how to get

Jonathan: out. It's fine for me.

Max: Yes. Save and exit, not just exit. Yeah,

Randal: hopefully. All right. If you're not saving it, it starts sticking.

Benedikt: All right. And Benedikt, same two questions. Yeah. For me, it's somewhere between Python and Bash. More fluent in Python though, I should say that.

And editor wise, I would say, yeah, not, not the biggest fan of him. I learned it. Normally I stick with Nano. That's fair.

Jonathan: That's fair. Alright. Thank you guys for being here. I wish we had more time. We'll have to bring you back maybe after, maybe after Blackhat. We'll bring you guys back on and talk about what's changed and get in all the questions we didn't get to today.

But thank you both so much. Thank you for having

Max: us. Have a good day.

Jonathan: Yup. Yup.

Max: And

Jonathan: bye bye.

Max: See

Jonathan: you. Alright. What do you think, Randall? You gonna go dig the firmware out of your router now and take a look and see if you can find some fresh CVEs?

Randal: Well, it's not my router, which is why I'm on Wi Fi. So It's I live in the basement of a, of a two main story house and they have the router upstairs.

And so that's why I don't have, which is why we were in troubles before the show. Yeah. Getting me on the screen of why I dropped it. After we're done taping, I've got some advice for you about how to set up the next show so that it's a little better bandwidth wise. So this was interesting in that I haven't spent a lot of time thinking about the firmware that's in all the devices that are now being used in my life.

And now it makes me more scared. Great. But at least there's some sort of diagnostic tool that at least somebody can run against some release of maybe something that I will use or eventually use or be kept from using because they found bad stuff in it. I like your open source question. Cause it's like that, unless they're getting a lot of community Contributions.

This isn't, this doesn't really pay off as open source. Unless most of this stuff was already open source and they're just bringing it together and kind of from a potluck perspective where everybody's bringing food and everybody takes food. Home, it sort of makes sense for it to be open source. So I like, I like it from that perspective.

It was a little out of my wheelhouse, so I didn't get a lot of questions. So

Jonathan: Randall, what you need to do, if you decide to start doing that, start working with this at all, is write yourself a custom module to see is any of your code in the firmware and just like make some sort of trumpet fanfare that plays automatically when it finds, you know, a Swartchian transform or whatever that, that is actually in there.

All right.

Randal: Right. Well, it's like, you know, the usual editor question, you know, for years, I said Emacs to that because there's a part of my code in every copy of GNU Emacs. So that kind of thing. And, and Yeah, Schwarzschild transforms show up everywhere. It's kind of crazy how that how that one's gone kind of spiraling out.

So, yeah yeah, cool. No, I was, it was a, it was a good show. The guests were knowledgeable and the project is worthwhile and in progress. Yep, absolutely. All right. Do you have anything you want to plug? Ah, yes. So so this is now a new addition to my semi regular schedule as often as Jonathan will let me back.

And, and I've opened for the day. Cause so typically my, my Tuesdays are pretty free. So I think it'll probably work out for at least as often as Jonathan's willing to put up with me. But every Wednesday I tape a live stream show as well. On Dart and Floss, which are my, not Dart and Floss, Dart and Flutter.

Ah, too many FL words. Dart and Flutter. I am a Google developer expert in the Dart and Flutter arena. I'm all in on Dart and Flutter. So, yes, although I was kind of hinting at Perl earlier in the show, I really, I personally haven't written more than 10 lines of Perl code probably in the last year. And but I've written thousands of lines of Dart code and Flutter code and, and things like that.

And so every Wednesday we do what we call Hump Day Q& A, Ask Me Anything, where we get a few experts together. And for some reason they let me on there. And I have no idea why. And we take live questions over YouTube. So if you go to YouTube and look for Flutter community and Flutter is anything vaguely interesting to you, please check that show out.

We really enjoy the live questions. We get some good stuff doing, we also do some live coding. So it's been a lot of fun doing that week for week for the last, I think, two, three years. So it's almost been the same burden that I used to have weekly, but now delivering it into the dart and flutter arena and the problem with the, not the problem, but the, one of the things to know about the hump day Q and a is it runs two or three or four hours long.

So. At about the third hour, I'm telling Simon, who's now madly typing in, trying to solve that last problem in his live coding. I say, Hey, we're starting the fourth hour there, Simon. And he'll finally go, Oh, I should probably wrap this up then. Yeah. Why don't you? Why don't you? We're down to 22 people watching, so we should probably stop.

Oh, that's great. Fun show. It's a great show. I love it. I'm, I look forward to it every week. I'm just making notes now about what we're trying to do tomorrow. But check that out. That's probably the biggest place to see me. I also have a dart and flutter YouTube channel, which I try to contribute to you on a regular basis.

So. You want to check that out. Just, just Google Randall Schwartz, Dart and Flutter. You'll find all sorts of good stuff about me and my new career.

Jonathan: Yeah. Excellent. Thank you, sir, for being here. It is good to have you back and we will make sure. We'll make sure and get Randall in the rotating cadre of co hosts.

So we'll see him again here in a few weeks and look forward to that. If you want to find my stuff, of course, Hackaday. We appreciate them being the new, the new home of Floss Weekly. You're not really new anymore. I've been here for almost a year. I've also got the security column that goes live every Friday morning and then still over on twit We've got the untitled Linux show, which is a lot of fun And if you're into Linux, which I imagine most of our listeners are you should go and check that out, too We appreciate everybody being here those that catch us live and on the download and we will see you next week on Floss Weekly