Developer Voices

I’ve often wondered how you build a text editor. Like many software projects, it’s a simple idea at the core with an almost infinite scope for features. How do you build a solid foundation to expand on? Which features matter for launch? And how do you hope to satisfy the needs of every programmer, working in every language?

My guest for this episode is Nathan Sobo. He’s tackled this problem once before with the Atom editor, and he’s back older & wiser with Zed - a new editor written completely from scratch in Rust. It has a modern UI, a wide spread of language support, and a completely different way of looking at team collaboration. But with so much ambition, what are Zed’s priorities, and what’s been left for a future version?

--

Zed Homepage: https://zed.dev/

Segment Trees: https://en.wikipedia.org/wiki/Segment_tree

Ropes: https://en.wikipedia.org/wiki/Rope_(data_structure)

Rust Executors: https://rust-lang.github.io/async-book/02_execution/04_executor.html

More about Roc: https://youtu.be/DzhIprQan68

More about TigerBeetle: https://youtu.be/ayG7ltGRRHs

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

This week on Developer Voices we’re talking to Ryan Worl, whose career in big data engineering has taken him from DataDog to Co-Founding WarpStream, an Apache Kafka-compatible streaming system that uses Golang for the brains and S3 for the storage.

Ryan tells us about his time at DataDog, along with the things he learnt from doing large-scale systems migration bit-by-bit, before we discuss how and why he started WarpStream. Why re-implement Kafka? What are the practical challenges and cost benefits of moving all your storage to S3? And would he choose Go a second time around?

--

WarpStream: https://www.warpstream.com/

DataDog: https://www.datadoghq.com/

Ryan on Twitter: https://x.com/ryanworl 

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

PostgreSQL is an incredible general-purpose database, but it can’t do everything. Every design decision is a tradeoff, and inevitably some of those tradeoffs get fundamentally baked into the way it’s built. Take storage for instance - Postgres tables are row-oriented; great for row-by-row access, but when it comes to analytics, it can’t compete with a dedicated OLAP database that uses column-oriented storage. Or can it?

Joining me this week is Philippe Noël of ParadeDB, who’s going to take us on a tour of Postgres’ extension mechanism, from creating custom functions and indexes to Rust code that changes the way Postgres stores data on disk. In his journey to bring Elasticsearch’s strengths to Postgres, he’s gone all the way down to raw datafiles and back through the optimiser to teach a venerable old dog some new data-access tricks. 

–

ParadeDB: https://paradedb.com

ParadeDB on Twitter: https://twitter.com/paradedb

ParadeDB on Github: https://github.com/paradedb/paradedb

pgrx (Postgres with Rust): https://github.com/pgcentralfoundation/pgrx

Tantivy (Rust FTS library): https://github.com/quickwit-oss/tantivy

PgMQ (Queues in Postgres): https://tembo.io/blog/introducing-pgmq

Apache Datafusion: https://datafusion.apache.org/

Lucene: https://lucene.apache.org/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

The actor model is a popular approach to building scalable software systems. And isn’t hard to understand when you’re just reading about the beginner’s examples. But how do you architect a complex design using the actor model? Which patterns work well? How do you think through it?

Joining me to take us through it is Hugh McKee. Hugh’s a total actor-model fan, and a Developer Advocate for Lightbend (the company that created the popular actor framework Akka). He takes us from his definition of actors to the designs he’s worked on, the patterns he’s found most useful, and the interesting meeting-point between actor-based designs and event-based ones.

—

Wikipedia - Actor Model: https://en.wikipedia.org/wiki/Actor_model

Hugh’s book, Designing Reactive Systems: https://go.lightbend.com/designing-reactive-systems-role-of-actor-model

Hugh on Twitter: https://twitter.com/mckeeh3

Hugh on LinkedIn: https://www.linkedin.com/in/mckeehugh

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Bytewax is a curious stream processing tool that blends a Python surface with a Rust core to produce something that’s in a similar vein to Kafka Streams or Apache Flink, but with a fundamentally different implementation. This week we’re going to take a look at what it does, how it works in theory, and how the marriage of Python and Rust works in practice…

–

The original Naiad Paper: https://dl.acm.org/doi/10.1145/2517349.2522738

Timely Dataflow: https://github.com/TimelyDataflow/timely-dataflow

Bytewax the Library: https://github.com/bytewax/bytewax

Bytewax the Service: https://bytewax.io/

PyO3, for calling Rust from Python: https://pyo3.rs/v0.21.2/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

--

#softwaredevelopment #dataengineering #apachekafka #timelydataflow

Mojo is the latest language from the creator of Swift and LLVM. It’s an attempt to take some of the best techniques from CPU/GPU-level programming and package them up in a Python-compatible syntax.

In this episode we explore why Mojo was created, and what it offers to Python programmers and non-Python programmers alike. How is it built for performance, and which performance features matter? What’s its take on functional programming and type systems? And can it marry the high-level programming of Python with the low-level programming of LLVM/MLIR?

If you’re a Python programmer who needs better performance, a C programmer who expects more from a ‘scripting language’, or just someone who’d be happier if Python had a first-class type system, Mojo might well be for you…

–

Mojo: https://www.modular.com/max/mojo

Mojo’s Roadmap: https://docs.modular.com/mojo/roadmap.html

The Mojo Discord: https://discord.com/invite/modular

MLIR: https://mlir.llvm.org/

Chris’s Talks: https://nondot.org/sabre/Resume.html#talks

Chris on Twitter: https://twitter.com/clattner_llvm

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#software #podcast #mojolang #ml #pythonml

Every database has to juggle the need to process new data and to query old data. That task falls to any system that “does stuff and remembers stuff”. But it’s quite hard to really optimise one system for both use cases. There are different constraints on new and old data, and as a system gets larger and larger, those differences multiply to breaking point. That’s something Twitter’s engineers were figuring out in the 2010s.

One solution that came up in those years was the Lambda Architecture. A two-pronged approach that recognises the divide between new and old data, and works hard to blend the two together seamlessly in userspace. But that seamless blending is easier said than done. It’s nearly all bespoke work.

What if you could get it off the shelf? Let someone else do the work of combining two different kinds of database into one neat package? That's the question of the week as we look at the recently open-sourced project Proton, and its attempt to be the Lambda Architecture in a box…

–

Proton Docs: https://docs.timeplus.com/proton

Proton Source: https://github.com/timeplus-io/proton

Timeplus: https://www.timeplus.com/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwareengineering #databases #dataengineering 

Rust changed the discussion around memory management - this week's guest hopes to push that discussion even further.

This week we're joined by Evan Ovadia, creator of the Vale programming language and collector of memory management techniques from far and wide. He takes us through his most important ones, including linear types, generation references and regions, to see what Evan hopes the future of memory management will look like.

If you've been interested in Rust's borrow-check and want more (or want different!) then Evan has some big ideas for you to sink your teeth into.

–

Vale: https://vale.dev/

The Vale Discord: https://discord.com/invite/SNB8yGH

Evan’s Blog: https://verdagon.dev/home

Evan’s 7DRL Entry: https://verdagon.dev/blog/higher-raii-7drl

7DRL: https://7drl.com/

https://verdagon.dev/grimoire/grimoire

What Colour Is Your Function?: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

42, the language: https://forty2.is/

Verona Language: https://www.microsoft.com/en-us/research/project/project-verona/

Austral language: https://austral-lang.org/

Surely You’re Joking, Mr Feynman! (book): https://www.goodreads.com/book/show/35167685-surely-you-re-joking-mr-feynman

Evan on Twitter: https://twitter.com/verdagon

Find Evan in the Vale Discord: https://discord.com/invite/SNB8yGH

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#software #programming #podcast #valelang

The “big data infrastructure” world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you’re in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow. 

So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we’re going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It’s a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible? 

Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren’s F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you’ve built? Let's find out.

– 

Quix Streams on Github: https://github.com/quixio/quix-streams

Quix Streams getting started guide: https://quix.io/get-started-with-quix-streams

Quix: https://quix.io/ 

Tomáš on LinkedIn: https://www.linkedin.com/in/tom%C3%A1%C5%A1-neubauer-a10bb144

Tomáš on Twitter: https://twitter.com/TomasNeubauer0

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins  

--

#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing

Erlang wears three hats - it’s a language, it’s a platform, and it’s an approach to making software run reliably once it’s in production. Those last two are so interesting I sometimes wonder why those ideas haven’t been ported to every language going.  How much work would it be?

This week we’re going to dig right down into that question with Leandro Ostera. He’s been working on Riot - a project to bring the best of Erlang’s runtime system and philosophy to OCaml. But why OCaml? Is it possible to marry together OCaml’s type system with Erlang’s dynamic dispatch systems? And what is it about the recent release of OCaml5 that makes the whole project easier?

–

Leandro’s Blog: https://www.abstractmachines.dev/

Why Typing Erlang is Hard: https://www.abstractmachines.dev/posts/am012-why-typing-erlang-is-hard/

Riot: https://riot.ml/

Riot source: https://github.com/riot-ml/riot

ReasonML: https://reasonml.github.io/

ReScript: https://rescript-lang.org/

Leandro on Twitter: https://twitter.com/leostera

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

--

#podcast #softwaredevelopment #erlang #ocaml #softwaredesign

The likes of LinkedIn and Uber use Pinot to power some astonishingly high-scale queries against realtime data. The numbers alone would make an impressive case-study. But behind the headline lies a fascinating set of architectural decisions and constraints to get there. So how does Pinot work? How does it process queries? How are the various roles split across a cluster? And equally important - what does it *not* try to achieve.

Joining me to go through the nuts and bolts of how Pinot handles SQL queries is Tim Berglund, veteran technology explainer of the realtime-data world. He takes us through Pinot step-by-step, covering the roles of brokers, servers, controllers and minions as we build up the picture of a query engine that's interesting in theory and massively performant in practice.

–

Apache Pinot: https://pinot.apache.org/

Apache Pinot Docs: https://docs.pinot.apache.org/

StarTree: https://startree.ai/

Event Driven Design episode with Bobby Calderwood: https://youtu.be/V7vhSHqMxus

Tim on Twitter: https://twitter.com/tlberglund

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwaredevelopment #apachepinot #database #dataengineering #sql

TJ DeVries is a core contributor to Neovim and several of its most interesting sub-projects, and he joins us this week to go in depth into how Neovim got started, how it’s structured, and what a truly programmable editor has to offer programmers who want the perfect environment.

Along the way we look at what we can learn from Neovim’s successful fork of the 30-year old codebase from Vim, how it still collaborates with the original project, and what putting Lua at the heart of the system has done for casual tinkerers and hardcore plugin writers alike.

Not everyone will come away from this discussion wanting to switch editors, but I’m sure you’ll get a newfound appreciation for digging deeper into the developer tools you use everyday.

–

Neovim: https://neovim.io/

Neovim Kickstarter: https://github.com/nvim-lua/kickstart.nvim

Kickstarter walkthrough video: https://www.youtube.com/watch?v=m8C0Cq9Uv9o

A directory of Neovim plugins: https://dotfyle.com/

Vimscript’s definition of true and false: https://vimhelp.org/eval.txt.html#Boolean

TJ on Twitter: https://twitter.com/teej_dv

TJ on Twitch: https://www.twitch.tv/teej_dv

TJ on YouTube: https://www.youtube.com/@teej_dv

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #software #softwareengineering #dx

Done right, a Hackathon can be a fantastic place to be a programmer - you get time and space to build and learn, in a room full of like-minded people, with swag and prizes to sweeten the deal. It’s a great way to pick up new ideas and run with them. But done wrong it can be a waste of time. What’s the difference between a good hackathon and a bad one? What do the good ones do right, and what can we learn from that?

This week we’re talking about the Joy of Hacks with Major League Hacking Co-Founder Jon Gottfried. He’s got over 10 years of experience building a Hackathon network that provides the right environment for “structured mucking about with computers”, so we’re going to pick his brains.

If you’re ever attending a Hackathon, organising one, or looking for a way to build or contribute to your local programming community, Jon can help guide you to events that work.

--

Major League Hacking: https://mlh.io/

Major League Hacking’s 2024 Event Calendar: https://mlh.io/seasons/2024/events

Games Week: https://events.mlh.io/events/10848 

Jon on Mastodon: https://hachyderm.io/@jonmarkgo

Jon on LinkedIn: https://www.linkedin.com/in/jonmarkgo

Jon on Twitter: https://twitter.com/jonmarkgo

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Bonus link - The Great American Baking Show 2023: https://www.youtube.com/watch?v=IlWLSAKEedk

--

#software #podcast #programming #hackathon

One of the most promising techniques for software reliability is property testing. The idea that, instead of writing unit tests we describe some property of our code that ought to always be true, then have the computer figure out thousands of unit tests that try to break that rule.

For example, you might say, “No matter which page you visit on my website, there should always be a login button or a logout button.” Then the test’s job is to try to break that rule, but clicking around until it finds some combination of clicks fails that assertion. Like, maybe it finds the 404 page, and you realise it was missing the website’s normal header.

At its best, property testing takes far less work than unit testing, but is far more thorough, because it lets us write the rules and has the computer write the examples. The downside is, it often seems theoretical. It can be hard to apply property testing to real-world cases. Let’s fix that.

We’re joined by Oskar Wickstrom, who’s been building all kinds of different systems and bringing property testing with him wherever he goes. We discuss the basics of property testing, then he goes into the advanced and cunning techniques that go beyond the ordinary into testing databases, webpages and more. With a bit of thought, he can help us test a ten times as much code with a tenth of the effort.

--

Oskar’s book, Property Testing a Screencast Editor [ebook]: https://leanpub.com/property-based-testing-in-a-screencast-editor

Quickstrom: https://quickstrom.io/

F# for Fun & Profit: Property Testing Series: https://fsharpforfunandprofit.com/series/property-based-testing/

Linear Temporal Logic: https://en.wikipedia.org/wiki/Linear_temporal_logic

The Quickstrom Paper: https://arxiv.org/abs/2203.11532

TodoMVC (One frontend app, many implementations): https://todomvc.com/

Oskar on Twitter: https://twitter.com/owickstrom

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

--

#softwaredevelopment #podcast #programming #testdrivendevelopment #propertytesting

If you ever feel overwhelmed by the number of different programming languages, this week’s episode might just offer you some solace, as we talk about an attempt to reunify many of the most popular languages by focussing on the bread & butter things that every language supports.

I’m joined by Martin Johansen, who’s been working on a new tool called Progsbase. With it, he’s created a spec based on all the things programming languages can agree on, and is building a library that can cross-compile between them. Write a program in Java, and it can be automatically translated to PHP, Python and a great deal more.

But how far can he take that idea? Is there really enough unity between these languages to build something universal? How do you bridge the divide between manual memory management languages like C and garbage-collected ones like Java? And what would it actually feel like to write code this way? Let’s put Martin’s plan under the spotlight and find out…

–

Martin on Twitter: https://twitter.com/martinfj

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Progsbase homepage: https://www.progsbase.com/

The Spec: https://www.progsbase.com/docs/programs/

The Progsbase library repository: https://repo.progsbase.com/

The Bug Bounty: https://www.progsbase.com/bug-bounty/

–

#software #programming #podcast #programminglanguages

A lot of programming is split into the mechanical work of writing what you know, and the creative work of figuring out what you don’t know. Wouldn’t it be nice to automate the mechanical stuff away?

Well the good news is we’re already automating a lot of it. Every time you run a refactoring tool or a pretty-printer, you’re handing boring work off to the computer. But how does that magic work, and how can we do more of it?

This week we’re joined by one of the authors of OpenRewrite—Jonathan Schneider—to learn how automated code-rewriting tools really work. From the basic approach to the hairy corner cases, to the reality of keeping developers happy with the subjective side of the results.

It takes a lot of work to automate work away - this week we’ll learn how the work gets done for us too.

–

OpenRewrite: https://docs.openrewrite.org/

Supported Languages: https://docs.openrewrite.org/recipes

Moderne: https://www.moderne.io/

Gradle Lint: https://github.com/nebula-plugins/gradle-lint-plugin

Chicory (Native JVM WASM): https://github.com/dylibso/chicory

Call Java from Haskell: https://github.com/tweag/inline-java#readme

Call Haskell from Java: https://github.com/nh2/call-haskell-from-anything

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #software #programming #softwareengineering #refactoring #parsers

SQLite could do with a little competition, so when I invited the co-creator of DuckDB in to talk, I thought we'd be discussing the perils of trying to build a new in-process database engine. I quickly realised things went much deeper than just a tech refresh.

Hannes Mühleisen joins me this week to blend his academic credentials as a database researcher with his vehement need to make that research practical. And so we dive into what modern database literature has to say on making queries faster, more parallelizable, and closer to the metal, and how it all comes together in a user-friendly package that’s found its way into my day-to-day workload, and might well help out yours.

If you’re curious about the gory details of database queries, how they can take advantage of modern hardware, or how all that research actually turns into a useful tool, Hannes has some great answers.

--

DuckDB: https://duckdb.org/

Database Systems Book: http://infolab.stanford.edu/~ullman/dscb.html

Kris’ first computer: https://en.wikipedia.org/wiki/File:ZX_Spectrum_Plus2_(retouched).jpg

Volcano Query Evaluation System [pdf]: https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b186061.pdf

Morsel Query Engine [pdf]: https://cs.brown.edu/~kayhan/papers/morsel_cp.pdf

Unnesting Arbitrary Queries [pdf]: https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf

Papers Hannes' team have published: https://duckdb.org/why_duckdb#peer-reviewed-papers-and-thesis-works

DuckDB on Mastodon: https://mastodon.social/@duckdb

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

--

#softwaredevelopment #podcast #programming #database #duckdb #sql #sqlite

This week we talk to Simon Peyton Jones, a veteran language designer and researcher, and key figure in the development of Haskell. Haskell. Simon has made countless contributions to advancement of functional programming, and computer programming in general, and is currently working at Epic Games, working on the foundations of their new programming language, Verse.

We discuss how programming languages are made, focussing on a big design idea from both Haskell and Verse: building a large language from a small, tightly designed core. Then we move into Simon's current work exploring Functional Logic Programming, the big new idea that underpins Verse. It's an idea that blends the fundamentals FP with the core ideas of logic languages like Prolog in an entirely new way. Not even Simon knows exactly where the idea will lead, but it's a fascinating idea that could potentially bring constraint-solving and deduction right into the heart of modern software.

Additionally, Simon discusses his involvement in reshaping the way we teach computing in England. He's been working hard to give computing education the same importance as the teaching of mathematics and sciences - something we should all have a fundamental understanding of.

Simon's one of the smartest, nicest people in programming. Come as hear his brilliant brain at work. :-D

–

Verse: https://github.com/UnrealVerseGuru/VerseProgrammingLanguage

The Verse Language Reference: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-reference

The Verse Calculus [pdf]: https://simon.peytonjones.org/assets/pdfs/verse-March23.pdf

https://en.wikipedia.org/wiki/Simon_Peyton_Jones

The LogicT monad: https://hackage.haskell.org/package/logict

Can programming be liberated from the von Neumann style?: https://dl.acm.org/doi/10.1145/359576.359579

CAS - Computing At School: https://www.computingatschool.org.uk/

Computer Science Teachers Association: https://csteachers.org/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Benthos wants to be part of your Data Engineering toolkit - it’s there as a quick and easy way to set up data pipelines and start streaming data out of A and into B. In contrast to a lot of the tools we’ve talked about on Developer Voices, Benthos seems focussed on cutting development time down to a minimum, so you can quickly configure a new pipeline and test it out, without making a whole sprint of the task. As quick as a quick-and-dirty shell script, without the dirt. 😉

So this week we’re talking to the creator of Benthos, Ashley Jeffs, to hear why he created Benthos, what it can do for you, and what its strengths and weaknesses are. And Jeff’s refreshingly candid about when you should and shouldn’t use it. If you ever need to get data from an HTTP connection into S3, or S3 into Kafka, or Kafka into a flat file, Benthos might just save you a few hours of development.

–

Benthos: https://www.benthos.dev/

A list of supported inputs, processors & outputs: https://www.benthos.dev/docs/about#components

All their cute blobfish logos: https://www.benthos.dev/blobfish/

IDML: https://idml.io/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #podcast #dataengineering #datascience

The world of game programming might seem a million miles away from 'regular' programming. But they still have to deal with the same kinds of data, scale and concurrency problems that we’re all familiar with in the software world. And that makes the gaming world an interesting place for new ideas - under the hood they’re solving those same problems we face, but often with some novel ideas about the solutions.  So this week we’re off to the massive open world that is game development, to see what we can learn that might make lives easier in the non-gaming space. Joining us for that is Tyler Cloutier, the founder of Clockwork Labs. They’re building SpaceTimeDB, a curiously-distributed database built to be the underlying platform for their new MMORPG, BitCraft. Digging down into the architecture of SpaceTimeDB, we pick Tyler’s brain for nuggets of information on event sourcing, request/response vs. subscriptions, transactions, security and much more. All in an effort to make our programmers and data scientists’ lives easier.

--

SpaceTimeDB: https://spacetimedb.com/

BitCraft: https://bitcraftonline.com/

“4X games” defined: https://en.wikipedia.org/wiki/4X

Plan 9 O.S.: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs

Tyler on LinkedIn: https://www.linkedin.com/in/tylercloutier/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

--

#programming #podcast #softwaredevelopment #software #gamedev #gamedevelopment

Odin’s creator, Bill Hall, makes some bold claims about the language, including that it’s “programming done right”. Before that starts a war on the internet, we’d best ask him to explain what that means, and how Odin tries to achieve it. And while we get deep into the details, overall his answer seems to be, “By gathering masses of feedback and then refining C until it feels joyous again.

Of all the C-like languages we’ve looked at on Developer Voices, Odin seems to be the most at-ease with its progenitor. It’s not trying to be a revolutionary new way of thinking about systems programming; it’s just trying to rethink C for modern conventions. If Bill’s hit his goals, it might be the most comfortable way to get a language that’s C, but C done better…

–

Odin: https://odin-lang.org/

Odin Packages: https://pkg.odin-lang.org/

Newsqueak [pdf]: https://swtch.com/~rsc/thread/newsqueak.pdf

EmberGen: https://jangafx.com/software/embergen/

Raylib: https://www.raylib.com/

RayLib bindings for Odin: https://github.com/odin-lang/Odin/tree/master/vendor/raylib

Verse language: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-reference

Algorithms + Data Structures = Programs: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs

Bill on Twitter: https://twitter.com/TheGingerBill

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#podcast #software #softwareprogramming #programming #odin #odinlang

This week’s guest describes Event Sourcing as, “all I’m going to use for the rest of my career.” But what is Event Sourcing? How should we think about it, and how does it encourage us to think about writing software?

In this episode we take a close look at systems designed around the idea of Events, with guest Bobby Calderwood. Bobby’s been designing (and helping others design) event based architectures for many years, and enthusiastically recommends it not only as a system-design technique, but as a way of solving business problems faster and more reliably.

During this discussion we look at the various ways of defining event systems, what tools we need to implement them, and the advantages of thinking about software from an event-based perspective. Along the way we discuss everything from Clojure, Bitemporality & Datomic to Kafka and more traditional databases - all in the service of capturing real-world events and building simple systems around them.

–

EventStoreDB: https://developers.eventstore.com/

The CloudEvents standard: https://cloudevents.io/

Datomic: https://www.datomic.com/

Adam Dymitruk’s Event Modelling Explanation: https://eventmodeling.org/

Bobby’s Event Modelling course: https://developer.confluent.io/courses/event-modeling/intro/

Bobby on Twitter: https://twitter.com/bobbycalderwood

Boddy on LinkedIn: https://www.linkedin.com/in/bobbycalderwood/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #softwarepodcast #programming #eventsourcing #eventdrivenarchitecture #kafka

One of our oldest languages meets one of our newest sciences in this episode, as we talk with Professor Christian Schafmeister, an award-winning nanotech researcher who's been developing a language and a design suite to help research the future molecular machines.

In this episode Christian gives us a quick chemistry lesson to explain what his research is trying to achieve, then we get into the software that's doing it: A new flavour of Common Lisp. But why Lisp? What advantages does a 60 year old language design offer? How does he strike a balance between high-level language features and the need for exceptional performance and parallelism?  And what tricks does his development environment have that modern IDEs could still learn a thing or two from?

--

Clasp (the Lisp): https://github.com/clasp-developers/clasp

Cando (the design language): https://github.com/cando-developers/cando

The Feynman Prize: https://en.wikipedia.org/wiki/Feynman_Prize_in_Nanotechnology

Alphafold: https://alphafold.ebi.ac.uk/

More on LEaP: https://ambermd.org/tutorials/pengfei/

Interactive Development of Crash Bandicoot: https://all-things-andy-gavin.com/2011/03/12/making-crash-bandicoot-gool-part-9/ 

Christian's Research Group: https://www.schafmeistergroup.com/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#programming #software #lisplang #commonlisp #nanotech

Sometimes, what a programming language makes harder is just as important as what it makes easier. For a simple example, think of GOTO. We’ve been wisely avoiding it for decades because it makes confusing control flow desperately easy. Types and tests are other examples - they’re as much about specifying what shouldn’t work as what should. And perspective is what makes this week’s topic particularly interesting: Roc is a language that’s functional, fast, friendly, and extremely interested in making your life easier by enabling some possibilities and restricting others.

So this week we’re joined by Richard Feldman, the creator of Roc. He’s been an advocate of the Elm programming language for years, for its tight focus on taking the best bits of Functional Programming to the browser. And in recent years he’s been inspired to build his own language, taking that philosophy to other places and platforms.

But which bits are “the best bits”? And how do they change when the domain you’re coding for changes? How is Roc built and how would we build systems in it? Let’s find out…

--
Roc’s homepage: https://www.roc-lang.org/

Richard’s GOTO Copenhagen 2021 talk: https://www.youtube.com/watch?v=3n17wHe5wEw

Richard on Twitter: https://twitter.com/rtfeldman

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

Kris on Twitter: https://twitter.com/krisajenkins

One of the recurring themes in the big data & data streaming worlds at the moment is developer experience. It seems like every major tool is trying to answer this question: how do we make large-scale data processing feel trivial?

In some places the answer is any library you like as long as it’s Python. In other realms, a mixture of Java and SQL shows promise. But as this week’s guest—Luca Pette—would say, the Unix design metaphor has plenty to give and keep on giving.

So in this episode of Developer Voices we look at TypeStream - his Kotlin project that provides a shell-like interface to data pipelines, and is gradually expanding to make integration pipelines as simple as `cat /dev/kafka | tee /dev/postgres`.

--

Luca on Twitter: https://twitter.com/lucapette

Luca on LinkedIn: https://www.linkedin.com/in/lucapette/

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

TypeStream homepage: https://www.typestream.io/

TypeStream installation guide: https://docs.typestream.io/tutorial/installation

Crafting interpreters: https://craftinginterpreters.com/

…by Bob Nystrom: https://twitter.com/munificentbob

NuShell: https://github.com/nushell/nushell

#podcast #apachekafka #bigdata

This week we're back on systems programming with Hare. A C-like language for the ages. We talk to its creator, Drew DeVault, about what he thinks we can learn from the past 50 years of programming, and how we can build that hindsight into a new language that will last for the next 100. 

In among all that long-term ambition we talk cover everything from error handling, typed unions and linear types, to metaprogramming and Drew's microkernel operating system. It's called Ares, and it is, of course, built in Hare.

--

Drew's Homepage: https://drewdevault.com/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ 

A summary of Hare’s features: https://harelang.org/tutorials/introduction/

Hare Community Resources: https://harelang.org/community/

SXMO Mobile: https://sxmo.org/

QBE Compiler Backend: https://c9x.me/compile/users.html

Ares OS Source Code: https://sr.ht/~sircmpwn/helios/

OSDev Wiki: https://wiki.osdev.org/Expanded_Main_Page

The Ares System [pdf]: https://mirror.drewdevault.com/ares.pdf

#programming #podcast #harelang #qbe #microkernel

A few months ago, Michael Drogalis quit his job and decided launch 4 viable startup business ideas in 4 months, publically documenting every step of the journey. Over here at Developer Voices it seemed fun, inspired, and just crazy enough to work.

We had him on the podcast a few months back just as that journey was beginning, and since he launched his first startup things have changed,. The reception has been better than he expected and the plan has been updated to go all-in on idea number one. But why? What's changed? What happened between brainstorming 4 ideas and launching #1 into the world? How is he figuring out what problems to solve, and how is he coping with the workload of being a solopreneur with a business idea and only himself to rely on?

It's definitely time for an update, and to see what we can learn from a fellow geek who wants to start a business, but most of all wants to build technology that people find useful and valuable. Let's hope he succeeds...

--

ShadowTraffic: https://shadowtraffic.io/
Michael’s Previous Appearance: https://youtu.be/jqS2TbxssQE
Follow Michael’s journey: https://michaeldrogalis.substack.com/
Michael on LinkedIn: https://www.linkedin.com/in/michael-drogalis-01029924/
Michael on Twitter: https://twitter.com/MichaelDrogalis
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#podcast #softwareprogramming #programming #startup #technology #kafka #postgres #shadowtraffic #entrepreneur

Integration is probably the last, hardest, and least well thought-out part of any large software project. So anything that makes the data-streaming job easier is worth knowing about. So this week we turn our attention to Apache Flink, a flexible system for grabbing, transforming and shipping data between systems using Java, Python or good ol’ SQL.

So this week Robert Metzger—Apache Flink expert and PMC member—joins us to explain what problems Flink solves and how it solves them reliably. We cover the range from simple use cases to realtime aggregations & joins to its high availability strategy.

If you’re working on systems that include more than one database, then you’re definitely going to face the kinds of problems that Flink tackles.

--

Apache Flink: https://flink.apache.org/
Robert on Twitter: https://twitter.com/rmetzger_
Robert on LinkedIn: https://www.linkedin.com/in/metzgerrobert/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #programming #podcast #flink #apacheflink #dataintegration

Zig is a programming language that’s attempting to become “the new C” - the language of choice for low-level systems programming and embedded hardware. Going into that space not only puts it in competition with C and C++, but also other newcomers like Rust and Go. So what makes Zig special?

Joining us to discuss it is Loris Cro from the Zig Foundation. We talk through Zig’s reasons to exist, its language design features, which parts of the C ecosystem it's tackling, and how the Zig Foundation is set up for the long-term health of the language.

–

Loris’ website: https://kristoff.it/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Zig homepage: https://ziglang.org/
The “learn zig” guide: https://ziglearn.org/
Learn Zig with Ziglings: https://ziglings.org/
Find the Zig community: https://github.com/ziglang/zig/wiki/Community
Rust’s cargo-zigbuild: https://github.com/rust-cross/cargo-zigbuild
Using zig as a better linker: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html
"The Economics of Programming Languages" by Evan Czaplicki (Strange Loop 2023) - https://youtu.be/XZ3w_jec1v8

–

#programming #programminglanguages #software #zig #llvm #rust #go

Would you ever take on a rewrite of one of the largest and most popular Apache projects? And if so, what would you keep the same and what would you change?

This week we’re talking to Christina Lin, who’s part of Redpanda, a company that’s rewriting parts of the Apache Kafka ecosystem in C++, with the aim of getting performance gains that aren’t feasible in Java. It seems like a huge mountain to climb, and a fascinating journey to be on, so let’s ask why and how they’ve taken on this challenge…

Christina on Twitter: https://twitter.com/Christina_wm
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Redpanda: https://redpanda.com/
Redpanda University: https://university.redpanda.com/
Seestar framework: https://seastar.io/
Apache Flink: https://flink.apache.org/

#redpanda #kafka #apachekafka #streaming #python

This week we’re looking at Debezium - an open source project that taps into a huge number of databases and lets you stream data to other systems in real time. It’s a huge project that covers a wide range of uses: Some people use it to replicate from Oracle to MySQL, others to do smart cache invalidation, and others to build a bridge from an existing relational database to the event-sourcing world. If you’re working on a system that has more than one kind of database, it may be an essential tool. But what exactly does it do, and how does it do it?

Joining us for a deep dive is Debezium expert and former project lead, Gunnar Morling. He takes us through all things Debezium, from who uses Debezium and why; which systems you can connect to and what data you get out from it; and how a project of this scope is developed.

Gunnar on Twitter: https://twitter.com/gunnarmorling
Gunnar on LinkedIn: https://www.linkedin.com/in/gunnar-morling-2b44b7229/
Gunnar’s blog: https://www.morling.dev/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Gunnar’s talk on the new incremental snapshot mechanism: https://events.bizzabo.com/468544/agenda/session/1136877
Getting into Real Time data: https://youtu.be/ftAVFxa5AwI
Stripe’s talk at Flink Forward: https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture
Francesco’s “When JDBC Goes Wrong” talk: https://www.confluent.io/events/kafka-summit-london-2022/jdbc-source-connector-what-could-go-wrong/

#debezium #microservicesarchitecture #cdc #database #postgresql #oracle #mysql #eventsourcing #kafka #jdbc #realtime

Ever read a bad README? We all have, and most of the time, we’ve just moved right along. A programmer that can’t communicate their ideas will find no-one uses their software. And that’s true even outside of the open-source world. The best software doesn’t win - the best software _that people can understand_ wins. So how do we get better at communicating our code? What do we talk about when we talk about software?

Joining to discuss that question is a data-streaming expert and skilled communicator, Francesco Tisiot. Unusually, this episode is recorded on location, as we met up in the hallway of a recent tech conference.

Francesco on Twitter: https://twitter.com/FTisiot
Francesco on LinkedIn: https://www.linkedin.com/in/francescotisiot/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

#podcast #podcasts #devrel #opensource #software #presentations

As interesting and useful as LLMs (Large Language Models) are proving, they have a severe limitation: they only know about the information they were trained on. If you train it on a snapshot of the internet from 2023, it’ll think it’s 2023 forever. So what do you do if you want to teach it some new information, but don’t want to burn a million AWS credits to get there?

In exploring that answer, we dive deep into the world of semantic search, augmented LLMs, and exactly how vector databases bridge that gap from the old dog to the new tricks. Along the way we’ll go from an easy trick to teach ChatGPT some new information by hand, all the way down to how vector databases store documents by their meaning, and how they efficiently search through those meanings to give custom, relevant answers to your questions.

--

Zain on Twitter: https://twitter.com/zainhasan6
Zain on LinkedIn: https://www.linkedin.com/in/zainhas
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
HNSW Paper: https://arxiv.org/abs/1603.09320
ImageBind - One Embedding Space To Bind Them All (pdf): https://openaccess.thecvf.com/content/CVPR2023/papers/Girdhar_ImageBind_One_Embedding_Space_To_Bind_Them_All_CVPR_2023_paper.pdf
Weaviate: https://weaviate.io/
Source: https://github.com/weaviate/weaviate
Examples: https://github.com/weaviate/weaviate-examples
Community Links: https://forum.weaviate.io/ and https://weaviate.io/slack

--

#vectordb #vectordatabase #semanticsearch #openai #chatgpt #weaviate #knn

Real-time data is gradually becoming a standard requirement in systems design. Our customers are beginning to demand it, our colleagues in other departments are starting to expect it. Whether you’re letting people book a taxi, recommending their next binge-watch, or delivering business reports to management, faster data is just obviously better. Or is it?

Does real-time data matter everywhere, or does it just have sweet spots in some sectors and some use-cases? Is it a cost-benefit question - is the idea great in theory, but still too hard to adopt in practice? Would everyone be streaming their data live if streaming their data live was easier? If the future of data is, “now, not later,” then what’s holding that future back?

In this week’s Developer Voices we talk to Thomas Camp of Ably, and chew through the use-cases, software stacks, and education needed to speed up the way we process data. We consider everything from the front-end to the back, from user experience to business needs, and from greenfield projects to incrementally adapting existing systems.

If you’re wondering what all the batch vs. streaming fuss is about, or you want to know how you can drag the industry there sooner, we have some answers. It’ll only take an hour. 😉

--

Thomas on LinkedIn: https://www.linkedin.com/in/thomascamp333/
Ably: https://ably.com/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

How far would you go to get the kind of database you want? How deep into the stack would you dive to re-architect a system for the kind of performance, reliability and scale you believe in? Today's guest has decided to go all in, as he’s tackling the database problem from the fsync up.

In this week’s Developer Voices we talk to Joran Dirk Greef, whose ambitions—combined with the lacklustre performance of his project's payment system—have led him to build a new database called TigerBeetle, that tackles some meaty problems. They’re attempting to build a database that can be durable in the face of fsync-corner cases, highly available in the face of all kinds of hidden network problems, and performant enough to outpace existing financial systems. And on top of all those goals, they’re doing it with an interesting new language you may not have heard of - Zig.

What makes him want to take on this big a challenge? What problems keep him awake at night? And what is he doing to turn all that ambition into an achievable launch strategy? Listen on and find out…

–

TigerBeetle on Twitter: https://twitter.com/TigerBeetleDB
TigerBeetle on YouTube: https://www.youtube.com/channel/UC3TlyQ3h6lC_jSWust2leGg
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Joran’s QCon ‘23 Talk: https://www.youtube.com/channel/UC3TlyQ3h6lC_jSWust2leGg
Viewstamped Replication Revisited (paper): https://pmg.csail.mit.edu/papers/vr-revisited.pdf
Github Test Cases for Journal recovery code: https://github.com/tigerbeetle/tigerbeetle/blob/b4dd441502894cbe9d48cb90ff0bc6a12c378591/src/vsr/journal.zig#L1181-L1213MySQL transactions per second vs fsyncs per second: https://sirupsen.com/napkin/problem-10-mysql-transactions-per-second

Ever wanted to start a tech business? Michael Drogalis has done it successfully in the past, and now he’s trying again (and again and…) as he makes a very public attempt to start 4 new tech businesses in the next 4 quarters.

He’d sound completely mad, except he’s got form: His last Kafka-based company got bought out by a tech giant, giving him enough of a safety net to try something new. And for his new approach, he’s doing the exact opposite of ‘stealth mode’. He’s publishing every step of his 4-by-4 challenge, wins and losses, for all to see. It's entrepreneurship for the Reality TV era. 😁

In an unusually vulnerable episode of Developer Voices, we talk about everything from solo software development and marketing strategies, to the inner struggle of starting out on your own, with no guarantees that the world will care. And this time with everyone watching.

--

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Follow Michael’s journey: https://michaeldrogalis.substack.com/
Michael on LinkedIn: https://www.linkedin.com/in/michael-drogalis-01029924/
Michael on Twitter: https://twitter.com/MichaelDrogalis
Steal Like An Artist by Austin Kleon: https://austinkleon.com/steal/

Ask any software developer about their favourite stack, and they'll probably have strong opinions. (They might even have a snappy acronym for it, like LAMP or JAM or...) But if you ask any business about their stack, things aren’t so clear. They'll probably tell you the key components, and then add a laundry list of extra parts that got glued on later. The reality is, assembling large systems gets complicated. And with more and more options coming out every year, that potential complexity is only growing. These days, a good software developer must also be part researcher, and part explorer. 

So this week we're going to survey the landscape, and head out on the hunt for workable architectures. And joining us to do it is Ben Gamble. He's worked in fields as diverse as video games, enterprise software and AR headsets, and these days works for an Everything As A Service company helping people find and assemble the missing pieces their system needs. Who better to guide us through the software landscape? 

If you've ever worried that your system is too complicated, or is missing important parts, or you're just yearning for an architecture that feels more cohesive and less like a box of parts, come exploring with us.

--

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Ben on Twitter: https://twitter.com/BenGamble7
The Schemaverse: https://schemaverse.com/
Nerdsniping: https://xkcd.com/356/
Tremor: https://www.tremor.rs/
Anna McDonald’s talk on Completion Criteria: https://www.confluent.io/events/kafka-summit-london-2023/pragmatic-patterns-and-pitfalls-for-event-streaming-in-brownfield/
Temporal.IO: https://temporal.io/

In modern systems, the amount of data keeps getting larger, and the time available keeps getting shorter. So it's almost inevitable that we're augmenting our general-purpose databases with dedicated analytics databases.

This week we dive into the world of OLAP with a thorough look at Clickhouse, a high-performance, columnar database designed to "query billions of rows in microseconds."

Alasdair Brown joins us to discuss what Clickhouse is, how it performs queries so quickly, and where it fits into a wider system. We talk about its origins as a Google Analytics-like, and how it's grown into one of the most popular OLAP databases around.

There's a lot of ground to cover, and a lot of questions to ask, all in the service of faster answers...

--

Alasdair's Blog: alasdairb.com
Alasdair on Threads: https://www.threads.net/@sdairsab
Alasdair on LinkedIn https://www.linkedin.com/in/alasdair-brown
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Clickhouse: https://clickhouse.com/
Tinybird: https://www.tinybird.co/
Birdhouse in your Soul: https://youtu.be/vn_or9gEB6g

Neil Buesing is a seasoned Apache Kafka® user, and a respected voice from the Kafka community who specialises in helping companies make the best use of Kafka. And that makes him the ideal person to ask the $64,000 question: What problems can Kafka actually solve for me? Because Kafka's definitely interesting, and it can be fun, but to earn a place in the toolbox it has to make life easier. 

In answering that question, Neil covers a tonne of ground, from queuing and quasi-databases, transitioning from batch to real-time, and solving general software integration headaches. 

If you have data problems, big or small, join us to figure out if Kafka is the answer. 

-- 

Kinetic Edge: https://www.kineticedge.io
Neil on Twitter: https://twitter.com/nbuesing
Neil on LinkedIn: https://www.linkedin.com/in/nbuesing/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Building an Enterprise Eventing Framework (Conference Talk): https://www.confluent.io/en-gb/kafka-summit-san-francisco-2019/building-an-enterprise-eventing-framework/
Synchronous Kafka (Conference Talk): https://www.confluent.io/resources/kafka-summit-2020/synchronous-commands-over-apache-kafka/
Dev Local: https://github.com/kineticedge/dev-local
KIP-714 - Client metrics and observability: https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability

How do distributed systems work? If you’ve got a database spread over three servers, how do they elect a leader? How does that change when we spread those machines out across data centers, situated around the globe? Do we even need to understand how it works, or can we relegate those problems to an off the shelf tool like Zookeeper?

Joining me this week is Distributed Systems Doctor—Benjamin Bengfort—for a deep dive into consensus algorithms. We start off by discussing how much of “the clustering problem” is your problem, and how much can be handled by a library. We go through many of the constraints and tradeoffs that you need to understand either way. And we eventually reach Benjamin’s surprising message - maybe the time is ripe to roll your own. Should we be writing our own bespoke Raft implementations? And if so, how hard would that be? What guidance can he offer us? 

Somewhere in the recording of this episode, I decided I want to sit down and try to implement a leader election protocol. Maybe you will too. And if not, you’ll at least have a better appreciation for what it takes. Distributed systems used to be rocket science, but they’re becoming deployment as usual. This episode should help us all to keep up!

--

KubeCon talk on the FCD bug: https://kccncna2022.sched.com/event/182N9/lessons-learned-from-etcd-the-data-inconsistency-issues-marek-siarkowicz-google-benjamin-wang-vmware
The Raft paper by Diego Ongaro and John Ousterhout: https://raft.github.io/raft.pdf
The EPaxos Algorithm: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf
LevelDB: https://github.com/google/leveldb

Benjamin on Twitter: https://twitter.com/bbengfort
Benjamin on LinkedIn: https://www.linkedin.com/in/bbengfort
Benjamin on GitHub: https://github.com/bbengfort
Rotational Labs: https://rotational.io (check out the blog!)
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

How do you get started as a programmer? And how do experienced programmers help them as they start their journey?

This week's guest is a developer-turned-teacher, James Q. Quick. A former coder and developer advocate, he's s been been working on tutorials, courses and bootcamps to teach aspiring developers how to get started with JavaScript. We talk about why people get into programming as a new career, what they need to succeed, and what James thinks is the best indicator of success.

We also discuss what's new and exciting in JavaScript, James' favourite up & coming libraries, and dive into the JavaScript vs. TypeScript debate.

James’s YouTube channel: https://www.youtube.com/c/jamesqquick
James’s website: https://www.jamesqquick.com/
Astro website builder: https://astro.build/
James’ Astro course: https://astrocourse.dev/
Svelte framework: https://svelte.dev/
Vite build tool: https://vitejs.dev/

James on Twitter: https://twitter.com/jamesqquick
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Sooner or later, every programmer will have to cross the gap between their programming language and their database. It feels like it should be easy, but in practice it’s always a much wider chasm than it seems, and every tool that bridges that gaps comes with its own strengths, weaknesses and opinions.

This week we take a look at a relatively new library for database access—jOOQ—by chatting with its author, Lukas Eder. This episode takes in the simple questions like syntax, the thornier ones like supporting multiple databases, and the deeply philosophical ones like how we even think about data and data-processing.

If you’re a Java (or JVM) programmer, there’s a new tool to learn here, and even if you're not there’s food for thought and ideas to borrow for the next time you need to SELECT…

jOOQ: https://www.jooq.org/
YesQL: https://github.com/krisajenkins/yesql
Datomic: https://www.datomic.com/
XTDB: https://www.xtdb.com/
The Elm Architecture: https://guide.elm-lang.org/architecture/

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Whether you’re trying to ace the coding interview, sharpen your programming skills or just have some fun learning new things, the world of competitive coding has something to offer you. Some people join with dreams of hitting the podium, and plenty of others are just competing to be their better selves.

Either way, Mathis Hammel is a veteran of the competitive coding scene and he’s going to give us a view into that world, tell a few war stories and share some tips how you can play better, faster and stronger…

ICPC: https://icpc.global/
Advent of Code: https://adventofcode.com/
Advent of Code, Day 18, 2022: https://adventofcode.com/2022/day/18
Clash of Code: https://www.codingame.com/multiplayer/clashofcode
CodeForces: https://codeforces.com/
CodeWars: https://www.codewars.com/
HackerRank: https://www.hackerrank.com/

Mathis on Twitter: https://twitter.com/mathishammel
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

“Software development has not caught up with the internet age.” So says this week’s guest, Rúnar Bjarnason. But what does that mean? What would a programming language for the internet age look like?

Rúnar’s answer is Unison. A language that completely rethinks the way distributing computing can work, from the source code up. Borrowing some key ideas from git, it challenges the way we think about code-sharing, compilation, versioning and more. 

--

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Rúnar on Twitter: https://twitter.com/runarorama
Rúnar’s book, Function Programming in Scala: https://amzn.to/46I9jew
Unison website: https://unison-lang.org

Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism (pdf): https://www.cl.cam.ac.uk/~nk480/bidir.pdf
Do Be Do Be Do (pdf): https://arxiv.org/pdf/1611.09259.pdf
Rúnar’s Øredev conference talk: https://youtu.be/EgIVzOobD48

Cloud icons created by Freepik - Flaticon: https://www.flaticon.com/free-icons/cloud
Computer icons created by xnimrodx - Flaticon: https://www.flaticon.com/free-icons/computer

Have you ever been overwhelmed by the number of databases on offer? This week we welcome database expert Ben Stopford as a guide to help us map the database landscape and make sense of it all!

Join us as we embark on a journey through the history of databases, tracing the path from Edgar Codd to the multitude cloud-era of options available today. Discover the strengths of various database styles and explore the tradeoffs between general-purpose databases like #PostgreSQL and highly customised ones like #Cassandra or #Snowflake.

We delve into the realm of the cloud and the opportunities it brings, both for users and the database vendors themselves. And then we examine the challenges that arise when you're forced to connect multiple databases across an organisation. Should you look at Event Sourcing? Or Event Streaming, and how exactly do they differ?

Finally, we look towards the future, discussing Ben's vision of an ideal database and which programming language he would choose to build it in.

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Mastodon: https://mastodon.social/@krisajenkins

LLMs like ChatGPT are not just fascinating, they're becoming increasing useful in our working lives. They've graduated from novelty to valuable tool. But building those tools is still in the hands of huge companies. Or is it?

In this week's episode of Developer Voices, we're learning how you can run LLMs on your own laptop, and how you can customize the system to make a tailored research assistant, a better documentation-searcher, and much more. All you need is a guide on which pieces you need, and how they fit together, and that's exactly what this week's guest—Tobi Fankhänel—is here to take us through.

A leaked memo from Google recently outlined how the Big Company Advantage has almost completely eroded, and how the next wave of LLM development is going to come from the open source community. So hackers rise up - the open source AI revolution begins now!

--

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Tobias on LinkedIn: https://www.linkedin.com/in/tobias-fankh%C3%A4nel-749712180/
Tobias’ blog: https://blog.exxample.eu
LangChain: https://python.langchain.com/docs/get_started/introduction.html
Embeddings: https://weaviate.io/blog/vector-embeddings-explained
Vector Databases: https://en.wikipedia.org/wiki/Vector_database
"We have no moat" – Google Employee on Open-source LLMs: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
“Attention is all you need” - https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Timeline since Meta open-sourced their first-gen models: https://www.semianalysis.com/i/119223672/the-timeline
Run LLMs on CPU only or, since May, mix CPU and GPU usage: https://github.com/abetlen/llama-cpp-python
Samantha: https://erichartford.com/meet-samantha
Embedding model leaderboards: https://huggingface.co/spaces/mteb/leaderboard
Open-source LLMs: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/
Blog post: Design-pattern ‘In-context learning’ https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/#section--2
Tobi's GitHub branch ‘In-context learning with LangChain’ https://github.com/aviav/turmbauten/blob/spaghetti-code/CHANGELOG.md
Prompt Syntax Cheat Sheet: https://github.com/oobabooga/text-generation-webui/tree/main/characters/instruction-following
Google Workspace Labs Sign-Up: https://workspace.google.com/labs-sign-up/
GMail Workspace Labs Demo Video, click ‘See it in action’: https://workspace.google.com/solutions/ai/#m10
Prediction trading on open-source LLMs vs GPT-4: https://manifold.markets/PeterWildeford/will-i-peter-wildeford-think-that-t-c95ff3c1b385

Time to put another new #programming language - and its creator - under the spotlight, as we talk to Robin Heggelund Hansen, the creator of Gren. Gren is a Norwegian word meaning 'branch', which is appropriate for a language that started as a fork from its roots in Elm.

With Gren, Robin's trying to create a safe, sane, #fullstack language that puts the power and elegance of functional programming working seamlessly on the server and the browser. But how and why do you do that? What design choices to do make, how much time do you spend on adding new features vs. improving the developer's experience of the existing ones? And most importantly...how do you pronounce Gren correctly? 😅

Gren Website: https://gren-lang.org
Gren Zulip: https://gren.zulipchat.com/
Gren package site: https://packages.gren-lang.org/
Example projects written in Gren: https://github.com/gren-lang/example-projects
Gren on Mastodon: https://fosstodon.org/@gren_lang
Gren on Twitter: https://twitter.com/gren_lang
Gren language proposal on parameterized modules: https://github.com/gren-lang/compiler/issues/81
Elm Website: https://elm-lang.org/  

Robin on Mastodon: https://snabelen.no/@robinheghan
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Mastodon: https://mastodon.social/@krisajenkins

What's going on with Postgres? Joining us for his perspective is Raouf Chebri, a Developer Advocate for PostgreSQL and the cloud service Neon. We catch up on what's new and important in recent versions of Postgres, what Neon have been doing to make Postgres work well in the Cloud, and what Raouf's life is like as a professional singer of Postgres's virtues.

Neon: https://neon.tech/
Neon’s architecture: https://neon.tech/docs/introduction/architecture-overview
That CEO quote: https://twitter.com/nikitabase/status/1563913187862335489
Raouf on Twitter: https://twitter.com/raoufdevrel
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Are you trying to build the community around your software? And what does “building a community” really mean? What are we building communities for?

Join us on Developer Voices as host Kris Jenkins sits down with Ale Murray, a seasoned community manager with nearly a decade of experience, to discuss her tips for building a thriving tech community. 

Ale shares her insights on why community building is essential, how to identify your target audience, and how to approach community building with the right mindset. She also offers practical advice on how to handle challenging situations, such as dealing with negative feedback and managing conflicts within the community. 

Whether you're just starting out or looking to improve your existing community, this conversation offers valuable insights and actionable tips to help you succeed.

Ale on Twitter: https://twitter.com/ale_amurray
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins
Kris on Mastodon: https://mastodon.social/@krisajenkins

As a developer, it's crucial to understand the various types of databases available so you can choose the right tool for the job. In this episode, we're shining a spotlight on bitemporal databases with James Henderson, lead developer of of a new bitemporal database called XTDB.

You may have already created an ad-hoc bitemporal database without realizing it, but James and his team have been hard at work building a custom database that's tailor-made for situations where having two notions of time are essential. Join us to learn about the what and why of bitemporality and explore the process of designing and building a database in Clojure.

Ready to get started with XTDB? Visit https://www.xtdb.com/v2 to learn more.
Want to get involved with the XTDB community? Head over to https://discuss.xtdb.com.
Follow XTDB on Twitter at https://twitter.com/xtdb_com
 and Kris Jenkins at https://twitter.com/krisajenkins.
Connect with Kris on LinkedIn at https://www.linkedin.com/in/krisjenkins/.

Every business deals in data, but the internet age has ushered in an explosion of the number of different data formats we have to process from a number of different databases to get the job done. Ask any Data Scientist, and they’ll tell you a huge part of their job isn’t data science - it’s data engineering. Acquiring, processing and shipping data, and above all, understanding it. We’ve always had that task in the IT world, but these days it’s a whole discipline.

Today’s podcast sits down with the author of “The Fundamentals Of Data Engineering” to survey the landscape. To understand what data engineering is, and go deep into what a data engineer needs to understand to succeed.

In this episode we're exploring the Kotlin programming language with an expert from Google. What does Kotlin have to offer? Is it just a Java alternative, or has it turned into something much more ambitious? Along the way we manage to discuss Scala, Function Programming vs. Object Orientation, Editors, Higher Kinded Types, Elm, React, UI architectures and multiplatform programming. A very full episode!

Happy Path Programming - Haskell is not as scary as you think: https://pod.link/1531666706/episode/710a605d605fda251f2e83b8858615b8
The Kotlin Programming Language: https://kotlinlang.org/
Android Studio: https://developer.android.com/studio
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

AI is the new hotness, but with new approaches come new problems of scale. How do you make raw data accessible to teams of data scientists and model-builders? How do build a repeatable deployment pipeline? And how do you monitor your models once they're in production? Today's episode goes through all these reality checks with author and expert Adi Polak, who's just released a new book on the subject.

Scaling Machine Learning With Spark (book): https://www.oreilly.com/library/view/scaling-machine-learning/9781098106812/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Louis Pilfold joins us to talk about Gleam, a new language that runs on Erlang's BEAM. In creating Gleam, Louis has tried to bring a new level of developer-friendliness to Erlang's famously fault-tolerant platform.

If you're interested in building reliable distributed systems, exploring functional programming, learning new languages or finding out what it takes to design your own language, Louis is here to talk us through it all.

The Gleam language: https://gleam.run/
Gleam’s package repo: https://packages.gleam.run/
Gleam on Twitter: https://twitter.com/gleamlang
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/