Open Source Archives – Software Engineering Daily

The shift to microservices architectures and distributed systems has been a challenge for systems using conventional security practices, such filtering IP addresses using network policies. In addition, the increasing intersection of development and operations exemplified by the DevOps methodology has expanded the scope responsibilities in implementing secure systems. 

Part of CNCF, SPIFFE is a set of open-source specifications for issuing identity to services in heterogenous, distributed environments such as a cloud-native microservices architecture. Systems implementing SPIFFE bypass the need for application-level authentication and network-level ACL configuration. SPIRE, or the SPIFFE Runtime Environment, is a system that implements the SPIFFE standards to manage platform and workload attestation, providing an API for controlling policies, and coordinating certificate issuance and rotation.

Derek Edwards is the head of engineering at Anthem.ai, and Ryan Turner is a software engineer at Uber. They join the show today to talk about the challenges of managing security in a distributed system, how adopting SPIFFE represented a paradigm shift in their authentication workflow, and how the SPIFFE and SPIRE projects are evolving to meet the needs of the next generation of cloud-native applications. HPE sponsored this podcast.

Sponsorship inquiries: [email protected]

The post SPIFFE and SPIRE with Derek Edwards and Ryan Turner appeared first on Software Engineering Daily.

A data warehouse is a centralized repository that an enterprise may use to store selected data from production systems. Data is transformed into a structured form that makes it easily accessible for business intelligence or other operational users. SQL-compliant databases are frequently used for data warehouses due to the popularity of SQL as a tool in business data analytics.

PostgreSQL is a free and open-source relational database management system. Postgres-based databases are widespread and are used by a variety of organizations, from Reddit to the International Space Station, and Postgres databases are a common offering from cloud providers such as AWS, Alibaba Cloud, and Heroku. 

Josh Drake and Thomas Richter are experts on Postgres data warehousing. They join the show today to talk about the staying power of Postgres, why Postgres is a good choice for data warehousing, and how cloud technology is changing relational database management systems.

Sponsorship inquiries: [email protected]

The post Postgres Data Warehouse with Josh Drake and Thomas Richter appeared first on Software Engineering Daily.

Open source software is software distributed along with its source code, using a permissive license that allows anyone to view, use, or modify it. The term “open source” also refers more broadly to a philosophy of technology development which prioritizes transparency and community development of a project. Typically, development is managed by a governing body, whether a company, foundation, or just a group of passionate users, and work is done in public repositories like Github. Nearly every corner of the software engineering world has been impacted in some way by open source. Well-known open source projects include Linux, Kubernetes, and WordPress.

Kevin Xu is the author of Interconnected, a bilingual newsletter on tech, business, and U.S-China relations. He is an investor in open source startups at OSS Capital, and formerly served in the Obama White House. He joins the show today to talk about the benefits of open source in the public and private sectors, and how open source will be critical to the development of high-tech industry in our country as we pivot to facing some of the 21st century’s most pressing challenges.

Sponsorship inquiries: [email protected]

The post Open Source Industrialization with Kevin Xu appeared first on Software Engineering Daily.

Originally published October 18, 2019

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics.

Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn.

Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management.

Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

Sponsorship inquiries: [email protected]

The post LinkedIn Kafka with Nacho Solis (Repeat) appeared first on Software Engineering Daily.

Originally published September 17, 2019

Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering.

Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that multiple systems can consume the messages from a single sender. 

In previous shows, we have covered design patterns within Kafka, Kafka streams, event sourcing with Kafka, and many other subjects relating to the technology. Kafka is broadly useful, and new strategies for using Kafka continue to emerge as the open source project develops new functionality and becomes a platform for data applications.

In today’s episode, Tim Berglund returns to Software Engineering Daily for a discussion of how applications are built today using Kafka–including systems that are undergoing a refactoring, data engineering applications, and systems with a large number of communicating services.

If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.

Sponsorship inquiries: [email protected]

The post Kafka Applications with Tim Berglund (Repeat) appeared first on Software Engineering Daily.

Originally published May 16, 2019

React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native).

React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks.

After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components.

After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project.

The post Facebook React with Dan Abramov (Repeat) appeared first on Software Engineering Daily.

Business intelligence is crucial for both internal and external applications at any company. There is a wide array of proprietary BI tools. Today, there is an increasing number of options for open source business intelligence, one of which is CubeJS. CubeJS is an open source analytical API platform for building BI. Artyom and Pavel from CubeJS join the show to talk about what they have built and their vision for the platform.

Sponsorship inquiries: [email protected]

The post CubeJS with Artyom Keydunov and Pavel Tiunov appeared first on Software Engineering Daily.

Infrastructure at Spotify runs at high speeds. Developers work autonomously, building and deploying services all the time. Backstage is an open source platform built at Spotify that allows developers to build portals for making sense of their infrastructure. Backstage developer portals are powered by a central service catalog, with centralized services and streamlined development. Stefan Alund joins the show to explain how Backstage works and their role in developing it.

Sponsorship inquiries: [email protected]

The post Backstage: Spotify Developer Portals with Stefan Ålund appeared first on Software Engineering Daily.

Firebase is well-known as a platform that makes it easy to build real-time applications quickly and easily. Firebase was acquired by Google, and has been turned into a large platform that runs on top of Google Cloud. Firebase is closed-source, which leads to a different ecosystem than open source platforms. Supabase is a new open source alternative to Firebase, built on Postgres and Elixir. Paul Copplestone is the founder of Supabase and he joins the show to talk through what he is building.

Sponsorship inquiries: [email protected]

The post Supabase: Open Source Firebase with Paul Copplestone appeared first on Software Engineering Daily.

Business intelligence tooling allows analysts to see large quantities of data presented to them in a flexible interface including charts, graphs, and other visualizations. BI tools have been around for decades, and as the world moves towards increased open source software, the business intelligence tools are following that trend.

Metabase is an open source business intelligence system that has been widely adopted by enterprises. It includes all the common tools that are expected from a business intelligence system: large-scale data ingestion, visualization software, and a flexible user interface.

Sameer Al-Sakran is the CEO of Metabase and he joins the show to talk about Metabase’s design, engineering, and usage.

Sponsorship inquiries: [email protected]

The post Metabase: Business Intelligence Open Source with Sameer Al-Sakran appeared first on Software Engineering Daily.

Dev.to has become one of the most popular places for developers to write about engineering, programming languages, and everyday life. For those who have not seen it, DEV is like a cross between Twitter and Medium, but targeted at developers. The content on DEV ranges from serious to humorous to technically useful.

DEV contains a set of features which appeal to a developer community, such as the ability to embed code snippets in a post, but for the most part the entire app is generalizable to other types of communities. Hence, the motivation for “Forem”. Forem is an open source project to make it possible to spin up instances of communities that are like DEV, but for other communities such as mixed martial arts, or doctors.

Ben Halpern is the creator of DEV and Forem, and he joins the show to talk about the DEV Community and his long-term goals for what the DEV team is building.

Sponsorship inquiries: [email protected]

The post DEV and Forem with Ben Halpern appeared first on Software Engineering Daily.

WordPress has been a dominant force in the world of online publishing for many years because of how battle-tested it is. WordPress is the definitive leader in CMS technology. But there have always been alternatives. 

Drupal, Ghost, and other open source CMSes. More recently, there has been an emergence of the headless CMS, such as Contentful, which decouples the CMS backend from the frontend presentation layer.

Strapi is a popular open source headless CMS. Pierre Burgy is the founder of Strapi, and he joins the show to talk about the CMS category, the role that Strapi fills, and the technology behind Strapi.

Sponsorship inquiries: [email protected]

The post Strapi: Headless CMS with Pierre Burgy appeared first on Software Engineering Daily.

Originally published July 25, 2019. We are taking a few weeks off. We’ll be back soon with new episodes.

Envoy is an open source edge and service proxy that was originally developed at Lyft. 

Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio.

Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone.

Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC.

Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.

The post Envoy Mobile with Matt Klein (Summer Break Repeat) appeared first on Software Engineering Daily.

Originally published April 14, 2017. We are taking a few weeks off. We’ll be back soon with new episodes.

Facebook’s open source projects include React, GraphQL, and Cassandra. These projects are key pieces of infrastructure used by thousands of developers–including engineers at Facebook itself.

These projects are able to gain traction because Facebook takes time to decouple the projects from their internal infrastructure and clean up the code before releasing them into the wild. Facebook has high standards for what they are willing to release.

Tom Occhino manages the React team at Facebook and works closely with engineers to determine what projects make sense to open source. In this episode, Preethi Kasireddy interviews Tom about how Facebook thinks about open source–what went right with React, why it makes sense for Facebook to continue to release new open source projects, and how full-time employees at Facebook interact with that open source codebase.

The post Facebook Open Source with Tom Occhino (Summer Break Repeat) appeared first on Software Engineering Daily.

Grafana is an open source visualization and monitoring tool that is used for creating dashboards and charting time series data. Grafana is used by thousands of companies to monitor their infrastructure. It is a popular component in monitoring stacks, and is often used together with Prometheus, ElasticSearch, MySQL, and other data sources.

The engineering complexities around building Grafana involve the large number of integrations, the highly configurable ReactJS frontend, and the ability to query and display large data sets. Grafana also must be deployable to cloud and on-prem environments.

Torkel Ödegaard is a co-founder of Grafana Labs, and joins the show to talk about his work on the open source project and the company he is building around it.

Sponsorship inquiries: [email protected]

The post Grafana with Torkel Ödegaard appeared first on Software Engineering Daily.

Facebook applications use maps for showing users where to go. These maps can display businesses, roads, and event locations. Understanding the geographical world is also important for performing search queries that take into account a user’s location. For all of these different purposes, Facebook needs up-to-date, reliable mapping data.

OpenStreetMap is an open system for accessing mapping data. Anyone can use OpenStreetMap to add maps to their application. The data in OpenStreetMap is crowdsourced by users who submit updates to the OpenStreetMap database. Since anyone can submit data to OpenStreetMap, there is a potential for bad data to appear in the system.

Facebook uses OpenStreetMap for its mapping data, including for important applications where bad data would impact a map user in a meaningfully negative way. In order to avoid this, Facebook builds infrastructure tools to improve the quality of its maps. Saurav Mohapatra and Jacob Wasserman work at Facebook on its mapping infrastructure, and join the show to talk about the tooling Facebook has built around OpenStreetMap data.

Sponsorship inquiries: [email protected]

The post Facebook OpenStreetMap Engineering with Saurav Mohapatra and Jacob Wasserman appeared first on Software Engineering Daily.

Building a game is not easy. The development team needs to figure out a unique design and gameplay mechanics that will attract players. There is a great deal of creative work that goes into making a game successful, and these games are often built with low budgets by people who are driven by the art and passion of game creation.

A game engine is a system used to build and run games. Game engines let the programmer work at a high level of abstraction, by providing interfaces for graphics, physics, and scripting. Popular game engines include Unreal Engine and Unity, both of which require a license that reduces the amount of money received by the game developer.

Godot is an open source and free to use game engine. The project was started by Juan Linietsky, who joins the show to discuss his motivation for making Godot. We have done some great shows on gaming in the past, which you can find on SoftwareDaily.com. Also, if you are interested in writing about game development, we have a new writing feature that you can check out by going to SoftwareDaily.com/write.

Sponsorship inquiries: [email protected]

The post Godot Game Engine with Juan Linietsky appeared first on Software Engineering Daily.

The Changelog is a podcast about the world of open source. As open source has become closely tied with the entire software development lifecycle, The Changelog has expanded its coverage to the broader software industry.

Since starting the podcast ten years ago, Adam Stacoviak and Jerod Santo have become full-time podcasters, and they have started several other podcasts within the Changelog network, including Go Time, JS Party, and Practical AI. Throughout all of their shows, there is a consistent theme of technical, entertaining conversations about software.

In the last decade, so much has changed within open source: GitHub became the de facto social network for open source; Kubernetes created a widely used platform for distributed systems; React has given frontend developers a component system to consolidate around. Adam and Jerod return to the show to discuss their perspective on the past and future of open source, and their learnings from interviewing influential software professionals for 10 years.

Sponsorship inquiries: [email protected]

The post Changelog Podcasting with Adam Stacoviak and Jerod Santo appeared first on Software Engineering Daily.

Originally published January 4, 2019

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging.

API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled.

Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway.

Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.

The post Kong API Platform with Marco Palladino Holiday Repeat appeared first on Software Engineering Daily.

Every company has a software supply chain.

A company builds its products from custom code, paid APIs, paid proprietary binaries, and open source software libraries. As the types of software available have increased, the management of the software supply chain has become complex. 

Large software companies have always needed to ensure the security of their software. With the growing variety of open source licenses, these companies also have to deal with an increased set of legal complexities. If an open source project is used in a way that violates an open source license, the company is subject to legal risk.

FOSSA is a company that focuses on automating the management of open source compliance and security. Kevin Wang is the CEO of FOSSA, and he joins the show to discuss the modern issues of software licensing and his experience building a company.

Sponsorship inquiries: [email protected]

Check out our active projects:

• We are hiring a head of growth. If you like Software Engineering Daily and consider yourself competent in sales, marketing, and strategy, send me an email: [email protected]
• FindCollabs is a place to build open source software.
• The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

The post FOSSA: Open Source Management with Kevin Wang appeared first on Software Engineering Daily.

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics.

Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn.

Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management.

Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

Sponsorship inquiries: [email protected]

Check out our active projects:

• We are hiring a head of growth. If you like Software Engineering Daily and consider yourself competent in sales, marketing, and strategy, send me an email: [email protected]
• FindCollabs is a place to build open source software.
• The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

The post LinkedIn Kafka with Nacho Solis appeared first on Software Engineering Daily.

Open source software is very new. Open source has existed for less than 30-40 years, depending on who you ask. The idea of open source was popularized by Linux, and open source software started to get heavily commercialized in the 1990s.

By the early 2000s, open source was used by nearly every large software company. In recent years, most of the new databases, web frameworks, user interface libraries, and other software primitives are being built in the open.

Dirk Hohndel is VP and chief open source officer at VMware, where he works on strategies around Linux, containers, Kubernetes, and other open source software. Dirk joins the show to talk about his long history with open source and the current state of the open source ecosystem.

Sponsorship inquiries: [email protected]

Check out our active companies and projects:

• FindCollabs is a place to find collaborators and build projects. Find a project to work on
• Podsheets is an open source podcast hosting platform built with the learnings from Software Engineering Daily. Our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

The post Open Source Ecosystem with Dirk Hohndel appeared first on Software Engineering Daily.

Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering.

Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that multiple systems can consume the messages from a single sender. 

In previous shows, we have covered design patterns within Kafka, Kafka streams, event sourcing with Kafka, and many other subjects relating to the technology. Kafka is broadly useful, and new strategies for using Kafka continue to emerge as the open source project develops new functionality and becomes a platform for data applications.

In today’s episode, Tim Berglund returns to Software Engineering Daily for a discussion of how applications are built today using Kafka–including systems that are undergoing a refactoring, data engineering applications, and systems with a large number of communicating services.

If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.

Sponsorship inquiries: [email protected]

Check out our active companies and projects:

• FindCollabs is a place to find collaborators and build projects. Find a project to work on
• Podsheets is an open source podcast hosting platform built with the learnings from Software Engineering Daily. Our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

The post Kafka Applications with Tim Berglund appeared first on Software Engineering Daily.

Open source plays a key role in today’s world of technology businesses. Today, the impact of open source seems obvious. From Kubernetes to distributed databases to cloud providers, so much of our software is powered by open source. But it was not always this way.  

Bruce Perens was one of the earliest figures in the world of open source. He collaborated with Eric S. Raymond, the author of Cathedral and the Bazaar. He developed a differing belief set from Richard Stallman, who was another foundational advocate of open source. Bruce worked as an engineer at Pixar from 1987 to the year 2000.

Bruce joins the show to give a history of open source and his experience in the software industry.

Sponsorship inquiries: [email protected]

Check out our active companies and projects:

• FindCollabs is a place to find collaborators and build projects. Find a project to work on
• Podsheets is an open source podcast hosting platform built with the learnings from Software Engineering Daily. Our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

The post Open Source Policy with Bruce Perens appeared first on Software Engineering Daily.

Cryptocurrencies are decentralized monetary systems built on open source software.

The open source software movement has evolved from the world of Linux, MySQL, and Apache to a thriving ecosystem of commercial enterprises built around open source software. This ecosystem includes projects such as Kubernetes, MongoDB, and ReactJS. It includes large organizations such as Amazon Web Services, Elastic, and Facebook.

In a parallel software universe, the crypto ecosystem has built revolutionary new financial tools, creating billions of dollars of value, but not very many massive commercial companies.  In the world of cryptocurrencies, many of the same rules apply to the classic open source world. But other rules do not. 

How do these two worlds differ from each other? How are they the same? And how might they end up colliding?

Haseeb Qureshi, Joseph Jacks, and Alok Vasudev join the show for a spirited discussion of cryptocurrencies and open source. Haseeb is a cryptocurrency investor, JJ is the founder of OSS Capital, and Alok is an engineer and the founder of crypto venture capital firm Standard Crypto.

Sponsorship inquiries: [email protected]

ANNOUNCEMENTS

• FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
• Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.   

The post Crypto and OSS with Haseeb Qureshi, Joseph Jacks, and Alok Vasudev appeared first on Software Engineering Daily.

Open source software has evolved into a thriving, multifaceted ecosystem.

Open source encompasses operating systems and databases. Open source embodies both altruism and self-interest. And open source enables thriving businesses from WordPress blogs to hundred billion dollar cloud providers.

There is a large set of business models that can be built around a successful open source project. Some of these business models are more defensible than others. A company based around open source must make a deliberate set of tradeoffs around how their private company will relate to the public open source ecosystem.

Today’s episode is a discussion of open source business models with Karthik Ranganathan, Heather Meeker, and Matt Asay. Karthik is the CTO of YugaByte, an open source distributed SQL database. Heather Meeker is an open source licensing specialist and a founding partner of OSS Capital. Matt Asay is a longtime executive who has worked in several open source based companies including MongoDB–he has also written at length about open source. Full disclosure: YugaByte, where Karthik works, is a sponsor of SE Daily.

Sponsorship inquiries: [email protected]

ANNOUNCEMENTS

• FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
• Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.   

The post Open Source Business Models with Karthik Ranganathan, Heather Meeker, and Matt Asay appeared first on Software Engineering Daily.

During 2015, Uber was going through rapid scalability. The internal engineering systems were constantly tested by the growing user base. Over the next two years, the number of internal services at Uber would grow from 500 to 2000, and standardizing the monitoring of all these different services became a priority.

After working with a variety of available tools, Uber’s engineering team decided that something new needed to be built internally. Jaeger is an open source distributed tracing tool that provides observability features throughout Uber’s microservices architecture.

Yuri Shkuro is an engineer at Uber, where he works on Jaeger and other infrastructure projects. He joins the show to discuss the history of engineering at Uber, the architecture of Jaeger, and the requirements for building and scaling a distributed tracing tool.

Sponsorship inquiries: [email protected]

Show Notes

Mastering Distributed Tracing

ANNOUNCEMENTS

• FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
• Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
• New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.   

The post Jaeger: Distributed Tracing at Uber with Yuri Shkuro appeared first on Software Engineering Daily.

Envoy is an open source edge and service proxy that was originally developed at Lyft. 

Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio.

Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone.

Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC.

Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.

ANNOUNCEMENTS

• New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.    
• FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
• Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco
• We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Envoy Mobile with Matt Klein appeared first on Software Engineering Daily.

Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native.

Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community.

The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do.

As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well.

Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.

ANNOUNCEMENTS

• New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.    
• FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
• Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco.
• We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Facebook Open Source Management with Tom Occhino appeared first on Software Engineering Daily.

Open source software allows developers to take code from the Internet and modify it for their own use. Open source has allowed innovation to occur on a massive scale. Today, open source software powers our consumer client applications and our backend cloud server infrastructure. 

Linux powers single node operating systems and Kubernetes is the foundation for new distributed systems. Hadoop created an open source distributed file system, Spark gave us a computational runtime on top of it, and Kafka created a middleware platform for shuttling data from one place to another.

There are numerous other examples of how open source has changed the world of software development. Open source has also reshaped the business landscape of infrastructure software companies. 

A common business structure for a modern infrastructure company is the “open core” model. An open core company maintains an open source project that is free to use, but also sells a product or service around that product. Companies with an open core model include Red Hat, HashiCorp, and GitLab.

Many companies are building a thriving business with the open core business model. But these companies do not directly control the most important part of the infrastructure supply chain: the cloud provider.

Cloud providers have a fundamental tension with open core companies because the cloud providers offer services that compete with the open core companies. 

In addition to the issue of cloud providers competing directly with the open core companies, some people have questioned whether Amazon Web Services is capturing an unfair portion of the value that is being created by open source.

Amazon Web Services is the biggest cloud provider, and it has built a large catalog of services that are built off of open source software. But AWS has not historically contributed heavily to open source relative to the value it has captured. 

One example of an open core company which has lost market share to an AWS cloud-hosted offering is Elastic, the open core company which maintains the ElasticSearch open source project. Amazon ElasticSearch Service is a closed-source hosted offering built on top of the ElasticSearch.

Elastic (the company) has increasingly intermingled proprietary software with their open source repository, making it less clear how that open source repository can be used by companies that want to deploy it for their commercial use.

Open core companies such as MongoDB, Redis Labs, and Cockroach Labs have responded to the competitive pressures of AWS by changing their licenses and making it more expensive for cloud providers to offer a cloud-hosted offering of their open source project.

The dynamics between cloud providers and open core companies will continue to evolve in the coming years. The norms around open source are up for debate.

Joseph Jacks is the founder of OSS Capital, a venture firm focused on investments in commercial open source software companies. He returns to the show to discuss the changing landscape of open core companies, and the benefits of permissionless innovation.

ANNOUNCEMENTS

• FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
• New Software Daily app for iOS. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.
• Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco.
• We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Permissionless Innovation with Joseph Jacks appeared first on Software Engineering Daily.

React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native).

React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks.

After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components.

After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project.

RECENT UPDATES:

The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open!

The FindCollabs Podcast is out!

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

The post Facebook React with Dan Abramov appeared first on Software Engineering Daily.

RECENT UPDATES:

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Software Daily is looking for help with Android engineering, QA, machine learning, and more

FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned

Open source policy has become a business issue as well as a political one.

Businesses like Elastic, MongoDB (the company), and Redis Labs have started to view the open source licenses of the projects they work on as a means for business defensibility against cloud providers offering similar services. It remains to be seen how viable this strategy will be for the commercial open source vendors.

Companies that do not directly sell commercial open source are also grappling with questions around open source licensing. Facebook has become a force in the open source world through projects like React and GraphQL. Facebook leads these projects, but Facebook is not monetizing them other than to the extent that they use the projects to build Facebook.com.

Facebook’s incentives are aligned with the rest of the industry on the quality of the GraphQL and React projects. Proper licensing can help Facebook keep those incentives in alignment.

Joel Marcey, Michael Cheng, and Kathy Kam from Facebook join me for a discussion of the state of open source licensing, and how that impacts Facebook.

The post Facebook OSS License Policy with Joel Marcey, Michael Cheng, and Kathy Kam appeared first on Software Engineering Daily.

Upcoming events:

A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019

FindCollabs Hackathon at App Academy on April 6, 2019

Data engineering touches every area of an organization.

Engineers need a data platform to build search indexes and microservices. Data scientists need data pipelines to build machine learning models. Business analysts need flexible dashboards to understand the trends and customer use for a product.

Max Beauchemin is a data engineer who has worked at Airbnb, Lyft, and Facebook. He’s the creator of two successful open source projects: Apache Airflow and Apache Superset. In a previous show, Max discussed data engineering at Airbnb, and the usage of Airflow. In today’s show, Max discusses the engineering of Apache Superset.

Superset is an open source business intelligence web application. Superset allows users to create visualizations, slice and dice their data, and query it. Superset integrates with Druid, a database that supports exploratory, OLAP-style workloads.

One reason Superset is distinctive is that it is a full open source application. Many open source projects are tools like databases, command line tools, and web frameworks. Superset is an open source application that can be used by individuals who are not developers–so the audience is wider than the typical open source tool built for engineers.

Max joins the show to talk about his experience as a data engineer at Airbnb and Lyft, and the open source projects he has started.

The post Apache Superset with Maxime Beauchemin appeared first on Software Engineering Daily.

Upcoming events:

A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019

FindCollabs Hackathon at App Academy on April 6, 2019

Red Hat was the first commercial open source software company. For years, investors and entrepreneurs assumed there would never be another Red Hat.

Red Hat’s business was built around enterprise operating system distribution and support. Since the operating system is at the core of how users within a company are doing their job, Red Hat had a lot of leverage and a strong business model. But how many enterprise software products could be so critical to a business that they could manage to offer their software as an open source option yet still make money?

As it turns out, there are many ways to make money in open source.

MySQL ate away at the dominance of Oracle’s database business for similar reasons to Red Hat’s success: much like an operating system, the database layer is critical infrastructure. Cloudera and Hortonworks were able to monetize the open source Hadoop project because Hadoop was hard to deploy and manage.

As cloud infrastructure matured, it became easier to start companies that offered open source software as-a-service. Elastic offers an easy way to use ElasticSearch. RedisLabs offers Redis as a service. MongoDB (the company) offers MongoDB as a service. As it turns out, engineers love to see the source code for databases, but they do not enjoy deploying and managing them and they are happy to pay providers to save them time.

Still, there is continued skepticism of open source businesses.

Today’s debates center around whether individual providers like Elastic can offer a service that competes with an ElasticSearch service offered by AWS. We could just as easily be asking the inverse question– how can AWS compete with an entire company that is dedicated to the deeply technical problem of solving search?

The reality is that most of these open source product categories have an enormous total addressable market, and extremely good unit economics. This applies to both cloud providers and point solution providers. Investors often talk about how much they love subscription businesses. When a company starts purchasing infrastructure-as-a-service from you, it is like they are buying a subscription where the annuity increases over time!

When an investor says they are worried about a giant cloud provider offering the same service as an open source company, it is similar to the investor being worried that a new sales tool is going to be duplicated by Salesforce. The market always needs new sales tools–and the market needs those tools to be offered both by Salesforce and by smaller CRM companies.

In the world of commercial open source, there is plenty of room for both point solution providers and cloud providers. But they are competing for the same customers, and the competitive battlefield is expanding to the nuanced world of software licensing. By changing their licenses, open source projects like Kafka, MongoDB, and Redis can prohibit AWS from certain usage patterns. This might offer some protection for companies based around the point solutions–companies like Confluent and RedisLabs.

Beyond the fracas of the battle between cloud providers and point solutions, there are newer open source companies with models that do not fit tightly into any historical business models. HashiCorp makes a suite of differentiated open source tools that have not been seriously contested or offered as a service by cloud providers. GitLab makes an open source platform that is built with monitoring, logging, CI, and code hosting out of the box.

As the world of open source business models expands, more companies will find opportunity in open sourcing the code that runs their products. In many cases, they will find that it strengthens their advantage rather than weakens it. The defensibility of many businesses relies more on data and network effects than the contents of the codebase. We may see the default question gradually shift from “why should I open source my codebase?” to “why shouldn’t I open source my codebase?”

Mike Volpi is a partner at Index Ventures and has invested in many open source businesses over the last decade. He is on the board of Confluent, Cockroach Labs, Kong, and Elastic. Mike joins the show to share his perspective on open source business models of the past, present, and future.

The post OSS Businesses with Mike Volpi appeared first on Software Engineering Daily.

GitLab is an open source platform for software development.

GitLab started with the ability to manage git repositories and now has functionality for collaboration, issue tracking, continuous integration, logging, and tracing. GitLab’s core business is selling to enterprises who want a self-hosted git installation, such as banks or other companies who prefer not to use a git service in the cloud.

The vision for GitLab is to provide a platform for managing the full software development lifecycle, from code hosting to deployment–as well as tools for observability and project management.

Sid Sijbrandij is the CEO of GitLab and he joins the show to talk about the product, the business, and the company’s vision for the future. GitLab’s strategy is to offer a set of tools that work for developers out of the box, cutting down on time spent integrating each individual vendor.

The post GitLab with Sid Sijbrandij appeared first on Software Engineering Daily.

An operating system kernel manages the system resources that are needed to run applications. The Linux kernel runs most of the smart devices that we interact with, and is the largest open source project in history.

Shuah Khan has worked on operating systems for two decades, including 13 years at HP and 5 years at Samsung. She has worked on proprietary operating systems and a variety of Linux operating system environments, including mobile devices. Shuah joins the show to discuss her work within Linux and her experience contributing to open source.

Shuah has made significant contributions to kselftest, a set of tests for Linux. Testing the Linux kernel is complicated. Because there is so much depth to the codebase, and such a variety of ways that Linux can be used, there is also a variety of ways that the operating system gets tested. There is smoke testing, performance testing, and regression testing. There are trees of tests, and as a developer you may only want to run a subset of the tests in that tree.

The conversation with Shuah ranged from the low level practices of testing the kernel to a high level discussion of how the Linux kernel can reveal dynamics of human nature.

The post Linux Kernel Development with Shuah Khan appeared first on Software Engineering Daily.

Edge computing refers to computation involving drones, connected cars, smart factories, or IoT sensors. Any software deployment that is not a large centralized server installation could qualify as an edge device–even a smartphone.

Today, much of our heavy computation takes place in the cloud–a set of remote data centers some distance away from our client devices. For many use cases, this works fine. But there are a growing number of use cases with lower latency and higher bandwidth requirements at the edge.

A simple example is video. Let’s say you want to record a video stream, and detect people in that video stream in real time. Based on who those people in the video stream are, you want to do different things–maybe you want to send them a text message, or report to the police that a dangerous person has entered the premises. This video stream could be captured by a drone, or by a smart car, or by a video camera mounted somewhere.

Where is the video stream getting stored? Where is the machine learning model running? How do you deploy new machine learning models to the operating system with the machine learning model? This is a simple example, and there are many open questions as to how to best solve such a problem.

With the increased resource constraints at the edge, there is a need for new hardware and software to power these edge applications. This led to the creation of LF Edge, a new open source group under the Linux Foundation. The goal of LF Edge is to build an open source framework for the edge.

Arpit Joshipura is the general manager of networking, orchestration, edge computing, and IoT with the Linux Foundation. He joins the show to describe the state of edge computation, and the mission of LF Edge.

This episode was exciting for several reasons. After seeing the rise of Kubernetes for container orchestration, we know that a popular open source technology that solves a widespread problem can have dramatic influence on the software world. And when multiple large companies get involved in that open source project, it can gain traction quite quickly.

Edge computing has a large set of unanswered questions, but telecom providers like AT&T and large infrastructure companies like Dell EMC are getting heavily involved with the Linux Foundation Edge group. This represents a significant expansion of the open source model, and a suggestion of further investment into open source projects in the near future.

The post Edge Computing Open Source with Arpit Joshipura appeared first on Software Engineering Daily.

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging.

API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled.

Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway.

Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.

The post Kong API Platform with Marco Palladino appeared first on Software Engineering Daily.

Building software was simplified by cloud providers. With the cloud, it became much easier to deploy a server, spin up a database, and scale an application. Cloud providers like AWS gave developers access to these infrastructure primitives like storage and compute.

On top of those primitives, numerous API companies have been built. An API company offers a more specific set of services. Twilio offers SMS text messaging API services. Stripe offers payment API services. These APIs give developers another level of tooling to build software out of.

Developers can now think of entire applications in terms of APIs, and the number of APIs is growing rapidly. From business services such as booking a flight to machine learning models like image classification, the “API economy” has given developers a huge catalog of tools.

Since developers have this additional leverage, software can be built with smaller teams. The codebase can also be smaller. But one area where the complexity is growing is the number of APIs that need to be managed. For each API, there is a different system for integrating the API into your application. Different API providers have different levels of reliability.

Another area of difficulty is the discoverability of APIs. If I don’t know about a flight search API, I am never going to think of what applications I could build on top of that. There are APIs for generating memes, and APIs for easily querying what music is trending across the world.

RapidAPI is a marketplace for APIs. It includes search and discovery features for the wide variety of different APIs that can be found across the internet. RapidAPI is also a system for integrating with multiple APIs through it’s API management system.

Iddo Gino is the CEO and founder of RapidAPI, and he joins the show to discuss the motivation for creating an API marketplace, as well as the engineering behind RapidAPI.

The post RapidAPI: API Marketplace with Iddo Gino appeared first on Software Engineering Daily.

Open source projects benefit from the network effects of a large audience of developers. A popular open source project will be contributed to and used by thousands of developers, who are continuously testing, deploying, and improving the software. The open source movement has created massive communities and a thriving, collaborative economy.

Infrastructure software companies are increasingly built within an open source business model. Databases, queueing systems, orchestrators, operating systems, and search engines have been started as freely available open source projects, and leveraged into billion dollar businesses. In previous shows we have talked about business strategy, go-to-market tactics, and licensing of infrastructure software.

There remains plenty of room for more open source infrastructure companies. We still need better databases and distributed systems management. But over time, open source will move up the stack. From Netflix to Uber to social networks to payments systems–all software verticals will become open source because the benefits of making your software open source outweigh the costs.

For many software business models, the competitive advantage is not found in their source code–it’s in their data, their network effects, their sales strategy, and their brand.

Therefore, it makes sense that someday the source code will be freely available, democratizing the infrastructure concerns and letting these software businesses move up the value chain and become less operationally intensive at the bottom of the stack. Rather than asking “why should we open source our code”, these companies will be asking “why shouldn’t we open source our code?”

Joseph Jacks is the founder of OSS Capital, a venture capital firm that invests exclusively in commercial OSS startup companies. Joe believes that over time, open source eats everything. In today’s show, we talk about the future of open source businesses, the impact of licensing, cloud providers, and cryptocurrencies.

Show Notes

• OSS.cash ($100M+ Revenue Commercial Open Source Software Company Index)
• Quantifying Decentralization

The post OSS Capital with Joseph Jacks appeared first on Software Engineering Daily.

Open source software powers everything we do on the Internet. Google runs on Linux servers. Content sites are served by WordPress. Our data is queued in Kafka clusters and stored in MongoDB instances.

The success of an open source project often leads to the creator of that open source software becoming wealthy. An open source project can be monetized through enterprise add-ons, or consultation, or simplified hosting. The creators of open source software know their domains so well, that they are usually well-suited to operate this kind of open source business.

Open source business model success stories include Elastic (ElasticSearch), Cloudera (Hadoop), and Red Hat.

The rise in usage of cloud providers has changed the viability of some open source business models. AWS can monetize almost any open source project more profitably than the creators. This is because AWS has established distribution channels.

If I already run my application on AWS, and I am looking for someone to provide me with a hosted version of a database, I will probably choose the hosted database that AWS provides.

The Commons Clause is a license that open source projects can use to protect their code from being profited from. Redis, an open source in-memory object storage system, recently licensed their code with the Commons Clause with the goal of improving the business of Redis Labs, a company built by the creators of the Redis project.

Kevin Wang joins the show to discuss everything open source–from business models to security vulnerabilities to licensing. Kevin is the CEO at FOSSA, a system for managing open source licensing and security. Kevin was involved in the creation of the Commons Clause and has written about it in detail.

The post Commons Clause with Kevin Wang appeared first on Software Engineering Daily.

Linux has created much more value for Google than it has for Linus Torvalds. Ruby on Rails has created more value for Airbnb than it has for David Heinemeier Hansson. Successful open source projects create more value than their creators’ capture–and that’s one reason why collections of people on the Internet are often inspired to work together on open source.

When an engineer creates an open source project, and that open source project finds a large audience, that engineer can often build a successful business. SpringSource, Cloudera, and Elastic are examples of massively successful enterprises that were founded by the creators of open source software. But in other cases, the value of an open source project gets largely captured by cloud providers that create a closed source version of the open source project and offer it as a service.

Shaun Connolly has worked in senior strategic roles at software companies such as SpringSource, VMWare, and Hortonworks. Throughout his decades of experience, much of his time has been spent figuring out how to monetize open source projects intelligently. Shaun joins the show to talk about his past experiences building enterprises, as well as modern issues–such as how to compete with major cloud providers. We also discuss the Commons Clause license, a new software license that open source projects can use to try to protect their value from being entirely captured by a cloud provider.

Software Engineering Daily is looking for sponsors. If you are interested in reaching over 50,000 developers, you can go to softwareengineeringdaily.com/sponsor to find out more, and you can send us a message. We’d love to hear from you. And if you are an engineer working at a company that is marketing to developers, or hiring developers, if you tell your marketing department or your recruiting department about softwareengineeringdaily.com/sponsor, that is one way to help us out.

The post Open Source Product Strategy with Shaun Connolly appeared first on Software Engineering Daily.

The DEV Community is a platform where developers share ideas, programming advice, and tools. Ben Halpern started it after running an extremely successful Twitter account creating humorous tweets for developers. One way to describe DEV Community is as a cross between Medium, Stack Overflow, and Reddit–but it has its own personality, so I recommend checking it out.

The DEV Community was open sourced, and we discussed the challenges and the opportunities of having an open source social network. We also talked about his plans for the future, and where he is taking DEV Community.

Ben is an entrepreneur who tries lots of different creative projects, so his perspective has always resonated with me. Ben has been on the show a few times before, when we talked about the state of developer media, side projects, and the identity of the software engineer.

DEV Community was originally called Practical Dev, which I mistakenly referred to it as in the earlier parts of the show.

The post DEV Community with Ben Halpern appeared first on Software Engineering Daily.

Blogging is more than 20 years old. Over that period of time, numerous publishing platforms have been created. Squarespace, Blogger, Medium, and Twitter are popular closed source platforms. WordPress has been the most popular open source blogging platform–and much of the Internet (including Software Engineering Daily) runs on WordPress.

WordPress is a powerful platform. News companies, ecommerce websites, and many other kinds of businesses use WordPress as their central publishing tool. But WordPress has been around for 15 years–and there are some potential conflicts of interest between WordPress the open source project and WordPress.com (a company started to host WordPress websites).

John O’Nolan was working as a WordPress developer when he decided to start a new publishing platform called Ghost. Five years later, the Ghost project is a success–with a thriving open source community, a profitable SaaS business, and companies like Digital Ocean and Mozilla using Ghost to host their blogs.

John and I discussed his background with WordPress, what he wanted to do differently with Ghost, and the software architecture of Ghost. We also touched on the Ghost SaaS business and the management of the open source project.

The post Ghost: Open Source Publishing Platform with John O’Nolan appeared first on Software Engineering Daily.

Flutter allows developers to build cross-platform mobile apps. In our previous show about Flutter, Eric Seidel from Google described the goals of Flutter, why he founded the project, and how Flutter is built. In today’s show, Randal Schwartz talks about Flutter in more detail–including the developer experience of building Flutter apps and why he finds Flutter so exciting.

Randal is a longtime software developer who has focused mostly on web applications. He’s also the host of the popular podcast FLOSS Weekly, a show about open source software. Like many developers, Randal has stayed away from mobile development in the past. If you write a mobile application, you have historically had to build iOS and Android apps separately. That is a large up-front cost, and Flutter reduces the cost by allowing users to develop mobile apps for iOS and Android with a single codebase.

Randal has been podcasting about open source software for 12 years, so he brings historical depth to this conversation. He has also been working with Dart for several years–Dart is a language developed by Google that is used in Flutter.

The post Flutter in Practice with Randal Schwartz appeared first on Software Engineering Daily.

Flutter is a project from Google that is rebuilding user interface engineering from the ground up.

Today, most engineering teams have dedicated engineering resources for web, iOS, and Android. These different platforms have their own design constraints, their own toolset, and their own programming languages. But each platform is merely building a user interface. Why should development across these three user surfaces be so different?

This was the question that Eric Seidel was asking himself three years ago, when he co-founded the Flutter project. The Flutter project had a few rough starts, as the team tried to figure out exactly what layer of abstraction they were trying to provide.

Around that time, ReactJS and React Native were growing in popularity. Seeing the React projects provided some data points, and some inspiration. But Flutter takes a lower level approach to cross platform app development, by presenting a rendering layer and a runtime API that are interfacing with the hardware in the same way that OpenGL does.

In today’s episode, Eric joins the show to explain how the Flutter project came to life, and his lessons from starting an ambitious project that took several years to pick up steam. I enjoyed this episode because Flutter could have massive improvements for how quickly we can build apps–and also because Eric is a serious engineer and there are so many insights in this episode about computer science, software engineering, and project management.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The post Flutter with Eric Seidel appeared first on Software Engineering Daily.

Chromium is an open source browser that shares code with the Chrome browser from Google. A browser is a large piece of software, with engineering challenges around threading, rendering, resource management, and networking. To add to the complexity, Chrome runs on iOS, Android, MacOSX, Windows, and other platforms.

Chrome OS is an operating system based on Chrome. There is also Chromium OS, the open source version of Chrome OS. The Chrome/Chromium operating systems are based off of Linux.

Through this entire episode, the line between browser and operating system is blurry. There is so much resource management involved in the Chrome browser that it has its own task manager. For many people (including myself) the browser is the main application you are interfacing with throughout the day. It handles all of your business applications. Even many of your desktop apps, such as Slack, are running on Electron, which is a framework for building cross-platform apps that uses Chromium.

David Bokan is an engineer on the Chromium team at Google, and he joins the show to describe the engineering of Chrome and the development and release process. David also gives his thoughts on future developments for browsers, apps, and the Internet.

The post Chrome and Chromium with David Bokan appeared first on Software Engineering Daily.

Different browsers consume JavaScript in different ways.

When a new version of JavaScript comes out, developers are eager to use the new functionality of that language version. But if you are writing frontend JavaScript code, that code needs to be interpretable by every browser that might consume it–whether the consumer is on an iPhone running Safari or a Windows machine running Internet Explorer 11.

Babel is a transpiler for JavaScript. Babel allows new versions of JavaScript to be consumed by older browsers by translating new language features of JavaScript into code that is readable by an older JavaScript interpreter. Babel does this by parsing JavaScript code, creating an abstract syntax tree, and manipulating the AST to make that code comply with the old browser.

Henry Zhu is a core maintainer of Babel and a full-time open source developer. In today’s episode, Henry explains how Babel works and its various applications. He also talks about life as a full-time open source developer, where he earns a living through Patreon and OpenCollective.

The post Babel with Henry Zhu appeared first on Software Engineering Daily.

When software is performing suboptimally, the programmer can use a variety of tools to diagnose problems and improve the quality of the code. A profiler is a tool for examining where a program is spending time.

Every program consists of a set of different functions. These functions call each other. The total amount of time that your program runs is the sum of the time your program spends in all of the different functions. When you run a program, you can execute a profiler on that program, and the profiler will give you a breakdown of which of the different functions time is being spent in.

If you have functions A, B, and C, your profiler might say that your program is spending 30% of its time in function A, 20% of its time in function B, and 50% of its time in function C.

Julia Evans is a software engineer at Stripe and the creator of a Ruby profiler called rbspy. rbspy can execute on a running Ruby program and report back with a profile. As Julia explains, a profiler turns out to be a non-trivial piece of software to build. To introspect a Ruby program, you need to understand how the Ruby interpreter is translating Ruby code into C structs for execution.

This episode is about profilers–but in order to talk about profilers, we also have to talk about Ruby, the Ruby interpreter, and the way that executing programs are laid out in memory.

The post Profilers with Julia Evans appeared first on Software Engineering Daily.

Message broker systems decouple the consumers and producers of a message channel. In previous shows, we have explored ZeroMQ, PubNub, Apache Kafka, and NATS. In this episode, we talk about another message broker: Apache Pulsar.

Pulsar is an open source distributed pub-sub message system originally created at Yahoo. It was used to scale products with high volumes of users–such as Yahoo Mail.

There are three components of a Pulsar deployment: the Pulsar broker (which handles the message brokering), Apache Bookkeeper (which handles the durable storage of the messages), and Apache Zookeeper, which manages the distributed coordination.

Lewis Kaneshiro joins the show to describe how Apache Pulsar works, and how it compares to other messaging systems like Apache Kafka. Lewis is the CEO of Streamlio, a company that builds messaging and stream processing systems for enterprises, and uses Pulsar in its core product.

The post Pulsar Messaging with Lewis Kaneshiro appeared first on Software Engineering Daily.

YouTube runs a large MySQL database to hold the metadata about its videos. As YouTube scaled, the database was sharded, and applications within YouTube had to write queries that were aware of the sharding layout of that database.

This is problematic because it pushes complexity to the application developer. An application developer shouldn’t have to be aware of how a database is laid out among different nodes. The developer should be able to issue a query and have the cluster simply return the data.

Vitess is an open source system for scaling large MySQL databases. Sugu Sougoumarane co-created Vitess at YouTube. Since YouTube is owned by Google, Vitess was able to leverage the Borg cluster manager developed at Google. Once Kubernetes came to market, it became more viable to make Vitess accessible to open source developers.

Sugu joins the show to talk about the scalability problems that YouTube’s database infrastructure encountered and the motivations for building Vitess.

The post Vitess: Scaling MySQL with Sugu Sougoumarane appeared first on Software Engineering Daily.

Social networks can make you feel connected to a global society. But those social networks are controlled by a corporate entity. The profit motivations of the corporation are not directly aligned with the experience of the users.

Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like Facebook and Twitter. In the Mastodon model, users can run their own nodes, and other users can connect to them. You can follow users whose accounts reside in other nodes.

Eugen joins the show to discuss how Mastodon works, and how its thousands of users interact on the platform. We explore the open source community that is building Mastodon, and speculate on the future of social networks.

The post Mastodon: Federated Social Network with Eugen Rochko appeared first on Software Engineering Daily.

Most technology companies rely on open source software projects. But open source software projects are often maintained by a group of people that is not affiliated with any particular company. When an open source project develops too much technical debt, it can become a tragedy of the commons. Who is responsible for maintaining these open source projects?

This is the motivation for open source bounties. Companies and individuals who rely on open source create bounties, which are financial incentives for developers to solve problems within the open source project.

Kevin Owocki is the creator of Gitcoin, a platform for open source bounties that is mediated by an Ethereum smart contract. Kevin joins the show to discuss his experience building Gitcoin–as well as some of the problems with the blockchain space, such as rampant ICOs. Gitcoin is NOT a cryptocurrency or token itself–it is a platform for open source software to be built more efficiently. Kevin was an awesome guest and you will enjoy the conversation.

Gitcoin is a nice example of a real-world Ethereum use case–it uses Ethereum for escrow: if I post a $25 bounty for someone to fix a bug in my open source project, I will lock up ether in a smart contract. When the bug is fixed, the programmer who fixed it will submit a pull request on Github, and I will release the ether from the smart contract to pay them.

We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on.

Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our Software Engineering Daily apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking.

The post Gitcoin: Open Source Bounties with Kevin Owocki appeared first on Software Engineering Daily.

You have probably missed some of the best episodes of Software Engineering Daily. If you listen to just a few episodes a week, it can be difficult to identify the high quality shows. And if you are new to the podcast, you have no idea how to find episodes that might appeal to you.

Software Engineering Daily has a discovery problem.

We have 600 episodes, and much of the content is evergreen. The shows we did a year ago on Apache Spark, or Ethereum, or ReactJS are still relevant today, and they get plenty of listens.

Keith and Craig Holliday built a recommendation system for Software Engineering Daily. Then they built a Software Engineering Daily iOS app to improve the experience of SE Daily listeners. You can use the SE Daily app to find the most popular episodes of this podcast, and to find episode recommendations based on what you have listened to.

In this episode, Keith and Craig join the show to explain why they built an app for Software Engineering Daily. You can find all the code for the SE Daily app at github.com/softwareengineeringdaily in case you want to fork it for your own podcast–or if you want to contribute to it.

Question of the Week: What is your favorite continuous delivery or continuous integration tool? Email [email protected] and a winner will be chosen at random to receive a Software Engineering Daily hoodie. 

The post Software Engineering Daily App with Keith and Craig Holliday appeared first on Software Engineering Daily.

Augmented reality is coming at us fast. Every large tech company is rumored to be building an AR product. Microsoft HoloLens is already available to developers. Pokemon Go, the most popular augmented reality product today, was made by a company that was spun out of Google. But Apple seems to be ahead of everyone.

Apple’s ARKit is a set of tools for developers to build augmented reality applications. The applications people are building with ARKit are remarkable, and two of those early adopters join the show today for an interview.

Jesse Bounds and Siyu Song work at Mapbox, a company that makes mapping, navigation, and location search SDKs. Location is natural companion to augmented reality. If I am walking down the street with a pair of augmented reality glasses on, those glasses can augment the world with information based on my location.

Because the fit between AR and mapping is so natural, Mapbox has been rapidly experimenting to build up an expertise in AR. As a result, Jesse and Siyu make for great guests to talk about what engineers can build with ARKit today and what might be possible in the future.

The post Augmented Reality with Jesse Bounds and Siyu Song appeared first on Software Engineering Daily.

Containers are widely used in projects that have adopted Docker, Kubernetes, or Mesos. Containers allow for better resource isolation and scalability. With all of the adoption of containers, companies like Red Hat, Google, and CoreOS are working on improved standards within the community.

Standards are important to this community because of its pace of growth and the number of concurrent projects. If you heard our recent episode about the Linux Kernel’s open source governance, you know that having some rules in place will help encourage the right kind of creativity to thrive.

In the world of containers, networking is not well addressed as it is highly environment specific. The Container Networking Interface is an effort to add specifications around how networks of containers can form.

Dan Williams is an engineer at Red Hat. In today’s episode, he explores the ideas behind the container networking interface, which gives insights into how the broader community of cloud native technologies is evolving as a whole.

The post Container Networking with Dan Williams appeared first on Software Engineering Daily.

The code in the Linux kernel changes all the time–11k lines are added, 5.8k lines are removed, and 2k lines are modified DAILY. Linux is an open source operating system that has been worked on for 25 years, and one reason the project is able to move so fast is its governance and release structure.

Greg Kroah-Hartman is a fellow at the Linux Foundation, where he takes part in many of the developments in the kernel. This episode was a dive into how open source software gets built at scale and what is in store for the future. The Kubernetes project has drawn comparable attention to the size of Linux, and the Kubernetes project is learning how to manage open source from the Linux community.

Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details.

Show Notes

Kernel Development

The post Linux Kernel Governance with Greg Kroah-Hartman appeared first on Software Engineering Daily.

Modern architectures often consist of containers that run services. Those containers scale up and down depending on the demand for the services. These large software systems often use a technique known as event sourcing, where every change to the system is kept in an event log. When an event on the log is processed, several different data stores might be updated in response.

In these architectures, containers are interacting with each other. Multiple databases are responding to events in the event log. To connect these systems together, engineers can write small functions to pass data around–you might call these small connecting functions “glue.” Glue functions are a great use for a serverless tool such as AWS Lambda or Google Cloud Functions. As these glue functions grow in popularity, there is an increased need for an open source way to deploy serverless functions.

Sebastien Goasgoen works on Kubeless, a serverless execution tool built on top of Kubernetes. In this episode, we explore his take on the “serverless on Kubernetes” problem. This is a great companion episode to yesterday’s interview with Soam Vasani.

Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: [email protected]

The post Event Driven Serverless with Sebastien Goasgoen appeared first on Software Engineering Daily.

Kubernetes is an orchestration system for managing containers. Since it was open sourced by Google, Kubernetes has created a wave of innovation in the infrastructure technology space.

Another recent innovation has been the “serverless” execution tools–such as AWS Lambda and Google Cloud Functions. Serverless execution, otherwise known as functions-as-a-service, allows a developer to execute code against cloud servers without specifying which cloud servers they are executing on.

Serverless execution is a cheap, flexible resource that any large company wants to have access to. But AWS Lambda and the other popular serverless tools are closed source. This led Soam Vasani to work on Fission, a serverless executor that sits on top of Kubernetes.

If you have not heard about either Kubernetes or Serverless, you can check out our previous episodes about either topic. If you are familiar with the two topics, I think you’ll enjoy this episode, in which Soam explains the motivation for serverless on Kubernetes, and the architecture of Fission.

Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: [email protected]

The post Serverless on Kubernetes with Soam Vasani appeared first on Software Engineering Daily.

Cloud computing changed how we develop applications for the web. Over the last decade, engineers have been learning how to build software in this new paradigm. The costs have gone down, but our nodes can fail at any time. We no longer have to manage individual servers, but the layers of virtualization and containerization require new strategies for communicating between services.

As we have adjusted to this new way of building applications, the term “cloud-native” has become a useful descriptor. Cloud native applications are modern distributed systems capable of scaling to tens of thousands of self healing multi-tenant nodes.

Open source projects such as Kubernetes, Prometheus, and Linkerd draw on the lessons of big technology companies like Google, Twitter, and SoundCloud. Engineers today don’t need to reinvent the key infrastructure of those companies. We can combine the open source cloud-native technologies and use them to build powerful systems.

Dan Kohn is the executive director of the Cloud Native Computing Foundation, and he joins me for an interview about the projects within the CNCF, how they fit together, and the future of computing.

Stay tuned at the end of the episode for Jeff Meyerson’s tip about job searching brought to you by Indeed Prime.

Show Notes

Measuring the Popularity of Kubernetes Using BigQuery

The post Cloud Native Projects with Dan Kohn appeared first on Software Engineering Daily.

Swift is a language that is most commonly used to write apps for Apple client devices, such as iPhones. Since being released in 2014, Swift has become one of the most popular languages due to its high performance and developer ergonomics. In 2015, Swift was open sourced, creating the opportunity for Swift to be used outside of the Apple ecosystem.

If you write an iPhone app today, your frontend is in Swift and your backend is probably in NodeJS, Java, or Ruby. Engineers are working to port Swift to the server so that the Swift developer experience is isomorphic: the same language on the backend and the frontend.

Chris Bailey is an engineer at IBM working on Kitura, a Swift web framework. In this episode, we discuss the history of Swift, why it is so appealing to developers, and why Swift could become a server side language with as much popularity as Java.

Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

The post Swift on the Server with Chris Bailey appeared first on Software Engineering Daily.

Facebook’s open source projects include React, GraphQL, and Cassandra. These projects are key pieces of infrastructure used by thousands of developers–including engineers at Facebook itself.

These projects are able to gain traction because Facebook takes time to decouple the projects from their internal infrastructure and clean up the code before releasing them into the wild. Facebook has high standards for what they are willing to release.

Tom Occhino manages the React team at Facebook and works closely with engineers to determine what projects make sense to open source. In this episode, Preethi Kasireddy interviews Tom about how Facebook thinks about open source–what went right with React, why it makes sense for Facebook to continue to release new open source projects, and how full-time employees at Facebook interact with that open source codebase.

We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Show Notes

Pete Hunt: React: Rethinking best practices — JSConf EU 2013

The post Facebook Open Source with Tom Occhino appeared first on Software Engineering Daily.

Every program gets compiled down to 1s and 0s before it can be executed against hardware. Before being translated to machine code, programs that are written in a language like Rust, Swift, or Java spend time in an intermediate representation.

In Java, this intermediate representation is Java bytecode. Many different languages–such as Scala–translate to Java bytecode, because there has been lots of optimization written to speed up Java bytecode. Java bytecode runs on the JVM–the Java Virtual Machine.

LLVM is a project that draws inspiration from the Java Virtual Machine. LLVM originally meant “low level virtual machine” but today it is just called LLVM and describes a set of compiler tools.

In today’s interview with Morgan Wilde, we explore how compilers work, how different processor hardware architectures present a problem for compilers, and why LLVM’s intermediate representation creates a layer of interoperability for any language that compiles down to that intermediate representation.

Whether you are new to compilers or have experience, this episode will appeal to you. Morgan is an excellent teacher and his enthusiasm for the subject comes through. He has a 30-minute YouTube video–A Brief Introduction to LLVM that I highly recommend.

The post LLVM with Morgan Wilde appeared first on Software Engineering Daily.

Open source software is publicly available code that is worked on in the open by large crowds of developers. Almost all new software today uses some open source software in its code. But most people never contribute to open source themselves. Some people would love to get involved in open source, but they don’t know how to get started.

Shubheksha Jalan found herself in this position. In her early attempts to break into open source, she failed to get much traction and found the open source community hostile to newcomers. She would fix beginner issues, but had trouble figuring out how to make more complex contributions. Eventually she found success and wrote about it. In this episode she takes us through that journey.

If you are wondering how to get into open source, this episode is for you.

The post Open Source Contribution with Shubheksha Jalan appeared first on Software Engineering Daily.

Whatever engineering problem you have right now, the solution is probably not to write a new programming language. But sometimes it does make sense.

JetBrains makes IDEs–the interactive development environments that many people code in, like IntelliJ and Webstorm. And all of these IDEs are written in Java. So the JetBrains team is very familiar with Java and the JVM. Since JetBrains spends so much time working on tooling for developers, they also have an intimate understanding of the problems that developers encounter.

Kotlin was developed as an alternative to JVM languages such as Java and Scala. In this episode, Hadi Hariri from JetBrains explains why the company decided to build a new JVM language. We discuss many of its features, such as safety, concision, and its ability to compile to JavaScript.

Show Notes

Why Kotlin is My Next Programming Language: Ode to the Language You’ve Never Heard Of, by Mike Hearn

YouTube: GOTO 2016 Conference, Kotlin – Ready for Production, Hadi Hariri

The post Kotlin with Hadi Hariri appeared first on Software Engineering Daily.

Urbit is a completely new way of looking at computing. Every user gets a personal server, which runs your apps, wrangles your connected devices, and defines your secure identity. Your urbit presents your whole digital life as a single web service.

Urbit feels foreign and confusing for those of us coming from the traditional web because the normal paradigm is to iterate and paper over the problems of the current platform with new things built on top. Curtis Yarvin, the creator of Urbit, argues that the current model is too fundamentally broken for that to work. As he says: “the Internet as a client-server network has won. The Internet as a peer-to-peer network has failed.”

This sounds like yet another quirky, overambitious developer side project–but Urbit has serious legs. The github repo has had 51 committers over its four years of activity. Last year, a public crowdsale of Urbit address space raised more than $200,000. Peter Thiel was an early investor in the project, perhaps partly due to the combination of persistence, technical skill, and unusual opinions of Curtis Yarvin.

In this episode, Curtis and Galen Wolfe-Pauly join me for a conversation about Urbit–its strange computing platform and the contrarian philosophies that drive its creators.

Show Notes
Urbit Explanation 

The post Urbit with Curtis Yarvin and Galen Wolfe-Pauly appeared first on Software Engineering Daily.

Free Code Camp is an online learning platform that takes people from knowing nothing about code to having enough knowledge to build software for a living. We have already done a show with Quincy Larson, the founder of Free Code Camp, in which we discussed his motivation for starting the organization.

The economics of running a free interactive platform with thousands of users informs the architectural strategy. Free Code Camp does not even have ads, so it needs to be frugal. Berkeley Martinez is the CTO of Free Code Camp, and in today’s episode he discusses how Free Code Camp works from a technical perspective, including how the site’s sandboxed coding environment is built.

Show Notes:
Free Code Camp with Quincy Larson SE Daily episode 

The post Architecture of Free Code Camp with Berkeley Martinez appeared first on Software Engineering Daily.

Mesos and Kubernetes are tools for distributed systems management. Kubernetes is built with an emphasis on running services, whereas Mesos is commonly used for a wider variety of workloads, including data infrastructure like Spark and Kafka. Mesos can also be used as a platform to provide resource management for Kubernetes.

Dharmesh Kakadia is the author of Apache Mesos Essentials, and has spent time studying both Mesos and Kubernetes. He joins the show to explain the strengths of both platforms, and how they can be used together. In addition to dissecting Mesos and Kubernetes, we had a wide-ranging discussion, from cloud providers to software architecture to business strategy.

***

If you are in the Bay Area on January 11th, come to the Software Engineering Daily meetup. There will be awesome speakers, food, and a positive atmosphere. Find more information on softwareengineeringdaily.com or at the Software Engineering Daily meetup page.

The post Mesos, Kubernetes, and Infrastructure of the Future with Dharmesh Kakadia appeared first on Software Engineering Daily.

Cloud Foundry is an open-source platform as a service. Companies use Cloud Foundry as a control plane to deploy and manage applications. It provides abstractions for microservices and continuous integration.

Cornelia Davis joins the show to discuss Cloud Foundry. I interviewed her at the DevOps Enterprise Summit, where enterprises share their stories of improving their culture and their technology stack. Cornelia explained how and why a large enterprise adopts Cloud Foundry, and why they often hire a company like Pivotal to help them as they migrate their infrastructure and change their processes.

The post Cloud Foundry with Cornelia Davis appeared first on Software Engineering Daily.

C# .NET is the framework that is most often used to write software for the Microsoft Windows operating system. For many years, the C# .NET framework was closed source, but Microsoft’s recent push towards open source has led to the creation of .NET Core, a fork of C# .NET composed of a small subset of features from the original C# .NET stack.

This episode takes us through a history of .NET, with two program managers who have worked on .NET for many years–Immo Landwerth and Lee Coward. We also explored the present and future of .NET–discussing .NET Core and .NET usage on operating systems other than Microsoft Windows, which is another recent development.

The post DotNet Core with Lee Coward and Immo Landwerth appeared first on Software Engineering Daily.

Rust is a systems level language that is built to prevent crashes and eliminate data races. A language like C++ gives you high speed and lots of control, but it is easy to have segfaults, data races, and other problems if you aren’t careful. On this spectrum of control versus safety, we can plot other languages like Java, Go, and Haskell–but none of these languages have the unique feature set of Rust.

Rust’s concurrency model is built on the foundation of ownership–a piece of data can only be owned by a single thing at a time. In today’s interview, Alex Crichton explains ownership and the other abstractions that give Rust its unique model of concurrency.

The post Rust Concurrency with Alex Crichton appeared first on Software Engineering Daily.

There is a hierarchy of ways to access and store data in a computer system. The cheapest, slowest way to store and retrieve data is disk. On the faster end, we have memory. As we architect systems with increasing complexity, we have additional considerations–network latency, transient compute nodes, and numerous caching layers.

Apache Geode is a distributed, in-memory system for the JVM. It provides fast data storage and retrieval. Swapnil Bawaskar is an engineer who works on Geode, and he joins the show today to explain the architecture of Geode, and how Geode is different from other in-memory systems that get used for caching, such as memcache or Redis.

The post Apache Geode with Swapnil Bawaskar appeared first on Software Engineering Daily.

Jeff Bezanson’s university thesis described the motivation for a new programming language. He discussed the shortcomings of “array based programming environments” and his desire to create a more performant language with the best qualities of Lisp, Python, Ruby, Perl, Mathematica, R, and C.

The Julia Language is a high performance language designed to suit technical users that crave the flexibility to pick their own notation. The language has support for a wide variety of operators, allowing scientists to create domain specific equations that can be elegantly expressed as code.

Jeff Bezanson joins the show to discuss his motivations for Julia, and how the language has evolved since he started working on it.

The post Julia Language with Jeff Bezanson appeared first on Software Engineering Daily.

Docker containers changed how engineers think about building software, and the company most responsible for the widespread adoption of containers is Docker itself. Since containerization has caught on in the mainstream, companies like RedHat, Google, Huawei, and many other big players have built platform products that utilize Docker containers.

Docker containers are the unit that many engineers use to deploy their applications, but the servers that run those containers are usually on infrastructure providers like Amazon, Google, or Rackspace. This raises the question–how will Docker the company make money? Having raised $180M in equity, Docker has plenty of time to find something that works, but there are suspicions in the open source community that aggressive changes to the open source Docker project are being made by Docker the company with the long-term objective of monetization.

These changes–namely the default support for Docker Swarm–are not nefarious, but they might be less likely to occur of Docker were just an open source project, as opposed to an open source project being tended by a well-financed company. 

The New Stack broke the story about the potential Docker Fork, and Alex Williams and Joab Jackson of The New Stack join me on today’s episode. We go deep into the facts and speculations about why a fork might occur, the motivations of different key players, and the disputes within the community that have led us to this moment.

The post Docker Fork with Alex Williams and Joab Jackson appeared first on Software Engineering Daily.

Unikernels allow us to specify the minimum features of an operating system we need to deploy our applications. We’ve had many shows about containers, which allow you to deploy your application on top of a segregated portion of an operating system. Unikernels are different because they can be deployed directly to bare metal or to a hypervisor.

Idit Levine works on Unikernels at EMC. We had an interview a few weeks ago with her colleague Scott Weiss, and today’s conversation goes deeper into the motivations for Unikernels, the problems they solve, and the project she is leading called Unik, which seeks to add the degree of usability and adoption to Unikernels that Docker brought to containers.

The post Unikernels with Idit Levine appeared first on Software Engineering Daily.

The Haskell programming language is often thought of as an academic tool, useful for teaching students about monads and functors, but not much else. But there are advantages to using Haskell as a production backend language.

Better is a company built with Haskell on the backend, and Carl Baatz wrote a blog post detailing his experiences using Haskell. He joins the show to give a detailed explanation of why a company might want to use Haskell on the backend, from software architecture, to testing, to hiring.

The post Haskell in Production with Carl Baatz appeared first on Software Engineering Daily.

Google’s infrastructure has been the source inspiration for research papers, software projects, and entire companies. Google pioneered the idea that we care less about the individual machines we are running our applications on, and more about the applications themselves.

Containers are the abstraction we use to separate the concerns of the application from those of the underlying hardware. CoreOS is an operating system built with this paradigm shift in mind. In a data center, the main job of the operating system is to be a platform for containers to run smoothly on. Brandon Philips is the CTO of CoreOS and he joins the show to explain what CoreOS does differently to power the applications that get deployed on top of it.

The post CoreOS with Brandon Philips appeared first on Software Engineering Daily.

Akka is a toolkit for building concurrent, distributed, message-driven applications on the JVM. Akka provides an implementation of the actor model of concurrency, which simplifies concurrency by adding a lighter weight abstraction than threads and thread pools.
Konrad Malawski joins the show today to discuss Akka and reactive streams. Reactive streams is an initiative to provide a standard for asynchronous stream processing. This show goes deep into modern concurrent programming and is a great companion to some of the shows we have done about reactive programming.

The post Akka Reactive Streams with Konrad Malawski appeared first on Software Engineering Daily.

The Linux kernel of many popular operating system distributions contains 200-500 million lines of code. The average user never touches many of the libraries that are contained in these operating system distributions. For example, if you spin up a virtual machine on a cloud service provider, the virtual machine will have a USB driver. This is wasted space, because you can’t even interact with the USB port on a virtual machine.

Unikernels are a way to rethink our usage of operating systems. A unikernel uses a stripped down operating system called a library operating system–it contains only the libraries you need for the applications you are running. Today’s guest Scott Weiss joins the show to talk about unikernels, and a project he is working on called UniK, a tool for compiling application sources into unikernels.

The post unikernels and unik with Scott Weiss appeared first on Software Engineering Daily.

GraphQL was open sourced out of Facebook, and gave developers a way to unify their different data sources into a single endpoint. Although the promise of GraphQL is appealing, the process of setting up a GraphQL server that can communicate with each disparate data source can prove to be complex.

Scaphold.io provides GraphQL as a service, and today’s guests are the creators of Scaphold, Vince Ning and Michael Paris. Scaphold.io lets developers configure their schema, and hosts their data. Vince and Michael explain the basics of GraphQL, and also discuss how they are building a GraphQL as a service platform. 

The post GraphQL as a Service with Scaphold.io appeared first on Software Engineering Daily.

Clojure is a dynamically typed functional language that runs on the JVM. Today’s guest Alex Miller gives us an overview of Clojure’s core functionality. Alex is a developer of Cognitect, and a founder of the the Strange Loop conference.

We discuss the data structures, garbage collection, and concurrency support. How does Clojure compare to other JVM languages like Scala and Groovy? How does Clojure copy immutable data structures without copying all of the data? How does a Clojure program get evaluated and converted to Java bytecode? These questions, and many others are discussed in this episode.

The post Clojure with Alex Miller appeared first on Software Engineering Daily.

Odd Networks is building a platform for anyone to launch their own over-the-top streaming video service. With Odd Networks, you can deploy your own video channel using a Roku, Amazon Fire TV, Apple TV, and other services.

Creating a streaming video service with interoperability between these different platforms presents numerous technical challenges, and today’s guest Kris Walker explains how Odd Networks is addressing those challenges. We discuss the open source projects of Odd Networks, including oddworks, which encompasses the SDKs, stores, services, and middleware, and the event bus, called oddcast.

The post Odd Networks with Kris Walker appeared first on Software Engineering Daily.

Prometheus is an open-source monitoring tool built at SoundCloud. It can be used to produce detailed time-series data about a distributed architecture. Prometheus is based on the monitoring system inside Google’s infrastructure, called Borgmon.

Julius Volz is the creator of Prometheus, and he joins the show to explain why he built Prometheus and how it differs from previous monitoring tools. Prometheus is widely used to monitor Kubernetes clusters, just like Google uses Borgmon to monitor its Borg clusters.

The post Prometheus with Julius Volz appeared first on Software Engineering Daily.

github has grown to have 10 million users and 30 million repositories. Getting to this scale has required innovation in many places–github has significantly altered the code for git itself, and has built unique infrastructure and written low level code to architect for git repository management at scale.

Despite the need for cutting-edge technologies to support github, the development culture at github values tried and true technologies, and today’s guest Sam Lambert explains the value of maintaining a Rails monolith as the frontend of github, and the other battle-tested tools that github uses.

The post Scaling github with Sam Lambert appeared first on Software Engineering Daily.

Open source software has become the rule for how software is written rather than the exception. OSCON is O’Reilly’s open source conference, where companies and individuals talk about where the open-source world is going.

Rachel Roumeliotis is the chair of OSCON, and she joins the show today to talk about the state of open source, and how the conference has developed since she started working at O’Reilly.

We are giving away a free ticket to O’Reilly’s Velocity 2016 conference in New York. If you want to be entered to win that ticket, send a tweet about your favorite SE Daily episode about dev ops or web performance, and tag software_daily, as well as #velocityconf.

The post Open Source Culture with Rachel Roumeliotis appeared first on Software Engineering Daily.

The container management system Kubernetes was open sourced by Google with the intention of creating a cloud service based on the project. Today, the Kubernetes ecosystem is looking similar to the Android ecosystem, with different vendors providing different ways to use Kubernetes, from RedHat’s OpenShift to Google Container Engine.

Craig Mcluckie was a member of the team who originally devised Kubernetes, and he joins the show to talk about the origins of Kubernetes and where the project is headed, exploring both the technical architecture and Google’s business strategy around Kubernetes.

The post Kubernetes Origins with Craig Mcluckie appeared first on Software Engineering Daily.

Container management systems like Kubernetes and Docker Swarm give us a higher level management tool for architectures built out of distributed containers.

Container platforms like Rancher provide a higher layer of usability, and today’s guest Darren Shepherd of Rancher Labs takes us through what a container platform is. This interview is part of our continued coverage of Kubernetes, Docker, and the other components of the growing container ecosystem.

The post Container Platforms with Darren Shepherd appeared first on Software Engineering Daily.

Twenty years ago, Bill Joy talked about the eight fallacies of distributed computing–these are things such as “the network is reliable”, and “latency is zero”, and “bandwidth is infinite”, and these fallacies are even more relevant today. With the popularity of Docker containers, the networks of distributed systems that we deal with have become even more complex, and with this growing complexity comes new problems and new fallacies.
WeaveWorks is a company building a simple, reliable way to manage containers and microservices. The CEO of Alexis Richardson joins the show today to discuss how Weave uses convergent replicated data types to manage distributed systems. The use of the CRDT data structure is in contrast to the use of Paxos or Raft to algorithmically manage our distributed systems. Our conversation also touches on the Cloud Native Computing Foundation, which is similar to the Apache Foundation, but for cloud native application technologies.

The post Container Management with Alexis Richardson appeared first on Software Engineering Daily.

Virtual machines were the unit of cloud computation for many years. Amazon Web Services pioneered the democratized model of allowing anyone to deploy a service to the cloud, running on a virtual machine on Amazon’s servers. After virtual machines, containers have become the unit of scale in the cloud. We break up our virtualized servers into even smaller units of computation called containers. Today, the unit of compute is getting reduced even more, with the introduction of serverless architecture.

Serverless architectures started getting talked about after Amazon Web Services released a service called AWS Lambda, which allows users to have pieces of code run in response to events. Programmers write a function and hand it off to Amazon, and Amazon will run that function call, and only charge the programmer when the function is actually called. This is in contrast to the cost model of containers or virtual machines, which users pay for even while they are running idle.

Today’s guest Austen Collins believes that serverless computing is the model of the future, and he created a company called Serverless around this idea. His company Serverless provides an application framework for building applications exclusively on Amazon Web Services Lambda.

The post Serverless Framework with Austen Collins appeared first on Software Engineering Daily.

Kubernetes is the container management platform that came out of Google’s experiences managing data centers. Kubernetes abstracts away many of the frustrations of distributed systems management. OpenShift is a platform built on top of Kubernetes to provide an additional layer of usability.
Clayton Coleman is the lead engineer of OpenShift, and in our conversation today, we start with the basics of Kubernetes, then talk about OpenShift–Clayton explains why we need another abstraction on top of Kubernetes. Near the end of the conversation, we discuss the current state of cloud products–which can be confusing. Mesosphere, Swarm, Kubernetes, OpenStack, ECS–why do we need all of these different products? Clayton gives his perspective, and explains why it is not going to get any less confusing any time soon.

The post Kubernetes and OpenShift with Clayton Coleman appeared first on Software Engineering Daily.

Infrastructure-as-code is a trend that has been popularized over the past decade, as cloud computing and distributed systems have become a part of every technology company. Tools like Salt, Puppet, Chef, and Ansible allow us to manage servers and processes from the command line.
David Boucha works at SaltStack, the company that makes Salt. Salt is a platform that provides configuration management and remote execution.

In our conversation today, we discuss how the distributed systems architecture of Salt works, and how it can be used in practice. We also discuss infrastructure-as-code in the context of modern infrastructure tools like containers and Kubernetes.

The post Infrastructure as Code with SaltStack’s David Boucha appeared first on Software Engineering Daily.

Rust is a systems programming language being developed at Mozilla. Rust has features of a high-level functional language like Scala and a low-level, performance-driven language like C++.

Steve Klabnik is a developer program member with Mozilla. In this episode, he discusses how Rust looks at memory management, type safety, mutability, and concurrency. We also dive into a discussion of the low level virtual machine, also known as the LLVM, which the language Swift is built on.

The post Rust with Steve Klabnik appeared first on Software Engineering Daily.

Erlang is a programming language with primitives that help software engineers build distributed systems. When a process is malfunctioning in Erlang, the philosophy of the language is to let the process crash–and in a distributed system where unexpected faults happen on a regular basis, this philosophy of “let it crash” simplifies how we reason about an Erlang system.

Other distributed systems advantages of Erlang include the garbage collection strategy. Each process in Erlang has its own garbage collector which means makes it easier to construct systems without a stop-the-world garbage collection.

Francesco Cesarini is the founder of Erlang Solutions, and he joins the show today to discuss the book that he wrote with Steve Vinoski, called Designing for Scalability with Erlang/OTP.

The post Erlang Systems Design with Francesco Cesarini appeared first on Software Engineering Daily.

Google has built a microservices architecture on top of the internal container management system called Borg. These services communicate over an internal protocol known as Stubby. Borg and Stubby are tightly coupled to Google’s infrastructure–it would not make sense for Google to open source them–but Google has worked with the open source community to develop open source projects with the core functionality of Borg and Stubby. These projects are known as Kubernetes and gRPC.

Sandeep Dinesh works on the Google Cloud Platform team as a developer advocate. Our conversation explores how a client application request from an app like Gmail would communicate with Google’s servers, where the request is handled by a network of microservices. We also talk about where Google Cloud Platform is evolving, and how it offers a competitive, differentiated model from Amazon Web Services.

The post Google’s Microservices: Kubernetes and gRPC with Sandeep Dinesh appeared first on Software Engineering Daily.

At Netflix, building a user interface requires data about movies, actors, users, accounts, and much more. If each piece of data is coming from a different source, a programmer has to call each of the services that has access to that data. Falcor is a piece of technology designed at Netflix to solve this problem.
Falcor creates a unified JSON model of all of the data you could want–and Netflix engineer Brian Holt joins us today to discuss how Falcor works. We also discuss React, ES6, and JavaScript at Netflix, and Brian explains why Falcor is extremely similar to GraphQL, and why that is OK.

The post Falcor at Netflix with Brian Holt appeared first on Software Engineering Daily.

Kubernetes is an open-source distributed operating system that allows normal engineers to manage their resources at the scale that Google does. With Docker providing the APIs to make containers easy to work with, Kubernetes allows processes to be easily distributed and replicated across a data center.

Kubernetes was created at Google, and today’s guest Kelsey Hightower is a staff developer advocate working at Google. He joins me in a conversation about how Kubernetes works, and why it is important to an average developer. We touch on how software engineering is going to look in the next 5-10 years, and how a lot of the work around distributed systems will become much easier over time.

The post Kubernetes, Docker, and the Distributed Operating System with Kelsey Hightower appeared first on Software Engineering Daily.

Docker is an open platform for distributed applications. Docker allows for standardized container runtimes and will be an important piece of infrastructure for many years to come. But if you are new to Docker, the technology can be confusing and intimidating.

What is a container? Why do I want to put my application in a container? Today’s guest Prakhar Srivastav wrote a long tutorial explaining the fundamentals of Docker. Prakhar explains why Docker is useful, how Docker works, and how you can get started.

The post Docker for Beginners with Prakhar Srivastav appeared first on Software Engineering Daily.

Kubernetes has been a focus of several previous shows on Software Engineering Daily. The architecture and management of our distributed systems have changed, and widespread adoption of containers and container management systems is just beginning.
Joseph Jacks from Kismatic joins the show today to give us a perspective on the past, present, and future of container management. He also talks about his company Kismatic, which provides enterprise support for Docker and Kubernetes.

The post Kubernetes, Mesos, and Kismatic with Joseph Jacks appeared first on Software Engineering Daily.

“Functional programming is about making the complex parts of your system explicit.”

Elixir is a programming language built on top of the Erlang virtual machine. Elixir allows metaprogramming, polymorphism, and a web framework called Phoenix that has drawn positive comparisons to Ruby on Rails.

Jose Valim is today’s guest. He built Elixir to augment a language that he loved–Erlang. On Software Engineering Daily, we interviewed Joe Armstrong, the creator of Erlang, which was a very popular show–I encourage any listeners who are fans of Erlang to check it out. Erlang was built with concurrency in mind, and it has been rising in popularity as more of our applications are written to be distributed. In today’s episode we will discuss what Jose is building on top of Erlang.

Questions

• What are the strengths of Erlang?
• Why is the programming world becoming more interested in concurrency, distributed systems and functional languages?
• Can you contrast how errors are handled in Erlang and Elixir, compared to other languages?
• What was the reason for building Elixir? What did Erlang lack?
• Elixir compiles to bytecode for the Erlang VM – what does this mean?
• What is meta programming?
• How is the adoption of Phoenix and where is it headed?

Links

• Erlang
• Riak
• Elixir
• Managing two million web servers
• “In Elixir everything is an expression”
• Domain-specific Language
• Ecto
• Phoenix
• Platformatec
• Orleans
• José on Twitter
  

The post Elixir and Erlang with Jose Valim appeared first on Software Engineering Daily.

Nadia Eghbal joins us as the host of today’s show to discuss the past, present, and future of open-source software. This monologue was adapted from Nadia’s article, We’re in a brave, new post open source world.

In this episode, Nadia describes the origins of open-source in the Free software movement, its rise to popularity, and today’s golden era where mainstream and popular technologies like React, Spark and Docker are all open source projects.

Yet there are several serious issues that threaten the foundation upon which the current open-source renaissance is happening. Nadia discusses what these are, and how to resolve these issues so that we can continue to have a vibrant ecosystem for developers.

Thank you to Nadia Eghbal for coming on the show and for standing up for open source software.

The post The Past, Present, and Future of Open Source appeared first on Software Engineering Daily.

“Git is a fairly flexible tool – one of the ways that people often describe it is that it gives you enough rope to hang yourself.”

Git is the most popular version control system. If you have been programming for less than a decade, it’s likely that you haven’t used any other method of version control. The git workflow of a software team defines how that team collaborates, builds, and ships software.

Tim Pettersen is a developer advocate at Atlassian, where he has been building software around git for many years. He joins us on today’s episode to talk about strategies for git–including branching, merging, continuous integration, and software as a service.

Questions

• Why should we use Git?
• How does Mercurial compare to Git, and why is it a compelling alternative?
• What are the characteristics of a good Git workflow?
• What is the difference between a good workflow and a great one?
• What is a rebase?
• Why is code review so important?
• What are the advanced ways you work with Git?
• Are there any underutilized features of Git?

Links

• Git
• Atlassian
• Mercurial
• JIRA
• Forking workflow
• Git rebase
• Pro Git
• Fast-Forward Git Merge
  
• Tim on Twitter
  

The post Git Workflows with Tim Pettersen appeared first on Software Engineering Daily.

“Why do we need any open source versions of proprietary implementations? I would argue that first of all, it’s just good for industry and the ecosystem.”

Cloud service providers like Amazon, Google, and Microsoft provide both infrastructure as a service and platform as a service. Infrastructure as a service gives developers access to virtual machines, servers, and network infrastructure. Platform as a service is the software that runs on top of that infrastructure as a service–this includes things like Amazon DynamoDB, Microsoft Azure Machine Learning, and Google App Engine.

OpenStack is an open-source cloud operating system. Today’s guest is John Purrier, a founder of OpenStack and the CTO of Automic Software.

Questions

• What is Infrastructure-as-a-Service (IaaS)?
• Why do we need an open source cloud operating system?
• The hypervisors managed by OpenStack can be different, but the user has a consistent interface – why is this advantageous?
• How much more efficient are containers when compared to virtual machines?
• What is the difference between scalability and elasticity?
• Why is shared state dangerous?
• How do you manage to test a system like OpenStack?
• How will OpenStack, AWS and other cloud platforms evolve going into the future?
• Are we headed towards a polyglot cloud world?

Links

• OpenStack
• Virtualization
• Hypervisor
• EC2
• Rackspace
• Appfog
• Job schedulers
• Nicira
• Message Bus
• Eventual consistency
• Automic
  

The post OpenStack and the Future of Cloud Computing with John Purrier appeared first on Software Engineering Daily.

“The most important thing behind it is to think about developers in the way that product people think about consumers, and that the first time experience of your API needs to be ridiculously simple.”

Software developers have been socializing on chat rooms for decades. In the nineties, we began using IRC and AOL instant messenger. In the early 2000s, we turned to Google Hangouts and Yammer. Today, we are using modern apps like Slack and Hipchat.

On today’s show, we take a deep dive into Gitter, the chat client specifically designed for developers. Our guests are Mike Bartlett and Andrew Newdigate, the creators of Gitter. Gitter is a highly scalable, real time social application for developers to talk about writing their software. This is a great episode that spans topics like front end development, back end distributed systems, how to compete with Slack, and how to scale a chat room to tens of thousands of active users.

Questions

• What are the chat platforms that developers use to communicate right now?
• Why were you compelled to build Gitter?
• What is the stack behind Gitter?
• How often do you decide to snapshot your messages for replication?
• What are the challenges around having so many chat integrations with other applications?
• How do you build an effective API that developers are compelled to use?
• How does Gitter compare to Slack, and how do you see the two evolving?
• Does the emergence of more sophisticated chat clients hint toward more distributed development teams?

Links

• IRC
• Gitter
• Free Code Camp with Quincy Larson
• CometD
• Why Bayeux still matters
• Redis Sentinel
• Datadog
• webpack
• Never Go Full Retard
• WebSockets
• Dear Github…
• Andrew on Twitter
• Mike on Twitter

The post Gitter Engineering with Mike Bartlett and Andrew Newdigate appeared first on Software Engineering Daily.

“I have a really fun time being in open source, and I want other people in the community to feel the same way as well. Programming isn’t just my job – its also my hobby.”

Ruby on Rails has had the most commits made by its creator, David Heinemeier Hansson. The next most frequent contributor to Rails is Aaron Patterson, our guest on today’s episode of Software Engineering Daily. Topics discussed in today’s episode include Ruby relative to JavaScript, Rails relative to Linux, and DHH relative to Tenderlove.

Questions

• What is it about Ruby on Rails that motivates you to contribute so much to the project?
• How was Rails different from the frameworks you had used previously?
• What impact has Rails had on the world of software development?
• Why is working on open source projects more emotionally rewarding for you?
• Is the Rails OSS community unique when compared to other open source projects?
• Are you also in agreement that the tradeoff in performance for developer ergonomics is a crucial reason for ruby and rails’ success?
• Why is blogging a good complement to the other work you do in the community?
• What is the future of Ruby on Rails?

Links

• Rails Repo
• J2EE
• ATG DYNAMO
• SemVer
• Active record
• H2
• Monkey patching
• Tenderlove on Twitter
  

The post Open Source and Rails with Aaron Patterson appeared first on Software Engineering Daily.

“Does the real world match the truth? If not, make the world match the truth.”

You keep hearing about containers, and maybe you have even used Docker in production. Now, it’s time to move beyond Docker. In today’s episode of Software Engineering Daily, Steve Pousty talks about OpenShift, a platform from RedHat that helps engineers leverage the power of containers and the dev ops harmony of microservices.

To hear these buzzwords, you can always listen to Software Engineering Daily. But to see these buzzwords demonstrated in live presentations, the 2016 O’Reilly Fluent Conference is coming up March 8-10 in San Francisco, and you can get a chance to win a free ticket by tweeting at us about your favorite episode of Software Engineering Daily. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet.

Steve Pousty is a Developer Advocate (and PaaS Dust Spreader) at Red Hat.

Questions

• What drives the difference in perspective between dev and ops?
• What is OpenShift and how does it compare to other Paas products?
• Why is Docker useful for a platform as a service like OpenShift?
• Why would I use Openshift to build my microservices architecture?
• What are some best practices for dev and ops to manage their working relationship?
• How does testing fit into the relationship between developers and operations?
• What is the future of the platform as a service?

Links

• OpenShift
• Steve on Twitter
• Steve’s talk at Fluent
• PaaS
• IaaS
• Tomcat
• Docker Swarm
• CoreOS
• Blazemeter
• NoOps
  

The post Containers and PaaS with Steve Pousty appeared first on Software Engineering Daily.

“The business of technology and the technology of technology are kind of converging if you ask me. And there is definitely a space for some publications that don’t have decades of technical debt in the software space.”

In this episode, Jeff and Derrick sit down to discuss the state of tech journalism, and how it has kept up with the rapid pace of technology innovation. Additionally, they discuss the evolution of the “big data” stack and where it is headed. The conversation focuses in on Apache Mesos and Mesosphere, the company building the Datacenter Operating System (DCOS).

Derrick Harris is a long time tech journalist, having worked at GigaOm and Fortune as a writer covering tech topics. He currently works at Mesosphere, where he is a tech evangelist.

Questions

• What were your goals as a software journalist?
• What is Mesosphere?
• Which technologies do you love covering?
• What are the aspects of Google and Facebook infrastructure that Mesosphere is helping to democratize?
• What are the challenges with building a great user experience for the data center?
• What is a single-technology company?

Links

• Mesosphere
• Apache Mesos
• Marathon
• Berkeley AMPLab
• Scale BLOG
• SQRL
• Investing in Big Data Technologies with Ben Horowitz
• Derrick on Twitter
  

The post Mesosphere and Tech Journalism with Derrick Harris appeared first on Software Engineering Daily.

“Don’t be so rigid in your methodologies and beliefs and whatever ecosystem you’re in. Everybody’s gotta be a little bit flexible because that rigidity doesn’t work. The only thing that we do as technologists that is rigid is that the compiler isn’t going to take a syntax error.”

Django is an open source web development framework that was initially released in 2005, around the same time as Ruby on Rails. In many ways, the two frameworks are similar – they expedited web development by modularizing common components of web apps and enabling developers to quickly prototype and launch their products. They both encouraged the use of the MVC architectural pattern, with each respectively acting as the controller, allowing developers to use their own front end and database technologies.

But the path of the two frameworks diverged in other parameters, including mainstream adoption. Ruby on Rails became the sweetheart in Silicon Valley and elsewhere, while Django was loved by a core constituency of pythonistas who had used Python in other applications and fallen in love with it. At the peak of the frameworks’ popularity, there was a common question from new developers asking whether they should first learn Django or Rails – the answer often boiled down to whether you wanted to use Python or Ruby. In this episode, Frank discusses the history and growth of Django, and explains how it evolved in comparison with other frameworks.

Frank Wiles is the President of the Django Software Foundation, and partner at Revolution Systems. He worked at the Lawrence Journal-World newspaper, where Django was initially conceived and created.

Questions

• How did Django evolve from a newspaper office in Lawrence, Kansas?
• How did your career grow with the evolution of Django?
• What are the tradeoffs for a new programmer in the onboarding process?
• What is Django’s philosophy on asynchronous or event-driven programming?
• What architectural decisions would you make differently if Django was started today?
• What are some common mistakes you see people make in their Django applications?
• What does an organization look like once it achieves “DevOps nirvana”?

Links

• Django Software Foundation
• Django
• The Lawrence Journal-World
• Flask
• Celery
• Podcast.__init__
• Revolution Systems
• Spectrum
• Frank’s personal page
  

The post Django with Frank Wiles appeared first on Software Engineering Daily.

“Actually it’s more work to turn a table into a chair, than it is to just make a damn chair.”

Ruby on Rails is a web app framework released 10 years ago that influenced the way websites were being built. Rails skyrocketed to popularity in the late 2000’s and empowered many small companies to quickly build and maintain robust web apps.

While it is still a mainstay in web development, it has been overshadowed as of late by JavaScript, and Node.js on the backend. David joins Software Engineering Daily to discuss how Ruby on Rails has evolved to keep up with newer programming paradigms and why it is still an excellent choice to build web apps with nowadays.

David Heinemeier Hansson is the creator of Ruby on Rails and a partner at Basecamp. Rails was extracted from the work done at Basecamp, and open-sourced for the public to commit to and use for their applications.

Questions

• How has the world of software engineering changed since Rails was first open sourced 10 years ago?
• Are there any canonical problems that have driven Rails over the last decade?
• Why did you rewrite Basecamp from scratch?
• Which programming language best fits your brain?
• What is the difference between software writing and software engineering?
• What are the practices of software development that are pseudoscientific?
• Why do startup founders feel compelled to raise so much money when costs are historically low?

Links

• Basecamp
• Ruby on Rails
• AJAX
• Technical Bankruptcy
• Web 2.0
• CoffeeScript
• RailsConf 2014 – Keynote: Writing Software by David Heinemeier Hansson
• Reconsider
• Zero to One
• David’s personal page
  

The post The Evolution of Rails with David Heinemeier Hansson appeared first on Software Engineering Daily.

“Every time I’ve gone to a conference, there’s always been some new idea or some new way of working that I’ve been exposed to.”

Marty Haught is a director of Ruby Central, a nonprofit dedicated to the support and advocacy of the worldwide ruby community. Ruby Central also hosts the RubyConf and RailsConf software conferences.

Questions

• When did you start working with Ruby?
• What lead to Ruby groups having a different ethos than other language user groups?
• How did RubyCentral get started, and what is its goal?
• What is the value of going to developer conferences?
• How can the programmer community encourage more people to become programmers?
• Has the emergence and new popularity of JavaScript displaced Ruby on Rails?

Links

• Ruby Central
• RubyConf
• RailsConf
• MINASWAN
• Marty’s personal page
  

The post The Ruby Community with Marty Haught appeared first on Software Engineering Daily.

“It really seemed like Rails just put everything together, so I thought well let’s give this thing a try, and I liked it.”

Michael Hartl is the author of The Ruby on Rails Tutorial, a widely acclaimed guide to learning how to build Ruby on Rails webapps.

Questions

• You have a background in physics, including getting a PhD studying black hole dynamics – why did you switch careers to software?
• What was unique about the Ruby on Rails Tutorial when it first came out?
• Why didn’t Python and Django take off in the same way Ruby on Rails did?
• What do you think about the shift from Rails to full-stack JavaScript?
• Is there anything about Rails that you’re not a fan of?
• What are you building at your new company?

Links

• OpenAI
• Ruby on Rails Tutorial
• Softcover
• Michael’s page
  

The post Learning Rails with Michael Hartl appeared first on Software Engineering Daily.

“Docker has made containers much more accessible to developers, and the timing is actually very good because of the whole DevOps infrastructure as code movement.”

Kitematic is an open source project built to simplify and streamline using Docker on a Mac or Windows machine. It allows users to get up and running on Docker with a single click, and provides a user interface that makes running and managing Docker files easy. Kitematic was acquired by Docker in March 2015.

Sean Li is the co-founder and Chief Product Officer of Kitematic. He is now working on product design and developer experience at Docker.

Questions

• What was new about Docker that got people, including yourself, so excited?
• Is Docker a technology that is useful for consumers?
• Why did you build Kitematic?
• What type of user did you have in mind when you first designed the Kitematic UI?
• How do you see the Docker ecosystem evolving?
• What are the major technical bottlenecks in the Docker ecosystem?
• How do you use React components in a desktop app?

Links

• Kitematic
• VirtualBox
• Docker Machine
• Github Desktop App
• Lightspeed Ventures Summer Fellowsship
• Hackers and Painters – Paul Graham
• Electron – Atom
• Sean on Twitter
  

The post Simplifying Docker with Sean Li appeared first on Software Engineering Daily.

Apache Mesos is an open-source cluster manager that enables resource sharing in a fine-grained manner, improving cluster utilization.

Michael Hausenblas is a developer and cloud advocate with Mesosphere, which builds the Datacenter Operating System (DCOS), a distributed OS that uses Apache Mesos as its kernel.

Questions

• Can you give the historical context for cluster computing?
• How are the distributed systems needs of different companies changing?
• What is Apache Mesos and how does it fit into this ecosystem?
• What are the operational problems that Mesos alleviates?
• What is the history of Kubernetes?
• How does a container work?
• Where is the interface between Mesos and Kubernetes?
• Does Kubernetes provide any cluster-wide services that Mesos can take advantage of?

Links

• Apache Mesos
• Kubernetes
• Marathon
• Borg: The Predecessor to Kubernetes
• Can we have Kubernetes on Mesos?
• Michael on Twitter

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Mesos and Docker in Practice with Michael Hausenblas appeared first on Software Engineering Daily.

“If you want to use the newest technology, there is always a segment of viewers that will not be able to view the website well.”

Free Lossless Image Format (FLIF) is a novel lossless image format which outperforms PNG, lossless WebP, lossless BPG and lossless JPEG2000 in terms of compression ratio.

Jon Sneyers is one of the core contributors to FLIF.

Questions

• Why is image compression so important?
• What is FLIF?
• What is progressive decoding?
• How did you come up with maniac compression?
• Why don’t we see more webp images on the internet?
• What are the differences between Webp and FLIF?

Links

• FLIF
• FLIF Compression Benchmarks
• WebP
• Jon on Twitter

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Free Lossless Image Format with Jon Sneyers appeared first on Software Engineering Daily.

“It was claimed that these [object-oriented] languages were better for modularity – and I don’t believe so.”

Caml is a functional programming language that is a dialect of the ML programming language family, developed at INRIA and formerly at ENS.

Gérard Huet is a senior researcher at INRIA, the French Institute for Research and Automation. He helped develop the Caml programming language in the 80s, and has a variety of other accomplishments in the world of computer science, including developing the Coq Proof Assistant System.

Questions

• What is ML?
• Why was LISP so popular?
• What was the purpose of proof assistants?
• The first implementation of Caml appeared in 1987. What was the spec for the language?
• What new features did Objective Caml (OCaml) bring to the language?
• What are the numerical humanities?

Links

• The Caml Language
• The Coq Proof Assistant
• Gérard’s personal page

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Caml with Gérard Huet appeared first on Software Engineering Daily.

“I think what functional programming does is codify what we’ve discovered are good ways to think about problems most of the time.”

Racket is a functional programming language similar to Lisp and Scheme.

Matthew Flatt is a member of the Racket core development team and also a professor of computer science at the University of Utah.

Questions

• What characterizes a Scheme language or a Lisp language?
• What is an s-expression?
• Why is Racket a good language to teach students how programming languages work?
• What’s the difference between a functional and a pure-functional language?
• What are the most important evolutionary moments in the history of programming languages?
• How would you define a domain specific language?
• What kinds of companies are using Racket?

Links

• Racket is …
• Why Racket? Why Lisp?
• Domain Specific Languages
• Dr. Flatt’s home page

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Racket with Matthew Flatt appeared first on Software Engineering Daily.

“There are entire days where I don’t even look at the browser when I’m coding in Elm because I just know its going to work.”

Elm is a functional programming language for web browsers.

Richard Feldman and Srinivas Rao are front­-end developers who work with Elm at NoRedInk.

Questions

• What are the problems with web development that Elm is solving?
• What is functional reactive programming?
• What is a signal?
• Can you describe the Elm architecture?
• How does the Elm architecture compare to the classic model view controller strategy?
• What’s the future of Elm?

Links

• Elm
• Why I designed a front-end programming language from scratch
• Building a Live-Validated Signup Form in Elm
• Richard on Twitter
• Rao on Twitter

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Elm with Richard Feldman and Srinivas Rao appeared first on Software Engineering Daily.

“Mutable state is the root of all evil.”

Erlang is a functional, concurrent programming language that was originally designed within Ericsson in the 1980’s. It was built to support distributed, fault-tolerant, non-stop applications suitable for telecommunications infrastructure.

Joe Armstrong is one of the designers of Erlang, and the chief architect of the Open Telecom Platform (OTP), a framework for building Erlang applications.

Questions

• Why has Erlang persisted as such a popular tool for communications platforms?
• In the domain of telecom, what were the valuable features of languages like Lisp and Prolog?
• How did Erlang germinate?
• Can you give a high level description for how someone using Erlang should be thinking about concurrency?
• What are the advantages of functional programming languages?
• Why did object oriented programming become so prevalent and so widely used?
• How has the Erlang community changed over time?

Links

• Erlang
• Development of the AXE System
• Joe on Twitter

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The post Erlang with Joe Armstrong appeared first on Software Engineering Daily.

GraphQL is a query language for managing client-server interactions.

Relay is a new way of structuring client applications that co-locates data fetching with view logic.

These technologies complement the paradigm shift of React and other Facebook technologies.

Nick Schrock is the co-creator of GraphQL and an engineer on product infrastructure at Facebook.

Questions

• How does REST manage client server interaction?
• Before GraphQL, how did Facebook deal with typical client-server issues?
• If a developer wants to work with GraphQL, do they need to use Relay as well?
• Could you describe a mutation in Relay?
• Why do Relay and GraphQL work nicely with React?
• How does GraphQL fit into the Flux architecture?

Links

• GraphQL Introduction
• GraphQL Specification
• Relay:  A JavaScript Framework for Building Data-Driven React Applications
• Nick Schrock on JavaScript Jabber

The post GraphQL and Relay with Nick Schrock appeared first on Software Engineering Daily.

Functional languages encourage practices and patterns that can simplify concurrent programming.

Scala, Clojure, and Akka are functional tools built on the Java Virtual Machine.

Jessica Kerr is a functional developer on the JVM. She currently works at Monsanto. At QCon San Francisco, she will be giving a talk called Contracts in Clojure: Settling Types vs. Tests.

Questions

• How does Scala compare to Clojure?
• What are some best practices for using threadpools?
• What is the actor model?
• Does functional programming simplify concurrency?
• What are the most important lessons from your talk Functional Principles for Object Oriented Development?

Links

• Jessica Kerr at QCon SF
• YouTube: Functional Principles for Object-Oriented Development
• Jessica Kerr Blog
• Abstractivate: Functional Programming Blog
• Stereotype Threat

The post Functional Programming with Jessica Kerr appeared first on Software Engineering Daily.

Microsoft is open-sourcing an increasing range of software.

Phil Haack is a former PM on Microsoft’s ASP.NET MVC Framework, as well as NuGet, an open-source package manager. He currently works on Github Desktop at Github.

Questions

• What is .NET?
• How is Microsoft’s policy around open-source changing?
• When Microsoft moved code to open-source at Github, how did communication change?
• What was your strategy for managing the development of NuGet?
• How does working at Github differ from working at Microsoft?
• What are the biggest criticisms of git?

Links

• QCon SF: The Amazing Potential of Microsoft Open-Source
• Github Flow Aliases
• Microsoft Open Source
• Phil Haack’s blog
• SE Daily: TypeScript with Jon Turner
• Why You Should Care About the New Open Source .NET

The post Microsoft Open-Source with Phil Haack appeared first on Software Engineering Daily.

Microservices on container architecture lead to effective workflows associated with DevOps.

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run. This allows for quick flexibility and scaleability in business applications, a key driver of the DevOps methodology.

James Turnbull is VP of engineering at Kickstarter, an advisor at Docker and author of The Docker Book.

Questions:

• How can Docker help you with change management in high-frequency trading?
• Should you build microservices from day 1?
• How does Docker interact with clustering technology like Mesos?
• What does the Docker ecosystem imply about the future of operating systems?

Links:

• What is Docker?
• Software Engineering Radio: James Turnbull on Docker
• Docker Slideshares
• Knight Capital: A DevOps Cautionary Tale

The post Docker and Microservices with James Turnbull appeared first on Software Engineering Daily.

Apache ZooKeeper enables highly reliable distributed coordination.

Flavio Junqueira is a committer and PMC of Apache ZooKeeper, and former VP of ZooKeeper.

Questions include:

• Why is master election so important in Hadoop?
• How does a new user begin working with ZooKeeper?
• How do nodes “watch” each other?
• Should ZooKeeper be used as a message queue or notification system?
• What is ZooKeeper’s place in a data center made increasingly complicated by tools like Mesos and Docker?
• How will the interactions between managed Big Data companies change in the future?
• Why wasn’t a company built around ZooKeeper?
• What is the future of the data center?

Links:

• ZooKeeper: Distributed Process Coordination by Flavio Junqueira and Ben Reed
• Apache ZooKeeper home page
• Keep moving forward: Liveness in distributed systems by Flavio Junqueira
• Libsei: scalable error isolation for distributed systems
• Apache Kafka with Flavio on SE Radio

The post Apache ZooKeeper with Flavio Junqueira appeared first on Software Engineering Daily.