Start / Data Engineering Podcast / Building applications with data as code on the dataos

Building Applications With Data As Code On The DataOS

49 min • 16 januari 2023

Summary

The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. Unfortunately it has also introduced new overhead to manage the full experience as a single workflow. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery. In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL. Go to dataengineeringpodcast.com/materialize today and sign up for early access to get started. If you like what you see and want to help make it better, they're hiring across all functions!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, dbt models, Airflow jobs, and business intelligence tools, reducing time to detection and resolution from weeks to just minutes. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more.
Data and analytics leaders, 2023 is your year to sharpen your leadership skills, refine your strategies and lead with purpose. Join your peers at Gartner Data & Analytics Summit, March 20 – 22 in Orlando, FL for 3 days of expert guidance, peer networking and collaboration. Listeners can save $375 off standard rates with code GARTNERDA. Go to dataengineeringpodcast.com/gartnerda today to find out more.
Your host is Tobias Macey and today I'm interviewing Srujan Akula about DataOS, a pre-integrated and managed data platform built by The Modern Data Company

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what your mission at The Modern Data Company is and the story behind it?
Your flagship (only?) product is a platform that you're calling DataOS. What is the scope and goal of that platform?
- Who is the target audience?
On your site you refer to the idea of "data as software". What are the principles and ways of thinking that are encompassed by that concept?
- What are the platform capabilities that are required to make it possible?
There are 11 "Key Features" listed on your site for the DataOS. What was your process for identifying the "must have" vs "nice to have" features for launching the platform?
Can you describe the technical architecture that powers your DataOS product?
- What are the core principles that you are optimizing for in the design of your platform?
- How have the design and goals of the system changed or evolved since you started working on DataOS?
Can you describe the workflow for the different practitioners and stakeholders working on an installation of DataOS?
What are the interfaces and escape hatches that are available for integrating with and extending the operation of the DataOS?
What are the features or capabilities that you are expressly choosing not to implement? (e.g. ML pipelines, data sharing, etc.)
What are the design elements that you are focused on to make DataOS approachable and understandable by different members of an organization?
What are the most interesting, innovative, or unexpected ways that you have seen DataOS used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on DataOS?
When is DataOS the wrong choice?
What do you have planned for the future of DataOS?

Contact Info

LinkedIn
@srujanakula on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA