166: Data Processing Fundamentals and Building a Unified Execution Engine Featuring Pedro Pedreira of Meta

Highlights from this week’s conversation include:

• The concept of composable at a lower level of data infrastructure (1:28)
• New architectures and components that allow developers to build databases (3:44)
• Pedro's background and experience in data infrastructure (6:18)
• The Spectrum of Latency and Analytics (12:59)
• Different Query Engines for Different Use Cases (16:32)
• Vectorized vs Code Gen Data Processing (19:33)
• Vectorization and Code Generation (21:21)
• Examples of Vectorized Engines (24:33)
• Rewriting Execution Engine in C++ (27:22)
• Different Organization of Presto and Spark (33:17)
• Arrow and its Extensions (37:15)
• The similarities between analytics and ML (44:33)
• Offline feature engineering and data preprocessing for training (48:00)
• Dialect and semantic differences in using Velox for different engines (50:01)
• The convergence of dialects (52:23)
• Challenges of substrate and semantics (53:18)
• Future plans for Velox (58:09)
• The discussion on evolving Parquet (1:03:38)
• The integration of the relational model and the tensor model (1:07:29)

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.