Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Patrick and Jacob sit down with Matei Zaharia, Co-Founder and CTO at Databricks and Professor at Stanford. They discuss how companies are training and serving models in production with Databricks, where LLMs fall short for search and how to improve them, the state of the art AI research at Stanford, and how the size and cost of models is likely to change with technological advances in the coming years.

(0:00) - Introduction

(2:04) - Founding story of Databricks

(6:03) - PhD classmates using early version of spark for Netflix competition

(6:55) - Building applications with MLFlow

(9:55) - LLMs and ChatGPT

(12:05) - Working with and fine-tuning foundation models

(13:00) - Prompt engineering here to stay or temporary?

(15:12) - Matei’s research at Stanford. The Demonstrate-Search-Predict framework (DSP)

(17:42) - How LLMs will be combined with classic information retrieval systems for world-class search

(19:38) - LLMs writing programs to orchestrate LLMs

(20:36) - Using LLMs in Databricks cloud product

(24:21) - Scaling LLM training and serving

(27:29) - How much will cost to train LLMs go down in coming years?

(29:22) - How many parameters is too many?

(31:14) - Open source vs closed source?

(35:19) - Stanford AI research - Snorkel, ColBERT, and More

(38:58) - Matei getting a $50 amazon gift card for weeks of work

(43:23) - Quick-fire round

With your co-hosts:

@jasoncwarner

- Former CTO GitHub, VP Eng Heroku & Canonical

@ericabrescia

- Former COO Github, Founder Bitnami (acq’d by VMWare)

@patrickachase

- Partner at Redpoint, Former ML Engineer LinkedIn

@jacobeffron

- Partner at Redpoint, Former PM Flatiron Health