32 avsnitt • Längd: 50 min • Månadsvis
Join us as we train our neural nets on the theme of the century: AI. Sonya Huang, Pat Grady and more Sequoia Capital partners host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies—and their implications for technology, business and society.
The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.
The podcast Training Data is created by Sequoia Capital. The podcast and the artwork on this page are embedded on this page using the public podcast feed (RSS).
Palo Alto Networks’s CEO Nikesh Arora dispels DeepSeek hype by detailing all of the guardrails enterprises need to have in place to give AI agents “arms and legs.” No matter the model, deploying applications for precision-use cases means superimposing better controls. Arora emphasizes that the real challenge isn’t just blocking threats but matching the accelerated pace of AI-powered attacks, requiring a fundamental shift from prevention-focused to real-time detection and response systems. CISOs are risk managers, but legacy companies competing with more risk-tolerant startups need to move quickly and embrace change.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
MongoDB product leader Sahir Azam explains how vector databases have evolved from semantic search to become the essential memory and state layer for AI applications. He describes his view of how AI is transforming software development generally, and how combining vectors, graphs and traditional data structures enables high-quality retrieval needed for mission-critical enterprise AI use cases. Drawing from MongoDB's successful cloud transformation, Azam shares his vision for democratizing AI development by making sophisticated capabilities accessible to mainstream developers through integrated tools and abstractions.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
Stef Corazza leads generative AI development at Roblox after previously building Adobe’s 3D and AR platforms. His technical expertise, combined with Roblox’s unique relationship with its users, has led to the infusion of AI into its creation tools. Roblox has assembled the world’s largest multimodal dataset. Stef previews the Roblox Assistant and the company’s new 3D foundation model, while emphasizing the importance of maintaining positive experiences and civility on the platform.
Mentioned in this episode:
Hosted by: Konstantine Buhler and Sonya Huang, Sequoia Capital
Ioannis Antonoglou, founding engineer at DeepMind and co-founder of ReflectionAI, has seen the triumphs of reinforcement learning firsthand. From AlphaGo to AlphaZero and MuZero, Ioannis has built the most powerful agents in the world. Ioannis breaks down key moments in AlphaGo's game against Lee Sodol (Moves 37 and 78), the importance of self-play and the impact of scale, reliability, planning and in-context learning as core factors that will unlock the next level of progress in AI.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Hema Raghavan is co-founder of Kumo, a company that makes graph neural networks accessible to enterprises by connecting to their relational data stored in Snowflake and Databricks. Hema talks about how running GNNs on GPUs has led to breakthroughs in performance as well as the query language Kumo developed to help companies predict future data points. Although approachable for non-technical users, the product provides full control for data scientists who use Kumo to automate time-consuming feature engineering pipelines.
Mentioned in this episode:
Hosted by: Konstantine Buhler and Sonya Huang, Sequoia Capital
Berkeley professor Ion Stoica, co-founder of Databricks and Anyscale, transformed the open source projects Spark and Ray into successful AI infrastructure companies. He talks about what mattered most for Databricks' success -- the focus on making Spark win and making Databricks the best place to run Spark. He highlights the importance of striking key partnerships -- the Microsoft partnership in particular that accelerated Databricks' growth and contributed to Spark's dominance among data scientists and AI engineers. He also shares his perspective on finding new problems to work on, which holds lessons for aspiring founders and builders: 1) building systems in new areas that, if widely adopted, put you in the best position to understand the new problem space, and 2) focusing on a problem that is more important tomorrow than today.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Oege de Moor, the creator of GitHub Copilot, discusses how XBOW’s AI offensive security system matches and even outperforms top human penetration testers, completing security assessments in minutes instead of days. The team’s speed and focus is transforming the niche market of pen testing with an always-on service-as-a-software platform. Oege describes how he is building a large and sustainable business while also creating a product that will “protect all the software in the free world.” XBOW shows how AI is essential for protecting software systems as the amount of AI-generated code increases along with the scale and sophistication of cyber threats.
Hosted by: Konstantine Buhler and Sonya Huang, Sequoia Capital
Mentioned in this episode:
When ChatGPT ushered in a new paradigm of AI in everyday use, many companies attempted to adapt to the new paradigm by rushing to add chat interfaces to their products. Eric has a different take—he doesn’t think chatbots are the right form factor for everything. He thinks “zero-touch” automation that works invisibly in the background can be more valuable in many cases. He cites self-driving cars as an analogy—or in this case, “self-driving money.” Ramp is a new kind of finance management company for businesses, offering AI-powered financial tools to help companies handle spending and expense processes. We’ll hear why Eric thinks AI that you never see is one of the most powerful instruments for reducing time spent on drudgery and unlocking more time for meaningful work.
Hosted by: Ravi Gupta and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Founded in early 2023 after spending years at Stripe and OpenAI, Gabriel Hubert and Stanislas Polu started Dust with the view that one model will not rule them all, and that multi-model integration will be key to getting the most value out of AI assistants. In this episode we’ll hear why they believe the proprietary data you have in silos will be key to unlocking the full power of AI, get their perspective on the evolving model landscape, and how AI can augment rather than replace human capabilities.
Hosted by: Konstantine Buhler and Pat Grady, Sequoia Capital
00:00 - Introduction
02:16 - One model will not rule them all
07:15 - Reasoning breakthroughs
11:15 - Trends in AI models
13:32 - The future of the open source ecosystem
16:16 - Model quality and performance
21:44 - “No GPUs before PMF”
27:24 - Dust in action
37:40 - How do you find “the makers”
42:36 - The beliefs Dust lives by
50:03 - Keeping the human in the loop
52:33 - Second time founders
56:15 - Lightning round
Clay is leveraging AI to help go-to-market teams unleash creativity and be more effective in their work, powering custom workflows for everything from targeted outreach to personalized landing pages. It’s one of the fastest growing AI-native applications, with over 4,500 customers and 100,000 users. Founder and CEO Kareem Amin describes Clay’s technology, and its approach to balancing imagination and automation in order to help its customers achieve new levels of go-to-market success.
Hosted by: Alfred Lin, Sequoia Capital
Can GenAI allow us to connect our imagination to what we see on our screens? Decart’s Dean Leitersdorf believes it can.
In this episode, Dean Leitersdorf breaks down how Decart is pushing the boundaries of compute in order to create AI-generated consumer experiences, from fully playable video games to immersive worlds. From achieving real-time video inference on existing hardware to building a fully vertically integrated stack, Dean explains why solving fundamental limitations rather than specific problems could lead to the next trillion-dollar company.
Hosted by: Sonya Huang and Shaun Maguire, Sequoia Capital
00:00 Introduction
03:22 About Oasis
05:25 Solving a problem vs overcoming a limitation
08:42 The role of game engines
11:15 How video real-time inference works
14:10 World model vs pixel representation
17:17 Vertical integration
34:20 Building a moat
41:35 The future of consumer entertainment
43:17 Rapid fire questions
Years before co-founding Glean, Arvind was an early Google employee who helped design the search algorithm. Today, Glean is building search and work assistants inside the enterprise, which is arguably an even harder problem. One of the reasons enterprise search is so difficult is that each individual at the company has different permissions and access to different documents and information, meaning that every search needs to be fully personalized. Solving this difficult ingestion and ranking problem also unlocks a key problem for AI: feeding the right context into LLMs to make them useful for your enterprise context. Arvind and his team are harnessing generative AI to synthesize, make connections, and turbo-change knowledge work. Hear Arvind’s vision for what kind of work we’ll do when work AI assistants reach their potential.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
00:00 - Introduction
08:35 - Search rankings
11:30 - Retrieval-Augmented Generation
15:52 - Where enterprise search meets RAG
19:13 - How is Glean changing work?
26:08 - Agentic reasoning
31:18 - Act 2: application platform
33:36 - Developers building on Glean
35:54 - 5 years into the future
38:48 - Advice for founders
In recent years there’s been an influx of theoretical physicists into the leading AI labs. Do they have unique capabilities suited to studying large models or is it just herd behavior? To find out, we talked to our former AI Fellow (and now OpenAI researcher) Dan Roberts.
Roberts, co-author of The Principles of Deep Learning Theory, is at the forefront of research that applies the tools of theoretical physics to another type of large complex system, deep neural networks. Dan believes that DLLs, and eventually LLMs, are interpretable in the same way a large collection of atoms is—at the system level. He also thinks that emphasis on scaling laws will balance with new ideas and architectures over time as scaling asymptotes economically.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
AI Math Olympiad: Dan is on the prize committee
NotebookLM from Google Labs has become the breakout viral AI product of the year. The feature that catapulted it to viral fame is Audio Overview, which generates eerily realistic two-host podcast audio from any input you upload—written doc, audio or video file, or even a PDF. But to describe NotebookLM as a “podcast generator” is to vastly undersell it. The real magic of the product is in offering multi-modal dimensions to explore your own content in new ways—with context that’s surprisingly additive. 200-page training manuals become synthesized into digestible chapters, turned into a 10-minute podcast—or both—and shared with the sales team, just to cite one example. Raiza Martin and Jason Speilman join us to discuss how the magic happens, and what’s next for source-grounded AI.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
All of us as consumers have felt the magic of ChatGPT—but also the occasional errors and hallucinations that make off-the-shelf language models problematic for business use cases with no tolerance for errors. Case in point: A model deployed to help create a summary for this episode stated that Sridhar Ramaswamy previously led PyTorch at Meta. He did not. He spent years running Google’s ads business and now serves as CEO of Snowflake, which he describes as the data cloud for the AI era.
Ramaswamy discusses how smart systems design helped Snowflake create reliable "talk-to-your-data" applications with over 90% accuracy, compared to around 45% for out-of-the-box solutions using off the shelf LLMs. He describes Snowflake's commitment to making reliable AI simple for their customers, turning complex software engineering projects into straightforward tasks.
Finally, he stresses that even as frontier models progress, there is significant value to be unlocked from current models by applying them more effectively across various domains.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
Cortex Analyst: Snowflake’s talk-to-your-data API
Document AI: Snowflake feature that extracts in structured information from documents
Combining LLMs with AlphaGo-style deep reinforcement learning has been a holy grail for many leading AI labs, and with o1 (aka Strawberry) we are seeing the most general merging of the two modes to date. o1 is admittedly better at math than essay writing, but it has already achieved SOTA on a number of math, coding and reasoning benchmarks.
Deep RL legend and now OpenAI researcher Noam Brown and teammates Ilge Akkaya and Hunter Lightman discuss the ah-ha moments on the way to the release of o1, how it uses chains of thought and backtracking to think through problems, the discovery of strong test-time compute scaling laws and what to expect as the model gets better.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
00:00 - Introduction
01:33 - Conviction in o1
04:24 - How o1 works
05:04 - What is reasoning?
07:02 - Lessons from gameplay
09:14 - Generation vs verification
10:31 - What is surprising about o1 so far
11:37 - The trough of disillusionment
14:03 - Applying deep RL
14:45 - o1’s AlphaGo moment?
17:38 - A-ha moments
21:10 - Why is o1 good at STEM?
24:10 - Capabilities vs usefulness
25:29 - Defining AGI
26:13 - The importance of reasoning
28:39 - Chain of thought
30:41 - Implication of inference-time scaling laws
35:10 - Bottlenecks to scaling test-time compute
38:46 - Biggest misunderstanding about o1?
41:13 - o1-mini
42:15 - How should founders think about o1?
Adding code to LLM training data is a known method of improving a model’s reasoning skills. But wouldn’t math, the basis of all reasoning, be even better? Up until recently, there just wasn’t enough usable data that describes mathematics to make this feasible.
A few years ago, Vlad Tenev (also founder of Robinhood) and Tudor Achim noticed the rise of the community around an esoteric programming language called Lean that was gaining traction among mathematicians. The combination of that and the past decade’s rise of autoregressive models capable of fast, flexible learning made them think the time was now and they founded Harmonic. Their mission is both lofty—mathematical superintelligence—and imminently practical, verifying all safety-critical software.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
00:00 - Introduction
01:42 - Math is reasoning
06:16 - Studying with the world's greatest living mathematician
10:18 - What does the math community think of AI math?
15:11 - Recursive self-improvement
18:31 - What is Lean?
21:05 - Why now?
22:46 - Synthetic data is the fuel for the model
27:29 - How fast will your model get better?
29:45 - Exploring the frontiers of human knowledge
34:11 - Lightning round
AI researcher Jim Fan has had a charmed career. He was OpenAI’s first intern before he did his PhD at Stanford with “godmother of AI,” Fei-Fei Li. He graduated into a research scientist position at Nvidia and now leads its Embodied AI “GEAR” group. The lab’s current work spans foundation models for humanoid robots to agents for virtual worlds.
Jim describes a three-pronged data strategy for robotics, combining internet-scale data, simulation data and real world robot data. He believes that in the next few years it will be possible to create a “foundation agent” that can generalize across skills, embodiments and realities—both physical and virtual. He also supports Jensen Huang’s idea that “Everything that moves will eventually be autonomous.”
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
00:00 Introduction
01:35 Jim’s journey to embodied intelligence
04:53 The GEAR Group
07:32 Three kinds of data for robotics
10:32 A GPT-3 moment for robotics
16:05 Choosing the humanoid robot form factor
19:37 Specialized generalists
21:59 GR00T gets its own chip
23:35 Eureka and Issac Sim
25:23 Why now for robotics?
28:53 Exploring virtual worlds
36:28 Implications for games
39:13 Is the virtual world in service of the physical world?
42:10 Alternative architectures to Transformers
44:15 Lightning round
There’s a new archetype in Silicon Valley, the AI researcher turned founder. Instead of tinkering in a garage they write papers that earn them the right to collaborate with cutting-edge labs until they break out and start their own.
This is the story of wunderkind Eric Steinberger, the founder and CEO of Magic.dev. Eric came to programming through his obsession with AI and caught the attention of DeepMind researchers as a high school student. In 2022 he realized that AGI was closer than he had previously thought and started Magic to automate the software engineering necessary to get there. Among his counterintuitive ideas are the need to train proprietary large models, that value will not accrue in the application layer and that the best agents will manage themselves. Eric also talks about Magic’s recent 100M token context window model and the HashHop eval they’re open sourcing.
Hosted by: Sonya Huang, Sequoia Capital
Mentioned in this episode:
00:00 - Introduction
01:39 - Vienna-born wunderkind
04:56 - Working with Noam Brown
8:00 - “I can do two things. I cannot do three.”
10:37 - AGI to-do list
13:27 - Advice for young researchers
20:35 - Reading every paper voraciously
23:06 - The army of Noams
26:46 - The leaps still needed in research
29:59 - What is Magic?
36:12 - Competing against the 800-pound gorillas
38:21 - Ideal team size for researchers
40:10 - AI that feels like a colleague
44:30 - Lightning round
47:50 - Bonus round: 200M token context announcement
On Training Data, we learn from innovators pushing forward the frontier of AI’s capabilities. Today we’re bringing you something different. It’s the story of a company currently implementing AI at scale in the enterprise, and how it was built from a bootstrapped idea in the pre-AI era to a 150 billion dollar market cap giant.
It’s the Season 2 premiere of Sequoia’s other podcast, Crucible Moments, where we hear from the founders and leaders of some legendary companies about the crossroads and inflection points that shaped their journeys. In this episode, you’ll hear from Fred Luddy and Frank Slootman about building and scaling ServiceNow. Listen to Crucible Moments wherever you get your podcasts or go to:
Spotify: https://open.spotify.com/show/40bWCUSan0boCn0GZJNpPn
Apple: https://podcasts.apple.com/us/podcast/crucible-moments/id1705282398
Hosted by: Roelof Botha, Sequoia Capital
Transcript: https://www.sequoiacap.com/podcast/crucible-moments-servicenow/
Customer service is hands down the first killer app of generative AI for businesses. The reasons are simple: the costs of existing solutions are so high, the satisfaction so low and the margin for ROI so wide. But trusting your interactions with customers to hallucination-prone LLMs can be daunting.
Enter Sierra. Co-founder Clay Bavor walks us through the sophisticated engineering challenges his team solved along the way to delivering AI agents for all aspects of the customer experience that are delightful, safe and reliable—and being deployed widely by Sierra’s customers. The Company’s AgentOS enables businesses to create branded AI agents to interact with customers, follow nuanced policies and even handle customer retention and upsell. Clay describes how companies can capture their brand voice, values and internal processes to create AI agents that truly represent the business.
Hosted by: Ravi Gupta and Pat Grady, Sequoia Capital
Mentioned in this episode:
00:00:00 Introduction
00:01:21 Clay’s background
00:03:20 Google before the ChatGPT moment
00:07:31 What is Sierra?
00:12:03 What’s possible now that wasn’t possible 18 months ago?
00:17:11 AgentOS
00:23:45 The solution to many problems with AI is more AI
00:28:37 𝛕-bench
00:33:19 Engineering task vs research task
00:37:27 What tasks can you trust an agent with now?
00:43:21 What metrics will move?
00:46:22 The reality of deploying AI to customers today
00:53:33 The experience manager
01:03:54 Outcome-based pricing
01:05:55 Lightning Round
After AlphaGo beat Lee Sedol, a young mechanical engineer at Google thought of another game reinforcement learning could win: energy optimization at data centers. Jim Gao convinced his bosses at the Google data center team to let him work with the DeepMind team to try. The initial pilot resulted in a 40% energy savings and led he and his co-founders to start Phaidra to turn this technology into a product.
Jim discusses the challenges of AI readiness in industrial settings and how we have to build on top of the control systems of the 70s and 80s to achieve the promise of the Fourth Industrial Revolution. He believes this new world of self-learning systems and self-improving infrastructure is a key factor in addressing global climate change.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
In the first wave of the generative AI revolution, startups and enterprises built on top of the best closed-source models available, mostly from OpenAI. The AI customer journey moves from training to inference, and as these first products find PMF, many are hitting a wall on latency and cost.
Fireworks Founder and CEO Lin Qiao led the PyTorch team at Meta that rebuilt the whole stack to meet the complex needs of the world’s largest B2C company. Meta moved PyTorch to its own non-profit foundation in 2022 and Lin started Fireworks with the mission to compress the timeframe of training and inference and democratize access to GenAI beyond the hyperscalers to let a diversity of AI applications thrive.
Lin predicts when open and closed source models will converge and reveals her goal to build simple API access to the totality of knowledge.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
00:00 - Introduction
02:01 - What is Fireworks?
02:48 - Leading Pytorch
05:01 - What do researchers like about PyTorch?
07:50 - How Fireworks compares to open source
10:38 - Simplicity scales
12:51 - From training to inference
17:46 - Will open and closed source converge?
22:18 - Can you match OpenAI on the Fireworks stack?
26:53 - What is your vision for the Fireworks platform?
31:17 - Competition for Nvidia?
32:47 - Are returns to scale starting to slow down?
34:28 - Competition
36:32 - Lightning round
GithHub invented collaborative coding and in the process changed how open source projects, startups and eventually enterprises write code. GitHub Copilot is the first blockbuster product built on top of OpenAI’s GPT models. It now accounts for more than 40 percent of GitHub revenue growth for an annual revenue run rate of $2 billion. Copilot itself is already a larger business than all of GitHub was when Microsoft acquired it in 2018.
We talk to CEO Thomas Dohmke about how a small team at GitHub built on top of GPT-3 and quickly created a product that developers love—and can’t live without. Thomas describes how the product has grown from simple autocomplete to a fully featured workspace for enterprise teams. He also believes that tools like Copilot will bring the power of coding to a billion developers by 2030.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
00:00:00 - Introduction
00:01:18 - Getting started with code
00:03:43 - Microsoft’s acquisition of GitHub
00:11:40 - Evolving Copilot beyond autocomplete
00:14:18 - In hindsight, you can always move faster
00:15:56 - Building on top of OpenAI
00:20:21 - The latest metrics
00:22:11 - The surprise of Copilot’s impact
00:25:11 - Teaching kids to code in the age of Copilot
00:26:38 - The momentum mindset
00:29:46 - Agents vs Copilots
00:32:06 - The Roadmap
00:37:31 - Making maintaining software easier
00:38:48 - The creative new world
00:42:38 - The AI 10x software engineer
00:45:12 - Creativity and systems engineering in AI
00:48:55 - What about COBOL?
00:50:23 - Will GitHub build its own models?
00:57:19 - Rapid incubation at GitHub Next
00:59:21 - The future of AI?
01:03:18 - Advice for founders
01:05:08 - Lightning round
As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem.
Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1.
The Bitter Lesson by Rich Sutton
00:00 Introduction
01:28 The Llama 3.1 405B launch
05:02 The open source license
07:01 What's in it for Meta?
10:19 Why not open source?
11:16 Will frontier models commoditize?
12:41 What about startups?
16:29 The Mistral team
19:36 Are all frontier strategies comparable?
22:38 Is model development becoming more like software development?
26:34 Agentic reasoning
29:09 What future levers will unlock reasoning?
31:20 Will coding and math lead to unlocks?
33:09 Small models
34:08 7X more data
37:36 Are we going to hit a wall?
39:49 Lightning round
In February, Sebastian Siemiatkowski boldly announced that Klarna’s new OpenAI-powered assistant handled two thirds of the Swedish fintech’s customer service chats in its first month. Not only were customer satisfaction metrics better, but by replacing 700 full-time contractors the bottom line impact is projected to be $40M. Since then, every company we talk to wants to know, “How do we get the Klarna customer support thing?”
Co-founder and CEO Sebastian Siemiatkowski tells us how the Klarna team shipped this new product in record time—and how embracing AI internally with an experimental mindset is transforming the company. He discusses how AI development is proliferating inside the company, from customer support to marketing to internal knowledge to customer-facing experiences.
Sebastian also reflects on the impacts of AI on employment, society, and the arts while encouraging lawmakers to be open minded about the benefits.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned in this episode:
DeepL: Language translation app that Sebastian says makes 10,000 translators in Brussels redundant
The Klarna brand: The offbeat optimism that the company is now augmenting with AI
Neo4j: The graph database management system that Klarna is using to build Kiki, their internal knowledge base
00:00 Introduction
01:57 Klarna’s business
03:00 Pitching OpenAI
08:51 How we built this
10:46 Will Klara ever completely replace its CS team with AI?
14:22 The benefits
17:25 If you had a policy magic wand…
21:12 What jobs will be most affected by AI?
23:58 How about marketing?
27:55 How creative are LLMs?
30:11 Klarna’s knowledge graph, Kiki
33:10 Reducing the number of enterprise systems
35:24 Build vs buy?
39:59 What’s next for Klarna with AI?
48:48 Lightning round
LLMs are democratizing digital intelligence, but we’re all waiting for AI agents to take this to the next level by planning tasks and executing actions to actually transform the way we work and live our lives.
Yet despite incredible hype around AI agents, we’re still far from that “tipping point” with best in class models today. As one measure: coding agents are now scoring in the high-teens % on the SWE-bench benchmark for resolving GitHub issues, which far exceeds the previous unassisted baseline of 2% and the assisted baseline of 5%, but we’ve still got a long way to go.
Why is that? What do we need to truly unlock agentic capability for LLMs? What can we learn from researchers who have built both the most powerful agents in the world, like AlphaGo, and the most powerful LLMs in the world?
To find out, we’re talking to Misha Laskin, former research scientist at DeepMind. Misha is embarking on his vision to build the best agent models by bringing the search capabilities of RL together with LLMs at his new company, Reflection AI. He and his cofounder Ioannis Antonoglou, co-creator of AlphaGo and AlphaZero and RLHF lead for Gemini, are leveraging their unique insights to train the most reliable models for developers building agentic workflows.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
00:00 Introduction
01:11 Leaving Russia, discovering science
10:01 Getting into AI with Ioannis Antonoglou
15:54 Reflection AI and agents
25:41 The current state of Ai agents
29:17 AlphaGo, AlphaZero and Gemini
32:58 LLMs don’t have a ground truth reward
37:53 The importance of post-training
44:12 Task categories for agents
45:54 Attracting talent
50:52 How far away are capable agents?
56:01 Lightning round
Mentioned:
The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation.
Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform.
Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents.
Hosted by: Pat Grady and Bill Coughran, Sequoia Capital
Mentioned:
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, the 2018 Google paper that convinced Kevin that Microsoft wasn’t moving fast enough on AI.
Dennard scaling: The scaling law that describes the proportional relationship between transistor size and power use; has not held since 2012 and is often confused with Moore’s Law.
Textbooks Are All You Need: Microsoft paper that introduces a new large language model for code, phi-1, that achieves smaller size by using higher quality “textbook” data.
GPQA and MMLU: Benchmarks for reasoning
Copilot: Microsoft product line of GPT consumer assistants from general productivity to design, vacation planning, cooking and fitness.
Devin: Autonomous AI code agent from Cognition Labs that Microsoft recently announced a partnership with.
Ray Solomonoff: Participant in the 1956 Dartmouth Summer Research Project on Artificial Intelligence that named the field; Kevin admires his prescience about the importance of probabilistic methods decades before anyone else.
00:00 - Introduction
01:20 - Kevin’s backstory
06:56 - The role of PhDs in AI engineering
09:56 - Microsoft’s AI strategy
12:40 - Highlights and lowlights
16:28 - Accelerating investments
18:38 - The OpenAI partnership
22:46 - Soon inference will dwarf training
27:56 - Will the demand/supply balance change?
30:51 - Business models for data
36:54 - The value function
39:58 - Copilots
44:47 - The 98/2 rule
49:34 - Solving zero-sum games
57:13 - Lightning round
As impressive as LLMs are, the growing consensus is that language, scale and compute won’t get us to AGI. Although many AI benchmarks have quickly achieved human-level performance, there is one eval that has barely budged since it was created in 2019.
Google researcher François Chollet wrote a paper that year defining intelligence as skill-acquisition efficiency—the ability to learn new skills as humans do, from a small number of examples. To make it testable he proposed a new benchmark, the Abstraction and Reasoning Corpus (ARC), designed to be easy for humans, but hard for AI. Notably, it doesn’t rely on language.
Zapier co-founder Mike Knoop read Chollet’s paper as the LLM wave was rising. He worked quickly to integrate generative AI into Zapier’s product, but kept coming back to the lack of progress on the ARC benchmark. In June, Knoop and Chollet launched the ARC Prize, a public competition offering more than $1M to beat and open-source a solution to the ARC-AGI eval.
In this episode Mike talks about the new ideas required to solve ARC, shares updates from the first two weeks of the competition, and shares why he’s excited for AGI systems that can innovate alongside humans.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned:
(00:00) Introduction
(01:51) AI at Zapier
(08:31) What is ARC AGI?
(13:25) What does it mean to efficiently acquire a new skill?
(19:03) What approaches will succeed?
(21:11) A little bit of a different shape
(25:59) The role of code generation and program synthesis
(29:11) What types of people are working on this?
(31:45) Trying to prove you wrong
(34:50) Where are the big labs?
(38:21) The world post-AGI
(42:51) When will we cross 85% on ARC AGI?
(46:12) Will LLMs be part of the solution?
(50:13) Lightning round
Archimedes said that with a large enough lever, you can move the world. For decades, software engineering has been that lever. And now, AI is compounding that lever. How will we use AI to apply 100 or 1000x leverage to the greatest lever to move the world?
Matan Grinberg and Eno Reyes, co-founders of Factory, have chosen to do things differently than many of their peers in this white-hot space. They sell a fleet of “Droids,” purpose-built dev agents which accomplish different tasks in the software development lifecycle (like code review, testing, pull requests or writing code). Rather than training their own foundation model, their approach is to build something useful for engineering orgs today on top of the rapidly improving models, aligning with the developer and evolving with them.
Matan and Eno are optimistic about the effects of autonomy in software development and on building a company in the application layer. Their advice to founders, “The only way you can win is by executing faster and being more obsessed.”
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned:
Transcript: https://www.sequoiacap.com/podcast/training-data-factory/
00:00 Introduction
01:36 Personal backgrounds
10:54 The compound lever
12:41 What is Factory?
16:29 Cognitive architectures
21:13 800 engineers at OpenAI are working on my margins
24:00 Jeff Dean doesn't understand your code base
25:40 Individual dev productivity vs system-wide optimization
30:04 Results: Factory in action
32:54 Learnings along the way
35:36 Fully autonomous Jeff Deans
37:56 Beacons of the upcoming age
40:04 How far are we?
43:02 Competition
45:32 Lightning round
49:34 Bonus round: Factory's SWE-bench results
Last year, AutoGPT and Baby AGI captured our imaginations—agents quickly became the buzzword of the day…and then things went quiet. AutoGPT and Baby AGI may have marked a peak in the hype cycle, but this year has seen a wave of agentic breakouts on the product side, from Klarna’s customer support AI to Cognition’s Devin, etc.
Harrison Chase of LangChain is focused on enabling the orchestration layer for agents. In this conversation, he explains what’s changed that’s allowing agents to improve performance and find traction.
Harrison shares what he’s optimistic about, where he sees promise for agents vs. what he thinks will be trained into models themselves, and discusses novel kinds of UX that he imagines might transform how we experience agents in the future.
Hosted by: Sonya Huang and Pat Grady, Sequoia Capital
Mentioned:
Transcript: https://www.sequoiacap.com/podcast/training-data-harrison-chase/
00:00 Introduction
01:21 What are agents?
05:00 What is LangChain’s role in the agent ecosystem?
11:13 What is a cognitive architecture?
13:20 Is bespoke and hard coded the way the world is going, or a stop gap?
18:48 Focus on what makes your beer taste better
20:37 So what?
22:20 Where are agents getting traction?
25:35 Reflection, chain of thought, other techniques?
30:42 UX can influence the effectiveness of the architecture
35:30 What’s out of scope?
38:04 Fine tuning vs prompting?
42:17 Existing observability tools for LLMs vs needing a new architecture/approach
45:38 Lightning round
Join us as we train our neural nets on the theme of the century: AI. Sequoia Capital partners Sonya Huang and Pat Grady host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies and their implications for technology, business and society.
The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.
En liten tjänst av I'm With Friends. Finns även på engelska.