Interviewing Andrew Trask on how language models should store (and access) information

Andrew Trask is one of the bright spots in engaging with AI policy for me in the last year. He is a passionate idealist, trying to create a future for AI that enables privacy, academic research, and government involvement in a rapidly transforming ecosystem. Trask is a leader of the OpenMined organization facilitating researcher access to non-public data and AIs, a senior research scientist at Google DeepMind, a PhD student at the University of Oxford, an author and educator on Deep Learning.

You can find more about Trask on Twitter or Google Scholar. You may want to watch his recent talk at Cohere on the future of AI (and why data breakthroughs dominate), his lecture at MIT on privacy preserving ML, or his book on deep learning that has a substantial GitHub component. Here’s a slide I liked from his recent Cohere talk:

The organization he helps run, OpenMined, has a few principles that say a lot about his ambitions and approaches to modern AI:

We believe we can inspire all data owners to open their data for research by building open-source privacy software that empowers them to receive more benefits (co-authorships, citations, grants, etc.) while mitigating risks related to privacy, security, and IP.

We cover privacy of LLMs, retrieval LLMs, secure enclaves, o1, Apple's new models, and many more topics.

More on Andrew: https://x.com/iamtrask

Transcript and more information: https://www.interconnects.ai/p/interviewing-andrew-trask

Interconnects (https://www.interconnects.ai/)...

... on YouTube: https://www.youtube.com/@interconnects

... on Twitter: https://x.com/interconnectsai

... on Linkedin: https://www.linkedin.com/company/interconnects-ai

... on Spotify: https://open.spotify.com/show/2UE6s7wZC4kiXYOnWRuxGv

We Mention

* Claude 3.5 launch and “pre release testing with UK AISI” (and the US AI Safety Institute)

* OpenMined and PySyft

* CSET (Center for Security and Emerging Technology)

* NAIRR

* The “open data wall”

* Apple’s Secure Enclaves, Nvidia Secure Enclave

* Data-store language models literature

* RETRO: Retrieval-Enhanced Transformer from DeepMind (2021)

* SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023)

* Scaling Retrieval-Based Language Models with a Trillion-Token Datastore (2024)

Chapters

[00:00:00] Introduction

[00:03:12] Secure enclaves and pre-release testing with Anthropic and UK Safety Institute

[00:16:31] Discussion on public AI and government involvement

[00:20:55] Data store language models and better approaches to “open training data”

[00:42:18] History and development of OpenMined

[00:48:57] Use of language models on air-gapped networks

[00:52:10] Near future of secure enclave technology and industry adoption

[00:58:01] Conclusions and future trajectory of AI development