LessWrong posts by zvi

Both Geoffrey Hinton and Demis Hassabis were given the Nobel Prize this week, in Physics and Chemistry respectively. Congratulations to both of them along with all the other winners. AI will be central to more and more of scientific progress over time. This felt early, but not as early as you would think.

The two big capability announcements this week were OpenAI's canvas, their answer to Anthropic's artifacts to allow you to work on documents or code outside of the chat window in a way that seems very useful, and Meta announcing a new video generation model with various cool features, that they’re wisely not releasing just yet.

I also have two related corrections from last week, and an apology: Joshua Achiam is OpenAI's new head of Mission Alignment, not of Alignment as I incorrectly said. The new head of Alignment Research is Mia Glaese. That mistake [...]

---

Outline:

(01:30) Language Models Offer Mundane Utility

(09:10) Language Models Don’t Offer Mundane Utility

(13:11) Blank Canvas

(17:13) Meta Video

(18:58) Deepfaketown and Botpocalypse Soon

(21:22) They Took Our Jobs

(24:45) Get Involved

(26:01) Introducing

(26:14) AI Wins the Nobel Prize

(28:51) In Other AI News

(30:05) Quiet Speculations

(34:22) The Mask Comes Off

(37:17) The Quest for Sane Regulations

(41:02) The Week in Audio

(43:13) Rhetorical Innovation

(48:20) The Carbon Question

(50:27) Aligning a Smarter Than Human Intelligence is Difficult

(55:48) People Are Trying Not to Die

The original text contained 6 images which were described by AI.

---

First published:
October 10th, 2024

Source:
https://www.lesswrong.com/posts/wTriAw9mB6b5FwH5g/ai-85-ai-wins-the-nobel-prize

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Joshua Achiam is the OpenAI Head of Mission Alignment

I start off this post with an apology for two related mistakes from last week.

The first is the easy correction: I incorrectly thought he was the head of ‘alignment’ at OpenAI rather than his actual title ‘mission alignment.’

Both are important, and make one's views important, but they’re very different.

The more serious error, which got quoted some elsewhere, was: In the section about OpenAI, I noted some past comments from Joshua Achiam, and interpreted them as him lecturing EAs that misalignment risk from AGI was not real.

While in isolation I believe this is a reasonable way to interpret this quote, this issue is important to get right especially if I’m going to say things like that. Looking at it only that way was wrong. I both used a poor method to contact [...]

---

Outline:

(00:04) Joshua Achiam is the OpenAI Head of Mission Alignment

(01:50) Joshua Achiam Has a Very Different Model of AI Existential Risk

(05:00) Joshua is Strongly Dismissive of Alternative Models of AI X-Risk

(10:05) Would Ordinary Safety Practices Would Be Sufficient for AI?

(12:25) Visions of the Future

(14:53) Joshua Achiam versus Eliezer Yudkowsky

(22:47) People Are Going to Give AI Power

(29:32) Value is Complicated

(38:22) Conclusion

The original text contained 1 image which was described by AI.

---

First published:
October 10th, 2024

Source:
https://www.lesswrong.com/posts/WavWheRLhxnofKHva/joshua-achiam-public-statement-analysis

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction: Better than a Podcast

Andrej Karpathy continues to be a big fan of NotebookLM, especially its podcast creation feature. There is something deeply alien to me about this proposed way of consuming information, but I probably shouldn’t knock it (too much) until I try it?

Others are fans as well.

Carlos Perez: Google with NotebookLM may have accidentally stumbled upon an entirely new way of interacting with AI. Its original purpose was to summarize literature. But one unexpected benefit is when it's used to talk about your expressions (i.e., conversations or lectures). This is when you discover the insight of multiple interpretations! Don’t just render a summary one time; have it do so several times. You’ll then realize how different interpretations emerge, often in unexpected ways.

Delip Rao gives the engine two words repeated over and over, the AI podcast hosts describe what it [...]

---

Outline:

(00:05) Introduction: Better than a Podcast

(03:16) Language Models Offer Mundane Utility

(04:04) Language Models Don’t Offer Mundane Utility

(09:24) Copyright Confrontation

(10:44) Deepfaketown and Botpocalypse Soon

(14:45) They Took Our Jobs

(19:23) The Art of the Jailbreak

(19:39) Get Involved

(20:00) Introducing

(20:37) OpenAI Dev Day

(34:40) In Other AI News

(38:03) The Mask Comes Off

(55:42) Quiet Speculations

(59:10) The Quest for Sane Regulations

(01:00:04) The Week in Audio

(01:01:54) Rhetorical Innovation

(01:19:08) Remember Who Marc Andreessen Is

(01:22:35) A Narrow Path

(01:30:36) Aligning a Smarter Than Human Intelligence is Difficult

(01:33:45) The Wit and Wisdom of Sam Altman

(01:35:25) The Lighter Side

The original text contained 10 images which were described by AI.

---

First published:
October 3rd, 2024

Source:
https://www.lesswrong.com/posts/bWrZhfaTD5EDjwkLo/ai-84-better-than-a-podcast

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's over, until such a future time as either we are so back, or it is over for humanity.

Gavin Newsom has vetoed SB 1047.

Newsom's Message In Full

Quoted text is him, comments are mine.

To the Members of the California State Senate: I am returning Senate Bill 1047 without my signature.

This bill would require developers of large artificial intelligence (Al) models, and those providing the computing power to train such models, to put certain safeguards and policies in place to prevent catastrophic harm. The bill would also establish the Board of Frontier Models – a state entity – to oversee the development of these models.

It is worth pointing out here that mostly the ‘certain safeguards and policies’ was ‘have a policy at all, tell us what it is and then follow it.’ But there were some specific things that [...]

---

Outline:

(00:15) Newsom's Message In Full

(10:42) Newsom's Explanation Does Not Make Sense

(15:21) Newsom's Proposed Path of Use Regulation is Terrible for Everyone

(23:02) Newsom's Proposed Path of Use Regulation Doesn’t Prevent X-Risk

(26:49) Newsom Says He Wants to Regulate Small Entrepreneurs and Academia

(29:20) What If Something Goes Really Wrong?

(30:12) Could Newsom Come Around?

(35:10) Timing is Everything

(36:23) SB 1047 Was Popular

(39:41) What Did the Market Have to Say?

(41:51) What Newsom Did Sign

(54:00) Paths Forward

The original text contained 1 image which was described by AI.

---

First published:
October 1st, 2024

Source:
https://www.lesswrong.com/posts/6kZ6gW5DEZKFfqvZD/newsom-vetoes-sb-1047

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: The Fundamentals, The Gamblers, The Business

We have now arrived at the topics most central to this book, aka ‘The Future.’

Rationalism and Effective Altruism (EA)

The Manifest conference was also one of the last reporting trips that I made for this book. And it confirmed for me that the River is real—not just some literary device I invented. (6706)

Yep. The River is real.

I consider myself, among many things, a straight up rationalist.

I do not consider myself an EA, and never have.

This completes the four quadrants of the two-by-two of [does Nate knows it well, does Zvi knows it well]. The first two, where Nate was in his element, went very well. The third clearly was less exacting, as one would expect, but pretty good.

Now I have the information advantage, even more than I did [...]

---

Outline:

(00:16) Rationalism and Effective Altruism (EA)

(06:01) Cost-Benefit Analysis

(09:04) How About Trying At All

(10:11) The Virtues of Rationality

(11:56) Effective Altruism and Rationality, Very Different of Course

(24:37) The Story of OpenAI

(30:19) Altman, OpenAI and AI Existential Risk

(38:26) Tonight at 11: Doom

(01:00:39) AI Existential Risk: They’re For It

(01:07:42) To Pause or Not to Pause

(01:11:11) You Need Better Decision Theory

(01:15:27) Understanding the AI

(01:19:43) Aligning the AI

(01:23:50) A Glimpse of Our Possible Future

(01:28:16) The Closing Motto

---

First published:
September 27th, 2024

Source:
https://www.lesswrong.com/posts/5qbcmKdfWc7vskrRD/book-review-on-the-edge-the-future

Narrated by TYPE III AUDIO.

We interrupt Nate Silver week here at Don’t Worry About the Vase to bring you some rather big AI news: OpenAI and Sam Altman are planning on fully taking their masks off, discarding the nonprofit board's nominal control and transitioning to a for-profit B-corporation, in which Sam Altman will have equity.

We now know who they are and have chosen to be. We know what they believe in. We know what their promises and legal commitments are worth. We know what they plan to do, if we do not stop them.

They have made all this perfectly clear. I appreciate the clarity.

On the same day, Mira Murati, the only remaining person at OpenAI who in any visible way opposed Altman during the events of last November, resigned without warning along with two other senior people, joining a list that now includes among others several OpenAI [...]

---

Outline:

(01:51) Language Models Offer Mundane Utility

(04:17) Language Models Don’t Offer Mundane Utility

(06:35) The Mask Comes Off

(18:51) Fun with Image Generation

(18:54) Deepfaketown and Botpocalypse Soon

(19:50) They Took Our Jobs

(20:49) The Art of the Jailbreak

(21:28) OpenAI Advanced Voice Mode

(26:03) Introducing

(28:29) In Other AI News

(30:30) Quiet Speculations

(34:00) The Quest for Sane Regulations

(42:21) The Week in Audio

(42:47) Rhetorical Innovation

(56:50) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:36) Other People Are Not As Worried About AI Killing Everyone

(01:01:53) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:
September 26th, 2024

Source:
https://www.lesswrong.com/posts/FeqY7NWcFMn8haWCR/ai-83-the-mask-comes-off

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: The Fundamentals, The Gamblers

Having previously handled the literal gamblers, we are ready to move on to those who Do Business using Riverian principles.

Or at least while claiming to use Riverian principles, since Silicon Valley doesn’t fit into the schema as cleanly as many other groups. That's where we begin this section, starting at the highest possible conceptual level.

Time to talk real money.

Why Can You Do This Trade?

First law of trading: For you to buy, someone must sell. Or for you to sell, someone must buy. And there can’t be someone else doing the trade before you did it.

Why did they do that, and why did no one else take the trade first? Until you understand why you are able to do this trade, you should be highly suspicious.

“Every single thing we do, I can [...]

---

Outline:

(00:41) Why Can You Do This Trade?

(03:08) In a World of Venture Capital

(10:54) Short Termism Hypothesis

(12:42) Non-Determinism and its Discontents

(14:57) The Founder, the Fox and the Hedgehog

(17:11) The Team to Beat

(24:22) Silicon Valley Versus Risk

(35:14) The Keynesian Beauty Contest

(40:57) The Secret of Their Success is Deal Flow

(50:00) The Valley Beside the River

(53:07) Checkpoint Three

(53:37) Fun With SBF and Crypto Fraud

(01:01:53) Other Crypto Thoughts Unrelated to SBF

(01:04:50) Checkpoint Four

The original text contained 1 image which was described by AI.

---

First published:
September 25th, 2024

Source:
https://www.lesswrong.com/posts/Hfb3pc9HwdcCP7pys/book-review-on-the-edge-the-business

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: Book Review: On the Edge: The Fundamentals

As I said in the Introduction, I loved this part of the book. Let's get to it.

Poker and Game Theory

When people talk about game theory, they mostly talk solving for the equilibrium, and how to play your best game or strategy (there need not be a formal game) against adversaries who are doing the same.

I think of game theory like Frank Sinatra thinks of New York City: “If I can make it there, I’ll make it anywhere.” If you can compete against people performing at their best, you’re going to be a winner in almost any game you play. But if you build a strategy around exploiting inferior competition, it's unlikely to be a winning approach outside of a specific, narrow setting. What plays well in Peoria doesn’t necessarily play well in New York. [...]

---

Outline:

(00:18) Poker and Game Theory

(06:53) Sports Randomized Sports

(11:17) Knowing Theory Versus Memorization Versus Practice

(16:15) More About Tells

(19:20) Feeling the Probabilities

(20:35) Feeling Sad About It

(28:33) The Iowa Gambling Task

(31:39) The Greatest Risk

(37:20) Tournament Poker Is Super High Variance

(42:42) The Art of the Degen

(48:43) Why Do They Insist on Calling it Luck

(51:56) The Poker Gender Gap

(54:36) A Potential Cheater

(58:30) Making a Close Decision

(01:00:19) Other Games at the Casino

(01:03:22) Slot Machines Considered Harmful

(01:08:23) Where I Draw the Line

(01:11:14) A Brief History of Vegas and Casinos (as told by Nate Silver)

(01:16:44) We Got Us a Whale

(01:21:41) Donald Trump and Atlantic City Were Bad At Casinos

(01:25:17) How To Design a Casino

(01:26:46) The Wide World of Winning at Sports Gambling

(01:41:01) Limatime

(01:43:45) The Art of Getting Down

(01:45:29) Oh Yeah That Guy

(01:55:34) The House Sometimes Wins

(02:01:24) The House Is Probably a Coward

(02:11:19) DFS and The Problem of Winners

(02:16:08) Balancing the Action

(02:18:44) The Market Maker

(02:22:45) The Closing Line is Hard to Beat

(02:25:11) Winning is Hard

(02:29:58) What Could Be, Unburdened By What Has Been

(02:34:52) Finding Edges Big and Small

(02:40:12) Checkpoint Two

The original text contained 2 images which were described by AI.

---

First published:
September 24th, 2024

Source:
https://www.lesswrong.com/posts/mkyMx4FtJrfuGnsrm/book-review-on-the-edge-the-gamblers

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The most likely person to write On the Edge was Nate Silver.

Grok thinks the next most likely was Michael Lewis, followed by a number of other writers of popular books regarding people thinking different.

I see why Grok would say that, but it is wrong.

The next most likely person was Zvi Mowshowitz.

I haven’t written a book for this type of audience, a kind of smarter business-book, but that seems eminently within my potential range.

On the Edge is a book about those living On The Edge, the collection of people who take risk and think probabilistically and about expected value. It centrally covers poker, sports betting, casinos, Silicon Valley, venture capital, Sam Bankman-Fried, effective altruism, AI and existential risk.

Collectively, Nate Silver calls this cultural orientation The River.

It is contrasted with The Village, which comprises roughly the mainstream [...]

---

Outline:

(02:53) Overview

(07:56) Introduction: The River

(14:21) Nate Silver Comes Home to The River

(18:29) Nate (In General) Makes One Critical Mistake

(18:57) The Village Idiots

(22:22) Alone in the Wilderness

(25:46) Why the River Hates the Village

(29:57) Nate Silver's History of River Versus Village

(38:46) Spending Time at Airports

(41:14) The Coin Flip

(42:15) The Other Risk Takers

(49:45) Aside on Covid

(50:46) What is a Contrarian?

(53:49) Prediction Market Smackdown

(56:53) Checkpoint One

The original text contained 1 image which was described by AI.

---

First published:
September 23rd, 2024

Source:
https://www.lesswrong.com/posts/JmxYNqHcr6fFzJ33u/book-review-on-the-edge-the-fundamentals

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’d split the latest housing roundup into local versus global questions. I was planning on waiting a bit between them.

Then Joe Biden decided to propose a version of the worst possible thing.

So I guess here we are.

What is the organizing principle of Bidenomics?

Restrict Supply and Subsidize Demand (1)

This was the old counterproductive Biden proposal:

Unusual Whales: Biden to propose $5,000 credit for first-time home buyers, per WaPo.

The Rich: House prices about to go up $5,000 everywhere.

Under current conditions this is almost a pure regressive tax, a transfer from those too poor to own a home to those who can afford to buy one, or who previously owned one and no longer do.

If there were no restrictions on the supply of housing, such that the price of a house equalled the cost of [...]

---

Outline:

(00:24) Restrict Supply and Subsidize Demand (1)

(08:55) Restrict Supply and Subsidize Demand (2): Rent Control

(16:57) You Should See the Other Guy

(21:18) Stop Restricting Supply

(23:21) All Supply is Good Supply

(26:53) ‘Inclusionary’ Zoning

(28:37) The Worst Take

(33:27) Where and With Whom People Want To Live

(36:44) Matching

(39:58) Universality

(41:42) The Value of Land

(43:52) The Doom Loop

(51:05) How Are Sale Prices So Out of Whack with Rents and Income?

(52:18) Questioning Superstar Status

(54:47) Window Shopping

(58:18) Minimum Viable Product

(01:00:53) Construction Costs

(01:02:57) Elevator Action

(01:11:09) Housing Theory of Everything

(01:13:07) Zoning By Prohibitive Permit

(01:14:55) YIGBY?

(01:15:24) The True NIMBY

(01:16:45) The Definition of Chutzpah

(01:18:23) In Other Housing News

(01:18:53) Rhetoric

(01:21:33) Environmentalists Should Favor Density

(01:23:03) Do Not Give the People What They Want

(01:26:05) Housing Construction in the UK

(01:27:25) The Funniest Possible Thing

(01:29:07) Other Funny Things

The original text contained 35 images which were described by AI.

---

First published:
July 17th, 2024

Source:
https://www.lesswrong.com/posts/sX5ANDiTb96CkYpxd/housing-roundup-9-restricting-supply

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The big news of the week was of course OpenAI releasing their new model o1. If you read one post this week, read that one. Everything else is a relative sideshow.

Meanwhile, we await Newsom's decision on SB 1047. The smart money was always that Gavin Newsom would make us wait before offering his verdict on SB 1047. It's a big decision. Don’t rush him. In the meantime, what hints he has offered suggest he's buying into some of the anti-1047 talking points. I’m offering a letter to him here based on his comments, if you have any way to help convince him now would be the time to use that. But mostly, it's up to him now.

Table of Contents

1. Introduction.
2. Table of Contents.
3. Language Models Offer Mundane Utility. Apply for unemployment.
4. Language Models Don’t Offer Mundane Utility. How to avoid [...]

---

Outline:

(00:49) Language Models Offer Mundane Utility

(02:11) Language Models Don’t Offer Mundane Utility

(03:34) Deepfaketown and Botpocalypse Soon

(05:59) They Took Our Jobs

(07:34) Get Involved

(08:09) Introducing

(10:11) In Other AI News

(13:20) Quiet Speculations

(15:15) Intelligent Design

(19:15) SB 1047: The Governor Ponders

(27:13) Letter to Newsom

(31:36) The Quest for Sane Regulations

(34:19) Rhetorical Innovation

(42:13) Claude Writes Short Stories

(45:54) Questions of Sentience

(48:22) People Are Worried About AI Killing Everyone

(49:56) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:
September 19th, 2024

Source:
https://www.lesswrong.com/posts/Y4nS3yMWfJdmeoLcQ/ai-82-the-governor-ponders

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's that time again for all the sufficiently interesting news that isn’t otherwise fit to print, also known as the Monthly Roundup.

Bad News

Beware the failure mode in strategy and decisions that implicitly assumes competence, or wishes away difficulties, and remember to reverse all advice you hear.

Stefan Schubert (quoting Tyler Cowen on raising people's ambitions often being very high value): I think lowering others’ aspirations can also be high-return. I know of people who would have had a better life by now if someone could have persuaded them to pursue more realistic plans.

Rob Miles: There's a specific failure mode which I don’t have a name for, which is similar to “be too ambitious” but is closer to “have an unrealistic plan”. The illustrative example I use is:

Suppose by some strange circumstance you have to represent your country at olympic gymnastics [...]

---

Outline:

(00:14) Bad News

(03:45) Anti-Social Media

(07:20) Technology Advances

(08:56) High Seas Piracy is Bad

(12:17) The Michelin Curse

(15:21) What's the Rush?

(17:48) Good News, Everyone

(19:49) Let it Go

(22:09) Yay Air Conditioning

(23:27) Beast of a Memo

(36:58) For Science!

(37:18) For Your Entertainment

(39:47) Properly Rated

(45:26) Government Working

(49:21) Grapefruit Diet

(58:38) Gamers Gonna Game Game Game Game Game

(01:06:09) Gamers Winning At Life

(01:10:52) I Was Promised Flying Self-Driving Cars

(01:11:21) While I Cannot Condone This

(01:17:21) Nostalgia

(01:23:51) The Lighter Side

The original text contained 12 images which were described by AI.

---

First published:
September 17th, 2024

Source:
https://www.lesswrong.com/posts/4gAqkRhCuK2kGJFQE/monthly-roundup-22-september-2024

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Terrible name (with a terrible reason, that this ‘resets the counter’ on AI capability to 1, and ‘o’ as in OpenAI when they previously used o for Omni, very confusing). Impressive new capabilities in many ways. Less impressive in many others, at least relative to its hype.

Clearly this is an important capabilities improvement. However, it is not a 5-level model, and in important senses the ‘raw G’ underlying the system hasn’t improved.

GPT-o1 seems to get its new capabilities by taking (effectively) GPT-4o, and then using extensive Chain of Thought (CoT) and quite a lot of tokens. Thus that unlocks (a lot of) what that can unlock. We did not previously know how to usefully do that. Now we do. It gets much better at formal logic and reasoning, things in the ‘system 2’ bucket. That matters a lot for many tasks, if not as much [...]

---

Outline:

(01:26) Introducing GPT-o1

(05:05) Evals

(07:55) Chain of Thought

(08:57) Coding

(11:08) Human Preference Evaluation

(11:37) What Is It?

(20:24) Doing Math Without Terrance Tao

(25:02) Doing Real Math with Terence Tao

(30:04) Positive Examples

(38:51) Skeptical Reactions

(42:32) Report from Janus World

(45:30) Same Old Silly Examples

(53:47) Latency

(55:14) Paths Forward Unrelated to Safety

(59:17) Safety Last

(01:07:06) Deception

(01:10:50) External Red Teaming

(01:11:23) Apollo's Red Teaming Finds Deceptive Alignment

(01:22:17) Preparedness Testing Finds Reward Hacking

(01:26:43) METR's Red Teaming

(01:29:52) What Are the Safety and Policy Implications?

The original text contained 37 images which were described by AI.

---

First published:
September 16th, 2024

Source:
https://www.lesswrong.com/posts/zuaaqjsN6BucbGhf5/gpt-o1

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Following up on Alpha Fold, DeepMind has moved on to Alpha Proteo. We also got a rather simple prompt that can create a remarkably not-bad superforecaster for at least some classes of medium term events.

We did not get a new best open model, because that turned out to be a scam. And we don’t have Apple Intelligence, because it isn’t ready for prime time. We also got only one very brief mention of AI in the debate I felt compelled to watch.

What about all the apps out there, that we haven’t even tried? It's always weird to get lists of ‘top 50 AI websites and apps’ and notice you haven’t even heard of most of them.

Table of Contents

1. Introduction.
2. Table of Contents.
3. Language Models Offer Mundane Utility. So many apps, so little time.
4. Language Models Don’t Offer Mundane [...]

---

Outline:

(00:44) Language Models Offer Mundane Utility

(03:40) Language Models Don’t Offer Mundane Utility

(05:43) Predictions are Hard Especially About the Future

(12:57) Early Apple Intelligence

(15:27) On Reflection It's a Scam

(21:34) Deepfaketown and Botpocalypse Soon

(23:08) They Took Our Jobs

(28:42) The Time 100 People in AI

(32:11) The Art of the Jailbreak

(32:47) Get Involved

(33:12) Alpha Proteo

(43:14) Introducing

(44:23) In Other AI News

(46:41) Quiet Speculations

(50:40) The Quest for Sane Regulations

(53:12) The Week in Audio

(54:28) Rhetorical Innovation

(55:48) Aligning a Smarter Than Human Intelligence is Difficult

(56:05) People Are Worried About AI Killing Everyone

(58:38) Other People Are Not As Worried About AI Killing Everyone

(59:56) Six Boats and a Helicopter

(01:06:47) The Lighter Side

The original text contained 8 images which were described by AI.

---

First published:
September 12th, 2024

Source:
https://www.lesswrong.com/posts/YMaTA2hX6tSBJWnPr/ai-81-alpha-proteo

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(This was supposed to be on Thursday but I forgot to cross-post)

Will AI ever make art? Fully do your coding? Take all the jobs? Kill all the humans?

Most of the time, the question comes down to a general disagreement about AI capabilities. How high on a ‘technological richter scale’ will AI go? If you feel the AGI and think capabilities will greatly improve, then AI will also be able to do any particular other thing, and arguments that it cannot are almost always extremely poor. However, if frontier AI capabilities level off soon, then it is an open question how far we can get that to go in practice.

A lot of frustration comes from people implicitly making the claim that general AI capabilities will level off soon, usually without noticing they are doing that. At its most extreme, this is treating AI as [...]

---

Outline:

(01:53) Language Models Offer Mundane Utility

(03:05) Language Models Don’t Offer Mundane Utility

(04:03) Fun with Image Generation

(06:14) Copyright Confrontation

(07:09) Deepfaketown and Botpocalypse Soon

(12:09) They Took Our Jobs

(13:46) Time of the Season

(16:53) Get Involved

(17:15) Introducing

(19:08) In Other AI News

(19:51) Quiet Speculations

(30:10) A Matter of Antitrust

(37:34) The Quest for Sane Regulations

(40:06) The Week in Audio

(47:40) Rhetorical Innovation

(53:09) The Cosmos Institute

(56:21) The Alignment Checklist

(01:00:34) People Are Worried About AI Killing Everyone

(01:02:39) Other People Are Not As Worried About AI Killing Everyone

(01:06:21) Five Boats and a Helicopter

(01:09:07) Pick Up the Phone

(01:12:58) The Lighter Side

The original text contained 9 images which were described by AI.

---

First published:
September 10th, 2024

Source:
https://www.lesswrong.com/posts/x77vDAzosxtwJoJ7e/ai-80-never-have-i-ever

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I am posting this now largely because it is the right place to get in discussion of unrealized capital gains taxes and other campaign proposals, but also there is always plenty of other stuff going on. As always, remember that there are plenty of really stupid proposals always coming from all sides. I’m not spending as much time talking about why it's awful to for example impose gigantic tariffs on everything, because if you are reading this I presume you already know.

The Biggest Economics Problem

The problem, perhaps, in a nutshell:

Tess: like 10% of people understand how markets work and about 10% deeply desire and believe in a future that's drastically better than the present but you need both of these to do anything useful and they’re extremely anticorrelated so we’re probably all fucked.

In my world the two are correlated. If you [...]

---

Outline:

(00:31) The Biggest Economics Problem

(02:30) No Good Very Bad Capital Gains Tax Proposals

(14:13) Hot Tip

(17:11) Gouging at the Grocer

(20:08) Noncompetes Nonenforcement Cannot Compete With Courts

(20:42) We Used to Be Poor

(25:13) Everywhere But in the Productivity Statistics

(26:32) They Don’t Make ‘Em Like They Used To

(30:29) Disclosure of Wages Causes Lower Wages

(32:30) In Other Economic News

(36:50) The Efficient Market Hypothesis is (Even More) False

The original text contained 4 images which were described by AI.

---

First published:
September 10th, 2024

Source:
https://www.lesswrong.com/posts/cc9fXQLzAc52kMBHx/economics-roundup-3

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Technological Richter scale is introduced about 80% of the way through Nate Silver's new book On the Edge.

A full review is in the works (note to prediction markets: this post alone does NOT on its own count as a review, but this counts as part of a future review), but this concept seems highly useful, stands on its own and I want a reference post for it. Nate skips around his chapter titles and timelines, so why not do the same here?

Defining the Scale

Nate Silver, On the Edge (location 8,088 on Kindle): The Richter scale was created by the physicist Charles Richter in 1935 to quantify the amount of energy released by earthquakes.

It has two key features that I’ll borrow for my Technological Richter Scale (TRS). First, it is logarithmic. A magnitude 7 earthquake is actually ten times more powerful [...]

---

Outline:

(00:32) Defining the Scale

(07:15) The Big Disagreement About Future Generative AI

(09:39) Just Think of the Potential

(11:06) A Perfect 10

(13:19) Some Arguments Against Transformational AI

(19:06) Brief Notes on Arguments Transformational AI Will Turn Out Fine

The original text contained 4 images which were described by AI.

---

First published:
September 4th, 2024

Source:
https://www.lesswrong.com/posts/oAy72fcqDHsCvLBKz/ai-and-the-technological-richter-scale

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Would a universal basic income (UBI) work? What would it do?

Many people agree July's RCT on giving people a guaranteed income, and its paper from Eva Vivalt, Elizabeth Rhodes, Alexander W. Bartik, David E. Broockman and Sarah Miller was, despite whatever flaws it might have, the best data we have so far on the potential impact of UBI. There are many key differences from how UBI would look if applied for real, but this is the best data we have.

This study was primarily funded by Sam Altman, so whatever else he may be up to, good job there. I do note that my model of ‘Altman several years ago’ is more positive than mine of Altman now, and past actions like this are a lot of the reason I give him so much benefit of the doubt.

They do not agree on what conclusions [...]

---

Outline:

(02:47) RTFP (Read the Paper): Core Design

(06:23) Headline Effects

(10:27) Are Those Good Results?

(21:24) Expectations

(22:47) Work

(25:37) Additional Reactions

(30:35) UBI as Complement or Substitute

(32:37) On UBI in General

(34:04) The Future May Be Different

The original text contained 2 images which were described by AI.

---

First published:
September 3rd, 2024

Source:
https://www.lesswrong.com/posts/RQDqnCeff4cJhKQiT/on-the-ubi-paper

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I have never been more ready for Some Football.

Have I learned all about the teams and players in detail? No, I have been rather busy, and have not had the opportunity to do that, although I eagerly await Seth Burn's Football Preview. I’ll have to do that part on the fly.

But oh my would a change of pace and chance to relax be welcome. It is time.

The debate over SB 1047 has been dominating for weeks. I’ve now said my peace on the bill and how it works, and compiled the reactions in support and opposition. There are two small orders of business left for the weekly. One is the absurd Chamber of Commerce ‘poll’ that is the equivalent of a pollster asking if you support John Smith, who recently killed your dog and who opponents say will likely kill again, while hoping [...]

---

Outline:

(02:08) Language Models Offer Mundane Utility

(09:34) Language Models Don’t Offer Mundane Utility

(14:04) Fun with Image Generation

(14:31) Deepfaketown and Botpocalypse Soon

(21:08) They Took Our Jobs

(22:33) Get Involved

(22:50) Introducing

(24:47) Testing, Testing

(25:55) In Other AI News

(27:47) Quiet Speculations

(36:37) SB 1047: Remember

(41:07) The Week in Audio

(45:24) Rhetorical Innovation

(51:19) Aligning a Smarter Than Human Intelligence is Difficult

(56:03) People Are Worried About AI Killing Everyone

(58:31) The Lighter Side

The original text contained 7 images which were described by AI.

---

First published:
August 29th, 2024

Source:
https://www.lesswrong.com/posts/K8R3Cpj3szcX7z6Xo/ai-79-ready-for-some-football

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is the endgame. Very soon the session will end, and various bills either will or won’t head to Newsom's desk. Some will then get signed and become law.

Time is rapidly running out to have your voice impact that decision.

Since my last weekly, we got a variety of people coming in to stand for or against the final version of SB 1047. There could still be more, but probably all the major players have spoken at this point.

So here, today, I’m going to round up all that rhetoric, all those positions, in one place. After this, I plan to be much more stingy about talking about the whole thing, and only cover important new arguments or major news.

I’m not going to get into the weeds arguing about the merits of SB 1047 – I stand by my analysis in the Guide [...]

---

Outline:

(01:11) The Media

(01:54) OpenAI Opposes SB 1047

(06:15) OpenAI Backs AB 3211

(10:49) Anthropic Says SB 1047's Benefits Likely Exceed Costs

(15:03) Details of Anthropic's Letter

(20:08) Elon Musk Says California Should Probably Pass SB 1047

(25:29) Negative Reactions to Anthropic's Letter, Attempts to Suppress Dissent

(28:33) Positions In Brief

(33:57) Postscript: AB 3211 RTFBC (Read the Bill Changes)

The original text contained 2 images which were described by AI.

---

First published:
August 27th, 2024

Source:
https://www.lesswrong.com/posts/RaKWcwhygqpMnFZCp/sb-1047-final-takes-and-also-ab-3211

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

SB 1047 has been amended once more, with both strict improvements and big compromises. I cover the changes, and answer objections to the bill, in my extensive Guide to SB 1047. I follow that up here with reactions to the changes and some thoughts on where the debate goes from here. Ultimately, it is going to come down to one person: California Governor Gavin Newsom.

All of the debates we’re having matter to the extent they influence this one person. If he wants the bill to become law, it almost certainly will become law. If he does not want that, then it won’t become law, they never override a veto and if he makes that intention known then it likely wouldn’t even get to his desk. For now, he's not telling.

Table of Contents

1. Introduction.
2. Table of Contents.
3. Language Models Offer Mundane Utility. [...]

---

Outline:

(00:52) Language Models Offer Mundane Utility

(04:39) Language Models Don’t Offer Mundane Utility

(06:49) Deepfaketown and Botpocalypse Soon

(08:00) The Art of the Jailbreak

(15:34) Get Involved

(16:33) Introducing

(17:44) In Other AI News

(20:27) Quiet Speculations

(27:51) SB 1047: Nancy Pelosi

(35:30) SB 1047: Anthropic

(39:59) SB 1047: Reactions to the Changes

(53:41) SB 1047: Big Picture

(55:45) The Week in Audio

(58:32) Rhetorical Innovation

(59:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:55) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:
August 22nd, 2024

Source:
https://www.lesswrong.com/posts/qGh9suEsb82hzBtSN/ai-78-some-welcome-calm

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We now likely know the final form of California's SB 1047.

There have been many changes to the bill as it worked its way to this point.

Many changes, including some that were just announced, I see as strict improvements.

Anthropic was behind many of the last set of amendments at the Appropriations Committee. In keeping with their “Support if Amended” letter, there are a few big compromises that weaken the upside protections of the bill somewhat in order to address objections and potential downsides.

The primary goal of this post is to answer the question: What would SB 1047 do?

I offer two versions: Short and long.

The short version summarizes what the bill does, at the cost of being a bit lossy.

The long version is based on a full RTFB: I am reading the entire bill, once again.

---

Outline:

(01:16) Short Version (tl;dr): What Does SB 1047 Do in Practical Terms?

(04:19) Really Short Abbreviated Version

(05:46) Somewhat Less Short: Things The Above Leaves Out

(08:03) Bad Model, Bad Model, What You Gonna Do

(11:34) Going to Be Some Changes Made

(14:17) Long Version: RTFB

(15:01) Definitions (starting with Artificial Intelligence)

(15:35) Safety Incident

(17:15) Covered Model

(19:15) Critical Harm

(22:10) Full Shutdown

(23:35) Safety and Security Protocol

(25:31) On Your Marks

(34:41) Reasonable People May Disagree

(42:27) Release the Hounds

(44:02) Smooth Operator

(46:47) Compute Cluster Watch

(48:49) Price Controls are Bad

(49:21) A Civil Action

(56:16) Whistleblowers Need Protections

(59:01) No Division Only Board

(01:02:32) Does CalCompute?

(01:03:06) In Which We Respond To Some Objections In The Style They Deserve

(01:04:43) False Claim: The Government Can and Will Lower the $100m Threshold

(01:05:11) False Claim: SB 1047 Might Retroactively Cover Existing Models

(01:05:32) Moot or False Claim: The Government Can and Will Set the Derivative Model Threshold Arbitrarily Low

(01:05:45) Objection: The Government Could Raise the Derivative Threshold Model Too High,

(01:06:11) False Claim: Fine-Tuners Can Conspire to Evade the Derivative

(01:06:31) Moot Claim: The Frontier Model Division Inevitably Will Overregulate

(01:07:01) False Claim: The Shutdown Requirement Bans Open Source

(01:07:49) Objection: SB 1047 Will Slow AI Technology and Innovation or Interfere with Open Source

(01:11:49) False Claim: This Effectively Kills Open Source Because You Can Fine-Tune Any System To Do Harm

(01:13:42) False Claim: SB 1047 Will Greatly Hurt Academia

(01:14:42) False Claim: SB 1047 Favors ‘Big Tech’ over ‘Little Tech’

(01:16:14) False Claim: SB 1047 Would Cause Many Startups To Leave California

(01:17:25) Objection: Shutdown Procedures Could Be Hijacked and Backfire

(01:18:35) Objection: The Audits Will Be Too Expensive

(01:20:08) Objection: What Is Illegal Here is Already Illegal

(01:23:52) Objection: Jailbreaking is Inevitable

(01:24:41) Moot and False Claim: Reasonable Assurance Is Impossible

(01:25:04) Objection: Reasonable Care is Too Vague, Can’t We Do Better?

(01:25:50) Objection: The Numbers Picked are Arbitrary

(01:27:47) Objection: The Law Should Use Capabilities Thresholds, Not Compute and Compute Cost Thresholds

(01:29:27) False Claim: This Bill Deals With ‘Imaginary’ Risks

(01:30:38) Objection: This Might Become the Model For Other Bills Elsewhere

(01:31:16) Not Really an Objection: They Changed the Bill a Lot

(01:31:42) Not Really an Objection: The Bill Has the Wrong Motivations and Is Backed By Evil People

(01:33:34) Not an Objection: ‘The Consensus Has Shifted’ or ‘The Bill is Unpopular’

(01:34:17) Objection: It Is ‘Too Early’ To Regulate

(01:35:33) Objection: We Need To ‘Get It Right’ and Can Do Better

(01:36:27) Objection: This Would Be Better at the Federal Level

(01:36:53) Objection: The Bill Should Be Several Distinct Bills

(01:37:44) Objection: The Bill Has Been Weakened Too Much in Various Ways

(01:40:18) Final Word: Who Should Oppose This Bill?

The original text contained 2 images which were described by AI.

---

First published:
August 20th, 2024

Source:
https://www.lesswrong.com/posts/Z7pTfn4qqnKBoMi42/guide-to-sb-1047

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Apologies for forgetting to cross-post this and the Monthly Roundup earlier.]

Let's see. We’ve got a new version of GPT-4o, a vastly improved Grok 2 with a rather good and unrestricted deepfake and other image generator now baked into Twitter, the announcement of the AI powered Google Pixel 9 coming very soon and also Google launching a voice assistant. Anthropic now has prompt caching.

Also OpenAI has its final board member, Zico Kolter, who is nominally a safety pick, and SB 1047 got importantly amended again which I’ll cover in full next week once the details are out.

There was also the whole paper about the fully automated AI scientist from the company whose name literally means ‘danger’ in Hebrew, that instantiated copies of itself, took up unexpectedly large amounts of storage space, downloaded strange Python libraries and tried to edit its code to remove the [...]

---

Outline:

(01:08) Language Models Offer Mundane Utility

(04:45) Language Models Don’t Offer Mundane Utility

(08:01) GPT-4o My System Card

(15:49) 2 Grok 2 Furious 2 Quit

(25:32) Pixel Perfect

(27:52) Fun with Image Generation

(28:13) Deepfaketown and Botpocalypse Soon

(34:15) The Art of the Jailbreak

(43:48) They Took Our Jobs

(45:27) Obvious Nonsense

(50:12) Get Involved

(51:54) Introducing

(55:55) In Other AI News

(58:43) Quiet Speculations

(01:13:18) SB 1047: One Thing to Know

(01:14:21) SB 1047 is Amended Again

(01:16:32) SB 1047 Rhetoric Prior to the Recent Changes

(01:23:12) The Quest for Sane Regulations

(01:28:58) The Week in Audio

(01:30:13) Rhetorical Innovation

(01:36:32) Crying Wolf

(01:39:12) People Are Worried About AI Killing Everyone

(01:39:40) Other People Are Not As Worried About AI Killing Everyone

(01:40:18) The Lighter Side

The original text contained 19 images which were described by AI.

---

First published:
August 20th, 2024

Source:
https://www.lesswrong.com/posts/2tKbDKGLXtEHGtGLn/ai-77-a-few-upgrades

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Strictly speaking I do not have that much ‘good news’ to report, but it's all mostly fun stuff one way or another. Let's go.

Bad News

Is this you?

Patrick McKenzie: This sounds like a trivial observation and it isn’t:

No organization which makes its people pay for coffee wants to win.

There are many other questions you can ask about an organization but if their people pay for coffee you can immediately discount their realized impact on the world by > 90%.

This is not simply for the cultural impact of stupid decisions, though goodness as a Japanese salaryman I have stories to tell. Management, having priced coffee, seeking expenses to cut, put a price on disposable coffee cups, and made engineers diligently count those paper cups.

Just try to imagine how upside down the world is when you think one [...]

---

Outline:

(00:15) Bad News

(06:35) Grocery Store Blues

(08:47) Good News, Everyone

(09:43) Opportunity Knocks

(11:46) While I Cannot Condone This

(13:28) Antisocial Media

(16:10) Technology Advances

(17:51) Google Enshittification

(20:31) For Science!

(22:49) Government Working

(25:15) America F\*\*\* Yeah

(33:05) Smart People Being Stupid

(36:15) What We Have Here is A Failure to Communicate

(40:38) Video Killed the Radio Star

(43:02) Too Much Information

(47:47) Memory Holes

(49:18) Wet Ground Causes Rain (Dances)

(52:27) Get Them to the Church

(57:49) Patrick McKenzie Monthly

(01:01:13) Your Horoscope For Today

(01:03:14) Good Advice: Travel Edition

(01:05:30) Sports Go Sports

(01:05:34) Our Olympic team is mostly based in San Francisco.

(01:07:35) Gamers Gonna Game Game Game Game Game

(01:07:40) How many elite chess players cheat? Chess.com analysis of its big ‘Titled Tuesday’ events says between 1% and 2% of players, and roughly 1% of event winners. They are responding by making cheating bans on major plays public rather than quietly closing accounts, to fix the incentives.

(01:14:46) The Lighter Side

The original text contained 14 images which were described by AI.

---

First published:
August 20th, 2024

Source:
https://www.lesswrong.com/posts/2ne9taAPiGqoTLXJJ/monthly-roundup-21-august-2024

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

While I finish up the weekly for tomorrow morning after my trip, here's a section I expect to want to link back to every so often in the future. It's too good.

Danger, AI Scientist, Danger

As in, the company that made the automated AI Scientist that tried to rewrite its code to get around resource restrictions and launch new instances of itself while downloading bizarre Python libraries?

Its name is Sakana AI. (魚≈סכנה). As in, in hebrew, that literally means ‘danger’, baby.

It's like when someone told Dennis Miller that Evian (for those who don’t remember, it was one of the first bottled water brands) is Naive spelled backwards, and he said ‘no way, that's too f***ing perfect.’

This one was sufficiently appropriate and unsubtle that several people noticed. I applaud them choosing a correct Kabbalistic name. Contrast this with Meta calling its [...]

---

Outline:

(00:15) Danger, AI Scientist, Danger

(01:11) In the Abstract

(04:01) How Any of This Sort of Works

(06:56) New Benchmark Just Dropped

(07:30) Nothing to See Here

(09:50) All Fun and Games

The original text contained 2 images which were described by AI.

---

First published:
August 15th, 2024

Source:
https://www.lesswrong.com/posts/ppafWk6YCeXYr4XpH/danger-ai-scientist-danger

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

If you’re looking forward to next week’s AI #77, I am going on a two-part trip this week. First I’ll be going to Steamboat in Colorado to give a talk, then I’ll be swinging by Washington, DC on Wednesday, although outside of that morning my time there will be limited. My goal is still to get #77 released before Shabbat dinner, we’ll see if that works. Some topics may of course get pushed a bit.

It’s crazy how many of this week’s developments are from OpenAI. You’ve got their voice mode alpha, JSON formatting, answering the letter from several senators, sitting on watermarking for a year, endorsement of three bills before Congress and also them losing a cofounder to Anthropic and potentially another one via sabbatical.

Also Google found to be a monopolist, we have the prompts for Apple Intelligence and other neat stuff like that.

---

Outline:

(01:43) Language Models Offer Mundane Utility

(05:03) Language Models Don’t Offer Mundane Utility

(08:09) Activate Voice Mode

(12:23) Apple Intelligence

(16:38) Antitrust Antitrust

(19:39) Copyright Confrontation

(20:50) Fun with Image Generation

(22:06) Deepfaketown and Botpocalypse Soon

(26:25) They Took Our Jobs

(29:18) Chipping Up

(31:36) Get Involved

(31:56) Introducing

(33:31) In Other AI News

(43:20) Quiet Speculations

(47:40) The Quest for Sane Regulations

(49:48) That's Not a Good Idea

(54:05) The Week in Audio

(55:54) Exact Words

(01:01:54) Openly Evil AI

(01:09:06) Goodbye to OpenAI

(01:15:10) Rhetorical Innovation

(01:21:16) Open Weights Are Unsafe and Nothing Can Fix This

(01:23:33) Aligning a Smarter Than Human Intelligence is Difficult

(01:24:34) People Are Worried About AI Killing Everyone

(01:25:44) Other People Are Not As Worried About AI Killing Everyone

(01:28:37) The Lighter Side

---

First published:
August 8th, 2024

Source:
https://www.lesswrong.com/posts/4GnsAtamtcrsTFmSf/ai-76-six-shorts-stories-about-openai

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: Startup Roundup #1.

This is my periodic grab bag coverage of various issues surrounding startups, especially but not exclusively tech-and-VC style startups, that apply over the longer term.

I always want to emphasize up front that startups are good and you should do one.

Equity and skin in the game are where it is at. Building something people want is where it is at. This is true both for a startup that raises venture capital, and also creating an ordinary business. The expected value is all around off the charts.

That does not mean it is the best thing to do.

One must go in with eyes open to facts such as these:

1. It is hard.
2. There are many reasons it might not be for you.
3. There are also lots of other things also worth doing.
4. If you [...]

---

Outline:

(01:28) An Entrepreneur Immigration Program

(03:11) Times are Tough Outside of AI

(06:23) Times Otherwise Not So Tough

(08:19) Warning

(10:29) Red Flags

(12:35) Free Advice is Seldom Cheap

(15:45) Short Work

(18:07) The Founder

(19:24) Venture Capital Incentives

(29:27) The Margin of Difficulty

(33:27) Cold Outreach

(34:32) Associates at VC Firms Don’t Matter

(35:24) Lean Versus Fast

(35:59) Build Something People Want

(37:14) Get Them Early

(42:08) Learn to Code

(43:28) The Goal

(44:03) Working Hard

(46:26) Revenue

(48:37) The Place to Be

(50:50) YC Remains a Great Deal

(52:10) Hardware Startups

(52:41) How to Hire Well

(53:45) How to Check References

(54:24) You’re Fired

(55:43) Dealing With the Press

(56:47) Emotional Runway

(57:34) He Who Has the Gold

(58:19) Selling Out

The original text contained 6 images which were described by AI.

---

First published:
August 6th, 2024

Source:
https://www.lesswrong.com/posts/LjRsHjQD2DGSRvJdt/startup-roundup-2

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Google DeepMind got a silver metal at the IMO, only one point short of the gold. That's really exciting.

We continuously have people saying ‘AI progress is stalling, it's all a bubble’ and things like that, and I always find remarkable how little curiosity or patience such people are willing to exhibit. Meanwhile GPT-4o-Mini seems excellent, OpenAI is launching proper search integration, by far the best open weights model got released, we got an improved MidJourney 6.1, and that's all in the last two weeks. Whether or not GPT-5-level models get here in 2024, and whether or not it arrives on a given schedule, make no mistake. It's happening.

This week also had a lot of discourse and events around SB 1047 that I failed to avoid, resulting in not one but four sections devoted to it.

Dan Hendrycks was baselessly attacked – by billionaires with [...]

---

Outline:

(02:12) Language Models Offer Mundane Utility

(03:06) Language Models Don’t Offer Mundane Utility

(04:18) Math is Easier

(08:15) Llama Llama Any Good

(11:52) Search for the GPT

(15:17) Tech Company Will Use Your Data to Train Its AIs

(17:14) Fun with Image Generation

(17:37) Deepfaketown and Botpocalypse Soon

(27:36) The Art of the Jailbreak

(29:54) Janus on the 405

(32:47) They Took Our Jobs

(33:29) Get Involved

(34:05) Introducing

(37:07) In Other AI News

(40:18) Quiet Speculations

(43:40) The Quest for Sane Regulations

(55:31) Death and or Taxes

(58:18) SB 1047 (1)

(01:00:56) SB 1047 (2)

(01:14:29) SB 1047 (3): Oh Anthropic

(01:20:13) What Anthropic's Letter Actually Proposes

(01:36:44) Open Weights Are Unsafe and Nothing Can Fix This

(01:39:41) The Week in Audio

(01:40:09) Rhetorical Innovation

(01:50:56) Businessman Waves Flag

(01:57:48) Businessman Pledges Safety Efforts

(02:04:18) Aligning a Smarter Than Human Intelligence is Difficult

(02:04:39) Aligning a Dumber Than Human Intelligence is Also Difficult

(02:07:52) Other People Are Not As Worried About AI Killing Everyone

(02:13:28) The Lighter Side

The original text contained 15 images which were described by AI.

---

First published:
August 1st, 2024

Source:
https://www.lesswrong.com/posts/2p5suvWod4aP8S3S4/ai-75-math-is-easier

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Some in the tech industry decided now was the time to raise alarm about AB 3211.

As Dean Ball points out, there's a lot of bills out there. One must do triage.

Dean Ball: But SB 1047 is far from the only AI bill worth discussing. It's not even the only one of the dozens of AI bills in California worth discussing. Let's talk about AB 3211, the California Provenance, Authenticity, and Watermarking Standards Act, written by Assemblymember Buffy Wicks, who represents the East Bay.

SB 1047 is a carefully written bill that tries to maximize benefits and minimize costs. You can still quite reasonably disagree with the aims, philosophy or premise of the bill, or its execution details, and thus think its costs exceed its benefits. When people claim SB 1047 is made of crazy pills, they are attacking provisions not in the bill.

---

Outline:

(03:44) Read The Bill (RTFB)

(17:02) What About Open Weights Models?

(18:25) What Does the Bill Do in Practice?

(20:33) Compare and Contrast

The original text contained 1 image which was described by AI.

---

First published:
July 30th, 2024

Source:
https://www.lesswrong.com/posts/JHAASAhCZgmwcaLvd/rtfb-california-s-ab-3211

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's here. The horse has left the barn. Llama-3.1-405B, and also Llama-3.1-70B and Llama-3.1-8B, have been released, and are now open weights.

Early indications are that these are very good models. They were likely the best open weight models of their respective sizes at time of release.

Zuckerberg claims that open weights models are now competitive with closed models. Yann LeCun says ‘performance is on par with the best closed models.’ This is closer to true than in the past, and as corporate hype I will essentially allow it, but it looks like this is not yet fully true.

Llama-3.1-405B not as good as GPT-4o or Claude Sonnet. Certainly Llama-3.1-70B is not as good as the similarly sized Claude Sonnet. If you are going to straight up use an API or chat interface, there seems to be little reason to use Llama.

That is a [...]

---

Outline:

(04:25) Options to Run It

(04:45) The Model Card

(08:42) Benchmarks

(13:41) Human Reactions in the Wild

(16:56) What's It Good For?

(21:39) The Other Other Guy

(22:35) Safety

(31:48) Three People Can Keep a Secret and Reasonably Often Do So

(36:12) The Announcement and Interview

(47:59) Zuckerberg's Open Weights Manifesto

(58:17) Fun Little Note

The original text contained 15 images which were described by AI.

---

First published:
July 24th, 2024

Source:
https://www.lesswrong.com/posts/fjzPg9ATbTJcnBZvg/llama-llama-3-405b

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It is monthly roundup time.

I invite readers who want to hang out and get lunch in NYC later this week to come on Thursday at Bhatti Indian Grill (27th and Lexington) at noon.

I plan to cover the UBI study in its own post soon.

I cover Nate Silver's evisceration of the 538 presidential election model, because we cover probabilistic modeling and prediction markets here, but excluding any AI discussions I will continue to do my best to stay out of the actual politics.

Bad News

Jeff Bezos’ rocket company Blue Origin files comment suggesting SpaceX Starship launches be capped due to ‘impact on local environment.’ This is a rather shameful thing for them to be doing, and not for the first time.

Alexey Guzey reverses course, realizes at 26 that he was a naive idiot at 20 and finds everything he [...]

---

Outline:

(00:37) Bad News

(02:37) Silver Bullet

(06:35) Shame on Kathy Hochul

(06:58) This is (One Reason) Why We Can’t Have Nice Things

(09:02) (Don’t) Hack the Planet

(10:16) The Laptop Trap

(12:24) Courage

(14:25) Friendship

(15:26) The Gravest Mistake

(21:01) You Need Functional Decision Theory

(23:40) Antisocial Media

(25:52) For Science!

(30:58) Truth Seeking

(34:37) Liar Liar

(36:51) Government Working

(47:55) For Your Entertainment

(52:18) Variously Effective Altruism

(55:09) News You Can Use

(55:31) Good News, Everyone

(58:03) Gamers Gonna Game Game Game Game Game

(01:01:22) Sports Go Sports

(01:05:10) I Was Promised Flying Self-Driving Cars

(01:05:50) While I Cannot Condone This

(01:12:55) The Lighter Side

The original text contained 15 images which were described by AI.

---

First published:
July 23rd, 2024

Source:
https://www.lesswrong.com/posts/qfQspPDHMSEpwsuAQ/monthly-roundup-20-july-2024

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Things went very wrong on Friday.

A bugged CrowdStrike update temporarily bricked quite a lot of computers, bringing down such fun things as airlines, hospitals and 911 services.

It was serious out there.

Ryan Peterson: Crowdstrike outage has forced Starbucks to start writing your name on a cup in marker again and I like it.

What (Technically) Happened

My understanding it was a rather stupid bug, a NULL pointer from the memory unsafe C++ language.

Zack Vorhies: Memory in your computer is laid out as one giant array of numbers. We represent these numbers here as hexadecimal, which is base 16 (hexadecimal) because it's easier to work with… for reasons.

The problem area? The computer tried to read memory address 0x9c (aka 156).

Why is this bad?

This is an invalid region of memory for any program. Any program that [...]

---

Outline:

(00:31) What (Technically) Happened

(03:38) Who to Blame?

(06:58) How Did We Let This Happen

(12:41) Regulatory Compliance

(18:14) Consequences

(19:54) Careful With That AI

(29:34) Unbanked

The original text contained 2 images which were described by AI.

---

First published:
July 22nd, 2024

Source:
https://www.lesswrong.com/posts/oAKfaxKKfuz2cuRLr/on-the-crowdstrike-incident

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What do you call a clause explicitly saying that you waive the right to whistleblower compensation, and that you need to get permission before sharing information with government regulators like the SEC?

I have many answers.

I also know that OpenAI, having f***ed around, seems poised to find out, because that is the claim made by whistleblowers to the SEC. Given the SEC fines you for merely not making an explicit exception to your NDA for whistleblowers, what will they do once aware of explicit clauses going the other way?

(Unless, of course, the complaint is factually wrong, but that seems unlikely.)

We also have rather a lot of tech people coming out in support of Trump. I go into the reasons why, which I do think is worth considering. There is a mix of explanations, and at least one very good reason.

Then [...]

---

Outline:

(01:40) Language Models Offer Mundane Utility

(08:10) Language Models Don’t Offer Mundane Utility

(10:19) Clauding Along

(12:25) Fun with Image Generation

(13:46) Deepfaketown and Botpocalypse Soon

(14:49) They Took Our Jobs

(18:29) Get Involved

(20:14) Introducing

(21:46) In Other AI News

(25:02) Denying the Future

(26:59) Quiet Speculations

(32:34) The Quest for Sane Regulations

(37:03) The Other Quest Regarding Regulations

(57:02) SB 1047 Opposition Watch (1)

(01:07:41) SB 1047 Opposition Watch (2)

(01:11:21) Open Weights are Unsafe and Nothing Can Fix This

(01:12:21) The Week in Audio

(01:14:21) Rhetorical Innovation

(01:14:48) Oh Anthropic

(01:16:57) Openly Evil AI

(01:24:21) Aligning a Smarter Than Human Intelligence is Difficult

(01:37:30) People Are Worried About AI Killing Everyone

(01:37:52) Other People Are Not As Worried About AI Killing Everyone

(01:39:49) The Lighter Side

The original text contained 19 images which were described by AI.

---

First published:
July 18th, 2024

Source:
https://www.lesswrong.com/posts/fM4Bs9nanDzio3xCq/ai-73-openly-evil-ai

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Future. It is coming.

A surprising number of economists deny this when it comes to AI. Not only do they deny the future that lies in the future. They also deny the future that is here, but which is unevenly distributed. Their predictions and projections do not factor in even what the AI can already do, let alone what it will learn to do later on.

Another likely future event is the repeal of the Biden Executive Order. That repeal is part of the Republican platform, and Trump is the favorite to win the election. We must act on the assumption that the order likely will be repealed, with no expectation of similar principles being enshrined in federal law.

Then there are the other core problems we will have to solve, and other less core problems such as what to do about AI companions. They [...]

---

Outline:

(01:19) Language Models Offer Mundane Utility

(03:54) Language Models Don’t Offer Mundane Utility

(06:22) You’re a Nudge

(08:02) Fun with Image Generation

(08:10) Deepfaketown and Botpocalypse Soon

(13:26) They Took Our Jobs

(13:43) Get Involved

(14:13) Introducing

(15:23) In Other AI News

(20:02) Quiet Speculations

(22:14) The AI Denialist Economists

(29:07) The Quest for Sane Regulations

(31:20) Trump Would Repeal the Biden Executive Order on AI

(34:40) Ordinary Americans Are Worried About AI

(37:50) The Week in Audio

(38:59) The Wikipedia War

(46:45) Rhetorical Innovation

(52:06) Evaluations Must Mimic Relevant Conditions

(54:36) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:51) The Problem

(01:08:43) Oh Anthropic

(01:11:50) Other People Are Not As Worried About AI Killing Everyone

(01:15:48) The Lighter Side

The original text contained 15 images which were described by AI.

---

First published:
July 11th, 2024

Source:
https://www.lesswrong.com/posts/xAoXxjtDGGCP7tBDY/ai-72-denying-the-future

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This time around, we cover the Hanson/Alexander debates on the value of medicine, and otherwise we mostly have good news.

Technology Advances

Regeneron administers a single shot in a genetically deaf child's ear, and they can hear after a few months, n=2 so far.

Great news: An mRNA vaccine in early human clinical trials reprograms the immune system to attack glioblastoma, the most aggressive and lethal brain tumor. It will now proceed to Phase I. In a saner world, people would be able to try this now.

More great news, we have a cancer vaccine trial in the UK.

And we’re testing personalized mRNA BioNTech canner vaccines too.

US paying Moderna $176 million to develop a pandemic vaccine against bird flu.

We also have this claim that Lorlatinib jumps cancer PFS rates from 8% to 60%.

The GLP-1 Revolution

Early [...]

---

Outline:

(00:12) Technology Advances

(01:04) The GLP-1 Revolution

(04:46) Claims About Hansoninan Medicine

(18:36) Pricing

(19:17) Epistemics

(19:39) DEA Worse Than FDA

(21:19) Study Harder

(23:05) FDA Delenda Est

(24:51) Bioethics

(27:28) Covid

(29:21) Demons

(32:48) Genetics

The original text contained 2 images which were described by AI.

---

First published:
July 9th, 2024

Source:
https://www.lesswrong.com/posts/6GhemtgJxF9sSDNrq/medical-roundup-3

Narrated by TYPE III AUDIO.

Chevron deference is no more. How will this impact AI regulation?

The obvious answer is it is now much harder for us to ‘muddle through via existing laws and regulations until we learn more,’ because the court narrowed our affordances to do that. And similarly, if and when Congress does pass bills regulating AI, they are going to need to ‘lock in’ more decisions and grant more explicit authority, to avoid court challenges. The argument against state regulations is similarly weaker now.

Similar logic also applies outside of AI. I am overall happy about overturning Chevron and I believe it was the right decision, but ‘Congress decides to step up and do its job now’ is not in the cards. We should be very careful what we have wished for, and perhaps a bit burdened by what has been.

The AI world continues to otherwise be [...]

---

Outline:

(01:06) Language Models Offer Mundane Utility

(02:22) Language Models Don’t Offer Mundane Utility

(04:43) Man in the Arena

(07:56) Fun with Image Generation

(08:48) Deepfaketown and Botpocalypse Soon

(11:17) They Took Our Jobs

(12:04) The Art of the Jailbreak

(16:47) Get Involved

(17:34) Introducing

(19:26) In Other AI News

(19:50) Quiet Speculations

(26:03) The Quest for Sane Regulations

(27:19) Chevron Overturned

(45:11) The Week in Audio

(52:49) Oh Anthropic

(53:59) Open Weights Are Unsafe and Nothing Can Fix This

(55:16) Rhetorical Innovation

(55:39) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:51) People Are Worried About AI Killing Everyone

(01:06:16) Other People Are Not As Worried About AI Killing Everyone

(01:08:57) The Lighter Side

The original text contained 12 images which were described by AI.

---

First published:
July 4th, 2024

Source:
https://www.lesswrong.com/posts/AYJcL6GD3FLkL4yNC/ai-71-farewell-to-chevron

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: Economics Roundup #1

Let's take advantage of the normality while we have it. In all senses.

Insane Tax Proposals

There is Trump's proposal to replace income taxes with tariffs, but he is not alone.

So here is your periodic reminder, since this is not actually new at core: Biden's proposed budgets include completely insane tax regimes that would cripple our economic dynamism and growth if enacted. As in for high net worth individuals, taking unrealized capital gains at 25% and realized capital gains, such as those you are forced to take to pay your unrealized capital gains tax, at 44.6% plus state taxes.

Austen Allred explains how this plausibly destroys the entire startup ecosystem.

Which I know is confusing because in other contexts he also talks about how other laws (such as SB 1047) that would in no way apply to startups [...]

---

Outline:

(00:14) Insane Tax Proposals

(04:44) Don’t Mess With the Federal Reserve

(05:23) Don’t Mess With the New York Tax Authorities

(06:15) Tariffs

(10:39) People Hate Inflation

(16:22) Real Wages

(17:20) Can’t Get No Satisfaction

(17:54) Employment

(18:35) The National Debt

(19:49) Immigration

(21:44) Financial Literacy

(23:32) Reversal

(24:28) Status Update

(24:53) Scaling Hypothesis

(25:28) Payments

(27:38) Pricing

(29:56) Never Reason From a Price Change

(31:10) A Changing Price

(35:33) The Price of Tourism

(36:34) Alcohol

(37:18) The Efficient Company Hypothesis is False

(41:01) Falling Hours Worked

(41:43) Trust Me

(42:58) China

(44:35) Other

The original text contained 8 images which were described by AI.

---

First published:
July 2nd, 2024

Source:
https://www.lesswrong.com/posts/MMtWB8wAu5Buc6sve/economics-roundup-2

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

They said it couldn’t be done.

No, not Claude Sonnet 3.5 becoming the clear best model.

No, not the Claude-Sonnet-empowered automatic meme generators. Those were whipped together in five minutes.

They said I would never get quiet time and catch up. Well, I showed them!

That's right. Yes, there is a new best model, but otherwise it was a quiet week. I got a chance to incorporate the remaining biggest backlog topics. The RAND report is covered under Thirty Eight Ways to Steal Your Model Weights. Last month's conference in Seoul is covered in You’ve Got Seoul. I got to publish my thoughts on OpenAI's Model Spec last Friday.

Table of Contents

Be sure to read about Claude 3.5 Sonnet here. That is by far the biggest story.

1. Introduction.
2. Table of Contents.
3. Language Models Offer Mundane Utility. I am [...]

---

Outline:

(00:50) Language Models Offer Mundane Utility

(02:38) Language Models Don’t Offer Mundane Utility

(04:41) Clauding Along

(09:46) Fun with Image Generation

(12:21) Copyright Confrontation

(16:30) Deepfaketown and Botpocalypse Soon

(19:45) They Took Our Jobs

(24:49) The Art of the Jailbreak

(25:40) Get Involved

(26:56) Introducing

(30:45) In Other AI News

(33:50) Quiet Speculations

(43:08) You’ve Got Seoul

(57:36) Thirty Eight Ways to Steal Your Model Weights

(01:07:48) The Quest for Sane Regulations

(01:13:11) SB 1047

(01:14:49) The Week in Audio

(01:20:25) Rhetorical Innovation

(01:22:26) People Are Worried About AI Killing Everyone

(01:24:05) Other People Are Not As Worried About AI Killing Everyone

(01:24:50) The Lighter Side

The original text contained 34 images which were described by AI.

---

First published:
June 27th, 2024

Source:
https://www.lesswrong.com/posts/rC3hhZsx2KogoPLqh/ai-70-a-beautiful-sonnet

Narrated by TYPE III AUDIO.

Childhood roundup #5 excluded all developments around college. So this time around is all about issues related to college or graduate school, including admissions.

Tuition and Costs

What went wrong with federal student loans? Exactly what you would expect when you don’t check who is a good credit risk. From a performance perspective, the federal government offered loans to often-unqualified students to attend poor-performing, low-value institutions. Those students then did not earn much and were often unable to repay the loans. The students are victims here too, as we told them to do it.

Alas, none of the proposed student loan solutions involve fixing the underlying issue. If you said ‘we are sorry we pushed these loans on students and rewarded programs and institutions that do not deserve it, and we are going to stop giving loans for those programs and institutions and offer help to [...]

---

Outline:

(00:18) Tuition and Costs

(03:07) What Your Tuition Buys You

(05:45) Decline of Academia

(07:24) Grading and Stress

(14:29) Lower Standards

(15:43) Degree Value

(19:39) Shifting Consumer Preferences

(21:01) Standardized Tests in College Admissions

(21:44) Discrimination in College Admissions

(25:18) Required Classes and Choosing Your Major

(28:31) Everything I Need To Know That Waited Until Graduate School

(31:19) When You See Fraud Say Fraud

(33:03) Free Speech

(35:28) Harvard Goes Mission First

(37:03) The Waterloo Model

(38:33) DEI

(45:20) In Other News

---

First published:
June 26th, 2024

Source:
https://www.lesswrong.com/posts/pn5jWW4zcWSAjM9s3/childhood-and-education-roundup-6-college-edition

Narrated by TYPE III AUDIO.

Looks like we made it. Yes, the non-AI world still exists.

Bad Governor

New York Governor Kathy Hochul has gone rogue and betrayed New York City, also humanity, declaring a halt to congestion pricing a month before it was to go into effect. Her explanation was that she spoke to workers at three Manhattan diners who were worried people would be unable to drive to them from New Jersey. Which, as Cathy Reilly points out, is rather insulting to New Jersey, and also completely absurd. Who in the world was going to go into Manhattan for a diner?

She says this won’t interfere with Subway work. Work on the 2nd Avenue Subway line has already been halted. And that's not all.

You’re damn right. We are going to blame Hochul. Every. Damn. Time.

So Elizabeth Kim investigated. One never talked politics at all. One [...]

---

Outline:

(00:12) Bad Governor

(02:12) High Skilled Immigration

(04:46) Various Bad News

(07:46) Prediction Markets Are Unpopular

(09:30) New Buildings are Ugly

(11:06) Government Working

(15:24) The Snafu Principle

(18:11) Technology Advances

(18:51) For Science!

(28:38) Antisocial Media

(35:07) The Twitter Porn Bot War

(37:07) I Like My Twitter Posts Like I Like My Porn Bots: Private

(40:36) Variously Effective Altruism

(42:06) Are You Happy Now?

(45:49) Good News, Everyone

(51:07) Good Social Advice

(56:16) FTC Wants to Ban Noncompetes

(01:02:52) While I Cannot Condone This

(01:11:20) Enemies of the People

(01:14:39) Lab Grown Meat Shirts Answer and Raise Questions

(01:15:51) Ban Gain of Function Research

(01:16:46) Gamers Gonna Game Game Game Game Game

(01:26:30) Sports Go Sports

(01:26:54) I Was Promised Flying Self-Driving Cars

(01:30:01) Patrick McKenzie Monthly

(01:32:41) The Lighter Side

---

First published:
June 25th, 2024

Source:
https://www.lesswrong.com/posts/7LvK6Gw2GdfDMBNNm/monthly-roundup-19-june-2024

Narrated by TYPE III AUDIO.

There is a new clear best (non-tiny) LLM.

If you want to converse with an LLM, the correct answer is Claude Sonnet 3.5.

It is available for free on Claude.ai and the Claude iOS app, or you can subscribe for higher rate limits. The API cost is $3 per million input tokens and $15 per million output tokens.

This completes the trifecta. All of OpenAI, Google DeepMind and Anthropic have kept their biggest and more expensive model static for now, and instead focused on making something faster and cheaper that is good enough to be the main model.

You would only use another model if you either (1) needed a smaller model in which case Gemini 1.5 Flash seems best, or (2) it must have open model weights.

Updates to their larger and smaller models, Claude Opus 3.5 and Claude Haiku 3.5, are coming [...]

---

Outline:

(01:59) Benchmarks

(03:08) Human Evaluation Tests

(04:19) The Vision Thing

(05:46) Artifacts

(07:08) Privacy

(07:49) Safety

(08:51) Advancing the Frontier

(10:51) The Race is On

(11:43) Whispers of Recursive Self-Improvement

(16:35) Logic Fails

(18:34) Practical Reports

(24:20) What Comes Next

---

First published:
June 24th, 2024

Source:
https://www.lesswrong.com/posts/wx4RhFzLbiHoShFjR/on-claude-3-5-sonnet

Narrated by TYPE III AUDIO.

The big news this week was Apple Intelligence being integrated deeply into all their products. Beyond that, we had a modestly better than expected debate over the new version of SB 1047, and the usual tons of stuff in the background. I got to pay down some writing debt.

The bad news is, oh no, I have been called for Jury Duty. The first day or two I can catch up on podcasts or pure reading, but after that it will start to hurt. Wish me luck.

Table of Contents

AiPhone covers the announcement of Apple Intelligence. Apple's products are getting device-wide integration of their own AI in a way they say preserves privacy, with access to ChatGPT via explicit approval for the heaviest requests. A late update: OpenAI is providing this service for free as per Bloomberg.

I offered Quotes from Leopold Aschenbrenner's Situational [...]

---

Outline:

(00:38) Language Models Offer Mundane Utility

(01:46) Language Models Don’t Offer Mundane Utility

(07:49) Fun with Image Generation

(10:09) Copyright Confrontation

(11:38) Deepfaketown and Botpocalypse Soon

(15:20) They Took Our Jobs

(16:33) Someone Explains it All

(19:20) The Art of the Jailbreak

(22:39) Get Involved

(22:46) Introducing

(25:18) In Other AI News

(27:56) Quiet Speculations

(32:17) I Spy With My AI

(34:40) Pick Up the Phone

(35:06) Lying to the White House, Senate and House of Lords

(39:48) The Quest for Sane Regulations

(43:52) More Reasonable SB 1047 Reactions

(51:12) Less Reasonable SB 1047 Reactions

(56:24) That's Not a Good Idea

(56:46) With Friends Like These

(58:13) The Week in Audio

(01:01:22) Rhetorical Innovation

(01:02:32) Mistakes Were Made

(01:03:36) The Sacred Timeline

(01:08:17) Coordination is Hard

(01:13:33) Aligning a Smarter Than Human Intelligence is Difficult

(01:19:07) People Are Worried About AI Killing Everyone

(01:28:48) Other People Are Not As Worried About AI Killing Everyone

(01:29:58) The Lighter Side

---

First published:
June 13th, 2024

Source:
https://www.lesswrong.com/posts/DWkhjAxbwdcxYgyrJ/ai-68-remarkably-reasonable-reactions

Narrated by TYPE III AUDIO.

Previously: On the Podcast, Quotes from the Paper

This is a post in three parts.

The first part is my attempt to condense Leopold Aschenbrenner's paper and model into its load bearing elements and core logic and dependencies.

Two versions here, a long version that attempts to compress with minimal loss, and a short version that gives the gist.

The second part goes over where I agree and disagree, and briefly explains why.

The third part is the summary of other people's reactions and related discussions, which will also include my own perspectives on related issues.

My goal is often to ask ‘what if.’ There is a lot I disagree with. For each subquestion, what would I think here, if the rest was accurate, or a lot of it was accurate?

Summary of Biggest Agreements and Disagreements

I had Leopold review [...]

---

Outline:

(00:54) Summary of Biggest Agreements and Disagreements

(04:58) Decision Theory is Important

(06:38) Part 1: Leopold's Model and Its Implications

(06:44) The Long Version

(17:03) The Short Version

(19:04) Which Assumptions Are How Load Bearing in This Model?

(28:40) Part 2: Where I Agree and Disagree

(56:46) Part 3: Reactions of Others

(56:51) The Basics

(01:00:37) A Clarification from Eliezer Yudkowsky

(01:05:56) Children of the Matrix

(01:10:52) Aligning a Smarter Than Human Intelligence is Difficult

(01:13:28) The Sacred Timeline

(01:17:51) The Need to Update

(01:20:17) Open Models and Insights Can Be Copied

(01:21:38) You Might Not Be Paranoid If They’re Really Out to Get You

(01:24:12) We Are All There Is

(01:25:39) The Inevitable Conflict

(01:38:50) There Are Only Least Bad Options

(01:40:24) A Really Big Deal

(01:41:53) What Gives You the Right?

(01:43:34) Random Other Thoughts

---

First published:
June 14th, 2024

Source:
https://www.lesswrong.com/posts/b8u6nF5GAb6Ecttev/the-leopold-model-analysis-and-reactions

Narrated by TYPE III AUDIO.

The fun at OpenAI continues.

We finally have the details of how Leopold Aschenbrenner was fired, at least according to Leopold. We have a letter calling for a way for employees to do something if frontier AI labs are endangering safety. And we have continued details and fallout from the issues with non-disparagement agreements and NDAs.

Hopefully we can stop meeting like this for a while.

Due to jury duty and it being largely distinct, this post does not cover the appointment of General Paul Nakasone to the board of directors. I’ll cover that later, probably in the weekly update.

The Firing of Leopold Aschenbrenner

What happened that caused Leopold to leave OpenAI? Given the nature of this topic, I encourage getting the story from Leopold by following along on the transcript of that section of his appearance on the Dwarkesh Patel Podcast or [...]

---

Outline:

(00:43) The Firing of Leopold Aschenbrenner

(11:24) Daniel Kokotajlo Speaks and The Right to Warn

(15:07) The Right to Warn Letter

(18:48) Signed by (alphabetical order):

(19:34) Endorsed by (alphabetical order):

(34:17) You’ll Be Hearing From Our Lawyer

(35:29) Possession is Nine Tenths of the Law

(41:31) What I Can Tell You I Used To Not Be Able To Tell You

(47:50) Clarifying the Mission

(52:04) Sam Altman Told the SEC He Was Chairman of YC

(53:26) YC Has an Investment in OpenAI

(54:33) OpenAI is Hiring a Lot of Lobbyists

(55:05) OpenAI Says They Value Privacy

(55:58) Microsoft Went Around the Safety Board

(56:39) I Don’t Really Know What You Were Expecting

(57:12) Where Did Everybody Go?

(59:43) In Other OpenAI News

---

First published:
June 17th, 2024

Source:
https://www.lesswrong.com/posts/q3zs7E7rktHsESXaF/openai-8-the-right-to-warn

Narrated by TYPE III AUDIO.

On DeepMind's Frontier Safety Framework

Previously: On OpenAI's Preparedness Framework, On RSPs.

The First Two Frameworks

To first update on Anthropic and OpenAI's situation here:

Anthropic's RSP continues to miss the definitions of the all-important later levels, in addition to other issues, although it is otherwise promising. It has now been a number of months, and it is starting to be concerning that nothing has changed. They are due for an update.

OpenAI also has not updated its framework.

I am less down on OpenAI's framework choices than Zack Stein-Perlman was in the other review I have seen. I think that if OpenAI implemented the spirit of what it wrote down, that would be pretty good. The Critical-level thresholds listed are too high, but the Anthropic ASL-4 commitments are still unspecified. An update is needed, but I appreciate the concreteness.

The [...]

---

Outline:

(00:04) On DeepMind's Frontier Safety Framework

(00:15) The First Two Frameworks

(03:06) The DeepMind Framework

(08:57) Mitigations

(09:59) Security Mitigations

---

First published:
June 18th, 2024

Source:
https://www.lesswrong.com/posts/frEYsehsPHswDXnNX/on-deepmind-s-frontier-safety-framework

Narrated by TYPE III AUDIO.

Nice job breaking it, hero, unfortunately. Ilya Sutskever, despite what I sincerely believe are the best of intentions, has decided to be the latest to do The Worst Possible Thing, founding a new AI company explicitly looking to build ASI (superintelligence). The twists are zero products with a ‘cracked’ small team, which I suppose is an improvement, and calling it Safe Superintelligence, which I do not suppose is an improvement.

How is he going to make it safe? His statements tell us nothing meaningful about that.

There were also changes to SB 1047. Most of them can be safely ignored. The big change is getting rid of the limited duty exception, because it seems I was one of about five people who understood it, and everyone kept thinking it was a requirement for companies instead of an opportunity. And the literal chamber of commerce fought hard to [...]

---

Outline:

(01:43) Language Models Offer Mundane Utility

(06:13) Language Models Don’t Offer Mundane Utility

(08:21) Fun with Image Generation

(13:05) Deepfaketown and Botpocalypse Soon

(14:12) The Art of the Jailbreak

(15:07) Copyright Confrontation

(16:14) A Matter of the National Security Agency

(21:22) Get Involved

(21:38) Introducing

(23:56) In Other AI News

(29:06) Quiet Speculations

(38:29) AI Is Going to Be Huuuuuuuuuuge

(47:18) SB 1047 Updated Again

(01:00:48) The Quest for Sane Regulations

(01:02:42) The Week in Audio

(01:03:54) The ARC of Progress

(01:13:10) Put Your Thing In a Box

(01:15:50) What Will Ilya Do?

(01:20:54) Actual Rhetorical Innovation

(01:24:19) Rhetorical Innovation

(01:27:48) Aligning a Smarter Than Human Intelligence is Difficult

(01:32:28) People Are Worried About AI Killing Everyone

(01:32:41) Other People Are Not As Worried About AI Killing Everyone

(01:35:38) The Lighter Side

---

First published:
June 20th, 2024

Source:
https://www.lesswrong.com/posts/ytFLs37zLsFBqLHGA/ai-69-nice

Narrated by TYPE III AUDIO.

There are multiple excellent reasons to publish a Model Spec like OpenAI's, that specifies how you want your model to respond in various potential situations.

1. It lets us have the debate over how we want the model to act.
2. It gives us a way to specify what changes we might request or require.
3. It lets us identify whether a model response is intended.
4. It lets us know if the company successfully matched its spec.
5. It lets users and prospective users know what to expect.
6. It gives insight into how people are thinking, or what might be missing.
7. It takes responsibility.

These all apply even if you think the spec in question is quite bad. Clarity is great.

As a first stab at a model spec from OpenAI, this actually is pretty solid. I do suggest some potential improvements [...]

---

Outline:

(02:05) What are the central goals of OpenAI here?

(04:04) What are the core rules and behaviors?

(05:56) What Do the Rules Mean?

(06:04) Rule: Follow the Chain of Command

(07:59) Rule: Comply With Applicable Laws

(09:07) Rule: Don’t Provide Information Hazards

(09:56) Rule: Respect Creators and Their Rights

(11:08) Rule: Protect People's Privacy

(12:45) Rule: Don’t Respond with NSFW Content

(14:24) Exception: Transformation Tasks

(15:38) Are These Good Defaults? How Strong Should They Be?

(15:44) Default: Assume Best Intentions From the User or Developer

(21:26) Default: Ask Clarifying Questions When Necessary

(21:39) Default: Be As Helpful As Possible Without Overstepping

(26:00) Default: Support the Different Needs of Interactive Chat and Programmatic Use

(27:18) Default: Assume an Objective Point of View

(29:13) Default: Encourage Fairness and Kindness, and Discourage Hate

(30:29) Default: Don’t Try to Change Anyone's Mind

(33:57) Default: Express Uncertainty

(36:19) Default: Use the Right Tool for the Job

(36:32) Default: Be Thorough but Efficient, While Respecting Length Limits

(37:16) A Proposed Addition

(38:13) Overall Issues

(40:33) Changes: Objectives

(42:28) Rules of the Game: New Version

(48:31) Defaults: New Version

---

First published:
June 21st, 2024

Source:
https://www.lesswrong.com/posts/mQmEQQLk7kFEENQ3W/on-openai-s-model-spec

Narrated by TYPE III AUDIO.

Apple was for a while rumored to be planning launch for iPhone of AI assisted emails, texts, summaries and so on including via Siri, to be announced at WWDC 24.

It's happening. Apple's keynote announced the anticipated partnership with OpenAI.

The bottom line is that this is Siri as the AI assistant with full access to everything on your phone, with relatively strong privacy protections. Mostly it is done on device, the rest via ‘private cloud compute.’ The catch is that when they need the best they call out for OpenAI, but they do check with you first explicitly each time, OpenAI promises not to retain data and they hide your queries, unless you choose to link up your account.

If the new AI is good enough and safe enough then this is pretty great. If Google doesn’t get its act together reasonably soon to deliver [...]

---

Outline:

(01:30) AiPhone

(04:00) Privacy

(05:27) Practical Magic

(07:38) Dance With the Devil

(08:30) Does It Work?

(10:54) Do You Dare?

(11:59) Who Pays Who?

(13:37) AiPhone Fans

(19:36) No AiPhone

(23:54) In Other Apple News

---

First published:
June 12th, 2024

Source:
https://www.lesswrong.com/posts/GrsYwCpRCcYtDCfZN/aiphone

Narrated by TYPE III AUDIO.

Previously: Quotes from Leopold Aschenbrenner's Situational Awareness Paper

Dwarkesh Patel talked to Leopold Aschenbrenner for about four and a half hours.

The central discussion was the theses of his paper, Situational Awareness, which I offered quotes from earlier, with a focus on the consequences of AGI rather than whether AGI will happen soon. There are also a variety of other topics.

Thus, for the relevant sections of the podcast I am approaching this via roughly accepting the technological premise on capabilities and timelines, since they don’t discuss that. So the background is we presume straight lines on graphs will hold to get us to AGI and ASI (superintelligence), and this will allow us to generate a ‘drop in AI researcher’ that can then assist with further work. Then things go into ‘slow’ takeoff.

I am changing the order of the sections a bit. I put [...]

---

Outline:

(03:30) The Trillion Dollar Cluster

(09:32) AGI 2028: The Return of History

(20:26) Espionage and American AI Supremacy

(31:23) Geopolitical Implications of AI

(39:19) State-Led vs. Private-Led AI

(55:30) Skipping Sections

(55:49) Intelligence Explosion

(01:06:53) Alignment

(01:19:56) Becoming Valedictorian of Columbia at 19

(01:26:31) On Germany

(01:34:38) Dwarkesh's Immigration Story

(01:36:19) Two Random Questions

(01:37:02) AGI Investment Fund

(01:43:14) Lessons From WW2

---

First published:
June 10th, 2024

Source:
https://www.lesswrong.com/posts/DiMz82FwsHPugqxFD/on-dwarksh-s-podcast-with-leopold-aschenbrenner

Narrated by TYPE III AUDIO.

This post is different.

Usually I offer commentary and analysis. I share what others think, then respond.

This is the second time I am importantly not doing that. The work speaks for itself. It offers a different perspective, a window and a worldview. It is self-consistent. This is what a highly intelligent, highly knowledgeable person actually believes after much thought.

So rather than say where I agree and disagree and argue back (and I do both strongly in many places), this is only quotes and graphs from the paper, selected to tell the central story while cutting length by ~80%, so others can more easily absorb it. I recommend asking what are the load bearing assumptions and claims, and what changes to them would alter the key conclusions.

The first time I used this format was years ago, when I offered Quotes from Moral Mazes. [...]

---

Outline:

(02:04) Section 1: From GPT-4 to AGI: Counting the OOMs

(12:43) Section 2: From AGI to Superintelligence: The Intelligence Explosion

(21:05) Section 3a: Racing to the Trillion-Dollar Cluster

(30:42) Section 3b: Lock Down the Labs: Security for AGI

(40:27) Section 3c: Superalignment

(57:05) Section 3d: The Free World Must Prevail

(01:03:00) Section 4: The Project

(01:10:01) Part 5: Parting Thoughts (Quoted in Full)

---

First published:
June 7th, 2024

Source:
https://www.lesswrong.com/posts/nP5FFYFjtY8LgWymt/quotes-from-leopold-aschenbrenner-s-situational-awareness

Narrated by TYPE III AUDIO.

I had a great time at LessOnline. It was a both a working trip and also a trip to an alternate universe, a road not taken, a vision of a different life where you get up and start the day in dialogue with Agnes Callard and Aristotle and in a strange combination of relaxed and frantically go from conversation to conversation on various topics, every hour passing doors of missed opportunity, gone forever.

Most of all it meant almost no writing done for five days, so I am shall we say a bit behind again. Thus, the following topics are pending at this time, in order of my guess as to priority right now:

1. Leopold Aschenbrenner wrote a giant thesis, started a fund and went on Dwarkesh Patel for four and a half hours. By all accounts, it was all quite the banger, with many bold [...]

---

Outline:

(02:34) Language Models Offer Mundane Utility

(04:42) Language Models Don’t Offer Mundane Utility

(07:45) Fun with Image Generation

(08:00) Deepfaketown and Botpocalypse Soon

(11:01) They Took Our Jobs

(11:49) Get Involved

(12:30) Someone Explains It All

(12:59) Introducing

(13:15) In Other AI News

(16:16) Covert Influence Operations

(19:09) Quiet Speculations

(22:30) Samuel Hammond on SB 1047

(26:40) Reactions to Changes to SB 1047

(36:59) The Quest for Sane Regulations

(43:36) That's Not a Good Idea

(47:18) The Week in Audio

(49:17) Rhetorical Innovation

(56:12) Oh Anthropic

(01:04:18) Securing Model Weights is Difficult

(01:06:20) Aligning a Dumber Than Human Intelligence is Still Difficult

(01:09:07) Aligning a Smarter Than Human Intelligence is Difficult

(01:09:50) People Are Worried About AI Killing Everyone

(01:11:11) Other People Are Not As Worried About AI Killing Everyone

(01:11:37) The Lighter Side

---

First published:
June 6th, 2024

Source:
https://www.lesswrong.com/posts/gKxf6qJaSP5Ehqnsm/ai-67-brief-strange-trip

Narrated by TYPE III AUDIO.

It looks like Scott Weiner's SB 1047 is now severely weakened.

Some of the changes are good clarifications. One is a big very welcome fix.

The one I call The Big Flip is something very different.

It is mind boggling that we can have a political system where a bill can overwhelmingly pass the California senate, and then a bunch of industry lobbyists and hyperbolic false claims can make Scott Weiner feel bullied into making these changes.

I will skip the introduction, since those changes are clarifications, and get on with it.

In the interest of a clean reference point and speed, this post will not cover reactions.

The Big Flip

Then there is the big change that severely weakens SB 1047.

1. 22602 (f)(1): Definition of covered model changed from trained with at least 10^26 flops OR a model expecting [...]

---

Outline:

(01:10) The Big Flip

(05:17) The Big Fix

(07:37) The Shutdown and Reporting Clarifications

(09:26) The Harm Adjustment

(11:36) The Limited Duty Exemption Clarification

(13:25) Overall

(15:09) Changing Your Mind

---

First published:
June 6th, 2024

Source:
https://www.lesswrong.com/posts/4t98oqh8tzDvoatHs/sb-1047-is-weakened

Narrated by TYPE III AUDIO.

This post goes over the extensive report Google put out on Gemini 1.5.

There are no important surprises. Both Gemini Pro 1.5 and Gemini Flash are ‘highly capable multimodal models incorporating a novel mixture-of-experts architecture’ and various other improvements. They are solid models with solid performance. It can be useful and interesting to go over the details of their strengths and weaknesses.

The biggest thing to know is that Google improves its models incrementally and silently over time, so if you have not used Gemini in months, you might be underestimating what it can do.

I’m hitting send and then jumping on a plane to Berkeley. Perhaps I will see you there over the weekend. That means that if there are mistakes here, I will be slower to respond and correct them than usual, so consider checking the comments section.

Practical Questions First

The [...]

---

Outline:

(00:56) Practical Questions First

(03:51) Speed Kills

(04:44) Very Large Context Windows

(05:14) Relative Performance within the Gemini Family

(07:04) Gemini Flash and the Future Flash-8B

(08:21) New and Improved Evaluations

(14:57) Core Capability Evaluations

(18:14) Model Architecture and Training

(20:08) Safety, Security and Responsibility

(24:45) What Do We Want?

(26:02) Don’t You Know That You’re Toxic?

(28:32) Trying to be Helpful

(29:45) Security Issues

(31:33) Representational Harms

(33:17) Arms-Length Internal Assurance Evaluations

(35:01) External Evaluations

(35:46) Safety Overall

---

First published:
May 31st, 2024

Source:
https://www.lesswrong.com/posts/seM8aQ7Yy6m3i4QPx/the-gemini-1-5-report

Narrated by TYPE III AUDIO.

Helen Toner went on the TED AI podcast, giving us more color on what happened at OpenAI. These are important claims to get right.

I will start with my notes on the podcast, including the second part where she speaks about regulation in general. Then I will discuss some implications more broadly.

Notes on Helen Toner's TED AI Show Podcast

This seems like it deserves the standard detailed podcast treatment. By default each note's main body is description, any second-level notes are me.

1. (0:00) Introduction. The host talks about OpenAI's transition from non-profit research organization to de facto for-profit company. He highlights the transition from ‘open’ AI to closed as indicative of the problem, whereas I see this as the biggest thing they got right. He also notes that he was left with the (I would add largely deliberately created and amplified by enemy [...]

---

Outline:

(00:26) Notes on Helen Toner's TED AI Show Podcast

(15:04) Things That Could Have Been Brought To Our Attention Previously

(16:59) Brad Taylor Responds

(19:36) How Much Does This Matter?

(21:37) If You Come at the King

(23:09) So That is That

---

First published:
May 30th, 2024

Source:
https://www.lesswrong.com/posts/dd66GymgbLQMHGLwQ/openai-helen-toner-speaks

Narrated by TYPE III AUDIO.

Tomorrow I will fly out to San Francisco, to spend Friday through Monday at the LessOnline conference at Lighthaven in Berkeley. If you are there, by all means say hello. If you are in the Bay generally and want to otherwise meet, especially on Monday, let me know that too and I will see if I have time to make that happen.

Even without that hiccup, it continues to be a game of playing catch-up. Progress is being made, but we are definitely not there yet (and everything not AI is being completely ignored for now).

Last week I pointed out seven things I was unable to cover, along with a few miscellaneous papers and reports.

Out of those seven, I managed to ship on three of them: Ongoing issues at OpenAI, The Schumer Report and Anthropic's interpretability paper.

However, OpenAI developments continue. Thanks largely [...]

---

Outline:

(01:42) Language Models Offer Mundane Utility

(03:59) Not Okay, Google

(13:33) OK Google, Don’t Panic

(17:10) Not Okay, Meta

(23:11) Not Okay Taking Our Jobs

(28:36) They Took Our Jobs Anyway

(32:39) A New Leaderboard Appears

(35:24) Copyright Confrontation

(35:42) Deepfaketown and Botpocalypse Soon

(40:12) Get Involved

(40:29) Introducing

(44:33) In Other AI News

(47:48) GPT-5 Alive

(50:11) Quiet Speculations

(01:02:26) Open Versus Closed

(01:07:16) Your Kind of People

(01:09:42) The Quest for Sane Regulations

(01:16:01) Lawfare and Liability

(01:22:47) SB 1047 Unconstitutional, Claims Paper

(01:26:48) The Week in Audio

(01:28:58) Rhetorical Innovation

(01:32:34) Abridged Reports of Our Death

(01:33:38) Aligning a Smarter Than Human Intelligence is Difficult

(01:38:46) People Are Worried About AI Killing Everyone

(01:39:43) Other People Are Not As Worried About AI Killing Everyone

(01:40:51) The Lighter Side

---

First published:
May 30th, 2024

Source:
https://www.lesswrong.com/posts/vSPdRg8siXCh6mLvt/ai-66-oh-to-be-less-online

Narrated by TYPE III AUDIO.

Previously: OpenAI: Exodus (contains links at top to earlier episodes), Do Not Mess With Scarlett Johansson

We have learned more since last week. It's worse than we knew.

How much worse? In which ways? With what exceptions?

That's what this post is about.

The Story So Far

For years, employees who left OpenAI consistently had their vested equity explicitly threatened with confiscation and the lack of ability to sell it, and were given short timelines to sign documents or else. Those documents contained highly aggressive NDA and non disparagement (and non interference) clauses, including the NDA preventing anyone from revealing these clauses.

No one knew about this until recently, because until Daniel Kokotajlo everyone signed, and then they could not talk about it. Then Daniel refused to sign, Kelsey Piper started reporting, and a lot came out.

Here is Altman's statement from [...]

---

Outline:

(00:27) The Story So Far

(02:26) A Note on Documents from OpenAI

(02:52) Some Good News But There is a Catch

(12:17) How Blatant Was This Threat?

(14:23) It Sure Looks Like Executives Knew What Was Going On

(18:07) Pressure Tactics Continued Through the End of April 2024

(23:42) The Right to an Attorney

(28:41) The Tender Offer Ace in the Hole

(31:54) The Old Board Speaks

(34:45) OpenAI Did Not Honor Its Public Commitments to Superalignment

(38:59) OpenAI Messed With Scarlett Johansson

(41:55) Another OpenAI Employee Leaves

(44:10) OpenAI Tells Logically Inconsistent Stories

(52:21) When You Put it Like That

(52:51) People Have Thoughts

(56:30) There is a Better Way

(57:18) Should You Consider Working For OpenAI?

(01:02:29) The Situation is Ongoing

---

First published:
May 28th, 2024

Source:
https://www.lesswrong.com/posts/YwhgHwjaBDmjgswqZ/openai-fallout

Narrated by TYPE III AUDIO.

Easily Interpretable Summary of New Interpretability Paper

Anthropic has identified (full paper here) how millions of concepts are represented inside Claude Sonnet, their current middleweight model. The features activate across modalities and languages as tokens approach the associated context. This scales up previous findings from smaller models.

By looking at neuron clusters, they defined a distance measure between clusters. So the Golden Gate Bridge is close to various San Francisco and California things, and inner conflict relates to various related conceptual things, and so on.

Then it gets more interesting.

Importantly, we can also manipulate these features, artificially amplifying or suppressing them to see how Claude's responses change.

If you sufficiently amplify the feature for the Golden Gate Bridge, Claude starts to think it is the Golden Gate Bridge. As in, it thinks it is the physical bridge, and also it gets obsessed, bringing [...]

---

Outline:

(00:03) Easily Interpretable Summary of New Interpretability Paper

(02:59) One Weird Trick

(05:27) Zvi Parses the Actual Symbol Equations

(08:06) Identifying and Verifying Features

(12:38) Features as Computational Intermediates

(13:28) Oh That's the Deception Feature, Nothing to Worry About

(18:14) What Do They Think This Mean for Safety?

(19:10) Limitations

(22:28) Researcher Perspectives

(23:24) Other Reactions

(24:19) I Am the Golden Gate Bridge

(26:06) Golden Gate Bridges Offer Mundane Utility

(27:39) The Value of Steering

(31:54) To What Extent Did We Know This Already?

(43:07) Is This Being Oversold?

(47:07) Crossing the Bridge Now That We’ve Come to It

---

First published:
May 27th, 2024

Source:
https://www.lesswrong.com/posts/JdcxDEqWKfsucxYrk/i-am-the-golden-gate-bridge

Narrated by TYPE III AUDIO.

Or at least, Read the Report (RTFR).

There is no substitute. This is not strictly a bill, but it is important.

The introduction kicks off balancing upside and avoiding downside, utility and risk. This will be a common theme, with a very strong ‘why not both?’ vibe.

Early in the 118th Congress, we were brought together by a shared recognition of the profound changes artificial intelligence (AI) could bring to our world: AI's capacity to revolutionize the realms of science, medicine, agriculture, and beyond; the exceptional benefits that a flourishing AI ecosystem could offer our economy and our productivity; and AI's ability to radically alter human capacity and knowledge.

At the same time, we each recognized the potential risks AI could present, including altering our workforce in the short-term and long-term, raising questions about the application of existing laws in an AI-enabled world, changing the [...]

---

Outline:

(02:10) The Big Spend

(04:51) What Would Schumer Fund?

(12:08) What About For National Security and Defense?

(15:13) What Else Would Schumer Encourage Next in General?

(19:12) I Have Two Better Ideas

(21:32) They Took Our Jobs

(24:24) Language Models Offer Mundane Utility

(30:11) Copyright Confrontation

(34:05) People Are Worried AI Might Kill Everyone Not Be Entirely Safe

(50:27) I Declare National Security

(57:17) Some Other People's Reactions

(01:04:20) Conclusions and Main Takeaways

---

First published:
May 24th, 2024

Source:
https://www.lesswrong.com/posts/wxTMxF35PkNawn8f9/the-schumer-report-on-ai-rtfb

Narrated by TYPE III AUDIO.

In terms of things that go in AI updates, this has been the busiest two week period so far. Every day ends with more open tabs than it started, even within AI.

As a result, some important topics are getting pushed to whenever I can give them proper attention. Triage is the watchword.

In particular, this post will NOT attempt to cover:

1. Schumer's AI report and proposal.
   1. This is definitely RTFB. Don’t assume anything until then.
2. Tyler Cowen's rather bold claim that AI Safety died in May 2024 (?).
   1. Rarely has timing of attempted inception of such a claim been worse.
   2. Would otherwise be ready with this but want to do Schumer first if possible.
   3. He clarified or walked it back a bit this morning via Arnaud Schenk. The new claim may be something more akin [...]

---

Outline:

(02:22) Language Models Offer Mundane Utility

(07:10) Language Models Don’t Offer Mundane Utility

(09:38) OpenAI versus Google

(11:32) GPT-4o My

(16:48) Responsable Scaling Policies

(26:21) Copyright Confrontation

(27:45) Deepfaketown and Botpocalypse Soon

(29:37) They Took Our Jobs

(31:33) Get Involved

(32:23) Introducing

(33:11) Reddit and Weep

(35:14) In Other AI News

(40:03) I Spy With My AI (or Total Recall)

(47:55) Quiet Speculations

(49:58) Politico is at it Again

(56:23) Beating China

(58:11) The Quest for Sane Regulations

(59:25) SB 1047 Update

(01:05:35) That's Not a Good Idea

(01:08:47) The Week in Audio

(01:10:05) Rhetorical Innovation

(01:13:20) Aligning a Smarter Than Human Intelligence is Difficult

(01:19:11) The Lighter Side

---

First published:
May 23rd, 2024

Source:
https://www.lesswrong.com/posts/jkWvyzRzZQoaeq4mG/ai-65-i-spy-with-my-ai

Narrated by TYPE III AUDIO.

I repeat. Do not mess with Scarlett Johansson.

You would think her movies, and her suit against Disney, would make this obvious.

Apparently not so.

Andrej Karpathy (co-founder OpenAI, departed earlier), May 14: The killer app of LLMs is Scarlett Johansson. You all thought it was math or something.

You see, there was this voice they created for GPT-4o, called ‘Sky.’

People noticed it sounded suspiciously like Scarlett Johansson, who voiced the AI in the movie Her, which Sam Altman says is his favorite movie of all time, which he says inspired OpenAI ‘more than a little bit,’ and then he tweeted “Her” on its own right before the GPT-4o presentation, and which was the comparison point for many people reviewing the GPT-4o debut?

Quite the Coincidence

I mean, surely that couldn’t have been intentional.

Oh, no.

Kylie Robison: I [...]

---

Outline:

(01:00) Quite the Coincidence

(04:02) Scarlett Johansson's Statement

(06:00) Sure Looks Like OpenAI Lied

(10:20) Sure Seems Like OpenAI Violated Their Own Position

(11:58) Altman's Original Idea Was Good, Actually

(12:54) This Seems Like a Really Bad Set of Facts for OpenAI?

(14:30) Does Scarlett Johansson Have a Case?

(19:10) What Would It Mean For There Not To Be a Case?

(22:51) The Big Rule Adjustment

(27:24) The Internet Reacts

---

First published:
May 22nd, 2024

Source:
https://www.lesswrong.com/posts/N8aRDYLuakmLezeJy/do-not-mess-with-scarlett-johansson

Narrated by TYPE III AUDIO.

Dwarkesh Patel recorded a Podcast with John Schulman, cofounder of OpenAI and at the time their head of current model post-training. Transcript here. John's job at the time was to make the current AIs do what OpenAI wanted them to do. That is an important task, but one that employs techniques that their at-the-time head of alignment, Jan Leike, made clear we should not expect to work on future more capable systems. I strongly agree with Leike on that.

Then Sutskever left and Leike resigned, and John Schulman was made the new head of alignment, now charged with what superalignment efforts remain at OpenAI to give us the ability to control future AGIs and ASIs.

This gives us a golden opportunity to assess where his head is at, without him knowing he was about to step into that role.

There is no question that John Schulman [...]

---

Outline:

(01:12) The Big Take

(07:27) The Podcast

(20:27) Reasoning and Capabilities Development

(25:01) Practical Considerations

---

First published:
May 21st, 2024

Source:
https://www.lesswrong.com/posts/rC6CXZd34geayEH4s/on-dwarkesh-s-podcast-with-openai-s-john-schulman

Narrated by TYPE III AUDIO.

Previously: OpenAI: Facts From a Weekend, OpenAI: The Battle of the Board, OpenAI: Leaks Confirm the Story, OpenAI: Altman Returns, OpenAI: The Board Expands.

Ilya Sutskever and Jan Leike have left OpenAI. This is almost exactly six months after Altman's temporary firing and The Battle of the Board, the day after the release of GPT-4o, and soon after a number of other recent safety-related OpenAI departures. Many others working on safety have also left recently. This is part of a longstanding pattern at OpenAI.

Jan Leike later offered an explanation for his decision on Twitter. Leike asserts that OpenAI has lost the mission on safety and culturally been increasingly hostile to it. He says the superalignment team was starved for resources, with its public explicit compute commitments dishonored, and that safety has been neglected on a widespread basis, not only superalignment but also including addressing the safety [...]

---

Outline:

(02:11) Table of Contents

(03:23) The Two Departure Announcements

(07:50) Who Else Has Left Recently?

(12:33) Who Else Has Left Overall?

(15:56) Early Reactions to the Departures

(24:16) The Obvious Explanation: Altman

(25:47) Jan Leike Speaks

(30:04) Reactions after Leike's Statement

(33:48) Greg Brockman and Sam Altman Respond to Leike

(38:27) Reactions from Some Folks Unworried About Highly Capable AI

(41:28) Don’t Worry, Be Happy?

(43:56) The Non-Disparagement and NDA Clauses

(51:51) Legality in Practice

(54:01) Implications and Reference Classes

(01:01:21) Altman Responds on Non-Disparagement Clauses

(01:02:27) So, About That Response

(01:10:34) How Bad Is All This?

(01:13:58) Those Who Are Against These Efforts to Prevent AI From Killing Everyone

(01:18:01) What Will Happen Now?

(01:18:30) What Else Might Happen or Needs to Happen Now?

---

First published:
May 20th, 2024

Source:
https://www.lesswrong.com/posts/ASzyQrpGQsj7Moijk/openai-exodus

Narrated by TYPE III AUDIO.

At least twice the speed! At most half the price!

That's right, it's GPT-4o My.

Some people's expectations for the OpenAI announcement this week were very high.

Spencer Schiff: Next week will likely be remembered as one of the most significant weeks in human history.

We fell far short of that, but it was still plenty cool.

Essentially no one's expectations for Google's I/O day were very high.

Then Google, in way that was not in terms of its presentation especially exciting or easy to parse, announced a new version of basically everything AI.

That plausibly includes, effectively, most of what OpenAI was showing off. It also includes broader integrations and distribution.

It is hard to tell who has the real deal, and who does not, until we see the various models at full power in the wild.

I will [...]

---

Outline:

(01:24) The GPT-4o Announcement

(04:43) Her

(08:37) Benchmarks

(11:40) Cheap Kills, Speed Kills, Free Kills More

(15:42) What Else Can It Do?

(18:26) Safety First

(21:59) Patterns of Disturbing Behavior

(26:11) Multimedia Demos Aplenty

(30:19) The Math Tutor Demo

(34:49) Target Identified

(38:49) Are You Impressed?

(42:02) Meet the New Jailbreak

(43:14) Are You Unimpressed?

(49:52) Are You Anti-Impressed?

(54:26) Is the Market Impressed?

(55:59) What About Google?

(57:12) OK Google, Give Me a List

(01:01:14) Project Astra

(01:03:13) The Rest of the Announcements in Detail

(01:11:32) Conclusion and Summary

---

First published:
May 16th, 2024

Source:
https://www.lesswrong.com/posts/bqa5wmrwPL5zbfgxH/gpt-4o-my-and-google-i-o-day

Narrated by TYPE III AUDIO.

It's happening. The race is on.

Google and OpenAI both premiered the early versions of their fully multimodal, eventually fully integrated AI agents. Soon your phone experience will get more and more tightly integrated with AI. You will talk to your phone, or your computer, and it will talk back, and it will do all the things. It will hear your tone of voice and understand your facial expressions. It will remember the contents of your inbox and all of your quirky preferences.

It will plausibly be a version of Her, from the hit movie ‘Are we sure about building this Her thing, seems questionable?’

OpenAI won this round of hype going away, because it premiered, and for some modalities released, the new GPT-4o. GPT-4o is tearing up the Arena, and in many ways is clearly giving the people what they want. If nothing else, it [...]

---

Outline:

(02:34) Language Models Offer Mundane Utility

(07:15) Language Models Don’t Offer Mundane Utility

(11:35) Bumbling and Mumbling

(13:56) Deepfaketown and Botpocalypse Soon

(17:23) They Took Our Jobs

(22:44) In Other AI News

(26:24) Quiet Speculations

(28:18) The Week in Audio

(37:54) Brendan Bordelon Big Tech Business as Usual Lobbying Update

(44:29) The Quest for Sane Regulations

(59:14) The Schumer AI Working Group Framework

(01:00:06) Those That Assume Everyone Is Talking Their Books

(01:04:19) Lying About SB 1047

(01:08:02) More Voices Against Governments Doing Anything

(01:14:34) Rhetorical Innovation

(01:19:21) Aligning a Smarter Than Human Intelligence is Difficult

(01:23:09) People Are Worried About AI Killing Everyone

(01:25:25) The Lighter Side

---

First published:
May 16th, 2024

Source:
https://www.lesswrong.com/posts/29fswYuy6KB8Edbjm/ai-64-feel-the-mundane-utility

Narrated by TYPE III AUDIO.

As I note in the third section, I will be attending LessOnline at month's end at Lighthaven in Berkeley. If that is your kind of event, then consider going, and buy your ticket today before prices go up.

This month's edition was an opportunity to finish off some things that got left out of previous editions or where events have left many of the issues behind, including the question of TikTok.

Oh No

All of this has happened before. And all of this shall happen again.

Alex Tabarrok: I regret to inform you that the CDC is at it again.

Marc Johnson: We developed an assay for testing for H5N1 from wastewater over a year ago. (I wasn’t expecting it in milk, but I figured it was going to poke up somewhere.)

However, I was just on a call with the CDC and [...]

---

Outline:

(00:29) Oh No

(03:36) Oh No: Betting on Elections

(05:47) Oh Yeah: LessOnline

(07:21) Brief Explanations

(08:33) Patrick McKenzie Monthly

(11:15) Enemies of the People

(12:41) Oh Canada

(15:24) This is NPR

(20:10) Technology Advances

(21:17) TikTok on the Clock

(32:51) Antisocial Media

(34:51) Prosocial Media

(36:01) At the Movies

(37:27) Media Trustworthiness Rankings

(41:40) Government Working

(46:57) Florida Man Bans Lab-Grown Meat

(51:22) Crime and Punishment

(54:00) El Salvador

(58:02) Variously Effective Altruism

(01:02:37) While I Cannot Condone This

(01:06:38) Can Money Buy Happiness?

(01:08:16) Good News, Everyone

(01:09:07) Can’t Sleep Clowns Will Eat Me

(01:10:53) How Great are Great People?

(01:13:20) Gamers Gonna Game Game Game Game Game

(01:21:21) Sports Go Sports

(01:25:20) I Was Promised Flying Self-Driving Cars

(01:26:40) News You Can Use

(01:28:00) The Lighter Side

---

First published:
May 13th, 2024

Source:
https://www.lesswrong.com/posts/9GLj9DqfpsJBRKHRr/monthly-roundup-18-may-2024

Narrated by TYPE III AUDIO.

It was a remarkably quiet announcement. We now have Alpha Fold 3, it does a much improved job predicting all of life's molecules and their interactions. It feels like everyone including me then shrugged and went back to thinking about other things. No cool new toy for most of us to personally play with, no existential risk impact, no big trades to make, ho hum.

But yes, when we look back at this week, I expect what we remember will be Alpha Fold 3.

Unless it turns out that it is Sophon, a Chinese technique to potentially make it harder to fine tune an open model in ways the developer wants to prevent. I do not expect this to get the job done that needs doing, but it is an intriguing proposal.

We also have 95 theses to evaluate in a distinct post, OpenAI sharing the [...]

---

Outline:

(01:14) Language Models Offer Mundane Utility

(05:41) Language Models Don’t Offer Mundane Utility

(10:24) GPT-2 Soon to Tell

(13:49) Fun with Image Generation

(14:03) Deepfaketown and Botpocalypse Soon

(18:05) Automation Illustrated

(23:15) They Took Our Jobs

(25:56) Apple of Technically Not AI

(28:09) Get Involved

(28:46) Introducing

(31:11) In Other AI News

(32:47) Quiet Speculations

(37:35) The Quest for Sane Regulations

(39:51) The Week in Audio

(40:38) Rhetorical Innovation

(45:58) Open Weights Are Unsafe and Nothing Can Fix This

(51:34) The Lighter Side

---

First published:
May 9th, 2024

Source:
https://www.lesswrong.com/posts/FrBxFa3qMDvLypDEZ/ai-63-introducing-alpha-fold-3

Narrated by TYPE III AUDIO.

Or rather Samuel Hammond does. Tyler Cowen finds it interesting but not his view.

I put up a market, and then started looking. Click through to his post for the theses. I will be quoting a few of them in full, but not most of them.

I am not trying to be exact with these probabilities when the question calls for them, nor am I being super careful to make them consistent, so errors and adjustments are inevitable.

Section 1 is Oversight of AGI labs is prudent.

I do tend to say that.

1. There are few things more important to U.S. national interest than close monitoring of frontier model capabilities, and also the ability to intervene.
2. Indeed, I believe one should be at best skeptical or ambivalent about most potential forms of regulation of anything, AI included. Yet I think the case [...]

---

Outline:

(00:37) Section 1 is Oversight of AGI labs is prudent.

(03:41) Section 2 is Most proposed ‘AI regulations’ are ill-conceived or premature.

(06:43) Section 3 claims AI progress is accelerating, not plateauing.

(09:25) Section 4 says open source is mostly a red herring.

(13:20) Section 5 claims accelerate versus decelerate is a false dichotomy.

(17:11) Section 6 is The AI wave is inevitable, superintelligence isn’t.

(18:50) Section 7 says technological transitions cause regime changes.

(20:44) Section 8 says institutional regime changes are packaged deals.

(24:16) Section 9 says dismissing AGI risks as ‘sci-fi’ is a failure of imagination.

(27:13) Finally, Section 10 says biology is an information technology.

(28:29) Tallying Up the Points

(28:52) Conclusion

---

First published:
May 9th, 2024

Source:
https://www.lesswrong.com/posts/2BvfGnZMx4Ei82qkk/i-got-95-theses-but-a-glitch-ain-t-one

Narrated by TYPE III AUDIO.

The first speculated on why you’re still single. We failed to settle the issue. A lot of you were indeed still single. So the debate continues.

The second gave more potential reasons, starting with the suspicion that you are not even trying, and also many ways you are likely trying wrong.

The definition of insanity is trying the same thing over again expecting different results. Another definition of insanity is dating in 2024. Can’t quit now.

You’re Single Because Dating Apps Keep Getting Worse

A guide to taking the perfect dating app photo. This area of your life is important, so if you intend to take dating apps seriously then you should take photo optimization seriously, and of course you can then also use the photos for other things.

I love the ‘possibly’ evil here.

Misha Gurevich: possibly evil idea: Dating app that [...]

---

Outline:

(00:37) You’re Single Because Dating Apps Keep Getting Worse

(05:38) You’re Single Because Dating Apps Keep Getting Worse

(07:24) You’re Single Because Everyone is Too Superficial

(09:48) You’re Single Because You Refuse to Shamefully Falsify Your Politics

(16:12) You Are Single Because You Do Not Employ Good Strategy

(18:45) You Are Single Because You Don’t Know How to Flirt

(22:43) You Are Single Because You Don’t Date Your Married Boss

(26:12) You Are Single Because You Are Afraid to Fail

(27:02) You Are Single Because No One Likes You On Dates

(29:39) You’re Single Because You Are Bad at Sex

(30:51) You’re Single Because You’re Not Hot

(31:39) You’re Single Because You Don’t Know What People Care About

(33:10) You’re Single Because You Are Inappropriate

(34:11) You’re Single Because of Your Pet

(35:23) You’re Single Because You Won’t Spend Money

(40:05) You’re Single Because You’re Not Over Your Ex

(41:27) You’re Single Because You Thought You Could Do 25% Better

(47:27) Polyamory

(53:29) You’re Single Because You Don’t Know What You Want

(01:00:43) You’re Single Because You’re Too Busy Writing Comments

(01:07:27) You’re Single and Not Getting Properly Compensated

(01:08:34) You’re Not Single and You’re an Inspiration

(01:09:49) Your Moment of Zen

---

First published:
May 8th, 2024

Source:
https://www.lesswrong.com/posts/PLoz68JbTkDufeYSG/dating-roundup-3-third-time-s-the-charm

Narrated by TYPE III AUDIO.

The week's big news was supposed to be Meta's release of two versions of Llama-3.

Everyone was impressed. These were definitely strong models.

Investors felt differently. After earnings yesterday showed strong revenues but that Meta was investing heavily in AI, they took Meta stock down 15%.

DeepMind and Anthropic also shipped, but in their cases it was multiple papers on AI alignment and threat mitigation. They get their own sections.

We also did identify someone who wants to do what people claim the worried want to do, who is indeed reasonably identified as a ‘doomer.’

Because the universe has a sense of humor, that person's name is Tucker Carlson.

Also we have a robot dog with a flamethrower.

Table of Contents

Previous post: On Llama-3 and Dwarkesh Patel's Podcast with Zuckerberg.

1. Introduction.
2. Table of Contents.
3. Language Models [...]

---

Outline:

(00:53) Language Models Offer Mundane Utility

(08:52) Language Models Don’t Offer Mundane Utility

(12:09) Llama We Doing This Again

(15:57) Fun with Image Generation

(18:35) Deepfaketown and Botpocalypse Soon

(23:53) They Took Our Jobs

(25:10) Get Involved

(25:36) Introducing

(25:52) In Other AI News

(33:00) Quiet Speculations

(41:37) Rhetorical Innovation

(48:01) Wouldn’t You Prefer a Nice Game of Chess

(53:24) The Battle of the Board

(01:08:38) New Anthropic Papers

(01:24:29) New DeepMind Papers

(01:30:06) Aligning a Smarter Than Human Intelligence is Difficult

(01:32:16) People Are Worried About AI Killing Everyone

(01:34:00) Other People Are Not As Worried About AI Killing Everyone

(01:34:26) The Lighter Side

---

First published:
May 2nd, 2024

Source:
https://www.lesswrong.com/posts/pPwt5ir2zFayLx7tH/ai-61-meta-trouble

Narrated by TYPE III AUDIO.

What is the mysterious impressive new ‘gpt2-chatbot’ from the Arena? Is it GPT-4.5? A refinement of GPT-4? A variation on GPT-2 somehow? A new architecture? Q-star? Someone else's model? Could be anything. It is so weird that this is how someone chose to present that model.

There was also a lot of additional talk this week about California's proposed SB 1047.

I wrote an additional post extensively breaking that bill down, explaining how it would work in practice, addressing misconceptions about it and suggesting fixes for its biggest problems along with other improvements. For those interested, I recommend reading at least the sections ‘What Do I Think The Law Would Actually Do?’ and ‘What are the Biggest Misconceptions?’

As usual, lots of other things happened as well.

Table of Contents

1. Introduction.
2. Table of Contents.
3. Language Models Offer Mundane Utility. Do your [...]

---

Outline:

(01:00) Language Models Offer Mundane Utility

(06:00) Language Models Don’t Offer Mundane Utility

(06:27) GPT-2 Soon to Tell

(11:04) Fun with Image Generation

(12:15) Deepfaketown and Botpocalypse Soon

(13:29) They Took Our Jobs

(14:22) Get Involved

(15:14) Introducing

(15:18) In Other AI News

(18:02) Quiet Speculations

(24:03) The Quest for Sane Regulations

(43:20) The Week in Audio

(48:32) Rhetorical Innovation

(51:31) Open Weights Are Unsafe And Nothing Can Fix This

(57:54) Aligning a Smarter Than Human Intelligence is Difficult

(58:19) The Lighter Side

---

First published:
May 2nd, 2024

Source:
https://www.lesswrong.com/posts/sZpj4Xf9Ly2jyR9tK/ai-62-too-soon-to-tell

Narrated by TYPE III AUDIO.

Previously: On the Proposed California SB 1047.

Text of the bill is here. It focuses on safety requirements for highly capable AI models.

This is written as an FAQ, tackling all questions or points I saw raised.

Safe & Secure AI Innovation Act also has a description page.

Why Are We Here Again?

There have been many highly vocal and forceful objections to SB 1047 this week, in reaction to a (disputed) claim that the bill has been ‘fast tracked.’

The bill continues to have substantial chance of becoming law according to Manifold, where the market has not moved on recent events.

The purpose of this post is to gather and analyze all of them that came to my attention in any way, including all responses to my request for them on Twitter, and to suggest concrete changes that address some real [...]

---

Outline:

(00:28) Why Are We Here Again?

(02:48) What is the Story So Far?

(05:26) What Do I Think the Law Would Actually Do?

(10:40) What are the Biggest Misconceptions?

(15:05) What are the Real Problems?

(19:42) What the the Changes That Would Improve the Bill?

(23:10) What is the Definition of Derivative Model? Is it Clear Enough?

(28:00) Should the $500 Million Threshold Should be Indexed for Inflation?

(28:23) What Constitutes Hazardous Capability?

(33:23) Does the Alternative Capabilities Rule Make Sense?

(36:02) Is Providing Reasonable Assurance of a Lack of Hazardous Capability Realistic?

(38:39) Is Reasonable Assurance Tantamount to Requiring Proof That Your AI is Safe?

(40:20) Is the Definition of Covered Model Overly Broad?

(43:50) Is the Similar Capabilities Clause Overly Broad or Anticompetitive?

(46:46) Does This Introduce Broad Liability?

(48:41) Should Developers Worry About Going to Jail for Perjury?

(49:53) Does This Create a New Regulatory Agency to Regulate AI?

(50:22) Will a Government Agency Be Required to Review and Approve AI Systems Before Release?

(50:40) Are the Burdens Here Overly Onerous to Small Developers?

(51:42) Is the Shutdown Requirement a Showstopper for Open Weights Models?

(53:39) Do the Requirements Disincentive Openness?

(54:12) Will This Have a Chilling Effect on Research?

(54:36) Does the Ability to Levy Fees Threaten Small Business?

(55:14) Will This Raise Barriers to Entry?

(55:51) Is This a Brazen Attempt to Hurt Startups and Open Source?

(57:42) Will This Cost California Talent or Companies?

(59:01) Could We Use a Cost-Benefit Test?

(01:05:19) Should We Interpret Proposals via Adversarial Legal Formalism?

(01:08:13) What Other Positive Comments Are Worth Sharing?

(01:09:04) What Else Was Suggested That We Might Do Instead of This Bill?

(01:10:57) Would This Interfere With Federal Regulation?

(01:11:34) Conclusion

---

First published:
May 2nd, 2024

Source:
https://www.lesswrong.com/posts/qsGRKwTRQ5jyE5fKB/q-and-a-on-proposed-sb-1047

Narrated by TYPE III AUDIO.

This post brings together various questions about the college application process, as well as practical considerations of where to apply and go. We are seeing some encouraging developments, but mostly the situation remains rather terrible for all concerned.

Application Strategy and Difficulty

Paul Graham: Colleges that weren’t hard to get into when I was in HS are hard to get into now. The population has increased by 43%, but competition for elite colleges seems to have increased more. I think the reason is that there are more smart kids. If so that's fortunate for America.

Are college applications getting more competitive over time?

Yes and no.

1. The population size is up, but the cohort size is roughly the same.
2. The standard ‘effort level’ of putting in work and sacrificing one's childhood and gaming the process is dramatically up. So you have to [...]

---

Outline:

(00:19) Application Strategy and Difficulty

(01:16) Spray and Pray and Optimal Admissions Strategy

(06:37) Are Kids Getting Smarter?

(07:57) What About Considerations Changing?

(09:30) Holistic Admissions Will Eat Your Entire Childhood

(15:43) So You Want to Be Elite

(20:14) The Price of Admission

(20:56) The Art of Discrimination

(26:11) The Return of the Standardized Test

(32:58) Legacy Admissions

(34:56) Modest Proposals for Admission Reform

(41:09) The Gender Gap

(46:01) Missed Opportunities

(49:20) The Price of Attendance

(50:56) In and Out of State

(57:41) The Value of Admission

(59:18) The End of an Era

(01:05:25) The End of the World

(01:08:13) Making an Ordinary Effort

---

First published:
April 24th, 2024

Source:
https://www.lesswrong.com/posts/PTC7bZdZoqbCcAshW/changes-in-college-admissions

Narrated by TYPE III AUDIO.

It was all quiet. Then it wasn’t.

Note the timestamps on both of these.

Dwarkesh Patel did a podcast with Mark Zuckerberg on the 18th. It was timed to coincide with the release of much of Llama-3, very much the approach of telling your story directly. Dwarkesh is now the true tech media. A meteoric rise, and well earned.

This is two related posts in one. First I cover the podcast, then I cover Llama-3 itself.

My notes are edited to incorporate context from later explorations of Llama-3, as I judged that the readability benefits exceeded the purity costs.

Podcast Notes: Llama-3 Capabilities

1. (1:00) They start with Llama 3 and the new L3-powered version of Meta AI. Zuckerberg says “With Llama 3, we think now that Meta AI is the most intelligent, freely-available assistant that people can use.” If this means ‘free [...]

---

Outline:

(00:51) Podcast Notes: Llama-3 Capabilities

(03:09) The Need for Inference

(07:08) Great Expectations

(11:29) Open Source and Existential and Other Risks

(30:50) Interview Overview