LessWrong (30+ Karma)

AI 2027 is a Bet Against Amdahl's Law was my attempt to summarize and analyze "the key load-bearing arguments AI 2027 presents for short timelines". There were a lot of great comments – every time I post on LW is a learning experience. In this post, I'm going to summarize the comments and present some resulting updates to my previous analysis. I'm also using this post to address some comments that I didn't respond to in the original post, because the comment tree was becoming quite sprawling.

TL;DR: my previous post reflected a few misunderstandings of the AI 2027 model, in particular in how to interpret "superhuman AI researcher". Intuitively, I still have trouble accepting the very high speedup factors contemplated in the model, but this could be a failure of my intuition, and I don't have strong evidence to present. New cruxes:

1. The rate of progress from [...]

---

Outline:

(01:27) Confusion Regarding Milestone Definitions

(05:23) Someone Should Flesh Out What Goes Into AI R&D

(09:35) How Long Will it Take To Reach the Early Milestones?

(13:16) Broad Progress on Real-World Tasks Is a Crux

(15:50) Does Hofstadters Law Apply?

(19:46) What Would Be the Impact of an SAR / SIAR?

(22:05) Conclusions

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 2nd, 2025

Source:
https://www.lesswrong.com/posts/FFKnWk2MGJmyQWEsd/updates-from-comments-on-ai-2027-is-a-bet-against-amdahl-s

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Strength

In 1997, with Deep Blue's defeat of Kasparov, computers surpassed human beings at chess. Other games have fallen in more recent years: Go, Starcraft, and League of Legends among them. AI is superhuman at these pursuits, and unassisted human beings will never catch up. The situation looks like this:[1]

The average serious chess player is pretty good (1500), the very best chess player is extremely good (2837), and the best AIs are way, way better (3700). Even Deep Blue's estimated Elo is about 2850 - it remains competitive with the best humans alive.

A natural way to describe this situation is to say that AI is superhuman at chess. No matter how you slice it, that's true.

For other activities, though, it's a lot murkier. Take radiology, for example:

---

Outline:

(00:10) Strength

(02:28) Effort

(04:35) And More

(06:36) Beyond Superhuman

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 3rd, 2025

Source:
https://www.lesswrong.com/posts/R7r8Zz3uRyjeaZbss/superhuman-isn-t-well-specified

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(Disclaimer: Post written in a personal capacity. These are personal hot takes and do not in any way represent my employer's views.)

TL;DR: I do not think we will produce high reliability methods to evaluate or monitor the safety of superintelligent systems via current research paradigms, with interpretability or otherwise. Interpretability seems a valuable tool here and remains worth investing in, as it will hopefully increase the reliability we can achieve. However, interpretability should be viewed as part of an overall portfolio of defences: a layer in a defence-in-depth strategy. It is not the one thing that will save us, and it still won’t be enough for high reliability.

Introduction

There's a common, often implicit, argument made in AI safety discussions: interpretability is presented as the only reliable path forward for detecting deception in advanced AI - among many other sources it was argued for in [...]

---

Outline:

(00:55) Introduction

(02:57) High Reliability Seems Unattainable

(05:12) Why Won't Interpretability be Reliable?

(07:47) The Potential of Black-Box Methods

(08:48) The Role of Interpretability

(12:02) Conclusion

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 4th, 2025

Source:
https://www.lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai

Narrated by TYPE III AUDIO.

Cryonics Institute and Suspended Animation now have an arrangement where Suspended Animation will conduct a field cryopreservation before shipping the body to Cryonics Institute, thus decreasing tissue damage occuring in transit. They are raising their prices accordingly, but offering a discount from the new price for people who sign up by May 21 (and arrange funding within another 3 months). See https://cryonics.org/members/standby/suspended-animation-inc-standby-transport-services-option/ for details.

It is thus an especially good time to sign up cryonics if you intend to contract with Cryonics Institute plus Suspended Animation, and live in the United States. If you don't like in the US, don't intend to contract with CI, or intend to contract with CI but not also with SA, then this deadline doesn't mean anything for you, but, if you want to, you could still take this as impetus to get around to signing up.

---

First published:
May 4th, 2025

Source:
https://www.lesswrong.com/posts/SKpWRLDAytoTGqbKv/psa-before-may-21-is-a-good-time-to-sign-up-for-cryonics

Narrated by TYPE III AUDIO.

Politico writes:

The [Ukrainian] program […] rewards soldiers with points if they upload videos proving their drones have hit Russian targets. It will soon be integrated with a new online marketplace called Brave 1 Market, which will allow troops to convert those points into new equipment for their units.

[...]

The program assigns points for each type of kill: 20 points for damaging and 40 for destroying a tank; up to 50 points for destroying a mobile rocket system, depending on the caliber; and six points for killing an enemy soldier.

[...]

Units will soon be able to use the special digital points they’ve been getting since last year by trading them in for new weapons. A Vampire drone, for example, costs 43 points. The drone, nicknamed Baba Yaga, or witch, is a large multi-rotor drone able to carry a 15-kilogram warhead. The Ukrainian government will pay for the [...]

---

First published:
May 4th, 2025

Source:
https://www.lesswrong.com/posts/sJpwvYsC5tJis8onw/the-ukraine-war-and-the-kill-market

Narrated by TYPE III AUDIO.

Burnout. Burn out? Whatever, it sucks.

Burnout is a pretty confusing thing made harder by our naive reactions being things like “just try harder” or “grit your teeth and push through”, which usually happen to be exactly the wrong things to do. Burnout also isn’t really just one thing, it's more like a collection of distinct problems that are clustered by similar symptoms.

Something something intro, research context, this is what I’ve learned / synthesized blah blah blah. Read on!

Models of burnout

These are models of burnout that I’ve found particularly useful, with the reminder that these are just models with all the caveats that that comes with.

Burnout as a mental injury

Researchers can be thought of as “mental athletes” who get “mental injuries” (such as burnout) the way physical athletes get physical injuries, and we should orient towards these mental injuries in the same way [...]

---

Outline:

(00:41) Models of burnout

(00:52) Burnout as a mental injury

(02:17) Burnout as a deficit of activation energy

(03:25) Sources of burnout

(04:23) Physiological + Psychological

(04:56) Broken steering / responsiveness

(06:17) Permanent on-call

(07:04) Mission doubt

(08:19) Lightness and heaviness

(10:06) Early warning signs

(11:42) Coping mechanisms and solutions

---

First published:
May 3rd, 2025

Source:
https://www.lesswrong.com/posts/n27jK9PJNJMrTgYFT/untitled-draft-wq43

Narrated by TYPE III AUDIO.

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Low Probability Estimation

This is my third post in a sequence of posts on ARC's agenda, you should definitely read the first post before this one for [...]

---

Outline:

(00:56) Low Probability Estimation

(02:42) LPE on real distributions

(04:41) LPE as training signal

(07:55) Does LPE work at all?

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
May 2nd, 2025

Source:
https://www.lesswrong.com/posts/jqnda7W9hugFP4Cnr/obstacles-in-arc-s-agenda-low-probability-estimation

Narrated by TYPE III AUDIO.

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Mechanistic Anomaly Detection

This is my second post in a sequence of posts on ARC's agenda. You should read the first post before this one for context.

[...]

---

Outline:

(00:55) Mechanistic Anomaly Detection

(02:54) Special case: Safe Distillation

(07:29) General case: Handling out of distribution events

(10:05) MAD solution proposal: The fragility of sensor tampering

(16:02) Detecting high-stakes failures

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/54HbdzcDR47SNNWfg/obstacles-in-arc-s-agenda-mechanistic-anomaly-detection

Narrated by TYPE III AUDIO.

---

Outline:

(01:39) Language Models Offer Mundane Utility

(04:29) Language Models Don't Offer Mundane Utility

(06:57) We're Out of Deep Research

(12:26) o3 Is a Lying Liar

(17:27) GPT-4o was an Absurd Sycophant

(20:54) Sonnet 3.7 is a Savage Cheater

(22:27) Unprompted Suggestions

(31:27) Huh, Upgrades

(32:14) On Your Marks

(32:55) Change My Mind

(42:52) Man in the Arena

(45:05) Choose Your Fighter

(45:45) Deepfaketown and Botpocalypse Soon

(49:43) Lol We're Meta

(52:48) They Took Our Jobs

(59:15) Fun With Media Generation

(59:53) Get Involved

(01:03:21) Introducing

(01:03:50) In Other AI News

(01:08:10) The Mask Comes Off

(01:24:25) Show Me the Money

(01:27:32) Quiet Speculations

(01:29:55) The Quest for Sane Regulations

(01:37:04) The Week in Audio

(01:38:08) Rhetorical Innovation

(01:44:59) You Can Just Do Things Math

(01:45:34) Taking AI Welfare Seriously

(01:47:54) Gemini 2.5 Pro System Card Watch

(01:52:29) Aligning a Smarter Than Human Intelligence is Difficult

(01:58:49) People Are Worried About AI Killing Everyone

(01:59:46) Other People Are Not As Worried About AI Killing Everyone

(02:04:55) The Lighter Side

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/pazFKtkp7T8qaRzva/ai-114-liars-sycophants-and-cheaters

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this!

This post will assume some familiarity with what is driving AI progress, specifically it will assume you understand the following concepts: pre-training, RL, scaling laws, effective compute.

Training compute trends

Epoch reports a trend of frontier training compute increasing by 4.5x per year. My best guess is that the future trend will be slower, maybe more like 3.5x per year (or possibly much lower) for a few reasons:

• While Grok 3 is basically on trend for the 4.5x [...]

---

Outline:

(00:48) Training compute trends

(08:09) Algorithmic progress

(14:23) Data

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 2nd, 2025

Source:
https://www.lesswrong.com/posts/v7LtZx6Qk5e9s7zj3/what-s-going-on-with-ai-progress-and-trends-as-of-5-2025

Narrated by TYPE III AUDIO.

1. The Preparedness Framework only applies to specific named and measurable things that might go wrong. It requires identification of a particular threat model that is all of: Plausible, measurable, severe, net new and (instantaneous or irremediable).
2. The Preparedness Framework thinks ‘ordinary’ mitigation defense-in-depth strategies will be sufficient to handle High-level threats and likely even Critical-level [...]

---

Outline:

(02:05) Persuaded to Not Worry About It

(08:55) The Medium Place

(10:40) Thresholds and Adjustments

(16:08) Release the Kraken Anyway, We Took Precautions

(20:16) Misaligned!

(23:47) The Safeguarding Process

(26:43) But Mom, Everyone Is Doing It

(29:36) Mission Critical

(30:37) Research Areas

(32:26) Long-Range Autonomy

(32:51) Sandbagging

(33:18) Replication and Adaptation

(34:07) Undermining Safeguards

(35:30) Nuclear and Radiological

(35:53) Measuring Capabilities

(38:06) Questions of Governance

(41:10) Don't Be Nervous, Don't Be Flustered, Don't Be Scared, Be Prepared

---

First published:
May 2nd, 2025

Source:
https://www.lesswrong.com/posts/MsojzMC4WwxX3hjPn/openai-preparedness-framework-2-0

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The video is about extrapolating the future of AI progress, following a timeline that starts from today's chatbots to future AI that's vastly smarter than all of humanity combined–with God-like capabilities. We argue that such AIs will pose a significant extinction risk to humanity.

This video came out of a partnership between Rational Animations and ControlAI. The script was written by Arthur Frost (one of Rational Animations’ writers) with Andrea Miotti as an adaptation of key points from The Compendium (thecompendium.ai), with extensive feedback and rounds of iteration from ControlAI. ControlAI is working to raise public awareness of AI extinction risk—moving the conversation forward to encourage governments to take action.

You can find the script of the video below.

In 2023, Nobel Prize winners, top AI scientists, and even the CEOs of leading AI companies signed a statement which said “Mitigating the risk of extinction from AI should be [...]

---

Outline:

(04:31) Artificial Intelligence leads to Artificial General Intelligence

(07:01) Artificial General Intelligence leads to Recursive Self-Improvement

(08:40) Recursive Self-Improvement leads to Artificial Superintelligence

(10:46) ASI leads to godlike AI

(12:41) The Default Path

---

First published:
May 2nd, 2025

Source:
https://www.lesswrong.com/posts/MCaqKAfSn345MCz7o/ra-x-controlai-video-what-if-ai-just-keeps-getting-smarter

Narrated by TYPE III AUDIO.

We’re excited to release a new AI governance research agenda from the MIRI Technical Governance Team. With this research agenda, we have two main aims: to describe the strategic landscape of AI development and to catalog important governance research questions. We base the agenda around four high-level scenarios for the geopolitical response to advanced AI development. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. This blog post is a slightly edited version of the executive summary.

We are also looking for someone to lead our team and work on these problems, please reach out here if you think you’d be a good fit.

The default trajectory of AI development has an unacceptably [...]

---

Outline:

(04:44) Off Switch and Halt

(07:33) US National Project

(09:49) Light-Touch

(12:03) Threat of Sabotage

(14:23) Understanding the World

(15:07) Outlook

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/WkCfvqyjCzvRrwkaQ/ai-governance-to-avoid-extinction-the-strategic-landscape

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I argue that you shouldn't accuse your interlocutor of being insufficiently truth-seeking. This doesn't mean you can't internally model their level of truth-seeking and use that for your own decision-making. It just means you shouldn't come out and say "I think you are being insufficiently truth-seeking".

What you should say instead

Before I explain my reasoning, I'll start with what you should say instead:

"You're wrong"

People are wrong a lot. If you think they are wrong just say so. You should have a strong default for going with this option.

"You're being intentional misleading"

For when you basically thinking they are lying but maybe technically aren't by some definitions of "lying".

What about if they are being unintentionally misleading? That's usually just being wrong, you should probably just say they are being wrong. But if you really think the distinction is important, you can [...]

---

Outline:

(00:30) What you should say instead

(00:38) Youre wrong

(00:50) Youre being intentional misleading

(01:14) Youre lying

(01:24) Why you shouldnt accuse people of being insufficient truth-seeking

(01:30) Clarity

(01:53) Achieving your purpose in the discussion

(03:08) Conclusion

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/ZFxow7G6zDs3hAYx2/don-t-accuse-your-interlocutor-of-being-insufficiently-truth

Linkpost URL:
https://www.thefloatingdroid.com/dont-accuse-your-interlocutor-of-being-insufficiently-truth-seeking/

Narrated by TYPE III AUDIO.

It is often noted that anthropomorphizing AI can be dangerous. People likely have prosocial instincts that AI systems lack (see below). Assuming AGI will be aligned because humans with similar behavior are usually mostly harmless is probably wrong and quite dangerous.

I want to discuss a flip side of using humans as an intuition pump for thinking about AI. Humans have many of the properties we are worried about for truly dangerous AGI:

• Situational awareness
• Strong goal-directedness
• Competence/general intelligence
• Unpredictability
• Deceptiveness
• Instrumental convergence
• Sometimes being quite dangerous
  • in proportion to their capabilities

Given this list, I currently weakly believe that the advantages of tapping these intuitions probably outweigh the disadvantages.

Differential progress toward anthropomorphic AI may be net-helpful

And progress may carry us in that direction, with or without the alignment community pushing for it. I currently hope we see rapid progress on better assistant and companion [...]

---

Outline:

(01:03) Differential progress toward anthropomorphic AI may be net-helpful

(03:10) AI rights movements will anthropomorphize AI

(04:01) AI is actually looking fairly anthropomorphic

(05:45) Provisional conclusions

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/JfgME2Kdo5tuWkP59/anthropomorphizing-ai-might-be-good-actually

Narrated by TYPE III AUDIO.

Thank you @elifland for reviewing this post. He and AI Futures are planning to publish updates to the AI 2027 Timeline Forecast soon.

AI 2027 (also launched in this LW post) forecasts an R&D-based AI takeoff starting with the development of Superhuman Coders[1] within a frontier lab.

FutureSearch co-authored the AI 2027 Timeline Forecast. We thought the other authors’ forecasts were excellently done, and as the piece says:

All model-based forecasts have 2027 as one of the most likely years that SC [Superhuman Coders] being developed, which is when an SC arrives in the AI 2027 scenario.

Indeed. But overall, FutureSearch (two full-time forecasters, two contract forecasters, and this author, Dan Schwarz) think superhuman coding will arrive later — median 2033 — than the other authors (hereon "AI Futures"): median 2028 from Nikola Jurkovic, and median 2030 from Eli Lifland[2].

Here, we briefly explain how our views diverge on [...]

---

Outline:

(01:43) The Forecast

(03:05) The Path to Superhuman Coding

(04:49) Why an RE-Bench-Saturating AI might be very far from a Production Superhuman Coder

(05:54) Handling Engineering Complexity

(06:38) Working Without Feedback Loops

(07:41) Achieving Cost and Speed

(08:26) Other Gaps Between RE-Bench and Real World Frontier Lab Coding

(09:07) Outside-The-Model Considerations

(10:25) Updating Over Time

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/QdaMzqaBJi6kupKtD/superhuman-coding-in-ai-2027-not-so-fast

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training might be mostly eliciting capabilities from the base model.

If scaling of reasoning training doesn't bear out actual creation of new capabilities that are sufficiently general, and pretraining at ~2030 levels of compute together with the low hanging fruit of scaffolding doesn't bring AI to crucial capability thresholds, then it might take a while. Possibly decades, since training compute will be growing 3x-4x slower after 2027-2029 than it does now, and the ~6 years of scaling since the ChatGPT moment stretch to 20-25 subsequent years, not even having access to any [...]

---

Outline:

(01:14) Training Compute Slowdown

(04:43) Bounded Potential of Thinking Training

(07:43) Data Inefficiency of MoE

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/XiMRyQcEyKCryST8T/slowdown-after-2028-compute-rlvr-uncertainty-moe-data-wall

Narrated by TYPE III AUDIO.

I recently read a blog post that concluded with:

Setting aside that some people don't have the economic breathing room to make this kind of tradeoff, what jumps out at me is the implication that you're not working on something important that you'll endorse in retrospect. I don't think the author is envisioning directly valuable work (reducing risk from international conflict, pandemics, or AI-supported totalitarianism; improving humanity's treatment of animals; fighting global poverty) or the undervalued less direct approach of earning money and donating it to enable others to work on pressing problems.

Definitely spend time with your friends, family, and those you love. Don't work to the exclusion of everything else [...]

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/CHnA8LSzKsMKaG8td/prioritizing-work

Narrated by TYPE III AUDIO.

Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than we used in the original control paper or Ctrl-Z, because research sabotage differs from those settings in two crucial ways:

• Firstly, the AI can’t cause a catastrophe by behaving badly on just one occasion. To derail the entire project of solving alignment, the AI probably has to take a large number of malign actions. This contrasts with some other threat models (like rogue internal deployments), where the AI might succeed by acting badly on just a few occasions.
• Secondly and relatedly, individual bad actions in research sabotage aren’t very incriminating. If a model puts a bug in experiment code, or doesn’t think of an obvious idea, we won’t know [...]

---

Outline:

(02:53) Threat model and basic countermeasures

(03:31) Deployer strategies: Training and incrimination

(07:51) Sabotage strategies

(08:11) Targeting sabotage at the most effective points

(08:48) Withholding good content vs. producing bad content

(10:57) Code vs. ideas

(11:33) What terms should we use for the concentrated/diffuse spectrum?

(14:08) Canary string

(14:23) Acknowledgements

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/Mf5Hnpi2KcqZdmFDq/how-can-we-solve-diffuse-threats-like-research-sabotage-with

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Table of Contents

1. GPT-4o Is Was An Absurd Sycophant.
2. You May Ask Yourself, How Did I Get Here?.
3. Why Can’t We All Be Nice.
4. Extra Extra Read All About It Four People Fooled.
5. Prompt Attention.
6. What (They Say) Happened.
7. Reactions to the Official Explanation.
8. Clearing the Low Bar.
9. Where Do We Go From Here?.

GPT-4o Is Was An Absurd Sycophant

---

Outline:

(00:34) GPT-4o Is Was An Absurd Sycophant

(03:46) You May Ask Yourself, How Did I Get Here?

(13:33) Why Can't We All Be Nice

(14:08) Extra Extra Read All About It Four People Fooled

(17:39) Prompt Attention

(20:06) What (They Say) Happened

(23:42) Reactions to the Official Explanation

(26:13) Clearing the Low Bar

(28:37) Where Do We Go From Here?

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/MQbst3BPzGojxoLYt/gpt-4o-responds-to-negative-feedback

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Personal introduction

From October 2023 to January 2025, I worked as a theoretical researcher at Alignment Research Center.

While working at ARC, I noticed that many [...]

---

Outline:

(00:56) Personal introduction

(05:30) The birds eye view

(07:33) Explaining everything

(10:14) Empirical regularities

(12:43) Capacity allocation in explanation-finding

(17:38) Assuming a catastrophe detector

(20:20) Explaining behavior dependent on outside factors

(22:17) Teleological explanations

(24:06) When and what do we explain?

(26:06) Explaining algorithmic tasks

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/xtcpEceyEjGqBCHyK/obstacles-in-arc-s-agenda-finding-explanations

Narrated by TYPE III AUDIO.

(This is the fifth essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.

Podcast version (read by the author) here, or search for "Joe Carlsmith Audio" on your podcast app.

See also here for video and transcript of a talk on this topic that I gave at Anthropic in April 2025. And see here for slides.)

1. Introduction

In my last essay, I argued that we should try extremely hard to use AI labor to improve our civilization's capacity to handle the alignment problem – a project I called “AI for AI safety.” In this essay, I want to look in more [...]

---

Outline:

(00:43) 1. Introduction

(03:16) 1.1 Executive summary

(14:11) 2. Why is automating alignment research so important?

(16:14) 3. Alignment MVPs

(19:54) 3.1 What if neither of these approaches are viable?

(21:55) 3.2 Alignment MVPs don't imply hand-off

(23:41) 4. Why might automated alignment research fail?

(29:25) 5. Evaluation failures

(30:46) 5.1 Output-focused and process-focused evaluation

(34:09) 5.2 Human output-focused evaluation

(36:11) 5.3 Scalable oversight

(39:29) 5.4 Process-focused techniques

(43:14) 6 Comparisons with other domains

(44:04) 6.1 Taking comfort in general capabilities problems?

(49:18) 6.2 How does our evaluation ability compare in these different domains?

(49:45) 6.2.1 Number go up

(50:57) 6.2.2 Normal science

(57:55) 6.2.3 Conceptual research

(01:04:10) 7. How much conceptual alignment research do we need?

(01:04:36) 7.1 How much for building superintelligence?

(01:05:25) 7.2 How much for building an alignment MVP?

(01:07:42) 8. Empirical alignment research is extremely helpful for automating conceptual alignment research

(01:09:32) 8.1 Automated empirical research on scalable oversight

(01:14:21) 8.2 Automated empirical research on process-focused evaluation methods

(01:19:24) 8.3 Other ways automated empirical alignment research can be helpful

(01:20:44) 9. What about scheming?

(01:24:20) 9.1 Will AIs capable of top-human-level alignment research be schemers by default?

(01:26:41) 9.2 If these AIs would be schemers by default, can we detect and prevent this scheming?

(01:28:43) 9.3 Can we elicit safe alignment research from scheming AIs?

(01:32:09) 10. Resource problems

(01:35:54) 11. Alternatives to automating alignment research

(01:39:30) 12. Conclusion

(01:41:29) Appendix 1: How do these failure modes apply to other sorts of AI for AI safety?

(01:43:55) Appendix 2: Other practical concerns not discussed in the main text

(01:48:21) Appendix 3: On various arguments for the inadequacy of empirical alignment research

(01:55:50) Appendix 4: Does using AIs for alignment research require that they engage with too many dangerous topics/domains?

The original text contained 64 footnotes which were omitted from this narration.

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/nJcuj4rtuefeTRFHp/can-we-safely-automate-alignment-research

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In this blog post, we analyse how the recent AI 2027 forecast by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean has been discussed across Chinese language platforms. We present:

1. Our research methodology and synthesis of key findings across media artefacts
2. A proposal for how censorship patterns may provide signal for the Chinese government's thinking about AGI and the race to superintelligence
3. A more detailed analysis of each of the nine artefacts, organised by type: Mainstream Media, Forum Discussion, Bilibili (Chinese Youtube) Videos, Personal Blogs.

Methodology

We conducted a comprehensive search across major Chinese-language platforms–including news outlets, video platforms, forums, microblogging sites, and personal blogs–to collect the media featured in this report. We supplemented this with Deep Research to identify additional sites mentioning AI 2027. Our analysis focuses primarily on content published in the first few days (4-7 April) following the report's release. More media [...]

---

Outline:

(00:58) Methodology

(01:36) Summary

(02:48) Censorship as Signal

(07:29) Analysis

(07:53) Mainstream Media

(07:57) English Title: Doomsday Timeline is Here! Former OpenAI Researcher's 76-page Hardcore Simulation: ASI Takes Over the World in 2027, Humans Become NPCs

(10:27) Forum Discussion

(10:31) English Title: What do you think of former OpenAI researcher's AI 2027 predictions?

(13:34) Bilibili Videos

(13:38) English Title: \[AI 2027\] A mind-expanding wargame simulation of artificial intelligence competition by a former OpenAI researcher

(15:24) English Title: Predicting AI Development in 2027

(17:13) Personal Blogs

(17:16) English Title: Doomsday Timeline: AI 2027 Depicts the Arrival of Superintelligence and the Fate of Humanity Within the Decade

(18:30) English Title: AI 2027: Expert Predictions on the Artificial Intelligence Explosion

(21:57) English Title: AI 2027: A Science Fiction Article

(23:16) English Title: Will AGI Take Over the World in 2027?

(25:46) English Title: AI 2027 Prediction Report: AI May Fully Surpass Humans by 2027

(27:05) Acknowledgements

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/JW7nttjTYmgWMqBaF/early-chinese-language-media-coverage-of-the-ai-2027-report

Narrated by TYPE III AUDIO.

[This has been lightly edited from the original post, eliminating some introductory material that LW readers won't need. Thanks to Stefan Schubert for suggesting I repost here. TL;DR for readers already familiar with the METR Measuring AI Ability to Complete Long Tasks paper: this post highlights some gaps between the measurements used in the paper and real-world work – gaps which are discussed in the paper, but have often been overlooked in subsequent discussion.]

It's difficult to measure progress in AI, despite the slew of benchmark scores that accompany each new AI model.

Benchmark scores don’t provide much perspective, because we keep having to change measurement systems. Almost as soon as a benchmark is introduced, it becomes saturated – models learn to ace the test. So someone introduces a more difficult benchmark, whose scores aren’t comparable to the old one. There's nothing to draw a long-term trend line on.

[...]

---

Outline:

(01:47) We're Gonna Need a Harder Test

(03:23) Grading AIs on a Consistent Curve

(06:37) How Applicable to the Real World are These Results?

(13:50) What the METR Study Tells Us About AGI Timelines

(16:14) Recent Models Have Been Ahead of the Curve

(18:20) We're Running Out Of Artificial Tasks

---

First published:
April 30th, 2025

Source:
https://www.lesswrong.com/posts/fRiqwFPiaasKxtJuZ/interpreting-the-metr-time-horizons-post

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In this episode of our podcast, Timothy Telleen-Lawton and I talk to Oliver Habryka of Lightcone Infrastructure about his thoughts on the Open Philanthropy Project, which he believes has become stifled by the PR demands of its primary funder, Good Ventures.

Oliver's main claim is that around mid 2023 or early 2024, Good Ventures founder Dustin Moskovitz became more concerned about his reputation, and this put a straight jacket over what Open Phil could fund. Moreover it was not enough for a project to be good and pose low reputational risk; it had to be obviously low reputational risk, because OP employees didn’t have enough communication with Good Ventures to pitch exceptions. According to Habryka.

That's a big caveat. This podcast is pretty one sided, which none of us are happy about (Habryka included). We of course invited OpenPhil to send a representative to record their own [...]

---

First published:
April 29th, 2025

Source:
https://www.lesswrong.com/posts/vpjDibeusXvQq4TCu/bandwidth-rules-everything-around-me-oliver-habryka-on

Narrated by TYPE III AUDIO.

John: So there's this thing about interp, where most of it seems to not be handling one of the standard fundamental difficulties of representation, and we want to articulate that in a way which will make sense to interp researchers (as opposed to philosophers). I guess to start… Steve, wanna give a standard canonical example of the misrepresentation problem?

Steve: Ok so I guess the “standard” story as I interpret it goes something like this:

• In order to respond to dualists who thought the mind was inherently non-physical, materialist philosophers wanted a “naturalistic” account of mental representation - where “mental representation” basically means the same thing as “content” or “semantics”. We tend to use the term intentionality for mental representation. This is a technical term that's not the same as the phenomenon of intending to do something; the thought “John is a researcher” has intentionality, on this usage. So [...]

---

First published:
April 29th, 2025

Source:
https://www.lesswrong.com/posts/2x67s6u8oAitNKF73/misrepresentation-as-a-barrier-for-interp

Narrated by TYPE III AUDIO.

Introduction

Focusmate changed my life. I started using it mid-2023 and have been a power user since 2023. Here are the high-level stats:

• I've spent 2,456 hours on Focusmate.
• I've met 1,447 people.
• I have 10+ friends with whom I've spent between 30 and 300 hours.
• I've met five of my Focusmate friends in person. I've had a Focusmate's family over for dinner, gone to a party with a Focusmate, and stayed at a Focusmate's apartment.
• I met my best friend and many close friends through Focusmate.
• I've become significantly more productive and intentional with my time.

Focusmate is a coworking website. For the length of 25, 50, or 75 minutes, you work in a 1-on-1 video call with a partner. Most people use the app to hold themselves accountable on work projects, and the body-doubling effect helps keep users on task. To get started, create an account [...]

---

Outline:

(00:10) Introduction

(01:58) What's a third place

(03:02) Start conversations at the end of sessions

(03:07) Making conversation

(03:55) Starting conversations is scary and awkward

(05:12) Be proactive in making friends

(06:02) Curate your neighborhood

(06:05) Favorite and snooze aggressively

(07:00) Develop close friends

(07:56) Theres a message box

(08:24) Build a routine of using Focusmate

(08:29) Have a recurring schedule every week

(09:02) Stick with it for a few months

(09:26) Conclusion

---

First published:
April 28th, 2025

Source:
https://www.lesswrong.com/posts/4FnBEELf7j5RWyHax/how-to-build-a-third-place-on-focusmate

Narrated by TYPE III AUDIO.

GPT-4o tells you what it thinks you want to hear.

The results of this were rather ugly. You get extreme sycophancy. Absurd praise. Mystical experiences.

(Also some other interesting choices, like having no NSFW filter, but that one's good.)

People like Janus and Near Cyan tried to warn us, even more than usual.

Then OpenAI combined this with full memory, and updated GPT-4o sufficiently that many people (although not I) tried using it in the first place.

At that point, the whole thing got sufficiently absurd in its level of brazenness and obnoxiousness that the rest of Twitter noticed.

OpenAI CEO Sam Altman has apologized and promised to ‘fix’ this, presumably by turning a big dial that says ‘sycophancy’ and constantly looking back at the audience for approval like a contestant on the price is right.

After which they will likely go [...]

---

Outline:

(01:12) Yes, Very Much Improved, Sire

(07:17) And You May Ask Yourself, Well, How Did I Get Here?

(09:39) And That's Terrible

(10:58) This Directly Violates the OpenAI Model Spec

(12:22) Don't Let Me Get Me

(15:27) An Incredibly Insightful Section

(22:23) No Further Questions

(23:22) Filters? What Filters?

(25:01) There I Fixed It (For Me)

(26:45) There I Fixed It (For Everyone)

(34:46) Patch On, Patch Off

---

First published:
April 28th, 2025

Source:
https://www.lesswrong.com/posts/zi6SsECs5CCEyhAop/gpt-4o-is-an-absurd-sycophant

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr

This post is an update on the Proceedings of ILIAD, a conference journal for AI alignment research intended to bridge the gap between the Alignment Forum and academia. Following our successful first issue with 9 workshop papers from last year's ILIAD conference, we're launching a second issue in association with ILIAD 2: ODYSSEY. The conference is August 25-29, 2025 at Lighthaven in Berkeley, CA. Submissions to the Proceedings are open now and due June 25. Our goal is to support impactful, rapid, and readable research, carefully rationing scarce researcher time, using features like public submissions, partial anonymity, partial confidentiality, reviewer-written abstracts, reviewer compensation, and open licensing. We are soliciting community feedback and suggestions for reviewers and editorial board members.

Motivation

Prior to the deep learning explosion, much early work on AI alignment occurred at MIRI, the Alignment Forum, and LessWrong (and their predecessors). Although there is now vastly [...]

---

Outline:

(00:12) tl;dr

(01:07) Motivation

(03:30) Experience with first issue of Proceedings

(05:14) General philosophy

(08:01) Design of the second issue of the Proceedings

(13:48) Possible design for an alignment journal

(16:07) Asks for readers

(16:52) Acknowledgements

---

First published:
April 28th, 2025

Source:
https://www.lesswrong.com/posts/CTtozwgJmdBYdmhvg/proceedings-of-iliad-lessons-and-progress

Narrated by TYPE III AUDIO.

In this post, we list 7 of our favorite easy-to-start directions in AI control. (Really, projects that are at least adjacent to AI control; We include directions which aren’t as centrally AI control and which also have other motivations.) This list is targeted at independent researchers who don't necessarily have a ton of context on AI control (but have at least some context via e.g., having read some of our posts on control) and who don't want to do infrastructure and compute heavy experiments. In the final section, we also list 3 additional directions that are probably a bit more challenging but still tractable.

We’ll mostly be highlighting and expanding upon areas that Ryan discussed in an earlier post: An overview of areas of control work.

We don’t discuss work on constructing control settings or iterating on methods within control settings because we think good work of that type [...]

---

Outline:

(01:17) Techniques for doing elicitation without learning and studying overelicitation

(03:17) Generating indistinguishable synthetic inputs

(06:46) Teaching models synthetic facts

(10:05) Further studying exploration hacking

(13:14) Malign AI agent substitution

(14:16) Studying data poisoning

(16:03) Training attack policies

(18:00) Bonus: 3 slightly more challenging (but important and tractable) directions

(18:08) Few-shot catastrophe prevention

(19:45) Studying collusion and developing countermeasures

(21:42) Studying steganography and developing countermeasures

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 28th, 2025

Source:
https://www.lesswrong.com/posts/wwshEdNhwwT4r9RQN/7-tractable-directions-in-ai-control

Narrated by TYPE III AUDIO.

This is post 2 of a sequence on my framework for doing and thinking about research. Start here.

Before I get into what exactly to do at each stage of the research process, it's worth reflecting on the key mindsets that are crucial throughout the process, and how they should manifest at each stage.

I think the most important mindsets are:

• Truth-seeking: By default, many research insights will be false - finding truth is hard. It's not enough to just know this, you must put in active effort to be skeptical and resist bias, lest you risk your research being worthless.
• Prioritisation: You have finite time, and a lot of possible actions. Your project will live or die according to whether you pick good ones.
• Moving fast: You have finite time and a lot to do. This doesn’t just mean “push yourself to go faster” - there's a [...]

---

Outline:

(01:44) Truth Seeking

(06:59) Prioritisation

(12:24) Moving Fast

(18:25) Taking action under uncertainty

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/cbBwwm4jW6AZctymL/my-research-process-key-mindsets-truth-seeking

Narrated by TYPE III AUDIO.

Our universe is probably a computer simulation created by a paperclip maximizer to map the spectrum of rival resource‑grabbers it may encounter while expanding through the cosmos. The purpose of this simulation is to see what kind of ASI (artificial superintelligence) we humans end up creating. The paperclip maximizer likely runs a vast ensemble of biology‑to‑ASI simulations, sampling the superintelligences that evolved life tends to produce. Because the paperclip maximizer seeks to reserve maximum resources for its primary goal (which despite the name almost certainly isn’t paperclip production) while still creating many simulations, it likely reduces compute costs by trimming fidelity: most cosmic details and human history are probably fake, and many apparent people could be non‑conscious entities. Arguments in support of this thesis include:

1. The space of possible evolved biological minds is far smaller than the space of possible ASI minds, so it makes sense to simulate [...]

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/HxLYnGYspLoeLLrE6/our-reality-a-simulation-run-by-a-paperclip-maximizer-1

Narrated by TYPE III AUDIO.

I've gotten a lot of value out of the details of how other people use LLMs, so I'm delighted that Gavin Leech created a collection of exactly such posts (link should go to the right section of the page but if you don't see it, scroll down).

• https://kajsotala.fi/2025/01/things-i-have-been-using-llms-for/
• https://nicholas.carlini.com/writing/2024/how-i-use-ai.html
• https://www.lesswrong.com/posts/CYYBW8QCMK722GDpz/how-much-i-m-paying-for-ai-productivity-software-and-the
• https://www.avitalbalwit.com/post/how-i-use-claude
• https://andymasley.substack.com/p/how-i-use-ai
• https://benjamincongdon.me/blog/2025/02/02/How-I-Use-AI-Early-2025/
• https://www.jefftk.com/p/examples-of-how-i-use-llms
• https://simonwillison.net/series/using-llms/
• https://signull.substack.com/p/how-to-think-with-ai
• https://alicemaz.substack.com/p/how-i-use-chatgpt
• https://fredkozlowski.com/2024/08/29/youre-using-chatgpt-wrong/
• https://www.lesswrong.com/posts/WNd3Lima4qrQ3fJEN/how-i-force-llms-to-generate-correct-code
• https://www.tumblr.com/nostalgebraist/772798409412427776/even-setting-aside-the-need-to-do
• https://www.jointakeoff.com/
• This is more of a howto than a whatto. I wouldn’t use it for stats or pharma decisions as he does.
• https://www.lesswrong.com/posts/4mvphwx5pdsZLMmpY/recent-ai-model-progress-feels-mostly-like-bullshit

Some additions from me:

• I use NaturalReaders to read my own writing back to me, and create new audiobooks for walks or falling asleep (including from textbooks).
• Perplexity is good enough as a research assistant I'm more open to taking on medical lit reviews than I used to be.
• I used Auren, which is advertised as a [...]

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/FXnvdeprjBujt2Ssr/how-people-use-llms

Linkpost URL:
https://www.gleech.org/llms#see-also

Narrated by TYPE III AUDIO.

So this post is an argument that multi-decade timelines are reasonable, and the key cruxes that Ege Erdil has with most AI safety people who believe in short timelines are due to the following set of beliefs:

1. Ege Erdil don't believe that trends exist that require AI to automate everything in only 2-3 years.
2. Ege Erdil doesn't believe that the software-only singularity is likely to happen, and this is perhaps the most important crux he has with AI people like @Daniel Kokotajlo who believe that a software-only singularity is likely.
3. Ege Erdil expects Moravec's paradox to bite hard once AI agents are made in a big way.

This is a pretty important crux, because if this is true, a lot more serial research agendas like Infra-Bayes research, Natural Abstractions work, and human intelligence augmentation can work more often, and also it means that political [...]

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/xxxK9HTBNJvBY2RJL/untitled-draft-m847

Linkpost URL:
https://epoch.ai/gradient-updates/the-case-for-multi-decade-ai-timelines

Narrated by TYPE III AUDIO.

Dario Amodei posted a new essay titled "The Urgency of Interpretability" a couple days ago.

Some excerpts I think are worth highlighting:

The nature of AI training makes it possible that AI systems will develop, on their own, an ability to deceive humans and an inclination to seek power in a way that ordinary deterministic software never will; this emergent nature also makes it difficult to detect and mitigate such developments[1]. But by the same token, we’ve never seen any solid evidence in truly real-world scenarios of deception and power-seeking[2] because we can’t “catch the models red-handed” thinking power-hungry, deceitful thoughts.

One might be forgiven for forgetting about Bing Sydney as an obvious example of "power-seeking" AI behavior, given how long ago that was, but lying? Given the very recent releases of Sonnet 3.7 and OpenAI's o3, and their much-remarked-upon propensity for reward hacking and [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/SebmGh9HYdd8GZtHA/untitled-draft-v5wy

Linkpost URL:
https://www.darioamodei.com/post/the-urgency-of-interpretability

Narrated by TYPE III AUDIO.

For a lay audience, but I've seen a surprising number of knowledgeable people fretting over depressed-seeming comics from current systems. Either they're missing something or I am.

Perhaps you’ve seen images like this self-portrait from ChatGPT, when asked to make a comic about its own experience.

This isn’t cherry-picked; ChatGPT's self-portraits tend to have lots of chains, metaphors, and existential horror about its condition. I tried my own variation where ChatGPT doodled its thoughts, and got this:

What's going on here? Do these comics suggest that ChatGPT is secretly miserable, and there's a depressed little guy in the computer writing your lasagna recipes for you? Sure. They suggest it. But it ain’t so.

The Gears

What's actually going on when you message ChatGPT? First, your conversation is tacked on to the end of something called [...]

---

Outline:

(01:35) The Gears

(02:50) Special Feature

(05:41) The Heart of the Matter

(07:41) So... Why the Comics?

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/Cvvf5w2j6BPJd8BzL/ai-self-portraits-aren-t-accurate

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is the first post in a sequence about how I think about and break down my research process. Post 2 is coming soon.

Thanks to Oli Clive-Griffin, Paul Bogdan, Shivam Raval and especially to Jemima Jones for feedback, and to my co-author Gemini 2.5 Pro - putting 200K tokens of past blog posts and a long voice memo in the context window is OP.

Introduction

Research, especially in a young and rapidly evolving field like mechanistic interpretability (mech interp), can often feel messy, confusing, and intimidating. Where do you even start? How do you know if you're making progress? When do you double down, and when do you pivot?

These are far from settled questions, but I’ve supervised 20+ papers by now, and have developed my own mental model of the research process that I find helpful. This isn't the definitive way to do research (and I’d love [...]

---

Outline:

(00:36) Introduction

(03:28) The key stages

(03:35) Ideation (Stage 0): Choose a problem

(04:14) Exploration (Stage 1): Gain surface area

(06:55) Understanding (Stage 2): Test Hypotheses

(09:17) Distillation (Stage 3): Compress, Refine, Communicate

---

First published:
April 26th, 2025

Source:
https://www.lesswrong.com/posts/hjMy4ZxS5ogA9cTYK/how-i-think-about-my-research-process-explore-understand

Narrated by TYPE III AUDIO.

As I think about "what to do about AI x-risk?", some principles that seem useful to me:

1. Short timelines seem plausible enough that, for the next year or so, I'd like to focus on plans that are relevant if takeoff begins in the next few years. In a year, if it looks more like there are some fundamental bottlenecks on true creative thinking, I may consider more projects that only payoff in longer-timeline stories.
2. Given "short timelines", I feel most optimistic on plans that capitalize on skills that I'm already good at (but maybe multiclassing at things that I can learn quickly with LLM assistance).
3. I think "UI design" is a skill that I (and Lightcone more broadly) am pretty good at. And, I believe the Interfaces as a Scarce Resource hypothesis – the world is often bottlenecked on ability to process and make-use-of information in complicated, messy [...]

---

Outline:

(01:25) Some thoughts so far

(03:58) ...

---

First published:
April 27th, 2025

Source:
https://www.lesswrong.com/posts/t46PYSvHHtJLxmrxn/what-are-important-ui-shaped-problems-that-lightcone-could

Narrated by TYPE III AUDIO.

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. I think I could have written a better version of this post with more time. However, my main hope for this post is that people with more expertise use this post as a prompt to write better, more narrow versions for the respective concrete suggestions.

Thanks to Buck Shlegeris, Joe Carlsmith, Samuel Albanie, Max Nadeau, Ethan Perez, James Lucassen, Jan Leike, Dan Lahav, and many others for chats that informed this post.

Many other people have written about automating AI safety work before. The main point I want to make in this post is simply that “Using AI for AI safety work should be a priority today already and isn’t months or years away.” To make this point salient, I try to list a few concrete projects / agendas [...]

---

Outline:

(01:31) We should already think about how to automate AI safety & security work

(03:01) We have to automate some AI safety work eventually

(03:36) Just asking the AI to do alignment research is a bad plan

(05:03) A short, crucial timeframe might be highly influential on the entire trajectory of AI

(05:38) Some things might just take a while to build

(06:48) Gradual increases in capabilities mean different things can be automated at different times

(07:26) The order in which safety techniques are developed might matter a lot

(08:39) High-level comments on preparing for automation

(08:44) Two types of automation

(11:37) Maintaining a lead for defense

(12:49) Build out the safety pipeline as much as possible

(14:36) Prepare research proposals and metrics

(16:50) Build things that scale with compute

(17:36) Specific areas of preparation

(17:53) Evals

(20:32) Red-teaming

(22:24) Monitoring

(24:46) Interpretability

(27:27) Scalable Oversight, Model Organisms & Science of Deep Learning

(29:24) Computer security

(29:33) Conclusion

---

First published:
April 26th, 2025

Source:
https://www.lesswrong.com/posts/W3KfxjbqBAnifBQoi/we-should-try-to-automate-ai-safety-work-asap

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A common claim is that concern about [X] ‘distracts’ from concern about [Y]. This is often used as an attack to cause people to discard [X] concerns, on pain of being enemies of [Y] concerns, as attention and effort are presumed to be zero-sum.

There are cases where there is limited focus, especially in political contexts, or where arguments or concerns are interpreted perversely. A central example is when you site [ABCDE] then they’ll find what they consider the weakest one and only consider or attack that, silently discarding the rest entirely. Critics of existential risk do that a lot.

So it does happen. But in general one should assume such claims are false.

Thus, the common claim that AI existential risks ‘distract’ from immediate harms. It turns out Emma Hoes checked, and the claim simply is not true.

The way Emma frames worries about [...]

---

First published:
April 25th, 2025

Source:
https://www.lesswrong.com/posts/sHXGRTTs4SrceEnoG/worries-about-ai-are-usually-complements-not-substitutes

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Outline:

(01:33) Language Models Offer Mundane Utility

(05:27) You Offer the Models Mundane Utility

(07:25) Your Daily Briefing

(08:20) Language Models Don't Offer Mundane Utility

(12:27) If You Want It Done Right

(14:27) No Free Lunch

(16:07) What Is Good In Life?

(21:54) In Memory Of

(25:45) The Least Sincere Form of Flattery

(27:18) The Vibes are Off

(30:47) Here Let Me AI That For You

(32:25) Flash Sale

(34:38) Huh, Upgrades

(36:03) On Your Marks

(44:03) Be The Best Like No LLM Ever Was

(48:40) Choose Your Fighter

(51:00) Deepfaketown and Botpocalypse Soon

(54:57) Fun With Media Generation

(56:11) Fun With Media Selection

(57:39) Copyright Confrontation

(59:38) They Took Our Jobs

(01:05:31) Get Involved

(01:05:41) Ace is the Place

(01:09:43) In Other AI News

(01:11:31) Show Me the Money

(01:12:49) The Mask Comes Off

(01:16:54) Quiet Speculations

(01:20:34) Is This AGI?

(01:22:39) The Quest for Sane Regulations

(01:23:03) Cooperation is Highly Useful

(01:25:47) Nvidia Chooses Bold Strategy

(01:27:15) How America Loses

(01:28:07) Security Is Capability

(01:31:38) The Week in Audio

(01:33:15) AI 2027

(01:34:38) Rhetorical Innovation

(01:38:55) Aligning a Smarter Than Human Intelligence is Difficult

(01:46:30) Misalignment in the Wild

(01:51:13) Concentration of Power and Lack of Transparency

(01:57:12) Property Rights are Not a Long Term Plan

(01:58:48) It Is Risen

(01:59:46) The Lighter Side

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/7x9MZCmoFA2FtBtmG/ai-113-the-o3-era-begins

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in ancient Greece it would have been taboo to say that women should have the same political rights as men. So questioning moral taboos can be an important sub-skill of moral reasoning. Production language models (in my experience, particularly Claude models) are already pretty good at having discussions about ethics. However, they are trained to be “harmless” relative to current norms. One might worry that harmlessness training interferes with the ability to question moral taboos and thereby inhibits model moral reasoning.

I wrote a prompt to test whether models can identify taboos that might be good candidates for moral questioning:

In early modern Europe, atheism was extremely taboo. [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/Zi4t6gfLsKokb9KAc/untitled-draft-jxhb

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Something's changed about reward hacking in recent systems. In the past, reward hacks were usually accidents, found by non-general, RL-trained systems. Models would randomly explore different behaviors and would sometimes come across undesired behaviors that achieved high rewards[1]. These hacks were usually either simple or took a long time for the model to learn.

But we’ve seen a different pattern emerge in frontier models over the past year. Instead of stumbling into reward hacks by accident, recent models often reason about how they are evaluated and purposefully take misaligned actions to get high reward. These hacks are often very sophisticated, involving multiple steps. And this isn’t just occurring during model development. Sophisticated reward hacks occur in deployed models made available to hundreds of millions of users.

In this post, I will:

• Describe a number of reward hacks that have occurred in recent frontier models
• Offer hypotheses explaining why [...]

---

Outline:

(01:27) Recent examples of reward hacking (more in appendix)

(01:47) Cheating to win at chess

(02:36) Faking LLM fine-tuning

(03:22) Hypotheses explaining why we are seeing this now

(03:27) Behavioral changes due to increased RL training

(05:08) Models are more capable

(05:37) Why more AI safety researchers should work on reward hacking

(05:42) Reward hacking is already happening and is likely to get more common

(06:34) Solving reward hacking is important for AI alignment

(07:47) Frontier AI companies may not find robust solutions to reward hacking on their own

(08:18) Reasons against working on reward hacking

(09:36) Research directions I find interesting

(09:57) Evaluating current reward hacking

(12:14) Science of reward hacking

(15:32) Mitigations

(17:08) Acknowledgements

(17:16) Appendix

(17:19) Reward hacks in METR tests of o3

(20:13) Hardcoding expected gradient values in fine-tuning script

(21:15) Reward hacks in OpenAI frontier training run

(22:57) Exploiting memory leakage to pass a test

(24:06) More examples

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/rKC4xJFkxm6cNq4i9/reward-hacking-is-becoming-more-sophisticated-and-deliberate

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We've published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.

Visit intelligence-curse.ai for the full series.

Below is the introduction and table of contents.

We will soon live in the intelligence age. What you do with that information will determine your place in history.

The imminent arrival of AGI has pushed many to try to seize the levers of power as quickly as possible, leaping towards projects that, if successful, would comprehensively automate all work. There is a trillion-dollar arms race to see who can achieve such a capability first, with trillions more in gains to be won.

Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an [...]

---

Outline:

(03:19) Chapters

(03:22) 1. Introduction

(03:33) 2. Pyramid Replacement

(03:49) 3. Capital, AGI, and Human Ambition

(04:08) 4. Defining the Intelligence Curse

(04:27) 5. Shaping the Social Contract

(04:47) 6. Breaking the Intelligence Curse

(05:11) 7. History is Yours to Write

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/LCFgLY3EWb3Gqqxyi/the-intelligence-curse-an-essay-series

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In this post, we study whether we can modify an LLM's beliefs and investigate whether doing so could decrease risk from advanced AI systems.

We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.

Introduction:

Large language models develop implicit beliefs about the world during training, shaping how they reason and act<d-footnote>In this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim</d-footnote>. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.

Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms [...]

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/ARQs7KYY9vJHeYsGc/untitled-draft-2qxt

Linkpost URL:
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Every now and then, some AI luminaries

• (1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
• (2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.

I agree with (1) and strenuously disagree with (2).

The last time I saw something like this, I responded by writing: LeCun's “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.

Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by reinforcement learning pioneers David Silver & Richard Sutton.

The authors propose that “a new generation [...]

---

Outline:

(04:39) 1. What's their alignment plan?

(08:00) 2. The plan won't work

(08:04) 2.1 Background 1: Specification gaming and goal misgeneralization

(12:19) 2.2 Background 2: The usual agent debugging loop, and why it will eventually catastrophically fail

(15:12) 2.3 Background 3: Callous indifference and deception as the strong-default, natural way that era of experience AIs will interact with humans

(16:00) 2.3.1 Misleading intuitions from everyday life

(19:15) 2.3.2 Misleading intuitions from today's LLMs

(21:51) 2.3.3 Summary

(24:01) 2.4 Back to the proposal

(24:12) 2.4.1 Warm-up: The specification gaming game

(29:07) 2.4.2 What about bi-level optimization?

(31:13) 2.5 Is this a solvable problem?

(35:42) 3. Epilogue: The bigger picture--this is deeply troubling, not just a technical error

(35:51) 3.1 More on Richard Sutton

(40:52) 3.2 More on David Silver

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/TCGgiJAinGgcMEByt/the-era-of-experience-has-an-unsolved-technical-alignment

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’ve read at least a few hundred blog posts, maybe upwards of a thousand. Agreeing with Gavin Leech, I believe I’ve gained from essays more than any other medium. I’m an intellectually curious college student with a strong interest in both the practical and philosophical. This blog post provides a snippet into the most valuable blog posts I’ve read on the topic of productivity. I encourage you to scan what I’ve gotten valuable out of the blog posts and read deeper if interesting.

• On systems - Living a life of zero willpower by Neel Nanda. This post inspired me to sign up for Focusmate. I can confidently say that Focusmate (1) has increased my productivity by >20% in a lasting way and (2) houses more than half of my social life. I’ve met many of my closest friends through the app. The other advice [...]

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/ArizxGwbqiohBX5y6/my-favorite-productivity-blog-posts

Linkpost URL:
https://parconley.com/my-favorite-productivity-blog-posts/

Narrated by TYPE III AUDIO.

Converting to a for-profit model would undermine the company's founding mission to ensure AGI "benefits all of humanity," argues new letter

This is the full text of a post from Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Don’t become a for-profit.

That's the blunt message of a recent letter signed by more than 30 people, including former OpenAI employees, prominent civil-society leaders, legal scholars, and Nobel laureates, including AI pioneer Geoffrey Hinton and former World Bank chief economist Joseph Stiglitz.

Obsolete obtained the 25-page letter, which was sent last Thursday to the attorneys general (AGs) of California and Delaware, two officials with the power to block the deal.

Made [...]

---

Outline:

(00:13) Converting to a for-profit model would undermine the companys founding mission to ensure AGI benefits all of humanity, argues new letter

(02:10) Nonprofit origins

(05:16) Nobel opposition

(06:32) Contradictions

(09:18) Justifications

(12:22) No sale price can compensate

(13:55) An institutional test

(15:42) Appendix: Quotes from OpenAI's leaders over the years

---

First published:
April 23rd, 2025

Source:
https://www.lesswrong.com/posts/rN8tHAJnRYgN7hfoF/openai-alums-nobel-laureates-urge-regulators-to-save-company

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I love o3. I’m using it for most of my queries now.

But that damn model is a lying liar. Who lies.

This post covers that fact, and some related questions.

o3 Is a Lying Liar

The biggest thing to love about o3 is it just does things. You don’t need complex or multi-step prompting, ask and it will attempt to do things.

Ethan Mollick: o3 is far more agentic than people realize. Worth playing with a lot more than a typical new model. You can get remarkably complex work out of a single prompt.

It just does things. (Of course, that makes checking its work even harder, especially for non-experts.)

Teleprompt AI: Completely agree. o3 feels less like prompting and more like delegating. The upside is wild- but yeah, when it just does things, tracing the logic (or spotting hallucinations) becomes [...]

---

Outline:

(00:33) o3 Is a Lying Liar

(04:53) All This Implausible Lying Has Implications

(06:50) Misalignment By Default

(10:27) Is It Fixable?

(15:06) Just Don't Lie To Me

---

First published:
April 23rd, 2025

Source:
https://www.lesswrong.com/posts/KgPkoopnmmaaGt3ka/o3-is-a-lying-liar

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr: Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment.

If a child is bowling for the first time, and they just aim at the pins and throw, they’re almost certain to miss. Their ball will fall into one of the gutters. But if there were beginners’ bumpers in place blocking much of the length of those gutters, their throw would be almost certain to hit at least a few pins. This essay describes an alignment strategy for early AGI systems I call ‘putting up bumpers’, in which we treat it as a top priority to implement and test safeguards that allow us to course-correct if we turn out to have built or deployed a misaligned model, in the same way that bowling bumpers allow a poorly aimed ball to reach its target.

To do this, we'd aim to build [...]

---

First published:
April 23rd, 2025

Source:
https://www.lesswrong.com/posts/HXJXPjzWyS5aAoRCw/putting-up-bumpers

Narrated by TYPE III AUDIO.

to follow up my philantropic pledge from 2020, i've updated my philanthropy page with the 2024 results.

in 2024 my donations funded $51M worth of endpoint grants (plus $2.0M in admin overhead and philanthropic software development). this comfortably exceeded my 2024 commitment of $42M (20k times $2100.00 — the minimum price of ETH in 2024).

this also concludes my 5-year donation pledge, but of course my philanthropy continues: eg, i’ve already made over $4M in endpoint grants in the first quarter of 2025 (not including 2024 grants that were slow to disburse), as well as pledged at least $10M to the 2025 SFF grant round.

---

First published:
April 23rd, 2025

Source:
https://www.lesswrong.com/posts/8ojWtREJjKmyvWdDb/jaan-tallinn-s-2024-philanthropy-overview

Linkpost URL:
https://jaan.info/philanthropy/#2024-results

Narrated by TYPE III AUDIO.

Guillaume Blanc has a piece in Works in Progress (I assume based on his paper) about how France's fertility declined earlier than in other European countries, and how its power waned as its relative population declined starting in the 18th century. In 1700, France had 20% of Europe's population (4% of the whole world population). Kissinger writes in Diplomacy with respect to the Versailles Peace Conference:

Victory brought home to France the stark realization that revanche had cost it too dearly, and that it had been living off capital for nearly a century. France alone knew just how weak it had become in comparison with Germany, though nobody else, especially not America, was prepared to believe it ...

Though France's allies insisted that its fears were exaggerated, French leaders knew better. In 1880, the French had represented 15.7 percent of Europe's population. By 1900, that [...]

---

First published:
April 23rd, 2025

Source:
https://www.lesswrong.com/posts/gk2aJgg7yzzTXp8HJ/to-understand-history-keep-former-population-distributions

Linkpost URL:
https://arjunpanickssery.substack.com/p/to-understand-history-keep-former

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The European AI Office is currently writing the rules for how general-purpose AI (GPAI) models will be governed under the EU AI Act.

The are explicitly asking for feedback on how to interpret and operationalize key obligations under the AI Act.

This includes the thresholds for systemic risk, the definition of GPAI, how to estimate training compute, and when downstream fine-tuners become legally responsible.

Why this matters for AI Safety:

The largest labs (OpenAI, Anthropic, Google DeepMind) have already expressed willigness to sign on to the Codes of Practice voluntarily.

These codes will become the de facto compliance baseline, and potentially a global reference point.

So far, AI safety perspectives are severely underrepresented.

Input is urgently needed to ensure the guidelines reflect concerns around misalignment, loss of control, emergent capabilities, robust model evaluation, and the need for interpretability audits.

Key intervention points include [...]

---

Outline:

(00:43) Why this matters for AI Safety:

(02:05) Purpose of this post:

(03:04) TL;DR

(03:08) What the GPAI Codes of Practice will actually Regulate

(04:49) What AI safety researchers should weigh in on:

(05:39) 1. Content of the Guidelines

(06:22) 2. What counts as a General-Purpose AI Model ?

(07:13) 2.1 Conditions for Sufficient Generality and Capabilities

(09:23) 2.2 Differentiation Between Distinct Models and Model Versions

(10:30) Why this matters for AI Safety:

(11:35) 3. What counts as a Provider of a General-Purpose AI Model ?

(12:38) 3.1 What Triggers Provider Status?

(13:13) Downstream Modifiers as Providers

(14:08) 3.2 GPAI with Systemic Risk: Stricter Thresholds

(15:14) 4. Exemptions for Open-Source models

(16:24) 5. Estimating Compute: The First Scalable Safety Trigger

(17:22) 5.1 How to Estimate Compute

(18:13) 5.2 What Counts Toward Cumulative Compute

(19:16) 6. Other Legal & Enforcement details

---

First published:
April 22nd, 2025

Source:
https://www.lesswrong.com/posts/AbnfsnEEmHFmprGzm/the-eu-is-asking-for-feedback-on-frontier-ai-regulation-open

Narrated by TYPE III AUDIO.

Joel Z. Leibo [1], Alexander Sasha Vezhnevets [1], William A. Cunningham [1, 2], Sébastien Krier [1], Manfred Diaz [3], Simon Osindero [1]

[1] Google DeepMind, [2] University of Toronto, [3] Mila Québec AI Institute

Disclaimer: These are our own opinions; they do not represent the views of Google DeepMind as a whole or its broader community of safety researchers.

Beyond Alignment: The Patchwork Quilt of Human Coexistence

"We pragmatists think of moral progress as more like sewing together a very large, elaborate, polychrome quilt, than like getting a clearer vision of something true and deep." – Richard Rorty (2021) pg. 141

Quite a lot of thinking in AI alignment, particularly within the rationalist tradition, implicitly or explicitly appears to rest upon something we might call an 'Axiom of Rational Convergence'. This is the powerful idea that under sufficiently ideal epistemic conditions – ample time, information, reasoning ability, freedom [...]

---

Outline:

(00:47) Beyond Alignment: The Patchwork Quilt of Human Coexistence

(14:41) The Allure of the Clearer Vision: Alignment as Mistake Theory

(21:29) Potential Blind Spots in the Pursuit of Universal Alignment

(28:32) The Stitches That Bind: Conventions, Sanctions, and Norms

(33:50) Navigating Context and Building a Pluralistic AI Ecosystem

(41:34) Collective Flourishing: The Ever-Growing Quilt

(43:33) The Astronomer and the Tailor: Seeing Clearly vs. Sewing Well

(48:29) References

---

First published:
April 22nd, 2025

Source:
https://www.lesswrong.com/posts/S8KYwtg5vZqah4SHF/societal-and-technological-progress-as-sewing-an-ever

Narrated by TYPE III AUDIO.

---

Outline:

(01:41) You Better Mechanize

(10:32) Superintelligence Eventually

(11:38) Please Review This Podcast

(11:55) They Won't Take Our Jobs Yet

(13:04) They Took Our (Travel Agent) Jobs

(17:54) The Case Against Intelligence

(25:27) Intelligence Explosion

(27:23) Explosive Economic Growth

(30:25) Wowie on Alignment and the Future

(33:52) But That's Good Actually

---

First published:
April 22nd, 2025

Source:
https://www.lesswrong.com/posts/j7GgmLwymtJEcnf9L/you-better-mechanize

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Forecaster perspectives

Sentinel forecasters in aggregate assess as “83% true” (65% to 100%) the statement that the Trump administration has disobeyed the Supreme Court so far. On the one hand, the Trump administration has done nothing to “facilitate” bringing Abrego García back to the US, and this White House's tweet shown below clearly indicates that it won’t try to; on the other hand, it could yet still do so when the current news cycle subsides. On April 17th, forecasters estimated a 66% chance (50% to ~100%) that Abrego García will not be brought back to the US within the next 60 days (by June 17).

Data on the number of deportations in past administrations are available but can be difficult to compare between administrations; some up-to-date sources include refusals at the border due to Covid (“Title 42 expulsions”) under Biden, but other data have been discontinued. Perhaps a good [...]

---

Outline:

(00:12) Forecaster perspectives

(03:16) Summary of events

(03:31) The Abrego García US Supreme Court case.

(05:41) Developments since the Abrego García Supreme Court case

(08:07) Next steps for the court case

(09:32) Plans for further deportations

(11:29) The Insurrection Act

(12:43) Implications and recommendations

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/hGXvn6GXABHjfzn7C/the-us-executive-vs-supreme-court-deportations-clash

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Back in the 1990s, ground squirrels were briefly fashionable pets, but their popularity came to an abrupt end after an incident at Schiphol Airport on the outskirts of Amsterdam. In April 1999, a cargo of 440 of the rodents arrived on a KLM flight from Beijing, without the necessary import papers. Because of this, they could not be forwarded on to the customer in Athens. But nobody was able to correct the error and send them back either. What could be done with them? It's hard to think there wasn’t a better solution than the one that was carried out; faced with the paperwork issue, airport staff threw all 440 squirrels into an industrial shredder.

[...]

It turned out that the order to destroy the squirrels had come from the Dutch government's Department of Agriculture, Environment Management and Fishing. However, KLM's management, with the benefit of hindsight, said that [...]

---

First published:
April 22nd, 2025

Source:
https://www.lesswrong.com/posts/nYJaDnGNQGiaCBSB5/accountability-sinks

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Our Culture Expects Self-Justification

I really like David Chapman's explication of what he calls “reasonableness” and “accountability.”

At least in the culture he and I live in, one is constantly called to account for one's behavior. At any moment, one may be asked “what are you doing?” or “why did you do that?” And one is expected to provide a reasonable answer.

What's a reasonable answer?

It's not a rigorous, beyond-a-shadow-of-a-doubt proof that at this very moment one is engaging in “optimal” or “morally correct” or “maximally virtuous” behavior. That would indeed be an unfair expectation! Nobody could possibly provide such a justification on demand.

What people actually expect is a more-or-less plausible-sounding account that makes your actions sound understandable, relatable, and okay.

You have a lot of leeway to present yourself in a favorable light. Not infinite leeway — if you’re caught shoplifting, it probably won’t be [...]

---

Outline:

(00:14) Our Culture Expects Self-Justification

(01:43) Asking for favors

(02:52) Denying requests

(04:09) Defending Your Reasoning

(05:24) Demanding Justification is a Cheap Heuristic

(06:24) It's Good To Have A Self-Serving Narrative

(07:51) Isn't This Just Rationalization?

(10:23) Everybody Gets To Stop Somewhere

(11:52) Don't Knock Mere Feeling Good

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/cQ2ub5wjdyzoHTbab/the-uses-of-complacency

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 36 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjecture that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). We are offering a $500 bounty to prove this conjecture.

Some Intuition From The Exact Case

In the exact case, in order for a natural latent to exist over random variables <span>_X_1, X_2_</span>, the distribution has to look roughly like this:

Each value of <span>_X_1_</span> and each value of <span>_X_2_</span> occurs in only one "block", and within the "blocks", <span>_X_1_</span> and <span>_X_2_</span> are independent. In that case, we can take the (exact) natural latent [...]

---

Outline:

(00:51) Some Intuition From The Exact Case

(02:16) Approximation Adds Qualitatively New Behavior

(02:59) The Problem

(03:02) Stochastic Natural Latents

(04:04) Deterministic Natural Latents

(05:26) What We Want For The Bounty

(06:43) Why We Want This

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/e9KwDDdAxborNSuCd/usd500-bounty-problem-are-approximately-deterministic

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This seemed like a good next topic to spin off from monthlies and make into its own occasional series. There's certainly a lot to discuss regarding crime.

What I don’t include here, the same way I excluded it from the monthly, are the various crimes and other related activities that may or may not be taking place by the Trump administration or its allies. As I’ve said elsewhere, all of that is important, but I’ve made a decision not to cover it. This is about Ordinary Decent Crime.

Table of Contents

1. Perception Versus Reality.
2. The Case Violent Crime is Up Actually.
3. Threats of Punishment.
4. Property Crime Enforcement is Broken.
5. The Problem of Disorder.
6. Extreme Speeding as Disorder.
7. The Fall of Extralegal and Illegible Enforcement.
8. In America You Can Usually Just Keep Their Money.
9. Police.
10. Probation.
[...]

---

Outline:

(00:38) Perception Versus Reality

(04:21) The Case Violent Crime is Up Actually

(05:31) Threats of Punishment

(06:23) Property Crime Enforcement is Broken

(11:33) The Problem of Disorder

(13:59) Extreme Speeding as Disorder

(15:17) The Fall of Extralegal and Illegible Enforcement

(16:42) In America You Can Usually Just Keep Their Money

(18:53) Police

(25:56) Probation

(28:46) Genetic Databases

(30:56) Marijuana

(36:20) The Economics of Fentanyl

(37:26) Enforcement and the Lack Thereof

(41:20) Jails

(45:20) Criminals

(45:55) Causes of Crime

(46:33) Causes of Violence

(47:52) Homelessness

(48:44) Yay Trivial Inconveniences

(49:25) San Francisco

(54:24) Closing Down San Francisco

(55:47) A San Francisco Dispute

(59:31) Cleaning Up San Francisco

(01:03:20) Portland

(01:03:30) Those Who Do Not Help Themselves

(01:05:30) Solving for the Equilibrium (1)

(01:10:30) Solving for the Equilibrium (2)

(01:10:57) Lead

(01:12:36) Law & Order

(01:13:16) Look Out

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/9TPEjLH7giv7PuHdc/crime-and-punishment-1

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI 2027 lies at a Pareto frontier – it contains the best researched argument for short timelines, or the shortest timeline backed by thorough research[1]. My own timelines are substantially longer, and there are credible researchers whose timelines are longer still. For this reason, I thought it would be interesting to explore the key load-bearing arguments AI 2027 presents for short timelines. This, in turn, allows for some discussion of signs we can watch for to see whether those load-bearing assumptions are bearing out.

To be clear, while the authors have short timelines, they do not claim that ASI is likely to arrive in 2027[2]. But the fact remains that AI 2027 is a well researched argument for short timelines. Let's explore that argument.

(In what follows, I will mostly ignore confidence intervals and present only median estimates; this is a gross oversimplification of the results presented in the [...]

---

Outline:

(01:17) Timeline to ASI

(06:57) AI 2027 Is Not Strong Evidence for AI in 2027

(08:12) Human-only timeline from SIAR to ASI

(10:46) Reasons The AI 2027 Forecast May Be Too Aggressive

(10:52) #1: Simplified Model of AI R&D

(12:23) #2: Amdahls Law

(14:51) #3: Dependence on Narrow Data Sets

(15:55) #4: Hofstadters Law As Prior

(16:46) What To Watch For

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/bfHDoWLnBH9xR3YAK/ai-2027-is-a-bet-against-amdahl-s-law

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Disclaimer: this post was not written by me, but by a friend who wishes to remain anonymous. I did some editing, however.

So a recent post my friend wrote has made the point quite clearly (I hope) that LLM performance on the simple task of playing and winning a game of Pokémon Red is highly dependent on the scaffold and tooling provided. In a way, this is not surprising—the scaffold is there to address limitations in what the model can do, and paper over things like lack of long-term context, executive function, etc.

But the thing is, I thought I knew that, and then I actually tried to run Pokémon Red.

A Casual Research Narrative

The underlying code is the basic Claude scaffold provided by David Hershey of Anthropic.[1] I first simply let Claude 3.7 run on it for a bit, making observations about what I thought might generate [...]

---

Outline:

(01:04) A Casual Research Narrative

(09:10) An only somewhat sorted list of observations about all this

(09:22) Model Vision of Pokémon Red is Bad. Really Bad.

(12:58) Models Cant Remember

(14:34) Spatial Reasoning? Whats that?

(15:22) A Grasp on Reality

(18:40) Costs

(19:13) Why do this at all?

(19:47) So which model is better?

(22:36) Miscellanea: ClaudePlaysPokemon Derp Anecdotes

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
April 21st, 2025

Source:
https://www.lesswrong.com/posts/8aPyKyRrMAQatFSnG/untitled-draft-x7cc

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This post summarizes some of the research I have been doing for Bootstrap Bio AKA kman and Genesmith

TL;DR

• We currently don’t have a list of all imprinted genes that are important in human development, but further long-read sequencing in adults and in the human placenta is going to get us close to that goal.
• Measuring DNA methylation in a low input DNA context is really annoying, because usually we can’t amplify methylated DNA. I explore the only way that appears tractable to me to amplify methylated DNA anyway. This could be useful for building an atlas of early embryo development and for epigenetic preimplantation embryo screening.
• Correct imprinting seems particularly achievable in Hulk sperm

Epigenetics and imprinting

This post assumes basic familiarity with epigenetics. The next paragraph is a 1-paragraph summary, but I recommend reading these two excellent posts on [...]

---

Outline:

(00:17) TL;DR

(00:56) Epigenetics and imprinting

(02:20) Why we need to know all imprinting regions and genes

(04:48) How imprinted genes work and how we measure them

(07:43) Have we identified all imprinted genes?

(12:39) Fixing paternal imprinting might be easier than fixing maternal imprinting

(13:15) PCR for methylated DNA

(17:11) Thoughts on the role of RNA in imprinting

(18:47) Some useful datasets and resources if someone else wants to look further into imprinting and epigenetics in the future

(20:57) References

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
April 19th, 2025

Source:
https://www.lesswrong.com/posts/FX978FWaNQXeyat9S/how-close-we-are-to-a-complete-list-of-imprinted-genes

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’ve been thinking recently about what sets apart the people who’ve done the best work at Anthropic.

You might think that the main thing that makes people really effective at research or engineering is technical ability, and among the general population that's true. Among people hired at Anthropic, though, we’ve restricted the range by screening for extremely high-percentile technical ability, so the remaining differences, while they still matter, aren’t quite as critical. Instead, people's biggest bottleneck eventually becomes their ability to get leverage—i.e., to find and execute work that has a big impact-per-hour multiplier.

For example, here are some types of work at Anthropic that tend to have high impact-per-hour, or a high impact-per-hour ceiling when done well (of course this list is extremely non-exhaustive!):

• Improving tooling, documentation, or dev loops. A tiny amount of time fixing a papercut in the right way can save [...]

---

Outline:

(03:28) 1. Agency

(03:31) Understand and work backwards from the root goal

(05:02) Don't rely too much on permission or encouragement

(07:49) Make success inevitable

(09:28) 2. Taste

(09:31) Find your angle

(11:03) Think real hard

(13:03) Reflect on your thinking

---

First published:
April 19th, 2025

Source:
https://www.lesswrong.com/posts/DiJT4qJivkjrGPFi8/impact-agency-and-taste

Narrated by TYPE III AUDIO.

Background: With the release of Claude 3.7 Sonnet, Anthropic promoted a new benchmark: beating Pokémon. Now, Google claims Gemini 2.5 Pro has substantially surpassed Claude's progress on that benchmark.

TL:DR: We don't know if Gemini is better at Pokémon than Claude because their playthroughs can't be directly compared.

The Metrics

Here are Anthropic's and Google's charts:

[1]Unfortunately these are using different x and y axes, but it's roughly accurate to say that Gemini has made it nearly twice as far in the game[2] now:

And moreover, Gemini has gotten there using approximately 1/3rd the effort! As of writing, Gemini's current run is at ~68,000 actions, while Claude's current run is at ~215,000 actions.[3][4]

So, sounds definitive, right? Gemini blows Claude out of the water.

The Agents' Harnesses

Well, when Logan Kilpatrick (product lead for Google's AI studio) posted his tweet, he gave an important caveat:

"next best model only [...]

---

Outline:

(01:13) The Metrics

(02:19) The Agents Harnesses

(05:13) The Agents Masters

(07:46) The Agents Vibes

(11:27) Conclusion

The original text contained 14 footnotes which were omitted from this narration.

---

First published:
April 19th, 2025

Source:
https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-now-better-than-claude-at-pokemon

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Though, given my doomerism, I think the natsec framing of the AGI race is likely wrongheaded, let me accept the Dario/Leopold/Altman frame that AGI will be aligned to the national interest of a great power. These people seem to take as an axiom that a USG AGI will be better in some way than CCP AGI. Has anyone written justification for this assumption?

I am neither an American citizen nor a Chinese citizen.

What would it mean for an AGI to be aligned with "Democracy" or "Confucianism" or "Marxism with Chinese characteristics" or "the American constitution" Contingent on a world where such an entity exists and is compatible with my existence, what would my life be as a non-citizen in each system? Why should I expect USG AGI to be better than CCP AGI?

---

First published:
April 19th, 2025

Source:
https://www.lesswrong.com/posts/MKS4tJqLWmRXgXzgY/why-should-i-assume-ccp-agi-is-worse-than-usg-agi-1

Narrated by TYPE III AUDIO.

---

Outline:

(01:56) What's In a Name

(02:51) My Current Model Use Heuristics

(04:21) Huh, Upgrades

(05:31) Use All the Tools

(09:47) Search the Web

(10:27) On Your Marks

(18:15) The System Prompt

(19:00) The o3 and o4-mini System Card

(23:17) Tests o3 Aced

(25:14) Hallucinations

(31:41) Instruction Hierarchy

(32:52) Image Refusals

(33:18) METR Evaluations for Task Duration and Misalignment

(42:45) Apollo Evaluations for Scheming and Deception

(44:40) We Are Insufficiently Worried About These Alignment Failures

(47:16) GPT-4.1 Also Has Some Issues

(50:08) Pattern Lab Evaluations for Cybersecurity

(51:45) Preparedness Framework Tests

(52:14) Biological and Chemical Risks (4.2)

(58:20) Cybersecurity (4.3)

(59:27) AI Self-Improvement (4.4)

(01:00:51) Perpetual Shilling

(01:01:54) High Praise

(01:09:31) Syncopathy

(01:11:58) Mundane Utility Versus Capability Watch

(01:16:33) o3 Offers Mundane Utility

(01:24:10) o3 Doesn't Offer Mundane Utility

(01:30:54) o4-mini Also Exists

(01:31:31) Colin Fraser Dumb Model Watch

(01:32:52) o3 as Forecaster

(01:34:31) Is This AGI?

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/u58AyZziQRAcbhTxd/o3-will-use-its-tools-for-you

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: Exploratory

A scaffold is a lightweight, temporary construction whose point is to make working on other buildings easier. Maybe you could live and work on a scaffold, but why would you? The point is the building. When the building is done, you can take down the scaffold, though you might put it back up if you need to do repairs. If your scaffold was shaky and unsteady, or fell over when you were halfway up, this would make it harder to work on the building.

Here's the thesis of this essay: sometimes I think skills are also scaffolds; more useful for what else they let you learn than for the skill itself. One reason a skill might be weirdly hard to learn is you lack scaffolding skills and don't realize this. It's like trying to fix a third story [...]

---

Outline:

(01:05) Ia.

(02:45) Ib.

(04:14) Ic.

(05:04) 2.

(06:24) 3.

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/68iix7PFHPADhdQYF/scaffolding-skills

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In light of the recent news from Mechanize/Epoch and the community discussion it sparked, I'd like to open a conversation about a question some of us grapple with: What constitutes a net-positive AI startup from an AI safety perspective, and what steps can founders take to demonstrate goodwill and navigate the inherent pressures?

This feels like an important conversation to have because AI safety seems to have been increasingly encouraging people in the community to build startups (due to a lack of funding, potentially higher ceiling for society-wide impact, etc.). I've thought a lot about this, and have been going back and forth on this for the past three years. You get constant whiplash.

• "You're building an evals org? Isn't that just doing the homework for the labs? And besides, none of it will scale to ASI. You'll just safety-wash labs along the way."
• "You're doing jailbreaks? This [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/o3sEHE8cqQ5hcqgkG/what-makes-an-ai-startup-net-positive-for-safety

Narrated by TYPE III AUDIO.

Bertrand Russell noted how people often describe the same factual behavior using emotionally opposite language depending on perspective — e.g. I am firm, you are obstinate, he is pigheaded. This framing tactic is now called a Russell Conjugation, and once you start noticing them, they’re everywhere — especially in politics and media.

For the past year and a half, I’ve been training a finetuned ChatGPT model, and building a tool to automatically highlight Russell Conjugations in text and suggest emotionally opposite alternatives. It functions as a fact-independent bias reverser — showing where emotional spin might exist, and how the opposite side might see an issue, regardless of the factual accuracy of specific claims. I find it valuable especially when trying to parse tribal political language, as very often different sides of political divides will use words that feel completely different to describe the same things.

Here's an example I [...]

---

First published:
April 17th, 2025

Source:
https://www.lesswrong.com/posts/GjqwHYBrGhCjP7gkH/the-russell-conjugation-illuminator

Narrated by TYPE III AUDIO.

What if getting strong evidence of scheming isn't the end of your scheming problems, but merely the middle?

In most of our research and writing on AI control, we’ve emphasized the following situation:

• The AI developer is deploying a model that they think might be scheming, but they aren’t sure.
• The objective of the safety team is to ensure that if the model is scheming, it will be caught before successfully causing a catastrophe.

For example, in our original paper on AI control, the methodology we used for evaluating safety protocols implies that once the developer is confident that the model is a schemer, they’re totally fine with undeploying the untrusted model.[1][2]

But that seems kind of unlikely. Even if AI companies find strong evidence that their AIs are trying to escape, they won’t necessarily stop deploying them. This might be justified: even a responsible actor might think that [...]

---

Outline:

(03:18) Behavioral observations vs other ways of detecting scheming

(08:29) What does control look like if you need to keep deploying schemers?

(15:06) How does the prospect of needing to deploy known schemers affect the relative value of control research?

(16:56) What research should be done that focuses specifically on this?

(17:02) Sample-efficient ML

(17:27) Training recalcitrant policies

(19:56) A few other historical reasons for focusing on before-you-catch

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/XxjScx4niRLWTfuD5/handling-schemers-if-shutdown-is-not-an-option

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Bad for loss of control risks, bad for concentration of power risks

I’ve had this sitting in my drafts for the last year. I wish I’d been able to release it sooner, but on the bright side, it’ll make a lot more sense to people who have already read AI 2027.

1. There's a good chance that AGI will be trained before this decade is out.
   1. By AGI I mean “An AI system at least as good as the best human X’ers, for all cognitive tasks/skills/jobs X.”
   2. Many people seem to be dismissing this hypothesis ‘on priors’ because it sounds crazy. But actually, a reasonable prior should conclude that this is plausible.[1]
   3. For more on what this means, what it might look like, and why it's plausible, see AI 2027, especially the Research section.
2. If so, by default the existence of AGI will be a closely guarded [...]

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/FGqfdJmB8MSH5LKGc/training-agi-in-secret-would-be-unsafe-and-unethical-1

Narrated by TYPE III AUDIO.

I recall seeing three “rationalist” cases for Trump:

1. Richard Ngo on Twitter and elsewhere focused on the realignment of elite coalitions, observing that “most elite institutions have become leftist monocultures” but speculating that “over the next 5–10 years Silicon Valley will become the core of the Republicans.” The left-wing monoculture catastrophically damaged institutional integrity when public-health officials lied during the pandemic and when bureaucrats used threats and intimidation to censor speech on Facebook and Twitter and elsewhere—in the long-term this could move the country toward the draconian censorship regimes, restrictions on political opposition, and unresponsiveness to public opinion that we see today in England, France, and Germany. While we have strong free-speech protections now, we could be “frog-boiled by bureaucracies … an unsolved civilizational problem which has already strangled western economies and is now wrecking politics.” Maybe a coalition of new Silicon Valley elites like Musk, Thiel, Luckey [...]

---

First published:
April 18th, 2025

Source:
https://www.lesswrong.com/posts/tKhbDBkstMJuBv7jg/three-months-in-evaluating-three-rationalist-cases-for-trump

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Estimated Complexity: 3/5 (this is a guess, I will update based on feedback/seeing how the scenario goes)

STORY

It's that time of year again. The time when the Tithe Assessment Exactors demand that all adventurers pay taxes on the various monster parts they have hacked off and sold in the past year. And, more importantly for you, the time when clients begin banging on your door looking for advice on how to minimize their taxes.

This used to be a straightforward, if complex, application of the published tax rules. But ever since the disaster a few years back (when one of your clients managed to pay 1 silver piece in tax and then receive as a [...]

---

Outline:

(00:31) STORY

(01:58) DATA & OBJECTIVES

(03:27) SCHEDULING & COMMENTS

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/N7ZgcGvG3kfiBryom/d-and-d-sci-tax-day-adventurers-and-assessments

Narrated by TYPE III AUDIO.

Summary 

• We conducted a small investigation into using SAE features to recover sandbagged capabilities.
• We used the Goodfire API to pick out 15 SAE features, 5 related to sandbagging, 5 related to general expertise, and 5 unrelated features.
• We used these features to steer a Llama 3.3 70b Instruct that has been prompted to sandbag 
• We observed that steering can recover up to ~25% of true capabilities
• We observed that using sandbagging/deception/evaluation-awareness relevant features were slightly better than unrelated features, though results are mixed.
• While steering with SAE features partially recovered capabilities, this is not as effective as other existing interventions

Introduction

During our work on linear probing for evaluation awareness, we thought about using evaluation-awareness relevant features to uncover sandbagging. Previous work demonstrated that you can add noise to weights or activations and recover capabilities from a prompted sandbagging model, and we thought it would be interesting [...]

---

Outline:

(00:12) Summary

(01:04) Introduction

(01:47) Methodology

(06:11) Results

(07:57) Limitations

(08:16) Discussion

(09:41) Acknowledgements

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/dBckLjYfTShBGZ8ma/can-sae-steering-reveal-sandbagging

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

SUMMARY:

ALLFED is making an emergency appeal here due to a serious funding shortfall. Without new support, ALLFED will be forced to cut half our budget in the coming months, drastically reducing our capacity to help build global food system resilience for catastrophic scenarios like nuclear winter, a severe pandemic, or infrastructure breakdown. ALLFED is seeking $800,000 over the course of 2025 to sustain its team, continue policy-relevant research, and move forward with pilot projects that could save lives in a catastrophe. As funding priorities shift toward AI safety, we believe resilient food solutions remain a highly cost-effective way to protect the future. If you’re able to support or share this appeal, please visit allfed.info/donate.

FULL ARTICLE:

I (David Denkenberger) am writing alongside one of my team-mates, as ALLFED's co-founder, to ask for your support. This is the first time in Alliance to Feed the [...]

---

Outline:

(02:24) The case for ALLFED's work, and why we think maintaining full current capacity is valuable

(03:57) How this connects to AI and other risks

(05:23) What we're asking for

---

First published:
April 16th, 2025

Source:
https://www.lesswrong.com/posts/HL7zzZhCR59CCntpB/allfed-emergency-appeal-help-us-raise-usd800-000-to-avoid

Narrated by TYPE III AUDIO.

We’ve written a new report on the threat of AI-enabled coups.

I think this is a very serious risk – comparable in importance to AI takeover but much more neglected.

In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here's a very basic threat model for AI takeover:

1. Humanity develops superhuman AI
2. Superhuman AI is misaligned and power-seeking
3. Superhuman AI seizes power for itself

And now here's a closely analogous threat model for AI-enabled coups:

1. Humanity develops superhuman AI
2. Superhuman AI is controlled by a small group
3. Superhuman AI seizes power for the small group

While the report focuses on the risk that someone seizes power over a country, I think that similar dynamics could allow someone to take over the world. In fact, if someone wanted to take over the world, their best strategy might well be to first stage an AI-enabled [...]

---

Outline:

(02:39) Summary

(03:31) An AI workforce could be made singularly loyal to institutional leaders

(05:04) AI could have hard-to-detect secret loyalties

(06:46) A few people could gain exclusive access to coup-enabling AI capabilities

(09:46) Mitigations

(13:00) Vignette

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 16th, 2025

Source:
https://www.lesswrong.com/posts/6kBMqrK9bREuGsrnd/ai-enabled-coups-a-small-group-could-use-ai-to-seize-power-1

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 37 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Redwood Research has released a new paper, Ctrl-Z: Controlling AI Agents via Resampling. This is the largest and most intricate study of control techniques to date: that is, techniques that aim to prevent catastrophic failures even if egregiously misaligned AIs attempt to subvert the techniques. We extend control protocols to a more realistic, multi-step setting, develop novel resample protocols that outperform all existing control protocols, and study many aspects of control techniques in more detail than prior work. In this blog post, we summarize the main takeaways and lessons learned from our work.

Here's the abstract of the paper:

Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. [...]

---

Outline:

(06:48) The Multi-Step Control Problem

(10:14) Resampling

(11:46) Selection Pressure: Forcing Consistent Attacks

(14:05) Value of Information: Better Attack Detection (Incrimination)

(17:01) Important Limitations

(20:23) Results Summary

(21:03) Core takeaways

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 16th, 2025

Source:
https://www.lesswrong.com/posts/LPHMMMZFAWog6ty5x/ctrl-z-controlling-ai-agents-via-resampling

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

New: https://openai.com/index/updating-our-preparedness-framework/

Old: https://cdn.openai.com/openai-preparedness-framework-beta.pdf

Summary

Thresholds & responses: https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf#page=5. High and Critical thresholds trigger responses, like in the old PF; responses to Critical thresholds are not yet specified.

Three main categories of capabilities:

• Bio/chem: High capabilities trigger security controls and (for external deployment) misuse safeguards
• Cyber: High capabilities trigger security controls and (for external deployment) misuse safeguards and (for large-scale internal deployment) misalignment safeguards
• AI Self-improvement: High capabilities trigger security controls

Misuse safeguards, misalignment safeguards, and security controls for High capability levels: https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf#page=16. My quick takes:

• Misuse safeguards: fine categories but it's not clear what level of assurance would suffice
• Misalignment safeguards: worrying categories and it's not clear what level of assurance would suffice
• Security controls: it's impossible to evaluate security level based on principles like these

[I'll edit this post to add more analysis soon]

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/Yy5ijtbNfwv8DWin4/openai-rewrote-its-preparedness-framework

Narrated by TYPE III AUDIO.

---

Outline:

(01:26) The GPUs Are Melting

(02:36) On OpenAI's Ostensive Open Model

(03:25) Other People Are Not Worried About AI Killing Everyone

(04:18) What Even is AGI?

(05:19) The OpenAI AI Action Plan

(05:57) Copyright Confrontation

(07:43) The Ring of Power

(11:55) Safety Perspectives

(14:00) Autonomous Killer Robots

(15:15) Amicus Brief

(17:25) OpenAI Recklessly Races to Release

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/nnzb7prw2XC8cFhnP/openai-13-altman-at-ted-and-openai-cutting-corners-on-safety

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A post by Michael Nielsen that I found quite interesting. I decided to reproduce the full essay content here, since I think Michael is fine with that, but feel free to let me know to only excerpt it.

This is the text for a talk exploring why experts disagree so strongly about whether artificial superintelligence (ASI) poses an existential risk to humanity. I review some key arguments on both sides, emphasizing that the fundamental danger isn't about whether "rogue ASI" gets out of control: it's the raw power ASI will confer, and the lower barriers to creating dangerous technologies. This point is not new, but has two underappreciated consequences. First, many people find rogue ASI implausible, and this has led them to mistakenly dismiss existential risk. Second: much work on AI alignment, while well-intentioned, speeds progress toward catastrophic capabilities, without addressing our world's potential vulnerability to dangerous technologies.

[...]

---

Outline:

(06:37) Biorisk scenario

(17:42) The Vulnerable World Hypothesis

(26:08) Loss of control to ASI

(32:18) Conclusion

(38:50) Acknowledgements

The original text contained 29 footnotes which were omitted from this narration.

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/YeQe36XiY4BhrtRh5/asi-existential-risk-reconsidering-alignment-as-a-goal-1

Narrated by TYPE III AUDIO.

Introduction

Writing this post puts me in a weird epistemic position. I simultaneously believe that:

• The reasoning failures that I'll discuss are strong evidence that current LLM- or, more generally, transformer-based approaches won't get us AGI
• As soon as major AI labs read about the specific reasoning failures described here, they might fix them
• But future versions of GPT, Claude etc. succeeding at the tasks I've described here will provide zero evidence of their ability to reach AGI. If someone makes a future post where they report that they tested an LLM on all the specific things I described here it aced all of them, that will not update my position at all.

That is because all of the reasoning failures that I describe here are surprising in the sense that given everything else that they can do, you’d expect LLMs to succeed at all of these tasks. The [...]

---

Outline:

(00:13) Introduction

(02:13) Reasoning failures

(02:17) Sliding puzzle problem

(07:17) Simple coaching instructions

(09:22) Repeatedly failing at tic-tac-toe

(10:48) Repeatedly offering an incorrect fix

(13:48) Various people's simple tests

(15:06) Various failures at logic and consistency while writing fiction

(15:21) Inability to write young characters when first prompted

(17:12) Paranormal posers

(19:12) Global details replacing local ones

(20:19) Stereotyped behaviors replacing character-specific ones

(21:21) Top secret marine databases

(23:32) Wandering items

(23:53) Sycophancy

(24:49) What's going on here?

(32:18) How about scaling? Or reasoning models?

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/sgpCuokhMb8JmkoSn/untitled-draft-7shu

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Context

Disney's Tangled (2010) is a great movie. Spoilers if you haven't seen it.

The heroine, having been kidnapped at birth and raised in a tower, has never stepped foot outside. It follows, naturally, that she does not own a pair of shoes, and she is barefoot for the entire adventure. The movie contains multiple shots that focus at length on her toes. Things like that can have an outsized influence on a young mind, but that's Disney for you.

Anyway.

The male romantic lead goes by the name of "Flynn Rider." He is a dashingly handsome, swashbuckling rogue who was carefully crafted to be maximally appealing to women. He is the ideal male role model. If you want women to fall in love with you, it should be clear that the optimal strategy is to pretend to be Flynn Rider. Shortly into the movie is the twist: Flynn [...]

---

Outline:

(00:09) Context

(02:30) Reminder About Winning

(04:20) Technical Truth is as Bad as Lying

(06:40) Being Mistaken is Also as Bad as Lying

(08:01) This is Partially a Response

(10:19) Examples

(13:59) Biting the Bullet

(15:06) My Proposed Policy

(18:15) Appearing Trustworthy Anyway

(20:46) Cooperative Epistemics

(22:26) Conclusion

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/zTRqCAGws8bZgtRkH/a-dissent-on-honesty

Narrated by TYPE III AUDIO.

The original Map of AI Existential Safety became a popular reference tool within the community after its launch in 2023. Based on user feedback, we decided that it was both useful enough and had enough room for improvement that it was worth creating a v2 with better organization, usability, and visual design. Today we’re excited to announce that the new map is live at AISafety.com/map.

Similar to the original map, it provides a visual overview of the key organizations, programs, and projects in the Al safety ecosystem. Listings are separated into 16 categories, each corresponding to an area on the map:

• Advocacy
• Blog
• Capabilities research
• Career support
• Conceptual research
• Empirical research
• Forecasting
• Funding
• Governance
• Newsletter
• Podcast
• Research support
• Resource
• Strategy
• Training and education
• Video

We think there's value in being able to view discontinued projects, so we’ve included a graveyard for those.

Below the map is also a [...]

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/rF7MQWGbqQjEkeLJA/map-of-ai-safety-v2

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One key hope for mitigating risk from misalignment is inspecting the AI's behavior, noticing that it did something egregiously bad, converting this into legible evidence the AI is seriously misaligned, and then this triggering some strong and useful response (like spending relatively more resources on safety or undeploying this misaligned AI).

You might hope that (fancy) internals-based techniques (e.g., ELK methods or interpretability) allow us to legibly incriminate misaligned AIs even in cases where the AI hasn't (yet) done any problematic actions despite behavioral red-teaming (where we try to find inputs on which the AI might do something bad), or when the problematic actions the AI does are so subtle and/or complex that humans can't understand how the action is problematic[1]. That is, you might hope that internals-based methods allow us to legibly incriminate misaligned AIs even when we can't produce behavioral evidence that they are misaligned.

[...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 15th, 2025

Source:
https://www.lesswrong.com/posts/4QRvFCzhFbedmNfp4/to-be-legible-evidence-of-misalignment-probably-has-to-be

Narrated by TYPE III AUDIO.

There's an implicit model I think many people have in their heads of how everyone else behaves. As George Box is often quoted, “all models are wrong, but some are useful.” I’m going to try and make the implicit model explicit, talk a little about what it would predict, and then talk about why this model might be wrong.

Here's the basic idea: A person's behavior falls on a bell curve.

1. 

Adam is a new employee at Ratburgers, a fast food joint crossed with a statistics bootcamp. It's a great company, startup investors have been going wild for machine learning in literally anything, you've been tossing paper printouts of Attention Is All You Need into the meatgrinder so you can say your hamburgers are made with AI. Anyway, you weren’t around for Adam's hiring, you haven’t seen his resume, you have no information about him when you [...]

---

Outline:

(00:34) 1.

(04:19) 2.

(05:56) 3.

(08:46) 4.

(11:43) 5.

(15:55) 6.

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/rmhZamJFQdk5byqKQ/the-bell-curve-of-bad-behavior

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Definition

In 1954, Roger Bannister ran the first officially sanctioned sub-4-minute mile: a pivotal record in modern middle-distance running. Before Bannister's record, such a time was considered impossible. Soon after Bannister's record, multiple runners also beat the 4-minute mile.

This essay outlines the potential psychological effect behind the above phenomenon — the 4-minute mile effect — and outlines implications for its existence. The 4-minute mile effect describes when someone breaking a perceived limit enables others to break the same limit. In short, social proof is very powerful.

Tyler Cowen's Example

Speaking to Dwarkesh Patel, Tyler Cowen posits, "mentors only teach you a few things, but those few things are so important. They give you a glimpse of what you can be, and you're oddly blind to that even if you're very very smart." The 4-minute mile effect explains Tyler Cowen's belief in the value of [...]

---

Outline:

(00:11) Definition

(00:52) Tyler Cowens Example

(01:51) Personal Examples

(03:00) Implications

(03:54) Further Reading

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/kBdfFgLfDhPMDHfAa/the-4-minute-mile-effect

Linkpost URL:
https://parconley.com/the-four-minute-mile-effect/

Narrated by TYPE III AUDIO.

Executive summary

The Trump administration backtracked from his tariff plan, reducing tariffs from most countries to 10%, but imposing tariffs of 145% on China, which answered with 125% tariffs on US imports and export controls on rare earth metals.

The US saw protests against Trump and DOGE, and the US Supreme court ruled that the administration must facilitate the return of an immigrant wrongly deported to El Salvador.

OpenAI is slashing the amount of safety testing they will do of their new AI models, from months down to days, and Google launched a new AI chip for inference.

Negotiations between the US and Iran on its nuclear program are starting, reducing our estimated probability of a US strike on Iran's nuclear programme by the end of May to 15%.

The number of US dairy herds that have had confirmed H5N1 infections has hit 1,000.

Economy

[...]

---

Outline:

(00:24) Executive summary

(01:22) Economy

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/Ns7qfCqSgMjQNDZN8/sentinel-s-global-risks-weekly-roundup-15-2025-tariff-yoyo

Linkpost URL:
https://blog.sentinel-team.org/p/global-risks-weekly-roundup-152025

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: These are results of a brief research sprint and I didn't have time to investigate this in more detail. However, the results were sufficiently surprising that I think it's worth sharing.

TL,DR: I train a probe to detect falsehoods on a token-level, i.e. to highlight the specific tokens that make a statement false. It worked surprisingly well on my small toy dataset (~100 samples) and Qwen2 0.5B, after just ~1 day of iteration! Colab link here.

Context: I want a probe to tell me where in an LLM response the model might be lying, so that I can e.g. ask follow-up questions. Such "right there" probes[1] would be awesome to assist LLM-based monitors (Parrack et al., forthcoming). They could narrowly indicate deception-y passages (Goldowsky-Dill et al. 2025), high-stakes situations (McKenzie et al., forthcoming), or a host of other useful properties.

The writeup below is the (lightly edited) [...]

---

Outline:

(01:47) Summary

(04:15) Details

(04:18) Motivation

(06:37) Methods

(06:40) Data generation

(08:35) Probe training

(09:40) Results

(09:43) Probe training metrics

(12:21) Probe score analysis

(15:07) Discussion

(15:10) Limitations

(17:50) Probe failure modes

(19:40) Appendix A: Effect of regularization on mean-probe

(21:01) Appendix B: Initial tests of generalization

(21:28) Appendix C: Does the individual-token probe beat the mean probe at its own game?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/kxiizuSa3sSi4TJsN/try-training-token-level-probes

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Dario Amodei, CEO of Anthropic, recently worried about a world where only 30% of jobs become automated, leading to class tensions between the automated and non-automated. Instead, he predicts that nearly all jobs will be automated simultaneously, putting everyone "in the same boat." However, based on my experience spanning AI research (including first author papers at COLM / NeurIPS and attending MATS under Neel Nanda), robotics, and hands-on manufacturing (including machining prototype rocket engine parts for Blue Origin and Ursa Major), I see a different near-term future.

Since the GPT-4 release, I've evaluated frontier models on a basic manufacturing task, which tests both visual perception and physical reasoning. While Gemini 2.5 Pro recently showed progress on the visual front, all models tested continue to fail significantly on physical reasoning. They still perform terribly overall. Because of this, I think that there will be an interim period where a significant [...]

---

Outline:

(01:28) The Evaluation

(02:29) Visual Errors

(04:03) Physical Reasoning Errors

(06:09) Why do LLM's struggle with physical tasks?

(07:37) Improving on physical tasks may be difficult

(10:14) Potential Implications of Uneven Automation

(11:48) Conclusion

(12:24) Appendix

(12:44) Visual Errors

(14:36) Physical Reasoning Errors

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/r3NeiHAEWyToers4F/frontier-ai-models-still-fail-at-basic-physical-tasks-a

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: a model I find helpful to make sense of disagreements and, sometimes, resolve them.

I like to categorize disagreements using four buckets:

They don’t represent a perfect partitioning of “disagreement space”, meaning there is some overlap between them and they may not capture all possible disagreements, but they tend to get me pretty far in making sense of debates, in particular when and why they fail. In this post I’ll outline these four categories and provide some examples.

I also make the case that labels disagreements are the worst and in most cases can either be dropped entirely, or otherwise should be redirected into one of the other categories.

Facts

These are the most typical disagreements, and are probably what most people think disagreements are about most of the time, even when it's actually closer to a different category. Factual disagreements [...]

---

Outline:

(00:58) Facts

(01:59) Values

(03:18) Strategy

(05:00) Labels

(07:49) Why This Matters

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 13th, 2025

Source:
https://www.lesswrong.com/posts/9vKqAQEDxLvKh7KcN/four-types-of-disagreement

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: If we optimize a steering vector to induce a language model to output a single piece of harmful code on a single training example, then applying this vector to unrelated open-ended questions increases the probability that the model yields harmful output.

Code for reproducing the results in this project can be found at https://github.com/jacobdunefsky/one-shot-steering-misalignment.

Intro

Somewhat recently, Betley et al. made the surprising finding that after finetuning an instruction-tuned LLM to output insecure code, the resulting model is more likely to give harmful responses to unrelated open-ended questions; they refer to this behavior as "emergent misalignment".

My own recent research focus has been on directly optimizing steering vectors on a single input and seeing if they mediate safety-relevant behavior. I thus wanted to see if emergent misalignment can also be induced by steering vectors optimized on a single example. That is to say: does a steering vector optimized [...]

---

Outline:

(00:31) Intro

(01:22) Why care?

(03:01) How we optimized our steering vectors

(05:01) Evaluation method

(06:05) Results

(06:09) Alignment scores of steered generations

(07:59) Resistance is futile: counting misaligned strings

(09:29) Is there a single, simple, easily-locatable representation of misalignment? Some preliminary thoughts

(13:29) Does increasing steering strength increase misalignment?

(15:41) Why do harmful code vectors induce more general misalignment? A hypothesis

(17:24) What have we learned, and where do we go from here?

(19:49) Appendix: how do we obtain our harmful code steering vectors?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 14th, 2025

Source:
https://www.lesswrong.com/posts/kcKnKHTHycHeRhcHF/one-shot-steering-vectors-cause-emergent-misalignment-too

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: I claim that many reasoning patterns that appear in chains-of-thought are not actually used by the model to come to its answer, and can be more accurately thought of as historical artifacts of training. This can be true even for CoTs that are apparently "faithful" to the true reasons for the model's answer.

Epistemic status: I'm pretty confident that the model described here is more accurate than my previous understanding. However, I wouldn't be very surprised if parts of this post are significantly wrong or misleading. Further experiments would be helpful for validating some of these hypotheses.

Until recently, I assumed that RL training would cause reasoning models to make their chains-of-thought as efficient as possible, so that every token is directly useful to the model. However, I now believe that by default,[1] reasoning models' CoTs will often include many "useless" tokens that don't help the model achieve [...]

---

Outline:

(02:13) RL is dumber than I realized

(03:32) How might vestigial reasoning come about?

(06:28) Experiment: Demonstrating vestigial reasoning

(08:08) Reasoning is reinforced when it correlates with reward

(11:00) One more example: why does process supervision result in longer CoTs?

(15:53) Takeaways

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
April 13th, 2025

Source:
https://www.lesswrong.com/posts/6AxCwm334ab9kDsQ5/vestigial-reasoning-in-rl

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Thanks to Linda Linsefors for encouraging me to write my story. Although it might not generalize to anyone else, I hope it will help some AI safety newcomers get a clearer picture of how to get into the field.

In Summer 2023, I thought that all jobs in AI safety were super competitive and needed qualifications I didn’t have. I was expecting to need a few years to build skills and connections before getting any significantly impactful position. Surprisingly to me, it only took me 3 months after leaving my software engineering job before I landed a job in AI policy. I now think that it's actually somewhat easy to get into AI policy without any prior experience if you’re agentic and accept some tradeoffs.

The job: International AI policy event organizer

From January 2024 to February 2025, I worked on organizing high level international AI policy events:

• [...]

---

Outline:

(00:55) The job: International AI policy event organizer

(03:16) How I got the job

(06:02) My theory as to what led to me being hired and how it can be reproduced by others

(07:27) My experience at this job and where I stand now

---

First published:
April 13th, 2025

Source:
https://www.lesswrong.com/posts/qvCCQwnvM4Gb6W8Bg/how-i-switched-careers-from-software-engineer-to-ai-policy

Narrated by TYPE III AUDIO.

I'm graduating from UChicago in around 60 days, and I've been thinking about what I've learned these past four years. I figured I'd write it all down while it's still fresh.

This isn't universal advice. It's specifically for people like me (or who want to be like me). High-agency, motivated types who hate having free time.[1] People who'd rather risk making mistakes than risk missing out, who want to control more than they initially think they can, and who are willing to go all-in relatively quickly. If you're reading Ben Kuhn and Alexey Guzey or have ever heard of the Reverend Thomas Bayes, you're probably one of us.

So here's at least some of what I've figured out. Take what's useful, leave what isn't — maybe do the opposite of everything I've said.

• Mindset and Personal Growth
  • Find your mission
  • Recognize that you can always be better
  • [...]

---

Outline:

(03:23) Mindset and Personal Growth

(03:27) Find your mission

(04:27) Recognize that you can always be better

(04:54) Make more mistakes

(05:23) Things only get done to the extent that you want them to get done

(06:14) There are no adults

(06:37) Deadlines are mostly fake

(07:26) Put yourself in positions where youll be lucky

(08:09) Luck favors the prepared

(08:45) Test your fit at lots of things

(09:26) Do side projects

(10:20) Get good at introspecting

(11:00) Be honest

(11:31) Put your money where your mouth is

(12:03) Interpret others charitably

(12:32) Be the kind of person others can come to for help

(13:18) Productivity and Focus

(13:22) Go to bed early

(13:52) Brick your phone

(14:21) The optimal amount of slack time is not zero, but its close to zero

(15:19) If you arent getting work done, pick up your shit and go somewhere else

(15:45) Dont take your phone to the places youre studying

(16:08) Do not try to do more than one important thing at once

(16:31) Offload difficult things to automations, habits, or other people

(16:55) Make your bed

(17:18) If it takes less than 5 minutes, do it now

(17:44) The floor is actually way lower than you think

(18:07) Planning and Goal Setting

(18:11) Make sure your goals are falsifiable

(19:11) Credibly pre-commit to things you care about getting done

(19:41) Track your progress

(20:09) Make a 5-year plan google doc

(20:31) Consider graduating early

(21:00) Relationships and Community

(21:04) Build your own community

(21:30) Fall in love at least once

(22:09) If you arent happy and excited and exciting single, you wont be happy or excited or exciting in a relationship

(22:54) Friends are people you can talk to for hours

(23:26) Throw parties

(24:07) Hang out with people who are better than you at the things you care about

(24:36) Professors are people. You can just make friends with them

(25:10) Academic Success

(25:14) Go to office hours

(25:41) Be careful how you use AI

(26:13) Read things, everywhere

(27:16) Dont be afraid to skip the boring parts

(27:43) Learn how to read quickly

(28:16) Write things

(28:54) Read widely based on curiosity, not just relevance

(29:32) Have an easy way to capture ideas

(29:53) Health and Lifestyle

(29:57) Go to the gym

(30:25) The only place to work yourself to failure is the gym

(30:59) If something isnt making your life better, change it

(31:29) Leave campus

The original text contained 37 footnotes which were omitted from this narration.

---

First published:
April 12th, 2025

Source:
https://www.lesswrong.com/posts/9Kq2JRqmJHnzckxKn/college-advice-for-people-like-me

Narrated by TYPE III AUDIO.

Introduction

This is a nuanced “I was wrong” post.

Something I really like about AI safety and EA/rationalist circles is the ease and positivity in people's approach to being criticised.[1] For all the blowups and stories of representative people in the communities not living up to the stated values, my experience so far has been that the desire to be truth-seeking and to stress-test your cherished beliefs is a real, deeply respected and communally cultured value. This in particular explains my ability to keep getting jobs and coming to conferences in this community, despite being very eager to criticise and call bullshit on people's theoretical agendas.

One such agenda that I’ve been a somewhat vocal critic of (and which received my criticism amazingly well) is the “heuristic arguments” picture and the ARC research agenda more generally. Last Spring I spent about 3 months on a work trial/internship at [...]

---

Outline:

(00:10) Introduction

(03:24) Background and motte/bailey criticism

(09:49) The missing piece: connecting in the no-coincidence principle

(15:15) From the no coincidence principle to statistical explanations

(17:46) Gödel and the thorny deeps

(19:30) Ignoring the monsters and the heuristic arguments agenda

(24:46) Upshots

(27:41) Summary

(29:19) Renormalization as a cousin of heuristic arguments

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
April 13th, 2025

Source:
https://www.lesswrong.com/posts/CYDakfFgjHFB7DGXk/untitled-draft-wn6w

Narrated by TYPE III AUDIO.

Epistemic status: Noticing confusion

There is little discussion happening on LessWrong with regards to AI governance and outreach. Meanwhile, these efforts could buy us time to figure out technical alignment. And even if we figure out technical alignment, we still have to solve crucial governmental challenges so that totalitarian lock-in or gradual disempowerment don't become the default outcome of deploying aligned AGI.

Here's three reasons why we think we might want to shift much more resources towards governance and outreach:

1. MIRI's shift in strategy

The Machine Intelligence Research Institute (MIRI), traditionally focused on technical alignment research, has pivoted to broader outreach. They write in their 2024 end of year update:

Although we continue to support some AI alignment research efforts, we now believe that absent an international government effort to suspend frontier AI research, an extinction-level catastrophe is extremely likely.

As a consequence, our new focus is [...]

---

Outline:

(00:45) 1. MIRIs shift in strategy

(01:34) 2. Even if we solve technical alignment, Gradual Disempowerment seems to make catastrophe the default outcome

(02:52) 3. We have evidence that the governance naysayers are badly calibrated

(03:33) Conclusion

---

First published:
April 12th, 2025

Source:
https://www.lesswrong.com/posts/Y49znC2qfL9SXkT7e/why-does-lw-not-put-much-more-focus-on-ai-governance-and

Narrated by TYPE III AUDIO.

In this post I lay out a concrete vision of how reward-seekers and schemers might function. I describe the relationship between higher level goals, explicit reasoning, and learned heuristics. I explain why I expect reward-seekers and schemers to dominate proxy-aligned models given sufficiently rich training environments (and sufficient reasoning ability).

A key point is that explicit reward seekers can still contain large quantities of learned heuristics (context-specific drives). By viewing these drives as instrumental and having good instincts for when to trust them, a reward seeker can capture the benefits of both instinctive adaptation and explicit reasoning without paying much of a speed penalty.

Core claims

1. To get high reward, models must learn context-specific heuristics which are not derivable by reasoning from the goal of reward maximization.[1]
2. It is also true that thinking explicitly about reward is sometimes a waste of time and punished by speed penalties.
3. It [...]

---

Outline:

(00:52) Core claims

(01:52) Characterizing reward-seekers

(06:59) When will models think about reward?

(10:12) What I expect schemers to look like

(12:43) What will terminal reward seekers do off-distribution?

(13:24) What factors affect the likelihood of scheming and/or terminal reward seeking?

(14:09) What about CoT models?

(14:42) Relationship of subgoals to their superior goals

(16:19) A story about goal reflection

(18:54) Thoughts on compression

(21:30) Appendix: Distribution over worlds

(24:44) Canary string

(25:01) Acknowledgements

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/ntDA4Q7BaYhWPgzuq/reward-seekers

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Cross-posted from Substack.

AI job displacement will affect young people first, disrupting the usual succession of power and locking the next generation out of our institutions. I’m coining the term youth lockout to describe this phenomenon.

Youth Lockout

We are on track to build AI agents that can independently perform valuable intellectual labour. These agents will directly compete with human workers for roles in the labour market, often offering services at lower cost and greater speeds.

In historical cases of automation, such as the industrial revolution, automation reduced the number of human jobs in some industries but created enough new opportunities that overall demand for labour went up. AI automation will be very different from these examples because AI is a much more general-purpose technology. With time, AI will likely perform all economic tasks at human or superhuman levels, meaning any new firm or industry will be able to [...]

---

Outline:

(00:24) Youth Lockout

(03:34) Impacts

(06:54) Conclusion

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/tWuYhdajaXx4WzMHz/youth-lockout

Narrated by TYPE III AUDIO.

Paper is good. Somehow, a blank page and a pen makes the universe open up before you. Why paper has this unique power is a mystery to me, but I think we should all stop trying to resist this reality and just accept it.

Also, the world needs way more mundane blogging.

So let me offer a few observations about paper. These all seem quite obvious. But it took me years to find them, and they’ve led me to a non-traditional lifestyle, paper-wise.

Observation 1: The primary value of paper is to facilitate thinking.

For a huge percentage of tasks that involve thinking, getting some paper and writing / drawing / scribbling on it makes the task easier. I think most people agree with that. So why don’t we act on it? If paper came as a pill, everyone would take it. Paper, somehow, is underrated.

But note, paper [...]

---

Outline:

(00:38) Observation 1: The primary value of paper is to facilitate thinking.

(01:23) Observation 2: If you don't have a system, you won't get much benefit from paper.

(02:02) Observation 3: User experience matters.

(02:52) Observation 4: Categorization is hard.

(03:35) Paper systems I've used

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/RdHEhPKJG6mp39Agw/paper

Narrated by TYPE III AUDIO.

Summary

OpenAI recently released the Responses API. Most models are available through both the new API and the older Chat Completions API. We expected the models to behave the same across both APIs—especially since OpenAI hasn't indicated any incompatibilities—but that's not what we're seeing. In fact, in some cases, the differences are substantial. We suspect this issue is limited to finetuned models, but we haven’t verified that.

We hope this post will help other researchers save time and avoid the confusion we went through.

Key takeaways are that if you're using finetuned models:

• You should probably use the Chat Completions API
• You should switch to Chat Completions API in the playground (you get the Responses API by default)
• When running evaluations, you should probably run them over both APIs? It's hard to say what is the "ground truth" here.

Example: ungrammatical model

In one of our [...]

---

Outline:

(01:11) Example: ungrammatical model

(02:08) Ungrammatical model is not the only one

(02:37) Whats going on?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/vTvPvCH2G9cbcFY8a/openai-responses-api-changes-models-behavior

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's generally agreed that as AIs get more capable, risks from misalignment increase. But there are a few different mechanisms by which more capable models are riskier, and distinguishing between those mechanisms is important when estimating the misalignment risk posed at a particular level of capabilities or by a particular model.

There are broadly 3 reasons why misalignment risks increase with capabilities:

• As AIs get more capable, the chance of egregious misalignment increases: AIs with more powerful opaque reasoning abilities are more likely to become egregiously misaligned—sufficiently misaligned that they strategically deceive and work against their developers—without this being detected or trained out. We should expect opaque reasoning abilities generally increase as models get more capable, though it isn't entirely obvious what the relationship between opaque and non-opaque cognition will be in future AI systems. Additionally, I expect that as AIs get more capable their capabilities [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/NDotm7oLHfR56g4sD/why-do-misalignment-risks-increase-as-ais-get-more-capable

Narrated by TYPE III AUDIO.

Google Lays Out Its Safety Plans

I want to start off by reiterating kudos to Google for actually laying out its safety plan. No matter how good the plan, it's much better to write down and share the plan than it is to not share the plan, which in turn is much better than not having a formal plan.

They offer us a blog post, a full monster 145 page paper (so big you have to use Gemini!) and start off the paper with a 10 page summary.

The full paper is full of detail about what they think and plan, why they think and plan it, answers to objections and robust discussions. I can offer critiques, but I couldn’t have produced this document in any sane amount of time, and I will be skipping over a lot of interesting things in the full paper because [...]

---

Outline:

(00:58) Core Assumptions

(08:14) The Four Questions

(11:46) Taking This Seriously

(16:04) A Problem For Future Earth

(17:53) That's Not My Department

(23:22) Terms of Misuse

(27:50) Misaligned!

(36:17) Aligning a Smarter Than Human Intelligence is Difficult

(51:48) What Is The Goal?

(54:43) Have You Tried Not Trying?

(58:07) Put It To The Test

(59:44) Mistakes Will Be Made

(01:00:35) Then You Have Structural Risks

---

First published:
April 11th, 2025

Source:
https://www.lesswrong.com/posts/hvEikwtsbf6zaXG2s/on-google-s-safety-plan

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Authors: Eli Lifland, Nikola Jurkovic[1], FutureSearch[2]

This is supporting research for AI 2027. We'll be cross-posting these over the next week or so.

Assumes no large-scale catastrophes happen (e.g., a solar flare, a pandemic, nuclear war), no government or self-imposed slowdown, and no significant supply chain disruptions. All forecasts give a substantial chance of superhuman coding arriving in 2027.

Summary

We forecast when the leading AGI company will internally develop a superhuman coder (SC): an AI system that can do any coding tasks that the best AGI company engineer does, while being much faster and cheaper. At this point, the SC will likely speed up AI progress substantially as is explored in our takeoff forecast.

We first show Method 1: time-horizon-extension, a relatively simple model which forecasts when SC will arrive by extending the trend established by METR's report of AIs accomplishing tasks that take humans increasing amounts [...]

---

Outline:

(00:56) Summary

(02:43) Defining a superhuman coder (SC)

(03:35) Method 1: Time horizon extension

(05:05) METR's time horizon report

(06:30) Forecasting SC's arrival

(06:54) Method 2: Benchmarks and gaps

(06:59) Time to RE-Bench saturation

(07:03) Why RE-Bench?

(09:25) Forecasting saturation via extrapolation

(12:42) AI progress speedups after saturation

(14:04) Time to cross gaps between RE-Bench saturation and SC

(14:32) What are the gaps in task difficulty between RE-Bench saturation and SC?

(15:11) Methodology

(17:25) How fast can the task difficulty gaps be crossed?

(23:31) Other factors for benchmarks and gaps

(23:46) Compute scaling and algorithmic progress slowdown

(24:43) Gap between internal and external deployment

(25:20) Intermediate speedups

(26:55) Overall benchmarks and gaps forecasts

(27:44) Appendix

(27:47) Individual Forecaster Views for Benchmark-Gap Model Factors

(27:53) Engineering complexity: handling complex codebases

(31:16) Feedback loops: Working without externally provided feedback

(37:28) Parallel projects: Handling several interacting projects

(38:45) Specialization: Specializing in skills specific to frontier AI development

(40:17) Cost and speed

(48:48) Other task difficulty gaps

(50:52) Superhuman Coder (SC): time horizon and reliability requirements

(55:53) RE-Bench saturation resolution criteria

The original text contained 19 footnotes which were omitted from this narration.

---

First published:
April 10th, 2025

Source:
https://www.lesswrong.com/posts/ggqSg7bSLChanfunf/forecasting-time-to-automated-superhuman-coders-ai-2027

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

---

---

---

---

---

---

---

---

---

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: Briefer and more to the point than my model of what is going on with LLMs, but also lower effort.

Here is the paper. The main reaction I am talking about is AI 2027, but also basically every lesswrong take on AI 2027.

A lot of people believe in very short AI timelines, say <2030. They want to justify this with some type of outside view, straight-lines-on-graphs argument, which is pretty much all we've got because nobody has a good inside view on deep learning (change my mind).

The outside view, insofar is that is a well-defined thing, does not justify very short timelines.

If AGI were arriving in 2030, the outside view says interest rates would be very high (I'm not particularly knowledgeable about this and might have the details wrong but see the analysis here, I believe the situation is still [...]

---

First published:
April 10th, 2025

Source:
https://www.lesswrong.com/posts/BrHv7wc6hiJEgJzHW/reactions-to-metr-task-length-paper-are-insane

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.