“I am the Golden Gate Bridge” by Zvi

Easily Interpretable Summary of New Interpretability Paper

Anthropic has identified (full paper here) how millions of concepts are represented inside Claude Sonnet, their current middleweight model. The features activate across modalities and languages as tokens approach the associated context. This scales up previous findings from smaller models.

By looking at neuron clusters, they defined a distance measure between clusters. So the Golden Gate Bridge is close to various San Francisco and California things, and inner conflict relates to various related conceptual things, and so on.

Then it gets more interesting.

Importantly, we can also manipulate these features, artificially amplifying or suppressing them to see how Claude's responses change.

If you sufficiently amplify the feature for the Golden Gate Bridge, Claude starts to think it is the Golden Gate Bridge. As in, it thinks it is the physical bridge, and also it gets obsessed, bringing [...]

---

Outline:

(00:03) Easily Interpretable Summary of New Interpretability Paper

(02:59) One Weird Trick

(05:27) Zvi Parses the Actual Symbol Equations

(08:06) Identifying and Verifying Features

(12:38) Features as Computational Intermediates

(13:28) Oh That's the Deception Feature, Nothing to Worry About

(18:14) What Do They Think This Mean for Safety?

(19:10) Limitations

(22:28) Researcher Perspectives

(23:24) Other Reactions

(24:19) I Am the Golden Gate Bridge

(26:06) Golden Gate Bridges Offer Mundane Utility

(27:39) The Value of Steering

(31:54) To What Extent Did We Know This Already?

(43:07) Is This Being Oversold?

(47:07) Crossing the Bridge Now That We’ve Come to It

---

First published:
May 27th, 2024

Source:
https://www.lesswrong.com/posts/JdcxDEqWKfsucxYrk/i-am-the-golden-gate-bridge

---

Narrated by TYPE III AUDIO.