“SHIFT relies on token-level features to de-bias Bias in Bios probes” by Tim Hua

In Sparse Feature Circuits (Marks et al. 2024), the authors introduced Spurious Human-Interpretable Feature Trimming (SHIFT), a technique designed to eliminate unwanted features from a model's computational process. They validate SHIFT on the Bias in Bios task, which we think is too simple to serve as meaningful validation. To summarize:

1. SHIFT ablates SAE latents related to undesirable concepts (in this case, gender) from an LM-based classifier trained on a biased dataset. The authors show that this procedure de-biases the classifier and argue that their experiment demonstrates real-world utility from SAEs.
2. We believe the Bias in Bios task is too simple. If SAEs only picked up on latents related to specific tokens, it would have been sufficient for them to do well on this de-biasing task.
3. Replicating appendix A4 of Karvonen et al. (2025), we show that we can de-bias the probe by ablating only the SAE latents immediately after the [...]

---

Outline:

(03:07) Background on SHIFT

(06:59) The SHIFT experiment in Marks et al. 2024 relies on embedding features.

(07:51) You can train an unbiased classifier just by deleting gender-related tokens from the data.

(08:33) In fact, for some models, you can train directly on the embedding (and de-bias by removing gender-related tokens)

(09:21) If not BiB, how do we check that SHIFT works?

(10:00) SHIFT applied to classifiers and reward models

(10:51) SHIFT for cognition-based oversight/disambiguating behaviorally identical classifiers

(12:03) Next steps: Focus on what to disentangle, and not just how well you can disentangle them

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
March 19th, 2025

Source:
https://www.lesswrong.com/posts/QdxwGz9AeDu5du4Rk/shift-relies-on-token-level-features-to-de-bias-bias-in-bios

---

Narrated by TYPE III AUDIO.