“On Claude 3.5 Sonnet” by Zvi

There is a new clear best (non-tiny) LLM.

If you want to converse with an LLM, the correct answer is Claude Sonnet 3.5.

It is available for free on Claude.ai and the Claude iOS app, or you can subscribe for higher rate limits. The API cost is $3 per million input tokens and $15 per million output tokens.

This completes the trifecta. All of OpenAI, Google DeepMind and Anthropic have kept their biggest and more expensive model static for now, and instead focused on making something faster and cheaper that is good enough to be the main model.

You would only use another model if you either (1) needed a smaller model in which case Gemini 1.5 Flash seems best, or (2) it must have open model weights.

Updates to their larger and smaller models, Claude Opus 3.5 and Claude Haiku 3.5, are coming [...]

---

Outline:

(01:59) Benchmarks

(03:08) Human Evaluation Tests

(04:19) The Vision Thing

(05:46) Artifacts

(07:08) Privacy

(07:49) Safety

(08:51) Advancing the Frontier

(10:51) The Race is On

(11:43) Whispers of Recursive Self-Improvement

(16:35) Logic Fails

(18:34) Practical Reports

(24:20) What Comes Next

---

First published:
June 24th, 2024

Source:
https://www.lesswrong.com/posts/wx4RhFzLbiHoShFjR/on-claude-3-5-sonnet

---

Narrated by TYPE III AUDIO.