Start / LessWrong posts by zvi / O3 oh my by zvi

“o3, Oh My” by Zvi

77 min • 30 december 2024

OpenAI presented o3 on the Friday before Thanksgiving, at the tail end of the 12 Days of Shipmas.

I was very much expecting the announcement to be something like a price drop. What better way to say ‘Merry Christmas,’ no?

They disagreed. Instead, we got this (here's the announcement, in which Sam Altman says ‘they thought it would be fun’ to go from one frontier model to their next frontier model, yeah, that's what I’m feeling, fun):

Greg Brockman (President of OpenAI): o3, our latest reasoning model, is a breakthrough, with a step function improvement on our most challenging benchmarks. We are starting safety testing and red teaming now.

Nat McAleese (OpenAI): o3 represents substantial progress in general-domain reasoning with reinforcement learning—excited that we were able to announce some results today! Here is a summary of what we shared about o3 in the livestream.

---

Outline:

(03:48) GPQA Has Fallen

(04:21) Codeforces Has Fallen

(05:32) Arc Has Kinda of Fallen But For Now Only Kinda

(09:27) They Trained on the Train Set

(15:26) AIME Has Fallen

(15:58) Frontier of Frontier Math Shifting Rapidly

(19:09) FrontierMath 4: We're Going To Need a Bigger Benchmark

(23:10) What is o3 Under the Hood?

(25:17) Not So Fast!

(28:38) Deep Thought