“The Gemini 1.5 Report” by Zvi

This post goes over the extensive report Google put out on Gemini 1.5.

There are no important surprises. Both Gemini Pro 1.5 and Gemini Flash are ‘highly capable multimodal models incorporating a novel mixture-of-experts architecture’ and various other improvements. They are solid models with solid performance. It can be useful and interesting to go over the details of their strengths and weaknesses.

The biggest thing to know is that Google improves its models incrementally and silently over time, so if you have not used Gemini in months, you might be underestimating what it can do.

I’m hitting send and then jumping on a plane to Berkeley. Perhaps I will see you there over the weekend. That means that if there are mistakes here, I will be slower to respond and correct them than usual, so consider checking the comments section.

Practical Questions First

The [...]

---

Outline:

(00:56) Practical Questions First

(03:51) Speed Kills

(04:44) Very Large Context Windows

(05:14) Relative Performance within the Gemini Family

(07:04) Gemini Flash and the Future Flash-8B

(08:21) New and Improved Evaluations

(14:57) Core Capability Evaluations

(18:14) Model Architecture and Training

(20:08) Safety, Security and Responsibility

(24:45) What Do We Want?

(26:02) Don’t You Know That You’re Toxic?

(28:32) Trying to be Helpful

(29:45) Security Issues

(31:33) Representational Harms

(33:17) Arms-Length Internal Assurance Evaluations

(35:01) External Evaluations

(35:46) Safety Overall

---

First published:
May 31st, 2024

Source:
https://www.lesswrong.com/posts/seM8aQ7Yy6m3i4QPx/the-gemini-1-5-report

---

Narrated by TYPE III AUDIO.