Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo

On the podcast we talk with Osman about Duolingo’s culture of experimentation, data and testing as a moat, and why passive aggressive push notifications actually work in the right context.

Top Takeaways
🧪 Leverage a culture of experimentation to create a top user experience
📊 Data and testing are Duolingo’s best moat
👍 Passive-aggressive push notifications might work in the right context

About Osman Mansur
👨‍💻 Product Manager (PM) at Duolingo, the global language learning app with close to 60 million active users.
💪 As PM on the retention team, Osman plays a key role in maximizing user engagement and retention through specific mechanics, with a dedicated testing and experimentation regimen.
💡 “Just by the sheer amount of data that we collect, we're really able to drill down and optimize a lot of things on the app. And it keeps us busy as a product team, because there's so much stuff that we know we can improve.”
👋  LinkedIn | Medium

Links & Resources
‣ Duolingo’s findings on notifications (Twitter thread)
‣ Duolingo’s findings on streak rewards (Twitter thread)
‣ How does Duo decide what message to send? The secret is in the AI!
‣ The habit-building research behind your Duolingo streak
‣ Join the Duolingo team

Follow us on Twitter
‣ David Barnard
‣ Jacob Eiting
‣ RevenueCat
‣ Sub Club

Episode Highlights

[2:08] Test everything: Osman explains how the data Duolingo collects and analyzes is one of the company’s best moats.
[5:26] Ideation generation: Sometimes the bottom-up approach works best for driving team roadmaps, but iteration and experimentation is at the heart of Duolingo’s testing process.
[11:02] Cooperation, working together: At Duolingo, teams share what they’ve learned with each other to create a better product, but they also cross over on analysis and experimentation.
[17:48] Looking back: To track long-term impact, Duolingo uses holdout experiments and looks at feature-level metrics via dashboards
[25:34] Notifications 101: Osman explains how a big driver of retention and company growth has been its notification strategy, learning a lot about what does and doesn’t work along the way.
[33:35] Let’s get creative: The secret to impactful retentive notifications is getting the tone right, and even conversing with users. Sterile voices don’t work — opinionated voices just might.
[37:30] Keep it simple: Messaging and theme matters for notifications, and so does copy length. Reduce cognitive load to increase willingness to engage. But once you’re in the app, you can get more complex for engaged users.
[40:10] Emoji titles: Osman’s team discovered that emojis are actually better in the title than in the body. Why it’s an attention grabber is still a mystery.
[41:53] Falling flat: Not every experiment works, but there are still great lessons to learn. The tone of an organic character works better than a brand talking to users like marketers.
[46:33] Keeping the streak going: The streak is one of the best Duolingo retention mechanics, Osman explains. He dives into how the company tinkered with it to prevent domino effect user drop offs, and how the streak widget works in iOS.