Which AI model is best at which PM task?

The only AI benchmark that matters to product execs. Frontier LLMs tested to the standards of world-class product and growth leaders.

Very cool! Love this.
Lenny Rachitsky

9 models23 task types414 outputs graded

GPT-6 Astra leads by 2 points.

Combined score by model and task type. Choose a column heading to sort by that task.
ModelProduct CraftGrowthMetrics & ExperimentationStrategyLeadership
GPT-6 AstraOpenAI89#196Best8184Best94Best7093918598Best100Best7696Best8999Best88Best82919298Best8495Best8490Best
GPT-6.1 SolOpenAI87#294837293899488878496737893998290Best8995Best81808596Best87
Sonnet 5.5Anthropic87#38087Best839293Best96Best93Best92Best808991Best9194Best96837780838681828976
Haiku 5.5Anthropic84#477786786929388718010089768395849092Best798374847587
Opus 5.5Anthropic82#57482689377917973909378888183746084958285Best808483
GPT-6 LunaOpenAI80#68578678485798868889873709078778983867970698179
Gemini 4 ArgonGoogle69#76770547789716867697666647071587756577252657589
Gemini 3.8 FlashGoogle61#86257396076545965767356527059585044606548616979
Gemini 3.5 Flash-LiteGoogle50#95342404549554744667632534450524542385840616056
Combined scoreEach cell is the model’s best setup on that task type.

Tested with criteria sourced from the very best.

We carefully research and design the rubric for each task around the advice of guests on Lenny’s Podcast.

30 product leaders behind the checks

We credit their ideas. None of them has reviewed or endorsed the benchmark.

Inside the benchmark

The checks AI fails most

  1. Names an advantage that's hard to copyDevelop product strategy10%
  2. Marks what to cut if the call runs overCustomer research call guide28%
  3. Says who it's for, and who it isn'tWrite a PRD35%
  4. Says why a test is worth it hereExperiment specification42%
What AI gets right and wrong, task by task

What you get for the cost

Show

Haiku 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at about a twenty-fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 89, 68× the cheapest (10 of 46 outputs estimated)GPT-6.1 Sol · API: task score 87, 14× the cheapestSonnet 5.5 · API: task score 87, 44× the cheapestHaiku 5.5 · API: task score 84, 2.4× the cheapestOpus 5.5 · Claude: task score 82, 65× the cheapest (10 of 46 outputs estimated), 1 critical failureGPT-6 Luna · API: task score 80, the cheapestGemini 4 Argon · API: task score 69, 16× the cheapest, 1 critical failureGemini 3.8 Flash · API: task score 61, 9.9× the cheapest, 2 critical failuresGemini 3.5 Flash-Lite · Gemini: task score 50, 2.2× the cheapest (11 of 46 outputs estimated), 5 critical failuresGPT-6 Astra · ChatGPTGPT-6.1 Sol · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API801.0×
Gemini 3.5 Flash-Lite · Gemini502.2× (partly estimated)
Haiku 5.5 · API842.4×
Gemini 3.8 Flash · API619.9×
GPT-6.1 Sol · API8714×
Gemini 4 Argon · API6916×
Sonnet 5.5 · API8744×
Opus 5.5 · Claude8265× (partly estimated)
GPT-6 Astra · ChatGPT8968× (partly estimated)

Score is the Combined score across every task type; cost is each setup’s typical output, relative to the cheapest.How we compare cost

Recent drops

All drops