Example Company
Software engineer
- Location
- San Francisco
- Type
- Full time
- Compensation
- $150k-$200k
- Posted
- 2 days ago
Join a team building useful software. Work closely with engineering and design to turn new ideas into products.
Loading job details
Benchmarks Storyteller
- Type
- Full time
- Compensation
- $100K - $150K; 0.20% - 0.60% equity
- Posted
- 2D AGO
About the role
About the role
About Openbenchmarks
Agents are becoming first-class users of the internet. They research, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need its product picked by agents, and agents prefer open, independent, grounded benchmarks to make that call.
Openbenchmarks is the evaluation infrastructure for agents: domain-specific, reproducible evaluations that help agents and people pick tools with confidence. Our mission is to be the trusted evaluation layer for agents.
Founders previously led AI research and infra teams at Oracle and Appfolio. We're a small team of researchers and engineers working with fast-growing AI-first companies like Parallel, Firecrawl, Telnyx, TinyFish and more.
About the role
Every benchmark we run produces a lot of data: accuracy, latency, cost, failure modes, and how results change between versions. You make sure what we publish is correct, clear and convincing, and you communicate it. That means rigorous statistics, graphs that show the real trade-off at a glance, and technical writing that holds up, from reports to posts.
The bar is a benchmark report a skeptical researcher can't poke holes in. Every number has an error bar, every comparison is controlled, and every claim is backed by a test.
What you'll work on
Technical writing - Write up each benchmark: methodology, results, limitations, and enough detail for anyone to reproduce our work. Technical reports by default, research papers when the work warrants it.
Statistics - Standard errors and confidence intervals on every result, stated sample sizes, and proper tests (e.g. mixed-effects models with per-task effects) before we call one tool better than another. Explain the outliers, don't hide them.
Methodology - Design controlled comparisons: isolate one variable, hold the rest fixed, and write up the grading rules and judge setup so anyone can reproduce the result.
Graphs - Score vs. cost with Pareto frontiers, cost broken into components, latency and cost distributions, and per-category results with error bars. Built in code, clear before pretty.
Communicating results - Turn each benchmark into reports, launch posts and threads for X, LinkedIn, Reddit and HN. Lead with one sharp finding and one graph.
Finding the story - Work with our researchers on the raw results. Dig into individual failures, spot what changed between runs, and decide what's worth saying.
You might be a fit if
- You have a strong background in statistics: regression, mixed-effects models, significance testing, and you know when a result isn't significant
- You've designed or run evals (ML or LLM evaluation) and know where they go wrong
- You write clear technical prose: methodology, results and limitations that a skeptical reader trusts
- You've made graphs that changed how people saw a result, and can show them
Nice to have
- You've written research papers (published or not)
- Familiarity with existing benchmarks (SWE-bench, Terminal-bench, BrowseComp) and how vendors cite them
- An audience on X, LinkedIn or a blog
About Openbenchmarks
About OpenBenchmarks
Agents are becoming first-class users and consumers of the internet. They research, evaluate, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need to get their products picked and used by agents.
Agents increasingly prefer open, independent and grounded benchmarks to make decisions.
Openbenchmarks is the evaluation infrastructure for agents - domain-specific, reproducible evaluations that help agents pick tools with confidence.
Our mission is to be the trusted evaluation layer for agents.
Founders previously led AI research and Infra teams at Oracle and Appfolio;
We started Openbenchmarks as an output of our research in the field of model behavior and how agents actually chose between different tools.
We're a team of researchers, engineers and work with the fastest growing AI first companies like Parallel, Firecrawl, Telnyx, TinyFish and more.
Similar roles
Engineering Delivery Manager IIDeloitte
Full Stack Software EngineerApoha · London- Senior Process Engineering Specialist – Industrial MineralsOmya
Data Platform & Engineering LeadBarclays Innovation Hub powered by Eagle Labs · Pune
R&D Drilling Engineer - Tool (m/w/d)Hades Mining · Munich
DevOps EngineerBitsight · Tel Aviv