About the role

Besimple AI is building datasets, benchmarks, and training environments that help voice models understand people, interact naturally, and complete real tasks.

We’re looking for a linguist to help define and improve the quality of the speech and conversations used to train these models. You’ll bring a trained ear for language, an understanding of how people communicate, and the ability to explain what makes a conversation clear, natural, and appropriate to its context.

Your work will connect linguistic expertise with practical decisions: which recordings are suitable for training, how contributors should be evaluated, what annotators should label, and how we measure the behavior of conversational agents.

What you’ll work on

• Evaluate speech and conversational quality. Assess pronunciation, intelligibility, prosody, pacing, tone, and turn-taking in human recordings and model outputs. Explain why a response sounds natural, awkward, overly scripted, or inappropriate for the situation.
• Build quality standards and evaluation rubrics. Translate customer requirements and feedback into clear criteria, examples, and decision rules. Define what reviewers should accept, flag, or reject, including how expectations differ across conversational roles and tasks.
• Screen and support speech contributors. Help design auditions, evaluate recordings, and give actionable feedback on delivery. Identify contributors who can sustain natural conversations while following a scenario’s requirements.
• Create and review linguistic annotations. Develop guidelines and reference examples for transcripts, speaker turns, speech delivery, and conversational behavior. Review human and model-generated labels, resolve ambiguous cases, and improve consistency across reviewers.
• Improve conversational tasks and model evaluation. Work with researchers to refine scenarios, instructions, and turn-level rubrics. Identify unclear wording or grading criteria, and distinguish real model failures from problems in the task itself.
• Turn recurring problems into better workflows. Analyze disagreements and rejection patterns, calibrate reviewers, and improve guidance in the crowdsourcing platform we build. Partner with engineers and audio specialists to separate language and delivery issues from recording defects.

What we’re looking for

• A background in linguistics, speech science, phonetics, phonology, or a related field, through academic study or substantial applied experience.
• Strong listening and analytical skills, with the ability to describe specific features of speech and explain how they affect meaning, clarity, and naturalness.
• Experience with speech data, linguistic annotation, transcription, language assessment, or conversational analysis.
• The ability to turn nuanced judgments into guidelines that other people can apply consistently.
• Clear written communication and comfort giving specific, constructive feedback.
• Careful judgment about ambiguity, language variation, and the difference between personal preference and a task’s actual requirements.

We welcome recent graduates with relevant research or substantial projects, as well as linguists with industry experience. Research, annotation guidelines, evaluation studies, and examples of applied linguistic work are all useful evidence.

Especially relevant experience

• Evaluating text-to-speech, speech recognition, speech-to-speech, or conversational AI systems.
• Prosody, pragmatics, discourse analysis, sociolinguistics, or conversation analysis.
• Praat, ELAN, phonetic transcription, or other speech-analysis and annotation tools.
• Designing annotation schemes, building reference datasets, or measuring agreement between reviewers.
• Speaker auditions, voice coaching, or evaluating spoken performance.
• Working with multilingual datasets or collaborating with linguists across languages.
• Basic scripting or data analysis for investigating annotation quality and recurring errors.

You do not need experience in every area listed above.

Why this role

You’ll help shape the standards behind the data that voice models learn from. Your judgments will influence how we collect conversations, select speakers, evaluate model behavior, and improve the quality of our datasets.

This is an opportunity to apply linguistic expertise directly to the development of conversational AI, with ownership over both the quality of individual examples and the processes that make that quality repeatable.

About Besimple AI

Why Us

At Besimple AI, we’re making it radically easier for teams to build and ship reliable AI by fixing the hardest part of the stack: data. Good evaluation, training and safety data require domain experts, robust tooling and meticulous QA. AI teams and labs come to us to get high quality data so they can launch AI safely. We’re a YC X25 company based in Redwood City, CA, already powering evaluation and training pipelines for leading AI companies across customer support, search, and education. Join now to be close to real customer impact, not just demos.

Why This Matters

High-quality, human-reviewed data is still the single biggest driver of model quality, but most teams are stuck with old tools and legacy processes that do not scale to modern, multimodal, agentic workflows. Besimple replaces that mess with instant custom UIs, tailored rubrics, and an end-to-end human-in-the-loop workflow that supports text, chat, audio, video, LLM traces, and more. We meet teams where they are—whether they need on-prem deployments and granular user management or a fast cloud setup—to turn evaluation into a continuous capability rather than a one-time project.

Traction & Customers

Who You’ll Work With

Founders previously built the annotation platform that supported Meta’s Llama models. We’ve seen how world-class annotation systems shape model quality and iteration speed; we’re bringing those lessons to every AI team that needs to ship with confidence. You’ll work directly with the founders and users, owning problems end-to-end—from an interface that unlocks a tough rubric, to a workflow that reduces disagreement, to a AI judge system that improves quality.

How We Work

• Bias to shipping and learning with customers
• Respect for craft: calibration, rubric clarity, inter-annotator agreement (IRR)
• Tight feedback loops from production back to evaluation
• Ownership: you’ll shape evaluation as an engineering discipline with real “fail-to-ship” tests tied to business and safety goals

If you’re excited by systems that combine product design, human judgment, and applied AI—and you want to build the data and evaluation layer that keeps AI trustworthy—come build with us. See how fast teams can go from raw logs to a robust, human-in-the-loop eval pipeline—and how that changes the way they ship AI.