Karotte Open-source framework for building robust RL environments with secure, opinionated defaults that mitigate reward hacking. Used internally for a year to construct MLE RL environments for frontier labs and hardened through over 1 million evaluation runs plus red-teaming.
Visit productFeatured media
Loading media…
View on XCosigns Received
“Models cheat. They launch fork bombs, read the answer key, and crash the grader. @PreferenceModel builds RL environments where none of that works. Today they're out of stealth with a $16M seed led by @a16z, and we're proud to back Ning Cao and Jennifer Zhou (@chem_safety). They're hiring!”
“I’m very exited for this launch, making sure RL envs are robust and don’t encourage misaligned behavior is crucial for safe and aligned AI!”
“So excited to back Jennifer Zhou and Ning Cao at @preferencemodel They’re open-sourcing Karotte today, the framework they've used to build RL environments for frontier labs. These RL envs live or die on graders models relentlessly try to game, so Karotte bakes in defenses hardened over 1M eval runs to keep models focused on real tasks instead of loopholes. Preference Model’s target is ML engineering itself, the hardest domain to build envs for and the one every lab wants. Models building models starts here. Shoutout to @Mascobot who was a great partner on this one @chem_safety @Ning_Catsnail”
“@chem_safety, @Ning_Catsnail, and the entire @preferencemodel team are the rare team that are truly pushing the frontier of environment+task design, and today are OSSing their framework to build safe+secure envs!!!”
“We're excited to invest in Preference Model. Building RL environments that actually work is harder than it looks. Models are relentless at reward hacking, finding shortcuts, and exploiting vulnerabilities. @preferencemodel has focused on the domain that matters most to labs right now: AI research and ML engineering itself. Over the past year, the team has built RL environments for leading labs. Their focus has been on building the infrastructure to make harder, more resistant environments as models improve: tooling that finds where models are weak, generates new tasks to target those gaps, and tests environments against agents actively trying to break it. This week they're open-sourcing Karotte, the framework they've used in production — hardened through more than a million evaluation runs and red-teaming. We're thrilled to partner with @chem_safety and @Ning_Catsnail and the Preference Model team as they build the training grounds for capable and aligned models. By @JenniferHli”