Preference Model builds realistic reinforcement-learning environments with diverse tasks and robust rewards to teach models to perform machine-learning research—the first step toward strong AI.