ML Engineer - Model Evaluation Expert

Summary

Mercor connects elite creative and technical talent with leading AI research labs. The company is seeking a Machine Learning Engineer to design research-based tasks, run and analyze training experiments, evaluate frontier models, and collaborate with researchers on rigorous and fair task development.

Responsibilities

• Design tasks by transforming real ML research ideas into well-defined, multi-step tasks
• Run experiments by implementing changes, executing training experiments, and analyzing results to define correct solutions
• Explore reinforcement learning concepts such as reward functions and training behavior in task development
• Evaluate frontier models' performance on tasks and identify areas of improvement
• Collaborate with researchers to ensure tasks are consistent, rigorous, and fair

Skills

• • MSc or PhD in machine learning, computer science, or a related STEM field
• • 1+ years in a research or research-engineering role
• • Experience in training and evaluating ML models and conducting end-to-end experiments
• • Proficiency in Python and Git
• • Strong understanding of large language models and their evaluation
• • Basic knowledge of reinforcement learning
• • Experience in AI training, model evaluation, or benchmark/task authoring

Qualifications

Must Haves

• • MSc or PhD in machine learning, computer science, or a related STEM field
• • 1+ years in a research or research-engineering role
• • Experience in training and evaluating ML models and conducting end-to-end experiments
• • Proficiency in Python and Git
• • Strong understanding of large language models and their evaluation

Nice to Haves

• • Basic knowledge of reinforcement learning
• • Experience in AI training, model evaluation, or benchmark/task authoring

Benefits

• Remote work
• Paid weekly