[Contract] Machine Learning Researchers (PhD)

Mercor

Remote

Role Overview

Mercor is partnering with a leading AI research lab on Project Vesuvius, an initiative designed to evaluate and enhance the ability of large language models (LLMs) to generate structured, high-quality research plans for open-ended machine learning problems.

We are seeking Machine Learning Researchers and PhDs to serve as annotators who will assess and provide structured feedback on AI-generated research plans. The goal is to improve how LLMs function as brainstorming partners for machine learning research.

This is a remote, short-term engagement with flexible hours and opportunities to contribute to frontier AI evaluation and research.

Key Responsibilities

Evaluate and compare AI-generated research plans for clarity, feasibility, and technical soundness.
Design and compile ML tasks based on real-world challenges and research competitions.
Draft detailed, executable natural language plans for machine learning workflows.
Implement and validate research plans in Python within a Docker environment.
Assess outputs against structured rubrics, provide usefulness scores, and deliver concise, objective feedback.

Ideal Qualifications

5+ years of experience in applied machine learning or a PhD in machine learning or related fields.
Strong understanding of ML research methodologies, experimental design, and evaluation practices.
Excellent analytical and technical writing skills.
Experience with reproducibility or benchmarking in ML research preferred.
Detail-oriented and able to deliver high-quality, structured feedback independently.