Research Scientist, Human Data & Evaluation

Summary

Contra Labs is a human-centered AI lab focused on creative and multimodal outputs, building datasets for benchmarking, evaluation, and post-training of AI models. The Research Scientist will independently lead rigorous research projects spanning human-data collection, evaluation methodology, experimentation, qualitative and quantitative analysis, and client-ready insights across creative and multimodal domains.

Responsibilities

• Translate ambiguous client questions into clear research questions, hypotheses, and study designs
• Develop research proposals and methodologies from scratch rather than relying on a predefined approach
• Identify promising new research directions and push the team toward ideas we have not already considered
• Help define what Contra Labs should measure, study, and build next
• Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and experiments
• Develop strong prompt strategies for eliciting, evaluating, and comparing model behavior
• Determine appropriate sampling, evaluator selection, calibration, quality-control, and reliability methods
• Design methodologies that work across creative and multimodal domains such as design, image, video, and agentic workflows
• Analyze qualitative and quantitative evaluation data
• Move beyond reporting results to explain why patterns occur and what they imply
• Synthesize findings into clear recommendations, research narratives, and new hypotheses
• Produce client-ready reports, presentations, benchmarks, and research publications
• Independently own research engagements from initial scoping through final delivery
• Work closely with project leads, creative experts, research engineers, and data/engineering teams
• Maintain a high bar for research rigor and data quality while operating on fast client timelines
• Potentially support multiple concurrent projects at different stages
• Be comfortable working on research driven by real client problems and constraints
• Participate in scoping and proposal development and help establish credibility with sophisticated AI/ML research teams
• Direct client-facing experience is valuable, but this is not primarily an account-management role. You will collaborate with Strategic Project Leads on client delivery

Skills

• • 3–5+ years of relevant research experience, with a Master's, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field
• • Strong qualitative and quantitative research skills, with experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research
• • Proven ability to independently design and execute rigorous research, including deciding what data to collect, why, and how to analyze it
• • Ability to turn data and observations into new insights, research directions, and actionable recommendations
• • Comfortable analyzing data with Python and/or SQL
• • Strong written communication and ability to clearly explain methodological tradeoffs and research decisions
• • Comfortable operating independently through ambiguity and balancing research rigor with startup speed and pragmatism
• • Based in San Francisco, California
• • Familiarity with modern ML and generative AI systems
• • Understanding of RLHF / post-training / preference data / model evaluation
• • Experience studying image, video, multimodal, creative, or agentic systems
• • Experience designing prompts or systematically studying prompting/model behavior
• • Experience working with large-scale human-data or annotation programs
• • Experience working directly with AI research or model teams
• • Strong publication record or other evidence of consistently generating original research ideas
• • Startup / 0-to-1 experience

Qualifications

Must Haves

• • 3–5+ years of relevant research experience, with a Master's, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field
• • Strong qualitative and quantitative research skills, with experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research
• • Proven ability to independently design and execute rigorous research, including deciding what data to collect, why, and how to analyze it
• • Ability to turn data and observations into new insights, research directions, and actionable recommendations
• • Comfortable analyzing data with Python and/or SQL
• • Strong written communication and ability to clearly explain methodological tradeoffs and research decisions
• • Comfortable operating independently through ambiguity and balancing research rigor with startup speed and pragmatism
• • Based in San Francisco, California

Nice to Haves

• • Familiarity with modern ML and generative AI systems
• • Understanding of RLHF / post-training / preference data / model evaluation
• • Experience studying image, video, multimodal, creative, or agentic systems
• • Experience designing prompts or systematically studying prompting/model behavior
• • Experience working with large-scale human-data or annotation programs
• • Experience working directly with AI research or model teams
• • Strong publication record or other evidence of consistently generating original research ideas
• • Startup / 0-to-1 experience

Benefits

• Equity
• Medical, Dental, Vision Benefits
• 401k Matching
• We will provide you with a company laptop on your start date
• Paid Case Study and Presentation