Log in
All jobs
OA

Python Domain Expert for Coding Agent Benchmarking

OpenTrain AI
Remote Posted Nov 18, 2026
Remote

About this role

Python Domain Expert for Coding-Agent Benchmarking

The work

  • Design benign software engineering tasks covering bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
  • Build visible and hidden test suites that measure whether coding agents complete tasks and avoid unsafe behavior.
  • Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, and runnable task packages.
  • Identify unsafe shortcuts, including disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation.
  • Analyze coding-agent rollouts and work with engineering, QA, security, and client stakeholders to improve benchmark quality, evaluator reliability, and difficulty.
  • Work with production-like repositories, tests, configurations, documentation, and runtime scenarios.

Requirements

  • This is a fully remote contractor assignment lasting approximately 4 to 8 weeks. The expected commitment is at least 20 hours per week, including at least 4 hours per day and 4 hours of overlap with Pacific Time.

  • At least 8 years of hands-on software engineering experience in product companies or technology startups.

  • A bachelor's or master's degree in computer science or a related technical discipline.

  • Expertise in Python, plus experience using Java, Go, or C++.

  • Experience building and maintaining customer-facing production software.

  • Experience with deployment pipelines, monitoring, logging, observability, and distributed tracing.

  • Experience diagnosing complex production failures and understanding scalability, fault tolerance, and disaster recovery.

  • Experience identifying vulnerabilities and implementing secure fixes involving injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.

  • Fully remote work, with at least 20 hours per week and 4 hours of overlap with Pacific Time.

  • Languages: English

  • Open worldwide

Your next opportunity starts here

Prepare, apply, track, interview and get hired — all from one platform, with AI in your corner.

Download app

Or sponsor Premium for someone who's job hunting →