About this role
Python Domain Expert for Coding-Agent Benchmarking
The work
- Design benign software engineering tasks covering bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
- Build visible and hidden test suites that measure whether coding agents complete tasks and avoid unsafe behavior.
- Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, and runnable task packages.
- Identify unsafe shortcuts, including disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation.
- Analyze coding-agent rollouts and work with engineering, QA, security, and client stakeholders to improve benchmark quality, evaluator reliability, and difficulty.
- Work with production-like repositories, tests, configurations, documentation, and runtime scenarios.
Requirements
-
This is a fully remote contractor assignment lasting approximately 4 to 8 weeks. The expected commitment is at least 20 hours per week, including at least 4 hours per day and 4 hours of overlap with Pacific Time.
-
At least 8 years of hands-on software engineering experience in product companies or technology startups.
-
A bachelor's or master's degree in computer science or a related technical discipline.
-
Expertise in Python, plus experience using Java, Go, or C++.
-
Experience building and maintaining customer-facing production software.
-
Experience with deployment pipelines, monitoring, logging, observability, and distributed tracing.
-
Experience diagnosing complex production failures and understanding scalability, fault tolerance, and disaster recovery.
-
Experience identifying vulnerabilities and implementing secure fixes involving injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
-
Fully remote work, with at least 20 hours per week and 4 hours of overlap with Pacific Time.
-
Languages: English
-
Open worldwide