Research Engineer

Research Engineer

Full time

Apply to Job

San Francisco, CA | Hybrid

3 - 12 Years of Experience

Apply to Job

Share this job

Who is Recruiting from Scratch:

Recruiting from Scratch is a premier talent firm that focuses on placing the best product managers, software, and hardware talent at innovative companies. Our team is 100% remote and we work with teams across the United States to help them hire.

Research Engineer

Location

San Francisco, CA / Bay Area, CA

Hybrid role. Remote-friendly culture with an SF hub; candidates in the Bay Area are preferred for hybrid collaboration.

Compensation

$250,000 – $290,000 Base + Competitive Equity

Visa

Open to visa transfers, including H-1B and OPT transfers.

Exceptional candidates may be considered for additional sponsorship on a case-by-case basis.

Company Stage

High-Growth / Venture-Backed Technology Company

Industry

Artificial Intelligence, AI Infrastructure, Developer Tools, Machine Learning, LLMs, Web Data, Data Infrastructure, Web Scraping, Developer Platforms

About the Company

Our client is building AI infrastructure that enables developers and AI-native companies to reliably turn the web into clean, structured, LLM-ready data.

The platform processes web content at massive scale, converting URLs into structured data and markdown that can be consumed by LLM applications, AI agents, search systems, and modern AI workflows.

The company has experienced exceptional growth, reaching eight figures in ARR during its first year and continuing to scale rapidly. Its developer-focused product has become widely adopted across the AI ecosystem, with a large open-source community and thousands of developers and AI companies building on the platform.

This is a small, highly technical, high-ownership engineering organization where engineers own meaningful pieces of the product end-to-end.

The Research Engineer role sits at the intersection of machine learning, research methodology, backend engineering, data engineering, and evaluation.

This is not a traditional research role where the primary responsibility is publishing papers or experimenting with models in isolation. It is also not a benchmark-operator role where someone simply runs pre-existing evaluation frameworks.

Instead, the engineer will define what "good" means, design the metrics used to measure it, build the evaluation infrastructure, generate and curate datasets, and create feedback loops that directly influence model and product decisions.

The ideal candidate combines strong engineering ability with rigorous analytical thinking and is comfortable working in ambiguous environments where the evaluation methodology itself needs to be invented.

What You'll Do

• Design evaluation systems that rigorously measure the quality of AI and web-data outputs

• Define metrics that determine what "good output" means across millions of websites and edge cases

• Build scalable evaluation pipelines and testing infrastructure

• Build evaluation harnesses capable of measuring model and product performance at scale

• Generate and curate high-quality datasets for evaluation

• Design datasets that accurately represent real-world web diversity and edge cases

• Develop methodologies for evaluating LLM-powered systems

• Measure the accuracy, reliability, completeness, and usefulness of generated outputs

• Build automated evaluation systems for web extraction and structured data generation

• Create statistical and quantitative methods for comparing system performance

• Investigate failure cases and identify patterns in model or product behavior

• Develop feedback loops connecting evaluation results to model improvements

• Translate evaluation findings into actionable product and engineering decisions

• Work closely with ML engineers and researchers to improve model performance

• Partner with product and engineering teams to determine evaluation priorities

• Build tooling that makes evaluation results accessible and actionable across the company

• Develop systems for continuously monitoring output quality

• Create regression testing frameworks for AI-powered systems

• Establish benchmarks and quality standards for production AI systems

• Build data pipelines supporting evaluation workflows

• Automate dataset generation, labeling, validation, and analysis

• Work with large-scale web and text datasets

• Investigate edge cases across different websites, formats, languages, and content types

• Analyze NLP and LLM outputs for quality and consistency

• Develop methods for evaluating structured and unstructured outputs

• Design experiments to determine whether product and model changes actually improve performance

• Analyze statistical significance and reliability of evaluation results

• Identify limitations and biases within evaluation datasets

• Improve evaluation methodology as models and product capabilities evolve

• Build internal tools for researchers and engineers to investigate system behavior

• Own evaluation infrastructure from initial concept through production deployment

• Move quickly from hypothesis to implementation, measurement, and iteration

• Close the loop between research, engineering, evaluation, and product decisions

• Work on ambiguous problems where there may not be an established methodology

• Contribute to technical direction around AI evaluation and measurement

• Operate with high autonomy in a fast-moving AI infrastructure environment

• Work directly with founders and senior technical leadership on high-impact problems

• Build systems that help the company understand whether its core product actually works

• Ideal Candidate Background

• Experience Requirements

• 3+ years of experience in Machine Learning Engineering, Research Engineering, Software Engineering, Data Engineering, Applied Science, or related technical roles

• Strong engineering foundation with experience building production systems

• Experience building evaluation systems, testing infrastructure, experimentation frameworks, or data-heavy systems

• Experience working with machine learning or LLM-powered systems

• Experience designing metrics or methodologies for measuring system quality

• Experience building data pipelines or infrastructure supporting evaluation and experimentation

• Experience working with large datasets

• Experience analyzing complex or noisy data

• Experience solving ambiguous technical problems independently

• Experience taking research or analytical concepts and turning them into production systems

• Experience designing experiments and interpreting results

• Strong understanding of how to measure model or system performance

• Experience working with NLP, LLMs, web data, search, recommendation systems, or similar domains is highly relevant

• Experience working across engineering and research-oriented teams

• Strong ability to balance methodological rigor with practical engineering constraints

• Comfortable owning projects end-to-end

• Comfortable operating without fully specified requirements

• Strong bias toward shipping, measuring, and iterating

• Experience working in startups or high-growth technology environments strongly preferred

• Technical Requirements

• Strong Python engineering skills

• Strong software engineering fundamentals

• Experience building production-grade data pipelines

• Experience working with machine learning systems

• Experience working with LLMs or NLP systems

• Experience designing evaluation frameworks

• Experience creating quantitative metrics

• Strong statistical analysis fundamentals

• Strong experimental design skills

• Experience analyzing model outputs

• Experience evaluating AI-generated content

• Experience working with large-scale datasets

• Experience with data generation and dataset curation

• Experience with data cleaning and preprocessing

• Experience building automated testing or evaluation systems

• Experience designing regression testing frameworks

• Experience building scalable evaluation infrastructure

• Experience with machine learning experimentation

• Experience with statistical or scientific computing libraries

• Experienc