AI QA Engineer (GenAI & ML Systems)

AI QA Engineer (GenAI & ML Systems)

We hiring an AI QA Engineer to own the quality, reliability and safety of our GenAI and ML-driven engineering applications. You will define how we test non-deterministic systems, from prompts and RAG pipelines to agentic workflows, and set the release bar for AI features used across Powertrain Engineering.

Key Responsibilities

• Design evaluation frameworks for LLM, RAG and agentic systems: accuracy, hallucination, consistency, latency and cost

• Build golden datasets, regression suites and automated eval pipelines integrated into CI/CD

• Run red-teaming and adversarial tests: prompt injection, jailbreaks, data leakage, bias and toxicity

• Validate retrieval quality, tool-calling and multi-step agent behavior

• Define quality gates and release criteria for AI features

• Monitor production drift and failure modes; lead root-cause analysis with AI engineers

• Report quality metrics and risks to engineering leadership

Qualifications

Required Qualifications :

• 4-8 years in QA/SDET, including 1-2+ years testing ML or GenAI systems

• Strong Python and test automation (pytest, API testing)

• Hands-on with LLM evaluation tools such as Ragas, DeepEval, promptfoo or LangSmith

• Working knowledge of LLM behavior, RAG architectures, embeddings and prompt engineering

• Statistical thinking: sampling, metric design, confidence, and the limits of LLM-as-judge

• CI/CD (GitHub Actions or Azure DevOps), Git and cloud fundamentals (Azure preferred)

• Preferred Qualifications

• Experience with agent frameworks and MCP/tool integrations

• Observability tooling for tracing and drift monitoring

• Exposure to engineering or automotive domains

Who Qualifies :

• Skeptical by default, precise, comfortable with non-deterministic outputs, a clear writer, and a close collaborator to AI engineers.

• Experience: 4-8 years

• Employment: Full-time

Benefits

• Discounts for Employees Possible

• Health Benefits

• Mobile Phone for Employees Possible

• Meal-Discounts

• Company Retirement

• Hybrid Work Possible

• Mobility Offers

• Events for Employees

• Coaching

• Flextime Possible

• Near-Site Childcare

• Parking

• Canteen, Café

• Good Public Transport

• Barrier-Free Workplace

• Inhouse Doctor

• Plot No. 5-P, EPIP 1st Phase560066 BangaloreDetails to location