AI Observability Engineer

Job title: AI Observability Engineer

About the Role The AI Observability Engineer plays a critical role in interaction with our AI product teams to ensure the AI solutions are aligned with the enterprise Responsible AI principles and standards. This role emphasizes in-person collaboration and is primarily onsite. You will support the design and execution of the AI Governance program and help communicate Responsible AI practices organization-wide.

What You'll Do

• Monitor model performance KPIs (context relevance, accuracy, precision, recall, F1) for ML, LLM, and agentic AI solutions
• Monitor data quality and bias; detect anomalies, drift to protect fairness and accuracy
• Use observability tools to produce structured, trustworthy insights from detailed data on AI system activities
• Manage ongoing operations of our AI model management platform and pipelines, evolving KPIs and model observability capabilities
• Review AI solutions for alignment with AIG’s Policies, AI Governance Framework and Responsible AI principles
• Provide support to AI product teams, conduct independent reviews, respond to ad-hoc queries, and participate in workshops
• Collaborate with legal, compliance, security, and other teams to coordinate Responsible AI reviews and ensure cohesive approach
• Create executive-level reports on Responsible AI and provide strategic recommendations to senior leadership
• Create and update RAI training content for technical and non-technical audiences
• Contribute to development and improvement of RAI assets: methods, tutorials, tools
• Monitor and stay up to date on emerging AI regulatory frameworks and translate into RAI framework updates

What We're Looking For

• Minimum 3 years’ experience and a Bachelor’s degree
• ML, LLM knowledge of models, algorithms, and evaluation metrics essential for monitoring
• Data pipelines and analysis skills to handle, clean, and transform data for monitoring
• Experience scaling human oversight of AI systems; familiarity with LLM development
• Comfort with ambiguity and curiosity; strong ownership
• Passion for Responsible AI or AI ethics; understanding of AI technologies and frameworks
• Ability to collaborate with cross-functional teams; experience in financial services or regulated industries
• Working knowledge of policies around usage of emerging AI tools and products; bringing best practices from the industry
• Track record of solving complex problems, developing mitigation strategies, and influencing others on key AI initiatives
• Interest in staying ahead of global AI regulations and translating frameworks into actionable updates

Nice to Have

• Production LLM, RAG or model inference monitoring pipelines
• Instrumenting applications using OpenTelemetry distributed tracing or structured logs
• Experience with AI Observability tools such as Fiddler, Arize, LangSmith or similar tools
• Ability to build dashboards and alerts tied to AI model behavior beyond system uptime

Compensation & Benefits

• Salary range and benefits not disclosed in posting.