About this role
Principal Software Engineer
How do you know an AI product is actually getting better—and how do you prove it at scale, before millions of people experience the difference?
We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality. Teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that inform what ships. We do this at the scale of one of the world’s largest AI products, within a strict enterprise compliance boundary. As Copilot evolves into a fleet of autonomous agents, the way we measure quality must evolve with it—and you will lead that transformation.
As a Principal Software Engineer on the Copilot evaluation team, you will own the technical vision for a platform built AI-first and agent-first: one where autonomous agents are not only measured, but also help operate, extend, and heal the system itself.
You will take on our hardest problem: delivering trustworthy results across a deep chain of dependencies—including Copilot’s model, retrieval, and scoring services—that we push far beyond their normal operating envelope, with hundreds of calls per job under relentless and growing load. You will archit