About this role
About the Role ABC Fitness is seeking a Principal Engineer to architect, build, and evolve the MCP/ACP layer for API and AI orchestration, exposing ABC Fitness capabilities as standardized, agent-consumable tools. The role focuses on production-ready, auditable AI workflows with strong emphasis on reliability, security, and scalable orchestration. You will lead technical design and define reusable patterns, SDKs, and paved paths for safe AI adoption across the platform.
What You'll Do
- Architect, build, and evolve the MCP / ACP ecosystem that exposes ABC Fitness capabilities as standardized, agent-consumable tools.
- Design schemas, tool definitions, and abstraction layers that shield AI agents from complexity and provide clean, structured, hallucination-resistant interfaces.
- Define and implement reusable orchestration patterns for multi-step AI workflows, including tool use, validation, retries, state management, fallback handling, and long-running execution flows.
- Build for deterministic execution by ensuring AI-driven tool interactions are predictable, safe, observable, auditable, and reversible where necessary.
- Establish company-wide standards for AI interaction patterns, structured outputs, input/output validation, schema governance, and agent orchestration workflows.
- Design and implement security, permissions, and access control patterns for non-human actors and autonomous systems, including auditability, rate controls, and operational boundaries.
- Build instrumentation and observability capabilities that trace agent interactions, identify schema friction, measure execution quality, and improve orchestration success rates over time.
- Partner closely with API modernization, platform engineering, infrastructure, security, and SRE teams to define AI-ready integration patterns and ensure scalable production operations.
- Lead technical design reviews and influence architecture decisions across teams building AI-native systems on the platform.
- Define and document reusable patterns, reference implementations, SDKs, and paved paths that accelerate safe AI adoption across ABC Fitness.
What We're Looking For
- Typically 10+ years of software engineering experience, including deep expertise building distributed backend systems, APIs, orchestration layers, or platform infrastructure; 8–14 year principal range is most aligned for this level.
- Proven experience designing and operating high-scale backend systems with strong understanding of state management, concurrency, reliability, and distributed execution patterns.
- Strong understanding of how LLMs interact with tools and workflows, including schema design, structured outputs, predictable execution patterns, and orchestration reliability.
- Deep expertise in abstraction design and wrapping legacy or complex systems with clean, scalable, and reusable interfaces.
- Experience building workflow orchestration systems handling retries, failure states, long-running processes, and multi-step execution logic.
- Demonstrated ability to define technical standards and influence engineering direction across multiple teams through architecture, design review, and strong engineering judgment.
- Strong production mindset, including experience designing for observability, operational resilience, auditability, scalability, and performance optimization.
- Excellent cross-functional collaboration skills and the ability to work effectively with Engineering, Product, Platform, Infrastructure, Security, and SRE teams.
- A growth mindset and strong learning agility in a rapidly evolving AI and platform engineering landscape.
- A One Team orientation: able to enable other engineering teams, establish scalable paved paths, and create reusable platform capabilities rather than isolated solutions.
Nice to Have
- Experience with Model Context Protocol (MCP), ACP patterns, LangChain, LangGraph, or similar AI orchestration frameworks.
- Experience with middleware platforms, API gateways, service mesh architectures, or distributed control planes.
- Experience building internal developer platforms, SDKs, or shared engineering tooling.
- Experience implementing AI observability, tracing, prompt tracking, model evaluation, or orchestration telemetry systems.
- Familiarity with policy enforcement, safe model execution, permissioning systems, and AI governance patterns in production environments.
- Experience helping engineering organizations adopt new technical standards through documentation, mentoring, and reference implementations.
Compensation & Benefits
- Salary and benefits not disclosed in the posting.