Principal Azure Databricks Platform Engineer

Principal Azure Databricks Platform Engineer

Syneos Health is a leading fully-integrated life sciences services organization built to accelerate customer success. We partner with innovators at every point across the drug development and commercialization continuum, helping them navigate complexity, anticipate change and accelerate progress.

Every day we perform better because of how we work together, as one team, each the best at what we do. We bring together talented experts across a broad spectrum of business critical corporate functions. Every role plays an essential part in enabling our customers to achieve their goals. Our teams are agile, collaborative, and committed to delivering—for each other, for our customers, and ultimately for the people who rely on the services we support. Discover what your 25,000 future colleagues already know:

Why Syneos Health

• We are passionate about developing our people, through career development and progression; supportive and engaged line management; technical and therapeutic area training; peer recognition and total rewards program.
• We are committed to building an inclusive culture – where you can authentically be yourself. Central to this is our purpose – Driven to Deliver – which captures the passion of our colleagues to show up each day and shape solutions that have the ability to dramatically impact someone’s life.
• We are continuously building the company we all want to work for and our customers want to work with. Why? Because we know that when we bring together smart colleagues from across the world, we can shape the future of healthcare, driving impact for customers and defining the pace of patient progress.

Job Responsibilities

We are seeking an experienced Principal Azure Databricks Platform Engineer to lead the engineering, administration, automation, governance, and continuous modernization of our enterprise data platform built on Microsoft Azure and Azure Databricks.

This is a senior, hands-on technical leadership role requiring deep expertise in Databricks platform capabilities, beyond basic workspace provisioning or cluster management. The ideal candidate has a proven track record of operating enterprise-scale Databricks environments, implementing platform engineering best practices, building reusable automation, and enabling data engineering, analytics, and AI teams through secure, scalable, and governed platform services.

The successful candidate will combine strong Azure infrastructure expertise with advanced Databricks platform administration, proactively evaluate new platform capabilities, establish technical standards, and mentor engineers while driving platform modernization initiatives

This role focuses on engineering and operating the data platform, rather than developing ETL pipelines or implementing individual data engineering solutions

Key Responsibilities

• 1. Azure Databricks Platform Engineering & Administration

• Lead the design, implementation, administration, and optimization of enterprise Azure Databricks environments across development, staging, and production.

• Manage Databricks account-level and workspace-level configurations, including identity, permissions, networking, compute, workspace settings, and governance.

• Establish and enforce enterprise Databricks architecture, administration standards, platform security policies, and operational best practices.

• Design and manage compute policies, SQL warehouses, serverless capabilities, jobs compute, workload isolation, and resource utilization.

• Troubleshoot complex platform issues involving performance, configuration, permissions, networking, availability, and integration.

• Implement platform monitoring, audit logging, cost observability, and proactive operational controls.

• Enable multi-team and multi-data-product environments with appropriate security, isolation, and access boundaries.

• 2. Databricks Automation, Terraform & DevOps

• Develop and maintain reusable Infrastructure as Code (IaC) using the Databricks Terraform Provider and Azure Terraform Provider.

• Automate Databricks account and workspace administration, Unity Catalog resources, permissions, compute policies, and other platform configurations

• Implement Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles /DAB) for standardized deployment and management of Databricks resources.

• Establish reusable Terraform modules, deployment templates, CI/CD pipelines, and automated environment provisioning.

• Develop automation using Databricks CLI, REST APIs, Python, PowerShell, or other scripting technologies.

• Implement Git-based deployment practices, configuration management, infrastructure drift detection, and automated policy enforcement.

• Drive consistent, repeatable, and auditable deployments across environments.

• 3. Unity Catalog, Governance & Platform Security

• Lead enterprise implementation and administration of Databricks Unity Catalog including catalogs, schemas, permissions, external locations, storage credentials, and governance policies.

• Define scalable data access models using Entra ID groups, service principals, managed identities, and role-based access controls.

• Establish standards for data product isolation, least-privilege access, environment separation, ownership, and secure data sharing.

• Implement and enhance auditing, lineage, data discovery, and platform governance capabilities.

• Work with enterprise governance and security teams to integrate Databricks with broader data governance solutions such as Microsoft Purview.

• Ensure compliance with organizational security, privacy, and data management standards.

• 4. Databricks Platform Modernization & Emerging Capabilities

• Continuously evaluate and introduce new Databricks capabilities to improve enterprise platform functionality, developer experience, scalability, and cost efficiency.

• Lead platform enablement and establish operational patterns for capabilities such as:

• Unity Gateway / AI Gateway: Governed AI model access, routing, access controls, monitoring, and cost management.

• Lakebase: Managed operational database services, provisioning, security, and integration patterns.

• Lakeflow: Platform enablement, orchestration capabilities, deployment standards, and operational governance.

• Delta Sharing: Secure internal and external data-sharing capabilities.

• Databricks Apps and AI capabilities: Platform services and operational standards supporting AI and application workloads.

• Assess platform feature maturity, architectural fit, security implications, and operational readiness before enterprise adoption.

• Develop reference architectures, reusable platform components, and standardized onboarding patterns for engineering teams.

• 5. Azure Infrastructure, Security & Integration

• Design and manage the Azure infrastructure supporting enterprise Databricks environments, including networking, storage, identity, and security integrations.

• Demonstrate strong hands-on expertise in Azure networking, private endpoints, VNets, DNS, Azure Data Lake Storage, Key Vault, and Entra ID.

• Implement secure connectivity and integration between Databricks and enterprise Azure services.

• Design and enforce infrastructure security, high availability, backup, disaster recovery, monitoring, and cost optimization practices.

• Collaborate with enterprise cloud, security, networking, and operations teams to maintain reliable, compliant platform services.

• 6. Technical Leadership, Mentorship & Platform Standards

• Serve as a technical authority for Azure Databricks platform engineering and administration.

• Lead complex technical initiatives from architecture and design through implementation and operational handover.

• Mentor engineers through design reviews, technical coaching, knowledge sharing, and hands-on problem-solving.

• Establish engineering standards, reusable architectural patterns, operational runbooks, and platform best practices.

• Conduct technical assessments, identify platform gaps, and drive continuous improvement initiatives.

• Collaborate with data engineers, solution architects, security teams, and business stakeholders to understand platform requirements and deliver scalable solutions.

• Demonstrate strong ownership, independent decision-making, and the ability to influence technical direction across teams.

• Required Qualifications

Experience

• 8+ years of relevant experience in cloud engineering, platform engineering, infrastructure automation, or enterprise data platform technologies.

• Strong hands-on experience with Microsoft Azure, including infrastructure architecture, networking, storage, identity, and security.

• Significant hands-on production experience administering and engineering Azure Databricks platforms preferably 4+ years.

• Proven experience implementing Databricks platform capabilities beyond basic workspace creation, cluster provisioning, or notebook execution.

• Demonstrated experience leading technical initiatives, mentoring engineers, and establishing platform engineering standards.

• Essential Technical Skills

• Azure Databricks Platform

• Advanced Databricks administration, account and workspace configuration, compute governance, operational troubleshooting, and platform performance optimization.

• Deep understanding of Unity Catalog architecture, permissions, governance, security, and resource administration.

• Experience with Databricks SQL warehouses, serverless compute, workload management, and cost optimization.

• Strong knowledge of Databricks security models, service principals, identity management, and enterprise deployment patterns.

• Infrastructure as Code & Automation

• Hands-on experience with the Databricks Terraform Provider, including management of Databricks-specific resources.

• Strong Terraform skills covering reusable modules, state management, environment configuration, and automation.

• Experience with Declarative Automation Bundles (Databricks Asset Bundles), Databricks CLI, APIs, and CI/CD.

• Proficiency with Git-based engineering practices and automated deployments using GitHub Actions, Azure DevOps, or similar tools.

• Scripting and automation experience using Python, PowerShell, or Bash.

• Microsoft Azure

• Strong knowledge of Azure networking, private endpoints, DNS, ADLS Gen2, Key Vault, and Entra ID.

• Experience securing and operating Azure-hosted enterprise data platforms.

• Familiarity with Azure monitoring, logging, infrastructure governance, and disaster recovery practices.

• Leadership & Communication

• Proven ability to independently own complex technical solutions and make architectural decisions.

• Experience mentoring engineers and providing technical direction to teams.

• Ability to translate enterprise requirements into practical platform engineering solutions.

• Strong troubleshooting, analytical, communication, and stakeholder management skills.

• Preferred Qualifications

• Hands-on experience with newer Databricks capabilities such as Unity Gateway / AI Gateway, Lakebase, Lakeflow, Delta Sharing, Databricks Apps, or AI platform services.

• Experience implementing platform governance, automation, or reusable deployment frameworks for large enterprise Databricks environments.

• Experience with Databricks system tables, audit logs, platform observability, and FinOps.

• Experience designing multi-tenant or data-mesh-oriented enterprise data platforms.

• Familiarity with Azure-based data engineering and orchestration services such as Azure DataFactory, without necessarily being a hands-on ETL developer.

• Relevant Databricks, Terraform, or Microsoft Azure certifications.

• What Success Looks Like

The successful candidate will be able to:

• Independently own and improve the enterprise Databricks platform rather than relying on others for platform administration decisions
• Create standardized an