About this role
Lead Data Engineer – Databricks & Azure
- We are looking for a Lead Data Engineer with 5–7 years of experience in Azure, Databricks, and Microsoft Fabric to lead and mentor the data engineering team. The role involves implementing scalable data solutions using Databricks as the core platform, building modern data pipelines, and enabling seamless integration with Microsoft Fabric and Power BI.
Responsibilities & Requirements:
- Lead and mentor data engineers while driving hands-on technical delivery.
- Implement approved Databricks and Microsoft Fabric architectures and solutions.
- Develop metadata-driven bronze, silver, and gold data pipelines.
- Build reusable batch, incremental, CDC, and streaming ingestion pipelines.
- Strong hands-on experience with Azure Databricks, PySpark, SQL, and Delta Lake.
- Experience with Microsoft Fabric, OneLake, Direct Lake, and Power BI integration.
- Implement data quality checks, reconciliation, source-to-target mapping, and metadata management.
- Configure Unity Catalog for data governance, access control, auditing, and lineage.
- Implement CI/CD, automated deployments, and Datadog monitoring.
- Drive testing, documentation, operational readiness, and knowledge transfer.
Good to have:
- Experience with MDM and master data integration.
- Exposure to Azure DevOps or GitHub Actions.
- Experience with automated data testing and reconciliation frameworks.
- Experience working with onshore stakeholders and distributed engineering teams.
- Strong communication, problem-solving, and team leadership skills.
About Improving
Improving is a modern digital services company dedicated to positively changing the perception of the IT professional. We offer innovative solutions through consulting, software development, and training to help thousands of our clients achieve new heights in a competitive and ever-changing market.
As our company continues to grow, we are looking for enthusiastic thought leaders to join our team. Improving has a unique mix of passionate professionals who strive to grow and thrive in new ways. We are committed to establishing and maintaining an inclusive culture that allows all Improvers to bring their authentic selves to work each day. This is why we work hard to build inclusion and diversity in our workplace, so we can all do amazing things and succeed together.
Discover Our Company Culture
Join the Improving Team Apply For This Position