Lead Databricks Data Engineer

Lead Databricks Data Engineer

Job Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer Experience: 10-12 Years Employment Type: Full-Time

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

Key

Responsibilities

• Data Engineering & Architecture

• Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.

• Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.

• Define enterprise data models, ingestion frameworks, and transformation strategies.

• Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.

• Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.

• Drive cloud data platform modernization and migration initiatives.

• Technical Leadership

• Serve as the technical lead for Databricks and Data Engineering programs.

• Establish engineering best practices, coding standards, and architecture guidelines.

• Conduct architecture reviews, code reviews, and solution assessments.

• Mentor and guide Data Engineers, Technical Leads, and junior team members.

• Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.

• Cloud & Platform Engineering

• Design cloud-native data solutions on Azure, AWS, or GCP.

• Implement secure, scalable, and resilient data architectures.

• Optimize costs, performance, reliability, and availability of data platforms.

• Lead CI/CD and Infrastructure-as-Code implementation for data workloads.

• Define and implement monitoring, observability, and operational excellence practices.

• Governance & Data Quality

• Establish data governance, lineage, cataloging, and security controls.

• Ensure compliance with organizational and regulatory requirements.

• Implement data quality frameworks and monitoring solutions.

• Collaborate with security and compliance teams to enforce data protection standards.

• Stakeholder & Delivery Management

• Partner with business leaders to understand strategic data requirements.

• Translate business requirements into scalable technical solutions.

• Drive delivery planning, risk management, and resource coordination.

• Present architecture decisions and technical recommendations to leadership teams.

• Required Technical

Skills

• Databricks

• Databricks Lakehouse Platform

• Databricks Workflows

• Delta Lake

• Unity Catalog

• Structured Streaming

• Databricks SQL

• Delta Live Tables (DLT)

• Databricks Asset Bundles

• Big Data Technologies

• Apache Spark

• PySpark

• Spark SQL

• Kafka

• Real-Time Streaming Architectures

• Data Lake & Lakehouse Architectures

• Cloud Platforms (Strong Experience in One or More) Azure

• Azure Databricks

• Azure Data Factory (ADF)

• ADLS Gen2

• Azure Synapse Analytics

• Azure Key Vault

• Event Hub

• AWS

• S3

• EMR

• Glue

• Redshift

• Lambda

• IAM

• GCP

• BigQuery

• Cloud Storage

• Dataflow

• Pub/Sub

• Composer

• Programming

• Python

• SQL

• PySpark

• Scala (Preferred)

• Shell Scripting

• DevOps & Automation

• Git

• Azure DevOps

• GitHub Actions

• Jenkins

• Terraform

• Docker

• Kubernetes

• Data Warehousing

• Dimensional Modeling

• Star & Snowflake Schema

• Data Vault

• Master Data Management

• Data Governance Frameworks

• Required Experience

• 12-15 years of overall IT experience.

• Minimum 6+ years of hands-on Databricks experience.

• 7+ years of experience in Spark/PySpark-based data engineering.

• Experience designing enterprise-scale Data Lake and Lakehouse solutions.

• Strong experience leading large data transformation and modernization programs.

• Experience handling structured, semi-structured, and unstructured data processing.

• Experience supporting analytics, AI/ML, and reporting ecosystems.

• Proven experience managing technical teams and driving architectural decisions.

• Preferred Qualifications

• Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.

• Databricks Certified Data Engineer Professional.

• Azure Data Engineer Associate.

• AWS Data Analytics Specialty.

• Google Professional Data Engineer Certification.

• TOGAF or Cloud Architecture certifications are a plus.

• Nice to Have

• MLOps and AI Engineering exposure.

• Snowflake and Databricks integration experience.

• Microsoft Fabric exposure.

• Data Mesh implementation experience.

• Data Governance tools such as Collibra, Alation, or Informatica.

• Expertise in Data Quality frameworks such as Great Expectations or Deequ.

• Key Competencies

• Solution Architecture

• Technical Leadership

• Stakeholder Management

• Program Delivery

• Data Strategy & Governance

• Team Mentoring & Coaching

• Problem Solving & Decision Making

• Communication & Executive Presentation Skills

• Performance Optimization

• Innovation & Continuous Improvement

• Technology Classification

• Technology Tower: Data & AI

• Practice Area: Data Engineering

• Primary Skill: Databricks Lakehouse Engineering

• Secondary

Skills

• Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

Responsibilities

• Management of DB Systems, Performance Tuning, Security, Compliance Standards,

Qualifications

• Graduate with 4 years BS or Professional Degree. 15 - 18 Years

• Required Skills

• Adapting To Change

• Attention To Consistency

• Group Policy Management

• Interpersonal Dynamics with Coworkers

• Microsoft Active Directory

• Networking Equipment

• Operating System Management

• Remote System Administration

• Results Orientation

• Time Management Skills

• Virtualization

• Working under Pressure

• Writing Communication Skills