Senior Databricks Data Engineering Consultant

Data Technologies

• Multiple locations

• Hybrid

Job Title: Senior Databricks Data Engineering Consultant

Location: Hybrid – Indianapolis, IN Job Type: Full-Time

About SDG Group

SDG Group is a global consulting firm specialized in Data, Analytics & AI. We are committed to unlocking organizations’ potential and hidden value by offering in-depth expertise that empowers our clients’ business models to become successful data-driven enterprises.

Innovation is in our DNA. We constantly evolve our value proposition through cutting-edge laboratories, advanced analytics, and AI technologies — enabling our clients to accelerate their data transformation journeys.

SDG Group operates on a model of Triple Expertise — combining deep industry domain knowledge, technological proficiency, and strategic partnerships with leading software providers such as Snowflake, Matillion, Databricks, and Microsoft.

With proven experience across 700+ clients, we deliver Data & AI solutions tailored to diverse industries. Our unique IP includes over 50 technology accelerators, frameworks, and tools designed to fast-track implementation. We have strong alliances with major AI hyperscalers including Microsoft, Google, and AWS, and are recognized globally as a leader in Data and AI services.

Our AI practice is anchored by Orbitae®, SDG’s agentic AI brand, and INSIGHTGEN®, together with 50+ accelerators and industry agent frameworks that shorten the distance between fragmented enterprise data and deployed, governed AI agents — in the cloud, on-premises, or across hybrid and sovereign environments.

If you’re seeking to join a team fully specialized in Data, Analytics, and AI, we can’t wait to meet you.

Position Overview

As a Senior Databricks Data Engineering Consultant, you will lead the design, development, and delivery of modern data solutions for SDG Group’s clients, serving as a senior technical contributor and trusted partner across complex engagements.

Working as part of a consulting delivery team, you will partner closely with technical and business stakeholders to shape data requirements, guide solution design, and ensure reliable, scalable delivery that supports analytics, reporting, AI, and broader data-driven initiatives.

This opportunity requires deep, hands-on Databricks experience, with particular emphasis on Databricks SQL and SQL-based data engineering workloads. You will work across complex enterprise data environments, lead technical decisions, optimize transformations and pipelines, validate data, and help establish scalable patterns and best practices for downstream business and analytical use cases.

This role is well suited for an experienced data engineer who enjoys combining hands-on delivery with technical leadership, client partnership, mentoring, and ownership of complex data engineering workstreams.

Key Responsibilities

• Databricks Architecture & Development: Lead the design, build, testing, and maintenance of scalable data solutions within Databricks, with a strong focus on SQL-based transformation and data processing workloads.

• Advanced SQL Development: Design, troubleshoot, and optimize complex SQL queries and transformation logic used for data processing, validation, reconciliation, and analytical workloads.

• Technical Leadership: Own or lead data engineering workstreams, guide technical decisions, establish development approaches, and help ensure high-quality delivery across complex client engagements.

• Data Integration: Design and implement integrations across structured and unstructured data sources using modern ETL/ELT methodologies and data integration tools.

• Data Modeling & Warehousing: Design and review scalable data models and warehouse structures that support reporting, analytics, AI, and downstream applications.

• Cloud Data Solutions: Lead or contribute to the architecture and modernization of cloud-based data platforms using Databricks and related cloud technologies.

• Technical Delivery & Governance: Drive technical delivery while promoting strong practices for development, testing, documentation, deployment, maintainability, and governance.

• Data Quality & Performance: Lead troubleshooting and optimization efforts across data quality, reliability, scalability, and query or pipeline performance.

• Client Leadership: Partner directly with client stakeholders to clarify requirements, explain tradeoffs, recommend solutions, manage technical expectations, and connect engineering decisions to business outcomes.

• Team Leadership & Mentorship: Collaborate across multidisciplinary teams, review technical work, share best practices, and mentor less experienced engineers and consultants.

• Continuous Learning: Stay current with evolving Databricks capabilities, data engineering methodologies, and industry best practices while contributing knowledge and technical leadership to the broader SDG community.

Qualifications

• Professional Experience: Approximately 5–7+ years of experience in data engineering, data warehousing, data integration, analytics engineering, or a related technical discipline, with demonstrated ownership of complex technical delivery. Consulting or other client-facing experience is strongly preferred.

• Databricks Experience: Extensive hands-on professional experience working within Databricks is required, including production data engineering and analytical workloads.

• Databricks SQL: Advanced experience using Databricks SQL, including designing, troubleshooting, and optimizing complex queries and transformations for enterprise-scale workloads.

• SQL & Programming: Advanced SQL skills, including joins, CTEs, window functions, aggregations, complex transformations, performance tuning, and data validation. Strong working proficiency in Python or another relevant programming or scripting language.

• Data Engineering Fundamentals: Deep understanding of data pipelines, ETL/ELT processes, data integration, data warehousing, transformation patterns, and production data engineering best practices.

• Data Modeling: Strong understanding of dimensional modeling and modern data modeling concepts, with the ability to design scalable analytical data structures.

• Cloud & Data Platforms: Strong experience working with modern cloud environments and enterprise-scale data platforms, including architecture and modernization initiatives.

• Data Integration Tools: Experience with modern data engineering, orchestration, or transformation tools. Experience with Matillion and/or dbt is highly preferred.

• Data Quality & Performance: Demonstrated ability to lead data validation, reconciliation, troubleshooting, and performance optimization across multiple source and target systems.

• Problem Solving & Technical Judgment: Strong analytical and architectural judgment with the ability to evaluate tradeoffs, solve ambiguous technical problems, and translate complex business requirements into scalable solutions.

• Consulting & Communication Skills: Strong written and verbal communication skills with the ability to lead technical discussions, influence stakeholders, explain complex concepts clearly, and build credibility with clients.

• Leadership & Growth Mindset: A proactive, collaborative approach with demonstrated ability to mentor others, raise delivery standards, and continuously expand technical and consulting expertise.

Preferred Experience

• Databricks SQL Warehouses

• Delta Lake / Delta Tables

• Databricks Workflows and orchestration

• Unity Catalog and data governance

• Databricks query, cluster, and performance optimization

• Matillion

• dbt

• Python / PySpark

• Snowflake

• Experience leading enterprise-scale data migration or modernization initiatives

• Experience working within pharmaceutical, life sciences, healthcare, or another highly regulated industry

What We Offer

• Stability and Growth: Full-time employment in a growing high-tech industry with a meritocratic culture that supports career advancement and professional development.

• Competitive Compensation: Competitive salary and a 401(k) retirement plan with employer match.

• Comprehensive Benefits: Robust healthcare and benefits package including Medical, Dental, Vision, and Disability coverage.

• Work-Life Balance: 3+ weeks of paid vacation, a flexible holiday schedule, and hybrid working arrangements that balance on-site and remote work.

• Supportive Community: A collaborative environment where you will work alongside experienced professionals, learn from your colleagues, and contribute to meaningful client engagements.

• Focus on Technology: Continuous training and exposure to modern Data, Analytics, and AI technologies, supported by SDG Group’s partnerships with leading technology providers.

Team

Data Technologies

Role

Data Engineer (Data Technologies)

Locations

Chicago, Massachusetts, New Jersey, Pennsylvania

Remote status

Hybrid

Employment type

Full-time

Employment level

First /Mid-Le