Lead Data Engineer

About the Role

The Lead Data Engineer is a pivotal member of MetLife's Data & Analytics USB Eligibility team, responsible for designing and delivering data architecture, data hubs/products, and governance-aligned solutions. This role serves as a big data development expert, leading end-to-end ETL, data warehousing, and reusable components using cloud and big data technologies. You will partner with architects, tech leads, and business teams to enable data products across LOBs and drive scalable data enablement. What You'll Do

• Design and solution end-to-end data architecture for data hubs/data products, from source systems to consumption.

• Oversee the design and management of data solutions to ensure data is stored, processed, curated, and utilized effectively.

• Own and build a reusable data pipeline utilizing Big Data and Azure/Databricks for Eligibility Program data products.

• Ingest large volumes of data from various platforms for analytics and write high-performance, maintainable ETL code.

• Leadership: Lead and mentor a team of data engineers, ensuring efficient data flow with defined processes and tools.

• Collect, store, process, and analyze large datasets to implement ETL processes.

• Develop reusable frameworks to reduce development effort and costs.

• Utilize CI/CD pipelines to automate delivery of data solutions with reliability and scalability.

• Use Azure Databricks and other cloud tech to enable data product solutions.

• Develop quality code with performance optimizations at development stage.

• Eagerness to learn new technologies and work on cutting-edge cloud tech.

• Partner with Tech, Business, BI, and Data Science teams to create reusable data products.

• Work with global teams to drive delivery and suggest improvements.

• Track and report KPIs for solution delivery and data quality.

• Communicate use cases, solutions, and impact to business stakeholders and mid/senior management.

• Optimize reusable frameworks and Spark jobs for performance and cost efficiency at scale.

• Interact with business stakeholders to gather requirements and implement solutions.

• Analyze data using SQL and exploratory tools across platforms like Bigdata, Oracle, SQL Server, Databricks, etc.

• Collaborate with IT, business, and architects to define technical design requirements. What We're Looking For

• 8+ years of Data Solutions, development, and delivery experience with 4+ years in Azure/Databricks.

• Proficiency with SQL, Spark and/or Scala/Python and performance tuning.

• Hands-on expertise in Big Data (Hive/HBase), Azure Databricks, Azure Functions, Cosmos DB and/or Data Factory.

• Strong experience in building/designing data warehouses/data stores for analytics on Cloud (real-time and batch).

• Design, build, deploy robust data ingestion and curation pipelines using Azure Data Factory, Spark (Scala or Python), Azure Databricks, Delta Lake.

• Scripting experience (shell/bash/PowerShell).

• Strong SQL knowledge for data anomaly detection and data quality assurance.

• Experience implementing data governance and data quality with enterprise tools.

• Drafting functional/technical requirements and data-flow diagrams.

• Writing validation scripts to validate data, integrations, and ETL transformations.

• Problem solving and excellent written and verbal communication. Nice to Have

• Python expertise and Azure Functions with Python/Node.js.

• Databricks certifications and/or Microsoft Azure Certifications.

• Experience using Event Hub for data integrations.

• Eagerness to learn new technologies and ship to production.

• Hive database management and performance tuning (partitioning/bucketing).

• Ability to interact with senior IT/business leaders. Compensation & Benefits

• Salary range: $120,000 - $175,600 per year. This role may be eligible for annual short-term incentive compensation. All incentives and benefits subject to plan terms.