Senior Data Engineer

Senior Data Engineer

Project description This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines. During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration. The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision. This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

Responsibilities

• Design, build, and maintain ETL/ELT pipelines on AWS
• Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS
• Develop and optimize data models for data warehouse/data lake (Redshift, S3)
• Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)
• Implement data quality checks, validation, and reconciliation processes
• Optimize pipeline performance and manage compute/storage costs
• Ensure data security and access control (IAM, KMS, encryption)
• Monitor and troubleshoot pipeline failures (CloudWatch, logging)
• Collaborate with data analysts, BI developers, and architects on data requirements
• Document data pipelines, architecture, and operational runbooks

Skills

• Must have

• 5+ years of experience

• Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)

• Strong SQL and experience with data warehousing (Redshift, dimensional modeling)

• Proficiency in Python or Scala for data engineering

• Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)

• Experience migrating data pipelines from on-prem or other cloud platforms

• Understanding of data governance, security, and access control on AWS

• Experience with CI/CD for data pipelines

• Nice to have

• AWS certification (Data Analytics Specialty or Data Engineer Associate)

• Experience with streaming data (Kinesis, MSK/Kafka)

• Familiarity with Infrastructure as Code (Terraform/CloudFormation)

• Experience with orchestration tools (Airflow, dbt)

• Experience in financial domain

• Languages

• English: C2 Proficient

• Seniority

• Senior

Location

Bengaluru, India

Req. VR-124691 Strategy & Design Consulting (SI) BCM Industry 30/09/2026