Sr Data Engineer

A Senior Data Engineer at Illumina, located in Bengaluru, India (Manyata), is a hands-on, senior individual-contributor role with end-to-end ownership and leadership across multiple domains such as Supply Chain, Manufacturing, and Quality. You will design, build, and scale data products on our cloud lakehouse to power analytics, reporting, and AI/ML initiatives. The role requires strong proficiency in Python, SQL, and data modeling, with a solid understanding of distributed systems and system design. You will mentor engineers on our India-based team and collaborate with business, AI, and platform teams to deliver scalable, governed data products on modern cloud platforms such as Databricks and Snowflake.

Responsibilities

• Partner with business, AI, and platform teams to translate domain needs (e.g., SAP, Manufacturing, Quality) into well-modeled, governed, and scalable data products.
• Design, build, and scale end-to-end data products on Databricks, interoperating with Snowflake, following a medallion (Bronze/Silver/Gold) architecture from ingestion to analytics-ready datasets.
• Develop reusable Python frameworks, libraries, and standardized patterns for ingestion, transformation, validation, and publishing.
• Design robust data models (relational, dimensional, and lakehouse) and build performant, reliable distributed data pipelines using Spark, Delta Lake/Open table formats, dbt, and SQL.
• Embed data quality, reconciliation, validation, and governance into pipelines (Unity Catalog: lineage, RBAC, masking, PII handling).
• Monitor, alert, troubleshoot, perform root-cause analysis, and ensure SLA adherence for business-critical datasets.
• Adopt AI to accelerate development, testing, and optimization in data and analytics workflows.
• Act as a technical leader, set standards, lead code reviews, contribute to architecture decisions, mentor engineers, and communicate trade-offs to peers and stakeholders.

Requirements

• 8+ years of professional data engineering experience building and scaling data products on cloud platforms such as Databricks and/or Snowflake.

• Strong proficiency in Python, including reusable framework development using functional and object-oriented programming.

• Advanced SQL and strong data modeling skills (relational, dimensional, and lakehouse).

• Solid understanding of distributed systems and system design for large-scale data processing.

• Hands-on experience with open table formats (Delta Lake and/or Apache Iceberg) and big-data file formats (Parquet).

• Experience with Spark and modern ELT tooling (e.g., dbt).

• Experience with data observability, governance, security, and compliance practices (RBAC, PII, SOX).

• Demonstrated adoption of AI in data and analytics engineering workflows.

• Solid software engineering foundation (Git, REST APIs, JSON, CI/CD) on at least one cloud environment (AWS preferred).

• Strong written and verbal communication skills, with the ability to engage across business, AI, and platform teams and lead technical discussions.

• Bachelor's degree in Computer Science, Data Science, Information Systems, Engineering, Mathematics, or a related field, or equivalent demonstrable experience.

• Preferred qualifications

• Domain knowledge in SAP, Manufacturing, and/or Quality data and processes.

• Experience delivering in regulated environments (e.g., GxP / 21 CFR Part 11).

• Experience with Unity Catalog and lakehouse governance at scale.

• Snowflake-to-Databricks migration experience.

• Exposure to SAP data (ECC/S/4HANA, CDS views) and SAP data integration patterns.

• Databricks and/or dbt certifications.

• Familiarity with Power BI/Tableau for BI and conversational analytics.

• Competencies

• Ownership: end-to-end accountability for data products from design through production support.

• Engineering craft: clean, reusable, well-designed code with auditable, reliable data.

• System thinking: solid design judgment across scale, performance, and cost.

• Technical leadership: raising the bar through standards, reviews, and mentorship.

• Learning velocity: quick to adopt new tools, including AI, and apply them pragmatically.

• Cross-functional communication: effective collaboration with business, AI, and platform teams, clearly explaining trade-offs.