About this role
EPAM Systems
We are seeking a Senior Data Engineer to design, build and run reliable ETL and ELT pipelines for enterprise data platforms. You will use Python, SQL and Apache Airflow with cloud services across Microsoft Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS). You will troubleshoot issues, improve delivery automation and partner with clients on requirements and trade-offs.
Design and build scalable ETL and ELT pipelines using Python, SQL and cloud-native services
Write, optimize and troubleshoot complex SQL for transformations, validation and performance
Create and maintain data models and warehouse structures that support quality, availability and scale
Orchestrate workflows in Apache Airflow including scheduling, monitoring and incident resolution
Support pipeline delivery and platform operations across Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS)
Improve continuous integration and continuous delivery (CI/CD), source control and automated deployment for data solutions
Investigate data, integration and infrastructure issues, drive root-cause analysis and implement fixes
Collaborate with clients and cross-functional teams to align on requirements, risks, dependencies and delivery plans
Hands-on experience building ETL and ELT pipelines and supporting data warehousing
Advanced SQL skill sets including optimization, data validation and complex transformations
Python proficiency for pipeline development and software engineering practices
Experience with distributed processing using PySpark or Apache Beam
Practical cloud experience with Microsoft Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS) for data workload
Workflow orchestration background with Apache Airflow or Google Cloud Composer
Knowledge of data modeling, data lifecycle management and data quality control
Clear communication with the ability to explain trade-offs, risks and solution options to stakeholders
Infrastructure as code exposure using Terraform or Kubernetes
Streaming familiarity with Kafka and event-driven patterns
Data integration tooling experience with Fivetran