About this role
Senior/Lead Data Quality Engineer
Data Quality Engineering& 5 others
apply
FacebookLinkedInSend via email
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer. Find me a job
We are seeking a skilled Senior/Lead Data Quality Engineer to join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms.
Responsibilities
-
-
Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
-
-
Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
-
-
Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
-
-
Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
-
-
Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
-
-
Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues
Requirements
-
-
Minimum of 3 years of experience in Data Quality Engineering or a related field
-
-
Solid understanding of automated testing frameworks and data engineering principles
-
-
Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs
-
-
Experience in integrating testing frameworks with Azure DevOps and managing source control with Git
-
-
Knowledge of pipeline configuration and deployments utilizing Terraform
-
-
Familiarity with Jira
-
-
Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges
-
-
Fluency in English with a B2+ level
-
Nice to have
-
-
Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools
-
-
Prior experience in a data-intensive industry or with large-scale data environments
-
-
Proficiency in implementing solutions through Infrastructure as Code (IaC)