Azure Databricks Engineer (ID: 4047)

Azure Databricks Engineer (ID: 4047)

As an Azure Databricks Engineer, you will:

• Design, develop, and maintain robust and reliable data processing solutions using Azure Databricks, Python, and PySpark.
• Build and manage both batch and streaming data processing workflows within Databricks.
• Design appropriate data models and processing architectures within the Databricks environment.
• Manage multiple parallel data processes working on shared source data.
• Develop scalable and maintainable data solutions while ensuring high standards of quality, reliability, and performance.
• Implement and maintain CI/CD pipelines using Azure and YAML.
• Work with Infrastructure as Code (IaC) using ARM/Bicep.
• Continuously improve existing solutions, reduce technical debt, and make pragmatic technical decisions.
• Collaborate effectively within an Agile engineering environment.

What You Bring to the Table:

• 8+ years of overall professional experience in data engineering/software engineering.
• Strong hands-on experience with Azure Databricks as a core technology.
• Deep expertise in Python, PySpark, and SQL.
• Proven experience designing and managing data processing solutions within Databricks.
• Strong experience handling both streaming and batch data.
• Experience designing data models specifically for Databricks-based processing environments.
• Solid experience with Microsoft Azure cloud services.
• Hands-on experience with CI/CD pipelines using YAML.
• Experience with Infrastructure as Code, particularly ARM/Bicep.
• Experience working in an Agile environment.

You should possess the ability to:

• Build stable, scalable, reliable, and maintainable Databricks solutions.
• Apply strong problem-solving and analytical skills to complex data-processing challenges.
• Make well-founded technical decisions in complex and evolving architectures.
• Work effectively with large-scale batch and streaming data processes.
• Manage and coordinate multiple parallel data-processing workloads.
• Identify and address technical debt proactively.
• Continuously improve data-processing solutions and engineering practices.
• Collaborate and communicate effectively with technical and cross-functional teams.
• Follow a structured and pragmatic engineering approach without unnecessary over-engineering.
• Balance engineering best practices with practical business and technical requirements.

What We Bring to the Table:

• An opportunity to work on complex Azure Databricks and data-processing solutions.
• Exposure to modern cloud, data engineering, CI/CD, and Infrastructure as Code practices.
• A collaborative Agile engineering environment focused on quality and continuous improvement.
• Opportunities to work with both batch and streaming data-processing architectures.
• A technically challenging environment where pragmatic, scalable, and maintainable solutions are valued