Pyspark Spark Python Developer

Pyspark Spark Python Developer

Job Description

Skill Set: SQL, Python, Spark, pyspark, Airflow

Total Experience: 4.00 to 15.00 Years

No of Openings: 1

Job Post Date: 05/08/2026

Job Expiry Date: 27/10/2026

Domain: IT

Location

CHENNAI [India]

Job Reference No: 4099888

Job Summary

Seeking a Hadoop Developer with 8 years of experience in big data engineering to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies. The role involves batch/stream processing, data integration, performance tuning, and close collaboration with business teams.

Key Responsibilities

• Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs using Spark and/or Hive.
• Develop workflows using Airflow.
• Write complex SQL/HiveQL for data transformation and reporting needs.
• Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).
• Ensure data quality, lineage, and governance standards are followed.
• Monitor, troubleshoot, and tune jobs for performance and scalability.
• Create technical documentation and follow coding best practices.

Required Skills

• 8 years of hands-on experience in Hadoop/big data development.
• Strong knowledge of HDFS, Hive, Spark,
• String knowledge in Python/Scala
• Strong SQL and data modelling fundamentals.
• Experience with workflow orchestration tools (Airflow).
• Good understanding of Linux, shell scripting, and distributed systems.
• Familiarity with version control (Git) and Agile delivery practices.