Lead Software Engineer - Pyspark, AWS, Python

Lead Software Engineer - Pyspark, AWS, Python

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorganChase within the Consumer and Community Banking, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.

Job responsibilities

• Lead the design, development, and maintenance of robust, scalable cloud-based data processing pipelines and infrastructure, ensuring adherence to engineering standards, governance frameworks, and industry best practices.
• Architect and refine data models for large-scale datasets, optimizing for efficient storage, high-performance retrieval, and advanced analytics while upholding data integrity and quality.
• Partner with cross-functional teams to translate complex business requirements into effective, scalable data engineering solutions that drive organizational value.
• Champion a culture of innovation and continuous improvement, proactively identifying and implementing enhancements to data infrastructure, processing workflows, and analytics capabilities.
• Define and execute data strategy, including the development of enterprise data models and the management of end-to-end data infrastructure—from design and construction to installation and ongoing maintenance of large-scale processing systems.
• Drive data quality initiatives, ensure seamless data accessibility for analysts and data scientists, and maintain strict compliance with data governance and regulatory requirements.
• Align data engineering practices with business objectives, ensuring solutions are both technically sound and strategically relevant.
• Author, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Expert in at least one distributed data processing framework (Spark). Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, if not Hadoop),
• Expert in at least one scheduling/orchestration tools ( Airflow, alternatively AWS Step Functions or similar) & Expert with relational and NoSQL databases. Expert in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar)
• Hands-on professional experience in one or more programming language(s), including Java or Python, proficiency in Python, SQL, and at least one additional language (e.g. Java or Scala) for data engineering tasks
• Hands-on experience utilizing Apache Spark for large-scale data processing, including developing and optimizing data pipelines, performing real-time and batch analytics, and leveraging Spark’s libraries for machine learning and data transformation to drive actionable business insights.
• Proficiency in microservices architecture, serverless computing and distributed cluster computing tools such as Docker, Kubernetes etc. Experience in one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.)
• Experience with test-driven development (TDD) or behavior-driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.
• Experience organizing and leading design workshops, coding sessions, and hackathons to promote a culture of excellence and innovation in data engineering. Expertise in architecting reusable, future-ready design patterns that address diverse use cases across the organization.
• Expertise in working with streaming platforms like Kafka, MQ etc.
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices

Preferred qualifications, capabilities, and skills

• Hands-on experience with Infrastructure as Code (IaC) tools, preferably Terraform; experience with AWS CloudFormation is also valued.
• Proficiency in cloud-based data pipeline technologies such as Spinnaker or similar platforms.
• Strong working knowledge of the Snowflake data platform.
• Experience in budgeting and resource allocation for data engineering projects.
• Proven ability to manage vendor relationships effectively.

Apply Now

Similar Jobs

Lead Software Engineer-Python , Java ,Gen AI, Architect

Mumbai, Maharashtra, India

Posted on 10/15/2026

Lead Software Engineer in Corporate Technology specializing in end-to-end architecture and delivery of secure, scalable full-stack platforms. Experienced with Java/Spring Boot, Python, MongoDB, and GenAI-enabled solutions.

Lead Software Engineer - Production Support, Python & SRE Bengaluru, Karnataka, India and 1 more

Posted on 10/23/2026 Lead and conduct resiliency design reviews, break up complex problems, and act as a technical lead for medium to large sized products

Sr Manager of Software Engineering - Java, AWS, React Mumbai, Maharashtra, India Posted on 10/12/2026 Drive impactful technology solutions as a Senior Lead Software Engineer, enhancing secure, scalable products for Finance Technology.

Software Engineer II - Fullstack (Java, React, AWS) Mumbai, Maharashtra, India Posted on 10/26/2026 Serve as an emerging member of an agile team to design and deliver market-leading technology products in a secure and scalable way

Careers

Students

How we hire

Work with us

Grow with us

I am an employee

Are You Still With Us?

It seems you've been gone for a while. For security reasons we will end your session automatically in 03:00 unless you would like to continue working.

End Session

Continue Working

Work Summary This summary is generated by AI Assist. Click inside the summary text box to make changes as necessary.

1000

Required

Discard

Add Summary