Lead PySpark Developer

Project description

The Surveillance Enhancement Program (SEP) is a multi-year initiative aimed at developing and improving the surveillance platform to identify potential market misconduct, manipulation, and abuse. Our client fulfilled its commitment to the Financial Conduct Authority (FCA) by meeting Market Abuse Regulation (MAR) requirements for critical data sources in September 2017.

Subsequent projects have been launched to further enhance, expand, and complement the existing automated surveillance platform. This project will focus on alert enhancements, strategic data sourcing from FM order and trade lakes, and user interface and workflow improvements to comply with regulatory obligations such as Market Abuse Regulation and Dodd-Frank.

Additionally, the scope includes enhancements related to Control Room and Suitability Monitoring data, interfaces, and scenarios.

Responsibilities

• Develop software applications based on business requirements
• Maintain software applications and make enhancements according to project specifications
• Participate in requirement analysis, design, development, testing, and implementation activities
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality
• Ensure to complete the assigned development tasks within the deadlines
• Work in compliance with coding standards and best practices
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Skills

• Must have

• 10+ years of experience in data intensive Pyspark development, Window Functions, Performance Optimization, Spark Architecture, Salting, Column Pruning

• Must have experience in leading a Development Team

• Proven experience as a Core Python developer

• Good understanding of Gen AI

• Experience developing Classes, OOPS, exception handling, parallel processing

• Strong knowledge of DB connectivity, data loading, transformation, calculation

• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations

• Lambda Functions, Decorators

• Vector operations on Pandas dataframes /series

• Application of apply map, apply, map functions

• Concurrency and error handling data pipeline batch of size [1-10 gb]

• Ability to understand business requirements and translate them into technical requirements

• Ability to design architecture of data pipeline for concurrent data processing

• Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services

• Debugging application

• Nice to have

• Knowledge of web backend technology

• Django, Python, PostgreSQL.

• Apache Airflow

• Other

• Languages

• English: C1 Advanced

• Senioriy

• Lead

• Luxoft Benefits

• Relocation options

• Experience in an international environment

• Cross-cultural experience

• Feedback culture

• Regular appraisals

• Annual holiday - 20 or 26 days. The duration of the leave depends on the overall seniority

• Occasional leave - 1 or 2 days/ depending on the circumstances

• Child care leave - 2 days or 16 hours per year

• Absence due to force majeure - 2 days or 16 hours per year

• Maternity Leave - 20 weeks

• Parental Leave - 41 weeks

• Paternity Leave - 14 days

• Expert-led tech courses covering basic to advanced topics

• Internal instructor-led soft skills courses

• Comprehensive in-house self-learning resources for both soft and hard skills

• Access to external self-learning libraries like ProQuest eBook and Udemy for Business

• Cloud Programs: MS Cloud Academy, AWS Partner Academy, Google Cloud Academy

• Custom Learning Programs: upskilling, reskilling, technical mentorship

• Leadership Programs for Managers

• Multisport card

• Possibility to order Multisport card at the corporate rate for family members

• LuxGood Program: wellbeing seminars, contests, relaxation sessions, yoga sessions, etc.

• One Team Program: Buddy for each New Joiner; seminars, meeting and workplace space to support integration with local community and culture; “Hire me” workshops for partners

• Preferential banking offer

• Preferential car leasing offer

• Cafeteria program discounts for shops, cinema tickets, holiday offers

• Luxoft Social Benefit Fund: sport and recreation benefits, the possibility to receive financial support

• Private Healthcare Insurance with unlimited access to specialists

• Full dental support

• Travel Insurance

• Possibility to add private healthcare coverage for family members at the corporate rate

• Life insurance at the corporate rate for employees and family members, including payment of the basic package for the employee by the employer

• Reimbursement for corrective glasses

• Many fun social activities organized by the Luxoft team offline in your city

• Online entertainment events for whole company and local team events

• A workplace where you’re treated with respect within a multicultural team

• Rotation between projects and accounts

• New career opportunities

• The acquisition of rights to the above benefits depends on the form of cooperation. Benefits apply to those employed under a contract of employment.

• Please note that relocation is not available for all open positions. At Luxoft Poland it is possible to work remotely only from the territory of Poland.

• Options offered by the Polish government.

• Krakow, Poland

• Req. VR-125805

• Python

• BCM Industry

• 29/10/2026

• Req. VR-125805

• Krakow, Poland