3781 IT & Software Developer jobs in the UK
Lead PySpark Engineer
£42,000 - 75,000
Randstad Technologies
Throgmorton Avenue 10, London
£42,000 - 75,000
Requirements
Must:
- Expert in PySpark
- Proficient in clean Python code and SOLID principles
- Experience with AWS services: EMR, Glue, S3, Athena, IAM, Lambda
- Strong knowledge of Data Modeling, including SCD Type 2, Fact/Dimension tables, Data Vault/Star Schema
- Proficient in reading and debugging SAS (Base, Macros, DI Studio)
- Familiar with Git-based workflows, Jenkins/GitLab CI, and Terraform
- Experience in engineering leadership and team collaboration
- Strong problem-solving and performance tuning skills
Responsibilities
- Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue
- Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring
- Optimize Spark execution (partitioning, shuffling, caching) to ensure cost-efficient processing of massive financial datasets
- Implement rigorous CI/CD, unit testing, and data reconciliation frameworks to ensure penny-perfect accuracy
Description
We are seeking a PySpark Engineer Lead to drive the migration of legacy SAS analytics to a modern cloud-native PySpark ecosystem on AWS. This role involves refactoring complex procedural logic into scalable, production-ready distributed pipelines within a Tier-1 financial services environment. We foster a culture of engineering excellence and innovation, aiming to deliver high-quality results. Join our dynamic team and take on challenging projects in a fast-paced, collaborative environment.
Something wrong or incorrect with this job? Tell us in the chat 💬 on the right ➡️
You can find DevOps salaries in the UK here.
How many DevOps jobs are in the UK?
Currently, there are 3781 DevOps openings. Check also: Cloud jobs, AWS jobs, Azure jobs, GCP jobs, Kubernetes jobs, Docker jobs, Terraform jobs - all with salary brackets.
Is the UK a good place for DevOps?
The UK is one of the best countries to work as a DevOps. It has a vibrant startup community, growing tech hubs and, most important: lots of interesting jobs for people who work in tech.
Which companies are hiring for DevOps jobs in the UK?
Arctic Shores, SwapTix, Element Materials Technology, HM Land Registry, Companies House, FCDO Services, TBG Solutions Ltd among others, are currently hiring for DevOps roles in the UK.
The company with most openings is Experis as they are hiring for 111 different DevOps jobs in the UK. They are probably quite committed to find good DevOps.
The company with most openings is Experis as they are hiring for 111 different DevOps jobs in the UK. They are probably quite committed to find good DevOps.