Data Engineer - Databricks, Apache Spark, Scala/Python at TSG, London, £Contract Rate

We are seeking a highly skilled Data Engineer with expertise in Databricks, Apache Spark, and Scala/Python to join our dynamic team in London on a contingent worker basis. The ideal candidate will have hands-on experience in data engineering, along with experience or a strong understanding of data governance. You will play a key role in designing and implementing robust data pipelines and ensuring compliance with data governance best practices.

Key Responsibilities:

• Develop, optimize, and maintain big data pipelines using Databricks and Apache Spark.
• Write efficient, scalable, and maintainable Scala/Python code for data processing and transformation.
• Collaborate with data architects, analysts, and business teams to understand data requirements.
• Ensure data quality, lineage, and security within data platforms.
• Work with structured and unstructured data from multiple sources, integrating them into a unified data lake.
• Optimize performance of big data workflows and troubleshoot data processing issues.
• Document technical processes, best practices, and governance policies.

Key Requirements:

• 3+ years of experience in data engineering with a focus on Databricks and Apache Spark.
• Strong programming skills in Scala and/or Python.
• Experience or a strong understanding of data governance, metadata management, and regulatory compliance.
• Knowledge of cloud platforms (Azure, AWS, or GCP) for big data processing.
• Experience with Delta Lake, Lakehouse architecture, and data cataloging.
• Strong understanding of ETL/ELT pipelines, SQL, and NoSQL databases.
• Familiarity with data security, privacy standards (GDPR, ISO, etc.), and access controls.
• Excellent problem-solving and communication skills
• Mandatory Banking Experience