Azure Databricks Platform Engineer at Department for Education, Remote, 8 Months, £Contract Rate

Role Overview

We are seeking a Databricks Platform Engineer to support the Analytical Data Access (ADA) programme, a large-scale transformation initiative focused on migrating legacy SQL Server data into a centralised Delta Lake using Azure Data Factory (ADF) and Databricks.

This role will primarily focus on Databricks platform administration, including user access management, governance, and day-to-day operational support, while also providing support for data engineering activities across the wider platform.

Key Responsibilities

Databricks Platform Administration 

•    Administer and maintain the Azure Databricks workspace, ensuring platform stability, security, and performance.

•    Manage user access and permissions (RBAC, groups, roles, and service principals) on a day-to-day basis.

•    Support user onboarding and offboarding in compliance with data governance and security requirements.

•    Monitor workspace usage, cluster performance, and resource consumption.

•    Implement and maintain access controls aligned with organisational governance standards.

•    Manage secrets, tokens, and secure configurations within Databricks.

•    Support audit activities, including access reviews, compliance reporting, and security assessments.

Data Engineering & Integration 

•    Develop, maintain, and support data pipelines using Azure Data Factory (ADF).

•    Support the ingestion and transformation of data from legacy systems into the Delta Lake environment.

•    Work with structured and semi-structured data within a lakehouse architecture.

•    Assist with the optimisation of Databricks notebooks, jobs, and workflows.

•    Ensure the quality, reliability, and performance of data pipelines and integrations.

Collaboration & Stakeholder Support

•    Work closely with Product Owners, Business Analysts, Data Owners, and Governance teams.

•    Support the onboarding of new analyst teams onto the ADA platform, ensuring smooth access and setup.

•    Provide guidance and best practice recommendations for using Databricks and Delta Lake.

•    Contribute to platform documentation, including process guides, access procedures, and operational runbooks.

Governance, Security & Compliance

•    Ensure adherence to data governance, security, and compliance standards.

•    Support the implementation of Data Sharing Agreements (DSAs) and access approval processes.

•    Maintain accurate documentation of data access, permissions, and lineage where required.

•    Ensure platform operations align with Agile delivery principles and GDS standards, where applicable.

Key Skills & Experience

Essential

•    Strong experience administering Azure Databricks environments.

•    Experience managing user access, roles, permissions, and RBAC controls.

•    Hands-on experience with Azure Data Factory (ADF).

•    Understanding of Data Lake, Delta Lake, and Lakehouse architectures.

•    Experience supporting users on live data platforms.

•    Knowledge of data governance, security, and access management principles.

•    Strong analytical, problem-solving, and stakeholder management skills.

Desirable

•    Experience working within Agile delivery environments.

•    Knowledge of Azure Active Directory (Azure AD/Entra ID) and identity management.

•    Familiarity with Unity Catalog and other data access governance tools.

•    Experience with SQL, Python, and/or Apache Spark.

•    Experience supporting data migration and cloud transformation programmes.

Please be aware that this role can only be worked within the UK and not Overseas.