Senior Data Engineer - Energy Markets at The ProEx Group, London Area, £Contract Rate (Outside IR35)

Senior Energy Markets Data Engineer

Location: London/Hybrid (3/4 Days in London Office)

Reports to: Delivery Lead

Level: Senior Manager / Lead Individual Contributor

Contract: Contract – Outside IR35

Please note – only candidates with recent Energy Markets experience will be considered for this role.

Role Purpose

We are seeking an exceptional Senior / Lead Data Engineer to design, build, and govern enterprise-grade data platforms that power analytics, AI, and operational decision-making.

This role blends hands-on engineering, technical architecture, and stakeholder leadership. You will work across business and technology teams to translate complex requirements into scalable, secure, and high-quality data products.

Key Responsibilities

Data Engineering & Platform Delivery

• Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure) using tools such as Azure Data Factory, Databricks, PySpark, SQL, and Python.
• Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems.
• Implement data quality monitoring, lineage, validation, and governance to ensure trusted data products.
• Build and optimise data models for analytical and AI workloads (e.g., Azure SQL, Synapse, PostgreSQL, Databricks Delta).
• Integrate data from complex ecosystems including APIs, geospatial services, SaaS platforms, on-prem systems, and third-party vendors.
• Develop production-ready pipelines with CI/CD using Azure DevOps, Git, and release pipelines.

Technical Architecture & Design

• Own high-level and low-level technical designs across data platforms, APIs, and integration services.
• Perform technical impact assessments for new data sources, features, and business requirements.
• Define scalable and secure architectures considering performance, cost, resilience, and regulatory requirements.
• Produce options papers, technical roadmaps, and cost-of-ownership models to support investment decisions.
• Ensure operational readiness by defining service wrap, monitoring, incident management, and interface agreements with system owners.

Analytics, AI & Business Enablement

• Collaborate with data scientists, analysts, product owners, and business leaders to define KPIs and data products that drive measurable value.
• Support feature engineering, model data pipelines, and AI use cases (e.g., demand forecasting, optimisation, clustering, ML pipelines).
• Enable business intelligence platforms (e.g., Power BI, Tableau) with trusted, well-modelled datasets.

Leadership & Stakeholder Management

• Lead, coach, and mentor data engineers (including graduates and diverse teams).
• Act as a trusted technical advisor to senior stakeholders and clients.
• Translate business needs into technical solutions and clearly communicate risks, trade-offs, and recommendations.
• Support sales, pre-sales, and roadmap definition for future data initiatives.

Required Experience & Skills

Experience

• Recent Energy Markets experience is a must
• 10+ years in data engineering, analytics, or data architecture roles.
• Proven delivery across enterprise-scale data platforms in regulated or complex environments (e.g., public sector, infrastructure, utilities, retail, FMCG).
• Experience operating in consulting or matrixed organisations with multiple stakeholders.

Technical Skills

• Advanced: SQL, Python, PySpark
• Cloud: Azure (ADF, Databricks, Synapse, Azure SQL, Blob Storage, Power Automate)
• Data Platforms: Databricks, relational & analytical data stores, API integration
• DevOps: Azure DevOps, CI/CD pipelines, source control
• Analytics & AI: Data modelling, feature engineering, ML data pipelines
• Integration: APIs, geospatial data, third-party systems, event-based pipelines

Architecture & Governance

• Data architecture and platform design
• Data quality, lineage, and governance frameworks
• Security, GDPR, and data compliance considerations

Soft Skills

• Exceptional stakeholder management and written/verbal communication
• Strong risk management and problem-solving
• Proven team leadership and coaching
• Ability to bridge business and technical teams

Be sure to follow our LinkedIn profile for more exciting global opportunities across a range of industries and sectors