Data Engineer at Distributed, Birmingham, £Contract Rate

Data Engineer

Birmingham - Onsite

6 months

InsideWho are we?

We're a software development company building the world's Elastic Workforce, reinventing work and challenging the assumption that a local team the best team.

We help businesses deliver technical projects better than ever before through our platform and on-demand Elastic Teams.

What's in it for you? Our mission is to create freelance jobs with more benefits than permanent.

Want to know more? read: https://distributed.co/about

About this role

We're partnering with a global organisation as they replace their legacy threat-hunting and intelligence platform with Elastic to enhance efficiency and speed. The new solution will simplify data searching and correlation across large-scale telemetry datasets, enabling quick detection of enterprise threats. With the capacity to ingest approximately 80TB of data daily, it will leverage AI and machine learning technologies. The implementation plan includes testing use cases to ensure effective performance in a production environment while understanding existing customisations.

Your Responsibilities

Data Ingestion Pipeline Design: Build and implement scalable data ingestion pipelines to handle high-volume data from diverse sources

Data Enrichment & Integration: Develop data enrichment processes and seamlessly integrate them with existing systems to enhance data quality and functionality

Data Storage Optimisation: Optimise data storage and retrieval systems for efficient querying, ensuring fast access to large datasets

Pipeline Monitoring & Management: Manage and monitor data pipelines using Elastic ingest pipelines and Apache Kafka, ensuring reliability and performance

ETL Process Development: Create and optimise ETL processes, including data modelling, to streamline data transformation and analysis

About You

We’re looking for passionate technologists who enjoy working in collaborative agile teams. You’ll need to be a clear, concise & engaging communicator with people on your team. We enjoy the big picture and the detail; we want people who excel at both.

• Elastic Ingest Pipelines: Proficiency in designing and managing Elastic ingest pipelines for efficient data processing.
• Apache Kafka: Experience with Apache Kafka for real-time data streaming and pipeline management.
• Data Modelling & ETL: Strong skills in data modelling and ETL (Extract, Transform, Load) processes for effective data transformation and analysis.
• Hadoop Ecosystem: Hands-on experience with Hadoop technologies for large-scale data processing and optimisation.
• Programming Languages: Proficiency in Python and Java for building and automating data workflows.
• SQL & NoSQL Databases: Expertise in both SQL and NoSQL databases for optimising data storage, querying, and retrieval.

About us

Distributed is proud to be an equal opportunities employer. Employees and contractors, as well as prospective employees and contractors, will all be treated equally and fairly. Distributed is committed to ensuring no less favourable treatment is experienced by any current or prospective employee because of any of the protected characteristics under the UK Equality Act 2010 or equivalent local equality legislation.

By submitting your application you give us permission to store and use the information from your CV and your answers to application questions.