Overview
A data organisation is moving from a collection of departmental pipelines toward a governed lakehouse platform that can support operational analytics and advanced modelling. The consultant will help establish the engineering foundations rather than simply build individual pipelines, with particular attention to scalability, data contracts and the practical realities of running a shared platform.
Responsibilities
- Define platform patterns for ingestion, transformation, storage and data access.
- Establish engineering standards for pipeline reliability, observability and deployment.
- Work with data architects to turn platform principles into implementable services.
- Review existing pipelines and identify areas where architecture creates unnecessary cost or operational risk.
- Mentor senior engineers through difficult platform design decisions.
Requirements
- 10+ years of data engineering or platform engineering experience.
- Deep knowledge of modern lakehouse architecture and distributed data processing.
- Strong Python and SQL skills.
- Experience with Databricks, Spark or comparable technologies.
- Proven understanding of data governance, lineage and platform reliability.
- Comfortable influencing architecture across several engineering teams.
Expertise
Apache Spark
Data Architecture
Data Governance
Databricks
Lakehouse
Python
SQL