Responsibilities
- Lead the design, development, and implementation of high-performance, scalable, and reliable data pipelines and ETL/ELT processes using Azure Data Factory, Databricks, and other Azure data services.
- Architect and manage data solutions within the Azure ecosystem, including Azure Data Lake Storage, Databricks, Databricks DLT and Streaming and Event Based Architectures.
- Drive the adoption of best practices for data governance, data quality, data security, and data lineage.
- Collaborate closely with data scientists, analysts, and other engineering teams to understand data requirements and translate them into technical solutions.
- Optimise data processing performance and cost efficiency on Azure Databricks, leveraging Spark capabilities effectively.
- Develop and maintain robust monitoring, alerting, and logging for data pipelines.
- Mentor and provide technical guidance to junior and mid-level data engineers, fostering a culture of continuous learning and improvement.
- Evaluate and recommend new data technologies and tools to enhance our data platform capabilities.
- Contribute to the overall data strategy and roadmap, ensuring alignment with business objectives.
- Troubleshoot and resolve complex data-related issues in a timely manner.
Requirements
- Extensive experience as a Data Engineer, with a significant portion in a principal or lead capacity.
- Deep expertise in Azure data platform services, including: Azure Databricks (extensive hands-on experience with Spark, Python/Scala for real time data processing).
- Azure Data Factory (maintaining complex data pipelines).
- Azure Data Lake Storage.
- Azure SQL Database and/or Azure Synapse Analytics.
- Strong proficiency in SQL.
- Exposure to Infrastructure as Code and CICD deployments.
- Excellent programming skills in Python (Scala is a strong advantage).
- Proven experience with data modelling, schema design, and data warehousing concepts.
- Solid understanding of data governance, data quality, and data security principles.
- Experience with version control systems (e.g., Git).
- Strong problem-solving abilities and a methodical approach to complex technical challenges.
- Excellent communication and interpersonal skills, with the ability to articulate complex technical concepts to both technical and non-technical stakeholders.
- Proven ability to lead and mentor other engineers.
Nice to Have
- Experience with real-time data streaming technologies (e.g., Azure Event Hubs, Kafka).
- Knowledge of CI/CD pipelines for data solutions.
- Familiarity with containerisation technologies (e.g., Docker, Kubernetes).
- Experience with other cloud platforms (AWS, GCP) is a plus.
- Relevant Microsoft Azure certifications (e.g., Azure Data Engineer Associate)
