Responsibilities
- Designing, implementing and maintaining a fundamental data system processing 50+ billion rows of data per day
- Ensuring the data system is reliable, resilient, and low-latency
- Being responsible for data pipelines end to end: acquisition, loading, transformation, implementing business rules/analytics, and delivery to the end user
- Partnering closely with business stakeholders and engineering teams to understand their data requirements
- Delivering the necessary data infrastructure to support business activities
- Focusing on performance optimisation, improving data access times and reducing latency
Requirements
- Strong experience working in MPP platforms and/or Spark
- Experience with 'big data' such as weather forecasts, AIS pings, satellite imagery
- Proven ability to develop resilient and reliable data pipelines
- Strong coding skills in SQL and Python
- Deep understanding of how to leverage the AWS stack
- Strong communication skills to translate technical concepts to non-technical users
- Ability to turn business requirements into clear, actionable technical designs
