Responsibilities
- Design, develop, and maintain scalable and efficient data pipelines from a wide variety of sources
- Use dbt (Data Build Tool) to transform data through the various layers (Cleansed, Conformed, Presentation etc)
- Ability to write custom connectors (Python) and leverage out-of-the-box data-loading tools
- Operationalise enterprise data model by curating appropriate data models to service reporting, analytics, and data science use cases
- Embed real-time, automated data quality checks, validations, exception handling, and alerting across all data pipelines
- Work within Dagster as the primary orchestration and observability platform for all data pipelines
- Manage CI/CD workflows using GitHub, GitHub Actions, and Kubernetes
- Use zero-copy cloning and containerised on-demand development environments
- Implement and review RBAC policies within Snowflake and related platforms
- Embrace DataOps and a code-first approach to all data engineering work
- Identify and promote best practices in data engineering; recommend improvements to existing processes and systems
- Leverage AI-assisted development tools (Anthropic Claude, GitHub Copilot) to accelerate delivery and improve code quality
Requirements
- High standard of expertise in SQL and Python
- Specialist knowledge of modern approaches to delivering data projects using a code-first approach
- Excellent problem-solving skills and ability to embrace change
- Effective communication and collaboration skills
- Natural self-starter, with enthusiasm for learning and research
