Endava is hiring a Data Engineer

Role Overview and Expected Deliverables The engineer will design, build and operate scalable, secure and analytics‑ready data pipelines and models across GCP BigQuery and AWS, delivering: - Airflow‑orchestrated data workflows and automations. - DBT Cloud transformations, testing frameworks and environment governance. - Robust CI/CD and Infrastructure as Code (IaC) deployments (GitHub Actions, Bitbucket, Bamboo, Terraform or CloudFormation). - Performance optimisation across data pipelines, SQL, BigQuery and AWS data services. - High‑quality technical documentation, production support, incident response and handover to internal teams. - Close collaboration with cross‑functional stakeholders including analytics, product and platform engineering. Must have skills GCP, BigQuery, Cloud Storage, Apache Airflow, dbt, Python, SQL, data pipelines, ingestion, transformation, Git/Bitbucket, data quality checks Good to have skills Dataplex, Collibra integration, CI/CD, Grafana, Splunk, Power BI data preparation, financial services data Core Technical Requirements Cloud & Platform Engineering: - Strong proficiency across AWS and GCP. - On AWS: IAM roles and policies, Lambda, S3, RDS, Redshift. - On GCP: BigQuery, workload optimisation, cost management. Data Engineering Tooling: - Hands‑on experience building data pipelines for data warehouse environments. - Strong SQL capability including analytical and window functions. - Advanced dbt Cloud experience (models, SCDs, tests, macros, release management). - Airflow workflow design including idempotency, backfills and monitoring. DevOps, CI/CD & IaC: - Experience with CI/CD (GitHub, Bitbucket, GitHub Actions, Bamboo). - Infrastructure as Code using Terraform or CloudFormation. - Familiarity with data quality frameworks and automated testing of pipelines. Programming & Modelling: - Proficiency in Python. - Strong data modelling expertise: facts/dimensions, SCD patterns, performance optimisation. Security, Data Governance & Operational Resilience: - Solid grounding in cloud security best practices and least‑privilege IAM. - Understanding of data access controls, regulated environments and governance expectations. Delivery Approach & Ways of Working - Experience working within Agile delivery frameworks. - Use of JIRA and Confluence for delivery management and documentation. - Proven track record collaborating effectively in multi‑disciplinary teams across engineering, product and analytics. Specialisations - BigQuery performance tuning and cost optimisation. - DBT Cloud governance (contracts, tests, macros, release processes). - Airflow workflow optimisation and monitoring. - Cross‑cloud pipelines and data movement (AWS ↔ GCP). - Security, IAM and data‑access design for regulated financial services environments. Experience & Other - 8 or more years experience in data engineering or equivalent technical roles. - Strong problem‑solving capability, comfortable troubleshooting pipeline failures and performance issues. - Strong communication and stakeholder management skills. - Self‑starter able to lead components independently while partnering closely with internal teams. - Experience supporting production environments, including incident response and root‑cause analysis. - All intellectual property rights retained by client