Responsibilities
- Take part in planning and carrying out the strategy to migrate data and ETL processes from SQL Server to Google Cloud Platform.
- Build and refine scripts and data pipelines for efficient data transfer and transformation into BigQuery.
- Create, implement, and support scalable, high-performance ETL and ELT workflows on GCP, replacing legacy systems.
- Use Google Cloud Composer or Apache Airflow to manage and schedule pipeline execution for consistent operations.
- Write and tune advanced SQL queries for BigQuery, considering migration-specific factors like partitioning and clustering.
- Help design and deploy efficient data schemas in BigQuery aligned with modern cloud infrastructure.
- Use Python to automate data migration, transformation, and integration with other GCP tools.
- Establish procedures to maintain data accuracy and consistency throughout migration.
- Work closely with existing database teams to understand current setups and support smooth transition.
- Engage with business stakeholders and data scientists to capture requirements and validate platform capabilities.
- Take part in code reviews and help enforce development standards and testing approaches for the new system.
- Monitor pipelines, troubleshoot issues, and resolve incidents during migration and after go-live.
- Keep current with GCP migration best practices and platform updates.
- Suggest enhancements and performance improvements for the new data platform on GCP.
Work Arrangement
Hybrid — Porto, Lisbon