Responsibilities
- Collaborate with data scientists and analysts to identify data requirements and build efficient data pipelines
- Design, develop, and manage data pipelines for ingestion, processing, and transformation using cloud platforms and modern data tools
- Build and support data solutions in Azure with technologies such as Data Factory, Synapse, Databricks/Spark, Polars/Pandas, or Fabric
- Apply data validation and cleansing methods to maintain high standards of data quality, accuracy, and reliability
- Optimize data pipeline performance for scalability, efficiency, and cost efficiency
- Monitor pipeline operations and troubleshoot issues to ensure data consistency and system availability
