Responsibilities
- Construct and sustain data ingestion workflows from multiple sources into the Data Lake.
- Develop, refine, and scale advanced data pipelines to support consistent data transfer.
- Build and manage reusable frameworks that streamline data pipeline development.
- Establish thorough testing protocols to validate data pipeline integrity and accuracy.
- Work closely with data analysts and scientists to deliver reliable, well-structured datasets.
- Maintain strong data governance, security standards, and regulatory compliance.
- Explore and adopt emerging technologies to enhance pipeline efficiency and scalability.
- Integrate and process data from heterogeneous systems such as Kafka, MQ, SFTP, databases, APIs, and file shares.
