Responsibilities
- Develop and maintain ETL data engineering processes using Python (PySpark) within Azure Synapse Analytics Notebooks, and/or Azure Synapse Analytics Pipelines, to ensure efficient data extractions, transformation, and loading.
- Apply expertise in data warehousing, understanding star schemas, facts, and dimensions, to design and build effective data storage structures in a Massively Parallel Processing (MPP) SWL Pool.
- Extract data from various sources, including REST APIs, SWL database tables, and CSV files.
- Utilize deep knowledge of Azure Synapse Analytics to design and optimize data notebooks/pipelines for scalability and performance.
- Contribute to the implementation and understanding of other Data Fabric concepts, such as data lakes, lakehouses, delta lakes, and data cataloging, to enhance data management capabilities.
- Collaborate with data architects to create data models and schemas that align with business requirements.
- Implement data quality checks and validation processes to maintain data accuracy and consistency.
- Identify and resolve performance bottlenecks and optimize ETL data notebooks/pipelines to meet SLAs.
- Monitoring ETL jobs, diagnose issues, and implement solutions to ensure data pipeline reliability.
- Maintain comprehensive documentation of ETL data engineering processes, data flows, and data transformations.
- Work closely with cross-functional teams to understand data requirements and provide support for data-related initiatives.
- Ensure data security and compliance with data governance and privacy standards.
Benefits
- Competitive salary and bonuses, including performance-based salary increases.
- Generous paid-time-off policy
- Flexible working hours
- Work remotely
- Continuing education, training, conferences
- Company-sponsored coursework, exams, and certifications
Work Arrangement
Remote (Worldwide)
Team
Structure: Cross-functional teams
Additional Information
- Flexible working hours
- Work remotely
- Continuing education, training, conferences
- Company-sponsored coursework, exams, and certifications
