About the Role
The candidate will develop and manage data engineering solutions using Azure Databricks, ensuring reliable data pipelines and integration with analytics systems.
Responsibilities
- Design and implement data pipelines using Azure Databricks
- Develop and maintain ETL workflows for large-scale datasets
- Collaborate with data analysts and scientists to support analytics needs
- Optimize data storage and query performance in cloud environments
- Ensure data quality and integrity across processing stages
- Integrate data from multiple sources into centralized platforms
- Support the deployment of data solutions in production environments
- Monitor and troubleshoot data pipeline performance issues
- Apply data modeling techniques to support business reporting
- Work with infrastructure teams to manage cloud resources
- Document technical designs and system configurations
- Follow best practices for data security and compliance
- Participate in code reviews and technical planning sessions
- Contribute to continuous improvement of data engineering processes
- Assist in defining standards for data architecture and tooling
Nice to Have
- Experience with Azure Data Factory or similar orchestration tools
- Exposure to DevOps practices in data projects
- Knowledge of containerization and microservices architecture
- Familiarity with monitoring and logging tools for data pipelines
- Background in financial or enterprise-scale data environments
Compensation
Competitive salary based on experience and qualifications
Work Arrangement
Hybrid work model with flexibility for remote and office-based work
Team
Collaborative environment within a technology-driven team focused on data platforms and cloud integration
Technology Stack
- Primary tools include Azure Databricks, Azure Data Lake, Azure Blob Storage, and Azure Synapse Analytics
- Development practices emphasize automation, testing, and infrastructure as code
Project Focus
- Current initiatives involve building scalable data pipelines for business intelligence and machine learning use cases
- Projects aim to improve data accessibility and processing efficiency across departments
Visa sponsorship may be available for qualified candidates