Responsibilities
- Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, Azure Data Factory, Azure Event Hub, Azure SQL, and related Azure services
- Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads
- Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance
- Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns
- Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities
- Deploy and Configure CI/CD pipelines utilizing Azure DevOps and Databricks Asset Bundles (DAB), Infrastructure as Code (Terraform/Bicep where applicable), deployment automation, release management, and environment promotion strategies
- Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices
- Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables
- Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation
- Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture
- Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure
Requirements
- Fluent in English
- Experience with Azure Databricks
- Experience with Delta Lake
- Experience with Unity Catalog
- Experience with Azure Data Lake Storage Gen2
- Experience with Azure Data Factory
- Experience with Azure Event Hub
- Experience with Azure SQL
- Experience with Microsoft Entra ID (Azure AD)
- Experience with Role-Based Access Control (RBAC)
- Experience with Privileged Identity Management (PIM)
- Experience with data encryption
- Experience with private networking
- Experience with managed identities
- Experience with Key Vault integration
- Experience with secure access patterns
- Experience with CI/CD pipelines using Azure DevOps
- Experience with Databricks Asset Bundles (DAB)
- Experience with Infrastructure as Code (Terraform/Bicep)
- Experience with performance tuning of Databricks workloads
- Experience with Spark optimization
- Experience with cluster sizing
- Experience with workload isolation
- Experience with caching strategies
- Experience with partitioning
- Experience with Delta optimization
- Experience with Photon
- Experience with cost optimization best practices
- Experience with monitoring using Azure Monitor, Log Analytics, Application Insights
- Experience with Databricks system tables
Work Arrangement
Remote (Worldwide)
Additional Information
- Fluent in English
