Responsibilities
- Lead production support for the Databricks platform, managing monitoring, alerts, and incident resolution across all live data workflows.
- Track and report on service level agreement metrics for data pipeline performance, ensuring transparency and accountability to internal and external teams.
- Discover and deploy optimizations to lower Databricks compute expenses, increase data throughput, and shorten processing times, measured through defined KPIs.
- Transition outdated ETL and ELT systems to Databricks, creating automated tools to minimize manual work and maintain continuous data flow during migration.
- Assist new clients during onboarding by setting up, verifying, and securing tenant-specific data pipelines that provide accurate, isolated data from launch.
- Develop high-efficiency data pipelines on Databricks to process and deliver large-scale ERP and CRM data across Azure and AWS cloud platforms.
- Manage Delta Lake architecture, including schema modeling, partitioning, data quality checks, and incremental update logic.
- Enforce security standards in Databricks, including access controls, secret handling, and compliance safeguards for sensitive enterprise data.
- Establish data quality monitoring and pipeline observability to ensure integrity of data feeding machine learning models and maintain prediction accuracy.
- Implement multi-tenant isolation patterns to guarantee secure and dependable data delivery for each enterprise client.
- Collaborate with Enterprise Architecture to align data pipeline design with the broader product ecosystem.
- Support global operations through on-call duties and after-hours incident response, meeting SLAs across diverse time zones.
- Maintain up-to-date technical documentation, runbooks, and architectural records to support team knowledge sharing and incident readiness.
- Apply CI/CD principles to data pipeline development, including version control, automated testing, and deployment automation for consistent delivery.
Benefits
- Comprehensive health insurance coverage for employees and their families
- 401(k) savings plan with company match
- Unlimited paid time off policy
- Paid leave for new parents
- Access to online legal consultation services
- Personal financial planning support
- Subsidized pet insurance coverage
- Exclusive travel and entertainment discounts through a corporate benefits program
- Reimbursement for health and wellness expenses
- Discounted travel options
- Access to educational materials and learning platforms
- Programs supporting career and personal growth
- Bonus incentives for employee referrals
- Merit-driven culture offering clear paths for learning, advancement, and career progression
Compensation
Not specified
Work Arrangement
Remote (Worldwide)
Team
Not specified
Not specified
