Responsibilities
- Lead the end-to-end ownership of the quality engineering function, including strategy, vision, operating model, and long-term roadmap.
- Collaborate with data engineering teams to ensure data pipelines are reliable, observable, and meet business needs.
- Recruit, grow, and retain a high-performing team of quality engineers and analysts, including both onshore and offshore resources.
- Set performance expectations, deliver consistent feedback, and define career development paths for team members.
- Define, implement, and enforce quality engineering standards, processes, and KPIs across all business units.
- Foster a culture where quality is embedded throughout development, not treated as a final checkpoint.
- Develop a centralized knowledge base to document test strategies, reusable patterns, and standards, reducing dependency on individual contributors.
- Assess, select, and manage data quality and observability tools, whether commercial or custom-built, including frameworks native to Databricks.
- Integrate quality early in the data lifecycle through proactive design, automated testing, and CI/CD integration.
- Design and maintain automated checks for data freshness, completeness, accuracy, validity, volume, and schema changes.
- Establish enterprise-wide data quality frameworks, performance scorecards, and service level agreements for critical data assets.
- Engage directly in technical discussions, design reviews, and architecture planning to ensure quality is built into solutions from the start.
- Guide the creation of automated data validation systems using Python, PySpark, and SQL for regression testing, report validation, and pipeline checks.
- Implement quality gates within CI/CD pipelines to validate data attributes before production deployment.
- Architect and manage observability systems for data quality, including dashboards, alerts, thresholds aligned to SLAs, and escalation protocols.
- Lead response efforts for critical data incidents, including root cause analysis, post-mortem reviews, and corrective action planning.
- Improve mean time to resolution by implementing automation and standardized operational procedures.
- Contribute directly to high-impact technical initiatives such as proof-of-concept projects, automation frameworks, and complex debugging efforts.
- Work closely with data engineering to ensure pipeline resilience, monitoring, and alignment with business objectives.
- Engage with analytics, product, and business teams to ensure quality metrics support measurable business outcomes.
- Support artificial intelligence and machine learning programs by ensuring training and inference data meet quality standards.
- Partner with platform teams managing Databricks, Azure, and CI/CD tooling to integrate quality checks directly into orchestration and release workflows.
