Responsibilities
- Lead pre-deployment validation
- Produce model cards/transparency (intended use, limits, monitoring, rollback)
- Partner on data quality (provenance, representativeness, refresh cadence, privacy)
- Build LLM evaluation datasets/gold standards; run clinical red-teams and prompt testing; refine prompts/guardrails; assess RAG fidelity
- Train clinicians to oversee AI tools and adjudicate exceptions; create SOPs/checklists/escalations for LLM use
- Collaborate with engineering and product, security, compliance, legal, care and service line leaders
- Administer clinical AI governance (ethics, safety, regulatory, privacy, change control, documentation) in partnership with the Patient Safety Officer
- Define clinical use criteria and human-in-the-loop guardrails
- Maintain governance artifacts (charter, SOPs, decision logs) and audit readiness
- Support clinical QA; URAC/NCQA activities; policy/standards/rubrics management
- Lead/participate in event detection, RCA/FMEA, CAPA; education (CME/CE, targeted training)
- Support client/payer audits, RFPs, VBC metrics (HEDIS, MIPS), implementations, and escalations, grievances/appeals
- Lead and/or support cross-functional efforts to analyze clinical quality performance data, identify improvement opportunities, and translate value-based care insights into health plan design enhancements that optimize outcomes, member experience, and cost efficiency
- Conduct ongoing assessments of clinical services, applying continuous improvement and evidence-based methodologies to ensure service effectiveness, measurable patient outcomes, and alignment with organizational quality standards
Requirements
- MD/DO with active, unrestricted license
- 3+ years clinical practice; 2+ years in quality, patient safety, or clinical operations
- Experience with clinical review of LLM outputs; building eval datasets; red-teaming/prompt testing; RAG assessment
- QI/patient safety expertise (PDSA, Lean/Six Sigma, RCA/RCA2, FMEA, CAPA)
- Working knowledge of URAC, NCQA, CMS; familiarity with HEDIS/MIPS
- Proven track record of quality improvement within value-based care models
- Experience partnering with analytics/data science; interpret core performance metrics
- Excellent documentation and stakeholder communication
Nice to Have
- Experience building, deploying, and iterating, clinical AI agents
- Proficiency in business intelligence and analytics. (e.g., Tableau/Power BI; basic SQL/statistics)
- Experience in virtual care/care navigation and enterprise audits
Additional Information
- Remote-first; limited travel (~10-15%)
- This is a remote role; however, candidates based in or near the San Francisco Bay Area are preferred
