Responsibilities
- Contribute to evaluation frameworks for clinical AI products, including defining acceptance criteria, test plans, clinically relevant rubrics, and performance benchmarks.
- Conduct structured sampling reviews of AI-generated outputs, assessing across criteria alignment with ABA principles and BACB ethical standards.
- Design and implement monitoring automation in collaboration with engineering, including automated quality checks, alerting thresholds, and drift detection for AI systems.
- Contribute to the creation and maintenance of reusable governance artifacts including system cards, monitoring playbooks, evaluation of SOP templates, and risk assessment documentation.
- Assist with pre-deployment validation testing for new AI products and features, executing defined testing and results of documentation.
- Participate in risk tiering assessments for new and evolving AI products, providing clinical and evaluative perspectives.
- Support incident investigation for AI-related issues if/as needed and in alignment with clinical best practice and CR Development Policy.
- Collaborate with product management and engineering teams to provide input during product planning and development, ensuring governance considerations are integrated early.
- Contribute to the development of evaluation methodology and standards for clinical AI use cases, documenting approaches and refining processes based on findings and emerging best practices.
- Contribute to AI literacy and training content for internal teams and customers, drawing on hands-on clinical evaluation experience.
- Stay current with developments in AI evaluation methodology, LLM behavior, ABA best practices, and relevant regulatory and ethical standards.
- Support CentralReach's thought leadership in responsible AI through contributions to conference presentations, professional publications, or applied research as opportunities arise.
Requirements
- Board Certified Behavior Analyst (BCBA) in good standing
- 2+ years of demonstrated experience in quality assurance, evaluation, data analysis, or technical assessment roles, preferably within healthcare or behavioral health settings.
- Demonstrated experience with large language models, generative AI, and their applications and limitations in clinical or professional contexts.
- Experience with prompt/context engineering, AI output evaluation, or AI workflow design.
- Experience with statistical sampling methodologies or healthcare quality assurance frameworks.
- Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act risk categories, AMA governance guidance).
- Experience developing evaluation rubrics, scoring frameworks, or quality measurement tools.
- Background in clinical informatics, research methodology, or systematic quality improvement.
- Demonstrated proficiency with data analysis tools and comfort working with quantitative and qualitative evaluation data.
- Strong analytical skills with the ability to develop systematic, evidence-based evaluation approaches.
- Familiarity with HIPAA compliance, healthcare data standards, and ethical considerations in AI.
- Excellent written and verbal communication skills with the ability to document methodology clearly and convey findings to clinical, technical, and leadership audiences.
- Demonstrated ability to work collaboratively across clinical, technical, and product teams.
- Experience in a remote work environment with strong self-direction and accountability.
