Third Way Health is hiring a Senior QA Engineer to own and evolve the quality and evaluation infrastructure for our AI-powered patient engagement platform. This is a high-impact individual contributor role where you will define how quality is measured across automated and manual workflows and build the tooling to make that definition actionable.

What You'll Do

Own and extend our multi-layered evaluation pipeline and verification portfolio, including deterministic quality checks, risk-factor heuristics, and LLM-graded transcript evaluation.
Advance our capabilities to evaluate end-to-end system performance across orchestrated agents, RAG-supported responses, and multi-party voice conversations.
Drive improvements to our observability stack to surface evaluation metrics, detect regressions, and enable data-driven quality decisions.
Build real-time monitoring and verification loops that catch issues in production interactions and feed back for system refinement.
Partner with ML engineers, product managers, and operations leads to translate real-world failure modes into automated checks.
Build and maintain adversarial and edge-case test suites for prompt injection resistance, guardrail robustness, and graceful degradation.
Champion “shift-left” quality practices by embedding evaluation criteria into prompt engineering workflows and defining acceptance criteria for new agent behaviors.
Contribute to the design of our QA pipeline orchestration to improve throughput, reliability, and developer experience.

What We're Looking For

5+ years of software or test engineering experience, with 3+ years focused on quality infrastructure for AI/ML or data-intensive systems.
Strong proficiency in Python for building test frameworks, evaluation pipelines, and API-level integration tests.
Demonstrated experience designing evaluation systems for LLM-based applications, with a clear understanding of the model as a generation layer, not the quality layer.
Familiarity with the architectural tradeoffs of relying on LLM outputs in production, including variance across model versions and prompt sensitivity.
Experience building extensible, rule-based validation systems that scale across a growing surface area of features.
Solid understanding of voice AI or conversational AI systems, including tool-calling patterns, transcript analysis, and interaction-level quality metrics.
Hands-on experience with observability and metrics instrumentation in production environments.
Excellent communication skills and the ability to collaborate effectively across engineering, product, and non-technical stakeholders.
Strong interest in healthcare innovation and building AI systems that meaningfully improve health outcomes.

Nice to Have

Experience building QA or evaluation systems in healthcare or regulated environments, with familiarity with standards such as HIPAA, GDPR, or FDA guidance.
Proven experience leading complex technical initiatives and mentoring junior engineers.
Experience building systems where quality guarantees live in the verification infrastructure rather than in any single model.
Familiarity with risk-scoring systems, anomaly detection, or production safety nets for autonomous AI agents.
Experience with AI safety testing, including adversarial evaluation, jailbreak testing, and bias detection in LLM outputs.
Hands-on experience with CI/CD pipelines for evaluation automation and infrastructure-as-code deployment patterns.
Experience with voice UI testing tools and platforms focused on evaluating speech generation and response quality.
Knowledge of accessibility testing and inclusive design principles.