The QA Lead, AI Agent will define and execute the quality strategy for the AI Agent product line, ensuring AI-driven interactions are dependable, precise, and human-centered at scale. This role blends technical leadership with strategic QA innovation, focusing on testing non-deterministic large language models and expanding QA operations in tandem with product growth.
Responsibilities
- Design and manage a scalable, end-to-end testing framework for AI agent systems using LLM-based methods such as automated simulations, LLM-evaluated rubrics, and adversarial testing techniques.
- Collaborate with Forward Deployed Engineers and Product Managers to resolve deployment issues, identify system bottlenecks, and rapidly develop new test cases in response to live customer environments.
- Perform manual user acceptance and voice-call testing to capture the end-user experience, identifying and communicating subtle issues like tone, clarity, or lack of empathy to technical and client teams.
- Lead and grow a QA pod, establishing best practices, communication protocols, and shared knowledge systems to scale QA effectiveness alongside product expansion.
Requirements
- Minimum of 5 years in Quality Engineering, Technical QA, or Deployment roles, preferably in AI or high-growth SaaS environments.
- Strong systems thinking with technical curiosity about LLMs, including understanding of RAG, prompt engineering, and multi-turn conversational logic.
- Demonstrated leadership in managing complex end-to-end technical initiatives across teams, improving collaboration and efficiency between QA, Engineering, and Product.
- Exceptional ability to detect edge cases and proactively address potential failures, with a fast, action-oriented approach to problem-solving.
- High level of empathy and a consultative mindset, capable of representing the human aspects of customer support interactions.
- Adaptability in fast-moving startup environments, with a track record of executing effectively amid ambiguity and a hands-on attitude.
Nice to Have
- Experience with Contact Center as a Service (CCaaS), telephony systems, or Speech-to-Text and Text-to-Speech technologies.
- Background in Conversation Design or Software Development in Test (SDET) roles.
- Proven experience managing teams with direct reports.
Tech Stack
LLM, RAG, prompt logic, multi-turn conversational flows, no-code test and evaluation tools
Benefits
- Comprehensive medical, dental, and vision insurance with options for individuals and families
- Flexible paid time off policy allowing employees to take leave as needed
- Paid parental leave for all new parents
- Retirement savings plan to support long-term financial planning
- Budget for remote work setup to support a functional home office
- Monthly stipend for wellness and communication expenses
- Onsite meal program and commuter benefits for in-office employees
Compensation
Base salary plus bonus and equity; total compensation is competitive and adjusted based on location, market standards, and individual performance. Equity and bonus are included in the overall package.
Work Arrangement
Hybrid model with support for remote work, including a dedicated budget for home office setup; in-office benefits available for those working onsite.
Team
Leads a pod of QA analysts and external partners; works closely with Forward Deployed Engineers and Product Managers to align quality with deployment and product goals.
- Comprised of leading experts in artificial intelligence and machine learning
- Operates in a fast-moving, high-impact startup environment
- Committed to transforming the workforce through AI innovation
- Offers people-first benefits and prioritizes work-life balance
- Focused on excellence and cutting-edge deployment of AI technologies
Additional Information
- Founded by Sebastian Thrun, Ping Wu, and Tim Shi
- Executive leadership includes former executives from Google, OpenAI, and AT&T
- Backed by prominent investors such as Andreessen Horowitz, Greylock Partners, Sequoia, and former AT&T CEO John Donovan
- Clients include Intuit, Cox Communications, Hilton, and CarMax
- Recognized by Forbes and Bain Consulting as one of the world's leading private AI companies
- All official recruiting communications originate from the @cresta.ai domain; candidates should disregard any impersonation attempts
- No information provided regarding visa sponsorship or relocation assistance
No information provided
