As a Senior AI Software Developer in Test, you will lead the development of advanced testing strategies tailored for AI-driven systems within a high-growth SaaS environment. You'll establish robust, scalable quality frameworks that ensure reliability across intelligent agents, LLMs, and complex RAG pipelines.
Key Responsibilities
- Design and implement automated evaluation systems to assess correctness, retrieval quality, and behavioral consistency across AI components.
- Integrate AI-specific testing into CI/CD workflows, enabling predictive flakiness detection, self-healing test scripts, and automated model validation.
- Develop deterministic and statistical methods to validate non-deterministic AI behaviors, addressing risks like hallucinations, bias, prompt injection, and concept drift.
- Build and maintain test frameworks covering the full AI lifecycle—from prompts and datasets to embeddings, model versions, and tool-calling sequences.
- Lead red-teaming efforts, implement fairness audits, and ensure compliance with ethical AI standards.
- Collaborate with data science, AI engineering, and product teams to simulate multi-agent scenarios and validate roadmap features.
- Define key quality metrics, including hallucination rates and context precision, and integrate observability into real-time dashboards.
- Mentor SDET team members, lead technical workshops, and promote best practices in AI testing across the organization.
Qualifications
You bring 7+ years in quality engineering within cloud-native environments, with at least 2 years focused on AI/ML systems. You have hands-on experience testing LLMs, agents, and retrieval-augmented pipelines, and are proficient in JavaScript/TypeScript with working knowledge of Python or Java.
Experience with evaluation tools such as Ragas, DeepEval, or LangChain ecosystem tools is essential. Familiarity with CI/CD platforms like Jenkins or GitHub Actions, observability with NewRelic, and statistical testing methods is required. ISTQB AI Testing certification is a plus, as is experience with performance tools like K6 or JMeter.
Environment & Benefits
This is a 100% remote role open to candidates in Colombia, offering a flexible schedule and strong work-life balance. You'll join a forward-thinking team that values innovation, collaboration, and accountability.
We offer competitive compensation above market average, performance bonuses, and a training budget to support your growth. Benefits include prepaid medicine, life and funeral insurance, internet and home office allowances, generous personal and sick leave, and recognition programs tied to tenure and impact.
Our culture thrives on inclusivity, knowledge sharing, and trust. We’re committed to ethical AI development and empower individuals to shape processes, influence design, and drive quality at scale.
Commitment to Inclusion
We believe diverse perspectives strengthen our product and team. We actively encourage applicants from all backgrounds and are happy to provide accommodations during the hiring process. If you need support, please contact our People Operations team.
