Responsibilities
- Develop comprehensive quality strategies, test plans, and automation coverage for machine learning-driven services and platform elements.
- Leverage large language models and AI-based methods to create, enhance, and sustain high-impact test cases for ML-integrated workflows.
- Construct scenario-driven test suites targeting AI capabilities, including adversarial inputs, boundary conditions, unclear prompts, and minority user cases.
- Lead quality engineering initiatives across cross-functional projects, guiding risk analysis, dependency coordination, and release preparedness.
- Build, implement, and support scalable test automation frameworks for backend systems, APIs, and ML inference pipelines using Python or Java.
- Create automated checks for ML and LLM outputs, evaluating ranking logic, score patterns, prompt and response quality, hallucination signals, and probabilistic model behavior.
- Investigate and resolve test failures, system anomalies, model discrepancies, and regressions in AI behavior to determine root causes.
- Conduct functional, integration, regression, API, end-to-end, performance, and reliability testing across distributed architectures.
- Enhance the stability of automated tests, minimize intermittent failures, and improve execution speed and efficiency.
- Work closely with development and machine learning teams to embed automated testing into continuous integration and deployment pipelines.
- Collaborate with multiple teams to define and promote scalable quality benchmarks, tooling solutions, and best practices.
Compensation
Competitive salary and benefits package commensurate with experience
Work Arrangement
Hybrid or remote options available based on role and location
Team
Part of a global quality engineering team focused on AI and machine learning systems within a major interactive entertainment platform
Responsibilities
- Define and complete quality strategies, test plans, and automation coverage for ML-powered services and platform components.
- Use LLMs and other AI-assisted techniques to generate, expand, and maintain high-value test cases for ML-powered workflows.
- Design scenario-based test suites for AI features, including adversarial prompts, edge cases, ambiguous inputs, and underrepresented user scenarios.
- Lead QE efforts for multi-functional projects, driving risk assessment, dependency management, and release readiness.
- Design, develop, and maintain scalable automation frameworks for backend services, APIs, and ML inference systems using Python and/or Java.
- Build automated validation for ML and LLM outputs, including ranking behavior, score distributions, prompt/response quality, hallucination indicators, and probabilistic model evaluation.
- Debug test failures, service anomalies, model inconsistencies, and AI behavior regressions to identify root causes and drive resolution.
- Perform functional, integration, regression, API, end-to-end, performance, and reliability testing for distributed systems.
- Improve automation reliability, reduce flakiness, and optimize execution efficiency.
- Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines and release workflows.
- Collaborate across teams to establish scalable quality standards, tooling, and guidelines.
Available for qualified candidates requiring sponsorship
