Mindrift is seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios for LLM agents. This role blends manual scenario validation, automated test thinking, and collaboration.

What You'll Do

Review and validate test scenarios from Evaluation Writers.
Spot logical inconsistencies, ambiguities, or missing checks.
Suggest improvements to structure, edge cases, or scoring logic.
Collaborate with infrastructure and tool developers to automate parts of the review.
Create clean and testable examples for others to follow.

What We're Looking For

Strong QA background (manual or automation), preferably in complex testing environments.
Understanding of test design, regression testing, and edge case detection.
Ability to evaluate logic and structure of test scenarios (even if written by others).
Experience reviewing and debugging structured test case formats (JSON, YAML).
Familiarity with Python and JS scripting for test automation or validation.
Clear communication and documentation skills.
Willingness to occasionally write or refactor test scenarios.

Nice to Have

Experience testing AI-based systems or NLP applications.
Familiarity with scoring systems and behavioral evaluation.
Git/GitHub workflow familiarity (PR review, versioning of test cases).
Experience using test management systems or tracking tools.

Technical Stack

Python, JS (JavaScript)
JSON, YAML
Git/GitHub

Team & Environment

You will collaborate directly with writers and engineers.

Benefits & Compensation

Contribute on your own schedule, from anywhere in the world.
Get paid for your expertise, with rates that can go up to $55/hour depending on skills, experience, and project needs.
Take part in a flexible, remote, freelance project that fits around primary professional or academic commitments.
Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
Influence how future AI models understand and communicate in your field of expertise.