Mindrift is seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios for LLM agents. This role blends manual scenario validation, automated test thinking, and collaboration.
What You'll Do
- Review and validate test scenarios from Evaluation Writers.
- Spot logical inconsistencies, ambiguities, or missing checks.
- Suggest improvements to structure, edge cases, or scoring logic.
- Collaborate with infrastructure and tool developers to automate parts of the review.
- Create clean and testable examples for others to follow.
What We're Looking For
- Strong QA background (manual or automation), preferably in complex testing environments.
- Understanding of test design, regression testing, and edge case detection.
- Ability to evaluate logic and structure of test scenarios (even if written by others).
- Experience reviewing and debugging structured test case formats (JSON, YAML).
- Familiarity with Python and JS scripting for test automation or validation.
- Clear communication and documentation skills.
- Willingness to occasionally write or refactor test scenarios.
Nice to Have
- Experience testing AI-based systems or NLP applications.
- Familiarity with scoring systems and behavioral evaluation.
- Git/GitHub workflow familiarity (PR review, versioning of test cases).
- Experience using test management systems or tracking tools.
Technical Stack
- Python, JS (JavaScript)
- JSON, YAML
- Git/GitHub
Team & Environment
You will collaborate directly with writers and engineers.
Benefits & Compensation
- Contribute on your own schedule, from anywhere in the world.
- Get paid for your expertise, with rates that can go up to $55/hour depending on skills, experience, and project needs.
- Take part in a flexible, remote, freelance project that fits around primary professional or academic commitments.
- Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
- Influence how future AI models understand and communicate in your field of expertise.
Work Mode
This is a global, fully remote freelance opportunity. You can work from anywhere in the world.
Mindrift believes in using the power of collective human intelligence to ethically shape the future of AI.
