MagicSchool AI is looking for an Associate LLM Quality Analyst to join our Trust, Safety and Quality team. In this early-career role, you will support the quality and reliability of our AI outputs, working on technology that directly impacts educators and students. This is a great opportunity for someone who wants to work in tech and explore different career paths like product management or data analysis.

What You'll Do

Intake, triage, replicate, and prioritize user feedback on LLM outputs.
Identify patterns in quality issues and escalate them appropriately.
Act as a human-in-the-loop for quality checks.
Manage and develop test cases.
Review and label LLM-generated responses for evaluators.
Generate ground truth datasets to validate evaluator accuracy and consistency.
Help test and validate new judges (evaluators) as they are developed.
Draft and iterate on judge prompts and scoring rubrics to improve coverage and quality.
Assist with prompt adjustments and writing.
Create and maintain test suites for regular evaluation of LLM performance and evaluator rubrics.
Compile collections of prompts, expected responses, and target evaluation scores.
Ensure test coverage includes edge cases and diverse content.
Assist in building dashboards to visualize and monitor evaluator outputs, consistency, and regressions.
Collect and prepare data for analysis.

What We're Looking For

Gets a lot done: Works hard, is resourceful, and does whatever it takes.
Self-starter: Takes initiative and doesn't wait to be told what to do.
Organized and detail-oriented: Can manage multiple workstreams and maintain clear documentation.
Adaptable: Smart, learns fast, and curious.
Builds relationships easily: Emotionally intelligent and a warm communicator.
Strong communication skills: Team-first mindset and highly collaborative.
Comfortable working in spreadsheets and organizing data.
Basic analytical skills and the ability to identify patterns in data.
Strong written communication skills for drafting prompts and documentation.
Experience with LLMs or AI products.
Education background—either professional experience in education or an education degree.
1-3 years of professional experience (or a recent graduate with relevant internship experience).
Passionate about solving education problems with technology.
Excited to explore different areas of tech (product, data analysis, quality assurance).

Nice to Have

Startup or fast-paced environment experience.

Team & Environment

You will work with the Trust, Safety and Quality team and report to a Senior LLM Quality Analyst.

Benefits & Compensation

Work on AI technology that directly impacts educators and students.
Join a mission-driven team passionate about making education more efficient and equitable.
Flexibility of working from home.
Unlimited time off.
Choice of employer-paid health insurance plans.
Dental and vision offered at very low premiums.
Generous stock options, vested over 4 years.
401k match.
Monthly wellness stipend.