Responsibilities
- Review complete AI-generated coding sessions from start to finish
- Determine if the output is practical and applicable in real-world scenarios
- Assess high-level accuracy of generated solutions
- Evaluate whether the logic reflects the mindset of a skilled software engineer
- Judge the strength of explanations and thought process, beyond just code correctness
- Differentiate response quality across a scoring scale, identifying what elevates performance
- Give direct, well-reasoned feedback on effective elements
- Give direct, well-reasoned feedback on shortcomings
- Identify subtle issues that feel incorrect or deceptive, even if technically sound
- Contribute to defining excellence in interactions with AI coding tools
Compensation
$100–$200/hour
Work Arrangement
Contract role
What We Mean by “Taste”
- Does the response resemble how a capable engineer would naturally communicate?
- Is the explanation useful in practice, or merely accurate in theory?
- Does the AI guide the user thoughtfully, or simply produce output?
- Would this exchange strengthen or weaken user confidence?
- You must be able to make consistent, defensible subjective judgments
Engagement Details
- Rate: $100–$200/hour
- Hours: ~10–20 hours/week
- Duration: Through early May (with possible extension)
- Start: ASAP
- Process: Take-home evaluation exercise
- Process: One behavioral interview
Other
- Contract role
- Start date: ASAP
- Duration: Through early May (with possible extension)
- Hours: ~10–20 hours/week
- Take-home evaluation exercise
- One behavioral interview
