About the Role
This position involves reviewing and assessing code and technical explanations produced by AI systems, providing feedback on correctness and quality, and helping refine outputs to meet engineering standards.
Responsibilities
- Analyze AI-generated code for logical accuracy and syntax correctness
- Review technical explanations for clarity and completeness
- Identify errors or inefficiencies in proposed software solutions
- Rate responses based on predefined quality criteria
- Provide detailed feedback to improve AI model performance
- Ensure outputs align with standard software engineering practices
- Evaluate solutions for adherence to best practices in coding
- Flag ambiguous or incorrect technical reasoning
- Assess whether code samples meet functional requirements
- Verify that explanations match the complexity of the task
- Judge the readability and maintainability of generated code
- Check for proper use of data structures and algorithms
- Review API usage and integration suggestions
- Evaluate debugging recommendations for validity
- Assess documentation quality within technical outputs
- Determine if security considerations are properly addressed
- Review testing strategies suggested by AI
- Ensure responses avoid overcomplication or under-explanation
- Monitor consistency across multiple AI-generated answers
- Help refine evaluation rubrics based on real-world examples
- Report edge cases where AI performance degrades
- Contribute to improving training data through annotations
- Support iterative development of evaluation workflows
- Maintain accuracy in high-volume assessment environments
- Adapt to evolving AI model versions and behaviors
Nice to Have
- Experience working with AI or machine learning systems
- Background in technical education or mentoring
- Familiarity with natural language processing models
- Prior experience in quality assurance roles
- Involvement in open-source software projects
- Exposure to large-scale codebases
- Experience evaluating technical content
- Knowledge of prompt engineering principles
- Participation in coding competitions
- Teaching or grading experience in computer science
Compensation
Competitive hourly rate
Work Arrangement
Remote
Team
Distributed team across the US and Canada
Location Requirements
- Applicants must reside in the United States or Canada
- Work authorization must be valid in one of these regions
- No relocation or visa support is provided
Time Commitment
- Flexible hours available
- Weekly time expectations communicated in advance
- Opportunities for ongoing involvement based on performance
Not available
