TaskUs is looking for a Manager, AI Data Quality to lead quality assurance for our AI data annotation projects. In this leadership role, you'll drive process development, establish comprehensive metrics, and manage a team, providing data-driven insights to stakeholders.
What You'll Do
- Drive the development, refinement, and documentation of quality assurance processes and standard operating procedures.
- Establish comprehensive quality metrics (e.g., F1 score, inter-annotator agreement) aligned with business objectives.
- Continuously review and refine annotation workflows to proactively identify risks and increase efficiency.
- Act as the subject matter expert on annotation quality, providing feedback, training, and support to annotators and teams.
- Lead in-depth data analysis to diagnose quality issues and uncover root causes of recurring errors.
- Develop and maintain dashboards providing real-time insights into quality metrics and project performance.
- Prepare and deliver strategic quality reports to senior management and clients.
- Partner with cross-functional teams to align on project goals and quality assurance initiatives.
- Lead a team of Data Quality Analysts, providing mentorship and fostering continuous improvement.
- Manage the configuration and integration of annotation and quality control tools (e.g., Labelbox, Dataloop, LabelStudio).
- Identify, evaluate, and implement innovative quality control tools and automation technologies.
What We're Looking For
- Bachelor’s degree in a technical field (e.g., Computer Science, Data Science) or equivalent professional experience.
- 3+ years of experience in data quality management, data operations, or related roles within AI/ML or data annotation environments.
- Proven track record in designing and executing quality assurance strategies for large-scale, multi-modal data annotation projects.
- Proven track record in a leadership role managing and developing high-performing, remote or distributed teams.
- Deep understanding of data annotation processes, quality assurance methodologies, and statistical quality metrics (e.g., F1 score, inter-annotator agreement).
- Strong data-analysis skills, with the ability to interrogate large datasets and translate findings into actionable recommendations.
- Excellent communication skills, with experience presenting complex data to technical and non-technical stakeholders.
- Proficiency with annotation and QA tools (e.g., Labelbox, Dataloop, LabelStudio).
- High-level proficiency in common data-analysis tools, such as Excel and Google Sheets.
- Familiarity with programmatic data analysis techniques (e.g., Python, SQL).
- Familiarity with the core concepts of AI/ML pipelines, including data preparation, model training, and evaluation.
Nice to Have
- Prior experience in an agile or fast-paced tech environment with exposure to AI/ML pipelines.
- Experience in a managed services or vendor-driven environment.
- Familiarity with prompt engineering and large-language-model assisted workflows to optimise annotation and validation processes.
- In-depth knowledge of ethical AI practices and compliance frameworks.
Technical Stack
- Annotation Platforms: Labelbox, Dataloop, LabelStudio
- Data Analysis: Excel, Google Sheets
- Programmatic Analysis: Python, SQL
Team & Environment
This is a leadership role managing a team of Data Quality Analysts.
Benefits & Compensation
- Competitive industry salaries.
- Comprehensive benefits packages.
- Internal mobility and professional growth.
TaskUs is committed to providing equal access to opportunities.
