Responsibilities
- Own the development and execution of the evaluation system for internal AI agent capabilities, beginning with adoption in Agent Studio.
- Design the external evaluation interface that enables users to test, track, and enhance the agents they build.
- Determine the appropriate level of evaluation detail to expose to users, balancing accuracy and ease of use.
- Collaborate with the Build Experience product lead to embed evaluation naturally within the user workflow.
- Coordinate with machine learning and platform engineering teams to align evaluation methods with technical feasibility.
- Define success metrics for agent quality and for how effectively customers adopt evaluation tools.
- Conduct customer research to identify challenges in assessing agent performance and understand existing user approaches.
Compensation
For California applicants, the pay for this role may range between $185,000 - $250,000 plus benefits, perks, and equity. The final package will depend on the interview process, we're open to negotiation.
Work Arrangement
Remote (Worldwide)
Other
- For California applicants, the pay for this role may range between $185,000 - $250,000 plus benefits, perks, and equity.
- The final package will depend on the interview process, we're open to negotiation.
- REQ ID: 2538
