Glean is looking for a Machine Learning Engineer to build the measurement and quality layer for our AI Assistant and Agents. You will own evaluation pipelines, quality evalsets, LLM-powered judges, agent observability, and tooling to understand changes in performance. This role combines infrastructure engineering, applied ML, and direct product impact.
What You'll Do
- Design and curate evaluation datasets with sampling strategies, query diversity, and golden sets for reliable coverage.
- Build and maintain large-scale evaluation pipelines measuring assistant quality across thousands of real user queries.
- Build LLM-powered judges that score metrics like correctness, completeness, and response quality, aligning them with human judgment.
- Evaluate new models and product changes before they ship to gate launches and prevent regressions.
- Build observability infrastructure for AI agents: trace enrichment, data pipelines, and dashboards for inspectable behavior.
- Close the loop between quality measurement and improvement using eval results, feedback, and automated prompt iteration.
- Collaborate with engineers across the company to make evaluations a first-class part of the shipping process.
What We're Looking For
- 2+ years of software engineering experience with strong coding skills.
- Strong backend fundamentals in Go and Python; comfortable with distributed data pipelines.
- Experience working with LLM evaluation, reinforcement learning from human feedback, natural language processing, or other large ML systems.
- Analytically rigorous – you think carefully about what offline metrics predict about real user experience.
- Thrive in a customer-focused, tight-knit, cross-functional environment; a team player willing to take on impactful work.
- You care about quality – both in the systems you build and the product you're helping measure and improve.
Technical Stack
- Go
- Python
Benefits & Compensation
- Compensation: $200,000 - $300,000 annually + equity. Certain roles may be eligible for equity.
- Medical, Vision, and Dental coverage
- Generous time-off policy
- 401k plan
- Home office improvement stipend
- Annual education and wellness stipends
- Regular company events
- Healthy daily lunches
Work Mode
This is a hybrid position based out of our SF Bay Area offices.
We do not discriminate based on gender, ethnicity, sexual orientation, religion, civil or family status, age, disability, or race.
