Lead and expand the offline evaluation framework across products, including test datasets for capabilities and regressions, judging mechanisms, and performance metrics
Develop and manage real-time quality monitoring dashboards tracking resolution rates, customer satisfaction, user feedback signals, LLM-based evaluations, error rates, and response latency
Analyze live traffic to identify failure patterns, convert them into testable regression cases, and collaborate with Product and subject matter experts to implement solutions
Improve evaluation methodologies by addressing judge consistency and handling non-deterministic model behaviors
Transform analytical findings into strategic decisions through regular syncs, emphasizing clear trade-offs and purpose-driven reporting

Proficiency in Python and SQL with the ability to independently conduct full-cycle data analysis
Strong grasp of statistical principles including sampling, hypothesis testing, variance, and interpreting noisy metrics
Problem-first thinking approach, starting with business objectives rather than tools or techniques
Minimum of three years of experience as an analyst or data scientist, with at least one year focused on product analytics

Background in quality measurement for machine learning systems such as ranking, recommendation, or classification models
Practical experience evaluating LLM-powered applications including RAG, agents, tool integration, and automated judging
Hands-on work building LLM agents, whether through personal projects, prototypes, or experimental setups

Direct influence on product development and decision-making
Opportunity to work remotely or in a hybrid setup across Europe, with EU-based operations
Participation in a stock options program
Strong organizational support and employee care policies
Access to the Work & Swim program, offering one month in a fully furnished corporate apartment in Cyprus

Hybrid — Germany, France, Netherlands, Italy, Spain

Finom is hiring a Product Data Scientist — AI Evaluation & Quality