Responsibilities
- Design, execute, and analyze A/B and multivariate experiments to evaluate product changes, learning interventions, and personalization strategies.
- Apply causal inference techniques (e.g., difference-in-differences, instrumental variables, regression discontinuity) where randomized experiments are not feasible.
- Develop robust frameworks for measuring treatment effects, handling interference, and addressing novelty/primacy effects in experimentation.
- Partner with product and engineering teams to define success metrics, set experiment guardrails, and ship decisions with confidence.
- Build statistical and ML models to support product roadmap decisions, learner segmentation, and personalization at scale.
- Apply predictive modeling, survival analysis, and Bayesian inference to understand learner behavior and forecast outcomes.
- Develop decision frameworks that weigh trade-offs across multiple business and learning objectives.
- Leverage GenAI tools and automation agents to accelerate analysis workflows and scale insight generation.
- Apply psychometric methods (e.g., item response theory, latent variable models, reliability and validity analysis) to measure learning outcomes and assessment quality.
- Design and evaluate instrumentation strategies that capture meaningful signals of learner knowledge and progress—not just activity.
- Partner with curriculum and learning design teams to define and operationalize constructs like mastery, engagement, and skill acquisition.
- Design and implement instrumentation strategies for accurate tracking of user interactions and data collection.
Requirements
- Bachelor’s or Master’s degree (or PhD) in Economics, Statistics, Computer Science, Cognitive Science, Psychometrics, Educational Measurement, or a related quantitative field.
- 7+ years of experience applying data science to product or business problems, with a strong track record of influencing decisions through rigorous analysis.
- Expert-level SQL and advanced Python proficiency, including fluency with data manipulation libraries (Pandas, NumPy) and scientific computing (SciPy, Statsmodels, scikit-learn).
- Deep applied statistics background: statistical inference, hypothesis testing, causal inference, Bayesian methods, and experimental design.
- Demonstrated experience designing and analyzing controlled experiments (A/B tests) at scale, including power analysis, sequential testing, and dealing with violations of standard assumptions.
- Experience with ML modeling in production contexts: feature engineering, model validation, bias-variance trade-offs, and model monitoring.
- Strong command of data visualization and the ability to translate complex statistical findings into clear, compelling narratives for non-technical audiences.
- Excellent written and verbal communication; comfortable presenting to senior leadership and cross-functional stakeholders.
Nice to Have
- Graduate study of psychometric modeling, item response theory (IRT), latent trait models, or educational measurement in a research or applied context.
- Familiarity with learning analytics frameworks: measuring knowledge acquisition, skill development, or learner progression in digital environments.
- Experience applying causal inference methods beyond A/B testing (e.g., synthetic control, propensity score matching, uplift modeling).
- Background in the educational technology sector, specifically with large-scale online learning environments.
- Experience with Airflow, Databricks, and/or Looker for pipeline orchestration and self-serve analytics.
- Experience with Amplitude or equivalent product analytics platforms.
- Exposure to survival analysis, time-series forecasting, or longitudinal data modeling.
