The University of Florida College of Medicine is seeking a Data Scientist II to support the Assistant Dean for Assessment, Evaluation, and Scholarship and the Medical Director of the Anaclerio Learning and Assessment Center (ALAC). This role leads the analysis and reporting of medical education assessments, applying advanced psychometric methodologies to enhance the validity of high-stakes evaluations in a mission-driven environment committed to advancing health through education, research, and patient care.
What You'll Do
- Create psychometric analyses (CTT, IRT, factor, and generalizability/reliability studies) to evaluate and improve high stakes assessments.
- Conduct inter-rater reliability analyses to improve validity and provide consistency by reducing score drift.
- Collaborate with the Medical Director of ALAC to design and facilitate standard-setting panels, justify defensible passing scores, and revise grading schemas and scoring models.
- Hypothesize ideas to strengthen the reliability, validity, and fairness of assessments.
- Present actionable psychometric reports to provide continuous quality improvement.
- Lead analysis, interpretation, and reporting of performance for medical education assessments, including performance data for various types of assessment.
- Coordinate and facilitate standard setting sessions using an expert panel for all Clinical Skills Examination (CSE) cases developed and for overall exams.
- Develop and maintain learning analytics that inform progress, continuous quality improvement, and summative reporting to program leadership.
- Design a system to create competency profiles for individual students to track learning progress.
- Design and develop dashboards using computation psychometrics to identify patterns and trends in performance and learning outcomes to inform the leadership of strengths and opportunities for improvement for the curriculum.
- Report recurring updates and data summaries to leadership.
What We're Looking For
- Bachelor’s Degree in data science, statistics, bioinformatics, analytics, or similar field and three years of experience.
- Master’s Degree in data science, statistics, bioinformatics, analytics, or similar field and one year of experience.
- Doctoral Degree in data science, statistics, bioinformatics, analytics, or similar field.
Nice to Have
- Doctorate in psychometrics, educational measurement, statistics, data science, analytics, or a closely related field.
- Demonstrated expertise with psychometric procedures (e.g., CTT, IRT; factor analysis; adaptive testing preferred) and assessment development/validation.
- Proficiency with statistical/analytic software (e.g., R, Python, SAS, SPSS, Stata) and strong skills in data cleaning, management, and quality assurance.
- Experience conducting analyses in medical education or related social/health sciences, with strong written and verbal communication for research dissemination.
- Experience with standard setting, building weighted grading/score models, and producing longitudinal competency profiles in clinical skills and clerkship contexts.
- Demonstrated ability to manage multiple complex projects with minimal supervision and to collaborate effectively with faculty, staff, and trainees.
Technical Stack
- R
- Python
- SAS
- SPSS
- Stata
Team & Environment
Provides support to the Assistant Dean for Assessment, Evaluation, and Scholarship and the Medical Director of the Anaclerio Learning and Assessment Center (ALAC).
Benefits & Compensation
- Compensation: $70,000 - $80,000 annually
- Low-cost State Health Plans: Medical, Dental, and Vision Insurance
- Life and Disability Insurance
- Generous Retirement Options to secure your future
- Comprehensive Paid Time Off Packages: includes over 10 paid holidays, as well as paid family, sick and vacation leave
- Exceptional Personal and Professional Development Opportunities: Access to UF Training & Organizational Development programs, leadership development, LinkedIn Learning, and more
- Tuition Assistance through the UF Employee Education Program
- Public Service Loan Forgiveness (PSLF) Eligible Employer
The University of Florida is an equal opportunity employer.
