Keeper Security, Inc. is seeking a Senior QA Analyst, AI & Threat Analytics to lead testing efforts for our AI-driven features and machine learning models. You will be responsible for designing and executing comprehensive test strategies that validate model accuracy, reliability, guardrails, and integration across Keeper's platform.
What You'll Do
- Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
- Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
- Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
- Validate data pipelines, input/output transformations and model integration points across Keeper's platform
- Conduct regression testing on ML models and AI workflows to ensure reliability across releases
- Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
- Validate security, privacy and zero-trust requirements for AI model interactions and data handling
- Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
- Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures
What We're Looking For
- 3+ years of experience in QA, test automation or evaluation of complex systems
- Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
- Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
- Experience testing LLM-integrated features or AI-driven workflows
- Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
- Experience with API testing, JSON validation and test data generation
- Ability to test nondeterministic systems and evaluate model quality with structured methodologies
- Strong analytical, troubleshooting and communication skills
- Experience testing cloud-integrated applications or systems deployed on AWS
- Bachelor's degree in Computer Science, Information Systems or related field, or equivalent practical experience
Nice to Have
- Experience testing security-focused AI or threat analytics systems
- Familiarity with ML observability tools, vector databases or data processing frameworks
- Experience testing video, image or behavior analysis pipelines
- Knowledge of zero-trust principles and secure data handling
- Experience working with LLM providers or AI guardrail frameworks
Technical Stack
- Python, Pytest, Hugging Face libraries, Jupyter Notebooks, AWS, JSON
Team & Environment
You will join the AI & Threat Analytics team.
Benefits & Compensation
- Medical, Dental & Vision (inclusive of domestic partnerships)
- Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
- Voluntary Short/Long Term Disability Insurance
- 401K (Roth/Traditional)
- A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
- Above market annual bonuses
Work Mode
This is a hybrid position open to candidates in the El Dorado Hills, CA metro area or Chicago, IL metro area.
Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program.
