Keeper Security, Inc.
Senior QA Analyst, AI & Threat Analytics
Keeper Security, Inc., Cameron Park, California, United States
About Keeper
Keeper Security is transforming cybersecurity for organizations around the world with next-generation privileged access management. Keeper’s zero-trust and zero-knowledge cybersecurity solutions are FedRAMP and StateRAMP Authorized, FIPS 140-2 validated, as well as SOC 2 and ISO 27001 certified. Keeper deploys in minutes, not months, and seamlessly integrates with any tech stack to prevent breaches, reduce help desk costs and ensure compliance. Trusted by thousands of organizations to protect every user on every device, Keeper is the industry leader for best-in-class password management, secrets management, privileged access, secure remote access and encrypted messaging. Learn more at KeeperSecurity.com.
About the Role The Senior QA Analyst AI & Threat Analytics will lead testing efforts for AI-driven features and machine learning models across Keeper’s platform, including LLM-based summarization, autofill and classification systems, and behavior analysis pipelines. You will design and execute test strategies that validate model accuracy, reliability, guardrails, data quality, and integration with production systems. This role requires hands-on Python automation expertise, strong analytical and evaluation skills, and the ability to test nondeterministic systems with rigor and consistency. You will collaborate closely with ML engineers, security teams and product engineering to ensure AI features meet Keeper’s quality, safety and performance standards.
Responsibilities
Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
Validate data pipelines, input/output transformations and model integration points across Keeper’s platform
Conduct regression testing on ML models and AI workflows to ensure reliability across releases
Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
Validate security, privacy and zero-trust requirements for AI model interactions and data handling
Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures
Qualifications
3+ years of experience in QA, test automation or evaluation of complex systems
Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
Experience testing LLM-integrated features or AI-driven workflows
Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
Experience with API testing, JSON validation and test data generation
Ability to test nondeterministic systems and evaluate model quality with structured methodologies
Strong analytical, troubleshooting and communication skills
Experience testing cloud-integrated applications or systems deployed on AWS
Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience
Preferred Qualifications
Experience testing security-focused AI or threat analytics systems
Familiarity with ML observability tools, vector databases or data processing frameworks
Experience testing video, image or behavior analysis pipelines
Knowledge of zero-trust principles and secure data handling
Experience working with LLM providers or AI guardrail frameworks
Benefits
Medical, Dental & Vision (inclusive of domestic partnerships)
Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
Voluntary Short/Long Term Disability Insurance
401K (Roth/Traditional)
A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
Above market annual bonuses
Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program. We celebrate diversity and are committed to creating an inclusive environment for all employees.
#J-18808-Ljbffr
About the Role The Senior QA Analyst AI & Threat Analytics will lead testing efforts for AI-driven features and machine learning models across Keeper’s platform, including LLM-based summarization, autofill and classification systems, and behavior analysis pipelines. You will design and execute test strategies that validate model accuracy, reliability, guardrails, data quality, and integration with production systems. This role requires hands-on Python automation expertise, strong analytical and evaluation skills, and the ability to test nondeterministic systems with rigor and consistency. You will collaborate closely with ML engineers, security teams and product engineering to ensure AI features meet Keeper’s quality, safety and performance standards.
Responsibilities
Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
Validate data pipelines, input/output transformations and model integration points across Keeper’s platform
Conduct regression testing on ML models and AI workflows to ensure reliability across releases
Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
Validate security, privacy and zero-trust requirements for AI model interactions and data handling
Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures
Qualifications
3+ years of experience in QA, test automation or evaluation of complex systems
Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
Experience testing LLM-integrated features or AI-driven workflows
Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
Experience with API testing, JSON validation and test data generation
Ability to test nondeterministic systems and evaluate model quality with structured methodologies
Strong analytical, troubleshooting and communication skills
Experience testing cloud-integrated applications or systems deployed on AWS
Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience
Preferred Qualifications
Experience testing security-focused AI or threat analytics systems
Familiarity with ML observability tools, vector databases or data processing frameworks
Experience testing video, image or behavior analysis pipelines
Knowledge of zero-trust principles and secure data handling
Experience working with LLM providers or AI guardrail frameworks
Benefits
Medical, Dental & Vision (inclusive of domestic partnerships)
Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
Voluntary Short/Long Term Disability Insurance
401K (Roth/Traditional)
A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
Above market annual bonuses
Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program. We celebrate diversity and are committed to creating an inclusive environment for all employees.
#J-18808-Ljbffr