The QA Engineer will specialize in testing Large Language Models and AI Agents, ensuring accuracy, safety, and integration quality in financial services applications. This role operates within a hybrid work model and collaborates with a multidisciplinary team advancing emerging technologies.
Responsibilities
- Evaluate business and technical requirements to inform testing strategies.
- Develop and execute functional test cases, including regression testing to ensure system stability.
- Identify, document, and track defects using bug tracking systems such as Jira, and validate resolutions.
- Execute and maintain automated test suites using Selenium with Python.
- Build test automation scripts leveraging the Pytest framework.
- Automate testing of web services using SoapUI or Python-based tools.
- Design and implement test plans for Large Language Models, covering prompt validation, accuracy assessment, and compliance with safety standards.
- Conduct testing of AI Agents to verify reliability, robustness, and seamless integration within workflows.
Requirements
- Eight or more years of professional experience in software quality assurance testing.
- Proficiency with tools including PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, and Jira.
- Proven involvement in projects utilizing CI/CD pipelines.
Nice to Have
- Capable of interpreting customer and business user needs, motivations, and behaviors to shape testable concepts and prototypes that enable direct feedback.
- Demonstrates strong analytical, technical, and problem-solving abilities, with attention to detail and the ability to translate complex user requirements into effective testing approaches.
- Able to manage time effectively by prioritizing tasks and meeting deadlines independently.
- Experience applying Behavior-Driven Development (BDD) and Test-Driven Development (TDD) using Gherkin syntax.
- Familiarity with using Jira for agile project management.
- Hands-on experience with Selenium and Python for test automation.
- Experience testing Large Language Models, including validating prompt responses, assessing model accuracy and relevance, and automating LLM workflows.
- Conduct red teaming exercises on LLMs using adversarial prompts to uncover vulnerabilities and edge cases.
- Perform safety and ethical compliance testing on LLM behaviors.
- Knowledge of LLM evaluation tools and frameworks such as OpenAI Evals, Ragas, and custom benchmarking solutions.
- Practical experience in testing both LLMs and AI Agents is strongly preferred.
- Basic understanding of artificial intelligence and machine learning concepts is desired.
- Familiarity with AI governance practices, bias detection methods, and safety validation techniques is a plus.
Tech Stack
Selenium, Python, Pytest, SoapUI, Jira, Postman, Fiddler, Git, Visual Studio Code, PyCharm, JMeter, Jenkins
Benefits
- Hybrid work model supporting a balance between remote and office work.
- Access to inclusive professional development opportunities.
- Flexible support for work-life integration.
- Paid time off for volunteer activities.
- Engagement with vibrant employee networks.
Work Arrangement
hybrid — Balance work from home and office based on needs and role requirements
Team
multidisciplinary team focused on emerging trends and cutting-edge technologies in financial services
- Committed to fostering an environment where every employee feels valued and empowered
- Employees are essential to continued success
- Inclusive development opportunities
- Flexible work-life support
- Vibrant employee networks
- Equal Opportunity Employer
Additional Information
- The role supports a hybrid work model.
- Basic understanding of artificial intelligence and machine learning algorithms is preferred.
- Familiarity with AI governance, bias detection, and safety validation will be an added advantage.
- Equal Opportunity Employer: considers all qualified applicants without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
