The QA Engineer will design and execute test plans for AI-driven systems, focusing on accuracy, safety, and integration quality in financial services applications. The role requires expertise in testing Large Language Models and AI Agents, along with strong automation and analytical skills to ensure system reliability and compliance.
Responsibilities
- Analyze both business and technical requirements to define testing scope and objectives.
- Develop functional test cases and conduct regression testing to validate system behavior.
- Document, track, and verify bug fixes using a bug tracking system such as Jira.
- Maintain and run automated test suites using Selenium with Python.
- Develop automated test scripts using the Pytest framework.
- Automate testing of web services using SoapUI or Python-based tools.
- Design and execute test strategies for Large Language Models, including prompt validation, accuracy assessment, and safety compliance.
- Test AI Agents to ensure reliability, robustness, and seamless integration within workflows.
Requirements
- Eight or more years of experience in software quality assurance and testing.
- Proficiency with tools including PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, and Jira.
- Experience working on projects utilizing CI/CD pipelines.
Nice to Have
- Ability to interpret customer and business user needs and translate them into testable concepts and prototypes.
- Strong analytical, technical, and problem-solving abilities with attention to detail and understanding of complex user requirements.
- Capable of managing time effectively by prioritizing tasks and meeting deadlines.
- Experience with Behavior-Driven Development and Test-Driven Development using Gherkin syntax.
- Familiarity with using Jira for agile project management.
- Hands-on experience in test automation using Selenium with Python.
- Experience testing Large Language Models, including evaluating prompt responses, model accuracy, relevance, and automating LLM workflows.
- Conduct red teaming exercises on LLMs using adversarial prompts to uncover vulnerabilities and edge cases.
- Perform safety and ethical compliance testing on LLM behaviors.
- Knowledge of LLM evaluation tools such as OpenAI Evals, Ragas, and custom benchmarking frameworks.
- Practical experience in testing both LLMs and AI Agents is highly desirable.
- Basic knowledge of artificial intelligence and machine learning algorithms is preferred.
- Familiarity with AI governance, bias detection, and safety validation practices is a plus.
Tech Stack
Selenium, Python, Pytest, SoapUI, Jira, PyCharm, Visual Studio Code, Git, Fiddler, Postman, JMeter, Jenkins, CI/CD
Benefits
- Hybrid work model supporting a balance between remote and office work.
- Access to inclusive professional development opportunities.
- Flexible support for work-life integration.
- Paid time off for volunteering activities.
- Engagement through vibrant employee networks focused on shared interests and values.
Work Arrangement
hybrid — Balance work from home and office based on your needs and role requirements
Team
multidisciplinary team within an AI initiatives group focused on emerging trends and advanced technologies in financial services
- Committed to creating an environment where every employee feels valued and empowered
- Relies on smart, engaged professionals to drive success
- Offers inclusive development opportunities
- Supports flexible work-life balance
- Fosters connection through employee networks
- Equal Opportunity Employer
Additional Information
- The role operates under a hybrid work model.
- A basic understanding of artificial intelligence and machine learning algorithms is preferred.
- Familiarity with AI governance, bias detection, and safety validation is considered an advantage.
