State Street is looking for a QA Engineer to join our Artificial Intelligence initiatives. You will work within a multidisciplinary team focused on emerging technologies and their application in financial services, analyzing specifications, defining test cases, and tracking defects.

What You'll Do

Analyze business and technical specifications.
Write functional test cases and perform regression testing.
Report bugs, verify fixes, and maintain the bug tracking system.
Maintain and execute automated tests using Selenium with Python.
Create automated test scripts using the Pytest framework.
Automate web services using SoapUI and/or Python.
Design and execute test plans for Large Language Models (LLMs), including prompt validation, accuracy checks, and safety compliance.
Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.

What We're Looking For

8+ years of software quality assurance testing experience.
Ability to translate customer needs into concepts and prototypes to elicit feedback.
Strong analytical, technical, and problem-solving skills with attention to detail.
Ability to manage your own time by appropriately prioritizing tasks within deadlines.
Experience with BDD & TDD (Gherkin).
Experience in using Jira as an agile management tool.
Experience in using Selenium with Python for test automation.
Experience in testing Large Language Models (LLMs), including validation of prompt responses, evaluation of model accuracy and relevance, and automation of LLM-based workflows.
Red teaming of LLMs to identify vulnerabilities and edge cases through adversarial prompts.
Safety validation and ethical compliance testing of LLM behavior.
Familiarity with tools and frameworks for LLM evaluation such as OpenAI Evals, Ragas, and custom benchmarking suites.
Experience with tools like PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins.
Experience in CI/CD-based projects.

Nice to Have

Hands-on experience in LLM testing and Agents testing.
Basic understanding of artificial intelligence and machine learning algorithms.
Familiarity with AI governance, bias detection, and safety validation.

Technical Stack

Selenium, Python, Pytest, SoapUI, Jira, PyCharm, Visual Studio Code, Git, Fiddler, Postman, JMeter, Jenkins

Work Mode

This role follows a hybrid work model.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.