State Street is looking for a Data Engineering Lead to own the development, maintenance, and optimization of data pipelines and systems. You will be responsible for the acquisition, storage, transformation, and analysis of large volumes of data, collaborating with teams to ensure data availability, reliability, and integrity.
What You'll Do
- Design, develop, and maintain Azure data pipelines for ingesting and loading data into centralized data lakes and Databricks Delta Lake.
- Implement efficient ELT/ETL processes to ensure data quality, consistency, and reliability.
- Design and implement efficient data models and database schemas.
- Implement modern data processing principles to streamline import/transformation.
- Work closely with cross-functional teams to understand data requirements and translate them into technical solutions.
- Monitor data pipelines, troubleshoot issues, and ensure data integrity and security.
- Implement data quality controls and validation processes to identify and rectify anomalies.
- Identify performance bottlenecks and optimize queries, data structures, and infrastructure.
- Implement security measures to protect sensitive data and ensure regulatory compliance.
- Stay updated with the latest trends, tools, and technologies in data engineering.
- Provide support to the development team with managing database instances, implementing complex queries, and managing Azure data infrastructure.
What We're Looking For
- Bachelor’s Degree in a computer or IT related subject.
- 10+ years of overall IT industry experience.
- 8+ years of overall Big Data pipeline experience.
- 8+ years as a Data Engineer, with a focus on designing and implementing solutions on Azure Databricks.
- 8+ years of experience with cloud-based development including Azure Services, Azure DevOps, Kubernetes, Docker.
- Hands-on experience with Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure SQL Database.
- Strong experience in data engineering, including data pipeline development, ETL/ELT processes, and data modeling.
- Strong proficiency in SQL and experience with programming languages such as Scala or Java.
- In-depth knowledge of SQL and experience with relational and non-relational databases such as Snowflake, SQLServer, Oracle.
- Knowledge of data warehousing concepts, dimensional modeling, and best practices.
- Experience working in a multi-developer environment, using version control (Git).
- Excellent problem-solving skills and ability to work independently and as part of a team.
- Excellent communication skills and ability to collaborate effectively with cross-functional teams.
- Team player with a positive attitude, enthusiasm, initiative, and self-motivation.
- Ability to multi-task, energetic, fast learner & problem solver.
- Experience with agile development methodology.
Nice to Have
- Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect.
- Experience on Snowflake.
- Experience of working in the financial industry.
Technical Stack
- Azure, Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure SQL Database
- SQL, Scala, Java
- Snowflake, SQLServer, Oracle
- Git, Kubernetes, Docker
Team & Environment
You will collaborate with cross-functional teams, including analysts, software engineers, and operations teams, and work with other globally situated team members.
Benefits & Compensation
- Compensation: $120,000 - $202,500 Annual
- Competitive and comprehensive benefits packages
- Generous medical care, insurance and savings plans
- Flexible Work Programs
- Wealth of development programs and educational support
- Paid volunteer days
- Matching gift programs
- Access to employee networks
- Retirement savings plan (401K) with company match
Work Mode
This role is onsite.
State Street is an equal opportunity and affirmative action employer.
