The World Bank Group is hiring a Data Engineer to design, build, and maintain the data infrastructure that supports our mission-driven, data-informed decision-making. This role is central to our data integration and transformation initiatives, enabling insights that address urgent global challenges.
What You'll Do
- Design, develop, and maintain data pipelines for ingestion, transformation, and serving across batch and streaming workloads.
- Build ETL/ELT workflows to integrate data from diverse sources into enterprise data platforms.
- Develop data transformation logic using Apache Spark, PySpark, SparkSQL, and SQL.
- Implement change data capture (CDC) patterns for real-time and near-real-time data synchronization.
- Build streaming data pipelines for real-time analytics and operational use cases.
- Optimize pipeline performance, resource utilization, and cost efficiency.
- Support a federated data pipeline architecture that enables Line of Business (LOB) teams to own and manage their domain data.
- Contribute to self-serve data infrastructure that abstracts complexity and allows domain teams to build pipelines independently.
- Develop standardized pipeline deployment patterns that LOB teams can adopt while maintaining autonomy.
- Support domain teams in building data products that are discoverable, interoperable, and compliant with enterprise standards.
- Enable distributed data processing across domains while ensuring consistency through federated governance.
- Assist in establishing data contracts and interoperability standards that allow seamless data sharing.
- Support the balance between domain autonomy and enterprise-wide governance requirements.
- Develop reusable pipeline templates and Infrastructure as Code (IaC) patterns for common data product types.
- Create blueprints for data ingestion, transformation, quality validation, and serving that LOB teams can customize.
- Build standardized patterns for batch pipelines, streaming pipelines, CDC implementations, and API-based integrations.
- Contribute to a pattern library covering medallion architecture, dimensional modeling, and data product packaging.
What We're Looking For
- Demonstrated experience designing, building, and maintaining production data pipelines for batch and streaming workloads.
- Proficiency in developing ETL/ELT workflows and data transformation logic.
- Hands-on experience implementing change data capture (CDC) patterns.
- Proven ability to optimize data pipeline performance and resource utilization.
- Experience supporting federated or domain-oriented data architectures.
- Strong collaboration skills to partner with stakeholders and domain teams.
- A commitment to building scalable, reusable, and standardized data infrastructure components.
Technical Stack
- Apache Spark
- PySpark
- SparkSQL
- SQL
Team & Environment
This role is part of the Platforms & Tools unit within the ITS Data Office.
Work Mode
This position is based on-site in our Chennai, India office.
The World Bank Group is dedicated to ending extreme poverty, increasing shared prosperity, and promoting sustainable development. We are a unique global partnership of five institutions working with public and private sector partners, investing in groundbreaking projects and using data, research, and technology to develop solutions to the most urgent global challenges.
