Responsibilities
- Lead and drive development of large-scale web crawling systems, data ingestion pipelines, and processing frameworks.
- Develop, sustain, and refine backend and frontend components for crawler services, covering storage, data access, and management dashboards.
- Work closely with Search and Infrastructure teams to maintain consistent, high-quality processing of billions of web documents.
- Design and deploy fullstack features and distributed systems capable of handling heavy traffic and real-time data workflows.
- Quickly prototype, test innovative methods, and improve system efficiency, user experience, and operational stability.