Responsibilities
- Construct and manage web-scale crawling systems that gather data from billions of URLs on an ongoing basis
- Create and refine systems for discovering, prioritizing, scheduling, and coordinating web crawls
- Develop distributed crawling solutions that gather content efficiently while adhering to website policies and operational limits
- Implement systems to extract, render, parse, and standardize content from a wide range of web formats
- Enhance the scope, timeliness, speed, and quality of web crawls through data analysis and experimentation
- Design infrastructure to support repeated crawling, detection of changes, and incremental data updates
- Build targeted crawlers for high-priority domains, dynamic content, and hard-to-reach sources
- Evaluate crawl performance and web coverage to detect shortcomings and areas for enhancement
- Develop monitoring, observability, and reliability tools for large-scale crawling operations
- Troubleshoot live system issues and iteratively improve the scalability and robustness of crawl infrastructure
Benefits
- Competitive compensation packages including salary and equity, tailored to global talent
- Equity participation through stock options for all contributors to company success
- Full health coverage including medical, dental, vision, and life insurance, plus an annual wellness stipend
- Daily provision of lunch and dinner for on-site employees
- Generous 22-week paid parental leave for all parents, including adoptive and surrogate cases
- Unlimited paid time off in the U.S.; 30 days annually in the U.K.
- Visa sponsorship available to support international hires and long-term immigration pathways
- Regular team off-sites, social events, and company-wide celebrations
Compensation
Salary and equity structured to recognize and retain global talent
Work Arrangement
Not specified
Team
Not specified
Export Control Notice
This role may involve access to technology or source code regulated under U.S. Export Administration Regulations. Employment may depend on the company obtaining government authorization to grant such access in compliance with export control laws.
Yes, with support for long-term immigration pathways
