Responsibilities
- Design, implement, and document robust Scrapy spiders
- Ensure resilience against future website changes while maintaining existing ones
- Conduct comprehensive code review and data validation to maintain the highest standards of code and data quality
- Leverage in-depth knowledge of the HTTP protocol, the mechanisms of modern web browsers, and internal tools to design sophisticated web crawling solutions that bypass advanced anti-bot countermeasures
- Architect and contribute to significant new developments across multiple codebases
- Provide clear technical documentation and conduct training sessions for relevant teams when necessary
Requirements
- Python development experience
- Experience with Scrapy framework
- Ability to design and maintain web spiders
- Comprehensive code review skills
- Data validation skills
- In-depth knowledge of the HTTP protocol
- Understanding of modern web browser mechanisms
- Ability to design web crawling solutions that bypass anti-bot countermeasures
