Senior Web Crawler Engineer - Remote

Overview

As a Senior Web Crawler Engineer at you.com, you will design, develop, and maintain automated systems to extract and index data from a wide variety of websites. Your work will directly power our search engine, ensuring users have access to the freshest and most comprehensive information available.

In Short

• Design, implement, and maintain scalable web crawlers.
• Develop robust scraping solutions for structured and unstructured data.
• Optimize crawling strategies for efficiency and coverage.
• Monitor and troubleshoot crawler performance.
• Collaborate with data engineering and AI teams.
• Ensure compliance with ethical web scraping practices.
• Stay updated with web technologies and anti-bot measures.

Requirements

• Strong programming skills in Python, Rust, C++, or similar languages.
• Familiarity with distributed systems and cloud infrastructure.
• Experience with HTML parsing and dynamic sites.
• Knowledge of anti-crawling techniques.
• Practical experience equivalent to a BS/MS degree in Computer Science or related field.
• Proven experience building large-scale web crawlers is a plus.

Benefits

• Flexible PTO with 11 U.S. holidays and a week shutdown in December.
• 100% health insurance coverage for policyholder and 75% for dependents.
• 12 weeks of paid parental leave.
• 401k program with 3% match.
• $500 work-from-home stipend.
• $1,200 per year Health & Wellness Allowance.
• Opportunity to collaborate with a leading AI research team.