Software Engineer, Data Ingestion - Remote

Overview

Anthropic is seeking a Software Engineer to lead the 'Tokens: Data Acquisition' team, focusing on acquiring data from the internet through a large-scale web crawler and partnerships.

In Short

• Lead the 'Tokens: Data Acquisition' team.
• Develop and maintain a large-scale web crawler.
• Build data ingestion pipelines.
• Improve observability of crawler systems.
• Collaborate with team members on data acquisition processes.
• Participate in code reviews and debugging.
• Focus on acquiring openly accessible information.
• Address data quality assessments.
• Work with specialized crawlers for high-value data sources.
• Engage in discussions about AI systems' transformative potential.

Requirements

• Experience with large distributed systems.
• Understanding of internet-scale crawling tradeoffs.
• Technical expertise in systems design.
• Familiarity with cloud-based solutions.
• Proficiency in Python.

Benefits

• Competitive compensation and benefits.
• Optional equity donation matching.
• Generous vacation and parental leave.
• Flexible working hours.
• Collaborative office space.