Independent Evaluation Contractor for AI Research - Remote

Overview

The role involves engaging independent contractors to evaluate the quality of AI model interactions, focusing on search tool usage and query effectiveness.

In Short

• Evaluate model interaction logs for appropriate search tool usage.
• Assess rewritten search queries for clarity and intent fidelity.
• Provide evidence-based rationales for evaluations.
• Score query quality and tool-use timing.
• Calibrate against gold examples and propose improvements.
• Maintain high inter-rater agreement and throughput targets.
• Flag sensitive content as per guidelines.

Requirements

• Excellent written communication skills.
• Meticulous attention to detail and ability to work independently.
• Familiarity with annotation tools and basic scripting is a plus.

Benefits

• Flexible, project-based workload with self-set hours.
• Support team available for inquiries.
• Resource sharing and best-practice guides provided.