AI Evaluation Engineer - Remote

Overview

As an AI Evaluation Engineer at P-1 AI, you will be responsible for ensuring that our AI, Archie, learns and retains necessary engineering skills through effective evaluation benchmarks.

In Short

• Implement systems for organizing and reporting on eval benchmarks.
• Ensure evals run effectively within our CI/CD system.
• Collaborate with industrial partners and experts to refine evals.
• Create methods for detecting common AI quality challenges.
• Lead the implementation of automated tests across technology stacks.

Requirements

• Experience in constructing test suites for software/AI systems.
• Ability to design metrics for system evaluation and performance visualization.
• Experience with LLM-based systems is a plus.
• Strong communication skills with stakeholders.
• Proficiency in Python and modern software development tools.
• Ability to thrive in a fast-paced startup environment.

Benefits

• Work remotely with flexibility.
• Opportunity to collaborate with top minds in AI and engineering.
• Participation in a dynamic startup culture.
• Travel to the San Francisco office for co-working sessions.