Voice AI Evaluation Lead - Remote

Overview

Deepgram is looking for a Voice AI Evaluation Lead to take ownership of how we benchmark and evaluate the performance of our voice AI models. This role is pivotal to the integrity and impact of our AI offerings.

In Short

• Build and maintain scalable benchmarking pipelines for model evaluations.
• Run regular evaluations of production and pre-release models.
• Partner with Research, Data, and Engineering teams for new methodologies.
• Design and refine evaluation metrics reflecting product goals.
• Author comprehensive model cards and internal reports.
• Work closely with Data Labeling Ops for evaluation datasets.
• Collaborate with QA Engineers for model tests integration.
• Support Marketing and Product with data-backed comparisons.
• Track market developments and maintain competitive benchmarks.
• Support GTM teams with benchmarking best practices.

Requirements

• Experience designing evaluation pipelines for ML models.
• Proficiency in Python and data analysis libraries.
• Ability to develop automated evaluation systems.
• Comfort with large-scale datasets and performance metrics.
• Experience using LLMs for analysis or pipeline prototyping.
• Strong communication skills for translating data insights.
• Proven success working cross-functionally.

Benefits

• Opportunity to work on cutting-edge technology.
• Collaborative and inclusive work environment.
• Equal opportunity employer.
• Support for applicants needing accommodations.