Data Scientist – AI & Agentic Applications & Benchmarking - Remote

Overview

CloudBees is seeking a startup-savvy Data Scientist to help define, measure, and evangelize the impact of Agentic Applications across our platform. You’ll work closely with engineers and product teams to prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks to help the company drive the conversation in the market and with customers.

In Short

• Lead the charge in developing and prototyping telemetry and eval frameworks for agentic systems.
• Measure AI outcomes and usage across customers and teams.
• Define KPIs and success metrics for AI and LLM-driven features.
• Use Python notebooks for data exploration and visualization.
• Write internal documentation and performance summaries.
• Enable engineering teams to evaluate agent performance effectively.
• Stay updated with metrics and evaluation techniques in AI.

Requirements

• 3+ years of experience in data science or ML analytics.
• Proficiency in Python and Jupyter notebooks.
• Experience with evals, telemetry, and user-facing ML systems.
• Familiarity with AI/ML tools like MLFlow and Hugging Face.
• Clear communication skills for writing about outcomes.
• Comfort in fast-paced environments.

Benefits

• Shape the future of AI-driven DevOps.
• Join a passionate team at the forefront of agentic systems.
• Work in a flexible, remote-first culture.
• Competitive salary and excellent benefits.