Applied AI Software Engineer - Remote

Overview

Canvas Medical is seeking an Applied AI Software Engineer to lead evaluations for AI agents, focusing on automating work for customers in the healthcare sector.

In Short

• Lead evaluations for AI agents in healthcare.
• Work with foundation model inference and fine-tuning APIs.
• Design and run rigorous evaluation experiments.
• Collaborate with product, ML engineering, and clinical informatics teams.
• Develop high-fidelity unit evaluations and end-to-end evaluations.
• Define expert-determined ground truth outcomes.
• Analyze results and summarize tradeoffs for stakeholders.
• Ensure AI agents are trustworthy and robust.
• Build test harnesses to validate model behavior.
• Engage with marketing to communicate AI features.

Requirements

• 5+ years of experience in applied machine learning or AI engineering.
• Proficiency with foundation model APIs.
• Experience in designing and running evaluation pipelines.
• Superlative Python engineering skills.
• Familiarity with clinical or healthcare data is a plus.
• Experience with reinforcement fine-tuning or model monitoring is a plus.
• Ability to collaborate across teams.
• Comfortable with complexity in healthcare deployments.
• Strong focus on correctness and reproducibility.
• Encouragement for diverse applicants to apply.

Benefits

• Work with a leading healthcare technology company.
• Collaborate with top technology investors.
• Opportunity to influence AI features in production settings.
• Engage with a diverse and skilled team.
• Contribute to meaningful healthcare solutions.