AI Evaluation Specialist - Remote

Overview

Binance is a leading global blockchain ecosystem behind the world’s largest cryptocurrency exchange by trading volume and registered users. We are trusted by over 280 million people in 100+ countries for our industry-leading security, user fund transparency, trading engine speed, deep liquidity, and an unmatched portfolio of digital-asset products. Binance offerings range from trading and finance to education, research, payments, institutional services, Web3 features, and more. We leverage the power of digital assets and blockchain to build an inclusive financial ecosystem to advance the freedom of money and improve financial access for people around the world.

In Short

• Participate in the entire software development lifecycle, encompassing all stages from requirements analysis to test planning, execution, defect tracking, through to product release and maintenance.
• Act as the go-to person for AI Agents evaluation and continuous monitoring.
• Create comprehensive and effective test strategies and conduct hands-on testing to ensure the accuracy, reliability, and performance of AI and data applications.
• Perform root cause analysis of test failures and product issues effectively, driving optimization for future enhancements.
• Design and develop internal tools leveraging AI technology to improve engineering and testing work efficiency.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Science, or a related field.
• Strong understanding of Large Language Models (LLMs), autonomous AI agents, and their system architectures.
• Experience with AI evaluation methodologies, including offline benchmarking, online monitoring, and hybrid human-AI evaluation approaches.
• Familiarity with software engineering best practices such as Test-Driven Development (TDD), Behavior-Driven Development (BDD), and their limitations in AI contexts.
• Proficiency in designing adaptive, lifecycle-spanning evaluation frameworks incorporating both quantitative and qualitative metrics.
• Experience with evaluation tools and frameworks (e.g., Opik, LangSmith) is a plus.
• Ability to analyze complex system-level behaviors, including reasoning pipelines, tool integrations, and emergent agent actions.
• Strong analytical skills with experience in data-driven diagnostics and root cause analysis.
• Excellent communication skills to document evaluation plans, results, and recommendations clearly.
• Experience working in cross-functional teams and managing feedback loops between evaluation and development.
• Experience collaborating with infrastructure or platform teams to improve AI tooling and automation platforms.

Benefits

• Opportunity to work in a leading global blockchain ecosystem.
• Engage with cutting-edge AI technologies.
• Collaborate with a diverse and talented team.
• Contribute to innovative projects in the cryptocurrency space.
• Flexible working environment.