AI Evaluation Engineer - Remote

Overview

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

In Short

• We are looking for an AI Evaluation Engineer specialized in data analysis.
• Design benchmark tasks that simulate real-world analytical workflows.
• Create scenarios for AI systems to analyze large, messy, multi-source datasets.
• Decompose tasks across multiple agents and produce clear, verifiable conclusions.
• Commitments required: 8 hours per day with 4 hours overlap with PST.
• Employment type: Contractor assignment (no medical/paid leave).
• Duration of contract: 4 weeks+
• Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria, Turkey, Vietnam.
• Interview: take home assessment (60min).

Requirements

• Design and develop multi-agent benchmark tasks focused on complex data analysis workflows.
• Create or curate realistic datasets (CSV, JSON, logs, reports, financial or operational data).
• Build tasks requiring cross-referencing across multiple data sources.
• Conduct anomaly detection and contradiction identification.
• Perform statistical analysis and interpretation.
• Define task decomposition strategies across specialized sub-agents.
• Develop verification logic to validate precise analytical outputs.
• Implement evaluation pipelines using Python and SQL.
• Create reproducible environments using Docker.
• Analyze task performance and refine for clarity, difficulty, and scoring accuracy.

Benefits

• Flexible working hours.
• Opportunity to work with diverse datasets.
• Engagement with cutting-edge AI technologies.
• Collaboration with a global team.
• Enhance your skills in data analysis and AI evaluation.