Responsibilities

• Develop a dataset for evaluating AI coding agents, focusing on their ability to manage real-world developer tasks.
• Create realistic developer environments, including a virtual company with a codebase, infrastructure, and contextual elements such as tickets, documentation, and conversations that reflect a believable development history.
• Design tasks derived from intermediate states of these environments, ensuring clarity in prompts, defining success criteria, and verifying that tasks are solvable by AI agents.
• Write comprehensive tests to validate agent solutions, ensuring that valid approaches are accepted while incorrect ones are rejected with a balanced level of strictness.
• Iterate on tasks and tests based on feedback from quality assurance, reviewing agent solutions, analyzing failures, and refining the evaluation process to ensure fairness and robustness.

Requirements

• Proven experience in AI evaluation, software testing, or a related field.
• Strong understanding of software development processes and practices.
• Ability to create realistic developer environments and design relevant tasks.
• Exceptional problem-solving skills and attention to detail.
• Excellent communication skills in English, both written and verbal.
• Experience with programming languages and software development tools is preferred.

Benefits

• Flexible project-based work that allows you to set your own schedule.
• Opportunity to work with leading tech companies and contribute to innovative AI solutions.
• Competitive compensation based on project scope and complexity.
• Gain valuable experience in the rapidly evolving field of artificial intelligence.
• Connect with a network of professionals in the tech industry.