Responsibilities

• Develop realistic developer environments that simulate a virtual company, including a codebase, infrastructure, and contextual elements such as tickets, documentation, and conversations.
• Create challenging tasks and evaluation criteria based on these environments, ensuring that tasks accurately reflect real-world developer scenarios.
• Design prompts and define success criteria for tasks, ensuring that they are solvable by AI agents and clearly communicate expected outcomes.
• Write comprehensive tests to validate agent solutions, allowing for a range of valid approaches while carefully excluding incorrect solutions.
• Iterate on tasks and tests by analyzing QA feedback, reviewing agent solutions, and refining evaluation methods to ensure fairness and robustness.

Requirements

• Proven experience in software engineering or a related field.
• Strong understanding of AI systems and their evaluation processes.
• Ability to create and analyze complex software tasks in simulated environments.
• Excellent problem-solving skills and attention to detail.
• Fluency in English, both written and spoken, is essential.
• Experience with virtual development environments is a plus.
• Available for project-based work, with flexibility in scheduling.

Benefits

• Opportunity to work on cutting-edge AI projects with leading technology companies.
• Flexible work hours and the ability to work remotely from LATAM.
• Collaborate with a team of skilled professionals dedicated to advancing AI technology.
• Gain valuable experience in AI evaluation and contribute to meaningful projects.
• Competitive compensation based on project scope and expertise.