Help push the limits of today's most advanced AI models.
Are you an experienced technical professional who enjoys solving complex problems and challenging the capabilities of AI systems?
We are looking for specialists in software engineering, systems, machine learning, cybersecurity, hardware, and related technical fields to contribute to an expert-level AI benchmarking project.
About the project
Today's AI agents can perform many technical tasks, but they still struggle with complex, real-world problems. In this project, you'll use your expertise to design challenging tasks that AI agents must solve in a Linux terminal.
Your work will involve:
- Designing original, technically challenging problems based on real-world scenarios.
- Developing automated verifiers to determine whether an AI agent's final result is correct.
- Anticipating edge cases and ensuring tasks have clear specifications and reliable evaluation criteria.
This is not a traditional data annotation project. You'll apply your technical knowledge to evaluate the capabilities and limitations of advanced AI systems.
Areas of expertise
We welcome candidates with experience in one or more of the following areas:
- Software engineering and coding
- Systems engineering and Linux environments
- Machine learning and artificial intelligence
- Cybersecurity and information security
- Hardware engineering
- Related technical disciplines
Requirements
- Bachelor's degree or higher from a top-500 university.
- Approximately 6–7+ years of relevant professional experience, depending on the specialization.
- Strong analytical, technical problem-solving, and critical-thinking skills.
- Practical expertise in your field, ideally including programming, scripting, Linux, automation, testing, or software verification.
- Ability to design challenging, clearly specified problems and define reliable criteria for evaluating solutions.
- Residence in an eligible country.
Compensation and commitment
- Competitive, project-based compensation based on expertise, assessment results, and project requirements.
- Higher compensation may be available for particularly strong submissions and specialized expertise.
- The project is ongoing, with an expected duration of at least seven weeks.