Coding Agent Evaluation Specialist at Thoth AI, Remote, £Freelance Rate

Position : Coding Agent Evaluation Specialist (LLM Eval & AI Training)

Employment Type : Remote Freelance

Working Hour : Flexible

About the Opportunity

We are building a talent pool of qualified professionals for upcoming and future Coding Agent Evaluation and AI Training projects. This freelance, fully remote opportunity is ideal for software engineers, programmers, and coding professionals who are passionate about improving the quality, safety, and performance of Large Language Models (LLMs) and AI coding agents.

In this role, you will evaluate AI-generated code, assess coding agent performance, identify errors, and provide structured feedback that helps train and improve next-generation AI systems.

Job Responsibilities

·      Apply and adapt coding evaluation rubrics to assess LLM-generated code responses and coding agent trajectories, including correctness, code quality, efficiency, security, and tool usage.

·      Run, test, and debug code to verify outputs against defined test cases and acceptance criteria.

·      Reproduce technical issues and identify their root causes.

·      Accurately categorize errors related to reasoning, code implementation, tool usage, or the execution environment.

·      Provide clear, structured, and actionable written feedback according to established evaluation guidelines.

·      Maintain consistent quality and accuracy across scoring, ranking, and rubric-based evaluation tasks.

·      Complete rubric interpretation and golden-task training.

·      Pass the required post-training qualification before starting live project work.

Requirements

·      Relevant software development experience demonstrated through professional employment, internships, academic work, or personal projects.

·      At least 1 year of relevant experience preferred, with 3 or more years considered an advantage.

·      Previous experience in LLM code evaluation, AI training, code annotation, or data labeling, including code-response scoring, ranking, and rubric-based reviews.

·      Proficiency in Python or another major programming language, such as Java, JavaScript, C++, C#, or Go.

·      Ability to read, run, test, and debug code effectively.

·      Working knowledge of Git, version control, and terminal or command-line tools.

·      Ability to understand complex guidelines, apply evaluation rubrics consistently, identify errors accurately, and provide reliable feedback.

·      Strong analytical skills, task-execution ability, attention to detail, and commitment to quality.

·      English proficiency at B2 level or above, with C1 level preferred.

·      Strong written English skills for feedback, training, documentation, and collaboration.

·      A personal laptop or desktop computer with a stable internet connection.

·      A bachelor’s, master’s, or doctoral degree in Computer Science, Software Engineering, or a related field is preferred. Equivalent professional experience is also accepted.

·      Experience with coding agents, AI-assisted development tools, open-source contributions, or personal software projects is an advantage.