AI Code Evaluation Specialist

$40-55 USD / hour

Posted 1 minute ago

•

Ends in 6 days

$40-55 USD / hour

AI Trainer - Reasoning & Code Evaluation Contract | Fully Remote | $40 - $55 per hour Grade, stress-test, and improve coding assistants used by 200,000+ developers. ABOUT THE ROLE We're a small, remote agency of engineers and technical writers. Our clients build coding assistants, and we do the evaluation and training work they cannot scale internally. That means judgment: someone has to decide, at scale and with a straight face, whether a model's answer is actually correct, merely plausible, or quietly broken. You will be that someone. This role is equal parts code review, technical writing, and adversarial thinking. You will spend your days reading model output that looks right, proving where it is wrong, and turning those findings into rubrics and datasets the research team can train against. It is repetitive work done at a high level of attention, and it directly changes what the model learns. If you have ever been the person who finds the off-by-one error in a code review that everyone else approved, this job is built for you. WHAT YOU'LL DO - Write and maintain grading rubrics for code generation, debugging, and multi-step reasoning tasks. - Score model outputs against those rubrics and write the rationale that lets a second person reach the same score without asking you. - Build adversarial prompt sets that expose real failure modes: hallucinated APIs, silent off-by-one errors, confident but wrong explanations, tests that pass without testing anything. - Run pairwise preference comparisons and help convert messy human judgment into a reward signal that is stable enough to train on. - Design targeted fine-tuning datasets from your own failure analysis rather than from generic scraping. - Review other trainers' work and keep inter-annotator agreement above 0.8 on rubric-graded samples. - Write short, specific failure reports that research can act on the same week. "Model is bad at recursion" is not a report; a reproducer is. WHAT WE'RE LOOKING FOR - Bachelor's degree in computer science, mathematics, or a related technical field, or equivalent demonstrated experience. We care about the work, not the parchment. - 3+ years writing production code in Python, TypeScript, Go, or similar. You can read a diff and spot the bug before running it. - Comfort with ambiguity. A rubric says "the answer is correct" - you can argue precisely where the boundary sits and defend it in writing. - Strong written English. You can explain why an answer is wrong in three sentences without hedging. - Tolerance for repetitive, high-attention work. Throughput matters; consistency matters more. - 20+ hours per week for at least three months. NICE TO HAVE - Prior AI training, annotation, or RLHF work (any provider). - A math or competitive programming background. - Experience with evaluation harnesses: lm-eval-harness, custom pytest-style suites, or an internal eval platform. - A second language, for multilingual evaluation work. HOW WE WORK Tools: our internal annotation platform, GitHub, Linear, Slack. - Async-first. Written feedback beats meetings; decisions get recorded. - Weekly calibration sessions: the whole team grades the same 10 samples and reconciles every disagreement. This is the most useful hour of the week. - No screenshot surveillance or keystroke tracking. We measure the quality and consistency of what you produce, not how long your cursor moved. RATE AND TERMS - $40 - $55 per hour on a 6-month, extendable contract, set by domain depth and calibration performance. Invoiced twice monthly, paid within 10 business days. - Onboarding and ramp-up are paid at the full rate. The first two weeks are structured: rubric training, shadow grading, then supervised grading. - No exclusivity. You may hold other contracts as long as quality holds. - Contractor and referral candidates are both welcome. HOW WE'LL KNOW IT'S WORKING - Weeks 1-2: calibrated on the rubric set, with agreement against senior graders at or above 0.75. - Months 1-2: you own one evaluation suite end to end, from rubric to published results. - Month 3 onward: your rubric proposals ship with light edits, and your failure analyses start appearing in release notes. We hire on demonstrated ability. We do not discriminate on the basis of race, religion, gender, age, disability, or national origin.

Project ID: 40812072

About the project

33 proposals

Open for bidding

Remote project

Active 4 secs ago

Place your bid

Bid amount Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

It's free to sign up and bid on jobs

33 freelancers are bidding on average $45 USD/hour for this job

User Avatar
Flag of UNITED STATES

Hi, This is Elias from Miami. I checked your project description and understand you’re looking for an AI Code Evaluation Specialist to help with reasoning and code evaluation. It sounds like you need someone who can effectively train AI models and evaluate their coding capabilities. I’ve worked on several AI projects that involve training models and evaluating their performance, so I’m familiar with the key technical challenges. I’d be happy to go through the details and suggest the best technical approach. I have a few questions to get a better understanding: Q1 – What specific programming languages or frameworks will the AI be evaluated against? Q2 – Are there any existing models or systems that you want to integrate with? Q3 – What is the expected volume of code or data that the AI will handle for evaluation? Looking forward to hearing from you.

User Avatar
Flag of PAKISTAN

Hi, I understand you need someone who can test coding assistant answers carefully, find hidden bugs, and explain each decision clearly. I can use Python and Coding experience to review generated solutions, reproduce failures, check edge cases, and separate correct answers from convincing but broken ones. My Debugging approach would include focused test cases, API checks, boundary testing, and short failure reports that another grader can follow. I can also support Automation for repeatable evaluation suites and help keep scoring consistent during calibration. Which programming areas and evaluation harness will I handle first, and how are rubric scores calibrated across graders? I can commit to 20+ hours each week and work well in an async team using written feedback. Please start chat and let’s discuss the evaluation workflow and first suite.

User Avatar
Flag of UNITED STATES

Hello!, This is James from Hollywood... This role fits what I do best: careful reasoning, code review, and catching the small issues that usually slip through fast-paced AI training work. I understand you are not just looking for someone who can label code, but someone who can judge correctness, explain why, and keep feedback consistent and useful. My approach is practical: 1. Read the task and intent carefully 2. Evaluate the code or reasoning step by step 3. Flag logic gaps, edge cases, hallucinations, and weak explanations 4. Give clear feedback that improves model quality I’m used to technical systems where precision matters, so I treat every evaluation like it will affect the next production output. If needed, I can also help tighten the rubric so scoring stays consistent across tasks. A few quick questions: 1. Is the main focus code correctness, reasoning quality, or both? 2. Do you already have a rubric for edge cases and partial credit? 3. What does a “perfect” submission look like for your team? Relevant work includes: - AI review workflow for a SaaS support assistant - Python-based code QA pipeline for internal engineering - LLM evaluation tooling for reasoning consistency - Data annotation guidelines for a technical AI team If you want someone who pays attention to details and keeps the quality bar high, I’d be glad to jump in.

User Avatar
Flag of TANZANIA, UNITED REPUBLIC OF

Hello, I catch the bugs that look fine until you dig in: off-by-ones, APIs that never existed, tests that pass because they never assert anything. That judgment call is the core of grading coding assistants at scale, and it is how I work day to day. My approach stays practical. I lock a clear rubric for each task type so a second person can land on the same score without pinging me. Then I build adversarial prompt packs aimed at the failure modes you already care about, and turn every miss into a short reproducer research can act on the same week. For pairwise prefs I keep the rationale tight so the reward signal stays stable enough to train on. I can put in 40 hours each week and stick with weekly calibration so agreement holds. Happy to ramp through your paid onboarding path: rubric training, shadow grading, then supervised scoring.

User Avatar
Flag of PAKISTAN

Hi, Evaluating coding assistants requires precise code review, reproducible test cases, and clear grading rubrics to identify subtle reasoning errors and produce reliable training datasets. A few questions: - Which programming languages and technical domains will the initial evaluation tasks primarily cover? - Are grading rubrics already established, or will trainers develop them from scratch? - Which evaluation platform or framework is used to manage scoring, pairwise comparisons, and inter-annotator agreement? Best regards, Muhammad Usman

User Avatar
Flag of PAKISTAN

I understand you need an AI Code Evaluation Specialist to rigorously assess and refine coding assistants. My recent work involved developing automated test suites for a similar AI-powered code generation tool, achieving a 15% reduction in false positive evaluations by identifying subtle logical errors and edge cases that manual review often missed. I'm confident I can bring that same analytical rigor to your project. My approach will involve a multi-faceted strategy: first, I'll establish a baseline by manually evaluating a representative sample of code outputs against predefined correctness criteria and common pitfalls. Concurrently, I'll identify opportunities to automate repetitive checks using Python scripting and leverage static analysis tools like Pylint and Bandit to flag syntactic and security vulnerabilities. For reasoning evaluation, I will design targeted prompt templates to probe logical consistency and identify flawed deductive steps, documenting each finding with clear explanations and actionable improvement suggestions. To ensure the highest quality evaluations, could you clarify the specific programming languages and frameworks most critical for your clients' coding assistants? Also, what existing evaluation metrics or frameworks are currently in place? I'm eager to discuss how my skills can directly contribute to enhancing your clients' AI coding tools.

User Avatar
Flag of PAKISTAN

I can contribute strong technical judgment to evaluating coding assistants, particularly in identifying incorrect implementations, flawed API usage, edge cases and misleading explanations. My experience with PHP, Laravel, JavaScript, API integrations and SaaS applications gives me practical exposure to reviewing application logic and troubleshooting real-world issues. I can write precise evaluation rationales, develop reproducible test cases, compare alternative solutions and turn identified failures into actionable feedback. I’m comfortable working with structured rubrics, Git-based workflows and detailed technical documentation, with a strong focus on consistency and accuracy.

User Avatar
Flag of UNITED ARAB EMIRATES

I can help you rigorously grade and stress test. Here's how you get it: we map your current process, then you get Python built and connected to Training, tested with real examples, plus a short video walkthrough so your team can run it confidently. You'd be working with an AI implementation specialist who builds LLM workflows, chatbots and voice AI with n8n, Make, Claude AI and Relevance AI, and founded an AI automation platform for clinics. Which tools do you use today, and which step takes your team the most time? Let me know if you'd like to move forward with this plan. Best regards, Salma Noreen

User Avatar
Flag of GERMANY

Hello, "Python Code Evaluation" - finding bugs that pass superficial review and turning model failures into reproducible evidence. I'll assess outputs against clear correctness criteria, verify questionable claims with focused test cases, and write concise rationales that another evaluator can apply consistently. I'll also look for edge cases such as off-by-one errors, misleading tests, and hallucinated APIs, then turn confirmed failures into actionable evaluation examples. Are the initial tasks focused mainly on Python generation and debugging, or will they also require TypeScript or Go? Looking forward to working with you. Beate

User Avatar
Flag of NETHERLANDS

Hello, Evaluating code generation models requires distinguishing between code that merely executes and code that is genuinely correct under subtle edge cases, such as concurrency pitfalls or off-by-one errors in state management. My background in production software engineering involves rigorous code review where spotting these hidden logic flaws before deployment is standard practice, which directly translates to writing reproducible failure reports and stable grading rubrics for AI training pipelines. Maintaining high inter-annotator agreement relies on clear boundary definitions when a model output sits right on the edge of plausibility, ensuring the resulting reward signal remains consistent for the research team. What specific domain or language subset currently presents the highest rate of false positives in your existing evaluation suites? Have a nice day

User Avatar
Flag of INDIA

I can start right now I have checked the details. I have 3+ years writing production Python and TypeScript, and I regularly hunt down those quiet off-by-one errors and hallucinated APIs during code reviews. I can build adversarial prompt sets and write clear grading rationales that keep inter-annotator agreement above 0.8. I am also fully available for your weekly calibration sessions and can commit to 20+ hours per week. I have one question: for the adversarial prompt sets, do you prefer we target specific language ecosystems first, or branch across Python, TypeScript, and Go concurrently during the first two weeks of rubric training? lets do it

User Avatar
Flag of UNITED STATES

Hello, Thank you for outlining such a thoughtful and high-impact opportunity. I’m excited by the prospect of helping to shape coding assistants used by thousands of developers and to contribute to the rigorous evaluation processes that advance this technology. With over five years of experience writing production Python code and conducting code reviews for both startups and established platforms, I’ve developed a keen eye for subtle bugs, logical inconsistencies, and quietly broken edge cases. My background includes building and maintaining grading rubrics for technical assessments, crafting adversarial test cases, and translating nuanced code errors into clear, actionable feedback for both human and machine learning audiences. I’ve previously worked on AI training and annotation projects, including RLHF tasks and evaluation harnesses such as lm-eval-harness and custom pytest-style suites. My approach centers on consistency, clarity, and high attention to detail—qualities I see are critical for this role. I thrive in async-first, feedback-driven environments and am comfortable defending and iterating on evaluation boundaries in writing. If selected, I would dive in by quickly calibrating to your rubrics through structured learning and collaborative grading sessions, then focus on producing reliable, reproducible evaluation data and failure analysis that directly inform model improvement. I look forward to helping uphold and raise your standards for code evaluation, and to partnering closely with your research and engineering teams. Thank you for considering my application. I would love to discuss how my experience aligns with your mission and how I can contribute to your team’s success.

User Avatar
Flag of VIETNAM

Hello, Production Python engineering in AI systems makes me well suited to evaluate coding assistants where plausible answers must be separated from genuinely correct ones. I build real-world legal technology, including document-processing pipelines, RAG systems, contract automation, and API integrations, so I am accustomed to tracing multi-step behavior and identifying subtle failures rather than judging code by appearance. My work includes parsing 200-page scanned court filings, extracting contract clauses with Python and FastAPI, and building an AI contract search assistant for a 40,000+ NDA library with source-grounded answers. That experience directly supports reviewing hallucinated APIs, incomplete reasoning, incorrect citations, edge cases, and outputs that appear convincing but fail under scrutiny. I can turn those findings into concise, reproducible technical feedback and consistent evaluations. I would be a strong choice for careful, high-attention code evaluation and reasoning-quality assessment, and I am ready to contribute as a reliable Python-focused trainer. Thanks!

User Avatar
Flag of PAKISTAN

Hi I have a Bachelor's degree in Physics and strong analytical, logical reasoning, and problem-solving skills, along with experience in full-stack web development and debugging. My development work involves reviewing code, identifying errors, testing functionality, and finding practical solutions to technical problems. I’m also comfortable writing clear technical explanations, evaluating AI-generated outputs, and following structured guidelines. I’m interested in contributing to your AI evaluation projects with accuracy, consistency, and attention to detail. Thanks

User Avatar
Flag of PAKISTAN

Hello, I have gone through "AI Code Evaluation Specialist" and I know how to deliver this cleanly. How I would approach it: - Break the design or requirements into reusable components and agree the scope with you up front. - Build a fast, responsive and accessible interface that matches your design and connects to your data or API. - Test across devices and browsers, then deploy and hand over clean, readable code. I have delivered many modern frontend projects and I keep progress visible at every milestone. Do you have designs or a reference site I can look at, so I can confirm the exact scope? Tarim

User Avatar
Flag of AUSTRALIA

Hello, Your team needs more than basic code review: it needs precise evaluation of AI-generated code, reproducible failure cases, and consistent reasoning that can directly improve model performance. I would approach this through structured grading rubrics, Python-based evaluation tests, adversarial prompts, and pairwise comparisons. Each evaluation would include a clear score, technical rationale, reproducible test case, and specific failure classification. I would also develop targeted datasets from identified weaknesses rather than relying on generic examples. A key challenge is distinguishing code that appears correct from code that actually satisfies the requirements. I would address this through edge-case testing, independent verification, regression tests, and rubric calibration to maintain consistent grading standards. The work would produce actionable evaluation reports, reusable test suites, and documented failure patterns that support model improvement. Would the initial assignment focus primarily on Python code generation and debugging, or would it cover multiple programming languages and mathematical reasoning from the beginning? Thank you,

User Avatar
Flag of UNITED STATES

Hi, I'd be happy to help move "AI Code Evaluation Specialist" forward. I'm Silas, a Full Stack Developer and AI Engineer. I build web applications, backend services, API integrations, and AI-powered features using technologies like React, Node.js, TypeScript, and Python. My focus isn't just on getting features working. I also pay attention to performance, reliability, code quality, and how everything fits together in the long run. Could you tell me a bit about the current stage of your project? Kind regards, Silas

内容来自 Freelancer News,版权归原作者所有。

阅读原文 ↗

把喜欢的内容,留在你的新标签页。

打开我的订阅