Senior Software Engineer - Agent Evaluation - Freelance/Remote 100 openings
Open to candidates in the North America, South America, Asia and Europe.
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involves
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What this is NOT
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What we look for
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Why this is hard
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Effort estimate
Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.
The process is designed to move quickly and typically includes the following steps:
- App review and invitation to a virtual project introduction session (approximately 30 minutes)
- Platform registration and identity verification
- Technical assessment (approximately 35 minutes)
- Background check (completed at no cost to candidates)
- Onboarding and project-specific training tasks
- Begin production work!
- Willingness to complete identity verification as part of the onboarding process.
- Ability to complete a technical assessment.
- Willingness to join and participate in Discord, which will be used for project communication and updates.
- Successful completion of a background check is required prior to onboarding.
- Reliable internet connection and ability to communicate effectively in a remote environment.
Company
Braintrust is a global talent network that connects top independent professionals with leading companies for high-quality, flexible work. We help organizations hire skilled talent faster while giving professionals access to vetted opportunities with innovative teams.
Similar roles
Software Engineer - Engineering Enablement
Phocas Software·Christchurch, New Zealand
More About Phocas: Phocas Software is a profitable, growing SaaS company with a loyal customer base and a category-leading product. With a clear path toward scaling from $70M to $250M and a reimagined GTM structure, this is a prime moment to join and shape the future of how we deliver value to customers. We know that fun is different for…
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Senior Java Backend Engineer - WCS Migration
Mindera·Bengaluru, India
We are looking for a Senior Java Backend Engineer with strong experience in Java, Spring, MongoDB and IBM WebSphere Commerce (WCS) to support a major platform modernisation initiative. The team is migrating away from WCS as the source of truth towards a modern backend stack based on Java/Spring and MongoDB. The key requirement is strong hands-on WCS knowledge, particularly around…
- Remote
- Full-time
Full Stack Developer - Java & React (f/m/d)
ecosio·Portugal
<div> <h2 class="title">Company Description</h2> </div> <div class="wysiwyg"> <p>ecosio is a fast-growing, innovative service company and a leading provider of B2B integration, specialising in electronic data interchange (EDI), Web EDI and e-invoicing. ecosio is part of Vertex, Inc., a leading global provider of indirect tax solutions listed on Nasdaq (VERX).<br><br>Our brand slogan is <em data-stringify-type="italic">Connections That Work</em> as we believe strong connections…
- On-site
- Full-time
Senior Software Engineer
Teleport·Gurugram, India
About Us At Teleport, our mission is to move things across Southeast Asia better than anyone else. We are an integrated end-to-end logistics solution, making it seamless for freight forwarders, shippers, eCommerce platforms, and small merchants to transport goods in a fast, reliable, and cost-effective manner. As we scale and expand into new markets, we are looking for Software Engineers…
- Hybrid
- Full-time
Working Student Frontend Engineer (m/f/d)
raisin·Berlin, Germany
<div class="content-intro"><p class="sc-gEkIjz cjtSrT MuiTypography-root MuiTypography-body1 sc-hoLEA">Raisin is the world’s leading platform for savings and investment products. Founded in 2012, the <strong>FinTech</strong> connects consumers with banks in the EU, the UK and the US. This gives consumers <strong>better interest rates</strong> and banks a <strong>diversified form of refinancing</strong>. Our vision is to offer savings and investments without barriers and thus open up…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Software Engineer (M/W/D) – AI-Native Development
INSIDE M2M GmbH·Garbsen, Germany
WAS WIR TUN Als IoT Pioniere entwickeln wir mit über 40 Kollegen*innen an drei Standorten IoT-Komplettlösungen für unsere Kunden. Aktuell sind weltweit über 500.000 Geräte und Anlagen mit unseren Lösungen für Machine-to-Machine-Kommunikation ausgestattet. Unsere Kompetenz liegt in der Entwicklung kundenspezifischer IoT-Lösungen, basierend auf unserer Erfahrung, unseren erprobten Software-Bausteinen und einer Vielzahl von bereits realisierten Projekten. UNSER TEAM Seit der Firmengründung…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Integration .Net Developer
Centorrino Technologies·Darwin, Australia
At Centorrino Technologies (CT), we’re more than just tech—we’re a community that goes beyond expectations. We’ve been recognised as a Great Place to Work in 2025 and Top 5 Inspiring Workplaces for 2026, with an outstanding eNPS score of 68. And we’re not stopping there. We're on a mission to redefine the customer experience, and we are seeking an experienced…
- Full-time
- Skilled Independent 189 — PR, no sponsor
Application .Net Developer
Centorrino Technologies·Darwin, Australia
At Centorrino Technologies (CT), we’re more than just tech—we’re a community that goes beyond expectations. We’ve been recognised as a Great Place to Work in 2025 and Top 5 Inspiring Workplaces for 2026, with an outstanding eNPS score of 68. And we’re not stopping there. We're on a mission to redefine the customer experience, and we need a passionate Application…
- Full-time
- Skilled Independent 189 — PR, no sponsor
