Senior Software Engineer (Python) - Agent Evaluation - Freelance/Remote 100 openings
***This opportunity is intended for experienced Senior Python Engineers only****
Open to candidates in the North America, South America, Asia and Europe.
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involves
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What this is NOT
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What we look for - You must meet all the requirements in order to be considered for this project:
- 5+ years of professional experience with Python.
- Strong experience with FastAPI, pytest, and async/await.
- Hands-on experience with Docker, PostgreSQL, and CI/CD pipelines.
- Proven experience writing and maintaining automated tests (not just executing them).
- Full-stack experience with React and TypeScript is a plus.
- English proficiency at B2 level or higher.
- Availability to work 30+ hours per week.
Why this is hard
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
The process is designed to move quickly and typically includes the following steps:
- App review and invitation to a virtual project introduction session (approximately 30 minutes)
- Platform registration and identity verification
- Technical assessment (approximately 35 minutes)
- Background check (completed at no cost to candidates)
- Onboarding and project-specific training tasks
- Begin production work!
- Willingness to complete identity verification as part of the onboarding process.
- Ability to complete a technical assessment.
- Willingness to join and participate in Discord, which will be used for project communication and updates.
- Successful completion of a background check is required prior to onboarding.
- Reliable internet connection and ability to communicate effectively in a remote environment.
Company
Braintrust is a global talent network that connects top independent professionals with leading companies for high-quality, flexible work. We help organizations hire skilled talent faster while giving professionals access to vetted opportunities with innovative teams.
Similar roles
Software Engineer - Engineering Enablement
Phocas Software·Christchurch, New Zealand
More About Phocas: Phocas Software is a profitable, growing SaaS company with a loyal customer base and a category-leading product. With a clear path toward scaling from $70M to $250M and a reimagined GTM structure, this is a prime moment to join and shape the future of how we deliver value to customers. We know that fun is different for…
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Senior HubSpot & Clio Solutions Architect / CRM Automation Developer - Remote
Oikonomakis Law Firm·India
The Role An international law firm is looking for one highly capable technical specialist to take end-to-end ownership of its HubSpot and Clio ecosystem. This is not a conventional CRM administrator position. We need someone who can combine HubSpot architecture and administration, Clio, advanced automation, API integrations and hands-on development and who is comfortable independently solving technical problems from design…
- Remote
- Full-time
Software Developer C#/.NET (m/w/d) - DevOps
IBS Schreiber GmbH·Hamburg, Germany
Wir sind IBS Schreiber – seit über 45 Jahren ein inhabergeführtes Unternehmen für SAP-Security mit Sitz in der wunderschönen Hansestadt Hamburg. Zu unseren Geschäftsbereichen zählen unsere Akademie, das Prüfungs- und Beratungsgeschäft sowie die Entwicklung von Softwarelösungen, die Unternehmen helfen, Zugriffsrisiken zu beherrschen und IT-Security einfach zugänglich zu machen. Unser technisches Herzstück: Der Bereich Product & Technology. Hier entwickeln wir skalierbare,…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Senior Java Backend Engineer - WCS Migration
Mindera·Bengaluru, India
We are looking for a Senior Java Backend Engineer with strong experience in Java, Spring, MongoDB and IBM WebSphere Commerce (WCS) to support a major platform modernisation initiative. The team is migrating away from WCS as the source of truth towards a modern backend stack based on Java/Spring and MongoDB. The key requirement is strong hands-on WCS knowledge, particularly around…
- Remote
- Full-time
Full Stack Developer - Java & React (f/m/d)
ecosio·Portugal
<div> <h2 class="title">Company Description</h2> </div> <div class="wysiwyg"> <p>ecosio is a fast-growing, innovative service company and a leading provider of B2B integration, specialising in electronic data interchange (EDI), Web EDI and e-invoicing. ecosio is part of Vertex, Inc., a leading global provider of indirect tax solutions listed on Nasdaq (VERX).<br><br>Our brand slogan is <em data-stringify-type="italic">Connections That Work</em> as we believe strong connections…
- On-site
- Full-time
Senior Software Engineer
Teleport·Gurugram, India
About Us At Teleport, our mission is to move things across Southeast Asia better than anyone else. We are an integrated end-to-end logistics solution, making it seamless for freight forwarders, shippers, eCommerce platforms, and small merchants to transport goods in a fast, reliable, and cost-effective manner. As we scale and expand into new markets, we are looking for Software Engineers…
- Hybrid
- Full-time
Working Student Frontend Engineer (m/f/d)
raisin·Berlin, Germany
<div class="content-intro"><p class="sc-gEkIjz cjtSrT MuiTypography-root MuiTypography-body1 sc-hoLEA">Raisin is the world’s leading platform for savings and investment products. Founded in 2012, the <strong>FinTech</strong> connects consumers with banks in the EU, the UK and the US. This gives consumers <strong>better interest rates</strong> and banks a <strong>diversified form of refinancing</strong>. Our vision is to offer savings and investments without barriers and thus open up…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Software Engineer (M/W/D) – AI-Native Development
INSIDE M2M GmbH·Garbsen, Germany
WAS WIR TUN Als IoT Pioniere entwickeln wir mit über 40 Kollegen*innen an drei Standorten IoT-Komplettlösungen für unsere Kunden. Aktuell sind weltweit über 500.000 Geräte und Anlagen mit unseren Lösungen für Machine-to-Machine-Kommunikation ausgestattet. Unsere Kompetenz liegt in der Entwicklung kundenspezifischer IoT-Lösungen, basierend auf unserer Erfahrung, unseren erprobten Software-Bausteinen und einer Vielzahl von bereits realisierten Projekten. UNSER TEAM Seit der Firmengründung…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
