AI Evaluation Engineer (Python, QA or Security)
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What this is NOT
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What we look for
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Why this is hard
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Compensation
Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.
Similar roles
Senior Project Engineer – Security Systems & ELV
The Supreme HR Advisory Pte Ltd·Singapore, Singapore
The Supreme HR Advisory Pte Ltd is hiring a Senior/ Project Engineer for the Camps Team in Singapore. The role focuses on leading security equipment installation, testing, commissioning and system programming across projects islandwide, including site travel. Strong project management experience in security systems and ELV knowledge are essential. Expect a Monday–Friday schedule with on-site requirements; night/weekend work may be…
- Full-time
- SGD 600–SGD 732 / yr
- Employment Pass — tied to employer
Client Advisor- Hamilton
Amplifon·New Zealand
Description Client Advisor – Hamilton - Part time, Monday/Thursday/Friday, 8:30am–5:00pm. This role requires flexibility to cover clinics across the region as needed. Responsibilities for this role include; * Providing gold standard client care * Welcome customers with warmth and professionalism * Manage appointments and front desk duties * Assist with hearing aid repairs and over-the-counter sales * Conduct basic hearing…
- Part-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Operations Manager
Evolution Healthcare HR·Wellington, New Zealand
Introduction About Wakefield Hospital Wakefield Hospital is entering an exciting new chapter following the opening of our state-of-the-art hospital facility. We're seeking an experienced Operations Manager to join our senior leadership team and play a key role in shaping the future performance, culture and growth of the hospital. This is a highly visible leadership role with responsibility for multiple operational…
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Account Executive Industry Sales, AWS Aotearoa / New Zealand
Amazon·New Zealand
As an Account Manager, Enterprise, you will own strategic enterprise accounts across New Zealand and drive adoption of AWS services to accelerate digital transformation and business growth. Working with C-level executives, you will lead transformation efforts helping customers modernize applications, leverage Generative AI and agentic AI capabilities, optimize cloud economics, and scale their operations globally. Engaging technology leaders, product teams,…
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Team Member Wairau Park
Nando’s·New Zealand
At Nando’s, it’s all about the people, with chicken on the side! Join a lively crew, earn money, learn life skills, and have fun discovering your best self. Whether you’re greeting customers with a smile or perfecting PERi-PERi magic on the grill, we’ll help you grow—no experience needed, just energy, passion and a hunger to learn. It’s more than just…
- Part-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Kaiārahi (Navigator) Whānau Ora
Te Korowai Hauora o Hauraki·Thames, New Zealand
Our Whānau Ora team are looking to recruit a Kaiārahi (Navigator) Whānau Ora! Our Thames-based Whānau Ora Navigators work alongside identified priority whānau to develop plans, identify aspirations and goals, and provide ‘wrap around’/multi-disciplinary support by drawing on a range of approaches to support whānau in achieving their outcomes. Employment type: Fixed Term until 30 June 2027 (1.0 FTE/40hrs per…
- Contract
- Skilled Migrant (Residence) — PR, points-based, no single employer
Customer Support Specialist
The Network for Learning·Christchurch, New Zealand
* Grow your technical knowledge and skills in a supportive and collaborative environment * Varied role with a balance between handling phone calls and managing email queries * 12 Month Fixed Term Contract * WFH Flexibility About Us Network for Learning (N4L) is a Crown-owned technology company that provides faster, safer internet for Aotearoa New Zealand’s schools and kura. But…
- Contract
- Skilled Migrant (Residence) — PR, points-based, no single employer
Warehouse & Customer Service Assistant - Palmerston North
Anzor·Palmerston North, New Zealand
Company Description Anzor is the leading supplier of stainless steel fasteners in New Zealand, offering the widest range and largest stock of stainless fasteners and fittings in the country. As a 100% Kiwi-owned and operated company, Anzor has been providing top-quality products and exceptional customer service for over three decades. Job Description We are literally the 'Nuts and Bolts' of…
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
