AI Quality Engineer (Contract)
AI Quality Engineer
Location:- Hyderabad
Work Mode:- WFO (5 days a week)
Role Type:- Contractual (3 months and extension would depend on project requirement and performance)
Key Responsibilities
- Test and validate AI-generated insights, recommendations, and decision-making workflows.
- Evaluate LLM and RAG systems for accuracy, relevance, consistency, factuality, and hallucinations.
- Validate retrieval quality, context relevance, grounding, and response quality in RAG systems.
- Test AI agents and autonomous workflows across functional, negative, and edge-case scenarios.
- Define AI evaluation criteria, test datasets, quality metrics, and validation processes.
- Perform regression testing for models, prompts, RAG configurations, and AI workflows.
- Collaborate with AI/ML engineers to identify issues and improve AI system quality.
Required Skills
- Strong understanding of AI/ML and Generative AI testing.
- Hands-on experience testing LLM and RAG-based applications.
- Knowledge of LLM evaluation, hallucination detection, relevance, and response quality.
- Understanding of AI agents and recommendation systems.
- Strong analytical and problem-solving skills.
Good to Have
- Experience with PyTest and automated testing frameworks.
- Experience building automated AI evaluation and regression frameworks.
- Familiarity with tools such as RAGAS, DeepEval, LangSmith, or equivalent.
- Experience with CI/CD-based test automation, performance testing, or AI guardrails.
- Familiarity with cloud platforms (AWS/Azure/GCP) and observability tools.
Success Metrics
- High accuracy, relevance, and reliability of AI outputs.
- Strong evaluation coverage across critical AI workflows.
- Early detection and reduction of hallucinations and AI regressions.
- Reduced production AI quality issues.
- Increased confidence and trust in AI-generated insights and recommendations.
Similar roles
Sales Manager (IC role) - Hyderabad - Kompally
Design Cafe·Chennai, India
Design Cafe is hiring a Sales Manager- Hyderabad Kompally location Address: Unit No 206, Raichandani Mall, Ruby Block, Brundavan Colony, Kompally, Hyderabad, Telangana 500100 Minimum Experience - 3 plus years in inside sales or similar Industry Preference - Interior Design Firms,Real Estate, Ed tech,Retail, Automotive Roles & Responsibilities: Build pipeline and build strong relationships with prospects Own the experience we…
- Full-time
Senior Sales Manager
Datacipher Solutions Pvt Ltd·Mumbai, India
Employment Type: Full-time Required Experience: 5-10 years. Budget: As per company standard. Job Location: Mumbai, Kolkata, Pune, Ahmedabad, Cochin, Coimbatore & Bangalore Company Description Datacipher Solutions Pvt Ltd is a system integrator and professional services consulting firm specializing in advanced networking and security solutions. Founded in 2009, the company serves leading businesses across multiple continents and industry verticals. Datacipher focuses…
- Full-time
Senior Lead, HV Protection & Substation Projects
Power Automation Pte Ltd·Singapore, Singapore
Power Automation Pte Ltd is seeking an experienced Senior Project Manager for high voltage protection and control systems in Singapore. You will lead cross-functional teams through sales handover, engineering, procurement, configuration, installation, testing, and commissioning from award to closeout. The role requires a degree in Electrical/Electronic Engineering with 8 years in project delivery and 3 years in team leadership, plus…
- Full-time
- SGD 2k–SGD 3k / yr
- Employment Pass — tied to employer
Product Engineering Intern
MeisterPfad·Berlin, Germany
Product Engineering Intern (m/w/d) mit Option auf Übernahme Wir bauen das KI-native ERP-System für Sanitär-, Heizungs- und Klimabetriebe (SHK). Unsere Pre-Seed-Runde ist abgeschlossen. Seit dem Go-Live haben wir 10 Kunden für uns gewonnen. Für die nächste Wachstumsphase suchen wir einen Founder's Associate im Praktikum. Du arbeitest eng mit dem Gründerteam und übernimmst von Beginn an echte Verantwortung in einer Rolle,…
- On-site
- Internship
- Blue Card — tied; settle 21–33mo
Director of Packaging
Cambridge GaN Devices·Cambridge, United Kingdom
Director of Packaging Team – Research and Development Location – Cambridge, UK Contract – Permanent About CGD Making Sustainable Power Electronics Possible A spin-out of the Cambridge University, Cambridge GaN Devices (CGD) is a fabless semiconductor company that develops a range of energy-efficient GaN-based power devices to make greener electronics possible. The global power semiconductor market is expected to exceed…
- Full-time
Paid Social Executive
Soar With Us·India
PAID SOCIAL EXECUTIVE (CONTRACTOR) Function: Media Buying | Meta & TikTok Ads | Acquisition Strategy Responsibility: Own and drive return on ad spend for a portfolio of e-commerce clients Reports to: Growth Strategist Compensation: Contract — $2,000–$2,500 per month (dependent on experience) Note: Proficiency in English (written and spoken) is required. Candidates must be able to work within approximately 2…
- Remote
- Full-time
Produktentwickler/in - Elektrotechnik
JÜRGEN TAUSCH Recruitment und Coaching·Dresden, Germany
JÜRGEN TAUSCH Recruitment • Coaching vermittelt qualifizierte Fach- und Führungskräfte in alle Bereiche der Wirtschaft, der Dienstleistungsbranche und des Handwerks. Gemeinsam mit unserem Mandanten, einem etablierten Entwicklungs- und Produktionsunternehmen im Raum Dresden, besetzen wir zur Festanstellung die Position* Produktentwicklung/Konstruktion* Produkte entwickeln. Kunden verstehen. Serienreife gestalten Sie entwickeln anspruchsvolle elektromechanische Komponenten von der ersten Idee bis zur serienreifen Lösung. Dabei verbinden…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Product Engineer - AI Stockbroking
Bjak·Germany
About BJAK The original mission of BJAK is we believe people deserve smarter ways to plan, save and grow their money. This is the origin of our name. Started in 2019, we built the first mobile-first, insurance platform, enabling insurance to be accessible online by millions in the region. Today, its the leading insurance platform in Southeast Asia. Today, we…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
