AI Specialist - TEKsystems (Allegis Group Singapore Pte Ltd)
- Lead Rigorous Model Evaluations
- Advanced Scoring Frameworks
- Latent Pattern Recognition
We are seeking an AI Evaluation Engineer to help understand, analyse, and improve the performance of advanced AI systems. This role combines data-driven analysis, evaluation design, and human-centered insights to assess AI behavior and drive continuous improvement. You will develop evaluation methodologies, analyse model outputs, identify areas for optimization, and translate findings into actionable recommendations for technical and business stakeholders.
Key Responsibilities:
- Lead Rigorous Model Evaluations: Architect and execute comprehensive evaluation suites for LLMs and multimodal models, identifying edge cases in multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
- Advanced Scoring Frameworks: Develop deterministic, heuristic, and LLM-assisted evaluation frameworks (e.g., LLM-as-a-judge, reward modeling) to quantify human-perceived quality metrics (e.g., helpfulness, hallucination rates).
- Actionable Signal Extraction: Translate qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training and inference.
- Improve Performance: Partner with engineering teams to refine model behavior, leveraging evaluation telemetry to inform prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
- Latent Pattern Recognition: Apply advanced ML techniques (e.g., embedding-based clustering, representation learning, perturbation analysis) to systematically map error taxonomies and latent failure manifolds in model outputs.
- MLOps & Automation: Develop robust MLOps workflows to codify evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines.
- Distributed Evaluation Pipelines: Architect scalable, distributed inference and processing pipelines (e.g., Ray, vLLM) for high-throughput model evaluation, automated annotation, and output analysis at scale.
- Human-Centric Metrics: Define quantitative evaluation frameworks that capture nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
- Auto-Evaluator Systems: Build automated evaluation pipelines utilizing LLMs to assess outputs at scale, optimizing for high correlation with human baseline annotations.
- Cross-Functional Partnership: Collaborate with ML researchers, software developers, and product managers across Apple to translate product requirements into scalable, reliable, and efficient model evaluation infrastructure.
- Bachelor's or Master's degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field
- Relevant industry experience in ML Engineering or Applied Research.
- Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face).
- Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
- Strong ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
- Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
- Deep familiarity with AI quality metrics, hallucination detection techniques (e.g., SelfCheckGPT), model alignment (RLHF/DPO), and LLM-as-a-judge frameworks (e.g., G-Eval, DeepEval).
- Experience building internal tools or automated pipelines for ML workflows using tools like MLflow, Weights & Biases, or similar platforms.
- Strong familiarity with advanced prompt engineering, RAG architectures (vector databases, semantic search), and Fine-Tuning.
Nice to haves:
- Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
We regret to inform that only shortlisted candidates will be notified / contacted.
EA Registration number : YAP JIA YI , R25157934 Allegis Group Singapore Pte Ltd, Company Reg No. 200909448N, EA Licence No. 10C4544
Similar roles
Senior Manager / Manager / Assistant Manager (deployment to MOH)
MOH Holdings·Singapore, Singapore
As part of the MOH HealthTech Policy Division, you will play a critical role in shaping and implementing health technology policies to ensure compliance and alignment with strategic objectives. You will develop, provide advice on and support the implementation of ICT and data policies and standards in the public healthcare sector that are in line with legislation such as the…
- Full-time
- Tech.Pass — self-sponsored, 2-yr
Biomarker Discovery Research Officer
Mirxes·Singapore, Singapore
MiRXES is a Singapore-headquartered molecular diagnostic company with R&D, manufacturing, and clinical lab operations in Singapore, USA, Japan, and China. We specialize in microRNA technologies and the development, manufacturing, and provision of diagnostic test kits and clinical services. Our mission is to enable early disease detection and provide insights for better patient care by harnessing the power of microRNA to…
- Full-time
- Tech.Pass — self-sponsored, 2-yr
Group Financial Controller / Chief Financial Officer
LHT HOLDINGS LTD·Singapore, Singapore
We are an SGX Mainboard-listed manufacturing group with multiple operating entities across Singapore and Malaysia. Our Group is at an important inflection point, with a new Board and management team driving functional transformation to strengthen our systems, controls, management capabilities and platform for future growth. We are now seeking a hands-on senior finance leader to take ownership of the Group…
- Full-time
- Tech.Pass — self-sponsored, 2-yr
Laboratory Assistant
Memorial Health·Riverton, New Zealand
Min USD $17.14/Hr. Max USD $26.56/Hr. Overview *$5,000 Sign On Bonus/2 Year Commitment* The function of this position is to perform basic phlebotomy and laboratory assistant duties. A variety of tasks fall within this classification such as blood collection...
- Full-time
- Skilled Migrant (Residence) — PR, points-based, no single employer
Senior MRI Application Specialist (f/m/d)
Orbem·Munich, Germany
We see the potential inside everything and everyone. Starting with you. Orbem uses AI to industrialize MRI, delivering non-invasive insight into everything from fruits to eggs to the human body. We are transforming what is becoming the world's largest biological dataset into actionable intelligence to reduce waste, increase quality, and accelerate the shift toward a more sustainable and healthier future.…
- On-site
- Full-time
- Blue Card — tied; settle 21–33mo
Engagement Manager, Professional Services (SEA/GCN)
Stripe·Singapore
WHO WE ARE ABOUT STRIPE Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world’s largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead. That means you have…
- Full-time
- Tech.Pass — self-sponsored, 2-yr
APAC GTM Operations Lead for Revenue & Forecasting
Digital Industries·Singapore, Singapore
Siemens is seeking a Head of APAC GTM Operations for MendixRapidMiner (MXRM) to act as the operational backbone of the APAC Specialized Sales organization. You will own the MOR, forecasting, and revenue reporting across ANZ, Japan, SEA, India, Korea, and Greater China, with Singapore preferred. You will lead 1–2 GTM Ops analysts, drive data-informed decisions, ensure Salesforce and CPQ data…
- Full-time
- SGD 3k–SGD 4k / yr
- Employment Pass — tied to employer
Senior Manager, Channel Marketing - APAC
NVIDIA Gruppe·Singapore, Singapore
What you\'ll be doing: * Lead APAC (excluding China) channel strategy formulation, aligning category sales and growth targets with internal sales teams and channel partners. * Continuously evaluate and identify key retail platform partners in each country, prioritizing engagement levels. * Collaborate with HQ campaign teams and regional teams to ensure successful execution of global campaigns and new product launches…
- Full-time
- SGD 2k–SGD 3k / yr
- Employment Pass — tied to employer
