Bilingual Llm Evaluator

 

Description:

Role Responsibilities
 

  • Evaluate LLM-generated responses on their ability to effectively answer user queries.
  • Conduct fact-checking using trusted public sources and external tools.
  • Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.
  • Assess reasoning quality, clarity, tone, and completeness of responses.
  • Ensure model responses align with expected conversational behavior and system guidelines.
  • Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.
     

Qualifications

Must-Have
 

  • Bachelor’s degree
  • Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in French
  • Significant experience using large language models (LLMs)
  • Excellent writing skills
  • Strong attention to detail
  • Adaptable and comfortable moving across topics
  • Background or experience in domains requiring structured analytical thinking
  • Excellent college-level mathematics skills

Apply Now
You'll sign in or register, then come back to this job.
No CV yet? Make one free and apply.
Organization Mercor
Industry Other Jobs
Occupational Category Bilingual LLM Evaluator
Job Location Dublin, Ireland
Shift Type Morning
Job Type Full Time
Gender No Preference
Career Level Intermediate
Experience 2 Years
Posted at 2026-03-26 10:16 pm
Expires on 2026-12-22