Technical AI Evaluation Analyst – LATAM

Job not on LinkedIn

🔥 0 minutes ago

🌐 Argentina, Brazil, +4 more countries – Remote

🇦🇷 Argentina – Remote 🇧🇷 Brazil – Remote 🇲🇽 Mexico – Remote 🇺🇾 Uruguay – Remote 🇵🇪 Peru – Remote 🇨🇱 Chile – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 12%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Gramian Consulting

WebsiteLinkedInAll Job Openings

2 - 10 employees

Founded 2025

💼 Consulting

📦 Logistics

📣 Marketing

Consulting • Logistics • Marketing

Gramian Consulting is a remote-first consulting firm that connects engineering and data/AI talent with organizations through talent augmentation, recruiting, dedicated teams, and contractor management. The firm provides Data & AI services including LLM training and fine-tuning, AI agents and assistants, MLOps, and AI infrastructure, and it offers mentorship and education programs for career readiness, interview preparation, and international market orientation. Rooted in hands-on engineering and recruiting experience, Gramian helps clients scale technical teams and extract business value from AI while developing individual talent.

📋 Description

• Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness. • Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified. • Audit grading logic to identify brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties for valid alternative solutions. • Investigate discrepancies between model performance, grader results, and expected outcomes. • Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures. • Independently assess automated QC findings rather than accepting them without verification. • Document concise, evidence-backed findings and provide actionable, reproducible feedback. • Flag uncertainty and verify that implemented revisions resolve previously identified issues.

🎯 Requirements

• Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs. • Experience reviewing technical workflows, software behavior, data outputs, or evaluation logic. • Strong analytical skills, including the ability to verify calculations and reconcile conflicting evidence. • Demonstrated ability to assess the correctness and completeness of technical deliverables. • Strong written English with experience providing clear, specific, and reproducible feedback. • High attention to detail when identifying inconsistencies, missing information, and evaluation defects.