Senior Evaluation AI Engineer

Job not on LinkedIn

🔥 0 minutes ago

🌐 Mexico, Brazil, +1 more countries – Remote

🇲🇽 Mexico – Remote 🇧🇷 Brazil – Remote 🇦🇷 Argentina – Remote

⏰ Full Time

🟠 Senior

🤖 AI Engineer

👻 Ghost score 10%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Helpware

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 2015

📦 Logistics

📣 Marketing

💼 Consulting

💰 Seed Round on 2016-01

Logistics • Marketing • Consulting

Helpware is a digitally enabled People as a Services Outsourcer dedicated to enhancing customer experiences for high-growth, market-disrupting, and technology companies. They specialize in providing a wide array of services such as digital customer experience, content control and verification, AI-driven customer service, back-office support, and data labeling. Helpware also assists in areas like digital marketing support, cybersecurity, and software development. Operating globally, they offer culturally aligned and scalable team solutions across multiple countries. Helpware aims to drive brand loyalty and efficiency for its clients by providing human task services integrated with technology solutions.

📋 Description

• Architect, design, implement, and scale custom automated evaluation pipelines for multi-agent LLM systems • Evaluate intent classification, tool-calling correctness, state-transition integrity, checkpointer manipulation, and multi-turn memory replay • Develop advanced domain-specific metrics using G-Eval, LLM-as-a-judge architectures, and deterministic verification methods • Evaluate complex document parsing, including leases and invoices, and real estate financial modeling • Implement defensive alignment testing, adversarial red-teaming, and prompt-injection mitigation • Prevent hallucination, data leakage, and unauthorized tool execution within corporate cloud environments • Construct evaluation datasets, golden regression sets, and synthetic-data pipelines modeling property manager, tenant, and broker behaviors • Monitor production agent workflows for behavioral drift, latency regressions, cost inefficiencies, and token utilization • Design, develop, and maintain scalable evaluation frameworks and pipelines using Python, SQL, APIs, and CI/CD • Execute hundreds or thousands of evaluation cases across models, prompts, RAG pipelines, and AI agents • Implement automated grading using deterministic validation, reference-based evaluation, human-defined rubrics, LLM-as-a-judge, and human-in-the-loop evaluation • Establish scoring criteria and quality thresholds for AI use cases • Partner with Engineering, Product, Data, and Quality Engineering teams to ensure AI capabilities are measurable, explainable, reliable, and production-ready

🎯 Requirements

• 5 to 7 years of experience in software engineering, quality engineering, test automation, AI engineering, machine learning, or a related technical role • Hands-on experience evaluating or testing LLM, NLP, machine learning, or AI applications • Strong Python skills and experience building automated test or evaluation workflows • Experience creating and maintaining golden datasets, test datasets, or benchmark datasets • Experience with adversarial testing • Familiarity with AI Safety, security & Privacy, and Responsible AI evaluation • Deep structural expertise in LangChain, LangGraph (including sub-graphs and conditional routing) and LangSmith • Hands-on experience using LangSmith for tracing, debugging, experiments, datasets, and evaluations • Strong GitHub experience, including repositories, branching, pull requests, code reviews, and CI/CD workflows • Experience with Claude Code or similar AI-assisted development tools • Experience with AWS and Amazon Bedrock • Experience with REST APIs, JSON, SQL, and modern application architectures • Strong understanding of LLMs, prompt engineering, embeddings, vector search, RAG, and AI agents • Experience with automated testing, regression testing, integration testing, and end-to-end testing • Hands-on experience evaluating LangGraph-based or multi-agent applications at the node and state-transition level • Strong analytical, problem-solving, and communication skills • Ability to work independently and take ownership of ambiguous technical problems • Preferred: Experience with Ragas, DeepEval, Arize Phoenix, TruLens, Promptfoo, or similar evaluation tools • Preferred: Experience with LLM-as-a-judge and human-in-the-loop evaluation approaches • Preferred: Experience with Bedrock Agents, Knowledge Bases for Amazon Bedrock, Lambda, S3, or CloudWatch • Preferred: Experience integrating AI evaluations into GitHub Actions

🏖️ Benefits

• 6-month duration with a high chance of prolongation for good performance • Long-term role potential

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume