AI Evaluation Engineer

Job not on LinkedIn

🔥 9 minutes ago

🌏 Anywhere in the World

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 10%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Scopic

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2006

🏥 Healthcare

📣 Marketing

📦 Logistics

Healthcare • Marketing • Logistics

Scopic is a software development company specializing in custom software solutions, mobile and web development, AI consulting, and digital marketing services. With expertise in various technologies such as PHP, Node. js, and AWS, Scopic caters to diverse industries including healthcare, finance, and education. They focus on transforming client ideas into innovative applications while ensuring high-quality user experiences and effective marketing strategies.

📋 Description

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment • Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency • Implement fast, low-cost code-based checks, including JSON schema validation, regex matching, API status checks, and embedding similarity • Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops • Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to secure sandboxes and guardrails • Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools • Reduce LLM API costs through caching and mocking

🎯 Requirements

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems • Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs • Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution • Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads • Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging • Ability to independently design and own an evaluation architecture from scratch • Strong written and spoken English • At least 3 hours of daily overlap with US Central Time business hours • Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI • Deep understanding of embedding models and vector databases • Experience load testing non-deterministic endpoints

🏖️ Benefits

• Paid training and other professional opportunities • Flexible schedule and preferred working hours • Ability to work from anywhere with a stable internet connection • Consistent flow of engaging, challenging product work • Annual pay increases for diligence and strong performance • Monthly compensation payment via wire transfer