Senior Software Engineer, AI Evaluation Infrastructure

Job not on LinkedIn

🔥 10 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🤖 AI Engineer

👻 Ghost score 12%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Riva Scientific

WebsiteLinkedInAll Job Openings

11 - 50 employees

💼 Consulting

🏥 Healthcare

📦 Logistics

Consulting • Healthcare • Logistics

Riva Scientific is a company that offers innovative solutions in the Life Sciences and Biotechnology sectors, as well as Information Technology services. They provide a comprehensive range of products and services including managed services, strategic consulting, and staffing solutions tailored to the unique challenges faced by Pharmaceuticals and Technology companies. Riva Scientific specializes in laboratory automation, software validation, custom application development, and staffing, helping clients navigate complex regulatory environments and improve operational efficiencies.

📋 Description

• Build and operate production backend systems • Own services from design through on-call • Develop and maintain AI/LLM evaluation infrastructure • Build benchmark tasks, score agent trials, and aggregate and compare results • Assess task quality, verifiers, answer leakage, and test validity • Make pragmatic system design decisions and extend existing systems where appropriate • Implement migrations, tests, and handling for user-facing edge cases • Collaborate with TypeScript and React frontend changes when needed • Document design rationale in pull requests and design notes

🎯 Requirements

• Typically 5+ years building and operating production backend systems • Ownership of at least one service from design through on-call • Strong Python • Experience with Django or a similar framework • Experience with PostgreSQL • Experience with a task queue such as Celery • Hands-on experience in LLM evaluation beyond calling a model API • Understanding of benchmark task construction, agent trial scoring, and result aggregation/comparison • Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar harness is a strong plus • Ability to identify ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason • Ability to make pragmatic design tradeoffs and explain what was not built • Ability to take ownership of ambiguous requests and complete work including migrations, tests, and edge cases • Ability to quickly adopt team conventions • Comfort with a TypeScript and React frontend when changes are visible there • Clear technical writing in pull request descriptions and design notes

🏖️ Benefits

• Remote work • Full-time employment