LLM Evaluation Engineer

🕒 March 28

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🔴 Lead

👷🏻‍♀️ Engineer

👻 Ghost score 60%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

ThirdLaw Molecular

WebsiteLinkedInAll Job Openings

1 - 10 employees

Founded 2020

🧬 Biotechnology

💊 Pharmaceuticals

⚕️ Healthcare Insurance

Biotechnology • Pharmaceuticals • Healthcare Insurance

ThirdLaw Molecular is a pioneering biotechnology company that has developed a groundbreaking platform utilizing Spiroligomer™ molecules for targeted drug discovery and diagnostics. Founded by Dr. Christian Schafmeister, the company focuses on creating a new class of therapeutics that are safer and more effective by employing a unique chemical platform for the design of synthetically engineered molecules. Their innovative approach enables highly selective binding to proteins, enhancing the potential for both drug therapies and diagnostic applications.

📋 Description

• Build the evaluation layer in the ThirdLaw platform for LLM prompts and responses • Design and tune guardrails, classifiers, and semantic judgment systems in real-time • Implement evaluation strategies with semantic similarity, foundation model scoring, and rule-based systems • Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking) • Prototype, tune, and productize small language models for classification, labeling, or scoring • Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage • Build tools to observe, debug, and improve evaluator performance across data distributions • Define abstractions for reusable evaluation components that can scale across use cases

🎯 Requirements

• 7+ years of experience in ML systems or AI engineering roles • At least 1–2 years working directly with LLMs, NLP pipelines, or semantic search • Deep understanding of foundation models (e.g. OpenAI, Claude, Mistral, Llama) and APIs • Hands-on experience with vector search (e.g. FAISS, Qdrant, Weaviate) and embeddings pipelines • Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules • Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow • Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production

🏖️ Benefits

• Generous benefits • Market cash compensation • Above-market equity • Well-designed benefits