Member of Technical Staff, Benchmarking

Job not on LinkedIn

🔥 0 minutes ago

🇺🇸 United States – Remote

💵 $150k - $210k / year

⏰ Full Time

🔴 Lead

🖥 Software Engineer

👻 Ghost score 0%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

GenMD

WebsiteLinkedInAll Job Openings

2 - 10 employees

🏥 Healthcare

🤖 Artificial Intelligence

☁️ SaaS

Healthcare • Artificial Intelligence • SaaS

GenMD is a healthcare data company that creates privacy-preserved digital twins of clinical records so researchers, health systems, and AI teams can collaboratively analyze and build models without exposing individual patient data. GenMD deploys inside a health system’s cloud account, generates synthetic replicas that preserve statistical structure while minimizing re-identification risk, validates privacy and fidelity, and provides vetted researchers and frontier AI teams fast access to usable clinical datasets. Health systems retain governance, can decline partners, audit use, and earn revenue when partners license access to data.

📋 Description

• Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use • Build end-to-end evaluation pipelines, including runs, scoring, dashboards and reports comparing models and training runs at scale • Create rubrics, automated graders and LLM-as-a-judge systems, and calibrate them against clinician review • Run systematic failure analysis, categorize failure modes, measure their frequency and feed results into dataset design and post-training • Work with clinical experts and partner AI teams to turn benchmark results into research papers and products • Build benchmarks and evaluation systems for real-world healthcare data and use findings to create data that closes model performance gaps

🎯 Requirements

• Hands-on experience evaluating or training LLMs (PyTorch, Hugging Face or similar) • Strong Python and data engineering skills • Comfortable with large tabular and text datasets • SQL/BigQuery is a plus • Ability to design evaluations that measure meaningful outcomes and resist shortcuts • B.Sc. or higher in CS, ML, NLP, biomedical informatics or a related field • About 4 hours of daily overlap with Pacific Time (roughly 9 AM to 1 PM PT) • LLM/NLP publications preferred • Experience with clinical data or medical benchmarks is nice to have • Experience with RL from verifiable rewards, reward modeling or post-training data is nice to have

🏖️ Benefits

• Equity • Well funded, revenue-generating company • Remote work arrangement

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume