Member of Technical Staff, Benchmarking

Job not on LinkedIn

πŸ•’ November 3

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $150k - $210k / year

⏰ Full Time

🟑 Mid-level

🟠 Senior

πŸ–₯ Software Engineer

Where this job comes from

Posted on GenMD's careers page β†—GenMD's careers page, not through a job board

First seen by Remote Rocketship November 3

Still open on GenMD's site. Checked 3 hours ago.

Apply Now
Logo of GenMD

GenMD

2 - 10 employees

πŸ₯ Healthcare

πŸ€– Artificial Intelligence

☁️ SaaS

Healthcare β€’ Artificial Intelligence β€’ SaaS

GenMD is a healthcare data company that creates privacy-preserved digital twins of clinical records so researchers, health systems, and AI teams can collaboratively analyze and build models without exposing individual patient data. GenMD deploys inside a health system’s cloud account, generates synthetic replicas that preserve statistical structure while minimizing re-identification risk, validates privacy and fidelity, and provides vetted researchers and frontier AI teams fast access to usable clinical datasets. Health systems retain governance, can decline partners, audit use, and earn revenue when partners license access to data.

πŸ“‹ Description

β€’ Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use β€’ Build end-to-end evaluation pipelines, including runs, scoring, dashboards and reports comparing models and training runs at scale β€’ Create rubrics, automated graders and LLM-as-a-judge systems, and calibrate them against clinician review β€’ Run systematic failure analysis, categorize failure modes, measure their frequency and feed results into dataset design and post-training β€’ Work with clinical experts and partner AI teams to turn benchmark results into research papers and products β€’ Build benchmarks and evaluation systems for real-world healthcare data and use findings to create data that closes model performance gaps

🎯 Requirements

β€’ Hands-on experience evaluating or training LLMs (PyTorch, Hugging Face or similar) β€’ Strong Python and data engineering skills β€’ Comfortable with large tabular and text datasets β€’ SQL/BigQuery is a plus β€’ Ability to design evaluations that measure meaningful outcomes and resist shortcuts β€’ B.Sc. or higher in CS, ML, NLP, biomedical informatics or a related field β€’ About 4 hours of daily overlap with Pacific Time (roughly 9 AM to 1 PM PT) β€’ LLM/NLP publications preferred β€’ Experience with clinical data or medical benchmarks is nice to have β€’ Experience with RL from verifiable rewards, reward modeling or post-training data is nice to have

πŸ–οΈ Benefits

β€’ Equity β€’ Well funded, revenue-generating company β€’ Remote work arrangement

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ November 3

Davis Wright Tremaine LLP

1001 - 5000

βš–οΈ Legal

πŸ’Ό Consulting

🀝 B2B

Modern workplace engineering manager leading endpoint operations and cloud modernization for AmLaw 100 law firm. Managing Intune, Autopilot, security, virtual desktops, and five engineers.

πŸ•’ November 2

Pennant

1001 - 5000

πŸ₯ Healthcare

βš•οΈ Healthcare Insurance

🧘 Wellness

Director of Engineering leading AI-native software for Pennant’s post-acute healthcare businesses. Owning architecture, delivery, engineering teams, and HIPAA-grade platform operations.

πŸ•’ November 2

Brown and Caldwell

1001 - 5000

πŸ’Ό Consulting

πŸ—οΈ Construction

πŸ₯ Healthcare

Digital Technology Consultant leading data governance, BI, AI, and digital transformation for Brown and Caldwell’s water and wastewater clients. Developing technology roadmaps and implementing integrated analytics solutions.

πŸ•’ November 2

Cordis

1001 - 5000

🏭 Manufacturing

πŸ₯ Healthcare

πŸ’Š Pharmaceuticals

AI applications developer building governed automations and intelligent workflows for Cordis, a cardiovascular technology company. Improving business processes through low-code platforms, integrations, and emerging AI.

πŸ•’ November 2

Koniag Government Services

1001 - 5000

πŸ›οΈ Government

πŸŽ–οΈ Defense

πŸ’Ό Consulting

Senior web, mobile, and DevOps developer modernizing federal government systems on Microsoft Azure. Building responsive applications, CI/CD pipelines, and compliant cloud infrastructure for federal clients.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok