Software Engineer – Benchmarking

🔥 11 horas atrás

🏄 California, New York – Remoto

🏄 California – Remote 🗽 New York – Remote

💵 $160.000 - $210.000 / ano

⏰ Tempo Integral

🟡 Pleno

🟠 Sênior

🧑‍💻 Engenheiro Full-stack

Candidatar-se

Encontrar Vagas Remotas Similares

Configurar Alerta de Email para Novas Vagas

📊 Verifique sua pontuação de currículo para esta vaga

Melhore suas chances de conseguir uma entrevista verificando sua pontuação de currículo antes de se candidatar.

Enviar Currículo

Office Hours

SiteLinkedInTodas as Vagas

11 - 50 funcionários

Fundada em 2020

💼 Consultoria

🏥 Saúde

📣 Marketing

💰 $5.000.000 Seed Round - Office Hours em 2021-02

Consulting • Healthcare • Marketing

Office Hours é uma plataforma que conecta profissionais das áreas de tecnologia e ciências da vida com empresas, pesquisadores e equipes de produto para ganhar dinheiro compartilhando sua expertise. Os especialistas participam de consultas pagas de 30 a 60 minutos, pequenos questionários e tarefas de treinamento de modelos de IA (rotulagem de dados, RLHF, avaliação especializada), enquanto a plataforma cuida do agendamento, pagamentos e correspondência de oportunidades. Funciona como um mercado global e serviço semelhante a SaaS, permitindo que especialistas em domínio monetizem seu conhecimento e ajudem a moldar decisões de IA e produtos.

Descrição

• Prepare and maintain benchmark datasets, including cleaning, preparation, conversion into runnable formats, validation, and ongoing maintenance • Build and maintain infrastructure and evaluation pipelines across model APIs and terminal agents • Create lightweight, containerized environments and viewers for tasking and tool-use evaluation • Support fine-tuning of small open-source LLMs and compare baseline with post-training performance • Develop scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level performance • Build analysis tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time • Collaborate with researchers and engineers to ensure evaluation data and outputs are accurate, consistent, and integrated into published work • Build HTML viewers and internal tools for task authoring, review, quality control, and structured data collection

🎯 Requisitos

• 4+ years of professional experience building and maintaining complex systems • Strong Python skills • Ability to write robust, maintainable code and work deeply in existing codebases and infrastructure • Experience preparing, cleaning, and maintaining datasets • Experience with Docker and reproducible execution environments • Ability to collaborate with researchers and scientists and translate methodology into working systems • Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus • Experience with Python, model APIs, agent/evaluation frameworks, and custom evaluation tooling • Experience with terminal agents and open-source models via the Hugging Face ecosystem and PyTorch • Experience with Docker • Experience with React, Next.js, and Tailwind • Experience with GitHub, Slack, Notion, and Linear • Fine-tuning or post-training open-source LLM experience, or other hands-on machine learning work, is bonus experience • Experience with agentic, multi-turn, long-context, or tool-use evaluation is bonus experience • Experience validating LLM-as-judge or rubric-based grading setups is bonus experience • Background or strong interest in a scientific or technical domain is bonus experience • Experience building data-heavy dashboards, leaderboards, or visualizations is bonus experience • Open-source contributions or published work related to benchmarks and measurement is bonus experience

🏖️ Benefícios

• Competitive salary and equity • Medical, dental, and vision coverage • 401(k) • Monthly wellness and fitness stipend • Paid time off policy, along with company holidays • Annual company off-sites (Tahoe, Mendocino, Mexico City, San Diego, Park City) • Parent-friendly policies • Remote flexibility • Paid family leave