LLM Red Team & Benchmark Evaluation Specialist

🕒 Julho 27

🗽 New York – Remoto

🗽 New York – Remote

💵 $55 - $85 / hora

⏰ Tempo Integral

🟢 Júnior

Candidatar-se

Encontrar Vagas Remotas Similares

Configurar Alerta de Email para Novas Vagas

📊 Verifique sua pontuação de currículo para esta vaga

Melhore suas chances de conseguir uma entrevista verificando sua pontuação de currículo antes de se candidatar.

Enviar Currículo

24-MAG

SiteLinkedInTodas as Vagas

2 - 10 funcionários

🤝 B2B

💼 Consultoria

B2B • Consulting

A 24-MAG é uma empresa de estratégia e execução comercial que auxilia organizações B2B a desenhar e implementar sistemas, fluxos de trabalho e ritmos operacionais para vendas, gerenciamento de clientes e projetos multifuncionais. Eles se concentram em transformar processos dispersos em funções comerciais alinhadas, mensuráveis e escaláveis—abrangendo a estrutura de pipeline, frameworks de gerenciamento de contas e disciplina operacional para equipes que buscam um crescimento eficiente e intencional.

Descrição

• Adversarial Model Evaluation • Probe frontier AI models across coding, machine learning, analysis, and multi-step agentic tasks • Identify subtle errors, vulnerabilities, edge cases, and misleadingly plausible outputs • Investigate situations where models appear capable while reaching incorrect or unsupported conclusions • Design reproducible experiments to isolate and validate model failure modes • Benchmark & Challenge Design • Convert observed model weaknesses into rigorous benchmark tasks • Develop challenges that are technically demanding while remaining fair and objectively assessable • Define clear task requirements, expected outcomes, and evaluation criteria • Ensure tasks require genuine reasoning rather than allowing shortcuts or superficial pattern matching • Failure Analysis & Documentation • Document findings with clear evidence, methodology, and reproducible steps • Explain why a model failed and which capabilities or assumptions contributed to the error • Produce detailed technical write-ups for researchers and task authors • Track recurring failure patterns across models, prompts, and evaluation environments • Task Strengthening & Research Collaboration • Work with task authors to close loopholes, grading gaps, and unintended shortcuts • Review benchmark tasks for ambiguity, exploitability, and evaluation reliability • Share insights with researchers and other specialists to improve benchmark coverage • Participate in iterative calibration, peer review, and task-refinement workflows

🎯 Requisitos

• At least 1 year of experience in research, research engineering, security, AI evaluation, or a related technical role • Demonstrated experience identifying vulnerabilities, edge cases, or failure modes in LLMs or ML systems • Background in red teaming, adversarial testing, security research, benchmark development, or rigorous model evaluation • Working proficiency in Python and Git • Ability to develop scripts, probes, and analyses independently • Strong familiarity with LLM capabilities, limitations, and evaluation techniques • Excellent written communication and technical documentation skills • Creativity, precision, and persistence when working through ambiguous research problems • Reliable availability for approximately 35 hours per week • A master's degree or PhD in a STEM field is highly relevant