LLMOps Engineer

Job not on LinkedIn

🔥 2 hours ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

The Disruptors Den

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2023

🤝 B2B

🌍 Social Impact

B2B • Startup • Social Impact

The Disruptors Den is an organization that focuses on innovation and entrepreneurship, providing a platform for startups and new ideas to thrive. It serves as a community for disruptors and visionaries seeking to challenge the status quo and drive change in their respective industries. The goal of The Disruptors Den is to connect like-minded individuals and foster an environment conducive to growth and collaboration.

📋 Description

• Build and run product-specific evaluation suites by failure mode, on schedule and with every release • Maintain cost-effective evaluations as the product line grows • Own the reporting loop by presenting evaluation findings and platform data to the team and stakeholders • Surface unintended or problematic model behaviour before learners encounter it • Steward core datasets, including classified customer-support data • Partner with the AI Product Manager on product instrumentation and build evaluations over captured data • Collaborate with Anthropic Engineers, AI engineers, product managers, and data scientists to design learning experiences • Grow from supporting the technical lead’s designs toward full ownership of the analyzer function

🎯 Requirements

• Solid Python skills and comfort shaping and analysing messy LLM-generated data • Data inclination • Ability to decompose AI products into success and failure metrics • Familiarity with Langfuse, RAGAS, DSPy, or similar evaluation tools; depth in one and awareness of the others • Evidence of measuring an AI system honestly in a project • Willingness to own a function rather than only execute tickets • Strong communication and collaboration ability • Self-drive to communicate constantly using web collaboration and video conferencing • Preferred time zone: GMT+2 • Desirable but not required: experience keeping evaluations cheap at scale, dashboarding and reporting, and classical statistics

🏖️ Benefits

• Encouraged to continually develop capability and attain certifications to reflect individual growth • Virtual working and collaboration across international locations