Software Engineer L5 – AI Observability, Agent Evaluation

🔥 0 minutes ago

☕ Washington – Remote

☕ Washington – Remote

💵 $388k - $619k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

🧑‍💻 Full-stack Engineer

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Netflix

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1997

📱 Media

👥 B2C

Media • B2C

Netflix is a global streaming entertainment company and content producer whose stated mission is "to entertain the world. " It operates a consumer-facing subscription platform offering on-demand TV shows, films, and original programming, and also runs a public careers site emphasizing culture, inclusion, and hiring accommodations. The provided text highlights Netflix’s focus on recruiting talent worldwide, its work-life and culture pages, and its public-facing employer materials.

📋 Description

• Build observability framework and platform capabilities providing metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration • Build primitives for teams to monitor model performance, data quality, drift, and degradation on their own systems • Build and extend evaluation frameworks for LLM and agentic systems supporting response quality, grounding and hallucination, task success, tool-use and trajectory correctness, LLM-as-a-judge, and human-in-the-loop scoring • Lead build-vs-buy evaluations for observability and evaluation tooling • Own SDKs, connectors, and APIs integrating vendor platforms into a consistent interface • Build reusable libraries, SDKs, and templates that make observability and evaluation the default for new systems • Provide dashboarding, alerting, and SLO/SLI building blocks and out-of-the-box templates • Partner with engineering, product, machine learning, and data teams to turn needs into reusable platform capabilities • Enable ML/AI practitioners to monitor model quality in production, evaluate LLM and agentic systems, and adopt vendor tooling through a self-serve platform

🎯 Requirements

• Experience in software, AI/ML, or platform engineering, with hands-on time in production observability, monitoring, or ML/LLM evaluation • Proven track record of designing standards, libraries, SDKs, or frameworks that other teams build on and adopt • Strong coding skills in Python and at least one of Java, Go, or Scala, with experience building production services • Practical experience operating ML models in production (online serving and/or batch), including drift, model performance, and data quality • Hands-on experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, Jaeger/Tempo, or similar • Solid understanding of distributed systems, microservices, and at least one major cloud platform (AWS, GCP, or Azure) • Ability to work cross-functionally with ML, data, infra, and product teams, and to communicate clearly about system behavior, quality, and risk • Experience with vendor integration and VPC deployment • AI-Native Engineering Mindset who uses AI tools as a core part of their own workflow to accelerate design, development, testing, and code review • Nice to have: Hands-on experience with ML/LLM observability and evaluation tools such as Arize, Braintrust, LangFuse, Weights & Biases, Galileo, Vertex AI Model Monitoring, or SageMaker Model Monitor • Nice to have: Experience building or shipping LLM/GenAI applications and evaluating them, including prompt/result logging, evaluation metrics, LLM-as-a-judge, and human-in-the-loop review • Nice to have: Experience evaluating agentic systems, including tool use, multi-step reasoning, and trajectory/task-success measurement

🏖️ Benefits

• Health Plans • Mental Health support • 401(k) Retirement Plan with employer match • Stock Option Program • Disability Programs • Health Savings and Flexible Spending Accounts • Family-forming benefits • Life and Serious Injury Benefits • Paid leave of absence programs • Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off • Full-time salaried employees are immediately entitled to flexible time off