Staff AI Engineer

🕒 April 13

🇺🇸 United States – Remote

💵 $175k - $210k / year

⏰ Full Time

🔴 Lead

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

Apply Now

Find Similar Remote Jobs

Receive Emails For Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Grafana Labs

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2014

🏢 Enterprise

☁️ SaaS

🤖 Artificial Intelligence

Enterprise • SaaS • Artificial Intelligence

Grafana Labs is a company that specializes in open-source observability technologies and solutions. It offers a comprehensive suite of tools for logging, metrics, tracing, and profile management with products like Grafana, Loki, Tempo, and Mimir. Their offerings are designed to help businesses visualize, monitor, and alert on data from various sources, providing capabilities such as anomaly detection, root cause analysis, and service level objective management using AI/ML insights. Grafana Labs provides both cloud-based and self-managed solutions, ideal for infrastructure, application, and frontend observability. Additionally, their platform supports integration with various data sources like Prometheus and OpenTelemetry, making them a key player in the observability and infrastructure monitoring space.

📋 Description

• Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation • Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams • Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs) • Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management • Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths • Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools) • Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context • Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure • Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope high-impact automation problems, identify bottlenecks, and build solutions with measurable business outcomes • Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards • Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently

🎯 Requirements

• 8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering • 2+ years hands-on experience applying LLMs/AI to production workflows, not just prototypes • Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline • Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation • Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns (sequential chains, router/dispatcher, parallel fan-out), state management, and production monitoring • Diagnose business problems before writing code; think in workflows and outcomes, not just functions • Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run) • Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management • Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction • Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code); use AI to build AI systems • Clear technical communicator—able to explain complex systems in simple terms to both engineers and business stakeholders.

🏖️ Benefits

• equity • bonus (if applicable) • Restricted Stock Units (RSUs) • 30 days annual leave