Staff AI Observability, Telemetry Engineer

🔥 0 minutes ago

🏄 California, Texas – Remote

🏄 California – Remote 🤠 Texas – Remote

⏰ Full Time

🔴 Lead

🤖 Artificial Intelligence

👻 Ghost score 10%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Bitdeer Group

WebsiteLinkedInAll Job Openings

201 - 500 employees

💼 Consulting

📦 Logistics

🏗️ Construction

💰 Post-IPO Equity on 2023-05

Consulting • Logistics • Construction

Bitdeer Group is a leader in the blockchain and high-performance computing industry. It is one of the world’s largest holders and suppliers of hash rate, offering specialized mining infrastructure and high-quality hash rate sharing products. Founded by cryptocurrency pioneer Jihan Wu and led by CEO Matt Linghui Kong, the company is headquartered in Singapore with mining datacenters in the United States, Norway, and Bhutan. Bitdeer is committed to providing comprehensive computing solutions, including cloud services and AI capabilities, while emphasizing dedication, authenticity, and trustworthiness in its mission to be the most reliable provider in the industry.

📋 Description

• Architect and scale high-cardinality telemetry infrastructure using highly available time-series databases such as VictoriaMetrics, Thanos, or Mimir • Integrate hardware-level exporters, including NVIDIA DCGM, network switch telemetry, and IPMI/Redfish, into the Kubernetes observability stack • Build eBPF-based diagnostic tools to trace network congestion, kernel-level I/O latency, and distributed training bottlenecks • Develop automated dashboards and alerting pipelines that proactively cordon degraded hardware before it impacts customer training jobs • Design metric pipelines for accurate, multi-tenant consumption billing based on real-time GPU and network utilization metrics • Collaborate with GPU Systems and Scheduling teams to create observability standards for AI-native workloads • Lead technical design reviews for observability architecture • Mentor team members on high-performance telemetry collection and analysis best practices

🎯 Requirements

• Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related field • 6+ years of software or site reliability engineering experience • Deep, hands-on expertise in the Prometheus/OpenTelemetry ecosystem • Advanced proficiency in Go • Extensive experience writing custom Kubernetes metric exporters and operators • Hands-on experience with kernel-level tracing tools, including eBPF and BCC • Deep performance tuning experience with Linux systems • Strong familiarity with AI hardware metrics, including GPU power states, SM utilization, and memory bandwidth • Strong familiarity with high-performance network telemetry • Proven track record of operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments • Strong technical leadership skills and ability to influence architectural decisions and align cross-functional teams around observability standards • Excellent communication skills and ability to translate complex system requirements into manageable engineering milestones • Experience working in high-velocity, high-growth engineering environments is strongly preferred

🏖️ Benefits

• Equal employment opportunities in accordance with country, state, and local laws • Non-discrimination protections based on race, color, gender identity and/or expression, sexual orientation, marital and/or parental status, religion, political opinion, nationality, ethnic background or social origin, social status, disability, age, indigenous status, and union