Principal Engineer – Data Retrieval

Job not on LinkedIn

🔥 5 minutes ago

⛰️ Colorado, New York, +2 more states – Remote

⛰️ Colorado – Remote 🗽 New York – Remote 🍂 Massachusetts – Remote ☕ Washington – Remote

💵 $272k - $431.3k / year

⏰ Full Time

🔴 Lead

🚰 Data Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 1%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

NVIDIA

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1993

🏥 Healthcare

🏭 Manufacturing

🤖 Artificial Intelligence

Healthcare • Manufacturing • Artificial Intelligence

NVIDIA is a leading technology company specializing in accelerated computing and artificial intelligence. NVIDIA pioneers advancements in graphical processing units (GPUs), cloud computing, data centers, and virtual reality, with a focus on gaming, automotive, healthcare, and robotics industries. The company's innovations, such as NVIDIA Omniverse, transform traditional digital processes by enabling high-fidelity simulations and rendering tasks. Their applications span various industries, from autonomous vehicles using NVIDIA DRIVE to healthcare solutions with NVIDIA Clara, and AI-driven analytics and workflows.

📋 Description

• Architect and scale data acquisition infrastructure powering NVIDIA’s foundation models • Build systems that discover valuable sources, crawl the open web at scale, and deliver reliable, high-quality data to training pipelines • Own architecture of a unified platform covering source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion • Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and quality-, diversity-, and freshness-based scheduling • Build a continuous discovery loop that extracts links and feeds new candidates into the crawl frontier • Develop a shared crawl execution layer that fetches reliably and politely at scale, including dynamic websites • Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets • Define stable interfaces for source connectors • Partner with parsing and curation teams to preserve valuable code, math, and structured content and improve accepted data yield • Implement provenance and policy enforcement with legal, privacy, security, and data governance partners • Translate publisher preferences, opt-outs, licensing terms, and use restrictions into auditable downstream controls • Own production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency • Instrument the pipeline end to end and set service objectives for throughput, freshness, reliability, and training cutoffs • Evaluate internal crawls and external datasets through comparable pilots with research and product teams • Deliver reproducible data releases tied to source coverage and processing quality in controlled training experiments • Coordinate cross-team dependencies and balance immediate data delivery with long-term platform development • Lead architecture reviews and critical implementation work • Mentor engineers and help build the team

🎯 Requirements

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience • 15+ years of relevant experience in software engineering • Experience designing and operating large-scale distributed systems • Experience setting technical direction and delivering complex infrastructure across multiple teams • Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure • Experience with distributed scheduling, stateful services, storage, and high-throughput data processing • Strong hands-on engineering and system design skills • Ability to move between architecture, implementation, performance analysis, and production debugging • Experience building batch or streaming pipelines • Knowledge of consistency, retries, duplicate processing, backpressure, and recovery • Track record of improving throughput, latency, resource efficiency, or operational reliability • Experience delivering data platforms for research, ML, search, or other demanding technical users • Ability to translate evolving requirements into clear interfaces and measurable engineering outcomes • Strong technical communication and leadership through influence across engineering, product, research, and governance teams • Experience architecting a web-wide crawl platform, scaling crawl frontier storage, or consolidating multiple acquisition systems into shared infrastructure • Direct experience supplying foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content • Experience implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale • Background with browser-based or agentic data collection, programmable extraction, and sandboxed environments for tool-use data or post-training evaluation • Experience onboarding licensed or multimodal datasets, building reusable partner integrations, or delivering streaming feeds with low latency for retrieval and grounding

🏖️ Benefits

• Equity • Benefits