Research Crawling Engineer

Job not on LinkedIn

🕒 April 20

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

👷🏻‍♀️ Engineer

👻 Ghost score 60%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Wynd Labs

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2018

💼 Consulting

📣 Marketing

🤖 Artificial Intelligence

Consulting • Marketing • Artificial Intelligence

Wynd Labs is focused on making public web data accessible for artificial intelligence applications. The company provides enterprise customers with robust web scraping networks for accessing diverse data, such as financial insights for small to medium cap equities. Additionally, Wynd Labs contributes to AI development by offering AI-oriented datasets and designing native AI models to assist in data scraping and structuring. Their platform supports a wide range of business solutions, including AI training data, price scraping, ad verification, and financial data services.

📋 Description

• Build and maintain large-scale web crawlers across diverse domains • Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day) • Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites • Develop pipelines for cleaning, deduplication, filtering, and normalization • Construct and maintain datasets for research and model training • Monitor crawl performance, coverage, and data quality; iterate quickly • Collaborate with research teams to align data collection with modeling needs • Optimize infrastructure for cost, latency, and reliability

🎯 Requirements

• Strong programming experience in one or more of: Go, Rust, Python, Java, or C++ • Experience building web crawlers or large-scale data pipelines • Solid understanding of HTTP, networking, and browser behavior • Familiarity with distributed systems and parallel processing • Experience working with large datasets (TB–PB scale preferred) • Ability to debug unstable or adversarial environments

🏖️ Benefits

• Competitive salary • Benefits and equity package