Research Crawling Engineer

πŸ•’ April 20

πŸ‡ΊπŸ‡Έ United States – Remote

⏰ Full Time

🟑 Mid-level

🟠 Senior

πŸ‘·πŸ»β€β™€οΈ Engineer

Where this job comes from

Posted on Wynd Labs's careers page β†—Wynd Labs's careers page, not through a job board

First seen by Remote Rocketship April 20

Still open on Wynd Labs's site. Checked 3 hours ago.

Apply Now
Logo of Wynd Labs

Wynd Labs

11 - 50 employees

Founded 2018

πŸ’Ό Consulting

πŸ“£ Marketing

πŸ€– Artificial Intelligence

Consulting β€’ Marketing β€’ Artificial Intelligence

Wynd Labs is focused on making public web data accessible for artificial intelligence applications. The company provides enterprise customers with robust web scraping networks for accessing diverse data, such as financial insights for small to medium cap equities. Additionally, Wynd Labs contributes to AI development by offering AI-oriented datasets and designing native AI models to assist in data scraping and structuring. Their platform supports a wide range of business solutions, including AI training data, price scraping, ad verification, and financial data services.

πŸ“‹ Description

β€’ Build and maintain large-scale web crawlers across diverse domains β€’ Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day) β€’ Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites β€’ Develop pipelines for cleaning, deduplication, filtering, and normalization β€’ Construct and maintain datasets for research and model training β€’ Monitor crawl performance, coverage, and data quality; iterate quickly β€’ Collaborate with research teams to align data collection with modeling needs β€’ Optimize infrastructure for cost, latency, and reliability

🎯 Requirements

β€’ Strong programming experience in one or more of: Go, Rust, Python, Java, or C++ β€’ Experience building web crawlers or large-scale data pipelines β€’ Solid understanding of HTTP, networking, and browser behavior β€’ Familiarity with distributed systems and parallel processing β€’ Experience working with large datasets (TB–PB scale preferred) β€’ Ability to debug unstable or adversarial environments

πŸ–οΈ Benefits

β€’ Competitive salary β€’ Benefits and equity package

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ April 20

Liatrio

51 - 200

🏒 Enterprise

☁️ SaaS

πŸ’Ό Consulting

Principal Engineer leading technical direction for modernization in AI-driven projects at Liatrio. Collaborating with clients to deliver innovative solutions and guide engineering practices.

πŸ•’ April 16

Accenture Federal Services

10,000+ employees

πŸ’Ό Consulting

πŸŽ–οΈ Defense

πŸ“¦ Logistics

Microsoft 365 Records & Automation Engineer supporting federal records management initiatives. Implementing Microsoft Purview capabilities and workflow automation in a collaborative environment.

πŸ•’ April 15

Slate Auto

201 - 500

🚘 Automotive

🏭 Manufacturing

πŸš— Transport

Salesforce Development Engineer optimizing Salesforce for electric vehicle startup Slate. Leading system architecture, implementation, and integration while mentoring team members.

πŸ•’ April 15

Salas O'Brien

1001 - 5000

πŸ’Ό Consulting

πŸ—οΈ Construction

πŸ₯ Healthcare

Structural Project Engineer for Axiom developing structural designs and calculations for various projects. Collaborating with teams to ensure project execution and maintaining quality standards in engineering solutions.

πŸ•’ April 15

Atlas Technica

201 - 500

πŸ”’ Cybersecurity

πŸ’Έ Finance

Project Engineer delivering IT management and security solutions for hedge funds and investment firms. Collaborating with teams to enhance client infrastructures and support technical projects.

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok