Crawling Engineer

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Ant-Tech

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2016

💼 Consulting

📣 Marketing

🔌 API

Consulting • Marketing • API

Ant-Tech is a French technology company specializing in innovative digital solutions, offering services in design, web development, and mobile app creation. The company focuses on user experience and embraces the latest technology trends to deliver high-quality products customized to client needs. Ant-Tech takes pride in their expertise in project management and support, ensuring that all masterpieces run smoothly and efficiently.

📋 Description

• Build and maintain large-scale web crawlers across diverse domains • Design high-throughput, fault-tolerant systems for data collection at millions to billions of URLs per day • Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites • Develop pipelines for cleaning, deduplication, filtering, and normalisation • Construct and maintain datasets for research and model training • Monitor crawl performance, coverage, and data quality; iterate quickly • Collaborate with research teams to align data collection with modeling needs • Optimize infrastructure for cost, latency, and reliability • Operate at the boundary of scale and reliability • Adapt to constantly changing web environments • Balance throughput, coverage, and data quality • Own end-to-end data acquisition pipelines

🎯 Requirements

• Strong programming experience in one or more of Go, Rust, Python, Java, or C++ • Experience building web crawlers or large-scale data pipelines • Solid understanding of HTTP, networking, and browser behavior • Familiarity with distributed systems and parallel processing • Experience working with large datasets; TB–PB scale preferred • Ability to debug unstable or adversarial environments • Experience with NLP pipelines or dataset curation for ML preferred • Familiarity with LLM pretraining data or retrieval systems preferred • Experience with headless browsers such as Chrome DevTools Protocol, Playwright, or Puppeteer preferred • Knowledge of proxy systems, IP rotation, and large-scale request orchestration preferred • Background in data quality evaluation or benchmarking preferred • Experience running workloads on cloud or bare-metal infrastructure preferred

🏖️ Benefits

• Competitive salary • Benefits • Equity package • Fully remote work