Data Engineer

Job not on LinkedIn

🕒 May 19

Apply Now
Find Similar Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Logo of Viaduct

Viaduct

11 - 50 employees

🤖 Artificial Intelligence

☁️ SaaS

💰 Series B on 2024-06

Automotive • Artificial Intelligence • SaaS

Viaduct is an AI platform designed to assist automotive original equipment manufacturers (OEMs), fleets, and suppliers in reducing quality costs and implementing predictive maintenance strategies at scale. By processing over 100 billion miles tracked via telematics and more than 100 TB of data monthly, Viaduct enables quality and maintenance leaders to minimize warranty expenses, predict vehicle health, and enhance preventative maintenance routines. The platform provides insights to identify high-risk vehicles early, aiding in servicing and maintenance before roadside failures occur. Viaduct offers various pricing plans tailored to different data needs and team sizes, supporting scalable and customized analytics solutions.

📋 Description

• Creating and supporting batch, incremental, and real-time ETL pipelines • Standardizing ingestion, validation, and cleaning processes across clients • Automating data validation to increase data quality • Managing and evolving schemas in all parts of our pipeline • Monitoring, tuning, and optimizing Spark jobs • Becoming a domain-expert in connected vehicle data

🎯 Requirements

• 3+ years as a data engineer • Experience as a tech lead or mentor • Proficiency in Python/Scala/Java/C++ and SQL • 3+ years of experience with Spark or equivalent technologies • 2+ years of experience with a workflow scheduler (Airflow, Prefect, Argo, etc) • 2+ years of experience with distributed file-systems (HDFS, S3, etc) • Familiar with the tools in open-source data ecosystem (Apache, CNCF, etc) • Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming, etc)

Apply Now