Data Processing Principal Engineer

Job not on LinkedIn

October 22

Apply Now

Receive Emails with Similar Jobs

DataPelago

WebsiteLinkedInAll Job Openings

Artificial Intelligence • Enterprise • B2B

DataPelago is a company that provides a universal data processing engine (DataPelago Nucleus) and the DataPelago Accelerator for Spark to dramatically accelerate GenAI and analytics workloads. Their platform leverages heterogeneous accelerated computing (GPUs, FPGAs, and CPUs), integrates with open-source frameworks like Spark, Trino, and Gluten/Substrait, and aims to deliver orders-of-magnitude performance and significant cost reductions with zero changes to applications or vendor lock-in. DataPelago targets enterprise data and AI practitioners, offering deployment guides, benchmarks, and case studies demonstrating large savings and faster processing for large-scale structured, semi-structured, and unstructured datasets.

51 - 200 employees

Founded 2022

🤖 Artificial Intelligence

🏢 Enterprise

🤝 B2B

📋 Description

• Lead architecture, design, and implementation of next-generation data processing engine built to exploit accelerated computing. • Work with cross-functional teams to advance the architecture at the core of Datapelago’s parallel and distributed execution engine based on accelerated computing • Lead the execution engine team in the design, implementation, and rollout of enterprise grade, highly reliable acceleration engine for data processing • Design, implement, test, and maintain major components of the execution engine • Analyze and identify areas for differentiation and improvements in the execution engine • Collaborate with other teams and team members to drive code reviews, design reviews, performance and reliability reviews, and development process and to drive continuous improvement in all of these

🎯 Requirements

• B.S. EE/CS or equivalent with 15+ years experience or MS with 10+ years experience • 10+ years of experience developing core components of an enterprise-grade database or analytics execution engine serving large-scale data processing workloads. Experience developing for platforms such as Apache Spark, Gluten, Velox, DataFusion preferred. • Experienced developing high-performance parallel implementations of data processing operators and functions, such as joins, aggregations, sorts • Experience leading 10+ teams in designing, developing, and releasing high-performance data processing engines for large production deployments • Strong programming ability in C, C++, and Rust • Strong development experience on Linux platforms