Data Processing Software Developer

Stelle nicht auf LinkedIn

🕒 vor 7 Monaten

🇺🇸 Vereinigte Staaten – Remote

⏰ Vollzeit

🟡 Mittelstufe

🟠 Senior

🧑‍💻 Full-Stack-Entwickler

🦅 H1B-Visum-Sponsor

Dieses Unternehmen hat in der Vergangenheit H1B-Visa gesponsert.

Jetzt Bewerben

Ähnliche Remote-Jobs finden

E-Mails für Remote-Jobs erhalten

📊 Überprüfen Sie Ihre Lebenslauf-Bewertung für diese Stelle

Verbessern Sie Ihre Chancen auf ein Vorstellungsgespräch, indem Sie Ihre Lebenslauf-Bewertung vor der Bewerbung überprüfen.

Lebenslauf hochladen

DataPelago

WebsiteLinkedInAlle Stellen

51 - 200 Mitarbeiter

Gegründet 2022

🤖 Künstliche Intelligenz

🏢 Unternehmen

🤝 B2B

Artificial Intelligence • Enterprise • B2B

DataPelago ist ein Unternehmen, das eine universelle Datenverarbeitungs-Engine (DataPelago Nucleus) und den DataPelago Accelerator für Spark bereitstellt, um GenAI- und Analyse-Workloads drastisch zu beschleunigen. Ihre Plattform nutzt heterogenes beschleunigtes Computing (GPUs, FPGAs und CPUs), integriert sich in Open-Source-Frameworks wie Spark, Trino und Gluten/Substrait und zielt darauf ab, Leistungssteigerungen um Größenordnungen und signifikante Kostensenkungen zu liefern, ohne Änderungen an Anwendungen oder Anbieterbindung. DataPelago richtet sich an Unternehmensdaten- und KI-Anwender und bietet Bereitstellungshandbücher, Benchmarks und Fallstudien, die große Einsparungen und schnellere Verarbeitung für großflächige strukturierte, semi-strukturierte und unstrukturierte Datensätze demonstrieren.

Beschreibung

• Implement software to perform data processing operations efficiently on large data sets as part of an overall data processing engine. • Work closely with technical leads and other individual developers to develop this software and ensure data is processed accurately, efficiently, and reliably as part of the overall analytics engine. • Investigate and address issues in the software and develop necessary enhancements in an ongoing manner. • Working with cross-functional teams to deliver new algorithms and techniques in the core Datapelago GPU-powered, parallel and distributed execution engine • Developing enterprise-grade, highly reliable code and owning the end-to-end delivery of solutions ensuring smooth production rollouts • Designing, implementing, testing, and maintaining enhancements in different components of our engine. • Developing highly modular components and ensuring clean integration with the entire stack • Define algorithms, mechanisms, procedures, and policies for the required functionality. Document them, and secure review and approval from the technical lead(s) • Continuously analyzing and identifying areas for differentiation and improvements in the execution engine • Identifying bottlenecks in scaling and performance and implementing solutions • Investigate and resolve all bugs before and after the release of the software • Support troubleshooting and resolution of issues, incidents, and problems, working in collaboration with others as needed. • Document findings, resolutions, and identify required enhancements to avoid recurrence • Collaborating with other teams and team members and assisting with code reviews, design reviews, and development process improvements

🎯 Anforderungen

• S. EE/CS or equivalent with 5+ years of experience or MS with 3+ years’ experience • 5+ years of experience developing database execution engine components including core database operations - joins, aggregations, sorts, analytic functions like window functions, etc., in an enterprise-class database serving large-scale data processing workloads • Experience developing modern analytic databases and familiarity with vectorized execution, parallel and distributed database algorithms, memory management, etc. • Good understanding of SQL and familiarity with Lakehouse architectures and technologies • Solid experience in developing performant code that minimizes overheads and latency and maximizes throughput • Experienced with the best practices in developing high quality, reliable software • Experience developing, evaluating, and troubleshooting high-performance software • Strong programming ability in C, C++. Rust experiences a strong plus • Strong development experience on Linux platforms