Data Processing Software Developer

Emploi pas sur LinkedIn

🕒 il y a 7 mois

🇺🇸 États-Unis – Télétravail

⏰ Temps Plein

🟡 Intermédiaire

🟠 Senior

🧑‍💻 Développeur Full-Stack

🦅 Parrain de Visa H1B

Cette entreprise a parrainé des visas H1B par le passé.

Postuler Maintenant

Trouver des Emplois à Distance Similaires

Recevoir des emails pour des emplois à distance

📊 Vérifiez votre score de CV pour ce poste

Améliorez vos chances d'obtenir un entretien en vérifiant votre score de CV avant de postuler.

Télécharger le CV

DataPelago

Site WebLinkedInTous les Emplois

51 - 200 employés

Fondée en 2022

🤖 Intelligence artificielle

🏢 Entreprise

🤝 B2B

Artificial Intelligence • Enterprise • B2B

DataPelago est une entreprise qui propose un moteur universel de traitement de données (DataPelago Nucleus) et le DataPelago Accelerator pour Spark afin d'accélérer drastiquement les charges de travail en GenAI et analytique. Leur plateforme exploite l'informatique accélérée hétérogène (GPU, FPGA et CPU), s'intègre aux frameworks open-source tels que Spark, Trino, et Gluten/Substrait, et vise à offrir des performances inédites et des réductions de coûts significatives sans apporter de modifications aux applications ni d'enfermement propriétaire. DataPelago cible les praticiens de la donnée et de l'IA en entreprise, offrant des guides de déploiement, des benchmarks et des études de cas démontrant de grandes économies et un traitement plus rapide pour les ensembles de données à grande échelle, structurées, semi-structurées et non structurées.

Description

• Implement software to perform data processing operations efficiently on large data sets as part of an overall data processing engine. • Work closely with technical leads and other individual developers to develop this software and ensure data is processed accurately, efficiently, and reliably as part of the overall analytics engine. • Investigate and address issues in the software and develop necessary enhancements in an ongoing manner. • Working with cross-functional teams to deliver new algorithms and techniques in the core Datapelago GPU-powered, parallel and distributed execution engine • Developing enterprise-grade, highly reliable code and owning the end-to-end delivery of solutions ensuring smooth production rollouts • Designing, implementing, testing, and maintaining enhancements in different components of our engine. • Developing highly modular components and ensuring clean integration with the entire stack • Define algorithms, mechanisms, procedures, and policies for the required functionality. Document them, and secure review and approval from the technical lead(s) • Continuously analyzing and identifying areas for differentiation and improvements in the execution engine • Identifying bottlenecks in scaling and performance and implementing solutions • Investigate and resolve all bugs before and after the release of the software • Support troubleshooting and resolution of issues, incidents, and problems, working in collaboration with others as needed. • Document findings, resolutions, and identify required enhancements to avoid recurrence • Collaborating with other teams and team members and assisting with code reviews, design reviews, and development process improvements

🎯 Exigences

• S. EE/CS or equivalent with 5+ years of experience or MS with 3+ years’ experience • 5+ years of experience developing database execution engine components including core database operations - joins, aggregations, sorts, analytic functions like window functions, etc., in an enterprise-class database serving large-scale data processing workloads • Experience developing modern analytic databases and familiarity with vectorized execution, parallel and distributed database algorithms, memory management, etc. • Good understanding of SQL and familiarity with Lakehouse architectures and technologies • Solid experience in developing performant code that minimizes overheads and latency and maximizes throughput • Experienced with the best practices in developing high quality, reliable software • Experience developing, evaluating, and troubleshooting high-performance software • Strong programming ability in C, C++. Rust experiences a strong plus • Strong development experience on Linux platforms