Senior Databricks Engineer

Emploi pas sur LinkedIn

🕒 il y a 1 mois

Postuler Maintenant

Trouver des Emplois à Distance Similaires

Configurer une alerte email pour les nouveaux emplois

📊 Vérifiez votre score de CV pour ce poste

Améliorez vos chances d'obtenir un entretien en vérifiant votre score de CV avant de postuler.

Télécharger le CV

EXL

Site WebLinkedInTous les Emplois

10 000+ employés

🏥 Santé

🛡️ Assurance

📦 Logistique

💰 €2 000 000 Venture Round en 2015-01

Healthcare • Insurance • Logistics

Choisir un partenaire digital ne concerne pas seulement les capacités — c’est également une question de collaboration et de caractère.

Description

• Ingestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers. • Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling. • Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption. • Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA). • DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments. • Open Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF). • Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses. • Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.

🎯 Exigences

• 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform. • Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns. • Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments. • Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data). • Technical Toolset: • Cloud Environment: Azure (preferred), AWS. • Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow. • Core Languages: Expert-level SQL, Python, and PySpark. • Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools.

🏖️ Avantages

• Work From Home