Senior Databricks Engineer

🕒 June 19

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

EXL

WebsiteLinkedInAll Job Openings

10,000+ employees

🏥 Healthcare

🛡️ Insurance

📦 Logistics

💰 $2M Venture Round on 2015-01

Healthcare • Insurance • Logistics

EXL is a business consulting and services firm that focuses on leveraging data to enhance business operations and decision-making. With a strong emphasis on collaboration and adaptability, EXL partners with organizations to address their unique needs and culture while integrating data science and technology solutions. The company's areas of expertise include operations management, decision analytics, digital transformation, and various industries such as healthcare, finance, and insurance. EXL's mission is to help clients drive business evolution and maintain competitive advantage through tailored solutions and effective use of data.

📋 Description

• Ingestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers. • Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling. • Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption. • Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA). • DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments. • Open Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF). • Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses. • Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.

🎯 Requirements

• 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform. • Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns. • Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments. • Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data). • Technical Toolset: • Cloud Environment: Azure (preferred), AWS. • Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow. • Core Languages: Expert-level SQL, Python, and PySpark. • Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools.

🏖️ Benefits

• Work From Home