Data Engineer, Databricks Lakehouse – Senior

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Compass

WebsiteLinkedInAll Job Openings

10,000+ employees

🏠 Real Estate

📱 Media

Real Estate • Media

Compass is a real-estate-focused content and services site that provides detailed market analysis, buying/selling/renting guides, mortgage and financing information, and home improvement and renovation advice. The site offers resources for homebuyers, sellers, renters, agents, and real estate investors — including articles on appraisals, affordable housing, investment strategies, staging and property maintenance. Compass aims to help users make informed decisions across the housing lifecycle through timely market updates and practical how-to content.

📋 Description

• Collaborate in building the Enterprise Lakehouse platform on Databricks, organized into data domains according to Data Mesh principles. • Implement and manage Unity Catalog as the central governance layer and enable data sharing between domains using Delta Sharing. • Develop data ingestion patterns (streaming and batch) and the medallion architecture template (bronze/silver/gold) for domain use. • Develop the platform's data contract framework and Data Quality framework. • Establish CI/CD pipelines for the Databricks platform and automate infrastructure provisioning. • Integrate the Databricks platform with the existing Data Mesh product on AWS, ensuring catalog interoperability and adherence to the data contract model. • Contribute to architecture decisions, technical documentation, and team mentoring.

🎯 Requirements

• Databricks: strong experience with the platform — workspace administration, clusters, jobs and workflows management; • Unity Catalog: data governance, permission models, lineage, external locations and storage integration; • Delta Lake and medallion architecture (bronze, silver, gold) in production environments; • Data ingestion patterns: streaming and batch at scale; • CI/CD for Databricks (Databricks Asset Bundles, Repos, integration with pipelines such as GitHub Actions, Azure DevOps or similar); • AWS ecosystem: S3, Glue, Lake Formation, EMR (EC2 and Serverless), Athena, Lambda, IAM, DMS, Kinesis, Step Functions, SNS, SQS and EventBridge; • IaC (Terraform, including the Databricks provider); • Advanced Python and PySpark for pipeline development and platform automations; • Advanced SQL; • Data Contracts and metadata tools like OpenMetadata; • Datadog: for observability controls and monitoring; • Desirable: • Experience with Data Mesh projects (domains, data products, federated governance) — significant plus; • Experience in the financial or credit industry — significant plus; • Observability and monitoring of data platforms (system tables, job metrics, data quality); • FinOps: monitoring, allocation and cost optimization in Databricks (DBUs, cluster sizing, compute policies); • Delta Sharing in cross-company, cross-cloud or cross-region scenarios; • AWS certifications (e.g., Data Engineer Associate, Solutions Architect); • Databricks certifications (Data Engineer Associate, Data Engineer Professional, Platform Administrator); • Open data contract specifications (Open Data Contract Standard, ODPS).

🏖️ Benefits

• Role also open to candidates with disabilities (PWD)