Senior Databricks Data Engineer

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Compass

WebsiteLinkedInAll Job Openings

10,000+ employees

🏠 Real Estate

📱 Media

Real Estate • Media

Compass is a real-estate-focused content and services site that provides detailed market analysis, buying/selling/renting guides, mortgage and financing information, and home improvement and renovation advice. The site offers resources for homebuyers, sellers, renters, agents, and real estate investors — including articles on appraisals, affordable housing, investment strategies, staging and property maintenance. Compass aims to help users make informed decisions across the housing lifecycle through timely market updates and practical how-to content.

📋 Description

• Rebuild legacy data warehouse pipelines in Databricks (PySpark / Spark SQL) based on specifications produced during reverse engineering; • Implement bronze, silver and gold layers following the medallion architecture template, the project's ingestion patterns (CDC/batch) and the defined reconstruction standard; • Execute reconstruction waves by domain, coexisting with the legacy data warehouse until cutover; • Implement business rules and transformations with automated tests in the pipelines; • Perform reconciliation and parity validation of data between the legacy data warehouse and the Lakehouse; • Optimize pipeline performance and cost (partitioning, OPTIMIZE/Z-ORDER, job sizing); • Contribute to technical documentation of migrated rules and support prioritization of reconstruction waves.

🎯 Requirements

• Minimum 5 years of experience in Data Engineering; • Advanced PySpark and Python: development of large-scale batch pipelines with tests and engineering best practices; • Advanced SQL and data modeling: complex transformations, performance tuning and relational/dimensional modeling in a data warehouse context; • Databricks and Delta Lake: jobs, workflows and medallion (bronze, silver, gold) architecture in production; Delta Live Tables / Lakeflow and Asset Bundles; • Migration / reconstruction of ETL pipelines: translating business rules from legacy tools to Spark (not a lift-and-shift), with CDC/batch ingestion from relational databases; • AWS ecosystem: S3, Glue, EMR, Athena, Lambda, DMS and Step Functions; • Azure Synapse: experience with the legacy environment (SQL pools and pipelines) to support reverse engineering; • Data quality and reconciliation: automated tests, DQ gates and legacy vs. new comparisons for parity acceptance; • Git and CI/CD applied to data pipelines. • Knowledge of DataStage (reading jobs for reverse engineering) — important advantage; • Prior experience in data warehouse-to-lakehouse migration programs — important advantage; • Unity Catalog (permissions, lineage) and data contracts; • Experience in the financial or credit industry; • Databricks certifications (Data Engineer Associate / Professional).