Senior Databricks Data Engineer

Job not on LinkedIn

🕒 August 1

🇧🇷 Brazil – Remote

⏰ Full Time

🟠 Senior

🚰 Data Engineer

👻 Ghost score 28%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Compass

WebsiteLinkedInAll Job Openings

10,000+ employees

🏠 Real Estate

📱 Media

Real Estate • Media

Compass is a real-estate-focused content and services site that provides detailed market analysis, buying/selling/renting guides, mortgage and financing information, and home improvement and renovation advice. The site offers resources for homebuyers, sellers, renters, agents, and real estate investors — including articles on appraisals, affordable housing, investment strategies, staging and property maintenance. Compass aims to help users make informed decisions across the housing lifecycle through timely market updates and practical how-to content.

📋 Description

• Rebuild legacy data warehouse pipelines in Databricks (PySpark / Spark SQL) based on specifications produced during reverse engineering; • Implement bronze, silver and gold layers following the medallion architecture template, the project's ingestion patterns (CDC/batch) and the defined reconstruction standard; • Execute reconstruction waves by domain, coexisting with the legacy data warehouse until cutover; • Implement business rules and transformations with automated tests in the pipelines; • Perform reconciliation and parity validation of data between the legacy data warehouse and the Lakehouse; • Optimize pipeline performance and cost (partitioning, OPTIMIZE/Z-ORDER, job sizing); • Contribute to technical documentation of migrated rules and support prioritization of reconstruction waves.

🎯 Requirements

• Minimum 5 years of experience in Data Engineering; • Advanced PySpark and Python: development of large-scale batch pipelines with tests and engineering best practices; • Advanced SQL and data modeling: complex transformations, performance tuning and relational/dimensional modeling in a data warehouse context; • Databricks and Delta Lake: jobs, workflows and medallion (bronze, silver, gold) architecture in production; Delta Live Tables / Lakeflow and Asset Bundles; • Migration / reconstruction of ETL pipelines: translating business rules from legacy tools to Spark (not a lift-and-shift), with CDC/batch ingestion from relational databases; • AWS ecosystem: S3, Glue, EMR, Athena, Lambda, DMS and Step Functions; • Azure Synapse: experience with the legacy environment (SQL pools and pipelines) to support reverse engineering; • Data quality and reconciliation: automated tests, DQ gates and legacy vs. new comparisons for parity acceptance; • Git and CI/CD applied to data pipelines. • Knowledge of DataStage (reading jobs for reverse engineering) — important advantage; • Prior experience in data warehouse-to-lakehouse migration programs — important advantage; • Unity Catalog (permissions, lineage) and data contracts; • Experience in the financial or credit industry; • Databricks certifications (Data Engineer Associate / Professional).