Senior Data Engineer, Databricks

🔥 12 hours ago

🇧🇷 Brazil – Remote

⏰ Full Time

🟠 Senior

🚰 Data Engineer

👻 Ghost score 13%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Compass

WebsiteLinkedInAll Job Openings

10,000+ employees

🏠 Real Estate

📱 Media

Real Estate • Media

Compass is a real-estate-focused content and services site that provides detailed market analysis, buying/selling/renting guides, mortgage and financing information, and home improvement and renovation advice. The site offers resources for homebuyers, sellers, renters, agents, and real estate investors — including articles on appraisals, affordable housing, investment strategies, staging and property maintenance. Compass aims to help users make informed decisions across the housing lifecycle through timely market updates and practical how-to content.

📋 Description

• Define and evolve data architectures using Databricks, ensuring pipeline optimization, processing efficiency, and resolution of highly complex technical issues. • Build and maintain scalable, reliable data pipelines using modern distributed processing technologies to ensure high-quality, high-performance data ingestion, transformation, and delivery. • Work with Apache Spark, Airflow, Kafka, dbt, SQL/NoSQL databases (PostgreSQL, MongoDB, Cassandra), and cloud platforms (Databricks, AWS Glue, BigQuery). • Design, define, and implement scalable data architectures using the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold). • Implement batch and streaming data processing solutions, including CDC and Structured Streaming. • Develop and orchestrate data ingestion and integration processes using Azure Data Factory (ADF), Auto Loader, and Lakeflow. • Implement data governance, security, and access control practices through Unity Catalog. • Develop REST APIs, integrations, and automations using Python. • Implement CI/CD, version control, and deployment automation processes using Git and Databricks Workflows.

🎯 Requirements

• Advanced proficiency in Databricks, PySpark, Spark SQL, and Delta Lake. • Strong experience with DLT / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming. • Hands-on experience with Liquid Clustering, Auto Loader, and Delta table optimization. • Expertise in Unity Catalog, governance, security, lineage, and access control. • Experience defining and implementing data architecture, including the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold). • Hands-on experience with Azure Data Factory (ADF) for orchestration, data ingestion, and integration. • Experience developing REST APIs and advanced Python programming for building integrations, microservices, and automations. • Advanced knowledge of Spark performance tuning, Photon, partitioning, shuffle, and execution plan analysis. • Experience with Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles. • Ability to optimize workloads, scale compute resources, and reduce processing costs. • Experience with observability, monitoring, and supporting production pipelines. • Ability to serve as a technical reference, mentor professionals, and establish data engineering best practices. • Certifications: Databricks Certified Data Engineer Associate; Databricks Certified Data Engineer Professional.

🏖️ Benefits

• Meal and/or food allowance • Health and dental insurance • Health insurance for your pet • Home office allowance (for remote positions) • Welcome kit as part of onboarding • Life insurance • Zenklub, offering psychological and nutritional support • TotalPass • Certification and Continuous Learning Program with partners (Adobe, Anthropic, AWS, Google, Microsoft, Salesforce) • Partnerships and discounts to support your professional development • Access to UOL content • Maternity kit • Extended maternity and paternity leave • Employee referral program

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume