Data Architect, AWS, Databricks

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Receive Emails For Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Compass

WebsiteLinkedInAll Job Openings

10,000+ employees

🏠 Real Estate

📱 Media

Real Estate • Media

Compass is a real-estate-focused content and services site that provides detailed market analysis, buying/selling/renting guides, mortgage and financing information, and home improvement and renovation advice. The site offers resources for homebuyers, sellers, renters, agents, and real estate investors — including articles on appraisals, affordable housing, investment strategies, staging and property maintenance. Compass aims to help users make informed decisions across the housing lifecycle through timely market updates and practical how-to content.

📋 Description

• Perform the complete migration of the current environment, today hosted on Databricks on Azure, to AWS, including creating a new data model and restructuring legacy pipelines and routines; • Define and evolve the Corporate Data Platform architecture (Lakehouse); • Ensure adherence to the target model based on AWS + Databricks; • Define architecture standards, frameworks and best practices; • Drive the definition of the migration strategy (waves, prioritization, dependencies); • Migration and Modernization: Lead the modernization of the legacy Data Warehouse (Azure/DataStage → AWS/Databricks); • Define migration approaches: Incremental vs Big Bang; • Ensure operational continuity during the transition; • Governance & Security: Define and implement standards for: Data governance/Access control/Data quality and lineage; • Ensure compliance with corporate policies and LGPD (Brazilian data protection law); • DataOps & Standardization: Structure standardized and reusable pipelines; • Implement best practices for CI/CD for data; • Reduce dependence on manual processes and low standardization; • Integration and Ecosystem: Design integrations with multiple sources and on-premises systems;

🎯 Requirements

• Experience with Cloud & AWS Platform, S3, Glue, IAM, Lake Formation, CloudWatch, CloudTrail; • Experience with Databricks: Unity Catalog, Delta Lake, notebooks, clusters and policies; • Knowledge of modern Lakehouse-based architecture; • Experience with data modeling (DW, Lakehouse – Bronze/Silver/Gold); • Experience with data pipelines (ETL/ELT); • Experience with: advanced SQL, Python, tools such as: Airflow / Control-M / distributed orchestration; • Experience with ADF / DataStage (legacy); • Experience with CI/CD for data (Azure DevOps, Git, pipelines); • Experience with Data Quality, Data Contracts, Data Lineage; • Experience with data catalog and corporate governance; • Experience with security and compliance (LGPD, access control, sensitive data); • Knowledge of integration with multiple sources: APIs, relational databases, NoSQL, mainframe; • Experience in distributed and domain-driven architecture; • Migration strategies: Replatform, Refactor, Rewrite; • Knowledge of monitoring (Datadog, CloudWatch); • Definition of SLAs/SLOs; • Experience troubleshooting critical pipelines;