Data Manager, Databricks

Job not on LinkedIn

🔥 1 minute ago

🤠 Texas – Remote

🤠 Texas – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

📊 Data Scientist

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 10%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Allata

WebsiteLinkedInAll Job Openings

201 - 500 employees

🚘 Automotive

🏥 Healthcare

📦 Logistics

Automotive • Healthcare • Logistics

Allata is a global consulting firm that helps businesses achieve digital excellence by crafting unique customer experiences, identifying revenue-generating opportunities, and improving operational efficiencies. They offer a wide range of services including strategic services, technology and cloud solutions, data and insights, and artificial intelligence development. Allata's approach is experience-led, strategy-aligned, tech-enabled, and data-driven. They work across multiple industries such as automotive, financial services, health and life sciences, high tech, logistics and transportation, among others, to drive modernization, personalization, innovation, and efficiency. With a team of former IT leaders and consulting professionals, Allata guides clients through complex digital landscapes to deliver value-based outcomes and help organizations integrate digital strategies seamlessly into their overall business strategies.

📋 Description

• Define overall data platform architecture for Lakehouse/EDW environments, including reference patterns, technology selection, and integration blueprints • Design conceptual, logical, and physical data models for multi-tenant and vertical-specific data products • Standardize logical data layers, including ingest/raw, staged/curated, and serving • Establish data governance, metadata, cataloging, lineage, data contracts, and classification practices • Define security and compliance controls, including access management, data masking, encryption, network segmentation, and audit policies • Architect scalability, high availability, disaster recovery, capacity, and cost management strategies • Lead selection and integration of Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory/Data Fabric, orchestration, and monitoring components • Design and enforce CI/CD patterns for data artifacts, including testing, automated deployments, and rollback strategies • Define batch and streaming ingestion patterns, compaction strategies, partitioning schemes, and storage layouts • Specify observability practices including metrics, SLAs, dashboards, logging, tracing, and alerting • Serve as technical authority and mentor for Data Engineering teams • Perform architecture and code reviews for critical components • Collaborate with Data Product Owners, Security, Infrastructure, BI, and ML stakeholders to translate requirements into solutions and roadmaps • Design, develop, test, and deploy processing modules using Spark, PySpark/Scala, Spark SQL, and stored procedures • Build and optimize data pipelines on Databricks and database platforms • Implement DevOps practices, infrastructure-as-code, CI/CD pipelines, automated testing, and version control • Troubleshoot complex data quality, performance, and availability issues and implement continuous improvements

🎯 Requirements

• Previous experience as architect or lead technical role on enterprise data platforms • Hands-on experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming • Strong expertise in Spark, PySpark and/or Scala, Spark SQL, and distributed job optimization • Solid background in data warehouse and lakehouse design • Practical familiarity with Medallion, Lambda, and Kappa patterns • Experience integrating SaaS/ETL/connectors such as Fivetran • Experience with orchestration platforms such as Airflow, Azure Data Factory, and Data Fabric • Experience with relational and hybrid databases, including MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or equivalents • Proficiency in CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar • Experience packaging and deploying artifacts such as .whl files and containers • Experience with batch and streaming processing, file compaction, partitioning strategies, and storage tuning • Understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking • Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, or native cloud monitoring • Automation experience with CI/CD pipelines and trunk-based development is a plus • Advanced PySpark proficiency is a plus • Advanced Spark workflow optimization and orchestration proficiency is a plus • Databricks, cloud architect, or data certifications are a plus • Ability to identify, troubleshoot, and resolve complex data issues • Strong teamwork and communication skills • Intellectual curiosity and willingness to embrace new tools, technologies, and methodologies

🏖️ Benefits

• Full-time employment • Equal opportunity and inclusive work environment • Diversity commitment