Data Manager, Databricks

πŸ•’ October 21

Where this job comes from

Posted on Allata's careers page β†—Allata's careers page, not through a job board

First seen by Remote Rocketship October 21

Still open on Allata's site. Checked 2 hours ago.

Apply Now
Logo of Allata

Allata

201 - 500 employees

🚘 Automotive

πŸ₯ Healthcare

πŸ“¦ Logistics

Automotive β€’ Healthcare β€’ Logistics

Allata is a global consulting firm that helps businesses achieve digital excellence by crafting unique customer experiences, identifying revenue-generating opportunities, and improving operational efficiencies. They offer a wide range of services including strategic services, technology and cloud solutions, data and insights, and artificial intelligence development. Allata's approach is experience-led, strategy-aligned, tech-enabled, and data-driven. They work across multiple industries such as automotive, financial services, health and life sciences, high tech, logistics and transportation, among others, to drive modernization, personalization, innovation, and efficiency. With a team of former IT leaders and consulting professionals, Allata guides clients through complex digital landscapes to deliver value-based outcomes and help organizations integrate digital strategies seamlessly into their overall business strategies.

πŸ“‹ Description

β€’ Define overall data platform architecture for Lakehouse/EDW environments, including reference patterns, technology selection, and integration blueprints β€’ Design conceptual, logical, and physical data models for multi-tenant and vertical-specific data products β€’ Standardize logical data layers, including ingest/raw, staged/curated, and serving β€’ Establish data governance, metadata, cataloging, lineage, data contracts, and classification practices β€’ Define security and compliance controls, including access management, data masking, encryption, network segmentation, and audit policies β€’ Architect scalability, high availability, disaster recovery, capacity, and cost management strategies β€’ Lead selection and integration of Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory/Data Fabric, orchestration, and monitoring components β€’ Design and enforce CI/CD patterns for data artifacts, including testing, automated deployments, and rollback strategies β€’ Define batch and streaming ingestion patterns, compaction strategies, partitioning schemes, and storage layouts β€’ Specify observability practices including metrics, SLAs, dashboards, logging, tracing, and alerting β€’ Serve as technical authority and mentor for Data Engineering teams β€’ Perform architecture and code reviews for critical components β€’ Collaborate with Data Product Owners, Security, Infrastructure, BI, and ML stakeholders to translate requirements into solutions and roadmaps β€’ Design, develop, test, and deploy processing modules using Spark, PySpark/Scala, Spark SQL, and stored procedures β€’ Build and optimize data pipelines on Databricks and database platforms β€’ Implement DevOps practices, infrastructure-as-code, CI/CD pipelines, automated testing, and version control β€’ Troubleshoot complex data quality, performance, and availability issues and implement continuous improvements

🎯 Requirements

β€’ Previous experience as architect or lead technical role on enterprise data platforms β€’ Hands-on experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming β€’ Strong expertise in Spark, PySpark and/or Scala, Spark SQL, and distributed job optimization β€’ Solid background in data warehouse and lakehouse design β€’ Practical familiarity with Medallion, Lambda, and Kappa patterns β€’ Experience integrating SaaS/ETL/connectors such as Fivetran β€’ Experience with orchestration platforms such as Airflow, Azure Data Factory, and Data Fabric β€’ Experience with relational and hybrid databases, including MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or equivalents β€’ Proficiency in CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar β€’ Experience packaging and deploying artifacts such as .whl files and containers β€’ Experience with batch and streaming processing, file compaction, partitioning strategies, and storage tuning β€’ Understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking β€’ Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, or native cloud monitoring β€’ Automation experience with CI/CD pipelines and trunk-based development is a plus β€’ Advanced PySpark proficiency is a plus β€’ Advanced Spark workflow optimization and orchestration proficiency is a plus β€’ Databricks, cloud architect, or data certifications are a plus β€’ Ability to identify, troubleshoot, and resolve complex data issues β€’ Strong teamwork and communication skills β€’ Intellectual curiosity and willingness to embrace new tools, technologies, and methodologies

πŸ–οΈ Benefits

β€’ Full-time employment β€’ Equal opportunity and inclusive work environment β€’ Diversity commitment

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ October 21

Autolus Therapeutics

501 - 1000

πŸ₯ Healthcare

πŸ’Ό Consulting

🏭 Manufacturing

Clinical data management leader overseeing outsourced vendors, database lock, and regulatory deliverables for Autolus CAR T trials. Supporting life-changing cancer and autoimmune therapies.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’° $350M Post-IPO Equity - Autolus on 2024-02

⏰ Full Time

🟑 Mid-level

🟠 Senior

πŸ“Š Data Scientist

πŸ•’ October 21

InMarket

201 - 500

πŸ“£ Marketing

Senior Data Scientist developing causal inference, machine learning, and AI solutions for InMarket’s advertising measurement platform. Taking prototypes from research through production at scale.

πŸ•’ October 20

CVS Health

10,000+ employees

πŸ₯ Healthcare

βš•οΈ Healthcare Insurance

πŸ›’ Retail

CVS Health Manager advancing data governance, analytics reporting, and quality assurance. Coordinating SQL, Tableau, cloud analytics, validation, and cross-functional projects.

πŸ•’ October 20

Wiley

5001 - 10000

πŸ“š Education

πŸ”¬ Science

Principal Data Scientist building production NLP and applied AI pipelines for Wiley’s scientific publishing corpus. Extracting research intelligence for downstream agentic applications.

πŸ•’ October 20

Wiley

5001 - 10000

πŸ“š Education

πŸ”¬ Science

Senior Data Scientist building production NLP and AI systems for Wiley’s scientific publishing corpus. Owning evaluations, model pipelines, and agentic applications from development through deployment.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok