Tech Lead – Databricks, Data Platform

Job not on LinkedIn

🕒 September 4

🇮🇳 India – Remote

⏰ Full Time

🔴 Lead

🚰 Data Engineer

Where this job comes from

Posted on Palo Alto Labs's careers page ↗Palo Alto Labs's careers page, not through a job board

First seen by Remote Rocketship September 4

Still open on Palo Alto Labs's site. Checked 2 hours ago.

Apply Now
Logo of Palo Alto Labs

Palo Alto Labs

51 - 200 employees

Founded 2025

💼 Consulting

🔒 Cybersecurity

🤖 Artificial Intelligence

Consulting • Cybersecurity • Artificial Intelligence

Palo Alto Labs is a technology managed-services and innovation firm that partners with businesses to deliver next-generation management technology, digital transformation, and operational excellence. The company provides managed cloud services, cybersecurity, SAP and AI capabilities, finance-as-a-service (FaaS), and data transformation and management, while designing prototypes and processes for new services and experiences. Its focus is on empowering enterprise customers and global capability centers to improve performance, security, and go-to-market readiness.

📋 Description

• Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines • Design and implement CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads • Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies • Define and implement automated testing and quality gates within CI/CD pipelines • Drive Infrastructure as Code practices using Terraform and reusable modules for Databricks and Azure infrastructure • Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management • Establish and maintain IAM across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls • Implement secure and scalable Unity Catalog and data governance patterns with architecture and security teams • Develop reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks • Define and govern code promotion and release management across development, QA, and production environments • Lead end-to-end orchestration using managed Airflow, ensuring scheduling, dependency management, monitoring, and recovery • Define and track platform SLAs, SLOs, and SLIs; lead incident triage, root cause analysis, and corrective actions • Implement FinOps best practices for monitoring, optimizing, and allocating Databricks and cloud infrastructure costs • Evaluate and improve platform tooling, architecture, reliability, security, performance, and developer productivity • Provide technical guidance and mentorship to engineers and establish engineering best practices, standards, and reusable patterns • Partner with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to deliver enterprise data solutions • Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness • Drive performance optimization and reliability improvements for production data workloads

🎯 Requirements

• 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines • Strong hands-on experience with Databricks in production environments, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms • Strong experience with Microsoft Azure and cloud-native data platforms • Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar • Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation • Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration • Strong programming/scripting skills in Python, Bash, or PowerShell • Experience implementing automated testing, validation, and quality gates for data pipelines and data models • Experience with production workload management, including monitoring, logging, troubleshooting, performance tuning, and incident management • Strong understanding of cloud security, IAM, RBAC, data access controls, and governance • Experience with FinOps, cloud cost optimization, and resource utilization monitoring • Ability to provide technical leadership, mentor engineers, and drive engineering standards across teams • Strong communication and stakeholder management skills with the ability to work effectively with technical and business stakeholders • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience (preferred) • Experience with dbt, SQL, Delta Lake, and modern data engineering practices (preferred) • Experience with Unity Catalog and enterprise data governance (preferred) • Experience designing reusable platform frameworks and accelerators (preferred) • Experience in CPG, retail, manufacturing, or distribution environments (preferred) • Experience with disaster recovery, business continuity, and highly available cloud architectures (preferred) • Databricks or Azure certifications are a plus

🏖️ Benefits

• Remote work arrangement

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🕒 September 3

Exavalu

201 - 500

💼 Consulting

🏥 Healthcare

🛡️ Insurance

Azure Data Architect designing secure, scalable Azure warehouses, lakes, and pipelines for IT services company Exavalu. Leading ingestion, governance, and big-data processing architecture.

🕒 August 21

TD SYNNEX

10,000+ employees

💼 Consulting

📦 Logistics

📣 Marketing

Global data center fabric architect standardizing TD SYNNEX’s spine-leaf, VXLAN/EVPN, and segmentation infrastructure. Serving as technical authority across India, Americas, EMEA, and APJ.

🕒 August 10

n Human Resources & Management Systems [ nHRMS ]

11 - 50

💼 Consulting

🏥 Healthcare

📦 Logistics

Principal Data Engineer designing enterprise-scale data and analytics solutions across GCP and Azure. Leading architecture, modeling, cloud implementation, and technical strategy for client engagements.

🕒 August 5

CLIQHR Recruitment Services

11 - 50

💼 Consulting

📣 Marketing

🎯 Recruiter

Data Architect designing and modernizing cloud-based enterprise data platforms for analytics, AI, ML, and mission-critical applications. Collaborating with Engineering, Product, Data Science, and DevOps teams on scalable data solutions.

🕒 August 3

TWO95 International, Inc

51 - 200

☁️ SaaS

🔒 Cybersecurity

🤖 Artificial Intelligence

Big Data Platform Engineer managing AKS-based infrastructure, administering Azure services, and ensuring platform compliance. Optimizing performance through infrastructure automation and collaborative support with Data Engineering teams.

Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok