Tech Lead – Databricks, Data Platform

Job not on LinkedIn

🕒 2 days ago

🇮🇳 India – Remote

⏰ Full Time

🟠 Senior

🚰 Data Engineer

👻 Ghost score 15%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Palo Alto Labs

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2025

💼 Consulting

🔒 Cybersecurity

🤖 Artificial Intelligence

Consulting • Cybersecurity • Artificial Intelligence

Palo Alto Labs is a technology managed-services and innovation firm that partners with businesses to deliver next-generation management technology, digital transformation, and operational excellence. The company provides managed cloud services, cybersecurity, SAP and AI capabilities, finance-as-a-service (FaaS), and data transformation and management, while designing prototypes and processes for new services and experiences. Its focus is on empowering enterprise customers and global capability centers to improve performance, security, and go-to-market readiness.

📋 Description

• Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines • Design and implement CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads • Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies • Define and implement automated testing and quality gates within CI/CD pipelines • Drive Infrastructure as Code practices using Terraform and reusable modules for Databricks and Azure infrastructure • Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management • Establish and maintain IAM across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls • Implement secure and scalable Unity Catalog and data governance patterns with architecture and security teams • Develop reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks • Define and govern code promotion and release management across development, QA, and production environments • Lead end-to-end orchestration using managed Airflow, ensuring scheduling, dependency management, monitoring, and recovery • Define and track platform SLAs, SLOs, and SLIs; lead incident triage, root cause analysis, and corrective actions • Implement FinOps best practices for monitoring, optimizing, and allocating Databricks and cloud infrastructure costs • Evaluate and improve platform tooling, architecture, reliability, security, performance, and developer productivity • Provide technical guidance and mentorship to engineers and establish engineering best practices, standards, and reusable patterns • Partner with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to deliver enterprise data solutions • Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness • Drive performance optimization and reliability improvements for production data workloads

🎯 Requirements

• 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines • Strong hands-on experience with Databricks in production environments, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms • Strong experience with Microsoft Azure and cloud-native data platforms • Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar • Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation • Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration • Strong programming/scripting skills in Python, Bash, or PowerShell • Experience implementing automated testing, validation, and quality gates for data pipelines and data models • Experience with production workload management, including monitoring, logging, troubleshooting, performance tuning, and incident management • Strong understanding of cloud security, IAM, RBAC, data access controls, and governance • Experience with FinOps, cloud cost optimization, and resource utilization monitoring • Ability to provide technical leadership, mentor engineers, and drive engineering standards across teams • Strong communication and stakeholder management skills with the ability to work effectively with technical and business stakeholders • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience (preferred) • Experience with dbt, SQL, Delta Lake, and modern data engineering practices (preferred) • Experience with Unity Catalog and enterprise data governance (preferred) • Experience designing reusable platform frameworks and accelerators (preferred) • Experience in CPG, retail, manufacturing, or distribution environments (preferred) • Experience with disaster recovery, business continuity, and highly available cloud architectures (preferred) • Databricks or Azure certifications are a plus

🏖️ Benefits

• Remote work arrangement