Lead/Sr AI Engineer, Azure Data Bricks

Job not on LinkedIn

🔥 1 minute ago

🇮🇳 India – Remote

⏰ Full Time

🟠 Senior

☁️ Cloud Engineer

👻 Ghost score 15%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Enable Data

WebsiteLinkedInAll Job Openings

51 - 200 employees

💼 Consulting

🏥 Healthcare

🏭 Manufacturing

Consulting • Healthcare • Manufacturing

Enable Data is a company that offers modern IT solutions to empower its customers by leveraging innovative data analytics and cloud platforms to drive increased value across their business ecosystem. They specialize in the implementation of large-scale SAS applications, distributed rules engines for processing health-related data, and managing data ingest frameworks. Enable Data serves industries requiring advanced data analytics and IT solutions, with a particular focus on healthcare quality and big data management.

📋 Description

• Build production-ready AI applications and deploy them on Azure Databricks and Azure cloud infrastructure • Design end-to-end data and AI pipelines using Azure Databricks • Develop ETL/ELT workflows with Python/PySpark and SQL • Implement CI/CD pipelines for Databricks jobs, notebooks, and workflows • Integrate Databricks with Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions, and related services • Optimize jobs for performance, cost, and reliability • Build reusable, modular code • Collaborate with data scientists, platform engineers, and business stakeholders to move models from experimentation to production • Implement logging, monitoring, and error handling for production pipelines • Develop and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines, for NLP, computer vision, and predictive analytics • Fine-tune LLMs using LoRA/QLoRA and integrate them with Azure OpenAI or Hugging Face models • Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search • Apply responsible AI practices, including bias detection and model governance • Ingest data into Databricks from sources such as SAP and website scraping • Write scenario-based and edge-case tests and finalize code changes and acceptance criteria with stakeholders • Send custom notifications from Databricks using APIs and custom Teams channel webhooks • Build chat UI interfaces and Claude-based applications • Use graphs to detect rule dependencies and circular references • Reengineer code and map it to user requirements • Implement AI safety guardrails and caching mechanisms

🎯 Requirements

• Experience with Azure Databricks (jobs, workflows, clusters, Unity Catalog preferred) • Strong Python skills, especially PySpark rather than only pandas • SQL proficiency, including complex joins, window functions, and analytical queries • Azure Cloud knowledge, including ADLS Gen2, ADF, Key Vault, and IAM concepts • Pipeline orchestration and deployment experience, including CI/CD and environment promotion • Experience with Azure DevOps • Strong understanding of ML lifecycle and MLOps best practices • Experience deploying models using MLflow or similar frameworks • Experience with ML and Generative AI workloads on Databricks • Experience with RAG, embeddings, or inference pipelines • Knowledge of Terraform, ARM, or Bicep • Knowledge of Databricks Asset Bundles • Knowledge of Airflow or ADF orchestration • Production monitoring and cost optimization experience • Knowledge of LangChain or similar frameworks • Experience with Azure AI services, including Azure Machine Learning and Azure Cognitive Services • Knowledge of Databricks Unity Catalog permissions, OBO tokens, and Service Principals • Ability to build and deploy Databricks Apps using Visual Studio Code • Ability to use FastAPI and React to build minimal single-page applications • Prompt engineering skills, including system and user prompts • Basic knowledge of token economics and prompt rule enforcement • Ability to use Claude Vision API and standard text processing • Understanding of Retrieval Augmented Generation, including AI Search indexes and hierarchical chunking • Knowledge of basic AI safety guardrails and caching mechanisms