Senior Databricks Platform Engineer

🔥 8 minutes ago

⚔️ Virginia – Remote

⚔️ Virginia – Remote

💵 $98.6k - $202.8k / year

⏰ Full Time

🟠 Senior

🏗️ Platform Engineer

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

ICF

WebsiteLinkedInAll Job Openings

5001 - 10000 employees

Founded 1969

💼 Consulting

🏛️ Government

🏥 Healthcare

💰 $29M Grant on 2023-03

Consulting • Government • Healthcare

ICF is a global consulting firm founded over 55 years ago that partners with governments, businesses, and communities to deliver data-driven, technology-enabled solutions for complex public- and private-sector challenges. Its work areas include digital modernization, data analytics, resilience and adaptation, energy planning, federal health programs, disaster management, and infrastructure and transportation support. ICF provides consulting, program implementation, and advisory services to drive measurable outcomes and build community resilience.

📋 Description

• Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals • Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark • Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns • Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage • Develop frameworks and interfaces for governed training, inference, and reporting data across agency initiatives • Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools • Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services • Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and conduct root cause analysis • Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities • Participate in Agile planning, code reviews, demos, and retrospectives • Document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories • Explain architecture, trade-offs, risks, and operational impacts to technical and nontechnical stakeholders • Collaborate across data engineering, AI, cloud, security, application, and operations teams

🎯 Requirements

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience • More than 5 years of data engineering experience • At least 3 years specifically supporting Azure Databricks and related Azure data services • Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala • Experience designing and optimizing large-scale ETL/ELT pipelines • Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies • Experience managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization • Experience with CI/CD or infrastructure as code using Git, Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable technologies • Experience with Agile delivery, data governance, data security, performance troubleshooting, and stakeholder communication • Ability to satisfy FDIC personnel eligibility and background-investigation requirements • U.S. Citizenship required • Must reside in the U.S., be authorized to work in the U.S., and perform all work in the U.S. • Preferred: experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse • Preferred: Databricks certification or comparable demonstrated experience • Preferred: streaming technologies including Kafka, Confluent, and Structured Streaming • Preferred: medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms • Preferred: Generative AI, NLP, RAG, vector stores, agents, or Databricks AI/ML workloads • Preferred: MLflow, model deployment pipelines, AWS SageMaker, AWS Bedrock, or cross-cloud architectures • Preferred: BCDR, high availability, disaster recovery, data retention, and enterprise security controls • Preferred: ATO processes, NIST or federal compliance controls, vulnerability remediation, SonarQube, Checkmarx, or Zero Trust • Preferred: Docker, Kubernetes, REST APIs, FastAPI, Node.js, or cloud-native application integration • Preferred: federal data modernization programs

🏖️ Benefits

• Reasonable accommodations for disability, veteran, and sincerely held religious beliefs • Equal opportunity employment • Confidential accommodation support • Benefits information provided under the Transparency in Coverage Act