Senior Staff Data Engineer, Databricks

🕒 6 days ago

🇺🇸 United States – Remote

⏰ Full Time

🔴 Lead

🚰 Data Engineer

Where this job comes from

Posted on Shield AI's careers page ↗Shield AI's careers page, not through a job board

First seen by Remote Rocketship 6 days ago

Still open on Shield AI's site. Checked 1 hour ago.

Apply Now
Logo of Shield AI

Shield AI

501 - 1000 employees

Founded 2015

🤖 Artificial Intelligence

🚀 Aerospace

🎖️ Defense

Artificial Intelligence • Aerospace • Defense

Shield AI is a leading developer of AI-driven military solutions, focusing on enhancing mission autonomy and battlefield awareness. Their platform, Hivemind, enables rapid deployment of intelligent systems for various defense applications, including drone operation and surveillance. With a commitment to utilizing advanced technology, Shield AI aims to protect service members and civilians by revolutionizing defense technologies through autonomous systems.

📋 Description

• Design and build batch and streaming ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake • Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design • Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement • Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards • Implement and maintain Databricks platform constructs, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns • Build and maintain CI/CD pipelines for ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod • Apply data classification, segregation, and handling requirements within pipeline design • Build data quality controls reflecting business meaning, record integrity, completeness, and expected domain behavior • Quarantine, flag, and route problematic records or datasets according to quality and compliance rules • Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and known limitations • Partner with the Analytics Engineer and domain teams to ensure reliable, governed Silver-layer data for Gold-layer modeling • Collaborate with domain engineering teams on ownership boundaries, onboarding patterns, data contracts, and support expectations

🎯 Requirements

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts • Experience with batch, CDC, and/or streaming ingestion patterns and associated operational trade-offs • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments • Strong SQL skills and solid grounding in data modeling fundamentals • Demonstrated ability to understand the business and regulatory context of processed data • Experience applying data classification, segregation, security, retention, or compliance requirements in regulated or security-sensitive environments • Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact • Comfort operating in a fast-moving platform build and shaping engineering standards • Databricks certification such as Data Engineer Associate or Professional (preferred) • Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting (preferred) • Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms (preferred) • Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment (preferred) • Experience enabling multiple business domains on a shared data platform while maintaining governance and ownership boundaries (preferred) • Must be authorized to work in the United States • Must satisfy applicable export-control requirements, such as being a United States citizen, lawful permanent resident, or protected individual

🏖️ Benefits

• Bonus • Benefits • Equity • Temporary benefits package applicable after 60 days of employment for temporary employees • Full-time regular employee offer package includes pay within range listed, bonus, benefits, and equity

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🕒 November 13

Agiloft

201 - 500

⚖️ Legal

💼 Consulting

🏥 Healthcare

Staff Data Engineer leading contract data models, pipelines, and analytics infrastructure. Building data foundations for Agiloft’s AI-powered contract lifecycle management platform.

🕒 November 12

Voyager Technologies

501 - 1000

🏭 Manufacturing

📦 Logistics

🎖️ Defense

Senior DataOps Engineer building regulated cloud data platforms and pipelines for Voyager Technologies’ space, defense, and national security missions. Leading AWS, Azure, Fabric, and DataOps modernization.

🕒 November 12

Signet Jewelers

10,000+ employees

🛒 Retail

🛍️ eCommerce

👗 Fashion

Senior AWS Data Architect shaping Signet Jewelers’ secure AWS data foundation. Enabling enterprise analytics, machine learning, GenAI, and agentic AI across its jewelry retail business.

🕒 November 12

Qualified Health

51 - 200

🏥 Healthcare

🤖 Artificial Intelligence

☁️ SaaS

Senior Staff Engineer setting architecture and building Databricks and Temporal pipelines for Qualified Health's generative AI healthcare platform. Governing data and enabling safe LLM workloads.

🇺🇸 United States – Remote

💵 $215k - $235k / year

💰 $30M Seed on 2025-02

⏰ Full Time

🔴 Lead

🚰 Data Engineer

🕒 November 11

Zscaler

5001 - 10000

🔒 Cybersecurity

☁️ SaaS

🏢 Enterprise

AI and data platform manager leading RAG, knowledge graph, and cloud data infrastructure engineering. Managing distributed teams for Zscaler’s cloud security platform.

🇺🇸 United States – Remote

💵 $164k - $205k / year

💰 Secondary Market on 2017-11

⏰ Full Time

🔴 Lead

🚰 Data Engineer

Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok