Senior Staff Data Engineer, Databricks

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Shield AI

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2015

🤖 Artificial Intelligence

🚀 Aerospace

🎖️ Defense

Artificial Intelligence • Aerospace • Defense

Shield AI is a leading developer of AI-driven military solutions, focusing on enhancing mission autonomy and battlefield awareness. Their platform, Hivemind, enables rapid deployment of intelligent systems for various defense applications, including drone operation and surveillance. With a commitment to utilizing advanced technology, Shield AI aims to protect service members and civilians by revolutionizing defense technologies through autonomous systems.

📋 Description

• The Senior Staff Data Engineer will help build and operate the enterprise lakehouse on Databricks, creating the governed data foundation that supports multiple business domains and downstream analytics. • Responsible for scalable ingestion, reliable data processing, and strong technical controls across the Bronze and Silver layers of the medallion architecture. • Design and build ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake. • Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design. • Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement. • Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards. • Implement and maintain Databricks platform constructs needed for secure delivery. • Build and maintain CI/CD pipelines for data platform assets. • Apply data classification, segregation, and handling requirements within the pipeline design. • Build data quality controls that reflect actual business meaning, record integrity, completeness, and expected domain behavior. • Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations. • Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling. • Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

🎯 Requirements

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines. • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL. • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts. • Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each. • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments. • Strong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role. • Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development. • Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment. • Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact. • Comfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

🏖️ Benefits

• Pay within range listed + Bonus + Benefits + Equity • Temporary benefits package (applicable after 60 days of employment)