Senior Staff Data Engineer, Databricks

🔥 0 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🚰 Data Engineer

👻 Ghost score 10%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 52 days.
• No salary listed.

Apply Now

Check your resume score

Customize resume + cover letter

Find Similar Remote Jobs

☆ Save☑️ Mark as applied❌ Hide

Report problem

Shield AI

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2015

🤖 Artificial Intelligence

🚀 Aerospace

🎖️ Defense

Artificial Intelligence • Aerospace • Defense

Shield AI is a leading developer of AI-driven military solutions, focusing on enhancing mission autonomy and battlefield awareness. Their platform, Hivemind, enables rapid deployment of intelligent systems for various defense applications, including drone operation and surveillance. With a commitment to utilizing advanced technology, Shield AI aims to protect service members and civilians by revolutionizing defense technologies through autonomous systems.

📋 Description

• Design and build batch and streaming ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake • Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design • Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement • Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards • Implement and maintain Databricks platform constructs, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns • Build and maintain CI/CD pipelines for ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod • Apply data classification, segregation, and handling requirements within pipeline design • Build data quality controls reflecting business meaning, record integrity, completeness, and expected domain behavior • Quarantine, flag, and route problematic records or datasets according to quality and compliance rules • Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and known limitations • Partner with the Analytics Engineer and domain teams to ensure reliable, governed Silver-layer data for Gold-layer modeling • Collaborate with domain engineering teams on ownership boundaries, onboarding patterns, data contracts, and support expectations

🎯 Requirements

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts • Experience with batch, CDC, and/or streaming ingestion patterns and associated operational trade-offs • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments • Strong SQL skills and solid grounding in data modeling fundamentals • Demonstrated ability to understand the business and regulatory context of processed data • Experience applying data classification, segregation, security, retention, or compliance requirements in regulated or security-sensitive environments • Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact • Comfort operating in a fast-moving platform build and shaping engineering standards • Databricks certification such as Data Engineer Associate or Professional (preferred) • Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting (preferred) • Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms (preferred) • Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment (preferred) • Experience enabling multiple business domains on a shared data platform while maintaining governance and ownership boundaries (preferred) • Must be authorized to work in the United States • Must satisfy applicable export-control requirements, such as being a United States citizen, lawful permanent resident, or protected individual

🏖️ Benefits

• Bonus • Benefits • Equity • Temporary benefits package applicable after 60 days of employment for temporary employees • Full-time regular employee offer package includes pay within range listed, bonus, benefits, and equity

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume