Lead Data Engineer

Job not on LinkedIn

🔥 6 minutes ago

🦀 Maryland – Remote

🦀 Maryland – Remote

⏰ Full Time

🟠 Senior

🚰 Data Engineer

👻 Ghost score 15%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Black Canyon Group Inc.

WebsiteLinkedInAll Job Openings

1 - 10 employees

💼 Consulting

📣 Marketing

📚 Education

Consulting • Marketing • Education

Black Canyon Group Inc. is an E-Learning provider that specializes in equipping B2B executives with the skills and strategies needed to compete, win, grow, and thrive in their markets. Focused on small-to-medium-sized and privately-held companies, the group emphasizes B2B marketing, professional development, and strategic business practices. They are known for their creation of the NanoMarketing Method ®, a precise system aimed at enhancing business development for organizations whose customers are other businesses and not individual consumers.

📋 Description

• Design and implement the core DuckDB / Apache Iceberg data platform on AWS • Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables • Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog • Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns • Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance • Build and maintain analytical datasets supporting application services, reporting, analytics, and data science • Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction • Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings • Lead migration and harmonization of datasets from multiple source systems • Implement reconciliation processes demonstrating completeness and accuracy between source and target systems • Establish automated data-quality monitoring, operational observability, and testing • Implement agentic and AI-enabled data engineering patterns for source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance • Support integration of structured data with associated document and geospatial datasets • Lead development of certified data products, analytical datasets, and business metrics • Optimize DuckDB queries and Iceberg datasets for performance and efficiency • Develop engineering documentation, operational procedures, and technical handoff materials

🎯 Requirements

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field • 6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers • Strong hands-on experience with DuckDB and Apache Iceberg • Advanced SQL and strong Python development experience • Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet • Strong understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization • Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets • Experience with AWS data services, particularly Amazon S3 and AWS Glue • Experience designing reusable, metadata-driven ingestion and transformation frameworks • Experience implementing automated data quality, reconciliation, monitoring, and operational controls • Experience with agentic or AI-enabled data engineering solutions, including the use of agents or AI-assisted workflows for data mapping, pipeline generation, validation, testing, or troubleshooting • Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows • Strong troubleshooting, performance tuning, and technical leadership skills

🏖️ Benefits

• Medical, dental and vision coverage • 401k plan with employer contribution • Paid holidays • Vacation • Tuition reimbursement

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume