Data Pipeline Engineer

Job not on LinkedIn

🔥 1 minute ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

PeerIslands

WebsiteLinkedInAll Job Openings

11 - 50 employees

💼 Consulting

🏥 Healthcare

🤖 Artificial Intelligence

Consulting • Healthcare • Artificial Intelligence

PeerIslands is a technology company that specializes in transforming enterprise architecture with innovative strategies and advanced technology solutions. They excel in AI-enabled rapid application modernization and data engineering services, aiming to drive innovation and efficiency for their clients. With products like Migration Copilot, Developer Copilot, and Intelligent App Copilot, PeerIslands offers tailored solutions to rapidly modernize software applications and infrastructure. Their approach focuses on leveraging AI to ensure swift transformation and enhance user experiences while upholding strict security and compliance standards. Additionally, PeerIslands collaborates with partners in various industries including healthcare and financial services to provide modernization and data solutions.

📋 Description

• Build batch-seed and event-tail ingestion per source system, including seed→tail watermark hand-off, idempotent upserts, and dedup ledgers • Build and operate medallion layers with reprocess-from-Bronze, pipeline orchestration (checkpoints, retry/backoff, DLQ), and full observability • Build data-quality gates (quarantine / pass-with-flag), quality scoring, and a reconciliation engine covering count, record, and financial reconciliation — financial is zero-tolerance • Build identity matching combining deterministic rules with probabilistic scoring and confidence bands; deliver deduplication, golden-record materialization, and survivorship rules, calibrating match thresholds with labelled data • Author and maintain source→canonical structural mappings and value crosswalks (e.g., collapsing 1,800+ raw employment-status values to ~20 standard ones) as governed, versioned configuration • Enforce data contracts at the boundary: schema registry, fail-fast validation, and semver-compatible schema evolution

🎯 Requirements

• 5+ years building production data pipelines at scale • Kafka depth: consumers/producers, replay, DLQ, exactly-once / idempotent processing patterns • Strong SQL and solid ETL fundamentals • Java and/or Python in production • Medallion / lakehouse layering, CDC, watermark/checkpoint patterns, and batch–stream hand-off • Data-quality frameworks: validation rules, quarantine and re-entry, quality scoring, reconciliation • Entity resolution / MDM exposure: record matching, dedup, survivorship — via commercial tools (Informatica MDM, Reltio) or custom builds • Data mapping and crosswalk discipline: profiling messy datasets, authoring governed reference data, config-as-code (YAML/JSON, Git)

🏖️ Benefits

• Health insurance • Retirement plans • Paid time off • Flexible work arrangements • Professional development