Data Pipeline Engineer

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

PeerIslands

WebsiteLinkedInAll Job Openings

11 - 50 employees

🤖 Artificial Intelligence

☁️ SaaS

🏢 Enterprise

Artificial Intelligence • SaaS • Enterprise

PeerIslands is a technology company that specializes in transforming enterprise architecture with innovative strategies and advanced technology solutions. They excel in AI-enabled rapid application modernization and data engineering services, aiming to drive innovation and efficiency for their clients. With products like Migration Copilot, Developer Copilot, and Intelligent App Copilot, PeerIslands offers tailored solutions to rapidly modernize software applications and infrastructure. Their approach focuses on leveraging AI to ensure swift transformation and enhance user experiences while upholding strict security and compliance standards. Additionally, PeerIslands collaborates with partners in various industries including healthcare and financial services to provide modernization and data solutions.

📋 Description

• Design, build, and maintain scalable batch and real-time data ingestion pipelines. • Develop and manage Bronze, Silver, and Gold (Medallion) data layers. • Implement CDC, watermarking, checkpointing, and batch-to-stream data processing. • Build robust data quality, validation, reconciliation, and monitoring frameworks. • Develop identity resolution, deduplication, and Golden Record (MDM) solutions. • Create and maintain source-to-canonical data mappings and crosswalks. • Ensure schema validation, versioning, and data contract enforcement. • Collaborate with cross-functional teams to onboard new data sources and optimize data pipelines.

🎯 Requirements

• 7+ years of experience in Data Engineering / Data Pipeline development. • Strong experience with Apache Kafka (Producers, Consumers, Replay, DLQ, Exactly-once/Idempotent processing). • Strong SQL and ETL/ELT fundamentals. • Hands-on experience in Java and/or Python. • Experience with CDC, Batch & Streaming pipelines, and Medallion/Lakehouse architecture. • Experience implementing Data Quality, Validation, and Reconciliation frameworks. • Knowledge of Master Data Management (MDM), Identity Resolution, Deduplication, and Golden Record concepts. • Experience with source-to-target mapping, canonical data models, YAML/JSON configuration, and Git. • Good to Have Experience with probabilistic record matching and record linkage. • Schema Registry (Avro/Protobuf). • Experience extracting data from legacy/Mainframe systems. • Financial reconciliation experience. • Healthcare or Benefits Administration domain experience.