Data Pipeline Engineer – Microsoft Fabric

Job not on LinkedIn

🔥 1 minute ago

🇧🇷 Brazil – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🚰 Data Engineer

👻 Ghost score 25%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

iblue

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2016

💼 Consulting

🏥 Healthcare

🛡️ Insurance

Consulting • Healthcare • Insurance

iblue is a Brazilian IT consulting firm that accelerates digital transformation for the financial sector, including banks, insurers and other financial services. The company focuses on Data & AI as its core, delivering tailored solutions across Big Data & AI, cloud & security, process integration and automation, and agile squads & engineering to improve efficiency and growth. iblue positions itself as a strategic partner offering deep technology expertise and domain knowledge to convert operational challenges into measurable business results.

📋 Description

• Configure and test the connection to SharePoint Online for the defined pilot scope • Develop pipeline components to ingest selected documents and metadata into the OneLake Bronze layer • Implement incremental loading patterns, change detection, retries, execution logging, and exception handling • Develop Silver layer logic for cleansing, normalization, deduplication, version handling, and conflict resolution • Implement Gold layer components to prepare validated information products in accordance with business rules • Support attribute extraction, information enrichment, and validation by business stakeholders • Record lineage between source documents, pipeline executions, transformed records, and validated results • Test document creation, update, versioning, duplication, deletion, exception, and modification scenarios • Implement operational dashboards, alerts, execution history, logs, and support diagnostics • Collaborate with SharePoint and information architecture teams to understand source structures, metadata, permissions, and change behavior • Document technical constraints, unsupported behaviors, workarounds, performance, and production considerations • Package code, notebooks, configurations, and pipeline definitions using approved version control and release practices • Produce tests, technical documentation, deployment instructions, and operational runbooks • Build and validate the ingestion flow for documents, metadata, and selected data from SharePoint Online into OneLake • Work under the guidance of the Fabric and data architecture teams, in collaboration with SharePoint, information architecture, AI, and business teams

🎯 Requirements

• Hands-on experience with Microsoft Fabric, OneLake, lakehouses, notebooks, and data pipelines • Knowledge of Spark-, PySpark-, or SQL-based transformations • Experience with pipeline architecture and development, incremental loads, and change detection • Knowledge of schema management, data quality, observability, recovery, and exception handling • Experience integrating REST APIs, files, document metadata, or Microsoft 365 data sources • Proficiency in SQL and at least one data engineering language, such as Python or PySpark • Knowledge of medallion architecture and data organization across the Bronze, Silver, and Gold layers • Experience with source control, parameterization, environment configuration, deployment, and testing • Ability to develop repeatable tests that demonstrate the platform’s actual behavior • Experience creating technical documentation, runbooks, deployment procedures, and support materials • Intermediate to advanced English (B2 or higher), with the ability to communicate verbally and in writing in a corporate and technical environment • Availability to work in the Eastern Standard Time (EST) time zone • Analytical thinking and technical investigation skills • Rigor in data validation, test execution, and results documentation • Ability to troubleshoot and investigate integration or processing failures • Organization, attention to detail, and engineering discipline • Collaboration with architects, developers, and business professionals • Clear communication regarding platform constraints, risks, failures, and behavior • Proactivity and a focus on automation and continuous improvement • Commitment to quality, traceability, observability, and sustainability

🏖️ Benefits

• CLT employment option • SulAmerica health insurance for you (nationwide coverage, ward accommodation, and copayments) • TotalPass gym membership • Latest-generation laptop • Transportation allowance • Meal allowance: BRL 770.00 (based on an average of 22 business days per month), with the option to allocate the amount to VR or a Flash card • Candidate referral bonus • Creditas financial services partnership • Annual performance review with an IDP (Individual Development Plan) • Training through iblue Academy • Udemy training • Certifications (AWS, Microsoft, IBM, and H2O) • Educational partnerships (with possible financial assistance tied to your performance review) • Structured Y-shaped career path (you can choose a management or specialist track) • Benefit packages offered by the cooperative, which you can select according to your needs • 15 days of paid leave after the 12th month