Senior Data Engineer

Job not on LinkedIn

🕒 August 11

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Veltris

WebsiteLinkedInAll Job Openings

501 - 1000 employees

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

Veltris is a trusted technology partner that drives innovation by leveraging digital technologies, artificial intelligence, and Industry 4. 0 capabilities. The company specializes in digital product engineering, enabling organizations across various sectors to accelerate technology transformations and develop smarter, connected products. With a design thinking approach and agile software development practices, Veltris helps clients optimize their product development processes and achieve strategic advantages in the market.

📋 Description

• Build, maintain, and optimize Bronze, Silver, and Gold layer pipelines in Microsoft Fabric using ADF and PySpark notebooks • Own the ingestion-to-consumption chain for source systems including Acumatica ERP, HubSpot CRM, AutoStore, RingCentral, website analytics, and marketplace channels • Implement and maintain event-triggered ADF pipelines using Acumatica SaaS push notifications, Azure Service Bus, and Azure Function App receivers • Build OData watermark fallback pipelines with incremental loading, pagination, and session concurrency management • Design and manage CDC strategies, CDF versioning, watermark tracking, and initial versus incremental load logic • Ensure Silver batching runs within SLA and scales as new sources are onboarded • Develop and maintain Silver transformation notebooks for deduplication, null handling, type casting, and data-quality rule application • Build and optimize Gold KPI layer notebooks using star-schema fact and dimension tables • Manage orchestration notebook execution sequences and dependencies • Validate Bronze Delta schemas and implement schema evolution through the Delta transaction log • Maintain source_object_config and watermark_registry control tables • Implement and maintain data-quality rules across Silver-layer tables; route failed records to the failed-records table • Manage Git promotion from development to production using Azure DevOps and maintain environment parity • Instrument notebooks with SDK logging to Azure Log Analytics and monitor pipeline health dashboards • Maintain data pipeline feeds powering Pecan AI forecasting and recommendation models • Ensure Fabric SQL endpoint integration with Pecan AI and model retraining pipelines remain reliable

🎯 Requirements

• 5+ years of experience in data analysis, data engineering, or a combined analytics and engineering role • Experience preferably in a fast-paced ecommerce or B2B distribution environment • Hands-on experience with Microsoft Fabric Lakehouses, Warehouses, ADF pipelines, PySpark notebooks, Delta Lake medallion architecture, and Power BI semantic models • Experience building and maintaining production-grade ingestion pipelines, CDC, watermark and OData incremental-load strategies, schema management, and orchestration • Working knowledge of Delta table format, transaction logs, schema enforcement and evolution, time travel, VACUUM, and CDF • Expertise in complex SQL queries • Proficiency in Python, pandas, PySpark, NumPy, and ETL scripting • Experience writing KQL queries for Azure Log Analytics, Microsoft Fabric real-time analytics, or Azure Data Explorer • Hands-on experience with Azure services including ADF, Key Vault, Service Bus, Function Apps, Log Analytics, and DevOps, and/or AWS • Hands-on experience with Power BI semantic model design, DAX measures, and certified dashboards • Familiarity with Microsoft Purview, data cataloging, lineage, sensitivity classification, KPI certification workflows, data-quality frameworks, and DQ rule implementation • Proficiency with Git and Azure DevOps for source control, branching, and pipeline-based promotion • Ability to translate business objectives into analytical questions and engineering requirements • Ability to communicate findings and technical trade-offs to technical and non-technical stakeholders, including executive audiences • Experience with Fabric CI/CD, workspace management, Azure Service Bus, Azure Function Apps, Event Hubs, Fabric F-SKU capacity management, and FinOps cost governance is a plus