Principal Data Engineer

May 3

Apply Now

Get a free AI resume review

Swiftly

WebsiteLinkedInAll Job Openings

Your Retail Technology & Solutions Partner.

SwiftLane Checkout • Retail Media • Retail Media Platform • Digital Platform • AI

51 - 200

Description

• Design, build, deploy, and maintain ETL (Extract, Transform, Load) processes using Azure hosted technologies • Orchestrate data workflows to integrate structured and unstructured data from multiple sources • Manage customer import and export processes that involve Secure File Transfer Protocol (SFTP), Azure Storage, and AWS S3 for data exchange • Ensure data security and compliance while transferring data between systems • Support our inclusive and supportive engineering culture, Mentor engineering team • Advocate for Agile development methodologies and participate in the software development lifecycle (SDLC) activities • Implement and maintain security measures to protect company assets and customer data • Develop contingency plans to address potential issues • Collaborate with data scientists, software engineers, and other stakeholders to create efficient data pipelines using Databricks • Optimize data processing at scale using Databricks’ capabilities while managing costs • Author REST servers and client to enhance data and provide admin portal functionality • Develop models and data sets to enhance personalized services for retail grocery shoppers • Demonstrate how to leverage data to improve customer experiences through effective targeting • Other related duties as assigned

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or a related field • 10+ years of Data Engineering experience • 5+ years in a small team, startup environment • Strong knowledge of Python • Foundational understanding of Azure, Eventhubs, storage accounts and infrastructure considerations • Experience with SQL, message queuing, stream processing, and advertising platforms • Deep experience with data pipelines (ETL/ELT, Spark, dbt, Airflow, Databricks, etc.) • Deep experience with OLAP database design, build, run (Star/ Snowflake, Fact / Dim, Snowflake, etc.) • Deep experience with distributed application design, build, run (api, queues, idempotent, encapsulated, etc.) • Experienced with cloud infrastructure (infra as code, cdk, Terraform, Kubernetes, containers & images • Proven experience in a leadership role within engineering, with a track record of successful project delivery • Excellent communication, and interpersonal skills

Benefits

• 11 paid holidays • Generous Accrued Time Off increasing with years of service • Generous paid sick time • Annual day of service