Senior Data Engineer – AWS, Python

Job not on LinkedIn

20 hours ago

Apply Now

Receive Emails with Similar Jobs

3Pillar Global

WebsiteLinkedInAll Job Openings

SaaS • Enterprise • Artificial Intelligence

3Pillar Global is a modern application strategy, design, and engineering firm that specializes in delivering strategic software development initiatives for various industries. They offer a range of services, including application technology strategy, digital product engineering, data and analytics, and artificial intelligence development. 3Pillar Global focuses on helping organizations transform their bold ideas into breakthrough solutions by leveraging cutting-edge technologies such as generative and multimodal AI. They work with partners and clients across multiple sectors, including healthcare, financial services, insurance, media, and information services, to solve complex technology challenges and deliver high-performing results.

1001 - 5000 employees

☁️ SaaS

🏢 Enterprise

🤖 Artificial Intelligence

💰 Private Equity Round on 2021-10

📋 Description

• Team Management: Manage the team of 3-4 Data Engineers. • Roster & Scheduling: Create, manage, and maintain team rosters and schedules to ensure full 8-hour coverage, 7 days a week, including rotational weekend and holiday coverage. • Availability: Must be able to work a day shift and manage a team that provides 7-day-a-week coverage, including weekends and holidays on rotation. • Coverage Assurance: Handle all team leave requests (planned and unplanned). You are responsible for ensuring coverage is always maintained, especially during medical or other unexpected absences. • Performance & SLAs: Act as the primary operational point of contact, ensuring the team executes all tasks efficiently, makes sound decisions, and adheres to all agreed-upon SLAs. • Final Escalation Point: Serve as the final technical escalation point for the team; personally handle the most critical job failures and system issues. • Expert RCA: Lead all high-priority Root Cause Analysis (RCA) efforts, documenting solutions and implementing permanent fixes to prevent recurrence. • Strategic Improvement: Drive the team's efforts in performance tuning and AWS Glue cost optimization. • Automation Strategy: Define and lead the development of new automation jobs (using Glue, Lambda, and Step Functions) to reduce manual support tasks. • Complex Data Management: Oversee all data integrity and synchronization tasks, including CDC reloads and write-backs to external databases like Teradata. • Knowledge Transfer: Own the knowledge transfer process to internal resources and ensure the team maintains excellent documentation and runbooks.

🎯 Requirements

• Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services. • Demonstrated expertise with a minimum of 5-8 years of relevant experience. • Deep understanding of Data engineering, ETL principles and data warehouse design. • Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue. • Automation: Proven ability to design and develop automation solutions from the ground up. • Problem-Solving: Expert-level troubleshooting and RCA skills. • Leadership: Ability to work independently, make critical decisions, and provide technical guidance. • Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis. • Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata. • Preferred Skills: AWS Certified.