Data Engineering Manager

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Penn Foster

WebsiteLinkedInAll Job Openings

201 - 500 employees

📚 Education

👥 B2C

🤝 B2B

Education • B2C • B2B

Penn Foster is an online, accredited career school and college that provides self-paced, career-focused diplomas, certificates, associate and bachelor’s degrees across vocational and professional fields. Its programs cover healthcare, veterinary technology (including an AVMA-CVTEA accredited Vet Tech program), skilled trades (HVACR, electrician, plumbing), business, IT, criminal justice, design, and high school diplomas accredited through the Distance Education Accrediting Commission. Penn Foster serves primarily individual learners with flexible, online study options and also offers training solutions for organizations and employers.

📋 Description

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices • Mentor and develop junior and mid-level Data Engineers • Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence • Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices • Serve as technical lead for Penn Foster Group's Databricks Lakehouse platform • Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks • Define Databricks development best practices across Workflows, Repos, Jobs, notebooks, Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation • Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon • Build reusable ingestion frameworks for batch, streaming, CDC, and API-based integration patterns • Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency • Develop trusted semantic data products, Genie Spaces, and Genie Ontologies for Databricks Genie • Establish semantic modeling, governed metrics, business metadata, and AI-ready dataset practices • Collaborate with Data Science and MLOps teams on machine learning, generative AI, and advanced analytics initiatives • Design scalable Lakehouse architectures for analytics, reporting, AI, and operational data products • Implement enterprise governance using Unity Catalog, including lineage, security, metadata management, and access controls • Champion automated testing, monitoring, observability, data quality, and production reliability • Serve as technical escalation point for complex production issues and lead root cause analysis • Drive platform modernization initiatives and contribute to technical roadmaps and platform strategy • Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams • Translate business requirements into technical solutions for reporting, analytics, AI, and strategic decision-making

🎯 Requirements

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field, or equivalent practical experience • 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering • 3+ years of experience leading technical initiatives and mentoring engineering teams • Demonstrated success developing engineers and driving engineering excellence and technical standards • Strong communication and collaboration skills with technical and business stakeholders • Expert knowledge of Databricks Lakehouse Platform, including Apache Spark/PySpark, Delta Lake, Unity Catalog, Workflows & Jobs, Repos, SQL Warehouses, Cluster Policies, Serverless Compute, MLflow, Lakehouse Monitoring, Auto Loader, and Delta Live Tables/Lakeflow • Expert knowledge of SQL and Python • Strong experience with BI tools such as Power BI, Tableau, or Business Objects • Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID, RBAC, networking, and cloud security • Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products • Deep understanding of Spark optimization, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization • Experience implementing CI/CD pipelines, Git-based workflows, Infrastructure as Code, and automated testing • Strong understanding of data governance, metadata management, data quality, observability, security, and compliance • Successful completion of a role-specific assessment • Successful completion of applicable pre-employment screening • Completion of federal employment eligibility verification through Form I-9 • Preferred: Databricks Genie, semantic models, AI-ready data products, machine learning/MLOps, Generative AI, dbt, Great Expectations, Databricks Certified Data Engineer Professional, and/or Microsoft Azure certifications

🏖️ Benefits

• Medical insurance • Dental insurance • Vision insurance • Flexible spending • Generous paid time off • Sponsored volunteer opportunities • 401K with a company match • Free access to all of our online programs • Remote work arrangement • On-camera collaborative work environment