Spark Data Engineer

Job not on LinkedIn

November 18

Apply Now

Receive Emails with Similar Jobs

Mactores

WebsiteLinkedInAll Job Openings

Enterprise • Data

Mactores is a company that provides end-to-end data platform solutions aimed at accelerating business value through automation. Since 2008, Mactores has been helping businesses with digital transformation, offering services like Enterprise Data Lakes, Scalable Databases, Modern Data Warehouses, Automated DataOps, MLOps, and Generative AI solutions. They focus on enabling faster and cost-effective migrations and modernizations in data analytics, partnering with leading platforms to drive innovation and success. Mactores works alongside tech teams to strategize and implement the right data solutions timely and efficiently.

51 - 200 employees

Founded 2008

🏢 Enterprise

📋 Description

• Architect, design, and build scalable data pipelines and distributed applications using Apache Spark (Spark SQL, DataFrames, RDDs) • Develop and manage ETL/ELT pipelines to process structured and unstructured data at scale. • Write high-performance code in Scala or PySpark for distributed data processing workloads. • Optimize Spark jobs by tuning shuffle, caching, partitioning, memory, executor cores, and cluster resource allocation. • Monitor and troubleshoot Spark job failures, cluster performance, bottlenecks, and degraded workloads. • Debug production issues using logs, metrics, and execution plans to maintain SLA-driven pipeline reliability. • Deploy and manage Spark applications on on-prem or cloud platforms (AWS, Azure, or GCP). • Collaborate with data scientists, analysts, and engineers to design data models and enable self-serve analytics. • Implement best practices around data quality, data reliability, security, and observability. • Support cluster provisioning, configuration, and workload optimization on platforms like Kubernetes, YARN, or EMR/Databricks. • Maintain version-controlled codebases, CI/CD pipelines, and deployment automation. • Document architecture, data flows, pipelines, and runbooks for operational excellence.

🎯 Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field. • 4+ years of experience building distributed data processing pipelines, with deep expertise in Apache Spark. • Strong understanding of Spark internals (Catalyst optimizer, DAG scheduling, shuffle, partitioning, caching). • Proficiency in Scala and/or PySpark with strong software engineering fundamentals. • Solid expertise in ETL/ELT, distributed computing, and large-scale data processing. • Experience with cluster and job orchestration frameworks. • Strong ability to identify and resolve performance bottlenecks and production issues. • Familiarity with data security, governance, and data quality frameworks. • Excellent communication and collaboration skills to work with distributed engineering teams. • Ability to work independently and deliver scalable solutions in a fast-paced environment

🏖️ Benefits

• Equal opportunities in all of our employment practices • Recruitment, compensation, promotions, transfers, disciplinary action, layoff, training, and social programs respect non-discrimination principles.