Senior Data Engineer, Databricks Migration

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Sigma Software Group

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 2002

💼 Consulting

🏥 Healthcare

🚘 Automotive

Consulting • Healthcare • Automotive

Sigma Software Group is a multinational company, established in 2002, that specializes in providing high-quality software development, graphic design, testing, and support services. The company focuses on delivering solutions across various industries such as automotive, telecommunications, aviation, advertising, gaming, banking, real estate, and healthcare. Sigma Software values professional growth, offers remote work opportunities worldwide, and caters to world-renowned clients like AstraZeneca, Scania, and SAS. The company emphasizes a culture of continuous education, mentorship, and flexible work environments, making it a preferred workplace for IT specialists aiming to work on complex solutions utilizing cutting-edge technologies. Sigma Software is committed to innovative solutions and engineering the future while also contributing to social causes such as charitable work in Ukraine.

📋 Description

• Participate in migrating a large-scale analytical platform from BigQuery to Databricks • Design and implement scalable Lakehouse architectures using Databricks and Delta Lake • Analyze ETL/ELT workloads and define migration approaches • Develop and optimize high-volume retail and analytical data pipelines • Implement incremental processing strategies and scalable transformation frameworks • Build and maintain Spark-based data processing solutions with PySpark • Design and maintain Bronze, Silver, and Gold medallion architecture layers • Implement data governance and security best practices using Unity Catalog • Collaborate with Data Science, Analytics, Product, and Customer Engineering teams • Participate in architecture discussions and technical solution design • Develop reusable data platform components and engineering standards • Conduct code reviews and improve platform reliability and maintainability • Troubleshoot and optimize complex SQL and Spark workloads • Support production deployments and platform modernization activities

🎯 Requirements

• 5+ years of professional experience as a Data Engineer • Strong programming skills in Python and advanced SQL • Hands-on commercial experience with Databricks • Strong knowledge of Apache Spark, primarily PySpark • Experience designing and building modern cloud-based data platforms • Experience developing ETL/ELT pipelines and large-scale data processing solutions • Hands-on experience with Delta Lake • Experience with Spark Declarative Pipelines • Experience with cluster monitoring, metrics analysis, and performance optimization • Strong understanding of distributed data processing architectures • Solid understanding of data warehousing concepts and dimensional modeling • Experience with Airflow or similar orchestration tools • Experience optimizing complex analytical SQL workloads • Experience implementing CI/CD practices for data engineering platforms • Strong troubleshooting and performance optimization skills • Ability to work collaboratively in cross-functional international teams • Upper-Intermediate or higher English level • Experience with GCP cloud services is a plus • Experience with AWS or Azure cloud platforms is a plus • Experience in retail analytics or pricing optimization domains is a plus • Experience supporting machine learning or AI-related data workloads is a plus • Experience with platform modernization and cloud migration initiatives is a plus

🏖️ Benefits

• Remote work option • Continuous learning opportunities • Technology growth opportunities • Meaningful engineering impact • Opportunity to work on complex international projects