Senior Data Engineer – Platform Modernization, Migration

Job not on LinkedIn

πŸ•’ October 20

Where this job comes from

Posted on In All Media's careers page β†—In All Media's careers page, not through a job board

First seen by Remote Rocketship October 20

Still open on In All Media's site. Checked 4 hours ago.

Apply Now
Logo of In All Media

In All Media

1001 - 5000 employees

πŸ’Ό Consulting

πŸ“£ Marketing

☁️ SaaS

Consulting β€’ Marketing β€’ SaaS

In All Media is a global community-driven service organization specializing in agile development services. The company focuses on developing digital solutions through a collaborative approach that leverages swarming intelligence and team dynamics. They are dedicated to aligning industry needs with the interests of their technical community members, who possess extensive experience in various domains such as UX/UI design, quality engineering, cloud computing, big data, Salesforce, and more. In All Media emphasizes a secure development environment and a strong collaborative spirit, striving for innovation and excellence in software production. Their model relies on pre-established teams with substantial past experience working together, ensuring efficient and effective delivery of customized development solutions.

πŸ“‹ Description

β€’ Execute the end-to-end migration of approximately 1,000 datasets to Snowflake β€’ Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3 β€’ Migrate legacy Hive data structures and datasets to Apache Iceberg using the Snowpark Connect framework/connectors β€’ Reconfigure, update, and migrate Confluent/Apache Kafka streaming connections across cloud regions β€’ Adapt existing data pipelines to target Snowflake environments β€’ Optimize SQL queries using CTEs, window functions, and complex joins β€’ Manage, test, and debug Apache Airflow DAGs and workflow scheduling β€’ Implement testing strategies and conduct hands-on code debugging to ensure data quality and consistency β€’ Modernize streaming infrastructure across cloud regions β€’ Uphold strict data quality and performance standards during platform modernization

🎯 Requirements

β€’ Minimum 2+ years of hands-on data engineering experience working with large-scale datasets, Apache Airflow, and Apache Spark β€’ Strong software engineering fundamentals β€’ Hands-on Python development experience β€’ Advanced SQL skills, including complex joins, CTEs, window functions, and query tuning β€’ Proven experience with Snowflake data modeling and schema architecture β€’ Direct experience building and testing pipelines using Apache Airflow, Apache Kafka, and Apache Spark / PySpark β€’ Expertise with Apache Iceberg and Apache Parquet data formats β€’ Experience with AWS S3, including data lakes, bucket partitioning, and lifecycle policies β€’ Ability to define testing strategies and independently debug complex distributed data pipelines β€’ Strong communication skills and ability to work independently in a fast-paced environment β€’ Must use a client-provisioned AWS WorkSpace β€’ Must demonstrate good connectivity to an AWS WorkSpace region by completing the connectivity test and submitting a screenshot as supporting evidence β€’ English required for all interviews, documentation, and daily communication

πŸ–οΈ Benefits

β€’ Fully remote work arrangement β€’ Full-time, permanent employment β€’ Client-provisioned AWS WorkSpace

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ October 16

Lateral Group

201 - 500

πŸ’Ό Consulting

πŸ“£ Marketing

πŸ“¦ Logistics

Senior Data Engineer stabilizing Snowflake platforms and automating reporting for Lateral Group, a global technology and venture-building company. Supporting real estate investment decisions through reliable data infrastructure.

πŸ•’ September 23

Azumo

51 - 200

πŸ’Ό Consulting

πŸ₯ Healthcare

πŸ“¦ Logistics

Data Engineer building production pipelines, warehouses, and RAG retrieval infrastructure for Azumo’s client AI systems. Fully remote across Latin America, using modern cloud and data platforms.

πŸ•’ September 1

Blend360

501 - 1000

πŸ₯ Healthcare

🏨 Hospitality

✈️ Travel

Lead Backend Data Engineer building scalable Java distributed systems and big data pipelines. Supporting Blend’s AI and data services through Kubernetes, observability, and performance optimization.

πŸ‡¦πŸ‡· Argentina – Remote

πŸ’° $100M Private Equity Round on 2022-08

⏰ Full Time

🟑 Mid-level

🟠 Senior

🚰 Data Engineer

πŸ•’ September 1

Blend360

501 - 1000

πŸ₯ Healthcare

🏨 Hospitality

✈️ Travel

Lead Backend Data Engineer building scalable Java distributed systems and big-data pipelines for Blend’s AI services. Managing Kubernetes workloads, observability, and production reliability.

πŸ‡¦πŸ‡· Argentina – Remote

πŸ’° $100M Private Equity Round on 2022-08

⏰ Full Time

🟑 Mid-level

🟠 Senior

🚰 Data Engineer

πŸ•’ August 23

Power Digital Marketing

501 - 1000

πŸ’Ό Consulting

πŸ‘₯ B2C

πŸ“£ Marketing

Data Engineer building unified, AI-ready pipelines and semantic layers for Power Digital, a tech-enabled growth marketing firm. Supporting agency, client, product, and AI initiatives.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok