Senior Serverless Spark Migration Engineer

Job not on LinkedIn

🔥 0 minutes ago

🌐 Brazil, Mexico, +2 more countries – Remote

🇧🇷 Brazil – Remote 🇲🇽 Mexico – Remote 🇺🇸 United States – Remote 🇨🇦 Canada – Remote

⏰ Full Time

🟠 Senior

👷🏻‍♀️ Engineer

👻 Ghost score 13%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Virtasant

WebsiteLinkedInAll Job Openings

51 - 200 employees

💼 Consulting

🏢 Enterprise

🤝 B2B

Consulting • Enterprise • B2B

Virtasant is a cloud services and optimization company that helps businesses migrate to, manage, and build on public cloud platforms. They combine a proprietary automation platform with managed services and FinOps expertise to reduce cloud costs (claiming average savings of over 50%), support Cloud FinOps programs, and deliver outcome-based engagements rather than hourly or seat-based billing. Virtasant also provides enterprise AI guidance and partners with major cloud providers (AWS, Google Cloud, Azure) to deliver migration, optimization, and 24/7 managed operations.

📋 Description

• Lead migrations of enterprise Spark workloads from on-premise environments to AWS and GCP • Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization • Determine migration approaches across rehost, replatform, refactor, modernize, or retire • Modernize traditional cluster-based workloads for serverless Spark where appropriate • Design and implement architectures using AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery • Refactor legacy PySpark/Scala/Spark SQL applications for cloud portability, scalability, and reliability • Migrate workloads using Hadoop, HDFS, YARN, Hive, and on-premise Spark clusters • Troubleshoot and optimize Spark workloads, including partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution • Benchmark performance and optimize serverless workloads for performance, reliability, and cloud cost • Build reusable migration tooling, automation, templates, and frameworks • Implement CI/CD and Infrastructure as Code using tools such as Terraform • Define testing, validation, cutover, rollback, observability, and production-readiness patterns • Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams • Own the complete migration lifecycle: Discover, Assess, Design, Refactor, Migrate, Validate, Optimize, Operate

🎯 Requirements

• 8+ years of experience across data engineering, distributed systems, cloud engineering, or platform engineering • 5+ years of hands-on Apache Spark experience in enterprise environments • Strong PySpark and/or Scala development experience • Proven experience migrating large-scale Spark workloads between infrastructure platforms • Hands-on experience with both AWS and GCP • Experience with on-premise Hadoop/Spark ecosystems, including HDFS, YARN, and Hive • Deep understanding of Spark internals and distributed processing • Strong SQL and data engineering fundamentals • Experience with cloud data lakes and object storage • Strong production troubleshooting and performance-tuning experience • Experience with CI/CD, Git, and Infrastructure as Code • Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization • Experience with EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is valuable

🏖️ Benefits

• Remote work arrangement • Coverage during Pacific Hours (8:00 AM–5:00 PM PST)