Senior Data Engineer – AWS, PySpark, Kafka

Job not on LinkedIn

🔥 5 minutes ago

🇧🇷 Brazil – Remote

⏳ Contract/Temporary

🟠 Senior

☁️ Cloud Engineer

👻 Ghost score 13%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Leega

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2010

💼 Consulting

📣 Marketing

🔌 API

Consulting • Marketing • API

Leega is a leading technology solutions provider in Latin America, specializing in data analytics and cloud solutions. As the first company in the region certified by Google Cloud for Data Analytics, Leega offers a range of services including application development, machine learning, and risk management analytics. The firm partners with major cloud services such as AWS and Microsoft Azure to help businesses enhance their data management and transition effectively to the cloud, ultimately driving digital transformation and innovation.

📋 Description

• Design and evolve event-driven architecture on AWS, using Kafka as the backbone between transactional systems and the analytics platform • Develop Kafka producers and consumers with PySpark (Spark Structured Streaming), managing offsets, checkpointing, idempotency, late data, and delivery semantics • Define and advocate for the design of topics, keys, partitioning, retention, compaction, and event contracts • Govern schemas in Schema Registry, defining compatibility and evolution without breaking consumers • Implement CDC for transactional databases via Kafka Connect or Debezium • Integrate sources such as custodians, distributors, and market data providers’ APIs • Build event persistence in the S3 Data Lake using Iceberg or Delta, integrating Glue, EMR, and Redshift • Optimize performance and costs in Kafka and Spark, including lag, rebalancing, brokers, micro-batches, skew, and spill • Instrument observability and automated quality controls for event flows • Ensure lineage, traceability, access controls, encryption, sensitivity-based segregation, and compliance with Brazil’s LGPD and industry regulations • Automate provisioning and deployment using infrastructure as code and CI/CD • Document architectural decisions (ADRs), participate in code reviews and pair programming, and help define standards • Collaborate with Wealth Management, Sales, Risk, Compliance, and Operations • Serve as a technical reference for mid-level and junior engineers

🎯 Requirements

• 6+ years of experience in data engineering, including at least 2 years with architectural responsibility for event-driven platforms • Proven track record of launching and operating a Kafka streaming platform in production, including incident response and on-call support • Deep knowledge of Apache Kafka: topics, partitions, keys, consumer groups, offsets, retention, compaction, rebalancing, producer and consumer configuration, and lag troubleshooting • Hands-on experience with producers and consumers using PySpark (Spark Structured Streaming) integrated with Kafka, including checkpointing, triggers, watermarks, and state management • Strong experience with the AWS ecosystem: Amazon MSK, S3, Glue, EMR, Lambda, IAM, and Redshift or equivalent technologies • Python applied to data engineering, with modular, testable, and version-controlled code, type annotations, and automated tests • Advanced SQL, including window functions, CTEs, and execution plan analysis • Experience with Schema Registry and Avro, Protobuf, or JSON Schema formats, including schema compatibility and evolution • Experience with Kafka Connect or Debezium for Change Data Capture (CDC) • Infrastructure as code: Terraform, CloudFormation, or CDK • Git, code review, and CI/CD applied to data engineering • Experience with open table formats such as Iceberg or Delta and Lakehouse architecture on S3 • Strong focus on numerical correctness and delivery guarantees • Clear written communication for documenting event architecture • Ability to independently investigate issues end to end • Ability to serve as a technical reference through code reviews, pair programming, and mentoring • Ability to translate business needs into sustainable event designs • Honesty when estimating scope, risk, and timelines • Nice to have: financial markets experience; regulatory data involving CVM, BACEN, BSM, or ANBIMA; Confluent; Kafka Streams; ksqlDB; Apache Flink; Kinesis; Airflow; Step Functions; dbt; data quality, catalog, or observability tools; and AWS or Confluent certifications

🏖️ Benefits

• Ongoing professional development • Dynamic and collaborative environment • 100% remote work • Opportunity to work on financial markets and investment projects • Work with AWS, Kafka, PySpark, and data engineering technologies • Inclusive company: applications from people with disabilities are welcome