Mid-Level Data Engineer – GCP, DBT

Job not on LinkedIn

🕒 June 24

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Leega

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2010

🔌 API

🤖 Artificial Intelligence

API • Artificial Intelligence • Cloud Solutions

Leega is a leading technology solutions provider in Latin America, specializing in data analytics and cloud solutions. As the first company in the region certified by Google Cloud for Data Analytics, Leega offers a range of services including application development, machine learning, and risk management analytics. The firm partners with major cloud services such as AWS and Microsoft Azure to help businesses enhance their data management and transition effectively to the cloud, ultimately driving digital transformation and innovation.

📋 Description

• Load/Pipeline Analysis and Planning: • Evaluate the data warehouse architecture and requirements. • Map data, transformations and processes to GCP services (Cloud Storage, BigQuery, Dataproc). • Define the data migration strategy (full load, incremental, CDC). • Develop a data architecture plan on GCP. • Data Design and Modeling on GCP: • Design table schemas in BigQuery considering performance, cost and scalability. • Define partitioning and clustering strategies for BigQuery. • Model data zones in Cloud Storage (Bronze, Silver and Gold). • ELT/ETL Pipeline Development: • Create data transformation routines using Dataproc (Spark) or Dataflow to load data into BigQuery. • Translate business logic and existing transformations into GCP. • Implement data validation and quality mechanisms. • Performance and Cost Optimization: • Optimize BigQuery queries to reduce costs and improve performance. • Tune and optimize Spark jobs on Dataproc. • Monitor and optimize GCP resource usage to control costs. • Data Security and Governance: • Implement and ensure data security in transit and at rest. • Define and enforce IAM policies to control access to data and resources. • Ensure compliance with data governance policies. • Monitoring and Support: • Troubleshoot performance and functionality issues of data pipelines and GCP resources. • Documentation: • Document the architecture, data pipelines, data models and operational procedures. • Communication: • Communicate effectively with team members, stakeholders and other company areas. • Ensure clear communication between architecture definitions and software components, and oversee the evolution and quality of the team's development. • Jira / Agile Methodologies: • Be familiar with agile methodologies, their rituals, and be proficient with Jira.

🎯 Requirements

• Proven experience with DBT for a minimum of 3 years; • Proficiency with: • models (staging, intermediate, marts) • ref() and source() • macros (Jinja) • seeds and snapshots • tests (not null, unique, custom) • Layered organization: • Staging → Transform → Mart (Data Warehouse) • Google Cloud Platform (GCP): • BigQuery: Deep knowledge of data modeling, query optimization, partitioning, clustering, data ingestion (streaming and batch), security and data governance. • Cloud Storage: Experience managing buckets, storage classes, lifecycle policies, access control (IAM) and data security. • Dataproc: Skills in provisioning, configuring and managing Spark/Hadoop clusters, optimizing jobs, and integrating with other GCP services. • Dataflow/Composer/DBT: Knowledge of orchestration and data processing tools for ELT/ETL pipelines. • Cloud IAM (Identity and Access Management): Implementing security policies and granular access control. • VPC, Networking and Security: Understanding of networks, subnets, firewall rules and cloud security best practices. • Programming Languages: • Python and PySpark: Essential for automation scripts, data pipeline development and integration with GCP APIs. • SQL (advanced): For BigQuery, DBT and data transformations. • Shell Scripting: For task automation. • Version Control: • Git/GitHub/Bitbucket.

🏖️ Benefits

• 🏥 Health plan (Porto Seguro) • 🦷 Dental plan (Porto Seguro) • 💰 Profit Sharing (PLR) • 👶 Childcare allowance • 🍽️ Meal and food vouchers (Alelo) • 💻 Home office allowance • 📚 Partnerships with educational institutions • 🚀 Incentives for certifications, including Cloud certifications • 🎁 Livelo points • 🏋️‍♂️ TotalPass • 🧘‍♂️ Mindself