Data Engineer

🔥 0 minutes ago

🇲🇾 Malaysia – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🚰 Data Engineer

👻 Ghost score 10%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Experian

WebsiteLinkedInAll Job Openings

10,000+ employees

Founded 1996

💼 Consulting

📣 Marketing

📦 Logistics

Consulting • Marketing • Logistics

Experian is a global leader in digital experience, technology, and transformation. They partner with recognized brands to enhance customer understanding, innovate product strategies, and implement agile technology solutions. With a focus on delivering superior customer experiences through AI, cloud architecture, and project management, Experian helps businesses streamline their operations and achieve their objectives effectively.

📋 Description

• Design, build, test, deploy and operate scalable batch and streaming data pipelines for the Ascend Sandbox analytical platform • Develop Python, PySpark and SQL solutions to ingest, transform and process large-scale datasets • Design data models and reusable, trusted datasets/data products for analytics, data science and downstream applications • Engineer and optimise AWS-based data solutions using Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift and Amazon EMR • Apply data lake and lakehouse engineering practices, including storage formats, partitioning, schema evolution and storage/query optimisation • Design robust ETL/ELT patterns with incremental processing, dependency management, idempotency, retries, backfills and failure recovery • Implement workflow orchestration and scheduling using Airflow/MWAA, AWS Step Functions or AWS Glue Workflows • Build automated data quality controls and monitoring for completeness, validity, reconciliation, schema compliance, freshness, pipeline health and processing SLAs/SLOs • Optimise Spark workloads and distributed data processing for performance, scalability and cost • Implement data governance, metadata, lineage, cataloguing, security and access-control requirements • Collaborate with clients, business users, data scientists and technical teams to translate requirements into scalable solutions and integrations • Investigate and resolve production data and pipeline issues, perform root-cause analysis and implement reliability improvements • Contribute to code reviews, CI/CD and automated deployment practices while ensuring architecture, coding, testing, security and quality standards

🎯 Requirements

• Must possess a Degree / Advanced Diploma in Computer Science, Data Engineering, Software Engineering or equivalent • At least 6 years of experience in data engineering or developing data-intensive systems, including the design, build and production support of data pipelines • Strong hands-on experience with Python and PySpark for large-scale and distributed data processing • Strong SQL skills, including complex transformations, joins, window functions, query optimisation and performance tuning • Hands-on experience designing and operating scalable ETL/ELT data pipelines, including incremental loads, retries, backfills, dependency management and failure recovery • Experience with data modelling for analytical use cases, including dimensional modelling, facts/dimensions, normalization/denormalization and handling historical data • Hands-on experience with AWS data services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift and Amazon EMR • Experience with data lake/lakehouse concepts, storage layout, partitioning, schema evolution and efficient data formats such as Parquet • Strong understanding of Spark performance engineering, including partitioning, shuffles, joins, data skew, parallelism, memory/resource optimisation and troubleshooting • Experience with pipeline orchestration and workflow scheduling using tools such as Airflow/MWAA, AWS Step Functions, AWS Glue Workflows or equivalent • Experience implementing automated data quality checks and data observability • Practical knowledge of data governance, metadata management, data cataloguing, lineage and schema management • Practical understanding of AWS data security, including IAM/least privilege, encryption/KMS, secrets management, S3 security and data-access controls • Understanding of cloud cost and performance optimisation • Experience with Git-based workflows, branching strategies, code reviews, CI/CD pipelines and automated deployment • Experience writing automated unit, integration and data-pipeline tests for Python/PySpark transformations and data quality • Strong testing, debugging, troubleshooting and root-cause analysis skills in production data environments • Good understanding of Agile, SCRUM, SDLC and software engineering practices • Strong communication and stakeholder management skills • Strong analytical thinking, problem-solving, teamwork and willingness to learn, with the ability to work independently and deliver under tight deadlines

🏖️ Benefits

• Hybrid work arrangement: home and office • Inclusive paid leave • People-first and inclusive workplace