543 - Senior Big Data Engineer

April 25

GoFasti

WebsiteLinkedInAll Job Openings

✓LatAm's Top Talent ✓Hire Faster ✓Stay Compliant ✓English Proficient ✓US Timezone

technology • software development • fullstack development • front end development • back end development

51 - 200

Description

• Design, build, and maintain efficient, reusable, and reliable PySpark code for processing large datasets. • Develop Python, Shell Scripts to automate tasks and enhance data processing capabilities. • Write optimized Hive queries for data extraction and to support analytics and reporting. • Develop and optimize PySpark jobs for processing large datasets. • Use Apache Airflow to orchestrate and schedule data pipelines. • Collaborate with other team members and stakeholders to ensure the company's data infrastructure will support business requirements. • Perform regular data audits to ensure data integrity and quality. • Participate in design and code reviews, ensuring adherence to best practices.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field. • 5-8 years of experience in a Big Data role. • Proficiency in Python and PySpark is essential. • Experience with Big Data technologies, specifically Spark, Hadoop and Hive. • Proven experience in Spark code optimization. • Experience in processing large volumes of data, ideally terabytes. • Experience with Apache Airflow and Control-M . • Excellent problem-solving skills and attention to detail. • Strong verbal and written communication skills.

Benefits

• The Salary range offered for this position varies from (USD) $3,500 - $5,000 per month, depending on seniority and skillset. • This position is for an independent contractor, through a payroll platform. • The talent will work REMOTELY allocated at our client.