Senior Azure Data Engineer

March 19

Enable Data

WebsiteLinkedInAll Job Openings

A leading provider of advanced data, application, and cloud engineering services.

Analytics • Big Data • Cloud Infrastructure • DevOps • Distributed Computing

51 - 200

Description

• Design, implement, and maintain cloud-based data pipelines to enable efficient storage, processing, and analysis of large volumes of data. • Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures. • Evaluate and select appropriate cloud infrastructure and services to support data storage, processing, and analytics needs, considering factors such as scalability, performance, cost, and security. • Implement data ingestion pipelines, integrating various data sources and ensuring data quality, integrity, and timeliness. • Design and optimize data storage solutions, including data lakes, data warehouses, and data marts, to support different types of data processing and analysis. • Implement data processing workflows using technologies such as Apache Spark, Apache Hadoop, or cloud-native data processing services. • Develop and maintain data governance and security policies, ensuring compliance with data protection regulations and industry best practices. • Monitor and optimize the performance of data platforms, identifying bottlenecks and implementing optimizations to improve data processing speed and efficiency. • Collaborate with DevOps teams to automate deployment, monitoring, and management of data platforms using infrastructure-as-code and CI/CD practices. • Stay up-to-date with emerging technologies and industry trends in cloud computing, big data processing, and data engineering. • Provide guidance and mentorship to junior team members, fostering a culture of learning and innovation.

Requirements

• Bachelor's or Master's degree in computer science, engineering, or a related field. • 8+ years of proven experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data platforms. • Strong experience in Microsoft Azure • Proficiency in the latest Big Data tools/technologies like Hive, Hadoop, Yarn, Kafka, Spark Stream. • Proficiency in data processing frameworks such as Apache Spark, Apache Hadoop, or cloud-native data processing services (Azure Data Lake, Azure Data factory, Azure Databricks, Azure Synapse, Snowflake, CosmosDB) • Experience with data integration and ETL (Extract, Transform, Load) processes, including tools like Apache Airflow or cloud-native orchestration services. • Experience with workload and job optimization for cost reduction • Knowledge of database systems (relational and NoSQL) and data modeling principles. • Familiarity with data governance, data security, and compliance frameworks. • Understanding of distributed computing principles and scalable architectures. • Experience with infrastructure-as-code tools like Terraform • Excellent problem-solving and troubleshooting skills, with the ability to address complex data platform challenges. • Strong communication and collaboration skills to work effectively with cross-functional teams

Benefits

• Opportunities to work with cutting-edge technologies in the Information Technology and Services industry • Chance to advance skills as a Senior Spark Engineer