Senior Data Engineer

Job not on LinkedIn

🔥 0 minutes ago

🇺🇸 United States – Remote

💵 $150k - $180k / year

⏰ Full Time

🟠 Senior

🚰 Data Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

Apply Now

Find Similar Remote Jobs

Receive Emails For Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Effectual

WebsiteLinkedInAll Job Openings

201 - 500 employees

🏢 Enterprise

🤖 Artificial Intelligence

💰 Private Equity Round on 2019-11

Cloud • Enterprise • Artificial Intelligence

Effectual is an AWS Premier Tier Services Partner specializing in enterprise digital transformation, focusing on cloud migration and modernization. They offer a range of services including cloud strategy and design, application development, and data analytics, leveraging cutting-edge technology like Generative AI to drive innovation and efficiency. Effectual works across various sectors such as financial services, aerospace, and the public sector, aiming to optimize cloud costs and enhance data security for their clients.

📋 Description

• Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis • Develop real-time data pipelines that handle high-volume, high-velocity data streams • Implement event-driven architectures and microservices patterns for streaming data processing • Create and optimize data streaming topologies for complex event processing scenarios • Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms • Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments • Implement Kafka Connect pipelines for streaming data integration • Design optimal Kafka topic partitioning strategies and replication configurations • Monitor and optimize Kafka cluster performance, throughput, and latency • Implement Kafka security configurations including SSL/TLS, SASL, and ACLs • Manage Kafka Schema Registry for data serialization and evolution • Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions • Configure Kinesis Analytics applications for real-time stream processing • Optimize Kinesis shard management and auto-scaling configurations • Implement Kinesis data retention and archival strategies • Integrate Kinesis with other AWS services for comprehensive streaming solutions • Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda • Implement complex event processing (CEP) patterns for real-time analytics • Build streaming ETL pipelines that transform data in motion • Create real-time aggregations, windowing operations, and stateful stream processing • Optimize streaming query performance and resource utilization • Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases • Implement data lineage and monitoring for streaming data pipelines • Create automated data quality checks and validation for streaming data • Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution • Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements • Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation • Automate deployment and management of streaming infrastructure through CI/CD pipelines • Monitor streaming system health, performance metrics, and alerting • Implement disaster recovery and high availability strategies for streaming systems • Stay current with emerging trends in streaming technologies and cloud-native solutions • Collaborate with data architects, data scientists, and application teams on streaming data requirements • Support rigorous project governance through daily progress reviews and time tracking • Provide technical leadership and mentorship to junior data engineers • Communicate complex streaming concepts to technical and non-technical stakeholders • Operate with transparency and responsiveness to support high-performing teams.

🎯 Requirements

• 7+ years of experience in the data engineering field with significant streaming data specialization • Bachelor's degree in Computer Science, Engineering, or related STEM field • Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools • Proven experience with AWS MSK and Amazon Kinesis services in production environments • Strong background in real-time data processing and stream analytics.

🏖️ Benefits

• Medical, dental, and vision health insurances, • Short term disability, long term disability and life insurances, • 401k with Company match • Paid time off (PTO) (120 hours PTO that accrue over one year) • Paid time off for major holidays (14 days per year) • These and any other employee benefit offerings are subject to management’s discretion and may change at any time.