Senior Data Engineer

April 3

Intellectsoft

WebsiteLinkedInAll Job Openings

Engineering Your Vision

Custom Software Development • Mobile App Development • Enterprise solutions • Team Extension • Dedicated Teams

51 - 200

Description

• Develop connectors for Kafka to streamline the syncing of updates from source data repositories. • Establish partitioned Kafka topics to optimize the synchronization of updates to destination data marts. • Leverage Apache Flink for crafting intricate data analytics workloads to enable real-time monitoring and transformations. • Deploy dashboards using Datadog and Cloudwatch to uphold system health and fulfill user needs. • Institute schema registries to uphold data governance standards while catering to diverse data requirements. • Collaborate closely with a West Coast-based scrum team, contributing to daily pull request submissions, code reviews, documentation maintenance, backlog management, and build validation across environments within sprint cycles lasting 2-4 weeks. • Coordinate with other scrum teams to ensure coherence on data contracts, API specifications, and deployment timelines. • Architect database schemas with a focus on query access patterns. • Establish and manage CI/CD pipelines using infrastructure-as-code principles. • Transition on-premises ETL jobs from PHP to AWS Flink and Glue processes gradually. • Work alongside QA Engineers to develop automated test suites. • Engage with end-users to troubleshoot service interruptions and champion our data product offerings. • Maintain vigilant oversight of data quality, promptly addressing discrepancies, latency issues, and defects.

Requirements

• Proficiency in Apache Kafka (preferably MSK flavour), Debezium, Python, Apache Flink or PySpark Streaming, MySQL (preferably RDS flavours), CDK or Terraform, Athena, Glue, Lambda, Appflow, HANA/4, PHP, Redis, Docker, and JavaScript. • At least 6 years of hands-on experience collaborating within professional scrum teams or equivalent educational background. • A minimum of 3 years of practical experience in designing and indexing relational databases. • At least 2 years of practical experience in constructing and managing real-time data streams. • Demonstrated proficiency with at least 1 year of experience in developing monitoring dashboards.

Benefits

• 36 paid absence days per year for work-life balance of each specialist + 1 additional day for each following year of cooperation with the company • Up to 10 unused absence days can be added to income after 12 months of cooperation • Health insurance compensation • Depreciation coverage for personal laptop usage for project needs • Udemy courses of your choice • Regular soft-skills training • Excellence Centers meetups