Senior Data Engineer

Job not on LinkedIn

September 10

Apply Now

Receive Emails with Similar Jobs

Spiritual Data

WebsiteLinkedInAll Job Openings

Artificial Intelligence • SaaS • Productivity

Spiritual Data is an advanced analytics platform that empowers data teams through the use of artificial intelligence to enhance productivity and efficiency. Their tools, such as DataPilot, aim to automate and optimize data management tasks, revolutionizing how data engineering teams operate. By providing AI-driven insights and recommendations, Spiritual Data enables teams to improve their data documentation, quality testing, and query handling, resulting in significant cost savings and operational improvements. The company focuses on aiding analytics, platform, and data engineers by integrating their solutions seamlessly into daily workflows and popular platforms like VSCode, Python, and Slack, offering tools that auto-generate models, conduct tests, and provide documentation. Spiritual Data is a trusted solution among leading enterprises, offering robust security features without compromising on AI functionality.

- employees

🤖 Artificial Intelligence

☁️ SaaS

⚡ Productivity

📋 Description

• This remote position requires an overlap with US Pacific Timezone. • Build PB scale data pipelines that can scale to 100K+ jobs and handle PB scale data per day • Design and implement cloud-native data infrastructure on AWS using Kubernetes and Airflow • Design and develop SQL intelligence systems for query optimization, dynamic pipeline generation, and data lineage tracking • Contribute to open-source initiatives • Work with team of engineers to integrate AI into data operations

🎯 Requirements

• 8+ years of experience in data engineering, with a focus on building scalable data pipelines and systems • Strong proficiency in Python and SQL • Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake • Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs • Experience in building data pipelines using Airflow or dbt • [Optional] Solid understanding of cloud platforms, particularly AWS • [Optional] Familiarity with Kubernetes (K8s) for containerized deployments

🏖️ Benefits

• Competitive salary and equity • Extensive professional development opportunities and resources for continuous learning • Dynamic and intellectually stimulating work environment with a team of talented engineers • Opportunities to shape the direction of the company and leave a lasting impact