Databricks Engineer

🔥 23 hours ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Quantiphi

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 2013

🤖 Artificial Intelligence

🏢 Enterprise

📚 Education

💰 Series A on 2019-12

Artificial Intelligence • Enterprise • Education

Quantiphi is a leading AI-first digital engineering company that leverages a decade of industry expertise to empower businesses through scalable, secure, and adaptable AI solutions. By integrating cutting-edge technology with real-world applications, Quantiphi transforms organizations across various sectors including healthcare, finance, education, and retail. Their services span AI applications, data analytics, cloud infrastructure modernization, and custom AI implementations. Quantiphi partners with technology giants like AWS, Google Cloud, NVIDIA, and others to drive AI adoption and deliver transformational opportunities for enterprises.

📋 Description

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark. • Build and optimize ETL/ELT workflows to ingest, transform, and curate large datasets. • Implement data solutions following Medallion Architecture (Bronze, Silver, Gold). • Develop high-performance PySpark and Spark SQL transformations. • Optimize Delta Lake tables using partitioning, optimization, vacuuming, and time travel capabilities. • Orchestrate and monitor Databricks Workflows and scheduled jobs. • Collaborate with analytics, engineering, and product teams to deliver reliable data products. • Apply software engineering best practices, including version control, automated testing, and CI/CD. • Utilize AI-assisted development tools for code generation, debugging, documentation, and productivity while validating outputs to ensure quality and accuracy. • Contribute to data quality, performance tuning, and operational excellence across the platform.

🎯 Requirements

• 7+ years of experience in data engineering • Bachelor's degree in Computer Science, Software Engineering, or a related field (or equivalent work experience) • Hands-on experience with Databricks Workspaces, Clusters, Jobs, and Unity Catalog • Strong knowledge of Delta Lake, including ACID transactions, Time Travel, OPTIMIZE, and VACUUM • Experience building and managing Databricks Workflows • Strong experience with PySpark, Spark SQL, and Scala • Expertise designing and building ETL/ELT pipelines • Experience implementing Medallion Architecture (Bronze/Silver/Gold) • Programming Languages: Python (Advanced), SQL (Advanced), Scala (Strong working knowledge) • Experience with at least one of the following cloud platform: AWS (Amazon S3, AWS Glue, Redshift), Azure (Azure Data Lake Storage Gen2 (ADLS Gen2),Azure Data Factory (ADF), Azure DevOps) • Experience using AI-powered developer tools for coding, debugging, testing, and documentation • Understanding of responsible AI usage, code validation, and quality assurance. • Git version control, CI/CD pipelines, Unit testing and automated validation for data transformations, Strong software engineering and code quality practices.

🏖️ Benefits

• Join one of the world’s fastest-growing AI-first digital engineering companies and make a real impact at scale. • Lead and collaborate with a high-energy team of talented, driven individuals solving complex, meaningful challenges. • Work with Fortune 500 companies and disruptive innovators in a research-driven environment with 60+ patents. • Stay ahead of the curve by gaining hands-on experience with cutting-edge AI, ML, data, and cloud technologies while continuously upskilling.