Data Engineer, Databricks

November 24

🤠 Texas – Remote

Although this job is listed in Texas, this company may be open to hiring remote in other states.

⏰ Full Time

🟡 Mid-level

🟠 Senior

🚰 Data Engineer

🦅 H1B Visa Sponsor

Apply Now

Receive Emails with Similar Jobs

Allata

WebsiteLinkedInAll Job Openings

B2B • Consulting • Technology

Allata is a global consulting firm that helps businesses achieve digital excellence by crafting unique customer experiences, identifying revenue-generating opportunities, and improving operational efficiencies. They offer a wide range of services including strategic services, technology and cloud solutions, data and insights, and artificial intelligence development. Allata's approach is experience-led, strategy-aligned, tech-enabled, and data-driven. They work across multiple industries such as automotive, financial services, health and life sciences, high tech, logistics and transportation, among others, to drive modernization, personalization, innovation, and efficiency. With a team of former IT leaders and consulting professionals, Allata guides clients through complex digital landscapes to deliver value-based outcomes and help organizations integrate digital strategies seamlessly into their overall business strategies.

201 - 500 employees

🤝 B2B

📋 Description

• Contribute to transformative enterprise data platform projects focused on developing data pipelines and logic engines to manage ingest, staging, and multi-tier data product modeling. • Data enrichment using various OEM-specific data warehouse and data lake house platform implementations. • Full life cycle design, build, deployment and optimization data products for multiple large enterprise industry vertical-specific implementations by processing datasets through a defined series of logically conformed layers, models, and views.

🎯 Requirements

• Collaborate in defining the overall architecture of the solution. This includes knowledge of modern Enterprise Data Warehouse and Data Lakehouse architectures that implement Medallion or Lamda architectures • Design, develop, test, and deploy processing modules to implement data-driven rules using SQL, Stored Procedures, and Pyspark. • Understand and own data product engineering deliverables relative to a CI-CD pipeline and standard devops practices and principles • Build and optimize data pipelines on platforms like Databricks, SQL Server, or Azure Data Fabric. • Current knowledge of modern data tools like (Databricks,FiveTran, Data Fabric and others); Core experience with data architecture, data integrations, data warehousing, and ETL/ELT processes. • Applied experience with developing and deploying custom whl and or in session notebook scripts for custom execution across parallel executor and worker nodes. • Applied experience in SQL, Stored Procedures, and Pyspark based on area of data platform specialization. • Strong knowledge of cloud and hybrid relational database systems, such as MS SQL Server, PostgresSQL, Oracle, Azure SQL, AWS RDS, Aurora or a comparable engine. • Strong experience with batch and streaming data processing techniques and file compactization strategies. • Automation experience with CICD pipelines to support deployment and integration workflows including trunk-based development using automation services such as Azure DevOps, Jenkins, Octopus. • Advanced proficiency in Pyspark for advanced data processing tasks. • Advance proficiency in spark workflow optimization and orchestration using tools such as Asset Bundles or DAG (Directed Acyclic Graph) orchestration. • Ability to identify, troubleshoot, and resolve complex data issues effectively. • Strong teamwork, communication skills and intellectual curiosity to work collaboratively and effectively with cross-functional teams. • Commitment to delivering high-quality, accurate, and reliable data products solutions. • Willingness to embrace new tools, technologies, and methodologies. • Innovative thinker with a proactive approach to overcoming challenges.

🏖️ Benefits

• At Allata, we value differences. • Allata is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.