Lead Data Engineer

Overview

Work with one of our U.S.-based clients.

Develop and implement data pipelines using PySpark.

Collaborate with cross-functional teams including machine learning engineers, software developers, analytics engineers, and product managers.

Translate business requirements into highly available data solutions.

Maintain and improve existing data pipelines and systems.

Requirements

• 8+ years of total experience collaborating with cross-functional teams, including machine learning engineers, software developers, analytics engineers, and product managers to translate business requirements into highly available data solutions

• 3+ years of experience developing and implementing data pipelines using PySpark

• Proven experience with SQL

• Advanced written and spoken English (B2 or C1/C2)

• Databricks Certification (nice to have)

• Experience providing practical direction within Azure native services, implementing data migration and data processing using Azure services such as ADLS, Azure Data Factory, Synapse, Azure SQL DB, Fabric

• Experience with Azure Event Hub, IoT Hub, Azure Stream Analytics, Azure Analysis Services, Cosmos DB

• Knowledge of SAP HANA

• Intermediate knowledge of Power BI

• Knowledge of DevOps and CI/CD deployments, cloud migration methodologies and processes

ATS Optimization Keywords

Below are skills and terms extracted directly from this job posting to improve Applicant Tracking System (ATS) visibility. This feature helps candidates tailor their applications more effectively.

Hard Skills

• PySpark

• SQL

• data pipelines

• data migration

• data processing

• Power BI

• DevOps

• CI/CD

• Azure services

• Cloud migration methodologies

Soft Skills

• Collaboration

• Communication

• Translation of business requirements

• Practical direction

Certifications & Qualifications

• Databricks Certification