data engineer for ML modeling teams

Описание: EPAM builds enterprise software products, open source solutions, accelerators, and data products for ML modeling teams.

Задачи

• Support a team of data engineers building pipelines used by MLOps and ML Engineers on ML modeling teams;
• Develop, optimize, and maintain data transformation pipelines in Databricks using PySpark;
• Work with data stored in ADLS Gen2 and SAP HANA Data Lake, primarily in Delta/Parquet format;
• Implement and maintain data quality checks, including schema validation, deduplication, enrichment, and tagging;
• Communicate with stakeholders to understand business processes and model input data;
• Tune performance for large-scale datasets to ensure efficient processing.

Требования

• 3+ Years of experience in data engineering with proficiency in Python, PySpark, and Databricks;
• Hands-on experience with Delta Lake and Azure data lake technologies;
• Familiarity with software version control tools such as GitHub and Git;
• Experience with CI/CD frameworks such as GitHub Actions;
• Knowledge of data lake technologies and large-scale dataset performance tuning;
• Proficiency in English at B2+ level;
• Nice to have: familiarity with at least one other programming or scripting language, such as Java, SQL, or Scala.

Условия

Remote work in Kazakhstan.