Senior Data Engineer (Remote)

• Own data pipeline development: Design, develop, and maintain reliable ETL/ELT pipelines for ingesting, transforming, and delivering large-scale datasets across multiple platforms and sources.
• Lead technical design: Define processing logic, data models, integration approaches, and workflow dependencies, considering scalability, maintainability, security, and cost.
• Develop and optimize data processing: Write efficient SQL, Hive queries, and Python code. Apply distributed processing frameworks such as Spark where appropriate.
• Build and manage AWS workflows: Develop and support data solutions using Amazon S3, Athena, EC2, and EMR, including monitoring and resource optimization.
• Manage workflow orchestration: Build and maintain automated workflows using Apache Airflow or comparable orchestration tools, including scheduling, dependencies, retries, alerting, and recovery.
• Ensure production reliability: Monitor pipeline performance and data freshness, investigate failures, perform root cause analysis, and implement improvements that prevent recurring issues.
• Maintain data quality and governance: Establish validation checks, reconciliation processes, lineage documentation, and appropriate access controls throughout the data lifecycle.
• Lead and mentor a small team: Guide 2-3 engineers, break requirements into actionable tasks, review code and designs, support troubleshooting, and track delivery commitments.
• C ollaborate with stakeholders: Work closely with analytics, product, engineering, and data science teams to clarify requirements, agree on priorities, communicate progress, and manage dependencies.
• Maintain technical documentation: Create and update solution designs, data flow diagrams, data dictionaries, operating procedures, and support runbooks using Confluence or similar platforms.
• Apply AI to engineering workflows: Identify practical opportunities to use AI for development, documentation, troubleshooting, and automation. Build simple AI agents and validate their outputs before operational use.
• Support agile delivery: Manage work through Jira, contribute to sprint planning and technical reviews, and coordinate across regions and work schedules as required.

Requirements

• 5+ years of professional experience in data engineering, with demonstrated ownership of production data pipelines and large-scale data processing.
• Strong SQL expertise, including complex joins, window functions, transformations, query tuning, and troubleshooting.
• Strong working knowledge of Hive and the Hadoop ecosystem, including partitioning, storage formats, and distributed data processing concepts.
• Proficiency in Python for data processing, automation, integration, and maintainable production code.
• Hands-on experience with Amazon S3, Athena, EC2, and EMR.
• Strong experience with Apache Airflow or another ETL/workflow orchestration tool, with Airflow preferred.
• Experience implementing data quality checks, monitoring, error handling, and operational support processes.
• Ability to translate business requirements into technical designs and deliver solutions with minimal supervision.
• Strong documentation and communication skills, including the ability to explain technical issues clearly to business and technical stakeholders.
• Working experience with Jira and Confluence for task tracking, delivery coordination, and documentation.
• Practical understanding of AI capabilities and limitations, along with hands-on experience creating simple AI agents or AI-assisted automations.
• Experience with version control, code reviews, and collaborative development practices.

Preferred Qualifications

• Hands-on experience with Apache Spark using PySpark or Scala.
• Experience with Snowflake, including data transformation, query optimization, and integration with cloud data pipelines.
• Practical experience using OpenAI Codex for engineering tasks and Snowflake Cortex for AI-enabled data workflows.
• Previous experience mentoring engineers, leading technical delivery, or managing a small team. Formal people management experience is an added advantage.
• Experience supporting marketing technology, audience data, campaign reporting, or