Data Engineer

About the Role

We are looking for a skilled Data Engineer with hands-on experience in building and managing scalable data pipelines across cloud platforms. The ideal candidate should have strong expertise in ETL frameworks and exposure to multiple cloud environments.

Key Responsibilities

• Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines (batch and near real-time) across AWS, Azure, and GCP environments (AWS preferred).
• Implement robust data ingestion patterns including CDC, API-based, file-based, and streaming integrations ensuring data quality, reliability, and performance.
• Develop and maintain cloud-native data solutions using MySQL/SQL Server, Azure Data Factory, MFTS, GCP BigQuery, and GCS (preferred: AWS Glue, Redshift, EC2, RDS).
• Build and manage workflow orchestration using Apache Airflow (or equivalent tools) for scheduling, monitoring, and dependency management.
• Work on data ingestion, transformation, and integration across multiple enterprise systems and platforms.
• Develop CI/CD pipelines for data infrastructure and ETL code using Git-based workflows and Infrastructure-as-Code (IaC) best practices.
• Collaborate with cross-functional teams to understand data requirements and deliver scalable and efficient data solutions.
• Work closely with security and compliance teams to implement encryption, IAM least-privilege access, network segmentation (VPN/VPC), and governance controls.
• Build monitoring, alerting, and auditing mechanisms for data pipelines including latency tracking, failure monitoring, retries, DLQ, and data quality metrics.
• Optimize data storage, partitioning, and query performance for analytics workloads in BigQuery and Redshift environments.
• Troubleshoot production issues, perform root-cause analysis, and implement corrective and preventive actions.
• Produce and maintain technical documentation including architecture diagrams, runbooks, and data contracts while supporting onboarding and mentoring activities.
• Provide ongoing sustainment and operational support for production data platforms and pipelines.

Required Qualifications

• 5–8 years of total experience in data engineering with at least 3–5 years specifically building cloud-native data pipelines
• Strong experience with Azure Cloud services, especially Azure Data Factory (ADF), data ingestion, and transformation workflows
• Hands-on expertise with GCP services such as BigQuery, Google Cloud Storage (GCS), and cloud-native analytics solutions
• Proficiency in Apache Airflow for workflow orchestration, scheduling, monitoring, and dependency management
• Experience implementing batch and near real-time data pipelines using CDC, API, file-based, and streaming ingestion patterns
• Strong understanding of data engineering concepts, data modeling, pipeline optimization, and performance tuning
• Experience with CI/CD pipelines, Git-based workflows, and Infrastructure-as-Code (IaC) practices
• Proficiency in SQL and scripting languages such as Python for data processing and automation
• Experience with monitoring, troubleshooting, and production support for enterprise data platforms

Preferred Qualifications

• Exposure to AWS services such as Glue, Redshift, EC2, and RDS
• Experience working in multi-cloud enterprise environments (Azure, GCP, AWS)
• Knowledge of data governance, IAM, encryption, and security best practices
• Experience handling large-scale distributed and analytics data systems
• Strong problem-solving skills and experience working in Agile environments