Compunnel, Inc.
Overview
We are seeking a highly skilled Data Engineering Specialist with expertise in Azure Cloud and DevOps practices. The ideal candidate will be passionate about building scalable data solutions, optimizing performance, and collaborating with cross-functional teams to deliver high-quality products. This role involves working on Lakehouse architectures, data orchestration, and secure data management using modern tools and frameworks. Key Responsibilities
Design, develop, and maintain Lakehouse solutions using Azure Databricks and PySpark. Optimize Spark jobs and manage large-scale data processing using RDD/DataFrame APIs. Govern and manage data access using Unity Catalog, including permissions, lineage, and audit trails. Build modular and reusable workflows using Azure Data Factory and Databricks Workflows. Implement secure, hierarchical namespace-based data lake storage using ADLS Gen2. Develop T-SQL queries, stored procedures, and manage metadata layers on Azure SQL. Work across the Azure ecosystem including networking, security, monitoring, and cost management. Write modular, testable Python code for data transformations and reusable components. Lead solution design discussions, prepare technical documentation, and mentor junior team members. Ensure adherence to coding guidelines, design patterns, and peer review processes. Collaborate with stakeholders and cross-functional teams to translate requirements into deliverables. Participate in Agile/Scrum processes and provide regular updates on progress and issues. Required Qualifications
5+ years of experience in Azure Databricks with PySpark, Databricks Workflows, Unity Catalog, and Azure Cloud. 4+ years of experience in Azure Data Factory (ADF), ADLS Gen2, and Azure SQL. 3+ years of experience in Python programming and package development. Strong understanding of Spark optimization, file formats (Parquet/Delta), and performance tuning. Experience with CI/CD tools (Azure DevOps, Terraform, ARM/Bicep). Familiarity with version control systems (Git), code quality tools (SonarQube, pylint), and testing frameworks (Pytest). Excellent communication and collaboration skills. Experience preparing HLD/LLD and architecture diagrams. Exposure to Agile tools like Jira or Azure DevOps. Preferred Qualifications
Experience with Azure Entra/AD and GitHub Actions. Orchestration experience using Airflow, Dagster, or Logic Apps. Exposure to event-driven architectures using Kafka, Azure Event Hub, or Google Cloud Pub/Sub. Experience with Change Data Capture (CDC) solutions using Debezium. Hands-on experience with Azure Synapse and Databricks Lakehouse migration projects. Experience managing cloud storage solutions on Azure and Google Cloud.
#J-18808-Ljbffr
We are seeking a highly skilled Data Engineering Specialist with expertise in Azure Cloud and DevOps practices. The ideal candidate will be passionate about building scalable data solutions, optimizing performance, and collaborating with cross-functional teams to deliver high-quality products. This role involves working on Lakehouse architectures, data orchestration, and secure data management using modern tools and frameworks. Key Responsibilities
Design, develop, and maintain Lakehouse solutions using Azure Databricks and PySpark. Optimize Spark jobs and manage large-scale data processing using RDD/DataFrame APIs. Govern and manage data access using Unity Catalog, including permissions, lineage, and audit trails. Build modular and reusable workflows using Azure Data Factory and Databricks Workflows. Implement secure, hierarchical namespace-based data lake storage using ADLS Gen2. Develop T-SQL queries, stored procedures, and manage metadata layers on Azure SQL. Work across the Azure ecosystem including networking, security, monitoring, and cost management. Write modular, testable Python code for data transformations and reusable components. Lead solution design discussions, prepare technical documentation, and mentor junior team members. Ensure adherence to coding guidelines, design patterns, and peer review processes. Collaborate with stakeholders and cross-functional teams to translate requirements into deliverables. Participate in Agile/Scrum processes and provide regular updates on progress and issues. Required Qualifications
5+ years of experience in Azure Databricks with PySpark, Databricks Workflows, Unity Catalog, and Azure Cloud. 4+ years of experience in Azure Data Factory (ADF), ADLS Gen2, and Azure SQL. 3+ years of experience in Python programming and package development. Strong understanding of Spark optimization, file formats (Parquet/Delta), and performance tuning. Experience with CI/CD tools (Azure DevOps, Terraform, ARM/Bicep). Familiarity with version control systems (Git), code quality tools (SonarQube, pylint), and testing frameworks (Pytest). Excellent communication and collaboration skills. Experience preparing HLD/LLD and architecture diagrams. Exposure to Agile tools like Jira or Azure DevOps. Preferred Qualifications
Experience with Azure Entra/AD and GitHub Actions. Orchestration experience using Airflow, Dagster, or Logic Apps. Exposure to event-driven architectures using Kafka, Azure Event Hub, or Google Cloud Pub/Sub. Experience with Change Data Capture (CDC) solutions using Debezium. Hands-on experience with Azure Synapse and Databricks Lakehouse migration projects. Experience managing cloud storage solutions on Azure and Google Cloud.
#J-18808-Ljbffr