Compunnel, Inc.
Principal Software Engineer Data Platform Engineering
Compunnel, Inc., Raleigh, North Carolina, United States, 27601
Principal Software Engineer Data Platform Engineering
08/19/2025 Full Time Active Job Description:
We are enabling the transformation of how energy and water are managed. As part of this mission, we are investing in cloud-native data platform technologies that power intelligent analytics at scale — particularly for large IoT data workloads. We are looking for a Principal Software Engineer - Data Platform Engineering with a strong background in data engineering to help design, build, and scale our next-generation data platform and services. This role will focus on developing performant, scalable, and secure data analytics and data pipelines using Python, Apache Spark, Databricks, and Azure-native technologies. The ideal candidate is passionate about working with large-scale datasets, understands Spark query execution plans, and has experience deploying infrastructure using Terraform and container technologies like Docker. Duties & Responsibilities: Design and implement scalable data pipelines for high-volume IoT telemetry data using PySpark, Spark SQL, and Databricks Analyze and optimize Spark jobs using execution plans, caching strategies, and memory tuning Develop Python-based micro-services and reusable libraries to support data processing workflows Work in a DevOps-driven Agile/Scrum environment to deliver high-quality code in fast iterations Collaborate with product managers, data scientists, and software engineers to define system requirements and data integration needs Build and manage infrastructure as code using Terraform to deploy workloads on Azure Create CI/CD pipelines for deploying and testing data workflows and services Package and deploy applications in Docker containers and manage runtime environments Apply strong testing practices, including unit tests, integration tests, and test automation Participate in architectural discussions and technical design reviews Ensure secure, reliable, and cost-effective operation of data workloads on cloud platforms Required Skills & Experience: Minimum of 10 years of software engineering experience, with at least 3 years focused on data platform or data engineering roles Proficient in Python, with strong knowledge of PySpark and Apache Spark Hands-on experience with Databricks and Spark SQL Strong understanding of Spark performance tuning and query execution plans Experience writing infrastructure-as-code using Terraform Working knowledge of Docker and containerized deployment Experience with CI/CD pipelines, version control (e.g., Git), and automated testing Experience with Agile/Scrum methodologies Bachelor’s degree in Computer Science, Engineering, or related field — or equivalent experience Preferred Skills & Experience: Experience with large-scale IoT or telemetry data processing Familiarity with Kafka, Event Hubs, or other streaming platforms Familiarity with cost optimization in cloud-based data processing Experience with monitoring and observability tools like Prometheus, Grafana, or Datadog Email ID * This field is required Please enter valid emailId. Cell phone * This field is required Please enter valid cell phone. First Name * This field is required Please enter valid first name. Last Name * This field is required Please enter valid last name.
#J-18808-Ljbffr
08/19/2025 Full Time Active Job Description:
We are enabling the transformation of how energy and water are managed. As part of this mission, we are investing in cloud-native data platform technologies that power intelligent analytics at scale — particularly for large IoT data workloads. We are looking for a Principal Software Engineer - Data Platform Engineering with a strong background in data engineering to help design, build, and scale our next-generation data platform and services. This role will focus on developing performant, scalable, and secure data analytics and data pipelines using Python, Apache Spark, Databricks, and Azure-native technologies. The ideal candidate is passionate about working with large-scale datasets, understands Spark query execution plans, and has experience deploying infrastructure using Terraform and container technologies like Docker. Duties & Responsibilities: Design and implement scalable data pipelines for high-volume IoT telemetry data using PySpark, Spark SQL, and Databricks Analyze and optimize Spark jobs using execution plans, caching strategies, and memory tuning Develop Python-based micro-services and reusable libraries to support data processing workflows Work in a DevOps-driven Agile/Scrum environment to deliver high-quality code in fast iterations Collaborate with product managers, data scientists, and software engineers to define system requirements and data integration needs Build and manage infrastructure as code using Terraform to deploy workloads on Azure Create CI/CD pipelines for deploying and testing data workflows and services Package and deploy applications in Docker containers and manage runtime environments Apply strong testing practices, including unit tests, integration tests, and test automation Participate in architectural discussions and technical design reviews Ensure secure, reliable, and cost-effective operation of data workloads on cloud platforms Required Skills & Experience: Minimum of 10 years of software engineering experience, with at least 3 years focused on data platform or data engineering roles Proficient in Python, with strong knowledge of PySpark and Apache Spark Hands-on experience with Databricks and Spark SQL Strong understanding of Spark performance tuning and query execution plans Experience writing infrastructure-as-code using Terraform Working knowledge of Docker and containerized deployment Experience with CI/CD pipelines, version control (e.g., Git), and automated testing Experience with Agile/Scrum methodologies Bachelor’s degree in Computer Science, Engineering, or related field — or equivalent experience Preferred Skills & Experience: Experience with large-scale IoT or telemetry data processing Familiarity with Kafka, Event Hubs, or other streaming platforms Familiarity with cost optimization in cloud-based data processing Experience with monitoring and observability tools like Prometheus, Grafana, or Datadog Email ID * This field is required Please enter valid emailId. Cell phone * This field is required Please enter valid cell phone. First Name * This field is required Please enter valid first name. Last Name * This field is required Please enter valid last name.
#J-18808-Ljbffr