Compunnel, Inc.
We are seeking a Senior Data Engineer to join a cross-functional team focused on building a forecasting platform.
This role involves designing and developing highly scalable data pipelines, modernizing architecture, and ensuring data quality and accessibility.
The ideal candidate will have strong experience in data engineering, API integration, and cloud technologies, with a passion for delivering efficient and reliable data solutions.
Key Responsibilities
Develop, test, document, and support scalable data pipelines. Build and evolve data integrations, including APIs, to handle increasing data volume and complexity. Establish and follow data governance processes to ensure data availability, consistency, integrity, and security. Implement scalable solutions aligned with architectural roadmaps for data storage, reporting, and analytics. Collaborate with analytics and business teams to improve data models for BI tools and foster data-driven decision-making. Design and develop data integrations and a data quality framework; write unit/integration tests and document work. Automate deployment of distributed systems for collecting and processing streaming events. Perform data analysis to troubleshoot and resolve data-related issues. Mentor junior engineers on coding best practices and optimization. Required Qualifications
Bachelor’s degree in Computer Science, Mathematics, Statistics, or related field (or equivalent experience). Minimum 5 years of experience in analytics, data engineering, or business intelligence. Strong proficiency in Python, PySpark, and SQL. Hands-on experience with Databricks. Skilled in object-oriented programming (Python preferred). Experience developing integrations across multiple systems and APIs. Familiarity with Agile software development methodologies. Experience with cloud-based databases and Azure technologies (Azure Data Lake, ADF, Azure DevOps, Azure Functions). Expertise in SQL queries and working with large-scale, complex datasets. Experience with data warehouse technologies and creating ETL/ELT jobs. Excellent problem-solving and troubleshooting skills. Strong documentation and process-oriented approach. Preferred Qualifications
Experience with Kafka, Flink, Fivetran, and Matillion. Knowledge of Data Science and Machine Learning. Experience with Snowflake. Background in software engineering. Familiarity with designing data schemas and operating SQL/NoSQL databases.
#J-18808-Ljbffr
Develop, test, document, and support scalable data pipelines. Build and evolve data integrations, including APIs, to handle increasing data volume and complexity. Establish and follow data governance processes to ensure data availability, consistency, integrity, and security. Implement scalable solutions aligned with architectural roadmaps for data storage, reporting, and analytics. Collaborate with analytics and business teams to improve data models for BI tools and foster data-driven decision-making. Design and develop data integrations and a data quality framework; write unit/integration tests and document work. Automate deployment of distributed systems for collecting and processing streaming events. Perform data analysis to troubleshoot and resolve data-related issues. Mentor junior engineers on coding best practices and optimization. Required Qualifications
Bachelor’s degree in Computer Science, Mathematics, Statistics, or related field (or equivalent experience). Minimum 5 years of experience in analytics, data engineering, or business intelligence. Strong proficiency in Python, PySpark, and SQL. Hands-on experience with Databricks. Skilled in object-oriented programming (Python preferred). Experience developing integrations across multiple systems and APIs. Familiarity with Agile software development methodologies. Experience with cloud-based databases and Azure technologies (Azure Data Lake, ADF, Azure DevOps, Azure Functions). Expertise in SQL queries and working with large-scale, complex datasets. Experience with data warehouse technologies and creating ETL/ELT jobs. Excellent problem-solving and troubleshooting skills. Strong documentation and process-oriented approach. Preferred Qualifications
Experience with Kafka, Flink, Fivetran, and Matillion. Knowledge of Data Science and Machine Learning. Experience with Snowflake. Background in software engineering. Familiarity with designing data schemas and operating SQL/NoSQL databases.
#J-18808-Ljbffr