Gunvor Group
Senior Data Engineer page is loaded## Senior Data Engineerlocations:
Houstonposted on:
Posted Todayjob requisition id:
JR102486**Job Title:**Senior Data Engineer**Contract Type:******Time Type:******Job Description:**Main Responsibilities**Data Engineering Leadership*** Contribute to the definition of the target data architecture (lake/lakehouse, streaming, event-driven) and technology choices for high-volume time-series, market, and transactional data.* Build and operate end-to-end pipelines (ingest → quality → transform → serve) with strong SLAs/SLOs, lineage, and observability.* Establish coding, testing, CI/CD, and infrastructure-as-code standards; drive adoption of data governance, cataloging, and access controls.* Own capacity, performance, reliability, and cost efficiency of data platforms across environments (dev/test/prod).* Ensure data quality, lineage, and observability across all layers of the data stack.**Strategic Collaboration & Business Alignment*** Partner with trading desks, quantitative teams, and risk functions to translate business needs into data solutions that enhance decision-making and operational efficiency.* Act as a senior liaison between engineering and business stakeholders, ensuring alignment on data priorities and delivery timelines.* Prioritize a value-based backlog (e.g., faster close/settlement, improved forecast accuracy, reduced balancing penalties) and measure business impact.* Align data models and domain ownership with business processes (bids/offers, nominations, positions, exposures,outages).* Coordinate with Cybersecurity, Compliance, and Legal on sector-specific controls (e.g., REMIT/NERC-CIP considerations, data retention, segregation).**Innovation & Product Development*** Incubate and industrialize data products: curated marts, feature stores, real-time decision APIs, and event streams for forecasting and optimization.* Introduce modern patterns (CDC, schema evolution, Delta/Iceberg, stream–batch unification) to improve freshness and resilience.* Evaluate and integrate external data (weather, fundamentals, congestion, capacity postings), internal and external vendor systems (ETRM) safely and at scale.* Collaborate with quantitative analysts to productionize ML pipelines (forecasting load/renewables, anomaly detection, etc.. ) with monitoring and rollback.**Mentorship & Technical Oversight*** Lead incident reviews and architectural forums; provide pragmatic guidance on trade-offs (latency vs. cost, simplicity vs. flexibility).* Develop growth paths and learning plans focused on energy domain fluency and modern data engineering practices.**Operational Excellence*** Implement robust monitoring/alerting, runbooks, and on-call rotations; drive MTTR down and availability up for critical data services.* Enforce data quality contracts (SLAs/SLOs), lineage, and reconciliation for market submissions, settlements, and reporting.* Optimize cloud spend and storage/compute footprints; plan capacity for market events and seasonal peaks.* Ensure security and compliance by design: least-privilege access, secrets management, encryption, auditability, and disaster recovery testing.Profile* Master’s or Bachelor’s degree in Computer Science, Data Engineering, Applied Mathematics, or a related technical field.* 5+ years of experience in data engineering, with at least 3 years in a senior role.* Proven experience in the energy trading sector, ideally with exposure to Natural Gas and Power markets, balancing mechanisms, and regulatory frameworks (e.g., REMIT, EMIR).* Azure: ADLS Gen2, Event Hubs, Synapse Analytics, Azure Databricks (Spark), Azure Functions, Azure Data* Factory/Databricks Workflows, Key Vault, Azure Monitoring/Log Analytics; IaC with Terraform/Bicep; CI/CD with Azure DevOps or GitHub Actions.* Snowflake (on Azure or multi-cloud): Warehousing design, Streams & Tasks, Snowpipe/Snowpipe Streaming, Time* Travel & Fail-safe, RBAC & row/column security, external tables over ADLS, performance tuning & cost governance.* Programming & Engineering Practices: Strong OOP in Python and Java/Scala; SDLC leadership, DevOps mindset, TDD/BDD, code reviews, automated testing (unit/integration/contract), packaging and dependency management, API design (REST/gRPC).* Web & Data Acquisition: Robust web scraping and ingestion with Scrapy, Requests, Playwright/Selenium; scheduling, retries/exponential backoff, change-data capture; ethical/legal collection practices (robots.txt, terms).* Orchestration & Quality: Airflow/ADF/Databricks Jobs, data contracts, Great Expectations (or similar), lineage/catalog (e.g., Purview), metrics/observability (Prometheus/Grafana/Application Insights).* Dataframe oriented programming: pandas, spark/snowpark dataframes, SQL data transformation* Additional Skills* Designing low-latency pipelines for sub-second to minute-level telemetry, weather and market data; tuning Spark* Structured Streaming/Flink/Kafka Streams.* Quality & Reconciliation for telemetry and market submissions (gap fill, resampling, deduplication, anomaly detection, schema evolution).* Serving Patterns: time-series stores and query layers (e.g., Delta Lake over ADLS, Iceberg, materialized views in Snowflake), APIs and event streams for downstream consumption.* English (fluent), any additional language is an assetIf you think the open position you see is right for you, we encourage you to apply!***Our people make all the difference in our success.***locations:
Houstonposted on:
Posted 17 Days Ago #J-18808-Ljbffr
Houstonposted on:
Posted Todayjob requisition id:
JR102486**Job Title:**Senior Data Engineer**Contract Type:******Time Type:******Job Description:**Main Responsibilities**Data Engineering Leadership*** Contribute to the definition of the target data architecture (lake/lakehouse, streaming, event-driven) and technology choices for high-volume time-series, market, and transactional data.* Build and operate end-to-end pipelines (ingest → quality → transform → serve) with strong SLAs/SLOs, lineage, and observability.* Establish coding, testing, CI/CD, and infrastructure-as-code standards; drive adoption of data governance, cataloging, and access controls.* Own capacity, performance, reliability, and cost efficiency of data platforms across environments (dev/test/prod).* Ensure data quality, lineage, and observability across all layers of the data stack.**Strategic Collaboration & Business Alignment*** Partner with trading desks, quantitative teams, and risk functions to translate business needs into data solutions that enhance decision-making and operational efficiency.* Act as a senior liaison between engineering and business stakeholders, ensuring alignment on data priorities and delivery timelines.* Prioritize a value-based backlog (e.g., faster close/settlement, improved forecast accuracy, reduced balancing penalties) and measure business impact.* Align data models and domain ownership with business processes (bids/offers, nominations, positions, exposures,outages).* Coordinate with Cybersecurity, Compliance, and Legal on sector-specific controls (e.g., REMIT/NERC-CIP considerations, data retention, segregation).**Innovation & Product Development*** Incubate and industrialize data products: curated marts, feature stores, real-time decision APIs, and event streams for forecasting and optimization.* Introduce modern patterns (CDC, schema evolution, Delta/Iceberg, stream–batch unification) to improve freshness and resilience.* Evaluate and integrate external data (weather, fundamentals, congestion, capacity postings), internal and external vendor systems (ETRM) safely and at scale.* Collaborate with quantitative analysts to productionize ML pipelines (forecasting load/renewables, anomaly detection, etc.. ) with monitoring and rollback.**Mentorship & Technical Oversight*** Lead incident reviews and architectural forums; provide pragmatic guidance on trade-offs (latency vs. cost, simplicity vs. flexibility).* Develop growth paths and learning plans focused on energy domain fluency and modern data engineering practices.**Operational Excellence*** Implement robust monitoring/alerting, runbooks, and on-call rotations; drive MTTR down and availability up for critical data services.* Enforce data quality contracts (SLAs/SLOs), lineage, and reconciliation for market submissions, settlements, and reporting.* Optimize cloud spend and storage/compute footprints; plan capacity for market events and seasonal peaks.* Ensure security and compliance by design: least-privilege access, secrets management, encryption, auditability, and disaster recovery testing.Profile* Master’s or Bachelor’s degree in Computer Science, Data Engineering, Applied Mathematics, or a related technical field.* 5+ years of experience in data engineering, with at least 3 years in a senior role.* Proven experience in the energy trading sector, ideally with exposure to Natural Gas and Power markets, balancing mechanisms, and regulatory frameworks (e.g., REMIT, EMIR).* Azure: ADLS Gen2, Event Hubs, Synapse Analytics, Azure Databricks (Spark), Azure Functions, Azure Data* Factory/Databricks Workflows, Key Vault, Azure Monitoring/Log Analytics; IaC with Terraform/Bicep; CI/CD with Azure DevOps or GitHub Actions.* Snowflake (on Azure or multi-cloud): Warehousing design, Streams & Tasks, Snowpipe/Snowpipe Streaming, Time* Travel & Fail-safe, RBAC & row/column security, external tables over ADLS, performance tuning & cost governance.* Programming & Engineering Practices: Strong OOP in Python and Java/Scala; SDLC leadership, DevOps mindset, TDD/BDD, code reviews, automated testing (unit/integration/contract), packaging and dependency management, API design (REST/gRPC).* Web & Data Acquisition: Robust web scraping and ingestion with Scrapy, Requests, Playwright/Selenium; scheduling, retries/exponential backoff, change-data capture; ethical/legal collection practices (robots.txt, terms).* Orchestration & Quality: Airflow/ADF/Databricks Jobs, data contracts, Great Expectations (or similar), lineage/catalog (e.g., Purview), metrics/observability (Prometheus/Grafana/Application Insights).* Dataframe oriented programming: pandas, spark/snowpark dataframes, SQL data transformation* Additional Skills* Designing low-latency pipelines for sub-second to minute-level telemetry, weather and market data; tuning Spark* Structured Streaming/Flink/Kafka Streams.* Quality & Reconciliation for telemetry and market submissions (gap fill, resampling, deduplication, anomaly detection, schema evolution).* Serving Patterns: time-series stores and query layers (e.g., Delta Lake over ADLS, Iceberg, materialized views in Snowflake), APIs and event streams for downstream consumption.* English (fluent), any additional language is an assetIf you think the open position you see is right for you, we encourage you to apply!***Our people make all the difference in our success.***locations:
Houstonposted on:
Posted 17 Days Ago #J-18808-Ljbffr