Vuesol
1SHLJP00005685 IT Data Engineering JG3|SC|ERM
Houston, Tx
Role Overview As a Data Engineering Technical Lead, you will design and deliver robust data platforms using Databricks (Delta Live Tables, Structured Streaming, PySpark), Python, and SQL on Microsoft Azure. You will work with to build, optimise data pipelines for optimal performance and cost, ensure governance and quality, and collaborate with cross-functional teams to unlock strategic insights from trading data. You will be a key contributor to Shell's digital transformation, enabling smarter trading decisions and supporting the energy transition through data-driven innovation. Key Responsibilities Technical Leadership
Architect, Design and implement scalable Azure Data Lakehouse solutions using ADF, ADLS, Synapse, Stream Analytics, and Databricks. Lead the development of real-time data pipelines using Databricks DLT and Structured Streaming for high-volume data ingestion and transformation. Optimise pipeline performance and cloud resource usage to ensure cost-effective operations. Leverage Unity Catalog for metadata management, data lineage, and governance. Domain Expertise
Integrate and manage data from ETRM systems to support Crude & Products trading analytics. Apply functional knowledge of commodity trading (Oil & Gas) to align data solutions with business needs. Collaborate with trading desks, analysts, and IT teams to deliver actionable insights. Governance & Quality Establish and enforce data governance frameworks to ensure compliance, security, and data quality. Monitor pipeline performance and proactively resolve production issues. Stakeholder Engagement
Communicate effectively with business and technical stakeholders, translating complex technical concepts into clear business goal/value. Lead executive-level discussions and presentations to align data initiatives with strategic priorities. Team & Capability Building Mentor and lead a high-performing team of data engineers and analysts. Drive continuous learning and capability development across Azure, Databricks, and streaming technologies. Required Skills & Experience • Technical Expertise:
Deep experience with Databricks DLT, Structured Streaming, PySpark, Python, and SQL. Strong knowledge of Azure data services (ADF, ADLS, Synapse, Stream Analytics). Proven ability to optimise Databricks for high-volume, real-time data processing. Experience with ETRM platforms and trading data workflows. Leadership & Communication: Strong stakeholder management and executive communication skills. Demonstrated ownership, accountability, and passion for delivering impactful solutions. Ability to lead cross-functional teams and drive strategic data initiatives. Education & Certification: Bachelor's or Master's degree in IT or related discipline. 12+ years of experience in data engineering and analytics. Certifications in Azure Data Engineering and Databricks (preferred). Nice to Have:
Experience with Kafka, Hadoop, Airflow, and PowerBI/QlikSense. Familiarity with AI/ML, NLP, and GenAI concepts. Exposure to Agile/Kanban methodologies and CI/CD practices.
Role Overview As a Data Engineering Technical Lead, you will design and deliver robust data platforms using Databricks (Delta Live Tables, Structured Streaming, PySpark), Python, and SQL on Microsoft Azure. You will work with to build, optimise data pipelines for optimal performance and cost, ensure governance and quality, and collaborate with cross-functional teams to unlock strategic insights from trading data. You will be a key contributor to Shell's digital transformation, enabling smarter trading decisions and supporting the energy transition through data-driven innovation. Key Responsibilities Technical Leadership
Architect, Design and implement scalable Azure Data Lakehouse solutions using ADF, ADLS, Synapse, Stream Analytics, and Databricks. Lead the development of real-time data pipelines using Databricks DLT and Structured Streaming for high-volume data ingestion and transformation. Optimise pipeline performance and cloud resource usage to ensure cost-effective operations. Leverage Unity Catalog for metadata management, data lineage, and governance. Domain Expertise
Integrate and manage data from ETRM systems to support Crude & Products trading analytics. Apply functional knowledge of commodity trading (Oil & Gas) to align data solutions with business needs. Collaborate with trading desks, analysts, and IT teams to deliver actionable insights. Governance & Quality Establish and enforce data governance frameworks to ensure compliance, security, and data quality. Monitor pipeline performance and proactively resolve production issues. Stakeholder Engagement
Communicate effectively with business and technical stakeholders, translating complex technical concepts into clear business goal/value. Lead executive-level discussions and presentations to align data initiatives with strategic priorities. Team & Capability Building Mentor and lead a high-performing team of data engineers and analysts. Drive continuous learning and capability development across Azure, Databricks, and streaming technologies. Required Skills & Experience • Technical Expertise:
Deep experience with Databricks DLT, Structured Streaming, PySpark, Python, and SQL. Strong knowledge of Azure data services (ADF, ADLS, Synapse, Stream Analytics). Proven ability to optimise Databricks for high-volume, real-time data processing. Experience with ETRM platforms and trading data workflows. Leadership & Communication: Strong stakeholder management and executive communication skills. Demonstrated ownership, accountability, and passion for delivering impactful solutions. Ability to lead cross-functional teams and drive strategic data initiatives. Education & Certification: Bachelor's or Master's degree in IT or related discipline. 12+ years of experience in data engineering and analytics. Certifications in Azure Data Engineering and Databricks (preferred). Nice to Have:
Experience with Kafka, Hadoop, Airflow, and PowerBI/QlikSense. Familiarity with AI/ML, NLP, and GenAI concepts. Exposure to Agile/Kanban methodologies and CI/CD practices.