Insight Global
Job Description
A client is looking for hands-on Lead Data Engineer to support AI-driven forecasting initiatives within their finance domain. This role involves working with modern cloud technologies, orchestrating data workflows, and collaborating with data science teams to ensure high-quality data delivery. They will be supporting our platform, Mosaic which delivers a systemized capability for the complete planning & forecasting process by each customer, product, and business unit across the sector, as well as reporting analytics for actuals. They will use Databricks to gather, cleanse, transform and analyze large datasets in preparation for data scientists to build data models with. They will then use Synapse Analytics to integrates big data and data warehousing to query structured and unstructured data as well ata Teradata to handle enterprise-level data warehousing with SQL-based transformations. Involved in stakeholder management, guiding them with data driven insights.
Key Responsibilities:
Design and develop data transformation jobs using PySpark and Databricks.
Build and manage data orchestration workflows using Azure Data Factory or equivalent tools.
Integrate data from multiple sources and prepare it for AI model consumption.
Collaborate with data analysts and data scientists to ensure functional logic is accurately implemented.
Maintain and optimize data lakes, warehouses, and cloud-based data platforms.
We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to Human Resources Request Form (https://airtable.com/app21VjYyxLDIX0ez/shrOg4IQS1J6dRiMo) . The EEOC "Know Your Rights" Poster is available here (https://www.eeoc.gov/sites/default/files/2023-06/22-088_EEOC_KnowYourRights6.12ScreenRdr.pdf) .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Skills and Requirements
Expertise in Databricks, Apache Spark, and Python.
Strong SQL skills (Databricks SQL, DB SQL, Snowflake).
Experience with cloud platforms (Azure preferred; AWS or GCP acceptable).
Solid understanding of data lake architecture, data warehousing, and data modeling techniques. Proven track record in data engineering roles within agile environments.
Ability to manage technical deliverables and ensure data quality across the pipeline.
Familiarity with AI/ML workflows and data preparation for model training. null
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.
A client is looking for hands-on Lead Data Engineer to support AI-driven forecasting initiatives within their finance domain. This role involves working with modern cloud technologies, orchestrating data workflows, and collaborating with data science teams to ensure high-quality data delivery. They will be supporting our platform, Mosaic which delivers a systemized capability for the complete planning & forecasting process by each customer, product, and business unit across the sector, as well as reporting analytics for actuals. They will use Databricks to gather, cleanse, transform and analyze large datasets in preparation for data scientists to build data models with. They will then use Synapse Analytics to integrates big data and data warehousing to query structured and unstructured data as well ata Teradata to handle enterprise-level data warehousing with SQL-based transformations. Involved in stakeholder management, guiding them with data driven insights.
Key Responsibilities:
Design and develop data transformation jobs using PySpark and Databricks.
Build and manage data orchestration workflows using Azure Data Factory or equivalent tools.
Integrate data from multiple sources and prepare it for AI model consumption.
Collaborate with data analysts and data scientists to ensure functional logic is accurately implemented.
Maintain and optimize data lakes, warehouses, and cloud-based data platforms.
We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to Human Resources Request Form (https://airtable.com/app21VjYyxLDIX0ez/shrOg4IQS1J6dRiMo) . The EEOC "Know Your Rights" Poster is available here (https://www.eeoc.gov/sites/default/files/2023-06/22-088_EEOC_KnowYourRights6.12ScreenRdr.pdf) .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Skills and Requirements
Expertise in Databricks, Apache Spark, and Python.
Strong SQL skills (Databricks SQL, DB SQL, Snowflake).
Experience with cloud platforms (Azure preferred; AWS or GCP acceptable).
Solid understanding of data lake architecture, data warehousing, and data modeling techniques. Proven track record in data engineering roles within agile environments.
Ability to manage technical deliverables and ensure data quality across the pipeline.
Familiarity with AI/ML workflows and data preparation for model training. null
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.