The Pittsburgh Paints Company
Data Engineer - Azure Databricks
The Pittsburgh Paints Company, Cranberry Township, Pennsylvania, United States
Description
We are looking for a skilled Data Engineer – Azure Databricks to join our Data & Analytics team in the US or Canada. The ideal candidate has extensive experience in data engineering, working with Databricks and Azure cloud services, designing and implementing scalable Azure Cloud data solutions. You will play a crucial role in developing, optimizing and maintaining data pipelines and architectures, ensuring the quality and availability of critical data through our Data Lake platform. You can work independently and as part of a team, with a strong desire to help shape our Data & Analytics platforms and strategy.
Key Responsibilities
Design, develop, and deploy robust and scalable data pipelines using Databricks, integrating data from various sources (databases, APIs, streaming platforms, etc.). Transform and clean data using SQL or Scala/Python and load data into target systems (data lakes, data warehouses, etc.). Optimize pipeline performance for greater efficiency and cost-effectiveness.
Use all capabilities of the Databricks platform, including Databricks SQL, Delta Lake, Databricks Runtime, and Databricks Workflows, to orchestrate complex data workflows and ensure data quality and pipeline reliability.
Contribute to the design and implementation of robust data models in the data lake environment, ensuring data consistency, integrity and accessibility for various use cases.
Implement rigorous data quality controls and validation procedures throughout the data pipeline to ensure high accuracy and reliability. Comply with data governance policies and best practices.
Seamlessly integrate Databricks with Azure cloud services for storage, compute and security, leveraging services like Azure Data Lake Storage.
Stay up to date on the latest Databricks features, cloud technologies and data engineering trends to continuously improve our solutions and deliver innovative results for our business.
Work closely with data analysts, IT Front Line and business stakeholders to understand data requirements and provide technical expertise on Databricks and data engineering best practices.
Performance Achievements
Proactively build strong working relationships with colleagues and external consultants, focusing on Azure Data Lake solutions.
Contribute to projects that ensure a stable and effective model around Azure Data Lake solutions after a brief acclimation period.
Familiarize with the specific solutions in place after the incubation period.
Key Competence Requirements
Bachelor's / Master’s degree in Computer Science, (Business) Engineering, or a specialized master’s in Machine Learning or AI.
7+ years of experience in data engineering with demonstrated expertise in Databricks and a focus on Azure Cloud Analytics Services. Proficiency in Azure Data Factory and Azure SQL Database acquired with Azure Data Lake. 5+ years of experience with Azure DevOps (git) with solid experience with CI/CD pipelines. Understanding of data governance, quality and security best practices.
Hands‑on experience with Azure cloud platform with a good understanding of cloud data services and infrastructure.
Strong programming skills in SQL, Python or Scala with experience in data manipulation libraries (e.g., PySpark, Spark SQL).
Solid understanding of data warehousing principles, ETL processes, data modeling techniques and database systems.
Advanced SQL skills for querying, transforming and analyzing data.
Excellent communication and interpersonal skills, with the ability to collaborate effectively with technical and non‑technical stakeholders.
Strong analytical and problem‑solving abilities, with a proactive approach to identifying and resolving data challenges.
Excellent communication skills in English.
Preferred Skills
Databricks certification (e.g. Data Engineer Professional)
Azure certification (e.g. Microsoft Certified: Azure Data Engineer Associate) is a plus.
Experience with Azure Logic Apps, Azure Functions, and API Management.
AI/ML experience (ability to develop predictive systems and deploy AI/ML models using Databricks & Microsoft tools)
Knowledge of Power BI or other data visualization tools.
Soft Skills
Excellent analytical and problem‑solving skills.
Highly organized, detail‑oriented and able to work independently as well as collaboratively.
Applicants must be authorized to work in the U.S. without employment‑based visa sponsorship (now or in the future). This includes H‑1B, L‑1, TN, O‑1, E‑3, H‑1B1, F‑1, J‑1, OPT, CPT or any other employment‑based visas.
#J-18808-Ljbffr
Key Responsibilities
Design, develop, and deploy robust and scalable data pipelines using Databricks, integrating data from various sources (databases, APIs, streaming platforms, etc.). Transform and clean data using SQL or Scala/Python and load data into target systems (data lakes, data warehouses, etc.). Optimize pipeline performance for greater efficiency and cost-effectiveness.
Use all capabilities of the Databricks platform, including Databricks SQL, Delta Lake, Databricks Runtime, and Databricks Workflows, to orchestrate complex data workflows and ensure data quality and pipeline reliability.
Contribute to the design and implementation of robust data models in the data lake environment, ensuring data consistency, integrity and accessibility for various use cases.
Implement rigorous data quality controls and validation procedures throughout the data pipeline to ensure high accuracy and reliability. Comply with data governance policies and best practices.
Seamlessly integrate Databricks with Azure cloud services for storage, compute and security, leveraging services like Azure Data Lake Storage.
Stay up to date on the latest Databricks features, cloud technologies and data engineering trends to continuously improve our solutions and deliver innovative results for our business.
Work closely with data analysts, IT Front Line and business stakeholders to understand data requirements and provide technical expertise on Databricks and data engineering best practices.
Performance Achievements
Proactively build strong working relationships with colleagues and external consultants, focusing on Azure Data Lake solutions.
Contribute to projects that ensure a stable and effective model around Azure Data Lake solutions after a brief acclimation period.
Familiarize with the specific solutions in place after the incubation period.
Key Competence Requirements
Bachelor's / Master’s degree in Computer Science, (Business) Engineering, or a specialized master’s in Machine Learning or AI.
7+ years of experience in data engineering with demonstrated expertise in Databricks and a focus on Azure Cloud Analytics Services. Proficiency in Azure Data Factory and Azure SQL Database acquired with Azure Data Lake. 5+ years of experience with Azure DevOps (git) with solid experience with CI/CD pipelines. Understanding of data governance, quality and security best practices.
Hands‑on experience with Azure cloud platform with a good understanding of cloud data services and infrastructure.
Strong programming skills in SQL, Python or Scala with experience in data manipulation libraries (e.g., PySpark, Spark SQL).
Solid understanding of data warehousing principles, ETL processes, data modeling techniques and database systems.
Advanced SQL skills for querying, transforming and analyzing data.
Excellent communication and interpersonal skills, with the ability to collaborate effectively with technical and non‑technical stakeholders.
Strong analytical and problem‑solving abilities, with a proactive approach to identifying and resolving data challenges.
Excellent communication skills in English.
Preferred Skills
Databricks certification (e.g. Data Engineer Professional)
Azure certification (e.g. Microsoft Certified: Azure Data Engineer Associate) is a plus.
Experience with Azure Logic Apps, Azure Functions, and API Management.
AI/ML experience (ability to develop predictive systems and deploy AI/ML models using Databricks & Microsoft tools)
Knowledge of Power BI or other data visualization tools.
Soft Skills
Excellent analytical and problem‑solving skills.
Highly organized, detail‑oriented and able to work independently as well as collaboratively.
Applicants must be authorized to work in the U.S. without employment‑based visa sponsorship (now or in the future). This includes H‑1B, L‑1, TN, O‑1, E‑3, H‑1B1, F‑1, J‑1, OPT, CPT or any other employment‑based visas.
#J-18808-Ljbffr