Spk Technologies inc
PositionID: 64002533
Client: Gainwell Technologies
Position: Azure Databricks Engineer
Responsibilities:
Data pipeline Development: Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Azure Data Factory, and other Azure services.
Data Processing: Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks.
Integration: Integrate data from various sources, such as databases, APIs, and streaming data, into the Databricks environment, often using tools like Azure Data Factory or Databricks Workflows.
Performance Optimization: Enhance the performance of data processing tasks by optimizing Spark jobs, managing cluster resources, and implementing best practices for data storage and retrieval.
Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure the solutions meet business needs.
Data Governance: Implement data governance policies and ensure data security and compliance with relevant regulations.
CI/CD Integration: Utilize Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure.
Client: Gainwell Technologies
Position: Azure Databricks Engineer
Responsibilities:
Data pipeline Development: Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Azure Data Factory, and other Azure services.
Data Processing: Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks.
Integration: Integrate data from various sources, such as databases, APIs, and streaming data, into the Databricks environment, often using tools like Azure Data Factory or Databricks Workflows.
Performance Optimization: Enhance the performance of data processing tasks by optimizing Spark jobs, managing cluster resources, and implementing best practices for data storage and retrieval.
Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure the solutions meet business needs.
Data Governance: Implement data governance policies and ensure data security and compliance with relevant regulations.
CI/CD Integration: Utilize Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure.