Azure Databricks Architect
US-CA-San Diego
Job ID:
Type: Regular Full-Time
# of Openings: 1
Category: Accounting/Finance
Corporate Office
Overview
An Azure Databricks Architect is a technical role focused on designing and implementing scalable data solutions using Azure Databricks, acting as a subject matter expert in the platform to create efficient data pipelines, manage data lakes, and ensure data quality while leveraging the full capabilities of the Azure cloud ecosystem; essentially, they are responsible for architecting robust data solutions on Azure Databricks, considering data ingestion, processing, storage, and analysis needs within the context of an organization's business requirements.
Responsibilities
Solution Design : Develop and implement data architecture using Azure Databricks, including data ingestion pipelines, data transformation logic, and data warehousing strategies. Building and optimizing ETL pipelines, particularly in Databricks using PySpark, Scala, or Spark SQL. Ability to create and manage Databricks notebooks for data exploration, pipeline development, and data visualization. Familiarity with scheduling and automating Databricks jobs.
Big Data Processing : Design and optimize ETL/ELT pipelines using Apache Spark, Python, Scala, or SQL. Experience building ETL/ELT pipelines, data cleansing, and data transformation processes.
Data Technologies: Proficiency in setting up, configuring, and managing Azure Databricks clusters, as well as optimizing performance and costs. Understanding of how to manage, process, and store big data in Azure Data Lake or other storage solutions.
Technical Leadership: Guide development teams, data engineers, and business analysts on best practices for data modeling, data processing, and optimization within the Databricks platform.
Data Integration : Design and build data integration solutions to connect diverse data sources, including on-premises systems, cloud-based applications, and APIs.
Performance Optimization: Monitor and optimize data processing performance on Databricks clusters to ensure efficient data processing at scale.
Security and Governance: Implement data security measures and access controls to protect sensitive data within the Azure Databricks environment. Knowledge of networking concepts within Azure, such as virtual networks, firewalls, private endpoints, and network security groups. Ensure compliance with security and governance best practices, including RBAC, encryption, and data masking.
Collaboration: Work closely with data engineers, data scientists, business intelligence analysts, and business stakeholders to understand requirements and translate them into technical solutions.
Compliance
Adhere to company policy and procedure, complete annual compliance training, and comply with business continuity and vendor management policy. These are crucial to the success of the organization and have regulatory compliance and/or safety and soundness for the sustainability of the credit union.
Comply with all Anti Money Laundering laws such as the