Highbridge Consulting LLC
We are seeking an experienced Azure Databricks Engineer to design, implement, and optimize modern Lakehouse architectures in a regulated, financial services environment. The ideal candidate blends deep expertise in Azure and Databricks, with proven experience enforcing governance and security at scale.
Responsibilities
rchitect, build, and maintain Azure-based Lakehouse solutions leveraging Databricks, Delta Lake, and Azure-native services. Implement medallion (bronze/silver/gold) architecture to ensure scalability, governance, and analytics agility. Develop and optimize data ingestion frameworks for both batch and streaming pipelines (PySpark, Spark SQL, etc). Configure and manage Unity Catalog, enforcing row- and column-level security, role-based access, and lineage tracking. Collaborate with cross-functional teams to integrate siloed systems into unified data platforms. Ensure solutions meet financial services regulatory requirements and deliver secure, compliant data environments. Optimize Spark workloads for performance and cost efficiency in large-scale environments. Support BI and analytics teams by delivering performant, governed semantic layers. Champion best practices for SDLC, CI/CD, testing, and DevOps in Azure and Databricks ecosystems. Requirements
7+ years in data engineering, with at least 3+ focused on Azure Databricks. Strong expertise in Azure services (Data Factory, SQL, Fabric). Hands-on experience implementing row- and column-level security via Unity Catalog. Deep understanding of investment banking or similarly regulated industries. Proficiency in Python, PySpark, Spark SQL, T-SQL. Experience in performance tuning and cost optimization of large-scale Spark workloads. Strong knowledge of data governance, compliance, and security protocols. Clear communicator who can bridge technical and business priorities. Strong problem-solver with a delivery mindset. Comfortable working in regulated, high-stakes environments. Collaborative team player with experience mentoring peers and enforcing best practices.
Responsibilities
rchitect, build, and maintain Azure-based Lakehouse solutions leveraging Databricks, Delta Lake, and Azure-native services. Implement medallion (bronze/silver/gold) architecture to ensure scalability, governance, and analytics agility. Develop and optimize data ingestion frameworks for both batch and streaming pipelines (PySpark, Spark SQL, etc). Configure and manage Unity Catalog, enforcing row- and column-level security, role-based access, and lineage tracking. Collaborate with cross-functional teams to integrate siloed systems into unified data platforms. Ensure solutions meet financial services regulatory requirements and deliver secure, compliant data environments. Optimize Spark workloads for performance and cost efficiency in large-scale environments. Support BI and analytics teams by delivering performant, governed semantic layers. Champion best practices for SDLC, CI/CD, testing, and DevOps in Azure and Databricks ecosystems. Requirements
7+ years in data engineering, with at least 3+ focused on Azure Databricks. Strong expertise in Azure services (Data Factory, SQL, Fabric). Hands-on experience implementing row- and column-level security via Unity Catalog. Deep understanding of investment banking or similarly regulated industries. Proficiency in Python, PySpark, Spark SQL, T-SQL. Experience in performance tuning and cost optimization of large-scale Spark workloads. Strong knowledge of data governance, compliance, and security protocols. Clear communicator who can bridge technical and business priorities. Strong problem-solver with a delivery mindset. Comfortable working in regulated, high-stakes environments. Collaborative team player with experience mentoring peers and enforcing best practices.