SMBC Group
Senior Data Engineer - Databricks
SMBC Group, Charlotte, North Carolina, United States, 28245
Overview
Senior Data Engineer - Databricks role at SMBC Group. This position targets candidates with strong Databricks, PySpark, and Azure cloud experience to contribute to fraud detection platform development and maintenance. Role Description
The Databricks Developer is responsible for implementing, supporting, and enhancing the internal fraud detection platform by developing scalable data pipelines, integrating batch processing methods, and ensuring the platform aligns with the banks risk management, legal, and regulatory requirements for fraud detection and prevention. This role requires deep functional and technical expertise in Databricks development, including strong development skills in PySpark and the Azure cloud ecosystem. Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar. The developer will work closely with business units and support teams to deliver the initial application, system enhancements, perform upgrades, and provide on-call user support. The ideal candidate holds a degree in Computer Science or a related field and has at least 5 years of professional experience in data engineering and cloud-based development. Key Responsibilities
Design, develop, and optimize large-scale batch data pipelines using Databricks and PySpark on the Azure cloud platform. Lead technical architecture and implementation of Azure-based solutions, supporting cloud migration and consolidation initiatives. Build and maintain ETL processes, ensuring seamless data integration and high data quality across diverse sources. Develop orchestration workflows using Azure Functions, Azure Data Factory (ADF), Logic Apps, and other Azure services. Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar Implement secure and scalable solutions leveraging Blob Storage, Key Vault, Managed Identities, and Azure DevOps. Provide technical guidance and support for architectural decisions and platform enhancements. Own end-to-end project delivery, working closely with business stakeholders, IT teams, and third-party vendors. Incorporating a variety of data processing techniquesincluding batch and streaming workflowswhile exposing and integrating APIs and external services into Databricks pipelines to enhance platform functionality and enable seamless data exchange across systems. Review and contribute to core code changes, ensuring best practices and supporting production deployments. Experience in developing and implementing disaster recovery strategies for cloud-based applications Qualifications And Skills
Required Qualifications: Bachelors degree in Computer Science, Information Systems, or a related technical field. Minimum 5 years of experience in data engineering, with a focus on Databricks, PySpark, and Azure. Strong understanding of data integration, transformation, and migration strategies. Experience with CI/CD pipelines and version control using Azure DevOps or githib. Excellent problem-solving skills and ability to resolve moderately complex technical challenges. Strong communication and collaboration skills; Additional Requirements
SMBCs employees participate in a Hybrid workforce model that provides employees with an opportunity to work from home, as well as from an SMBC office. SMBC requires that employees live within a reasonable commuting distance of their office location. Prospective candidates will learn more about their specific hybrid work schedule during their interview process. Hybrid work may not be permitted for certain roles, including, for example, certain FINRA-registered roles for which in-office attendance for the entire workweek is required. SMBC provides reasonable accommodations during candidacy for applicants with disabilities consistent with applicable federal, state, and local law. If you need a reasonable accommodation during the application process, please let us know at accommodations@smbcgroup.com. Job Details
Seniority level: Not Applicable Employment type: Full-time Job function: Information Technology Industries: Banking, Financial Services, and IT Services and IT Consulting
#J-18808-Ljbffr
Senior Data Engineer - Databricks role at SMBC Group. This position targets candidates with strong Databricks, PySpark, and Azure cloud experience to contribute to fraud detection platform development and maintenance. Role Description
The Databricks Developer is responsible for implementing, supporting, and enhancing the internal fraud detection platform by developing scalable data pipelines, integrating batch processing methods, and ensuring the platform aligns with the banks risk management, legal, and regulatory requirements for fraud detection and prevention. This role requires deep functional and technical expertise in Databricks development, including strong development skills in PySpark and the Azure cloud ecosystem. Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar. The developer will work closely with business units and support teams to deliver the initial application, system enhancements, perform upgrades, and provide on-call user support. The ideal candidate holds a degree in Computer Science or a related field and has at least 5 years of professional experience in data engineering and cloud-based development. Key Responsibilities
Design, develop, and optimize large-scale batch data pipelines using Databricks and PySpark on the Azure cloud platform. Lead technical architecture and implementation of Azure-based solutions, supporting cloud migration and consolidation initiatives. Build and maintain ETL processes, ensuring seamless data integration and high data quality across diverse sources. Develop orchestration workflows using Azure Functions, Azure Data Factory (ADF), Logic Apps, and other Azure services. Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar Implement secure and scalable solutions leveraging Blob Storage, Key Vault, Managed Identities, and Azure DevOps. Provide technical guidance and support for architectural decisions and platform enhancements. Own end-to-end project delivery, working closely with business stakeholders, IT teams, and third-party vendors. Incorporating a variety of data processing techniquesincluding batch and streaming workflowswhile exposing and integrating APIs and external services into Databricks pipelines to enhance platform functionality and enable seamless data exchange across systems. Review and contribute to core code changes, ensuring best practices and supporting production deployments. Experience in developing and implementing disaster recovery strategies for cloud-based applications Qualifications And Skills
Required Qualifications: Bachelors degree in Computer Science, Information Systems, or a related technical field. Minimum 5 years of experience in data engineering, with a focus on Databricks, PySpark, and Azure. Strong understanding of data integration, transformation, and migration strategies. Experience with CI/CD pipelines and version control using Azure DevOps or githib. Excellent problem-solving skills and ability to resolve moderately complex technical challenges. Strong communication and collaboration skills; Additional Requirements
SMBCs employees participate in a Hybrid workforce model that provides employees with an opportunity to work from home, as well as from an SMBC office. SMBC requires that employees live within a reasonable commuting distance of their office location. Prospective candidates will learn more about their specific hybrid work schedule during their interview process. Hybrid work may not be permitted for certain roles, including, for example, certain FINRA-registered roles for which in-office attendance for the entire workweek is required. SMBC provides reasonable accommodations during candidacy for applicants with disabilities consistent with applicable federal, state, and local law. If you need a reasonable accommodation during the application process, please let us know at accommodations@smbcgroup.com. Job Details
Seniority level: Not Applicable Employment type: Full-time Job function: Information Technology Industries: Banking, Financial Services, and IT Services and IT Consulting
#J-18808-Ljbffr