Compunnel
Senior Backend Engineer Metadata Catalog / Governance Automation Mission
Compunnel, Cincinnati, Ohio, United States, 45208
Job Summary
The Senior Backend Engineer will lead the design and development of automation capabilities that support metadata governance, data quality, and observability within Databricks and Unity Catalog.
This role is critical to enabling trusted, well-governed data infrastructure that powers the Metadata Catalog UI and supports enterprise-wide data transparency and compliance.
Key Responsibilities
Databricks & Unity Catalog Engineering Build and maintain backend services using Databricks (SQL, PySpark, Delta Lake, Jobs/Workflows). Administer Unity Catalog metadata, permissions, lineage, and tagging. Integrate Unity Catalog APIs to support Metadata Catalog UI functionality. Governance Automation
Develop automation scripts and pipelines to enforce access controls, tagging, and role-based policies. Implement governance workflows integrated with tools like ServiceNow. Automate compliance checks for IAM, PII handling, and encryption standards. Data Quality & Observability
Implement data quality frameworks (e.g., Great Expectations, Deequ). Build monitoring pipelines for logging, usage metrics, audit trails, and alerts. Ensure system reliability and proactive issue detection. API Development & Integration
Design APIs to integrate Databricks services with external platforms (e.g., ServiceNow, monitoring tools). Build reusable automation utilities and frameworks for scalable governance. DevOps & CI/CD
Manage source control and CI/CD pipelines using GitHub, Azure DevOps, or Jenkins. Deploy secure and scalable backend services in cloud environments (Azure preferred). Document, test, and productionize automation solutions. Required Qualifications
Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent professional experience. Minimum 5 years of backend engineering experience in data platforms. Minimum 3 years of experience with Databricks and/or Unity Catalog in enterprise environments. Strong proficiency in Databricks (SQL, PySpark, Delta Lake, Jobs/Workflows). Deep knowledge of Unity Catalog administration and APIs. Expertise in Python for automation, API integrations, and data quality checks. Experience with governance frameworks (access control, tagging, lineage, compliance). Solid understanding of IAM, encryption, and PII handling. Preferred Qualifications
Experience with CI/CD tools (GitHub Actions, Azure DevOps, Jenkins). Familiarity with monitoring and observability tools. Experience integrating with external systems such as ServiceNow. Knowledge of data quality frameworks (Great Expectations, Deequ). Strong problem-solving skills in distributed systems. Clear communication and documentation skills.
Education:
Bachelors Degree
The Senior Backend Engineer will lead the design and development of automation capabilities that support metadata governance, data quality, and observability within Databricks and Unity Catalog.
This role is critical to enabling trusted, well-governed data infrastructure that powers the Metadata Catalog UI and supports enterprise-wide data transparency and compliance.
Key Responsibilities
Databricks & Unity Catalog Engineering Build and maintain backend services using Databricks (SQL, PySpark, Delta Lake, Jobs/Workflows). Administer Unity Catalog metadata, permissions, lineage, and tagging. Integrate Unity Catalog APIs to support Metadata Catalog UI functionality. Governance Automation
Develop automation scripts and pipelines to enforce access controls, tagging, and role-based policies. Implement governance workflows integrated with tools like ServiceNow. Automate compliance checks for IAM, PII handling, and encryption standards. Data Quality & Observability
Implement data quality frameworks (e.g., Great Expectations, Deequ). Build monitoring pipelines for logging, usage metrics, audit trails, and alerts. Ensure system reliability and proactive issue detection. API Development & Integration
Design APIs to integrate Databricks services with external platforms (e.g., ServiceNow, monitoring tools). Build reusable automation utilities and frameworks for scalable governance. DevOps & CI/CD
Manage source control and CI/CD pipelines using GitHub, Azure DevOps, or Jenkins. Deploy secure and scalable backend services in cloud environments (Azure preferred). Document, test, and productionize automation solutions. Required Qualifications
Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent professional experience. Minimum 5 years of backend engineering experience in data platforms. Minimum 3 years of experience with Databricks and/or Unity Catalog in enterprise environments. Strong proficiency in Databricks (SQL, PySpark, Delta Lake, Jobs/Workflows). Deep knowledge of Unity Catalog administration and APIs. Expertise in Python for automation, API integrations, and data quality checks. Experience with governance frameworks (access control, tagging, lineage, compliance). Solid understanding of IAM, encryption, and PII handling. Preferred Qualifications
Experience with CI/CD tools (GitHub Actions, Azure DevOps, Jenkins). Familiarity with monitoring and observability tools. Experience integrating with external systems such as ServiceNow. Knowledge of data quality frameworks (Great Expectations, Deequ). Strong problem-solving skills in distributed systems. Clear communication and documentation skills.
Education:
Bachelors Degree