Data Freelance Hub
Sr. Databricks Solution Architect with Healthcare Domain (Only W2 or Selfcorp)
Data Freelance Hub, New York, New York, us, 10261
Sr. Databricks Solution Architect with Healthcare Domain (Only W2 or Selfcorp)
⭐ - Featured Role | Apply direct with Data Freelance Hub
This role is for a Sr. Databricks Solution Architect with Healthcare Domain, offering a contract-to-hire position. Pay rate is competitive. Key skills include Azure Databricks, ADF, CI/CD, and Terraform. Requires 5+ years in cloud data engineering and healthcare experience.
Location: United States (Remote, Washington, DC office visits as needed). Employment: Contract to Hire (W2 Contractor).
Job Description Top Skills Needed:
Deep hands-on expertise with Databricks platform architecture and governance
Unity Catalog, workspaces, external locations, compute, access controls, cluster governance
Reliability engineering, monitoring, and operational hardening of the Lakehouse
Observability, alerting, DR readiness, backup/restore, performance tuning, incident response
Strong experience with ADF, CI/CD, and Terraform for orchestrating and managing the Lakehouse
Pipeline orchestration, IaC, DevOps, environment promotion, compute policies
Typical Day-to-Day:
Design how the Databricks Lakehouse should work including the structure, tools, standards, and best practices
Guide engineering teams on how to build pipelines and use Databricks correctly
Solve technical issues when data jobs fail or performance slows
Work with stakeholders to understand data needs and deliver solutions
Set standards for security, governance, naming conventions, and architecture
Ensure the Databricks platform is stable, reliable, and always available
Build and maintain monitoring, alerting, logging, and health dashboards
Strengthen and fix ingestion pipelines (ADF → landing → raw → curated)
Improve data quality checks, anomaly detection, and pipeline reliability
Manage CI/CD pipelines and deployment processes using Azure DevOps or GitHub
Use Terraform (IaC) to deploy and manage Databricks and Azure infrastructure
Partner with Security and FinOps on access controls, compliance, and cost governance
Mentor the Data Engineer and support distributed data engineering teams across the organization
Key Responsibilities 1. Lakehouse Architecture & Platform Administration (Approximately 60% of role when combined with mentoring & code review)
Serve as the primary architect and administrator for the Azure Databricks Lakehouse (Unity Catalog, workspaces, external locations, compute, access controls).
Lead execution of a Minimal Viable Hardening Roadmap for the platform, prioritizing:
High availability and DR readiness
Backup/restore patterns for data and metadata
Platform observability and operational metrics
Secure and maintainable catalog/namespace structure
Robust and proactive data quality assurance
Implement and evolve naming conventions, environment strategies, and platform standards that enable long-term maintainability and safe scaling.
Act as the Lakehouse-facing counterpart to Enterprise Architecture and Security, collaborating on network architecture, identity & access, compliance controls, and platform guardrails.
2. Reliability, Monitoring, and Incident Management
Design, implement, and maintain comprehensive monitoring and alerting for Lakehouse platform components, ingestion jobs, key data assets, and system health indicators.
Oversee end-to-end reliability engineering, including capacity planning, throughput tuning, job performance optimization, and preventative maintenance (e.g., IR updates, compute policy reviews).
Participate in — and help shape — the on-call rotation for high-priority incidents affecting production workloads, including rapid diagnosis and mitigation during off-hours as needed.
Develop and maintain incident response runbooks, escalation pathways, stakeholder communication protocols, and operational readiness checklists.
Lead or participate in post-incident Root Cause Analyses, ensuring durable remediation and preventing recurrence.
Conduct periodic DR and failover simulations, validating RPO/RTO and documenting improvements.
3. Pipeline Reliability, Ingestion Patterns & Data Quality
Strengthen and standardize ingestion pipelines (ADF
Collaborate with the Data Engineer to modernize logging, automated anomaly detection, pipeline health dashboards, and DQ validation automation.
Provide architectural guidance, code reviews, mentoring, and best-practice patterns to distributed engineering teams across MedStar.
Support stabilization of existing ingestion and transformation pipelines across clinical (notes, OHDSI), financial, operational, and quality use cases.
4. DevOps, CI/CD, and Infrastructure as Code
Administer and improve CI/CD pipelines using Azure DevOps or GitHub Enterprise.
Support automated testing, environment promotion, and rollback patterns for Databricks and dbt assets.
Maintain and extend Terraform (or adopt Terraform from another IaC background) for Databricks, storage, networking, compute policies, and related infrastructure.
Promote version control standards, branching strategies, and deployment governance across data engineering teams.
5. Security, FinOps, and Guardrails Partnership
Partner with Enterprise Architecture and Security on platform access controls, identity strategy, encryption, networking, and compliance.
Implement and enforce cost tagging, compute policies, and alerts supporting FinOps transparency and cost governance.
Collaborate with the team defining agentic coding guardrails, ensuring the Lakehouse platform supports safe & compliant use of AI-assisted code generation and execution.
Help assess and optimize serverless SQL, serverless Python, and job compute patterns for cost-efficiency and reliability.
6. Mentorship, Collaboration, & Distributed Enablement
Mentor the mid-level Data Engineer on Databricks, ADF, dbt, observability, DevOps, Terraform, and operational engineering patterns.
Provide guidance, design patterns, and code review support to multiple distributed data engineering teams (Finance, MCPI, Safety/Risk, Quality, Digital Transformation, etc.).
Lead platform knowledge-sharing efforts through documentation, workshops, and best-practice guidance.
Demonstrate strong collaboration skills, balancing independence with alignment across teams.
7. Optional / Nice-to-Have: OHDSI Platform Support (Not required for hiring; can be learned on the job.)
Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments).
Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse.
Required Qualifications
5+ years in cloud data engineering, platform engineering, or solution architecture.
Strong hands-on expertise in Azure Databricks: Unity Catalog, workspaces & external locations, SQL/Python notebooks & Jobs, cluster/warehouse governance.
Solid working experience with Azure Data Factory (pipelines, IRs, linked services).
Strong SQL and Python engineering skills.
Experience with CI/CD in Azure DevOps or GitHub Enterprise.
Experience with Terraform or another IaC framework, and willingness to adopt Terraform.
Demonstrated ability to design or support monitoring, alerting, logging, or reliability systems.
Strong communication, collaboration, and problem-solving skills.
Preferred Qualifications (Optional)
Advanced Terraform experience.
Familiarity with healthcare, HIPAA, PHI, or regulated environments.
Experience with Purview or enterprise cataloging.
Exposure to OHDSI/OMOP.
Experience optimizing or refactoring legacy ingestion pipelines.
Experience supporting secure, controlled AI/agentic execution environments.
Experience with EPIC EHR data exchange and/or EPIC Caboodle or Cogito analytics suite.
Personal Attributes
Hands-on, pragmatic, and operationally minded.
Comfortable leading both architecture and implementation.
Collaborative and mentorship-oriented; thrives in small core teams with broad influence.
Values platform stability, observability, and hardening over shiny features.
Curious and adaptable, especially with emerging AI-assisted engineering patterns.
Ability to remain calm and effective during incidents and high-pressure situations.
If you believe you are qualified for this position and are currently in the job market or interested in making a change, please email me the resume along with contact details at
roshni@nytpcorp.com .
#J-18808-Ljbffr
This role is for a Sr. Databricks Solution Architect with Healthcare Domain, offering a contract-to-hire position. Pay rate is competitive. Key skills include Azure Databricks, ADF, CI/CD, and Terraform. Requires 5+ years in cloud data engineering and healthcare experience.
Location: United States (Remote, Washington, DC office visits as needed). Employment: Contract to Hire (W2 Contractor).
Job Description Top Skills Needed:
Deep hands-on expertise with Databricks platform architecture and governance
Unity Catalog, workspaces, external locations, compute, access controls, cluster governance
Reliability engineering, monitoring, and operational hardening of the Lakehouse
Observability, alerting, DR readiness, backup/restore, performance tuning, incident response
Strong experience with ADF, CI/CD, and Terraform for orchestrating and managing the Lakehouse
Pipeline orchestration, IaC, DevOps, environment promotion, compute policies
Typical Day-to-Day:
Design how the Databricks Lakehouse should work including the structure, tools, standards, and best practices
Guide engineering teams on how to build pipelines and use Databricks correctly
Solve technical issues when data jobs fail or performance slows
Work with stakeholders to understand data needs and deliver solutions
Set standards for security, governance, naming conventions, and architecture
Ensure the Databricks platform is stable, reliable, and always available
Build and maintain monitoring, alerting, logging, and health dashboards
Strengthen and fix ingestion pipelines (ADF → landing → raw → curated)
Improve data quality checks, anomaly detection, and pipeline reliability
Manage CI/CD pipelines and deployment processes using Azure DevOps or GitHub
Use Terraform (IaC) to deploy and manage Databricks and Azure infrastructure
Partner with Security and FinOps on access controls, compliance, and cost governance
Mentor the Data Engineer and support distributed data engineering teams across the organization
Key Responsibilities 1. Lakehouse Architecture & Platform Administration (Approximately 60% of role when combined with mentoring & code review)
Serve as the primary architect and administrator for the Azure Databricks Lakehouse (Unity Catalog, workspaces, external locations, compute, access controls).
Lead execution of a Minimal Viable Hardening Roadmap for the platform, prioritizing:
High availability and DR readiness
Backup/restore patterns for data and metadata
Platform observability and operational metrics
Secure and maintainable catalog/namespace structure
Robust and proactive data quality assurance
Implement and evolve naming conventions, environment strategies, and platform standards that enable long-term maintainability and safe scaling.
Act as the Lakehouse-facing counterpart to Enterprise Architecture and Security, collaborating on network architecture, identity & access, compliance controls, and platform guardrails.
2. Reliability, Monitoring, and Incident Management
Design, implement, and maintain comprehensive monitoring and alerting for Lakehouse platform components, ingestion jobs, key data assets, and system health indicators.
Oversee end-to-end reliability engineering, including capacity planning, throughput tuning, job performance optimization, and preventative maintenance (e.g., IR updates, compute policy reviews).
Participate in — and help shape — the on-call rotation for high-priority incidents affecting production workloads, including rapid diagnosis and mitigation during off-hours as needed.
Develop and maintain incident response runbooks, escalation pathways, stakeholder communication protocols, and operational readiness checklists.
Lead or participate in post-incident Root Cause Analyses, ensuring durable remediation and preventing recurrence.
Conduct periodic DR and failover simulations, validating RPO/RTO and documenting improvements.
3. Pipeline Reliability, Ingestion Patterns & Data Quality
Strengthen and standardize ingestion pipelines (ADF
Collaborate with the Data Engineer to modernize logging, automated anomaly detection, pipeline health dashboards, and DQ validation automation.
Provide architectural guidance, code reviews, mentoring, and best-practice patterns to distributed engineering teams across MedStar.
Support stabilization of existing ingestion and transformation pipelines across clinical (notes, OHDSI), financial, operational, and quality use cases.
4. DevOps, CI/CD, and Infrastructure as Code
Administer and improve CI/CD pipelines using Azure DevOps or GitHub Enterprise.
Support automated testing, environment promotion, and rollback patterns for Databricks and dbt assets.
Maintain and extend Terraform (or adopt Terraform from another IaC background) for Databricks, storage, networking, compute policies, and related infrastructure.
Promote version control standards, branching strategies, and deployment governance across data engineering teams.
5. Security, FinOps, and Guardrails Partnership
Partner with Enterprise Architecture and Security on platform access controls, identity strategy, encryption, networking, and compliance.
Implement and enforce cost tagging, compute policies, and alerts supporting FinOps transparency and cost governance.
Collaborate with the team defining agentic coding guardrails, ensuring the Lakehouse platform supports safe & compliant use of AI-assisted code generation and execution.
Help assess and optimize serverless SQL, serverless Python, and job compute patterns for cost-efficiency and reliability.
6. Mentorship, Collaboration, & Distributed Enablement
Mentor the mid-level Data Engineer on Databricks, ADF, dbt, observability, DevOps, Terraform, and operational engineering patterns.
Provide guidance, design patterns, and code review support to multiple distributed data engineering teams (Finance, MCPI, Safety/Risk, Quality, Digital Transformation, etc.).
Lead platform knowledge-sharing efforts through documentation, workshops, and best-practice guidance.
Demonstrate strong collaboration skills, balancing independence with alignment across teams.
7. Optional / Nice-to-Have: OHDSI Platform Support (Not required for hiring; can be learned on the job.)
Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments).
Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse.
Required Qualifications
5+ years in cloud data engineering, platform engineering, or solution architecture.
Strong hands-on expertise in Azure Databricks: Unity Catalog, workspaces & external locations, SQL/Python notebooks & Jobs, cluster/warehouse governance.
Solid working experience with Azure Data Factory (pipelines, IRs, linked services).
Strong SQL and Python engineering skills.
Experience with CI/CD in Azure DevOps or GitHub Enterprise.
Experience with Terraform or another IaC framework, and willingness to adopt Terraform.
Demonstrated ability to design or support monitoring, alerting, logging, or reliability systems.
Strong communication, collaboration, and problem-solving skills.
Preferred Qualifications (Optional)
Advanced Terraform experience.
Familiarity with healthcare, HIPAA, PHI, or regulated environments.
Experience with Purview or enterprise cataloging.
Exposure to OHDSI/OMOP.
Experience optimizing or refactoring legacy ingestion pipelines.
Experience supporting secure, controlled AI/agentic execution environments.
Experience with EPIC EHR data exchange and/or EPIC Caboodle or Cogito analytics suite.
Personal Attributes
Hands-on, pragmatic, and operationally minded.
Comfortable leading both architecture and implementation.
Collaborative and mentorship-oriented; thrives in small core teams with broad influence.
Values platform stability, observability, and hardening over shiny features.
Curious and adaptable, especially with emerging AI-assisted engineering patterns.
Ability to remain calm and effective during incidents and high-pressure situations.
If you believe you are qualified for this position and are currently in the job market or interested in making a change, please email me the resume along with contact details at
roshni@nytpcorp.com .
#J-18808-Ljbffr