Azure Databricks Architect

2 days ago Be among the first 25 applicants

This range is provided by Fractal. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

$115,000.00/yr - $165,000.00/yr

Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. Fractal is building a world where individual choices, freedom, and diversity are the greatest assets. An ecosystem where human imagination is at the heart of every decision. Where no possibility is written off, only challenged to get better. We believe that a true Fractalite is the one who empowers imagination with intelligence. Fractal has been featured as a Great Place to Work by The Economic Times in partnership with the Great Place to Work Institute and recognized as a ‘Cool Vendor’ and a ‘Vendor to Watch’ by Gartner.

Please visit Fractal | Intelligence for Imagination for more information about Fractal.

Position Overview :

Fractal is looking for a proactive and driven Azure Databricks Architect to join our Consulting team for a client in the Midwest. In this role, you will be working on a Databricks Lakehouse Platform modernization / migration. You will work designing the system architecture and solution, ensuring the platform is scalable while performant, and creating automated data pipelines.

Responsibilities:

• Design & Architecture of Scalable Data Platforms
• Design, develop, and maintain large-scale data processing architectures on the Databricks Lakehouse Platform to support business needs such as sales forecasting, trade promotions, supply chain optimization etc...
• Architect multi-layer data models including Bronze (raw), Silver (cleansed), and Gold (curated) layers for various domains (e.g., Retail Execution, Digital Commerce, Logistics, Category Management).
• Leverage Delta Lake, Unity Catalog, and advanced features of Databricks for governed data sharing, versioning, and reproducibility.
• Client & Business Stakeholder Engagement
• Partner with business stakeholders to translate functional requirements into scalable technical solutions.
• Conduct architecture workshops and solutioning sessions with enterprise IT and business teams to define data-driven use cases
• Collaborate with data engineers and data scientists to develop end-to-end pipelines using PySpark, SQL, DLT (Delta Live Tables), and Databricks Workflows.
• Enable data ingestion from diverse sources such as ERP (SAP), POS data, Syndicated Data, CRM, e-commerce platforms, and third-party datasets.
• Performance, Scalability, and Reliability
• Optimize Spark jobs for performance tuning, cost efficiency, and scalability by configuring appropriate cluster sizing, caching, and query optimization techniques.
• Implement monitoring and alerting using Databricks Observability, Ganglia, Cloud-native tools
• Security, Compliance & Governance
• Design secure architectures using Unity Catalog, role-based access control (RBAC), encryption, token-based access, and data lineage tools to meet compliance policies.
• Establish data governance practices including Data Fitness Index, Quality Scores, SLA Monitoring, and Metadata Cataloging.
• Adoption of AI Copilots & Agentic Development
• Utilize GitHub Copilot, Databricks Assistant, and other AI code agents for
• Writing PySpark, SQL, and Python code snippets for data engineering and ML tasks.
• Generating documentation and test cases to accelerate pipeline development.
• Interactive debugging and iterative code optimization within notebooks.
• Advocate for agentic AI workflows that use specialized agents for
• Data profiling and schema inference.
• Automated testing and validation.
• Innovation and Continuous Learning
• Stay abreast of emerging trends in Lakehouse architectures, Generative AI, and cloud-native tooling.
• Evaluate and pilot new features from Databricks releases and partner integrations for modern data stack improvements.

Requirements:

• Bachelor’s or master’s degree in computer science, Information Technology, or a related field.
• 12-18 years of hands-on experience in data engineering, with at least 5+ years on Databricks Architecture and Apache Spark.
• Expertise in building high-throughput, low-latency ETL/ELT pipelines on Azure Databricks using PySpark, SQL, and Databricks-native features.
• Familiarity with ingestion frameworks from structured/unstructured data sources including APIs, flat files, RDBMS, and cloud storage (Azure Data Lake Storage Gen2)
• Experience designing Lakehouse architectures with bronze, silver, gold layering.
• Expertise in optimizing Databricks performance using Delta Lake features such as OPTIMIZE, VACUUM, ZORDER, and Time Travel
• Strong understanding of data modelling concepts, star/snowflake schemas, dimensional modelling, and modern cloud-based data warehousing.
• Experience with designing Data marts using Databricks SQL warehouse and integrating with BI tools (Power BI, Tableau, etc.).
• Hands-on experience designing solutions using Workflows (Jobs), Delta Lake, Delta Live Tables (DLT), Unity Catalog, and MLflow.
• Familiarity with Databricks REST APIs, Notebooks, and cluster configurations for automated provisioning and orchestration.
• Experience in integrating Databricks with CI/CD pipelines using tools such as Azure DevOps, GitHub Actions.
• Knowledge of infrastructure-as-code (Terraform, ARM templates) for provisioning Databricks workspaces and resources
• In-depth experience with Azure Cloud services such as ADF, Synapse, ADLS, Key Vault, Azure Monitor, and Azure Security Centre.
• Strong understanding of data privacy, access controls, and governance best practices.
• Experience working with Unity Catalog, RBAC, tokenization, and data classification frameworks
• Excellent communication skills for stakeholder interaction, solution presentations, and team coordination.
• Proven experience leading or mentoring global, cross-functional teams across multiple time zones and engagements.
• Ability to work independently in agile or hybrid delivery models, while guiding junior engineers and ensuring solution quality

Pay:

The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Fractal, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is: $115,000 - $165,000. In addition, you may be eligible for a discretionary bonus for the current performance period.

As a full-time employee of the company or as an hourly employee working more than 30 hours per week, you will be eligible to participate in the health, dental, vision, life insurance, and disability plans in accordance with the plan documents, which may be amended from time to time. You will be eligible for benefits on the first day of employment with the Company. In addition, you are eligible to participate in the Company 401(k) Plan after 30 days of employment, in accordance with the applicable plan terms. The Company provides 11 paid holidays and 12 weeks of Parental Leave. We also follow a “free time” PTO policy, allowing you the flexibility to take the time needed for either sick time or vacation.

Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Engineering and Information Technology

• Industries

  IT Services and IT Consulting and Business Consulting and Services

Referrals increase your chances of interviewing at Fractal by 2x

Get notified about new Cloud Architect jobs in Chicago, IL .

Chicago, IL $218,000.00-$306,000.00 2 weeks ago

Chicago, IL $93,000.00-$151,000.00 1 week ago

Chicago, IL $167,032.32-$226,686.72 3 weeks ago

Practice Architect, Manufacturing/Auto, Google Cloud

Chicago, IL $168,000.00-$249,000.00 6 days ago

Chicago, IL $150,000.00-$190,000.00 3 weeks ago

Chicago, IL $214,900.00-$322,300.00 2 weeks ago

Global Practice Architect Lead, Partners, Google Cloud

Chicago, IL $218,000.00-$306,000.00 1 week ago

AWS Technical Solution Architect - Onsite(Fulltime)

Chicago, IL $168,000.00-$249,000.00 2 days ago

Chicago, IL $130,000.00-$160,000.00 1 week ago

Solutions Architect, Retail, Google Cloud

Chicago, IL $147,000.00-$216,000.00 1 week ago

Chicago, IL $90,000.00-$115,000.00 2 months ago

Senior Solutions Architect, ServiceNow, Full-time, Days

Rolling Meadows, IL $129,000.00-$251,500.00 2 weeks ago

Chicago, IL $188,000.00-$230,000.00 1 week ago

Chicago, IL $150,000.00-$170,000.00 3 weeks ago

Chicago, IL $70,000.00-$150,000.00 4 days ago

Greater Chicago Area $85.00-$100.00 2 weeks ago

Chicago, IL $125,825.00-$221,275.00 6 days ago

Software Engineering - Solutions Architect

Oak Brook, IL $130,000.00-$150,000.00 2 weeks ago

Solutions Architect, Amazon Web Services

Chicago, IL $118,200.00-$204,300.00 1 week ago

Principal Architect IV, Retail, Google Cloud

Chicago, IL $186,000.00-$261,000.00 2 weeks ago

Chicago, IL $168,000.00-$249,000.00 2 days ago

Principal Enterprise Architect - Hybrid Infrastructure

Chicago, IL $163,040.00-$244,560.00 1 week ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr