How a Luggage Brand Transformed Analytics with Automated Data Lakehouse

To track performance, one of India’s leading luggage brands relied on Excel sheets maintained across departments. The data was frequently duplicated or modified, leaving teams with conflicting versions of the numbers. By the time these files were consolidated for leadership reporting, the data was already days old.

Here’s how we replaced that chaotic reporting with an automated, single source of truth powered by a modern data lakehouse.

What's in this article:

• How the enterprise escaped data silos and established a single source of truth
• How we secured data pipelines for SAP HANA Rise cloud migration
• Cloud Platforms evaluated: Databricks on AWS vs Azure vs Microsoft Fabric
• How we implemented a medallion lakehouse with ADF and Unity Catalog
• How we estimated cloud compute costs based on operational schedules

The Challenge: Fragmented Data and Conflicting Numbers

The company tracks primary sales, billing variations, and warehouse inventory within its core ERP, SAP S/4HANA. Downstream, however, critical commercial data lived in disconnected silos: localized sales applications, partner portals, secondary distributor networks, and flat files distributed via email.

To assemble the leadership performance review, a dedicated analytics team manually downloaded, cleaned, and stitched together more than 60 separate Excel sheets. The biggest problem, however, was that the data could not be trusted. Business units pulled their numbers independently, leading to conflicting versions of core metrics.

How We Designed for SAP Migrations and Strict Network Security

The design had to respect hard operational realities beyond simple feature sets:

• Existing Identity and Security: The enterprise is standardized on Microsoft Office 365 and Microsoft Entra ID (Azure AD). The data platform had to plug directly into this identity layer out of the box to avoid forcing IT to maintain a second, isolated active directory.
• Surviving the SAP HANA Rise Migration: The brand was planning to move its core ERP from an on-premises data center to a cloud-hosted SAP HANA Rise environment. The ingestion pipelines had to be architected with a decoupled abstraction layer. By deploying on-premises SHIR VMs as a network bridge, we ensured that when the SAP system eventually cuts over to the cloud, the data lake pipelines will survive without a rewrite; it will require changing a single network connection string.
• The Private Networking Imperative: Security was a top priority. Financial ledgers, proprietary product volume metrics, and corporate pricing conditions had to be completely isolated from the public internet. Every stage of the data lifecycle—from extraction to transformation—had to live entirely within a locked-down virtual network (VNet), using private endpoints and dedicated secure subnets.

Cloud Paths Evaluated: Databricks on AWS, Azure, And Microsoft Fabric

We evaluated three environments to find the closest fit for the client's strict performance and security requirements: Databricks on AWS, Azure, and Microsoft Fabric.

Each option could have delivered a lakehouse. Azure Databricks won because it fit the client’s identity, security, scale, and timeline constraints without forcing a compromise elsewhere. Four factors decided it:

1. Native Fit with Microsoft Estate: Azure Databricks is a first-party Azure service, so it plugged directly into Entra ID, private networking, and Power BI. There was no second identity system to maintain and no cross-cloud networking or egress cost to absorb.
2. Control at Billion-Row Scale: Liquid Clustering, Z-Ordering, and exact VM sizing let us tune the heaviest ledger and pricing tables directly, instead of working around platform defaults.
3. Built-in Governance: Unity Catalog delivers row- and column-level security, column-level lineage, and auditing in one place, driven by the same Entra ID groups IT already manages. Achieving that elsewhere meant assembling it from separate services.
4. Open Format: Data sits as Delta tables on ADLS Gen2, an open format with no lock-in. The same platform also provides Delta Live Tables for pipelines, Genie for natural-language querying, and MLflow for future machine learning, so the next use case does not need a new tool.

How We Secured Network Control with Classic Compute

Serverless options work well for lighter workloads, but securing this enterprise data required total network control. We deployed Azure Databricks on Classic Compute to run clusters inside customer-managed Virtual Networks (VNets).

This gave the IT infrastructure team the ability to configure explicit network security groups (NSGs), route data through private endpoints, and establish direct, insulated connections back to SAP—all without exposing data nodes to the open web.

Beyond network isolation, this deployment model solved two architectural challenges:

1. Automated Governance via Entra ID: Azure Databricks authenticates natively against Entra ID using SCIM token synchronization. User groups created in the corporate network automatically mirror into Databricks' Unity Catalog. When IT adds a user to a corporate department group, that user instantly inherits the data access rules for that department.
2. Granular Control for High-Volume Scaling: Classic compute lets us select exact VM sizes for specific workloads. We provisioned memory-optimized nodes to handle heavy historical backfills and switched to cost-effective compute nodes for daily incremental processing.

The Medallion Lakehouse Architecture

The platform uses a structured medallion lakehouse architecture, organizing data into clear functional layers.

• Ingestion (Bronze): Azure Data Factory pulls data through the Self-Hosted Integration Runtime (SHIR) deployed locally in the network. High-volume ledger and pricing tables are handled via custom, timestamp-based delta extraction views to protect the live production ERP. To save on redundant infrastructure costs, we designed a Transient Inbound Landing Zone. Data lands as raw Parquet files, but native Azure lifecycle policies permanently purge these files after 14 days. Because Databricks streams this data incrementally into permanent Delta tables, the actual raw file directory stays clean, avoiding file fragmentation and saving massive storage costs.
• Transformation (Silver): Databricks Auto Loader monitors the landing zone and safely ingests data using its internal checkpoint state store. From there, Databricks jobs and pipelines take over. They handle type casting, normalize naming conventions across different sales channels, strip out null anomalies, and apply version-controlled business logic.
• Analytics Consumption (Gold): Refined tables are separated into distinct departmental schemas managed by Unity Catalog. Business users never connect directly to raw physical data. Instead, Power BI connects to optimized Gold semantic layers via DirectQuery. This setup allows corporate dashboards to refresh rapidly without making redundant local copies of the data, satisfying leadership's requirement for snappy, real-time performance tracking.

Built in Partnership with Databricks: As a certified partner, QBurst co-architected this environment alongside Databricks Solution Architects. By validating our Classic Compute VNet networking and tuning Liquid Clustering for billion-row tables against official reference patterns, we eliminated design risks and accelerated the path to production.

How Faster and Governed Insights Are Delivered

A single source of truth only matters if people can actually use it. The Gold layer supports two distinct consumption patterns, both reading from the same governed tables to guarantee consistent answers across the business.

• Standardized Reporting with Power BI: Power BI connects to the Gold layer through DirectQuery on a Databricks SQL warehouse, so dashboards query live tables rather than local copies. To maintain consistent metric logic, report authors query curated executive views rather than writing custom cross-schema joins. Z-ordered Gold tables and a right-sized warehouse keep response times low, and leadership gets the same dashboards on mobile.
• Natural Language Queries via Databricks Genie: Genie gives leadership a conversational layer over the curated Gold tables. Instead of raising a request with the analytics team every time a number needs a different cut, users ask questions in plain English, and Genie translates them into SQL. Because Genie runs on top of Unity Catalog, it only returns data the user is entitled to see. Its generated SQL executes on SQL warehouse compute, so it falls under the same sizing and cost discipline as the dashboards.
• One Security Model for Both: Access is managed through Entra ID groups mirrored into Unity Catalog, never through individual grants. Row-level security ensures channel heads only view their specific domains, while column-level masking protects sensitive financial fields. Defining these rules centrally guarantees that the same governance model secures both Power BI dashboards and Genie answers.

How We Modeled Cloud Costs for Predictable Budgets

Processing billions of rows daily requires serious compute power, making budget overruns a real threat. Instead of relying on generic sizing calculators, we ensured predictable pricing by modeling cloud costs directly against the client's actual operational schedules. We split the production footprint into three consumption buckets:

1. Batch Pipeline Compute: Modeled for nightly data engineering pipelines, assuming 150 cluster run-hours per month.  This cluster size is optimized specifically to handle the regular daily delta transformations once initial historical loads stabilize.
2. Reporting Compute: To keep Power BI dashboards responsive throughout the working day, we sized a dedicated SQL warehouse to cover a 240-hour monthly window (roughly 10 hours a day, 24 days a month) to align perfectly with core business operating hours.
3. Storage and Utility Base: Storage costs factor in physical capacity, transactional request volumes, and data redundancy, such as running a 4 TB hot-tier block storage bucket using Zone-Redundant Storage (ZRS) to guarantee high availability. Flat utility fees cover ADF pipeline runs, networking gateway uptimes, and secure Key Vault operations.

The Outcome: Timely Insights and Confident Decision-Making

By abandoning the chaotic spreadsheet workflow for a governed Lakehouse, the enterprise eliminated reporting latency and effort. 

Stop running your enterprise on fragmented spreadsheets. Reach out to us to engineer a secure, scalable single source of truth tailored to your operational realities.