Governed lakehouse and ML in production — Enterprise data platform

In one line: A global manufacturer went from fragmented sources and no ML in production to a governed lakehouse with hundreds of curated tables, production models, and support intelligence at scale.

Context

A global B2B and B2C manufacturer had data spread across more than 20 systems and no machine learning in production. Budget and product decisions needed a single, trustworthy data layer — not another export.

Why it was worth building

Without governed pipelines and catalogued tables, every new question restarted from raw sources. ML experiments could not ship because lineage, access, and refresh were unclear. The cost of inaction was slow decisions, duplicate work, and models that never left notebooks.

What I built

• Medallion lakehouse on Azure Databricks — Bronze → Silver → Gold with daily increments; 20 domain schemas and 20+ source systems (BigQuery, GA4, ads platforms, Dynamics 365, ServiceNow, Salesforce, e-commerce, and internal systems).
• Unity Catalog governance — column-level lineage, access controls, and documentation across 600+ curated tables (270+ tagged for discovery).
• 100 scheduled production jobs I built and operated across marketing ingest, support management, ML/AI, dealer portal analytics, and masterdata (including 1.46M product records and 71M attribute rows).
• Support intelligence — BERTopic across 1.46M+ cases (447 topic clusters), routing model at 93.6% accuracy, deflection knowledge base of 6,001 entries, plus scheduled AI analyst/editor/product-owner jobs on Databricks.
• ML in production — B2B/B2C churn models (95.1% val AUC on B2B), Prophet demand forecasting, cross-sell, RFM, attribution, and portal search signals; all versioned in 13 MLflow experiments.
• 17-page Streamlit app on Databricks SQL — B2C and B2B decision views, anomaly centre, and dealer portal analytics (5M+ sessions, 1.3M+ B2B orders).

How it was adopted

Domain experts shaped requirements and validated outcomes. I owned architecture, implementation, and day-to-day operations — monitoring failures, extending pipelines, and connecting outputs to Streamlit and downstream teams. Dealer visit and trade-in workflows fed structured data back into the lakehouse via n8n and lightweight apps.

Effect

• 100 production Databricks jobs and 201 notebooks in active use.
• 600+ governed tables under Unity Catalog.
• 1.46M+ support cases enriched for routing and product decisions.
• Churn, forecasting, and NLP outputs used in sales prioritization, support routing, and product backlog signals.

Where AI fits now

The same platform pattern extends naturally to natural-language analytics (validated SQL on governed data), agent-assisted data operations with scoped tools, and approval-gated automations that publish or act only after review. The throughline: trustworthy data first, then models and agents that can safely use it.