From questions to SQL: an AI analytics platform on Databricks

In one line: Petabytes of e-commerce and customer data in a lakehouse became a place where people ask questions in plain English and get validated SQL, narratives, and alerts — in twelve weeks.

Context

A product organisation had rich e-commerce, marketing, and customer data in Databricks while many budget conversations still happened outside the warehouse. Teams needed answers before meetings, not exports afterward.

Why it was worth building

Dashboards alone did not close the gap between data volume and decision speed. Natural-language access only works on governed, validated SQL — otherwise trust collapses. The alternative was continued shadow spreadsheets and delayed escalations on stock and revenue issues.

What I built

• 10 production Streamlit dashboards — B2C paid media ROI, revenue across 14 markets, LTV/cohorts, out-of-stock impact, attribution, ML intelligence, notification centre; B2B dealer performance, pipeline, market scorecard.
• AI querying — GPT-4o via LiteLLM generates validated SQL against Databricks; results return with narrative explanation and follow-up suggestions; successful patterns saved as few-shot examples.
• Anomaly engine — z-scores vs 28-day baseline (warning |z|>2, critical |z|>3) on a schedule.
• Operational pipelines — 29 instrumented tables, ~150M+ rows under ~3s query; seven scheduled workflows with dependency checks; two production failures caught and fixed during audit.
• Prototype phase — started as a conversational analytics prototype (plain-English exploration UI) before hardening for production SQL and Streamlit.

How it was adopted

Built for product, marketing, and commercial stakeholders who needed self-serve answers without writing SQL. Anomaly cards and out-of-stock models drove supply-chain escalations when metrics diverged from baseline.

Effect

• 10 dashboards live across 14 European markets.
• 29 curated tables powering decisions.
• Teams had answers before meetings — not after.

Where AI fits now

Text-to-SQL is one lane in a broader pattern: scoped agents that query governed data, eval fixtures for when models drift, and human approval before anything customer-facing ships. The prototype proved the UX; production added validation, monitoring, and catalog boundaries.

Read the longer architecture write-up.