Six Months with Cursor on Databricks — Why Did I Wait on Skills?
“I had taught the agent how we work. I had not taught it how Databricks works.”
For six months I have used Cursor against a Databricks data platform — custom channels, runbooks, AGENTS.md, .cursor/rules. It worked. Governance held. Deployments stayed sane. But something was always slightly off: the agent guessed at CLI flags, bundle layout, and API shapes that I had already internalised.
The fix was not a smarter model. It was official Databricks agent skills — instructions the agent reads at the right moment for bundles, jobs, Unity Catalog, SQL, and the Python SDK. They had been available for a while. I just had not installed them.
What my setup already covered
Our repo rules handle how we work: dev targets before prod, validate before deploy, naming conventions, governance guardrails. That layer is non-negotiable and team-specific. No vendor skill replaces it.
What it does not cover is what Databricks expects today — DABs bundle structure, Lakeflow Jobs patterns, Genie and Unity Catalog semantics, serverless defaults. That knowledge lived in my head and in scattered docs, not in anything the agent could load on demand.
Skills, rules, and MCP — three layers
| Layer | Job | Example |
|---|---|---|
| Your rules | Team policy, repo conventions, workflow | validate before deploy, dev target only |
| Platform skills | Current Databricks patterns and CLI | DABs, jobs, UC, SDK workflows |
| MCP (optional) | Live platform data | query tables, inspect job runs |
Skills and MCP complement each other; neither replaces your rules. Skills teach the agent to write correct code. MCP lets it read live state — a separate decision with its own governance implications.
Installation took minutes
I installed a curated set rather than everything at once:
databricks aitools install --agents cursorSee the Databricks agent skills docs and the source repo for the full catalog. Skills land under ~/.cursor/skills/ (symlinked from ~/.databricks/aitools/skills/). My starting list:
- databricks-core — CLI, authentication, profiles
- databricks-dabs — bundle structure and deployment
- databricks-jobs — Lakeflow Jobs orchestration
- databricks-unity-catalog — governance, lineage, audit
- databricks-dbsql — SQL warehouse patterns
- databricks-python-sdk — SDK, Connect, REST API
- databricks-pipelines — Lakeflow Spark Declarative Pipelines
- databricks-serverless-migration — serverless compute defaults
- databricks-genie-agents — Genie spaces and agents
Start narrow; expand when a workflow keeps tripping the agent.
What changed in practice
The difference showed up in boring, high-friction tasks — exactly where AI assistance should earn its keep:
- Bundles and jobs — fewer wrong assumptions about
databricks.ymlstructure and job API fields - Unity Catalog — better questions about grants, lineage, and naming without me re-explaining UC every session
- Genie — more grounded suggestions instead of generic “ask your data” answers
- CLI syntax — less time correcting flags the agent invented from older docs
I still write the HSQ rules. The skills fill the platform gap I had been papering over with repeated prompts.
Start narrow, update often
If you are already running Cursor against Databricks with custom prompts: you are not doing it wrong. You are probably just missing the platform layer. Install databricks-core, databricks-dabs, and databricks-jobs first. Add MCP later, deliberately scoped. Keep skills current with databricks aitools update.
The best move I made was not a new prompt. It was turning on something that already existed.
Six months with AI against Databricks. A quarter of an hour with the right skills. I should have done it earlier.