Databricks Cost Optimization: 5 Commands You Run Directly from Cursor

Databricks costs can easily spiral out of control if you don’t track what’s running, how long it’s running, and how efficiently it’s running.

Here are five practical optimizations you can run directly from Cursor — no manual clicks in the UI, no guessing. Just code and CLI commands that give you answers immediately.

1. Find forgotten clusters

Unused clusters that continue running are the most common cost leak in Databricks. An interactive cluster sitting idle costs you money every hour.

The command:

databricks clusters list --output json | \
  jq '.clusters[] | select(.state == "RUNNING") | {cluster_name, cluster_id, state, start_time}'

What it does:

• Lists all active clusters
• Filters those in “RUNNING” state
• Shows name, ID, state, and start time

Follow-up action:

If you see clusters that have been running longer than expected, stop them:

databricks clusters delete --cluster-id <cluster-id>

Optimization:

Set auto-termination on all interactive clusters to max 30 minutes. Add this to your Terraform or databricks.yml:

clusters:
  - name: "interactive-cluster"
    autotermination_minutes: 30

2. Analyze Spark configurations for over-allocation

Many teams run Databricks jobs with default settings that allocate more memory and CPU than the job actually needs. You’re paying for over-allocation.

The command:

Create a Python script in Cursor that analyzes your job configurations:

# analyze_spark_configs.py
from databricks_cli.sdk import JobsService
from databricks_cli.sdk.api_client import ApiClient
import json

api_client = ApiClient(host=DATABRICKS_HOST, token=DATABRICKS_TOKEN)
jobs_service = JobsService(api_client)

jobs = jobs_service.list_jobs()

for job in jobs['jobs']:
    job_id = job['job_id']
    settings = job['settings']
    
    if 'new_cluster' in settings:
        cluster_config = settings['new_cluster']
        driver_type = cluster_config.get('driver_node_type_id', 'N/A')
        worker_type = cluster_config.get('node_type_id', 'N/A')
        num_workers = cluster_config.get('num_workers', 0)
        
        print(f"Job: {job['settings']['name']}")
        print(f"  Driver: {driver_type}")
        print(f"  Workers: {num_workers} x {worker_type}")
        print(f"  Autoscale: {cluster_config.get('autoscale', 'No')}")
        print("---")

What to look for:

• Jobs using large instance types (e.g., i3.xlarge) for simple transformations
• Jobs with fixed worker count where autoscale would be better
• Driver nodes that are larger than worker nodes (unnecessary in most cases)

Follow-up action:

Switch to smaller instance types for jobs that aren’t memory- or CPU-intensive. Use r5 for memory-intensive jobs and c5 for CPU-intensive ones. For ETL jobs with variable load, enable autoscaling:

{
  "autoscale": {
    "min_workers": 2,
    "max_workers": 10
  }
}

3. Identify databases with data spill in Delta Lake

When Spark can’t fit data in memory, it spills to disk — this is called “spill”. Spill is slow and expensive. If it happens frequently, you’re paying for longer runtimes and larger clusters than necessary.

The command:

Add logging to your Spark jobs to capture spill metrics. Then run this Spark SQL query from a notebook (via Cursor with remote SSH or by pushing the code to a notebook):

SELECT 
  job_id,
  stage_id,
  SUM(disk_bytes_spilled) as total_disk_spill,
  SUM(memory_bytes_spilled) as total_memory_spill
FROM 
  system.spark.metrics
WHERE 
  disk_bytes_spilled > 0
GROUP BY 
  job_id, stage_id
ORDER BY 
  total_disk_spill DESC
LIMIT 20;

What it does:

• Shows which jobs and stages spill the most data to disk
• Sorts by largest spill first

Follow-up action:

For jobs with significant spill:

1. Increase spark.executor.memory if it’s memory-intensive
2. Increase spark.sql.shuffle.partitions if it’s join- or aggregation-intensive
3. Enable Adaptive Query Execution (AQE):

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

4. Find unused tables

Unused data costs money in storage. Delta Lake tables that haven’t been read in months are candidates for archival or deletion.

The command:

Use the Databricks SQL History API to see which tables are actually being queried:

# find_unused_tables.py
from databricks_cli.sdk import SqlAnalyticsService
from databricks_cli.sdk.api_client import ApiClient
from datetime import datetime, timedelta
import json

api_client = ApiClient(host=DATABRICKS_HOST, token=DATABRICKS_TOKEN)
sql_service = SqlAnalyticsService(api_client)

# Fetch query history for the last 90 days
end_time = datetime.now()
start_time = end_time - timedelta(days=90)

queries = sql_service.list_queries(
    filter_by={'start_time': start_time.isoformat()}
)

# Extract table names from queries
tables_accessed = set()
for query in queries:
    # Parse SQL for table names (simple regex)
    # In production: use a SQL parsing library
    if 'FROM' in query['query_text']:
        # Simplified extraction
        parts = query['query_text'].split('FROM')[1].split()
        if len(parts) > 0:
            tables_accessed.add(parts[0].strip())

# List all tables in your catalog
all_tables = []  # Fetch from INFORMATION_SCHEMA.TABLES

unused_tables = set(all_tables) - tables_accessed

print(f"Unused tables (last 90 days): {len(unused_tables)}")
for table in unused_tables:
    print(f"  - {table}")

Follow-up action:

For unused tables:

• Archive to cheaper storage (S3 Glacier, Azure Cool Tier)
• Delete if no longer needed
• Document why they should be retained

5. Optimize Delta Lake file sizes

Delta Lake creates small files over time when data is written frequently. Small files = slower queries = longer runtimes = higher costs.

The command:

Create a script that identifies tables with too many small files:

# check_delta_file_sizes.py
from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

tables_to_check = [
    "catalog.schema.table1",
    "catalog.schema.table2",
    # Add your tables
]

for table_path in tables_to_check:
    delta_table = DeltaTable.forName(spark, table_path)
    
    # Get file details
    detail = delta_table.detail().collect()[0]
    num_files = detail['numFiles']
    size_in_bytes = detail['sizeInBytes']
    
    avg_file_size_mb = (size_in_bytes / num_files) / (1024 * 1024)
    
    print(f"Table: {table_path}")
    print(f"  Files: {num_files}")
    print(f"  Avg size: {avg_file_size_mb:.2f} MB")
    
    # Flag tables with average file size below 128 MB
    if avg_file_size_mb < 128:
        print(f"  ⚠️  Consider OPTIMIZE")
    print("---")

Follow-up action:

For tables with too many small files, run OPTIMIZE:

OPTIMIZE catalog.schema.table_name;

For large tables, use Z-ORDER to co-locate data that’s often queried together:

OPTIMIZE catalog.schema.table_name
ZORDER BY (date_column, frequently_filtered_column);

Automate:

Schedule OPTIMIZE jobs to run nightly for your most-used tables. Add to your Databricks Workflow:

tasks:
  - task_key: "optimize_tables"
    notebook_task:
      notebook_path: "/Shared/maintenance/optimize_tables"
    schedule:
      quartz_cron_expression: "0 0 2 * * ?"  # 2 AM every night

Summary

The five optimization tips:

1. Stop forgotten clusters — find and terminate unused resources
2. Right-size clusters — don’t pay for over-allocation
3. Reduce data spill — faster jobs = lower costs
4. Delete unused data — storage costs add up over time
5. Optimize file sizes — fewer files = faster queries = cheaper

All commands can be run directly from Cursor with Databricks CLI or by pushing Python scripts to notebooks via Git integration.

Next steps:

Set up a monthly routine where you run these checks. Build a simple dashboard showing:

• Number of active clusters
• Cost per job over the last 30 days
• Tables with the most spill
• Tables with the worst file sizes

Cost control isn’t a one-time effort — it’s a continuous process.