Browse a local Spark/Hive warehouse the way Databricks Unity Catalog browses tables — and query it with a Databricks-style SQL editor — all without needing a running Spark/JVM process.
Features
- Catalog tree: Catalog → Database → Table → Column, read directly from your
spark-warehousedirectory layout (no Hive metastore / Derby DB required). - Schema viewer: click a table for a webview with its columns, types, nullability, partition keys, format (Parquet or Delta), location, and approximate row count.
- SQL editor: write real SQL against your tables —
SELECT * FROM sales.orders WHERE region = 'us'— and run it withCtrl+Enter/Cmd+Enter. Backed by DuckDB, which reads your Parquet files and Delta tables (_delta_logand all) directly. - Query results — pin & filter: pin columns (📌) to keep them stuck to the left while scrolling, and filter rows per-column with the inline text boxes under each header. Clear filters resets every column at once.
- Autocomplete: type
db.for real table names,table.(or an alias) for real column names, plus SQL keyword completion. - Works with both Parquet and Delta Lake tables, including partitioned ones (Hive-style
key=valuepartitioning is recovered automatically).
Requirements
- A local Spark warehouse directory (the default output of
spark.sql(...)/df.write.saveAsTable(...)in localpyspark/spark-shellsessions — typically a folder namedspark-warehouse). - No Java, JVM, or running Spark session needed to browse or query — everything reads the on-disk Parquet/Delta files directly.
Getting started
- Open a workspace folder that contains (or is near) your
spark-warehousedirectory, or click Add Warehouse Folder... (folder icon in the Spark Catalog view title bar) to point at one anywhere on disk. - Expand the tree to browse databases, tables, and columns.
- Right-click a table → New Query for Table to open a pre-filled
.sqlfile, or New SQL Query for a blank one. Files are created under<workspace>/query/. - Write SQL referencing tables as
database.table(matches the tree naming) and pressCtrl+Enter— results open in a panel beside the editor.
Configuration
| Setting | Description |
|---|---|
sparkCatalog.warehousePaths |
Paths to Spark warehouse directories (workspace-relative or absolute). If empty, auto-detects a spark-warehouse folder in each open workspace folder. |
Commands
| Command | Description |
|---|---|
| Spark Catalog: Refresh | Re-scan the warehouse(s), bypassing the schema cache |
| Spark Catalog: Add Warehouse Folder... | Pick a folder via a native dialog and add it to warehousePaths |
| Spark Catalog: Show Schema | Open the schema webview for a table |
| Spark Catalog: New SQL Query | Open a blank query file |
| Spark Catalog: New Query for Table | Open a query file pre-filled for a specific table |
| Spark Catalog: Run Spark SQL Query | Run the current selection (or whole file) — bound to Ctrl+Enter / Cmd+Enter in .sql files |
Known limitations
- SQL runs on DuckDB, not a real Spark session — the vast majority of everyday SQL (
SELECT/WHERE/JOIN/GROUP BY/aggregates/CTEs) behaves the same, but two Databricks/Spark-SQL habits don't carry over: backtick-quoted identifiers aren't supported (use double quotes), and double-quoted strings are always treated as identifiers, never string literals (use single quotes for string values). - Autocomplete uses a lightweight heuristic, not a full SQL parser — it won't follow CTEs or subquery-derived tables.
- Type mapping from Parquet/Delta to display names is best-effort, not a byte-perfect reproduction of Spark's
DESCRIBE TABLE.