See what your DataFrame looked like at every step — not just the last
print().
Runs your pandas or PySpark script unmodified and lets you click any transformation line to see the DataFrame's shape, schema, and sample data at that exact point — with instant diffs against the previous step.
⚡ The Problem: Data Pipelines are Black Boxes
When writing data transformation pipelines in pandas or PySpark, data engineers and analytics engineers face persistent debugging pain points:
- Broken VS Code Debug Console: VS Code's standard debug console silently fails to print PySpark DataFrame evaluation calls (
df.show()ordf.printSchema()) due to process stream redirection bugs ([VS Code Python Issue #18239](https://github.com/microsoft/vscode-python/issues)). - Data Wrangler Variable Inspection Failures: Microsoft's Data Wrangler extension frequently throws variable evaluation errors when inspecting PySpark DataFrames from debug frames ("Could not retrieve variable... Please file an issue").
- Driver-Only Debuggers: Databricks' VS Code extension and standard Python debuggers only step through driver execution lines without showing per-transformation data deltas or row-loss metrics.
- Intrusive Logging Libraries: Community libraries like
pandas-logorpdlogrequire developers to manually add decorators, wrap pipeline functions, or modify source code — creating opt-in friction and risking code churn.
🚀 How It's Different
| Feature | DataFrame Lens |
pandas-log / pdlog |
Standard Debugger / Wrangler | Databricks VS Code |
|---|---|---|---|---|
| Zero Code Changes | ✅ Yes | ❌ No (requires decorators) | ✅ Yes | ✅ Yes |
| Inline Editor CodeLens & Gutter | ✅ Yes | ❌ No (terminal only) | ❌ No | ❌ No |
| Instant Row-Loss & Schema Diffs | ✅ Yes | ⚠️ Partial text log | ❌ No | ❌ No |
| Crash-Safe (Disk Streamed) | ✅ Yes | ❌ Memory lost on crash | ❌ Session dies | ❌ Session dies |
| PySpark + pandas Unified | ✅ Yes | ❌ pandas only | ⚠️ Broken PySpark console | ⚠️ Driver-only |
| Fully Offline & Privacy Safe | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Requires Cloud |
🔥 Features
- Zero-Code-Change Instrumentation: Click
▶ Run with Time-TravelCodeLens at the top of any.pyscript. Your code runs unmodified. - Stacked Inline CodeLenses: Chained transformation calls on a single line (
df.filter(...).groupBy(...).agg(...)) get stacked CodeLenses (👁 987 rows · 6 cols · −217 rows vs prev), each independently clickable. - Gutter Summaries & Hover Tooltips: See step-by-step row transformations right in the editor gutter.
- Rich Step Detail Panel:
- Schema delta table highlighting added, removed, and retained columns.
- Side-by-side sample data preview with key-based row tracking.
- Null-count delta table per column.
- Plain-English offline step explanation ("This
.dropna()dropped 217 rows (18% of input)"). - One-click Copy as Markdown button for PR descriptions or Slack updates.
- Pipeline Timeline View: Horizontal timeline card view rendering the full shape evolution across all pipeline steps at a glance.
- Past Run Time-Travel: Switch between past execution runs to compare today's run vs yesterday's run snapshot-for-snapshot.
🛠 Supported Stacks
| Library / Environment | Support Level | Safety & Sampling Guarantees |
|---|---|---|
| pandas | Full | Instant .head(N) sampling, cell string truncation, wide-column truncation |
| PySpark (Local Mode) | Full | Never calls .collect(), uses .limit(N).toPandas(), countApprox safety guard |
| PySpark (Cluster Mode) | Roadmap | Planned remote JSONL transport adapter |
🏁 Quick Start
- Install DataFrame Lens from the VS Code Marketplace.
- Open any Python file containing pandas or PySpark operations.
- Click the
▶ Run with Time-TravelCodeLens at the top of your file (or pressCtrl+Shift+P→DataFrame Lens: Run with Time-Travel). - Click through the inline CodeLenses on transformation lines to open step diffs and timeline views.
⚙️ Configuration Reference
| Setting | Default | Description |
|---|---|---|
timeTravelDebugger.sampleSize |
20 |
Maximum number of sample rows captured in each snapshot. |
timeTravelDebugger.maxExactCountRows |
100000 |
Estimated rows threshold for exact PySpark count vs approximate count. |
timeTravelDebugger.excludeColumns |
[] |
List of column names or globs to exclude from snapshots for privacy. |
timeTravelDebugger.instrumentedMethods |
(20 methods) | List of DataFrame method names to instrument. |
timeTravelDebugger.retainRuns |
10 |
Number of past execution runs retained on disk before automatic pruning. |
timeTravelDebugger.autoShowTimeline |
false |
Automatically open Timeline view after run completion. |
🏗 How It Works (Architecture)
The extension injects a lightweight Python bootstrap package (python/tt_bootstrap) via PYTHONPATH injection when running your script.
- Monkey-Patching: Before your script runs,
tt_bootstrapsafely monkey-patches DataFrame transformation methods (filter,merge,join,groupBy, etc.). - Stack Frame Inspection: Each patched method inspects
inspect.currentframe().f_backto capture the caller's filename, line number, and monotonic call order on that line. - Crash-Safe JSONL Tail-Watch: Snapshots are written incrementally as newline-delimited JSON (
.jsonl) to$TEMP/tt-debugger/{run_id}.jsonl. The TypeScript extension host tails bytes live viafs.watch, updating CodeLenses and Webviews in real time. If your script crashes halfway through, all snapshots up to that point remain safely viewable.
🔒 Privacy & Safety
- 100% Offline & Local: No data ever leaves your local computer.
- Safety Limits: PySpark data is sampled via
.limit(20).toPandas()— never invoking.collect()on full datasets. - Column Exclusion: Configure
timeTravelDebugger.excludeColumns: ["ssn", "credit_card", "*password*"]to ensure sensitive fields are never captured in snapshot files.
🗺 Roadmap
- Remote PySpark cluster transport adapter (Databricks / EMR / Synapse)
- Polars & Dask DataFrame instrumentation patchers
- Jupyter Notebook (
.ipynb) cell-level inline time-travel view
📄 License
MIT License. See LICENSE for details.