SI Search

A Source Insight-style code search extension for VS Code, designed for C/C++ developers working on large-scale codebases.

SI Search builds a symbol index using tree-sitter parsing and stores it in an on-disk SQLite FTS5 database. This lets symbol lookup stay instant even on workspaces as large as the Linux kernel (70k+ files, tens of millions of symbols), without holding the index in the extension host's JS heap. When the index has no match for a query, SI Search falls back to ripgrep full-text search seamlessly.

Features

Symbol Index (Sync)

SI Search uses tree-sitter (WASM) to parse all C/C++ source files in your workspace and extract symbol definitions:

• Functions — int main(), void foo()
• Structs — struct device
• Enums — enum state
• Typedefs — typedef unsigned int uint32_t
• Macros — #define MAX_SIZE 1024, #define INIT(x) ...
• Classes (C++) — class Widget
• Namespaces (C++) — namespace std
• Unions — union data

The index is persisted to disk as a SQLite database at .sisearch/index.sqlite in the workspace root and reopened instantly on VS Code restart—no re-scanning needed.

Press Ctrl+Shift+S or click the sync button in the Search panel title bar to build/update the index.

Two-Tier Search Strategy

1. Index search (instant) — When the symbol index is ready, queries run against the SQLite FTS5 database. Exact name lookup (whole word) uses a B-tree index; substring search uses FTS5 MATCH + LIKE/GLOB for case-insensitive/case-sensitive filtering; regex extracts literal tokens for FTS5 pre-filtering and then runs RegExp.test in JS.
2. Ripgrep fallback — If the index has no results (e.g., searching for a string literal rather than a symbol name), SI Search falls back to ripgrep full-text search.

This hybrid approach gives you the speed of a pre-built index with the coverage of full-text search.

Search Results Panel

• Virtual scrolling + pagination — Results render through a virtual-scroll list and load incrementally (200 rows at a time by default) so 10k+ results stay responsive.
• Find in results — Press Ctrl+F (Cmd+F) inside the results panel to open a find widget styled like VS Code's native editor find. It searches within the currently loaded results and supports Match Case (Alt+C), Match Whole Word (Alt+W), and Use Regular Expression (Alt+R) toggles. Enter / Shift+Enter cycle through matches, F3 / Shift+F3 work too, Esc closes the widget.
• Syntax-highlighted preview — Hover over any result's code portion to see a multi-line preview with full syntax highlighting (powered by shiki), matching your current VS Code color theme.
• Jump to source — Click the arrow icon on any result to open the file at the exact line.
• Result navigation — Step through results one by one with Ctrl+Shift+F4 / Ctrl+Shift+F3.
• Gutter indicators — Source files that contain search results show blue triangle markers in the editor gutter.
• CodeLens links — "Jump to Search Result" CodeLens appears on matched lines, allowing quick navigation back to the results panel.

Search Filters (Include / Exclude)

Click the ⋯ button next to the regex toggle to reveal files to include and files to exclude input fields — just like VS Code's native search.

• Files to include — Comma-separated glob patterns (e.g. *.c, src/**). When specified, these replace the includeFileExtensions setting for that search.
• Files to exclude — Comma-separated glob patterns (e.g. **/test/**, **/build/**). These are merged with the excludePatterns setting.
• Press Enter in any of the three inputs (search, include, exclude) to trigger the search.

Manual Highlights

• Press Ctrl+Shift+F8 to highlight the selected text (or trigger a selection prompt in the results panel).
• Multiple highlight colors cycle automatically.
• Highlights appear both in the results panel and in all open editors.
• The Highlights tree view in the sidebar shows all active highlights with remove buttons.

Incremental File Watching

SI Search monitors your workspace for file changes:

• Modified/created files are marked as dirty; the status bar shows "stale".
• Deleted files are removed from the index (rows delete from files; FTS5 entries drop via ON DELETE CASCADE + delete trigger).
• Re-sync (Ctrl+Shift+S) only re-parses changed files, not the entire workspace.

Sync-Time Search Behavior

Search requests that arrive while Sync is in progress can behave four different ways; configure with siSearch.search.duringSyncBehavior (see Configuration).

Architecture

                                     ┌─ Worker Pool (N × parseWorker, tree-sitter WASM)
 File System ──▶ SyncOrchestrator ──▶│
                                     └─ DbWriterClient ──▶ dbWriterWorker (owns write handle)
                                                                    │
                                                                    ▼
                                                          .sisearch/index.sqlite
                                                          (SQLite WAL + FTS5)
                                                                    ▲
                                           DbBackend (readonly handle)
                                                                    │
 User query ──▶ SearchEngine ──▶ SymbolIndex ──▶ DbBackend.search ──┘
                      │
                      └─ (no index hits) ──▶ ripgrep fallback

Key components:

Commands

Configuration

All settings are under the siSearch.* namespace in VS Code settings.

Example .vscode/settings.json

{
    // Only index files under these subdirectories (empty = entire workspace)
    "siSearch.includePaths": ["src/wifi", "src/drivers"],

    // File extensions for search and indexing
    "siSearch.includeFileExtensions": [".c", ".h", ".cpp", ".hpp"],

    // Glob patterns excluded from all searches
    "siSearch.excludePatterns": ["**/build/**", "**/.git/**", "**/node_modules/**"],

    // Keep the ripgrep fallback prompt; don't auto-trigger grep during sync
    "siSearch.search.duringSyncBehavior": "prompt-cancel",

    // Bigger pages for large workspaces
    "siSearch.search.maxResults": 500
}

Search Options

The search input supports three toggle options:

• Case Sensitive (Aa) — Match exact letter casing. Substring mode uses SQLite GLOB (ASCII-binary); whole-word mode uses plain = equality.
• Whole Word (W) — Match the whole symbol name only. Uses B-tree equality against symbols.name rather than FTS5 (FTS5's unicode61 tokenizer keeps underscore-joined C identifiers intact, which would make sub-token matches silently miss).
• Regex (.*) — Interpret the query as a JavaScript regular expression. Literal tokens (length ≥ 2 alphanumeric runs) are extracted from the pattern and used as an FTS5 pre-filter, then RegExp.test runs on the candidate names.

Sidebar Views

SI Search contributes two views to its own activity bar container:

Status Bar

The status bar item (bottom-left) shows the current index state:

How It Works

Symbol Parsing

SI Search loads tree-sitter WASM grammars for C and C++ at runtime. For each source file, it runs tree-sitter S-expression queries to extract symbol definitions:

;; Example: extract function names
(function_definition
  declarator: (function_declarator
    declarator: (identifier) @name)) @def

Each extracted symbol records: name, kind, file path, line number, column.

Large/generated C headers (above siSearch.parser.maxFileSizeBytes) bypass tree-sitter WASM and go through a regex-based streaming extractor instead. This path is memory-flat (line-by-line readline) and still feeds the SQLite index, so macros from multi-megabyte register headers are fully searchable.

SQLite FTS5 Index

The index lives in {workspaceRoot}/.sisearch/index.sqlite as a SQLite database with:

• meta — schema version, created-at timestamp, workspace root, tokenizer name.
• files — one row per indexed file (relative_path UNIQUE, mtime_ms, size_bytes, symbol_count). Indexed on relative_path.
• symbols — one row per symbol (name, kind as enum int, file_id, line_number, column). Foreign key to files with ON DELETE CASCADE. Indexed on file_id and name.
• symbols_fts — FTS5 virtual table using unicode61 remove_diacritics 2 tokenizer. Stored as content='' (contentless FTS5 — only the inverted index, half the disk footprint).
• Two triggers on symbols keep FTS5 in sync:
  • AFTER INSERT: INSERT INTO symbols_fts(rowid, name) VALUES (NEW.id, NEW.name)
  • AFTER DELETE: INSERT INTO symbols_fts(symbols_fts, rowid, name) VALUES ('delete', OLD.id, OLD.name) (contentless-FTS5 delete protocol).

Pragmas at open:

journal_mode = WAL        crash-safe, readers don't block writers
synchronous  = NORMAL     WAL-safe, ~3× faster than FULL
cache_size   = -65536     64 MB SQLite page cache
temp_store   = MEMORY     keep temp tables off disk
foreign_keys = ON         enforce the CASCADE delete on files

The writer-worker additionally sets synchronous=OFF and a larger cache (-262144, 256 MB) for Sync throughput; readers keep NORMAL.

Concurrency: Writer Worker + Main-Thread Reader

SQLite write locks are exclusive, so SI Search isolates writes in a dedicated worker_threads worker (dbWriterWorker.ts) that owns the only write connection. The main thread holds a readonly connection (DbBackend with {readonly:true, fileMustExist:true}) for search queries.

During Sync:

1. SyncOrchestrator classifies files (new, dirty, deleted, unchanged).
2. Parse jobs go through WorkerPool; each completed batch is a ParseBatchResult posted back to the main thread.
3. onBatchResult forwards the batch to DbWriterClient.postBatch, which sends it to the worker.
4. The worker runs one db.transaction(() => { insertSymbol × N; upsertFile × M; deleteFileByPath × K })() per batch.
5. FTS5 triggers keep the inverted index consistent inside the same transaction.

Back-pressure: DbWriterClient.awaitBackpressure(hwm=20) polls the in-flight batch counter with a 20 ms sleep when pending > 20; this keeps the worker's message queue bounded so the terminating drain/checkpoint don't sit behind a flood of batches.

At Sync end the orchestrator awaits drain() then checkpoint() (both with a 60-second safety timeout, so a stalled worker can't lock the UI on "Saving Index").

Pagination & Virtual Scrolling

executeSearchWithIndex returns {results, totalCount}:

• results is DbBackend.search(query, options, {limit, offset}) (paged, bounded to maxResults per page).
• totalCount is DbBackend.countMatches(query, options) (a single COUNT(*) against the same WHERE).

The webview receives a showResults for page 0 and posts loadMore as the user scrolls near the bottom. The extension replies with appendResults carrying the next page. Both messages include loadedCount / totalCount so the results panel's footer label stays consistent. When the user scrolls past the loaded region, a bottom spacer reserves the row height for not-yet-loaded rows so the scrollbar thumb reflects the true total.

Incremental Sync

During sync, SI Search compares each file's mtime_ms and size_bytes against files. Only new, modified, or deleted files are processed; deletions cascade to symbols (and, via the trigger, to symbols_fts). This makes re-syncing a large codebase (e.g., Linux kernel) take seconds rather than minutes.

Sync-Time Search UX

When a search is dispatched while SymbolIndex.isSyncInProgress() returns true, executeSearchWithIndex branches on siSearch.search.duringSyncBehavior:

• cancel — return no results silently.
• grep-fallback — silently run ripgrep on the workspace.
• prompt-grep-fallback / prompt-cancel — show a VS Code information message with "Use Grep" / "Cancel" buttons. A 1-second dedup window suppresses repeats.

Resilience

• Corrupt database — at open, DbBackend.openOrInit runs PRAGMA quick_check. Failures move the file aside as .sisearch/index.sqlite.corrupt-<ts> and re-initialize a fresh schema.
• Schema version skew — a newer on-disk schema than the extension expects throws DbSchemaTooNewError with a descriptive message. Older schemas are silently rebuilt.
• Missing native addon — if better-sqlite3 fails to load (e.g. no prebuild for this platform/arch), the extension still activates: siSearch.nativeOk context key becomes false, Sync/Clear commands hide, and all searches automatically route to ripgrep.
• Legacy .sisearch/shards/ — old msgpack shards from pre-SQLite builds are removed on activation.

Diagnostic Logging

Set SISEARCH_WORKER_DIAG=1 before launching VS Code to record a JSON-Lines trace of the writer worker's IPC channel (postBatch / ack / drain / checkpoint / timeout) plus every search query on the main thread. Logs go to $TMPDIR/sisearch-writer-<pid>-main.log and -worker.log. After a crash or hang:

ls -t /tmp/sisearch-writer-*.log | head -2 | xargs tail -40

The diagnostic path is a compile-time no-op when the environment variable is unset (cheap process.env read), so production use is unaffected.

Requirements

• VS Code 1.85.0 or later.
• better-sqlite3 native addon. Prebuilt binaries for Linux / macOS / Windows × x64 / arm64 are shipped in the VSIX; if your platform isn't covered, the extension falls back to ripgrep-only mode with a status-bar warning.
• The ripgrep binary and all tree-sitter WASM grammars are bundled.

Build from Source

Prerequisites

• Node.js 18+ and npm
• VS Code 1.85.0+
• A C/C++ toolchain for better-sqlite3 to compile against (node-gyp; Python 3.8+; a working gcc/clang/MSVC). Only needed when no prebuild matches your Node/Electron ABI.

Install Dependencies

npm install

Rebuild Native Addon for VS Code (Electron) ABI

VS Code runs extensions inside Electron, which has a different Node ABI than plain Node. Before F5 debugging or packaging:

npm run rebuild-electron

To run node-only unit tests (see below), rebuild for plain Node instead:

npm rebuild better-sqlite3

Compile

npm run compile

This runs tsc -p ./ to compile TypeScript to out/.

Watch Mode (for development)

npm run watch

Run Tests

Node-runnable unit tests:

npm run compile
npx mocha --ui tdd out/test/suite/*.test.js

Host-only integration tests (launch a VS Code instance):

npm run test:host

Package as VSIX

npx @vscode/vsce package

Install VSIX

code --install-extension sisearch-<version>.vsix

Or in VS Code: Ctrl+Shift+P → Extensions: Install from VSIX...

Known Limitations

• Symbol indexing currently supports C and C++ only. Other languages fall back to ripgrep full-text search.
• The hover preview renders code using the current VS Code theme via shiki. Some custom themes may not render perfectly.
• The .sisearch/ directory is created in the workspace root. Add it to your .gitignore if needed.
• Multi-window concurrent Sync against the same workspace is not specifically handled; SQLite's BUSY timeout kicks in and one window will surface an error. Sync from one window at a time.
• regex searches on patterns with no literal token (e.g., .+) fall back to scanning up to 10,000 symbol names and filtering in JS.

License

MIT