Shift-left tooling for LLM app development in VS Code.

LLM Watch helps you catch LLM observability and safety issues while writing code, not after incidents in production.

Current MVP Features

Detects baseline LLM SDK call sites in TypeScript and JavaScript:
- openai.chat.completions.create(...)
- anthropic.messages.create(...)
Adds diagnostics for uninstrumented LLM calls.
Provides CodeLens and Quick Fix to wrap calls in OpenTelemetry GenAI span templates.
Includes a command to generate a local OpenTelemetry ConsoleSpanExporter bootstrap file.
Adds guardrails diagnostics for:
- Unsafe prompt concatenation with likely user input.
- PII-like values in prompt-like fields (email, phone, card-like patterns).
Adds prompt management basics:
- Prompt file discovery (.prompt.yaml, .prompt.md)
- Prompt sidebar tree view
- Prompt variable preview command

Commands

LLM Watch: Wrap LLM call in OTel span
LLM Watch: Set OpenAI API Key
LLM Watch: Create OTel Console Bootstrap File
LLM Watch: Preview Prompt Variables

Settings

llm-watch.enabled
llm-watch.observability.enabled
llm-watch.guardrails.enabled
llm-watch.gpu.enabled
llm-watch.prompts.enabled
llm-watch.telemetry.enabled

Privacy & Telemetry

LLM Watch includes opt-in anonymous telemetry to help improve the extension. You are asked once on first activation; you can change the choice any time in Settings.

What is collected (when opted in)

Extension version, OS type (e.g. darwin)
Anonymous event counts: CodeLens shown, span wrapped, guardrail rule fired, error type
No user identifiers, no workspace paths

What is never collected

Your code, prompts, or LLM outputs
Model names, token counts, or API costs
API keys or any secrets
Personally identifiable information

Disabling telemetry

Set llm-watch.telemetry.enabled to false in VS Code Settings, or choose No Thanks on first activation.

Local Development

Install dependencies:

npm install

Build and test:

npm run release:check

Run in Extension Development Host:

Use Run LLM Watch Extension or Run LLM Watch (Isolated) from Run and Debug.

Smoke Test Files

Use these files in the repo to validate behavior quickly:

smoke/openai-unsafe.ts
smoke/anthropic-pii.ts
smoke/template.prompt.md

Packaging

Create a VSIX package:

npm run release:package

This produces a .vsix file you can install locally via:

VS Code Command Palette -> Extensions: Install from VSIX...

Status

This is an MVP release focused on shift-left observability + guardrails for LLM call sites. Additional roadmap features will be added in future releases.