A thin front end for the TruthSeek Eval Harness Python project. Open the harness folder as your workspace, then drive it from the Command Palette.
Commands
All commands are under the TruthSeek category:
| Command | What it does |
|---|---|
| Configure target model… | Pick a local server (Ollama, LM Studio, vLLM, llama.cpp), auto-discovers the models loaded there, and writes the target block into config.yaml. |
| Run eval (config.yaml) | Runs python -m harness.cli run --config config.yaml --cases cases/ in a terminal. |
| Run mock eval (offline) | Runs the offline mock suite — no model, no network. |
| Run strict gate | Runs with --strict, the same gate CI uses. |
| Compare models… | Runs the same cases against several models head-to-head. |
| Open scorecard | Opens the generated out/scorecard.html in a webview. |
Model discovery mirrors the harness's own logic: it tries the Ollama /api/tags endpoint
first, then falls back to the OpenAI-compatible /models listing. Every "run" command shells
out to the harness CLI, so results are identical to running it by hand.
Settings
truthseek.pythonPath(defaultpython) — the Python executable used to run the harness. Point it at a virtualenv interpreter if the harness is installed there.
Requirements
- The workspace is a TruthSeek harness checkout (has
harness/,cases/,config.example.yaml). - Python with the harness dependencies installed (
pip install -r requirements.txt).
License
MIT