Azure DevOps Build AI Analyzer

An Azure DevOps extension that automatically explains why your pipeline failed using an LLM of your choice — including a local, self-hosted model (Ollama, LM Studio, vLLM, llama.cpp) or any OpenAI-compatible API.

When a build fails, a pipeline task reads the logs of the failed tasks, sends them to your LLM, and attaches a structured diagnosis (summary, root cause, key errors, suggested fixes) to the build. An AI Analyzer tab on the build results page then shows it.

No separate backend, no extra ports, no inbound network changes to your Azure DevOps server — everything ships inside the extension.

How it works

Pipeline fails
  └─> AIBuildAnalyzer task (last step, condition: failed()) runs on the agent
        ├─ reads the failed tasks' logs via the Azure DevOps REST API
        ├─ calls your LLM directly (agent → LLM)
        ├─ falls back to offline heuristics if the LLM is unreachable
        ├─ attaches the analysis JSON to the build
        └─ prints a direct link to the AI Analyzer report in the task log
AI Analyzer tab on the build results page
  └─ reads that attachment from Azure DevOps (same origin) and displays it

Why this design:

• The LLM is only ever called from the build agent, not the browser. So your model can live on a private network the browser can't reach.
• The tab only talks to Azure DevOps itself, so there is no cross-origin/SSL/mixed-content problem and nothing for users to configure.
• The task never fails your build — if anything goes wrong it logs a warning and exits 0.

Requirements

• Azure DevOps Services or Azure DevOps Server (2020+).
• An OpenAI-compatible chat-completions endpoint reachable from your build agents.
• tfx-cli to package the extension (npm i -g tfx-cli).

Install

1. Package the extension

npm install            # installs tfx-cli locally
npm run package        # produces <publisher>.<id>-<version>.vsix

If you are forking this for your own org, first set your own publisher in extension/vss-extension.json ("publisher"). Create one at https://marketplace.visualstudio.com/manage.

2. Upload it

Organization settings → Extensions → Manage Extensions → Upload new extension, then Install it into your project. (Or tfx extension publish --token <PAT>.)

3. Add the task to your pipeline

Add this as the last step of the job:

- task: AIBuildAnalyzer@2
  displayName: AI Build Analyzer
  condition: failed()
  inputs:
    llmUrl: http://localhost:11434/v1   # your LLM endpoint
    llmModel: llama3.1

That's it. From the next failed run onward, open the build's AI Analyzer tab, or use the direct Open AI Analyzer report link printed at the end of the task log.

A full sample is in examples/azure-pipelines.yml.

Use your own LLM

The task talks to any OpenAI-compatible /v1/chat/completions endpoint. Only llmUrl (base URL ending in /v1) and llmModel are required; every other input is optional. The build agent must be able to reach that URL.

Inputs

Sharper analysis with source context

By default the analyzer only reads the failure logs. Turn on readSource to let it also read the repo files the error points at (from the agent's checked-out sources — no extra network or credentials), so it can name the exact offending dependency/line instead of a generic suggestion:

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: http://localhost:11434/v1
    llmModel: llama3.1
    readSource: true          # read Dockerfile / package.json / etc. as extra context
    maxSourceBytes: '8000'    # keep small for small-context models

The source lookup tries exact paths first, then searches recursively (up to a bounded depth) for referenced filenames and common manifests. It skips .git, node_modules, bin, obj, dist, build, virtual environments, and other generated/dependency directories. Secrets are redacted before anything is sent, huge lockfiles are reduced to their suspect version lines, symlinks cannot escape the repository, and the total is capped by maxSourceBytes.

The complete request is independently capped by maxPromptChars. If the LLM returns an empty or incomplete JSON response, the task retries once with compact log/source context before falling back to offline heuristics. The task log reports prompt sizes, selected source filenames, response status, and finish_reason diagnostics without printing source contents or LLM response bodies.

For NuGet framework-compatibility failures (NU1701/NU1202), suggested fixes are also validated deterministically. Misleading PrivateAssets advice and unsupported application-retargeting suggestions are removed; warning suppression is labelled only as a temporary workaround because it does not make the package compatible.

When Publish Test Results fails with only a generic message, the analyzer also inspects a bounded set of preceding task logs from the same job. This catches test commands deliberately followed by exit 0, where the runner task succeeds even though its log contains [FAIL], the assertion message, and a stack trace. With readSource: true, local .trx and *.test.xml files under the checked-out sources are parsed as a second source of failure details. Only failed tests are included; coverage XML and passed results are ignored. Source files referenced by failed-test stack traces are prioritized over unrelated manifests.

Examples for common backends

Ollama (runs the model on the agent or a nearby host):

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: http://localhost:11434/v1
    llmModel: llama3.1            # or qwen2.5-coder, mistral, gpt-oss:20b, ...

LM Studio (local server, default port 1234):

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: http://localhost:1234/v1
    llmModel: your-loaded-model-name

vLLM / llama.cpp / text-generation-webui (OpenAI-compatible server):

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: http://your-gpu-host:8000/v1
    llmModel: meta-llama/Llama-3.1-8B-Instruct

Self-hosted HTTPS with a self-signed/internal certificate:

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: https://llm.internal.example:8443/v1
    llmModel: gpt-oss:20b
    insecureTls: true            # skips TLS verification (curl -k equivalent)

OpenAI / Azure OpenAI or any keyed API — store the key as a secret pipeline variable, then:

- task: AIBuildAnalyzer@2
  condition: failed()
  inputs:
    llmUrl: https://api.openai.com/v1
    llmModel: gpt-4o-mini
    llmApiKey: $(OPENAI_API_KEY)  # secret variable, never hard-code keys

Pick a model that fits the context. The task trims logs to the error-bearing sections, but very large failures still benefit from a model with a decent context window. For local use, a small instruct/coder model (7–20B) is usually plenty.

Heuristic fallback (no LLM)

If the LLM is unreachable or times out, the task falls back to a built-in, offline heuristic analyzer that recognizes common CI failures (npm/dependency errors, Docker build failures, TypeScript/C#/MSBuild errors, YAML syntax, failing tests, timeouts, auth/registry failures, non-zero exit codes). The tab labels the result ⚙️ heuristic analysis; LLM results are labeled 🤖 LLM analysis.

Repository layout

extension/
├── vss-extension.json          # extension manifest (tab + task contributions)
├── content/                    # the "AI Analyzer" build-results tab
│   ├── ai-tab.html / .js / styles.css
│   └── lib/SDK.min.js          # Azure DevOps Extension SDK
└── tasks/ai-analyzer/          # the AIBuildAnalyzer pipeline task
    ├── task.json               # task definition / inputs
    └── index.js                # self-contained Node (no npm dependencies)
examples/azure-pipelines.yml    # sample usage
test/                           # local smoke test + mock AzDO/LLM server

Local development & testing

The task is plain Node with no runtime dependencies. The local test suite covers log selection, failed-test parsing, recursive source discovery, prompt limits, LLM response handling, fallback behavior, and result formatting:

npm test

The tests use local fixtures and mocks only; they do not call your Azure DevOps instance or configured LLM endpoint.

To preview the tab UI, open extension/content/ai-tab.html in a browser (it shows the "no analysis" empty state outside Azure DevOps).

After changing anything under extension/, bump the version and repackage:

• extension/vss-extension.json → "version"
• extension/tasks/ai-analyzer/task.json → "version" (only if the task changed; agents cache the task by version)

npm run package

Troubleshooting

License

MIT © Mohammad Jafari