AI Code Reviewer for Azure DevOps

Pull request reviews that learn from your team — not from generic internet advice.

AI PR Review is a free Azure DevOps extension that automatically reviews every pull request using patterns your team already established. It reads resolved PR comments from past merges, extracts the rules your reviewers actually enforced, and applies them to every new PR — inline, like a human reviewer.

📦 Install from Marketplace: marketplace.visualstudio.com/items?itemName=AISRE.pr-review

How it works

  Past PR (merged)              New PR (opened)
  ─────────────────             ─────────────────
  Reviewer: "Always use         PrReviewLearning@1 runs,
  HikariCP here. See our        matches learned rule →
  #best-practice guide."        posts inline comment:
  Dev fixes, merges.            "⚠ Use HikariCP — your
                                team's rule (4 observations)"
        ↓
  ingest task learns rule,
  stores it in your ADO org

1. Add PrReviewLearning@1 (mode: review) to your PR validation pipeline — one task, zero config beyond that.
2. Add the same task (mode: ingest) to your post-merge pipeline to learn new rules automatically.
3. Configure your Azure OpenAI endpoint once in Project Settings → PR Review.

That's it. No servers to operate, no data leaving your Azure tenant.

Features

• Inline comments on every PR — AI findings appear directly on the changed lines
• Learns from your history — rules come from your team's own resolved comments, not internet opinions
• #best-practice shortcut — tag any comment to immediately create a rule, without waiting for the AI to classify it
• Project groups — one rule can apply across a set of related repositories (e.g. all payment-service repos)
• Three review modes — full AI review, rule-based only, or disabled — choose per pipeline
• Four learning modes — strict (default), tagged-only, conversation, all
• Rule approval workflow — non-tagged rules require admin approval in Hub before affecting PRs
• Rule usage counters — triggered / posted / accepted / rejected raw counters (informational, no auto-judgment)
• Rule provenance & audit log — every rule shows its source PR comments and full change history
• Feedback-loop protection — ingest skips AI-posted comments to prevent self-learning
• API key encrypted at rest — AES-GCM with per-organization key
• Your data, your tenant — BYOK Azure OpenAI, storage in ADO Extension Data Service, zero external dependencies
• Free — you pay only for your own Azure OpenAI token usage (~$5–50 / month typical)

Requirements

Installation

1. Install the extension from the Visual Studio Marketplace (link active after public release).
2. Open Project Settings → PR Review and enter your Azure OpenAI endpoint and API key.
3. Add the tasks to your pipelines (see below).

Pipeline setup

1. Review pipeline — runs on every PR

Save as ai-review-pipeline.yml in your repo root, register it as a pipeline in Azure DevOps, then set it as a Branch Policy → Build Validation on main (set as Optional to give feedback without blocking merge):

trigger: none

pr:
  branches:
    include: [main]
  paths:
    exclude:
      - '**/*.md'
      - docs/*

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: Review
    timeoutInMinutes: 5
    steps:
      - task: PrReviewLearning@1
        inputs:
          mode: review

2. Ingest pipeline — auto-runs on merge

Save as ai-ingest-pipeline.yml. Auto-triggers on every merge to your target branch. Extracts the PR number from the merge commit message (Merged PR N: Title, ADO's default format for merge-completion) and runs ingest automatically. This captures #best-practice tagged rules AND updates Accepted/Rejected metrics on existing rules based on each bot-comment's final thread status:

trigger:
  branches:
    include: [main]
  paths:
    exclude:
      - '**/*.md'
      - docs/*

parameters:
  - name: pullRequestId
    displayName: 'Pull Request ID (leave empty for auto-detect from merge commit)'
    type: string
    default: ''

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: IngestRules
    timeoutInMinutes: 10
    steps:
      - bash: |
          PR_ID='${{ parameters.pullRequestId }}'
          if [ -z "$PR_ID" ]; then
            PR_ID=$(echo "$BUILD_SOURCEVERSIONMESSAGE" | grep -oE 'Merged PR [0-9]+' | grep -oE '[0-9]+' | head -1)
          fi
          if [ -z "$PR_ID" ]; then
            echo "No PR ID — skipping (not a standard merge commit)."
            exit 0
          fi
          echo "##vso[task.setvariable variable=System.PullRequest.PullRequestId]$PR_ID"
          echo "Ingesting PR #$PR_ID"

      - task: PrReviewLearning@1
        inputs:
          mode: ingest

Compatibility: works with ADO's default "Merge (no fast-forward)" completion option. Squash and rebase merges skip (no PR ref in commit). Manual pullRequestId param always works — use it to re-ingest historical PRs or bootstrap rules from pre-extension PR history.

How to register the pipelines in Azure DevOps

After saving the YAML files in your repo, each one needs to be registered as a pipeline in ADO:

1. Pipelines → New pipeline
2. Azure Repos Git → pick your repo
3. Existing Azure Pipelines YAML file → select /ai-review-pipeline.yml (or /ai-ingest-pipeline.yml)
4. Save (don't run yet)
5. Optionally rename (three-dot menu → Rename), e.g. myrepo-ai-review and myrepo-ai-ingest

Set review pipeline as Branch Policy (one-time)

So the review runs on every PR targeting main:

1. Project Settings → Repositories → [your repo] → Policies
2. Under Branch Policies, click on main (or your target branch)
3. Build Validation → Add
4. Build pipeline: select myrepo-ai-review
5. Trigger: Automatic
6. Policy requirement: Optional (recommended — AI review is advisory, not blocking). Pick Required only once you trust its precision.
7. Save

Grant Build Service permission to post PR comments (one-time, per repo)

The pipeline uses $(System.AccessToken) — ADO's Build Service identity — to post comments. This identity needs explicit permission. Without it, pipeline runs cleanly but no comments appear and the log shows 403 Forbidden — TF401027: PullRequestContribute.

1. Project Settings → Repositories → [your repo] → Security
2. Find the identity <project> Build Service (<org>) (e.g. aisre-services Build Service (aisre-org))
3. Set Contribute to pull requests → Allow
4. Optionally also set Contribute → Allow
5. Change saves automatically — no button needed

Repeat per repo where you want AI review to post.

When does each mode actually run?

Review — automatic, every PR

Once the Branch Policy is set up:

• Dev opens a PR targeting the protected branch → review pipeline triggers automatically
• AI analyzes the diff, applies learned rules, posts inline comments on the PR
• Dev pushes new commits → pipeline re-runs; already-posted comments are not duplicated (idempotent by file+line+rule hash)
• Nothing manual — set it once, forget it.

Ingest — automatic, at merge time

With the YAML above, ingest triggers automatically on every merge:

1. Reviewer leaves a comment on a PR (optionally tagged #best-practice to make it a rule immediately, or untagged — strict mode gates learning on Resolved + commit-touched)
2. Author fixes, merges the PR — ADO creates Merged PR N: Title commit on main
3. CI trigger fires automatically
4. Bash step extracts PR #N from commit message
5. Task runs ingest: captures #best-practice rules, updates Accepted/Rejected metrics based on bot-comment thread statuses
6. Done — no human action

Manual re-ingest: use Pipelines → ai-ingest → Run pipeline with pullRequestId param to bootstrap from historical PRs or after scope changes.

Task inputs

The #best-practice tag

If a reviewer writes a comment containing #best-practice, the ingest task classifies it as immediately addressable — no AI resolution analysis needed. The rule is extracted directly from the comment text and stored with full confidence. This is the fastest path from "we agreed on something in a PR" to "it's automatically enforced on the next PR".

Project Groups — scope rules across related repos

By default, a rule learned in one repository applies only to that repository. If your team works across multiple repos that share conventions, you can group them so a rule learned in one applies to all.

When do you need it?

• You have microservices split across multiple repos that share coding standards (payment-api, payment-worker, payment-gateway)
• A reviewer notes "we always use HikariCP for DB pooling" in one of them — without project groups, this rule applies only to that one repo. With a payments group, it applies to all three.
• Not needed if you have one repo or each repo has different standards.

How to configure

1. In ADO: Project Settings → AI Code Reviewer → Project groups
2. Click + New group
3. Fill in:
   • Name — e.g. payment-services
   • Description (optional) — e.g. "All services for the payments domain"
   • Repositories (one per line, format org/project/repo):

     myorg/MyProject/payment-api
     myorg/MyProject/payment-worker
     myorg/MyProject/payment-gateway
     

4. Save

From the next ingest run onwards, the AI classifier sees your groups and can scope a rule to a group instead of just one repo. Add a new repo to the group later → all group-scoped rules apply to it automatically, no re-ingest needed.

⚠ Pipelines are still per-repo

A project group shares rules across repos, not pipelines. Each repo in the group still needs:

• Its own ai-review-pipeline.yml in the repo root
• Its own pipeline registration in ADO (one-time, via New pipeline wizard)
• Its own Branch Policy → Build Validation set on main

Same for ingest — each repo needs its own ai-ingest-pipeline.yml to run ingest on a merged PR from that repo. The group controls how rules match; the pipelines control where reviews run.

A practical setup for a 3-repo group: 6 pipelines total (review + ingest × 3 repos), all sharing the same learned rules store via the extension's per-organization data service.

Scope hierarchy

The AI picks the narrowest scope that fits each rule:

You can edit any rule's scope later in the Learned Rules tab if AI classified it too narrow or too broad.

Enterprise governance — learning mode, provenance, audit log

Learning mode — control which comments become rules

Set in Project Settings → AI Code Reviewer → Azure OpenAI → Learning mode. Four strictness levels:

Default is Strict. Non-tagged rules require both explicit human signals:

1. Thread marked Resolved in PR UI
2. Commit actually changed the commented line

Non-tagged rules are always PROJECT scope — they can't accidentally become cross-repo policies. To make a global rule, use #best-global-practice tag explicitly.

Rule provenance — click Details to see source PRs

Every rule in Learned Rules tab has a Details button. Expanding shows:

• Provenance — the exact PR comments that created or reinforced this rule (reviewer, date, file path, link to PR)
• Audit log — chronological history of every change to this rule

Audit log — who changed what

Every state change is recorded:

• Actor — system:ingest (PR #42) or user:<display name>
• Action — created / merged / confidence_changed / active_toggled / edited / deleted
• Before/after — only changed fields (compact footprint)
• Note — human-readable description

Stored sharded (~100 events per document) in Extension Data Service. Zero external infrastructure.

Privacy & security

AISRE Labs never sees your code, your diffs, or your PR comments. The pipeline task runs as a standard Azure Pipelines job authenticated by $(System.AccessToken). All network calls go to dev.azure.com/{your-org}/… and {your-resource}.openai.azure.com/….

Full details: aisre-labs.github.io/pr-review/privacy

Cost

Troubleshooting / FAQ

Pipeline runs but no comments appear on the PR — log shows 403 Forbidden — TF401027: PullRequestContribute

Full error:

ADO postThread failed: 403 Forbidden — TF401027: You need the Git 'PullRequestContribute'
permission to perform this action. Details: identity 'Build\<guid>', scope 'repository'.

Cause: The pipeline posts comments using $(System.AccessToken), which maps to your project's Build Service identity. By default, this identity does NOT have permission to comment on pull requests. You need to grant it once per repo.

Fix:

1. Project Settings → Repositories → [your repo] → Security
2. Find <project> Build Service (<org>) in the identity list (e.g. aisre-services Build Service (aisre-org))
3. Set Contribute to pull requests → Allow
4. Optionally also Contribute → Allow
5. Change saves automatically — retry the pipeline (empty commit push or manual run)

Pipeline log shows old extension version even after updating in Installed Extensions

If the pipeline log shows an older [pr-review] extension v0.1.X than what Installed Extensions reports, ADO's task cache can serve a stale bundle until the next task version bump. Force a refresh:

1. Organization Settings → Extensions → Installed → AI Code Reviewer → Uninstall
2. Shared tab → AI Code Reviewer → Install

Uninstall does NOT delete your Extension Data Service (rules, config, metrics, audit log persist). It only clears the bundled task code cache.

Ingest doesn't auto-run after merge

ai-ingest-pipeline.yml auto-triggers on master pushes AND extracts the PR number from the merge commit message (format Merged PR N: Title). Two common reasons it doesn't fire:

• ADO Pipeline not registered — every YAML in the repo must be saved as a pipeline (Pipelines → New pipeline → Existing YAML) before ADO watches it for triggers.
• Non-standard complete option — if your PR was completed with Squash or Rebase, the commit message doesn't contain Merged PR N. The pipeline skips silently. Either switch the PR complete option to Merge (no fast-forward), or run ingest manually with the pullRequestId parameter.

Rule usage counters stay at 0 even though comments are being posted

Counters live in Extension Data Service and are bumped by the task. Checklist:

1. Review pipeline is actually running (check its log history)
2. Log shows matched=N ruleCitations=M at end (new format) — not just files=X batches=Y posted=Z. If old format, see stale extension issue above.
3. For Accepted/Rejected specifically: they only update when the ingest pipeline runs on a merged PR. If you only see Triggered/Posted updating but never Accepted/Rejected, your ingest pipeline is not auto-triggering (see above).

AI posts inconsistent number of comments across re-runs of the same PR

temperature=0 with a stable seed makes output ~deterministic, but Azure OpenAI's determinism is best-effort. Small variance (0 or 1 extra comment) is possible, especially if the model's system_fingerprint changes between runs. Empirically: >90% of re-runs on the exact same PR produce identical output.

For true "same PR = same output" guarantee, Azure OpenAI does not currently offer a stronger contract than seed.

Rule applied across projects flags borderline cases

This is the known recall/precision tradeoff with #best-global-practice tags. The rule matches any file of the given LANGUAGE — context-specific exceptions (e.g. a derived Spring Data query that's naturally bounded) still get flagged. Options:

• Click Won't Fix on the individual comment — bumps Rejected counter on that rule (rule admin can then decide whether to deactivate)
• Click Closed (By Design) — also counts as rejection under current implementation (same effect)
• Edit the rule text in Hub to include explicit exclusions

Support & feedback

• Bug reports / feature requests: GitHub Issues
• Security vulnerabilities: [email protected] (do not open a public issue)

License

Proprietary — see LICENSE.
AI prompt templates are trade secrets of AISRE Labs. Reverse engineering is prohibited.