AI Writes, You Verify: A Documentation Review Pipeline for Skeptics

Last week I deleted a function that had been "documented" by a comment explaining a behavior the function hadn't had in three versions. The comment was confident. The function was gone.

This is the real failure mode of AI-generated docs: they can be fluent, plausible, and wrong. Not because the model is bad, but because no human verified what the text claims. The fix isn't to avoid AI. It's to build a checkpoint where the model drafts and the human signs off.

The Ownership Split

A model can summarize code, describe parameters, and turn commit messages into release notes. It cannot know why a decision was made, which edge cases are career-ending, or which comments are now dangerous.

My rule of thumb:

• The model drafts: API descriptions, usage examples, parameter tables, changelog bullets from git history.
• A human owns: security implications, business rules, architectural trade-offs, deprecation warnings, anything tied to customer promises.

The pipeline below makes that split explicit. It generates a draft, then forces a review issue with a checklist that separates the two categories.

The Pipeline

I run this as a GitHub Actions workflow on every merged PR that touches src/. It takes the diff, sends it to a language model with a strict output schema, and opens a documentation review issue.

Here's a condensed version of the workflow YAML:

name: docs-draft
on:
  pull_request:
    types: [closed]
    branches: [main]

jobs:
  draft:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Generate doc draft
        env:
          API_BASE: ${{ secrets.MONKEYCODE_API_BASE }}
          API_KEY: ${{ secrets.MONKEYCODE_API_KEY }}
        run: |
          git diff origin/main HEAD -- src/ > diff.txt
          python draft_docs.py diff.txt
      - name: Open review issue
        uses: actions/github-script@v7
        with:
          script: |
            const body = require('fs').readFileSync('review_body.md', 'utf8')
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Docs review: ${context.payload.pull_request.title}`,
              body: body
            })

The draft_docs.py script sends the diff to the model with a prompt that asks for three sections: What changed, Usage impact, and Needs human decision. The last section is left blank by the model unless it detects something ambiguous.

The Review Checklist

Each generated draft opens an issue with a table like this:

This isn't a formality. The checkbox item "Why the change matters" is the one most commonly ignored. When it's left blank, reviewers often investigate the actual business context — which is exactly what I want.

Why a Free Server Changes the Cost Equation

Running this on every PR can burn tokens quickly if you route everything through a paid API. That's where MonkeyCode's free model access becomes practically useful: it lets you run the same pipeline without monitoring your wallet every time a teammate merges a hotfix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode also offers a free server option, so you can host the workflow's API endpoint on infrastructure you control instead of embedding vendor calls directly into CI. I use it here to keep the diff processing inside my own network before the draft hits the issue tracker.

Is this the only way to get honest docs? No. But it lowers the barrier to asking the model to do the cheap 80% while preserving the expensive 20% for humans.

What the Model Still Gets Wrong

I've seen the model confidently document a parameter that was removed in a previous PR, purely because the diff included the removal but the commit message said "cleanup". The checkpoint caught it because the human reviewer had to tick the "Parameter descriptions" box against the actual signature.

Other failure modes:

• Legacy code: models infer intent from names that are misleading. timeout might actually mean retry_after.
• Deleted features: a diff doesn't explain why something was removed. Humans must decide whether to map it to a replacement.
• Internal jokes: commit messages like "magic number here" become release notes if you blindly trust them.
• Security context: a model may flag that a function uses eval, but only a human knows whether that function only receives internal constants.

None of these are reasons to abandon AI drafts. They're reasons to keep a human review step with teeth.

A Practical Proposal for Your Team

Start with one documentation target — say, your REST API reference. Wire up the pipeline above. Run it for a week. Keep a small log of how many changes to the model's draft the human reviewer made.

If the human changes fewer than 10% of the parameter descriptions, you can widen the model's authority. If the number is higher, you have a quick signal that your codebase's intent is encoded in places the model can't see.

That measurement is more valuable than any prompt tweak. It tells you where your documentation debt actually lives.

Who Should Not Use This

This workflow is overkill for a solo project with five public functions. It also doesn't help if your team already ignores documentation review issues; adding an AI draft to a process nobody reads just creates another ignored issue.

And if your codebase is so unstable that git diff between main and a PR routinely produces thousands of lines, then no draft pipeline will save you. Fix the branches first.

The Trust Boundary

Docs are a contract between code and the people who use it. A contract needs both a drafter and a signer. Let the model draft, but force the sign-off to happen where it can actually be questioned.

Your future self will thank you when a comment says something true for the first time in two years.