The Loop Doesn't Close Itself

The Copilot cloud agent finishes, opens a PR, and leaves it in draft. No signal about whether the code is good. No indication it ran the tests. No confirmation the type checker is clean. The PR just sits there, and you have no idea whether the implementation is worth looking at or whether you’re about to inherit a mess.

I saw this pattern enough times that I stopped treating it as a GitHub quirk and started treating it as a design problem. The agent isn’t broken — it’s doing exactly what it was built to do. The gap is in the feedback mechanisms around it.

For the ollama-mcp project — a Python MCP server that exposes locally-running Ollama models to Claude — I needed the agent to do more than ship a PR. I needed it to iterate until something external to itself confirmed the work was good. This post is about the feedback mechanisms I built to make that happen: the 4-agent dev loop, the CI gate hooks that feed failures back as context, and the setup-step investment that keeps the agent’s budget from evaporating on installs.

The starting condition

The Copilot agent session is iterative — you can steer it while it runs, send follow-up prompts, review diffs before the PR lands. The gap isn’t in the session model. It’s in the exit condition: the agent decides when it thinks it’s done, and nothing external validates that call.

Custom Agents narrow the brief — a scoped tool set, a tighter persona, a role-specific system prompt. There’s a second benefit that matters more in practice: context isolation. Each agent runs in its own context window. The explorer can scan the entire codebase without contaminating the developer’s working context. The developer’s full implementation doesn’t inflate the reviewer’s scoring context. Sub-agents can consume thousands of tokens exploring and then return a condensed summary — the orchestrator never sees the noise, only the signal.

But specialization and isolation together still don’t change how the session ends. A well-scoped agent with a clean context still exits when it decides it’s done, still leaves the PR in draft, still gives you no signal about whether the work is worth looking at.

The problem isn’t that the agent is wrong. It’s that you have no signal either way until a human goes and looks. In a codebase with a strict spec, strict typing, and a layered architecture, the surface area for quiet mistakes is large. The agent could satisfy the literal issue description while silently violating the design invariants that keep the codebase maintainable.

The only way to close that gap is to make the loop explicit — and automated.

The 4-agent dev loop

The core feedback mechanism is a dev loop with four roles: an orchestrator, an explorer, a developer, and a reviewer. Each role has a single job.

The orchestrator doesn’t write code. It parses the incoming task, extracts the actual problem statement (not just the requested action), checks for alignment with the project spec, routes work to the other agents, carries context between them, and decides when the loop terminates. If the task description contradicts the spec, the loop surfaces that contradiction before any code is written.

Before any implementation starts, the orchestrator invokes the explorer. The explorer is strictly read-only — it never modifies files. Its job is to map affected modules, locate the relevant spec section, identify reuse candidates, surface constraints, and flag open questions. The separation matters: if the explorer finds a contradiction between the issue and the spec, it stops the loop there, not after the developer has already built the wrong thing.

Once the exploration report is in hand, the orchestrator invokes the developer with the task description, the full explorer context, and — on iterations after the first — the reviewer’s list of blocking issues from the previous round.

The developer implements the change and runs task check before marking the work done. (task is Taskfile — the project uses it as a task runner for all dev operations.) task check runs format, lint, typecheck, and test in sequence. If any step fails, the developer fixes it before handing off to the reviewer. The reviewer never evaluates broken code.

Then the reviewer scores the implementation on a 1–10 scale against the original problem statement and the spec — not against a checklist of acceptance criteria. The core question the reviewer asks is whether the implementation actually solves the problem and respects the design, or whether it just satisfies the letter of the request.

Score 8 or higher exits the loop as APPROVED. Below 8, the reviewer returns two lists: Issues (blockers — things that must be fixed before the score can reach 8) and Suggestions (non-blocking improvements). The orchestrator passes only the Issues back to the developer. Suggestions are preserved in the review record but don’t drive the next iteration.

The iteration cap is 5. If the reviewer still scores below 8 after 5 rounds, the loop exits with MAX_ITERATIONS_REACHED and flags the PR for mandatory human review. In practice, the combination of the explorer’s upfront context and the specific blocking issues from each reviewer round means most tasks converge in 2–3 iterations.

The score threshold is the mechanism that makes this different from an unstructured handoff. Without it, the loop has no exit condition other than “the developer says it’s done.” With it, quality is quantified — the agent doesn’t stop until something external to itself agrees the bar has been met.

Whether an LLM’s numeric score is reliable matters less than what justifying it forces: the reviewer can’t return a sub-8 without specifying the Issues that block a higher one, and those blockers have to be concrete enough for the developer to act on in the next iteration. The score is the gate. The Issues list is the mechanism.

CI gate hooks

Between sub-agent handoffs, things can quietly break. A mypy violation introduced by the developer. A ruff error added mid-fix. These survive the reviewer loop because they’re mechanical, not logical — the reviewer is evaluating the implementation, not running the tool chain. The CI hooks catch them at the boundary, the moment an agent or sub-agent hands off control.

The configuration lives in .github/hooks/ci-gate.json. Two hooks:

• agentStop runs the full CI gate: format check, lint, typecheck, and the complete test suite.
• subagentStop runs a quick gate: ruff lint and mypy only, no tests.

(A note on naming: the hooks reference defines two equivalent formats. agentStop and subagentStop are the camelCase keys — they fire with camelCase payload fields. Stop and SubagentStop are the VS Code-compatible aliases for the same hooks, but the payload comes back in snake_case instead. The repo uses camelCase, which is the native format.)

The two-tier design is intentional. The 4-agent dev loop spawns multiple sub-agents per iteration — the developer and the reviewer — so subagentStop fires several times per loop iteration, while agentStop fires once when the orchestrator exits. Running the full test suite on every sub-agent stop would make the loop prohibitively slow.

The quick gate catches most mistakes fast — a ruff error or a mypy complaint — without paying the cost of running pytest on every sub-agent stop. The full gate runs when the whole session ends and everything has to pass.

The critical design decision is how failures are reported. Both scripts always exit 0. A non-zero exit would signal a hook infrastructure error, not a CI failure — those are different problems. Instead, failures are returned as structured JSON:

    Copy
    {
  "decision": "block",
  "reason": "CI gate failed. Fix all issues before finishing:\n\nsrc/ollama_mcp/tools/compare.py:87: error: Argument 1 to \"fan_out\" has incompatible type \"list[str]\"; expected \"Sequence[ModelRequest]\"\nFound 1 error in 1 file (checked 12 source files)"
}

The reason field carries the complete output. A ruff error includes the file and the specific rule violation. A mypy error includes the file, line, and the type mismatch. That output becomes the context the agent reads before its next action.

A silent block teaches the agent nothing. A block with the full error output is a prompt it can act on. The difference between those two outcomes is whether the feedback mechanism is honest about what went wrong.

The setup step investment

Every Copilot cloud agent session starts from a clean environment. Without pre-warmed caches, the cost shows up immediately:

• task lint — ruff runs cold
• task typecheck — mypy rebuilds its cache from scratch
• task test — every module re-imports

Each cold start is time the agent spends on infrastructure instead of the task. A full 5-iteration run means the dev loop executes the tool chain at least 5 times — and with the explorer, developer, and reviewer firing as sub-agents, subagentStop triggers across up to 15 agent exits. Mypy rebuilding its cache from scratch on each one is overhead that accumulates faster than the implementation work it’s supposed to validate.

The copilot-setup-steps workflow installs uv, Python 3.12, the project environment, and go-task, then deliberately runs ruff, mypy, and pytest collection before the agent ever starts.

Ruff and mypy populate per-file caches that persist across processes. When the agent runs its first lint or typecheck, it gets incremental results against only what changed — not a full rebuild.

Pytest collection is different: it doesn’t warm a shared import cache, but it surfaces module import errors before the agent ever runs task test, catching broken imports early rather than mid-iteration.

Gotcha: Two things must align for Copilot to pick this up. The workflow file must be named exactly .github/workflows/copilot-setup-steps.yml, and the job inside it must be named exactly copilot-setup-steps. Either one wrong and the warm-up doesn’t connect to the agent’s environment. A third requirement that’s easy to miss: the file must be on your default branch — putting it on a feature branch does nothing.

What this changes

Before these feedback mechanisms: assign issue → agent implements → PR in draft → wait. You open it not knowing whether it’s worth 20 minutes of review or a close-without-comment. That uncertainty is the actual cost — not the agent’s time, yours.

After: the explorer confirms the implementation doesn’t contradict the spec before any code is written. The developer runs task check before the reviewer sees anything. The reviewer iterates until the score hits 8, and documents what had to change to get there. The CI hooks enforce passing gates at every agent exit. By the time the PR lands in your inbox, something other than the agent itself has validated each of those claims.

The agent is no longer just a blind handoff. It iterates until something external to itself confirms the work meets the bar.

That’s the whole point. The agent didn’t become more capable. The feedback mechanisms around it became more honest about what “done” means.

The PR in draft was never the problem. The problem was that “draft” and “done” were indistinguishable, and nothing closed the gap between them. Building that gap into explicit feedback mechanisms — a scored review loop, a CI gate that returns context — is what makes the loop actually close.