Part 2 of 2 in the Pencils Down series. Part 1 is about the review machine; this one is about the reviewer.
The review machine from Part 1 answers one question well: is this code right? Agents write the code, and a stack of gates checks it: the spec, the plan, each task, the whole branch, and sometimes a second model. Nothing reaches my editor until it has passed the review that applies to it, and nothing merges without my confirmation. I stand behind every gate in that pipeline.
It also has a hole in it, and the hole isn’t in the gates. It’s me.
The machine verifies the artifact. It says nothing about whether I got better while verifying it. In the 2025 Stack Overflow survey, among developers who answered the question about AI frustrations, 20% said they’ve become less confident in their own problem-solving, and 16% said it’s hard to understand how or why the AI’s code works. Those numbers aren’t about review, but they point in the same direction.
Verifying and getting better felt like the same thing until Chelsea Troy’s talk at Craft Conference, covered by InfoQ, gave me words for the difference. Troy is a Senior Staff Engineer at Mozilla, where she leads machine learning operations, and she teaches in the University of Chicago’s Master’s Program in Computer Science.
Her argument, as InfoQ summarizes it: software engineering has two modes, figuring stuff out and doing stuff, and using tools to speed up the outcome often doesn’t support skill development. Here’s my one-line version:
AI can speed up the doing. It doesn’t build my ability to figure things out.
Verifying isn’t learning
For me, verifying counts as “doing stuff.” Tools speed that mode up, but they often don’t help with figuring stuff out, and learning is inefficient by design. That inefficiency is the mechanism, not a flaw:
We have to sit with something that we feel bad at, and we have to struggle with it. It is that uncomfortable emotion, the confusion, the frustration, that is necessary for us to build cognitive capacity.
In a follow-up interview, she uses the toy compiler she assigns her students. Nobody in that class needs a compiler:
Sure, Claude can write the compiler, but that completely misses the point, because the point was never the compiler.
The compiler is a delivery device for design questions. Generate the code and you end up with an artifact nobody learned from. In the same interview, she calls the broader failure conflating “work completed” with “learning objective accomplished.”
My implementation review has the same shape as that classroom, and it took me a day to admit it. I still make real decisions at the spec and plan gates. But once implementation starts, the agent ships the artifact, and I verify it, judge it, and merge it. Every one of those steps is “doing stuff.”
If I’d written the code myself, I’d have worked out the design space, the rejected alternatives, and why this shape and not another. Now the implementer subagent (the agent that writes the code) does that, and I mostly see the result. “The code works” is my compiler: the deliverable standing in for the learning.
In AI-assisted review, the deliverable standing in for the learning looks like this:
Explain-first review feels like diligence and works like reading a transcript. You come out knowing what the code does, which is the one thing the code already knew.
Guess first, then look
What explain-first review skips is the guess. Troy covers three concepts: outcome prediction, format diversification, and spaced repetition. I’ll get to the other two below. The one that changed my review habit is outcome prediction: you learn something better if you try to guess it before you see it. Her protocol, from InfoQ’s write-up of the talk:
As you are learning how something works, you can just stop at each step and ask yourself, “What do I think the next step should be and why, given what I know right now?”
As I read it, the guess isn’t a warm-up for the learning. It’s where the learning happens: the rep that makes me build a mental model good enough to be wrong in an interesting way. Skip it and go straight to the answer, and you’ve collected the fact without building the model.
A student reading a compiler textbook and a reviewer reading a diff are in the same position, because the answer arrives before any guess. Opening the diff before I’ve guessed does the same thing. The agent only makes skipping easier.
What I actually changed
The fix was smaller than I expected. I didn’t add gates to the machine. I changed the order of operations inside the one gate I already own: mine.
The guess happens before the diff, in writing. When a branch lands in my review queue, I don’t open the diff yet. I open the spec and write three or four sentences in the review notes: the modules I’d touch, the interfaces I’d route through, and the failure mode I’d watch. Three minutes. It feels ceremonial exactly once. After that it feels like what it is: the one place in review where I work the design out myself.
Three minutes of typing isn’t Troy’s “sit with something that we feel bad at,” and I won’t pretend it is. What the three minutes buys me is a guess that can be wrong: a specific prediction on the record that the diff then confirms or contradicts. It only counts as a rep if I write it before I open the diff, and the timestamps in my review notes are what make that checkable later.
The habit has a weak point, too. Nobody checks the notes but me, and I’m grading the guess against the agent’s diff, not against ground truth, so a wrong diff can make a bad guess look good. So the guess doesn’t replace verification. It goes in front of it.
Comprehension help comes after the guess, never before. The AI help in my review workflow didn’t go away. It changed jobs. I still use it for explanations, blast-radius context (what else the change could break), and questions like “why is this dependency here,” but only to resolve a gap between my prediction and the diff, not to form my first impression. If the agent’s explanation is my first exposure to the design, I skipped the rep.
Spikes test my design call instead of replacing it. The same order applies when my gut says a different path was possible. Before an agent builds a spike of it (a throwaway prototype), I write down my hypothesis and how I’d tell whether it worked. The agent builds it, and I compare the result with the design that shipped. The spike grades my judgment instead of handing me a menu of options I didn’t design, and the call is on record.
My review notes also touch Troy’s other two concepts, without extra work. Format diversification: I meet one design as spec prose, then as my guess, then as the agent’s diff, then as test output. Spaced repetition: the guesses are timestamped and searchable, so when I touch a module again I can find my old prediction, and I try to re-derive the design before I re-read it. Neither is scheduled. Both ride on the calendar the work already generates.
What the review queue is for
One more point from Troy’s interview has stayed with me: an exercise is about what changes in the person, not the artifact. That reframed my whole review queue. It’s not a list of diffs to clear. It’s a cheap, frequent place to practice. When I want to, I can have an agent build the alternatives I would have considered and grade my judgment against them. That used to be too expensive to do routinely. Now an agent does the building, and the part that’s mine is the guess.
In Part 1, going “pencils down” on hand-written code moved my pencil into review, the gap between “done” and “proven.” I still verify every diff. The first thing I put in that gap now is my guess.