When AI Writes the Code, Specs Want an Exit Technique – O’Reilly

The next article has been prolonged and rewritten by Markus Eisele from The Predominant Thread and is being republished right here with the creator’s permission.

Open a repository after six months of spec-driven agent work and you might discover a second system sitting subsequent to the code. Necessities, analysis notes, high-level designs, low-level designs, implementation plans, job lists, assessment stories, and a rising stack of Markdown information that designate what the code is meant to imply. Even when the code modified considerably final Tuesday, the final documentation replace was weeks in the past.

I perceive how groups get there. And it’s not a extremely new impact in spite of everything. We had software program evolving parallel to documentation since I can bear in mind. Now that brokers produce code so  shortly, we attempt to management the drift and the code technology by shifting extra thought in entrance of implementation. As a substitute of documenting code, we attempt to drive code technology with it, making Markdown information with necessities, determination data, design approaches, and acceptance standards the middle of gravity and turning them into our workflow drivers.

What successfully is turning into a really massive immediate can simply fill a good portion of the context window even of contemporary brokers earlier than any related supply code will get added to it. Pure language specification is a weak system for brokers to synchronize a codebase with. With out extra consideration and diligence, most brokers I work with slowly shift consideration away from it shortly and deal with the stronger indicators within the codebase, forgetting to replace the specification finally.

Even when it sounds prefer it, I’m not advocating for one-shot prompting or vibe coding right here. We nonetheless want some specs to construct profitable software program. The error is treating a specification as a everlasting natural-language copy of the software program. A helpful spec describes the following change, paperwork the choices that drive the change, units boundaries, and provides us and the brokers sufficient verification floor. However as quickly because the change ships, most of it must be eliminated.

What stays ought to transfer into the artifacts software program groups already know the right way to preserve. At first, clearly, the code. However I additionally rely schemas, configuration, and insurance policies as related artifacts. They carry which means about area data and system configuration. Two classes that I worth extremely get simply forgotten: exams because the secure verification layer and runtime telemetry. In actual fact, I do let my brokers have a look at proof from all these locations not solely to hunt for errors but in addition to constantly optimize present codebases. Oh, and I do preserve determination data. However solely a small quantity and solely when their content material actually has no different place in any of the talked about artifacts. They’ll even seem like Javadoc, however that can be one other article sometime.

Code is the actual fact

Code is precise conduct. As soon as code is deployed to manufacturing, customers and linked programs are relying on it. Even a mistake can develop into an noticed contract as a result of it has behaved the identical method for 3 years. The runtime conduct takes priority on this contract as a result of no person checks the specification anymore, even when it defines a really totally different conduct. That is the strongest sign for me to start out with the precise code within the manufacturing system. Studying a natural-language abstract as a substitute of the carried out reality can’t precisely mirror runtime conduct. Code to me is the last word, executable specification. Simply written in a really particular and deterministic language.

What manufacturing code can’t drive although is the following model or iteration of a function. Whereas brokers can infer technical patterns from well-structured codebases, there’s no method they might predict coverage adjustments or future function requests. Neither can they find out about regulatory necessities like retention durations or different particular exceptions, equivalent to why one export runs each night time for just one buyer. That particular context has to return from some place else. Nevertheless it doesn’t require us to maintain a everlasting prose description of the entire system. We want simply sufficient context to resolve the delta: the distinction between what exists and what ought to exist subsequent.

Written phrases are for the delta

A change specification ought to exist when it helps a group resolve and assessment that delta. It ought to title the end result, non-goals, constraints that differ from present conduct, and the proof required for acceptance. It’d even comprise technical design components when new options cross architectural boundaries or introduce new patterns that aren’t current within the code but. Typically it’s also value serious about how costly reversing the change is, particularly if the present system has varied implementations for a sure sample and the chance is excessive that an agent may invent one other new model.

The checklist vital for adjustments could be very brief:

• The supposed end result and non-goals (the place vital)
• Identified unknowns and choices that want human judgment
• Affected system boundaries and authoritative interface artifacts
• Purposeful and nonfunctional constraints that differ from in the present day
• Acceptance standards/take a look at situations overlaying the dangerous path

I desire calling this a “change temporary” as a substitute of a “specification.” Specification carries an excessive amount of negativity. It sounds heavyweight and jogs my memory of occasions long gone. It additionally pretends to be full. And this completeness is making it very costly.

We now have tried exhaustive specs earlier than and produced requirement paperwork and different  high- and low-level designs, adopted by structure determination data for the whole lot. I bear in mind studying folders filled with paper over the weekend to get began on a brand new mission on Monday. Method earlier than AI even entered all our lives and codebases. We known as this waterfall again within the day, and the strategy nonetheless has the identical damaging unwanted effects in the present day. The documentation was full in an administrative sense and was principally ineffective within the engineering sense. All of us have seen this taking place. Brokers simply recreate the identical erratic outcomes from overflowing documentation, like we did again within the day.

One explicit threat I’m seeing with many groups is that they let brokers generate the preliminary model of the spec. A protracted workflow run produces not solely the analysis however immediately derives the necessities, design, and planning, and opinions artifacts on high. Whereas the completeness makes the whole lot look very managed and outlined, it additionally generates much more materials to be reviewed and accredited. Even when fashions and harnesses proceed to evolve at breathtaking velocity, it’s nonetheless difficult for them to generate actual cohesiveness out of chaos. The possibility they put the unsuitable consideration on some tempting repetitive phrases is excessive. This leads to a fair greater burden on the human reviewer and makes it endlessly more durable to maintain the assorted paperwork aligned.

I feel that extra prose like analysis notes, prototypes, and design data ought to solely be added to a software program mission when uncertainty justifies them. They resolve a particular downside. Or assist navigate the terrain. I wrote about this earlier than. They need to completely not develop into required phases for each pull request.

The map will all the time be incomplete

A immediate, ticket, or change temporary captures what we all know earlier than the work begins. The codebase, runtime info, configuration, linked programs, and years of collected choices glued into code maintain the remainder. A few of these choices have been by no means written down.

When brokers get to work they expose the lacking info. Studying a module reveals an surprising dependency. A prototype reveals {that a} particular user-interaction is awkward. A take a look at uncovers an edge case. Manufacturing knowledge contradicts an assumption within the design. This discipline information on discovering unknowns in agent work describes the issue properly. We are able to determine some unknowns initially. Others seem solely after we examine the references, construct a prototype, or assessment a consequence utilizing judgment that was troublesome to put in writing down prematurely.

Discovery occurs and continues in the course of the work:

• Earlier than implementation, examine the present system and determine choices that might change the structure or person expertise. When preferences are troublesome to explain, construct an affordable prototype.
• Throughout implementation, document significant deviations. Cease and reassess when a brand new unknown adjustments the chance or course.
• After implementation, learn the code, run the checks, and evaluate the consequence with the unique intent.

The change temporary stays a part of this loop. It gives the place to begin and data the intent, whereas the work provides the data wanted to finish it. Solely promote sturdy constraints.

Maintain sturdy information of their native type

Once I say “promote sturdy constraints,” I don’t imply turning each determination into everlasting Markdown. That provides us the identical stale documentation downside differently. Software program engineering already gives higher variations for a lot of the vital, sturdy information:

• API form and compatibility belong in OpenAPI, AsyncAPI, protocol schemas, varieties, and compatibility exams.
• Information invariants belong in varieties, database constraints, validation, and migration checks.
• Safety guidelines belong in entry insurance policies, static evaluation, dependency insurance policies, and runtime enforcement.
• Structure boundaries belong in module construction, dependency guidelines, and centered structure exams.
• Reliability necessities belong in load exams, service goals, telemetry, and alerts.
• Launch guidelines belong in steady integration and deployment insurance policies.

These artifacts are already a part of supply. A failed schema test or alert must be fastened and dealt with whereas the corresponding paragraph in an previous design folder doesn’t.

Pure language and specification nonetheless have a spot in software program. Particular area data like enterprise coverage, trade-offs, and even architectural rationale don’t all the time match into an executable artifact or annotation. I preserve that prose brief and near the factor it explains. A small structure determination document is value conserving when a future group may in any other case repeat an costly investigation and a code remark can’t justify the implementation. Recording each native selection simply hides the few choices that matter and confuses the brokers which can be supposed to construct the software program. Ask which truth should survive and what its authoritative type must be.

Judgment belongs within the workflow

Heavyweight specification strategies attempt to management high quality by prescribing the trail. Each change goes by the identical paperwork, opinions, and take a look at classes. That strategy creates numerous consideration on low-risk work whereas avoiding the deep technical judgment wanted for more durable adjustments. A copyedit and a payment-flow change mustn’t need to comply with the identical course of or testing technique.

Simon Willison describes a less complicated strategy: give the coding agent the end result and let it decide how a lot course of the duty requires. His examples embrace deciding whether or not a change warrants automated exams and whether or not routine implementation may be delegated to a less expensive mannequin whereas conserving judgment-heavy work in the primary loop. This replaces a rising checklist of procedural branches with one expectation: Select ways that match the work. That matches how I need these programs to function. And I feel it extends to specification and the way we doc intent.

Agentic adjustments nonetheless require clear boundaries. The group defines the end result, security constraints, possession, and who has authority to just accept the consequence. Inside these boundaries, the agent can select its ways. When uncertainty introduces penalties past its authority, it ought to floor the issue and ask for a choice.

The workflow then begins matching the chance launched:

• A small, acquainted change can transfer from a brief temporary to implementation and assessment. Virtually a one-shot immediate change.
• Unfamiliar code requires factual analysis earlier than design. Discover codebases, determine implementation particulars. Preload intent and agent data.
• An unclear person expertise requires prototypes and comparability. And may even require person analysis in spite of everything.
• An architectural change requires specific human alignment.
• Excessive-consequence conduct requires stronger impartial proof and approval.

I’d slightly add processes and extra artifacts when the work turns into dangerous or unfamiliar. Beginning each change with the complete ceremony simply burns time and context.

Context is an engineering finances

Giant specs value greater than the time required to put in writing and preserve them. Additionally they  compete with the code and proof the agent wants for the present determination. Each requirement, design observe, repository instruction, and gear definition consumes a part of a restricted working context. Additional materials burns costly tokens, however the a lot greater value is misplaced consideration. Vital guidelines develop into more durable to comply with when they’re surrounded by stale or duplicated materials. A spec that leaves too little room for the repository defeats its personal objective.

Progressive disclosure is a greater match. Give the agent a small map, just a few secure guidelines that apply broadly, and tips to deeper materials. A concise AGENTS.md can doc construct instructions, repository format, and architectural boundaries. It mustn’t narrate each class or repeat API documentation. The file helps people for a similar motive: It tells them the place to look with out pretending to switch what we are going to discover.

Expertise with Analysis-Plan-Implement reveals what occurs when the context grows too massive. The unique workflow moved human assessment earlier than implementation, however groups ended up with massive prompts and plans that might attain 1,000 traces. Engineers reviewed these plans whereas treating generated code virtually like compiler output. The implementation might nonetheless drift from the accredited plan, which meant that finally somebody needed to reconstruct the choice from the code. That downside turns into worse in brownfield programs, whereas greenfield programs may even survive massive plans as a result of they inherited no hidden constraints. Complicated adjustments, in distinction, typically inherit conduct that plans might miss.

In “Every little thing We Received Improper About Analysis-Plan-Implement,” Dexter Horthy revisits the unique place. Groups shipped extra code after which spent a lot of the achieve time cleansing up earlier low-quality output. The implementation might additionally diverge from the reviewed plan, which pressured engineers to reconstruct what occurred from the code anyway. The revised workflow makes use of smaller contexts for factual analysis, design alignment, construction, implementation, and assessment. I take a easy lesson from this: Analysis and design give me leverage, however I nonetheless want to know and personal the code that’s generated.

Modernization makes this apparent

A mature utility accommodates a number of sorts of conduct in the identical codebase. Some logic represents sturdy enterprise logic or implements a broadcast interface. Some code exists as a result of an previous platform imposed a technical constraint. An incident repair stays lengthy after its  context is gone. And even defects can survive to the purpose the place they virtually look intentional when undiscovered.

An agent that treats each code variant as a brand new goal specification can translate these layers faithfully into a brand new language or structure. The interpretation could also be technically correct but in addition preserves defects and previous structure approaches in newer and cleaner code.

I design adjustments to brownfield tasks much like the best way I did modernizations earlier than the agentic age. Classification and statement are central elements that I put first. The targets are:

• Protect sturdy enterprise invariants and externally required conduct
• Confirm conduct that seems energetic however lacks clear possession or proof
• Redesign logic tied to out of date architectural constraints
• Take away useless paths, duplicated logic, and confirmed defects

You possibly can learn lots about static supply code evaluation in terms of brownfield assessments or modernization. You possibly can examine dependencies and present conduct by executing exams and perhaps even including take a look at circumstances to safe conduct. What I do suggest is to additionally embrace mutation testing approaches (e.g., PIT) to seek out hidden assumptions and failure conduct. Code protection can be seeing a renaissance as a result of it aids in figuring out useless code paths.

On high of that we nonetheless ignore operational context and telemetry knowledge. Each are very important components to not solely management but in addition to assist decide present conduct. All this collectively helps you decide which components belong within the system going ahead and which don’t. All of it begins from code. It’s the basis of the conduct we now have. The unique and main specification. A change temporary will all the time be short-term and its sole job is to explain the delta between present and future performance. The brand new implementation and its native checks develop into the following sturdy state.

Small specs nonetheless want actual proof

Maintaining specs small doesn’t imply returning to a unfastened immediate adopted by hopeful assessment and even vibe-coding approaches. An agent can flip an underspecified request right into a coherent implementation earlier than the lacking choices develop into seen to anybody. The consequence might compile, go the obtainable exams, and look internally constant. That coherent look is a part of the chance now. Unapproved enterprise choices disappear into one thing very ordinary-looking as a result of they bought resolved plausibly.

And this conduct is backed by analysis. If we have a look at repairing ambiguous natural-language necessities, for instance, we will see that immediately asking fashions to resolve ambiguity typically results in inconsistent and even irrelevant outcomes. Selecting a extra focused restore strategy across the recognized defects (change temporary) improved the outcomes by roughly 31%. SWT-Bench discovered that generated exams might filter proposed fixes and double the precision of a software program restore agent. They used one agent to generate a proposed change and gave one other the duty to supply proof to reject it. Lastly, the subject of formal specification technology: One attention-grabbing research I discovered gave 30 fashions the duty to translate pure language into TLA+ (Temporal Logic of Actions, a specification language created by Turing Award-winner Leslie Lamport). The most effective outcomes solely reached about 27% syntactic correctness and 9% semantic correctness. The formal notation helped to detect errors, but it surely didn’t assure correctness or that the interpretation preserved the unique which means.

These outcomes help centered clarification and impartial checks. Make clear the uncertainties that may change the end result, then confirm the implementation with proof that doesn’t come solely from the identical reasoning path. Producing an extended specification doesn’t remedy that downside in any respect.

I need the energy and independence of the proof to match the consequence of being unsuitable. A small inside refactor might have peculiar exams and code assessment. A change that includes safety or monetary elements, or that even touches regulated knowledge, wants a a lot stronger separation coupled with adversarial assessment and specific human approval. For these adjustments, the agent proposing the implementation mustn’t even be the one supply of its necessities and exams.

A lighter working mannequin

In apply, I need a workflow that I can clarify and not using a advanced movement diagram. It begins with the proof already within the system and makes the supposed change specific. Every little thing else is added solely when the potential threat of the change justifies it. Ideally, it is a easy five-step course of:

1. Begin from the code and operational proof that describe the present system
2. Outline the supposed delta, necessary boundaries, and recognized unknowns
3. Add analysis, prototypes, design alignment, or stronger verification the place threat requires them
4. Learn and assessment the implementation, not simply the plan
5. At launch, discard short-term reasoning and protect every surviving truth in its native authoritative artifact

That’s sufficient construction to information the work with out constructing a natural-language duplicate of the software program.

Earlier than implementation, the change temporary describes the supposed delta, and through implementation it helps folks and brokers align whereas new info adjustments the plan. However after the discharge the code and manufacturing conduct develop into the first proof of what the system does. Not separate documentation in any type that probably drifts over time.

Sturdy obligations stay within the artifacts we already know the right way to preserve: schemas, exams, insurance policies, configuration, telemetry, and brief data for rationale that can not be encoded elsewhere. Most planning particulars have accomplished their job by then and will expire.

I anticipate groups to get probably the most from coding brokers when they’re selective: specify what should be determined, uncover what the system can reply, confirm what carries threat, and let short-term planning go.

Sources

• A Area Information to Discovering Your Unknowns, Thariq Shihipar, 2026.
• Judgement, Simon Willison, 2026.
• Can LLMs Write Appropriate TLA+ Specs?, Bisharat et al., 2026.
• Automated Restore of Ambiguous Pure Language Necessities, Jia et al., 2025.
• SWT-Bench: Testing and Validating Actual-World Bug-Fixes with Code Brokers, Mündler et al., 2024.
• Giant Language Fashions for Necessities Engineering: A Systematic Literature Evaluation, Zadenoori et al., 2025.
• Superior Context Engineering for Coding Brokers, HumanLayer, 2025.
• Every little thing We Received Improper About Analysis-Plan-Implement, Dexter Horthy, 2026.
• Utilization, Results and Necessities for AI Coding Assistants within the Enterprise, IBM Analysis, 2026.
• From AI Governance to AI Assurance, IBM, 2026.