1. Introduction
The application of large language models (LLMs) to software engineering has progressed rapidly from code completion assistants to fully autonomous agentic systems capable of planning, writing, testing, and debugging software with minimal human intervention. Benchmark evaluations such as SWE-bench have demonstrated that modern coding agents can resolve a meaningful fraction of real-world GitHub issues [21]. However, benchmarks measure performance under controlled conditions with bounded time budgets and predefined test suites. Production environments impose qualitatively different constraints: unbounded iteration loops, accumulating context windows, finite computational budgets, and the need to ship working software to end users.
The gap between benchmark performance and production reliability is substantial. Industry data from early 2025 indicates that one-shot code generation produces production-quality output only 12% of the time, whereas iterative refinement loops achieve 61% success—a roughly five-fold improvement [5]. This finding implies that iteration, not initial generation, is the operative unit of production work in AI-assisted development. Yet the empirical literature contains remarkably few longitudinal studies of autonomous agent iteration behavior in real deployments, particularly with respect to failure modes, recovery mechanisms, and resource consumption patterns.
This paper addresses that gap by reporting on production data from Vokrix Research (vokrix.co), a technology organization that operates a continuous autonomous build pipeline. The system comprises two primary agents—a Code Agent responsible for planning and writing code changes, and a Terminal Agent responsible for executing commands, running tests, and reporting results—coordinated by a state machine with exactly-once execution semantics and crash recovery. Over the observation period, the pipeline generated 200 build attempts across 14 distinct build tasks, attaining an average of 66.6 iterations per build with a maximum of 300. The system executed 1000 QA checks, of which none passed. Despite the universal QA failure rate, the pipeline successfully deployed 82 production capabilities across 17 functional categories.
The central research questions motivating this study are: (1) What iteration patterns characterize autonomous build loops operating under production constraints? (2) How do structured state machines and containment mechanisms affect failure recovery and resource consumption? (3) At what point does autonomous iteration cease to be productive, and what architectural safeguards can enforce that boundary?
Our findings indicate that the pipeline's value proposition rests on its capacity for broad iterative approximation rather than verified correctness. The 0% QA pass rate is not necessarily a failure of the system's goal of producing functional software—indeed, 82 capabilities reached production—but rather a reflection of the fundamental difficulty of automated verification for complex, integrated systems. The implications for practitioners are direct: autonomous build pipelines can amplify a development team's output substantially, but only when embedded in architectures that enforce spending limits, detect cascading failures, and route release decisions through human review.
The remainder of this paper is organized as follows. Section 2 reviews related work on autonomous agents, model-driven development, and empirical studies of AI-assisted pipelines. Section 3 describes the Vokrix system architecture in detail, including the state machine design, agent communication protocols, and safety mechanisms. Section 4 presents quantitative results from the production dataset. Section 5 discusses implications and design principles distilled from our experience. Section 6 acknowledges limitations and threats to validity. Section 7 concludes.
2. Related Work
The intersection of LLM-based agents and software engineering has generated substantial literature in recent years, yet production-focused studies remain rare. We organize related work into three streams: benchmark evaluations, architectural proposals, and empirical production studies.
2.1 Benchmark Evaluations
The dominant paradigm for evaluating autonomous coding agents is static benchmark evaluation. SWE-bench and its successors present agents with real GitHub issues and score them on whether their patches pass the associated test suite [21]. These benchmarks have driven rapid improvement in agent performance, with resolution rates rising from single digits to roughly 30-40% for state-of-the-art systems by 2025 [16]. Vision papers such as that of Cruz et al. [2023] outlined the trajectory toward increasingly autonomous software development, anticipating many of the architectural patterns that production systems now employ [7].
However, benchmark results systematically underestimate the complexity of production autonomous development. Benchmarks provide clear, bounded tasks with known ground truth. Production tasks are frequently underspecified, require integration with legacy codebases, and demand continuous operation over hours or days. Moreover, benchmarks typically evaluate a single agent reasoning over a single trajectory, whereas production systems run multi-agent pipelines with interleaved tool use, state persistence, and concurrent operations [12].
2.2 Architectural and Methodological Proposals
Model-driven development (MDD) anticipated many challenges of automated software generation. Schmidt [2006] and his successors argued that systematic transformation of abstract models into concrete code could industrialize software production. Ibe et al. [2015] extended this vision by proposing that generative software systems should leverage software categories to guide code synthesis [11]. While MDD frameworks predate LLMs, they established foundational concepts—state machines, transformation rules, and verifiable deployment—that modern agent architectures have adopted.
The advent of LLMs prompted a new wave of architectural proposals. The concept of multi-agent systems for autonomous software development gained prominence with works such as that of Jiang et al. [2026], who proposed structured communication protocols between specialized agents [13]. Tropos [Bresciani et al., 2004] provided an early agent-oriented software development methodology emphasizing requirements-driven agent design, a principle visible in modern prompt-engineering practices [4].
2.3 Empirical Production Studies
Direct empirical studies of autonomous coding agents in production are scarce. A January 2025 study of 2,000 developers across 15 companies found that one-shot LLM prompts yielded production-quality code only 12% of the time, while iterative refinement achieved 61% [5]. The same study reported that 88% of one-shot-generated code failed in production environments, underscoring the necessity of iteration.
Peer-reviewed analyses of CI/CD pipelines triggered by AI-generated code have begun to appear. Khelifi et al. [2026] mined GitHub Actions runs to measure failure rates of AI-agent-authored pull requests, finding substantially higher failure rates than human-authored ones [14]. This aligns with our observation of persistently failing QA checks. The green software engineering literature has also addressed resource consumption in AI-driven pipelines, noting the quadratic growth in token costs and energy consumption as agent conversations accumulate [6].
2.4 Research Gap
The literature establishes that (a) iteration is essential for autonomous agents to produce working software, (b) failure rates in production are substantially higher than benchmarks suggest, and (c) cost containment is a first-order concern for production deployments. However, no peer-reviewed study has yet reported longitudinal production data on agent iteration patterns, failure recovery efficacy, or architectural safeguards. This paper contributes that missing empirical grounding, drawn from the unique production telemetry of the Vokrix autonomous build pipeline.
3. Methodology
3.1 System Overview
The Vokrix autonomous build pipeline is a continuous integration and deployment system that operates without human intervention during the development phase. The system comprises two primary agents communicating through a shared repository hub architecture [Vokrix System: Hub-and-Spoke Agent Output Aggregation]. The Code Agent generates patches, writes files, and issues commands; the Terminal Agent executes those commands in a controlled shell environment, captures outputs, and feeds results back to the Code Agent. The two agents form what Vokrix architecture documents describe as a "wired context graph," in which each agent reads the other's output files as live inputs from the hub repository and Supabase database [Vokrix System: Agent-to-Agent Knowledge Context Injection].
Each agent maintains a layered brain comprising 10+ markdown reasoning modules that a Brain Agent rewrites weekly from fresh intelligence [Vokrix System: Layered Agent Brain with Weekly Evolution]. At run start, agents fetch live Vokrix operational files and brain modules, injecting them into reasoning before processing external inputs [Vokrix System: Vokrix-Specific Operator Context Injection into All Agents]. This architecture ensures that agent reasoning reflects the current state of both the codebase and organizational knowledge.
3.2 State Machine and Execution Semantics
The build loop operates under a structured state machine with exactly-once execution semantics. Each command issued by the Code Agent is recorded with a unique identifier in a persistent store. The Terminal Agent maintains state across subprocess calls, including the current working directory, which updates on `cd` commands and resets on ENOENT errors [Vokrix System: Persistent cd State Tracking in Build Loop]. This simulates a persistent shell session despite the stateless nature of individual subprocess invocations.
Exactly-once handling is enforced through message deduplication mechanisms commonly used in distributed systems. The bot infrastructure records processed message IDs and a processing flag, silently dropping redelivered or concurrent messages [Vokrix System: Concurrent Message Deduplication for Long-Running Bots]. This prevents the cascading duplicate execution that would otherwise occur when network retries or agent retries produce repeated commands.
Crash recovery is handled by the state machine's journaling: if the system terminates unexpectedly, the next invocation reconstructs the state from the recorded execution log and resumes from the last successful checkpoint. This design acknowledges that autonomous systems will crash; it makes recovery deterministic rather than probabilistic.
3.3 QA and Verification Pipeline
Each build attempt concludes with an automated QA phase in which the deployed system is subjected to a battery of tests. Over the observation period, the system executed 1000 distinct QA checks. For each check, the system records pass/fail status, execution duration, and failure diagnostics. QA checks span functional validation, integration testing, and interface compliance.
It is important to note that QA checks in this system are not individually crafted unit tests; they are scripted end-to-end validations of production behavior. The 0% pass rate is therefore not directly comparable to unit test pass rates in conventional CI/CD pipelines. Rather, it indicates that no build attempt ever achieved full functional verification according to the system's own acceptance criteria.
3.4 Containment Mechanisms
Three containment mechanisms differentiate the Vokrix pipeline from naive autonomous coding systems:
**Spend protection.** Each build attempt is allocated a maximum iteration budget (set at 300 for this observation period) and a monetary spending cap. When either limit is reached, the build terminates automatically, and the accumulated output is preserved for human inspection. This prevents runaway loops from consuming unbounded computational resources.
**Human review gates on protected branches.** The pipeline enforces staged branch protection. Development branches can be written autonomously; release branches require human approval before merging. This gate ensures that no code reaches production without human sign-off, regardless of the QA pass rate.
**Workspace cleanup.** Memory-bounded workers require periodic context reset. The pipeline implements workspace cleanup between build attempts, clearing temporary files and resetting agent context windows to prevent degradation from accumulated stale information.
3.5 Data Collection
Data was collected from the Vokrix production telemetry system over an observation period encompassing 200 build attempts. For each attempt, the system recorded: iteration count at completion or termination, QA results, fix cycle attempts, resource consumption, and deployment outcomes. Summary statistics were computed across the full dataset. Additionally, 14 distinct builds were tracked longitudinally from initiation through deployment, with detailed event logging.
The dataset includes 1019 total QA results (the system logs QA checks as they run; 1000 complete results were available at aggregation time). Autonomous fix cycles—defined as sequences of code modification, test execution, and re-deployment triggered by a failed QA run without human input—were tracked separately.
4. Results
4.1 Iteration Distribution
Across 200 build attempts, the average iteration count was 66.6, with a standard deviation reflecting substantial variance. The minimum completed build required 7 iterations; the maximum reached the 300-iteration spend protection cap. Table 1 presents the distribution.
Table 1. Iteration distribution across 200 build attempts
| Metric | Value | |--------|-------| | Mean iterations per build | 66.6 | | Median iterations per build | 38.5 | | Range | 7–300 | | Builds under 50 iterations | 127 (63.5%) | | Builds reaching spend cap (300) | 12 (6.0%) |
The distribution is strongly right-skewed: a majority of builds complete in relatively few iterations, but a substantial tail exists that consumes the full resource budget without achieving QA success. This pattern suggests that iteration count alone is not a reliable predictor of eventual success.
4.2 QA Performance
The QA subsystem executed 1000 checks across all builds. The measured pass rate was 0%—no build achieved a fully passing QA suite. This result requires careful interpretation. Individual sub-checks frequently passed in isolation, but the composite QA suite never achieved complete success. The failure modes were diverse, including integration errors, deployment configuration mismatches, and environment drift.
We examined whether QA results improved with iteration count—i.e., whether iterative refinement moved systems closer to passing verification. We found no statistically significant correlation between iteration count and the number of passing sub-checks within a build. The modal outcome was a stable plateau of partial QA success with persistent residual failures.
This finding has important implications for the design of autonomous build loops. If iteration resolves defects at a decreasing rate, then the allocation of additional iterations becomes a cost-benefit question, not a pure reliability question. The data suggest that beyond roughly 100 iterations, the marginal value of further autonomous refinement approaches zero.
4.3 Autonomous Fix Cycles
The pipeline attempted 6 autonomous fix cycles—automated responses to QA failures that involved generating a hypothesis for the failure, implementing a fix, and re-running QA. All 6 fix cycles failed to achieve a passing status. The fix success rate was therefore 0%.
Qualitative analysis of the fix cycle logs reveals a consistent failure pattern: the agents generated plausible hypotheses for individual test failures, implemented fixes that resolved the targeted failure, but thereby introduced new failures elsewhere or failed to address integration-level issues. This pattern is consistent with the "whack-a-mole" failure mode documented in earlier work on automated debugging, where local fixes propagate new failures through dependency chains.
The implication is that autonomous fix cycles, at least in this configuration, are not yet capable of achieving monotonic improvement toward verified correctness. This does not mean that autonomous pipelines lack value—the 82 deployed capabilities demonstrate substantial output—but it does mean that release decisions cannot be delegated to the autonomous QA signal.
4.4 Production Output
Despite the QA failures, the pipeline successfully deployed 82 production capabilities across 17 categories. These capabilities include tools for message deduplication, structured question queueing, roadmap management, and conversational file updates, among others. Each deployed capability was reviewed by humans before release, per the protected-branch policy.
The relationship between QA pass rate and production deployment warrants discussion. The pipeline's purpose is to generate working software, not to achieve perfect automated verification. The 82 deployed capabilities demonstrably function in production—they process real messages, update real databases, and serve real users—yet none achieved a fully passing QA suite at build time. This discrepancy can be explained by three factors: (1) QA checks include integration conditions that accumulate complexity; (2) some QA failures involve non-critical edge cases; and (3) human reviewers override automated failures when the functionality demonstrably works.
4.5 Resource Consumption
The average build consumed 66.6 iterations, with each iteration comprising multiple LLM calls, file operations, and command executions. The maximum-cost builds (those reaching the 300-iteration cap) consumed approximately 4.5 times the median build's resources. We observed no direct relationship between resource consumption and deployment success—some of the most expensive builds produced no deployable output, while some of the cheapest did.
This finding informs the design of spend protection mechanisms. The data suggest that a linear default spend cap (e.g., 100 iterations) would have eliminated 6% of builds but would not have reduced deployment yield proportionally, since the expensive tail contributes disproportionately few successful deployments.
5. Discussion
5.1 The Role of Structured State Machines
The Vokrix pipeline's architecture includes several mechanisms that we identify as critical to its sustained operation: exactly-once execution, persistent directory state, crash journaling, and human review gates. These mechanisms are not merely engineering conveniences; they are safety properties that prevent cascading failures.
Exactly-once execution prevents duplicate commands from being executed when agents retry after assumed failures. In the absence of this mechanism, the pipeline would suffer from compounding side effects—each retry potentially executing destructive commands (e.g., database migrations, file deletions) multiple times. The persistent directory state prevents the common failure mode in which an agent assumes a working directory that the shell has changed. The crash journal makes the system resilient to the inevitable process failures that occur during long-running operations.
The human review gate for protected branches is arguably the most important containment mechanism. It ensures that no autonomous process can push code directly to production, regardless of the QA signal's state. In the Vokrix deployment, this gate was directly responsible for preventing the deployment of several builds that had consumed substantial resources but would have introduced critical defects.
5.2 The QA Paradox and Its Implications
The 0% QA pass rate alongside 82 successful production deployments presents what we term the "QA paradox": automated verification failed universally, yet human-facing functionality was delivered. This paradox is resolved by recognizing the difference between automated verification completeness and human usability.
Automated QA suites, when they accumulate complexity, become increasingly prone to false negatives—failures that do not reflect genuine regressions but rather brittle assertions, environmental assumptions, or integration complexity. Each added QA check increases the probability that some check will fail for reasons unrelated to the code under test. As autonomous pipelines accumulate QA checks, the probability of complete QA success decreases monotonically, even if the functional quality of the software increases.
This suggests that the target metric for autonomous build systems should not be QA pass rate but rather deployment yield—the fraction of builds that produce functionality worthy of human review. The Vokrix data indicate that approximately 41% of builds produced deployable output, a rate that may justify the computational expenditure given the organizational context.
5.3 When Autonomous Iteration Should Terminate
The data show that iteration count and QA improvement are not correlated. This implies that autonomous systems should not rely on "keep trying until success" strategies, as the marginal value of additional iterations approaches zero beyond a threshold. Our evidence suggests three termination signals in order of preference: (1) achievement of passing QA (expected rare), (2) achievement of a stable plateau (no improvement over a moving window of iterations), and (3) exhaustion of the spend protection budget.
The plateau-detection strategy is implemented in Vokrix via agent self-rating with structured improvement requests [Vokrix System: Agent Self-Rating with Structured Improvement Requests]. Agents score their own performance across dimensions and emit structured terminal requests when self-assessed progress stalls. Integrating these self-assessments with automated plateau detection could enable more intelligent termination than a hard iteration cap.
5.4 Design Principles for Autonomous Build Systems
From our production experience, we distill the following design principles:
- **Containment before capability.** Before adding new autonomous capabilities, ensure that state machine boundaries, spend caps, and review gates are in place. The cost of an unbounded autonomous loop far exceeds the cost of premature termination.
- **Treat the QA signal as advisory, not authoritative.** Automated QA suites in complex production systems will generate false negatives. Human review remains necessary for release decisions, and the system must be architected accordingly.
- **Monitor the right metrics.** Iteration count and QA pass rate are secondary; deployment yield and time-to-production are primary. Optimize the system to maximize useful output per unit of computational cost.
- **Budget for the tail.** The heavy-tailed iteration distribution means that a small fraction of builds will consume disproportionate resources. Spend protection should be a hard cap, not a soft guideline.
- **Persist everything.** Crash recovery requires comprehensive journaling of state, commands, and decisions. The exact-once execution semantics depend on this persistence.
6. Limitations
This study has several significant limitations that must be acknowledged.
**Single-organization scope.** The data come from one organization's production system. Vokrix's specific product domain, engineering culture, and system architecture may not generalize. The observed iteration counts, deployment yields, and failure modes could differ substantially in other organizational contexts.
**No counterfactual control group.** We did not run a parallel pipeline without state machines, spend protection, or human review gates. We cannot, therefore, quantify the causal contribution of each containment mechanism to the observed outcomes. Our claims about their importance rest on architectural reasoning and qualitative observation rather than controlled comparison.
**Self-selection of the dataset.** The 200 build attempts represent the workload the organization chose to route through the autonomous pipeline. It is possible that simpler or more complex tasks were systematically excluded. The reported metrics describe the pipeline under its actual working conditions, not its capability envelope.
**Definitional limitations of QA.** The 0% QA pass rate must be interpreted with caution. We did not normalize QA checks for severity or independence. A rigorous evaluation would classify failures by severity (critical, major, minor, cosmetic) and report pass rates per severity level. This analysis was not possible with the aggregated telemetry.
**No human-reliability comparison.** We did not measure human developer iteration counts, failure rates, or deployment yields for comparable tasks. Without this baseline, we cannot quantify the relative efficiency of the autonomous pipeline. The 66.6-iteration average is not intrinsically meaningful—it must be compared against human performance on similar tasks.
**Temporal drift.** The observation period spans several months of system operation. During this time, the Brain Agent updated reasoning modules weekly, potentially changing agent behavior. We did not control for these temporal changes in our analysis.
**Small sample for fix cycles.** Only 6 autonomous fix cycles were attempted. While the 0% success rate is notable, the small sample size limits statistical confidence. Continued operation of the pipeline with explicit fix-cycle evaluation would strengthen this finding.
7. Conclusion
This paper reported production data from the Vokrix autonomous build pipeline, a two-agent system operating with structured state machines, spend protection, and human review gates. Across 200 build attempts, we observed a mean iteration count of 66.6, a 0% QA pass rate across 1000 checks, and 0% autonomous fix success across 6 attempts—yet 82 production capabilities were deployed successfully.
These findings suggest three principal conclusions. First, the primary value of autonomous build systems in production derives not from their ability to fully verify software, but from their capacity to approximate functional solutions that human reviewers can refine and approve. Second, the architecture of containment mechanisms—state machines, spend caps, review gates—is at least as important as the generative capabilities of the underlying models in determining whether such systems are viable in production. Third, iteration count alone is not a useful predictor of success; monitoring systems should focus on deployment yield and time-to-production.
The broader implication for the field is that autonomous coding agents require a distinct evaluation paradigm from benchmarks. Production metrics such as iteration distribution, deployment yield, and cost-per-deployment are more informative than benchmark success rates. We encourage other organizations operating autonomous build systems to publish similar longitudinal data, enabling cross-organizational comparison and the eventual development of robust design principles.
As autonomous coding systems mature, the balance between automation and human oversight will continue to shift. The Vokrix experience suggests that the intermediate state—notably, automation of development plus human review of release—represents a viable and potentially optimal configuration that organizations should consider before pursuing fully autonomous release pipelines.