Immersion in the GitHub Universe: Scaling Coding Agents to Mastery

Authors: Jiale Zhao*, Guoxin Chen*, Fanzhe Meng*, Minghao Li, Jie Chen, Hui Xu, Yongshuai Sun, Wayne Xin Zhao†, Ruihua Song†, Yuan Zhang, Peng Wang, Cheng Chen, Ji-Rong Wen, Kai Jia†

Affiliations: Gaoling School of Artificial Intelligence, Renmin University of China; Band AI, ByteDance; Awe AI Team

Venue: arXiv preprint

Year: 2026

Date: February 10, 2026

Pages: 25

Code URL: Not explicitly provided in the paper

1. 研究摘要 (Research Summary)

The pursuit of autonomous software engineering has reached a critical inflection point. While large language models have demonstrated remarkable proficiency in generating code snippets and solving isolated programming puzzles, the translation of these capabilities into real-world software engineering tasks remains fundamentally constrained by a singular bottleneck: the scarcity of large-scale, high-quality training data. This is the central challenge that Scale-SWE confronts head-on, and its significance cannot be overstated. Unlike conventional code generation tasks that operate in isolated, self-contained environments, software engineering tasks demand interaction with complex, existing codebases, navigation of intricate dependency graphs, and validation against comprehensive test suites. The gap between the abundance of raw software artifacts on platforms like GitHub and the curated, executable datasets needed to train capable coding agents represents one of the most pressing barriers to advancing the field.

The intellectual contribution of this work lies in its radical reimagining of how software engineering datasets can be constructed. Rather than relying on labor-intensive manual curation or simplistic rule-based synthesis, the authors propose a paradigm shift: leveraging multi-agent systems operating within sandboxed environments to autonomously explore, validate, and transform raw pull requests into high-quality training instances. This approach fundamentally reframes dataset construction from a static, heuristic-driven filtering process to a dynamic, agent-driven exploration process. The key insight is that the same agentic capabilities we seek to train can be harnessed to create the very data needed for that training—a recursive, self-reinforcing loop that promises exponential scaling.

The paper makes three foundational contributions that collectively advance the state of the art. First, it introduces Scale-SWE, an automated sandboxed multi-agent workflow that orchestrates three specialized agents—an Environment Builder Agent (EBA), a Unit-test Creator Agent (UCA), and a Problem Statement Writer Agent (PSWA)—to collaboratively construct executable software engineering tasks. Second, it produces Scale-SWE-Data, comprising 100,000 verified instances derived from 6 million pull requests across 5,200 repositories, establishing it as the largest verified SWE benchmark to date with unprecedented repository diversity. Third, it demonstrates the practical utility of this dataset by distilling 71,498 high-quality trajectories and fine-tuning Qwen3-30B-A3B-Instruct to create Scale-SWE-Agent, which achieves a remarkable 64% resolve rate on SWE-Bench-Verified—representing an almost three-fold improvement over the base model.

The experimental findings are both convincing and transformative. The Scale-SWE-Agent not only dramatically surpasses its base model but also outperforms competitors of similar size by substantial margins, including Qwen3-Coder (51.6%) and GLM-4.7-Flash (59.2%). Perhaps most strikingly, it exceeds the performance of significantly larger models such as SWE-RL (Llama3-70B) and SWE-Fixer-72B, demonstrating that data quality and scale can compensate for model size. The comparative analysis reveals that real-world data, when properly curated through agentic workflows, inherently outperforms synthetic alternatives—a finding that challenges the prevailing trend toward purely synthetic dataset generation.

The broader impact of this work extends far beyond the immediate results. By providing a scalable, reproducible approach to dataset construction, Scale-SWE opens new avenues for the entire research community. It suggests a future where the limiting factor in coding agent development is no longer data availability but rather computational resources and algorithmic ingenuity. The work also establishes a methodological template that could be extended to other programming languages and domains, potentially catalyzing a new wave of progress in autonomous software engineering.

2. 理论框架 (Theoretical Framework)

The theoretical foundations of Scale-SWE are rooted in the evolving understanding of what constitutes effective training data for coding agents. The intellectual lineage of this work traces through several converging threads: the recognition that software engineering tasks require executable environments (epitomized by SWE-bench), the growing appreciation for agentic systems that can interact with their environment (embodied in frameworks like SWE-agent and OpenHands), and the scaling laws that have consistently shown data volume and quality to be primary drivers of model capability. This paper sits at the intersection of these threads, proposing that the construction of training data itself should be treated as an agentic task requiring environmental interaction.

At the heart of Scale-SWE's theoretical framework is the concept of "sandboxed multi-agent collaboration"—the idea that specialized agents, each with distinct capabilities and objectives, can collectively solve problems that would be intractable for any single agent or rule-based system. This draws upon broader theories of multi-agent systems and division of labor, but applies them in a novel context where the agents are not solving the end-user's problem directly, but rather constructing the artifacts needed for future agents to learn. The Environment Builder Agent embodies the principle that reproducible execution environments are not merely logistical necessities but theoretical prerequisites for valid software engineering tasks. Without the ability to execute and validate code, any task instance lacks the grounding needed for meaningful training. The UCA extends this by recognizing that test generation is itself an exploratory, feedback-driven process requiring dynamic code execution, not merely static analysis. The PSWA completes the triad by addressing the critical challenge of semantic alignment—ensuring that problem descriptions accurately reflect the validation criteria without leaking solution information.

The mathematical formulations in the paper, while relatively sparse, capture essential functional relationships. The core transformation performed by each agent can be understood as a mapping function. For the Environment Builder Agent, this mapping takes the form:

Here, $R$ represents the input repository that the agent analyzes to infer dependencies and configuration logic, $D_{init}$ denotes the base Docker image, and $D_{final}$ is the resulting functional, ready-to-use container image. This equation captures the essential idea that environment construction is not a deterministic function but an agent-driven transformation that must account for the unique characteristics of each repository. The agent must interpret project-specific documentation, resolve dependency conflicts, and synthesize a reproducible configuration—all tasks that resist simple rule-based automation.

Similarly, the Unit-test Creator Agent performs the transformation:

where $M$ denotes the input pull request metadata (including title, description, and diff patches), $R$ is the associated source repository, $D_{final}$ is the functional Docker environment built by the EBA, and $U_{final}$ is the resulting set of executable test cases. This formulation emphasizes that test generation is conditioned on multiple sources of information: the semantic intent conveyed by PR metadata, the structural and logical constraints of the codebase, and the executable context that enables dynamic validation.

The Problem Statement Writer Agent completes the pipeline with:

where $S_{final}$ represents the formal problem description. This equation highlights the critical insight that effective problem statements must be grounded in the executable test suite, not merely derived from PR metadata. The tests provide the objective criteria against which any solution must be evaluated, and the problem statement must articulate the requirements implied by these tests without revealing the implementation details.

The theoretical framework makes several important assumptions that define its scope and limitations. It assumes that real-world pull requests contain meaningful software engineering tasks suitable for agent training—a assumption validated by the authors' careful filtering but acknowledged to require quality control. It assumes that sandboxed Docker environments provide sufficient isolation and reproducibility for task validation, which holds for most Python projects but may face challenges with system-level dependencies or GPU-dependent code. The framework also assumes that three specialized agents, operating sequentially, can adequately cover the pipeline's requirements, which implies that the tasks of environment building, test creation, and problem statement synthesis are sufficiently separable and modular.

The theoretical connections between these components form a coherent logical chain. The EBA's output (a functional environment) is a prerequisite for the UCA's operation, since tests must be executed within a valid runtime. The UCA's output (an executable test suite) is in turn essential for the PSWA, as the problem statement must be semantically aligned with the validation criteria. This dependency chain means that failures at any stage propagate downstream, necessitating rigorous quality assurance throughout the pipeline. The framework also implicitly encodes a theory of what makes a good software engineering task: it must be executable (requiring a valid environment), verifiable (requiring tests), and well-specified (requiring a clear problem statement)—a triad that mirrors the classic software engineering emphasis on reproducibility, testability, and clarity.

3. 技术架构 (Technical Architecture)

The technical architecture of Scale-SWE represents a sophisticated orchestration of specialized agents, each designed to address a distinct challenge in the pipeline of transforming raw GitHub pull requests into validated, training-ready software engineering tasks. At the highest level, the system operates as a unified pipeline that processes 6 million pull requests from approximately 23,000 candidate repositories, ultimately yielding 100,000 verified instances. This scale is achieved not through brute-force computation but through intelligent design that maximizes resource efficiency while maintaining rigorous quality standards.

The data flow through the system begins with repository selection and PR extraction, proceeds through the three-agent transformation pipeline, and concludes with dataset verification and trajectory distillation. The initial filtering stages are themselves agent-assisted, employing an LLM-as-a-judge approach to automatically exclude repositories that are GPU-dependent, tutorial-based, API wrappers, or otherwise unsuitable for benchmarking code generation and bug-fixing capabilities. This filtering reduces the initial pool of 23,000 repositories to a curated subset from which 6 million pull requests are extracted. A second LLM-as-a-judge filter is then applied to the PRs themselves, discarding those involving only documentation changes, trivial parameter adjustments, or auto-generated file modifications. This two-stage filtering ensures that only substantive code changes enter the main pipeline.

The Environment Builder Agent (EBA) serves as the foundation of the technical architecture. Its primary function is to transform a generic Docker environment into a specialized runtime container tailored to a specific repository. The agent is initialized within a base Docker container containing the cloned repository, and it autonomously explores the repository structure, analyzing configuration files such as setup.py, pyproject.toml, and README.md to infer project dependencies. The critical insight here is that Python projects lack a universal setup protocol, making standardized installation approaches insufficient. The EBA overcomes this limitation through an interactive, feedback-driven process: it attempts installations, parses terminal feedback for errors, and iteratively resolves dependency conflicts. Once the environment is functional, the agent extracts all executed commands from its trajectory and uses an LLM to synthesize them into a reproducible Dockerfile. To scale efficiently, the system samples at most ten PRs per repository for full environment construction, then reuses the "nearest" available environment for remaining PRs based on proximity in PR ID (as a proxy for repository timeline). This strategy enables dramatically improved environment reuse, with an average of 19 test instances per repository.

The Unit-test Creator Agent (UCA) builds upon the EBA's output to generate comprehensive test suites. Its operation is perhaps the most technically intricate component of the architecture, as it must synthesize both Fail-to-Pass (F2P) tests—which verify that a specific bug is fixed—and Pass-to-Pass (P2P) tests—which ensure no regressions are introduced. The UCA begins by analyzing the PR metadata to comprehend the intent and scope of proposed code changes, then autonomously traverses the repository to map directory structure and identify key modules. However, the true innovation lies in its execution loop: deployed within the sandboxed Docker environment produced by the EBA, the UCA can dynamically run its proposed tests, observe outcomes, and iteratively revise test logic, assertions, and fixtures. This closed-loop, feedback-driven methodology addresses the fundamental challenge that effective test generation requires not just static code understanding but dynamic behavioral validation, including reasoning about cross-file interactions, data flow, exception handling, and edge cases.

The Problem Statement Writer Agent (PSWA) completes the pipeline by synthesizing high-quality, self-contained task descriptions. The critical challenge here is avoiding solution leakage: raw PR descriptions are often written retrospectively, after the fix is implemented, and may reference internal artifacts or inadvertently reveal implementation details. The PSWA addresses this by integrating both PR metadata and the executable unit tests produced by the UCA into its prompt context. This integration is essential because F2P tests may invoke functions or classes that do not exist in the original codebase, and the generated problem statement must explicitly articulate these requirements to make the task tractable. The agent synthesizes a coherent narrative specifying expected behavior, any new interfaces required by the tests, and the context necessary for an external solver—all while deliberately omitting hints about the implementation. Notably, the authors employ Gemini 3-Pro for this agent, as their experiments indicate it generates more consistent and rigorous problem statements while significantly minimizing information leakage.

The architecture also incorporates rigorous anti-cheating measures. A sanitization script executes immediately after environment initialization, performing a hard reset of the git repository, deleting all remote references and tags, and purging internal git files (logs, packed-refs, HEAD, and various reflog files). This eliminates any trace of future solution history that could be exploited by models using git commands to access ground truth patches.

The implementation details reveal careful engineering choices. The system is built upon SWE-agent and powered by the DeepSeek language model family (v3.1 or v3.2 for most tasks) and Gemini 3-Pro (specifically for the PSWA). The hyperparameters for the subsequent supervised fine-tuning include a learning rate of 1e-5, batch size of 128, maximum context length of 131,072 tokens, and cosine learning rate scheduling with a warmup ratio of 0.05 over 3 epochs. These choices reflect the computational demands of training on long-context software engineering tasks while maintaining training stability.

4. 实验评估 (Experimental Evaluation)

The experimental evaluation of Scale-SWE is designed to answer several interconnected questions: Does the dataset enable meaningful improvements in coding agent performance? How does it compare to existing datasets in terms of training efficacy? And what do the quantitative and qualitative characteristics of the dataset reveal about its quality and complexity? The authors address these questions through a carefully structured evaluation on SWE-Bench-Verified, the established benchmark comprising 500 high-quality, human-curated Python software issues.

The primary experimental result is striking in its magnitude. Scale-SWE-Agent, produced by fine-tuning Qwen3-30B-A3B-Instruct on 71,498 distilled trajectories from Scale-SWE-Data, achieves a 64.0% resolve rate on SWE-Bench-Verified. This represents an absolute improvement of 42.0 percentage points over the base model's 22.0%—an almost three-fold relative improvement. To contextualize this result, the authors provide a comprehensive comparison across multiple model categories, as summarized in the following table:

This comparison reveals several important patterns. First, among models of comparable size (approximately 30B activated parameters), Scale-SWE-Agent establishes a new state of the art, surpassing the previous best open-source result (KAT-Dev-32B at 62.4%) and outperforming other 30B-class models by substantial margins. Second, Scale-SWE-Agent demonstrates remarkable efficiency, exceeding the performance of significantly larger models such as SWE-RL (70B parameters) and SWE-Fixer-72B. This suggests that the quality and relevance of training data can compensate for model scale, a finding with significant implications for resource-constrained research settings.

To isolate the contribution of the dataset itself, the authors conduct a controlled experiment comparing Scale-SWE against existing alternatives using identical distillation and fine-tuning pipelines. The results, presented in the table below, are revealing:

This head-to-head comparison demonstrates that Scale-SWE outperforms both SWE-Gym (a real-world dataset of 2,400 instances from 11 repositories) and SWE-smith (a synthetic dataset of 50,000 instances from 128 repositories) by nearly 10 percentage points. The comparison with SWE-smith is particularly instructive: despite having a smaller total instance count than SWE-smith's 50,000, Scale-SWE's 100,000 instances (with training on a 25,000 subset) yield substantially better performance. The authors interpret this as evidence of "diminishing returns on purely synthetic data," suggesting that high-fidelity, real-world data is inherently more effective than massive-scale synthetic alternatives.

The dataset's statistical characteristics further illuminate its quality. With a median of 3 modified files, 43 added lines, and 6 total edited lines per instance, the tasks reflect non-trivial software engineering complexity. The test suite composition—with a median of 68 P2P tests and 2 F2P tests per instance—provides robust protection against regression while validating specific bug fixes. The distribution of bug categories across the dataset also reveals superior diversity compared to alternatives. While synthetic datasets like SWE-smith exhibit strong bias toward Logic Errors, Scale-SWE achieves a balanced distribution across all ten categories in the BugPilot taxonomy, including API Mismatches, State Synchronization issues, Constructor errors, and Security flaws. This diversity is attributed to the broad repository coverage (5,200 real repositories versus 11 for SWE-Gym or 128 for SWE-smith) and the automated environment-building pipeline that can handle heterogeneous codebases.

The trajectory distillation process itself provides insight into dataset complexity. Using DeepSeek-V3.2 with a temperature of 0.95 and a maximum budget of 100 interaction turns, the authors conducted five independent sampling trials per instance. The resulting 71,498 valid trajectories (from 25,000 instances) total approximately 3.5 billion tokens. The distribution of interaction turns and token counts, visualized in the paper's figures, shows that Scale-SWE tasks necessitate a greater number of turns for repository exploration and iterative debugging compared to tasks from other datasets, underscoring their inherent complexity and difficulty.

Quality assurance is further supported by expert validation: four senior Ph.D. students manually audited 100 randomly sampled instances using a cross-validation protocol, confirming that 94 instances were valid with correct environments, unit tests, and problem statements. This 94% validity rate is attributed to three factors: the use of state-of-the-art models (Gemini 3 Pro) for problem statement synthesis, the LLM-as-a-Judge approach for preemptive filtering of low-quality repositories and PRs, and the enforcement of a fixed execution order for P2P and F2P tests to prevent test pollution.

5. 综合价值与局限 (Synthesis — Value and Limitations)

The theoretical significance of Scale-SWE extends beyond its immediate empirical results to reshape how the research community conceptualizes the relationship between data and agent capability. By demonstrating that agentic systems can be deployed to construct their own training data, the paper introduces a recursive paradigm that could accelerate progress in autonomous software engineering. The work provides a new conceptual tool—the sandboxed multi-agent pipeline—that can be adapted, extended, and improved upon by subsequent researchers. It also validates the hypothesis that real-world data, when properly curated through intelligent automation, carries an intrinsic advantage over synthetic alternatives that cannot be compensated for by mere volume.

The practical impact of this work is substantial and multifaceted. For researchers and practitioners seeking to train or fine-tune coding agents, Scale-SWE-Data offers a ready-made resource of unprecedented scale and diversity. The dataset's 100,000 verified instances span 5,200 repositories, providing exposure to a wide array of coding styles, architectural patterns, and bug types. This breadth is particularly valuable for training generalist agents capable of handling diverse real-world tasks rather than overfitting to narrow domains. The demonstrated ability to boost a 30B parameter model to 64% resolve rate on SWE-Bench-Verified also suggests a pathway for developing highly capable coding assistants without the prohibitive costs associated with training or deploying hundred-billion-parameter models.

Among the paper's strengths, the most compelling is its holistic systems-level thinking. The authors do not merely propose a new dataset; they design, implement, and validate a complete pipeline from raw repository selection through final agent evaluation. Each component—the repository filtering, the environment building, the test generation, the problem statement synthesis, the anti-cheating measures, and the trajectory distillation—is carefully considered and rigorously evaluated. The experimental design is similarly commendable, with controlled comparisons against both real-world and synthetic datasets using identical training pipelines, enabling fair attribution of performance differences to data quality rather than training procedure.

However, honest assessment requires acknowledging several limitations. The most significant is the current restriction to Python repositories. While Python represents a substantial portion of open-source software, the generalizability of the approach to languages with different packaging ecosystems, build systems, and dependency management tools (such as Java with Maven/Gradle, C++ with CMake, or Rust with Cargo) remains to be demonstrated. The authors explicitly acknowledge this limitation and identify multi-language extension as a key direction for future work.

Another limitation concerns the computational cost of the pipeline itself. While the paper emphasizes scalability, constructing 100,000 verified instances required processing 6 million pull requests with multiple agent interactions per instance, including Docker environment builds and iterative test executions. The resource requirements, while manageable for well-resourced organizations, may pose barriers for smaller research groups seeking to replicate the full pipeline or extend it to new domains. The reliance on proprietary models (Gemini 3-Pro for PSWA, DeepSeek-V3.2 for distillation) also introduces dependencies that may affect reproducibility and accessibility.

The quality assurance, while impressive at 94% expert-validated accuracy, also implies a 6% error rate that could propagate to training data. In a dataset of 100,000 instances, this translates to approximately 6,000 potentially problematic cases. While the authors mitigate this through rigorous filtering and LLM-as-a-judge approaches, the presence of imperfect data remains a reality that downstream users must consider. Additionally, the fixed execution order for P2P and F2P tests, while necessary to prevent test pollution, may mask certain types of state-dependent bugs that only manifest under specific execution sequences.

Broader implications of this work connect to several trends in the field. The success of Scale-SWE reinforces the growing emphasis on data curation and quality over mere scale, echoing findings in other domains that carefully filtered, high-quality datasets often outperform larger but noisier alternatives. It also contributes to the evolving understanding of agentic workflows, suggesting that specialization—different agents for different subtasks—can be more effective than monolithic approaches. Looking forward, this work opens several promising research directions, including the extension to multilingual codebases, the exploration of reinforcement learning on the distilled trajectories (beyond the supervised fine-tuning demonstrated), and the potential for iterative dataset improvement where trained agents are used to construct even better training data for subsequent generations.

6. 延伸阅读与思考 (Further Reading and Reflection)

To fully appreciate the contributions of Scale-SWE, one must situate it within the broader landscape of software engineering benchmarks and agent development. The most direct intellectual predecessors are SWE-bench (Jimenez et al., 2023) and SWE-bench-Verified (Chowdhury et al., 2024), which established the foundational benchmark for evaluating language models on real-world GitHub issues. These works defined the task formulation—providing a problem statement and repository state, requiring the agent to produce a patch that passes validation tests—that Scale-SWE adopts and scales. The SWE-agent framework (Yang et al., 2024a) and OpenHands (Wang et al., 2025b) provided the scaffolding that enables agents to interact with development environments, executing shell commands, editing files, and browsing documentation—capabilities that Scale-SWE both leverages for its own pipeline and enhances through improved training data.

On the dataset construction side, several parallel efforts illuminate different approaches to the same challenge. SWE-Gym (Pan et al., 2024) pursued real-world data but was limited to 2,400 executable instances from 11 repositories due to the manual effort required for environment construction. SWE-rebench (Badertdinov et al., 2025) expanded this to 7,500 instances from 3,500 repositories through increased automation, but still fell short of Scale-SWE's scale. On the synthetic side, SWE-smith (Yang et al., 2025a) and SWE-Mirror (Wang et al., 2025a) demonstrated that artificial issue generation could produce large datasets (50,000 and 60,000 instances respectively), but the comparison in Scale-SWE reveals that these synthetic approaches suffer from diminished training efficacy and reduced bug category diversity. R2E-Gym (Jain et al., 2025) took yet another approach, using procedural environments and hybrid verifiers to generate training data, achieving 4,600 instances from 10 repositories.

In terms of model development, the competitive landscape includes SWE-RL (Wei et al., 2025), which applied reinforcement learning to Llama3-70B; KAT-Dev-32B (Zhan et al., 2025), which achieved the previous state-of-the-art open-source result of 62.4%; and various specialized models like SWE-Lego-32B (Tao et al., 2026) and SWE-Fixer-72B (Xie et al., 2025). The Scale-SWE-Agent's ability to surpass these models with a smaller parameter count underscores the critical role of training data quality.

Looking ahead, several promising research directions emerge from this work. The most immediate is the extension to other programming languages, which the authors explicitly identify as future work. Beyond language expansion, there is significant potential in exploring reinforcement learning on the distilled trajectories. The current work uses supervised fine-tuning, but the 71,498 trajectories provide a rich substrate for RLHF (Reinforcement Learning from Human Feedback) or RLAF (Reinforcement Learning from AI Feedback) approaches that could further refine agent behavior. Another direction involves iterative dataset construction: using the trained Scale-SWE-Agent to identify high-quality PRs or even generate synthetic but realistic issues, creating a virtuous cycle of data improvement.

The deepest unsolved challenges in this area transcend any single dataset or model. How can we train agents that generalize across radically different software ecosystems? How do we evaluate agents on tasks that require understanding of domain-specific knowledge, not just general programming skills? And how do we ensure that autonomous coding agents remain safe, secure, and aligned with human intentions as they take on increasingly complex engineering tasks? Scale-SWE's insights about the importance of real-world diversity and executable validation provide a foundation for addressing these challenges, but much work remains.

On a personal level, the most thought-provoking aspect of this work is its recursive structure: using agents to build data to train better agents. This echoes broader themes in artificial intelligence research about bootstrapping and self-improvement, and raises fascinating questions about whether we are approaching a threshold where the pace of progress in coding agents becomes self-accelerating. The fact that a 30B parameter model, when trained on carefully curated data, can outperform 70B parameter models suggests that we have not yet reached the limits of what can be achieved through intelligent data engineering. If the next generation of agents can not only solve software engineering tasks but also autonomously identify, validate, and curate new training instances, the scaling of coding agent capabilities may enter a qualitatively new phase—one where the agents themselves become active participants in their own development.

Topics:

• "multi_agent_systems"
• "software_engineering"
• "coding_agents"
• "self_evolving_agents"
• "data_scaling"
  References:
• "renmin_university"
• "bytedance_seed"
• "swe_bench"