Learning & Reasoning / ACL 2025

A Code Model Learns a Different Lesson From the Same File

Sliding the training window changes which earlier lines the model sees before predicting a token. A file-aware stride improves the reported Swift test with fewer updates than a fixed-overlap schedule.

Overlapping Context with Variable-Length Stride Increases Diversity when Training Large Language Model for Code

Geonmo Gu, Jaeho Kwak, Haksoo Moon, Hyun Seung Shim, Yu Jin Kim, Byoungjip Kim, Moontae Lee, Hyejeong Jeon

Overview

Imagine reading a function definition that starts halfway through a page. One training window ends just before the call that uses it. Another starts a little later and contains both. They share most of their text, yet the model practices a different prediction from the context that changed. This paper uses overlap to make more training examples from a limited code collection.

Key Findings

A program can teach a model different lessons depending on where the training example begins. Open the window at a function’s first line and the model sees the declaration before it predicts the body. Open later and it has to use a different context. The paper takes this small decision about packing and makes it an experimental variable.

Move the window, then respect the file boundaries

Tokens with matching colors show where the same part of the stream reappears as the context window moves.
Tokens with matching colors show where the same part of the stream reappears as the context window moves.

Training pipelines often concatenate tokenized code files and cut the stream into equal-size windows. A stride equal to the window length creates disjoint contexts. A shorter stride creates overlap. The same target token can now occur with a different prefix, giving a different next-token prediction problem. In the in-house dataset at context length 2,048, the authors report 1.263 billion unique examples with a disjoint stride of 2,048, and 10.078 billion with a stride of 256. More than 99% of the enumerated examples are unique by the paper's counting method in both cases. Packing and Table 1

Overlap can also repeat boundaries between unrelated files. The variable-stride method checks the current window for an end-of-file token. If it finds one, it advances past the rightmost such token. If it finds none, it uses the smaller default stride. This keeps overlap within long files while skipping some repeated cross-file contexts.

Count the examples that actually changed

The paper counts training examples as a prefix paired with the next token, so reusing a token can still create a new example. With contexts of 2,048 tokens, these are selected counts from Table 1 of the in-house data. The percentage is the share of all prefix–target pairs that were unique.

Fixed strideUnique prefix–target pairsShare unique
2,0481,263,146,56099.79%
5125,044,421,25199.67%
25610,078,484,17799.57%

At stride 256, the count is almost eight times the nonoverlapping count while the proportion of duplicates rises only slightly. The count measures diversity as the paper defines it. It leaves open whether each additional example has equal training value.

How much was gained on Swift?

CodeGen-350M context scheduleTraining epochsTraining stepsMultiPL-E Swift pass@1 (%)
Variable stride, default 12823,3027.9
Fixed stride 12825,5066.5
Disjoint stride 1,024103,4406.2
Disjoint stride 1,024206,8806.2

The rows use a context length of 1,024, the same batch size, and a learning rate of 0.0002. This is the public-data CodeGen-350M experiment. Pass@1 measures how often the top generated solution passes its tests. The values average the best three checkpoints, with a checkpoint saved every 10% of training. The study also evaluates larger models and private data. More overlap gives the optimizer more windows to process, so the cost depends on how much of the overlap the stride rule keeps. Table 2 and discussion

Change the stride and the length together

A fixed stride ss moves the window by the same number of tokens each time. For a window of length LL and 0<s<L0<s<L, adjacent windows share L−sL-s tokens. The variable-length approach preserves a full initial context while adding shorter training examples for later target segments. The paper describes several ways to implement this without storing all duplicated content as separate copies. That engineering work matters because extra examples otherwise incur a storage cost before training even starts.

Rule out more passes as the only explanation

Table 2 compares variable-length packing with fixed overlap and with additional epochs of disjoint examples. In the public-data Swift comparison for CodeGen-350M, the reported pass@1 is 7.9 for variable length with a stride of 128, 6.5 for fixed length with the same stride, and 6.2 for the disjoint setup after either ten or twenty epochs in the displayed rows. These are low absolute scores on one language. The comparison helps separate a change to the contexts a model sees from simply repeating the same examples.

Put the larger model through a different test

On the private U100 benchmark, the task is to generate a function body from its header and pass unit tests. The 8.8-billion-parameter model trained with overlapping windows peaks at 29% pass@1, versus 21% with nonoverlapping windows. This comparison uses fixed stride for the overlapping model, whereas the public Swift result compares fixed and variable stride in a smaller model. The two tests establish different parts of the case. See the in-house learning curves in Figure 3.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Does overlap introduce new code?

A: The token stream stays the same. What changes is the window containing each token and the prefix the model can use when predicting it. Some training examples therefore expose relationships that the disjoint boundaries would separate.

Q: Why advance to the end of a file?

A: Repeatedly stepping through a window containing unrelated files can spend training on relationships that carry little meaning. Advancing beyond the last file ending in the current window reduces some of that repetition.

Q: Can adding more windows hurt?

A: It increases training cost and can put more emphasis on particular examples. Continual training can also trade gains on a target language against earlier capabilities. The appendix tests general C++ retention on HumanEval-X. Models trained only on in-house code decline at the smallest strides, while mixing in 10% public code changes that pattern.

Q: Does a high count of unique windows mean the model learns a wider range of code?

A: Overlapping windows can put a token in a genuinely different prediction context, such as one where a function definition is visible. They can also create near-duplicates. The Swift result helps compare variable stride with a similar-step disjoint schedule. It still takes token-matched experiments on unseen repositories and multiple languages to show that nominally unique windows add useful context.

Future Ideas

Give a small codebase several deliberate views

For a code assistant trained on a licensed internal codebase, compare ordinary disjoint packing, the paper's variable stride, and a sampler that places windows around definitions and their nearby call sites. Fix the processed-token budget and keep holdout repositories entirely outside training. Use public data during training as a control for retention.

Measure held-out pass@1 and whether generated code correctly uses APIs declared outside the immediate edit region. Track wall-clock training and regressions on an unrelated language. The definition-aware sampler earns further work only if it improves unseen dependency use over file-aware sliding at comparable compute.

Track repeated exposure per file and per span as well as processed tokens. Use the same checkpoint-selection rule and repeat across seeds to separate the effect of the sampler from extra training opportunities.

Sources

Original abstract

The pretraining of code LLMs typically begins with general data and progresses to domain-specific data through sequential stages. In the latter stages, a challenging issue is that the data of a target domain can be limited in size, and conventional approach of increasing the number of epochs does not lead to a performance gain. In this paper, we propose a novel packing method, which is extracting overlapping contexts from the training data using variable-length stride. Our method can mitigate the data-scarcity issue by providing more diverse and abundant examples of next token prediction than non-overlapping contexts. While the training time of our approach is increased proportionally to the amount of augmented examples, we present space-efficient implementations to store overlapping contexts. Extensive experiments with real datasets show that our approach outperforms the conventional approach of controlling the number of epochs in terms of the pass@k rate.