Overview
An image classifier sees a new set of classes and rehearses the old ones from a small collection of saved images. The older classes may still be in the batch, yet the newest classes can dominate the statistics used to normalize the network’s features. Moontae and his colleagues ask whether changing that calculation can help the classifier retain its older tasks. The experiments discussed here are from the January 2022 arXiv revision.
Key Findings
A new training task can change the scale of an old one
A classifier learns a fresh group of categories while keeping only a few examples from earlier tasks. Those earlier examples are there to prevent forgetting, yet they still pass through layers whose idea of a typical feature comes from the current batch. The workshop paper follows what happens when the batch is mostly new data and the test set asks about every learned category. The authors also point out that the running means can drift each time a new task begins, changing how the network sees the remembered features during training itself.
Test which part of the layer needs balance
Batch normalization uses an estimated mean and variance to recenter and rescale each feature channel. It then applies a learned scale and offset. The authors take a fine-tuned model and perform an oracle recalibration with access to all earlier data. In Figure 2, recomputing the means and variances by itself slightly hurts average accuracy. Recalibrating the learned scale and offset too gives a clearer improvement. The oracle comparison is a diagnostic. During ordinary replay the model has only a limited set of saved examples.
Make each task count in the batch
The proposed layer reshapes the current examples into groups along the channel axis. It repeats the replay examples across those groups. The intent is that the number of new examples per group matches the mean number of replay examples per earlier task. Let be the current-example count, the replay count, and the task index. The target in Equation 5 is
The rule assumes the current-to-replay batch-size ratio is an integer. The ideal can still fail to divide the batch evenly. The authors choose the largest common divisor of and below that ideal in the fallback case. They also use the reshaped features to learn the channel scale and shift with less bias toward the current examples. These steps change training. The test-time normalization retains the ordinary batch-normalization form.
Follow the gain through two task schedules
Table 2 reports the following three-run averages for ImageNet-100 with ten tasks of ten classes. Accuracy measures performance after the final task. The forgetting measure looks at the loss of accuracy on earlier tasks.
| Replay method | Ordinary BN accuracy | Task-balanced BN accuracy | Ordinary BN forgetting | Task-balanced BN forgetting |
|---|---|---|---|---|
| Fine-tuning | 39.72 | 45.73 | 47.52 | 40.95 |
| EEIL | 40.91 | 45.51 | 45.87 | 41.13 |
| BiC | 52.50 | 52.01 | 11.21 | 12.60 |
The results vary by training method. BiC already has a mechanism for correcting biased predictions, and its result on this benchmark illustrates the limits of layering two changes together. In the 20-task ImageNet-100 schedule in Table 3, fine-tuning reaches 43.67 with the new layer and 38.89 with the ordinary one. The paper also reports improvement for the fine-tuning setup across the four tested backbones in Table 4. In the component ablation, fixing the group factor rather than adapting it produces the largest drop among the single-component changes. The supplementary timing comparison reports 12.7 hours for fine-tuning with ordinary batch normalization and 15.5 hours with the balanced layer on ImageNet-100.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Does repeating a replay feature give the model a new example?
A: The feature represents the same saved example. Repeating it changes its contribution to several normalization groups. The quality and coverage of the replay buffer still determine what information is available.
Q: Why use the average number of examples per past task?
A: The ratio gives a simple target from the batch counts and the number of tasks. If some tasks are underrepresented within the buffer, matching the average still leaves variation among past tasks. Measuring that variation matters when replay coverage is uneven.
Q: Should I expect the same gain with BiC?
A: The selected BiC result suggests checking the combination empirically. An intervention that helps simple fine-tuning can overlap with a correction already built into a stronger replay method. Evaluate forgetting and current-task learning together.
Q: What happens when old and new examples cannot be divided into matching groups?
A: The replication and grouping procedure depends on the counts available in the two parts of the batch. Feasible group sizes can change over a sequence of updates. The short-task results in Table 2 and the separate fine-tuning comparison in Table 1 use different settings. A deployment test should use the actual buffer and update schedule rather than assume the reported gain transfers.
Future Ideas
A normalization check before adding a new wildlife class
A camera-trap classifier could run a shadow update when a project adds a species. Keep the same small replay buffer for four pilots: ordinary BN, balanced replay sampling, task-balanced BN, and a post-training calibration pass. Use cameras and observation periods outside the training set, then measure recall for rare earlier species, recall for the new species, and training time. Repeat the comparison at several buffer sizes, and inspect cases with poor replay coverage. Deploy an update only when the improvement on rare earlier species is consistent across locations and the new-class performance remains within a predeclared margin.
Specify whether a new species, a new camera deployment, or both define a task. Log batch counts and any update where the intended group size is infeasible.
Sources
- January 2022 arXiv revision
- Version 1 PDF with the figures and experiment tables
- CLVISION 2022 accepted papers
Original abstract
Batch Normalization (BN) and its variants has been extensively studied for neural nets in various computer vision tasks, but relatively little work has been dedicated to studying the effect of BN in continual learning. To that end, we develop a new update patch for BN, particularly tailored for the exemplar-based class-incremental learning (CIL). The main issue of BN in CIL is the imbalance of training data between current and past tasks in a mini-batch, which makes the empirical mean and variance as well as the learnable affine transformation parameters of BN heavily biased toward the current task -- contributing to the forgetting of past tasks. While one of the recent BN variants has been developed for "online" CIL, in which the training is done with a single epoch, we show that their method does not necessarily bring gains for "offline" CIL, in which a model is trained with multiple epochs on the imbalanced training data. The main reason for the ineffectiveness of their method lies in not fully addressing the data imbalance issue, especially in computing the gradients for learning the affine transformation parameters of BN. Accordingly, our new hyperparameter-free variant, dubbed as Task-Balanced BN (TBBN), is proposed to more correctly resolve the imbalance issue by making a horizontally-concatenated task-balanced batch using both reshape and repeat operations during training. Based on our experiments on class incremental learning of CIFAR-100, ImageNet-100, and five dissimilar task datasets, we demonstrate that our TBBN, which works exactly the same as the vanilla BN in the inference time, is easily applicable to most existing exemplar-based offline CIL algorithms and consistently outperforms other BN variants.