Learning & Reasoning / CVPR 2023

A Small Training Change Helps Vision Models Remember

Task-Balanced Batch Normalization gives earlier classes more influence during training and keeps the usual batch-normalization computation at inference.

Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning

Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, Taesup Moon

Overview

Imagine updating a camera to recognize ten new objects while keeping only a few photographs of everything it learned last year. The model sees far more new examples than old ones. That imbalance reaches inside the network, where batch normalization uses the current training batch to calibrate the scale of its signals. Moontae and his colleagues found that these internal statistics can drift toward the newest task. Their CVPR paper describes a training change that gives earlier tasks more influence over the calibration.

Key Findings

The new class takes over the batch

Imagine an image classifier learning birds this week. It has folders full of new bird images and a short list of remembered photos from last week, when it learned to tell cars apart. Those remembered photos are meant to keep the old knowledge alive. Yet one operation in the network averages features over the whole batch. With birds filling most of the slots, the operation treats this week’s images as its picture of normal. The mismatch grows whenever a training stage brings plenty of fresh data and a small memory of previous classes.

The study follows that imbalance through batch normalization, a layer that recenters and rescales features as the network learns. Earlier work had identified biased means and variances. The authors trace a second effect to the learned scale and shift applied after standardization. Continual Normalization, an earlier method developed for one-pass online training, changes how the features are normalized. The authors find that its learned scale and shift can still receive more influence from the new task in offline training, where the model makes repeated passes through the data.

Updating the averages only gets partway there

A batch-normalization layer has one estimated mean μ^c\hat\mu_c and variance σ^c2\hat\sigma_c^2 per channel cc. It also learns a scale γc\gamma_c and offset βc\beta_c. This is the channelwise version of the paper’s Equation 2. Here hb,c,dh_{b,c,d} is one feature from example bb and position dd, and ϵ\epsilon stabilizes the denominator.

yb,c,d=γchb,c,d−μ^cσ^c2+ϵ+βcy_{b,c,d} = \gamma_c\frac{h_{b,c,d}-\hat\mu_c}{\sqrt{\hat\sigma_c^2+\epsilon}}+\beta_c

The authors ask what would happen if they could use all the old training examples for one controlled test. They first take a trained fine-tuning model, keep its learned parameters fixed, and recalculate the means and variances using balanced data. Then they test updating the scale and shift along with those statistics. Figure 2 shows the benefit of handling both pieces together. This experiment uses data the continual learner ordinarily cannot retain. It gives the team a diagnosis before they design a layer that works with the exemplar memory.

The diagnostic test compares ordinary fine-tuning, balanced recalculation of normalization statistics, and balanced adjustment of the scale and shift as well.
The diagnostic test compares ordinary fine-tuning, balanced recalculation of normalization statistics, and balanced adjustment of the scale and shift as well.

Making the old examples count during training

Task-Balanced Batch Normalization, or TBBN, reshapes the larger group of current examples into parallel sets of channels. It repeats the smaller group of remembered examples so it can pair their features with each group. This lets the layer compute statistics from more balanced groups without throwing away the new examples. After normalization, the current features return to their original shape, and the repeated branches for past examples are combined. The authors use a shared scale and shift across branches and balance how their gradients are computed.

The blue arrows in Figure 3 follow features through the reshape and repeat operations. The red arrows show how gradients return to the original feature groups.
The blue arrows in Figure 3 follow features through the reshape and repeat operations. The red arrows show how gradients return to the original feature groups.

The reshape and repeat factor is determined from the batch composition. At test time, the layer has the same form as ordinary batch normalization and uses the running statistics and learned parameters from training. The authors report extra peak GPU memory of up to about 1.6 times the original and training time of about 1.3 times the original in their ablation. They report no added test-time cost.

Seeing which mistakes disappear

The authors swap the normalization layer into existing class-incremental methods and evaluate CIFAR-100, ImageNet-100, and a sequence of five dissimilar tasks. The following final accuracies come from Table 2. All four rows use fine-tuning or a representative replay method. The CIFAR-100 setting uses ResNet-32 and the ImageNet-100 setting uses ResNet-18.

Training and normalizationCIFAR-100 final accuracyImageNet-100 final accuracy
Fine-tuning + ordinary BN35.4139.40
Fine-tuning + TBBN38.4643.20
EEIL + ordinary BN39.8240.06
EEIL + TBBN41.9345.18

The pattern depends on the method and backbone. For instance, PODNet on CIFAR-100 achieves 38.10 with ordinary batch normalization and 37.90 with TBBN in Table 2. The error analysis clarifies what a gain can mean. In the ImageNet-100 fine-tuning run, fewer images from past classes are incorrectly assigned to a newly introduced class under TBBN. Table 5 tests changes to the two pieces separately. Balancing the training and test statistics alone, or learning only the scale and shift in a balanced way, fails to match the combined layer.

Figure 4 separates four types of mistakes during the ImageNet-100 experiment. Look for the decline in past examples assigned to classes from the current task.
Figure 4 separates four types of mistakes during the ImageNet-100 experiment. Look for the decline in past examples assigned to classes from the current task.

Questions & Answers

Q: Why does the order of new classes matter?

A: Each update is trained with the classes available at that point, so the current data and replay buffer determine which distributions the model sees together. A sequence of closely related tasks presents a different calibration problem from one with abrupt changes in image domain.

Q: Why change the learned scale and offset if the feature statistics are balanced?

A: The loss still determines how each channel should be amplified or shifted. Gradients can be dominated by current-task examples even if the running statistics have been improved. The paper changes both pieces and tests them separately in its ablation.

Q: Can this use a buffer that contains no examples from a past task?

A: The rearrangement needs replay data to estimate past feature statistics. When a prior task is absent, repeated examples from the remaining tasks cannot recover its activations. A deployment should check the buffer’s coverage before relying on the calibration.

Q: If TBBN balances tasks, are rare classes within a task protected automatically?

A: The balance is set at the task level. A rare defect can still be underrepresented within a task, even when older and newer tasks contribute more evenly overall. Per-class false negatives and runs with different replay-buffer sizes reveal that failure more directly than final average accuracy.

Future Ideas

A replay audit for inspection cameras

A factory camera could update on new defect types in shadow mode. Train standard BN and TBBN with the same replay images, add a post-training recalibration baseline, and reserve a small independent audit set for old defect classes and lighting conditions. Before rollout, compare the false-negative rate on each retained defect, performance on the new defect, and the training memory required. Use several production shifts as evaluation blocks and estimate uncertainty by shift. Promote an update only if the reduction in missed older defects clears a predeclared threshold and the new-defect score and compute budget still meet their requirements.

Add a balanced-replay baseline and vary replay capacity while holding the update schedule fixed. Report per-class outcomes within each task and whether gains survive renewed lighting or camera shifts.

Sources

Original abstract

Batch Normalization (BN) and its variants has been ex- tensively studied for neural nets in various computer vision tasks, but relatively little work has been dedicated to studying the effect of BN in continual learning. To that end, we de- velop a new update patch for BN, particularly tailored for the exemplar-based class-incremental learning (CIL). The main issue of BN in CIL is the imbalance of training data between current and past tasks in a mini-batch, which makes the em- pirical mean and variance as well as the learnable affine transformation parameters of BN heavily biased toward the current task — contributing to the forgetting of past tasks. While one of the recent BN variants has been developed for “online” CIL, in which the training is done with a single epoch, we show that their method does not necessarily bring gains for “offline” CIL, in which a model is trained with multiple epochs on the imbalanced training data. The main reason for the ineffectiveness of their method lies in not fully addressing the data imbalance issue, especially in comput- ing the gradients for learning the affine transformation pa- rameters of BN. Accordingly, our new hyperparameter-free variant, dubbed as Task-Balanced BN (TBBN), is proposed to more correctly resolve the imbalance issue by making a horizontally-concatenated task-balanced batch using both reshape and repeat operations during training. Based on our experiments on class incremental learning of CIFAR- 100, ImageNet-100, and five dissimilar task datasets, we demonstrate that our TBBN, which works exactly the same as the vanilla BN in the inference time, is easily applicable to most existing exemplar-based offline CIL algorithms and consistently outperforms other BN variants.