Learning & Reasoning / NeurIPS 2022

Teaching a Smaller Image Model With Words It Can Compare

BeamCLIP transfers an image model's relationship to text prompts as well as its image representations. A compact ResNet learns useful features from unlabeled images.

Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, Honglak Lee

Overview

An image of a snapdragon can be close to the wrong meaning of the word if the text prompt gives no botanical context. This paper makes such prompts part of the way a large image-and-text model teaches a smaller image model. The student learns both how the teacher represents an image and how the image relates to a collection of text descriptions.

Key Findings

Can a smaller model inherit how CLIP sees?

Picture a dog photo from CIFAR-10, a small image benchmark with ten classes. A large image-and-text model called CLIP can place that photo and the phrases "A photo of cat" or "A photo of horse" in one shared space. The angles between the dog and each phrase carry a lot of information about what the dog looks like. Moontae and his colleagues asked whether a small ResNet could inherit that structure without the 400 million image-text pairs and hundreds of GPUs behind CLIP. Their method, BeamCLIP, treats the large CLIP vision transformer as a teacher and a compact ResNet as the student.

Why matching one vector leaves the student lost

The simplest transfer asks the student to regress the teacher's image vector (Eq. 2). After normalizing both vectors to unit length, call the student's qiq_i and the teacher's kik_i. Minimizing the squared distance then equals maximizing their dot product, the cosine similarity (Eq. 3). The authors call this instance similarity matching, and it becomes the loss in their Eq. 4:

LISM=−∑i=1Nqi⋅ki\mathcal{L}_\text{ISM} = -\sum_{i=1}^{N} q_i \cdot k_i

The authors point out a flaw. Two student vectors pointing in different directions can have the same cosine similarity to a single teacher vector, so one reference point leaves the geometry ambiguous. Figure 5 in the appendix illustrates this topological ambiguity.

Text phrases as extra reference points

The fix uses language. The teacher's text encoder turns MM class phrases into anchor vectors aja_j. For an image, the teacher computes a softmax over its similarities to every anchor, with temperature τ=0.01\tau = 0.01 (Eq. 5):

sj(ki,A)=exp⁡((ki⋅aj)/τ)∑m=1Mexp⁡((ki⋅am)/τ)s_j(k_i, A) = \frac{\exp\left((k_i \cdot a_j)/\tau\right)}{\sum_{m=1}^{M} \exp\left((k_i \cdot a_m)/\tau\right)}

Collecting these gives a distribution P(ki∣A)P(k_i|A) over anchors, and the student computes P(qi∣A)P(q_i|A) the same way. Cross-modal similarity matching trains the student to match the teacher's distribution with cross-entropy and also to minimize the entropy of its own distribution, which pulls it toward the anchors (Eq. 7). The full loss adds the instance term with weight λISM=10\lambda_\text{ISM} = 10 (Eq. 8). Training uses unlabeled images and a shared set of anchor phrases.

The teacher measures how similar a query image is to each text anchor. The student learns to mimic that pattern.
The teacher measures how similar a query image is to each text anchor. The student learns to mimic that pattern.

When a flower name misleads the text encoder

Anchors only help if their embeddings mean the right thing. The authors found that ambiguous labels can yield semantically incorrect text embeddings. Flowers102 includes "snapdragon," "bird of paradise" and "colt's foot." Their context-based prompt augmentation appends a Wikipedia description for Flowers102 and Pets37, as Table 1 shows for the snapdragon. For CIFAR100 and ImageNet it appends the coarse category from the class hierarchy, as in "A photo of a beaver, categorized as aquatic mammals." The authors list the extra engineering this requires as a limitation.

Two further details support training. The student is first pre-trained with SimCLR, a self-supervised method, and a momentum copy of the student updated by exponential moving average (Eq. 9, momentum 0.99) stabilizes optimization. The appendix also describes label smoothing on the teacher's distribution, with smoothing value 0.2.

Beating self-supervised learning on ImageNet

The headline test is a linear probe, which fits a simple classifier on frozen features. Table 3 reports ImageNet-1K top-1 accuracy for a ResNet-18 trained from scratch.

ResNet-18 method100 epochs200 epochs400 epochs
SimCLR47.149.951.8
SwAV57.761.263.7
OSS (ResNet-50 teacher)60.064.165.8
BeamCLIP63.864.866.2

Supervised ResNet-18 scores 69.8%. For ResNet-50, Table 2 shows BeamCLIP at 74.8%, 75.1% and 75.0% at 200, 400 and 800 epochs, ahead of SwAV's 72.0% and 74.3% where those values exist, with supervised training at 76.2%. Table 4 shows SwAV pre-training of the student lifting the 200-epoch result from 74.8% to 75.8%. The authors caution that self-supervised methods improve with longer training, so the margin may shrink.

Transfer across datasets and ablations

Table 6 compares ResNet-50 students against the CLIP teacher's zero-shot accuracy. Conventional distillation with KL divergence reaches 56.38% on ImageNet-1K and contrastive representation distillation reaches 30.90%. BeamCLIP reaches 66.17% against the teacher's 68.6%. It exceeds the teacher on CIFAR10 (92.10 against 91.6) and Flowers102 (75.86 against 70.4).

The ablation in Table 7 shows each piece contributing. Cosine matching alone gives 39.07% on CIFAR10 without pre-training and 87.00% with SimCLR. Flowers102 depends heavily on pre-training, rising from 12.94% to 62.18% with the cross-entropy anchor loss. The authors suspect the dataset's 1,020 training images are too few to probe the teacher's space. Prompt augmentation then lifts Flowers102 to 75.86%.

Top-5 text-to-image retrieval on Flowers102 for CLIP-RN50 and BeamCLIP-RN50. Red boxes mark errors.
Top-5 text-to-image retrieval on Flowers102 for CLIP-RN50 and BeamCLIP-RN50. Red boxes mark errors.

Zero-shot ability and imperfect prompts

Trained on unlabeled ImageNet-21K images, which do not overlap ImageNet-1K, BeamCLIP-RN50 reaches 57.5% zero-shot accuracy (Table 5). CLIP-RN50 scores 59.6% using WIT-400M, and OpenCLIP trained on CC-12M scores 36.5%. The authors describe their data as about 3% of CLIP's.

Finally, Figure 4 tests incomplete anchors on CIFAR100. With 20 of 100 class names, the cross-entropy plus entropy variant scores 21.72% in Table 15, rising to 67.35% with all 100. Random ImageNet class names as anchors still work, giving 58.93% with 1,000 prompts in Table 16. The KL variant stays flat near 46 to 56%. These results come from the authors' experiments on these datasets and a single teacher family.

Effect of random text prompts on CIFAR100, with the teacher's accuracy as an upper bound.
Effect of random text prompts on CIFAR100, with the teacher's accuracy as an upper bound.

Questions & Answers

Q: What does the student receive when the image has no class label?

A: The teacher supplies an image vector and similarities to the text anchors. The training loss asks the student to reproduce those signals using its own image representation.

Q: What makes a phrase a useful anchor?

A: It should give the text encoder a specific meaning that corresponds to what can be seen. A little context helps when a label is shared by different objects or concepts.

Q: Does a high linear-probe score imply zero-shot performance?

A: A linear probe is trained with labeled examples after the image encoder is frozen. Zero-shot classification compares an image with text descriptions without fitting that classifier. The paper evaluates these settings separately.

Q: What would show that language anchors help beyond better captions alone?

A: Use the same teacher, student initialization, images, and training budget while varying the anchoring loss and prompt descriptions independently. The paper’s linear-probe and zero-shot evaluations answer related questions with different evaluation signals. An additional unknown-class check would reveal whether the method makes uncertain predictions too sharp.

Future Ideas

Test which field-guide captions make a small model more useful

Train a compact plant recognition model with unlabeled photographs and botanist-written descriptions of visible features. Compare image-only distillation, bare species-name anchors, and descriptions that emphasize traits distinguishing close relatives. Use the same teacher and student budget in every condition.

Evaluate on held-out observers and locations, with a separate test for species that share a common name. Measure top-k accuracy and the rate of inappropriate confident predictions on unknown species. This test would reveal whether the richer anchors help when the visual distinction is subtle and the names are ambiguous.

Sources

Original abstract

Despite surprising performance on zero-shot transfer, pre-training a large-scale multimodal model is often prohibitive as it requires a huge amount of data and computing resources. In this paper, we propose a method (BeamCLIP) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP-ViT) into a small target model (e.g., ResNet-18). For unsupervised transfer, we introduce cross-modal similarity matching (CSM) that enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input text prompts. Our experiments show that unsupervised representation transfer of a pre-trained vision-language model enables a small ResNet-18 to achieve a better ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-only self-supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%), while closing the gap with supervised learning (69.8%).