Learning & Reasoning / CVPR Workshop 2024 (MMFM)

Before Captioning a Video, Give the Model a List of What It Might Be Seeing

CaptionT5 adds candidate objects and actions to a language model's frame features. Its advantage over a video-specific baseline is largest when only a few frames are available.

Show Think and Tell: Thought-Augmented Fine-Tuning of Large Language Models for Video Captioning

Byoungjip Kim, Dasol Hwang, Sungjun Cho, Youngsoo Jang, Honglak Lee, Moontae Lee

Overview

A video caption has to name what is on screen and what is happening. With only a handful of frames, that can mean turning incomplete visual evidence into a complete sentence. CaptionT5 helps the captioning model by suggesting words for likely objects and actions, then tests whether those suggestions improve the final description.

Key Findings

A short video may show a person taking an object from a shelf. The caption should mention the action that matters. CaptionT5 gives the text generator visual features and a few candidate words, then checks whether a caption fits the clip.

Turn some of the visual evidence into words

A frozen CLIP encoder produces features for sampled frames. The authors also use CLIP to compare the mean frame feature with a vocabulary drawn from 1,000 ImageNet object categories and 700 Kinetics action categories. During training, the method samples phrases using those similarities. At inference it selects the highest-ranked phrases. T5 receives both the projected frame features and the phrases. A separate CLIP score can rerank generated caption candidates. The authors call these short phrases “thought prompts.” Pipeline, Figure 2 and Section 3

CaptionT5 passes visual embeddings and selected object and action phrases to T5, then uses CLIP to rank generated captions.
CaptionT5 passes visual embeddings and selected object and action phrases to T5, then uses CLIP to rank generated captions.

Sparse frames change the comparison

Sampled framesSwinBERT CIDEr on VATEXCaptionT5 CIDEr on VATEX
247.460.0
448.264.2
865.267.5
1668.467.5

CIDEr scores agreement with human reference captions. In the eight-frame ablation, adding the object and action prompts raises VATEX CIDEr from 61.4 to 66.1. Reranking brings the score to 67.5. SwinBERT leads in the 16-frame row, so the size of the advantage depends on the amount of visual input. Ablation and frame comparison, Tables 2 and 3

Sample ideas during training, select them at use time

The candidate phrases come from object and action vocabularies. CLIP measures how closely they match the sampled frames. Sampling at training time gives the generator more than one way to describe a clip. Inference instead uses highly ranked candidates. The ablation compares that training strategy with providing the top-ranked phrases every time. At eight frames, sampling during training scored 66.10 on VATEX, against 62.94 for deterministic top-kk and 61.42 without these prompts. A separate ablation scored 65.03 with object words alone and 65.93 with action words alone. Both kinds of hint together achieved 66.10. Tables 4 and 5

A second score can choose among completed captions

Reranking compares candidate captions with the clip through a separate CLIP score. The hints can change what T5 proposes, while reranking chooses from the captions it has already written. At two frames, reranking lifts VATEX CIDEr from 57.28 with beam search to 60.00. At eight frames the scores are 66.10 and 67.52, respectively. The gains are sensitive to the comparison and the number of frames. The eight-frame model also scored 66.1 on MSR-VTT and 128.6 on YouCook2 in the paper's model comparison. These are benchmark measures of agreement with reference captions. They cannot establish that every event in a clip was described correctly. Tables 1 and 7

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What are the thought prompts?

A: They are candidate object and action phrases retrieved using visual similarity. The captioner sees them alongside the video features.

Q: Why sample phrases during training?

A: Sampling exposes T5 to different plausible cues for the same clip. In the reported comparison, similarity sampling improves VATEX CIDEr over training with deterministic top-ranked phrases.

Q: What could a high caption score still miss?

A: CIDEr rewards agreement with reference captions. A plausible but unsupported action can still resemble the references, and an event between sampled frames can go unseen by both the visual encoder and the phrase suggestions. For a tutorial archive, separately score whether the named object and action are supported by the video.

Future Ideas

Draft descriptions for an archive that can afford only a few frames

Pilot a search archive for short instructional clips. Caption each clip from a fixed four-frame budget, then acquire more frames when object or action scores computed for individual frames disagree. Compare with fixed four-frame and eight-frame systems at matched overall cost. Have annotators verify visible objects, actions, and ordering. Measure retrieval recall and unsupported details before using the captions in a public archive.

Sources