Retrieval & Search / CVPR Workshop 2024

Can the Video Answer the Question at All?

YTCommentQA starts with real questions under instructional videos and asks which evidence, if any, could answer them.

YTCommentQA: Video Question Answerability in Instructional Video

Saelyne Yang, Sunghyun Park, Yunseok Jang, Moontae Lee

Overview

Someone watching a repair tutorial asks how to remove the top part. The narrator may never name the step, but the motion on screen can make it clear. Another question needs a detail the video never shows. This manuscript evaluates whether an assistant can tell those situations apart.

Key Findings

A person watching a repair video asks, “What size is that wrench?” The transcript might never name the size. In another clip the speaker gives the size while the tool is out of frame. A question-answering system first has to work out which source of evidence could support an answer.

Viewer comments and timestamped replies help locate evidence that can be visual, spoken, both, or absent.
Viewer comments and timestamped replies help locate evidence that can be visual, spoken, both, or absent.

Treat missing evidence as an answerability label

The dataset uses viewer questions and timestamped replies from instructional videos. The timestamp helps locate candidate evidence, but a reply can clarify a question without identifying an answer. The annotators distinguish visual, script, and combined evidence, alongside unanswerable cases. Combined questions need both modalities. The paper reports that 13.98% of questions are deemed unanswerable. The initial snippets are capped at 60 seconds and annotators can extend them by up to three transcript sentences. Whole-video labels still inherit judgments focused around the reply timestamp.

Table 4 scores answerability for video segments using F1. Table 5 separately scores whole-video answerability and evidence modality using accuracy. The selected values below come from the archived arXiv manuscript.

Model and settingSegment F1Whole-video accuracy
Fine-tuned Llama-2 7B45.7836.24
Fine-tuned Llama-2 13B55.4937.70
Zero-shot GPT-433.0227.03
SeViLA multimodal baseline46.5535.27

The study reports that visual evidence is easy for a zero-shot language model to miss. In the segment test, 53% of GPT-4’s false cases are visual-answerable questions marked unanswerable.

Let the viewer’s question define the task

The viewer comments provide naturally occurring questions rather than a set written to match the transcript. Timestamped replies help locate likely evidence and can still be misleading. The annotation separates cases requiring visual information, spoken information, or both. Including unanswerable questions lets the evaluation test whether a system can tell when evidence is absent.

Check the short clip and the entire video

The benchmarks evaluate segments and whole videos separately. A short clip reduces the search space. A whole video gives the system more evidence and more opportunities to seize on a similar but irrelevant moment. The tables test how models behave under both conditions. The timestamp-centered label construction is important when interpreting whole-video errors, since useful material elsewhere may be harder to discover or annotate.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why not treat a timestamp as proof of answerability?

A: A reply can point to the confusing part without providing the requested detail. The labeled segment still needs to support an answer.

Q: What makes a question combined?

A: The evidence from the frames and the script is jointly necessary. Either one alone is insufficient.

Q: What does the whole-video label leave uncertain?

A: The study locates likely evidence using the timestamped reply, and the full-video labels inherit those localized judgments. This can miss evidence elsewhere in a tutorial. The benchmark also draws from videos where someone commented and replied, which shapes the kinds of questions that get annotated. Testing on held-out authors helps measure how much a system depends on those patterns.

Future Ideas

A tutorial player that cites the frame and transcript

Build a prototype that answers only when it can show the relevant clip or transcript passage. Compare transcript-only, frame-only, and combined retrieval at matched review effort. On videos held out by author, have annotators score support, unnecessary refusals by evidence modality, and whether the cited clip helps a viewer carry out the step.

Sources

Original abstract

Instructional videos provide detailed how-to guides for various tasks, with viewers often posing questions regarding the content. Addressing these questions is vital for comprehending the content, yet receiving immediate answers is difficult. While numerous computational models have been developed for Video Question Answering (Video QA) tasks, they are primarily trained on questions generated based on video content, aiming to produce answers from within the content. However, in real-world situations, users may pose questions that go beyond the video's informational boundaries, highlighting the necessity to determine if a video can provide the answer. Discerning whether a question can be answered by video content is challenging due to the multi-modal nature of videos, where visual and verbal information are intertwined. To bridge this gap, we present the YTCommentQA dataset, which contains naturally-generated questions from YouTube, categorized by their answerability and required modality to answer -- visual, script, or both. Experiments with answerability classification tasks demonstrate the complexity of YTCommentQA and emphasize the need to comprehend the combined role of visual and script information in video reasoning. The dataset is available at https://github.com/lgresearch/YTCommentQA.