Retrieval & Search / AAAI 2024

Can This Video Answer That Question?

YTCommentQA collects real questions from YouTube tutorials and asks what evidence the video offers. The tests distinguish answers that are shown, spoken, or only available when viewers put both together.

YTCommentQA: Video Question Answerability in Instructional Video

Saelyne Yang, Sunghyun Park, Yunseok Jang, Moontae Lee

Overview

A repair video may show a latch release while the presenter says only “pull this.” The transcript can tell a reader what to do without identifying the part. The picture can identify the part without conveying the instruction. YTCommentQA tests whether a system can tell when the video contains enough evidence to answer a viewer, and which kind of evidence it would need.

Key Findings

A timestamp can point to a question without answering it

A viewer asks about something in an instructional video. Someone replies with a timestamp. That moment might contain the answer, or it might only identify the tool or ingredient the viewer meant. The researchers use timestamped replies as leads for annotators. They collect English questions that real viewers posted, then check what the timestamp is doing before looking for evidence around it. Annotators can extend the window if the answer spans more time. The resulting YTCommentQA benchmark contains 2,332 questions across 2,004 videos. Collection and annotation workflow

Figure 1. Questions about one tutorial can require evidence from speech, images, or their combination.
Figure 1. Questions about one tutorial can require evidence from speech, images, or their combination.

Separate a visible answer from one that needs the narration

Annotators inspect silent video and the transcript. If a temperature is both visible and spoken, either source is enough. If a narrator refers to a particular part while the camera shows which part it is, the two sources may provide different pieces that are needed together. Whole-video evaluation distinguishes these two possibilities. The annotation table counts some questions as both visual-answerable and script-answerable, so its rows are overlapping. Table 2

TypeQuestionsShare
Visual answerable1,42761.19%
Script answerable1,15549.53%
Combined answerable1747.46%
Unanswerable32613.98%

Give the model a clip, then take away the timestamp

The first benchmark task asks whether the nearby segment can answer a question. The second asks which kind of evidence is needed across the whole video, with five choices including unanswerable. The language-only systems receive written descriptions of the visual material, including frame captions and on-screen text, alongside transcript text. For the full-video task, the authors summarize short chunks to fit the context budget. The multimodal baseline samples frames and selects keyframes.

ModelSegment answerability F1Whole-video classification accuracy
Fine-tuned Llama-2 7B45.7836.24
Fine-tuned Llama-2 13B55.4937.70
Zero-shot GPT-433.0227.03
SeViLA multimodal baseline46.5535.27

F1 combines precision and recall. Whole-video accuracy measures agreement with the reference class. The setups differ in adaptation and in the evidence they see, so the two columns represent different challenges. Table 4 · Table 5

The hardest question can need two incomplete clues

The authors report that 85% of combined-answerable instances were misclassified in their discussion of GPT-4’s whole-video results. They also observe a tendency to call visually answerable cases unanswerable in the segment task. These failures matter to a video assistant deciding whether it has enough evidence. A transcript can be informative and still omit the visual clue that makes a question answerable. The paper’s own annotations begin near human-supplied timestamps, so they offer stronger evidence about those segments than about evidence spread over distant parts of a video. Results and limitations

Questions & Answers

Q: Why does a timestamp fail to guarantee an answer?

A: A reply can point to an object or event without explaining what a viewer wants to know. The researchers use the timestamp to locate likely evidence and ask annotators to judge whether the material actually resolves the question.

Q: What is the difference between “both” and “combined”?

A: In the five-class task, both script and visual answerable means either source can independently answer. Combined answerable means each supplies part of the answer and the system needs them together. Seeing the object and hearing what happens to it is one example.

Q: Do the model scores measure the accuracy of their answers?

A: The experiments measure whether a model correctly classifies answerability and evidence type. Checking an actual response would also require a ground-truth answer or a person to judge whether the response is supported by the cited evidence.

Q: Could an answer marked unanswerable appear later in the video?

A: That is possible. Annotation starts near a reply’s timestamp and permits limited expansion. The authors identify full-video and cross-segment evidence as future directions. A broader annotation protocol could record when an answer appears in another part of the video.

Q: How should a tutorial system decide when it has enough evidence to answer?

A: Answerability labels identify which evidence would suffice, and they are concentrated near reply timestamps. A useful system would calibrate its confidence on questions with independently checked sources throughout the video. Reporting both accuracy among answered questions and the share answered makes the cost of abstaining visible.

Future Ideas

Let a tutorial answer with the clip that proves it

A help feature for a curated tutorial library could retrieve likely clips for a question, then test whether the transcript or frames suffice on their own. If the answer requires both, the player could show the exact passage and a still of the matching action. If the first clip is insufficient, the feature could search another segment or ask the viewer which part they mean. For unanswered questions it would say what evidence it found and where it fell short.

Begin with one tutorial category in which human reviewers can judge the answers. Use a sample of questions collected without relying on timestamped replies and label supporting intervals across the entire video. Compare a transcript-only search, ordinary multimodal retrieval, and the feature at the same review and inference budget. Track whether the cited evidence supports the response, how often the model answers despite missing evidence, and how long it takes viewers to reach the right clip.

Report evidence-supported accuracy at several coverage targets, and compare the time and accuracy effects of searching another clip when the first one fails.

Sources

Original abstract

Instructional videos provide detailed how-to guides for vari- ous tasks, with viewers often posing questions regarding the content. Addressing these questions is vital for comprehend- ing the content, yet receiving immediate answers is difficult. While numerous computational models have been developed for Video Question Answering (Video QA) tasks, they are pri- marily trained on questions generated based on video content, aiming to produce answers from within the content. However, in real-world situations, users may pose questions that go be- yond the video’s informational boundaries, highlighting the necessity to determine if a video can provide the answer. Dis- cerning whether a question can be answered by video content is challenging due to the multi-modal nature of videos, where visual and verbal information are intertwined. To bridge this gap, we present the YTCommentQA dataset, which contains naturally-generated questions from Y ouTube, categorized by their answerability and required modality to answer – visual, script, or both. Experiments with answerability classification tasks demonstrate the complexity of YTCommentQA and emphasize the need to comprehend the combined role of vi- sual and script information in video reasoning. The dataset is available at https://github.com/lgresearch/YTCommentQA.