Others & Etc. / DIS 2024

A Researcher’s Best Answer May Start With a Better Question

AQUA helps paper authors draft interview-style articles. In a small study, suggested questions helped authors get started, while generated answers still needed their judgment.

Co-Creating Question-and-Answer Style Articles with Large Language Models for Research Promotion

Hyunseung Lim, Ji Yong Cho, Taewan Kim, Jeongeon Park, Hyungyu Shin, Seulgi Choi, Sunghyun Park, Kyungjae Lee, Juho Kim, Moontae Lee, Hwajung Hong

Overview

An author who has spent months on a paper may no longer remember which detail will puzzle a first-time reader. A question from an interviewer can reveal that missing step. AQUA puts an AI-generated interviewer into a writing tool, offering questions and draft answers while giving the author control of what reaches the article.

Key Findings

For a researcher telling the story of a paper, the first obstacle can be finding the question that gives a reader a way in. The authors studied what happens when an AI system supplies some of those questions and leaves the scientist in charge of the article.

A desk full of question cards

AQUA offers general questions about a paper, questions tailored to the researcher’s background, and follow-ups to an existing answer. The author can add a question to the editing area, rewrite it, revise a generated answer, and decide whether the card is visible in the preview. Figures 1–3 show the interface. The authors used the title and abstract for the shared question prompt after finding that prompting with the whole paper could yield questions too technical for new readers.

The AQUA interface places question and answer cards in an editing space beside the reader's preview. The author decides which cards appear and in what order.
The AQUA interface places question and answer cards in an editing space beside the reader's preview. The author decides which cards appear and in what order.

Twelve early-career researchers used the system to write about their own published work. One participant used it for two papers, yielding 13 articles. The study observed how participants drafted and revised, then interviewed them about the experience.

Being suggested is only the first hurdle

Table 3 reports study averages for questions added to the editing area and accepted into the final article.

Question originAdded to editing areaAccepted into article
General suggestion6.313.00
Personalized suggestion1.460.38
Follow-up suggestion2.840.85
Author-written question6.542.61

Participants reported that recommendations helped with the blank page and sometimes prompted them to explain material they would otherwise take for granted. They also corrected generated errors, added details, changed the tone, and wrote answers that required opinions or context outside the paper. Seven of the twelve participants edited generated answers minimally or left them unchanged. On average, participants revised answers in about half of the final pairs. Participants rated their articles relatively high for readability and clarity and lower for depth and uniqueness. Those ratings were self-assessments.

One generated answer misread the letters in TAST-BN as “nearest neighbor.” The author corrected the explanation to refer to batch-normalization layers. Another changed the description of a framework from third person to first person. Those are distinct kinds of editing work. One repairs the science. The other helps the piece sound like its author. Table 4

The best questions fit a story the author already understands

Nine of the twelve participants said the system reduced the burden of starting or drafting. Some asked for fresh batches whenever the wording or angle fell flat. Six participants used the extra prompt field to narrow the topic or ask for a different tone. The study also observed authors entering their own questions when recommendations moved away from the story they wanted to tell. Useful personalization depended on more than a researcher's biography. It also depended on what that person wanted a particular reader to learn. Usage patterns and author interviews, Sections 6.2 and 6.3

Watch the writing as it happens

The study combines what people did in the interface with what they said about their experience. That gives context to the acceptance counts. An author may reject a suggestion and still find it useful for thinking of a different question. The results describe a small group of researchers writing about work they know firsthand. Measuring how an outside audience understands the final article would answer a separate question.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why present the work as a conversation?

A: A question can start with what a reader wants to know and introduce the method at the point it becomes relevant. A sequence of questions also gives authors a way to decide which background material needs to come first.

Q: Does an accepted answer count as verified?

A: Our study observed what authors accepted and how they revised. Acceptance is a behavior in the interface. Independent source checking and reader evaluation would measure different outcomes.

Q: Why did personalized suggestions appear less often in final articles?

A: Participants used and accepted them less often in this study. The prototype’s personalization and the participants’ goals may both matter. We would need a targeted comparison to identify why.

Q: Did the study establish that AQUA produces better articles faster?

A: The interaction logs distinguish a suggested card, one an author adopts into the draft, and one kept for readers. Interviews explain some of those choices. All participants used AQUA, so the study cannot estimate the time or quality gain over a plain editor. Reader comprehension and documentary accuracy require separate measurements.

Future Ideas

Put a source check beside each answer

Let a paper author approve claims one at a time, with supporting page excerpts beside the draft. Keep questions about interpretation or future work editable and require the author to supply the answer for those questions. Save approvals alongside the article so a later revision can reopen only claims affected by a source change.

Compare the source-checking editor with the original card workflow on the same papers. Ask independent readers to identify the main result and its limits, then ask domain reviewers to count unsupported claims. Measure time to a publishable draft and preserve a separate record of author edits to tone and emphasis.

Separate support that can be traced to a specific passage from an editorial inference that requires the author's judgment. Count the extra review time spent on each type alongside any improvement in the final article.

Sources