Evaluation & Judgment / CHI Workshop 2024

Can You Measure the Moment an AI Gives You an Idea?

A CHI workshop paper turns dozens of ways to evaluate inspiration into draft questions for researchers. It lays out how those questions could become a validated scale.

Towards an Evaluation of LLM-Generated Inspiration by Developing and Validating Inspiration Scale

Hyungyu Shin, Seulgi Choi, Ji Yong Cho, Sahar Admoni, Hyunseung Lim, Taewan Kim, Hwajung Hong, Moontae Lee, Juho Kim

Overview

A suggestion can be original and leave a researcher with nowhere to go. A familiar suggestion can connect two problems and change what the researcher tries next. If an AI is supposed to inspire, judging only the quality of the idea it produced misses part of the experience. The authors collect the ways prior work has measured that experience and use them to draft a research-specific questionnaire.

Key Findings

A model proposes a new research direction. The idea looks original, but the scientist reading it cannot see how to test it. A second proposal may sound less surprising and still change the scientist's next experiment. This workshop paper asks how a questionnaire could tell such reactions apart.

Five themes emerge from earlier studies

The authors began with 523 search results and retained 97 papers that used human evaluation with relevant measures. Those papers yielded 202 metric mentions. The authors consolidated overlapping terms into 30 measures, then organized them into five themes. The themes distinguish properties of the idea from what it does for the person receiving it.

The paper's measure map links five themes from the literature to draft questionnaire items for research brainstorming.
The paper's measure map links five themes from the literature to draft questionnaire items for research brainstorming.
Theme from the reviewExample of what it measures
Idea qualityNovelty, feasibility, or applicability
Idea spaceRange of alternatives and redundant ideas
Impact on usersPerceived usefulness or change in direction
Social acceptanceRelevance to the research community or ethical concerns
Human alignmentSpecificity, relevance to the prompt, or clarity

Turn broad concepts into questions people can answer

The paper adapts the themes to brainstorming about scientific research. That changes what a word like appropriateness means. A researcher may care whether an idea addresses a live question in their field, while someone evaluating a general writing tool may have a different test in mind. The authors wrote 48 draft items. Three authors tried them on responses to possible future research directions prompted by a paper's discussion, then reworded ambiguous items or split them into separate questions. The workshop paper presents the resulting draft. Item development, Section 5

Test whether those questions measure distinct experiences

The authors plan to recruit 150 people with research experience in the first study. Participants would select a paper and rate model responses to questions it inspires. The authors plan to explore which survey responses tend to move together, narrow the pool to about ten items, and then test the proposed grouping and reliability in a second study. They also plan to compare scores with measures of enjoyment and cognitive load, to check whether the new scale captures a distinct experience. The completed work is the literature map and draft questionnaire. The proposed studies would establish whether that instrument gives consistent and interpretable scores. Study plans, Section 6

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why ask the researcher instead of an outside judge?

A: An outside judge can assess properties of an idea. The recipient can report how the suggestion changed their own thinking, which is part of the effect we want to measure.

Q: Why propose two factor analyses?

A: Exploratory analysis looks for patterns among the ratings. Confirmatory analysis then asks whether a proposed structure fits data from a separate study.

Q: Is the proposed scale ready to compare systems with a single score?

A: The workshop paper provides an item pool and a validation plan. The wording pilot involved three authors. Before combining items into a single score, test how they group in a new sample and whether those groups mean the same thing across disciplines. Comparing ratings with later research actions would help establish what the items measure.

Future Ideas

Track whether a suggestion becomes a research action

Extend the scale-validation plan with a behavioral follow-up. Compare two brainstorming interfaces in a preregistered pilot. Collect responses to a prespecified set of candidate items immediately after each session. A week later, ask which suggestions became a literature search, an experiment sketch, or a documented change of direction. Have independent reviewers verify the follow-up artifacts with participant consent. Test whether the candidate items predict action beyond novelty ratings, and report that relationship separately by field.

Sources