Structure & Discovery / arXiv 2024

When a Music Model Learns to Repeat, Do Listeners Hear a Better Song?

Four signals extracted from raw MIDI help GPT-2 track musical structure. Sinusoidal initialization led on recurrence and chord-progression rationality, while random initialization earned the most listener wins among the tested models.

Practical Reconstruction of Structural Embeddings for Symbolic Music Generation with Large Language Models

Seungyeon Rhyu, Kichang Yang, Sungjun Cho, Jaehyeon Kim, Kyogu Lee, Moontae Lee

Overview

Give a music generator a few notes and ask it to continue. The next phrase may repeat a pattern, develop it, or drift into another idea. This study asks whether position, event-type, timing, and pitch-class signals extracted from MIDI help a language model make those choices, and whether the patterns it learns correspond to what listeners hear.

Key Findings

Raw piano events contain timing but leave out some structure

A MIDI sequence tells a model when notes begin, when they end, and how much time passes between events. It does not necessarily mark beats or bars, especially when the MIDI came from an audio transcription. This study revisits structural embeddings used by MuseNet and adapts them to raw piano MIDI. The question is whether giving a language model clearer musical coordinates changes what it writes next.

Figure 1. Beat detection can behave differently across piano excerpts, motivating the use of structural features derived directly from MIDI events.
Figure 1. Beat detection can behave differently across piano excerpts, motivating the use of structural features derived directly from MIDI events.

Give each note four extra coordinates

The model adds four features to the token representation. Part locates an event within the piece. Type distinguishes events such as note-on, note-off, and time shift. Time records elapsed time between non-time-shift events. Pitch class lets the model recognize a note such as C across octaves. The embeddings are concatenated with the token embedding, projected to the model’s hidden size, and combined with positional information.

The comparison trains plain GPT-2 and two variants under the same next-token task. GPT2-RE initializes all four structural embeddings randomly. GPT2-SE initializes part and time with sinusoidal weights and the remaining two randomly. The researchers train on augmented transcribed piano MIDI and prompt the trained models with the opening of MAESTRO performances. Features and training pipeline · Figure 3

Count patterns, then ask people what they hear

The objective tests look at recurrence over several time spans, the predictability of chord changes, and how much the chord sequence varies. GPT2-SE has the strongest recurrence and highest chord-progression variation rationality in the reported comparisons. It also has the lowest chord-progression irregularity. A score for orderly repetition can favor material that returns often. A listener can still prefer a continuation that handles the prompt differently. Table 1 and Figure 4

Four listeners compare pairs of continuations from twelve 64-token prompts. The table aggregates wins across the pairings, including the original performances as references. Table 3

ContinuationNaturalness winsPrompt maintenance wins
GPT23031
GPT2-RE6357
GPT2-SE3746
Ground truth10992

The random-initialized variant beats plain GPT-2 on naturalness and prompt maintenance in the reported pairwise tests. The sinusoidal version shows no statistically significant difference from plain GPT-2 on either criterion. The listening result ranks the variants differently from the recurrence metrics. Table 4

A short prompt may contain very little music

The study measures the share of prompt frames in which notes sound, called the prompt non-silence ratio. For 16-token prompts, GPT2-RE’s average objective scores fall sharply when notes sound for less than half the prompt. The association shows that prompt length alone can hide how much musical evidence the model receives. The authors also plot where repeated passages appear over time and at different durations, giving readers a way to see patterns that a single recurrence score compresses. Figure 5 · Figure 6

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why give the model pitch class when it already sees the note?

A: The note token identifies a specific event. Pitch class gives notes an explicit relation across octaves, which may help the model learn harmonic patterns from the event stream.

Q: Why can a model repeat more and sound less connected to the prompt?

A: A repeated pattern can be introduced after the prompt and repeated many times. Listeners can also recognize a prompt through a variation that simple recurrence measurements score weakly.

Q: How much should an editor rely on the listening result when choosing a model?

A: The study compares a small collection of piano excerpts with four raters. Its value is in showing that recurrence and perceived continuation quality can diverge. To choose a mode for new prompts, collect additional blind ratings and separate excerpts with few notes from long excerpts that simply contain extended rests.

Future Ideas

Offer a continuation mode that adapts to sparse prompts

In a MIDI editor, offer one continuation selected by a small router trained on prompt length and the fraction of time with notes sounding. Compare it against always choosing GPT2-RE or always choosing GPT2-SE on held-out clips. In blinded comparisons, measure motif preservation and how many edits musicians make before accepting the continuation. Use the router only if it improves on the strongest fixed option.

Stratify the held-out clips by active-note fraction and total duration before sampling listeners. This keeps long rests from becoming a proxy for short prompts and lets the experiment test where adaptive routing is actually useful.

Sources

Original abstract

Music generation introduces challenging complexities to large language models. Symbolic structures of music of- ten include vertical harmonization as well as horizontal counterpoint, urging various adaptations and enhancements for large-scale Transformers. However, existing works share three major drawbacks: 1) their tokenization requires domain-specific annotations, such as bars and beats, that are typically missing in raw MIDI data; 2) the pure impact of enhancing token embedding methods is hardly exam- ined without domain-specific annotations; and 3) existing works to overcome the aforementioned drawbacks, such as MuseNet, lack reproducibility. To tackle such limitations, we develop a MIDI-based music generation framework in- spired by MuseNet, empirically studying two structural embeddings that do not rely on domain-specific annotations. We provide various metrics and insights that can guide suitable encoding to deploy. We also verify that multiple embedding configurations can selectively boost certain mu- sical aspects. By providing open-source implementations via HuggingFace 1 , our findings shed light on leveraging large language models toward practical and reproducible music generation.