Structure & Discovery / arXiv 2024

For Forecasting, Some Neighbors Help More Than All of Them

PMformer learns from sampled groups of time series, between a model that isolates each series and one that combines every channel.

Partial-Multivariate Model for Forecasting

Jaehoon Lee, Hankook Lee, Sungik Choi, Sungjun Cho, Moontae Lee

Overview

A traffic sensor’s future readings can depend on nearby roads. Adding every sensor in the city can make the model search through many unrelated signals. This paper explores the middle ground by training on subsets of channels and combining predictions from different groupings.

Key Findings

The temperature in one part of a building may help predict the temperature next door. Bringing in every sensor from every floor can be expensive and may drown that relationship in irrelevant inputs. PMformer asks how much of the building a forecaster needs to see at once.

Train on part of the network at a time

Univariate models process each series separately. A complete multivariate model can use information from every series. PMformer trains a Transformer on subsets of the available series so it can learn cross-series relationships without attending to the whole set at once. At inference, the authors try repeated random partitions and aggregate forecasts. This increases the chance that useful predictors appear in the same subset. The headline comparisons use three partitions, and the appendix reports strong average results with a single partition.

Table 3 varies the subset size within the same model family. Lower mean squared error is better. The selected results include one case where the partial model trails the single-series version.

Dataset and horizonSingle seriesPartial multivariateAll series
Weather, 960.1410.1420.146
Electricity, 1920.1460.1420.147
Traffic, 1920.3880.3700.383
Traffic, 7200.4410.4260.441

The Traffic and Electricity comparisons make the largest channel counts particularly interesting. The paper’s PAC-Bayes analysis links subset size to the effective number of training examples and the entropy of the learned distribution. The paper also studies missing features. The accuracy of random partitions still depends on how often they place informative series together and how much inference repetition a deployment can afford.

PMformer samples a group of series and alternates attention across time and across the selected series.
PMformer samples a group of series and alternates attention across time and across the selected series.

Make more than one pass through the neighborhood

The authors study both random sampling and random partitions during training. At inference, a fresh partition changes which series share a Transformer input. For a particular series ii, a simplified description of aggregation over KK forecasts is

y^i=1K∑k=1Ky^i(k).\widehat y_i=\frac{1}{K}\sum_{k=1}^{K}\widehat y_i^{(k)}.

This gives a predictor that can use different neighbors on different passes while controlling the width of any single pass. It introduces a separate decision about how many passes the forecasting budget can afford.

The useful group size varies

The subset-size comparison asks whether a gain comes from cross-series information or from giving every series the largest possible context. A small group can omit a relevant series. A large one can make the model expensive and include irrelevant inputs. The benchmark and appendix examine both the number of series per group and the cost of making more than one prediction. This is useful when a practitioner has many measurements but expects only some to move together.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why use multiple partitions?

A: One partition may leave useful series in different groups. Repeating the partition gives the model additional combinations to use.

Q: Does the method require a known dependency graph?

A: The random grouping strategy can work without a graph. The paper’s attention-score-based grouping has higher error than random partitioning on Electricity and Traffic in Table 4. An available graph can serve as a separate baseline.

Q: When might random partitions miss a useful relationship?

A: A series can depend strongly on a rare partner that lands in a different subset. Drawing more partitions increases the opportunities to include useful partners and also adds work at inference. A single-series baseline is worth retaining, since the paper includes a Weather setting in which it performs slightly better than the selected multivariate setup.

Future Ideas

Forecast a building while sensors are offline

Use temperature and power readings from several buildings. Compare single-series, partial, and full models under matched inference latency. Simulate missing streams in contiguous blocks and hold out seasons and evaluate on channels seen in training. A separate study of unseen buildings would need a way to handle new feature identities. Report horizon-specific error and how often a missing neighbor causes a large miss.

Build a correlation-based grouping from the training period only and compare it with random groups after a seasonal or equipment change. Use a one-partition baseline and a fixed inference-latency budget to see whether additional partitions or better grouping buy the most accuracy.

Sources

Original abstract

When solving forecasting problems including multiple time-series features, existing approaches often fall into two extreme categories, depending on whether to utilize inter-feature information: univariate and complete-multivariate models. Unlike univariate cases which ignore the information, complete-multivariate models compute relationships among a complete set of features. However, despite the potential advantage of leveraging the additional information, complete-multivariate models sometimes underperform univariate ones. Therefore, our research aims to explore a middle ground between these two by introducing what we term Partial-Multivariate models where a neural network captures only partial relationships, that is, dependencies within subsets of all features. To this end, we propose PMformer, a Transformer-based partial-multivariate model, with its training algorithm. We demonstrate that PMformer outperforms various univariate and complete-multivariate models, providing a theoretical rationale and empirical analysis for its superiority. Additionally, by proposing an inference technique for PMformer, the forecasting accuracy is further enhanced. Finally, we highlight other advantages of PMformer: efficiency and robustness under missing features.