Overview
An image detector trained on churches sees a bridge. The colors and textures look familiar, and the object occupies a small patch of the frame. A detector that scores the whole image can miss the difference. Projection Regret uses a diffusion model to make a comparison image and asks how much of the original reconstruction error remains after accounting for changes that also happen to the comparison.
Key Findings
A bridge hides in a church detector’s blind spot
The authors show a scene the detector has never been trained to recognize: a bridge in a setting that looks familiar. A reconstruction shifts the central structure toward a church, but the feature difference across the full image remains modest. The detector can assign it a lower anomaly score than a genuine church image. The example asks a sharper question than whether the picture changes. How much of the change would the same reconstruction have produced on a familiar picture with similar surroundings?
Make the image its own comparison
A diffusion model can add controlled noise and reconstruct an image toward patterns in its training data. The paper calls this a projection. Its preliminary tests compare a single denoising step with a full return to a clean image, and pixel distance with learned perceptual distance. Full projection with the LPIPS perceptual distance makes the semantic change easier to detect in some examples. It still leaves the bridge problem. Figures 2–3
The researchers project the image once to create a comparison that should look more like training data. They then apply the same anomaly measure to the original and to that comparison. Let be the distance from an image to a projection at noise level using distance . Equation 4 subtracts the expected score for the comparison image, where supplies random noise in the first projection.
The second term estimates the movement caused by projection even after much of the unfamiliar content has been replaced. The interpretation depends on the first projection keeping enough of the original background. The authors use a consistency model to make full projections cheaper and combine scores across noise-level pairs so detection relies less on a single choice. Equation 4 and Section 3.2
Three comparisons separate reconstruction from detection
For CIFAR-10 as the familiar dataset and CIFAR-100 as the unfamiliar one, the paper’s ablation shows how much each step contributes. AUROC measures how well scores rank unfamiliar above familiar images across thresholds. Table 4
| Variant | CIFAR-10 versus CIFAR-100 AUROC |
|---|---|
| LMD baseline | 0.620 |
| Projection with LPIPS | 0.667 |
| Projection Regret, one noise-level pair | 0.760 |
| Projection Regret, ensemble | 0.775 |
The main comparison shows the same ensemble reaching 0.837 for CIFAR-10 versus LSUN, ahead of the reported LMD score of 0.734. The order reverses for CIFAR-100 versus SVHN: Projection Regret is 0.945, while LMD is 0.968 and MSMA is 0.974. Which images count as familiar is therefore part of the experimental result. Table 1
Give the background a stronger advantage
The team also tests whether the method helps when color is a spurious cue in a colored-digit dataset. On the spurious out-of-distribution set, Projection Regret reaches 0.990 AUROC and a 0.958 true-negative rate at a 95% true-positive rate. The comparison method reaches 0.867 and 0.696, respectively. A separate sampler comparison finds little additional gain from the more expensive multistep diffusion model at the smaller shared ensemble setting. The strong color result shows where the subtraction can help. The sampler comparison suggests that more expensive reconstructions alone are a poor explanation for it. Tables 6–7
Questions & Answers
Q: Why subtract anything from the first error?
A: Even a familiar image can change during projection. A second projection of an already projected image estimates how much change the process introduces for a related example. The difference emphasizes changes that are unusually large for the input.
Q: Does the score identify the object that caused the alert?
A: The experiments use image-level scores. A person can inspect the projections, but reliable localization would need its own procedure and ground-truth regions.
Q: Can I use any noise level?
A: The two timesteps control how strongly the projections change the input. These timesteps are hyperparameters. A deployment would need to test sensitivity to them and account for the cost of multiple reconstructions.
Q: What should an inspection team check before trusting the score under a camera change?
A: The method relies on the reconstructions preserving enough relevant background for the second comparison to estimate its contribution. It can trail alternatives, as in the CIFAR-100 versus SVHN comparison, and AUROC aggregates over detection thresholds. An inspection team would inspect what changed in object and background regions and test at the false-alarm threshold it can actually handle.
Future Ideas
Separate a new defect from a new camera setup
In a factory image archive, collect known products before and after a change in lighting, camera, or work surface. Hold out some defect types and camera changes. Compare one-pass projection, Projection Regret, and a feature-distance baseline using the same training images. Measure missed defects and false alarms by camera condition at the inspection queue’s capacity.
On a small subset, annotate foreground and background regions and inspect where the projections changed. Advance if the added projection improves detection at the same false-alarm budget and the latency fits the line.
Measure the latency of the extra projections at the same target defect-detection rate, and report whether the result still holds for the slowest images.
Sources
Original abstract
Novelty detection is a fundamental task of machine learning which aims to detect abnormal ( out-of-distribution (OOD)) samples. Since diffusion models have recently emerged as the de facto standard generative framework with surprising generation results, novelty detection via diffusion models has also gained much attention. Recent methods have mainly utilized the reconstruction property of in-distribution samples. However, they often suffer from detecting OOD samples that share similar background information to the in-distribution data. Based on our observation that diffusion models can \emph{project} any sample to an in-distribution sample with similar background information, we propose \emph{Projection Regret (PR)}, an efficient novelty detection method that mitigates the bias of non-semantic information. To be specific, PR computes the perceptual distance between the test image and its diffusion-based projection to detect abnormality. Since the perceptual distance often fails to capture semantic changes when the background information is dominant, we cancel out the background bias by comparing it against recursive projections. Extensive experiments demonstrate that PR outperforms the prior art of generative-model-based novelty detection methods by a significant margin.