Feeding AI "Noise" Can Also Boost Scores, This Work Enables Positive Transfer with Noise
Feeding "Noise" to AI Can Improve Performance: A Method Enables Positive Transfer from Noise
This work, Semi-Supervised Noise Adaptation (SSNA), introduces a Noise Adaptation Framework (NAF) that challenges traditional transfer learning. Instead of requiring a labeled source domain of real data (e.g., images, text), NAF uses randomly generated Gaussian noise as the source. For a target task with C classes, it constructs C noise clusters by sampling from Gaussian distributions. Although this synthetic noise contains no semantic meaning, NAF trains it to form a discriminative class structure in a shared representation space—clustering same-class noise and separating different classes.
The key is aligning this learned structure from the noise domain to the real, sparsely labeled target domain. A small number of target labels are still essential to establish the correspondence between noise clusters and actual classes. The training objective combines: 1) supervised loss on the few labeled target samples, 2) classification loss for the noise to build its structure, and 3) a distribution alignment loss (using Negative Domain Similarity) to minimize the gap between the noise and target domains in the shared space.
Experiments show significant gains in few-label settings. With just 4 labels per class, NAF with a ResNet-18 backbone improves accuracy over a standard supervised baseline (ERM) by +12.35% on CIFAR-10, +7.61% on CIFAR-100, +4.38% on DTD-47, and +2.74% on Caltech-101. It also benefits fine-grained datasets and scales to ImageNet-1K (with 100 labels/class) and text classification (AG News). NAF can be integrated into existing semi-supervised methods like FixMatch for further gains.
Ablation studies confirm the transferred benefit comes from the discriminative structure of the noise, not randomness itself. Collapsing all noise into a single point causes negative transfer, while increasing separation between noise cluster centers improves performance. The amount of noise per class is less critical once a basic structure forms.
In conclusion, this work demonstrates that for positive transfer, the semantic content of source data may not be necessary. What can be effectively transferred is the *organizational structure* of categories within a representation space. This offers a promising alternative for scenarios where real source data is unavailable due to privacy, copyright, or procurement constraints.
marsbit07/22 07:56
