109
Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Models
arXiv:2512.16963v1 Announce Type: new
Abstract: Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs.
Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: \textbf{``Compression is Routing.''} We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a \textbf{64x sequence length compression} (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of \textbf{99.47\%} on the in-domain (code) validation set; accuracy drops sharply to \textbf{47.76\%} on a semi-out-of-distribution domain (Wiki text); and further plummets to just \textbf{0.57\%} on a fully out-of-distribution domain (random sequences).
This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an \textbf{Intrinsic Distribution Fingerprint}. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.
Abstract: Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs.
Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: \textbf{``Compression is Routing.''} We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a \textbf{64x sequence length compression} (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of \textbf{99.47\%} on the in-domain (code) validation set; accuracy drops sharply to \textbf{47.76\%} on a semi-out-of-distribution domain (Wiki text); and further plummets to just \textbf{0.57\%} on a fully out-of-distribution domain (random sequences).
This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an \textbf{Intrinsic Distribution Fingerprint}. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.
Anonymous