|
On the Expressive Power of Weight Quantization in Deep Neural Networks |
Soundness: 2: fair
Presentation: 3: good
Contribution: 3: good
Rating: 6: marginally above the acceptance threshold
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
This paper studies the theoretical expressive power of neural networks with quantized weights. It shows that (1) quantized networks can still be universal approximators with sufficient depth and width, and (2) expressiveness degrades polynomially as the number of bits decreases.
* Provides a clear analysis of how weight quantization affects expressive power, which is important for model compression research.
* Establishes universal approximation for quantized networks and quantifies polynomial degradation with bit reduction.
* The paper does not cite Three Quantization Regimes for ReLU Networks (ca2024), which studies depth-precision trade-offs, minimax approximation error, and identifies under-, over-, and proper quantization regimes. This omission weakens both novelty and completeness of the literature review.
* While polynomial degradation is shown, the paper does not connect this to minimax approximation error or the mechanisms behind it, limiting practical guidance on bit allocation.
* How does your polynomial degradation result relate to the three quantization regimes identified in ca2024? Why was this prior work not discussed?
* Can you provide bounds or rates for the universal approximation property relative to network width and depth, to make the results more actionable? |
Fully AI-generated |
|
On the Expressive Power of Weight Quantization in Deep Neural Networks |
Soundness: 3: good
Presentation: 3: good
Contribution: 3: good
Rating: 4: marginally below the acceptance threshold
Confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
This paper proposes a theoretical framework for analyzing the expressive power of weight-quantized neural networks, proving that networks with two or more bits retain universal approximation while 1-bit networks suffer expressive collapse, and that expressive power degrades polynomially as bit-width decreases, with experimental validation.
1. This paper establishes a formal mathematical link between quantization bit-width and expressive power, providing a solid theoretical basis for weight quantization.
2. The paper rigorously formulates universal approximation, expressive collapse, and polynomial degradation with clear and reproducible logic.
3. The theoretical results are validated through simulation and ImageNet experiments, enhancing the credibility and practical relevance of the work.
1. The metric ln(accuracy/model complexity) used in the ImageNet experiments appears to be uncommon; have the authors considered providing the raw values to facilitate the evaluation of fitting accuracy?
2. Would the main theorems still hold if weight quantization were modeled as a stochastic process rather than deterministic rounding?
3. The paper repeatedly mentions “linear width” and “exponential depth”; could the authors provide a one-sentence explanation of their physical or intuitive meanings?
4. In Experiment 1, 1000 sample points were generated, but the sampling method (uniform or normal) was not specified—could this affect the results?
5. The paper contains rich theoretical proofs; it is recommended that the authors include a table of symbols and terminology to help readers better follow the subsequent derivations.
See the Weaknesses section. |
Moderately AI-edited |
|
On the Expressive Power of Weight Quantization in Deep Neural Networks |
Soundness: 2: fair
Presentation: 2: fair
Contribution: 3: good
Rating: 4: marginally below the acceptance threshold
Confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
The paper explores how weight quantization influences the expressive power of deep neural networks. It shows that networks using two or more bits can still approximate any continuous function when sufficiently deep, while one-bit networks restricted to {0, 1} weights lose this ability entirely. The authors further derive a quantitative relationship between precision and representational accuracy, demonstrating that the approximation error between quantized and full-precision models grows polynomially as the number of bits decreases. Empirical tests on synthetic and image-classification tasks follow the same general pattern, though the experiments are limited in scope and mainly serve as qualitative support for the theoretical claims.
1. The paper provides a unified theoritical framework connecting quantization to expressibity and approximation error.
2. The constructive proof for universality is mathematically sound and leverages ideas from deep-narrow network theory (Kidger & Lyons, 2020).
The main weakness lies in the fact that the paper’s “expressive collapse” theorem for 1-bit neural networks only applies to models whose weights are restricted to the set {0, 1}, rather than the practically relevant signed case {−1, +1}. Because {0, 1} weights can only form non-negative linear combinations, such networks lack the ability to perform subtraction or cancellation, which makes their limited expressiveness somewhat inevitable. This means the negative result demonstrates the weakness of a degenerate, unsigned quantization scheme rather than establishing a general limitation of binary networks. Without extending the analysis to signed weights or reconciling it with prior work that found universal approximation in the {−1, +1} setting, the paper’s main claim risks overstating its generality and practical relevance.
Minor writing error (do not affect score):
Reference list duplicates Courbariaux et al., 2015a/2015b entries. They are identical. keep one.
Would allowing signed 1-bit weights {−1, +1} restore universal approximation, or does the expressive-collapse phenomenon persist under that setting? |
Fully AI-generated |