|
Global optimization of graph acquisition functions for neural architecture search |
Soundness: 2: fair
Presentation: 3: good
Contribution: 2: fair
Rating: 8: accept, good paper
Confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
Briefly summarize the paper and its contributions. You can incorporate Markdown and Latex into your review.
This article proposes an equivalent representation of a general labeled graph in an optimized variable space, where each graph corresponds to a unique feasible solution. It further introduces a universal kernel formula to measure graph similarity, which is compatible with the proposed encoding. This method achieves global acquisition optimization based on graph Bayesian optimization in neural structure search.
1. The paper proposes an equivalent representation of general labeled graphs in the optimization variable space, ensuring that each graph corresponds to a unique feasible solution. Moreover, it introduces a unified kernel formulation that quantifies the similarity between two labeled graphs at the levels of graph structure, node labels, and edge labels. The advantages over baselines were demonstrated in NAS Bench 101, NAS Bench 201, and NAS Bench 301.
2. The formulas and derivation proofs in the article are very detailed and accompanied by complete code.
1. The benchmarks used (NAS Bench 101, NAS Bench 201, and NAS Bench 301) are all from before 2022. Similarly, the baseline methods such as GCN, NAS BOT, and NAS BOWL are also from before 2021. No experiments were conducted on the latest benchmarks or with more recent baseline methods.
2. This paper lacks an analysis of the algorithm's time complexity.
3. The evaluated benchmark is limited to NAS, lacking experiments on real-world tasks, which makes the contribution relatively limited.
1. Could experiments be added on more recent and broader benchmarks and baselines?
2. Could an analysis of the algorithm’s time complexity be provided? |
Moderately AI-edited |
|
Global optimization of graph acquisition functions for neural architecture search |
Soundness: 2: fair
Presentation: 3: good
Contribution: 2: fair
Rating: 2: reject
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
This paper proposes NAS-GOAT, a framework for globally optimizing graph-based acquisition functions in Bayesian optimization (BO) for neural architecture search (NAS). The authors formulate the graph search space—including reachability, shortest paths, and node/edge labels—as a mixed-integer program (MIP), enabling exact optimization of acquisition functions. The method generalizes prior graph BO formulations (e.g., BoGrape) to handle weakly-connected or disconnected DAGs common in NAS. Experiments on NAS-Bench-101, 201, and 301 show that NAS-GOAT efficiently finds near-optimal architectures, often outperforming or matching state-of-the-art baselines.
++ This method extends graph BO to NAS by relaxing the strong connectivity assumption of BoGrape.
++ Comprehensive experiments on three major NAS benchmarks under both deterministic and noisy settings demonstrate robustness and efficiency.
-- The MIP encoding for graph structures builds heavily on BoGrape, with the main adaptation being the relaxation of strong connectivity. While this is non-trivial, the paper could better highlight what specific constraints were modified or added to handle NAS-specific DAGs.
Specifically, the claim that BoGrape is unsuitable due to strong connectivity is not followed by a clear explanation of how this is resolved beyond "generalizing the graph encoding."
-- I am afraid that this method is not a "plug-and-play" solution. The MIP model must be manually re-derived and re-implemented for each new search space topology. This creates a significant barrier to practical adoption and limits its applicability to new or evolving NAS problems.
1. I suggest the authors provide more analyze about the differences between this method and BoGrape. As I am concerned, the contribution of this work lies in the adoption of BoGrape for NAS tasks. |
Moderately AI-edited |
|
Global optimization of graph acquisition functions for neural architecture search |
Soundness: 2: fair
Presentation: 3: good
Contribution: 2: fair
Rating: 4: marginally below the acceptance threshold
Confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
NAS-GOAT casts cell-based neural architecture search as a Mixed-Integer Program in which graph topology, reachability, shortest-path features and a GP acquisition function are jointly optimized. The resulting MIP is solved to global optimality at every BO step, eliminating hand-crafted mutations and providing certificates of optimality under the surrogate model. Experiments on three public NAS benchmarks demonstrate competitive or superior query efficiency versus recent sampling-based or evolutionary BO baselines.
1. The paper is clearly written and easy to follow.
2. The authors design a full condition plan of NAS graph space.
3. The code is supplied, and the hyper-parameters are reported.
1. The complexity of the method should be analyzed.
2. The main content in Theorem 1 is more likely a modeling plan of the graph space, but it takes too much space in the paper, which makes readers uncomfortable. In addition, Theorem 1 is unnecessary to be a theorem.
3. The experiments are all conducted on NB101~301, it is better to evaluate the method on more datasets. Besides, the method cannot achieve SOTA in some of cases.
See weakness. |
Fully human-written |