|
Bridging ML and algorithms: comparison of hyperbolic embeddings |
Soundness: 2: fair
Presentation: 2: fair
Contribution: 1: poor
Rating: 2: reject
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
The paper presents more like a survey with experiments bridging hyperbolic embedding methods in machine learning and classical network embedding algorithms. It reviews learning-based hyperbolic embeddings algorithms commonly used in ML with embeddings algorithms from network theory. The authors argue that the ML community often focuses on gradient-based methods, overlooking algorithmic approaches from network theory, that could show better efficiency with comparable performances.
The paper highlights the diversity of hyperbolic embedding methods and draws attention to classical embedding algorithms from network theory. Conducted experiments for both sides algorithms and draw comparison.
It makes the point that representations from network theory is more efficient and could even performs better than ML learning-based algorithms tracing from Nickel and Kiela (2017, 2018) and thus should be favored, cited and adopted.
The survey-style overview may help readers unfamiliar with the historical algorithmic side of hyperbolic embeddings algorithms.
The criticism that ML work “fails to cite/use” algorithmic hyperbolic embedding literature is not very compelling.
- The reason the ML community focuses more on works tracing from Nickel and Kiela (2017, 2018) is that, they allow gradient flow to the embedding layer, and trainable, end-to-end representations that adapt to downstream tasks, not just for embedding data (rather the reported performances serves as a demo of the learnt representation). Static embeddings from algorithmic approaches (network theory) cannot serve this role. Ultimately in my opinion, the hyperbolic learning community targets at an end-to-end hyperbolic networks that can replace existing Euclidean-based networks on some tasks. And it's already well-established how important learnable embedding layers are.
- The focus on metrics is also somewhat misplaced, since for downstream tasks (e.g., node or graph classification), the rank and precision of distances is very relevant to downstream performance, that truly matters, especially considering current e.g. classification algorithms are largely dependent on the hyperbolic distance.
- Some reported results appear unreliable—for example, the 2D Poincaré and Lorentz embeddings from Nickel and Kiela (2017, 2018) were not reproduced, even though they were already shown in many following ML works, can be reproduced.
- The framing of the paper is also misleading: it reads more like a survey rather than a genuine attempt to bridge ML and algorithms due to the reasons stated above.
- How do the authors envision static embeddings from network theory adapting to downstream ML tasks that require trainable representations?
- what causes the non-reproducibility of Poincaré/Lorentz experiment results? |
Fully human-written |
|
Bridging ML and algorithms: comparison of hyperbolic embeddings |
Soundness: 3: good
Presentation: 3: good
Contribution: 2: fair
Rating: 4: marginally below the acceptance threshold
Confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
The paper systematically compares hyperbolic embedding methods spanning three communities. The headline result is that the BFKL algorithm is typically ~100× faster than Poincaré (NIPS’17) and Lorentz (ICML’18) embeddings while achieving similar quality under MAP/MR and greedy routing metrics. The authors also introduce an Information Control Value (ICV) criterion to penalize excessive radius/dimension, arguing that apparent gains from higher dimension are often optimization artifacts.
1. The work fills a real gap by bridging communities that have studied hyperbolic embeddings in isolation, delivering a broad, apples-to-apples experimental comparison across many methods, datasets, and metrics.
2. The paper thoughtfully treats numerical stability and contributes the ICV criterion to counter dimension/radius inflation, leading to more balanced method selection. The inclusion of both real hierarchies and scale-free networks, plus synthetic HRG controls with regression analyses on temperature/size, strengthens the empirical narrative.
1. There is no theoretical support in the paper.
2. Figures 2 and 5 are difficult to read.
3. Despite the breadth, some baseline coverage choices limit conclusions, which may bias the landscape.
4. Some evaluations rely on discrete variants (dMAP/dMR) to avoid precision issues, which can shift scores relative to standard definitions.
No questions |
Moderately AI-edited |
|
Bridging ML and algorithms: comparison of hyperbolic embeddings |
Soundness: 1: poor
Presentation: 1: poor
Contribution: 1: poor
Rating: 2: reject
Confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully. |
The paper aims to provide a systematic, cross-community comparison of hyperbolic embedding algorithms developed in machine learning (ML), network theory (NT), and algorithmic graph research.
The authors aim to close the gap between these communities by experimentally evaluating 14 embedding methods on 30 real-world and 450 simulated networks, comparing both embedding quality and computational efficiency.
This work aims to bridge ML and algorithmic perspectives on hyperbolic embeddings, attempting to show that older algorithmic methods are far more computationally efficient without sacrificing accuracy, and proposes ICV which is intended to be a fairer, theory-grounded quality metric for comparing embeddings.
The main strength of the paper is the mission to offer a wide comparison of algorithms for hyperbolic embedding of networks across domains.
The comparison on artificial networks is misleading. The authors should use benchmarks that create ground-truth networks generated in the hyperbolic space using the nonuniform popularity similarity model (nPSO) that allows also the creation of mesoscale community structure in artificial networks as many real networks have. Then they should use measures for the order of nodes on angular coordinates, the hyperbolic distance correlation and the angular separability index of the community. In particular, the presence of community in the benchmark is fundamental because they are a property that justifies why methods such as Hypermap and HypermapCN did not work well, because being based on the simple PSO model they did not have a parameter to model the community organization of real networks.
The test on real networks are not convincing because as a matter of fact the authors do not offer any quantitative evidence that these networks are hyperbolic and the measures they use are all empirical and ill posed. For instance, assuming that a network is embedded better because their greedy stretch factor is higher is misleading. In reality, every network has a complex system on the back with an intrinsic navigability and over-estimating it is wrong.
The measure of Greedy stretch factor for navigability is old and surpassed. Measures such as the greedy routing efficiency and the geometrical congruency are more advanced tools to quantify the navigability.
The authors do not consider in their comparison important studies:
+ one of the recent states of the art proposed in this article:
CLOVE, a Travelling Salesman’s approach to hyperbolic embeddings of complex networks with communities. SG Balogh, B Sulyok, T Vicsek, G Palla. Communications Physics 8 (1), 397
+ a fast method based on network automata:
Minimum curvilinear automata with similarity attachment for network embedding and link prediction in the hyperbolic space.A Muscoloni, CV Cannistraci. arXiv preprint arXiv:1802.01183
+ the studies of Filippo Radicchi on this topic seems to be neglected.
+ It does not seem that the Authors mentions explicitly the algorithms Hypermap and HypermapCN in their initial review. These are inefficient algorithm with low performance but they are between the first methods offered. Papadopoulos, F., Psomas, C. & Krioukov, D. Network mapping by replaying hyperbolic growth. IEEE/ACM Trans. Netw 23, 198–211 (2015).
Papadopoulos, F., Aldecoa, R. & Krioukov, D. Network geometry inference using common neighbors. Phys. Rev. E 92, 22807 (2015).
+ It does not seem that the Authors mentions explicitly the algorithms of LPCS of Wang, Z., Wu, Y., Li, Q., Jin, F. & Xiong, W. Link prediction based on hyperbolic mapping with community structure for complex networks. Phys. A 450, 609–623 (2016).
This is one of the fastest ever algorithm proposed, and it should be considered together with the other evolutions proposed by the same authors in subsequent articles that the Authors can find themselves by reviewing the literature of these relevant scientists.
7. This study seems to be neglected together with the following of the same authors: Martin Keller‑Ressel and Stephanie Nargang authored “HYDRA: a method for strain-minimizing hyperbolic embedding of network- and distance-based data”.
Coalescent embedding, which is a machine learning method, was published on arXiv on [Submitted on 21 Feb 2016, Machine learning meets network science: dimensionality reduction for fast and efficient embedding of networks in the hyperbolic space], hence before the other machine learning studies listed.
The study does not report enough information to replicate the experiments. For many algorithms, it is not reported the way their versions or hyperparameters are selected.
The organization of the article needs a strong effort to improve the quality of the information reported.
For instance the authors could propose a first figure that organizes the different algorithms in a genealogical tree in order of date of publication and relationship of methodology used.
The main article could report the selected results for the best methods and the appendix the full results.
I kindly ask to the authors to address all the concerns I raised in the section Weaknesses above.
However, in my opinion this study needs to be totally restructured and re-written and my recommendation is to withdraw and resubmit in a next conference a majorly improved version of this study. |
Fully human-written |
|
Bridging ML and algorithms: comparison of hyperbolic embeddings |
Soundness: 4: excellent
Presentation: 3: good
Contribution: 3: good
Rating: 8: accept, good paper
Confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully. |
Hyperbolic embedding is a well-studied technique for graph representation learning. This paper presents a comprehensive comparison of algorithms from the ML/DL community and the traditional algorithm community. It presents a convincing survey that shows: the most popular methods derived by the ML/DL community (e.g. by Nickel and Kiela) are inferior to classic algorithms (e.g. by Blasius et al) in terms of both computational efficiency and embedding quality. This is a problem that many ML/DL researchers vaguely noticed but failed to explicitly point out. I highly appreciate the contributions of this work.
1. This paper presents a comprehensive and clear survey of both ML/DL approaches to hyperbolic embedding as well as traditional algorithmic approaches. The computational complexity of each approach is clearly discussed.
2. Very comprehensive experimental results to support the authors' claims.
3. Good writing styles -- the authors wrote a very clear preliminary section on hyperbolic geometry for the readers' information, followed by separate sections on algorithmic approaches and ML/DL approaches.
1. Some citations should be in brackets, e.g. on Line 42.
2. Figure 2, 3, and 5 are difficult to read -- the markers are too small and too similar. The same problem applies to the figures in the Appendix. If these figures are scaled to fit the page limit, I'd suggest re-design Table 1 to save some space instead of sacrificing the readability of your most illustrative figures.
3. Different hyperbolic embedding strategies perform differently on different structures, be it a tree, a deep but sparse graph, a shallow but dense graph, or a nearly fully-connected graph. You can refer to this ICLR workshop paper: https://arxiv.org/pdf/2407.16641?. The concept of local capacity explains why some methods work better on certain datasets. You may want to include more varied network datasets in your experiments, or generate datasets with distinct structures to test the methods.
1. Line 288-290, the authors claim that "The hMDA method from (Sala et al., 2018) looks interesting, but it depends on the scaling factor, and it is not clear how to learn this parameter; therefore we do not include this method in our experiments". Do you mean the hMDS method? You need to explain more clearly why this method does apply to your experimental setting. |
Fully human-written |