|
Harmonized Cone for Feasible and Non-conflict Directions in Training Physics-Informed Neural Networks |
Soundness: 3: good
Presentation: 4: excellent
Contribution: 3: good
Rating: 8: accept, good paper
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
This paper focuses on improving training physics-informed neural networks (PINN). PINN is known to be challenging for training due to different losses, e.g., PDE loss, boundary loss and initial value loss. Prior works can be categorized into two main streams: 1. Reweighting the losses; 2. Pursuing multi-objective optimization (MOO). It turns out that these two directions can be combined together, which is the method proposed in this paper.
1. The problem can be reformulated as seeking parameter updating direction that is both *feasible* and *non-conflict*, which is called *harmonic* direction. Feasible means nonnegative linear combination of gradients wrt each loss. Non-conflict means the direction in which none of losses increases.
2. Algorithm 1 *HARMONIC* is proposed to find a harmonic direction, which is built upon Double Description Method (DDM).
3. Theoretic analysis of algorithm 1 (Theorem 3) shows that convergence to Pareto front is guaranteed.
4. Experiment on benchmark PINNacle shows that the method is highly competitive without extra computation resource.
**Originality** is high. This paper summarizes existing works and sharply points out the limitation of either "feasible" or "non-conflict" methods. The proposed method is built on a principled framework, see summarization.
**Quality** is high. The method seems to be theoretically sound and the comprehensive experiment shows strong evidence of effectiveness.
**Clarity** is high. I found the paper mostly clear to follow.
**Significance** is high. The paper provides a unified and effective method that sets new SOTA to PINN methods.
See questions.
1. What is "the descent lemma" in the proof of Theorem 3 in Appendix B.4? I understand that equation (20) holds for gradient update rules. Please clarify this.
2. Also in the proof of Theorem 3 in line 776, why does the norm of $\mathcal{A}_h(G)$ equal to the sum? If $g_j$'s are not orthogonal, why is this true? |
Fully human-written |
|
Harmonized Cone for Feasible and Non-conflict Directions in Training Physics-Informed Neural Networks |
Soundness: 3: good
Presentation: 3: good
Contribution: 2: fair
Rating: 4: marginally below the acceptance threshold
Confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully. |
This paper proposes Harmonized Cone Gradient Descent (HARMONIC) for multi-loss optimization. HARMONIC constructs the update vector by aggregating the rays of the Harmonized Cone, which is defined as the intersection of the primal gradient cone (formed by conic combinations of individual loss gradients) and the dual gradient cone. The rays of the Harmonized Cone are computed using the Double Description method. HARMONIC outperforms existing loss-balancing and gradient-manipulation methods on multiple PDE benchmarks while maintaining comparable computational cost. Furthermore, applying the harmonized-cone constraint to existing methods (including ConFIG, ReLoBRaLo) improves their performance.
**Strengths**
1. The method of generating and aggregating the rays of the Harmonized Cone to determine the update direction is interesting and appears novel. Moreover, by showing performance improvements when combined with existing loss-balancing methods (e.g., ReLoBRaLO) and gradient-manipulation methods (e.g., ConFIG), the proposed approach empirically validates the motivation and effectiveness of enforcing the harmonized cone.
2. Although the proposed HARMONIC method is demonstrated on Physics-Informed Neural Networks (PINNs), it is widely applicable to other deep learning tasks involving multiple losses, suggesting potential contributions to various domains.
3. The proposed HARMONIC method shows superior performance compared to existing approaches in the presented experimental results.
**Weakness**
1. The explanation of the proposed algorithm is insufficient. The paper does not clearly describe how the Double Description method computes the rays of the Harmonized Cone, and it lacks references or related works that would help readers understand this process. It also remains unclear whether the Double Description method can always compute the rays exactly or if there are cases where approximation or numerical instability may occur. Furthermore, it would be helpful to explicitly include the Double Description step in the pseudocode to clarify how it is integrated into the overall algorithm.
2. The experiments presented in the paper were conducted on relatively simple benchmark problems. It would strengthen the work to evaluate HARMONIC on more challenging PDE benchmarks where PINNs are known to struggle, such as the Navier–Stokes or Kuramoto–Sivashinsky equations (see [1]). Demonstrating that HARMONIC can improve PINN performance in such difficult cases would provide stronger evidence of the algorithm’s effectiveness. Furthermore, it is important to analyze whether HARMONIC remains effective when combined with various PINN variants (for example, [2], [3]), as this would highlight its robustness and general applicability.
3. The example in Figure 1 is not sufficiently convincing in demonstrating the advantage of HARMONIC, as several compared algorithms such as CAGrad and Aligned-MTL also show convergence to the Pareto front. To better highlight the unique characteristics and advantages of HARMONIC, it would be more appropriate to include an example where other algorithms fail to converge while HARMONIC succeeds. Such a case would more clearly illustrate the distinctive behavior and strength of the proposed method.
4. The paper proposes a new optimizer and compares it with several existing methods, but it seems that the learning rate was fixed at 1e-3 for all experiments. Since different optimizers often require different optimal learning rates, it would be fairer to compare them after tuning the learning rate individually for each optimizer.
[1] Wang, S., Sankaran, S., & Perdikaris, P. (2024). Respecting causality for training physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 421, 116813.
[2] Zhao, Z., Ding, X., & Prakash, B. A. PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks. In The Twelfth International Conference on Learning Representations.
[3] Cho, J., Nam, S., Yang, H., Yun, S. B., Hong, Y., & Park, E. (2023). Separable physics-informed neural networks. Advances in Neural Information Processing Systems, 36, 23761-23788.
**Questions**
1. HARMONIC appears to be applicable not only to PINNs but also to other domains with multiple losses (for example, multi-task learning). Have you applied HARMONIC to such fields or conducted preliminary experiments beyond PINNs? (This question will not affect my rating since I understand that the rebuttal period is short.)
2. According to the experimental results provided in the Appendix, there are several benchmarks (e.g., Burgers, PInv, Poisson2d-CG) where HARMONIC does not outperform other competitors. In these cases, algorithms that do not satisfy the feasibility or non-conflict properties sometimes achieve better performance. Could you explain the limitations of HARMONIC and under what conditions such results are likely to occur?
3. In my understanding, HARMONIC appears to be a generalized version of DCGD, since DCGD also satisfies the characteristics listed in Table 1 but only for the two-loss setting. Therefore, as in the experiments of ConFIG [1], it would be valuable to include two-loss scenarios to clarify this relationship. More importantly, demonstrating the advantages of HARMONIC when extending to three or more losses, beyond the simple separation of the PINN loss into PDE residual and other term, would provide stronger evidence for the algorithm’s contribution.
[1] Liu, Q., Chu, M., & Thuerey, N. ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks. In The Thirteenth International Conference on Learning Representations. |
Lightly AI-edited |
|
Harmonized Cone for Feasible and Non-conflict Directions in Training Physics-Informed Neural Networks |
Soundness: 3: good
Presentation: 3: good
Contribution: 2: fair
Rating: 6: marginally above the acceptance threshold
Confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully. |
This paper proposes a geometric framework to address two key challenges in the multi-loss training of Physics-Informed Neural Networks: infeasible update directions caused by loss scaling and conflicting gradients across objectives. The authors introduce the harmonized cone, defined as the intersection of feasible and non-conflicting directions, and develop the HARMONIC algorithm based on this idea. Using the Double Description method, they compute valid update directions that satisfy both conditions. Theoretical results guarantee convergence to Pareto stationary points, and experiments on the PINNacle benchmark, particularly on the challenging Poisson2d-C problem, demonstrate superior robustness and stability compared to existing methods.
- (Clear motivation and good presentation) The paper convincingly explains the difficulties of PINN training into two factors, infeasibility and conflict, supported by well-designed motivating examples and illustrative figures. The theoretical framework is presented in a structured manner, and the connection between geometric intuition and optimization behavior is clearly articulated. Overall, the paper is well written.
- (New insight into the training of PINNs) This paper identifies a new and important issue, infeasibility, which has not been explicitly addressed by existing conflict-free methods. The paper provides a principled framework to resolve this issue by combining the idea of dual cone gradient descent methods.
- (Acceptable computational overhead) Although the Double Description method used to compute the harmonized cone can be computationally expensive in theory, the number of loss terms in PINN is typically very small. Given this practical context, the computational cost remains acceptable.
- (Limited novelty): The overall novelty of this paper is somewhat limited. The proposed method heavily relies on the dual cone gradient descent (DCGD) framework and differentiates itself mainly by introducing the additional infeasibility restriction compared to methods such as ConFIG.
- (Unclear effect of infeasibility restriction): The effect of the infeasibility restriction remains unclear. While the proposed harmonized cone integrates both feasibility and conflict-avoidance constraints, the paper does not provide sufficient empirical or theoretical analysis isolating the specific contribution of the infeasibility component.
- (Computational cost) The experiments in this paper appear to focus primarily on cases with three or more loss terms. However, in practical PINN applications such as inverse problems, the number of losses can easily increase to four or more. In such cases, the computational cost associated with the Double Description method may grow significantly and could become a clear disadvantage (acceptable though) compared to other existing methods.
Please refer to the Weaknesses section for the main points. In addition, I have the following specific questions:
- Would it be possible to conduct an ablation study to isolate the effect of the infeasibility component?
- The proposed method performs remarkably well on the Poisson2d-C problem, suggesting that avoiding infeasibility plays a crucial role in this particular case. Could the authors provide further explanation on why infeasibility is especially critical for this benchmark and what characteristics of Poisson2d-C make this effect more pronounced? |
Lightly AI-edited |