|
Concept-Based Steering of LLMs for Conditional Molecular Generation |
Soundness: 2: fair
Presentation: 1: poor
Contribution: 1: poor
Rating: 2: reject
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
The authors introduce concept-based activation steering (CAST) for conditional molecular generation with LLMs.
CAST enables precise control over multiple molecular properties without requiring model fine-tuning.
Extensive experiments on in-distribution and out-of-distribution datasets showcase CAST's superior property alignment and generative efficiency compared to existing methods.
Results demonstrate that CAST consistently outperforms base LLMs and even competitive fine-tuned models in most scenarios.
The method maintains molecular diversity and quality while achieving strong property alignment.
Qualitative assessments reveal that CAST generates molecules with higher novelty and synthesizability, improving the quality of generated molecules.
The method's robustness and efficiency are evident across various datasets and out-of-distribution scenarios.
- The paper transfers the idea of activation steering from general NLP tasks to conditional molecular generation, employing a Concept Bottleneck Model (CBM) to learn interpretable property directions (QED and LogP) for dynamic activation modulation. The paper clearly points out the limitations of traditional methods that rely on handcrafted contrastive prompts and fixed steering strengths, whereas CAST introduces a CBM-based, automated, and continuous way to generate property directions, with a logically coherent and well-structured design.
- The experiments cover multiple scenarios within the TDC datasets (ZINC, MOSES, and ChEMBL) and OOD settings, comparing CAST with Baseline, ActAdd, LoRA, and full SFT models, and reporting metrics such as MAE, Validity, and GenEff. CAST outperforms the baselines in most tables while using far fewer parameters than LoRA or full SFT, though in some specific settings SFT or LoRA achieve better results.
- Although the paper introduces a CBM to automatically generate steering vectors, the overall framework essentially remains a standard activation addition paradigm. Compared with existing approaches such as Activation Addition, Contrastive Steering, and Representation Engineering, the contribution is mainly at the application level, lacking new theoretical insights or substantial algorithmic innovation.
- While the CAST framework indeed extracts hidden states from intermediate layers and passes them through the CBM to generate concept vectors that are re-injected into the residual stream, the choice of layer position is an empirical design decision. The authors justify it only by citing prior work claiming that “intermediate layers retain rich semantic information,” without providing mechanistic explanations or conducting layer-wise ablation to validate this choice or its uniqueness.
- The paper does not justify the selection of QED and LogP as target properties, nor explain their chemical or pharmacological relevance to generative controllability.”
- The work lacks a discussion on the causes of OOD performance degradation, and the OOD task design itself is relatively simplistic.
Please see weaknesses. |
Lightly AI-edited |
|
Concept-Based Steering of LLMs for Conditional Molecular Generation |
Soundness: 1: poor
Presentation: 3: good
Contribution: 2: fair
Rating: 4: marginally below the acceptance threshold
Confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. |
This paper proposes CAST, a concept-based activation steering approach for molecular LLMs that aims to control continuous properties without fine-tuning the base LLM. The authors train a small concept bottleneck module on top of a frozen decoder-only LLM to map its hidden states to property-specific directions, which are later injected at an intermediate layer during decoding. On ZINC/MOSES/ChEMBL and OOD targets (incl. Conjugated-xTB), CAST reduces MAE versus frozen baselines, where the authors also argue that CAST also enables high validity, novelty, and synthesizability.
- The authors present a method that introduces a simple auxiliary network to learn appropriate hidden features for steering, without any direct training or modification of the base LLM.
- They conduct experiments across diverse application scenarios and, in addition to standard baselines, include both full fine-tuning and LoRA-based fine-tuning, enabling a thorough performance comparison.
- The overall performance of CAST still lacks adequate justification: the CBM module’s outputs are trained solely with a separate MLP-based property regression loss, and the training pipeline does not include any end-to-end objective that evaluates the effect of injecting these features into the LLM’s hidden states during generation. Consequently, there is no principled guarantee that adding these features will improve property control or overall generation quality.
- Similarly, the expectation that validity, novelty, and synthesizability would improve is insufficiently supported; indeed, the tendencies reported in Tables 5–6 do not consistently substantiate these claims and, in some cases, appear neutral or mixed with respect to such improvements.
- Considering that the target properties can be correlated (e.g., QED partially depends on LogP), how essential is the orthogonality loss? Could the authors quantify the performance sensitivity to including versus removing this term, and report whether any observed differences are significant?
- Is there a justification(e.g., an ablation or comparative study) for the choice of the intermediate layer at which CBM features are injected? How does performance vary across candidate layers and depths, and is there a principled criterion guiding this selection? |
Lightly AI-edited |
|
Concept-Based Steering of LLMs for Conditional Molecular Generation |
Soundness: 3: good
Presentation: 2: fair
Contribution: 3: good
Rating: 4: marginally below the acceptance threshold
Confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. |
The paper introduces Concept-based Activation STeering (CAST), a lightweight and flexible method for controlling large language models (LLMs) to generate molecules with specific properties, such as drug-likeliness (QED) and solubility (LogP). Unlike traditional methods that require costly retraining or fine-tuning of the LLM , CAST keeps the base LLM frozen. It works by training a small Concept Bottleneck Model (CBM) that learns to map the LLM's internal activations to property-specific steering vectors. During inference, these vectors are added back into the LLM's hidden states to "steer" the generation process toward the desired property values. Through extensive experiments, CAST is shown to consistently outperform baseline models and other activation steering methods, achieving strong performance on both in-distribution and out-of-distribution (OOD) generation tasks and demonstrating superior generative efficiency compared to even fine-tuned models.
1. **Efficiency and Flexibility**: The primary strength of CAST is that it does not require retraining or fine-tuning the base LLM, whose parameters remain frozen. This makes it a highly efficient and flexible alternative to SFT and LoRA, which require costly retraining for new property distributions.
2. **Novel Steering Mechanism**: The paper successfully improves upon traditional activation steering. By using a CBM to automatically compute property-conditioned steering vectors , CAST avoids the major limitations of prior work, such as the reliance on manually crafted contrastive prompts and fixed steering magnitudes.
3. **Strong OOD Performance**: The method demonstrates remarkable robustness on out-of-distribution (OOD) tasks. In settings where target properties are outside the training distribution, CAST consistently outperforms baseline models and remains highly competitive with fully fine-tuned models.
1. **Limited Property Control**: The method was only evaluated on two continuous molecular properties, QED and LogP. Though the authors note that QED and LogP are "relatively straightforward" benchmarks, the method's effectiveness on more complex or challenging properties, such as HOMO-LUMO gaps or excitation energies, remains untested.
2. **Lack of Qualitative Examples**: While the paper presents extensive quantitative results that are convincing, they can also be confusing: how this simple steering technique intuitively influences such a complex molecular generation process? To better elucidate the underlying mechanism, could the authors show how a specific molecule's SMILES string or 2D graph varies as they continuously change the steering strength for a single property?
1. As I am new to the concept of activation steering, I would appreciate a more detailed illustration of the inference-time procedure. Specifically, regarding the concept values $c_i$, are they predicted by the model during inference, with their strength then tuned by the users (i.e., scaled up or down) to find an optimal value? Or, alternatively, are the $c_i$ values entirely user-specified at inference, with the Concept Bottleneck Model (CBM) being completely dropped? My current understanding favors the first interpretation. If this is correct, does it imply that generating a specific molecule requires careful manual tuning of the steering strength to achieve an optimal result? This can be a weakness.
2. See W2.
3. If my understanding in Q1 is correct, I have a follow-up regarding the claim in Experiment 4.4 #1: "Take, for example, QED... the max MAE usually is close to 0.5 or 0.6 across all settings, implying that through the steering strength variable, CAST provides enough flexibility..." Could the authors please elaborate on the use of the term "usually" in this context? I am asking for clarification because, upon reviewing Table 7, some of the reported max MAE values appear to be substantially larger than 0.6. How is the claim reconciled with these larger values in Table 7?
I will consider raising my score if my questions or misunderstandings of this paper are properly addressed. |
Lightly AI-edited |