Overview
A drawing of a molecule shows which atoms are bonded. The same drawing can fold into different shapes in space, and some of the molecule’s properties depend on how those atoms sit relative to each other. Three-dimensional models can learn from that geometry, but downstream datasets may supply only the drawing. D&D asks how much of a geometric model’s knowledge a student can use when it sees only the graph.
Key Findings
A molecule has a shape, but a database may tell us only which atoms are bonded. The researchers want to bring some of the value of knowing the shape to a model that will often work from a bond diagram. Their teacher learns geometry by putting noisy structures back together.
Teach with noisy coordinates
A conformer gives each atom a position in three-dimensional space. The paper first perturbs those positions with Gaussian noise and trains a model to predict the noise on each atom. To estimate what moved, the teacher must use patterns in the spatial arrangement of the atoms. Its architecture handles rotations and translations so that a shift or turn of the whole molecule need not change the learned property representation.
The authors then freeze this teacher. A student sees atoms and bonds as a 2D graph and learns to match the teacher’s representations. D&D-Graph compares the average representation over all atoms. D&D-Node compares the two encoders’ representations atom by atom. Figure 2 shows these alternatives. Downstream training uses the 2D student and requires no new conformer for the target molecule.
The gains depend on the property
Table 1 reports averages across five seeds. These rows use the same TokenGT graph backbone. BACE and HIV report ROC-AUC, where higher is better. Lipo reports mean absolute error, where lower is better.
| Pretraining | BACE ROC-AUC ↑ | HIV ROC-AUC ↑ | Lipo MAE ↓ |
|---|---|---|---|
| None | 77.53 ± 1.97 | 70.44 ± 1.86 | 0.71 ± 0.01 |
| GraphMVP | 69.12 ± 0.84 | 68.17 ± 0.86 | 0.66 ± 0.02 |
| MoleculeSDE | 74.33 ± 3.14 | 72.21 ± 0.42 | 0.61 ± 0.02 |
| D&D | 81.11 ± 2.65 | 77.31 ± 1.03 | 0.52 ± 0.01 |
D&D also has weaker results in other columns. On ClinTox, the same backbone reaches 81.51 ROC-AUC with D&D, 87.65 with MoleculeSDE, and 85.44 without pretraining. The main table selects the better D&D variant for each task, so a practitioner still needs a procedure for deciding between atom-level and graph-level supervision.
Two scales for transferring a lesson
At the molecular level, D&D-Graph matches the summary vectors. At the atom level, D&D-Node matches corresponding nodes. In the graph version, the average has the familiar form
Here is the set of atoms and is the encoder’s representation of atom . Averaging yields a fixed-size fingerprint even as molecules vary in size. The node objective preserves the correspondence between individual atoms. The same geometry teacher can thus inform a student about the whole molecule and the local environments that make it up.
Learning before the property labels arrive
Denoising and distillation use molecules for which the teacher has conformers. The property benchmarks come later, when the student only needs the bond graph. This separates the cost of obtaining geometry for pretraining from the cost of making an individual prediction. The gains in the table support the transfer on those tasks. The authors pretrain on unlabeled graph–conformer pairs from PCQM4Mv2 and use scaffold splits for MoleculeNet. That split tests property prediction on molecules with scaffolds separated from the training set. The team also examines whether the student’s attention reflects physical distance. Its analysis compares attention and atom-pair distances in held-out molecules, even though the student did not receive coordinates at inference. Correlation is a clue that geometry was transferred. It cannot establish that the student recovered the whole three-dimensional structure. Figure 4 and experimental setup
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Does the student predict the actual 3D structure?
A: Our student matches internal representations from a 3D teacher. The objective does not ask it to recover a unique set of coordinates. A molecular graph can have several conformers, and the student may only learn the geometric regularities that are predictable from the graph.
Q: Why match representations atom by atom?
A: Atoms in the graph have corresponding atoms in the conformer. That correspondence gives us finer supervision than comparing one average for the whole molecule. We test both approaches because the useful level of detail can depend on the task.
Q: Where does this save computation?
A: The student needs 3D supervision during pretraining. Fine-tuning and prediction on new property tasks can use only the molecular graph, avoiding the need to generate a conformer for every new input.
Q: Should the geometry branch be treated as the correct answer when the two models disagree?
A: No. The 3D denoising teacher learns from conformers. The screening idea requires a separate 3D model trained on the target property, and its predictions still depend on whether the available conformers are reliable and relevant. Both paths need to be checked against measured properties to see whether geometry improves the decision enough to justify its cost.
Future Ideas
A screening queue that requests geometry when needed
Pretrain a 2D student on unlabeled graph–conformer pairs, then fine-tune it on the labeled property data. Fine-tune a 3D model separately on compounds with trusted conformers. Use the 2D student’s uncertainty and distance from known molecular families to select which new compounds receive additional 3D computation. Run the queue retrospectively on a dataset where both representations are available.
Compare uncertainty-based routing with random routing and a simple chemical-similarity rule at the same conformer budget. Hold out scaffold families. Measure prediction error, calibration, and how often the low-cost 2D path misses high-error cases. Use the 3D model as an additional measurement with its own uncertainty, and evaluate both against measured properties where available.
Sources
Original abstract
Pretraining molecular representations from large unlabeled data is essential for molecular property prediction due to the high cost of obtaining ground-truth labels. While there ex- ist various 2D graph-based molecular pretraining approaches, these methods struggle to show statistically significant gains in predictive performance. Recent work have thus instead proposed 3D conformer-based pretraining under the task of denoising, leading to promising results. During down- stream finetuning, however, models trained with 3D conform- ers require accurate atom-coordinates of previously unseen molecules, which are computationally expensive to acquire at scale. In this paper, we propose a simple solution of denoise- and-distill (D&D), a self-supervised molecular representa- tion learning method that pretrains a 2D graph encoder by distilling representations from a 3D denoiser. With denois- ing followed by cross-modal knowledge distillation, our ap- proach enjoys use of knowledge obtained from denoising as well as painless application to downstream tasks with no ac- cess to 3D conformers. Experiments on real-world molecu- lar property prediction datasets show that the graph encoder trained via D&D can infer 3D information based on the 2D graph and shows superior performance and label-efficiency against previous methods.