MolEncoder: towards optimal masked language modeling for molecules

MolEncoder: towards optimal masked language modeling for molecules Fabian P. Krüger, * abc Nicklas ¨ Osterbacka, a Mikhail Kabeshov, a Ola Engkvist ad and Igor Tetko c Predicting molecular properties is a key challenge in drug discovery. Machine learning models, especially those based on transformer architectures, are increasingly used to make these predictions from chemical structures. Inspired by recent progress in natural language processing, many studies have adopted encoder-only transformer architectures similar to BERT (Bidirectional Encoder Representations from Transformers) for this task. These models are pretrained using masked language modeling, where parts of the input are hidden and the model learns to recover them before fine-tuning on downstream tasks. In this work, we systematically investigate whether core assumptions from natural language processing, which are commonly adopted in molecular BERT-based models, actually hold when applied to molecules represented using the Simplified Molecular Input Line Entry System (SMILES). Specifically, we examine how masking ratio, pretraining dataset size, and model size affect performance in molecular property prediction. We find that higher masking ratios than commonly used significantly improve performance. In contrast, increasing model or pretraining dataset size quickly leads to diminishing returns, offering no consistent benefit while incurring significantly higher computational cost. Based on these insights, we develop MolEncoder, a BERT-based model that outperforms existing approaches on drug discovery tasks while being more computationally efficient. Our results highlight key differences between molecular pretraining and natural language processing, showing that they require different design choices. This enables more efficient model development and lowers barriers for researchers with limited computational resources. We release MolEncoder publicly to support future work and hope our findings help make molecular representation learning more accessible and cost-effective in drug discovery. Introduction Developing new drugs is an expensive and time-consuming process. A major reason for this is that many drug candidates fail during development due to poor properties such as low solubility, high toxicity, or metabolic instability. 1 By enabling the proling of compounds even before they are synthesized, computational approaches allow earlier identication of these issues, which reduces failed experiments and helps limit the need for animal testing. Recent regulatory developments encourage the use of computational models to reduce reliance on animal testing. 2 While early screening for undesirable properties alone cannot eliminate all failures, it can improve overall efficiency and reduce unnecessary resource expenditure. Therefore, predicting molecular properties using computational methods has the potential to make drug development more efficient, ultimately saving signicant costs and resources. 3 Machine learning models offer a promising approach to achieve this goal. 4 They have gained signicant traction in drug discovery, with numerous tools now available to predict molecular properties from chemical structure. 4 Although traditional machine learning models for this have existed for decades, recent breakthroughs in deep learning have sparked interest in applying these techniques to chemistry. 5,6 This renewed attention is driven by progress in elds such as natural language processing and computer vision, where advances in model architectures and computing power have produced substantial improvements in performance. 7 One of the most important developments in this context has been the transformer architecture introduced by Vaswani et al. in 2017. 8 This architecture became the foundation for many high-performing models in natural language processing by allowing them to capture contextual relationships in text through self-attention. A key variant of this architecture are bidirectional transformers for language understanding, rst a AstraZeneca R&D, Discovery Sciences, Molecular AI, 431 83 Mölndal, Sweden b TUM School of Computation, Information and Technology, Department of Mathematics, Technical University of Munich, 80333 Munich, Germany c Helmholtz Munich –German Research Center for Environmental Health (GmbH), Institute of Structural Biology, Molecular Targets and Therapeutics Center, 85764 Neuherberg, Germany d Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Sweden Cite this: DOI: 10.1039/d5dd00369e Received 18th August 2025 Accepted 25th October 2025 DOI: 10.1039/d5dd00369e rsc.li/digitaldiscovery © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Digital Discovery PAPER Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online View Journal introduced by Devlin et al. in 2018 in the form of BERT (Bidirectional Encoder Representations from Transformers). 9 BERT is an encoder-only transformer trained using masked language modeling and next sentence prediction. In masked language modeling parts of the input sequence are hidden and the model learns to predict the missing parts from their context. In next sentence prediction, the model has to determine whether the second sentence follows the rst in the original text or if it is a random sentence from the training corpus. These unsupervised training tasks allow the model to learn general features of the input domain, which can then be adapted to specic tasks such as classication or regression through ne-tuning. 9 Inspired by BERT's success, researchers started applying this model architecture to molecular data. 10–20 To do so, molecules are represented as text sequences using the Simplied Molecular Input Line Entry System (SMILES). 21 In this setting, masked language modeling can be applied directly by masking parts of the SMILES string, whereas next sentence prediction is typically omitted because molecules lack a natural analogue to ordered sentences in text. Using masked language modeling, researchers have built molecular BERT-like encoder models that learn contextual representations of molecular structure. 10,11,13–19,22 These representations can be ne-tuned to predict a wide range of molecular properties. A related line of work has adapted the same approach to biological sequences, such as proteins represented as amino acid sequences, 23 and used combinations of BERT-like encoder models to predict interactions between molecules and proteins. 24 Recent studies have also shown that combining masked language modeling with auxiliary tasks during pretraining, such as predicting physical or chemical properties, can improve model performance. This can be achieved through joint training or by adapting the model to the auxiliary tasks aer pretraining. 13,16,18 Although these methods have gained signicant attention, many modeling choices are still directly borrowed from natural language processing. One important example is the masking ratio. Molecular BERT models use a masking ratio of 15 percent, as in the original BERT paper. 10,11,13–19,22 However, there is little evidence that this value is optimal for SMILES strings, which differ from natural language in their structure and redundancy. In fact, even follow-up work in natural language processing found that higher masking ratios, such as 40 percent, lead to better model performance. 25 These ndings raise the question on whether the historical masking ratio from natural language processing is appropriate for molecular data. Another assumption that has been carried over is that scaling up model size and pretraining dataset size leads to better performance. Many existing BERT models for molecules are pretrained on tens or hundreds of millions of molecules, even up to over a billion. 10,13,15,17,24 This increases computational requirements signicantly. However, previous studies have reported mixed ndings on the benets of larger pretraining datasets. Some observe performance improvements, while others nd diminishing returns or no gains beyond certain thresholds. 15,18 These inconsistencies suggest that the relationship between pretraining dataset size and model performance in this domain is still unclear. Similarly, the effect of scaling model size has not been systematically studied. Model sizes used in previous work vary widely, and it is unclear how many parameters are actually needed to achieve good performance. While too few parameters can prevent the model from learning the necessary patterns in the data, using more than needed results in wasted computational resources during both training and deployment. In this work, we address these knowledge gaps by systematically evaluating how masking ratio, model size, and pretraining dataset size affect the performance of encoder-only transformers trained on SMILES strings using masked language modeling. To this end, we build upon the ModernBERT architecture, 26 the current state-of-the-art and most efficient variant of BERT in natural language processing. ModernBERT incorporates recent advances such as rotary positional embeddings, GeGLU activation functions, pre-normalization, the removal of bias terms, FlashAttention, and unpadding. These components are described in detail in the methods section. Our experiments employ statistically robust procedures, including 5 ×5 cross-validation, to capture the impact of variability from both data splitting and ne-tuning, following the guidelines suggested by Ash et al. 27 Our evaluation spans several datasets that reect key tasks in drug discovery, such as solubility, metabolic stability, permeability, and enzyme inhibition. Based on our ndings, we propose MolEncoder, a model that achieves improved performance over existing BERT-based models while requiring signicantly fewer computational resources. Results Here we present our ndings on how different pretraining settings for masked SMILES strings inuence performance on common tasks in drug discovery. Our goal is to provide practical guidance for future model development. The evaluation covers a diverse set of tasks, including prediction of human liver microsomal stability (HCLint), aqueous solubility (Solubility), membrane permeability (Permeability), lipophilicity (Lipophilicity), and CYP3A4 enzyme inhibition (CYP) for small molecules. 28–30 These ve datasets are used consistently throughout all experiments. We adopt a 5 ×5 cross-validation strategy, as recommended by Ash et al., 27 to ensure robust and reliable evaluation. In preliminary experiments, we assessed the stability of this setup by repeating the analysis across different random seeds, which affect both model ne-tuning and dataset splitting. The results were almost identical, conrming the reliability of the evaluation (see Fig. S1). Based on this consistency, we used this setup for all subsequent experiments. In order to improve masked language modeling for molecules and develop a better model, we rst investigate several key design choices that inuence pretraining effectiveness. We begin our analysis by examining how the masking ratio used during pretraining affects downstream performance. This ratio determines how much information is hidden from the model and is a design choice that has received little attention in existing work. We then explore two additional factors: the Digital Discovery © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online pretraining dataset size and the model size. To better understand the generalizability of our results, we also include ablation studies on our training hyperparameters. Finally, we put our ndings to the test by developing MolEncoder, a model that incorporates these insights to improve masked language modeling for molecules, and evaluating whether it achieves better performance at a lower training cost compared to existing approaches. This comparison allows us to assess the practical value of our design choices and understand the extent to which masked pretraining can be optimized for drug discovery tasks. Masking ratio We rst examined how the masking ratio used during pretraining affects downstream performance. To this end, we tested a range of masking ratios from 10% to 90% and evaluated the resulting models on all ve of our benchmark datasets. Fig. 1 summarizes the mean absolute error (MAE) across the ve tasks as a function of masking ratio. The condence intervals reect the variability due to data splitting and ne-tuning across cross-validation folds. They indicate how precisely we can distinguish between models' average performances given these sources of variation and are constructed using Tukey's Qcritical value as proposed by Hochberg and Tamhane. 31 If two intervals overlap, the observed difference may be explained by this variation alone, and we cannot condently conclude that the models differ in performance. Our results show a clear pattern: models trained with masking ratios below 20% and above 70% perform signicantly worse than those trained with higher ratios, as determined by repeated-measures ANOVA with post hoc Tukey HSD tests. This includes the widely used masking ratio of 15%. The performance drop is consistent across all ve datasets and is reected by signicant differences in MAE. In contrast, for masking ratios between 20% and 60%, the MAEs remain relatively consistent. Although the 30% masking ratio achieves the best overall performance, the differences between 20% and 60% are small and mostly not statistically signicant. The observed trend is consistent across all ve evaluation tasks, suggesting that the effect of masking ratio is robust to the specic molecular property being predicted. We conrmed this observation using alternative evaluation metrics, including mean squared error (MSE), coefficient of determination (R 2 ), and Spearman correlation (r), all of which show the same pattern (see Fig. S2–S4). We also explored how this effect generalizes to variations in SMILES representation. Specically, we repeated the experiments using SMILES strings that include explicit hydrogen atoms. While models trained on these SMILES strings performed notably worse than those using standard SMILES strings (Fig. S5), we again observed a strong dependence on the masking ratio. In particular, masking ratios above 20% again consistently outperformed lower ones (Fig. S6). In this setting, the 40% masking ratio gave the best results. These ndings suggest that the observed benets of higher masking ratios than Fig. 1 Mean absolute error (MAE) for five molecular property prediction tasks, shown for models pretrained with different masking ratios. Points show MAE averaged over 5 ×5 cross-validation. Horizontal bars and color coding are based on 95% simultaneous confidence intervals for each mean, constructed using Tukey's Qcritical value as proposed by Hochberg and Tamhane, 31 applied post-ANOVA. Within this construction, nonoverlapping intervals indicate statistically significant differences. They reflect how well we can distinguish models given the variability from data splitting and fine-tuning across cross-validation folds. The best-performing model (lowest MAE over 5 ×5 CV) is shown in blue. Models not significantly different from the best are shown in gray; significantly worse models are shown in red. The models were trained on log 10 -transformed values of the original measurements (as provided and recommended by Polaris 27 ). Therefore, an MAE of 0.25, for example, corresponds to a prediction error of approximately a factor of 1.8 in the original scale. Further details on the original measurement units are provided in the Methods section. © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Digital Discovery Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online the commonly used 15% are robust, holding consistently across different downstream tasks, evaluation metrics, and even variations in SMILES representation. Effect of pretraining dataset and model size Next, we investigated how pretraining dataset size and model size affect downstream performance. We evaluated all combinations of three model sizes: 5 million, 15 million, and 111 million parameters, and three pretraining dataset sizes: half of the ChEMBL dataset, the full ChEMBL dataset, and the PubChem dataset (which includes ChEMBL). Fig. 2 summarizes the results for all ve evaluation tasks. Our rst observation is that pretraining consistently improves performance. Models trained directly on the downstream task without any pretraining always performed signi- cantly worse than the best pretrained model. Second, we nd that the smallest model with only 5 million parameters consistently performs worse. Independent of the pretraining dataset size, its performance is signicantly worse than that of the best model. However, increasing model size and dataset size beyond a certain point does not lead to better performance. The combinations that yield the best results, or are not signicantly statistically different from the best, are those using the 15 million parameter model pretrained on either the full ChEMBL dataset or half of the ChEMBL dataset. This trend holds across all ve evaluation tasks and is consistent for all other metrics, including mean squared error (MSE), coefficient of determination (R 2 ), and Spearman correlation (r) (see Fig. S7–S9). The largest model, with 111 million parameters, performs well in some cases but not consistently. For some tasks and pretraining dataset sizes, its performance is signicantly worse than the best model. However, this varies across evaluation datasets and does not hold consistently across metrics. This suggests that increasing both model size and pretraining dataset size does not reliably improve performance and may even lead to a decrease in model performance for the masked language learning pretraining task. To investigate whether the observed differences in performance could be attributed to distributional differences rather than dataset size, we compared the chemical and drug-like property distributions of the pretraining datasets. Specically, we examined quantitative estimates of drug-likeness (QED), adherence to Lipinski's Rule of Five, and overall coverage of chemical space (Fig. S11A–C). We found that ChEMBL and PubChem exhibit very similar distributions across these metrics, suggesting that their overall molecular characteristics are comparable. Notably, although PubChem contains nearly all SMILES strings present in the downstream tasks (>98%)— substantially more than ChEMBL—it does not lead to improved performance (Fig. S11D). This further supports that differences Fig. 2 Mean absolute error (MAE) for five molecular property prediction tasks, reported for models of different sizes (in number of parameters) pretrained on different datasets. The pretraining dataset sizes are 114.4M molecules for PubChem, 2.3M for ChEMBL, and 1.2M for Half ChEMBL. All models were trained using masked language modeling with a masking ratio of 30%. Each cell shows the average MAE over 5 ×5 crossvalidation. Cell color indicates statistical difference relative to the best-performing model (see legend in figure). Digital Discovery © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online in dataset coverage or distribution are not the primary drivers of model performance. Both datasets broadly span the chemical space relevant to the downstream evaluation tasks (Fig. S12 and S13). Most importantly, the fact that pretraining on half of ChEMBL yields performance statistically indistinguishable from pretraining on the full dataset indicates that the lack of improvement with larger datasets is unlikely to stem from a distributional mismatch. Since the half-ChEMBL subset was obtained by uniformly drawing molecules from the full dataset, it preserves the original distribution of molecular properties, reinforcing that the observed plateau in performance reects a true scaling effect rather than differences in dataset composition. To assess whether pretraining dataset size could be reduced further for the 15 million parameter model, we extended the study to include smaller subsets of ChEMBL: one fourth and one eighth of the dataset. These experiments resulted in worse performance (see Fig. S10), indicating that further reducing dataset size below half of ChEMBL results in worse models. An additional question we investigated was whether downstream evaluation is necessary, or if pretraining performance alone can serve as a reliable proxy for downstream performance. To test this, we examined the correlation between pretraining performance and nal performance on each of our ve evaluation datasets. Pretraining performance was dened as the cross entropy loss for predicting the masked tokens and nal performance as the mean absolute error for predicting the label. As shown in Fig. S14, we found no meaningful correlation between these metrics. Both Pearson and Spearman correlation coefficients were low and not statistically signicant across all evaluation tasks. This result shows that pretraining loss does not predict downstream performance, highlighting the importance of directly evaluating models on the target tasks. Comparison to existing models To test whether our ndings lead to practical improvements, we compared a model trained using our insights with widely used baselines. We selected ChemBERTa-2 (ref. 17) and MolFormer 15 because both are based on the BERT architecture and employ masked language modeling as their pretraining objective, making them directly comparable to our approach. In addition, both models provide publicly available pretrained weights on Hugging Face, which enabled us to evaluate the original released models without retraining and thereby ensured fair comparability. Other potential baselines did not fulll all of these criteria, either differing in pretraining objectives or lacking publicly available model weights. Collectively, these considerations made ChemBERTa-2 and MolFormer the most appropriate baselines for assessing the effectiveness of our proposed approach. We also included a classical baseline using XGBoost trained on extended connectivity ngerprints with a radius of 2 (ECFP4), a method that remains widely used in cheminformatics. We also extended our analysis by adding four more datasets for a more comprehensive comparison between the models. These datasets include human plasma protein binding rate (hPP Binding), drug half-life in the body (Half-life), inhibitory constant for the delta opioid receptor (DOR Binding), and inhibitory constant for the dopamine D3 receptor (D3R Binding). 29,32,33 Our goal was to assess whether using a relatively small model, a modest pretraining dataset, and a higher masking ratio would result in competitive or superior downstream performance, while requiring much less computational effort. Therefore we used the model with 15M parameters pretrained on half of ChEMBL, which we call MolEncoder from now on. The results of this comparison are shown in Fig. 3. Fig. 3 Comparison of our model, MolEncoder, to ChemBERTa-2, 17 MoLFormer 15 and an XGBoost model. The number of molecules each model was pretrained on is shown in brackets after the model name. Model sizes are 15.2M parameters for MolEncoder, 3.4M parameters for ChemBERTa-2, and 45.5M parameters for MoLFormer. The best-performing model is shown in blue. Models not significantly different from the best are shown in gray; significantly worse models are shown in red. All MAEs are reported in log 10 -transformed scale, following the data format provided by Polaris. The only exceptions are the half-life dataset, reported in hours, and the hPP binding dataset, expressed as the percentage of a drug bound to plasma proteins in blood, as both were supplied in their original (non-log-transformed) scales. © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Digital Discovery Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online Across 9 evaluation datasets, our model achieved the best performance or showed no statistically signicant difference from the best in the majority of cases. It was best or not signicantly different from the best on 6 datasets. Among the BERT-based models (MoLFormer, ChemBERTa-2, and ours), this number increased to 8 out of 9 datasets. In direct comparisons, our model performed signicantly better than the ECFP4 with XGBoost baseline on 6 datasets. Against both versions of ChemBERTa-2, it was also signicantly better on 6 datasets and showed no signicant difference on the remaining 3. Compared to MoLFormer, it was signicantly better on 4 datasets, showed no signicant difference on another 4, and was signicantly worse on 1 dataset. Between the two ChemBERTa-2 variants, performance is either statistically indistinguishable or slightly better for the version pretrained on 10 million molecules, depending on the evaluation dataset. MoLFormer performs signicantly better than ChemBERTa-2 on ve datasets, signicantly worse on two, and shows no signicant difference on the remaining 2 datasets. Overall, our model MolEncoder achieved the strongest and most consistent performance among all tested models. Comparing to MolFormer, our model is being pretrained on a dataset 100 times smaller and the parameter count of our model is only one third of MoLFormer. These results suggest that for achieving high performance in masked language modeling for molecular property prediction in drug discovery, large-scale pretraining datasets or model sizes are not required. Models trained with a small computational budget can outperform more resource-intensive models when pretraining settings are chosen carefully. While this work mainly focuses on how to make masked language modeling better, we also wanted to see how masked language model pretraining compares to other modeling approaches. We therefore evaluated our nal model, MolEncoder, on the official benchmark test sets for the ve datasets used throughout this study. These benchmarks are hosted on the Polaris platform 34 and provide held-out test sets that we did not use at any point during model development. While we used the benchmark training data for our 5 ×5 cross-validation experiments, the benchmark test sets remained entirely untouched until this nal evaluation. MolEncoder achieved strong results across the benchmark tasks. It ranked third out of thirteen submissions on the permeability prediction task and fourth out of twelve on the liver microsomal stability prediction task. For the solubility prediction task, it placed thirteenth out of forty-two models. On the lipophilicity prediction task, it ranked fourth out of eight participants. Finally, it placed second out of two on the CYP3A4 inactivation rate task. Full benchmark results, including ranks, metric values, and number of competing models, are provided in Table S1, along with direct links to the corresponding public leaderboard pages at this time point. These rankings conrm that our MLM pretraining performs reliably across a range of molecular property prediction tasks compared to other machine learning models. It consistently outperforms traditional machine learning baselines in the leaderboard. In several tasks, it also outperforms other modern deep learning models. While it is not the top-performing method on every task, it consistently delivers strong performance across tasks. It is important to note that these leaderboards come with limitations. Polaris does not enforce blind evaluation, and performance relative to the leaderboard results is evident before being uploaded to the platform. This may lead to a reluctance in sharing results that achieve a low rank. Moreover, the leaderboard does not include statistical signicance testing, meaning that small differences in rank may not reect meaningful differences in performance. The Polaris platform itself acknowledges these limitations and cautions against overinterpreting the public rankings. 34 Despite these limitations, these rankings give some evidence that MLM pretraining is a strong approach that delivers consistently good results. While not delivering the best model for each task, it performs robustly across tasks and provides a solid foundation for further improvement as for example combination with other auxiliary prediction tasks, which we will discuss in the following section. Discussion In this study, we explored how to improve masked language modeling for molecular data by systematically varying pretraining conditions and evaluating their impact on downstream performance in drug discovery tasks. Our goal was to identify which design choices lead to models that achieve stronger downstream performance while reducing the computational cost required for pretraining, and to understand how these trade-offsdiffer from those observed in natural language processing. We found that higher masking ratios during pretraining consistently led to better performance across downstream tasks. In contrast, increasing the size of the model or the pretraining dataset only provided benets up to a point. Beyond that, further scaling did not improve performance and, in some cases, even reduced it. Finally, we demonstrated that by following our proposed pretraining guidelines, it is possible to train models that outperform the current state of the art while requiring signicantly less computational resources. These results challenge some common assumptions in the eld and offer a more efficient path forward. In the following paragraphs, we discuss each of these ndings in more detail and place them in the context of existing work. As with any empirical study, our ndings are subject to certain limitations. Most notably, it is not feasible to explore all possible combinations of hyperparameters. This means we cannot guarantee that our conclusions hold universally across every conceivable setting. However, we aimed to control for the most inuential factors within our computational budget. Our ablation studies indicate that common hyperparameter variations such as learning rate and batch size do not signicantly alter the observed trends, which is consistent with previous work on hyperparameter optimization. 35 This suggests that our conclusions are likely to hold across a range of realistic conditions. Digital Discovery © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online In addition, we focused on a variety of tasks within the drug discovery domain, which primarily involve drug-like molecules. While this provides a representative assessment across bioactive chemical space, future work will be needed to validate whether the same conclusions hold when applying our ndings outside of the drug discovery setting, for example in material science or other chemically distinct domains. A key nding of our study is the strong impact of the masking ratio on model performance. Most existing work that applies masked language modeling to SMILES strings uses a masking ratio of 15%. 10,11,13–19,22 This value was originally used in the BERT paper 9 and has likely been adopted into the chemistry domain without systematic investigation. Our results show that higher masking ratios lead to signicantly better performance (Fig. 1). This effect is consistent across different downstream tasks, evaluation metrics, and even across variations in the SMILES representation. These ndings have strong practical relevance, as simply increasing the masking ratio can substantially improve model quality without changing any other aspect of the training setup. A few years aer the initial BERT paper, Wettig et al. 25 showed that the default 15% masking ratio is suboptimal for natural language. They found that a higher rate of 40% leads to better performance, which has inuenced the development of subsequent models. Here we nd a similar trend for the SMILES language. We hope that our ndings will have a similar impact in the eld of molecular representation learning and help guide future pretraining strategies. Another key aspect we investigated is the effect of pretraining data size on model performance. Our results provide strong evidence that increasing the size of the pretraining dataset does not lead to better downstream performance beyond the size of 1 million molecules. In multiple settings, models pretrained on larger datasets performed similarly or slightly worse than those trained on smaller ones. Although the difference in performance could, in principle, be inuenced by variations in molecular distributions between PubChem and ChEMBL, our additional analyses suggest that this is not the primary cause. ChEMBL and PubChem have similar trends in distributions of key drug-like properties and chemical space coverage, which is consistent with previous observations. 36 Yet, models pretrained on PubChem do not outperform those trained on ChEMBL. Even more compellingly, pretraining on only half of ChEMBL yields performance not signicantly different than using the full dataset. Because this subset was uniformly drawn, it preserves the same molecular distribution, providing strong evidence that the observed performance plateau arises from scaling limitations rather than dataset composition. While this does not entirely exclude the possibility that subtle distributional biases play a role, the comparison closely mirrors a practical choice faced by model developers: whether to pretrain on a smaller, carefully curated dataset such as ChEMBL, or on a larger dataset such as PubChem. Our results challenge the common assumption that more pretraining data is always better. Additional evidence for our claim comes from comparing two variants of ChemBERTa-2 in Fig. 3. One is pretrained on 10 million molecules and the other on 77 million. In our experiments, the one pretrained on the larger dataset did not have better performance. This aligns with the ndings of the original paper, 17 which also showed that better performance varied across evaluation datasets, with no consistent advantage for the larger pretraining set, but did not investigate that further or more systematically. Other previous work has reported mixed results regarding this topic. For example, Ross et al. 15 found that larger pretraining datasets led to better performance, but did not assess whether the improvement was statistically signicant. Contrarily, a recent preprint by Sultan et al. 18 found no signi- cant gains beyond 400 000 molecules. While the exact number is different, our ndings agree mostly with the latter nding. We hope our statistically robust analysis helps clarify this ongoing debate and provides solid evidence that it is not necessary to increase pretraining dataset size beyond a relatively low amount, which we found to be one million molecules. We also investigated the effect of model size, an aspect that has received little attention in previous work but is important because it directly impacts the computational cost of training and deploying models. We found that increasing the number of parameters from 5 million to 15 million led to a clear improvement in performance. However, scaling the model further to 111 million parameters, which matches the size of the original ModernBERT model that exhibits state-of-the-art results in natural language processing, 26 did not lead to further gains. In fact, we observed a slight drop in performance in some tasks. We thereby conclude that masked language modeling on SMILES strings does not require as large model sizes as in natural language processing. Smaller, more efficient, model sizes suffice. When looking at the model comparison we can nd some more evidence for our ndings. Our model (15 million parameters) and MoLFormer (45.5 million parameters) performed better than ChemBERTa-2, which has only 3.4 million parameters. ChemBERTa-2's weaker performance may thus partially be explained by its small model size. While other factors may also contribute, our results offer a simple and plausible explanation based on model size. The comparison between our model and MoLFormer highlights the practical value of our ndings even more. Our model achieved better performance, despite having only one-third the number of parameters and being pretrained on a dataset 100 times smaller. This result demonstrates that larger models and massive pretraining datasets are not always necessary for strong downstream performance. We thereby show that our ndings can both reduce computational costs and improve model effectiveness. We hope this encourages more inclusive research by showing that strong models can be developed even with limited computational resources, thereby making progress in this domain more accessible to more researchers. Aer demonstrating how to improve masked language modeling (MLM) for molecular representations, we now place these ndings within the broader context of property and activity prediction in cheminformatics. Our results show that a model pretrained with MLM performs better than the traditional approach based on ECFP4 ngerprints and XGBoost for © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Digital Discovery Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online most tasks. This traditional method remains widely used in the eld, but our ndings suggest that MLM provides a more effective and exible foundation for molecular modeling tasks. This trend persisted for other traditional models using handcraed ngerprints, with our model consistently outperforming them on the Polaris benchmarks. While leaderboard comparisons have limitations due to the absence of rigorous statistical testing, the consistent ranking of our MLMbased model above traditional ones provides further evidence of its value. Compared to other modern architectures, our MLM model performed consistently well across tasks, although it was never the top-performing model. This pattern is expected. MLM is a general and task-agnostic pretraining method, which provides a strong foundation for a model. However, pretraining tasks that are more closely aligned with the target downstream task or that incorporate more prior knowledge can lead to further improvements in performance. 16,18,37 Prior studies have shown that combining MLM with other objectives, such as physicochemical property prediction, improves downstream results. 16,18 These combinations have been implemented either through joint pretraining or through later task-specic adaptation. 16,18 We believe that our ndings establish a solid foundation for developing these models further. By improving MLM for molecules, we offer a more reliable and efficient starting point that can be further extended with tailored pretraining strategies. Beyond property prediction, MLM-pretrained models are also increasingly applied in more complex tasks. For instance, they are used to encode molecules in tasks such as drug–target interaction prediction 24 and molecular knowledge graph construction. 38 In these applications, molecular encodings produced by a model pretrained on MLM are integrated with representations of other biological entities to predict interactions. Our model's strong performance and, in particular, its lower computational requirements make it well suited for such computationally demanding applications, where scalability is oen a critical concern. Moreover, our ndings contribute to this area more broadly by offering practical insights into how to design effective and efficient molecular encoders, which form a key component of these multientity modeling approaches, as well as making our small and well performing model MolEncoder publicly available. Conclusion We systematically investigated key design choices for masked language modeling on molecular SMILES strings and found that (1) higher masking ratios than the widely used 15% consistently improve downstream performance; (2) increasing model or pretraining dataset size beyond moderate levels yields no consistent benet and can even degrade performance; and (3) pretraining loss is not a reliable predictor of downstream performance. Putting these insights into practice, we trained MolEncoder, a 15M-parameter model pretrained on 1 million molecules, which outperforms or matches state-of-the-art models on most benchmark tasks while requiring vastly fewer computational resources. These results challenge common assumptions and provide practical guidelines for building more efficient, accessible molecular language models. Methods In this work, we studied different ways to use masked language modeling (MLM) to pretrain chemical language models with the goal of improving performance on common downstream tasks in drug discovery. We focused on small molecules, which we represented using the Simplied Molecular Input Line Entry System (SMILES). Our models were based on an encoder-only transformer architecture, following the design of the recently improved version of the Bidirectional Encoder Representations from Transformers (ModernBERT). 26 This section rst introduces the datasets and model congurations used in our experiments, and then outlines the training procedures, evaluation pipeline, and experimental setups used to assess the effects of masking ratio, model size, and dataset scale. An overview of our workow is shown in Fig. 4. Evaluation datasets To evaluate the effect of pretraining strategies on downstream performance, we selected ve benchmark prediction tasks that represent a broad range of challenges in drug discovery. All datasets were taken from the Polaris benchmark suite (Python package version 0.13.0). Three of the tasks, introduced by Fang et al., 28 focus on pharmacokinetic properties. The rst, HCLint, involves predicting human liver microsomal stability. It is reported as intrinsic clearance in mL min −1 kg −1 . The second, Solubility, targets aqueous solubility at pH 6.8, reported in mg mL −1 . The third, Permeability, assesses membrane permeability based on the MDR1-MDCK efflux ratio, a unitless value. All three targets are provided in logarithmic form in the benchmark suite. The remaining two tasks cover physicochemical and safetyrelevant endpoints. The Lipophilicity dataset, originally published by AstraZeneca on ChEMBL, 29 contains the logarithmic values of the distribution coefficient of a compound at pH 7.4. The CYP dataset, published by Fluetsch et al. 30 involves predicting the inactivation rate constant k obs for time-dependent inhibition of the CYP3A4 enzyme, a major contributor to drug metabolism and potential drug–drug interactions. It includes log-transformed k obs values (originally measured in min −1 ) based on high-throughput in vitro assays with human liver microsomes. Together, these ve tasks were selected to provide a diverse and representative set of benchmarks for evaluating the quality of molecular representations on common tasks in drug discovery. For the comparison between our developed model and other models, we added four additional evaluation datasets. This was computationally feasible since it involved fewer models than the previous experiments. The four datasets are based on data from ChEMBL and include the human plasma protein binding rate as the percentage of a drug bound to plasma proteins in the blood (hPP Binding), 29 drug half-life in the body in hours (Halflife), 33 inhibitory constant for the delta opioid receptor (DOR Binding), 32 and inhibitory constant for the dopamine D3 Digital Discovery © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online receptor (D3R Binding). 32 The latter two datasets have previously been characterized as containing activity cliffs. 32 All evaluation datasets were provided with predened training and test splits by the Polaris benchmark suite. For all evaluation experiments involving model selection and comparison, we used only the training sets. This includes all experiments for which we used a 5 ×5 cross-validation procedure, which is described in detail later. The test sets were reserved exclusively for the nal leaderboard evaluation in order to ensure an unbiased comparison with previously published models. Since the datasets were already preprocessed and cleaned as part of the Polaris benchmark, we used them as provided without applying any additional data cleaning. The number of molecules in each training and test split is summarized in Table 1. Pretraining datasets For pretraining, we used molecular data from ChEMBL (release 35), 29 and PubChem (downloaded in March 2025). 39 We cleaned molecules from both sources to ensure consistency and quality. This included removing isotope labels and explicit hydrogen atoms, sanitizing molecules to correct valences, aromaticity, and bond types, disconnecting metals, normalizing functional groups, reionizing, and assigning stereochemistry. We also removed common salts and solvents, but lemixtures of compounds in the data to promote generalization to different downstream tasks that would require it. All preprocessing steps were carried out using RDKit (release 02.03.2025) and the datasets Python package (version 3.3). To avoid biasing the model toward patterns introduced by canonicalization algorithms, we generated non-canonical (random) SMILES representations. We removed SMILES strings longer than 500 characters and eliminated duplicates, ensuring each SMILES string corresponds to a unique molecule. For all pretraining datasets, we used a held-out test set consisting of 50 000 randomly selected SMILES strings. These molecules were excluded from all training datasets and used solely for early stopping during pretraining to detect convergence. Aer cleaning and removing the molecules used for the test set, the ChEMBL training dataset contained approximately 2.32 million molecules. From this full dataset, we constructed three smaller subsets: one-half (1.2M), one-quarter (0.6M), and oneeighth (0.3M) of ChEMBL. Each subset was created by randomly sampling the corresponding fraction of molecules from the original cleaned ChEMBL training data. In addition, we prepared a large-scale dataset using PubChem. Although PubChem already contains ChEMBL data, we ensured it included the most recent entries by adding the molecules from our ChEMBL dataset to the PubChem dataset and then removing duplicates. The resulting PubChem dataset contained 114 million molecules. We also created a version of the Table 2 Sizes of the pretraining datasets after cleaning and deduplication Dataset Size (in molecules) PubChem 114.4M ChEMBL 2.3M Half of ChEMBL 1.2M Quarter of ChEMBL 0.6M Eighth of ChEMBL 0.3M Table 1 Number of molecules in each training and test split provided by Polaris. We only used the training part for our 5 ×5 cross validation procedure. The test set was only used for the final Polaris leaderboard evaluation Dataset Train size Test size CYP 30 16 372 368 Lipophilicity 29 3360 840 HCLint 28 2229 575 Permeability 28 1919 483 Solubility 28 1578 400 Fig. 4 Overview of the workflow used in this study. We pretrained encoder-only transformer models on SMILES representations of small molecules using a masked language modeling (MLM) objective. Pretraining experiments varied model size, masking ratio, and pretraining dataset size. We included datasets derived from ChEMBL 29 and PubChem. 39 All models were evaluated using a standardized 5 ×5 cross-validation pipeline on five benchmark tasks from Polaris. 34 Evaluation focused on downstream performance across pharmacokinetic, physicochemical, and safety-related endpoints. Statistical comparisons were made using repeated measures ANOVA and a post hoc Tukey HSD test if the former was significant. © 2025 The Author(s). Published by the Royal Society of Chemistry Digital Discovery Paper Digital Discovery Open Access Article. Published on 28 October 2025. Downloaded on 11/4/2025 7:41:13 AM. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. View Article Online