Navigating the maze of models: ASReview Model Selection Guideline

Navigating the maze of models: ASReview Model Selection Guideline Jelle Jasper Teijema1* 1Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, The Netherlands, *Corresponding author:, J. J. Teijema: Department of Methodology and Statistics, Utrecht University, P.O. Box 80.140, 3508TC, Utrecht, The Netherlands, Tel.: +31 302534468, E-mail address: [email protected]. 13 February 2024 Abstract The following work introduces key concepts such as feature extraction and classification, explains the implementation of models including TF-IDF, Doc2Vec, and SBERT, and outlines how these models interact with classifiers within an active learning pipeline. The post is designed to help researchers select suitable model combinations based on their dataset and review objectives. There’s also a discussion on computational considerations and performance implications. 1 Introduction Starting a systematic review can feel like navigating through a maze, with countless articles and endless decisions. Enter ASReview – your trusty AI tool. ASReview isn’t just a one-trick pony; it’s 1 more like a Swiss Army knife. Equipped with a variety of models, designed to tackle every different type of dataset. In this work, we’re diving into these models. Every model has their special powers and quirks. From the classic predictability of traditional classifiers to the complex power of neural networks, we’ll explore how each model can turn the task of literature screening into your own optimized research plan. Whether you’re a seasoned researcher or just dipping your toes into the ocean of systematic reviews, join us through the powerful and diverse models available in ASReview. 2 What Are Models? ASReview is a tool heavily dependent on machine learning algorithms. When you begin a new systematic review in ASReview, you have the opportunity to select which models you will use. ASReview, with its strong academic roots, emphasizes the values of openness and fairness. And being scientists, we don’t limit you to a model we think is best for you; instead, we give you the freedom to choose. After all, you are the Oracle. But with this freedom comes a question: which combination of models will you select for your review? ASReview uses Active Learning at its core. Active learning is a cycle in which you and an AI take turns doing what they do best to finish the screening phase of a systematic review as efficiently as possible. What you do better than anyone or anything is the labeling of literature for your study. What AI does best is efficiently and quickly going through tons and tons of data to provide you with the critical piece of data to be labeled. Together, you make an incredible team! Want to know more about how active learning works? This work explains it in great detail. So what exactly is AI? AI is a broad term, but we will let you in on a little secret — AI is just statistics on a large scale! Don’t tell anyone, though! By keeping this in mind, AI goes from being magic to something that can be understood. And that’s exactly what we will do in this work. Explain the small elements 2 that together make up the powerful AI pipeline. The ASReview active learning pipeline is made up of five elements: •A Feature Extractor algorithm •A Classification algorithm •A Query Strategy •A Balancing strategy •A stopping rule 1 These elements are algorithms. Give an algorithm some input, and in return, it will give you some output based on the rules of the algorithm. By stringing these algorithms together, we create an AI pipeline that will help you in your systematic review screening. In this work, we will touch on feature extractors and classifiers. 3 Feature Extractors Machines do not understand text in the way humans do. Machines are adept at processing numbers, but not so much at processing words or sentences. This limitation holds even for advanced models like the newest GPT models, which, despite their seemingly humanlike interaction, convert text into numerical representations before analyzing what it says. And since ASReview runs on a machine, we want to represent the text as numbers. Many rows of numbers. One row of numbers for each abstract, and we do this using feature extractors. Feature extractors are a crucial component in the field of machine learning. At its core, a feature extractor is a tool that transforms text data into a numerical format, a process often referred to as vectorization. It takes features from the text and extracts them. The result is a vector, a row of 1github.com/asreview/asreview/discussions/557 3 Figure 1: Different types of feature extractors. numbers in a table. Figure 2: Representation of an embedding So, what are features? Features are types of information. It can be the frequency of a word in a text, but also the location of a word in a sentence (position). And it can even be more complex; a feature can be the semantic meaning of a word, or the words a word references (attention). Any piece of information we can derive from an abstract using algorithms can be coded as a feature. In the context of ASReview, feature extractors enable machine learning models to process and analyze vast amounts of text. From the features extracted from text, models can identify patterns, trends, and correlations, which might be indicative of the relevance or irrelevance of an article to a 4 particular systematic review. ASReview has different feature extractors available for you to select. Below is an (often simplified) overview of options, together with PROS and CONS for each model. 3.1 TF-IDF TF-IDF [9], short for Term Frequency-Inverse Document Frequency, is a feature extraction technique used in text mining and information retrieval. It is a feature extractor available in ASReview. It operates on two key concepts: •Term Frequency (TF): This measures how frequently a term occurs in your abstract. •Inverse Document Frequency (IDF): This measures how often the term appears across all of your abstracts. IDF is calculated by dividing the total number of documents by the number of documents containing the term. The TF-IDF value is obtained by multiplying these two figures. This results in a numeric value for a word, signifying its importance in an abstract relative to your dataset. By doing this for every word in every abstract, we effectively transform text into numbers, with each column representing a specific word and its corresponding TF-IDF score and each row representing an abstract, allowing for subsequent machine learning analysis. PROS: •Lightweight: TF-IDF has near-instant computation. •Compatibility: TF-IDF produces no negative values, and will work with Na¨ıve Bayes (Na¨ıve Bayes? You’ll read about NB in the classifiers section!), while other feature extractors won’t. •Simple: Good in datasets with consistent terms for classification. •Interpretability: TF-IDF is easily interpretable when evaluating the feature matrix. 5 CONS: •Loss of word order and context: TF-IDF does not incorporate the order of words and their context into its representation. This can lead to a loss of meaning, especially in documents where phrasing and context are important. Without this information, do you know the difference between “not really” and “not really”? How about the difference between “bark” and “bark”? •Ignores Semantics: TF-IDF doesn’t put different words with the same meaning together. It is thus up to the classifier to reconnect similar words. Identical and indistinguishable are very different words for TF-IDF. Yet they are identical/indistinguishable for us. What do the vectors look like for TF-IDF? Here, each column is a different word. The final table will have one column for each word found across all abstracts. Each value is calculated as (Term Frequency / Document Frequency). Figure 3: TF-IDF Vector Representation 3.2 Doc2Vec Compared to TF-IDF, the Doc2Vec [6] model is a lot more complex. Doc2Vec is a simple form of neural network. This means that instead of using an algorithm made by a human to perform its logic, it has learned its logic by training on data first, and uses this learned logic to perform its task. Doc2Vec learns this logic in three steps: 1. During the initial setup, each document in your dataset is randomly assigned a vector. These 6 vectors are not meaningful yet. 2. Then, the model is trained. Training works as follows: the model reads a few words from an abstract and tries to predict the next word. It does this by using the vectors of the surrounding words and the vector assigned to this specific document. The prediction is a probability distribution over all words in the vocabulary – essentially, a guess at which word comes next. 3. If the prediction is wrong, the model adjusts the vectors slightly, using a method called backpropagation 2. Over many iterations, the predictions get better and better, as the vectors start capturing the essence of the words and documents – their meaning, usage, and context. 4. After sufficient training, the vectors stabilize. They don’t change much anymore because the model has become good at prediction. These trained vectors are the output of your Doc2Vec model, which you will use for your classification. Neural networks.The term neural network is indeed derived from the neurons one finds in our brains. However, they work, learn, and function quite differently. A neural network is modeled after a brain, like an airplane is modeled after a bird. The metaphor only works up to a point. The machine learning implementation of the brain has diverged significantly from how the brain works. The great thing about learning how a word is used in language is that we learn how similar words are used. And on top of that, it also learns in what context a word is used. Doc2Vec learns the meaning of words in a text by observing how they appear in relation to other words, essentially learning the associations and patterns from their usage. By doing this repeatedly across many contexts and documents, the model builds a multi-dimensional space where words with similar contexts and meanings are positioned closer to each other. And what is a multi-dimensional space? It’s just rows and columns of numbers! The rows represent the abstracts, and the columns represent 2https://towardsdatascience.com/understanding-backpropagation-algorithm 7 dimensions of meaning. Not One Meaning Per Dimension: In the multi-dimensional space of Doc2Vec, each dimension does not correspond to an interpretable meaning or concept. Instead, each dimension represents a feature learned from the text data. Sadly, these features are often abstract and not easily interpretable. Too bad! This multi-dimensional space makes Doc2Vec particularly powerful for tasks like document similarity, where you want to understand how closely related different documents are based on their content. PROS: •Contextual and Semantic Understanding: Doc2Vec is very good at understanding the context and semantics of words within documents. It captures the nuanced relationships between words based on their usage in different contexts, offering a richer representation of text data. •Preservation of Order: Unlike simpler models like TF-IDF, the representation that Doc2Vec makes uses the order of words. This helps in understanding the narrative or the flow of ideas in a document, which can be crucial for text analysis where the specific ordering of words has a lot of impact (think of legal documentation, for example, or a medical patient history). •No outside bias: Doc2Vec is trained only on your data, and therefore cannot include bias from outside of your dataset. CONS: •Computational Intensity: Doc2Vec is computationally demanding, particularly with large datasets. After all, it is learning a language from scratch! Keep this in mind when selecting Doc2Vec as a feature extractor, as this might pose challenges in terms of processing time and resource requirements. 8 •Hyperparameter Sensitivity: The performance of Doc2Vec can be sensitive to the choice of hyperparameters (the gears and knobs to optimize the learning capacity of the model). While ASReview’s hyperparameters are verified extensively with countless simulations, it is challenging to determine the optimal settings for Doc2Vec that would suit every user’s specific review scenario. •Low Explainability: The dimensions in the vector representations produced by Doc2Vec are abstract and not easily interpretable. This might be a concern in scientific applications where the quality of the feature extractor is important. •Single Language: Doc2Vec will not work for datasets with multiple languages. What do the vectors look like for Doc2Vec? Here, each column is an abstract concept dimension as extracted from the documents. The vectors will have as many dimensions as set via the vector size hyperparameter. For ASReview’s Doc2Vec model, there are 40 of these dimensions. Figure 4: Doc2Vec Vector Representation 3.3 SBERT SBERT [7], short for Sentence-BERT, is a pre-trained BERT model (Bidirectional Encoder Representations from Transformers), that’s specialized in evaluating sentences. It uses a transformer architecture, and yes, this is the same architecture that powers the latest GPT models! SBERT is designed to create embeddings for entire paragraphs, making it very useful for semantic similarity analysis. Embeddings: Embeddings are the same as vectors, which are simply rows of numbers. 9 foundation for all active learning cycles. Timing: The following indications are based on Teijema et al. [10], one of our simulation studies, performed in ASReview, using a dataset containing 46,376 records. From quickest to slowest embedding calculation time: 1. TF-IDF – times ranged from 13 to 23 seconds 2. Doc2Vec – times ranged from 15 to 17 minutes 3. SBERT – times ranged from 6 to 7 hours In contrast to feature extractors, classifiers in ASReview are repeatedly calculated, once per active learning cycle. In each cycle of active learning, the system builds the classifier from the ground up, and then asks for new predictions. The processing time for classifiers can vary based on the complexity of the algorithm, the size of the dataset, and the computational resources available. 1. nb: The fastest model around, ideal for fast computations ˘0.03 seconds per cycle 2. logistic: Low processing cost, scales well with increasing data size. ˘0.10 seconds per cycle 3. rf: Medium resource-intensiveness, involves building multiple decision trees. ˘1.5 seconds per cycle 4. nn-2-layer: Processing-intensive due to its multi-layered architecture. ˘2.4 seconds per cycle 5. svm: Complexity increases with data size and feature count, leading to higher processing demands for larger datasets. ˘9 seconds per cycle Note how each of these classifiers takes under 10 seconds to process. If you, the labeler, take more than this processing time to assess and label an abstract, ASReview will wait for you to finish before restarting the active learning cycle, and the calculation time of the classifier will have no impact on ASReview. Only when you take less time, ASReview will skip an active learning cycle 16 and continue with the previous iteration of the classifier. Even then, this will have little impact on the performance of ASReview. 5 Feature Extractors and Classifier Interactions Considering what we’ve touched on in this blogpost, your final question might be: So, what model is best for me? Ultimately, it is impossible to provide you with a best model combination. Only you know the situation of your research question and environment. Therefore, you are the best judge of models! Models can have complex interactions. Some classifiers will more efficiently utilize the embeddings from certain feature extractors than others. Some classifiers can only use embeddings with positive numbers (which is why Na¨ıve Bayes only works with TF-IDF). Some classifiers will require huge amounts of memory when using certain feature extractors. Therefore, we provide you with certain predefined combinations in ASReview, selected and verified by us to work well, based on many, many simulations. The combinations we selected for you in ASReview are: •ELAS Ultra - Rapid and excellent-performing model for most use cases. –ELAS u3 - TF-IDF + Naive Bayes –ELAS u4 - TF-IDF (with bigrams) + SVM •ELAS Multilingual - Designed for multilingual datasets. –ELAS l2 - multilingual-e5-large (similar to SBERT) + SVM •ELAS Heavy - Focuses on the semantic understanding of text. –ELAS h3 - mxbai-embed-large-v1 (similar to SBERT) + SVM 17 Plugins: If you’re interested in more models for ASReview, why not install asreview Dory 5[3], or even create your own? 6 ASReview offers a wide range of models for conducting systematic reviews, each with its unique strengths and limitations. The choice between feature extractors like TF-IDF, Doc2Vec, and SBERT, and classifiers such as Na¨ıve Bayes, Random Forest, and Logistic Regression, depends on your dataset and research question. Ultimately, your knowledge of the subject matter and the specific demands of your systematic review will allow you to make the right choice. 6 References References [1] Iz Beltagy, Kyle Lo, and Arman Cohan. “SciBERT: A pretrained language model for scientific text”. In: arXiv preprint arXiv:1903.10676 (2019). [2] Rob den Boer et al. “Advancing Multilingual Abstract Classification: A Comparative Analysis of Feature Extraction Models in Systematic Reviews”. In: (2024). [3] ASReview LAB developers. ASReview Dory - New and exciting models for ASReview. Version v1.1.1. June 2025. doi:10.5281/zenodo.15649248.url:https://doi.org/10.5281/ zenodo.15649248. [4] Fangxiaoyu Feng et al. Language-agnostic BERT Sentence Embedding. 2022. arXiv: 2007. 01852 [cs.CL].url:https://arxiv.org/abs/2007.01852. [5] Sophie Jentzsch and Cigdem Turan. “Gender Bias in BERT - Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task”. In: Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP). Association for Computational Linguistics, 2022, pp. 184–199. doi:10.18653/v1/2022.gebnlp-1.20. url:http://dx.doi.org/10.18653/v1/2022.gebnlp-1.20. 5https://github.com/asreview/asreview-dory 6https://github.com/asreview/template-extension-new-model 18 [6] Quoc Le and Tomas Mikolov. “Distributed representations of sentences and documents”. In: International conference on machine learning. PMLR. 2014, pp. 1188–1196. [7] Nils Reimers and Iryna Gurevych. “Sentence-bert: Sentence embeddings using siamese bertnetworks”. In: arXiv preprint arXiv:1908.10084 (2019). [8] Rens van de Schoot et al. “An open source machine learning framework for efficient and transparent systematic reviews”. In: NATURE MACHINE INTELLIGENCE 3.2 (2021), pp. 125– 133. doi:10.1038/s42256-020-00287-7. [9] Karen Sparck Jones. “A statistical interpretation of term specificity and its application in retrieval”. In: Journal of documentation 28.1 (1972), pp. 11–21. [10] Jelle Jasper Teijema et al. “Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders”. In: Frontiers in research metrics and analytics 8 (2023), p. 1178181. [11] Wietse de Vries et al. BERTje: A Dutch BERT Model. 2019. arXiv: 1912.09582 [cs.CL]. url:https://arxiv.org/abs/1912.09582. 19