ViSketch-GPT: Collaborative Multi-Scale Feature Extraction for Hand-Drawn Sketch Retrieval

ViSketch-GPT: Collaborative Multi-Scale Feature Extraction for Hand-Drawn Sketch Retrieval Giulio Federico1,2, Fabio Carrara2, Claudio Gennaro2, and Marco Di Benedetto2 1University of Pisa, Pisa (PI), Italy 2Institute of Information Science and Technologies (ISTI-CNR), Pisa (PI), Italy {name.surname}@isti.cnr.it Abstract. Understanding the nature of hand-drawn sketches is challenging due to the wide variation in their creation. Federico et al. (Federico, Amato, Carrara, Gennaro and Di Benedetto (2025)) demonstrated that recognizing complex structural patterns enhances both sketch recognition and generation. Building on this foundation, we explore how the extracted features can also be leveraged for hand-drawn sketch retrieval. In this work, we extend ViSketch-GPT, a multi-scale context extraction model originally designed for classification and generation, to the task of retrieval. The model’s ability to capture intricate details at multiple scales allows it to learn highly discriminative representations, making it well-suited for retrieval applications. Through extensive experiments on the QuickDraw and TU-Berlin datasets, we show that ViSketch-GPT surpasses state-of-the-art methods in sketch retrieval, achieving substantial improvements across multiple evaluation metrics. Our results show that the extracted feature representations, originally designed for classification and generation, are also highly effective for retrieval tasks. This highlights ViSketch-GPT as a versatile and high-powerful framework for various applications in computer vision and sketch analysis. Keywords: Sketch. Retrieval. Generative AI. AI. LLM. DDPM. Signed Distance Function. Machine Learning 1 Introduction Hand-drawn sketches have long been studied across diverse fields such as art, cognitive science, and computer vision. They represent fast and spontaneous visual expressions, often created to convey an idea, concept, or emotion. Due to their inherent variability and subjectivity — shaped by individual perception, artistic style, and intent — understanding and interpreting sketches remains a highly challenging task. Unlike structured forms of visual data such as photographs, diagrams, technical drawings, or maps that adhere to consistent conventions, sketches exhibit significant variability in style, detail, quality, and abstraction. This variability often reflects the creator’s personal interpretation or creative intention rather than strict visual norms. 2 G. Federico et al. For instance, datasets like QuickDraw (Ha and Eck, 2018) contain a large volume of sketches but of relatively low quality—often noisy, ambiguous, or heavily abstracted—leading to significant intra-class variability. In contrast, the TU-Berlin dataset (Eitz, Hays and Alexa, 2012), though much smaller, consists of cleaner, more representative sketches that consistently reflect the intended category. This contrast underscores the need for new mechanisms capable of capturing the full complexity of sketches and effectively applying it to downstream tasks such as generation, classification or retrieval. Federico et al. (Federico et al. (2025)) introduced a novel architecture designed to model sketch complexity through collaborative multi-scale feature extraction called ViSketch-GPT. The model outperforms existing state-of-theart performance in both sketch generation and classification. In this new study, we investigate the adaptability of this architecture to a different downstream task: sketch retrieval. Our results demonstrate that the proposed framework is not only flexible but also effective in this setting, outperforming state-of-theart results on both highly variable and curated sketch datasets, demonstrating robustness across extreme differences in style and quality. 2 Background and Related Works Previous research on freehand drawings has primarily represented sketches in a pixel-based format and leveraged convolutional neural networks (CNNs) for their interpretation. Following the introduction of Sketch-a-Net in 2015 (Yu, Yang, Song, Xiang and Hospedales, 2015), a pioneering CNN-based model for free-hand sketch generation, the field has experienced significant advancements in neural architectures, data representations, and sketch datasets. Subsequent work explored vector-based representations of sketches, motivated by their inherently sequential nature. In 2017, Google launched QuickDraw, a large-scale dataset containing over 50 million user-submitted sketches, and introduced SketchRNN (Ha and Eck, 2018), an RNN-based Variational Autoencoder capable of handling diverse and expressive sketches. However, the learned representations were primarily tailored for generative tasks rather than recognition or retrieval. To this end, Sketch-BERT (Lin, Fu, Xue and Jiang, 2020), inspired by the BERT architecture (Devlin, Chang, Lee and Toutanova, 2019), treats sketches as sequences of vector strokes, drawing an analogy to how sentences are processed in natural language processing. It adapts the transformer model to operate over sketch data, where each token corresponds to a drawing segment encoded with spatial coordinates and pen states (pen down, pen up, end of sketch). After undergoing a pre-training phase (Sketch Gestalt), the model is fine-tuned for sketch retrieval. During this stage, it learns to produce feature embeddings specifically optimized to distinguish between different sketches for retrieval tasks. Ideally, one could move beyond the dichotomy between raster and vector representations by leveraging both the visual and sequential nature of sketches, aiming to obtain richer representations better suited for discriminative tasks such as retrieval. This perspective aligns with the growing interest in Sketch- 2. BACKGROUND AND RELATED WORKS 3 Based Image Retrieval (SBIR)(Bhunia, Chowdhury, Sain, Yang, Xiang and Song, 2021; Bhunia, Koley, Khilji, Sain, Chowdhury, Xiang and Song, 2022; Chaudhuri, Mancini, Chen, Akata and Dutta, 2022; Dey, Riba, Dutta, Llados and Song, 2019; Dutta and Akata, 2020), and more broadly, content retrieval based on freehand drawings. Furthermore, it is emerging as a versatile and inclusive mode of visual interaction. In scenarios where linguistic barriers or vocabulary limitations hinder precise verbal descriptions — for instance, when non-native speakers attempt to search for specific objects they are unable to name — sketches serve as a powerful medium of universal expression. In this context, sketch retrieval not only facilitates access to visual content but also supports a more natural, intuitive, and accessible form of communication among individuals from diverse linguistic or cultural backgrounds. STNet (Gatti, Parikh, Paul, Gupta and Mishra, 2024) demonstrated that traditional retrieval systems may require additional information to support sketchbased search, especially when using only sketches is insufficient to express complex concepts. The CSTBIR (Composite Sketch+Text Based Image Retrieva) problem, introduced by them, specifically addresses scenarios where sketches are rough and text serves as a complementary input. They introduced a multimodal transformer-based model for CSTBIR that jointly processes sketches and text to retrieve relevant natural scene images. ARNet (Jiang, Tang, Jiang, Yu and Wu, 2025) proposes an innovative solution to improve Fine-Grained Sketch-Based Image Retrieval (FG-SBIR). It leverages dual weight-sharing networks to optimize the alignment between sketch and image domains, addressing the challenges of feature saturation seen with single-encoder architectures. Instead of using traditional Triplet Loss, ARNet introduces a Contrastive Loss function that enhances intraand inter-sample alignment and incorporates the Multi-Scale Token Recycling (MSTR) module for Vision Transformers (ViTs), which reuses discarded Patch Tokens, enhancing the model’s ability to capture diverse and semantically rich features from both sketches and images. Another challenge in retrieval is Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) (Lyou, Lee, Kim and Lee, 2024; Lin, Li, Li, Hospedales, Song and Qi, 2023), which aims to retrieve images from unseen categories during training. Traditional methods struggle with the high cost of paired sketch-image datasets. To address this, Lyou et al. (2024) propose Modality-Aware Encoders, a framework that aligns sketches and images indirectly through text descriptions. This eliminates the need for paired data during training. Recent research trends highlight the importance of data-free learning (DFL) (Chen, Wang, Xu, Yang, Liu, Shi, Xu, Xu and Tian, 2019), driven by privacy concerns and the high cost of collecting paired sketch-image datasets, as discussed earlier. In this context, it is common to use modality-specific encoders (e.g., trained separately on sketches and images) and transfer them to new tasks without requiring additional supervision (Chaudhuri, Bhunia, Song and Dutta, 2023). Developing an effective sketch representation mechanism would thus benefit not only SBIR but also data-free scenarios. 4 G. Federico et al. This is the motivation behind our work, which extends a previous study (Federico et al., 2025), where a novel architecture was introduced to capture sketchspecific features for generation and classification tasks. However, that model was not originally designed to produce a compact, global descriptor suitable for retrieval. This limitation motivated us to explore whether the learned representations could be adapted to retrieval settings, while preserving the fine-grained sensitivity and abstraction capabilities. In this paper, we propose a retrievaloriented adaptation of that architecture to produce discriminative sketch embeddings. Our results, which surpass state-of-the-art methods, suggest that this approach can serve as a strong sketch encoder for retrieval tasks and could play a key role in supporting SBIR and data-free learning approaches. 3 ViSkech-GPT for retrieval Federico et al. (2025) introduced a network called ViSketch-GPT, designed for sketch generation and classification tasks. The architecture consists of a diffusion model for the generative component, followed by a transformer-based encoder-decoder network for refinement. Specifically, after generating a sketch, the encoder of the transformer is provided with the current contextual information for each tile to be refined, while the decoder decodes the encoder’s output into a sequence of tokens. These tokens are then further processed by a VQ-VAE to produce the refined tile. In the classification task, only the encoder is used, where, given an input sketch to classify, the context for each tile is provided, and the resulting CLS tokens for each tile are processed by a final layer to generate a single CLS token for classification. Following the Sketch Gestalt concept introduced by Lin et al. (2020), they pretrain both the encoder and decoder of the transformer using an increasingly complex completion task. In this task, the model is trained to generate missing tiles in a partially completed sketch. But the core innovation of ViSketch-GPT lies in how sketches are represented: instead of processing the entire sketch as a single image or as a sequence of strokes, they decompose it into a sequence of tiles. For each tile, they extract a multi-level context that captures both local and global features, enabling a richer and more structured representation. In particular, given a sketch S, the algorithm first computes the signed distance field (SDF) of the sketch — an alternative and more informative representation commonly used in computer graphics — which, for readability, we define as ϕs in which, instead of assigning each pixel a binary value (0 for empty space, 1 for stroke), each pixel is assigned a distance value relative to the sketch (with the stroke considered as the reference). Then, the sketch is represented as a sequence of Ltiles extracted from its SDF: S→ {l(1) ϕs, ..., l(i−1) ϕs, l(i) ϕs, ..., l(L) ϕs} 3. VISKECH-GPT FOR RETRIEVAL 5 Fig. 1: The ViSketch-GPT algorithm first computes the SDF of the sketch (ϕs) and subsequently constructs the quadtree. These tiles, in particular, correspond to the leaves of a quadtree, obtained by recursively partitioning the SDF only if a certain tile contains significant information or until a predefined depth is reached (Figure 1). Finally, for each tile, a so-called leaf context ℵl(i) ϕsis extracted, which captures both global and local information of the corresponding leaf. The context of a leaf ℵl(i) ϕsis defined as the set of spatial neighbors of the leaf itself and all of its ancestors. Specifically, given the leaf l(i) ϕs, its spatial neighbors will correspond to the 3x3 grid of tiles where the leaf itself is at the center of the grid, and each tile has the same size as the leaf. The subsequent neighbors will be those of the 3x3 grid where the parent of the leaf is at the center, and each tile has the size of the parent. This process continues until the root itself is reached (the top-down view of the sketch). The entire process is illustrated in Figure 2. Fig. 2: The algorithm for extracting the informative context of each leaf. 6 G. Federico et al. Note that when we build the quadtree, we ensure that all leaves, regardless of their level, are resized to a fixed resolution. Therefore, the final representation of the sketch Swill be a sequence of leaf contexts: S→nℵl(1) ϕs, ..., ℵl(i−1) ϕs,ℵl(i) ϕs, ..., ℵl(L) ϕs o We will use these descriptors to represent the entire sketch. Specifically, the ViSketch-GPT architecture processes each context separately through a Transformer Encoder, extracting a summarizing embedding (CLS token) for each. Therefore, by processing Lleaves, LCLS tokens are produced. Finally, these are all processed together by an additional self-attention layer, which generates the final summarizing embedding (another CLS token) (Figure 3), which is used for the retrieval task: Logits, Embedding =ViSketch-GPT ℵl(1) ϕs, ..., ℵl(i) ϕs, ..., ℵl(L) ϕs  Fig. 3: The ViSketch-GPT architecture adapted to the retrieval task. To train the model for the retrieval task, a loss function is minimized that combines the classical cross entropy loss for the individual sketch and a triplet loss to ensure that final embeddings are similar to samples of the same class and as distinct as possible to samples belonging to different classes: L=αLCrossEntropy(LogitsA)+βLTriplet(EmbeddingA,EmbeddingP,EmbeddingN) 4. EXPERIMENTS 7 where A,P, and Nrepresent the anchor, positive, and negative examples, respectively, while αand βare the weights used during training to determine the contribution of each loss term to the final loss. 4 Experiments To assess the effectiveness of our approach, we evaluated ViSketch-GPT (adapted to retrieval) on two datasets: QuickDraw (Jongejan, Rowley, Kawashima, Kim and Fox-Gieg, 2016; Ha and Eck, 2018) and TU-Berlin (Eitz et al., 2012). 4.1 Datasets Our experiments are conducted initially using the QuickDraw dataset, a largescale collection of human-drawn sketches originating from the Google game Quick, Draw!. In this online game, users were prompted to rapidly produce drawings — each within a 20-second limit — corresponding to predefined categories. The dataset comprises approximately 50 million sketches spanning 345 distinct categories. Each sketch in QuickDraw is encoded as a sequence of pen stroke actions, each defined by a five-dimensional tuple: (∆x, ∆y, p1, p2, p3) where: –∆xand ∆ydenote the relative displacement from the previous point in the sketch. –p1is a binary flag indicating whether the pen is in contact with the surface (1) or not (0), thereby controlling line continuity. –p2is a binary value that determines whether the pen is lifted after reaching the current point (1) or remains in contact (0). –p3specifies whether the current point marks the end of the drawing sequence (1) or if it continues (0). To make the sketches compatible with our framework, each drawing is rendered as an image. We adopted the preprocessing procedures and dataset splits proposed by Ha and Eck (2018), allocating 70,000 sketches per class for training, 2,500 for validation, and 2,500 for testing. Additionally, to ensure comparability with prior work, we simplified the stroke sequences using the Ramer-Douglas-Peucker (RDP) algorithm, constraining the maximum sequence length to 321. In addition to QuickDraw, we also employed the TU-Berlin dataset, which, although smaller in scale, offers higher-quality sketch samples. The TU-Berlin dataset comprises 250 object categories, each represented by 80 human-drawn sketches. Compared to QuickDraw, the sketches in TU-Berlin are typically more refined and detailed. 8 G. Federico et al. 4.2 Competitors We conduct a comparative analysis of multiple baseline models following the same procedure of Sketch-BERT: 1. BiLSTM (Hochreiter and Schmidhuber, 1997): A three-layer bidirectional Long Short-Term Memory (BiLSTM) model is utilized to assess recognition and retrieval tasks on sequential sketch data. The hidden state dimensionality is set to 512. 2. Sketch-a-Net (Yu et al., 2015): A convolutional neural network specifically designed for sketch data. 3. DSSA (Song, Yu, Song, Xiang and Hospedales, 2017): This approach extends Sketch-a-Net by incorporating an attention module and a high-order energy triplet loss function. 4. ResNet (He, Zhang, Ren and Sun, 2016): A widely used residual neural network architecture in computer vision, primarily designed for image recognition tasks. 5. TC-Net (Lin, Fu, Lu, Gong, Xue and Jiang, 2019): A network based on DenseNet (Huang, Liu, Van Der Maaten and Weinberger, 2017), applied to sketch-based image retrieval tasks. We utilize a pre-trained version for classification and retrieval experiments. 6. Sketch-BERT (Lin et al., 2020): A BERT-based model, adapted for freehand sketches, uses BERT’s ability to capture contextual relationships in sequences to interpret sketches as temporal pen strokes. The training and validation subsets are used for model training, while evaluations are conducted on the test set. To evaluate the performance of different models, triplet loss and cross-entropy loss are applied to the sketch features extracted from each competitor, with each model serving as the backbone. 4.3 Implementation details For each sketch, we rendered the image at a resolution of 128x128 and used a maximum quadtree depth of 4, thus processing tiles with a resolution of 32x32. For the Transformer Encoder, we used a hidden size of 256, a depth of 6 and 8 attention heads. The encoder was pre-trained as discussed in the section (3). Finally, to aggregate all the CLS tokens from the various leaves into a single representative token, the final self-attention layer, with a depth of 2, hidden size of 256 and 4 attention heads, is used. For QuickDraw, we used α= 1, β = 1, while for fine-tuning TU-Berlin, we used α= 1, β = 0.1. 4.4 Results and discussion In Table 1, the retrieval results for both datasets under examination are presented. As done by Sketch-BERT , the results on TU-Berlin were obtained by 5. CONCLUSION 9 Methods QuickDraw TU-Berlin Top-1 Top-5 mAP Top-1 Top-5 mAP Bi-LSTM 70.91 89.52 60.11 31.40 59.60 23.71 Sketch-a-Net 74.88 90.10 65.13 37.25 63.50 26.18 DSSA 78.16 91.04 68.10 38.45 66.10 28.77 ResNet18 80.34 91.71 70.98 41.45 67.10 29.33 ResNet50 82.41 92.52 74.84 51.80 74.45 36.94 TCNet 83.59 92.57 76.38 55.30 79.45 38.78 Sketch-BERT 85.47 93.49 78.87 57.25 81.50 41.54 ViSketch-GPT (our) 96.69 99.92 87.35 67.36 89.40 59.99 Table 1: Performance comparison of various models on the QuickDraw and TUBerlin datasets. Metrics include Top-1 accuracy, Top-5 accuracy, and mean Average Precision (mAP). fine-tuning the network trained on QuickDraw. It is evident from the results that our model far surpasses the various competitors in each of the 3 metrics (Top1 and Top5 accuracy, Mean Average Precision (mAP)). It is important to emphasize that all results shown for the competitors were obtained using all the available data in QuickDraw, i.e., all 345 classes and 70K examples per class, but ViSketch-GPT achieved significantly better performance on both QuickDraw and TU-Berlin using only 100 classes and 5K examples per class, even for the pretraining phase before fine-tuning on TU-Berlin (where we instead used all the available data as done by the competitors). In Table 2, the performance difference obtained by the main competitor Sketch-BERT using different volumes of training data is shown compared to our network, which uses much less. Models Top-1 (%) Top-5 (%) Sketch-BERT (100 ×5K) 81.91 92.01 Sketch-BERT (200 ×5K) 81.87 92.07 Sketch-BERT (345 ×5K) 82.44 92.13 Sketch-BERT (345 ×70K) 85.47 93.49 ViSketch-GPT (100 ×5K ) 96.69 99.92 Table 2: The performance of Sketch-BERT on retrieval task on the QuickDraw dataset after different volumes of training data compared to the performance of ViSketch-GPT which uses much less data. 5 Conclusion In this work, we extended the ViSketch-GPT architecture — originally designed for sketch generation and classification — to the task of sketch retrieval. Our