Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation

Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation Luca Barsellotti∗1, Lorenzo Bianchi∗2,3, Nicola Messina2, Fabio Carrara2, Marcella Cornia1, Lorenzo Baraldi1, Fabrizio Falchi2, Rita Cucchiara1 1University of Modena and Reggio Emilia, Italy 2ISTI-CNR, Italy 3University of Pisa, Italy 1{name.surname}@unimore.it 2{name.surname}@isti.cnr.it Abstract Open-Vocabulary Segmentation (OVS) aims at segmenting images from free-form textual concepts without predefined training classes. While existing vision-language models such as CLIP can generate segmentation masks by leveraging coarse spatial information from Vision Transformers, they face challenges in spatial localization due to their global alignment of image and text features. Conversely, self-supervised visual models like DINO excel in fine-grained visual encoding but lack integration with language. To bridge this gap, we present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP. Our approach aligns the textual embeddings of CLIP to the patchlevel features of DINOv2 through a learned mapping function without the need to fine-tune the underlying backbones. At training time, we exploit the attention maps of DINOv2 to selectively align local visual patches with textual embeddings. We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process, resulting in more natural and less noisy segmentations, and that our approach can also effectively distinguish foreground objects from the background. Experimental results demonstrate that Talk2DINO achieves state-of-theart performance across several unsupervised OVS benchmarks. Source code and models are publicly available at: https://lorebianchi98.github.io/Talk2DINO/. 1. Introduction Open-Vocabulary Segmentation (OVS) [59] is a fundamental task in Computer Vision that aims to partition an input image into a set of coherent regions based on concepts provided at inference time [6,17,27]. The set of concepts used to partition the image is usually provided in freeform natural language, which effectively unchains the methods ∗Equal contribution. person dinosaur ⋯ vegetation ⋯ Learned Mapping Input Image Input Categories refinement & threshold sim (element-wise) DINOv2 CLIP Text Encoder Mapped Category Embeddings Dense Feature Map Figure 1. Overview. Our approach aligns the embedding spaces of CLIP and DINOv2 through a learned mapping function. This results in fine-grained visual encodings, which can be mapped to language to provide natural and less noisy semantic segmentation. from working with a specific fixed set of classes provided at training time. Solving this task requires a fine-grained understanding of the semantic interconnections between image pixels and the meaning conveyed by natural language. While previous works employed pixel-level annotations as a source of supervision [53,58,65], a recent trend in the field is to tackle this problem in an unsupervised manner [3, 9,54] leveraging the correlations learned by state-of-the-art backbones. Contrastive embedding spaces like CLIP [38], in particular, have demonstrated good performance on tasks that demand a holistic understanding of vision and language modalities [28,33,60,64], and have therefore been employed for unsupervised OVS [26,47,63]. Although CLIPbased backbones exhibit strong cross-modal capabilities, they are primarily trained to predict a global similarity score between text and images, which limits their spatial understanding and consequently affects tasks based on dense predictions. Recent efforts have tackled this limitation by introducing architectural modifications [18,47,63]. However, the spatial understanding constraints imposed by the training modality hinder the effectiveness of such backbones in OVS and highlight the potential benefits of exploring alternative models with enhanced perceptual capabilities. arXiv:2411.19331v3 [cs.CV] 16 Sep 2025 Self-supervised vision-only backbones like DINO and DINOv2 [8,12,37] have instead shown remarkable abilities in capturing fine-grained and localized spatial features without the reliance on annotated data. Specifically, the selfattention mechanism in such backbones generates attention maps that consistently pinpoint relevant regions within the image and has been widely leveraged for foreground object segmentation [42,43,48–50]. While this property makes them a powerful tool for tasks requiring fine-grained spatial understanding, the embedding space derived from visual self-supervised networks is not inherently aligned with textual concepts, making it incompatible with the OVS task. To close the gap between vision-language and selfsupervised embedding spaces, we propose Talk2DINO, a method that combines the spatial sensitivity of DINOv2 with the text-image alignment capabilities of CLIP, enabling a highly localized multimodal image understanding. Our approach, depicted in Fig. 1, learns a mapping function that translates the text embeddings of CLIP to interact with the patch-level embeddings of DINOv2 without fine-tuning the underlying backbones. Our alignment mechanism enhances text-image correspondence by exploiting the selfattention heads of DINOv2 to highlight diverse regions of the image. During training, we weight visual patch embeddings using the attention maps of DINOv2, dynamically selecting the head that best aligns with the provided caption. This embedding is then used to maximize similarity with the caption representation through contrastive learning. At inference time, we calculate the similarity of visual patches to each textual label, including a novel background cleaning procedure that weights class scores using attention maps. Our approach demonstrates state-of-the-art performance in unsupervised OVS with minimal parameter learning. Our results show that a self-supervised vision-only encoder can generate embeddings with semantic properties akin to textual representations, opening up new pathways for addressing spatial understanding limitations in CLIP-like models [45]. To sum up, our main contributions are as follows: • We propose Talk2DINO, the first model that provides language properties to DINOv2 by mapping the CLIP textual embeddings into the DINOv2 space through a non-linear warping function. • Our proposed model employs a novel training schema that selects the most relevant visual self-attention head and does not need fine-tuning on the backbones. • We showcase the capabilities of Talk2DINO on unsupervised OVS by devising a computationally efficient inference pipeline that also employs a novel approach based on DINOv2 self-attention to improve distinguishing foreground categories from the background. • Experimentally, we show that Talk2DINO achieves stateof-the-art results in standard OVS benchmarks, demonstrating the effectiveness of the proposed approach. 2. Related Work Vision-Language Pre-Training. In the last years, visionlanguage pre-training has gained increasing interest by learning multimodal representations that can be easily transferred to downstream tasks [16,20]. The popular CLIP model [38] is trained on large-scale web-scraped data through a contrastive objective, which maximizes the similarity of the representations of corresponding image-text pairs while minimizing the similarity of the other pairs within a batch. This approach has demonstrated remarkable zero-shot classification and retrieval performance. However, learning a multimodal representation by matching global images and texts poses challenges in localizing regions with their corresponding text, showing limited performance in dense prediction tasks [4,29,35,39,60]. Open-Vocabulary Segmentation. In zero-shot segmentation, a segmentation model is trained on a set of seen classes and must generalize to unseen classes. The first attempts of OVS inherit this paradigm by training the model on a closed set of classes for which segmentation data is available and exploiting vision-language pre-training to extend their capabilities on an open set of classes through text [10,19,53,58,65]. The two-step approach represents the most popular framework in this research direction, which proposes class-agnostic regions, trained on segmentation masks, and provides them to CLIP to be aligned with textual classes [13,17,30,56,57]. However, this approach is affected by performance gaps between seen and unseen classes and presents a significant computational overhead. On the contrary, another research direction investigates how to force the segmentation capabilities to emerge without relying on direct supervision from segmentation data. Cha et al. [9] identify this setting as unsupervised OVS and propose a unified evaluation protocol. Approaches in this line can be categorized into two main groups: (i) training-free methods that propose architectural adaptations to enable pre-trained models to produce localized multimodal features [5,18,25,26,44,46,47,51,63], and (ii) weakly-supervised methods that leverage large sets of image-text pairs with dedicated learning strategies that aim to improve the correspondence between regions and texts [9,32,39,40,54,55]. Our model lies in the latter category since we exploit a weak supervision to learn how to bridge the CLIP and DINOv2 feature spaces. Self-Supervised Backbones. Recent advances in selfsupervised learning have led to models that showcase impressive matching and localization capabilities. In particular, the DINO family of models [8,12,37] employs Vision Transformers [14] trained with self-distillation, and has shown that patch-level features learned through selfsupervision can yield semantic information. Also, a strong relationship has been observed between self-attention acti- “a cat sitting on a table next to a statue of a bird” softmax softmax softmax Σ ⋅ ⋯ DINOv2 ⋯ CLIP Text Encoder sim Σ ⋅ sim Σ ⋅ sim max ⋯ Figure 2. Overview of the training methodology of Talk2DINO. We learn a projection ψ(·)that maps the CLIP textual embeddings to the visual embedding space of DINOv2. Given the dense feature map and the attention maps extracted from DINOv2, we generate N visual embeddings by computing a weighted average of the feature map with each attention map. We then compute the similarity between each visual embedding and the projected text embedding, and use the maximum similarity as the global alignment score. vations and foreground regions of the image. Hence, researchers have recently focused on the usage of DINO for the unsupervised segmentation task [42,43,48–50]. While the aforementioned models are purely visual, recent works have also focused on connecting selfsupervised feature spaces with textual representations for OVS. ReCo [41], OVDiff [23], FOSSIL [2], and FreeDA [3] are training-free approaches that build prototypes in the visual space according to pre-defined textual categories. CLIP-DINOiser [52] demonstrates that CLIP can be finetuned with the supervision of DINO to retain improved localization capabilities. LaVG [22] employs DINO to propose class-agnostic regions and computes the average embedding from CLIP for each region. ProxyCLIP [26] proposes a proxy attention module to integrate DINO features with values from the last attention layer of CLIP, thus employing the two visual backbones at prediction time. Our method is closely related to this research field since we aim to combine the multimodal understanding capabilities of CLIP with the localization properties of DINOv2. However, in contrast to these methods, we propose to directly map the textual representations from the textual encoder of CLIP to the DINOv2 space, and demonstrate that our approach sets a new state-of-the-art without relying on multiple visual backbones or on external sources of knowledge. 3. Proposed Method 3.1. Preliminaries Task Definition. Open-vocabulary segmentation aims to segment objects of interest defined through natural language at inference time. Let I∈RH×W×3be an image and v(I)∈RH P×W P×Dvits dense feature map extracted by a Transformer-based visual backbone with input patch size Pand dimensionality of embedding space Dv. Let {Tj}j=1,...,M be a set of arbitrary textual categories and t(Tj)∈RDttheir embeddings extracted by a pre-trained textual backbone. To simplify the notation, in the following, we will refer to v(I)as vand to t(Tj)as tj. Assuming a multimodal setting in which Dt=Dv, we could define the similarity map S(I, Tj)∈RH P×W Pfor the image Iand category Tjas the cosine similarity between tjand each spatial entry of v. Formally, the similarity map is defined as S(I, Tj)[h,w]=v[h,w]·t⊺ j ||v[h,w]|| ||tj||,(1) where ·[h,w]represents indexing over spatial axes. The full resolution similarity map ˆ S(I, T)∈RH×Wis recovered by upsampling S(I, T)(e.g., via bilinear interpolation). Segmentation masks M(I, T1, ..., TM)are then derived by assigning pixels to the category with the highest similarity score, i.e., M(I, T1, ..., TM)[h,w]= argmax j=1,...,M ˆ S(I, Tj)[h,w].(2) In order for Eq. 1, and therefore the segmentation from Eq. 2, to work correctly, not only the two vand tspaces should share the same dimensionality, but they should also be constructed so that they also share the same semantics. CLIP and DINO Duality. Existing vision-language models trained on image-text pairs, such as CLIP [38], can naturally fit the formulation mentioned above, as they provide dense visual and textual embeddings in the same space. However, while CLIP can correctly align global features coming from texts and images (i.e., through the similarities corresponding to CLS tokens), it lacks a precise alignment between the textual feature tand spatial patches v. Conversely, purely visual self-supervised backbones like DINOv2 [37] have shown remarkable semantic and local consistency of spatial embeddings, enabling agnostic image segmentation [42,43,50]. These abilities occur naturally in the last attention layer of DINOv2, where the attention maps computed between the CLS token and the spatial tokens align with relevant objects within the image (see Fig. 2). Despite the remarkable results observed on imageonly tasks, DINOv2 lacks a solid bridge with natural language, making it impossible to directly compute the similarities with the text features, as expressed in Eq. 1. While DINOv2 and CLIP embedding spaces are traditionally thought as being uncorrelated spaces, we show that the CLIP textual embedding space can be projected into the DINOv2 space through a learnable nonlinear warping. 3.2. Augmenting DINO with Semantics Warping CLIP Embedding Space. We learn a projection ψ:RDt→RDvto map textual embeddings tinto the space of the visual patch embeddings vof DINOv2, leveraging weak supervision from image-text pairs. We build the projection ψapplied to textual features by composing two affine transformations with a hyperbolic tangent activation, which provides nonlinear warping. Formally, ψ(t) = W⊺ b(tanh(W⊺ at+ba)) + bb,(3) where Wa∈RDt×Dvand Wb∈RDv×Dvare learnable projection matrices and b∗are learnable bias vectors. Mapping DINO to the Warped CLIP Space. To learn the nonlinear projection ψ, we exploit the intrinsic segmentation capability of DINOv2 to identify the precise spatial subsets of vto which ψ(t)should be aligned with. Specifically, we first extract the Nattention maps Ai∈ RH P×W P(one for each of the i= 1, ..., N heads) which DINOv2 computes between the CLS vector and its patch features from the last layer. One of the key features of DINOv2 is that each Aihighlights different semantic regions within the image. For each attention map Ai, we compute a visual embedding vAi∈RDvas a weighted average of the dense feature map v, emphasizing the spatial areas that Ai highlights. We then calculate the cosine similarity between each vAiand the projected text embedding ψ(t), resulting in Nsimilarity scores. Formally, the cosine similarity score between a head and the text embedding is defined as sim(vAi, t) = vAi·ψ(t)⊺ ||vAi|| ||ψ(t)||,(4) with vAi=X h,w v[h,w]softmax(Ai)[h,w].(5) To obtain the most relevant score for alignment, we apply a selection function over the similarity scores obtained for different heads. In particular, we choose the maximum similarity maxi=1,...,N sim(vAi, t)score across all heads, therefore promoting a robust alignment between textual and visual representations that adapts to the most salient visual features corresponding to the text query. Training Procedure. To optimize the alignment between text and visual embeddings, we employ the InfoNCE loss, which leverages similarity scores across a batch of imagetext pairs. For each pair (Ii, Ti), we compute similarity scores between the projected text embedding ψ(ti)and the maximally-activated visual embedding ˜vi, where ˜viis the visual embedding derived from the most relevant attention head for the corresponding text ti,i.e., ˜vi=vAj i|j= argmax k=1,...,N sim(vAk i, ti).(6) Treating the true image-text pair as the positive instance and the remaining pairs within the batch as negatives, this contrastive approach drives the model to increase similarity for matching pairs and decrease it for non-matching pairs. Formally, the InfoNCE loss LInfoNCE for a batch of Bimagetext pairs is defined as LInfoNCE =−1 2B B X i=1 log exp(sim(˜vi, ti)) PB j=1 exp(sim(˜vj, ti)) −1 2B B X i=1 log exp(sim(˜vi, ti)) PB j=1 exp(sim(˜vi, tj)). This formulation effectively strengthens alignment by maximizing the similarity for true pairs and minimizing it for mismatched pairs across the batch. Inference. The projection learned during the training procedure that warps the CLIP embedding space into the DINOv2 space enables the textual embeddings to be directly comparable with the dense feature embeddings from DINOv2. Hence, at inference time, given an image I∈RH×W×3and a set of textual arbitrary categories {Tj}j=1,...,M , we can obtain the segmentation masks as defined in Eq. 2by considering the projected text embeddings ψ(tj)in the similarity map computation from Eq. 1. 3.3. Identifying Background Regions An additional challenge that OVS approaches need to face, especially when tasked with benchmarks like Pascal VOC [15] and COCO Objects [7], is that of identifying “background” regions, i.e. regions that do not belong to the set of categories considered in the benchmark. The standard approach consists in applying a threshold on the similarity or probability score to identify where the model is not softmax cat shoe plant Σ ⋅ ⋅ Σ norm soft. Σ norm soft. Σ norm soft. sim Mapped Category Embeddings Dense DINOv2 Embeddings + DINOv2 Attention Maps refinement & threshold Input Image Segmentation Masks Similarity Maps Background Cleaning Maps Background Cleaning Procedure Figure 3. Inference procedure. At the top, we compute the similarity between mapped text embeddings and the DINOv2 patches to produce the initial similarity maps. In the bottom part, we produce a background cleaning map for each class derived from the different DINOv2 attention heads. We obtain the final enhanced similarity map of each category through a convex combination of the similarity and background cleaning maps. The output segmentation then results from the final refinement and thresholding steps. certain about the predicted category and classify these locations as background. However, previous works [23,51,52] have introduced custom approaches to improve the capabilities of the model in recognizing the background. Following this line, we propose a background cleaning procedure, depicted in Fig. 3, that is based on the capabilities of the DINOv2 backbone in focusing on coherent areas and highlighting the foreground through the self-attention heads. Specifically, given Nattention maps Ai∈RH P×W P and Mprojected textual embeddings of classes ψ(tj), we first compute the average visual embeddings vAias in Eq. 5. Similarly to the training procedure, we then compute the similarity between each vAiand ψ(tj), resulting in a matrix of similarity scores R ∈ RM×N, which is additionally normalized row-wise through a softmax operation. These scores represent how much each self-attention head is related to each textual category. Formally, Ris defined as R= [R1, ..., Rj, ..., RM]⊺,with (7) Rj=softmax sim(vA1, ψ(tj)), ..., sim(vAN, ψ(tj)). Then, for each category Tjwe compute its average attention map Fj∈RH P×W Pas Fj(Ai,Rij) = N X i=1 RijAi,(8) and normalize Fby applying a softmax normalization over both spatial axes, and linearly re-projecting its values in the range [minj,h,w S(I, Tj)[h,w],maxj,h,w S(I, Tj)[h,w]], where S(·)is the similarity map defined in Eq. 1. We exploit the resulting normalized average attention per category to shape the similarity map by activating the foreground region and deactivating the background. The resulting shaped similarity map S ∈ RH×Wis defined as S(I, Tj)[h,w]=λS(I, Tj)[h,w]+ (1 −λ)Fj,[h,w],(9) where λis a hyperparameter representing the relevance of the background shaping in computing the segmentation masks. The background mask is then identified as the collection of pixels for which the shaped similarity map is lower than a threshold across all semantic categories. 4. Experiments 4.1. Experimental Setup Datasets. We evaluate our approach on eight widely-used semantic segmentation benchmarks, which we categorize based on the inclusion of a background class. Specifically, we conduct experiments on the validation sets of Pascal VOC 2012 [15], Pascal Context [34], COCO Stuff [7], Cityscapes [11], and ADE20K [61,62], that contain 20, 59, 171, 150, and 19 semantic categories respectively and do not include the “background” class. We report additional experiments on the COCO Objects dataset [7], which consists of 80 different foreground object classes, and on modified versions of Pascal VOC 2012 and Pascal Context in which the “background” category is, instead, included (i.e., with 21 and 60 semantic categories respectively). Implementation Details. For the main experiments, we employ DINOv2 ViT-B/14 as the base model and DINOv2 ViT-L/14 as the large model, both with the CLIP ViT-B/16 text encoder. We use the DINOv2 variant with registers [12] since our method benefits from the removal of artifacts in self-attention maps. We train the model with the Adam optimizer, a batch size of 128, and a learning rate of 1×10−4 for a total of 100 epochs on the COCO Captions 2014 training split [31], composed of around 80k images. Following previous works [9,18,23], we optionally employ a mask refinement stage to counteract any inaccuracies in the final masks. In particular, we adopt Pixel-Adaptive Mask Refinement (PAMR) [1], an iterative post-refinement method aimed at enhancing the fidelity of the similarities to the visual characteristics of the image. In our experiments, we use λequal to 5/6for background cleaning, a threshold of 0.55 on the similarity score to determine which pixels belong to the “background” category, and, when using mask refinement, employ PAMR with 10 iterations. Evaluation Protocol. We follow the standard evaluation protocol for unsupervised OVS [9], where prior access to ViT-Base (mIoU) ViT-Large (mIoU) Model Visual Encoder Frozen V20 C59 Stuff City ADE V21 C60 Object Avg V20 C59 Stuff City ADE V21 C60 Object Avg without Mask Refinement GroupViT [54] Custom ViT ✗79.7 23.4 15.3 11.1 9.2 50.4 18.7 27.5 29.4 - - - - - - - - - ReCo [41] CLIP ✗57.7 22.3 14.8 21.1 11.2 25.1 19.9 15.7 23.5 - - - - - - - - - TCL [9] CLIP ✗77.5 30.3 19.6 23.1 14.9 51.2 24.3 30.4 33.9 - - - - - - - - - SILC [36] Custom ViT ✗77.5 31.6 20.8 26.9 19.3 - - - - - - - - - - - - - MaskCLIP [63] CLIP ✓74.9 26.4 16.4 12.6 9.8 38.8 23.6 20.6 27.9 29.4 12.4 8.8 11.5 7.2 23.3 11.7 7.2 13.9 CLIP-DIY [51] CLIP+DINO ✓79.7 19.8 13.3 11.6 9.9 59.9 19.7 31.0 30.6 - - - - - - - - - SCLIP [47] CLIP ✓80.4 34.2 22.4 32.2 16.1 59.1 30.4 30.5 38.2 70.6 25.2 17.6 21.3 10.9 44.0 22.3 26.9 29.9 CLIP-DINOiser [52] CLIP ✓80.9 35.9 24.6 31.1 20.0 62.1 32.4 34.8 40.2 - - - - - - - - - ClearCLIP [25] CLIP ✓80.9 35.9 23.9 30.0 16.7 51.8 32.6 33.0 38.1 80.0 29.6 19.9 27.9 15.0 - - - - NACLIP [18] CLIP ✓79.7 35.2 23.3 35.5 17.4 58.9 32.2 33.2 39.4 78.7 32.1 21.4 31.4 17.3 52.2 28.7 29.9 36.5 dino.txt [21] DINOv2(reg) ✓- - - - - - - - - 62.1 30.9 20.9 32.1 20.6 - - - - FreeDA [3] DINOv2 ✓77.1 37.1 24.9 34.0 19.5 51.7 32.6 24.4 37.7 71.8 35.4 24.2 32.3 19.4 44.9 31.1 24.6 35.5 FreeDA [3] CLIP+DINOv2 ✓84.3 39.7 25.7 34.1 20.8 51.8 35.3 36.3 41.0 85.7 39.7 26.3 33.6 21.4 44.1 34.8 33.9 39.9 ProxyCLIP [26] CLIP+DINOv2(reg) ✓83.0 37.2 25.4 33.9 19.7 58.6 33.8 37.4 41.1 85.2 36.2 24.6 35.2 21.6 56.6 33.0 36.7 41.1 ProxyCLIP [26] CLIP+DINO ✓80.3 39.1 26.5 38.1 20.2 61.3 35.3 37.5 42.3 83.2 37.7 25.6 40.1 22.6 60.6 34.5 39.2 42.9 Talk2DINO (Ours) DINOv2(reg) ✓87.1 39.8 28.1 36.6 21.1 61.5 35.1 41.0 43.8 87.1 39.1 27.0 35.8 21.1 60.1 34.2 37.6 42.8 with Mask Refinement GroupViT [54] Custom ViT ✗81.5 23.8 15.4 11.6 9.4 51.1 19.0 27.9 30.0 - - - - - - - - - ReCo [41] CLIP ✗62.4 24.7 16.3 22.8 12.4 27.2 21.9 17.3 25.6 - - - - - - - - - TCL [9] CLIP ✗83.2 33.9 22.4 24.0 17.1 55.0 30.4 31.6 37.2 - - - - - - - - - MaskCLIP [63] CLIP ✓72.1 25.3 15.1 11.2 9.0 37.2 22.6 18.9 26.4 - - - - - - - - - SCLIP [47] CLIP ✓83.5 36.1 23.9 34.1 17.8 61.7 31.5 32.1 40.1 76.3 27.4 18.7 23.9 11.8 47.8 23.8 26.9 32.1 CLIP-DINOiser [52] CLIP ✓81.5 37.1 25.3 31.5 20.6 64.6 33.5 36.1 41.3 - - - - - - - - - LaVG [22] CLIP+DINO ✓82.5 34.7 23.2 26.2 15.8 62.1 31.6 34.2 38.3 - - - - - - - - - NACLIP [18] CLIP ✓83.0 38.4 25.7 38.3 19.1 64.1 35.0 36.2 42.5 84.5 36.4 24.6 37.1 19.6 57.9 36.4 34.6 41.4 FreeDA [3] DINOv2 ✓79.5 40.2 27.1 34.4 20.9 52.0 35.2 25.8 39.4 75.2 39.0 27.0 33.1 21.3 45.3 34.3 26.7 37.7 FreeDA [3] CLIP+DINOv2 ✓85.2 42.1 27.0 33.8 21.8 51.8 37.4 38.6 42.2 87.1 42.4 28.1 33.8 22.6 55.4 37.1 36.1 42.8 ProxyCLIP [26] CLIP+DINOv2(reg) ✓83.1 38.9 26.6 35.4 20.3 62.0 35.2 38.7 42.5 85.8 37.6 25.6 37.5 22.5 59.4 34.6 39.0 42.8 ProxyCLIP [26] CLIP+DINO ✓80.3 39.4 26.9 38.6 20.2 60.8 35.3 37.2 42.3 83.2 38.0 26.2 41.0 22.6 60.7 34.7 39.4 43.2 Talk2DINO (Ours) DINOv2(reg) ✓88.5 42.4 30.2 38.1 22.5 65.8 37.7 45.1 46.3 89.8 42.7 29.6 38.4 22.9 66.1 37.3 42.3 46.1 Table 1. Comparison with unsupervised OVS models on Pascal VOC [15], Pascal Context [34], COCO Stuff [7], COCO Object [7], Cityscapes [11], and ADE20K [61,62]. For each method, we specify the visual backbone used, along with whether it is frozen or finetuned. We report both the variants with and without background for Pascal VOC (V21 and V20) and Pascal Context (C60 and C59). Best results with and without mask refinement are highlighted in bold, overall best results are underlined. the target data before evaluation is not allowed, and use the default class names provided by the MMSegmentation toolbox. The images are resized to have a shorter side of 448, using a sliding window approach with a stride of 224 pixels. All models are evaluated using mean Intersectionover-Union (mIoU) on all the classes of each dataset. 4.2. Comparison with the State of the Art We compare Talk2DINO with previous state-of-the-art approaches for unsupervised OVS on the five benchmarks that do not include the “background” category and the three benchmarks with the “background” category. We consider as competitors: (i) prototype-based approaches, such as ReCo [41] and FreeDA [3], which aim to create visual prototypes associated with the textual categories, (ii) CLIP adaptations, as MaskCLIP [63], CLIP-DIY [51], SCLIP [47], ClearCLIP [25], and NACLIP [18], which propose architectural modifications to enhance its localization properties, (iii) methods trained on sets of imagecaption pairs with objectives designed to force the segmentation capabilities to emerge, like GroupViT [54], TCL [9], SILC [36], and dino.txt [21], and (iv) methods that aim to combine the properties of CLIP and DINO, as CLIPDINOiser [52], LaVG [22], and ProxyCLIP [26]. Table 1reports the results on the five benchmarks without background (i.e., Pascal VOC-20, Pascal Context59, COCO Stuff, Cityscapes, and ADE) and the three benchmarks with background (i.e., Pascal VOC-21, Pascal Context-60, and COCO Object). Specifically, we report the performance of both the base and large configurations of both Talk2DINO and the competitors, according to their definitions in the original papers. Moreover, we divide the table into two sections depending on whether a mask refinement technique is employed. Specifically, LaVG exploits a custom region proposer combined with DenseCRF [24], while all the other methods refine their masks with PAMR. Regarding datasets with background, Pascal VOC and COCO Objects present only foreground categories, also referred to as “things” in the literature, while Pascal Context presents both categories from foreground and background, also mentioned as “stuff”. Hence, following CLIP-DINOiser [52], we report the performance with mIoU Visual Backbone V20 C59 Stuff City ADE DINO ViT-S 27.7 13.2 7.4 14.8 5.2 DINOv2 (without registers) ViT-S 83.7 38.3 25.8 32.9 19.8 DINOv2 (with registers) ViT-S 86.9 35.3 24.5 27.2 16.9 MAE ViT-B 9.5 4.3 2.0 4.0 1.1 CLIP ViT-B 55.4 14.9 12.2 6.2 3.8 DINO ViT-B 27.3 11.2 7.9 13.6 4.5 DINOv2 (without registers) ViT-B 74.2 31.9 23.0 27.9 16.5 DINOv2 (with registers) ViT-B 87.1 39.8 28.1 36.6 21.1 MAE ViT-L 6.7 2.7 1.4 4.6 0.9 CLIP ViT-L 16.6 5.0 7.7 0.9 2.0 DINOv2 (without registers) ViT-L 56.0 20.1 14.9 18.1 8.2 DINOv2 (with registers) ViT-L 87.1 39.1 27.0 35.8 21.1 Table 2. Ablation study results using different visual backbones and with different sizes of the ViT architecture. the background cleaning procedure described in Sec. 3.3 only on Pascal VOC and COCO Objects. As it can be observed, our approach achieves the best average mIoU on all the configurations and presents a consistent improvement compared to the considered competitors, with and without the mask refinement, across all datasets except Cityscapes. The most straightforward comparison is the one with FreeDA without global similarity (i.e., with DINOv2 only as visual backbone). It builds a bridge between DINOv2 and the CLIP text encoder by retrieving from a collection of visual-textual embedding pairs and by building visual prototypes for each textual category. The significant improvement achieved by Talk2DINO demonstrates that training a direct projection from the CLIP text encoder to DINOv2 leads to a more accurate bridge between the two embedding spaces without the overhead in computation and memory provided by the retrieval procedure. 4.3. Ablation Studies and Analyses Choosing Different Visual Backbones. In Table 2we show the performance of our approach when varying the visual backbone and the size of the employed ViT architecture. We observe that backbones that differ from DINOv2 present unsatisfactory results and can not be aligned to the CLIP textual encoder with a learnable mapping. In particular, while DINO achieves the second best performance on average, Talk2DINO heavily benefits from the strong semantic representation of the dense features of DINOv2 and on the capabilities of its self-attention heads in highlighting coherent regions of the image – properties which are not reflected in other visual backbones. We refer to the supplementary materials for more details on self-attention heads. Moreover, our results emphasize the critical role of registers [12] in DINOv2, as demonstrated by the comparison between its variants with and without registers. Registers are a recently proposed mechanism to mitigate the presence of artifacts in the feature maps of ViT-based backbones. ArmIoU V20 C59 Stuff City ADE Effect of Projection Linear Projection (text only) 85.1 37.9 26.7 35.6 20.1 Our mapping (both vision and text) 59.2 27.3 18.9 23.5 13.5 Our mapping (vision only) 84.6 35.2 26.2 20.4 15.5 Our mapping (text only) 87.1 39.8 28.1 36.6 21.1 Effect of Self-Attention Selection and Aggregation CLS only (without self-attention) 84.5 30.6 23.0 22.6 17.2 Standard average 89.6 36.9 25.6 33.5 19.7 CLS-weighted average 87.6 35.2 23.1 29.3 17.5 CLS similarity-weighted sampling 88.2 32.9 22.6 27.0 17.9 Max CLS similarity 87.1 39.8 28.1 36.6 21.1 Table 3. Ablation study evaluating the impact of the core components of the proposed architecture on the final performance. We report the results using the base model of DINOv2. tifacts are tokens that exhibit a significantly higher norm with respect to the other tokens and retain less information about their original position in the image. The alignment process in our method relies on high-quality attention maps, and the presence of artifacts poses a challenge by limiting the selection of the most relevant self-attention heads. Interestingly, since register-related artifacts are more pronounced in larger backbones, the ViT-S variant without registers maintains competitive performance compared to its register-enabled counterpart. Finally, we observe that our approach maintains robust and consistent performance across different ViT sizes, achieving strong results even with the compact ViT-S backbone. This suggests that our method is effective across a range of model sizes, making it adaptable to varying computational constraints. Additional insights and detailed evaluations of the different visual backbones can be found in the supplementary. Impact of the Proposed Components. Table 3reports the results of Talk2DINO evaluating the impact of its core components on the overall performance. Specifically, in the first section of the table, we analyze the effect of the adopted projection ψ. Replacing it with a linear projection leads to a slight performance drop. The good performance obtained by a linear transform evidences how the DINOv2 and CLIP spaces are intrinsically compatible, as the former can be obtained through an affine transform of the latter without losing too much information. Interestingly, applying the proposed projection on top of DINOv2 or using two projections on both spaces significantly lowers performance, confirming the appropriateness of the proposed approach. In the second section of the table, we instead study the effect of the selection and aggregation strategy of selfattention heads Ai, i = 1, ..., N during training. In particular, we test aligning (i) directly the visual CLS token to the textual CLS token, (ii) the visual embedding from the standard average self-attention, (iii) the weighted mean of the head embeddings vAi, where the weights are given by their Image Ground-truth FreeDA [3] ProxyCLIP [26] CLIP-DINOiser [52] Talk2DINO (Ours) Figure 4. Qualitative results of Talk2DINO in comparison with FreeDA [3], ProxyCLIP [26], and CLIP-DINOiser [52]. mIoU Mask Refinement V21 Object without background cleaning ✗59.9 37.1 with background cleaning ✗61.5 41.0 without background cleaning ✓63.9 40.3 with background cleaning ✓65.8 45.1 Table 4. Ablation study on the impact of the background cleaning procedure. We report the results using DINOv2 ViT-B. softmaxed similarity with the textual CLS token, (iv) a strategy in which we sample a single head embedding, where the sampling probability is given by the softmaxed similarity with the CLS token, and (v) our adopted solution in which we select the head embedding which is the most similar to the textual CLS token. The results show that only on the Pascal VOC dataset – composed mostly by large subjects in foreground – the embedding from the standard average self-attention presents improved performance. On all other benchmarks, our approach proves to be the most effective, further validating the robustness of our selection method. Effect of Background Cleaning. Table 4shows how the performance is affected by the background cleaning mechanism and by the usage of PAMR for mask refinement. It can be observed that the background cleaning procedure has a significantly positive impact on Pascal VOC and COCO Object, leading, respectively, to a +1.6 and +3.9 increase in mIoU score. Further, it can be noticed that the effectiveness of the proposed background cleaning procedure is confirmed also when applying the mask refinement. Qualitative results showing the effect of the background cleaning procedure are available in the supplementary material. Qualitative Results. Fig. 4depicts qualitative segmentation results, in which we highlight the segmentation capabilities of Talk2DINO along with other state-of-the-art models (i.e., FreeDA [3], ProxyCLIP [26], and CLIPDINOiser [52]). We show two images from Pascal VOC, in which it can also be appreciated how the background cleaning procedure leads to high-quality masks and localization, and two images from COCO Stuff and Pascal Context where Talk2DINO effectively segments “things” in the scene, such as the boat and teddy bear, and “stuff” categories, such as sky and road. 5. Conclusion In this paper, we introduced Talk2DINO, a novel approach for OVS that bridges the spatially detailed embeddings of the DINOv2 self-supervised vision backbone with the highly semantic text embeddings of CLIP. Our method achieves fine-grained alignment between textual concepts and visual patches without the need for extensive finetuning of the backbone networks, only leveraging selfattention maps from DINOv2 and a lightweight languageto-vision mapping layer. Talk2DINO achieves state-of-theart results in standard OVS benchmarks, outperforming previous prototype-based, CLIP adaptation, and contrastive learning approaches. Our study highlights the potential of integrating vision-only and multimodal models, suggesting broader applications in fine-grained and cross-modal tasks. Acknowledgments We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support. This work has been conducted under a research grant co-funded by Leonardo S.p.A., and supported by the PNRRM4C2 project “FAIR - Future Artificial Intelligence Research”, by the PRIN 2022-PNRR project “MUCES” (CUP E53D23016290001 and B53D23026090001), and by the PNRR project “ITSERR - Italian Strengthening of Esfri RI Resilience” (CUP B53C22001770006), all funded by the European Union - NextGenerationEU, and by the SUN XR project funded by the Horizon Europe Research & Innovation Programme (GA n. 101092612). References [1] Nikita Araslanov and Stefan Roth. Single-Stage Semantic Segmentation from Image Labels. In CVPR, 2020. 5 [2] Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. FOSSIL: Free Open-Vocabulary Semantic Segmentation through Synthetic References Retrieval. In WACV, 2024. 3 [3] Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Training-free openvocabulary segmentation with offline diffusion-augmented prototype generation. In CVPR, 2024. 1,3,6,8,2,9 [4] Ioana Bica, Anastasija Ilic, Matthias Bauer, Goker Erdogan, Matko Boˇ snjak, Christos Kaplanis, Alexey A Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, et al. Improving fine-grained understanding in imagetext pre-training. In ICML, 2024. 2 [5] Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding Everything: Emerging Localization Properties in Vision-Language Transformers. In CVPR, 2024. 2 [6] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´ erez. Zero-shot semantic segmentation. In NeurIPS, 2019. 1 [7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCOStuff: Thing and Stuff Classes in Context. In CVPR, 2018. 4,5,6,3 [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, 2021. 2 [9] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning To Generate Text-Grounded Mask for Open-World Semantic Segmentation From Only Image-Text Pairs. In CVPR, 2023. 1,2,5,6,3 [10] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation. In CVPR, 2024. 2 [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR, 2016. 5,6,3 [12] Timoth´ ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers. In ICLR, 2024. 2,5,7 [13] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022. 2 [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021. 2 [15] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results, 2012. 4,5,6,3 [16] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. In CVPR, 2023. 2 [17] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, 2022. 1,2 [18] Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation. In WACV, 2025. 1,2,5,6,3 [19] Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network. In ICCV, 2023. 2 [20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In ICML, 2021. 2 [21] Cijo Jose, Th´ eo Moutakanni, Dahyun Kang, Federico Baldassarre, Timoth´ ee Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Micha¨ el Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24905–24916, 2025. 6 [22] Dahyun Kang and Minsu Cho. In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation. In ECCV, 2024. 3,6 [23] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion Models for Zero-Shot OpenVocabulary Segmentation. In ECCV, 2024. 3,5 [24] Philipp Kr¨ ahenb¨ uhl and Vladlen Koltun. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In NeurIPS, 2011. 6 [25] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference. In ECCV, 2024. 2,6 Model Visual Encoder V20 C59 Stuff City ADE V21 C60 Object Avg DINOv2 ViT-B/14 with registers (without Mask Refinement) FreeDA DINOv2 83.4 39.5 25.9 35.2 20.7 50.1 ▷43.6 34.3 23.8 ▷24.7 39.1 ▷38.4 FreeDA CLIP+DINOv2 87.0 40.6 25.7 34.2 21.2 49.3 ▷41.8 35.7 34.8 ▷34.7 41.1 ▷40.1 ProxyCLIP CLIP+DINOv2 83.0 37.2 25.4 33.9 19.7 58.6 ▷60.0 33.8 37.4 ▷37.3 41.1 ▷41.3 Talk2DINO DINOv2 87.1 39.8 28.1 39.6 21.1 59.9 ▷61.5 35.1 37.1 ▷41.0 43.5 ▷44.2 DINOv2 ViT-B/14 with registers (with Mask Refinement) FreeDA DINOv2 84.9 42.3 27.7 36.8 22.0 50.2 ▷43.7 36.7 24.5 ▷25.5 40.6 ▷40.0 FreeDA CLIP+DINOv2 87.4 42.4 26.6 34.8 22.1 49.4 ▷41.7 37.2 36.6 ▷36.7 42.1 ▷41.1 ProxyCLIP CLIP+DINOv2 83.1 38.9 26.6 35.4 20.3 62.0 ▷63.4 35.2 38.7 ▷38.6 42.5 ▷42.7 Talk2DINO DINOv2 88.5 42.4 30.2 41.6 22.5 63.9 ▷65.8 37.7 40.3 ▷45.1 45.9 ▷46.7 Table 10. Comparison between FreeDA, ProxyCLIP, and Talk2DINO when using DINOv2 with and without registers. For VOC21, Object, and the average, we report the results without background cleaning on the left and with background cleaning on the right. challenging or outlier queries. In addition to computing text-image similarities using cosine similarity between a global text token and a global image token, we experiment with a similarity function that mirrors the one used during training. Specifically, instead of representing the image with the mean of the vAiembeddings and calculating similarity as the cosine similarity between this representation and the text encoding, we represent the image using all vAiembeddings. We compute the similarity as maxi=1,...,N sim(vAi, t), taking the maximum similarity score across all heads. This alternative similarity function leads to significant performance improvements, allowing Talk2DINO to surpass CLIP on several metrics. This enhancement is likely due to the ability of the model to evaluate captions at a finer granularity. Captions often describe multiple aspects of an image, including both foreground and background elements. By individually examining different regions of the image as detected by distinct attention heads, the model can assign more precise scores, ultimately boosting retrieval accuracy. C. Activation Map Visualizations In Fig. 6, we show the distribution of attention heads selected for alignment with the text input during the final epoch of training. The results indicate that certain heads, particularly heads 1 and 3, are more often aligned with the text than others. However, aside from these, the remaining heads are relatively evenly distributed. These findings are noteworthy because they suggest that some heads specialize in capturing features that align more closely with the input caption, while all heads contribute meaningfully during training. Notably, no head shows a negligible activation frequency, highlighting the importance of the entire set of attention heads in the alignment process. Fig. 7presents examples from the training set, showcasing images paired with their corresponding captions and the attention maps selected for alignment. Despite describing the same scene, variations in the captions lead the alignment procedure to focus on different regions of the image. For instance, in the first row, the caption mentioning the fans also 01234567891011 Head number 0 2 5 7 10 12 15 % Occurrences DINOv2 Head Activations Figure 6. Percentage of times each attention head of ViT-B backbone is selected for alignment to textual embeddings on the final epoch of training. The dashed line denotes uniform distribution. focuses on the background, while captions that reference only the player, the ball, and the racket do not. D. Additional Qualitative Results Effect of Background Cleaning. Fig. 10 shows a set of qualitative results in which we highlight the advantages of using the proposed background cleaning procedure with respect to directly thresholding the similarities with the input categories to detect the background. In particular, the first two rows show four qualitatives on images from COCO Object and the last two rows from VOC. These results demonstrate that background cleaning removes the noise in the background from the image and improves the fitting of the masks on the foreground objects. These findings are reflected in the results reported in Table 4of the main paper. In-the-Wild Qualitative Examples. Fig. 11 depicts a few examples of “in-the-wild” segmentation, obtained by providing to Talk2DINO sample images from the web and asking it to segment uncommon categories, such as “pikachu”, “millennium falcon”, and “westminster abbey”, and free-form text, like “golden retriever puppy”. On the left, we show three examples in which we task the model with also finding the background, while exploiting the background clean- Head 1: A professional tennis player hits a ball as fans watch. Head 5: champion tennis player swats at the ball hoping to win Head 7: A man hitting a tennis ball with a racquet. Head 10: A man is hitting his tennis ball with a recket on the court. Head 11: a tennis player on a court with a racket Head 0: A gray and white building on the corner of bay street with building behind it. Head 1: A street in the ciy San franscisco on a good day Head 2: A tree leaning on a building on Bay Street. Head 5: A car driving down a street next to a tall building. Head 9: A street sign for Bay Street in a residential neighborhood. Head 1: A puppy holding a black disk on the couch Head 2: A dog sitting on a couch holding a frisbee in its mouth. Head 3: A dog holding a plate while sitting on a chair. Head 9: A bull dog holding a frisbee in it’s mouth. Head 10: a brown black and white dog and a black frisbee Head 0: several of the baseball players are in view Head 1: A large crowd is watching a baseball game. Head 3: Players on a baseball field during a game. Head 5: A man is up to bat at a professional baseball game. Head 6: a man holding up his baseball bat during a baseball game Head 1: A white round table filled with some assorted treats. Head 6: A number of pastry items on a table Head 7: A dessert tray with donuts, cupcakes, and muffins Head 10: A white plate topped with chocolates, donuts, and other goodies. Head 11: Donuts and other goodies on a table Figure 7. Sample images from the training set paired with their corresponding captions and the attention maps selected for alignment during the last epoch of training. ing procedure, and, on the right, three examples in which the model has to assign a provided category to each pixel. The high quality of the resulting masks demonstrates the efficacy of our approach, even on out-of-domain images. From these examples, we can appreciate the capabilities of the model in combining the knowledge from CLIP with the semantic localization of DINOv2 on unconventional concepts, such as fictional character names and proper nouns of historical buildings. Comparison with State-of-the-Art Methods. Finally, in Fig. 12 we report a set of qualitative results on the five datasets used for the evaluation of the models, in addition to the qualitative depicted in Fig. 4of the main paper. We compare the segmentation masks of Talk2DINO with the ones of FreeDA [3], ProxyCLIP [26], and CLIP-DINOiser [52], which represent our main competitors. In particular, we report a pair of images from Pascal VOC with background and eight pairs of images from Pascal Context, COCO Stuff, Cityscapes, and ADE20K, without background. As it can be seen, these qualitative results further highlight the impressive segmentation capabilities of Talk2DINO with both background and foreground categories. Image ViT-S ViT-S Reg. ViT-B ViT-B Reg. ViT-L ViT-L Reg. ViT-S ViT-S Reg. ViT-B ViT-B Reg. ViT-L ViT-L Reg. Figure 8. Comparison of DINOv2 with and without registers across different visual backbones (ViT-S, ViT-B, and ViT-L). The results highlight how the ViT-B and ViT-L backbones without registers exhibit artifacts that introduce noise during the alignment process. Image DINOv2 DINO MAE CLIP DINOv2DINO MAE CLIP 2 Figure 9. Self-attention activations of different visual backbones (i.e., DINOv2, DINO, MAE, CLIP). with without with without Image Ground-truth Bkg. Cleaning Bkg. Cleaning Image Ground-truth Bkg. Cleaning Bkg. Cleaning Figure 10. Qualitative results obtained with and without the proposed background cleaning strategy, on COCO Object and Pascal VOC. millennium falcon tordelli westminster abbey golden retriever puppy grass field vegetation pikachu traffic sign forest route axolotl stones leaves gravel Figure 11. ”In-the-wild” segmentation results obtained by prompting Talk2DINO with uncommon textual categories on images retrieved from the web. Image Ground-truth FreeDA [3] ProxyCLIP [26] CLIP-DINOiser [52] Talk2DINO (Ours) Pascal VOCPascal ContextCOCO StuffCityscapesADE20K Figure 12. Additional qualitative results of Talk2DINO in comparison with FreeDA [3], ProxyCLIP [26], and CLIP-DINOiser [52].