A Comparative Demonstration of Relevance Feedback Methods for Image Retrieval

A Comparative Demonstration of Relevance Feedback Methods for Image Retrieval Francesca Scotti  , Lucia Vadicamo , Giuseppe Amato , and Fabio Carrara Institute of Information Science and Technologies, CNR, Pisa, Italy {francesca.scotti,lucia.vadicamo,giuseppe.amato,fabio.carrara}@isti.cnr.it Abstract. Relevance feedback is a well-established approach to refine search results based on user input, but its comparative evaluation across different methods remains limited in practice. This demonstration paper introduces an interactive platform that supports and compares four relevance feedback methods — Rocchio, PicHunter, Polyadic Search, and SVM-based active learning — under consistent conditions. The primary goal is to enhance the understanding of how different relevance feedback methods affect retrieval performance from both a technical and user-centric perspective. The source code is available at https://github. com/francescascotti16/Demo-Relevance-Feedback, while the demonstration can be found at http://relevance-feedback.isti.cnr.it/. Keywords: Interactive Video and Image Retrieval ·Relevance Feedback ·Rocchio ·Pichunter ·SVM ·Polyadic Search 1 Introduction In the evolving field of content-based multimedia retrieval, modern systems leveraging state-of-the-art image and video features [9,15,13] have demonstrated strong performance. However, a fundamental challenge remains: ensuring that retrieval results accurately align with the user’s actual information needs, which can be dynamic during exploratory searches or very specific in ad-hoc and knownitem searches. Relevance feedback has played a crucial role in improving this alignment by enabling interactive systems to refine results based on user input [20,8,12]. For instance, the Exquisitor [11] image and video retrieval system employs linear SVM models to efficiently refine the search on the basis of user-selected relevant and non-relevant examples. The SOMHunter [14] system, on the other hand, employs the Bayesian PicHunter model [7]. Another approach by Claydon et al. [3] employs the MSED [6], an information distance measure, to evaluate polyadic queries where multiple query objects are derived from explicit user feedback. Despite the growing adoption of relevance feedback approaches, different methods may be better suited to different application domains, user tasks, and interaction models. However, current image and video retrieval systems typically focus on implementing a single relevance feedback technique, limiting the 2 F. Scotti et al. Fig. 1: System interface during the initial relevance feedback stage. Starting from the results of an initial query (e.g., “a man running”), the user can mark images as relevant (green thumbs-up) or non-relevant (red thumbs-down) to guide subsequent search refinement. ability to compare different approaches and understand their relative effectiveness in various search scenarios. The difficulty and effort required to set up and manage multiple algorithms in consistent conditions make such comparisons resource-intensive. This gap in comparative evaluation limits both practical adoption and further methodological advancements. Our demonstration system (Figure 1) addresses this limitation by providing a unified interactive platform where users can explore and compare multiple relevance feedback techniques in real time. In our current implementation, the system supports four distinct methods: the Rocchio Algorithm, PicHunter, Polyadic Query, and SVM-based relevance feedback. These techniques were selected as they represent different categories of feedback techniques, covering vector-based, probabilistic, polyadic, and classification-based approaches, and have also been adopted in practical interactive systems [14,3,11]. This work extends on our recent study [19], which focused on a quantitative analysis of these four relevance feedback methods for ad-hoc video search. In this demo, we enable a qualitative, interactive comparison across multiple feedback iterations and retrieval scenarios. Compared to our previous setup [19], where users started from a random display, the current system supports text-based queries as initial input, which are now integrated into the feedback algorithms for a more consistent and user-driven experience. Although our framework is not yet designed for seamless integration of new methods, its modular backend hints at the potential for future extensibility. Our goal is to offer a unified platform where users — both general and expert — can explore, compare, and reflect on the behavior of different relevance feedback approaches in a controlled and reproducible environment. A demonstrator of our system is publicly available at http://relevance-feedback. isti.cnr.it/. A Comparative Demonstration of Relevance Feedback Methods 3 2 Relevance Feedback Algorithms Overview Our system implements relevance feedback methods that refine search results based on user-provided labels (relevant or non-relevant) on a set of nDdisplayed images Dt. At each iteration t, the user selects a subset of relevant (A+ t) and non-relevant (A− t) examples, which the system uses to update the display Dt+1, aiming to better align results with the user’s intent. This section provides a high-level summary of the four relevance feedback algorithms implemented in our system: (1) Rocchio [16], a classic query point modification algorithm; (2) Polyadic Search, adapted from the method originally proposed by Connor et al. [4] and used in [3], which refines search results using multiple independent query objects; (3) PicHunter [7], and our enhanced version PicHunter⋆, which use the Bayesian theory to estimates data object relevance based on user feedback; and (4) the Support Vector Machines Active Learning [18,10], which leverages the distance from the hyperplane separating relevant from non-relevant examples to refine the search. We refer the reader to our previous study [19] for the quantitative comparative analysis of these methods in an ad-hoc video search scenario, as well as for a comprehensive discussion of their underlying formulations and performance. Rocchio The Rocchio algorithm [16] is a traditional approach that updates a query vector by shifting it closer to the centroid of relevant documents while moving it further from the centroid of non-relevant documents. The following formula defines the version of the algorithm implemented in our system: qt+1 =αq0+β1 |A+ ≤t|X o+ j∈A+ ≤t o+ j−γ1 |A− ≤t|X o− j∈A− ≤t o− j(1) where q0is the initial query, A≤t=Si≤tAi, and α,β, and γare weight parameters. The new query qt+1 is used to search the original space and create the new display as Dt+1 =kNN(qt+1)with k=nD, where the nearest-neighbor search is performed on the original data feature space PicHunter/PicHunter⋆PicHunter [7] employs a Bayesian approach to estimate the likelihood distribution of images that best align with the user’s target preferences, represented as a random variable T. At each iteration t, the algorithm updates the posterior probability P(T=oi|Ht)for each image oiin the dataset, based on the feedback history Ht, which includes both the displayed images and the corresponding user feedback. The probability update follows Bayes’ Theorem [7]: P(T=oi|Ht) = P(At|T=oi, Dt)P(T=oi|Ht−1) Pn j=1 P(At|T=oj, Dt)P(T=oj|Ht−1).(2) At each step, the top nDimages with the highest posterior probabilities are selected for display. 4 F. Scotti et al. The original PicHunter assumes only positive feedback (At=A+ t) and computes the user model P(At|T=oi, Dt)via a softmin function over the distances between data objects and positive examples. In our enhanced version, PicHunter⋆ [19], we also incorporate negative feedback by combining the softmin function for positive examples with a softmax function over the distances to negative examples. Additionally, instead of initializing P(T=oi|H0)with a uniform value, we compute it via a softmin over distances from the initial query q0, which improved recall in our tests. Polyadic Search Polyadic search [4,3] uses user-selected positive samples to form polyadic queries, which can be used in relevance feedback scenarios. Our system implements an enhanced version described in [19], which also incorporates negative examples and iteratively computes a relevance score ρt+1(o)for each data object oas ρt+1(o) = αρ0(o) + β sim A+ ≤t,o−γ sim A− ≤t,o,(3) where sim(·)is a polyadic similarity function and ρ0(o) = sim({q0},o)is the similarity to the initial query q0. The scores ρt+1 are used to rank the dataset and select the top nDitems for the new display. We implemented two polyadic similarity functions between a set of queries V={v1,...,vm}and a data object o: 1. sMSED(V, o) = 1 −MSED(f(V), f(o)), based on the MSED divergence [6] over over probability vectors derived from the data and query set; 2. sSED(V, o) = 1 −SED Pv∈Vf(v)/|V|, f(o), based on the Structural Entropic Distance (SED) [5] between probability vectors derived from the object and the centroid of the query set. Both functions use entropy-based similarity computed over feature vectors transformed using the softmax function (i.e., fis the softmax). SVM-based Relevance Feedback Tong and Chang [18] introduced an approach that integrates SVMs with an active learning strategy to enhance relevance feedback in image retrieval. In our system, user-provided feedback up to iteration tis used to train a linear binary SVM classifier, which separates relevant and non-relevant examples. The user’s initial query q0is considered as a relevant image. The system then updates the display by selecting the nDimages farthest from this hyperplane on the positive (relevant) side. This process involves conducting an inner product search based on the hyperplane’s parameters. Since SVM training requires both positive and negative examples, if the user does not provide any negative feedback, we randomly select 10 images from the dataset (excluding those currently displayed) as negative examples without prior knowledge of their actual relevance. Note that using only negative examples is not supported in our demo. A Comparative Demonstration of Relevance Feedback Methods 5 Table 1: Average recall@50 and average iteration time (in seconds) for each tested method, with and without the use of the initial textual query embedding q0. Polyadic Rocchio PicHunter PicHunter⋆MSED SED SVM recall@50 (with q0)0.74 0.67 0.70 0.70 0.75 0.68 recall@50 (without q0)0.69 0.62 0.68 0.68 0.70 0.67 Avg. iteration time 0.04s 0.09s 0.09s 3.49s 3.44s 0.04s 3 System Overview Our system enables searches over approximately 2.6 million images extracted from the V3C1 and V3C2 video collection [17]. Unlike our previous work [19], where search started from a random display, users can now initiate queries using natural language. Results are retrieved and displayed, and users can iteratively refine them by marking some images as relevant or non-relevant and selecting a relevance feedback method. The initial cross-modal retrieval step is powered by the VISIONE [2,1] system, which leverages CLIP ViT-H/14 - LAION-2B multimodal features [9,15]. This means that the user’s textual query is converted into a text embedding q0, which is then used to generate the initial display and to support subsequent feedback-based updates. The initial display is formed using the top nD= 400 images, while the top 10,000 retrieved images serve as the reference set that is re-ranked at each step based on user feedback. Interaction is straightforward: users click thumbs-up or thumbs-down icons to label results as relevant or non-relevant Feedback’s history is displayed on the left panel and can be modified at any time. A drop-down menu lets users choose the relevance feedback method (Rocchio,PicHunter,PicHunter⋆,Polyadic-SED, Polyadic-MSED,SVM and Compare All), and pressing “Update” button updates the results. This process can be repeated iteratively until the user finds the desired content. Figure 2 presents qualitative results obtained with different relevance feedback methods using the user feedback provided as an example in Figure 1. Table 1 reports the average Recall@50 and iteration time across 80 queries on a dataset of 10,000 images [19], using three positive and five negative examples per iteration. We compare the results obtained when incorporating the initial textual query embedding q0into the feedback update with those obtained using the implementation and parameters from [19], where the initial query is set to zero (q0=0). All parameters were selected via grid search. For Rocchio, we use (α=1.2, β=2.4, γ=2.6) when q0is included, and (β=0.75, γ=0.5) when it is excluded. For Polyadic search, we use (α=0.4, β=2.0, γ=1.4) for MSED and (α=1.0, β=1.6, γ=1.6) for SED when q0is included, and (β=1.0, γ=0.75) for both methods when q0=0. To provide additional performance insights, the interface also displays the update time for each method. Note that no indexing techniques were employed, 6 F. Scotti et al. (a) Initial query and user feedback (b) Rocchio algorithm (c) SVM algorithm (d) PicHunter algorithm (e) PicHunter⋆algorithm (f) Polyadic-MSED algorithm (g) Polyadic-SED algorithm Fig. 2: (a) The user first enters the initial query q0(1), then selects an algorithm from the dropdown menu or the “Compare All” option (2). Once the results are returned, they can provide relevance feedback by marking images as relevant (Liked) or non-relevant (Disliked) (3). Figures (b) to (g) illustrate the results of different relevance feedback algorithms, enabling a visual comparison of their behavior. A Comparative Demonstration of Relevance Feedback Methods 7 leaving room for potential efficiency improvements with optimized indexing strategies. Implementation and Technologies The user interface is built using JavaScript, HTML, and CSS and communicates via HTTP requests to a backend implemented with Python and Flask. The backend communicates with the VISIONE system and implements the relevance feedback algorithms. Flask manages HTTP requests, handling the interaction between the user and the server. When a user submits a query, Flask processes the request, retrieves results from the VISIONE server, applies the relevance feedback algorithms and returns the updated ranked list of results to the user. We used an on-disk forward index based on HDF5 to access the image features from the IDs returned by the VISIONE server. The implementation of the relevance feedback algorithms and the demonstration code are publicly available at https://github.com/francescascotti16/ Demo-Relevance-Feedback. 4 Conclusions We presented a demo system for comparing four relevance feedback algorithms in content-based image retrieval. Our system provides a user-friendly interface for both general users and researchers to explore these techniques. This demo, along with the source code, are publicly available and serve as a foundation for exploring and comparing additional relevance feedback methods. Future work will focus on improving the system’s time efficiency through optimized indexing strategies, ensuring faster query processing and more responsive user interactions. Acknowledgments. This work was partially funded by MUCES – a MUltimedia platform for Content Enrichment and Search in audiovisual archives (EU NextGenerationEU - PRIN 2022 PNRR P2022BW7CW - CUP: B53D23026090001) and SUN – Social and hUman ceNtered XR (EC, Horizon Europe n. 101092612). Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. Amato, G., Bolettieri, P., Carrara, F., Debole, F., Falchi, F., Gennaro, C., Vadicamo, L., Vairo, C.: The VISIONE video search system: exploiting off-theshelf text search engines for large-scale video retrieval. Journal of Imaging 7(5), 76 (2021) 2. Amato, G., Bolettieri, P., Carrara, F., Falchi, F., Gennaro, C., Messina, N., Vadicamo, L., Vairo, C.: Visione 5.0: Enhanced user interface and ai models for vbs2024. In: International Conference on Multimedia Modeling. pp. 332–339. Springer (2024) 8 F. Scotti et al. 3. Claydon, B., Connor, R., Dearle, A.: Demonstrating the efficacy of polyadic. In: Similarity Search and Applications: 17th International Conference, SISAP 2024, Providence, RI, USA, November 4-6, 2024, Proceedings. vol. 15268, p. 49. Springer Nature (2024) 4. Connor, R., Dearle, A., Morrison, D., Chávez, E.: Similarity search with multipleobject queries. In: Pedreira, O., Estivill-Castro, V. (eds.) Similarity Search and Applications. pp. 223–237. Springer (2023) 5. Connor, R., Dearle, A., Morrison, D., Chávez, E.: Similarity search with multipleobject queries. In: International Conference on Similarity Search and Applications. pp. 223–237. Springer (2023) 6. Connor, R., Simeoni, F., Iakovos, M., Moss, R.: A bounded distance metric for comparing tree structure. Information Systems 36(4), 748–764 (2011) 7. Cox, I.J., Miller, M.L., Minka, T.P., Papathomas, T.V., Yianilos, P.N.: The bayesian image retrieval system, pichunter: theory, implementation, and psychophysical experiments. IEEE transactions on image processing 9(1), 20–37 (2000) 8. Crucianu, M., Ferecatu, M., Boujemaa, N.: Relevance feedback for image retrieval: a short survey. Report of the DELOS2 European Network of Excellence (FP6) (2004) 9. Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., Schmidt, L.: Openclip (2021). https://doi.org/10.5281/zenodo.5143773 10. Khan, O.S., Aumüller, M., Jónsson, B.Þ.: Suitability of nearest neighbour indexes for multimedia relevance feedback. In: Pedreira, O., Estivill-Castro, V. (eds.) Similarity Search and Applications. pp. 133–147. Springer Nature Switzerland, Cham (2023) 11. Khan, O.S., Zhu, H., Sharma, U., Kanoulas, E., Rudinac, S., Jónsson, B.Þ.: Exquisitor at the video browser showdown 2024: Relevance feedback meets conversational search. In: International Conference on Multimedia Modeling. pp. 347–355. Springer (2024) 12. Li, J., Allinson, N.M.: Relevance feedback in content-based image retrieval: a survey. In: Handbook on neural information processing, pp. 433–469. Springer (2013) 13. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., et al.: Dinov2: Learning robust visual features without supervision (2023), https: //arxiv.org/abs/2304.07193 14. Peška, L., Vomlelová, M., Vesel`y, P., Škrhák, V., Lokoč, J.: Evaluating a bayesianlike relevance feedback model with text-to-image search initialization. Multimedia Tools and Applications 82(15), 22305–22341 (2023) 15. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (18–24 Jul 2021) 16. Rocchio Jr, J.J.: Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing (1971) 17. Rossetto, L., Schuldt, H., Awad, G., Butt, A.A.: V3C - A research video collection. In: International Conference on Multimedia Modeling. pp. 349–360. Springer (2019) 18. Tong, S., Chang, E.: Support vector machine active learning for image retrieval. In: Proceedings of the ninth ACM international conference on Multimedia. pp. 107–118 (2001) A Comparative Demonstration of Relevance Feedback Methods 9 19. Vadicamo, L., Scotti, F., Dearle, A., Connor, R.: Comparative analysis of relevance feedback techniques for image retrieval. In: International Conference on Multimedia Modeling. pp. 206–219. Springer (2025) 20. Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: A comprehensive review. Multimedia systems 8, 536–544 (2003)