Natural noise management in collaborative recommender systems over time-related information

Natural noise management in collaborative1 recommender systems over time-related information2 Francisco J. Baldána, Raciel Yeraa,∗, Luis Martíneza 3 aComputer Science Department, University of Jaén, Jaén, Spain4 Abstract5 Recommender systems are currently a suitable alternative for providing easy and appropriate access to information for users in today’s digital information-overloaded world. However, an important drawback of these systems is the inconsistent behavior of users in providing item preferences. To address this issue, several natural noise management (NNM) approaches have been proposed, which positively influence recommendation accuracy. However, a major limitation of such previous works is the disregarding of the time-related information coupled to the rating data in RSs. Based on this motivation, this paper proposes two novel methods, named SeqNNM and SeqNNM-p for NNM focused on an incremental, time-aware recommender system scenario that has not yet been considered, by performing a classification-based NNM over specific preference sequences, driven by their associated timestamps. Such methods have been evaluated by simulating a real-time scenario and using metrics such as Mean Absolute Error (MAE), Root-Mean-Square error (RMSE), Precision, Recall, NDCG, number of modified ratings, and running time. The obtained experimental results show that in the used settings, it is possible to achieve better recommendation accuracy with a low intrusion degree. Furthermore, the main innovation associated with the overall contribution is the screening of natural noise management approaches to be used on specific preferences subsets, and not ∗Corresponding author Email addresses: [email protected] (Francisco J. Baldán), [email protected] (Raciel Yera ), [email protected] (Luis Martínez) Preprint submitted to The Journal of Supercomputing June 5, 2024 Originally published in: Baldán, F. J., Yera, R., & Martínez, L. (2024). Natural noise management in collaborative recommender systems over time-related information. The Journal of Supercomputing, 80(16), 23628-23666. over the whole dataset as discussed by previous authors. These proposed approaches allow the use of natural noise management in large datasets, in which it would be very difficult to correct the entire data. Keywords: Time-aware noise correction,Natural noise management, 6 user inconsistencies,collaborative filtering,recommender systems7 1. Introduction8 Recommender systems (RS) currently represent an appropriate solution for9 the problem associated with efficient information access in today’s digital world10 (Adomavicius & Tuzhilin, 2005; Konstan & Riedl, 2012; Jannach et al., 2021;11 Yera et al., 2023). Specifically, collaborative filtering (CF) has been highlighted12 as a very appropriate paradigm to implement these systems because they are13 able to produce accurate recommendations requiring a minimum amount of14 information (Ekstrand et al., 2011).15 User-user collaborative filtering, as a pioneer recommendation approach, was16 initially introduced by (Resnick et al., 1994) as a way to predict the user’s17 preferences by using the information associated with similar peers associated18 with a community of people. Later,(Sarwar et al., 2001) extended the idea19 behind user similarity and defined item-item CF algorithms that are based on20 item-based similarities and outperform user-user methods in terms of accuracy21 and scalability (Deshpande & Karypis, 2004). (Koren et al., 2009) popularized a22 new paradigm for CF, proposing dimensionality reduction methods that notably23 improve the performance of traditional methods. In addition, a lot of notable24 research has been developed in order to do a deeper exploration of user ratings25 to enhance the recommendation process (Bobadilla et al., 2013; De Pessemier26 et al., 2014; Wei et al., 2017; Yera & Martínez, 2017; Yu et al., 2017; Wang &27 Mishra, 2017; Van Dat et al., 2022; Yannam et al., 2023).28 2 In this way, although the focus on recommender systems research began in29 30 the 90s (Resnick & Varian, 1997), it has been evolving across the time in a very 31 active and prolific research field in which currently there are several hot topics 32 such as the item recommendation from implicit feedback (Rendle, 2022), the 33 use of deep learning (Zhang et al., 2022), the context-aware recommendation 34 (Adomavicius et al., 2022), the group recommendation (Masthoff & Delić, 2022), 35 the cross-domain recommendation (Dacrema et al., 2022), and the explainability 36 (Tintarev & Masthoff, 2022), among other popular research efforts. On the other hand, the preprocessing process of inconsistent user37 preferences in RS has also become an emerging field of study, mainly focused38 on movie recommendations (Pham & Jung, 2013; Yera et al., 2015; Luo et al.,39 2023). Several authors have stated that user ratings in recommender systems40 are intrinsically inconsistent because of imperfect and even unintentional user41 behaviors while they express their preferences, limiting their performance with42 amagic barrier (Said et al., 2012).43 Extant research has provided different examples of the presence of natural44 noise in recommender systems:45 • (Amatriain et al., 2009) have suggested that preference values should not46 be regarded as ground-truth values because the rating gathering is a noisy47 process.48 • (Pham & Jung, 2013) pointed out two probable causes for the presence49 of natural noise in recommender systems datasets, which are: 1) the fact50 that user preferences change across time, and 2) the users are imprecise51 when they provide rating values.52 • (Said et al., 2012) and (Kluver et al., 2012) have indicated that users’53 imprecision can be caused by personal conditions, social influences,54 emotional states, or certain rating scales.55 3 • (Yera et al., 2015) have presented an illustrative example of natural noise,56 where a low rating is considered noisy if the corresponding user usually57 evaluates positively most of the items, and the associated item has been58 voted high by the majority of the items.59 • (Yera et al., 2019) also presents an illustrative example of natural noise,60 where the noise degree of a rating is characterized by the number and61 weights of the identified user’s behaviors/regularities that the rating62 contradicts or does not verify. In this way, it is identified as noisy, a63 rating that causes the user to not follow a pattern/regularity with high64 support.65 To overcome these issues, some works have been proposed in the last few66 years, focusing on detecting, removing, or correcting naturally noisy ratings67 by using their own rating information or information obtained from external68 sources (O’Mahony et al., 2006; Pham & Jung, 2013; Yera et al., 2015; Castro69 et al., 2017; Zhu et al., 2018; Luo et al., 2023). In addition, there have been70 studies centered on detecting whole noisy-but-non-malicious user profiles (Li71 et al., 2013).72 Previous research has focused on traditional evaluation setups that use73 ratings to create training and test sets and employ them to evaluate the74 accuracy of the corresponding method; in this scenario, these methods imply75 an improvement in the recommendation performance (Gunawardana & Shani,76 2009). However, the data associated with a real-time recommender system do77 not match these settings. In the real-world scenario, preferences are78 incrementally entered, and therefore, rating sorting begins to play a relevant79 role here (Alabduljabbar et al., 2023; Chen et al., 2021a; Tran & Huh, 2023).80 Furthermore, the system must simultaneously capture this temporal81 information and provide user rating predictions. Then, the addition of natural82 4 noise management into this new scenario requires a solution to several83 limitations that are connected with the application of the noise correction84 approach in RS datasets. Some of these limitations are: 1) to explore whether85 the application of natural noise management in some segment of recent data86 instead of the whole dataset would be effective in improving accuracy; 2) to87 explore the magnitude of the associated improvement degree; 3) to identify88 how this accuracy could vary across different lengths of the rating sequences;89 and 4) to evaluate other important criteria in natural noise management, such90 as the intrusion level and the running time, for performing a trade-off with the91 accuracy, and therefore suggest conclusions in order to use the new approaches92 in real scenarios. This paper focuses on supporting these issues by exploring93 approaches for performing natural noise management in such time-related94 recommendation scenarios.95 Specifically, the main novel contributions of this paper in relation to previous96 proposals and existing similar approaches are:97 98 99 100 101 102 103 104 105 106 107 108 • The screening of the natural noise management process, tailored to an incremental, time-aware recommendation scenario. • The development of a comparison protocol between the time-aware natural noise management and the traditional natural noise management approach without the time dimension. • An extensive evaluation of the time-aware natural noise management performance, using as new rating predictors in a natural noise management context, up to ten different state-of-the-art recommendation approaches. It includes 1) a clustering-based method (George & Merugu, 2005), 2) a basic neighborhood-based method (Resnick et al., 1994), 3) a neighborhoodbased method including average deviation (Resnick et al., 1994), 4) a neighborhood-based 109 5 110 111 112 113 114 115 116 method that includes biased-based baselines modeling (Resnick et al., 1994), 5) a method based on negative matrix factorization (Luo et al., 2013), 6) the Koren’s basic SVD approach (Koren et al., 2009), 7) the Koren’s SVD approach with temporal information (Koren, 2010), 8) the slope-one approach (Lemire & Maclachlan, 2005), 9) a baseline only approach predicting the biased-based baseline estimate for given user and item (Koren, 2010), and 10) a normal predictor based on the distribution of the training set, highlighted by (Hug, 2020). 117 • The overall evidence is that a natural noise management approach that118 incorporates time-related information and time windows is able to reduce119 the method’s intrusiveness, decrease the execution time, as well as lead to120 a similar or improved accuracy.121 The paper is organized as follows. Section 2 presents previous studies122 related to recommender systems and natural noise management in123 collaborative recommendations and justifies the selection of a specific124 approach to be used as the base for the following sections. Section 3 describes125 new approaches for performing the natural noise management task in an126 incremental, time-aware movie recommendation scenario. Section 4 plans and127 develops an experimental framework to evaluate the new NNM framework128 tailored to the time-related context. Finally, we present a final discussion129 about the obtained results. Section 5 concludes the paper.130 2. Preliminaries131 In this section, we present the necessary background for easy following and132 understanding of our proposal. It includes some antecedents for movie133 recommender systems, for previous works about the management of natural134 noise or unintentional inconsistencies that users can introduce in RS, as well as135 6 a detailed reference to a pioneer work on natural noise management that will136 be used as the basis for the current proposal.137 2.1. Antecedents on basic recommender systems138 The movie recommendation domain boosted initially the development of 139 140 modern recommender systems in the middle of the 90s (Resnick et al., 1994; 141 Adomavicius & Tuzhilin, 2005). Two main recommendation paradigms have 142 been modeled over the last 30 years for developing recommendation tools: •Content-based recommendation: The basis of content-based143 recommendation is the use of movie attributes for composing the user144 and item profiles, considering the relationship between item attributes145 and the rating values provided by the users (Adomavicius & Tuzhilin,146 2005). Then, several scoring approaches are employed to recommend the147 most appropriate movie profiles to each individual user. Some common148 movie attributes include genre, director, actors, country, year, or each149 movie’s associated tags (De Gemmis et al., 2015; Pérez-Almaguer et al.,150 2021; Yera et al., 2022b).151 In the last few years, several sophisticated approaches have been developed152 for building the aforementioned user and item profiles. It includes the use153 of advanced machine learning algorithms, as well as the use of semantic154 tools such as ontologies (Lops et al., 2019; Van Dat et al., 2022).155 •Collaborative filtering-based recommendation: In the case of156 collaborative filtering, the working principle relies on crowd preferences157 to suggest movies for the active user. Specifically, it is based on the158 discovery, in an explicit or implicit way, of users’ rating patterns that are159 similar to those associated with the current user and uses their160 associated information for the recommendation generation (Ning et al.,161 7 2015). Two large families of collaborative filtering approaches can be162 identified: 1) memory-based (Ning et al., 2015), focused on directly163 finding appropriate users’ neighborhoods for the current user and using164 the preferences of such nearest neighbors for recommendation165 generation, and 2) model-based (Koren et al., 2009), focused on building166 intermediate models that comprise the preferences of the user’s crowd167 and can facilitate recommendation generation.168 Collaborative filtering approaches have been very popular in movie169 recommendation because they can provide accurate recommendations170 using only ratings and without any additional information, in contrast to171 content-based approaches that depend on item attributes for an172 appropriate performance (Pilászy & Tikk, 2009).173 Across these traditional approaches, in the last few years, movie174 recommendations have continuously been a relevant research topic. In this175 way, (Deldjoo et al., 2019) model a new concept, titled Movie Genome, as a176 way of alleviating the new item cold start problem in movie recommendation177 and therefore improving the recommendation accuracy. (Kumar et al., 2020)178 introduces a movie recommender system using sentiment analysis from179 microblogging data, leveraging, in this way, the content-based recommendation180 paradigm. (Widiyaningtyas et al., 2021) explore advanced correlation-based181 similarities between the user profiles for introducing new algorithms for movie182 recommendation focused on outperforming previous proposals. In a different183 direction, (Chen et al., 2021b) exploits users’ positive and negative profiles and184 relies on preferences over movies to compose a novel movie recommendation185 method.186 Overall, while most of the available approaches in movie recommendation187 focus on improving recommendations through the proposal of more188 8 sophisticated recommendation algorithms (Goyani & Chaurasiya, 2020), the189 current paper follows an alternative research path. Specifically, it will focus on190 the improvement of recommendation accuracy supported by the management191 of natural noise (Yera et al., 2015; Martínez et al., 2016; Yera et al., 2019)192 associated with user preferences and the use of time-related information.193 The next subsection focuses briefly on the incorporation of time-related194 information into recommender systems.195 2.2. Antecedent of time-related information in recommender systems196 The value of time-related information in recommender systems was early197 pointed out by (Ding & Li, 2005), when they presented several weighting198 approaches to the basic memory-based collaborative filtering scenario, being199 used the time where the user’s opinion is provided as input for the weights’200 calculation.201 More recently, (Campos et al., 2014) presents a large-scale survey on the202 use of time-aware recommender systems, illustrating a taxonomy for203 classifying the developed works on the use of time as a contextual dimension204 in this scenario. They cover three main work categories: 1) continuous205 time-aware heuristic approaches, 2) categorical time-aware heuristic206 approaches, and 3) time-adaptive models.207 (Vinagre et al., 2015) have also developed a survey in a similar direction, 208 209 identifying several groups of research trends related to different challenges in 210 the field, such as the following: 211 212 213 214 •Time-aware algorithms focused on modeling time as context. It includes time-aware factorization and time-aware neighborhood models. In the context-aware framework, time features (day, day of the week, working/nonworking hours) were used in the prefiltering, postfiltering, and modeling stages. The main limitation of this approach is related to 215 9 in the rating data based on reconstruction errors after training. By removing376 the detected natural noise, the collaborative filtering approach can predict the377 unrated ratings using noise-free data.378 Table 1 summarizes the described methods in terms of four main features:379 380 381 382 383 384 385 386 387 388 •Avoid loss of information: It refers to the fact of avoiding the removal of user preferences across the natural noise management approach. •Does not use additional information: It refers to the performance of the noise management without the dependency on information beyond the user preference values. Examples of this additional information could be item attributes or tags. •Considers a time-related context: It refers to the use of rating timestamps or similar time-related variables in the developed models. •Tailored to a group scenario: It refers to natural noise management models specifically conceived or evaluated in group recommendation scenarios. 389 Works like (O’Mahony et al., 2006) and (Li et al., 2013), even though they390 do not depend on additional information beyond the rating values, remove391 important information from the dataset. Other research like the developed by392 (Pham & Jung, 2013), (Amatriain et al., 2009), (Bellogín et al., 2014), (Yu393 et al., 2016), and (Saia et al., 2016), although focusing on rating correction394 and therefore does not imply information loss, also depend on additional395 information beyond the rating matrix and therefore could be difficult to apply396 in some scenarios. Eventually, (Yera et al., 2019) introduces a regularity-based397 correction approach that does not depend on additional information but398 requires the discovery of intermediate knowledge in terms of association rules,399 which could be difficult to generalize in some scenarios.400 16 Approach Avoid loss of information Does not use additional information Considers a time-related context Tailored to a group scenario (O’Mahony et al., 2006) - -- (Amatriain et al., 2009) --- (Pham & Jung, 2013) --- (Li et al., 2013) - -- (Bellogín et al., 2014) --- (Yera et al., 2015)  -- (Yu et al., 2016) --- (Saia et al., 2016) --- (Castro et al., 2017)  - (Zhu et al., 2018)  -- (Bag et al., 2019)  -- (Yera et al., 2019)  -- (Wang et al., 2021)  -- (Luo et al., 2023)  -- (Park et al., 2023) - -- This work - Table 1: Comparative analysis of existing approaches. In contrast to the abovementioned works, the previous classification-based401 approach developed by (Yera et al., 2015) and (Castro et al., 2017), also402 featured recently by (Bag et al., 2019) and (Luo et al., 2023), correct ratings,403 does not remove important information from the dataset, and does not depend404 on additional information such as item attributes. In this way, while most of405 the considered approaches are centered on individual recommendation, (Castro406 et al., 2017) have introduced natural noise management in group recommender407 systems.408 Therefore, considering the advantages of the previous classification-based409 approach for natural noise management (Yera et al., 2015), as well as its410 increasing popularity according to recent works that have continued this411 approach (Bag et al., 2019; Luo et al., 2023), see Table 1, the rest of the paper412 will take the pioneering previous classification-based approach (Yera et al.,413 2015), as the base for the current proposal. As presented in Table 1, the414 17 current proposal provides a novel feature in managing a time-related context,415 which contrasts previous approaches that do not consider it. We leave to416 future work the tailoring to a group recommendation scenario.417 2.4. The classification-based approach for natural noise management in RS418 The classification-based approach for natural noise management in RS419 (Figure 1) was proposed as a way to perform this task without using420 additional information beyond the user ratings (Yera et al., 2015).421 This approach comprises two main stages: 1) the detection of possible noisy422 423 ratings, and 2) the correction of noisy ratings. The first stage performs a classification of users and items based on a 424 425 direct inspection of their ratings, to identify tendencies to have low, medium, 426 or high preferences. Overall, the ratings that do not match those 427 well-identified tendencies are considered as possible noisy. This is the 428 underlying technique behind this stage. Specifically, each user, item, and rating are respectively classified into three429 possible classes, which are presented in Table 2. Specifically, it focuses on430 classifying users in the classes benevolent, average, critical, or variable; and431 items in the classes strongly-preferred, averagely-preferred, weakly-preferred,432 or variably-preferred. Variable and variably-preferred classes are used433 respectively for users and items that can not be classified into specific classes.434 Besides, ratings can be classified as weak, average, or strong, depending on two435 thresholds. Algorithm A.1 (included in Appendix A) shows the pseudocode of436 this process also included in our new proposals. Moreover, the proposal437 considers three groups that establish matching among user, item, and rating438 classes. The proposed method assumes that for a certain rating, if its user and439 item classes belong to the same group (different from the variable class), then440 the rating should belong to the corresponding rating class in the same group.441 18 Otherwise, the rating should be classified as a possible inconsistency.442 Classify all users, items, and ratings For each rating. Its current user and item classes are homologous? Ratings class is homologous with user (or item) class? Correct the current rating Rating dataset No Yes No Yes Figure 1: Global scheme of the previous classification-based approach for natural noise management. Table 2: Group of homologous classes User class Item class Rating class Group 1 Critical Weakly-preferred Weak Group 2 Average Averagely-preferred Average Group 3 Benevolent Strongly-preferred Strong Table 3 presents criteria for classifying users and items using such rating443 classification. In the case of users, it assumes that for each user uthe sets444 |Wu|,|Au|and |Su|are the respective sets of weak, average, and strong ratings.445 Regarding the proportion of ratings in each class, the user is classified as critical,446 benevolent, or average, and those users who have a similar proportion of the447 three kinds of ratings are classified as variable users. In the case of items, it448 follows a very similar approach in relation to users but considers all the ratings449 associated with the item (see also Table 3). Here, sets |Wi|,|Ai|, and |Si|450 are used as the respective weakly-preferred, averagely-preferred, and strongly-451 preferred ratings for item i.452 19 Table 3: Classes definition User classes Definition Critical |Wu|>|Au|+|Su| Average |Au|>|Wu|+|Su| Benevolent |Su|>|Wu|+|Au| Item classes Definition Weakly-preferred |Wi|>|Ai|+|Si| Averagely-preferred |Ai|>|Wi|+|Si| Strongly-preferred |Si|>|Wi|+|Ai| The second stage of the proposal is focused on correcting the ratings 453 454 identified as possible inconsistencies, obtained in the previous stage. 455 Specifically, a new rating value is predicted for each user-item pair associated 4 56 with the possible noisy rating previously detected. This stage then uses an 457 underlying rating prediction algorithm, which is the well-known Resnick’s 458 user-based method with Pearson’s similarity (UserKNNPearson) (Resnick 459 et al., 1994), as the former collaborative filtering approach. In each case, if the 460 original rating is sufficiently different from the predicted value, the old rating 461 is replaced with the new one. In the proposal, the difference threshold was set 462 to δ = 1, as this value tends to be the minimum step between two ratings in 463 recommendation scenarios. Algorithm A.2 (included in Appendix A) presents 464 this procedure, which is included in our new proposals. As presented in this section, several authors have pointed out that user465 preferences evolve over time and that taking this issue into account leads to466 performance improvement in RS models (Vinagre et al., 2015; Huang et al., 2019;467 Chen et al., 2022). It is then necessary to explore how the use of time-related468 information affects the behavior of this natural noise management model, which469 has already been justified. Therefore, two new proposals for performing natural470 noise management in an incremental, time-related recommendation scenario are471 presented in the next section.472 20 3. Correcting noisy ratings in a time-aware recommendation scenario473 The recommendation tasks are intrinsically incremental, taking into account474 that the ratings stored behind a CF recommender system are provided by users475 who simultaneously request suggestions from the system itself. However, as476 presented in the Introduction section, the use of natural noise management477 approaches to this incremental scenario brings new issues that have not yet been478 regarded, and to the best of our knowledge, no previous studies have focused479 on solving this task. Typical natural noise management methods receive as480 input a set of ratings and optionally additional information about them and481 return as output the corrected set. Under these circumstances, its deployment482 in an incremental, time-aware scenario faces troubles like the selection of the483 ratings set to be corrected across time and the selection of the data that must484 be considered for the correction process. Taking into account the relevancy of485 the time dimension and sequential recommendation context as research trends,486 it is necessary to tailor formerly developed natural noise management models to487 these new requirements and scenarios. Therefore, the goal of the current study is488 to screen new models for the natural noise management process contextualized489 to an incremental, time-related recommendation scenario.490 These models use as underlying algorithms the approach for identifying 491 492 possibly noisy ratings, and the approach for correcting noisy ratings. Both 493 algorithms, formerly proposed by Yera et al. (2015), have been discussed in 494 Section 2.4 and detailed in Algorithms A.1 and A.2 Furthermore, this work developed a comprehensive experimental procedure495 over several recommendation approaches, with a higher, more general496 magnitude in relation to the previously referred works on natural noise497 management. Specifically, the current research work will then screen two498 frameworks for natural noise management in recommender systems, where it is499 21 assumed a sequential gathering of the rating data, which is the real context of500 a deployed recommender system. The next sections describe these approaches.501 3.1. Sequential natural noise management in collaborative filtering502 In the first stage, we propose a framework, named SeqNNM, that considers503 the continuous gathering of sequential rating data by RS. Figure 2 illustrates this504 framework. Herein, it is assumed that a set of rating sequences s1,s2,...,sk,...,sn 505 is continuously gathered by the system. Each newly gathered skis first added506 to the main RS dataset R. Then, the R+skdataset is corrected through the507 mentioned natural noise management approach. From the identification of noisy508 ratings, following Algorithm A.1, and the subsequent prediction of corrected509 ratings, following the guidelines in Algorithm A.2, a processed dataset is finally510 obtained with the noise corrected based on the available data up to that moment.511 The sequential processing of data in specific time steps is the main innovation512 of this proposal. After that, the data reached as output by the NNM approach513 started to be used as the main data of the recommender system for both the514 main recommendation generation process and for the subsequent runs of the515 NNM process. This procedure processes multiple times all the available data,516 so it is able to correct a large amount of noise through further intrusion into517 the original data. Algorithm 1 presents an overview of this framework.518 3.2. Sequential natural noise management in collaborative filtering covering the519 last p rating sequences520 The framework for sequential natural noise management presented in the521 previous subsection has a shortcoming of the high volume of data that is used522 for natural noise management across the processing of each new sequence, which523 could affect the time performance of the proposal. To alleviate this drawback,524 we propose an alternative approach, named SeqNNM-p, in which instead of525 22 Set of rating sequences s1 s2 sk sn Next sequence sk Addition of the sequence to the RS dataset Natural noise correction approach Replace the set of fixed ratings with the updated one skR Add Set of already corrected ratings Add Updated set of corrected ratings R + skR' R'R Figure 2: Overview for the approach for natural noise correction in a sequential scenario. correcting all data every time that a new rating sequence is processed, it would526 be corrected only the last psequences of the most recent ratings in the dataset.527 This approach significantly limits the data to be processed in each iteration,528 considerably reducing the final running time and the intrusiveness of the original529 proposal since the number of instances identified as noise is reduced with a530 shorter time horizon.531 Figure 3 illustrates this approach. Here, once the new rating sequence skis532 gathered, a temporal dataset Tcontaining such a sequence, as well as previous533 ones. The natural noise management used as the starting point for these models534 (Section 2.4) is applied over this temporal dataset T, and at the last stage, the535 values of the modified ratings in Tare updated in the original dataset Rused536 for recommendation generation. Algorithm 2 screens this approach.537 Overall, the computational cost of both approaches presented in this538 section depends on two main factors: 1) The cost of the classification-based539 approach for natural noise management, which is used at the initial step in540 both approaches, and 2) The cost of the inner approaches for rating541 23 Algorithm 1: Pseudocode for the incremental time-aware natural noise management proposal. seq method. Input : Rdata the sorted set of ratings to be checked for natural noise detection and correction slist a list of rating sequences, s1,s2,s3,...,sk,...,sn∈Rdata, continuously gathered by the recommender system th1 first classification thresholds th2 second classification thresholds δdifference threshold Output: Rcorrected list of corrected ratings SequentialNNCorrection(Rdata,slist) 1Rcorrected ={} 2foreach new skdo 3Rtemp=Rcorrected+sk 4Rnoise=identifyNoiseRatings(Rtemp,th1, th2) 5foreach rui in Rnoise do 6Rcorrected=corrrectNoisyRating(rui,Rtemp,δ) 7end foreach 8end foreach 9return Rcorrected 542 prediction. In the first case, considering that full inspection of the rating 543 matrix is necessary, the theoretical cost would be O(|U|∗|I|), where U and I 544 are the sets of users and items. However, due to the sparsity of RS datasets, 545 this matrix can be quickly inspected. In the second case, the complexity of the 546 different rating prediction methods varies from methods with constant time to 547 methods with higher complexity. Moreover, in the experimental section, it will 548 be proved that, in practice, the approach is able to correct several ratings in a 549 short period. In addition, it will be proved how the considered length of the 550 sequence can manage such execution time while maintaining positive values in 551 terms of accuracy for almost all the evaluated settings. 552 4. Experiments and results 553 This section executes an evaluation process to measure the impact of the 554 proposed alternatives to natural noise management in an incremental 555 time-aware recommendation scenario. We assume two main criteria for the 24 Set of rating sequences s1 s2 sk sn Next sequence sk Building of a temporal dataset for correction composed of the last p sequences Natural noise correction approach Add or replace the corresponding ratings in T' in the RS dataset R sk, sk-1, ..., sk-p+1 T Build Temporal set of ratings for correction Updated set of corrected ratings T'R Add T'T Figure 3: Overview on an improved approach for natural noise correction in a sequential scenario, considering the correction of the last ksequences. performance evaluation: recommendation accuracy after the execution of the556 correction method in the data and the amount of rating modified by the557 correction process. With this aim, we initially discuss the experimental setup558 and then present and analyze the obtained experimental findings.559 4.1. Evaluation protocol560 In this study, we evaluated how our natural noise preprocessing approach561 increases data quality, thereby affecting recommendation accuracy. Therefore,562 we will compare the results provided by our sequential approach versus two563 different cases: the case of not applying natural noise methods and the case of564 applying the natural noise method identified as baseline (Yera et al., 2015), but565 without considering the sequential nature. After applying the selected natural566 noise method, the recommendation results were evaluated using a five-fold cross-567 validation approach.568 It is important to highlight that the same rating prediction model is used569 for both the preprocessing natural noise step and the final recommendation.570 25 SVD seqk9 0.81917 0.6264 0.92756 0.70506 0.5509 0.61849 13636 29.1 SVD seqk11 0.80833 0.61691 0.92952 0.70614 0.55291 0.62019 14538 32.23 SVD++ no 0.91996 0.72135 0.92007 0.70957 0.55054 0.61999 0 161.33 SVD++ nn 0.76528 0.59833 0.94018 0.73672 0.57562 0.64626 9516 278.67 SVD++ seq 0.69185 0.53127 0.94812 0.73353 0.57863 0.64692 16526 1626.78 SVD++ seqk7 0.8214 0.6287 0.92771 0.70928 0.54968 0.61935 12639 209.55 SVD++ seqk9 0.80695 0.6157 0.93113 0.71169 0.55375 0.62283 13636 225.11 SVD++ seqk11 0.79739 0.60744 0.93202 0.71164 0.55571 0.62406 14538 231.04 SlopeOne no 0.94587 0.74335 0.91307 0.69392 0.5373 0.60563 0 17.9 SlopeOne nn 0.75652 0.58208 0.93877 0.73282 0.56969 0.64102 12353 100.62 SlopeOne seq 0.72831 0.5593 0.94202 0.73311 0.5726 0.64297 14467 945.27 SlopeOne seqk7 0.83611 0.63703 0.92545 0.70573 0.54892 0.61751 12639 36.77 SlopeOne seqk9 0.82315 0.62512 0.92824 0.70779 0.55154 0.61994 13636 40.56 SlopeOne seqk11 0.81254 0.61553 0.93024 0.70708 0.54976 0.61855 14538 44.05 In order to facilitate the comparison between the nn model and the seq652 model, Table 7 shows the percentage improvement obtained in each metric. In653 this case, a considerable improvement can be seen in all performance metrics in654 practically all cases. In the case of running time and the number of modified655 values, we see how seq is more time-consuming and intrusive. Both metrics656 show the cost of obtaining better results.657 Table 7: Percentage of improvement (%) of the seq proposal vs. the state-of-the-art nn proposal for MovieLens 100k dataset. Models RMSE (%) MAE (%) NDCG (%) Precision (%) Recall (%) F1 (%) Modified Ratings (%) Running Time (secs) (%) BaselineOnly 6.00446 6.92800 0.52926 0.02963 1.20722 0.68241 -29.42108 -893.61290 CoClustering 5.98928 7.64281 0.54906 -0.46047 0.66192 0.17511 -42.94195 -870.83685 KNNBaseline 6.87498 7.73906 0.77178 1.53723 2.65528 2.15870 -45.73371 -781.88837 KNNBasic 6.26627 6.57595 0.80584 1.08670 2.91848 2.09481 -47.00319 -818.95722 KNNWithMeans 6.77211 7.85861 0.64555 1.09681 1.93109 1.57057 -43.61111 -800.76381 NMF 9.46845 11.30371 1.25595 0.40551 1.09444 0.79709 -63.05034 -860.71504 NormalPredictor 15.66205 16.83099 4.08955 6.08314 3.21999 4.40989 -57.95576 -1007.35500 SVD 11.09867 12.78649 1.06445 -0.12754 0.88387 0.43942 -101.71337 -952.24599 SVD++ 9.59518 11.20786 0.84452 -0.43300 0.52291 0.10213 -73.66541 -483.75989 SlopeOne 3.72892 3.91355 0.34620 0.03957 0.51080 0.30420 -17.11325 -839.40943 To analyze the results more clearly, some graphical comparisons have been658 included.659 32 Figure 4: RMSE results for the multiple models and natural noise approaches selected in the MovieLens 100k dataset. Figure 4 includes the RMSE results for every model tested and all the660 natural noise approaches included in this work. The comparison shows how661 the application of any natural noise correction technique improves the final662 results obtained using all the methods. Our first proposal, sequential and663 cumulative natural noise correction (seq), provides the best results for all664 models, with significant improvements over the traditional approach (nn). Our665 second proposal (seqk) offers results that are progressively closer to those666 obtained by the traditional method (nn) as the value of kincreases. This667 behavior is relevant in massive data or Big Data environments considering that668 seqk works with a reduced subset of data while nn needs all available data.669 Moreover, in the case of SVD, seqk11 is able to provide better results than nn,670 so in these environments, the seqk approach becomes a desirable alternative.671 The NDCG metric (Table 7) shows a similar behavior to that observed for672 33 RMSE. The seq approach obtains the best results, nn obtains the second673 place, followed by the different seqk approaches, the higher the k, the better674 the performance. This metric shows reduced differences between seqk and nn,675 and it is possible to improve the results of the traditional approach with slight676 increases in the kparameter. It is important to note the reduction of resources677 associated with the seqk approach, both in time and memory, by processing a678 subset of the original data at each step.679 Figure 5: Running time (secs) results, in logarithmic scale, for the multiple models and natural noise approaches selected in the MovieLens 100k dataset. Figure 5 shows the running time obtained from each approach. The680 difference between the seq approach with respect to the rest is quite clear.681 This approach requires the longest running time. In the second place, we682 found the traditional approach (nn), while our second proposal (seqk) offers a683 considerable reduction in the running time concerning nn. Since the cost in684 the result performance is reduced, the seqk approach provides a robust685 34 alternative in environments where time is a constraint to be considered.686 Figure 6: Number of modified ratings produced for each model and natural noise approach evaluated combination in the MovieLens 100k dataset. Finally, Figure 6 shows the number of modified values for each model and687 approach. The seq approach is the most intrusive among the models, except688 for the SlopeOne and KNN-based models. The seqk approach is more intrusive689 as the value of kincreases. This behavior shows that a higher data availability690 leads to higher natural noise detection and, therefore, higher intrusiveness. Re-691 evaluation of the data when new data are sequentially added also leads to greater692 intrusiveness, as in the seq case.693 4.4.2. MovieLens last 1M of 25M rating dataset694 In this section, experimentation close to a real use case in a data-intensive695 environment is performed, allowing us to evaluate the performance and696 scalability of the proposals.697 The results shown in Table 8 show a clear dominance of the SVD++ model698 35 with the proposed seq approach, obtaining the best results in terms of RMSE,699 MAE, and NDCG metrics. BaselineOnly, also with the seq approach, obtains700 the best Precision and F1-Score results. Finally, the KNNBasic model, with the701 seq approach, obtains the best results in the Recall metric. Although the seq702 approach is more data-intrusive than traditional approaches, the differences in703 performance results are very significant, as can be seen in Figure 7.704 Table 8: Results obtained for MovieLens 1M dataset: no natural noise method (no), baseline natural noise proposal (nn), the sequential proposal (seq), and the sequential method considering the last k rating sequences (seqk). The best cases for each method are highlighted in blue. The best results are stressed in bold. Models Natural Noise Approach RMSE MAE NDCG Precision Recall F1 Modified Ratings Running Time (secs) BaselineOnly no 0.9045 0.71412 0.92298 0.71811 0.56994 0.63551 0 81.92 BaselineOnly nn 0.7166 0.55467 0.94705 0.75102 0.6053 0.67033 128249 8719.99 BaselineOnly seq 0.67968 0.52162 0.95098 0.75287 0.61615 0.67768 165506 217044.8 BaselineOnly seqk7 0.80165 0.6189 0.93435 0.72684 0.58262 0.64679 119606 413.4 BaselineOnly seqk9 0.79905 0.61663 0.93488 0.72732 0.58339 0.64745 121959 472.42 BaselineOnly seqk11 0.79576 0.61395 0.93543 0.72697 0.5834 0.64732 124459 558.41 CoClustering no 0.91263 0.71281 0.92464 0.7037 0.55873 0.62289 0 154.11 CoClustering nn 0.74229 0.57018 0.94482 0.73056 0.58374 0.64895 112804 7666.49 CoClustering seq 0.70363 0.53261 0.94903 0.72924 0.58704 0.65046 159584 216431.68 CoClustering seqk7 0.81639 0.62191 0.93382 0.71153 0.56614 0.63056 119606 512.84 CoClustering seqk9 0.81084 0.61768 0.93454 0.71241 0.56971 0.63312 121959 564.73 CoClustering seqk11 0.80871 0.61482 0.93519 0.71262 0.56884 0.63266 124459 698.63 KNNBaseline no 0.89624 0.70529 0.92375 0.71945 0.57261 0.63769 0 616.48 KNNBaseline nn 0.80037 0.63473 0.93192 0.72558 0.58042 0.64493 55729 8346.76 KNNBaseline seq 0.75403 0.59515 0.9372 0.72954 0.58999 0.65239 89148 220434.11 KNNBaseline seqk7 0.79971 0.61733 0.9335 0.72539 0.58096 0.64518 119606 936.94 KNNBaseline seqk9 0.79711 0.61523 0.9339 0.72647 0.582 0.64626 121959 987.28 KNNBaseline seqk11 0.79384 0.6126 0.93454 0.72578 0.58225 0.64614 124459 1067.88 KNNBasic no 0.93529 0.73738 0.92439 0.71818 0.5988 0.65308 0 747.07 KNNBasic nn 0.83961 0.66558 0.93129 0.72443 0.60933 0.66191 57210 9396.34 KNNBasic seq 0.79088 0.62373 0.93677 0.72915 0.61907 0.66962 97594 223299.58 KNNBasic seqk7 0.84285 0.64746 0.93391 0.72832 0.61471 0.6667 119606 1078.19 KNNBasic seqk9 0.84034 0.6453 0.93443 0.72934 0.61612 0.66797 121959 1167.65 36 KNNBasic seqk11 0.8372 0.64272 0.93504 0.72881 0.61685 0.66817 124459 1285.16 KNNWithMeans no 0.92434 0.73021 0.92135 0.68389 0.53334 0.59931 0 722.12 KNNWithMeans nn 0.82265 0.65337 0.9305 0.68713 0.53587 0.60214 64782 8877.83 KNNWithMeans seq 0.77313 0.60956 0.93567 0.68885 0.54257 0.60702 100863 220632.42 KNNWithMeans seqk7 0.82257 0.63621 0.93127 0.69373 0.54538 0.61067 119606 899.73 KNNWithMeans seqk9 0.81985 0.63386 0.93176 0.69437 0.54612 0.61138 121959 955.05 KNNWithMeans seqk11 0.81645 0.63102 0.93243 0.6951 0.54673 0.61205 124459 1037.78 NMF no 0.91588 0.7201 0.9193 0.69677 0.53433 0.60483 0 131.88 NMF nn 0.74417 0.57726 0.94004 0.71794 0.55391 0.62535 114743 7632.58 NMF seq 0.68188 0.5179 0.94891 0.71766 0.55887 0.62839 184003 215615.8 NMF seqk7 0.81572 0.62808 0.92993 0.70578 0.53999 0.61185 119606 481.98 NMF seqk9 0.81289 0.62539 0.93041 0.70684 0.54209 0.61359 121959 536.5 NMF seqk11 0.80971 0.62283 0.93099 0.70592 0.54181 0.61307 124459 613.05 NormalPredictor no 1.51028 1.21401 0.85059 0.55968 0.40268 0.46837 0 67.79 NormalPredictor nn 1.35526 1.08332 0.86377 0.56656 0.40661 0.47344 141563 8376.64 NormalPredictor seq 1.15084 0.90733 0.89786 0.60137 0.42608 0.49877 225176 219088.62 NormalPredictor seqk7 1.42381 1.14078 0.85739 0.5567 0.40052 0.46587 83481 443.44 NormalPredictor seqk9 1.42394 1.1409 0.85719 0.55744 0.40201 0.46713 84887 516.02 NormalPredictor seqk11 1.42287 1.13982 0.85737 0.55785 0.40191 0.46721 87066 639.33 SVD no 0.88207 0.69149 0.92781 0.72318 0.56249 0.6328 0 111.93 SVD nn 0.76815 0.60688 0.94119 0.73858 0.57867 0.64892 67717 7982.44 SVD seq 0.67975 0.52599 0.95112 0.73873 0.58529 0.65312 152278 215875.73 SVD seqk7 0.79717 0.6131 0.93402 0.72525 0.56777 0.63692 119606 443.4 SVD seqk9 0.79468 0.61109 0.93439 0.72634 0.56894 0.63808 121959 520.05 SVD seqk11 0.79082 0.60806 0.93486 0.7249 0.56773 0.63676 124459 568.96 SVD++ no 0.86704 0.67483 0.93271 0.73233 0.57421 0.6437 0 3048.93 SVD++ nn 0.72654 0.5658 0.94897 0.7514 0.59207 0.66228 87500 11406.44 SVD++ seq 0.66065 0.50447 0.95588 0.74983 0.59762 0.66513 157357 250978.33 SVD++ seqk7 0.78568 0.60084 0.93794 0.73112 0.57334 0.64269 119606 3904.18 SVD++ seqk9 0.78296 0.59857 0.93867 0.73246 0.57374 0.64345 121959 3958.81 SVD++ seqk11 0.78011 0.5961 0.93889 0.73168 0.57429 0.6435 124459 4038.17 SlopeOne no 0.90424 0.70993 0.92328 0.7088 0.55153 0.62035 0 285.17 SlopeOne nn 0.72927 0.56432 0.94552 0.73887 0.57676 0.64783 114407 7821.6 SlopeOne seq 0.70597 0.5449 0.94793 0.73975 0.58208 0.65151 134183 217920.66 SlopeOne seqk7 0.80414 0.61754 0.93414 0.71629 0.55896 0.62792 119606 631.97 SlopeOne seqk9 0.80164 0.61529 0.93465 0.71662 0.55938 0.62831 121959 681.4 SlopeOne seqk11 0.79827 0.61249 0.93518 0.71641 0.55967 0.62841 124459 760.12 37 Figure 7: RMSE results for the multiple models and natural noise approaches selected in the MovieLens last 1M of 25M ratings dataset. Additionally, Table 8 shows an increase in the running time for the seq705 approach, while the seqk approach shows a significant reduction in the time706 cost. Because each time window is seven days, the seq approach can be applied707 in a real-world application without a problem. In the case of time constraints,708 such as to prevent the use of the seq or nn approaches, the seqk approaches offer709 competitive results (Figure 7), with respect to the traditional approach (nn).710 Moreover, this approach allows us to improve its performance in terms of results711 by adapting the time horizon kto the time constraints of each problem.712 As in the previous case, the comparison between the nn model and the713 seq model, by percentage of improvement, is included in Table 9. The results714 show that a significant enhancement is evident across nearly all performance715 metrics. When examining factors such as running time and the number of716 modified values, it becomes clear that the seq approach is more time-consuming717 38 and invasive. Both metrics highlight the trade-off involved in achieving improved718 results.719 Table 9: Percentage of improvement (%) of the seq proposal vs. the state-of-the-art nn proposal for MovieLens 1M dataset. Models RMSE (%) MAE (%) NDCG (%) Precision (%) Recall (%) F1 (%) Modified Ratings (%) Running Time (secs) (%) BaselineOnly 5.15211 5.95850 0.41497 0.24633 1.79250 1.09647 -29.05052 -2389.04875 CoClustering 5.20821 6.58915 0.44559 -0.18068 0.56532 0.23268 -41.47016 -2723.08614 KNNBaseline 5.78982 6.23572 0.56657 0.54577 1.64881 1.15671 -59.96698 -2540.95459 KNNBasic 5.80389 6.28775 0.58843 0.65155 1.59848 1.16481 -70.58906 -2276.45257 KNNWithMeans 6.01957 6.70524 0.55562 0.25032 1.25030 0.81044 -55.69603 -2385.20671 NMF 8.37040 10.28306 0.94358 -0.03900 0.89545 0.48613 -60.36098 -2724.94092 NormalPredictor 15.08345 16.24543 3.94665 6.14410 4.78837 5.35020 -59.06416 -2515.47151 SVD 11.50817 13.32883 1.05505 0.02031 1.14400 0.64723 -124.87411 -2604.38175 SVD++ 9.06901 10.83952 0.72816 -0.20894 0.93739 0.43033 -79.83657 -2100.32074 SlopeOne 3.19498 3.44131 0.25489 0.11910 0.92239 0.56805 -17.28566 -2686.13991 Analyzing the results obtained by all models and approaches in both720 datasets, we can see that, except for the KNNBaseline model, the rest of the721 models obtain, in most cases, an improvement in the performance of the722 results. If we focus on the seqk approaches, we can appreciate that the723 performance differences for different values of kare more significant in the case724 of MovieLens100k than in MovieLens1M (Figures 4 and 7, respectively).725 These results show the importance of the temporal component in both726 scenarios, but it is more significant in small datasets. In addition, the727 importance of the parameter kcan be appreciated, and it is advisable to adapt728 it to the type of problem addressed.729 The results obtained for the seq approach, which are the best results in730 the vast majority of cases, require a large amount of running time (Figure 8).731 Because, in a real case, the accumulation of data takes weeks or even months,732 for the amount of data we are working on, the running time does not limit the733 application of our proposal in real scenarios. In the extreme case of working734 with large amounts of data and very tight model running time windows, we735 39 always have the option of using the seqk approach. This approach allows us736 to adjust the performance and running time using the parameter k, which is737 especially useful for this type of problem.738 Figure 8: Running time (secs) results, in logarithmic scale, for the multiple models and natural noise approaches selected in the MovieLens last 1M of 25M ratings dataset. Finally, in Figure 9, we analyze the intrusivity levels of our MovieLens1M739 dataset and compare it with those obtained for the MovieLens100k dataset,740 Figure 6, we can appreciate a significant reduction of the relative intrusivity741 in each dataset. This behavior can be observed numerically by comparing the742 results included in Tables 6 and 8. This is especially relevant in the case of743 the KNN-based models, where the seq approach is the second least intrusive,744 marking a significant difference from the trend shown by the rest of the cases745 in both datasets.746 40 Figure 9: Number of modified ratings produced for each model and natural noise approach evaluated combination in the MovieLens last 1M of 25M ratings dataset. 4.5. Discussion747 The results obtained in the previous section show a considerable748 improvement in the data quality after the application of the two natural noise749 correction techniques proposed in this study.750 The first approach proposed, seq, obtains considerable results improvement751 by adding and accumulating information sequentially (Tables 6 and 8).752 Although the intrusiveness is not high according to Figures 6 and 9, the753 computation required in high-dimensional problems may limit its use.754 The second proposed approach, seqk, focused on the use of data related to755 the last kweeks, is able to provide competitive results in a short time at the756 cost of higher intrusiveness, considering the traditional natural noise approach757 (Figures 6 and 9) and a correct setting of the parameter k. This proposal758 uses a smaller amount of data, which allows its use in real problems with large759 dimensions and running time limitations (Figures 5 and 8).760 41 filtering recommendation system with dynamic time decay. The Journal of912 Supercomputing,77 , 244–262.913 Chen, Y.-L., Yeh, Y.-H., & Ma, M.-R. (2021b). A movie recommendation914 method based on users’ positive and negative profiles. Information Processing915 & Management,58, 102531.916 Dacrema, M. F., Cantador, I., Fernández-Tobías, I., Berkovsky, S., &917 Cremonesi, P. (2022). Design and evaluation of cross-domain recommender918 systems. In Recommender Systems Handbook (pp. 485–516). Springer.919 De Gemmis, M., Lops, P., Musto, C., Narducci, F., & Semeraro, G. (2015).920 Semantics-aware content-based recommender systems. Recommender systems921 handbook, (pp. 119–159).922 De Pessemier, T., Dooms, S., & Martens, L. (2014). Comparison of group923 recommendation algorithms. Multimedia Tools and Applications,72, 2497–924 2541.925 Deldjoo, Y., Dacrema, M. F., Constantin, M. G., Eghbal-Zadeh, H., Cereda, S.,926 Schedl, M., Ionescu, B., & Cremonesi, P. (2019). Movie genome: alleviating927 new item cold start in movie recommendation. User Modeling and User-928 Adapted Interaction,29, 291–343.929 Deshpande, M., & Karypis, G. (2004). Item-based top-n recommendation930 algorithms. ACM Transactions on Information Systems,22, 143–177.931 Ding, Y., & Li, X. (2005). Time weight collaborative filtering. In Proceedings932 of the 14th ACM international conference on Information and knowledge933 management (pp. 485–492). ACM.934 Ekstrand, M. D., Riedl, J. T., & Konstan, J. A. (2011). Collaborative935 48 filtering recommender systems. Foundations and Trends in Human-Computer936 Interaction,4, 81–173.937 George, T., & Merugu, S. (2005). A scalable collaborative filtering framework938 based on co-clustering. In Fifth IEEE International Conference on Data939 Mining (ICDM’05) (pp. 4–pp). IEEE.940 Goyani, M., & Chaurasiya, N. (2020). A review of movie recommendation941 system: Limitations, survey and challenges. ELCVIA: electronic letters on942 computer vision and image analysis,19, 18–37.943 Gunawardana, A., & Shani, G. (2009). A Survey of Accuracy Evaluation Metrics944 of Recommendation Tasks. Journal of Machine Learning Research,10 , 2935–945 2962.946 Gunes, I., Kaleli, C., Bilge, A., & Polat, H. (2014). Shilling attacks against947 recommender systems: a comprehensive survey. Artificial Intelligence Review,948 42, 767–799.949 Harper, F. M., & Konstan, J. A. (2015). The movielens datasets: History and950 context. ACM Transactions on Interactive Intelligent Systems (TIIS),5,1–951 19.952 Huang, X., Fang, Q., Qian, S., Sang, J., Li, Y., & Xu, C. (2019).953 Explainable interaction-driven user modeling over knowledge graph for954 sequential recommendation. In Proceedings of the 27th ACM international955 conference on multimedia (pp. 548–556).956 Hug, N. (2020). Surprise: A python library for recommender systems. Journal957 of Open Source Software,5, 2174.958 Jannach, D., Manzoor, A., Cai, W., & Chen, L. (2021). A survey on959 49 conversational recommender systems. ACM Computing Surveys (CSUR),54,960 1–36.961 Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir962 techniques. ACM Transactions on Information Systems (TOIS),20, 422–446.963 Jin, Z., Zhang, Y., Mu, W., Wang, W., & Jin, H. (2018). Leveraging the dynamic964 changes from items to improve recommendation. In Conceptual Modeling:965 37th International Conference, ER 2018, Xi’an, China, October 22–25, 2018,966 Proceedings 37 (pp. 507–520). Springer.967 Kluver, D., Nguyen, T. T., Ekstrand, M., Sen, S., & Riedl, J. (2012). How many968 bits per rating? In Proceedings of the sixth ACM conference on Recommender969 systems (pp. 99–106).970 Konstan, J. A., & Riedl, J. (2012). Recommender systems: from algorithms to971 user experience. User Modeling and User-Adapted Interaction,22, 101–123.972 Koren, Y. (2008). Factorization meets the neighborhood: a multifaceted973 collaborative filtering model. In Proceedings of the 14th ACM SIGKDD974 international conference on Knowledge discovery and data mining (pp. 426–975 434).976 Koren, Y. (2010). Collaborative Filtering with Temporal Dynamics. Interacting977 with Computers OF THE ACM,53 , 89–97.978 Koren, Y. (2010). Factor in the neighbors: Scalable and accurate collaborative979 filtering. ACM Transactions on Knowledge Discovery from Data (TKDD),4,980 1–24.981 Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for982 recommender systems. Computer,42, 30–37.983 50 Kumar, S., De, K., & Roy, P. P. (2020). Movie recommendation system984 using sentiment analysis from microblogging data. IEEE Transactions on985 Computational Social Systems,7, 915–923.986 Latrech, J., Kodia, Z., & Ben Azzouna, N. (2023). Codfi-dl: a hybrid987 recommender system combining enhanced collaborative and demographic988 filtering based on deep learning. The Journal of Supercomputing, (pp. 1–23).989 Lemire, D., & Maclachlan, A. (2005). Slope one predictors for online rating-990 based collaborative filtering. In Proceedings of the 2005 SIAM International991 Conference on Data Mining (pp. 471–475). SIAM.992 Li, B., Chen, L., Zhu, X., & Zhang, C. (2013). Noisy but non-malicious user993 detection in social recommender systems. World Wide Web,16, 677–699.994 Li, X., Barajas, J. M., & Ding, Y. (2007). Collaborative filtering on streaming995 data with interest-drifting. Intelligent Data Analysis,11 , 75–87.996 Lops, P., Jannach, D., Musto, C., Bogers, T., & Koolen, M. (2019). Trends in997 content-based recommendation: Preface to the special issue on recommender998 systems based on rich item descriptions. User Modeling and User-Adapted999 Interaction,29, 239–249.1000 Luo, C., Wang, Y., Li, B., Liu, H., Wang, P., & Zhang, L. Y. (2023). An efficient1001 approach to manage natural noises in recommender systems. Algorithms,16,1002 228.1003 Luo, X., Xia, Y., Zhu, Q., & Li, Y. (2013). Boosting the k-nearest-neighborhood1004 based incremental collaborative filtering. Knowledge-Based Systems,53 , 90–1005 99.1006 Luo, X., Zhou, M., Xia, Y., & Zhu, Q. (2014). An efficient non-negative matrix-1007 51 factorization-based approach to collaborative filtering for recommender1008 systems. IEEE Transactions on Industrial Informatics,10, 1273–1284.1009 Martínez, L., Castro, J., & Yera, R. (2016). Managing natural noise1010 in recommender systems. In C. Martín-Vide, T. Mizuki, & M. A.1011 Vega-Rodríguez (Eds.), Theory and Practice of Natural Computing: 5th1012 International Conference, TPNC 2016, Sendai, Japan, December 12-13, 2016,1013 Proceedings (pp. 3–17). Springer International Publishing.1014 Masthoff, J., & Delić, A. (2022). Group recommender systems: Beyond1015 preference aggregation. In Recommender Systems Handbook (pp. 381–420).1016 Springer.1017 Mishra, R., Kumar, P., & Bhasker, B. (2015). A web recommendation system1018 considering sequential information. Decision Support Systems,75, 1–10.1019 Mobasher, B., Burke, R., Bhaumik, R., & Williams, C. (2007). Toward1020 trustworthy recommender systems: An analysis of attack models and1021 algorithm robustness. ACM Transactions on Internet Technology,7.1022 Mohammadpour, T., Bidgoli, A. M., Enayatifar, R., & Haj Seyyed Javadi,1023 H. (2023). Efficient recommendations in collaborative filtering recommender1024 system: A multi-objective evolutionary approach based on nsga-ii algorithm.1025 International Journal of Nonlinear Analysis and Applications,14, 785–804.1026 Ning, X., Desrosiers, C., & Karypis, G. (2015). A comprehensive survey of1027 neighborhood-based recommendation methods. In F. Ricci, L. Rokach, &1028 B. Shapira (Eds.), Recommender Systems Handbook (pp. 37–76). Springer1029 US.1030 O’Mahony, M. P., Hurley, N. J., & Silvestre, G. (2006). Detecting noise1031 52 in recommender system databases. In 11th international conference on1032 Intelligent user interfaces (pp. 109–115). ACM.1033 Park, H., Jeong, J., Oh, K.-W., & Kim, H. (2023). Autoencoder-based1034 recommender system exploiting natural noise removal. IEEE Access,11 ,1035 30609–30618.1036 Pérez-Almaguer, Y., Yera, R., Alzahrani, A. A., & Martínez, L. (2021).1037 Content-based group recommender systems: A general taxonomy and further1038 improvements. Expert Systems with Applications,184, 115444.1039 Pham, H. X., & Jung, J. J. (2013). Preference-based user rating correction1040 process for interactive recommendation systems. Multimedia tools and1041 applications,65, 119–132.1042 Pilászy, I., & Tikk, D. (2009). Recommending new movies: even a few ratings1043 are more valuable than metadata. In Proceedings of the third ACM conference1044 on Recommender systems (pp. 93–100).1045 Quadrana, M., Cremonesi, P., & Jannach, D. (2018). Sequence-aware1046 recommender systems. ACM computing surveys (CSUR),51, 1–36.1047 Rabiu, I., Salim, N., Da’u, A., & Osman, A. (2020a). Recommender system1048 based on temporal models: a systematic review. Applied Sciences,10, 2204.1049 Rabiu, I., Salim, N., Da’u, A., & Osman, A. (2020b). Recommender system1050 based on temporal models: a systematic review. Applied Sciences,10, 2204.1051 Rendle, S. (2022). Item recommendation from implicit feedback. In1052 Recommender Systems Handbook (pp. 143–171). Springer.1053 Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., & Riedl, J. (1994).1054 Grouplens: an open architecture for collaborative filtering of netnews. In1055 53 Proceedings of the 1994 ACM Conference on Computer Supported Cooperative1056 Work (pp. 175–186). New York, USA: ACM.1057 Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications1058 of the ACM,40, 56–58.1059 Ricci, F., Venturini, A., Cavada, D., Mirzadeh, N., Blaas, D., & Nones, M.1060 (2003). Product recommendation with interactive query management and1061 twofold similarity. In International Conference on Case-Based Reasoning (pp.1062 479–493). Springer.1063 Saia, R., Boratto, L., & Carta, S. (2016). A semantic approach to remove1064 incoherent items from a user profile and improve the accuracy of a1065 recommender system. Journal of Intelligent Information Systems,47, 111–1066 134.1067 Said, A., Jain, B. J., Narr, S., & Plumbaum, T. (2012). Users and noise:1068 The magic barrier of recommender systems. In J. Masthoff, B. Mobasher,1069 M. C. Desmarais, & R. Nkambou (Eds.), User Modeling, Adaptation, and1070 Personalization: 20th International Conference, UMAP 2012, Montreal,1071 Canada, July 16-20, 2012. Proceedings (pp. 237–248). Springer Berlin1072 Heidelberg.1073 Salakhutdinov, R., & Mnih, A. (2008). Bayesian probabilistic matrix1074 factorization using markov chain monte carlo. In Proceedings of the 25th1075 international conference on Machine learning (pp. 880–887).1076 Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001). Item-based1077 collaborative filtering recommendation algorithms. In 10th international1078 conference on World Wide Web (pp. 285–295). ACM.1079 Tintarev, N., & Masthoff, J. (2022). Beyond explaining single item1080 54 recommendations. In Recommender Systems Handbook (pp. 711–756).1081 Springer.1082 Tran, D. T., & Huh, J.-H. (2023). New machine learning model based on1083 the time factor for e-commerce recommendation systems. The Journal of1084 Supercomputing,79 , 6756–6801.1085 Van Dat, N., Van Toan, P., & Thanh, T. M. (2022). Solving distribution1086 problems in content-based recommendation system with gaussian mixture1087 model. Applied Intelligence,52, 1602–1614.1088 Vinagre, J., Jorge, A. M., & Gama, J. (2015). An overview on the exploitation of1089 time in collaborative filtering. Wiley interdisciplinary reviews: Data mining1090 and knowledge discovery,5, 195–215.1091 Wang, P., Wang, Y., Zhang, L. Y., & Zhu, H. (2021). An effective and1092 efficient fuzzy approach for managing natural noise in recommender systems.1093 Information Sciences,570, 623–637.1094 Wang, W., & Mishra, K. K. (2017). A novel stock trading prediction and1095 recommendation system. Multimedia Tools and Applications, (pp. 1–13).1096 Wang, Y., Wang, L., Li, Y., He, D., & Liu, T.-Y. (2013). A theoretical analysis1097 of ndcg type ranking measures. In Conference on learning theory (pp. 25–54).1098 PMLR.1099 Wei, J., He, J., Chen, K., Zhou, Y., & Tang, Z. (2017). Collaborative filtering1100 and deep learning based recommendation system for cold start items. Expert1101 Systems with Applications,69, 29–39.1102 Widiyaningtyas, T., Hidayah, I., & Adji, T. B. (2021). User profile correlation-1103 based similarity (upcsim) algorithm in movie recommendation system.1104 Journal of Big Data,8, 1–21.1105 55 Yannam, V. R., Kumar, J., Babu, K. S., & Patra, B. K. (2023). Enhancing1106 the accuracy of group recommendation using slope one. The Journal of1107 Supercomputing,79 , 499–540.1108 Yera, R., Alzahrani, A. A., & Martínez, L. (2022a). Exploring post-hoc agnostic1109 models for explainable cooking recipe recommendations. Knowledge-Based1110 Systems,251 , 109216.1111 Yera, R., Alzahrani, A. A., & Martínez, L. (2022b). A fuzzy content-based group1112 recommender system with dynamic selection of the aggregation functions.1113 International Journal of Approximate Reasoning,150 , 273–296.1114 Yera, R., Alzahrani, A. A., Martínez, L., & Rodríguez, R. M. (2023).1115 A systematic review on food recommender systems for diabetic patients.1116 International Journal of Environmental Research and Public Health,20, 4248.1117 Yera, R., Barranco, M. J., Alzahrani, A. A., & Martínez, L. (2019).1118 Exploring fuzzy rating regularities for managing natural noise in collaborative1119 recommendation. International Journal of Computational Intelligence1120 Systems,12 , 1382–1392.1121 Yera, R., Caballero Mota, Y., & Martínez, L. (2015). Correcting noisy ratings1122 in collaborative recommender systems. Knowledge-Based Systems,76 ,96–1123 108.1124 Yera, R., Castro, J., & Martínez, L. (2016). A fuzzy model for managing natural1125 noise in recommender systems. Applied Soft Computing,40, 187 – 198.1126 Yera, R., & Martínez, L. (2017). Fuzzy tools in recommender systems: A survey.1127 International Journal of Computational Intelligence Systems,10 , 776–803.1128 Yera, R., & Martínez, L. (2017). A recommendation approach for programming1129 56 online judges supported by data preprocessing techniques. Applied1130 Intelligence,47 , 277–290.1131 Yu, L., Han, F., Huang, S., & Luo, Y. (2017). A content-based goods image1132 recommendation system. Multimedia Tools and Applications, (pp. 1–15).1133 Yu, P., Lin, L., & Yao, Y. (2016). A novel framework to process the quantity and1134 quality of user behavior data in recommender systems. In B. Cui, N. Zhang,1135 J. Xu, X. Lian, & D. Liu (Eds.), Web-Age Information Management: 17th1136 International Conference, WAIM 2016, Nanchang, China, June 3-5, 2016,1137 Proceedings, Part I (pp. 231–243). Springer International Publishing.1138 Zhang, S., Tay, Y., Yao, L., Sun, A., & Zhang, C. (2022). Deep learning for1139 recommender systems. In Recommender Systems Handbook (pp. 173–210).1140 Springer.1141 Zhou, K., Wang, H., Zhao, W. X., Zhu, Y., Wang, S., Zhang, F., Wang,1142 Z., & Wen, J.-R. (2020). S3-rec: Self-supervised learning for sequential1143 recommendation with mutual information maximization. In Proceedings of the1144 29th ACM international conference on information & knowledge management1145 (pp. 1893–1902).1146 Zhu, J., Han, L., Gou, Z., & Yuan, X. (2018). A fuzzy clustering-based denoising1147 model for evaluating uncertainty in collaborative filtering recommender1148 systems. Journal of the Association for Information Science and Technology,1149 69, 1109–1121.1150 Appendix A1151 57